Ten thousand meters

I made a website with best learning resources on IT topics

2022-11-11T07:30:00+00:00

Learning new stuff is an integral part of a software engineer's job and a daily routine for any curious person working in IT. We constantly discover and learn new programming languages, frameworks, libraries, tools, concepts, theories, and ideas. We do it to get better at our jobs. And we do it because learning is fun.

Learning a topic becomes fun once you're on a learning path, once you find good books, docs, tutorials, and other resources to follow. But searching for good resources can be frustrating. The problem nowadays is not that good information is scarce. The problem is that there is too much information, and most of it isn't that good or doesn't suit your needs.

Search engines are good at finding some stuff but fail to filter out the best. Try to google "best resources to learn python". You'll get dozens of articles recommending hundreds of resources. Which one should you pick? You may refine your query to something like "intermediate python books", but the problem remains. So you probably type "site:news.ycombinator.com" or "site:reddit.com", read reviews, and eventually do a full research on the subject.

It would be cool if someone did this research for your and saved your a lot of time next time you learn something. They would learn about the topic in great detail, go through all the available resources, pick the best, and share the links, supplying each resource with a brief description of its contents and its target audience so you can quickly tell if it's for you.

I thought I could be that person since I often learn new stuff and do this research anyway. So I created bestresourcestolearnx.com – a curated collection of best learning resources on IT topics.

The project is in the very early stage. Right now I cover topics that I know or that I'm learning about. If you want to contribute, please go to this repo. I welcome issues with resource recommendations and any other suggestions. Just note that the content is opinionated, so I cannot guarantee everything gets published.

More content is on the way. People, keep on learning!

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #13: the GIL and its effects on Python multithreading

2021-09-22T08:55:00+00:00

As you probably know, the GIL stands for the Global Interpreter Lock, and its job is to make the CPython interpreter thread-safe. The GIL allows only one OS thread to execute Python bytecode at any given time, and the consequence of this is that it's not possible to speed up CPU-intensive Python code by distributing the work among multiple threads. This is, however, not the only negative effect of the GIL. The GIL introduces overhead that makes multi-threaded programs slower, and what is more surprising, it can even have an impact on I/O-bound threads.

In this post I'd like to tell you more about non-obvious effects of the GIL. Along the way, we'll discuss what the GIL really is, why it exists, how it works, and how it's going to affect Python concurrency in the future.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

OS threads, Python threads, and the GIL

Let me first remind you what Python threads are and how multithreading works in Python. When you run the python executable, the OS starts a new process with one thread of execution called the main thread. As in the case of any other C program, the main thread begins executing python by entering its main() function. All the main thread does next can be summarized by three steps:

The main thread is a regular OS thread that executes compiled C code. Its state includes values of CPU registers and the call stack of C functions. A Python thread, however, must capture the call stack of Python functions, the exception state, and other Python-related things. So what CPython does is put those things in a thread state struct and associate the thread state with the OS thread. In other words, Python thread = OS thread + Python thread state.

The evaluation loop is an infinite loop that contains a giant switch over all possible bytecode instructions. To enter the loop, a thread must hold the GIL. The main thread takes the GIL during the initialization, so it's free to enter. When it enters the loop, it just starts executing bytecode instructions one by one according to the switch.

From time to time, a thread has to suspend bytecode execution. It checks if there are any reasons to do that at the beginning of each iteration of the evaluation loop. We're interested in one such reason: another thread has requested the GIL. Here's how this logic is implemented in the code:

PyObject*
_PyEval_EvalFrameDefault(PyThreadState *tstate, PyFrameObject *f, int throwflag)
{
    // ... declaration of local variables and other boring stuff

    // the evaluation loop
    for (;;) {

        // `eval_breaker` tells whether we should suspend bytecode execution
        // e.g. other thread requested the GIL
        if (_Py_atomic_load_relaxed(eval_breaker)) {

            // `eval_frame_handle_pending()` suspends bytecode execution
            // e.g. when another thread requests the GIL,
            // this function drops the GIL and waits for the GIL again
            if (eval_frame_handle_pending(tstate) != 0) {
                goto error;
            }
        }

        // get next bytecode instruction
        NEXTOPARG();

        switch (opcode) {
            case TARGET(NOP) {
                FAST_DISPATCH(); // next iteration
            }

            case TARGET(LOAD_FAST) {
                // ... code for loading local variable
                FAST_DISPATCH(); // next iteration
            }

            // ... 117 more cases for every possible opcode
        }

        // ... error handling
    }

    // ... termination
}

In a single-threaded Python program, the main thread is the only thread, and it never releases the GIL. Let's now see what happens in a multi-threaded program. We use the threading standard module to start a new Python thread:

import threading

def f(a, b, c):
    # do something
    pass

t = threading.Thread(target=f, args=(1, 2), kwargs={'c': 3})
t.start()

The start() method of a Thread instance creates a new OS thread. On Unix-like systems including Linux and macOS, it calls the pthread_create() function for that purpose. The newly created thread starts executing the t_bootstrap() function with the boot argument. The boot argument is a struct that contains the target function, the passed arguments, and a thread state for the new OS thread. The t_bootstrap() function does a number of things, but most importantly, it acquires the GIL and then enters the evaluation loop to execute the bytecode of the target function.

To acquire the GIL, a thread first checks whether some other thread holds the GIL. If this is not the case, the thread acquires the GIL immediately. Otherwise, it waits until the GIL is released. It waits for a fixed time interval called the switch interval (5 ms by default), and if the GIL is not released during that time, it sets the eval_breaker and gil_drop_request flags. The eval_breaker flag tells the GIL-holding thread to suspend bytecode execution, and gil_drop_request explains why. The GIL-holding thread sees the flags when it starts the next iteration of the evaluation loop and releases the GIL. It notifies the GIL-awaiting threads, and one of them acquires the GIL. It's up to the OS to decide which thread to wake up, so it may or may not be the thread that set the flags.

That's the bare minimum of what we need to know about the GIL. Let me now demonstrate its effects that I was talking about earlier. If you find them interesting, proceed with the next sections in which we study the GIL in more detail.

The effects of the GIL

The first effect of the GIL is well-known: multiple Python threads cannot run in parallel. Thus, a multi-threaded program is not faster than its single-threaded equivalent even on a multi-core machine. As an naive attempt to parallelize Python code, consider the following CPU-bound function that performs the decrement operation a given number of times:

def countdown(n):
    while n > 0:
        n -= 1

Now suppose we want to perform 100,000,000 decrements. We may run countdown(100_000_000) in a single thread, or countdown(50_000_000) in two threads, or countdown(25_000_000) in four threads, and so forth. In the language without the GIL like C, we would see a speedup as we increase the number of threads. Running Python on my MacBook Pro with two cores and hyper-threading, I see the following:

Number of threads	Decrements per thread (n)	Time in seconds (best of 3)
1	100,000,000	6.52
2	50,000,000	6.57
4	25,000,000	6.59
8	12,500,000	6.58

The times don't change. In fact, multi-threaded programs may run slower because of the overhead associated with context switching. The default switch interval is 5 ms, so context switches do not happens that often. But if we decrease the switch interval, we will see a slowdown. More on why we might need to do that later.

Although Python threads cannot help us speed up CPU-intensive code, they are useful when we want to perform multiple I/O-bound tasks simultaneously. Consider a server that listens for incoming connections and, when it receives a connection, runs a handler function in a separate thread. The handler function talks to the client by reading from and writing to the client's socket. When reading from the socket, the thread just hangs until the client sends something. This is where multithreading helps: another thread can run in the meantime.

To allow other threads run while the GIL-holding thread is waiting for I/O, CPython implements all I/O operations using the following pattern:

release the GIL;
perform the operation, e.g. write(), recv(), accept();
acquire the GIL.

Thus, a thread may release the GIL voluntarily before another thread sets eval_breaker and gil_drop_request. In general, a thread needs to hold the GIL only while it works with Python objects. So CPython applies the release-perform-acquire pattern not just to I/O operations but also to other blocking calls into the OS like select() and pthread_mutex_lock(), and to heavy computations in pure C. For example, hash functions in the hashlib standard module release the GIL. This allows us to actually speed up Python code that calls such functions using multithreading.

Suppose we want to compute SHA-256 hashes of eight 128 MB messages. We may compute hashlib.sha256(message) for each message in a single thread, but we may also distribute the work among multiple threads. If I do the comparison on my machine, I get the following results:

Number of threads	Total size of messages per thread	Time in seconds (best of 3)
1	1 GB	3.30
2	512 MB	1.68
4	256 MB	1.50
8	128 MB	1.60

Going from one thread to two threads is almost a 2x speedup because the threads run in parallel. Adding more threads doesn't help much because my machine has only two physical cores. The conclusion here is that it's possible to speed up CPU-intensive Python code using multithreading if the code calls C functions that release the GIL. Note that such functions can be found not only in the standard library but also in computational-heavy third-party modules like NumPy. You can even write a C extension that releases the GIL yourself.

We've mentioned CPU-bound threads – threads that compute something most of the time, and I/O-bound threads – threads that wait for I/O most of the time. The most interesting effect of the GIL takes place when we mix the two. Consider a simple TCP echo server that listens for incoming connections and, when a client connects, spawns a new thread to handle the client:

from threading import Thread
import socket


def run_server(host='127.0.0.1', port=33333):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    while True:
        client_sock, addr = sock.accept()
        print('Connection from', addr)
        Thread(target=handle_client, args=(client_sock,)).start()


def handle_client(sock):
    while True:
        received_data = sock.recv(4096)
        if not received_data:
            break
        sock.sendall(received_data)

    print('Client disconnected:', sock.getpeername())
    sock.close()


if __name__ == '__main__':
    run_server()

How many requests per second can this sever handle? I wrote a simple client program that just sends and receives 1-byte messages to the server as fast as it can and got something about 30k RPS. This is most probably not an accurate measure since the client and the server run on the same machine, but that's not the point. The point is to see how the RPS drops when the server performs some CPU-bound task in a separate thread.

Consider the exact same server but with an additional dummy thread that increments and decrements a variable in an infinite loop (any CPU-bound task will do just the same):

# ... the same server code

def compute():
    n = 0
    while True:
        n += 1
        n -= 1

if __name__ == '__main__':
    Thread(target=compute).start()
    run_server()

How do you expect the RPS to change? Slightly? 2x less? 10x less? No. The RPS drops to 100, which is 300x less! And this is very surprising if you are used to the way operating systems schedule threads. To see what I mean, let's run the server and the CPU-bound thread as separate processes so that they are not affected by the GIL. We can split the code into two different files or just use the multiprocessing standard module to spawn a new process like so:

from multiprocessing import Process

# ... the same server code

if __name__ == '__main__':
    Process(target=compute).start()
    run_server()

And this yields about 20k RPS. Moreover, if we start two, three, or four CPU-bound processes, the RPS stays about the same. The OS scheduler prioritizes the I/O-bound thread, which is the right thing to do.

In the server example the I/O-bound thread waits for the socket to become ready for reading and writing, but the performance of any other I/O-bound thread would degrade just the same. Consider a UI thread that waits for user input. It would freeze regularly if you run it alongside a CPU-bound thread. Clearly, this is not how normal OS threads work, and the cause is the GIL. It interferes with the OS scheduler.

This problem is actually well-known among CPython developers. They refer to it as the convoy effect. David Beazley gave a talk about it in 2010 and also opened a related issue on bugs.python.org. In 2021, 11 years laters, the issue was closed. However, it hasn't been fixed. In the rest of this post we'll try to figure out why.

The convoy effect

The convoy effect takes place because each time the I/O-bound thread performs an I/O operation, it releases the GIL, and when it tries to reacquire the GIL after the operation, the GIL is likely to be already taken by the CPU-bound thread. So the I/O-bound thread must wait for at least 5 ms before it can set eval_breaker and gil_drop_request to force the CPU-bound thread release the GIL.

The OS can schedule the CPU-bound thread as soon as the I/O-bound thread releases the GIL. The I/O-bound thread can be scheduled only when the I/O operation completes, so it has less chances to take the GIL first. If the operation is really fast such as a non-blocking send(), the chances are actually quite good but only on a single-core machine where the OS has to decide which thread to schedule.

On a multi-core machine, the OS doesn't have to decide which of the two threads to schedule. It can schedule both on different cores. The result is that the CPU-bound thread is almost guaranteed to acquire the GIL first, and each I/O operation in the I/O-bound thread costs extra 5 ms.

Note that a thread that is forced to release the GIL waits until another thread takes it, so the I/O-bound thread acquires the GIL after one switch interval. Without this logic, the convoy effect would be even more severe.

Now, how much is 5 ms? It depends on how much time the I/O operations take. If a thread waits for seconds until the data on a socket becomes available for reading, extra 5 ms do not matter much. But some I/O operations are really fast. For example, send() blocks only when the send buffer is full and returns immediately otherwise. So if the I/O operations take microseconds, then milliseconds of waiting for the GIL may have a huge impact.

The echo server without the CPU-bound thread handles 30k RPS, which means that a single request takes about 1/30k ≈ 30 µs. With the CPU-bound thread, recv() and send() add extra 5 ms = 5,000 µs to every request each, and a single request now takes 10,030 µs. This is about 300x more. Thus, the throughput is 300x less. The numbers match.

You may ask: Is the convoy effect a problem in real-world applications? I don't know. I never ran into it, nor could I find evidence that anyone else did. People do not complain, and this is part of the reason why the issue hasn't been fixed.

But what if the convoy effect does cause performance problems in your application? Here are two ways to fix it.

Fixing the convoy effect

Since the problem is that the I/O-bound thread waits for the switch interval until it requests the GIL, we may try to set the switch interval to a smaller value. Python provides the sys.setswitchinterval(interval) function for that purpose. The interval argument is a floating-point value representing seconds. The switch interval is measured in microseconds, so the smallest value is 0.000001. Here's the RPS I get if I vary the switch interval and the number of CPU threads:

Switch interval in seconds	RPS with no CPU threads	RPS with one CPU thread	RPS with two CPU threads	RPS with four CPU threads
0.1	30,000	5	2	0
0.01	30,000	50	30	15
0.005	30,000	100	50	30
0.001	30,000	500	280	200
0.0001	30,000	3,200	1,700	1000
0.00001	30,000	11,000	5,500	2,800
0.000001	30,000	10,000	4,500	2,500

The results show several things:

The switch interval is irrelevant if the I/O-bound thread is the only thread.
As we add one CPU-bound thread, the RPS drops significantly.
As we double the number of CPU-bound threads, the RPS halves.
As we decrease the switch interval, the RPS increases almost proportionally until the switch interval becomes too small. This is because the cost of context switching becomes significant.

Smaller switch intervals make I/O-bound threads more responsive. But too small switch intervals introduce a lot of overhead caused by a high number of context switches. Recall the countdown() function. We saw that we cannot speed it up with multiple threads. If we set the switch interval too small, then we'll also see a slowdown:

Switch interval in seconds	Time in seconds (threads: 1)	Time in seconds (threads: 2)	Time in seconds (threads: 4)	Time in seconds (threads: 8)
0.1	7.29	6.80	6.50	6.61
0.01	6.62	6.61	7.15	6.71
0.005	6.53	6.58	7.20	7.19
0.001	7.02	7.36	7.56	7.12
0.0001	6.77	9.20	9.36	9.84
0.00001	6.68	12.29	19.15	30.53
0.000001	6.89	17.16	31.68	86.44

Again, the switch interval doesn't matter if there is only one thread. Also, the number of threads doesn't matter if the switch interval is large enough. A small switch interval and several threads is when you get poor performance.

The conclusion is that changing the switch interval is an option for fixing the convoy effect, but you should be careful to measure how the change affects your application.

The second way to fix the convoy effect is even more hacky. Since the problem is much less severe on single-core machines, we could try to restrict all Python threads to a single-core. This would force the OS to choose which thread to schedule, and the I/O-bound thread would have the priority.

Not every OS provides a way to restrict a group of threads to certain cores. As far as I understand, macOS provides only a mechanism to give hints to the OS scheduler. The mechanism that we need is available on Linux. It's the pthread_setaffinity_np() function. It takes a thread and a mask of CPU cores and tells the OS to schedule the thread only on the cores specified by the mask.

pthread_setaffinity_np() is a C function. To call it from Python, you may use something like ctypes. I didn't want to mess with ctypes, so I just modified the CPython source code. Then I compiled the executable, ran the echo server on a dual core Ubuntu machine and got the following results:

Number of CPU-bound threads	0	1	2	4	8
RPS	24k	12k	3k	30	10

The server can tolerate one CPU-bound thread quite well. But since the I/O-bound thread needs to compete with all CPU-bound threads for the GIL, as we add more threads, the performance drops massively. The fix is more of a hack. Why don't CPython developers just implement a proper GIL?

Update from October 7, 2021: I've now learned that restricting threads to one core helps with the convoy effect only when the client is restricted to the same core, which is how I set up the benchmark. See the notes for details.

A proper GIL

The fundamental problem with the GIL is that it interferes with the OS scheduler. Ideally, you would like to run an I/O-bound thread as soon the I/O operation it waits for completes. And that's what the OS scheduler usually does. In CPython, however, the thread then immediately gets stuck waiting for the GIL, so the OS scheduler's decision doesn't really mean anything. You may try to get rid of the switch interval so that a thread that wants the GIL gets it without delay, but then you have a problem with CPU-bound threads because they want the GIL all the time.

The proper solution is to differentiate between the threads. An I/O-bound thread should be able to take away the GIL from a CPU-bound thread without waiting, but threads with the same priority should wait for each other. The OS scheduler already differentiates between the threads, but you cannot rely on it because it knows nothing about the GIL. It seems that the only option is to implement the scheduling logic in the interpreter.

After David Beazley opened the issue, CPython developers made several attempts to solve it. Beazley himself proposed a simple patch. In short, this patch allows an I/O-bound thread to preempt a CPU-bound thread. By default, all threads are considered I/O-bound. Once a thread is forced to release the GIL, it's flagged as CPU-bound. When a thread releases the GIL voluntarily, the flag is reset, and the thread is considered I/O-bound again.

Beazley's patch solved all the GIL problems that we've discussed today. Why hasn't it been merged? The consensus seems to be that any simple implementation of the GIL would fail in some pathological cases. At most, you might need to try a bit harder to find them. A proper solution has to do scheduling like an OS, or as Nir Aides put it:

... Python really needs a scheduler, not a lock.

So Aides implemented a full-fledged scheduler in his patch. The patch worked, but a scheduler is never a trivial thing, so merging it to CPython required a lot of effort. Finally, the work was abandoned because at the time there wasn't enough evidence that the issue caused problems in production code. See the discussion for more details.

The GIL never had a huge fanbase. What we've seen today only makes it worse. We come back to the all time question.

Can't we remove the GIL?

The first step to remove the GIL is to understand why it exists. Think why you would typically use locks in a multi-threaded program, and you'll get the answer. It's to prevent race conditions and make certain operations atomic from the perspective of other threads. Say you have a sequence of statements that modifies some data structure. If you don't surround the sequence with a lock, then another thread can access the data structure somewhere in the middle of the modification and get a broken incomplete view.

Or say you increment the same variable from multiple threads. If the increment operation is not atomic and not protected by a lock, then the final value of the variable can be less than the the total number of increments. This is a typical data race:

Thread 1 reads the value x.
Thread 2 reads the value x.
Thread 1 writes back the value x + 1.
Thread 2 writes back the value x + 1, thus discarding the changes made by Thread 1.

In Python the += operation is not atomic because it consists of multiple bytecode instructions. To see how it can lead to data races, set the switch interval to 0.000001 and run the following function in multiple threads:

sum = 0

def f():
    global sum
    for _ in range(1000):
        sum += 1

Similarly, in C incrementing an integer like x++ or ++x is not atomic because the compiler translates such operations to a sequence of machine instructions. Threads can interleave in between.

The GIL is so helpful because CPython increments and decrements integers that can be shared between threads all over the place. This is CPython's way to do garbage collection. Every Python object has a reference count field. This field counts the number of places that reference the object: other Python objects, local and global C variables. One place more increments the reference count. One place less decrements it. When the reference count reaches zero, the object is deallocated. If not the GIL, some decrements could overwrite each other and the object would stay in memory forever. Worse still, overwritten increments could result in a deallocated object that has active references.

The GIL also simplifies the implementation of built-in mutable data structures. Lists, dicts and sets do not use locking internally, yet because of the GIL, they can be safely used in multi-threaded programs. Similarly, the GIL allows threads to safely access global and interpreter-wide data: loaded modules, preallocated objects, interned strings as so on.

Finally, the GIL simplifies the writing of C extensions. Developers can assume that only one thread runs their C extension at any given time. Thus, they don't need to use additional locking to make the code thread-safe. When they do want to run the code in parallel, they can release the GIL.

To sum up, what the GIL does is make the following thread-safe:

reference counting;
mutable data structures;
global and interpreter-wide data;
C extensions.

To remove the GIL and still have a working interpreter, you need to find alternative mechanisms for thread safety. People tried to do that in the past. The most notable attempt was Larry Hastings' Gilectomy project started in 2016. Hastings forked CPython, removed the GIL, modified reference counting to use atomic increments and decrements, and put a lot of fine-grained locks to protect mutable data structures and interpreter-wide data.

Gilectomy could run some Python code and run it in parallel. However, the single-threaded performance of CPython was compromised. Atomic increments and decrements alone added about 30% overhead. Hastings tried to address this by implementing buffered reference counting. In short, this technique confines all reference count updates to one special thread. Other threads only commit the increments and decrements to the log, and the special thread reads the log. This worked, but the overhead was still significant.

In the end, it became evident that Gilectomy is not going to be merged into CPython. Hastings stopped working on the project. It wasn't a complete failure, though. It taught us why removing the GIL from CPython is hard. There are two main reasons:

Garbage collection based on reference counting is not suited for multithreading. The only solution is to implement a tracing garbage collector that JVM, CLR, Go, and other runtimes without a GIL implement.
Removing the GIL breaks existing C extensions. There is no way around it.

Nowadays nobody thinks seriously about removing the GIL. Does it mean that we are to live with the GIL forever?

The future of the GIL and Python concurrency

This sounds scary, but it's much more probable that CPython will have many GILs than no GIL at all. Literally, there is an initiative to introduce multiple GILs to CPython. It's called subinterpreters. The idea is to have multiple interpreters within the same process. Threads within one interpreter still share the GIL, but multiple interpreters can run parallel. No GIL is needed to synchronize interpreters because they have no common global state and do not share Python objects. All global state is made per-interpreter, and interpreters communicate via message passing only. The ultimate goal is to introduce to Python a concurrency model based on communicating sequential processes found in languages like Go and Clojure.

Interpreters have been a part of CPython since version 1.5 but only as an isolation mechanism. They store data specific to a group of threads: loaded modules, builtins, import settings and so forth. They are not exposed in Python, but C extensions can use them via the Python/C API. A few actually do that, though, mod_wsgi being a notable example.

Today's interpreters are limited by the fact that they have to share the GIL. This can change only when all the global state is made per-interpreter. The work is being done in that direction, but few things remain global: some built-in types, singletons like None, True and False, and parts of the memory allocator. C extensions also need to get rid of the global state before they can work with subinterpreters.

Eric Snow wrote PEP 554 that adds the interpreters module to the standard library. The idea is to expose the existing interpreters C API to Python and provide mechanisms of communication between interpreters. The proposal targeted Python 3.9 but was postponed until the GIL is made per-interpreter. Even then it's not guaranteed to succeed. The matter of debate is whether Python really needs another concurrency model.

Another exciting project that's going on nowadays is Faster CPython. In October 2020, Mark Shannon proposed a plan to make CPython ≈5x faster over several years. And it's actually much more realistic than it may sound because CPython has a lot of potential for optimization. The addition of JIT alone can result in an enormous performance boost.

There were similar projects before, but they failed because they lacked proper funding or expertise. This time, Microsoft volunteered to sponsor Faster CPython and let Mark Shannon, Guido van Rossum, and Eric Snow work on the project. The incremental changes already go to CPython – they do not stale in a fork.

Faster CPython focuses on single-threaded performance. The team has no plans to change or remove the GIL. Nevertheless, if the project succeeds, one of the Python's major pain points will be fixed, and the GIL question may become more relevant than ever.

P.S.

The benchmarks used in this post are available on GitHub. Special thanks to David Beazley for his amazing talks. Larry Hastings' talks on the GIL and Gilectomy (one, two, three) were also very interesting to watch. To understand how modern OS schedulers work, I've read Robert Love's book Linux Kernel Development. Highly recommend it!

If you want to study the GIL in more detail, you should read the source code. The Python/ceval_gil.h file is a perfect place to start. To help you with this venture, I wrote the following bonus section.

The implementation details of the GIL *

Technically, the GIL is a flag indicating whether the GIL is locked or not, a set of mutexes and conditional variables that control how this flag is set, and some other utility variables like the switch interval. All these things are stored in the _gil_runtime_state struct:

struct _gil_runtime_state {
    /* microseconds (the Python API uses seconds, though) */
    unsigned long interval;
    /* Last PyThreadState holding / having held the GIL. This helps us
       know whether anyone else was scheduled after we dropped the GIL. */
    _Py_atomic_address last_holder;
    /* Whether the GIL is already taken (-1 if uninitialized). This is
       atomic because it can be read without any lock taken in ceval.c. */
    _Py_atomic_int locked;
    /* Number of GIL switches since the beginning. */
    unsigned long switch_number;
    /* This condition variable allows one or several threads to wait
       until the GIL is released. In addition, the mutex also protects
       the above variables. */
    PyCOND_T cond;
    PyMUTEX_T mutex;
#ifdef FORCE_SWITCHING
    /* This condition variable helps the GIL-releasing thread wait for
       a GIL-awaiting thread to be scheduled and take the GIL. */
    PyCOND_T switch_cond;
    PyMUTEX_T switch_mutex;
#endif
};

The _gil_runtime_state stuct is a part of the global state. It's stored in the _ceval_runtime_state struct, which in turn is a part of _PyRuntimeState that all Python threads have an access to:

struct _ceval_runtime_state {
    _Py_atomic_int signals_pending;
    struct _gil_runtime_state gil;
};

typedef struct pyruntimestate {
    // ...
    struct _ceval_runtime_state ceval;
    struct _gilstate_runtime_state gilstate;

    // ...
} _PyRuntimeState;

Note that _gilstate_runtime_state is a struct different from _gil_runtime_state. It stores information about the GIL-holding thread:

struct _gilstate_runtime_state {
    /* bpo-26558: Flag to disable PyGILState_Check().
       If set to non-zero, PyGILState_Check() always return 1. */
    int check_enabled;
    /* Assuming the current thread holds the GIL, this is the
       PyThreadState for the current thread. */
    _Py_atomic_address tstate_current;
    /* The single PyInterpreterState used by this process'
       GILState implementation
    */
    /* TODO: Given interp_main, it may be possible to kill this ref */
    PyInterpreterState *autoInterpreterState;
    Py_tss_t autoTSSkey;
};

Finally, there is a _ceval_state struct, which is a part of PyInterpreterState. It stores the eval_breaker and gil_drop_request flags:

struct _ceval_state {
    int recursion_limit;
    int tracing_possible;
    /* This single variable consolidates all requests to break out of
       the fast path in the eval loop. */
    _Py_atomic_int eval_breaker;
    /* Request for dropping the GIL */
    _Py_atomic_int gil_drop_request;
    struct _pending_calls pending;
};

The Python/C API provides the PyEval_RestoreThread() and PyEval_SaveThread() functions to acquire and release the GIL. These function also take care of setting gilstate->tstate_current. Under the hood, all the job is done by the take_gil() and drop_gil() functions. They are called by the GIL-holding thread when it suspends bytecode execution:

/* Handle signals, pending calls, GIL drop request
   and asynchronous exception */
static int
eval_frame_handle_pending(PyThreadState *tstate)
{
    _PyRuntimeState * const runtime = &_PyRuntime;
    struct _ceval_runtime_state *ceval = &runtime->ceval;

    /* Pending signals */
    // ...

    /* Pending calls */
    struct _ceval_state *ceval2 = &tstate->interp->ceval;
    // ...

    /* GIL drop request */
    if (_Py_atomic_load_relaxed(&ceval2->gil_drop_request)) {
        /* Give another thread a chance */
        if (_PyThreadState_Swap(&runtime->gilstate, NULL) != tstate) {
            Py_FatalError("tstate mix-up");
        }
        drop_gil(ceval, ceval2, tstate);

        /* Other threads may run now */

        take_gil(tstate);

        if (_PyThreadState_Swap(&runtime->gilstate, tstate) != NULL) {
            Py_FatalError("orphan tstate");
        }
    }

    /* Check for asynchronous exception. */
    // ...
}

On Unix-like systems the implementation of the GIL relies on primitives provided by the pthreads library. These include mutexes and conditional variables. In short, they work as follows. A thread calls pthread_mutex_lock(mutex) to lock the mutex. When another thread does the same, it blocks. The OS puts it on the queue of threads that wait for the mutex and wakes it up when the first thread calls pthread_mutex_unlock(mutex). Only one thread can run the protected code at a time.

Conditional variables allow one thread to wait until another thread makes some condition true. To wait on a conditional variable a thread locks a mutex and calls pthread_cond_wait(cond, mutex) or pthread_cond_timedwait(cond, mutex, time). These calls atomically unlock the mutex and make the thread block. The OS puts the thread on a waiting queue and wakes it up when another thread calls pthread_cond_signal(). The awakened thread locks the mutex again and proceeds. Here's how conditional variables are typically used:

# awaiting thread

mutex.lock()
while not condition:
    cond_wait(cond_variable, mutex)
# ... condition is True, do something
mutex.unlock()

# signaling thread

mutex.lock()
# ... do something and make condition True
cond_signal(cond_variable)
mutex.unlock()

Note that the awaiting thread should check the condition in a loop because it's not guaranteed to be true after the notification. The mutex ensures that the awaiting thread doesn't miss the condition going from false to true.

The take_gil() and drop_gil() functions use the gil->cond conditional variable to notify GIL-awaiting threads that the GIL has been released and gil->switch_cond to notify the GIL-holding thread that other thread took the GIL. These conditional variables are protected by two mutexes: gil->mutex and gil->switch_mutex.

Here's the steps of take_gil() :

Lock the GIL mutex: pthread_mutex_lock(&gil->mutex).
See if gil->locked. If it's not, go to step 4.
Wait for the GIL. While gil->locked:
1. Remember gil->switch_number.
2. Wait for the GIL-holding thread to drop the GIL: pthread_cond_timedwait(&gil->cond, &gil->mutex, switch_interval).
3. If timed out, and gil->locked, and gil->switch_number didn't change, tell the GIL-holding thread to drop the GIL: set ceval->gil_drop_request and ceval->eval_breaker.
Take the GIL and notify the GIL-holding thread that we took it:
1. Lock the switch mutex: pthread_mutex_lock(&gil->switch_mutex).
2. Set gil->locked.
3. If we're not the gil->last_holder thread, update gil->last_holder and increment gil->switch_number.
4. Notify the GIL-releasing thread that we took the GIL: pthread_cond_signal(&gil->switch_cond).
5. Unlock the switch mutex: pthread_mutex_unlock(&gil->switch_mutex).
Reset ceval->gil_drop_request.
Recompute ceval->eval_breaker.
Unlock the GIL mutex: pthread_mutex_unlock(&gil->mutex).

Note that while a thread waits for the GIL, another thread can took it, so it's necessary to check gil->switch_number to ensure that a thread that just took the GIL won't be forced to drop it.

Finally, here's the steps of drop_gil() :

Lock the GIL mutex: pthread_mutex_lock(&gil->mutex).
Reset gil->locked.
Notify the GIL-awaiting threads that we drop the GIL: pthread_cond_signal(&gil->cond).
Unlock the GIL mutex: pthread_mutex_unlock(&gil->mutex).
If ceval->gil_drop_request, wait for another thread to take the GIL:
1. Lock the switch mutex: pthread_mutex_lock(&gil->switch_mutex).
2. If we're still gil->last_holder, wait: pthread_cond_wait(&gil->switch_cond, &gil->switch_mutex).
3. Unlock the switch mutex: pthread_mutex_unlock(&gil->switch_mutex).

Note that the GIL-releasing thread doesn't need to wait for a condition in a loop. It calls pthread_cond_wait(&gil->switch_cond, &gil->switch_mutex) only to ensure that it doesn't reacquire the GIL immediately. If the switch occurred, this means that another thread took the GIL, and it's fine to compete for the GIL again.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Update from October 7, 2021: [1] Restricting threads to one core doesn't really fix the convoy effect. Yes, it forces the OS to choose which of the two threads to schedule, which gives the I/O-bound thread a good chance to reacquire the GIL on an I/O-operation, but if the I/O-operation is blocking, it doesn't help. In this case the I/O-bound thread is not ready for scheduling, so the OS schedules the CPU-bound thread.

In the echo server example, effectively every recv() is blocking – the server waits for the client to read the response and send the next message. Restricting threads to one core should not help. But we saw the RPS improved. Why is that? It's because the benchmark is flawed. I ran the client on the same machine and on the same core as the server threads. Such a setup forces the OS to choose between the server's CPU-bound thread and the client thread when the server's I/O-bound thread blocks on recv(). The client thread is more likely to be scheduled. It sends the next message and blocks on recv() too. But now the server's I/O-bound thread is ready and competes with the CPU-bound thread. Basically, running client on same core core, makes the OS to choose between the I/O-bound thread and the CPU-bound thread even on blocking recv().

Also, you don't need to modify the CPython source code or mess with ctypes to restrict Python threads to certain cores. In Linux, the pthread_setaffinity_np() function is implemented on top of the sched_setaffinity() syscall, and the os standard module exposes this syscall to Python. Thanks to Carl Bordum Hansen for pointing this out to me.

There is also the taskset command that allows you to set the CPU affinity of a process without touching the source code at all. Just run the program like this:

$ taskset -c {cpu_list} python program.py

Update from October 16, 2021: Sam Gross recently announced his CPython fork that removes the GIL. You can think of this project as Gilectomy 2.0: it replaces the GIL with alternative mechanisms for thread safety but, unlike Gilectomy, doesn't make single-threaded code much slower. In fact, Gross optimized the interpreter so that the single-threaded performance of the fork without the GIL became even faster than mainline CPython 3.9.

This project looks like the most promising attempt to remove the GIL from CPython. I'm sure some ideas proposed by Gross will find their way upstream. To learn more about the project and the ideas behind it, see the design document and the GitHub repo. These is also a nice writeup on LWN.

Python behind the scenes #12: how async/await works in Python

2021-08-24T03:30:00+00:00

Mark functions as async. Call them with await. All of a sudden, your program becomes asynchronous – it can do useful things while it waits for other things, such as I/O operations, to complete.

Code written in the async/await style looks like regular synchronous code but works very differently. To understand how it works, one should be familiar with many non-trivial concepts including concurrency, parallelism, event loops, I/O multiplexing, asynchrony, cooperative multitasking and coroutines. Python's implementation of async/await adds even more concepts to this list: generators, generator-based coroutines, native coroutines, yield and yield from. Because of this complexity, many Python programmers that use async/await do not realize how it actually works. I believe that it should not be the case. The async/await pattern can be explained in a simple manner if you start from the ground up. And that's what we're going to do today.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

It's all about concurrency

Computers execute programs sequentially – one instruction after another. But a typical program performs multiple tasks, and it doesn't always make sense to wait for some task to complete before starting the next one. For example, a chess program that waits for a player to make a move should be able to update the clock in the meantime. Such an ability of a program to deal with multiple things simultaneously is what we call concurrency. Concurrency doesn't mean that multiple tasks must run at the same physical time. They can run in an interleaved manner: a task runs for some time, then suspends and lets other tasks run, hoping it will get more time in the future. By this mechanism, an OS can run thousands of processes on a machine that has only a few cores. If multiple tasks do run at the same physical time, as in the case of a multi-core machine or a cluster, then we have parallelism, a special case of concurrency [1].

It's crucial to realize that you can write concurrent programs without any special support from the language. Suppose you write a program that performs two tasks, each task being represented by a separate function:

def do_task1():
    # ...

def do_task2():
    # ...

def main():
    do_task1()
    do_task2()

If the tasks are independent, then you can make the program concurrent by decomposing each function into several functions and call the decomposed functions in an interleaved manner, like so:

def do_task1_part1():
    # ...

def do_task1_part2():
    # ...

def do_task2_part1():
    # ...

def do_task2_part2():
    # ...

def main():
    do_task1_part1()
    do_task2_part1()
    do_task1_part2()
    do_task2_part2()

Of course, this is an oversimplified example. The point here is that the language doesn't determine whether you can write concurrent programs or not but may provide features that make concurrent programming more convenient. As we'll learn today, async/await is just such a feature.

To see how one goes from concurrency to async/await, we'll write a real-world concurrent program – a TCP echo server that supposed to handle multiple clients simultaneously. We'll start with the simplest, sequential version of the server that is not concurrent. Then we'll make it concurrent using OS threads. After that, we'll see how we can write a concurrent version that runs in a single thread using I/O multiplexing and an event loop. From this point onwards, we'll develop the single-threaded approach by introducing generators, coroutines and, finally, async/await.

A sequential server

Writing a TCP echo server that handles only one client at a time is straightforward. The server listens for incoming connections on some port, and when a client connects, the server talks to the client until the connection is closed. Then it continues to listen for new connections. This logic can be implemented using basic socket programming:

# echo_01_seq.py

import socket


def run_server(host='127.0.0.1', port=55555):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    while True:
        client_sock, addr = sock.accept()
        print('Connection from', addr)
        handle_client(client_sock)


def handle_client(sock):
    while True:
        received_data = sock.recv(4096)
        if not received_data:
            break
        sock.sendall(received_data)

    print('Client disconnected:', sock.getpeername())
    sock.close()


if __name__ == '__main__':
    run_server()

Take time to study this code. We'll be using it as a framework for subsequent, concurrent versions of the server. If you need a reminder on sockets, check out Beej's Guide to Network Programming and the docs on the socket module. What we do here in a nutshell is:

create a new TCP/IP socket with socket.socket()
bind the socket to an address and a port with sock.bind()
mark the socket as a "listening" socket with sock.listen()
accept new connections with sock.accept()
read data from the client with sock.recv() and send the data back to the client with sock.sendall().

This version of server is not concurrent by design. When multiple clients try to connect to the server at about the same time, one client connects and occupies the server, while other clients wait until the current client disconnects. I wrote a simple simulation program to demonstrate this:

$ python clients.py 
[00.097034] Client 0 tries to connect.
        [00.097670] Client 1 tries to connect.
                [00.098334] Client 2 tries to connect.
[00.099675] Client 0 connects.
[00.600378] Client 0 sends "Hello".
[00.601602] Client 0 receives "Hello".
[01.104952] Client 0 sends "world!".
[01.105166] Client 0 receives "world!".
[01.105276] Client 0 disconnects.
        [01.106323] Client 1 connects.
        [01.611248] Client 1 sends "Hello".
        [01.611609] Client 1 receives "Hello".
        [02.112496] Client 1 sends "world!".
        [02.112691] Client 1 receives "world!".
        [02.112772] Client 1 disconnects.
                [02.113569] Client 2 connects.
                [02.617032] Client 2 sends "Hello".
                [02.617288] Client 2 receives "Hello".
                [03.120725] Client 2 sends "world!".
                [03.120944] Client 2 receives "world!".
                [03.121044] Client 2 disconnects.

The clients connect, send the same two messages and disconnect. It takes half a second for a client to type a message, and thus it takes about three seconds for the server to serve all the clients. A single slow client, however, could make the server unavailable for an arbitrary long time. We should really make the server concurrent!

OS threads

The easiest way to make the server concurrent is by using OS threads. We just run the handle_client() function in a separate thread instead of calling it in the main thread and leave the rest of the code unchanged:

# echo_02_threads.py

import socket
import threading


def run_server(host='127.0.0.1', port=55555):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    while True:
        client_sock, addr = sock.accept()
        print('Connection from', addr)
        thread = threading.Thread(target=handle_client, args=[client_sock])
        thread.start()


def handle_client(sock):
    while True:
        received_data = sock.recv(4096)
        if not received_data:
            break
        sock.sendall(received_data)

    print('Client disconnected:', sock.getpeername())
    sock.close()


if __name__ == '__main__':
    run_server()

Now multiple clients can talk to the server simultaneously:

$ python clients.py 
[00.095948] Client 0 tries to connect.
        [00.096472] Client 1 tries to connect.
                [00.097019] Client 2 tries to connect.
[00.099666] Client 0 connects.
        [00.099768] Client 1 connects.
                [00.100916] Client 2 connects.
[00.602212] Client 0 sends "Hello".
        [00.602379] Client 1 sends "Hello".
                [00.602506] Client 2 sends "Hello".
[00.602702] Client 0 receives "Hello".
        [00.602779] Client 1 receives "Hello".
                [00.602896] Client 2 receives "Hello".
[01.106935] Client 0 sends "world!".
        [01.107088] Client 1 sends "world!".
                [01.107188] Client 2 sends "world!".
[01.107342] Client 0 receives "world!".
[01.107814] Client 0 disconnects.
        [01.108217] Client 1 receives "world!".
        [01.108305] Client 1 disconnects.
                [01.108345] Client 2 receives "world!".
                [01.108395] Client 2 disconnects.

The one-thread-per-client approach is easy to implement, but it doesn't scale well. OS threads are an expensive resource in terms of memory, so you can't have too many of them. For example, the Linux machine that serves this website is capable of running about 8k threads at most, though even fewer threads may be enough to swamp it. With this approach the server not only works poorly under heavy workloads but also becomes an easy target for a DoS attack.

Thread pools solve the problem of uncontrolled thread creation. Instead of submitting each task to a separate thread, we submit tasks to a queue and let a group of threads, called a thread pool, take and process the tasks from the queue. We predefine the maximum number of threads in a thread pool, so the server cannot start too many of them. Here's how we can write a thread pool version of the server using the Python standard concurrent.futures module:

# echo_03_thread_pool.py

import socket
from concurrent.futures import ThreadPoolExecutor


pool = ThreadPoolExecutor(max_workers=20)


def run_server(host='127.0.0.1', port=55555):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    while True:
        client_sock, addr = sock.accept()
        print('Connection from', addr)
        pool.submit(handle_client, client_sock)


def handle_client(sock):
    while True:
        received_data = sock.recv(4096)
        if not received_data:
            break
        sock.sendall(received_data)

    print('Client disconnected:', sock.getpeername())
    sock.close()


if __name__ == '__main__':
    run_server()

The thread pool approach is both simple and practical. Note, however, that you still need to do something to prevent slow clients from occupying the thread pool. You may drop long-living connections, require the clients to maintain some minimum throughput rate, let the threads return the tasks to the queue or combine any of the suggested methods. The conclusion here is that making the server concurrent using OS threads is not as straightforward as it may seem at first, and it's worthwhile to explore other approaches to concurrency.

I/O multiplexing and event loops

Think about the sequential server again. Such a server always waits for some specific event to happen. When it has no connected clients, it waits for a new client to connect. When it has a connected client, it waits for this client to send some data. To work concurrently, however, the server should instead be able to handle any event that happens next. If the current client doesn't send anything, but a new client tries to connect, the server should accept the new connection. It should maintain multiple active connections and reply to any client that sends data next.

But how can the server know what event it should handle next? By default, socket methods such as accept(), recv() and sendall() are all blocking. So if the server decides to call accept(), it will block until a new client connects and won't be not able to call recv() on the client sockets in the meantime. We could solve this problem by setting a timeout on blocking socket operations with sock.settimeout(timeout) or by turning a socket into a completely non-blocking mode with sock.setblocking(False). We could then maintain a set of active sockets and, for each socket, call the corresponding socket method in an infinite loop. So, we would call accept() on the socket that listens for new connections and recv() on the sockets that wait for clients to send data.

The problem with the described approach is that it's not clear how to do the polling right. If we make all the sockets non-blocking or set timeouts too short, the server will be making calls all the time and consume a lot of CPU. Conversely, if we set timeouts too long, the server will be slow to reply.

The better approach is to ask the OS which sockets are ready for reading and writing. Clearly, the OS has this information. When a new packet arrives on a network interface, the OS gets notified, decodes the packet, determines the socket to which the packet belongs and wakes up the processes that do a blocking read on that socket. But a process doesn't need to read from the socket to get notified. It can use an I/O multiplexing mechanism such as select(), poll() or epoll() to tell the OS that it's interested in reading from or writing to some socket. When the socket becomes ready, the OS will wake up such processes as well.

The Python standard selectors module wraps different I/O multiplexing mechanisms available on the system and exposes each of them via the same high-level API called a selector. So it exposes select() as SelectSelector and epoll() as EpollSelector. It also exposes the most efficient mechanism available on the system as DefaultSelector.

Let me show you how you're supposed to use the selectors module. You first create a selector object:

sel = selectors.DefaultSelector()

Then you register a socket that you want to monitor. You pass the socket, the types of events you're interested in (the socket becomes ready for reading or writing) and any auxiliary data to the selector's register() method:

sel.register(sock, selectors.EVENT_READ, my_data)

Finally, you call the selector's select() method:

keys_events = sel.select()

This call returns a list of (key, events) tuples. Each tuple describes a ready socket:

key is an object that stores the socket (key.fileobj) and the auxiliary data associated with the socket (key.data).
events is a bitmask of events ready on the socket (selectors.EVENT_READ or selectors.EVENT_WRITE or both).

If there are ready sockets when you call select(), then select() returns immediately. Otherwise, it blocks until some of the registered sockets become ready. The OS will notify select() as it notifies blocking socket methods like recv().

When you no longer need to monitor some socket, you just pass it to the selector's unregister() method.

One question remains. What should we do with a ready socket? We certainly had some idea of what do to with it when we registered it, so let's register every socket with a callback that should be called when the socket becomes ready. That's, by the way, what the auxiliary data parameter of the selector's register() method is for.

We're now ready to implement a single-threaded concurrent version of the server using I/O multiplexing:

# echo_04_io_multiplexing.py

import socket
import selectors


sel = selectors.DefaultSelector()


def setup_listening_socket(host='127.0.0.1', port=55555):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    sel.register(sock, selectors.EVENT_READ, accept)


def accept(sock):
    client_sock, addr = sock.accept()
    print('Connection from', addr)
    sel.register(client_sock, selectors.EVENT_READ, recv_and_send)


def recv_and_send(sock):
    received_data = sock.recv(4096)
    if received_data:
        # assume sendall won't block
        sock.sendall(received_data)
    else:
        print('Client disconnected:', sock.getpeername())
        sel.unregister(sock)
        sock.close()


def run_event_loop():
    while True:
        for key, _ in sel.select():
            callback = key.data
            sock = key.fileobj
            callback(sock)


if __name__ == '__main__':
    setup_listening_socket()
    run_event_loop()

Here we first register an accept() callback on the listening socket. This callback accepts new clients and registers a recv_and_send() callback on every client socket. The core of the program is the event loop – an infinite loop that on each iteration selects ready sockets and calls the corresponding registered callbacks.

The event loop version of the server handles multiple clients perfectly fine. Its main disadvantage compared to the multi-threaded versions is that the code is structured in a weird, callback-centered way. The code in our example doesn't look so bad, but this is in part because we do not handle all the things properly. For example, writing to a socket may block if the write queue is full, so we should also check whether the socket is ready for writing before calling sock.sendall(). This means that the recv_and_send() function must be decomposed into two functions, and one of these functions must be registered as a callback at any given time depending on the server's state. The problem would be even more apparent if implemented something more complex than the primitive echo protocol.

OS threads do not impose callback style programming on us, yet they provide concurrency. How do they do that? The key here is the ability of the OS to suspend and resume thread execution. If we had functions that can be suspended and resumed like OS threads, we could write concurrent single-threaded code. Guess what? Pyhon allows us to write such functions.

Generator functions and generators

A generator function is a function that has one or more yield expressions in its body, like this one:

$ python -q
>>> def gen():
...     yield 1
...     yield 2
...     return 3
... 
>>>

When you call a generator function, Python doesn't run the function's code as it does for ordinary functions but returns a generator object, or simply a generator:

>>> g = gen()
>>> g
<generator object gen at 0x105655660>

To actually run the code, you pass the generator to the built-in next() function. This function calls the generator's __next__() method that runs the generator to the first yield expression, at which point it suspends the execution and returns the argument of yield. Calling next() second time resumes the generator from the point where it was suspended, runs it to the next yield expression and returns its argument:

>>> next(g)
1
>>> next(g)
2

When no more yield expressions are left, calling next() raises a StopIteration exception:

>>> next(g)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration: 3

If the generator returns something, the exception holds the returned value:

>>> g = gen()
>>> next(g)
1
>>> next(g)
2
>>> try:
...     next(g)
... except StopIteration as e:
...     e.value
... 
3

Initially generators were introduced to Python as an alternative way to write iterators. Recall that in Python an object that can be iterated over (as with a for loop) is called an iterable. An iterable implements the __iter__() special method that returns an iterator. An iterator, in turn, implements __next__() that returns the next value every time you call it. You can get the values by calling next(), but you typically iterate over them with a for loop:

>>> for i in gen():
...     i
... 
1
2

Iterators can be iterated over because they are iterables too. Every iterator implements __iter__() that returns the iterator itself.

Generators allowed us to write iterators as functions that yield values instead of defining classes with special methods. Python fills the special methods for us so that generators become iterators automatically.

Generators produce values in a lazy, on-demand manner, so they are memory-efficient and can even be used to generate infinite sequences. See PEP 255 to learn more about such uses cases. We want to use generators for a completely different reason, though. What's important for us is not the values that a generator produces but the fact that it can be suspended and resumed.

Generators as coroutines

Take any program that performs multiple tasks. Turn functions that represent these tasks into generators by inserting few yield statements here and there. Then run the generators in a round-robin fashion: call next() on every generator in some fixed order and repeat this step until all the generators are exhausted. You'll get a concurrent program that runs like this:

Let's apply this strategy to the sequential server to make it concurrent. First we need to insert some yield statements. I suggest to insert them before every blocking operation. Then we need to run generators. I suggest to write a class that does this. The class should provide the create_task() method that adds a generator to a queue of scheduled generators (or simply tasks) and the run() method that runs the tasks in a loop in a round-robin fashion. We'll call this class EventLoopNoIO since it functions like an event loop except that it doesn't do I/O multiplexing. Here's the server code:

# echo_05_yield_no_io.py

import socket

from event_loop_01_no_io import EventLoopNoIO


loop = EventLoopNoIO()


def run_server(host='127.0.0.1', port=55555):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    while True:
        yield
        client_sock, addr = sock.accept()
        print('Connection from', addr)
        loop.create_task(handle_client(client_sock))


def handle_client(sock):
    while True:
        yield
        received_data = sock.recv(4096)
        if not received_data:
            break
        yield
        sock.sendall(received_data)

    print('Client disconnected:', sock.getpeername())
    sock.close()


if __name__ == '__main__':
    loop.create_task(run_server())
    loop.run()

And here's the event loop code:

# event_loop_01_no_io.py

from collections import deque


class EventLoopNoIO:
    def __init__(self):
        self.tasks_to_run = deque([])

    def create_task(self, coro):
        self.tasks_to_run.append(coro)

    def run(self):
        while self.tasks_to_run:
            task = self.tasks_to_run.popleft()
            try:
                next(task)
            except StopIteration:
                continue
            self.create_task(task)

This counts as a concurrent server. You may notice, however, that it has a problem. Its concurrency is very limited. The tasks run in an interleaved manner, but their order is fixed. For example, if the currently scheduled task is the task that accepts new connections, tasks that handle connected clients have to wait until a new client connects.

Another way to phrase this problem is to say that the event loop doesn't check whether socket operations will block. As we've learned, we can fix it by adding I/O multiplexing. Instead of rescheduling a task immediately after running it, the event loop should reschedule the task only when the socket that the task is waiting on becomes available for reading (or writing). A task can register its intention to read from or write to a socket by calling some event loop method. Or it can just yield this information to the event loop. Here's a version of the server that takes the latter approach:

# echo_06_yield_io.py

import socket

from event_loop_02_io import EventLoopIo


loop = EventLoopIo()


def run_server(host='127.0.0.1', port=55555):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    while True:
        yield 'wait_read', sock
        client_sock, addr = sock.accept()
        print('Connection from', addr)
        loop.create_task(handle_client(client_sock))


def handle_client(sock):
    while True:
        yield 'wait_read', sock
        received_data = sock.recv(4096)
        if not received_data:
            break
        yield 'wait_write', sock
        sock.sendall(received_data)

    print('Client disconnected:', sock.getpeername())
    sock.close()


if __name__ == '__main__':
    loop.create_task(run_server())
    loop.run()

And here's the new event loop that does I/O multiplexing:

# event_loop_02_io.py

from collections import deque
import selectors


class EventLoopIo:
    def __init__(self):
        self.tasks_to_run = deque([])
        self.sel = selectors.DefaultSelector()

    def create_task(self, coro):
        self.tasks_to_run.append(coro)

    def run(self):
        while True:
            if self.tasks_to_run:
                task = self.tasks_to_run.popleft()
                try:
                    op, arg = next(task)
                except StopIteration:
                    continue

                if op == 'wait_read':
                    self.sel.register(arg, selectors.EVENT_READ, task)
                elif op == 'wait_write':
                    self.sel.register(arg, selectors.EVENT_WRITE, task)
                else:
                    raise ValueError('Unknown event loop operation:', op)
            else:
                for key, _ in self.sel.select():
                    task = key.data
                    sock = key.fileobj
                    self.sel.unregister(sock)
                    self.create_task(task)

What do we get out of it? First, we get the server that handles multiple clients perfectly fine:

$ python clients.py 
[00.160966] Client 0 tries to connect.
        [00.161494] Client 1 tries to connect.
                [00.161783] Client 2 tries to connect.
[00.163256] Client 0 connects.
        [00.163409] Client 1 connects.
                [00.163470] Client 2 connects.
[00.667343] Client 0 sends "Hello".
        [00.667491] Client 1 sends "Hello".
                [00.667609] Client 2 sends "Hello".
[00.667886] Client 0 receives "Hello".
        [00.668160] Client 1 receives "Hello".
                [00.668237] Client 2 receives "Hello".
[01.171159] Client 0 sends "world!".
        [01.171320] Client 1 sends "world!".
                [01.171439] Client 2 sends "world!".
[01.171610] Client 0 receives "world!".
[01.171839] Client 0 disconnects.
        [01.172084] Client 1 receives "world!".
        [01.172154] Client 1 disconnects.
                [01.172190] Client 2 receives "world!".
                [01.172237] Client 2 disconnects.

Second, we get the code that looks like regular sequential code. Of course, we had to write the event loop, but this is not something you typically do yourself. Event loops come with libraries, and in Python you're most likely to use an event loop that comes with asyncio.

When you use generators for multitasking, as we did in this section, you typically refer to them as coroutines. Coroutines are functions that can be suspended by explicitly yielding the control. So, according to this definition, simple generators with yield expressions can be counted as coroutines. A true coroutine, however, should also be able to yield the control to other coroutines by calling them, but generators can yield the control only to the caller.

We'll see why we need true coroutines if try to factor out some generator's code into a subgenerator. Consider these two lines of code of the handle_client() generator:

yield 'wait_read', sock
received_data = sock.recv(4096)

It would be very handy to factor them out into a separate function:

def async_recv(sock, n):
    yield 'wait_read', sock
    return sock.recv(n)

and then call the function like this:

received_data = async_recv(sock, 4096)

But it won't work. The async_recv() function returns a generator, not the data. So the handle_client() generator has to run the async_recv() subgenerator with next(). However, it can't just keep calling next() until the subgenerator is exhausted. The subgenerator yields values to the event loop, so handle_client() has to reyield them. It also has to handle the StopIteration exception and extract the result. Obviously, the amount of work that it has to do exceeds all the benefits of factoring out two lines of code.

Python made several attempts at solving this issue. First, PEP 342 introduced enhanced generators in Python 2.5. Generators got the send() method that works like __next__() but also sends a value to the generator. The value becomes the value of the yield expression that the generator is suspended on:

>>> def consumer():
...     val = yield 1
...     print('Got', val)
...     val = yield
...     print('Got', val)
... 
>>> c = consumer()
>>> next(c)
1
>>> c.send(2)
Got 2
>>> c.send(3)
Got 3
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

The generators' __next__() method became simply a shorthand for send(None).

Generators also got the throw() method that runs the generator like send() or __next__() but also raises a specified exception at the suspension point and the close() method that raises a GeneratorExit exception.

Here's how this enhancement solved the subgenerator issue. Instead of running a subgenerator in place, a generator could now yield it to the event loop, and the event loop would run the subgenerator and then send() the result back to the generator (or throw an exception into the generator if the subgenerator raised one). The generator would call the subgenerator like this:

received_data = yield async_recv(sock)

And this call would work just as if one coroutine calls another.

This solution requires some non-trivial logic in the event loop, and you may find it hard to understand. Don't worry. You don't have to. PEP 380 introduced a much more intuitive solution for implementing coroutines in Python 3.3.

yield from

You've probably used yield from to yield values from an iterable. So you should know that this statement:

yield from iterable

works as a shorthand for this piece of code:

for i in iterable:
    yield i

But yield from does much more when you use it with generators. It does exactly what a generator has to do to run a subgenerator in place, and that's why we're discussing it. The main steps of yield from are:

Run the subgenerator once with send(None). If send() raises a StopIteration exception, catch the exception, extract the result, make it a value of the yield from expression and stop.
If subgenerator's send() returns a value without exceptions, yield the value and receive a value sent to the generator.
When received a value, repeat step 1 but this time send() the received value.

This algorithm requires some elaboration. First, yield from automatically propagates exceptions thrown by calling the generator's throw() and close() methods into the subgenerator. The implementation of these methods ensures this. Second, yield from applies the same algorithm to non-generator iterables except that it gets an iterator with iter(iterable) and then uses __next__() instead send() to run the iterator.

Here's how you can remember what yield from does: it makes the subgenerator work as if the subgenerator's code were a part of the generator. So this yield from call:

received_data = yield from async_recv(sock)

works as if the call were replaced with the code of async_recv(). This also counts as a coroutine call, and in contrast to the previous yield-based solution, the event loop logic stays the same.

Let's now take advantage of yield from to make the server's code more concise. First we factor out every boilerplate yield statement and the following socket operation to a separate generator function. We put these functions in the event loop:

# event_loop_03_yield_from.py

from collections import deque
import selectors


class EventLoopYieldFrom:
    def __init__(self):
        self.tasks_to_run = deque([])
        self.sel = selectors.DefaultSelector()

    def create_task(self, coro):
        self.tasks_to_run.append(coro)

    def sock_recv(self, sock, n):
        yield 'wait_read', sock
        return sock.recv(n)

    def sock_sendall(self, sock, data):
        yield 'wait_write', sock
        sock.sendall(data)

    def sock_accept(self, sock):
        yield 'wait_read', sock
        return sock.accept()

    def run(self):
        while True:
            if self.tasks_to_run:
                task = self.tasks_to_run.popleft()
                try:
                    op, arg = next(task)
                except StopIteration:
                    continue

                if op == 'wait_read':
                    self.sel.register(arg, selectors.EVENT_READ, task)
                elif op == 'wait_write':
                    self.sel.register(arg, selectors.EVENT_WRITE, task)
                else:
                    raise ValueError('Unknown event loop operation:', op)
            else:
                for key, _ in self.sel.select():
                    task = key.data
                    sock = key.fileobj
                    self.sel.unregister(sock)
                    self.create_task(task)

Then we yield from the generators in the server's code:

# echo_07_yield_from.py

import socket

from event_loop_03_yield_from import EventLoopYieldFrom


loop = EventLoopYieldFrom()


def run_server(host='127.0.0.1', port=55555):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    while True:
        client_sock, addr = yield from loop.sock_accept(sock)
        print('Connection from', addr)
        loop.create_task(handle_client(client_sock))


def handle_client(sock):
    while True:
        received_data = yield from loop.sock_recv(sock, 4096)
        if not received_data:
            break
        yield from loop.sock_sendall(sock, received_data)

    print('Client disconnected:', sock.getpeername())
    sock.close()


if __name__ == '__main__':
    loop.create_task(run_server())
    loop.run()

And that's it! Generators, yield and yield from are all we need to implement coroutines, and coroutines allow us to write asynchronous, concurrent code that looks like regular sequential code. What about async/await? Well, it's just a syntactic feature on top of generators that was introduced to Python to fix the generators' ambiguity.

async/await

When you see a generator function, you cannot always say immediately whether it's intended to be used as a regular generator or as a coroutine. In both cases, the function looks like any other function defined with def and contains a bunch of yield and yield from expressions. So to make coroutines a distinct concept, PEP 492 introduced the async and await keywords in Python 3.5.

You define a native coroutine function using the async def syntax:

>>> async def coro():
...     return 1
...

When you call such a function, it returns a native coroutine object, or simply a native coroutine. A native coroutine is pretty much the same thing as a generator except that it has a different type and doesn't implement __next__(). Event loops call send(None) to run native coroutines:

>>> coro().send(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration: 1

Native coroutines can call each other with the await keyword:

>>> async def coro2():
...     r = await coro()
...     return 1 + r
... 
>>> coro2().send(None)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration: 2

The await keyword does exactly what yield from does but for native coroutines. In fact, await is implemented as yield from with some additional checks to ensure that the object being awaited is not a generator or some other iterable.

When you use generators as coroutines, you must end every chain of yield from calls with a generator that does yield. Similarly, you must end every chain of await calls with a yield expression. However, if you try to use a yield expression in an async def function, what you'll get is not a native coroutine but something called an asynchronous generator:

>>> async def g():
...     yield 2
... 
>>> g()
<async_generator object g at 0x1046c6790>

We're not going spend time on asynchronous generators here, but in a nutshell, they implement the asynchronous version of the iterator protocol: the __aiter__() and __anext__() special methods (see PEP 525 to learn more). What's important for us at now is that __anext__() is awaitable, while asynchronous generators themeselves are not. Thus, we cannot end a chain of await calls with an async def function containing yield. What should we end the chain with? There are two options.

First, we can write a regular generator function and decorate it with @types.coroutine. This decorator sets a special flag on the function behind the generator so that the generator can be used in an await expression just like a native coroutine:

>>> import types
>>> @types.coroutine
... def gen_coro():
...     yield 3
... 
>>> async def coro3():
...     await gen_coro()
... 
>>> coro3().send(None)
3

A generator decorated with @types.coroutine is called a generator-based coroutine. Why do we need such coroutines? Well, if Python allowed us to await on regular generators, we would again mix the concepts of generators and coroutines and come back to the same ambiguity problem. The @types.coroutine decorator explicitly says that the generator is a coroutine.

As a second option, we can make any object awaitable by defining the __await__() special method. When we await on some object, await first checks whether the object is a native coroutine or a generator-based coroutine, in which case it "yields from" the coroutine. Otherwise, it "yields from" the iterator returned by the object's __await__() method. Since any generator is an iterator, __await__() can be a regular generator function:

>>> class A:
...     def __await__(self):
...             yield 4
... 
>>> async def coro4():
...     await A()
... 
>>> coro4().send(None)
4

Let's now write the final version of the server using async/await. First we mark the server's functions as async and change yield from calls to await calls:

# echo_08_async_await.py

import socket

from event_loop_04_async_await import EventLoopAsyncAwait


loop = EventLoopAsyncAwait()


async def run_server(host='127.0.0.1', port=55555):
    sock = socket.socket()
    sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    sock.bind((host, port))
    sock.listen()
    while True:
        client_sock, addr = await loop.sock_accept(sock)
        print('Connection from', addr)
        loop.create_task(handle_client(client_sock))


async def handle_client(sock):
    while True:
        received_data = await loop.sock_recv(sock, 4096)
        if not received_data:
            break
        await loop.sock_sendall(sock, received_data)

    print('Client disconnected:', sock.getpeername())
    sock.close()


if __name__ == '__main__':
    loop.create_task(run_server())
    loop.run()

Then we modify the event loop. We decorate generator functions with @types.coroutine so that they can be used with await and run the tasks by calling send(None) instead of next():

# event_loop_04_async_await.py

from collections import deque
import selectors
import types


class EventLoopAsyncAwait:
    def __init__(self):
        self.tasks_to_run = deque([])
        self.sel = selectors.DefaultSelector()

    def create_task(self, coro):
        self.tasks_to_run.append(coro)

    @types.coroutine
    def sock_recv(self, sock, n):
        yield 'wait_read', sock
        return sock.recv(n)

    @types.coroutine
    def sock_sendall(self, sock, data):
        yield 'wait_write', sock
        sock.sendall(data)

    @types.coroutine
    def sock_accept(self, sock):
        yield 'wait_read', sock
        return sock.accept()

    def run(self):
        while True:
            if self.tasks_to_run:
                task = self.tasks_to_run.popleft()
                try:
                    op, arg = task.send(None)
                except StopIteration:
                    continue

                if op == 'wait_read':
                    self.sel.register(arg, selectors.EVENT_READ, task)
                elif op == 'wait_write':
                    self.sel.register(arg, selectors.EVENT_WRITE, task)
                else:
                    raise ValueError('Unknown event loop operation:', op)
            else:
                for key, _ in self.sel.select():
                    task = key.data
                    sock = key.fileobj
                    self.sel.unregister(sock)
                    self.create_task(task)

And we're done! We've implemented an async/await-based concurrent server from scratch. It works exactly like the previous version of the server based on yield from and only has a slightly different syntax.

By now, you should understand what async/await is about. But you also should have questions about implementation details of generators, coroutines, yield, yield from and await. We're going to cover all of that in the next section.

How generators and coroutines are implemented *

If you've been following this series, you effectively know how Python implements generators. First recall that the compiler creates a code object for every code block that it encounters, where a code block can be a module, a function or a class body. A code object describes what the code block does. It contains the block's bytecode, constants, variable names and other relevant information. A function is an object that stores the function's code object and such things as the function's name, default arguments and __doc__ attribute.

A generator function is an ordinary function whose code object has a CO_GENERATOR flag set. When you call a generator function, Python checks for this flag, and if it sees the flag, it returns a generator object instead of executing the function. Similarly, a native coroutine function is an ordinary function whose code object has a CO_COROUTINE flag set. Python check for this flag too and returns a native coroutine object if it sees the flag.

To execute a function, Python first creates a frame for it and then executes the frame. A frame is an object that captures the state of the code object execution. It stores the code object itself as well as the values of local variables, the references to the dictionaries of global and built-in variables, the value stack, the instruction pointer and so on.

A generator object stores the frame created for the generator function and some utility data like the generator's name and a flag telling whether the generator is currently running or not. The generator's send() method executes the generator's frame just like Python executes frames of ordinary functions – it calls _PyEval_EvalFrameDefault() to enter the evaluation loop. The evaluation loop iterates over the bytecode instructions one by one and does whatever the instructions tell it to do. The only but crucial difference between calling a function and running a generator is that every time you call the function, Python creates a new frame for it, while the generator keeps the same frame between the runs, thus preserving the state.

How does Python execute yield expressions? Let's see. Every time the compiler encounters yield, it emits a YIELD_VALUE bytecode instruction. We can use the dis standard module to check this:

# yield.py

def g():
    yield 1
    val = yield 2
    return 3

$ python -m dis yield.py
...
Disassembly of <code object g at 0x105b1c710, file "yield.py", line 3>:
  4           0 LOAD_CONST               1 (1)
              2 YIELD_VALUE
              4 POP_TOP

  5           6 LOAD_CONST               2 (2)
              8 YIELD_VALUE
             10 STORE_FAST               0 (val)

  6          12 LOAD_CONST               3 (3)
             14 RETURN_VALUE

YIELD_VALUE tells the evaluation loop to stop executing the frame and return the value on top of the stack (to send() in our case). It works like a RETURN_VALUE instruction produced for a return statement with one exception. It sets the f_stacktop field of the frame to the top of the stack, whereas RETURN_VALUE leaves f_stacktop set to NULL. By this mechanism, send() understands whether the generator yielded or returned the value. In the first case, send() simply returns the value. In the second case, it raises a StopIteration exception that contains the value.

When send() executes a frame for the first time, it doesn't actually sends the provided argument to the generator. But it ensures that the argument is None so that a meaningful value is never ignored:

>>> def g():
...     val = yield
... 
>>> g().send(42)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't send non-None value to a just-started generator

On subsequent runs, send() pushes the argument onto the stack. The argument is then assigned to a variable by STORE_FAST (or similar instruction) or just popped by POP_TOP if yield does not receive a value. If you couldn't remember before whether generators first yield or receive, you should remember now: first YIELD_VALUE, then STORE_FAST.

The compiler emits GET_YIELD_FROM_ITER, LOAD_CONST and YIELD_FROM instructions when it encounters yield from:

# yield_from.py

def g():
    res = yield from another_gen

$ python -m dis yield_from.py
...
Disassembly of <code object g at 0x1051117c0, file "yield_from.py", line 3>:
  4           0 LOAD_GLOBAL              0 (another_gen)
              2 GET_YIELD_FROM_ITER
              4 LOAD_CONST               0 (None)
              6 YIELD_FROM
              8 STORE_FAST               0 (res)
...

The job of GET_YIELD_FROM_ITER is to ensure that the object to yield from, which is the value on top of the stack, is an iterator. If the object is a generator, GET_YIELD_FROM_ITER leaves it as is. Otherwise, GET_YIELD_FROM_ITER replaces the object with iter(obj).

The first thing YIELD_FROM does is pop a value from the stack. Usually, this value is a value pushed by send(). But send() pushes nothing on the first run, so the compiler emits a LOAD_CONST instruction that pushes None before YIELD_FROM.

The second thing YIELD_FROM does is peek the object to yield from. If the value to send is None, YIELD_FROM calls obj.__next__(). Otherwise, it calls obj.send(value). If the call raises a StopIteration exception, YIELD_FROM handles the exception: it replaces the object on top of the stack (i.e. the object to yield from) with the result, and the frame execution continues. If the call returns a value without exceptions, YIELD_FROM stops the frame execution and returns the value to send(). In the latter case, it also sets the instruction pointer in such a way so that the next execution of the frame starts with YIELD_FROM again. What will be different on the subsequent runs is the state of the object to yield from and the value to send.

A native coroutine is basically a generator object that has a different type. The difference between the types is that the generator type implements __iter__() and __next__(), while the coroutine type implements __await__(). The implementation of send() is the same.

The compiler emits the same bytecode instructions for an await expression as for yield from except that instead of a GET_YIELD_FROM_ITER instruction it emits GET_AWAITABLE:

# await.py

async def coro():
    res = await another_coro

$ python -m dis await.py 
...
Disassembly of <code object coro at 0x10d96e7c0, file "await.py", line 3>:
  4           0 LOAD_GLOBAL              0 (another_coro)
              2 GET_AWAITABLE
              4 LOAD_CONST               0 (None)
              6 YIELD_FROM
              8 STORE_FAST               0 (res)
...

GET_AWAITABLE checks whether the object to yield from is a native coroutine or a generator-based coroutine, in which case it leaves the object as is. Otherwise, it replaces the object with obj.__await__().

That's basically how generators and coroutines work. If you still have questions left, I recommend you study the CPython source code. See Include/cpython/code.h for the code object definition, Include/funcobject.h for the function object definition and Include/cpython/frameobject.h for the frame definition. Look at Objects/genobject.c to learn more about generators and coroutines, and look at Python/ceval.c to learn what different bytecode instructions do.

We've figured out how async/await works, but we also need an event loop to run async/await programs. You're unlikely to write your own event loops as we did in this post because that's a lot work. What you usually do instead is use some event loop library. So before we conclude this post, let me say a few words about the library you're most likely to use.

asyncio

asyncio came to the Python standard library around the same time async/await was introduced (see PEP 3156). It does a lot of things, but essentially it provides an event loop and a bunch of classes, functions and coroutines for asynchronous programming.

The asyncio event loop provides an interface similar to that of our final EventLoopAsyncAwait but works a bit differently. Recall that our event loop maintained a queue of scheduled coroutines and ran them by calling send(None). When a coroutine yielded a value, the event loop interpreted the value as an (event, socket) message telling that the coroutine waits for event on socket. The event loop then started monitoring the socket with a selector and rescheduled the coroutine when the event happened.

The asyncio event loop is different in that it does not maintain a queue of scheduled coroutines but only schedules and invokes callbacks. Nevertheless, it provides loop.create_task() and other methods to schedule and run coroutines. How does it do that? Let's see.

The event loop maintains three types of registered callbacks:

The ready callbacks. These are stored in the loop._ready queue and can be scheduled by calling the loop.call_soon() and loop.call_soon_threadsafe() methods.
The callbacks that become ready at some future time. These are stored in the loop._scheduled priority queue and can be scheduled by calling the loop.call_later() and loop.call_at() methods.
The callbacks that become ready when a file descriptor becomes ready for reading or writing. These are monitored using a selector and can be registered by calling the loop.add_reader() and loop.add_writer() methods.

The methods listed above wrap the callback to be scheduled in a Handle or a TimerHandle instance and then schedule and return the handle. Handle instances provide the handle.cancel() method that allows the caller to cancel the callback. TimerHandle is a subclass of Handle for wrapping callbacks scheduled at some future time. It implements the comparison special methods like __le__() so that the sooner a callback is scheduled the less it is. Due to TimerHandle, the loop._scheduled priority queue keeps callbacks sorted by time.

The loop._run_once() method runs one iteration of the event loop. The iteration consists of the following steps:

Remove cancelled callbacks from loop._scheduled.
Call loop._selector.select() and then process the events by adding the callbacks to loop._ready.
Move callbacks whose time has come from loop._scheduled to loop._ready.
Pop callbacks from loop._ready and invoke those that are not cancelled.

So, how does this callback-based event loop run coroutines? Let's take a look at the loop.create_task() method. To schedule a coroutine, it wraps the coroutine in a Task instance. The Task.__init__() method schedules task.__step() as a callback by calling loop.call_soon(). And this is the trick: task.__step() runs the coroutine.

The task.__step() method runs the coroutine once by calling coro.send(None). The coroutine doesn't yield messages. It can yield either None or a Future instance. None means that the coroutine simply wants to yield the control. This is what asyncio.sleep(0) does, for example. If a coroutine yields None, task.__step() simply reschedules itself.

A Future instance represents the result of some operation that may not be available yet. When a coroutine yields a future, it basically tells the event loop: "I'm waiting for this result. It may not be available yet, so I'm yielding the control. Wake me up when the result becomes available".

What does task.__step() do with a future? It calls future.add_done_callback() to add to the future a callback that reschedules task.__step(). If the result is already available, the callback is invoked immediately. Otherwise, it's invoked when someone/something sets the result by calling future.set_result().

Native coroutines cannot yield. Does it mean that we have to write a generator-based coroutine any time we need to yield a future? No. Native coroutines can simply await on futures, like so:

async def future_waiter():
    res = await some_future

To support this, futures implement __await__() that yields the future itself and then returns the result:

class Future:
    # ...

    def __await__(self):
        if not self.done():
            self._asyncio_future_blocking = True
            yield self  # This tells Task to wait for completion.
        if not self.done():
            raise RuntimeError("await wasn't used with future")
        return self.result()  # May raise too.

What sets the result on a future? Let's take a function that creates a future for the socket incoming data as an example. Such a function can be implemented as follows:

Create a new Future instance.
Call loop.add_reader() to register a callback for the socket. The callback should read data from the socket and set the data as the future's result.
Return the future to the caller.

When a task awaits on this future, it will yield the future to task.__step(). The task.__step() method will add a callback to the future, and this callback will reschedule the task when the callback from step 2 sets the result.

We know that a coroutine can wait for the result of another coroutine by awaiting on that coroutine:

async def coro():
    res = await another_coro()

But it can also schedule the coroutine, get a Task instance and then await on the task:

async def coro():
    task = asyncio.create_task(another_coro())
    res = await task

Task subclasses Future so that tasks can be awaited on. What sets the result on a task? It's task.__step(). If coro.send(None) raises a StopIteration exception, task.__step() handles the exception and sets the task's result.

And that's basically how the core of asyncio works. There two facts about it that we should remember. First, the event loop is based on callbacks, and the coroutine support is implemented on top of that. Second, coroutines do not yield messages to the event loop but yield futures. Futures allow coroutines to wait for different things, not only for I/O events. For example, a coroutine may submit a long-running computation to a separate thread and await on a future that represents the result of the computation. We could implement such a coroutine on top of sockets, but it would be less elegant and general than the solution with a future.

Conclusion

The async/await pattern has gained popularity in recent years. Concurrency is as relevant today as ever, and traditional approaches for achieving it, such as OS threads and callbacks, cannot always provide an adequate solution. OS threads work fine in some cases, but in many other cases the concurrency can be implemented much better at the language/application level. A callback-based event loop is technically as good as any async/await solution, but who likes writing callbacks?

It's not to say that async/await is the only right approach to concurrency. Many find other approaches to be better. Take the communicating sequential processes model implemented in Go and Clojure or the actor model implemented in Erlang and Akka as examples. Still, async/await seems to be the best model we have in Python today.

Python didn't invent async/await. You can also find it in C#, JavaScript, Rust, and Swift, to name a few. I'm biased towards Python's implementation because I understand it best, but objectively, it's not the most refined. It mixes generators, generator-based coroutines, native coroutines, yield from and await, which makes it harder to understand. Nevertheless, once you understand these concepts, Python's async/await seems pretty straightforward.

asyncio is a solid library, but it has its issues. The callback-based event loop allows asyncio to provide an API for both callback-style and async/await-style programming. But an event-loop that runs coroutines directly, like those that we wrote in this post, can be much simpler in both implementation and usage. The curio and trio modules are notable alternatives to asyncio that take this approach.

To sum up, concurrency is inherently hard, and no programming model can make it easy. Some models make it manageable, though, and this post should help you master one such model – Python's async/await.

P.S.

The code for this post is available on github. The post is inspired by David Beazley's Curious Course on Coroutines and Concurrency talk and by Eli Bendersky's Concurrent Servers series.

async/await completes the list of topics I wanted to cover in the Python behind the scenes series. I'm now planning to write about other interesting things, but the series is likely to get a sequel in the future. If you want to suggest a topic for the next post, you can write me an email to victor@tenthousandmeters.com.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Update from August 27, 2021: [1] The relationship between concurrency and parallelism is more subtle. Usually, concurrency is viewed as a property of a program and parallelism as a property of a program execution. Thus, you can have "parallelism without concurrency" – even the execution of a sequentially-looking program involves instruction-level or bit-level parallelism. Task-level parallelism is indeed a special case of concurrency.

Python behind the scenes #11: how the Python import system works

2021-07-13T06:36:00+00:00

If you ask me to name the most misunderstood aspect of Python, I will answer without a second thought: the Python import system. Just remember how many times you used relative imports and got something like ImportError: attempted relative import with no known parent package; or tried to figure out how to structure a project so that all the imports work correctly; or hacked sys.path when you couldn't find a better solution. Every Python programmer experienced something like this, and popular StackOverflow questions, such us Importing files from different folder (1822 votes), Relative imports in Python 3 (1064 votes) and Relative imports for the billionth time (993 votes), are a good indicator of that.

The Python import system doesn't just seem complicated – it is complicated. So even though the documentation is really good, it doesn't give you the full picture of what's going on. The only way to get such a picture is to study what happens behind the scenes when Python executes an import statement. And that's what we're going to do today.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

Our plan

Before we begin, let me present you a more detailed version of our plan. First, we'll discuss the core concepts of the import system: modules, submodules, packages, from <> import <> statements, relative imports, and so on. Then we'll desugar different import statements and see that they all eventually call the built-in __import__() function. Finally, we'll study how the default implementation of __import__() works. Let's go!

Modules and module objects

Consider a simple import statement:

import m

What do you think it does? You may say that it imports a module named m and assigns the module to the variable m. And you'll be right. But what is a module exactly? What gets assigned to the variable? In order to answer these questions, we need to give a bit more precise explanation: the statement import m searches for a module named m, creates a module object for that module, and assigns the module object to the variable. See how we differentiated between a module and a module object. We can now define these terms.

A module is anything that Python considers a module and knows how to create a module object for. This includes things like Python files, directories and built-in modules written in C. We'll look at the full list in the next section.

The reason why we import any module is because we want to get an access to functions, classes, constants and other names that the module defines. These names must be stored somewhere, and this is what module objects are for. A module object is a Python object that acts as a namespace for the module's names. The names are stored in the module object's dictionary (available as m.__dict__), so we can access them as attributes.

If you wonder how module objects are implemented, here's the definition from Objects/moduleobject.c:

typedef struct {
    PyObject ob_base;
    PyObject *md_dict;
    struct PyModuleDef *md_def;
    void *md_state;
    PyObject *md_weaklist;
    PyObject *md_name;
} PyModuleObject;

The md_dict field stores the module's dictionary. Other fields are not really important for our discussion.

Python creates module objects implicitly for us. To see that there is nothing magical about this process, let's create a module object ourselves. We usually create Python objects by calling their types, like MyClass() or set(). The type of a module object is PyModule_Type in the C code but it's not available in Python as a built-in. Fortunately, such "unavailable" types can be found in the types standard module:

$ python -q
>>> from types import ModuleType
>>> ModuleType
<class 'module'>

How does the types module define ModuleType? It just imports the sys module (any module will do) and then calls type() on the module object returned. We can do it as well:

>>> import sys
>>> ModuleType = type(sys)
>>> ModuleType
<class 'module'>

No matter how we get ModuleType, once we get it, we can easily create a module object:

>>> m = ModuleType('m')
>>> m
<module 'm'>

A newly created module object is not very interesting but has some special attributes preinitialized:

>>> m.__dict__
{'__name__': 'm', '__doc__': None, '__package__': None, '__loader__': None, '__spec__': None}

Most of these special attributes are mainly used by the import system itself, but some are used in the application code as well. The __name__ attribute, for example, is often used to get the name of the current module:

>>> __name__
'__main__'

Notice that __name__ is available as a global variable. This observation may seem evident, but it's crucial. It comes from the fact that the dictionary of global variables is set to the dictionary of the current module:

>>> import sys
>>> current_module = sys.modules[__name__] # sys.modules stores imported modules
>>> current_module.__dict__ is globals()
True

The current module acts as a namespace for the execution of Python code. When Python imports a Python file, it creates a new module object and then executes the contents of the file using the dictionary of the module object as the dictionary of global variables. Similarly, when Python executes a Python file directly, it first creates a special module called __main__ and then uses its dictionary as the dictionary of global variables. Thus, global variables are always attributes of some module, and this module is considered to be the current module from the perspective of the executing code.

Different kinds of modules

By default, Python recognizes the following things as modules:

Built-in modules.
Frozen modules.
C extensions.
Python source code files (.py files).
Python bytecode files (.pyc files).
Directories.

Built-in modules are C modules compiled into the python executable. Since they are part of the executable, they are always available. This is their key feature. The sys.builtin_module_names tuple stores their names:

$ python -q
>>> import sys
>>> sys.builtin_module_names
('_abc', '_ast', '_codecs', '_collections', '_functools', '_imp', '_io', '_locale', '_operator', '_peg_parser', '_signal', '_sre', '_stat', '_string', '_symtable', '_thread', '_tracemalloc', '_warnings', '_weakref', 'atexit', 'builtins', 'errno', 'faulthandler', 'gc', 'itertools', 'marshal', 'posix', 'pwd', 'sys', 'time', 'xxsubtype')

Frozen modules are too a part of the python executable, but they are written in Python. Python code is compiled to a code object and then the marshalled code object is incorporated into the executable. The examples of frozen modules are _frozen_importlib and _frozen_importlib_external . Python freezes them because they implement the core of the import system and, thus, cannot be imported like other Python files.

C extensions are a bit like built-in modules and a bit like Python files. On one hand, they are written in C or C++ and interact with Python via the Python/C API. On the other hand, they are not a part of the executable but loaded dynamically during the import. Some standard modules including array, math and select are C extensions. Many others including asyncio, heapq and json are written in Python but call C extensions under the hood. Technically, C extensions are shared libraries that expose a so called initialization function. They are usually named like modname.so, but the file extension may be different depending on the platform. On my macOS, for example, any of these extensions will work: .cpython-39-darwin.so, .abi3.so, .so. And on Windows, you'll see .dll and its variations.

Python bytecode files are typically live in a __pycache__ directory alongside regular Python files. They are the result of compiling Python code to bytecode. More specifically, a .pyc file contains some metadata followed by a marshalled code object of a module. Its purpose is to reduce module's loading time by skipping the compilation stage. When Python imports a .py file, it first searches for a corresponding .pyc file in the __pycache__ directory and executes it. If the .pyc file does not exist, Python compiles the code and creates the file.

However, we wouldn't call .pyc files modules if we couldn't execute and import them directly. Surprisingly, we can:

$ ls
module.pyc
$ python module.pyc 
I'm a .pyc file
$ python -c "import module"
I'm a .pyc file

To learn more about .pyc files, check out PEP 3147 -- PYC Repository Directories and PEP 552 -- Deterministic pycs.

As we'll later see, we can customize the import system to support even more kinds of modules. So anything can be called a module as long as Python can create a module object for it given a module name.

Submodules and packages

If module names were limited to simple identifiers like mymodule or utils, then they all must have been unique, and we would have to think very hard every time we give a new file a name. For this reason, Python allows modules to have submodules and module names to contain dots.

When Python executes this statements:

import a.b

it first imports the module a and then the submodule a.b. It adds the submodule to the module's dictionary and assigns the module to the variable a, so we can access the submodule as a module's attribute.

A module that can have submodules is called a package. Technically, a package is a module that has a __path__ attribute. This attribute tells Python where to look for submodules. When Python imports a top-level module, it searches for the module in the directories and ZIP archives listed in sys.path. But when it imports a submodule, it uses the __path__ attribute of the parent module instead of sys.path.

Regular packages

Directories are the most common way to organize modules into packages. If a directory contains a __init__.py file, it's considered to be a regular package. When Python imports such a directory, it executes the __init__.py file, so the names defined there become the attributes of the module.

The __init__.py file is typically left empty or contains package-related attributes such as __doc__ and __version__. It can also be used to decouple the public API of a package from its internal implementation. Suppose you develop a library with the following structure:

mylibrary/
    __init__.py
    module1.py
    module2.py

And you want to provide the users of your library with two functions: func1() defined in module1.py and func2() defined in module2.py. If you leave __init__.py empty, then the users must specify the submodules to import the functions:

from mylibrary.module1 import func1
from mylibrary.module2 import func2

It may be something you want, but you may also want to allow the users to import the functions like this:

from mylibrary import func1, func2

So you import the functions in __init__.py:

# mylibrary/__init__.py
from mylibrary.module1 import func1
from mylibrary.module2 import func2

A directory with a C extension named __init__.so or with a .pyc file named __init__.pyc is also a regular package. Python can import such packages perfectly fine:

$ ls
spam
$ ls spam/
__init__.so
$ python -q
>>> import spam
>>>

Namespace packages

Before version 3.3, Python had only regular packages. Directories without __init__.py were not considered packages at all. And this was a problem because people didn't like to create empty __init__.py files. PEP 420 made these files unnecessary by introducing namespace packages in Python 3.3.

Namespace packages solved another problem as well. They allowed developers to place contents of a package across multiple locations. For example, if you have the following directory structure:

mylibs/
    company_name/
        package1/...
morelibs/
    company_name/
        package2/...

And both mylibs and morelibs are in sys.path, then you can import both package1 and package2 like this:

>>> import company_name.package1
>>> import company_name.package2

This is because company_name is a namespace package that contains two locations:

>>> company_name.__path__
_NamespacePath(['/morelibs/company_name', '/mylibs/company_name'])

How does it work? When Python traverses path entries in the path (sys.path or parent's __path__) during the module search, it remembers the directories without __init__.py that match the module's name. If after traversing all the entries, it couldn't find a regular package, a Python file or a C extension, it creates a module object whose __path__ contains the memorized directories.

The initial idea of requiring __init__.py was to prevent directories named like string or site from shadowing standard modules. Namespace package do not shadow other modules because they have lower precedence during the module search.

Importing from modules

Besides importing modules, we can also import module attributes using a from <> import <> statement, like so:

from module import func, Class, submodule

This statement imports a module named module and assign the specified attributes to the corresponding variables:

func = module.func
Class = module.Class
submodule = module.submodule

Note that the module variable is not available after the import as if it was deleted:

del module

When Python sees that a module doesn't have a specified attribute, it considers the attribute to be a submodule and tries to import it. So if module defines func and Class but not submodule, Python will try to import module.submodule.

Wildcard imports

If we don't want to specify explicitly the names to import from a module, we can use the wildcard form of import:

from module import *

This statement works as if "*" was replaced with all the module's public names. These are the names in the module's dictionary that do not start with an underscore "_" or the names listed in the __all__ attribute if it's defined.

Relative imports

Up until now we've been telling Python what modules to import by specifying absolute module names. The from <> import <> statement allows us to specify relative module names as well. Here are a few examples:

from . import a
from .. import a
from .a import b
from ..a.b import c

The constructions like .. and ..a.b are relative module names, but what are they relative to? As we said, a Python file is executed in the context of the current module whose dictionary acts as a dictionary of global variables. The current module, as any other module, can belong to a package. This package is called the current package, and this is what relative module names are relative to.

The __package__ attribute of a module stores the name of the package to which the module belongs. If the module is a package, then the module belongs to itself, and __package__ is just the module's name (__name__). If the module is a submodule, then it belongs to the parent module, and __package__ is set to the parent module's name. Finally, if the module is not a package nor a submodule, then its package is undefined. In this case, __package__ can be set to an empty string (e.g. the module is a top-level module) or None (e.g. the module runs as a script).

A relative module name is a module name preceded by some number of dots. One leading dot represents the current package. So, when __package__ is defined, the following statement:

from . import a

works as if the dot was replaced with the value of __package__.

Each extra dot tells Python to move one level up from __package__ . If __package__ is set to "a.b", then this statement:

from .. import d

works as if the dots were replaced with a.

You cannot move outside the top-level package. If you try this:

from ... import e

Python will throw an error:

ImportError: attempted relative import beyond top-level package

This is because Python does not move through the file system to resolve relative imports. It just takes the value of __package__, strips some suffix and appends a new one to get an absolute module name.

Obviously, relative imports break when __package__ is not defined at all. In this case, you get the following error:

ImportError: attempted relative import with no known parent package

You most commonly see it when you run a program with relative imports as a script. Since you specify which program to run with a filesystem path and not with a module name, and since Python needs a module name to calculate __package__, the code is executed in the __main__ module whose __package__ attribute is set to None.

Running programs as modules

The standard way to avoid import errors when running a program with relative imports is to run it as a module using the -m switch:

$ python -m package.module

The -m switch tells Python to use the same mechanism to find the module as during the import. Python gets a module name and is able to calculate the current package. For example, if we run a module named package.module, where module refers to a regular .py file, then the code will be executed in the __main__ module whose __package__ attribute is set to "package". You can read more about the -m switch in the docs and in PEP 338.

Alright. This was a warm-up. Now we're going to see what exactly happens when we import a module.

Desugaring the import statement

If we desugar any import statement, we'll see that it eventually calls the built-in __import__() function. This function takes a module name and a bunch of other parameters, finds the module and returns a module object for it. At least, this is what it's supposed to do.

Python allows us to set __import__() to a custom function, so we can change the import process completely. Here's, for example, a change that just breaks everything:

>>> import builtins
>>> builtins.__import__ = None
>>> import math
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'NoneType' object is not callable

You rarely see people overriding __import__() for reasons other than logging or debugging. The default implementation already provides powerful mechanisms for customization, and we'll focus solely on it.

The default implementation of __import__() is importlib.__import__(). Well, it's almost true. The importlib module is a standard module that implements the core of the import system. It's written in Python because the import process involves path handling and other things that you would prefer to do in Python rather than in C. But some functions of importlib are ported to C for performance reasons. And default __import__() actually calls a C port of importlib.__import__(). For our purposes, we can safely ingore the difference and just study the Python version. Before we do that, let's see how different import statements call __import__().

Simple imports

Recall that a piece of Python code is executed in two steps:

The compiler compiles the code to bytecode.
The VM executes the bytecode.

To see what an import statement does, we can look at the bytecode produced for it and then find out what each bytecode instruction does by looking at the evaluation loop in Python/ceval.c.

To get the bytecode, we use the dis standard module:

$ echo "import m" | python -m dis
  1           0 LOAD_CONST               0 (0)
              2 LOAD_CONST               1 (None)
              4 IMPORT_NAME              0 (m)
              6 STORE_NAME               0 (m)
...

The first LOAD_CONST instruction pushes 0 onto the value stack. The second LOAD_CONST pushes None. Then the IMPORT_NAME instruction does something we'll look into in a moment. Finally, STORE_NAME assigns the value on top of the value stack to the variable m.

The code that executes the IMPORT_NAME instruction looks as follows:

case TARGET(IMPORT_NAME): {
    PyObject *name = GETITEM(names, oparg);
    PyObject *fromlist = POP();
    PyObject *level = TOP();
    PyObject *res;
    res = import_name(tstate, f, name, fromlist, level);
    Py_DECREF(level);
    Py_DECREF(fromlist);
    SET_TOP(res);
    if (res == NULL)
        goto error;
    DISPATCH();
}

All the action happens in the import_name() function. It calls __import__() to do the work, but if __import__() wasn't overridden, it takes a shortcut and calls the C port of importlib.__import__() called PyImport_ImportModuleLevelObject(). Here's how this logic is implemented in the code:

static PyObject *
import_name(PyThreadState *tstate, PyFrameObject *f,
            PyObject *name, PyObject *fromlist, PyObject *level)
{
    _Py_IDENTIFIER(__import__);
    PyObject *import_func, *res;
    PyObject* stack[5];

    import_func = _PyDict_GetItemIdWithError(f->f_builtins, &PyId___import__);
    if (import_func == NULL) {
        if (!_PyErr_Occurred(tstate)) {
            _PyErr_SetString(tstate, PyExc_ImportError, "__import__ not found");
        }
        return NULL;
    }

    /* Fast path for not overloaded __import__. */
    if (import_func == tstate->interp->import_func) {
        int ilevel = _PyLong_AsInt(level);
        if (ilevel == -1 && _PyErr_Occurred(tstate)) {
            return NULL;
        }
        res = PyImport_ImportModuleLevelObject(
                        name,
                        f->f_globals,
                        f->f_locals == NULL ? Py_None : f->f_locals,
                        fromlist,
                        ilevel);
        return res;
    }

    Py_INCREF(import_func);

    stack[0] = name;
    stack[1] = f->f_globals;
    stack[2] = f->f_locals == NULL ? Py_None : f->f_locals;
    stack[3] = fromlist;
    stack[4] = level;
    res = _PyObject_FastCall(import_func, stack, 5);
    Py_DECREF(import_func);
    return res;
}

If you carefully examine all of the above, you'll be able to conclude that this statement:

import m

is actually equivalent to this code:

m = __import__('m', globals(), locals(), None, 0)

the meaning of the arguments according to the docstring of importlib.__import__() being the following:

def __import__(name, globals=None, locals=None, fromlist=(), level=0):
    """Import a module.

    The 'globals' argument is used to infer where the import is occurring from
    to handle relative imports. The 'locals' argument is ignored. The
    'fromlist' argument specifies what should exist as attributes on the module
    being imported (e.g. ``from module import <fromlist>``).  The 'level'
    argument represents the package location to import from in a relative
    import (e.g. ``from ..pkg import mod`` would have a 'level' of 2).

    """

As we said, all import statements eventually call __import__(). They differ in what they do before and after the call and how they make the call. Relative imports, for example, pass non-zero level, and from <> import <> statements pass non-empty fromlist.

Let's now express other import statements via __import__() as we expressed import m but much faster this time.

Importing submodules

This statement:

import a.b.c

compiles to the following bytecode:

$ echo "import a.b.c" | python -m dis  
  1           0 LOAD_CONST               0 (0)
              2 LOAD_CONST               1 (None)
              4 IMPORT_NAME              0 (a.b.c)
              6 STORE_NAME               1 (a)
...

and is equivalent to the following code:

a = __import__('a.b.c', globals(), locals(), None, 0)

The arguments to __import__() are passed in the same way as in the case of import m. The only difference is that the VM assigns the result of __import__() not to the name of the module (a.b.c is not a valid variable name) but to the first identifier before the dot, i.e. a. As we'll see, __import__() returns the top-level module in this case.

from <> import <>

This statement:

from a.b import f, g

compiles to the following bytecode:

$ echo "from a.b import f, g" | python -m dis  
  1           0 LOAD_CONST               0 (0)
              2 LOAD_CONST               1 (('f', 'g'))
              4 IMPORT_NAME              0 (a.b)
              6 IMPORT_FROM              1 (f)
              8 STORE_NAME               1 (f)
             10 IMPORT_FROM              2 (g)
             12 STORE_NAME               2 (g)
             14 POP_TOP
...

and is equivalent to the following code:

a_b = __import__('a.b', globals(), locals(), ('f', 'g'), 0)
f = a_b.f
g = a_b.g
del a_b

The names to import are passed as fromlist. When fromlist is not empty, __import__() returns not the top-level module as in the case of a simple import but the specified module like a.b.

from <> import *

This statement:

from m import *

compiles to the following bytecode:

$ echo "from m import *" | python -m dis
  1           0 LOAD_CONST               0 (0)
              2 LOAD_CONST               1 (('*',))
              4 IMPORT_NAME              0 (m)
              6 IMPORT_STAR
...

and is equivalent to the following code:

m = __import__('m', globals(), locals(), ('*',), 0)
all_ = m.__dict__.get('__all__')
if all_ is None:
    all_ = [k for k in m.__dict__.keys() if not k.startswith('_')]
for name in all_:
    globals()[name] = getattr(m, name)
del m, all_, name

The __all__ attribute lists all public names of the module. If some names listed in __all__ are not defined, __import__() tries to import them as submodules.

Relative imports

This statement:

from .. import f

compiles to the following bytecode

$ echo "from .. import f" | python -m dis
  1           0 LOAD_CONST               0 (2)
              2 LOAD_CONST               1 (('f',))
              4 IMPORT_NAME              0
              6 IMPORT_FROM              1 (f)
              8 STORE_NAME               1 (f)
             10 POP_TOP
...

and is equivalent to the following code:

m = __import__('', globals(), locals(), ('f',), 2)
f = m.f
del m

The level argument tells __import__() how many leading dots the relative import has. Since it's set to 2, __import__() calculates the absolute name of the module by (1) taking the value of __package__ and (2) stripping its last portion. The __package__ attribute is available to __import__() because it's passed with globals().

We're now done with import statements and can focus solely on the __import__() function.

Inside import()

As I learned preparing this article, studying __import__() by following all its code paths is not the most entertaining experience. So I offer you a better option. I'll summarize the key algorithms of the import process in plain English and give links to the functions that implement these algorithms so you can read the code if something is left unclear.

The algorithm that __import__() implements can be summarized as follows:

If level > 0, resolve a relative module name to an absolute module name.
Import the module.
If fromlist is empty, drop everything after the first dot from the module name to get the name of the top-level module. Import and return the top-level module.
If fromlist contains names that are not in the module's dictionary, import them as submodules. That is, if submodule is not in the module's dictionary, import module.submodule. If "*" is in fromlist, use module's __all__ as new fromlist and repeat this step.
Return the module.

Step 2 is where all the action happens. We'll focus on it in the remaining sections, but let us first elaborate on step 1.

Resolving relative names

To resolve a relative module name, __import__() needs to know the current package of the module from which the import statement was executed. So it looks up __package__ in globals. If __package__ is None, __import__() tries to deduce the current package from __name__. Since Python always sets __package__ correctly, this fallback is typically unnecessary. It can only be useful for modules created by means other than the default import mechanism. You can look at the _calc___package__() function to see how the current package is calculated exactly. All we should remember is that relative imports break when __package__ is set to an empty string, as in the case of a top-level module, or to None, as in the case of a script, and have a chance of succeeding otherwise. The following function ensures this:

def _sanity_check(name, package, level):
    """Verify arguments are "sane"."""
    if not isinstance(name, str):
        raise TypeError('module name must be str, not {}'.format(type(name)))
    if level < 0:
        raise ValueError('level must be >= 0')
    if level > 0:
        if not isinstance(package, str):
            raise TypeError('__package__ not set to a string')
        elif not package:
            raise ImportError('attempted relative import with no known parent '
                              'package')
    if not name and level == 0:
        raise ValueError('Empty module name')

After the check, the relative name gets resolved:

def _resolve_name(name, package, level):
    """Resolve a relative module name to an absolute one."""
    # strip last `level - 1` portions of `package`
    bits = package.rsplit('.', level - 1)
    if len(bits) < level:
        # stripped less than `level - 1` portions
        raise ImportError('attempted relative import beyond top-level package')
    base = bits[0]
    return '{}.{}'.format(base, name) if name else base

And __import__() calls _find_and_load() to import the module.

The import process

The _find_and_load() function takes an absolute module name and performs the following steps:

If the module is in sys.modules, return it.
Initialize the module search path to None.
If the module has a parent module (the name contains at least one dot), import the parent module if it's not in sys.modules yet. Set the module search path to parent's __path__.
Find the module's spec using the module name and the module search path. If the spec is not found, raise ModuleNotFoundError.
Load the module from the spec.
Add the module to the dictionary of the parent module.
Return the module.

All imported modules are stored in the sys.modules dictionary. This dictionary maps module names to module objects and acts as a cache. Before searching for a module, _find_and_load() checks sys.modules and returns the module immideatly if it's there. Imported modules are added to sys.module at the end of step 5.

If the module is not in sys.modules, _find_and_load() proceeds with the import process. This process consists of finding the module and loading the module. Finders and loaders are objects that perform these tasks.

Finders and loaders

The job of a finder is to make sure that the module exists, determine which loader should be used for loading the module and provide the information needed for loading, such as a module's location. The job of a loader is to create a module object for the module and execute the module. The same object can function both as a finder and as a loader. Such an object is called an importer.

Finders implement the find_spec() method that takes a module name and a module search path and returns a module spec. A module spec is an object that encapsulates the loader and all the information needed for loading. This includes module's special attributes. They are simply copied from the spec after the module object is created. For example, __path__ is copied from spec.submodule_search_locations, and __package__ is copied from spec.parent. See the docs for the full list of spec attributes.

To find a spec, _find_and_load() iterates over the finders listed in sys.meta_path and calls find_spec() on each one until the spec is found. If the spec is not found, _find_and_load() raises ModuleNotFoundError.

By default, sys.meta_path stores three finders:

BuiltinImporter that searches for built-in modules
FrozenImporter that searches for frozen modules; and
PathFinder that searches for different kinds of modules including Python files, directories and C extensions.

These are called meta path finders. Python differentiates them from path entry finders that are a part of PathFinder. We'll discuss both types of finders in the next sections.

After the spec is found, _find_and_load() takes the loader from the spec and passes the spec to the loader's create_module() method to create a module object. If create_module() is not implemented or returns None, then _find_and_load() creates the new module object itself. If the module object does not define some special attributes, which is usually the case, the attributes are copied from the spec. Here's how this logic is implemented in the code:

def module_from_spec(spec):
    """Create a module based on the provided spec."""
    # Typically loaders will not implement create_module().
    module = None
    if hasattr(spec.loader, 'create_module'):
        # If create_module() returns `None` then it means default
        # module creation should be used.
        module = spec.loader.create_module(spec)
    elif hasattr(spec.loader, 'exec_module'):
        raise ImportError('loaders that define exec_module() '
                          'must also define create_module()')
    if module is None:
        # _new_module(name) returns type(sys)(name)
        module = _new_module(spec.name)

    # copy undefined module attributes (__loader__, __package__, etc.)
    # from the spec
    _init_module_attrs(spec, module)
    return module

After creating the module object, _find_and_load() executes the module by calling the loader's exec_module() method. What this method does depends on the loader, but typically it populates the module's dictionary with functions, classes, constants and other things that the module defines. The loader of Python files, for example, executes the contents of the file when exec_module() is called.

The full loading process is implemented as follows:

def _load_unlocked(spec):
    # ... compatibility stuff

    module = module_from_spec(spec)

    # needed for parallel imports
    spec._initializing = True
    try:
        sys.modules[spec.name] = module
        try:
            if spec.loader is None:
                if spec.submodule_search_locations is None:
                    raise ImportError('missing loader', name=spec.name)
                # A namespace package so do nothing.
            else:
                spec.loader.exec_module(module)
        except:
            try:
                del sys.modules[spec.name]
            except KeyError:
                pass
            raise
        # Move the module to the end of sys.modules.
        # This is to maintain the import order.
        # Yeah, Python dicts are ordered
        module = sys.modules.pop(spec.name)
        sys.modules[spec.name] = module
        _verbose_message('import {!r} # {!r}', spec.name, spec.loader)
    finally:
        spec._initializing = False

    return module

This piece of code is interesting for several reasons. First, a module is added to sys.modules before it is executed. Due to this logic, Python supports circular imports. If we have two modules that import each other like this:

# a.py
import b

X = "some constant"

# b.py
import a

We can import them without any issues:

$ python -q
>>> import a
>>>

The catch is that the module a is only partially initialized when the module b is executed. So if we use a.X in b:

# b.py
import a

print(a.X)

we get an error:

$ python -q
>>> import a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/a.py", line 1, in <module>
    import b
  File "/b.py", line 3, in <module>
    print(a.X)
AttributeError: partially initialized module 'a' has no attribute 'X' (most likely due to a circular import)

Second, a module is removed from sys.modules if the execution fails for any reason, but modules that were successfully imported as a side-effect remain in sys.modules.

Finally, the module in sys.modules can be replaced during the module execution. Thus, the module is looked up in sys.modules before it's returned.

We're now done with _find_and_load() and __import__() and ready to see how different finders and loaders work.

BuiltinImporter and FrozenImporter

As we can judge from the name, BuiltinImporter is both a finder and a loader of built-in modules. Its find_spec() method checks if the module is a built-in module and if so, creates a spec that contains nothing but the module's name and the loader. Its create_module() method finds the module's init function and calls it. Both methods are easy to implement because built-in module names are statically mapped to init functions:

struct _inittab _PyImport_Inittab[] = {
    {"posix", PyInit_posix},
    {"errno", PyInit_errno},
    {"pwd", PyInit_pwd},
    {"_sre", PyInit__sre},
    {"_codecs", PyInit__codecs},
    {"_weakref", PyInit__weakref},
    {"_functools", PyInit__functools},
    {"_operator", PyInit__operator},
    {"_collections", PyInit__collections},
    {"_abc", PyInit__abc},
    {"itertools", PyInit_itertools},
    {"atexit", PyInit_atexit},
    // ... more entries
};

The init functions are the same init functions that C extensions define. We're not going to discuss how they work here, so if you want to learn more about this, check out the Extending Python with C or C++ tutorial.

FrozenImporter finds frozen modules in the same way. Their names are statically mapped to code objects:

static const struct _frozen _PyImport_FrozenModules[] = {
    /* importlib */
    {"_frozen_importlib", _Py_M__importlib_bootstrap,
        (int)sizeof(_Py_M__importlib_bootstrap)},
    {"_frozen_importlib_external", _Py_M__importlib_bootstrap_external,
        (int)sizeof(_Py_M__importlib_bootstrap_external)},
    {"zipimport", _Py_M__zipimport,
        (int)sizeof(_Py_M__zipimport)},
    /* Test module */
    {"__hello__", M___hello__, SIZE},
    /* Test package (negative size indicates package-ness) */
    {"__phello__", M___hello__, -SIZE},
    {"__phello__.spam", M___hello__, SIZE},
    {0, 0, 0} /* sentinel */
};

The difference with BuiltinImporter is that create_module() returns None. Code objects are executed by exec_module().

We now focus on the meta path finder that application developers should care about the most.

PathFinder

PathFinder searches for modules on the module search path. The module search path is parent's __path__ passed as the path argument to find_spec() or sys.path if this argument is None. It's expected to be an iterable of strings. Each string, called a path entry, should specify a location to search for modules, such as a directory on the file system.

PathFinder doesn't actually do the search itself but associates each path entry with a path entry finder that knows how to find modules in the location specified by the path entry. To find a module, PathFinder iterates over the path entries and, for each entry, calls find_spec() of the corresponding path entry finder.

To find out which path entry finder to use for a particular entry, PathFinder calls path hooks listed in sys.path_hooks. A path hook is a callable that takes a path entry and returns a path entry finder. It can also raise ImportError, in which case PathFinder tries the next hook. To avoid calling hooks on each import, PathFinder caches the results in the sys.path_importer_cache dictionary that maps path entries to path entry finders.

By default, sys.path_hooks contains two path hooks:

a hook that returns zipimporter instances; and
a hook that returns FileFinder instances.

A zipimporter instance searches for modules in a ZIP archive or in a directory inside a ZIP archive. It supports the same kinds of modules as FileFinder except for C extensions. You can read more about zipimporter in the docs and in PEP 273. A FileFinder instance searches for modules in a directory. We'll discuss it in the next section.

Besides calling path entry finders, PathFinder creates specs for namespace packages. When a path entry finder returns a spec that doesn't specify a loader, this means that the spec describes a portion of a namespace package (typically just a directory). In this case, PathFinder remembers the submodule_search_locations attribute of this spec and continues with the next path entry hoping that it will find a Python file, a regular package or a C extension. If it doesn't find any of these eventually, it creates a new spec for a namespace package whose submodule_search_locations contains all the memorized portions.

To sum up what we said about PathFinder, here's the complete algorithm that its find_spec() implements:

If path is None, set path to sys.path.
Initialize the list of path entries of a potential namespace package: namespace_path = [].
For each path entry in path:
1. Look up the entry in sys.path_importer_cache to get a path entry finder.
2. If the entry is not in sys.path_importer_cache, call hooks listed in sys.path_hooks until some hook returns a path entry finder.
3. Store the path entry finder in sys.path_importer_cache. If no path entry finder is found, store None and continue with the next entry.
4. Call find_spec() of the path entry finder. If the spec is None, continue with the next entry.
5. If found a namespace package (spec.loader is None), extend namespace_path with spec.submodule_search_locations and continue with the next entry.
6. Otherwise, return the spec.
If namespace_path is empty, return None.
Create a new namespace package spec with submodule_search_locations based on namespace_path.
Return the spec.

All this complicated logic of PathFinder is unnecessary most of the time. Typically, a path entry is just a path to a directory, so PathFinder calls the find_spec() method of a FileFinder instance returned by the corresponding hook.

FileFinder

A FileFinder instance searches for modules in the directory specified by the path entry. A path entry can either be an absolute path or a relative path. In the latter case, it's resolved with respect to the current working directory.

The find_spec() method of FileFinder takes an absolute module name but needs only the "tail" portion after the last dot since the package portion was already used to determine the directory to search in. It extracts the "tail" like this:

modname_tail = modname.rpartition('.')[2]

Then it performs the search. It looks for a directory named {modname_tail} that contains __init__.py, __init__.pyc or __init__ with some shared library file extension like .so. It also looks for files named {modname_tail}.py, {modname_tail}.pyc and {modname_tail}.{any_shared_library_extension}. If it finds any of these, it creates a spec with the corresponding loader:

ExtensionFileLoader for a C extension
SourceFileLoader for a .py file; and
SourcelessFileLoader for a .pyc file.

If it finds a directory that is not a regular package, it creates a spec with the loader set to None. PathFinder collects a single namespace package spec from such specs.

The algorithm that find_spec() implements can be summarized as follows:

Get the last portion of the module name: modname_tail = modname.rpartition('.')[2].
Look for a directory named {modname_tail} that contains __init__.{any_shared_library_extension}. If found, create and return a regular package spec.
Look for a file named {modname_tail}.{any_shared_library_extension} If found, create and return a file spec.
Repeat steps 2 and 3 for .py files and for .pyc files.
If found a directory named {modname_tail} that is not a regular package, create and return a namespace package spec.
Otherwise, return None.

A regular package spec is created like this:

loader = SourceFileLoader(modname, path_to_init) # loader may be different
spec = ModuleSpec(modname, loader, origin=path_to_init)
spec.submodule_search_locations = [path_to_package]

a file spec like this:

loader = SourceFileLoader(modname, path_to_file) # loader may be different
spec = ModuleSpec(modname, loader, origin=path_to_file)
spec.submodule_search_locations = None

and a namespace package like this:

spec = ModuleSpec(modname, loader=None, origin=None)
spec.submodule_search_locations = [path_to_package]

Once the spec is created, the loading of the module begins. ExtensionFileLoader is worth studying, but we should leave it for another post on C extensions. SourcelessFileLoader is not very interesting, so we won't discuss it either. SourceFileLoader is the most relevant for us because it loads .py files. We'll briefly mention how it works.

SourceFileLoader

The create_module() method of SourceFileLoader always returns None. This means that _find_and_load() creates the new module object itself and initializes it by copying the attributes from the spec.

The exec_module() method of SourceFileLoader does exactly what you would expect:

def exec_module(self, module):
    """Execute the module."""
    code = self.get_code(module.__name__)
    if code is None:
        raise ImportError('cannot load module {!r} when get_code() '
                        'returns None'.format(module.__name__))
    _bootstrap._call_with_frames_removed(exec, code, module.__dict__)

It calls get_code() to create a code object from the file and then calls exec() to execute the code object in the module's namespace. Note that get_code() first tries to read the bytecode from the .pyc file in the __pycache__ directory and creates this file if it doesn't exist yet.

That's it! We completed our study of finders and loaders and saw what happens during the import process. Let's summarize what we've learned.

Summary of the import process

Any import statement compiles to a series of bytecode instructions, one of which, called IMPORT_NAME, imports the module by calling the built-in __import__() function. If the module was specified with a relative name, __import__() first resolves the relative name to an absolute one using the __package__ attribute of the current module. Then it looks up the module in sys.modules and returns the module if it's there. If the module is not there, __import__() tries to find the module's spec. It calls the find_spec() method of every finder listed in sys.meta_path until some finder returns the spec. If the module is a built-in module, BuiltinImporter returns the spec. If the module is a frozen module, FrozenImporter returns the spec. Otherwise, PathFinder searches for the module on the module search path, which is either the __path__ attribute of the parent module or sys.path if the former is undefined. PathFinder iterates over the path entries and, for each entry, calls the find_spec() method of the corresponding path entry finder. To get the corresponding path entry finder, PathFinder passes the path entry to callables listed in sys.path_hooks. If the path entry is a path to a directory, one of the callables returns a FileFinder instance that searches for modules in that directory. PathFinder calls its find_spec(). The find_spec() method of FileFinder checks if the directory specified by the path entry contains a C extension, a .py file, a .pyc file or a directory whose name matches the module name. If it finds anything, it create a module spec with the corresponding loader. When __import__() gets the spec, it calls the loader's create_module() method to create a module object and then the exec_module() method to execute the module. Finally, it puts the module in sys.modules and returns the module.

Do you have any questions left? I have one.

What's in sys.path?

By default, sys.path includes the following:

An invocation-dependent current directory. If you run a program as a script, it's the directory where the script is located. If you run a program as a module using the -m switch, it's the directory from which you run the python executable. If you run python in the interactive mode or execute a command using the -c switch, the first entry in sys.path will be an empty string.
Directories specified by the PYTHONPATH environment variable.
A zip archive that contains the standard library, e.g. /usr/local/lib/python39.zip. It's used for embeddable installations. Normal installation do not include this archive.
A directory that contains standard modules written in Python, e.g. /usr/local/lib/python3.9.
A directory that contains standard C extensions, e.g. /usr/local/lib/python3.9/lib-dynload.
Site-specific directories added by the site module, e.g. /usr/local/lib/python3.9/site-packages. That's where third-party modules installed by tools like pip go.

To construct these paths, Python first determines the location of the python executable. If we run the executable by specifying a path, Python already knows the location. Otherwise, it searches for the executable in PATH. Eventually, it gets something like /usr/local/bin/python3. Then it tries to find out where the standard modules are located. It moves one directory up from the executable until it finds the lib/python{X.Y}/os.py file. This file denotes the directory containing standard modules written in Python. The same process is repeated to find the directory containing standard C extensions, but the lib/python{X.Y}/lib-dynload/ directory is used as a marker this time. A pyvenv.cfg file alongside the executable or one directory up may specify another directory to start the search from. And the PYTHONHOME environment variable can be used to specify the "base" directory so that Python doesn't need to perform the search at all.

The site standard module takes the "base" directory found during the search or specified by PYTHONHOME and prepends lib/python{X.Y}/site-packages to it to get the directory containing third-party modules. This directory may contain .pth path configuration files that tell site to add more site-specific directories to sys.path. The added directories may contain .pth files as well so that the process repeats recursively.

If the pyvenv.cfg file exists, site uses the directory containing this file as the "base" directory. Note that this is not the directory that pyvenv.cfg specifies. By this mechanism, Python supports virtual environments that have their own site-specific directories but share the standard library with the system-wide installation. Check out the docs on site and PEP 405 -- Python Virtual Environments to learn more about this.

The process of calculating sys.path is actually even more nuanced. If you want to know those nuances, see this StackOverflow answer.

Conclusion

If you ask me to name the most misunderstood aspect of Python, I will answer without a second thought: the Python import system. Until I wrote this post, I couldn't really tell what a module is exactly; what a package is; what relative imports are relative to; how various customization points such as sys.meta_path, sys.path_hooks and sys.path fit together; and how sys.path is calculated. What can I tell now? First, modules and packages are simple concepts. I blame my misunderstanding on the docs that oversimplify the reality like this:

A module is a file containing Python definitions and statements.

or omit the details like this:

You can think of packages as the directories on a file system and modules as files within directories, but don’t take this analogy too literally since packages and modules need not originate from the file system. For the purposes of this documentation, we’ll use this convenient analogy of directories and files.

Relative imports are indeed unintuitive, but once you understand that they are just a way to specify a module name relative to the current package name, you should have no problems with them.

Meta path finders, path entry finders, path hooks, path entries and loaders make the import system more complex but also make it more flexible. PEP 302 and PEP 451 give some rationale for this trade-off.

What's about sys.path? It's crucial to understand what's there when you import a module, yet I couldn't find a satisfactory explanation in the docs. Perhaps, it's too complicated to describe precisely. But I think that the approximation like the one we gave in the previous section is good enough for practical purposes.

Overall, studying the import system was useful, but I think that the next time we should study something more exciting. How about async/await?

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #10: how Python dictionaries work

2021-04-05T06:15:00+00:00

Python dictionaries are an extremely important part of Python. Of course they are important because programmers use them a lot, but that's not the only reason. Another reason is that the interpreter uses them internally to run Python code. CPython does a dictionary lookup every time you access an object attribute or a class variable, and accessing a global or built-in variable also involves a dictionary lookup if the result is not cached. What makes a dictionary appealing is that lookups and other dictionary operations are fast and that they remain fast even as we add more and more elements to the dictionary. You probably know why this is the case: Python dictionaries are hash tables. A hash table is a fundamental data structure. The idea behind it is very simple and widely known. Yet, implementing a practical hash table is not a trivial task. There are different hash table designs that vary in complexity and performance. And new, better designs are constantly being developed.

The goal of this post is to learn how CPython implements hash tables. But understanding all the aspects of hash table design can be hard, and CPython's implementation is especially sophisticated, so we'll approach this topic gradually. In the first part of this post, we'll design a simple fully-functional hash table, discuss its capabilities and limitations and outline a general approach to design a hash table that works well in practice. In the second part, we'll focus on the specifics of CPython's implementation and finally see how Python dictionaries work behind the scenes.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

What is a dictionary

Let us first clarify that a dictionary and a hash table are not the same thing. A dictionary (also known as a map or associative array) is an interface that maintains a collection of (key, value) pairs and supports at least three operations:

Insert a (key, value) pair: d[key] = value.
Look up the value for a given key: d[key].
Delete the key and the associated value: del d[key].

A hash table is a data structure that is commonly used to implement dictionaries. We can, however, use other data structures to implement dictionaries as well. For example, we can store the (key, value) pairs in a linked list and do a linear search to look them up. A dictionary can also be implemented as a sorted array or as a search tree. Any of these data structures will do the job. The difference between them is that they have different performance characteristics. Hash tables are a popular choice because they exhibit excellent average-case performance. To see what it means, let's discuss how hash tables work.

Designing a simple hash table

In its essence, a hash table is an array of (key, value) pairs. A nice fact about arrays is that we can access the i-th element of an array in constant time. The main idea of a hash table is to map each key to an array index and then use this index to quickly locate the corresponding (key, value) pair.

Each position in a hash table is called a bucket. Instead of speaking about the mapping between keys and indices, we often speak about the mapping between keys and buckets. A function that maps keys to buckets is called a hash function. Generally speaking, a hash function is any function that maps arbitrary-size data to fixed-size values, so you may hear this term in other contexts as well. We now show one simple way to construct a hash function for hash tables.

To map (or hash) integer keys, we use a hash function of the form h(key) = key % number_of_buckets. It gives the values in the range [0, number_of_buckets - 1]. And this is exactly what we need! To hash other data types, we first convert them to integers. For example, we can convert a string to an integer if we interpret the characters of the string as digits in a certain base. So the integer value of a string of length $n$ is calculated like this:

$$str\_to\_int(s) = s[0] \times base ^{n-1} + s[1] \times base ^{n-2} + \cdots + s[n-1]$$

where $base$ is the size of the alphabet.

With this approach, different keys may map to the same bucket. In fact, if the number of possible keys is larger than the number of buckets, then some key will always map to the same bucket no matter what hash function we choose. So we have to find a way to handle hash collisions. One popular method to do that is called chaining. The idea of chaining is to associate an additional data structure with each bucket and store all the items that hash to the same bucket in that data structure. The following picture shows a hash table that uses linked lists for chaining:

To insert a (key, value) pair into such a table, we first hash the key to get its bucket and then search for the key in the corresponding linked list. If we find the key, we update the value. If we don't find the key, we add a new entry to the list. The lookup and delete operations are done in a similar manner.

Since the comparison of keys may take a long time (e.g. the keys are long strings), the hashes are typically compared first. If the hashes are not equal, then the keys are not equal either. It's a common practice to store hashes along with keys and values to avoid recomputing them each time.

We now have a working hash table. How well does it perform? The worst-case analysis is quite simple. If the set of possible keys is sufficiently large, then there is a non-zero chance that all the items inserted into the hash table happen to be in the same bucket. The average-case performance is more promising. It largely depends on two factors. First, it depends on how evenly the hash function distributes the keys among buckets. Second, it depends on the average number of items per bucket. This latter characteristic of a hash table is called a load factor:

$$load\_factor = \frac{number\_of\_items}{number\_of\_buckets}$$

Theory says that if every key is equally likely to hash to any bucket, independently of other keys, and if the load factor is bounded by a constant, then the expected time of a single insert, lookup and delete operation is $O(1)$.

To see why this statement is true, insert $n$ different keys into a hash table with $m$ buckets and calculate the expected length of any chain. It will be equal to the load factor:

$$E[len(chain_j)] = \sum_{i=1}^{n} \Pr[key_i \;maps \;to \;bucket\; j ] = n \times \Pr[a\; key \;maps \;to \;bucket\; j ] = n \times \frac{1}{m} = load\_factor$$

For more elaborate proofs, consult a textbook. Introduction to Algorithms (a.k.a. CLRS) is a good choice.

How reasonable are the assumptions of the statement? The load factor assumption is easy to satisfy. We just double the size of the hash table when the load factor exceeds some predefined limit. Let this limit be 2. Then, if upon insertion the load factor becomes more than 2, we allocate a new hash table that has twice as many buckets as the current one and reinsert all the items into it. This way, no matter how many items we insert, the load factor is always kept between 1 and 2. The cost of resizing the hash table is proportional to the number of items in it, so inserts that trigger resizing are expensive. Nevertheless, such inserts are rare because the size of the hash table grows in geometric progression. The expected time of a single insert remains $O(1)$.

The other assumption means that the probability of a key being mapped to a bucket must be the same for all buckets and equal to 1/number_of_buckets. In other words, the hash function must produce uniformly distributed hashes. It's not that easy to construct such a hash function because the distribution of hashes may depend on the distribution of keys. For example, if the keys are integers, and each integer is equally likely to be the next key, then the modulo hash function h(key) = key % number_of_buckets will give uniform distribution of hashes. But suppose that the keys are limited to even integers. Then, if the number of buckets is even, the modulo hash function will never map a key to an odd bucket. At least half of the buckets won't be used.

It's quite easy to choose a bad hash function. In the next section we'll discuss how to choose a good one.

Hash functions

If we cannot predict what the keys in every possible application will be, then we need to choose a hash function that is expected to uniformly distribute any set of keys. The way to do this is to generate the hash function randomly. That is, with equal probability, we assign a random hash to each possible key. Note that the hash function itself must be deterministic. Only the generation step is random.

In theory, a randomly generated hash function is the best hash function. Unfortunately, it's impractical. The only way to represent such a function in a program is to store it explicitly as a table of (key, hash) pairs, like so:

key	0	1	2	3	4	5	6	7	...
h(key)	43	521	883	118	302	91	339	16	...

And this requires too much memory.

The best thing we can do in practice is to choose a hash function that approximates a randomly generated hash function. There exists a number of approaches to do that. Before we delve into them, note that there is no need to choose a separate hash function for each possible hash table size. What real-world hash tables do instead is introduce an auxiliary hash function that maps keys to fixed-size integers, such as 32-bit or 64-bit ints, and another function that maps these integers to hash table buckets. Only the latter function changes when the size of the hash table changes. Typically, this function is just the modulo operation, so that the bucket for a given key is calculated as follows:

hash(key) % number_of_buckets

It's a common practice to use powers of 2 as the hash table size because in this case the modulo operation can be computed very efficiently. To compute hash(key) % (2 ** m), we just take m lower bits of hash(key):

hash(key) & (2 ** m - 1)

This approach may lead to many hash collisions if the hashes differ mainly in higher bits. To make this situation unlikely, the hash() function should be designed to give a close-to-uniform distribution of hashes.

Some hash table designers do not construct the hash() function properly and resort to certain tricks instead. A common advice is to use prime numbers as the hash table size, so that the bucket for a given key is calculated as follows:

hash(key) % prime_number

Composite numbers are considered to be a bad choice because of this identity:

$$ka\;\%\;kn = k (a \;\% \;n)$$

It means that if a key shares a common factor with the number of buckets, then the key will be mapped to a bucket that is a multiple of this factor. So the buckets will be filled disproportionately if such keys dominate. Prime numbers are recommended because they are more likely to break patterns in the input data.

Another trick is to use powers of 2 as the hash table size but scramble the bits of a hash before taking the modulus. You may find such a trick in the Java HashMap:

/**
* Computes key.hashCode() and spreads (XORs) higher bits of hash
* to lower.  Because the table uses power-of-two masking, sets of
* hashes that vary only in bits above the current mask will
* always collide. (Among known examples are sets of Float keys
* holding consecutive whole numbers in small tables.)  So we
* apply a transform that spreads the impact of higher bits
* downward. There is a tradeoff between speed, utility, and
* quality of bit-spreading. Because many common sets of hashes
* are already reasonably distributed (so don't benefit from
* spreading), and because we use trees to handle large sets of
* collisions in bins, we just XOR some shifted bits in the
* cheapest possible way to reduce systematic lossage, as well as
* to incorporate impact of the highest bits that would otherwise
* never be used in index calculations because of table bounds.
*/
static final int hash(Object key) {
    int h;
    return (key == null) ? 0 : (h = key.hashCode()) ^ (h >>> 16);
}

No tricks are needed if we choose a proper hash function in the first place. As we've already said, there exists a number of approaches to do that. Let us now see what they are.

Non-cryptographic hash functions

The first approach is to choose a well-known non-cryptographic hash function that was designed for hash tables. The list of such functions includes Jenkins hash, FNV hash, MurmurHash, CityHash, xxHash and many others. These functions take byte sequences as their inputs, so they can be used to hash all kinds of data. To get a rough idea of how they work, let's take a look at the FNV-1a hash. Here's what its Python implementation may look like:

OFFSET_BASIS = 2166136261
FNV_PRIME = 16777619
HASH_SIZE = 2 ** 32


def fvn1a(data: bytes) -> int:
    h = OFFSET_BASIS
    for byte in data:
        h = h ^ byte
        h = (h * FNV_PRIME) % HASH_SIZE
    return h

For each byte in the input, the function performs two steps:

combines the byte with the current hash value (xor); and
mixes the current hash value (multiplication).

Other hash functions have this structure too. To get an idea of why they work that way and why they use particular operations and constants, check out Bret Mulveyʼs excellent article on hash functions. Bret also explains how to evaluate the quality of a hash function, so we won't discuss it here. Some very interesting results can be found in this answer on StackExchange. Check them out too!

A fixed non-cryptographic hash function performs well in practice under normal circumstances. It performs very poorly when someone intentionally tries to supply bad inputs to the hash table. The reason is that a non-cryptographic hash function is not collision-resistant, so it's fairly easy to come up with a sequence of distinct keys that all have the same hash and thus map to the same bucket. If a malicious user inserts a sequence of $n$ such keys, then the hash table will handle the input in $O(n^2)$. This may take a long time and freeze the program. Such an attack is known as a Hash DoS attack or hash flooding. A potential target of hash flooding is a web application that automatically parses incoming query parameters or POST data into a dictionary. Since most web frameworks offer this functionality, the problem is real. Next we'll look at two approaches to choose a hash function that solve it.

Universal hashing

Note that attackers won't be able to come up with a sequence of colliding keys if they know nothing about the hash function used. So a randomly generated hash function is again the best solution. We said that we cannot use it in practice because it cannot be computed efficiently. But what if we randomly choose a hash function from a family of "good" functions that can be computed efficiently, won't it do the job? It will, though we need to find a suitable family of functions. A family won't be suitable, for example, if we can come up with a sequence of keys that collide for every function in the family. Ideally, we would like to have a family such that, for any set of keys, a function randomly chosen from the family is expected to distribute the keys uniformly among buckets. Such families exist, and they are called universal families. We say that a family of functions is universal if, for two fixed distinct keys, the probability to choose a function that maps the keys to the same bucket is less than 1/number_of_buckets:

$$ \forall x \ne y \in Keys\;\; \underset{h\in F}\Pr[h(x) = h(y)] \le \frac{1}{number\_of\_buckets}$$

It means that an average function from a universal family is unlikely to produce colliding hashes for two distinct keys.

Just to get an idea of what a universal family may look like, here's a classic example of a universal family for hashing integer keys:

$$h_{a, b}(x) = ((ax + b)\;\%\;p)\;\%\;number\_of\_buckets$$

where $p$ is any fixed prime number at least as large as the number of possible keys, and $a \in \{1, ...p-1\}$ and $b \in \{0, ...p-1\}$ are parameters chosen at random that specify a concrete hash function from the family.

What does universality give us? Suppose that we randomly choose a hash function from a universal family and use this hash function to insert a sequence of keys into a hash table with chaining and table resizing as described in the previous section. Then theory says that the expected length of each chain in the hash table is bounded by a constant. This implies that the expected time of a single insert, lookup and delete operation is $O(1)$. And it does not matter what keys we insert!

Note that we've made a similiar statement before:

Theory says that if every key is equally likely to hash to any bucket, independently of other keys, and if the load factor is bounded by a constant, then the expected time of a single insert, lookup and delete operation is $O(1)$.

The important difference is that in the case of universal hashing the word "expected" means averaging over hash functions, while the statement from the previous section refers to averaging over keys.

To learn more about the theory behind universal hashing, read the paper by Lawrence Carter and Mark Wegman that introduced this concept. For examples of universal families, see Mikkel Thorup's survey.

Universal hashing looks good in theory because it guarantees excellent average-case performance and protects against hash flooding. Nevertheless, you won't find many hash table implementations that actually use it. The reason is a combination of two facts:

Universal hash functions are not as fast as the fastest non-universal hash functions.
Universal hash functions do not protect against advanced types of hash flooding.

What does the second point mean? It is true that if a universal hash function is used, attackers cannot come up with a sequence of colliding keys beforehand. But if the attackers can observe how the hash function maps keys, they may be able to deduce how it works and come up with such a sequence. This situation is possible when users work with the hash table interactively: insert a key, then look up a key, then insert a key again and so on. To learn how the hash function maps keys, the attackers can perform a timing attack. First, they insert a single key into the hash table. Then they try to find some other key that maps to the same bucket. Such a key can be detected using a lookup because if a key maps to the same bucket the lookup takes more time. This is one way in which the information about the hash function may leak. Once it leaks, universal hashing doesn't give us any guarantees.

The described attack is known as advanced hash flooding. It was identified by Jean-Philippe Aumasson and Daniel J. Bernstein in 2012. At that time, most hash table implementations used non-cryptographic hash functions. Some of those hash functions employed an idea of universal hashing and took a randomly generated seed. Still, they were vulnerable to hash flooding. Aumasson and Bernstein pointed out this problem and argued that because of advanced hash flooding, even true universal hashing couldn't be a solution. As a solution, they developed a keyed hash function called SipHash, which is now widely used.

SipHash

SipHash takes a 128-bit secret key and a variable-length input and produces a 64-bit hash. Unlike non-cryptographic hash functions, SipHash is designed to have certain cryptographic properties. Specifically, it's designed to work as a message authentication code (MAC). MACs guarantee that it's not feasible to compute the hash of a given input without knowing the secret key even when the hash of any other input is at hand. Thus, if the secret key is randomly generated and unknown to attackers, SipHash protects against advanced hash flooding.

Note that no hash function including SipHash can prevent the attackers from finding the colliding keys by bruteforce as we've seen in the example of a timing attack. This approach, however, requires $O(n^2)$ requests to find $n$ colliding keys, so the potential damage caused by the attack is significantly reduced.

Note also that there is no formal proof of SipHash's security. Such proofs are beyond the state of the art of modern cryptography. Moreover, it's conceivable that someone will break SipHash in the future. Nevertheless, some cryptanalysis and evidence show that SipHash should work as a MAC.

SipHash is not as fast as some non-cryptographic hash functions, but its speed is comparable. The combination of speed and security made SipHash a safe bet for a general-purpose hash table. It's now used as a hash function in Python, Perl, Ruby, Rust, Swift and other languages. To learn more about SipHash, check out the paper by Aumasson and Bernstein.

The choice of the hash function plays a huge role in the performance of a hash table. It is, however, not the only choice hash table designers have to make. They also have to decide how to handle hash collisions. Chaining is one option, but there are other methods that often perform better. In fact, most state-of-the-art hash tables use methods other than chaining. Let us now see what those methods are.

Collision resolution methods

We saw that chaining can be used to implement a hash table whose average-case performance is constant. Asymptotically, we cannot do better. But asymptotic behavior is not what's important in practice. What's important in practice is the actual time it takes to process real-world data and the amount of memory required to do that. From this perspective, other collision resolution methods often perform better than chaining. Most of them are based on the same idea called open addressing.

In open addressing, all items are stored directly in the hash table. Hash collisions are resolved by using a hash function of a special form. Instead of mapping each key to a single bucket, a hash function of this form maps each key to a sequence of buckets. Such a sequence is called a probe sequence. Buckets in a probe sequence are called probes.

To insert a new (key, value) pair in a hash table with open addressing, we iterate over the buckets in the probe sequence until we find an empty bucket and store the key and the value in that bucket. We'll always find an empty bucket eventually if the hash table is not full and if the probe sequence covers all the buckets in the hash table. In addition to that, the probe sequence should be a permutation of buckets since visiting the same bucket more than once is a waste of time. The following picture illustrates the insertion process into a hash table with open addressing:

To look up the value of a key, we iterate over the buckets in the probe sequence until we either find the key or find an empty bucket. If we find an empty bucket, then the key is not in the hash table because otherwise it would be inserted into the empty bucket that we found.

Deleting a key from a hash table with open addressing is not that straightforward. If we just clear the bucket that the key occupies, then some lookups will break because lookups assume that probe sequences don't have gaps. This picture illustrates the problem:

The problem is typically solved by marking the item deleted instead of actually deleting it. This way, it continues to occupy the bucket, so lookups do not break. A deleted item goes away completely in one of two ways. It's either displaced by a new item or removed when the hash table resizes.

One advantage of open addressing over chaining is that the hash table doesn't store a linked list pointer for every item in the hash table. This saves space. On the other hand, empty buckets take more space because each bucket stores an item instead of a pointer. Whether a hash table with open addressing is more memory-efficient depends on the size of items. If the items are much larger than pointers, then chaining is better. But if the items take little space (e.g. the keys and the values are pointers themselves), then open addressing wins. The saved space can then be used to increase the number of buckets. More buckets means less hash collisions, and less hash collisions means the hash table is faster.

So, how do we construct a hash function that returns probe sequences? Typically, it's built of ordinary hash functions that we studied before. In linear probing, for example, an ordinary hash function is used to compute the first probe. Every next probe is just the next bucket in the hash table:

probes[i] = hash(key) + i % number_of_buckets

So if the first probe is bucket b, then the probe sequence is:

[b, b + 1, b + 2, ..., number_of_buckets - 1, 0, 1, ..., b - 1]

Despite its simplicity, linear probing guarantees constant average-case performance under two conditions. The first conditions is that the load factor must be strictly less than 1. The second condition is that the hash() function must map every key with equal probability to any bucket and independently of other keys.

As we've already discussed, the second condition is hard-to-impossible to satisfy. In practice, we choose a hash function that works well enough, but linear probing is very sensitive to the quality of the hash function, so it's harder to do. Another issue is that the load factor must be low if we want a decent performance. Consider the following estimate of the expected number of scanned buckets to insert a new key that Donald Knuth derives in his proof of the statement:

$$E[\#scanned\_buckets(load\_factor)] \approx \frac{1}{2}(1 + \frac{1}{(1-load\_factor)^2})$$

If we take a load factor of 90%, then we'll have about 50 buckets scanned on average assuming that the number of items in the hash table is sufficiently large. Thus, the load factor should be much lower. And that means more empty buckets and higher memory usage.

When we insert a new key or look up a key that is not in a hash table, we want to find an empty bucket as soon as possible. With linear probing it can be a problem because of contiguous clusters of occupied buckets. Such clusters tend to grow because the larger the cluster is, the more likely the next key will hash to a bucket in that cluster and will be inserted at its end. This problem is known as primary clustering.

Quadratic probing solves the primary clustering problem and is less sensitive to the quality of the hash function. It's similar to linear probing. The difference is that the value of the i-th probe depends quadratically on i:

probes[i] = hash(key) + a * i + b * (i ** 2) % number_of_buckets

The constants a and b must be chosen carefully for the probe sequence to cover all the buckets. When the size of the hash table is a power of 2, setting a = b = 1/2 guarantees that the probe sequence will cover all the buckets before it starts repeating them. What does the probe sequence look like in this case? If the first probe is bucket b, then the sequence goes like b, b + 1, b + 3, b + 6, b + 10, b + 15, b + 21 and so on (modulo number_of_buckets). Note that the intervals between consecutive probes increase by 1 at each step. This is a well-known sequence of triangular numbers, and triangular numbers are guaranteed to produce complete probe sequences. See this paper for the proof.

An alternative to quadratic probing is pseudo-random probing. Like other probing schemes, it calls an ordinary hash function to compute the first probe:

probes[0] = hash(key) % number_of_buckets

Then it passes the first probe as a seed to a pseudo-random number generator (PRNG) to compute the subsequent probes. Typically, the PRNG is implemented as a linear congruential generator, so the probes are computed as follows:

probes[i] = a * probes[i-1] + c % number_of_buckets

Hull–Dobell Theorem tells us how to choose the constants a and c so that the probe sequence covers all the buckets before it starts repeating them. If the size of the hash table is a power of 2, then setting a = 5 and c = 1 will do the job.

Quadratic probing and pseudo-random probing are still quite sensitive to the quality of the hash function because the probe sequences of two different keys will be identical whenever their first probes are the same. This situation is also a form of clustering known as secondary clustering. There is a probing scheme that mitigates it. It's called double hashing.

In double hashing, the interval between two consecutive probes depends on the key itself. More specifically, a second, independent hash function determines the interval, so the probe sequence is calculated as follows:

probes[i] = hash1(key) + hash2(key) * i % number_of_buckets

To ensure that the probe sequence covers all the buckets, the hash2() function must produce hashes that are relatively prime to the number of buckets, that is, hash2(key) and number_of_buckets must have no common factors except 1. This can be achieved by constructing the hash2() function in such a way so that it always returns an odd number and by setting the size of the hash table to a power of 2.

The more "random" probe sequences are, the less likely clustering is to occur and the less probes are needed. Thus, in theory, such sequences are better. But theory and practice do not always agree. Up until now we've been measuring the time complexity of algorithms in the number of elementary steps, such as the number of probes or the number of traversed linked list nodes. This metric works fine for asymptotic analysis, but it does not agree with the actual time measurements because it assumes that the cost of each elementary step is roughly the same, and that's not true in reality. In reality, the steps that access main memory are the most expensive. A single access to RAM takes about 100 ns. Compare it to the cost of accessing the fastest CPU cache – it's about 1 ns. Therefore, one of the most important aspects of hash table design is the effective use of the cache.

Linear probing may perform quite well because it's very cache-friendly. To see why, recall that data is moved from the main memory to the cache in cache lines, which are contiguous blocks of memory, typically 64 bytes long. When the contents of the first bucket in a probe sequence have been read, the contents of the next several buckets are already in the cache.

As a general rule, a data structure will be more cache-effective if the items that are often used together are placed close to each other in memory. Linear probing follows this rule much better than other probing schemes. And open addressing in general works better than chaining in this respect because in chaining each item sits in a separately allocated node.

To better comprehend how much the cache affects hash table performance, consider the following graph:

This graph shows how the time of a single lookup in a Python dictionary changes as the number of items in the dictionary increases. It is clear that the time is not constant but increases as well. Why? Hash collisions are not the reason because the keys were chosen at random from a uniform distribution. You might also think that it's a peculiarity of a Python dictionary, but it's not. Any other hash table would behave similarly. The real reason is that when the hash table is small, it fits completely into the cache, so the CPU doesn't need to access the main memory. As the hash table grows larger, the portion of the hash table that is not in the cache grows as well, and the CPU has to access the main memory more frequently.

By the way, have you noticed those zigzags in the graph? They indicate the moments when the hash table resizes.

We discussed a number of methods to resolve hash collisions: chaining and open addressing with various probing schemes. You probably think, "Why do we need all of them?" The reason is that different methods suit different use cases. Chaining makes sense when the items are large and when deletes are frequent. Linear probing works best when the items are small and when the hash function distributes the keys uniformly. And quadratic probing, pseudo-random probing and double hashing are a safe bet in most cases.

State-of-the-art hash tables are typically variations of open addressing with some improvements. Google's Swiss Table, for example, uses SIMD instructions to probe several buckets in parallel. This talk explains how it works in detail. Robin Hood hashing is perhaps the most popular advanced method to resolve hash collisions. To understand the idea behind it, observe that the number of probes to look up a key equals the number of probes that was required to insert it. Naturally, we would like to keep those numbers low. And that's what Robin Hood hashing tries to do. When a new key gets inserted, it doesn't just wait for an empty bucket but can also displace other keys. It displaces any key whose final probe number is less than the number of the current probe. The displaced key then continues on its probe sequence, possibly displacing other keys. As a result, large probe numbers do not emerge, and lookups become faster. To learn more about the benefits of Robin Hood hashing, check out this post. See also Malte Skarupke's talk for an overview of advanced methods to resolve hash collisions.

Well done! We've covered the essentials of hash table design. There is much more to say on this topic, but we now know enough to understand how Python dictionaries work. Without further ado, let's apply our knowledge.

Python dictionaries

Overview

A Python dictionary is a hash table with open addressing. Its size is always a power of 2 and is initially set to 8. When the load factor exceeds 2/3, the hash table resizes. Usually, the size just doubles, but it can also be set to some lesser power of 2 if deleted items occupy a lot of buckets. In short, the load factor varies between 1/3 and 2/3.

The hash of a Python object is a 32-bit or 64-bit singed integer (on 32-bit and 64-bit platforms respectively). We call the built-in hash() function to compute it, and this function works by calling the tp_hash slot of the object's type. Built-in types implement the tp_hash slot directly, and classes can implement it by defining the __hash__() special method. Thus, the hash function is different for different types. Strings and bytes objects are hashed with SipHash, while other types implement custom, simpler hashing algorithms.

The hash of an integer, for example, is usually the integer itself:

$ python -q
>>> hash(1)
1
>>> hash(2343)
2343
>>> hash(-54)
-54

This is not always the case because Python integers can be arbitrary large. So CPython implements a hashing algorithm that works like this:

MODULUS = 2 ** 61 - 1 # Mersenne prime; taking the modulus is efficient

def hash_unoptimized(integer):
    """Unoptimized version of hash() for integers"""
    hash_value = abs(integer) % MODULUS
    if integer < 0:
        hash_value = -hash_value

    if hash_value == -1: # -1 indicates an error; do not use it
        return -2

    return hash_value

Because the algorithm is so simple, it's very easy to come up with a sequence of integers that all have the same hash:

$ python -q
>>> modulus = 2 ** 61 - 1
>>> hash(0)
0
>>> hash(modulus)
0
>>> hash(modulus * 2)
0
>>> hash(modulus * 3)
0
>>> hash(modulus * 1000)
0

Isn't this a security issue? Apparently, CPython developers thought that nobody in a sane mind would cast keys to integers automatically when parsing untrusted user input, so they decided not to use SipHash in this case.

But even non-malicious inputs exhibit regularities that such a primitive hash function won't break. To mitigate the effects of poorly distributed hashes, CPython implements a clever probing scheme.

The probing scheme is pseudo-random probing with a modification. To see the reasoning behind this modification, recall that pseudo-random probing suffers from secondary clustering: the whole probe sequence is determined by the first probe, and the first probe depends only on lower bits of the hash (m lower bits when the size of the hash table is 2**m). CPython solves this problem by perturbing the first few probes with values that depend on higher bits of the hash. Here's what the algorithm that computes probes looks like:

def get_probes(hash_value, hash_table_size):
    mask = hash_table_size - 1 # used to take modulus fast
    perturb = hash_value # used to perturb the probe sequence
    probe = hash_value & mask

    while True:
        yield probe

        perturb >>= 5
        probe = (probe * 5 + perturb + 1) & mask

Initially, perturb is set to the hash value. Then, at each iteration, it is shifted 5 bits to the right and the result is added to the linear congruential generator to perturb the next probe. This way, every next probe depends on 5 extra bits of the hash until perturb becomes 0. When perturb becomes 0, the linear congruential generator is guaranteed to cover all the buckets by the Hull–Dobell Theorem.

Despite the clever probing scheme, CPython's hash tables seem very inefficient. First, their maximum load factor is 2/3, which is about 66.6%, and this is when state-of-the-art hash tables work well with load factors of 90% and more. So there is a huge room for improvement here. Second, pseudo-random probing is not cache-friendly. And we saw how important the cache is.

Are CPython's hash tables really as inefficient as they seem? Well, they certainly perform worse than Google's Swiss Table with hundreds of millions of items. But they are not optimized for such huge loads. They are optimized to be compact and to be fast when the hash table is small enough to fit into the cache. This is because the most important uses of Python dictionaries are the storage and retrieval of object attributes, class methods and global variables. And in this cases, the dictionaries are typically small and many.

CPython employs some interesting optimizations to better fit the use cases above. Let's take a look at them.

Compact dictionaries

Before version 3.6, the layout of CPython's hash tables was typical. Each bucket held a 24-byte entry that consisted of a hash, a key pointer and a value pointer. So the following dictionary:

d = {"one": 1, "two": 2, "three": 3}

would be represented like this:

hash_table = [
    ('--', '--', '--'),
    (542403711206072985, 'two', 2),
    ('--', '--', '--'),
    (4677866115915370763, 'three', 3),
    ('--', '--', '--'),
    (-1182584047114089363, 'one', 1),
    ('--', '--', '--'),
    ('--', '--', '--')
]

In CPython 3.6 the layout changed. Since then, the entries are stored in a separate, dense array, and the hash table stores only the indices to that array. The same dictionary is now represented like this:

hash_table = [None, 1, None, 2, None, 0, None, None]
entries = [
    (-1182584047114089363, 'one', 1),
    (542403711206072985, 'two', 2),
    (4677866115915370763, 'three', 3),
    ('--', '--', '--'),
    ('--', '--', '--')
]

Each index to the entries array takes 1, 2, 4 or 8 bytes depending on the size of the hash table. In any case it is much less than 24 bytes taken by an entry. As a result, empty buckets take less space, and dictionaries become more compact. Of course, the entries array should have extra space for future entries as well. Otherwise, it would have to resize on every insert. But CPython manages to save space nonetheless by setting the size of the entries array to 2/3 of the size of the hash table and resizing it when the hash table resizes.

This optimization has other benefits too. Iteration over a dictionary became faster because entries are densely packed. And dictionaries became ordered because items are added to the entries array in the insertion order.

Shared keys

CPython stores the attributes of an object in the object's dictionary. Since instances of the same class often have the same attributes, there can be a lot of dictionaries that have the same keys but different values. And that's another opportunity to save space!

Since CPython 3.3, object dictionaries of the same class share keys. The keys and hashes are stored in a separate data structure in the class, and the dictionaries store only a pointer to that structure and the values.

For example, consider a simple class whose instances have the same two attributes:

class Point:
    def __init__(self, x, y):
        self.x = x
        self.y = y

And consider two instances of this class:

p1 = Point(4, 4)
p2 = Point(5, 5)

The dictionaries of p1 and p2 will store their own arrays of values but will share everything else:

hash_table = [None, 1, None, None, 0, None, None, None]
entries = [
    (-8001793907708313420, 'x', None),
    (308703142051095673, 'y', None),
    ('--', '--', '--'),
    ('--', '--', '--'),
    ('--', '--', '--')
]

values_p1 = [4, 4, None, None, None]
values_p2 = [5, 5, None, None, None]

Of course, the keys can diverge. If we add a new attribute to an object, and this attribute is not among the shared keys, then the object's dictionary will be converted to an ordinary dictionary that doesn't share keys. And the dictionaries of new objects won't share keys as well. The conversion will not happen only when the object is the sole instance of the class. So you should define all the attributes on the first instance before you create other instances. One way to do this is to define the attributes in the __init__() special method.

To learn more about key-sharing dictionaries, check out PEP 412.

String interning

To look up a key in a hash table, CPython has to find an equal key in the probe sequence. If two keys have different hashes, then CPython may safely assume that the keys are not equal. But if the keys have the same hash, it must compare the keys to see if they are equal or not. The comparison of keys may take a long time, but it can be avoided altogether when the keys are in fact the same object. To check whether this is the case, we can just compare their ids (i.e. memory addresses). The only problem is to ensure that we always use the same object.

When we create two strings with the same contents, we often get two equal but distinct objects:

$ python -q
>>> a = 'hi!'
>>> b = 'hi!'
>>> a is b
False

To get a reference to the same object, we need to use the sys.intern() function:

>>> import sys
>>> a = sys.intern('hi!')
>>> b = sys.intern('hi!')
>>> a is b
True

The first call to sys.intern() will return the passed string but before that it will store the string in the dictionary of interned strings. The dictionary will map the string to itself, and the second call will find the string in the dictionary and return it.

CPython interns many strings automatically. For example, it interns some string constants:

>>> a = 'hi'
>>> b = 'hi'
>>> a is b
True

These are all the string constants that match this regex:

[a-zA-Z0-9_]*

CPython also interns the names of variables and attributes so we don't have to do that ourselves.

This concludes our study of Python dictionaries. We discussed the most important ideas behind them but left out some implementation details. If you want to know those details, take a look at the source code in Objects/dictobject.c.

A note on sets

Dictionaries are closely related to sets. In fact, sets are just dictionaries without values. Because of this, you might think that CPython implements sets in the same way as it implements dictionaries. But it doesn't. A set is a different object and the hash table behind it works a bit differently. For example, its maximum load factor is not 66.6% but 60%, and if there are less than 50,000 items in the set, its growth factor is not 2 but 4. The most important difference is in the probing scheme. Sets use the same pseudo-random probing but, for every probe, they also inspect 9 buckets that follow the probe. It's basically a combination of pseudo-random and linear probing.

CPython doesn't rely on sets internally as it relies on dictionaries so there is no need to optimize them for specific use cases. Moreover, the general use cases for sets are different. Here's a comment from the source code that explains this:

Use cases for sets differ considerably from dictionaries where looked-up keys are more likely to be present. In contrast, sets are primarily about membership testing where the presence of an element is not known in advance. Accordingly, the set implementation needs to optimize for both the found and not-found case.

The implementation of sets can be found in Objects/setobject.c.

Conclusion

It's not that hard to implement your own hash table once you've seen how others do it. Still, it is hard to choose a hash table design that fits your use case best. CPython implements hash tables that are optimized both for general and internal use. The result is a unique and clever design. But it is also controversial. For example, the probing scheme is designed to tolerate bad hash functions, and this may come at the expense of cache-friendliness. Of course, it's all talk, and only benchmarks can tell the truth. But we cannot just take some state-of-the-art hash table for C++ and compare it with a Python dictionary because Python objects introduce overhead. A proper benchmark would implement Python dictionaries with different hash table designs. It's a lot of work, though, and I don't know of anyone who did it. So, do you have any plans for the next weekend?

The dict type is a part of the builtins module, so we can always access it. Things that are not in builtins have to be imported before they can be used. And that's why we need the Python import system. Next time we'll see how it works.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #9: how Python strings work

2021-02-21T06:45:00+00:00

In 1991 Guido van Rossum released the first version of the Python programming language. About that time the world began to witness a major change in how computer systems represent written language. The internalization of the Internet increased the demand to support different writing systems, and the Unicode Standard was developed to meet this demand. Unicode defined a universal character set able to represent any written language, various non-alphanumeric symbols and, eventually, emoji 😀. Python wasn't designed with Unicode in mind, but it evolved towards Unicode support during the years. The major change happened when Python got a built-in support for Unicode strings – the unicode type that later became the str type in Python 3. Python strings have been proven to be a convenient way to work with text in the Unicode age. Today we'll see how they work behind the scenes.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

The scope of this post

This post doesn't try to cover all aspects of text encoding in relation to Python. You see, programming language designers have to make several text encoding decisions because they have to answer the following questions:

How to talk to the external world (the encodings of command-line parameters, environment variables, standard streams and the file system).
How to read the source code (the encoding of source files).
How to represent text internally (the encoding of strings).

This post focuses on the last problem. But before we dive into the internals of Python strings, let's briefly discuss the problem of text encoding on a real life example and clarify what Unicode really is.

The essence of text encoding

You see this text as a sequence of characters rendered by your browser and displayed on your screen. I see this text as the same sequence of characters as I type it into my editor. In order for us to see the same thing, your browser and my editor must be able to represent the same set of characters, that is, they must agree on a character set. They also need to choose some, possibly different, ways to represent the text internally to be able to work with it. For example, they may choose to map each character to a unit consisting of one or more bytes and represent the text as a sequence of those units. Such a mapping is usually referred to as a character encoding. A character encoding is also crucial for our communication. Your browser and my web server must agree on how to encode text into bytes and decode text from bytes, since bytes is what they transmit to talk to each other.

The character set that your browser and my editor use is Unicode. Unicode is able to represent English as well as any other written language you can think of (文言, Čeština, Ελληνικά, עברית, हिन्दी), 日本語, Português, Русский) and thousands of miscellaneous symbols (₤, ⅐, ↳, ∭, ⌘, , ♫, 👨🏼‍💻, 🍺) . My web server sends this text as a part of the HTML page in the UTF-8 encoding. You browser knows which encoding was used to encode the text because the Content-Type HTTP header declares the encoding:

Content-Type: text/html; charset=utf-8

Even if you save this HTML page locally, your browser will still be able to detect its encoding because the encoding is specified in the HTML itself:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="utf-8" />
    <!-- ... -->
</html>

This may seem absurd to you. How can a browser decode the HTML to read the encoding if it doesn't know the encoding yet? This is usually not a problem in practice because the beginning of an HTML page contains only ASCII characters and most encodings used on the web encode ASCII characters in the same way. Check out the HTML standard to learn more about the algorithm that browsers use to determine the encoding.

Note that the HTTP header and the HTML metatag specify "charset", i.e. a character set. This may seem confusing since UTF-8 is not a character set. What they really specify is a character encoding. The two terms are often used interchangeably because character encodings typically imply a character set of the same name. For example, the ASCII character encoding implies the ASCII character set. The Unicode Standard fixes the terminology by giving precise definitions to all important terms. We'll study them, but before, let's discuss why and how the Unicode project began.

The road to Unicode

Before the adoption of Unicode, most computer systems used the ASCII character encoding that encodes a set of 128 characters using a 7-bit pattern to encode each character. ASCII was sufficient to deal with English texts but that's about it. Other character encodings were developed to support more languages. Most of them extended ASCII to 256 characters and used one byte to encode each character. For example, the ISO 8859 standard defined a family of 15 such character encodings. Among them were:

Latin Western European ISO 8859-1 (German, French, Portuguese, Italian, etc.)
Central European ISO 8859-2 (Polish, Croatian, Czech, Slovak, etc.)
Latin/Cyrillic ISO 8859-5 (Russian, Serbian, Ukrainian, etc.)
Latin/Arabic ISO 8859-6
Latin/Greek ISO 8859-7.

Multi-lingual software had to handle many different character encodings. This complicated things a lot. Another problem was to choose the right encoding to decode text. Failing to do so resulted in a garbled text known as mojibake. For example, if you encode the Russian word for mojibake "кракозябры" using the KOI-8 encoding and decode it using ISO 8859-1, you'll get "ËÒÁËÏÚÑÂÒÙ".

The problems with different character encodings are not gone completely. Nevertheless, it became much more easier to write multi-lingual software nowadays. This is due to two independent initiatives that began in the late 1980s. One was ISO 10646, an international standard, and the other was Unicode, a project organized by a group of software companies. Both projects had the same goal: to replace hundreds of conflicting character encodings with a single universal one that covers all languages in widespread use. They quickly realized that having two different universal character sets wouldn't help achieve the goal, so in 1991 the Universal Coded Character Set (UCS) defined by ISO 10646 and Unicode's character set were unified. Today the projects define essentially the same character encoding model. Nevertheless, both continue to exist. The difference between them is that the Unicode Standard has a greater scope:

The assignment of characters is only a small fraction of what the Unicode Standard and its associated specifications provide. The specifications give programmers extensive descriptions and a vast amount of data about the handling of text, including how to:

divide words and break lines

sort text in different languages

format numbers, dates, times, and other elements appropriate to different locales

display text for languages whose written form flows from right to left, such as Arabic or Hebrew

display text in which the written form splits, combines, and reorders, such as for the languages of South Asia

deal with security concerns regarding the many look-alike characters from writing systems around the world

The most important thing that we need to understand about Unicode is how it encodes characters.

Unicode basics

Unicode defines characters as smallest components of written language that have semantic value. This means that such units as diacritical marks are considered to be characters on their own. Multiple Unicode characters can be combined to produce what visually looks like a single character. Such combinations of characters are called grapheme clusters in Unicode. For example, the string "á" is a grapheme cluster that consists of two characters: the Latin letter "a" and the acute accent "´". Unicode encodes some grapheme clusters as separate characters as well, but does that solely for compatibility with legacy encodings. Due to combining characters, Unicode can represent all sorts of grapheme clusters such as "ä́" and, at the same time, keep the character set relatively simple.

Unicode characters are abstract. The standard doesn't care about the exact shape a character takes when it's rendered. The shape, called a glyph, is considered to be a concern of a font designer. The connection between characters and glyphs can be quite complicated. Multiple characters can merge into a single glyph. A single character can be rendered as multiple glyphs. And how characters map to glyphs can depend on the context. Check out the Unicode Technical Report #17 for examples.

Unicode doesn't map characters to bytes directly. It does the mapping in two steps:

The coded character set maps characters to code points.
A character encoding form, such as UTF-8, maps code points to sequences of code units, where each code unit is a sequence of one or more bytes.

The Unicode coded character set is what we usually mean when we say Unicode. It's the same thing as the UCS defined by ISO 10646. The word "coded" means that it's not actually a set but a mapping. This mapping assigns a code point to each character in the character set. A code point is just an integer in the range [0, 1114111], which is written as U+0000..U+10FFFF in the Unicode hexadecimal notation and is called a code space. The current Unicode 13.0 assigns code points to 143,859 characters.

Technically, the coded character set is a collection of entries. Each entry defines a character and assigns a code point to it by specifying three pieces of information:

the code point value
the name of the character; and
a representative glyph.

For example, the entry for the letter "b" looks like this: (U+0062, LATIN SMALL LETTER B, b).

The standard also specifies various character properties such as whether the character is a letter, a numeral or some other symbol, whether it's written from left-to-right or from right-to-left and whether it's an uppercase letter, lowercase letter or doesn't have a case at all. All this information is contained in the Unicode Character Database. We can query this database from Python using the unicodedata standard module.

If we encode some text with the coded character set, what we get is a sequence of code points. Such a sequence is called a Unicode string. This is an appropriate level of abstraction to do text processing. Computers, however, know nothing about code points, so code points must be encoded to bytes. Unicode defines three character encoding forms to do that: UTF-8, UTF-16 and UTF-32. Each is capable of encoding the whole code space but has its own strengths and weaknesses.

UTF-32 is the most straightforward encoding form. Each code point is represented by a code unit of 32 bits. For example, the code point U+01F193 is encoded as 0x0001F193. The main advantage of UTF-32, besides simplicity, is that it's a fixed-width encoding form, i.e. each code point corresponds to a fixed number of code units (in this case – one). This allows fast code point indexing: we can access the nth code point of a UTF-32-encoded string in constant time.

Originally, Unicode defined only one encoding form that represented each code point by a code unit of 16 bits. It was possible to encode the whole code space using this encoding form because the code space was smaller and consisted of 2^16 = 65,536 code points. Over the time, Unicode people realized that 65,536 code points were not enough to cover all written language and extended the code space to 1,114,112 code points. The problem was that new code points, which constituted the range U+010000..U+10FFFF, could not be represented by a 16-bit code unit. Unicode solved this problem by encoding each new code points with a pair of 16-bit code units, called a surrogate pair. Two unassigned ranges of code points were reserved to be used only in surrogate pairs: U+D800..U+DBFF for higher parts of surrogate pairs and U+DC00..U+DFFF for lower parts of surrogate pairs. Each of these ranges consists of 1024 code points, so they can be used to encode 1024 × 1024 = 1,048,576 code points. This encoding form that uses one 16-bit code unit to encode code points in the range U+0000..U+FFFF and two 16-bit code units to encode code points in the range U+010000..U+10FFFF became known as UTF-16. Its original version is a part of the ISO 10646 standard and is called UCS-2. The only difference between UTF-16 and UCS-2 is that UCS-2 doesn't support surrogate pairs and is only capable of encoding code points in the range U+0000..U+FFFF known as the Basic Multilingual Plane (BMP). The ISO 10646 standard also defines the UCS-4 encoding form, which is effectively the same thing as UTF-32.

UTF-32 and UTF-16 are widely used for representing Unicode strings in programs. They are, however, not very suitable for text storage and transmission. The first problem is that they are space-inefficient. This is especially true when a text that consists mostly of ASCII characters is encoded using the UTF-32 encoding form. The second problem is that bytes within a code unit can be arranged in a little-endian or big-endian order, so UTF-32 and UTF-16 come in two flavors each. The special code point called the byte order mark (BOM) is often added to the beginning of a text to specify the endianness. And the proper handling of BOMs adds complexity. The UTF-8 encoding form doesn't have these issues. It represents each code point by a sequence of one, two, three or four bytes. The leading bits of the first byte indicate the length of the sequence. Other bytes always have the form 0b10xxxxxx to distinguish them from the first byte. The following table shows what sequences of each length look like and what ranges of code points they encode:

Range	Byte 1	Byte 2	Byte 3	Byte 4
U+0000..U+007F	`0b0xxxxxxx`
U+0080..U+07FF	`0b110xxxxx`	`0b10xxxxxx`
U+0800..U+FFFF	`0b1110xxxx`	`0b10xxxxxx`	`0b10xxxxxx`
U+010000..U+10FFFF	`0b11110xxx`	`0b10xxxxxx`	`0b10xxxxxx`	`0b10xxxxxx`

To encode a code point, we choose an appropriate template from the table above and replace xs in it with the binary representation of a code point. An appropriate template is the shortest template that is capable of encoding the code point. The binary representation of a code point is aligned to the right, and the leading xs are replaced with 0s.

Note that UTF-8 represents all ASCII characters using just one byte, so that any ASCII-encoded text is also a UTF-8-encoded text. This feature is one of the reasons why UTF-8 gained adoption and became the most dominant encoding on the web.

This section should give us a basic idea of how Unicode works. If you want to learn more about Unicode, I really recommend reading the first few chapters of the Unicode Standard.

A brief history of Python strings

The way Python strings work today is very different from the way Python strings worked when Python was first released. This aspect of the language changed significantly multiple times. To better understand why modern Python strings work the way they do, let's take a quick look into the past.

Initially, Python had one built-in type to represent strings – the str type. It was not the str type we know today. Python strings were byte strings, that is, sequences of bytes, and worked similar to how bytes objects work in Python 3. This is in contrast to Python 3 strings that are Unicode strings.

Since byte strings were sequences of bytes, they were used to represent all kinds of data: sequences of ASCII characters, UTF-8-encoded texts and arbitrary arrays of bytes. Byte strings themselves didn't hold any information about the encoding. It was up to a program to interpret the values. For example, we could put a UTF-8-encoded text into a byte string, print it to the stdout and see the actual Unicode characters if the terminal encoding was UTF-8:

$ python2.7
>>> s = '\xe2\x9c\x85'
>>> print(s)
✅

Though byte strings were sequences of bytes, they were called strings for a reason. The reason is that Python provided string methods for byte strings, such as str.split() and str.upper(). Think about what the str.upper() method should do on a sequence of bytes. It doesn't make sense to take a byte and convert it to an uppercase variant because bytes don't have case. It starts make sense if we assume that the sequence of bytes is a text in some encoding. That's exactly what Python did. The assumed encoding depended on the current locale. Typically, it was ASCII. But we could change the locale, so that string methods began to work on non-ASCII-encoded text:

$ python2.7
>>> s = '\xef\xe8\xf2\xee\xed' # Russian 'питон' in the encoding windows-1251
>>> '\xef\xe8\xf2\xee\xed'.upper() # does nothing since characters are non-ascii
'\xef\xe8\xf2\xee\xed'
>>> import locale
>>> locale.setlocale(locale.LC_ALL , 'ru_RU.CP1251')
'ru_RU.CP1251'
>>> '\xef\xe8\xf2\xee\xed'.upper() # converts to uppercase
'\xcf\xc8\xd2\xce\xcd'
>>> print('\xef\xe8\xf2\xee\xed'.upper().decode('windows-1251')) # let's print it
ПИТОН

The implementation of this logic relied on the C standard library. It worked for 8-bit fixed-width encodings but didn't work for UTF-8 or any other Unicode encoding. In short, Python had no Unicode strings back then.

Then the unicode type was introduced. This happened before Python 2 when PEPs hadn't existed yet. The change was only later described in PEP 100. The instances of unicode were true Unicode strings, that is, sequences of code points (or, if you prefer, sequences of Unicode characters). They worked much like strings we have today:

$ python2.7
>>> s = u'питон' # note unicode literal
>>> s # each element is a code point
u'\u043f\u0438\u0442\u043e\u043d'
>>> s[1] # can index code points
u'\u0438'
>>> print(s.upper()) # string methods work
ПИТОН

Python used the UCS-2 encoding to represent Unicode strings internally. UCS-2 was capable of encoding all the code points that were assigned at that moment. But then Unicode assigned first code points outside the Basic Multilingual Plane, and UCS-2 could no longer encode all the code points. Python switched from UCS-2 to UTF-16. Now any code point outside the Basic Multilingual Plane could be represented by a surrogate pair. This caused another problem. Since UTF-16 is a variable-width encoding, getting the nth code point of a string requires scanning the string until that code point is found. Python supported indexing into a string in constant time and didn't want to lose that. So, what happened is that Unicode objects seized to be true Unicode strings and became sequence of code units. This had the following consequences:

$ python2.7
>>> u'hello'[4] # indexing is still supported and works fast
u'o'
>>> len(u'😀') # but length of a character outside BMP is 2
2
>>> u'😀'[1] # and indexing returns code units, not code points
u'\ude00'

PEP 261 tried to revive true Unicode strings. It introduced a compile-time option that enabled the UCS-4 encoding. Now Python had two distinct builds: a "narrow" build and a "wide" build. The choice of the build affected the way Unicode objects worked. UCS-4 could not replace UTF-16 altogether because of its space-inefficiency, so both had to coexist. Internally, Unicode object was represented as an array of Py_UNICODE elements. The Py_UNICODE type was set to wchar_t if the size of wchar_t was compatible with the build. Otherwise, it was set to either unsigned short (UTF-16) or unsigned long (UCS-4).

In the meantime, Python developers focused their attention on another source of confusion: the coexistence of byte strings and Unicode strings. There were several problems with this. For example, it was possible to mix two types:

>>> "I'm str" + u" and I'm unicode"
u"I'm str and I'm unicode"

Unless it wasn't:

>>> "I'm str \x80" + u" and I'm unicode"
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
UnicodeDecodeError: 'ascii' codec can't decode byte 0x80 in position 8: ordinal not in range(128)

The famous Python 3.0 release renamed the unicode type to the str type and replaced the old str type with the bytes type. The essence of this change is summarized in the release notes:

The biggest difference with the 2.x situation is that any attempt to mix text and data in Python 3.0 raises TypeError, whereas if you were to mix Unicode and 8-bit strings in Python 2.x, it would work if the 8-bit string happened to contain only 7-bit (ASCII) bytes, but you would get UnicodeDecodeError if it contained non-ASCII values. This value-specific behavior has caused numerous sad faces over the years.

Python strings became the Python strings we know today with the release of Python 3.3. PEP 393 got rid of "narrow" and "wide" builds and introduced the flexible string representation. This representation made Python strings true Unicode strings without exceptions. Its essence can be summarized as follows. Three different fixed-width encodings are used to represent strings: UCS-1, UCS-2 and UCS-4. Which encoding is used for a given string depends on the largest code point in that string:

If all code points are in the range U+0000..U+00FF, then UCS-1 is used. UCS-1 encodes code points in that range with one byte and does not encode other code points at all. It's equivalent to the Latin-1 (ISO 8859-1) encoding.
If all code points are in the range U+0000..U+FFFF and at least one code point is in the range U+0100..U+FFFF, then UCS-2 is used.
Finally, if at least one code point is in the range U+010000..U+10FFFF, then UCS-4 is used.

In addition to this, CPython distinguishes the case when a string contains only ASCII characters. Such strings are encoded using UCS-1 but stored in a special way. Let's take a look at the actual code to understand the details.

Meet modern Python strings

CPython uses three structs to represent strings: PyASCIIObject, PyCompactUnicodeObject and PyUnicodeObject. The second one extends the first one, and the third one extends the second one:

typedef struct {
  PyObject_HEAD
  Py_ssize_t length;
  Py_hash_t hash;
  struct {
      unsigned int interned:2;
      unsigned int kind:2;
      unsigned int compact:1;
      unsigned int ascii:1;
      unsigned int ready:1;
  } state;
  wchar_t *wstr;
} PyASCIIObject;

typedef struct {
  PyASCIIObject _base;
  Py_ssize_t utf8_length;
  char *utf8;
  Py_ssize_t wstr_length;
} PyCompactUnicodeObject;

typedef struct {
  PyCompactUnicodeObject _base;
  union {
      void *any;
      Py_UCS1 *latin1;
      Py_UCS2 *ucs2;
      Py_UCS4 *ucs4;
  } data;
} PyUnicodeObject;

Why do we need all these structs? Recall that CPython provides the Python/C API that allows writing C extensions. In particular, it provides a set of functions to work with strings. Many of these functions expose the internal representation of strings, so PEP 393 could not get rid of the old representation without breaking C extensions. One of the reasons why the current representation of strings is more complicated than it should be is because CPython continues to provide the old API. For example, it provides the PyUnicode_AsUnicode() function that returns the Py_UNICODE* representation of a string.

Let's first see how CPython represents strings created using the new API. These are called "canonical" strings. They include all the strings that we create when we write Python code. The PyASCIIObject struct is used to represent ASCII-only strings. The buffer that holds a string is not a part of the struct but immediately follows it. The allocation is done at once like this:

obj = (PyObject *) PyObject_MALLOC(struct_size + (size + 1) * char_size);

The PyCompactUnicodeObject struct is used to represent all other Unicode strings. The buffer is allocated in the same way right after the struct. Only struct_size is different and char_size can be 1, 2 or 4.

The reason why both PyASCIIObject and PyCompactUnicodeObject exist is because of an optimization. It's often necessary to get a UTF-8 representation of a string. If a string is an ASCII-only string, then CPython can simply return the data stored in the buffer. But otherwise, CPython has to perform a conversion from the current encoding to UTF-8. The utf8 field of PyCompactUnicodeObject is used to store the cached UTF-8 representation. This representation is not always cached. The special API function PyUnicode_AsUTF8AndSize() should be called when the cache is needed.

If someone requests the old Py_UNICODE* representation of a "canonical" string, then CPython may need to perform a conversion. Similarly to utf8, the wstr field of PyASCIIObject is used to store the cached Py_UNICODE* representation.

The old API allowed creating strings with a NULL buffer and filling the buffer afterwards. Today the strings created in this way are called "legacy" strings. They are represented by the PyUnicodeObject struct. Initially, they have only the Py_UNICODE* representation. The wstr field is used to hold it. The users of the API must call the PyUnicode_READY() function on "legacy" strings to make them work with the new API. This function stores the canonical (USC-1, UCS-2 or UCS-4) representation of a string in the data field of PyUnicodeObject.

The old API is still supported but deprecated. PEP 623 lays down a plan to remove it in Python 3.12.

Perhaps the most interesting question about the flexible string representation is how to get it. Typically, a string is created by decoding a sequence of bytes using some encoding. This is how the parser creates strings from string literals. This is how contents of a file become strings. And this is what happens when we call the decode() method of a bytes object. In all of these cases Python uses the UTF-8 encoding by default, so let's discuss the algorithm that decodes a UTF-8-encoded text to a Python string. It's not immediately obvious how to implement such an algorithm because CPython needs to choose an appropriate struct and encoding to represent the string (ASCII, UCS-1, UCS-2 or UCS-4), and it must decode all the code points to do that. One solution would be to read the input twice: the first time to determine the largest code point in the input and the second time to convert the input from the UTF-8 encoding to the chosen internal encoding. This is not what CPython does. It tries to be optimistic and initially creates an instance of PyASCIIObject to represent the string. If it encounters a non-ASCII character as it reads the input, it creates an instance of PyCompactUnicodeObject, chooses the next most compact encoding that is capable of representing the character and converts the already decoded prefix to the new encoding. This way, it reads the input once but may change the internal representation up to three times. The algorithm is implemented in the unicode_decode_utf8() function in Objects/unicodeobject.c.

There is a lot more to say about Python strings. The implementation of string methods, such as str.find() and str.join(), is an interesting topic, but it probably deserves a separate port. Another topic worth discussing is string interning. We'll cover it when we take a look at how Python dictionaries work. This post focuses on how CPython implements strings and it won't be complete if we don't discuss alternative ways to implement strings in a programming language, so that's what we'll do now.

How other Python implementations represent strings

The flexible string representation is quite complex, so you might wonder whether other Python implementations, such as PyPy and MicroPython, use it. The short answer is: they don't. In fact, I'm not aware of any other language, not to say about Python implementation, that takes CPython's approach.

MicroPython uses UTF-8 for the string representation. Strings are true Unicode strings just like in CPython. Code point indexing is supported but implemented by scanning the string, so it takes $O(n)$ time to access the nth code point.

PyPy uses UTF-8 too. But it does code point indexing in constant time. The trick is simple. Here's how you can do it. Think about a UTF-8 representation as a sequence of blocks, each block (with the possible exception of the last) containing 64 code points. Create an array of integers such that ith element of the array is a starting byte position of the ith block. Then the nth code point of a string can be found as follows:

def get_code_point(buffer, n):
    block_num, code_point_in_block = divmod(n, 64)
    block_start_byte = block_index[block_num]
    return seek_forward(buffer[block_start_byte:], code_point_in_block)

This message on the pypy-dev mailing list explains the algorithm in more detail.

MicroPython and PyPy have to implement the same strings that CPython implements in order to stay compatible with it. But other languages have different views on what a string should be in the first place. It's especially interesting to look at those languages that were designed with Unicode in mind. This is the focus of the next section.

How strings work in other languages

C

The most primitive form of a string data type is an array of bytes. Python 2 strings are an example of this approach. It comes from C where strings are represented as arrays of char. The C standard library provides a set of functions like toupper() and isspace() that take bytes and treat them as characters in the encoding specified by the current locale. This allows working with encodings that use one byte per character. To support other encodings, the wchar_t type was introduced in the C90 standard. Unlike char, wchar_t is guaranteed to be large enough to represent all characters in any encoding specified by any supported locale. For example, if some locale specifies the UTF-8 encoding, then wchar_t must be large enough to represent all Unicode code points. The problem with wchar_t is that it is platform-dependent and its width can be as small as 8 bits. The C11 standard addressed this problem and introduced the char16_t and char32_t types that can be used to represent code units of UTF-16 and UTF-32 respectively in a platform-independent way. Chapter 5 of the Unicode Standard discusses Unicode data types in C in more detail.

Go

In Go, a string is a read-only slice of bytes, i.e. an array of bytes along with the number of bytes in the array. A string may hold arbitrary bytes just like an array of char in C, and indexing into a string returns a byte. Nevertheless, Go provides decent Unicode support. First, Go source code is always UTF-8. This means that string literals are valid UTF-8 sequences. Second, iterating over a string with the for loop yields Unicode code points. There is a separate type to represent code points – the rune type. Third, the standard library provides functions to work with Unicode. For example, we can use the ValidString() function provided by the unicode/utf8 package to check whether a given string is a valid UTF-8 sequence. To learn more about strings in Go, check out this excellent article written by Rob Pike.

Rust

Rust provides several string types. The main string type, called str, is used to represent UTF-8-encoded text. A string is a slice of bytes that cannot hold arbitrary bytes but only a valid UTF-8 sequence. Attempt to create a string from a sequence of bytes that is not a valid UTF-8 sequence results in an error. Indexing into a string by an integer is not supported. The docs give a reasoning for that:

Indexing is intended to be a constant-time operation, but UTF-8 encoding does not allow us to do this. Furthermore, it's not clear what sort of thing the index should return: a byte, a codepoint, or a grapheme cluster. The bytes and chars methods return iterators over the first two, respectively.

The iteration is the way to access code points. Nevertheless, it's possible to index into a string by a range, like &string[0..4]. This operation returns a substring consisting of bytes in the specified range. If the substring is not a valid UTF-8 sequence, the program will crash. It's always possible to access individual bytes of a string by converting it to a byte slice first. To learn more about strings in Rust, check out Chapter 8 of the Rust Programming Language book.

Swift

Swift takes the most radical approach when it comes to Unicode support. A string in Swift is a sequence of Unicode grapheme clusters, that is, a sequence of human-perceived characters. The count property returns the number of grapheme clusters:

let str = "\u{65}\u{301}"
print(str)
print(str.count)

// Output:
// é
// 1

And iterating over a string yields grapheme clusters:

let str = "Cluster:\u{1112}\u{1161}\u{11AB} "
for c in str {
    print(c, terminator:" ")
}

// Output:
// C l u s t e r : 한

To implement such behavior, a language must be able to detect boundaries of grapheme clusters. The Unicode Standard Annex #29 describes how to do that algorithmically.

Internally, a string is stored in the UTF-8 encoding. Indexing into a string by an integer is not supported. There is an API, though, that allows accessing grapheme clusters by indices:

let str = "Swift";
let c = str[str.index(str.startIndex, offsetBy: 3)]
print(c)

// Output:
// f

It looks intentionally clumsy to remind programmers about the expensiveness of the operation. To learn more about strings in Swift, check out the Language Guide.

Conclusion

In the modern world of programming, the word "string" means Unicode data. Programmers should be aware of how Unicode works, and language designers should provide the right abstraction to deal with it. Python strings are sequences of Unicode code points. The flexible string representation allows indexing into a string in constant time and, at the same time, tries to keep strings relatively compact. This approach seems to work well for Python because accessing elements of a string is easy, and in most cases programmers don't even think whether those elements should be characters or grapheme clusters. Modern languages, such as Go, Rust and Swift, questioned whether indexing into a string is important at all. They give us an idea of what the best approach for implementing strings may look like: represent strings internally as UTF-8 sequences and provide a set of iterators that yield bytes, code units, code points and grapheme clusters. Python evolves. Will it gravitate towards this approach in the future?

The implementation of built-in types is a fascinating topic. It's always interesting and useful to know how things you constantly deal with actually work. This is especially true of Python dictionaries. They are not only extensively used by programmers but also underlie important features of the language. Next time we'll see how they work.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #8: how Python integers work

2021-02-08T14:12:00+00:00

In the previous parts of this series we studied the core of the CPython interpreter and saw how the most fundamental aspects of Python are implemented. We made an overview of the CPython VM, took a look at the CPython compiler, stepped through the CPython source code, studied how the VM executes the bytecode and learned how variables work. In the two most recent posts we focused on the Python object system. We learned what Python objects and Python types are, how they are defined and what determines their behavior. This discussion gave us a good understanding of how Python objects work in general. What we haven't discussed is how particular objects, such as strings, integers and lists, are implemented. In this and several upcoming posts we'll cover the implementations of the most important and most interesting built-in types. The subject of today's post is int.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

Why Python integers are interesting

Integers require no introduction. They are so ubiquitous and seem so basic that you may doubt whether it's worth discussing how they are implemented at all. Yet, Python integers are interesting because they are not just 32-bit or 64-bit integers that CPUs work with natively. Python integers are arbitrary-precision integers, also known as bignums. This means that they can be as large as we want, and their sizes are only limited by the amount of available memory.

Bignums are handy to work with because we don't need to worry about such things as integer overflows and underflows. They are extensively used in fields like cryptography and computer algebra where large numbers arise all the time and must be represented precisely. So, many programming languages have bignums built-in. These include Python, JavaScript, Ruby, Haskell, Erlang, Julia, Racket. Others provide bignums as a part of the standard library. These include Go, Java, C#, D, PHP. Numerous third-party libraries implement bignums. The most popular one is the GNU Multiple Precision Arithmetic Library (GMP). It provides a C API but has bindings for all major languages.

There are a lot of bignum implementations. They're different in detail, but the general approach to implement bignums is the same. Today we'll see what this approach looks like and use CPython's implementation as a reference example. The two main questions we'll have to answer are:

how to represent bignums; and
how to perform arithmetic operations, such as addition and multiplication, on bignums.

We'll also discuss how CPython's implementation compares to others and what CPython does to make integers more efficient.

Bignum representation

Think for a moment how you would represent large integers in your program if you were to implement them yourself. Probably the most obvious way to do that is to store an integer as a sequence of digits, just like we usually write down numbers. For example, the integer 51090942171709440000 could be represented as [5, 1, 0, 9, 0, 9, 4, 2, 1, 7, 1, 7, 0, 9, 4, 4, 0, 0, 0, 0]. This is essentially how bignums are represented in practice. The only important difference is that instead of base 10, much larger bases are used. For example, CPython uses base 2^15 or base 2^30 depending on the platform. What's wrong with base 10? If we represent each digit in a sequence with a single byte but use only 10 out of 256 possible values, it would be very memory-inefficient. We could solve this memory-efficiency problem if we use base 256, so that each digit takes a value between 0 and 255. But still much larger bases are used in practice. The reason for that is because larger base means that numbers have less digits, and the less digits numbers have, the faster arithmetic operations are performed. The base cannot be arbitrary large. It's typically limited by the size of the integers that the CPU can work with. We'll see why this is the case when we discuss bignum arithmetic in the next section. Now let's take a look at how CPython represents bignums.

Everything related to the representation of Python integers can be found in Include/longintrepr.h. Technically, Python integers are instances of PyLongObject, which is defined in Include/longobject.h, but PyLongObject is actually a typedef for struct _longobject that is defined in Include/longintrepr.h:

struct _longobject {
    PyVarObject ob_base; // expansion of PyObject_VAR_HEAD macro
    digit ob_digit[1];
};

This struct extends PyVarObject, which in turn extends PyObject:

typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size; /* Number of items in variable part */
} PyVarObject;

So, besides a reference count and a type that all Python objects have, an integer object has two other members:

ob_size that comes from PyVarObject; and
ob_digit that is defined in struct _longobject.

The ob_digit member is a pointer to an array of digits. On 64-bit platforms, each digit is a 30-bit integer that takes values between 0 and 2^30-1 and is stored as an unsigned 32-bit int (digit is a typedef for uint32_t). On 32-bit platforms, each digit is a 15-bit integer that takes values between 0 and 2^15-1 and is stored as an unsigned 16-bit int (digit is a typedef for unsigned short). To make things concrete, in this post we'll assume that digits are 30 bits long.

The ob_size member is a signed int, whose absolute value tells us the number of digits in the ob_digit array. The sign of ob_size indicates the sign of the integer. Negative ob_size means that the integer is negative. If ob_size is 0, then the integer is 0.

Digits are stored in a little-endian order. The first digit (ob_digit[0]) is the least significant, and the last digit (ob_digit[abs(ob_size)-1]) is the most significant.

Finally, the absolute value of an integer is calculated as follows:

$$val = ob\_digit[0] \times (2 ^{30})^0 + ob\_digit[1] \times (2 ^{30})^1 + \cdots + ob\_digit[|ob\_size| - 1] \times (2 ^{30})^{|ob\_size| - 1}$$

Let's see what all of this means with an example. Suppose we have an integer object that has ob_digit = [3, 5, 1] and ob_size = -3. To compute its value, we can do the following:

$ python -q
>>> base = 2**30
>>> -(3 * base**0 + 5 * base**1 + 1 * base**2)
-1152921509975556099

Now let's do the reverse. Suppose we want to get the bignum representation of the number 51090942171709440000. Here's how we can do that:

>>> x = 51090942171709440000
>>> x % base
952369152
>>> (x // base) % base
337507546
>>> (x // base // base) % base
44
>>> (x // base // base // base) % base
0

So, ob_digit = [952369152, 337507546, 44] and ob_size = 3. Actually, we don't even have to compute the digits, we can get them by inspecting the integer object using the ctypes standard library:

import ctypes


MAX_DIGITS = 1000

# This is a class to map a C `PyLongObject` struct to a Python object
class PyLongObject(ctypes.Structure):
    _fields_ = [
        ("ob_refcnt", ctypes.c_ssize_t),
        ("ob_type", ctypes.c_void_p),
        ("ob_size", ctypes.c_ssize_t),
        ("ob_digit", MAX_DIGITS * ctypes.c_uint32)
    ]


def get_digits(num):
    obj = PyLongObject.from_address(id(num))
    digits_len = abs(obj.ob_size)
    return obj.ob_digit[:digits_len]

>>> from num_digits import get_digits
>>> x = 51090942171709440000
>>> get_digits(x)
[952369152, 337507546, 44]

As you might guess, the representation of bignums is an easy part. The main challenge is to implement arithmetic operations and to implement them efficiently.

Bignum arithmetic

We learned in part 6 that the behavior of a Python object is determined by the object's type. Each member of a type, called slot, is responsible for a particular aspect of the object's behavior. So, to understand how CPython performs arithmetic operations on integers, we need to study the slots of the int type that implement those operations.

In the C code, the int type is called PyLong_Type. It's defined in Objects/longobject.c as follows:

PyTypeObject PyLong_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    "int",                                      /* tp_name */
    offsetof(PyLongObject, ob_digit),           /* tp_basicsize */
    sizeof(digit),                              /* tp_itemsize */
    0,                                          /* tp_dealloc */
    0,                                          /* tp_vectorcall_offset */
    0,                                          /* tp_getattr */
    0,                                          /* tp_setattr */
    0,                                          /* tp_as_async */
    long_to_decimal_string,                     /* tp_repr */
    &long_as_number,                            /* tp_as_number */
    0,                                          /* tp_as_sequence */
    0,                                          /* tp_as_mapping */
    (hashfunc)long_hash,                        /* tp_hash */
    0,                                          /* tp_call */
    0,                                          /* tp_str */
    PyObject_GenericGetAttr,                    /* tp_getattro */
    0,                                          /* tp_setattro */
    0,                                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE |
        Py_TPFLAGS_LONG_SUBCLASS,               /* tp_flags */
    long_doc,                                   /* tp_doc */
    0,                                          /* tp_traverse */
    0,                                          /* tp_clear */
    long_richcompare,                           /* tp_richcompare */
    0,                                          /* tp_weaklistoffset */
    0,                                          /* tp_iter */
    0,                                          /* tp_iternext */
    long_methods,                               /* tp_methods */
    0,                                          /* tp_members */
    long_getset,                                /* tp_getset */
    0,                                          /* tp_base */
    0,                                          /* tp_dict */
    0,                                          /* tp_descr_get */
    0,                                          /* tp_descr_set */
    0,                                          /* tp_dictoffset */
    0,                                          /* tp_init */
    0,                                          /* tp_alloc */
    long_new,                                   /* tp_new */
    PyObject_Del,                               /* tp_free */
};

We can see the long_new() function that creates new integers, the long_hash() function that computes hashes and the implementations of some other important slots. In this post, we'll focus on the slots that implement basic arithmetic operations: addition, subtraction and multiplication. These slots are grouped together in the tp_as_number suite. Here's what it looks like:

static PyNumberMethods long_as_number = {
    (binaryfunc)long_add,       /*nb_add*/
    (binaryfunc)long_sub,       /*nb_subtract*/
    (binaryfunc)long_mul,       /*nb_multiply*/
    long_mod,                   /*nb_remainder*/
    long_divmod,                /*nb_divmod*/
    long_pow,                   /*nb_power*/
    (unaryfunc)long_neg,        /*nb_negative*/
    long_long,                  /*tp_positive*/
    (unaryfunc)long_abs,        /*tp_absolute*/
    (inquiry)long_bool,         /*tp_bool*/
    (unaryfunc)long_invert,     /*nb_invert*/
    long_lshift,                /*nb_lshift*/
    long_rshift,                /*nb_rshift*/
    long_and,                   /*nb_and*/
    long_xor,                   /*nb_xor*/
    long_or,                    /*nb_or*/
    long_long,                  /*nb_int*/
    0,                          /*nb_reserved*/
    long_float,                 /*nb_float*/
    0,                          /* nb_inplace_add */
    0,                          /* nb_inplace_subtract */
    0,                          /* nb_inplace_multiply */
    0,                          /* nb_inplace_remainder */
    0,                          /* nb_inplace_power */
    0,                          /* nb_inplace_lshift */
    0,                          /* nb_inplace_rshift */
    0,                          /* nb_inplace_and */
    0,                          /* nb_inplace_xor */
    0,                          /* nb_inplace_or */
    long_div,                   /* nb_floor_divide */
    long_true_divide,           /* nb_true_divide */
    0,                          /* nb_inplace_floor_divide */
    0,                          /* nb_inplace_true_divide */
    long_long,                  /* nb_index */
};

We'll begin by studying the long_add() function that implements integer addition.

Addition (and subtraction)

First note that a function that adds two integers can be expressed via two other functions that deal with absolute values only:

a function that adds the absolute values of two integers; and
a function that subtracts the absolute values of two integers.

It's possible because:

$$-|a|+(-|b|) = -(|a|+|b|)$$

$$|a|+(-|b|) = |a|-|b|$$

$$-|a|+|b| = |b|-|a|$$

CPython uses these simple identities to express the long_add() function via the x_add() function that adds the absolute values of two integers and the x_sub() function that subtracts the absolute values of two integers:

static PyObject *
long_add(PyLongObject *a, PyLongObject *b)
{
    PyLongObject *z;

    CHECK_BINOP(a, b);

    if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) {
        return PyLong_FromLong(MEDIUM_VALUE(a) + MEDIUM_VALUE(b));
    }
    if (Py_SIZE(a) < 0) {
        if (Py_SIZE(b) < 0) {
            z = x_add(a, b); // -|a|+(-|b|) = -(|a|+|b|)
            if (z != NULL) {
                /* x_add received at least one multiple-digit int,
                   and thus z must be a multiple-digit int.
                   That also means z is not an element of
                   small_ints, so negating it in-place is safe. */
                assert(Py_REFCNT(z) == 1);
                Py_SET_SIZE(z, -(Py_SIZE(z)));
            }
        }
        else
            z = x_sub(b, a); // -|a|+|b| = |b|-|a|
    }
    else {
        if (Py_SIZE(b) < 0)
            z = x_sub(a, b); // |a|+(-|b|) = |a|-|b|
        else
            z = x_add(a, b);
    }
    return (PyObject *)z;
}

So, we need to understand how x_add() and x_sub() are implemented.

It turns out that the best way to add the absolute values of two bignums is the column method taught in the elementary school. We take the least significant digit of the first bignum, take the least significant digit of the second bignum, add them up and write the result to the least significant digit of the output bignum. If the result of the addition doesn't fit into a single digit, we write the result modulo base and remember the carry. Then we take the second least significant digit of the first bignum, the second least significant digit of the second bignum, add them to the carry, write the result modulo base to the second least significant digit of the output bignum and remember the carry. The process continues until no digits are left and the last carry is written to the output bignum. Here's CPython's implementation of this algorithm:

// Some typedefs and macros used in the algorithm:
// typedef uint32_t digit;
// #define PyLong_SHIFT    30
// #define PyLong_BASE     ((digit)1 << PyLong_SHIFT)
// #define PyLong_MASK     ((digit)(PyLong_BASE - 1))


/* Add the absolute values of two integers. */
static PyLongObject *
x_add(PyLongObject *a, PyLongObject *b)
{
    Py_ssize_t size_a = Py_ABS(Py_SIZE(a)), size_b = Py_ABS(Py_SIZE(b));
    PyLongObject *z;
    Py_ssize_t i;
    digit carry = 0;

    /* Ensure a is the larger of the two: */
    if (size_a < size_b) {
        { PyLongObject *temp = a; a = b; b = temp; }
        { Py_ssize_t size_temp = size_a;
            size_a = size_b;
            size_b = size_temp; }
    }
    z = _PyLong_New(size_a+1);
    if (z == NULL)
        return NULL;
    for (i = 0; i < size_b; ++i) {
        carry += a->ob_digit[i] + b->ob_digit[i];
        z->ob_digit[i] = carry & PyLong_MASK;
        carry >>= PyLong_SHIFT;
    }
    for (; i < size_a; ++i) {
        carry += a->ob_digit[i];
        z->ob_digit[i] = carry & PyLong_MASK;
        carry >>= PyLong_SHIFT;
    }
    z->ob_digit[i] = carry;
    return long_normalize(z);
}

First note that Python integers are immutable. CPython returns a new integer as a result of an arithmetic operation. The size of the new integer is initially set to the maximum possible size of the result. Then if, after the operation is performed, some leading digits happen to be zeros, CPython shrinks the size of the integer by calling long_normalize(). In the case of addition, CPython creates a new integer that is one digit longer than the larger operand. Then if, after the operation is performed, the most significant digit of the result happens to be 0, CPython decrements the size of the result by one.

Note also that a digit takes lower 30 bits of a 32-bit int. When we add two digits, we get at most 31-bit integer, and a carry is stored at the bit 30 (counting from 0), so we can easily access it.

Subtraction of the absolute values of two bignums is done in a similar manner except that carrying is replaced with borrowing. We also need to ensure that the first bignum is the larger of the two. If this is not the case, we swap the bignums and change the sign of the result after the subtraction is performed. As it is implemented in CPython, the borrowing is easy because according to the C specification, unsigned ints are a subject to a modular arithmetic:

Otherwise, if the new type is unsigned, the value is converted by repeatedly adding or subtracting one more than the maximum value that can be represented in the new type until the value is in the range of the new type.

This means that when we subtract a larger digit from a smaller one, the maximum possible int is added to the result to get a value in the valid range. For example, 1 - 2 = -1 + (2**32 - 1) = 4294967294. To get the effect of borrowing, we just write the bits 0-29 to the result and check the bit 30 to see if the borrowing happened. Here's how CPython does all of that:

// Some typedefs and macros used in the algorithm:
// typedef uint32_t digit;
// #define PyLong_SHIFT    30
// #define PyLong_BASE     ((digit)1 << PyLong_SHIFT)
// #define PyLong_MASK     ((digit)(PyLong_BASE - 1))


static PyLongObject *
x_sub(PyLongObject *a, PyLongObject *b)
{
    Py_ssize_t size_a = Py_ABS(Py_SIZE(a)), size_b = Py_ABS(Py_SIZE(b));
    PyLongObject *z;
    Py_ssize_t i;
    int sign = 1;
    digit borrow = 0;

    /* Ensure a is the larger of the two: */
    if (size_a < size_b) {
        sign = -1;
        { PyLongObject *temp = a; a = b; b = temp; }
        { Py_ssize_t size_temp = size_a;
            size_a = size_b;
            size_b = size_temp; }
    }
    else if (size_a == size_b) {
        /* Find highest digit where a and b differ: */
        i = size_a;
        while (--i >= 0 && a->ob_digit[i] == b->ob_digit[i])
            ;
        if (i < 0)
            return (PyLongObject *)PyLong_FromLong(0);
        if (a->ob_digit[i] < b->ob_digit[i]) {
            sign = -1;
            { PyLongObject *temp = a; a = b; b = temp; }
        }
        size_a = size_b = i+1;
    }
    z = _PyLong_New(size_a);
    if (z == NULL)
        return NULL;
    for (i = 0; i < size_b; ++i) {
        /* The following assumes unsigned arithmetic
           works module 2**N for some N>PyLong_SHIFT. */
        borrow = a->ob_digit[i] - b->ob_digit[i] - borrow;
        z->ob_digit[i] = borrow & PyLong_MASK;
        borrow >>= PyLong_SHIFT;
        borrow &= 1; /* Keep only one sign bit */
    }
    for (; i < size_a; ++i) {
        borrow = a->ob_digit[i] - borrow;
        z->ob_digit[i] = borrow & PyLong_MASK;
        borrow >>= PyLong_SHIFT;
        borrow &= 1; /* Keep only one sign bit */
    }
    assert(borrow == 0);
    if (sign < 0) {
        Py_SET_SIZE(z, -Py_SIZE(z));
    }
    return maybe_small_long(long_normalize(z));
}

The long_sub() function that implements integer subtraction delegates the work to x_add() and x_sub(), just like long_add() does. Here it is:

static PyObject *
long_sub(PyLongObject *a, PyLongObject *b)
{
    PyLongObject *z;

    CHECK_BINOP(a, b);

    if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) {
        return PyLong_FromLong(MEDIUM_VALUE(a) - MEDIUM_VALUE(b));
    }
    if (Py_SIZE(a) < 0) {
        if (Py_SIZE(b) < 0) {
            z = x_sub(b, a);
        }
        else {
            z = x_add(a, b);
            if (z != NULL) {
                assert(Py_SIZE(z) == 0 || Py_REFCNT(z) == 1);
                Py_SET_SIZE(z, -(Py_SIZE(z)));
            }
        }
    }
    else {
        if (Py_SIZE(b) < 0)
            z = x_add(a, b);
        else
            z = x_sub(a, b);
    }
    return (PyObject *)z;
}

Arithmetic operations on bignums are much slower that the same arithmetic operations on native integers performed by a CPU. In particular, bignum addition is much slower than CPU addition. And it is slower not only because CPU performs multiple arithmetic operations to add two bignums but mainly because bignum addition usually involves multiple memory accesses, and a memory access can be quite expensive, i.e. hundreds of times more costly than an arithmetic operation. Fortunately, CPython employs an optimization to add and subtract small integers faster. This optimization is done by the following check:

static PyObject *
long_sub(PyLongObject *a, PyLongObject *b)
{
    //...

    if (Py_ABS(Py_SIZE(a)) <= 1 && Py_ABS(Py_SIZE(b)) <= 1) {
        // MEDIUM_VALUE macro converts integer to a signed int
        return PyLong_FromLong(MEDIUM_VALUE(a) - MEDIUM_VALUE(b));
    }

    //...
}

If both integers comprise of at most one digit, CPython doesn't call x_add() or x_sub() but simply computes the result with a single operation. If the result also fits into a single digit, no more calculations are required, and bignums are effectively added (or subtracted) as if they were native integers.

Multiplication

There is no silver-bullet algorithm for bignum multiplication. Several algorithms are used in practice because some perform better on relatively small bignums and others perform better on large and extremely large bignums. CPython implements two multiplication algorithms:

the grade-school multiplication algorithm that is used by default; and
the Karatsuba multiplication algorithm that is used when both integers have more than 70 digits.

Wikipedia summarizes the grade-school multiplication algorithm as follows:

Multiply the multiplicand by each digit of the multiplier and then add up all the properly shifted results.

The bignum implementation has one important difference. Instead of storing the results of multiplying by each digit and then adding them up in the end, we add these results to the output bignum as soon as we compute them. The following gif illustrates the idea:

This optimization saves both memory and time. The best way to understand other details of the algorithm is to look at the actual implementation. Here's CPython's one:

// Some typedefs and macros used in the algorithm:
// typedef uint32_t digit;
// typedef uint64_t twodigits;
// #define PyLong_SHIFT    30
// #define PyLong_BASE     ((digit)1 << PyLong_SHIFT)
// #define PyLong_MASK     ((digit)(PyLong_BASE - 1))


/* Grade school multiplication, ignoring the signs.
 * Returns the absolute value of the product, or NULL if error.
 */
static PyLongObject *
x_mul(PyLongObject *a, PyLongObject *b)
{
    PyLongObject *z;
    Py_ssize_t size_a = Py_ABS(Py_SIZE(a));
    Py_ssize_t size_b = Py_ABS(Py_SIZE(b));
    Py_ssize_t i;

    // The size of the result is at most size_a + size_b
    z = _PyLong_New(size_a + size_b);
    if (z == NULL)
        return NULL;

    memset(z->ob_digit, 0, Py_SIZE(z) * sizeof(digit));
    if (a == b) {
        // ... special path for computing a square
    }
    else { /* a is not the same as b -- gradeschool int mult */

        // Iterate over the digits of the multiplier
        for (i = 0; i < size_a; ++i) {
            twodigits carry = 0;
            twodigits f = a->ob_digit[i];
            digit *pz = z->ob_digit + i;
            digit *pb = b->ob_digit;
            digit *pbend = b->ob_digit + size_b;

            // ... signal handling

            // Iterate over the digits of the multiplicand
            while (pb < pbend) {
                carry += *pz + *pb++ * f;
                *pz++ = (digit)(carry & PyLong_MASK);
                carry >>= PyLong_SHIFT;
                assert(carry <= PyLong_MASK);
            }
            if (carry)
                *pz += (digit)(carry & PyLong_MASK);
            assert((carry >> PyLong_SHIFT) == 0);
        }
    }
    return long_normalize(z);
}

Note that when we multiply two 30-bit digits, we can get a 60-bit result. It doesn't fit into a 32-bit int, but this is not a problem since CPython uses 30-bit digits on 64-bit platforms, so 64-bit int can be used to perform the calculation. This convenience is the primary reason why CPython doesn't use larger digit sizes.

The grade-school multiplication algorithm takes $O(n^2)$ time when multiplying two n-digit bignums. The Karatsuba multiplication algorithm takes $O(n^{\log _{2}3})= O(n^{1.584...})$. CPython uses the latter when both operands have more than 70 digits.

The idea of the Karatsuba algorithm is based on two observations. First, observe that each operand can be splited into two parts: one consisting of low-order digits and the other consisting of high-order digits:

$$x = x_1 + x_2 \times base ^ {len(x_1)}$$

Then a multiplication of two n-digit bignums can be replaced with four multiplications of smaller bignums. Assuming the splitting is done so that $len(x_1) = len(y_1)$,

$$xy = (x_1 + x_2 \times base ^ {len(x_1)})(y_1 + y_2 \times base ^ {len(x_1)}) = x_1y_1 + (x_1y_2 + x_2y_1) \times base ^ {len(x_1)} + x_2y_2 \times base ^ {2len(x_1)}$$

The results of the four multiplications can then be calculated recursively. This algorithm, however, also works for $O(n^2)$. We can make it asymptotically faster using the following observation: four multiplications can be replaced with three multiplications at the cost of a few extra additions and subtractions because

$$x_1y_2 + x_2y_1 = (x_1+x_2) (y_1+y_2) - x_1y_1 - x_2y_2$$

So, we only need to calculate $x_1y_1$, $x_2y_2$ and $(x_1+x_2) (y_1+y_2)$. If we split each operand in such a way so that its parts have about half as many digits, then we get an algorithm that works for $O(n^{\log _{2}3})$. Success!

The bignum division is a bit harder to implement. We won't discuss it here, but it's essentially the familiar long division algorithm. Check out Objects/longobject.c to see how bignum division and other arithmetic operations are implemented in CPython. The descriptions of the implemented algorithms can be found in the chapter 14 of the Handbook of Applied Cryptography by Alfred Menezes (it's free!).

CPython's bignums vs other bignum implementations

How fast is CPython's implementation of bignums compared to other implementations? While it's not the easiest task to come up with a totally representative test, we can get some idea. The Benchmarks Game has a pidigits benchmark that measures the performance of bignums in different programming languages. The benchmark asks to implement a specific algorithm for generating digits of pi. You can find the results here. One important thing to know about these results is that the fastest programs use bignums provided by the GMP library and not the bignums provided by the language. If we exclude the programs that use GMP bindings, we get the following results:

#	source	secs
1	Haskell GHC #5 *	0.75
2	Chapel #2 *	0.76
3	Julia *	1.56
4	Go #8	2.68
5	Dart #2	3.33
6	Python 3 #4	3.85
7	OCaml #5	4.36
8	Lisp SBCL #2	5.77
9	Node js #4	6.15
10	Java	7.61
11	Erlang HiPE #3	7.94
12	VW Smalltalk #4	8.02
13	Racket	11.40
14	Free Pascal	14.65
15	Ruby	17.10
16	PHP	5 min

Some languages rely on GMP to implement built-in bignums. They are marked with an asterisk (*).

The results show that CPython's implementation has decent performance. Still, GMP proves that bignums can be implemented even more efficiently. The natural question to ask is: What makes GMP's bignums faster than CPython's bignums? I can think of three main reasons:

Some parts of GMP are written in assembly language. The code is highly optimized for different CPU architectures.
GMP uses bigger digit sizes. It uses 64-bit digits on 64-bit platforms and 32-bit digits on 32-bit platforms. As a result, GMP represents the same integers with fewer digits. Thus, arithmetic operations are performed faster. This is possible due to reason 1. For example, GMP can read the carry flag or use the adc instruction to add with carry. It can also get the 128-bit result of mutiplying two 64-bit integers with the mul instruction.
GMP uses more sophisticated algorithms to do bignum arithmetic. For example, the Karatsuba algorithm is not the asymptotically fastest multiplication algorithm. And GMP implements seven different multiplication algorithms. Which one is used depends on the operands' sizes.

The performance of CPython's bignums should be enough for most applications. When it's not enough, GMP's bignums can be used in a Python program via the gmpy2 module.

For more comments on the results of the pidigits benchmark, check out this article.

Memory usage considerations

Python integers take a considerable amount of memory. Even the smallest integers take 28 bytes on 64-bit platforms:

a reference count ob_refcnt: 8 bytes
a type ob_type: 8 bytes
an object's size ob_size: 8 bytes
ob_digit: 4 bytes.

Allocating a list of a million integers requires allocating the integers themselves plus a million references to them, which is about 35 megabytes in total. Compare it to 4 megabytes required to allocate an array of a million 32-bit ints. So, sometimes it makes sense to use the array module or numpy to store large amounts of homogenous data.

We said before that CPython creates a new integer object on every arithmetic operation. Fortunately, it employs an optimization to allocate small integers only once during the interpreter's lifetime. The integers in the range [-5, 256] are preallocated when CPython starts. Then, when CPython needs to create a new integer object, it first checks if the integer value is in the range [-5, 256] and, if it is in the range, returns the preallocated object. The elimination of extra memory allocations saves both memory and time.

The range [-5, 256] is chosen because the values in this range are extensively used throughout CPython and the Python standard library. For more details on the choice, check out this article.

Conclusion

The design of built-in types has certainly contributed to the Python's success. Python integers serve as an example of a quite efficient and, at the same time, accessible bignum implementation. We made use of this fact today to learn both about Python integers and about bignums. Next time we'll continue to study Python built-in types. Stay tuned to learn about how Python strings work.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #7: how Python attributes work

2020-12-28T07:10:00+00:00

What happens when we get or set an attribute of a Python object? This question is not as simple as it may seem at first. It is true that any experienced Python programmer has a good intuitive understanding of how attributes work, and the documentation helps a lot to strengthen the understanding. Yet, when a really non-trivial question regarding attributes comes up, the intuition fails and the documentation can no longer help. To gain a deep understanding and be able to answer such questions, one has to study how attributes are implemented. That's what we're going to do today.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

A quick refresher

Last time we studied how the Python object system works. Some of the things we've learned in that part are crucial for our current discussion, so let's recall them briefly.

A Python object is an instance of a C struct that has at least two members:

a reference count; and
a pointer to the object's type.

Every object must have a type because the type determines how the object behaves. A type is also a Python object, an instance of the PyTypeObject struct:

// PyTypeObject is a typedef for "struct _typeobject"

struct _typeobject {
    PyVarObject ob_base; // expansion of PyObject_VAR_HEAD macro
    const char *tp_name; /* For printing, in format "<module>.<name>" */
    Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */

    /* Methods to implement standard operations */

    destructor tp_dealloc;
    Py_ssize_t tp_vectorcall_offset;
    getattrfunc tp_getattr;
    setattrfunc tp_setattr;
    PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2)
                                    or tp_reserved (Python 3) */
    reprfunc tp_repr;

    /* Method suites for standard classes */

    PyNumberMethods *tp_as_number;
    PySequenceMethods *tp_as_sequence;
    PyMappingMethods *tp_as_mapping;

    /* More standard operations (here for binary compatibility) */

    hashfunc tp_hash;
    ternaryfunc tp_call;
    reprfunc tp_str;
    getattrofunc tp_getattro;
    setattrofunc tp_setattro;

    /* Functions to access object as input/output buffer */
    PyBufferProcs *tp_as_buffer;

    /* Flags to define presence of optional/expanded features */
    unsigned long tp_flags;

    const char *tp_doc; /* Documentation string */

    /* Assigned meaning in release 2.0 */
    /* call function for all accessible objects */
    traverseproc tp_traverse;

    /* delete references to contained objects */
    inquiry tp_clear;

    /* Assigned meaning in release 2.1 */
    /* rich comparisons */
    richcmpfunc tp_richcompare;

    /* weak reference enabler */
    Py_ssize_t tp_weaklistoffset;

    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;

    /* Attribute descriptor and subclassing stuff */
    struct PyMethodDef *tp_methods;
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
    struct _typeobject *tp_base;
    PyObject *tp_dict;
    descrgetfunc tp_descr_get;
    descrsetfunc tp_descr_set;
    Py_ssize_t tp_dictoffset;
    initproc tp_init;
    allocfunc tp_alloc;
    newfunc tp_new;
    freefunc tp_free; /* Low-level free-memory routine */
    inquiry tp_is_gc; /* For PyObject_IS_GC */
    PyObject *tp_bases;
    PyObject *tp_mro; /* method resolution order */
    PyObject *tp_cache;
    PyObject *tp_subclasses;
    PyObject *tp_weaklist;
    destructor tp_del;

    /* Type attribute cache version tag. Added in version 2.6 */
    unsigned int tp_version_tag;

    destructor tp_finalize;
    vectorcallfunc tp_vectorcall;
};

The members of a type are called slots. Each slot is responsible for a particular aspect of the object's behavior. For example, the tp_call slot of a type specifies what happens when we call the objects of that type. Some slots are grouped together in suites. An example of a suite is the "number" suite tp_as_number. Last time we studied its nb_add slot that specifies how to add objects. This and all other slots are very well described in the docs.

How slots of a type are set depends on how the type is defined. There are two ways to define a type in CPython:

statically; or
dynamically.

A statically defined type is just a statically initialized instance of PyTypeObject. All built-in types are defined statically. Here's, for example, the definition of the float type:

PyTypeObject PyFloat_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    "float",
    sizeof(PyFloatObject),
    0,
    (destructor)float_dealloc,                  /* tp_dealloc */
    0,                                          /* tp_vectorcall_offset */
    0,                                          /* tp_getattr */
    0,                                          /* tp_setattr */
    0,                                          /* tp_as_async */
    (reprfunc)float_repr,                       /* tp_repr */
    &float_as_number,                           /* tp_as_number */
    0,                                          /* tp_as_sequence */
    0,                                          /* tp_as_mapping */
    (hashfunc)float_hash,                       /* tp_hash */
    0,                                          /* tp_call */
    0,                                          /* tp_str */
    PyObject_GenericGetAttr,                    /* tp_getattro */
    0,                                          /* tp_setattro */
    0,                                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,   /* tp_flags */
    float_new__doc__,                           /* tp_doc */
    0,                                          /* tp_traverse */
    0,                                          /* tp_clear */
    float_richcompare,                          /* tp_richcompare */
    0,                                          /* tp_weaklistoffset */
    0,                                          /* tp_iter */
    0,                                          /* tp_iternext */
    float_methods,                              /* tp_methods */
    0,                                          /* tp_members */
    float_getset,                               /* tp_getset */
    0,                                          /* tp_base */
    0,                                          /* tp_dict */
    0,                                          /* tp_descr_get */
    0,                                          /* tp_descr_set */
    0,                                          /* tp_dictoffset */
    0,                                          /* tp_init */
    0,                                          /* tp_alloc */
    float_new,                                  /* tp_new */
};

To dynamically allocate a new type, we call a metatype. A metatype is a type whose instances are types. It determines how types behave. In particular, it creates new type instances. Python has one built-in metatype known as type. It's the metatype of all built-in types. It's also used as the default metatype to create classes. When CPython executes the class statement, it typically calls type() to create the class. We can create a class by calling type() directly as well:

MyClass = type(name, bases, namespace)

The tp_new slot of type is called to create a class. The implementation of this slot is the type_new() function. This function allocates the type object and sets it up.

Slots of a statically defined type are specified explicitly. Slots of a class are set automatically by the metatype. Both statically and dynamically defined types can inherit some slots from its bases.

Some slots are mapped to special methods. If a class defines a special method that corresponds to some slot, CPython automatically sets the slot to the default implementation that calls the special method. This is why we can add objects whose class defines __add__(). CPython does the reverse for a statically defined type. If such a type implements a slot that corresponds to some special method, CPython sets the special method to the implementation that wraps the slot. This is how the int type gets its __add__() special method.

All types must be initialized by calling the PyType_Ready() function. This function does a lot of things. For example, it does slot inheritance and adds special methods based on slots. For a class, PyType_Ready() is called by type_new(). For a statically defined type, PyType_Ready() must be called explicitly. When CPython starts, it calls PyType_Ready() for each built-in type.

With this in mind, let's turn our attention to attributes.

Attributes and the VM

What is an attribute? We might say that an attribute is a variable associated with an object, but it's more than that. It's hard to give a definition that captures all important aspects of attributes. So, instead of starting with a definition, let's start with something we know for sure.

We know for sure that in Python we can do three things with attributes:

get the value of an attribute: value = obj.attr
set an attribute to some value: obj.attr = value
delete an attribute: del obj.attr

What these operations do depends, like any other aspect of the object's behavior, on the object's type. A type has certain slots responsible for getting, setting and deleting attributes. The VM calls these slots to execute the statements like value = obj.attr and obj.attr = value. To see how the VM does that and what these slots are, let's apply the familiar method:

Write a piece of code that gets/sets/deletes an attribute.
Disassemble it to bytecode using the dis module.
Take a look at the implementation of the produced bytecode instructions in ceval.c.

Getting an attribute

Let's first see what the VM does when we get the value of an attribute. The compiler produces the LOAD_ATTR opcode to load the value:

$ echo 'obj.attr' | python -m dis
  1           0 LOAD_NAME                0 (obj)
              2 LOAD_ATTR                1 (attr)
...

And the VM executes this opcode as follows:

case TARGET(LOAD_ATTR): {
    PyObject *name = GETITEM(names, oparg);
    PyObject *owner = TOP();
    PyObject *res = PyObject_GetAttr(owner, name);
    Py_DECREF(owner);
    SET_TOP(res);
    if (res == NULL)
        goto error;
    DISPATCH();
}

We can see that the VM calls the PyObject_GetAttr() function to do the job. Here's what this function does:

PyObject *
PyObject_GetAttr(PyObject *v, PyObject *name)
{
    PyTypeObject *tp = Py_TYPE(v);

    if (!PyUnicode_Check(name)) {
        PyErr_Format(PyExc_TypeError,
                     "attribute name must be string, not '%.200s'",
                     Py_TYPE(name)->tp_name);
        return NULL;
    }
    if (tp->tp_getattro != NULL)
        return (*tp->tp_getattro)(v, name);
    if (tp->tp_getattr != NULL) {
        const char *name_str = PyUnicode_AsUTF8(name);
        if (name_str == NULL)
            return NULL;
        return (*tp->tp_getattr)(v, (char *)name_str);
    }
    PyErr_Format(PyExc_AttributeError,
                 "'%.50s' object has no attribute '%U'",
                 tp->tp_name, name);
    return NULL;
}

It first tries to call the tp_getattro slot of the object's type. If this slot is not implemented, it tries to call the tp_getattr slot. If tp_getattr is not implemented either, it raises AttributeError.

A type implements tp_getattro or tp_getattr or both to support attribute access. According to the documentation, the only difference between them is that tp_getattro takes a Python string as the name of an attribute and tp_getattr takes a C string. Though the choice exists, you won't find types in CPython that implement tp_getattr, because it has been deprecated in favor of tp_getattro.

Setting an attribute

From the VM's perspective, setting an attribute is not much different from getting it. The compiler produces the STORE_ATTR opcode to set an attribute to some value:

$ echo 'obj.attr = value' | python -m dis
  1           0 LOAD_NAME                0 (value)
              2 LOAD_NAME                1 (obj)
              4 STORE_ATTR               2 (attr)
...

And the VM executes STORE_ATTR as follows:

case TARGET(STORE_ATTR): {
    PyObject *name = GETITEM(names, oparg);
    PyObject *owner = TOP();
    PyObject *v = SECOND();
    int err;
    STACK_SHRINK(2);
    err = PyObject_SetAttr(owner, name, v);
    Py_DECREF(v);
    Py_DECREF(owner);
    if (err != 0)
        goto error;
    DISPATCH();
}

We find that PyObject_SetAttr() is the function that does the job:

int
PyObject_SetAttr(PyObject *v, PyObject *name, PyObject *value)
{
    PyTypeObject *tp = Py_TYPE(v);
    int err;

    if (!PyUnicode_Check(name)) {
        PyErr_Format(PyExc_TypeError,
                     "attribute name must be string, not '%.200s'",
                     Py_TYPE(name)->tp_name);
        return -1;
    }
    Py_INCREF(name);

    PyUnicode_InternInPlace(&name);
    if (tp->tp_setattro != NULL) {
        err = (*tp->tp_setattro)(v, name, value);
        Py_DECREF(name);
        return err;
    }
    if (tp->tp_setattr != NULL) {
        const char *name_str = PyUnicode_AsUTF8(name);
        if (name_str == NULL) {
            Py_DECREF(name);
            return -1;
        }
        err = (*tp->tp_setattr)(v, (char *)name_str, value);
        Py_DECREF(name);
        return err;
    }
    Py_DECREF(name);
    _PyObject_ASSERT(name, Py_REFCNT(name) >= 1);
    if (tp->tp_getattr == NULL && tp->tp_getattro == NULL)
        PyErr_Format(PyExc_TypeError,
                     "'%.100s' object has no attributes "
                     "(%s .%U)",
                     tp->tp_name,
                     value==NULL ? "del" : "assign to",
                     name);
    else
        PyErr_Format(PyExc_TypeError,
                     "'%.100s' object has only read-only attributes "
                     "(%s .%U)",
                     tp->tp_name,
                     value==NULL ? "del" : "assign to",
                     name);
    return -1;
}

This function calls the tp_setattro and tp_setattr slots the same way as PyObject_GetAttr() calls tp_getattro and tp_getattr. The tp_setattro slot comes in pair with tp_getattro, and tp_setattr comes in pair with tp_getattr. Just like tp_getattr, tp_setattr is deprecated.

Note that PyObject_SetAttr() checks whether a type defines tp_getattro or tp_getattr. A type must implement attribute access to support attribute assignment.

Deleting an attribute

Interestingly, a type has no special slot for deleting an attribute. What then specifies how to delete an attribute? Let's see. The compiler produces the DELETE_ATTR opcode to delete an attribute:

$ echo 'del obj.attr' | python -m dis
  1           0 LOAD_NAME                0 (obj)
              2 DELETE_ATTR              1 (attr)

The way the VM executes this opcode reveals the answer:

case TARGET(DELETE_ATTR): {
    PyObject *name = GETITEM(names, oparg);
    PyObject *owner = POP();
    int err;
    err = PyObject_SetAttr(owner, name, (PyObject *)NULL);
    Py_DECREF(owner);
    if (err != 0)
        goto error;
    DISPATCH();
}

To delete an attribute, the VM calls the same PyObject_SetAttr() function that it calls to set an attribute, so the same tp_setattro slot is responsible for deleting attributes. But how does it know which of two operations to perform? The NULL value indicates that the attribute should be deleted.

As this section shows, the tp_getattro and tp_setattro slots determine how attributes of an object work. The next question that comes to mind is: How are these slots implemented?

Slots implementations

Any function of the appropriate signature can be an implementation of tp_getattro and tp_setattro. A type can implement these slots in an absolutely arbitrary way. Fortunately, we need to study only a few implementations to understand how Python attributes work. This is because most types use the same generic implementation.

The generic functions for getting and setting attributes are PyObject_GenericGetAttr() and PyObject_GenericSetAttr(). All classes use them by default. Most built-in types specify them as slots implementations explicitly or inherit them from object that also uses the generic implementation.

In this post, we'll focus on the generic implementation, since it's basically what we mean by Python attributes. We'll also discuss two important cases when the generic implementation is not used. The first case is type. It implements the tp_getattro and tp_setattro slots in its own way, though its implementation is quite similar to the generic one. The second case is any class that customizes attribute access and assignment by defining the __getattribute__(), __getattr__(), __setattr__() and __delattr__() special methods. CPython sets the tp_getattro and tp_setattro slots of such a class to functions that call those methods.

Generic attribute management

The PyObject_GenericGetAttr() and PyObject_GenericSetAttr() functions implement the behavior of attributes that we're all accustomed to. When we set an attribute of an object to some value, CPython puts the value in the object's dictionary:

$ python -q
>>> class A:
...     pass
... 
>>> a = A()
>>> a.__dict__
{}
>>> a.x = 'instance attribute'
>>> a.__dict__
{'x': 'instance attribute'}

When we try to get the value of the attribute, CPython loads it from the object's dictionary:

>>> a.x
'instance attribute'

If the object's dictionary doesn't contain the attribute, CPython loads the value from the type's dictionary:

>>> A.y = 'class attribute'
>>> a.y
'class attribute'

If the type's dictionary doesn't contain the attribute either, CPython searches for the value in the dictionaries of the type's parents:

>>> class B(A): # note the inheritance
...     pass
... 
>>> b = B()
>>> b.y
'class attribute'

So, an attribute of an object is one of two things:

an instance variable; or
a type variable.

Instance variables are stored in the object's dictionary, and type variables are stored in the type's dictionary and in the dictionaries of the type's parents. To set an attribute to some value, CPython simply updates the object's dictionary. To get the value of an attribute, CPython searches for it first in the object's dictionary and then in the type's dictionary and in the dictionaries of the type's parents. The order in which CPython iterates over the types when it searches for the value is the Method Resolution Order (MRO).

Python attributes would be as simple as that if there were no descriptors.

Descriptors

Technically, a descriptor is a Python object whose type implements certain slots: tp_descr_get or tp_descr_set or both. Essentially, a descriptor is a Python object that, when used as an attribute, controls what happens we get, set or delete it. If PyObject_GenericGetAttr() finds that the attribute value is a descriptor whose type implements tp_descr_get, it doesn't just return the value as it normally does but calls tp_descr_get and returns the result of this call. The tp_descr_get slot takes three parameters: the descriptor itself, the object whose attribute is being looked up and the object's type. It's up to tp_descr_get to decide what to do with the parameters and what to return. Similarly, PyObject_GenericSetAttr() looks up the current attribute value. If it finds that the value is a descriptor whose type implements tp_descr_set, it calls tp_descr_set instead of just updating the object's dictionary. The arguments passed to tp_descr_set are the descriptor, the object, and the new attribute value. To delete an attribute, PyObject_GenericSetAttr() calls tp_descr_set with the new attribute value set to NULL.

On one side, descriptors make Python attributes a bit complex. On the other side, descriptors make Python attributes powerful. As Python's glossary says,

Understanding descriptors is a key to a deep understanding of Python because they are the basis for many features including functions, methods, properties, class methods, static methods, and reference to super classes.

Let's revise one important use case of descriptors that we discussed in the previous part: methods.

A function put in the type's dictionary works not like an ordinary function but like a method. That is, we don't need to explicitly pass the first argument when we call it:

>>> A.f = lambda self: self
>>> a.f()
<__main__.A object at 0x108a20d60>

The a.f attribute not only works like a method, it is a method:

>>> a.f
<bound method <lambda> of <__main__.A object at 0x108a20d60>>

However, if we look up the value of 'f' in the type's dictionary, we'll get the original function:

>>> A.__dict__['f']
<function <lambda> at 0x108a4ca60>

CPython returns not the value stored in the dictionary but something else. This is because functions are descriptors. The function type implements the tp_descr_get slot, so PyObject_GenericGetAttr() calls this slot and returns the result of the call. The result of the call is a method object that stores both the function and the instance. When we call a method object, the instance is prepended to the list of arguments and the function gets invoked.

Descriptors have their special behavior only when they are used as type variables. When they are used as instance variables, they behave like ordinary objects. For example, a function put in the object's dictionary does not become a method:

>>> a.g = lambda self: self
>>> a.g
<function <lambda> at 0x108a4cc10>

Apparently, the language designers haven't found a case when using a descriptor as an instance variable would be a good idea. A nice consequence of this decision is that instance variables are very straightforward. They are just data.

The function type is an example of a built-in descriptor type. We can also define our own descriptors. To do that, we create a class that implements the descriptor protocol: the __get__(), __set__() and __delete__() special methods:

>>> class DescrClass:
...     def __get__(self, obj, type=None):
...             print('I can do anything')
...             return self
...
>>> A.descr_attr = DescrClass()
>>> a.descr_attr 
I can do anything
<__main__.DescrClass object at 0x108b458e0>

If a class defines __get__(), CPython sets its tp_descr_get slot to the function that calls that method. If a class defines __set__() or __delete__(), CPython sets its tp_descr_set slot to the function that calls __delete__() when the value is NULL and calls __set__() otherwise.

If you wonder why anyone would want to define their our descriptors in the first place, check out the excellent Descriptor HowTo Guide by Raymond Hettinger.

Our goal is to study the actual algorithms for getting and setting attributes. Descriptors is one prerequisite for that. Another is the understanding of what the object's dictionary and the type's dictionary really are.

Object's dictionary and type's dictionary

An object's dictionary is a dictionary in which instance variables are stored. Every object of a type keeps a pointer to its own dictionary. For example, every function object has the func_dict member for that purpose:

typedef struct {
    // ...
    PyObject *func_dict;        /* The __dict__ attribute, a dict or NULL */
    // ...
} PyFunctionObject;

To tell CPython which member of an object is the pointer to the object's dictionary, the object's type specifies the offset of this member using the tp_dictoffset slot. Here's how the function type does this:

PyTypeObject PyFunction_Type = {
    // ...
    offsetof(PyFunctionObject, func_dict),      /* tp_dictoffset */
    // ... 
};

A positive value of tp_dictoffset specifies an offset from the start of the object's struct. A negative value specifies an offset from the end of the struct. The zero offset means that the objects of the type don't have dictionaries. Integers, for example, are such objects:

>>> (12).__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'int' object has no attribute '__dict__'

We can assure ourselves that tp_dictoffset of the int type is set to 0 by checking the __dictoffset__ attribute:

>>> int.__dictoffset__
0

Classes usually have a non-zero tp_dictoffset. The only exception is classes that define the __slots__ attribute. This attribute is an optimization. We'll cover the essentials first and discuss __slots__ later.

A type's dictionary is a dictionary of a type object. Just like the func_dict member of a function points to the function's dictionary, the tp_dict slot of a type points to the type's dictionary. The crucial difference between the dictionary of an ordinary object and the dictionary of a type is that CPython knows about tp_dict, so it can avoid locating the dictionary of a type via tp_dictoffset. Handling the dictionary of a type in a general way would introduce an additional level of indirection and wouldn't brings much benefit.

Now, when we know what descriptors are and where attributes are stored, we're ready to see what the PyObject_GenericGetAttr() and PyObject_GenericSetAttr() functions do.

PyObject_GenericSetAttr()

We begin with PyObject_GenericSetAttr(), a function whose job is set an attribute to a given value. This function turns out to be a thin wrapper around another function:

int
PyObject_GenericSetAttr(PyObject *obj, PyObject *name, PyObject *value)
{
    return _PyObject_GenericSetAttrWithDict(obj, name, value, NULL);
}

And that function actually does the work:

int
_PyObject_GenericSetAttrWithDict(PyObject *obj, PyObject *name,
                                 PyObject *value, PyObject *dict)
{
    PyTypeObject *tp = Py_TYPE(obj);
    PyObject *descr;
    descrsetfunc f;
    PyObject **dictptr;
    int res = -1;

    if (!PyUnicode_Check(name)){
        PyErr_Format(PyExc_TypeError,
                     "attribute name must be string, not '%.200s'",
                     Py_TYPE(name)->tp_name);
        return -1;
    }

    if (tp->tp_dict == NULL && PyType_Ready(tp) < 0)
        return -1;

    Py_INCREF(name);

    // Look up the current attribute value
    // in the type's dict and in the parent's dicts using the MRO.
    descr = _PyType_Lookup(tp, name);

    // If found a descriptor that implements `tp_descr_set`, call this slot.
    if (descr != NULL) {
        Py_INCREF(descr);
        f = Py_TYPE(descr)->tp_descr_set;
        if (f != NULL) {
            res = f(descr, obj, value);
            goto done;
        }
    }

    // `PyObject_GenericSetAttr()` calls us with `dict` set to `NULL`.
    // So, `if` will be executed.
    if (dict == NULL) {
        // Get the object's dict.
        dictptr = _PyObject_GetDictPtr(obj);
        if (dictptr == NULL) {
            if (descr == NULL) {
                PyErr_Format(PyExc_AttributeError,
                             "'%.100s' object has no attribute '%U'",
                             tp->tp_name, name);
            }
            else {
                PyErr_Format(PyExc_AttributeError,
                             "'%.50s' object attribute '%U' is read-only",
                             tp->tp_name, name);
            }
            goto done;
        }
        // Update the object's dict with the new value.
        // If `value` is `NULL`, delete the attribute from the dict.
        res = _PyObjectDict_SetItem(tp, dictptr, name, value);
    }
    else {
        Py_INCREF(dict);
        if (value == NULL)
            res = PyDict_DelItem(dict, name);
        else
            res = PyDict_SetItem(dict, name, value);
        Py_DECREF(dict);
    }
    if (res < 0 && PyErr_ExceptionMatches(PyExc_KeyError))
        PyErr_SetObject(PyExc_AttributeError, name);

  done:
    Py_XDECREF(descr);
    Py_DECREF(name);
    return res;
}

Despite its length, the function implements a simple algorithm:

Search for the attribute value among type variables. The order of the search is the MRO.
If the value is a descriptor whose type implements the tp_descr_set slot, call the slot.
Otherwise, update the object's dictionary with the new value.

We haven't discussed the descriptor types that implement the tp_descr_set slot, so you may wonder why we need them at all. Consider Python's property(). The following example from the docs demonstrates its canonical usage to create a managed attribute:

class C:
    def __init__(self):
        self._x = None

    def getx(self):
        return self._x

    def setx(self, value):
        self._x = value

    def delx(self):
        del self._x

    x = property(getx, setx, delx, "I'm the 'x' property.")

If c is an instance of C, c.x will invoke the getter, c.x = value will invoke the setter and del c.x the deleter.

How does property() work? The answer is simple: it's a descriptor type. It implements both the tp_descr_get and tp_descr_set slots that call the specified functions.

The example from the docs is only a framework and doesn't do much. However, it can easily be extended to do something useful. For example, we can write a setter that performs some validation of the new attribute value.

PyObject_GenericGetAttr()

Getting the value of an attribute is a bit more complicated than setting it. Let's see by how much. The PyObject_GenericGetAttr() function also delegates the work to another function:

PyObject *
PyObject_GenericGetAttr(PyObject *obj, PyObject *name)
{
    return _PyObject_GenericGetAttrWithDict(obj, name, NULL, 0);
}

And here's what that function does:

PyObject *
_PyObject_GenericGetAttrWithDict(PyObject *obj, PyObject *name,
                                 PyObject *dict, int suppress)
{
    /* Make sure the logic of _PyObject_GetMethod is in sync with
       this method.

       When suppress=1, this function suppress AttributeError.
    */

    PyTypeObject *tp = Py_TYPE(obj);
    PyObject *descr = NULL;
    PyObject *res = NULL;
    descrgetfunc f;
    Py_ssize_t dictoffset;
    PyObject **dictptr;

    if (!PyUnicode_Check(name)){
        PyErr_Format(PyExc_TypeError,
                     "attribute name must be string, not '%.200s'",
                     Py_TYPE(name)->tp_name);
        return NULL;
    }
    Py_INCREF(name);

    if (tp->tp_dict == NULL) {
        if (PyType_Ready(tp) < 0)
            goto done;
    }

    // Look up the attribute value
    // in the type's dict and in the parent's dicts using the MRO.
    descr = _PyType_Lookup(tp, name);

    // Check if the value is a descriptor that implements:
    // * `tp_descr_get`; and
    // * `tp_descr_set` (data descriptor)
    // In this case, call `tp_descr_get`
    f = NULL;
    if (descr != NULL) {
        Py_INCREF(descr);
        f = Py_TYPE(descr)->tp_descr_get;
        if (f != NULL && PyDescr_IsData(descr)) {
            res = f(descr, obj, (PyObject *)Py_TYPE(obj));
            if (res == NULL && suppress &&
                    PyErr_ExceptionMatches(PyExc_AttributeError)) {
                PyErr_Clear();
            }
            goto done;
        }
    }

    // Look up the attribute value in the object's dict
    // Return if found one
    if (dict == NULL) {
        /* Inline _PyObject_GetDictPtr */
        dictoffset = tp->tp_dictoffset;
        if (dictoffset != 0) {
            if (dictoffset < 0) {
                Py_ssize_t tsize = Py_SIZE(obj);
                if (tsize < 0) {
                    tsize = -tsize;
                }
                size_t size = _PyObject_VAR_SIZE(tp, tsize);
                _PyObject_ASSERT(obj, size <= PY_SSIZE_T_MAX);

                dictoffset += (Py_ssize_t)size;
                _PyObject_ASSERT(obj, dictoffset > 0);
                _PyObject_ASSERT(obj, dictoffset % SIZEOF_VOID_P == 0);
            }
            dictptr = (PyObject **) ((char *)obj + dictoffset);
            dict = *dictptr;
        }
    }
    if (dict != NULL) {
        Py_INCREF(dict);
        res = PyDict_GetItemWithError(dict, name);
        if (res != NULL) {
            Py_INCREF(res);
            Py_DECREF(dict);
            goto done;
        }
        else {
            Py_DECREF(dict);
            if (PyErr_Occurred()) {
                if (suppress && PyErr_ExceptionMatches(PyExc_AttributeError)) {
                    PyErr_Clear();
                }
                else {
                    goto done;
                }
            }
        }
    }

    // If _PyType_Lookup found a non-data desciptor,
    // call its `tp_descr_get`
    if (f != NULL) {
        res = f(descr, obj, (PyObject *)Py_TYPE(obj));
        if (res == NULL && suppress &&
                PyErr_ExceptionMatches(PyExc_AttributeError)) {
            PyErr_Clear();
        }
        goto done;
    }

    // If _PyType_Lookup found some value,
    // return it
    if (descr != NULL) {
        res = descr;
        descr = NULL;
        goto done;
    }

    if (!suppress) {
        PyErr_Format(PyExc_AttributeError,
                     "'%.50s' object has no attribute '%U'",
                     tp->tp_name, name);
    }
  done:
    Py_XDECREF(descr);
    Py_DECREF(name);
    return res;
}

The major steps of this algorithm are:

Search for the attribute value among type variables. The order of the search is the MRO.
If the value is a data descriptor whose type implements the tp_descr_get slot, call this slot and return the result of the call. Otherwise, remember the value and continue. A data descriptor is a descriptor whose type implements the tp_descr_set slot.
Locate the object's dictionary using tp_dictoffset. If the dictionary contains the value, return it.
If the value from step 2 is a descriptor whose type implements the tp_descr_get slot, call this slot and return the result of the call.
Return the value from step 2. The value can be NULL.

Since an attribute can be both an instance variable and a type variable, CPython must decide which one takes precedence over the other. What the algorithm does is essentially implement a certain order of precedence. This order is:

type data descriptors
instance variables
type non-data descriptors and other type variables.

The natural question to ask is: Why does it implement this particular order? More specifically, why do data descriptors take precedence over instance variables but non-data descriptors don't? First of all, note that some descriptors must take precedence over instance variables in order for attributes to work as expected. An example of such a descriptor is the __dict__ attribute of an object. You won't find it in the object's dictionary, because it's a data descriptor stored in the type's dictionary:

>>> a.__dict__['__dict__']
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
KeyError: '__dict__'
>>> A.__dict__['__dict__']
<attribute '__dict__' of 'A' objects>
>>> a.__dict__ is A.__dict__['__dict__'].__get__(a)
True

The tp_descr_get slot of this descriptor returns the object's dictionary located at tp_dictoffset. Now suppose that data descriptors don't take precedence over instance variables. What would happened then if we put '__dict__' in the object's dictionary and assigned it some other dictionary:

>>> a.__dict__['__dict__'] = {}

The a.__dict__ attribute would return not the object's dictionary but the dictionary we assigned! That would be totally unexpected for someone who relies on __dict__. Fortunately, data descriptors do take precedence over instance variables, so we get the object's dictionary:

>>> a.__dict__
{'x': 'instance attribute', 'g': <function <lambda> at 0x108a4cc10>, '__dict__': {}}

Non-data descriptors don't take precedence over instance variables, so that most of the time instance variables have a priority over type variables. Of course, the existing order of precedence is one of many design choices. Guido van Rossum explains the reasoning behind it in PEP 252:

In the more complicated case, there's a conflict between names stored in the instance dict and names stored in the type dict. If both dicts have an entry with the same key, which one should we return? Looking at classic Python for guidance, I find conflicting rules: for class instances, the instance dict overrides the class dict, except for the special attributes (like __dict__ and __class__), which have priority over the instance dict.

I resolved this with the following set of rules, implemented in PyObject_GenericGetAttr(): ...

Why is the __dict__ attribute implemented as a descriptor in the first place? Making it an instance variable would lead to the same problem. It would be possible to override the __dict__ attribute and hardly anyone wants to have this possibility.

We've learned how attributes of an ordinary object work. Let's see now how attributes of a type work.

Metatype attribute management

Basically, attributes of a type work just like attributes of an ordinary object. When we set an attribute of a type to some value, CPython puts the value in the type's dictionary:

>>> B.x = 'class attribute'
>>> B.__dict__
mappingproxy({'__module__': '__main__', '__doc__': None, 'x': 'class attribute'})

When we get the value of the attribute, CPython loads it from the type's dictionary:

>>> B.x
'class attribute'

If the type's dictionary doesn't contain the attribute, CPython loads the value from the metatype's dictionary:

>>> B.__class__
<class 'type'>
>>> B.__class__ is object.__class__
True

Finally, if the metatype's dictionary doesn't contain the attribute either, CPython searches for the value in the dictionaries of the metatype's parents...

The analogy with the generic implementation is clear. We just change the words "object" with "type" and "type" with "metatype". However, type implements the tp_getattro and tp_setattro slots in its own way. Why? Let's take a look at the code.

type_setattro()

We begin with the type_setattro() function, an implementation of the tp_setattro slot:

static int
type_setattro(PyTypeObject *type, PyObject *name, PyObject *value)
{
    int res;
    if (!(type->tp_flags & Py_TPFLAGS_HEAPTYPE)) {
        PyErr_Format(
            PyExc_TypeError,
            "can't set attributes of built-in/extension type '%s'",
            type->tp_name);
        return -1;
    }
    if (PyUnicode_Check(name)) {
        if (PyUnicode_CheckExact(name)) {
            if (PyUnicode_READY(name) == -1)
                return -1;
            Py_INCREF(name);
        }
        else {
            name = _PyUnicode_Copy(name);
            if (name == NULL)
                return -1;
        }
        // ... ifdef
    }
    else {
        /* Will fail in _PyObject_GenericSetAttrWithDict. */
        Py_INCREF(name);
    }

    // Call the generic set function.
    res = _PyObject_GenericSetAttrWithDict((PyObject *)type, name, value, NULL);
    if (res == 0) {
        PyType_Modified(type);

        // If attribute is a special method,
        // add update the corresponding slots.
        if (is_dunder_name(name)) {
            res = update_slot(type, name);
        }
        assert(_PyType_CheckConsistency(type));
    }
    Py_DECREF(name);
    return res;
}

This function calls generic _PyObject_GenericSetAttrWithDict() to set the attribute value, but it does something else too. First, it ensures that the type is not a statically defined type, because such types are designed to be immutable:

>>> int.x = 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: can't set attributes of built-in/extension type 'int'

It also checks whether the attribute is a special method. If the attribute is a special method, it updates the slots corresponding to that special method. For example, if we define the __add__() special method on an existing class, it will set the nb_add slot of the class to the default implementation that calls the method. Due to this mechanism, special methods and slots of a class are kept in sync.

type_getattro()

The type_getattro() function, an implementation of the tp_getattro slot, doesn't call the generic function but resembles it:

/* This is similar to PyObject_GenericGetAttr(),
   but uses _PyType_Lookup() instead of just looking in type->tp_dict. */
static PyObject *
type_getattro(PyTypeObject *type, PyObject *name)
{
    PyTypeObject *metatype = Py_TYPE(type);
    PyObject *meta_attribute, *attribute;
    descrgetfunc meta_get;
    PyObject* res;

    if (!PyUnicode_Check(name)) {
        PyErr_Format(PyExc_TypeError,
                     "attribute name must be string, not '%.200s'",
                     Py_TYPE(name)->tp_name);
        return NULL;
    }

    /* Initialize this type (we'll assume the metatype is initialized) */
    if (type->tp_dict == NULL) {
        if (PyType_Ready(type) < 0)
            return NULL;
    }

    /* No readable descriptor found yet */
    meta_get = NULL;

    /* Look for the attribute in the metatype */
    meta_attribute = _PyType_Lookup(metatype, name);

    if (meta_attribute != NULL) {
        Py_INCREF(meta_attribute);
        meta_get = Py_TYPE(meta_attribute)->tp_descr_get;

        if (meta_get != NULL && PyDescr_IsData(meta_attribute)) {
            /* Data descriptors implement tp_descr_set to intercept
             * writes. Assume the attribute is not overridden in
             * type's tp_dict (and bases): call the descriptor now.
             */
            res = meta_get(meta_attribute, (PyObject *)type,
                           (PyObject *)metatype);
            Py_DECREF(meta_attribute);
            return res;
        }
    }

    /* No data descriptor found on metatype. Look in tp_dict of this
     * type and its bases */
    attribute = _PyType_Lookup(type, name);
    if (attribute != NULL) {
        /* Implement descriptor functionality, if any */
        Py_INCREF(attribute);
        descrgetfunc local_get = Py_TYPE(attribute)->tp_descr_get;

        Py_XDECREF(meta_attribute);

        if (local_get != NULL) {
            /* NULL 2nd argument indicates the descriptor was
             * found on the target object itself (or a base)  */
            res = local_get(attribute, (PyObject *)NULL,
                            (PyObject *)type);
            Py_DECREF(attribute);
            return res;
        }

        return attribute;
    }

    /* No attribute found in local __dict__ (or bases): use the
     * descriptor from the metatype, if any */
    if (meta_get != NULL) {
        PyObject *res;
        res = meta_get(meta_attribute, (PyObject *)type,
                       (PyObject *)metatype);
        Py_DECREF(meta_attribute);
        return res;
    }

    /* If an ordinary attribute was found on the metatype, return it now */
    if (meta_attribute != NULL) {
        return meta_attribute;
    }

    /* Give up */
    PyErr_Format(PyExc_AttributeError,
                 "type object '%.50s' has no attribute '%U'",
                 type->tp_name, name);
    return NULL;
}

This algorithm indeed repeats the logic of the generic implementation but with three important differences:

It gets the type's dictionary via tp_dict. The generic implementation would try to locate it using metatype's tp_dictoffset.
It searches for the type variable not only in the type's dictionary but also in the dictionaries of the type's parents. The generic implementation would handle a type like an ordinary object that has no notions of inheritance.
It supports type descriptors. The generic implementation would support only metatype descriptors.

As a result, we have the following order of precedence:

metatype data descriptors
type descriptors and other type variables
metatype non-data descriptors and other metatype variables.

That's how type implements the tp_getattro and tp_setattro slots. Since type is the metatype of all built-in types and the metatype of all classes by default, attributes of most types work according to this implementation. Classes themselves, as we've already said, use the generic implementation by default. If we want to change the behavior of attributes of a class instance or the behavior of attributes of a class, we need to define a new class or a new metaclass that uses a custom implementation. Python provides an easy way to do this.

Custom attribute management

The tp_getattro and tp_setattro slots of a class are initially set by the type_new() function that creates new classes. The generic implementation is its default choice. A class can customize attribute access, assignment and deletion by defining the __getattribute__(), __getattr__(), __setattr__() and __delattr__() special methods. When a class defines __setattr__() or __delattr__(), its tp_setattro slot is set to the slot_tp_setattro() function. When a class defines __getattribute__() or __getattr__(), its tp_getattro slot is set to the slot_tp_getattr_hook() function.

The __setattr__() and __delattr__() special methods are quite straightforward. Basically, they allow us to implement the tp_setattro slot in Python. The slot_tp_setattro() function simply calls __delattr__(instance, attr_name) or __setattr__(instance, attr_name, value) depending on whether the value is NULL or not:

static int
slot_tp_setattro(PyObject *self, PyObject *name, PyObject *value)
{
    PyObject *stack[3];
    PyObject *res;
    _Py_IDENTIFIER(__delattr__);
    _Py_IDENTIFIER(__setattr__);

    stack[0] = self;
    stack[1] = name;
    if (value == NULL) {
        res = vectorcall_method(&PyId___delattr__, stack, 2);
    }
    else {
        stack[2] = value;
        res = vectorcall_method(&PyId___setattr__, stack, 3);
    }
    if (res == NULL)
        return -1;
    Py_DECREF(res);
    return 0;
}

The __getattribute__() and __getattr__() special methods provide a way to customize attribute access. Both take an instance and an attribute name as their parameters and return the attribute value. The difference between them is when they get invoked.

The __getattribute__() special method is the analog of __setattr__() and __delattr__() for getting the value of an attribute. It's invoked instead of the generic function. The __getattr__() special method is used in tandem with __getattribute__() or the generic function. It's invoked when __getattribute__() or the generic function raise AttributeError. This logic is implemented in the slot_tp_getattr_hook() function:

static PyObject *
slot_tp_getattr_hook(PyObject *self, PyObject *name)
{
    PyTypeObject *tp = Py_TYPE(self);
    PyObject *getattr, *getattribute, *res;
    _Py_IDENTIFIER(__getattr__);

    getattr = _PyType_LookupId(tp, &PyId___getattr__);
    if (getattr == NULL) {
        /* No __getattr__ hook: use a simpler dispatcher */
        tp->tp_getattro = slot_tp_getattro;
        return slot_tp_getattro(self, name);
    }
    Py_INCREF(getattr);

    getattribute = _PyType_LookupId(tp, &PyId___getattribute__);
    if (getattribute == NULL ||
        (Py_IS_TYPE(getattribute, &PyWrapperDescr_Type) &&
         ((PyWrapperDescrObject *)getattribute)->d_wrapped ==
         (void *)PyObject_GenericGetAttr))
        res = PyObject_GenericGetAttr(self, name);
    else {
        Py_INCREF(getattribute);
        res = call_attribute(self, getattribute, name);
        Py_DECREF(getattribute);
    }
    if (res == NULL && PyErr_ExceptionMatches(PyExc_AttributeError)) {
        PyErr_Clear();
        res = call_attribute(self, getattr, name);
    }
    Py_DECREF(getattr);
    return res;
}

Let's translate the code to English:

If the class doesn't define __getattr__(), first set its tp_getattro slot to another function, slot_tp_getattro(), then call this function and return the result of the call.
If the class defines __getattribute__(), call it. Otherwise call generic PyObject_GenericGetAttr().
If the call from the previous step raised AttributeError, call ___getattr__().
Return the result of the last call.

The slot_tp_getattro() function is an implementation of the tp_getattro slot that CPython uses when a class defines __getattribute__() but not __getattr__(). This function just calls __getattribute__():

static PyObject *
slot_tp_getattro(PyObject *self, PyObject *name)
{
    PyObject *stack[2] = {self, name};
    return vectorcall_method(&PyId___getattribute__, stack, 2);
}

Why doesn't CPython set the tp_getattro slot to the slot_tp_getattro() function instead of slot_tp_getattr_hook() initially? The reason is the design of the mechanism that maps special methods to slots. It requires special methods that map to the same slot to provide the same implementation for that slot. And the __getattribute__() and __getattr__() special methods map to the same tp_getattro slot.

Even a perfect understanding of how the __getattribute__() and __getattr__() special methods work doesn't tell us why we need both of them. Theoretically, __getattribute__() should be enough to make attribute access work in any way we want. Sometimes, though, it's more convenient to define __getattr__(). For example, the standard imaplib module provides the IMAP4 class that can be used to talk to a IMAP4 server. To issue the commands, we call the class methods. Both lowercase and uppercase versions of the commands work:

>>> from imaplib import IMAP4_SSL # subclass of IMAP4
>>> M = IMAP4_SSL("imap.gmail.com", port=993)
>>> M.noop()
('OK', [b'Nothing Accomplished. p11mb154389070lti'])
>>> M.NOOP()
('OK', [b'Nothing Accomplished. p11mb154389070lti'])

To support this feature, IMAP4 defines __getattr__():

class IMAP4:
    # ...

    def __getattr__(self, attr):
        #       Allow UPPERCASE variants of IMAP4 command methods.
        if attr in Commands:
            return getattr(self, attr.lower())
        raise AttributeError("Unknown IMAP4 command: '%s'" % attr)

    # ...

Achieving the same result with __getattribute__() would require us to explicitly call the generic function first: object.__getattribute__(self, attr). Is this inconvenient enough to introduce another special method? Perhaps. The real reason, tough, why both __getattribute__() and __getattr__() exist is historical. The __getattribute__() special method was introduced in Python 2.2 when __getattr__() had already existed. Here's how Guido van Rossum explained the need for the new feature:

The __getattr__() method is not really the implementation for the get-attribute operation; it is a hook that only gets invoked when an attribute cannot be found by normal means. This has often been cited as a shortcoming - some class designs have a legitimate need for a get-attribute method that gets called for all attribute references, and this problem is solved now by making __getattribute__() available.

What happens when we get or set an attribute of a Python object? I think we gave a detailed answer to this question. The answer, however, doesn't cover some important aspects of Python attributes. Let's discuss them as well.

Loading methods

We saw that a function object is a descriptor that returns a method object when we bound it to an instance:

>>> a.f
<bound method <lambda> of <__main__.A object at 0x108a20d60>>

But is it really necessary to create a method object if all we need to do is to call the method? Couldn't CPython just call the original function with the instance as the first argument? It could. In fact, this is exactly what CPython does.

When the compiler sees the method call with positional arguments like obj.method(arg1,...,argN), it does not produce the LOAD_ATTR opcode to load the method and the CALL_FUNCTION opcode to call the method. Instead, it produces a pair of the LOAD_METHOD and CALL_METHOD opcodes:

$ echo 'obj.method()' | python -m dis
  1           0 LOAD_NAME                0 (obj)
              2 LOAD_METHOD              1 (method)
              4 CALL_METHOD              0
...

When the VM executes the LOAD_METHOD opcode, it calls the _PyObject_GetMethod() function to search for the attribute value. This function works just like the generic function. The only difference is that it checks whether the value is an unbound method, i.e. a descriptor that returns a method-like object bound to the instance. In this case, it doesn't call the tp_descr_get slot of the descriptor's type but returns the descriptor itself. For example, if the attribute value is a function, _PyObject_GetMethod() returns the function. The function type and other descriptor types whose objects act as unbound methods specify the Py_TPFLAGS_METHOD_DESCRIPTOR flag in their tp_flags, so it's easy to identify them.

It should be noted that _PyObject_GetMethod() works as described only when the object's type uses the generic implementation of tp_getattro. Otherwise, it just calls the custom implementation and doesn't perform any checks.

If _PyObject_GetMethod() finds an unbound method, the method must be called with the instance prepended to the list of arguments. If it finds some other callable that doesn't need to be bound to the instance, the list of arguments must be kept unchanged. Therefore, after the VM has executed LOAD_METHOD, the values on the stack can be arranged in one of two ways:

an unbound method and a list of arguments including the instance: (method | self | arg1 | ... | argN)
other callable and a list of arguments without the instance (NULL | method | arg1 | ... | argN)

The CALL_METHOD opcode exists to call the method appropriately in each of these cases.

To learn more about this optimization, check out the issue that originated it.

Listing attributes of an object

Python provides the built-in dir() function that can be used to view what attributes an object has. Have you ever wondered how this function finds the attributes? It's implemented by calling the __dir__() special method of the object's type. Types rarely define their own __dir__(), yet all the types have it. This is because the object type defines __dir__(), and all other types inherit from object. The implementation provided by object lists all the attributes stored in the object's dictionary, in the type's dictionary and in the dictionaries of the type's parents. So, dir() effectively returns all the attributes of an ordinary object. However, when we call dir() on a type, we don't get all its attributes. This is because type provides its own implementation of __dir__(). This implementation returns attributes stored in the type's dictionary and in the dictionaries of the type's parents. It, however, ignores attributes stored in the metatype's dictionary and in the dictionaries of the metatype's parents. The documentation explains why this is the case:

Because dir() is supplied primarily as a convenience for use at an interactive prompt, it tries to supply an interesting set of names more than it tries to supply a rigorously or consistently defined set of names, and its detailed behavior may change across releases. For example, metaclass attributes are not in the result list when the argument is a class.

Where attributes of types come from

Take any built-in type and list its attributes. You'll get quite a few:

>>> dir(object)
['__class__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__']
>>> dir(int)
['__abs__', '__add__', '__and__', '__bool__', '__ceil__', '__class__', '__delattr__', '__dir__', '__divmod__', '__doc__', '__eq__', '__float__', '__floor__', '__floordiv__', '__format__', '__ge__', '__getattribute__', '__getnewargs__', '__gt__', '__hash__', '__index__', '__init__', '__init_subclass__', '__int__', '__invert__', '__le__', '__lshift__', '__lt__', '__mod__', '__mul__', '__ne__', '__neg__', '__new__', '__or__', '__pos__', '__pow__', '__radd__', '__rand__', '__rdivmod__', '__reduce__', '__reduce_ex__', '__repr__', '__rfloordiv__', '__rlshift__', '__rmod__', '__rmul__', '__ror__', '__round__', '__rpow__', '__rrshift__', '__rshift__', '__rsub__', '__rtruediv__', '__rxor__', '__setattr__', '__sizeof__', '__str__', '__sub__', '__subclasshook__', '__truediv__', '__trunc__', '__xor__', 'as_integer_ratio', 'bit_length', 'conjugate', 'denominator', 'from_bytes', 'imag', 'numerator', 'real', 'to_bytes']

We saw last time that the special methods that correspond to slots are added automatically by the PyType_Ready() function that initializes types. But where do the rest attributes come from? They all must be specified somehow and then be set to something at some point. This is a vague statement. Let's make it clear.

The most straightforward way to specify attributes of a type is to create a new dictionary, populate it with attributes and set type's tp_dict to that dictionary. We cannot do that before built-in types are defined, so tp_dict of built-in types is initialized to NULL. It turns out that the PyType_Ready() function creates dictionaries of built-in types at runtime. It is also responsible for adding all the attributes.

First, PyType_Ready() ensures that a type has a dictionary. Then, it adds attributes to the dictionary. A type tells PyType_Ready() which attributes to add by specifying the tp_methods, tp_members and tp_getset slots. Each slot is an array of structs that describe different kinds of attributes.

tp_methods

The tp_methods slot is an array of the PyMethodDef structs that describe methods:

struct PyMethodDef {
    const char  *ml_name;   /* The name of the built-in function/method */
    PyCFunction ml_meth;    /* The C function that implements it */
    int         ml_flags;   /* Combination of METH_xxx flags, which mostly
                               describe the args expected by the C func */
    const char  *ml_doc;    /* The __doc__ attribute, or NULL */
};
typedef struct PyMethodDef PyMethodDef;

The ml_meth member is a pointer to a C function that implements the method. Its signature can be one of many. The ml_flags bitfield is used to tell CPython how exactly to call the function.

For each struct in tp_methods, PyType_Ready() adds a callable object to the type's dictionary. This object encapsulates the struct. When we call it, the function pointed by ml_meth gets invoked. This is basically how a C function becomes a method of a Python type.

The object type, for example, defines __dir__() and a bunch of other methods using this mechanism:

static PyMethodDef object_methods[] = {
    {"__reduce_ex__", (PyCFunction)object___reduce_ex__, METH_O, object___reduce_ex____doc__},
    {"__reduce__", (PyCFunction)object___reduce__, METH_NOARGS, object___reduce____doc__},
    {"__subclasshook__", object_subclasshook, METH_CLASS | METH_VARARGS,
     object_subclasshook_doc},
    {"__init_subclass__", object_init_subclass, METH_CLASS | METH_NOARGS,
     object_init_subclass_doc},
    {"__format__", (PyCFunction)object___format__, METH_O, object___format____doc__},
    {"__sizeof__", (PyCFunction)object___sizeof__, METH_NOARGS, object___sizeof____doc__},
    {"__dir__", (PyCFunction)object___dir__, METH_NOARGS, object___dir____doc__},
    {0}
};

The callable object added to the dictionary is usually a method descriptor. We should probably discuss what a method descriptor is in another post on Python callables, but essentially it is an object that behaves like a function object, i.e. it binds to instances. The major difference is that a function bound to an instance returns a method object, and a method descriptor bound to an instance returns a built-in method object. A method object encapsulates a Python function and an instance, and a built-in method object encapsulates a C function and an instance.

For example, object.__dir__ is a method descriptor:

>>> object.__dir__
<method '__dir__' of 'object' objects>
>>> type(object.__dir__)
<class 'method_descriptor'>

If we bind __dir__ to an instance, we get a built-in method object:

>>> object().__dir__
<built-in method __dir__ of object object at 0x1088cc420>
>>> type(object().__dir__)
<class 'builtin_function_or_method'>

If ml_flags flags specifies that the method is static, a built-in method object is added to the dictionary instead of a method descriptor straight away.

Every method of any built-in type either wraps some slot or is added to the dictionary based on tp_methods.

tp_members

The tp_members slot is an array of the PyMemberDef structs. Each struct describes an attribute that exposes a C member of the objects of the type:

typedef struct PyMemberDef {
    const char *name;
    int type;
    Py_ssize_t offset;
    int flags;
    const char *doc;
} PyMemberDef;

The member is specified by offset. Its type is specified by type.

For each struct in tp_members, PyType_Ready() adds a member descriptor to the type's dictionary. A member descriptor is a data descriptor that encapsulates PyMemberDef. Its tp_descr_get slot takes an instance, finds the member of the instance located at offset, converts it to a corresponding Python object and returns the object. Its tp_descr_set slot takes an instance and a value, finds the member of the instance located at offset and sets it to the C equivalent of the value. A member can be made read-only by specifying flags.

By this mechanism, for example, type defines __dictoffset__ and other members:

static PyMemberDef type_members[] = {
    {"__basicsize__", T_PYSSIZET, offsetof(PyTypeObject,tp_basicsize),READONLY},
    {"__itemsize__", T_PYSSIZET, offsetof(PyTypeObject, tp_itemsize), READONLY},
    {"__flags__", T_ULONG, offsetof(PyTypeObject, tp_flags), READONLY},
    {"__weakrefoffset__", T_PYSSIZET,
     offsetof(PyTypeObject, tp_weaklistoffset), READONLY},
    {"__base__", T_OBJECT, offsetof(PyTypeObject, tp_base), READONLY},
    {"__dictoffset__", T_PYSSIZET,
     offsetof(PyTypeObject, tp_dictoffset), READONLY},
    {"__mro__", T_OBJECT, offsetof(PyTypeObject, tp_mro), READONLY},
    {0}
};

tp_getset

The tp_getset slot is an array of the PyGetSetDef structs that desribe arbitrary data descriptors like property():

typedef struct PyGetSetDef {
    const char *name;
    getter get;
    setter set;
    const char *doc;
    void *closure;
} PyGetSetDef;

For each struct in tp_getset, PyType_Ready() adds a getset descriptor to the type's dictionary. The tp_descr_get slot of a getset descriptor calls the specified get function, and the tp_descr_set slot of a getset descriptor calls the specified set function.

Types define the __dict__ attribute using this mechanism. Here's, for example, how the function type does that:

static PyGetSetDef func_getsetlist[] = {
    {"__code__", (getter)func_get_code, (setter)func_set_code},
    {"__defaults__", (getter)func_get_defaults,
     (setter)func_set_defaults},
    {"__kwdefaults__", (getter)func_get_kwdefaults,
     (setter)func_set_kwdefaults},
    {"__annotations__", (getter)func_get_annotations,
     (setter)func_set_annotations},
    {"__dict__", PyObject_GenericGetDict, PyObject_GenericSetDict},
    {"__name__", (getter)func_get_name, (setter)func_set_name},
    {"__qualname__", (getter)func_get_qualname, (setter)func_set_qualname},
    {NULL} /* Sentinel */
};

The __dict__ attribute is implemented not as a read-only member descriptor but as a geteset descriptor because it does more than simply return the dictionary located at tp_dictoffset. For instance, the descriptor creates the dictionary if it doesn't exist yet.

Classes also get the __dict__ attribute by this mechanism. The type_new() function that creates classes specifies tp_getset before it calls PyType_Ready(). Some classes, though, don't get this attribute because their instances don't have dictionaries. These are the classes that define __slots__.

slots

The __slots__ attribute of a class enumerates the attributes that the class can have:

>>> class D:
...     __slots__ = ('x', 'y')
...

If a class defines __slots__, the __dict__ attribute is not added to the class's dictionary and tp_dictoffset of the class is set to 0. The main effect of this is that the class instances don't have dictionaries:

>>> D.__dictoffset__
0
>>> d = D()
>>> d.__dict__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'D' object has no attribute '__dict__'

However, the attributes listed in __slots__ work fine:

>>> d.x = 4
>>> d.x
4

How is that possible? The attributes listed in __slots__ become members of class instances. For each member, the member descriptor is added to the class dictionary. The type_new() function specifies tp_members to do that.

>>> D.x
<member 'x' of 'D' objects>

Since instances don't have dictionaries, the __slots__ attribute saves memory. According to Descriptor HowTo Guide,

On a 64-bit Linux build, an instance with two attributes takes 48 bytes with __slots__ and 152 bytes without.

The guide also lists other benefits of using __slots__. I recommend you check them out.

Summary

The compiler produces the LOAD_ATTR, STORE_ATTR and DELETE_ATTR opcodes to get, set, and delete attributes. To executes these opcodes, the VM calls the tp_getattro and tp_setattro slots of the object's type. A type may implement these slots in an arbitrary way, but mostly we have to deal with three implementations:

the generic implementation used by most built-in types and classes
the implementation used by type
the implementation used by classes that define the __getattribute__(), __getattr__(), __setattr__() and __delattr__() special methods.

The generic implementation is straightforward once you understand what descriptors are. In a nutshell, descriptors are attributes that have control over attribute access, assignment and deletion. They allow CPython to implement many features including methods and properties.

Built-in types define attributes using three mechanisms:

tp_methods
tp_members; and
tp_getset.

Classes also use these mechanisms to define some attributes. For example, __dict__ is defined as a getset descriptor, and the attributes listed in __slots__ are defined as member descriptors.

P.S.

This post closes the first season of the Python behind the scenes series. We've learned a lot over this time. A lot remains to be covered. The topics on my list include: CPython's memory management, the GIL, the implementation of built-in types, the import system, concurrency and the internals of the standard modules. You can tell me what you would like to read about next time. Send your ideas and preferences to victor@tenthousandmeters.com.

See you in 2021. Stay tuned!

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #6: how Python object system works

2020-12-04T16:48:00+00:00

As we know from the previous parts of this series, the execution of a Python program consists of two major steps:

The CPython compiler translates Python code to bytecode.
The CPython VM executes the bytecode.

We've been focusing on the second step for quite a while. In part 4 we've looked at the evaluation loop, a place where Python bytecode gets executed. And in part 5 we've studied how the VM executes the instructions that are used to implement variables. What we haven't covered yet is how the VM actually computes something. We postponed this question because to answer it, we first need to understand how the most fundamental part of the language works. Today, we'll study the Python object system.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

Motivation

Consider an extremely simple piece of Python code:

def f(x):
    return x + 7

To compute the function f, CPython must evaluate the expression x + 7. The question I'd like to ask is: How does CPython do that? Special methods such as __add__() and __radd__() probably come to your mind. When we define these methods on a class, the instances of that class can be added using the + operator. So, you might think that CPython does something like this:

It calls x.__add__(7) or type(x).__add__(x, 7).
If x doesn't have __add__(), or if this method fails, it calls (7).__radd__(x) or int.__radd__(7, x).

The reality, though, is a bit more complicated. What really happens depends on what x is. For example, if x is an instance of a user-defined class, the algorithm described above resembles the truth. If, however, x is an instance of a built-in type, like int or float, CPython doesn't call any special methods at all.

To learn how some Python code is executed, we can do the following:

Disassemble the code into bytecode.
Study how the VM executes the disassembled bytecode instructions.

Let's apply this algorithm to the function f. The compiler translates the body of this function to the following bytecode:

$ python -m dis f.py
...
  2           0 LOAD_FAST                0 (x)
              2 LOAD_CONST               1 (7)
              4 BINARY_ADD
              6 RETURN_VALUE

And here's what these bytecode instructions do:

LOAD_FAST loads the value of the parameter x onto the stack.
LOAD_CONST loads the constant 7 onto the stack.
BINARY_ADD pops two values from the stack, adds them and pushes the result back onto the stack.
RETURN_VALUE pops the value from the stack and returns it.

How does the VM add two values? To answer this question, we need to understand what these values are. For us, 7 is an instance of int and x is, well, anything. For the VM, though, everything is a Python object. All values the VM pushes onto the stack and pops from the stack are pointers to PyObject structs (hence the phrase "Everything in Python is an object").

The VM doesn't need to know how to add integers or strings, that is, how to do the arithmetic or concatenate sequences. All it needs to know is that every Python object has a type. A type, in turn, knows everything about its objects. For example, the int type knows how to add integers, and the float type knows how to add floats. So, the VM asks the type to perform the operation.

This simplified explanation captures the essence of the solution, but it also omits a lot of important details. To get a more realistic picture, we need to understand what Python objects and types really are and how they work.

Python objects and types

We've discussed Python objects a little in part 3. This discussion is worth repeating here.

We begin with the definition of the PyObject struct:

typedef struct _object {
    _PyObject_HEAD_EXTRA // macro, for debugging purposes only
    Py_ssize_t ob_refcnt;
    PyTypeObject *ob_type;
} PyObject;

It has two members:

a reference count ob_refcnt that CPython uses for garbage collection; and
a pointer to the object's type ob_type.

We said that the VM treats any Python object as PyObject. How is that possible? The C programming language has no notion of classes and inheritance. Nevertheless, it's possible to implement in C something that can be called a single inheritance. The C standard states that a pointer to any struct can be converted to a pointer to its first member and vice versa. So, we can "extend" PyObject by defining a new struct whose first member is PyObject.

Here's, for example, how the float object is defined:

typedef struct {
    PyObject ob_base; // expansion of PyObject_HEAD macro
    double ob_fval;
} PyFloatObject;

A float object stores everything PyObject stores plus a floating-point value ob_fval. The C standard simply states that we can convert a pointer to PyFloatObject to a pointer to PyObject and vice versa:

PyFloatObject float_object;
// ...
PyObject *obj_ptr = (PyObject *)&float_object;
PyFloatObject *float_obj_ptr = (PyFloatObject *)obj_ptr;

The reason why the VM treats every Python object as PyObject is because all it needs to access is the object's type. A type is also a Python object, an instance of the PyTypeObject struct:

// PyTypeObject is a typedef for "struct _typeobject"

struct _typeobject {
    PyVarObject ob_base; // expansion of PyObject_VAR_HEAD macro
    const char *tp_name; /* For printing, in format "<module>.<name>" */
    Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */

    /* Methods to implement standard operations */

    destructor tp_dealloc;
    Py_ssize_t tp_vectorcall_offset;
    getattrfunc tp_getattr;
    setattrfunc tp_setattr;
    PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2)
                                    or tp_reserved (Python 3) */
    reprfunc tp_repr;

    /* Method suites for standard classes */

    PyNumberMethods *tp_as_number;
    PySequenceMethods *tp_as_sequence;
    PyMappingMethods *tp_as_mapping;

    /* More standard operations (here for binary compatibility) */

    hashfunc tp_hash;
    ternaryfunc tp_call;
    reprfunc tp_str;
    getattrofunc tp_getattro;
    setattrofunc tp_setattro;

    /* Functions to access object as input/output buffer */
    PyBufferProcs *tp_as_buffer;

    /* Flags to define presence of optional/expanded features */
    unsigned long tp_flags;

    const char *tp_doc; /* Documentation string */

    /* Assigned meaning in release 2.0 */
    /* call function for all accessible objects */
    traverseproc tp_traverse;

    /* delete references to contained objects */
    inquiry tp_clear;

    /* Assigned meaning in release 2.1 */
    /* rich comparisons */
    richcmpfunc tp_richcompare;

    /* weak reference enabler */
    Py_ssize_t tp_weaklistoffset;

    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;

    /* Attribute descriptor and subclassing stuff */
    struct PyMethodDef *tp_methods;
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
    struct _typeobject *tp_base;
    PyObject *tp_dict;
    descrgetfunc tp_descr_get;
    descrsetfunc tp_descr_set;
    Py_ssize_t tp_dictoffset;
    initproc tp_init;
    allocfunc tp_alloc;
    newfunc tp_new;
    freefunc tp_free; /* Low-level free-memory routine */
    inquiry tp_is_gc; /* For PyObject_IS_GC */
    PyObject *tp_bases;
    PyObject *tp_mro; /* method resolution order */
    PyObject *tp_cache;
    PyObject *tp_subclasses;
    PyObject *tp_weaklist;
    destructor tp_del;

    /* Type attribute cache version tag. Added in version 2.6 */
    unsigned int tp_version_tag;

    destructor tp_finalize;
    vectorcallfunc tp_vectorcall;
};

By the way, note that the first member of a type is not PyObject but PyVarObject, which is defined as follows:

typedef struct {
    PyObject ob_base;
    Py_ssize_t ob_size; /* Number of items in variable part */
} PyVarObject;

Nevertheless, since the first member of PyVarObject is PyObject, a pointer to a type can still be converted to a pointer to PyObject.

So, what is a type and why does it have so many members? A type determines how the objects of that type behave. Each member of a type, called slot, is responsible for a particular aspect of the object's behavior. For example:

tp_new is a pointer to a function that creates new objects of the type.
tp_str is a pointer to a function that implements str() for objects of the type.
tp_hash is a pointer to a function that implements hash() for objects of the type.

Some slots, called sub-slots, are grouped together in suites. A suite is just a struct that contains related slots. For example, the PySequenceMethods struct is a suite of sub-slots that implement the sequence protocol:

typedef struct {
    lenfunc sq_length;
    binaryfunc sq_concat;
    ssizeargfunc sq_repeat;
    ssizeargfunc sq_item;
    void *was_sq_slice;
    ssizeobjargproc sq_ass_item;
    void *was_sq_ass_slice;
    objobjproc sq_contains;

    binaryfunc sq_inplace_concat;
    ssizeargfunc sq_inplace_repeat;
} PySequenceMethods;

If you count all the slots and sub-slots, you'll get a scary number. Fortunately, each slot is very well documented in the Python/C API Reference Manual (I strongly recommend you to bookmark this link). Today we'll cover only a few slots. Nevertheless, it shall give us a general idea of how slots are used.

Since we're interested in how CPython adds objects, let's find the slots responsible for addition. There must be at least one such slot. After careful inspection of the PyTypeObject struct, we find that it has the "number" suite PyNumberMethods, and the first slot of this suite is a binary function called nd_add:

typedef struct {
    binaryfunc nb_add; // typedef PyObject * (*binaryfunc)(PyObject *, PyObject *)
    binaryfunc nb_subtract;
    binaryfunc nb_multiply;
    binaryfunc nb_remainder;
    binaryfunc nb_divmod;
    // ... more sub-slots
} PyNumberMethods;

It seems that the nb_add slot is what we're looking for. Two questions naturally arise regarding this slot:

What is it set to?
How is it used?

I think it's better to start with the second. We should expect that the VM calls nb_add to execute the BINARY_ADD opcode. So, let's, for a moment, suspend our discussion about types and take a look at how the BINARY_ADD opcode is implemented.

BINARY_ADD

Like any other opcode, BINARY_ADD is implemented in the evaluation loop in Python/ceval.c:

case TARGET(BINARY_ADD): {
    PyObject *right = POP();
    PyObject *left = TOP();
    PyObject *sum;
    /* NOTE(haypo): Please don't try to micro-optimize int+int on
        CPython using bytecode, it is simply worthless.
        See http://bugs.python.org/issue21955 and
        http://bugs.python.org/issue10044 for the discussion. In short,
        no patch shown any impact on a realistic benchmark, only a minor
        speedup on microbenchmarks. */
    if (PyUnicode_CheckExact(left) &&
                PyUnicode_CheckExact(right)) {
        sum = unicode_concatenate(tstate, left, right, f, next_instr);
        /* unicode_concatenate consumed the ref to left */
    }
    else {
        sum = PyNumber_Add(left, right);
        Py_DECREF(left);
    }
    Py_DECREF(right);
    SET_TOP(sum);
    if (sum == NULL)
        goto error;
    DISPATCH();
}

This code requires some comments. We can see that it calls PyNumber_Add() to add two objects, but if the objects are strings, it calls unicode_concatenate() instead. Why so? This is an optimization. Python strings seem immutable, but sometimes CPython mutates a string and thus avoids creating a new string. Consider appending one string to another:

output += some_string

If the output variable points to a string that has no other references, it's safe to mutate that string. This is exactly the logic that unicode_concatenate() implements.

It might be tempting to handle other special cases in the evaluation loop as well and optimize, for example, integers and floats. The comment explicitly warns against it. The problem is that a new special case comes with an additional check, and this check is only useful when it succeeds. Otherwise, it may have a negative effect on performance.

After this little digression, let's look at PyNumber_Add():

PyObject *
PyNumber_Add(PyObject *v, PyObject *w)
{
    // NB_SLOT(nb_add) expands to "offsetof(PyNumberMethods, nb_add)"
    PyObject *result = binary_op1(v, w, NB_SLOT(nb_add));
    if (result == Py_NotImplemented) {
        PySequenceMethods *m = Py_TYPE(v)->tp_as_sequence;
        Py_DECREF(result);
        if (m && m->sq_concat) {
            return (*m->sq_concat)(v, w);
        }
        result = binop_type_error(v, w, "+");
    }
    return result;
}

I suggest to step into binary_op1() straight away and figure out what the rest of PyNumber_Add() does later:

static PyObject *
binary_op1(PyObject *v, PyObject *w, const int op_slot)
{
    PyObject *x;
    binaryfunc slotv = NULL;
    binaryfunc slotw = NULL;

    if (Py_TYPE(v)->tp_as_number != NULL)
        slotv = NB_BINOP(Py_TYPE(v)->tp_as_number, op_slot);
    if (!Py_IS_TYPE(w, Py_TYPE(v)) &&
        Py_TYPE(w)->tp_as_number != NULL) {
        slotw = NB_BINOP(Py_TYPE(w)->tp_as_number, op_slot);
        if (slotw == slotv)
            slotw = NULL;
    }
    if (slotv) {
        if (slotw && PyType_IsSubtype(Py_TYPE(w), Py_TYPE(v))) {
            x = slotw(v, w);
            if (x != Py_NotImplemented)
                return x;
            Py_DECREF(x); /* can't do it */
            slotw = NULL;
        }
        x = slotv(v, w);
        if (x != Py_NotImplemented)
            return x;
        Py_DECREF(x); /* can't do it */
    }
    if (slotw) {
        x = slotw(v, w);
        if (x != Py_NotImplemented)
            return x;
        Py_DECREF(x); /* can't do it */
    }
    Py_RETURN_NOTIMPLEMENTED;
}

The binary_op1() function takes three parameters: the left operand, the right operand and an offset that identifies the slot. Types of both operands can implement the slot. Therefore, binary_op1() looks up both implementations. To calculate the result, it calls one implementation or another relying on the following logic:

If the type of one operand is a subtype of another, call the slot of the subtype.
If the left operand doesn't have the slot, call the slot of the right operand.
Otherwise, call the slot of the left operand.

The reason to prioritize the slot of a subtype is to allow the subtypes to override the behavior of their ancestors:

$ python -q
>>> class HungryInt(int):
...     def __add__(self, o):
...             return self
...
>>> x = HungryInt(5)
>>> x + 2
5
>>> 2 + x
7
>>> HungryInt.__radd__ = lambda self, o: self
>>> 2 + x
5

Let's turn back to PyNumber_Add(). If binary_op1() succeeds, PyNumber_Add() simply returns the result of binary_op1(). If, however, binary_op1() returns the NotImplemented constant, which means that the operation cannot be performed for a given combination of types, PyNumber_Add() calls the sq_concat "sequence" slot of the first operand and returns the result of this call:

PySequenceMethods *m = Py_TYPE(v)->tp_as_sequence;
if (m && m->sq_concat) {
    return (*m->sq_concat)(v, w);
}

A type can support the + operator either by implementing nb_add or sq_concat. These slots have different meanings:

nb_add means algebraic addition with properties like a + b = b + a.
sq_concat means the concatenation of sequences.

Built-in types such as int and float implement nb_add, and built-in types such as str and list implement sq_concat. Technically, there's no much difference. The main reason to choose one slot over another is to indicate the appropriate meaning. In fact, the sq_concat slot is so unnecessary that it's set to NULL for all user-defined types (i.e. classes).

We saw how the nb_add slot is used: it's called by the binary_op1() function. The next step is to see what it's set to.

What nb_add can be

Since addition is a different operation for different types, the nb_add slot of a type must be one of two things:

it's either a type-specific function that adds object of that type; or
it's a type-agnostic function that calls some type-specific functions, such as type's __add__() special method.

It's indeed one of these two, and which one depends on the type. For example, built-in types such as int and float have their own implementations of nb_add. In contrast, all classes share the same implementation. Fundamentally, built-in types and classes are the same thing – instances of PyTypeObject. The important difference between them is how they are created. This difference effects the way the slots are set, so we should discuss it.

Ways to create a type

There are two ways to create a type object:

by statically defining it; or
by dynamically allocating it.

Statically defined types

An example of a statically defined type is any built-in type. Here's, for instance, how CPython defines the float type:

PyTypeObject PyFloat_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    "float",
    sizeof(PyFloatObject),
    0,
    (destructor)float_dealloc,                  /* tp_dealloc */
    0,                                          /* tp_vectorcall_offset */
    0,                                          /* tp_getattr */
    0,                                          /* tp_setattr */
    0,                                          /* tp_as_async */
    (reprfunc)float_repr,                       /* tp_repr */
    &float_as_number,                           /* tp_as_number */
    0,                                          /* tp_as_sequence */
    0,                                          /* tp_as_mapping */
    (hashfunc)float_hash,                       /* tp_hash */
    0,                                          /* tp_call */
    0,                                          /* tp_str */
    PyObject_GenericGetAttr,                    /* tp_getattro */
    0,                                          /* tp_setattro */
    0,                                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_BASETYPE,   /* tp_flags */
    float_new__doc__,                           /* tp_doc */
    0,                                          /* tp_traverse */
    0,                                          /* tp_clear */
    float_richcompare,                          /* tp_richcompare */
    0,                                          /* tp_weaklistoffset */
    0,                                          /* tp_iter */
    0,                                          /* tp_iternext */
    float_methods,                              /* tp_methods */
    0,                                          /* tp_members */
    float_getset,                               /* tp_getset */
    0,                                          /* tp_base */
    0,                                          /* tp_dict */
    0,                                          /* tp_descr_get */
    0,                                          /* tp_descr_set */
    0,                                          /* tp_dictoffset */
    0,                                          /* tp_init */
    0,                                          /* tp_alloc */
    float_new,                                  /* tp_new */
};

The slots of a statically defined type are specified explicitly. We can easily see how the float type implements nb_add by looking at the "number" suite:

static PyNumberMethods float_as_number = {
    float_add,          /* nb_add */
    float_sub,          /* nb_subtract */
    float_mul,          /* nb_multiply */
    // ... more number slots
};

where we find the float_add() function, a straightforward implementation of nb_add:

static PyObject *
float_add(PyObject *v, PyObject *w)
{
    double a,b;
    CONVERT_TO_DOUBLE(v, a);
    CONVERT_TO_DOUBLE(w, b);
    a = a + b;
    return PyFloat_FromDouble(a);
}

The floating-point arithmetic is not that important for our discussion. This example demonstrates how to specify the behavior of a statically defined type. It turned out to be quite easy: just write the implementation of slots and point each slot to the corresponding implementation.

If you want to learn how to statically define your own types, check out Python's tutorial for C/C++ programmers.

Dynamically allocated types

Dynamically allocated types are the types we define using the class statement. As we've already said, they are instances of PyTypeObject, just like statically defined types. Traditionally, we call them classes but we might call them user-defined types as well.

From the programmer's perspective, it's easier to define a class in Python than a type in C. This is because CPython does a lot of things behind the scenes when it creates a class. Let's see what's involved in this process.

If we wouldn't know where to start, we could apply the familiar method:

1. Define a simple class

class A:
    pass

2. Run the disassembler:

$ python -m dis class_A.py

3. Study how the VM executes the produced bytecode instructions.

Feel free to do that if you find the time, or read the article on classes by Eli Bendersky. We'll take a shortcut.

An object is created by a call to a type, e.g. list() or MyClass(). A class is created by a call to a metatype. A metatype is just a type whose instances are types. Python has one built-in metatype called PyType_Type, which is known to us simply as type. Here's how it's defined:

PyTypeObject PyType_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    "type",                                     /* tp_name */
    sizeof(PyHeapTypeObject),                   /* tp_basicsize */
    sizeof(PyMemberDef),                        /* tp_itemsize */
    (destructor)type_dealloc,                   /* tp_dealloc */
    offsetof(PyTypeObject, tp_vectorcall),      /* tp_vectorcall_offset */
    0,                                          /* tp_getattr */
    0,                                          /* tp_setattr */
    0,                                          /* tp_as_async */
    (reprfunc)type_repr,                        /* tp_repr */
    0,                                          /* tp_as_number */
    0,                                          /* tp_as_sequence */
    0,                                          /* tp_as_mapping */
    0,                                          /* tp_hash */
    (ternaryfunc)type_call,                     /* tp_call */
    0,                                          /* tp_str */
    (getattrofunc)type_getattro,                /* tp_getattro */
    (setattrofunc)type_setattro,                /* tp_setattro */
    0,                                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC |
    Py_TPFLAGS_BASETYPE | Py_TPFLAGS_TYPE_SUBCLASS |
    Py_TPFLAGS_HAVE_VECTORCALL,                 /* tp_flags */
    type_doc,                                   /* tp_doc */
    (traverseproc)type_traverse,                /* tp_traverse */
    (inquiry)type_clear,                        /* tp_clear */
    0,                                          /* tp_richcompare */
    offsetof(PyTypeObject, tp_weaklist),        /* tp_weaklistoffset */
    0,                                          /* tp_iter */
    0,                                          /* tp_iternext */
    type_methods,                               /* tp_methods */
    type_members,                               /* tp_members */
    type_getsets,                               /* tp_getset */
    0,                                          /* tp_base */
    0,                                          /* tp_dict */
    0,                                          /* tp_descr_get */
    0,                                          /* tp_descr_set */
    offsetof(PyTypeObject, tp_dict),            /* tp_dictoffset */
    type_init,                                  /* tp_init */
    0,                                          /* tp_alloc */
    type_new,                                   /* tp_new */
    PyObject_GC_Del,                            /* tp_free */
    (inquiry)type_is_gc,                        /* tp_is_gc */
};

The type of all built-in types is type, and the type of all classes defaults to type. So, type determines how types behave. For example, what happens when we call a type, like list() or MyClass(), is specified by the tp_call slot of type. The implementation of the tp_call slot of type is the type_call() function. Its job is to create new objects. It calls two other slots to do that:

It calls tp_new of a type to create an object.
It calls tp_init of a type to initialize the created object.

The type of type is type itself. So, when we call type(), the type_call() function is invoked. It checks for the special case when we pass a single argument to type(). In this case, type_call() simply returns the type of the passed object:

$ python -q
>>> type(3)
<class 'int'>
>>> type(int)
<class 'type'>
>>> type(type)
<class 'type'>

But when we pass three arguments to type(), type_call() creates a new type by calling tp_new and tp_init of type as described above. The following example demonstrates how to use type() to create a class:

$ python -q
>>> MyClass = type('MyClass', (), {'__str__': lambda self: 'Hey!'})
>>> instance_of_my_class = MyClass()
>>> str(instance_of_my_class)
Hey!

The arguments we pass to type() are:

the name of a class
a tuple of its bases; and
a namespace.

Other metatypes take arguments in this form as well.

We saw that we can create a class by calling type(), but that's not what we typically do. Typically, we use the class statement to define a class. It turns out that in this case too the VM eventually calls some metatype, and most often it calls type().

To execute the class statement, the VM calls the __build_class__() function from the builtins module. What this function does can be summarized as follows:

Decide which metatype to call to create the class.
Prepare the namespace. The namespace will be used as a class's dictionary.
Execute the body of the class in the namespace, thus filling the namespace.
Call the metatype.

We can instruct __build_class__() which metatype it should call using the metaclass keyword. If no metaclass is specified, __build_class__() calls type() by default. It also takes metatypes of bases into account. The exact logic of choosing the metatype is nicely described in the docs.

Suppose we define a new class and do not specify metaclass. Where does the class actually get created? In this case, __build_class__() calls type(). This invokes the type_call() function that, in turn, calls the tp_new and tp_init slots of type. The tp_new slot of type points to the type_new() function. This is the function that creates classes. The tp_init slot of type points to the function that does nothing, so all the work is done by type_new().

The type_new() function is nearly 500 lines long and probably deserves a separate post. Its essence, though, can be briefly summarized as follows:

Allocate new type object.
Set up the allocated type object.

To accomplish the first step, type_new() must allocate an instance of PyTypeObject as well as suites. Suites must be allocated separately from PyTypeObject because PyTypeObject contains only pointers to suites, not suites themselves. To handle this inconvenience, type_new() allocates an instance of the PyHeapTypeObject struct that extends PyTypeObject and contains the suites:

/* The *real* layout of a type object when allocated on the heap */
typedef struct _heaptypeobject {
    PyTypeObject ht_type;
    PyAsyncMethods as_async;
    PyNumberMethods as_number;
    PyMappingMethods as_mapping;
    PySequenceMethods as_sequence;
    PyBufferProcs as_buffer;
    PyObject *ht_name, *ht_slots, *ht_qualname;
    struct _dictkeysobject *ht_cached_keys;
    PyObject *ht_module;
    /* here are optional user slots, followed by the members. */
} PyHeapTypeObject;

To set up a type object means to set up its slots. This is what type_new() does for the most part.

Type initialization

Before any type can be used, it should be initialized with the PyType_Ready() function. For a class, PyType_Ready() is called by type_new(). For a statically defined type, PyType_Ready() must be called explicitly. When CPython starts, it calls PyType_Ready() for each built-in type.

The PyType_Ready() function does a number of things. For example, it does slot inheritance.

Slot inheritance

When we define a class that inherits from other type, we expect the class to inherit some behavior of that type. For example, when we define a class that inherits from int, we expect it to support the addition:

$ python -q
>>> class MyInt(int):
...     pass
... 
>>> x = MyInt(2)
>>> y = MyInt(4)
>>> x + y
6

Does MyInt inherit the nb_add slot of int? Yes, it does. It's pretty straightforward to inherit the slots from a single ancestor: just copy those slots that the class doesn't have. It's a little bit more complicated when a class has multiple bases. Since bases, in turn, may inherit from other types, all these ancestor types combined form an hierarchy. The problem with the hierarchy is that it doesn't specify the order of inheritance. To solve this problem, PyType_Ready() converts this hierarchy into a list. The Method Resolution Order (MRO) determines how to perform this conversion. Once the MRO is calculated, it becomes easy to implement the inheritance in the general case. The PyType_Ready() function iterates over ancestors according to the MRO. From each ancestor, it copies those slots that haven't been set on the type before. Some slots support the inheritance and some don't. You can check in the docs whether a particular slot is inherited.

In contrast to a class, a statically defined type can specify at most one base. This is done by implementing the tp_base slot.

If no bases are specified, PyType_Ready() assumes that the object type is the only base. Every type directly or indirectly inherits from object. Why? Because it implements the slots that every type is expected to have. For example, it implements tp_alloc, tp_init and tp_repr slots.

The ultimate question

So far we've seen two ways in which a slot can be set:

It can be specified explicitly (if a type is a statically defined type).
It can be inherited from an ancestor.

It's still unclear how slots of a class are connected to its special methods. Moreover, we have a reverse problem for built-in types. How do they implement special methods? They certainly do:

$ python -q
>>> (3).__add__(4)
7

We come to the ultimate question of this post: What's the connection between special methods and slots?

Special methods and slots

The answer lies in the fact that CPython keeps a mapping between special methods and slots. This mapping is represented by the slotdefs array. It looks like this:

#define TPSLOT(NAME, SLOT, FUNCTION, WRAPPER, DOC) \
    {NAME, offsetof(PyTypeObject, SLOT), (void *)(FUNCTION), WRAPPER, \
     PyDoc_STR(DOC)}

static slotdef slotdefs[] = {
    TPSLOT("__getattribute__", tp_getattr, NULL, NULL, ""),
    TPSLOT("__getattr__", tp_getattr, NULL, NULL, ""),
    TPSLOT("__setattr__", tp_setattr, NULL, NULL, ""),
    TPSLOT("__delattr__", tp_setattr, NULL, NULL, ""),
    TPSLOT("__repr__", tp_repr, slot_tp_repr, wrap_unaryfunc,
           "__repr__($self, /)\n--\n\nReturn repr(self)."),
    TPSLOT("__hash__", tp_hash, slot_tp_hash, wrap_hashfunc,
           "__hash__($self, /)\n--\n\nReturn hash(self)."),
    // ... more slotdefs
}

Each entry of this array is a slotdef struct:

// typedef struct wrapperbase slotdef;

struct wrapperbase {
    const char *name;
    int offset;
    void *function;
    wrapperfunc wrapper;
    const char *doc;
    int flags;
    PyObject *name_strobj;
};

Four members of this struct are important for our discussion:

name is a name of a special method.
offset is an offset of a slot in the PyHeapTypeObject struct. It specifies the slot corresponding to the special method.
function is an implementation of a slot. When a special method is defined, the corresponding slot is set to function. Typically, function calls special methods to do the work.
wrapper is a wrapper function around a slot. When a slot is defined, wrapper provides an implementation for the corresponding special method. It calls the slot to do the work.

Here's, for example, an entry that maps __add__() special method to the nb_add slot:

name is "__add__".
offset is offsetof(PyHeapTypeObject, as_number.nb_add).
function is slot_nb_add().
wrapper is wrap_binaryfunc_l().

The slotdefs array is a many-to-many mapping. For example, as we'll see, both the __add__() and __radd__() special methods map to the same nb_add slot. Conversely, both the mp_subscript "mapping" slot and the sq_item "sequence" slot map to the same __getitem__() special method.

CPython uses the slotdefs array in two ways:

to set slots based on special methods; and
to set special methods based on slots.

Slots based on special methods

The type_new() function calls fixup_slot_dispatchers() to set slots based on special methods. The fixup_slot_dispatchers() function calls update_one_slot() for each slot in the slotdefs array, and update_one_slot() sets the slot to function if a class has the corresponding special method.

Let's take the nb_add slot as an example. The slotdefs array has two entries corresponding to that slot:

static slotdef slotdefs[] = {
    // ...
    BINSLOT("__add__", nb_add, slot_nb_add, "+"),
    RBINSLOT("__radd__", nb_add, slot_nb_add,"+"),
    // ...
}

BINSLOT() and RBINSLOT() are macros. Let's expand them:

static slotdef slotdefs[] = {
    // ...
    // {name, offset, function,
    //     wrapper, doc}
    // 
    {"__add__", offsetof(PyHeapTypeObject, as_number.nb_add), (void *)(slot_nb_add),
        wrap_binaryfunc_l, PyDoc_STR("__add__" "($self, value, /)\n--\n\nReturn self" "+" "value.")},

    {"__radd__", offsetof(PyHeapTypeObject, as_number.nb_add), (void *)(slot_nb_add),
        wrap_binaryfunc_r, PyDoc_STR("__radd__" "($self, value, /)\n--\n\nReturn value" "+" "self.")},
    // ...
}

What update_one_slot() does is look up class.__add__() and class.__radd__(). If either is defined, it sets nb_add of the class to slot_nb_add(). Note that both entries agree on slot_nb_add() as function. Otherwise, we would have a conflict when both are defined.

Now, what is slot_nb_add(), you ask? This function is defined with a macro that expands as follows:

static PyObject *
slot_nb_add(PyObject *self, PyObject *other) {
    PyObject* stack[2];
    PyThreadState *tstate = _PyThreadState_GET();
    _Py_static_string(op_id, "__add__");
    _Py_static_string(rop_id, "__radd__");
    int do_other = !Py_IS_TYPE(self, Py_TYPE(other)) && \
        Py_TYPE(other)->tp_as_number != NULL && \
        Py_TYPE(other)->tp_as_number->nb_add == slot_nb_add;
    if (Py_TYPE(self)->tp_as_number != NULL && \
        Py_TYPE(self)->tp_as_number->nb_add == slot_nb_add) {
        PyObject *r;
        if (do_other && PyType_IsSubtype(Py_TYPE(other), Py_TYPE(self))) {
            int ok = method_is_overloaded(self, other, &rop_id);
            if (ok < 0) {
                return NULL;
            }
            if (ok) {
                stack[0] = other;
                stack[1] = self;
                r = vectorcall_maybe(tstate, &rop_id, stack, 2);
                if (r != Py_NotImplemented)
                    return r;
                Py_DECREF(r); do_other = 0;
            }
        }
        stack[0] = self;
        stack[1] = other;
        r = vectorcall_maybe(tstate, &op_id, stack, 2);
        if (r != Py_NotImplemented || Py_IS_TYPE(other, Py_TYPE(self)))
            return r;
        Py_DECREF(r);
    }
    if (do_other) {
        stack[0] = other;
        stack[1] = self;
        return vectorcall_maybe(tstate, &rop_id, stack, 2);
    }
    Py_RETURN_NOTIMPLEMENTED;
}

You don't need to study this code carefully. Recall the binary_op1() function that calls the nb_add slot. The slot_nb_add() function basically repeats the logic of binary_op1(). The main difference is that slot_nb_add() eventually calls __add__() or __radd__().

Setting special method on existing class

Suppose that we create a class without the __add__() and __radd__() special methods. In this case, the nb_add slot of the class is set to NULL. As expected, we cannot add instances of that class. If we, however, set __add__() or __radd__() after the class has been created, the addition works as if the method was a part of the class definition. Here's what I mean:

$ python -q
>>> class A:
...     pass
... 
>>> x = A()
>>> x + 2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'A' and 'int'
>>> A.__add__ = lambda self, o: 5
>>> x + 2
5
>>>

How does that work? To set an attribute on an object, the VM calls the tp_setattro slot of the object's type. The tp_setattro slot of type points to the type_setattro() function, so when we set an attribute on a class, this function gets called. It stores the value of the attribute in the class's dictionary. Then it checks if the attribute is a special method and, if so, sets the corresponding slots by calling the update_one_slot() function.

Before we can learn how CPython does the reverse, that is, how it adds special methods to built-in types, we need to understand what a method is.

Methods

A method is an attribute, but a peculiar one. When we call a method from an instance, the method implicitly receives the instance as its first parameter, which we usually denote self:

$ python -q
>>> class A:
...     def method(self, x):
...             return self, x
...
>>> a = A()
>>> a.method(1)
(<__main__.A object at 0x10d10bfd0>, 1)

But when we call the same method from a class, we have to pass all arguments explicitly:

>>> A.method(a, 1)
(<__main__.A object at 0x10d10bfd0>, 1)

In our example, the method takes one argument in one case and two arguments in another. How is that possible that the same attribute is a different thing depending on how we access it?

First of all, realize that a method we define on a class is just a function. A function accessed through an instance differs from the same function accessed through the instance's type because the function type implements the descriptor protocol. If you're unfamiliar with descriptors, I highly recommend you to read Descriptor HowTo Guide by Raymond Hettinger. In a nutshell, a descriptor is an object that, when used as an attribute, determines by itself how you get, set and delete it. Technically, a descriptor is an object that implements __get__(), __set__(), or __delete__() special methods.

The function type implements __get__(). When we look up some method, what we get is the result of a call to __get__(). Three arguments are passed to it:

an attribute, i.e. a function
an instance
the instance's type.

If we look up a method on a type, the instance is NULL, and __get__() simply returns the function. If we look up a method on an instance, __get__() returns a method object:

>>> type(A.method)
<class 'function'>
>>> type(a.method)
<class 'method'>

A method object stores a function and an instance. When called, it prepends the instance to the list of arguments and calls the function.

Now we're ready to tackle the last question.

Special methods based on slots

Recall the PyType_Ready() function that initializes types and does slot inheritance. It also adds special methods to a type based on the implemented slots. PyType_Ready() calls add_operators() to do that. The add_operators() function iterates over the entries in the slotdefs array. For each entry, it checks whether the special method specified by the entry should be added to the type's dictionary. A special method is added if it's not already defined and if the type implements the slot specified by the entry. For example, if the __add__() special method is not defined on a type, but the type implements the nb_add slot, add_operators() puts __add__() in the type's dictionary.

What is __add__() set to? Like any other method, it must be set to some descriptor to behave like a method. While methods defined by a programmer are functions, methods set by add_operators() are wrapper descriptors. A wrapper descriptor is a descriptor that stores two things:

It stores a wrapped slot. A wrapped slot "does the work" for a special method. For example, the wrapper descriptor of the __add__() special method of the float type stores float_add() as a wrapped slot.
It stores a wrapper function. A wrapper function "knows" how to call the wrapped slot. It is wrapper of a slotdef entry.

When we call a special method that was added by add_operators(), we call a wrapper descriptor. When we call a wrapper descriptor, it calls a wrapper function. A wrapper descriptor passes to a wrapper function the same arguments that we pass to a special methods plus the wrapped slot. Finally, the wrapper function calls the wrapped slot.

Let's see how a built-in type that implements the nb_add slot gets its __add__() and __radd__() special methods. Recall the slotdef entries corresponding to nb_add:

static slotdef slotdefs[] = {
    // ...
    // {name, offset, function,
    //     wrapper, doc}
    // 
    {"__add__", offsetof(PyHeapTypeObject, as_number.nb_add), (void *)(slot_nb_add),
        wrap_binaryfunc_l, PyDoc_STR("__add__" "($self, value, /)\n--\n\nReturn self" "+" "value.")},

    {"__radd__", offsetof(PyHeapTypeObject, as_number.nb_add), (void *)(slot_nb_add),
        wrap_binaryfunc_r, PyDoc_STR("__radd__" "($self, value, /)\n--\n\nReturn value" "+" "self.")},
    // ...
}

If a type implements the nb_add slot, add_operators() sets __add__() of the type to a wrapper descriptor with wrap_binaryfunc_l() as a wrapper function and nb_add as a wrapped slot. It similarly sets __radd__() of the type with one exception: a wrapper function is wrap_binaryfunc_r().

Both wrap_binaryfunc_l() and wrap_binaryfunc_r() take two operands plus a wrapped slot as their parameters. The only difference is how they call the slot:

wrap_binaryfunc_l(x, y, slot_func) calls slot_func(x, y)
wrap_binaryfunc_r(x, y, slot_func) calls slot_func(y, x).

The result of this call is what we get when we call the special method.

Summary

Today we've demystified perhaps the most magical aspect of Python. We've learned that the behavior of a Python object is determined by the slots of the object's type. The slots of a statically defined type can be specified explicitly, and any type can inherit some slots from its ancestors. The real insight was that the slots of a class are set up automatically by CPython based on the defined special methods. CPython does the reverse too. It adds special methods to the type's dictionary if the type implements the corresponding slots.

We've learned a lot. Nevertheless, the Python object system is such a vast subject that at least as much remains to be covered. For example, we haven't really discussed how attributes work. This is what we're going to do next time.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #5: how variables are implemented in CPython

2020-11-14T14:09:00+00:00

Consider a simple assignment statement in Python:

a = b

The meaning of this statement may seem trivial. What we do here is take the value of the name b and assign it to the name a, but do we really? This is an ambiguous explanation that gives rise to a lot of questions:

What does it mean for a name to be associated with a value? What is a value?
What does CPython do to assign a value to a name? To get the value?
Are all variables implemented in the same way?

Today we'll answer these questions and understand how variables, so crucial aspect of a programming language, are implemented in CPython.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

Start of the investigation

Where should we start our investigation? We know from the previous parts that to run Python code, CPython compiles it to bytecode, so let's start by looking at the bytecode to which a = b compiles:

$ echo 'a = b' | python -m dis

  1           0 LOAD_NAME                0 (b)
              2 STORE_NAME               1 (a)
...

Last time we learned that the CPython VM operates using the value stack. A typical bytecode instruction pops values from the stack, does something with them and pushes the result of the computation back onto the stack. The LOAD_NAME and STORE_NAME instructions are typical in that respect. Here's what they do in our example:

LOAD_NAME gets the value of the name b and pushes it onto the stack.
STORE_NAME pops the value from the stack and associates the name a with that value.

Last time we also learned that all opcodes are implemented in a giant switch statement in Python/ceval.c, so we can see how the LOAD_NAME and STORE_NAME opcodes work by studying the corresponding cases of that switch. Let's start with the STORE_NAME opcode since we need to associate a name with some value before we can get the value of that name. Here's the case block that executes the STORE_NAME opcode:

case TARGET(STORE_NAME): {
    PyObject *name = GETITEM(names, oparg);
    PyObject *v = POP();
    PyObject *ns = f->f_locals;
    int err;
    if (ns == NULL) {
        _PyErr_Format(tstate, PyExc_SystemError,
                      "no locals found when storing %R", name);
        Py_DECREF(v);
        goto error;
    }
    if (PyDict_CheckExact(ns))
        err = PyDict_SetItem(ns, name, v);
    else
        err = PyObject_SetItem(ns, name, v);
    Py_DECREF(v);
    if (err != 0)
        goto error;
    DISPATCH();
}

Let's analyze what it does:

The names are strings. They are stored in a code object in a tuple called co_names. The names variable is just a shorthand for co_names. The argument of the STORE_NAME instruction is not a name but an index used to look up the name in co_names. The first thing the VM does is get the name, which it's going to assign a value to, from co_names.
The VM pops the value from the stack.
Values of variables are stored in a frame object. The f_locals field of a frame object is a mapping from the names of local variables to their values. The VM associates a name name with a value v by setting f_locals[name] = v.

We learn from this two crucial facts:

Python variables are names mapped to values.
Values of names are references to Python objects.

The logic for executing the LOAD_NAME opcode is a bit more complicated because the VM looks up the value of a name not only in f_locals but also in a few other places:

case TARGET(LOAD_NAME): {
    PyObject *name = GETITEM(names, oparg);
    PyObject *locals = f->f_locals;
    PyObject *v;

    if (locals == NULL) {
        _PyErr_Format(tstate, PyExc_SystemError,
                        "no locals when loading %R", name);
        goto error;
    }

    // look up the value in `f->f_locals`
    if (PyDict_CheckExact(locals)) {
        v = PyDict_GetItemWithError(locals, name);
        if (v != NULL) {
            Py_INCREF(v);
        }
        else if (_PyErr_Occurred(tstate)) {
            goto error;
        }
    }
    else {
        v = PyObject_GetItem(locals, name);
        if (v == NULL) {
            if (!_PyErr_ExceptionMatches(tstate, PyExc_KeyError))
                goto error;
            _PyErr_Clear(tstate);
        }
    }

    // look up the value in `f->f_globals` and `f->f_builtins`
    if (v == NULL) {
        v = PyDict_GetItemWithError(f->f_globals, name);
        if (v != NULL) {
            Py_INCREF(v);
        }
        else if (_PyErr_Occurred(tstate)) {
            goto error;
        }
        else {
            if (PyDict_CheckExact(f->f_builtins)) {
                v = PyDict_GetItemWithError(f->f_builtins, name);
                if (v == NULL) {
                    if (!_PyErr_Occurred(tstate)) {
                        format_exc_check_arg(
                                tstate, PyExc_NameError,
                                NAME_ERROR_MSG, name);
                    }
                    goto error;
                }
                Py_INCREF(v);
            }
            else {
                v = PyObject_GetItem(f->f_builtins, name);
                if (v == NULL) {
                    if (_PyErr_ExceptionMatches(tstate, PyExc_KeyError)) {
                        format_exc_check_arg(
                                    tstate, PyExc_NameError,
                                    NAME_ERROR_MSG, name);
                    }
                    goto error;
                }
            }
        }
    }
    PUSH(v);
    DISPATCH();
}

This code translates to English as follows:

As for the STORE_NAME opcode, the VM first gets the name of a variable.
The VM looks up the value of the name in the mapping of local variables: v = f_locals[name].
If the name is not in f_locals, the VM looks up the value in the dictionary of global variables f_globals. And if the name is not in f_globals either, the VM looks up the value in f_builtins. The f_builtins field of a frame object points to the dictionary of the builtins module, which contains built-in types, functions, exceptions and constants. If the name is not there, the VM gives up and sets the NameError exception.
If the VM finds the value, it pushes the value onto the stack.

The way the VM searches for a value has the following effects:

We always have the names from the builtin's dictionary, such as int, next, ValueError and None, at our disposal.
If we use a built-in name for a local variable or a global variable, the new variable will shadow the built-in one.
A local variable shadows the global variable with the same name.

Since all we need to be able to do with variables is to associate them with values and to get their values, you might think that the STORE_NAME and LOAD_NAME opcodes are sufficient to implement all variables in Python. This is not the case. Consider the example:

x = 1

def f(y, z):
    def _():
        return z

    return x + y + z

The function f has to load the values of variables x, y and z to add them and return the result. Note which opcodes the compiler produces to do that:

$ python -m dis global_fast_deref.py
...
  7          12 LOAD_GLOBAL              0 (x)
             14 LOAD_FAST                0 (y)
             16 BINARY_ADD
             18 LOAD_DEREF               0 (z)
             20 BINARY_ADD
             22 RETURN_VALUE
...

None of the opcodes are LOAD_NAME. The compiler produces the LOAD_GLOBAL opcode to load the value of x, the LOAD_FAST opcode to load the value of y and the LOAD_DEREF opcode to load the value of z. To see why the compiler produces different opcodes, we need to discuss two important concepts: namespaces and scopes.

Namespaces and scopes

A Python program consists of code blocks. A code block is a piece of code that the VM executes as a single unit. CPython distinguishes three types of code blocks:

module
function (comprehensions and lambdas are also functions)
class definition.

The compiler creates a code object for every code block in a program. A code object is a structure that describes what a code block does. In particular, it contains the bytecode of a block. To execute a code object, CPython creates a state of execution for it called a frame object. Besides other things, a frame object contains name-value mappings such as f_locals, f_globals and f_builtins. These mappings are referred to as namespaces. Each code block introduces a namespace: its local namespace. The same name in a program may refer to different variables in different namespaces:

x = y = "I'm a variable in a global namespace"

def f():
    x = "I'm a local variable"
    print(x)
    print(y)

print(x)
print(y)
f()

$ python namespaces.py 
I'm a variable in a global namespace
I'm a variable in a global namespace
I'm a local variable
I'm a variable in a global namespace

Another important notion is the notion of a scope. Here's what the Python documentation says about it:

A scope is a textual region of a Python program where a namespace is directly accessible. “Directly accessible” here means that an unqualified reference to a name attempts to find the name in the namespace.

We can think about a scope as a property of a name that tells where the value of that name is stored. The example of a scope is a local scope. The scope of a name is relative to a code block. The following example illustrates the point:

a = 1

def f():
    b = 3
    return a + b

Here, the name a refers to the same variable in both cases. From the function's perspective, it's a global variable, but from the module's perspective, it's both global and local. The variable b is local to the function f, but it doesn't exist at the module level at all.

The variable is considered to be local to a code block if it's bound in that code block. An assignment statement like a = 1 binds the name a to 1. An assignment statement, though, is not the only way to bind a name. The Python documentation lists a few more:

The following constructs bind names: formal parameters to functions, import statements, class and function definitions (these bind the class or function name in the defining block), and targets that are identifiers if occurring in an assignment, for loop header, or after as in a with statement or except clause. The import statement of the form from ... import * binds all names defined in the imported module, except those beginning with an underscore. This form may only be used at the module level.

Because any binding of a name makes the compiler think that the name is local, the following code raises an exception:

a = 1

def f():
    a += 1
    return a

print(f())

$ python unbound_local.py
...
    a += 1
UnboundLocalError: local variable 'a' referenced before assignment

The a += 1 statement is a form of assignment, so the compiler thinks that a is local. To perform the operation, the VM tries to load the value of a, fails and sets the exception. To tell the compiler that a is global despite the assignment, we can use the global statement:

a = 1

def f():
    global a
    a += 1
    print(a)

f()

$ python global_stmt.py 
2

Similarly, we can use the nonlocal statement to tell the compiler that a name bound in an enclosed (nested) function refers to a variable in an enclosing function:

a = "I'm not used"

def f():
    def g():
        nonlocal a
        a += 1
        print(a)
    a = 2
    g()

f()

$ python nonlocal_stmt.py
3

This is the work of the compiler to analyze the usage of names within a code block, take statements like global and nonlocal into account and produce the right opcodes to load and store values. In general, which opcode the compiler produces for a name depends on the scope of that name and on the type of the code block that is currently being compiled. The VM executes different opcodes differently. All of that is done to make Python variables work the way they do.

CPython uses four pairs of load/store opcodes and one more load opcode in total:

LOAD_FAST and STORE_FAST
LOAD_DEREF and STORE_DEREF
LOAD_GLOBAL and STORE_GLOBAL
LOAD_NAME and STORE_NAME; and
LOAD_CLASSDEREF.

Let's figure out what they do and why CPython needs all of them.

LOAD_FAST and STORE_FAST

The compiler produces the LOAD_FAST and STORE_FAST opcodes for variables local to a function. Here's an example:

def f(x):
    y = x
    return y

$ python -m dis fast_variables.py
...
  2           0 LOAD_FAST                0 (x)
              2 STORE_FAST               1 (y)

  3           4 LOAD_FAST                1 (y)
              6 RETURN_VALUE

The y variable is local to f because it's bound in f by the assignment. The x variable is local to f because it's bound in f as its parameter.

Let's look at the code that executes the STORE_FAST opcode:

case TARGET(STORE_FAST): {
    PREDICTED(STORE_FAST);
    PyObject *value = POP();
    SETLOCAL(oparg, value);
    FAST_DISPATCH();
}

SETLOCAL() is a macro that essentially expands to fastlocals[oparg] = value. The fastlocals variable is just a shorthand for the f_localsplus field of a frame object. This field is an array of pointers to Python objects. It stores values of local variables, cell variables, free variables and the value stack. Last time we learned that the f_localsplus array is used to store the value stack. In the next section of this post we'll see how it's used to store values of cell and free variables. For now, we're interested in the first part of the array that's used for local variables.

We've seen that in the case of the STORE_NAME opcode, the VM first gets the name from co_names and then maps that name to the value on top of the stack. It uses f_locals as a name-value mapping, which is usually a dictionary. In the case of the STORE_FAST opcode, the VM doesn't need to get the name. The number of local variables can be calculated statically by the compiler, so the VM can use an array to store their values. Each local variable can the be associated with an index of that array. To map a name to a value, the VM simply stores the value in the corresponding index.

The VM doesn't need to get the names of variables local to a function to load and store their values. Nevertheless, it stores these names in a function's code object in the co_varnames tuple. Why? Names are necessary for debugging and error messages. They are also used by the tools such as dis that reads co_varnames to display names in parentheses:

              2 STORE_FAST               1 (y)

CPython provides the locals() built-in function that returns the local namespace of the current code block in the form of a dictionary. The VM doesn't keep such a dictionary for functions but it can built one on the fly by mapping keys from co_varnames to values from f_localsplus.

The LOAD_FAST opcode simply pushes f_localsplus[oparg] onto the stack:

case TARGET(LOAD_FAST): {
    PyObject *value = GETLOCAL(oparg);
    if (value == NULL) {
        format_exc_check_arg(tstate, PyExc_UnboundLocalError,
                             UNBOUNDLOCAL_ERROR_MSG,
                             PyTuple_GetItem(co->co_varnames, oparg));
        goto error;
    }
    Py_INCREF(value);
    PUSH(value);
    FAST_DISPATCH();
}

The LOAD_FAST and STORE_FAST opcodes exist only for performance reasons. They are called *_FAST because the VM uses an array for the mapping, which works faster than a dictionary. What's the speed gain? Let's measure the difference between STORE_FAST and STORE_NAME. The following piece of code stores the value of the variable i 100 million times:

for i in range(10**8):
    pass

If we place it in a module, the compiler produces the STORE_NAME opcode. If we place it in a function, the compiler produces the STORE_FAST opcode. Let's do both and compare the running times:

import time


# measure STORE_NAME
times = []
for _ in range(5):
    start = time.time()
    for i in range(10**8):
        pass
    times.append(time.time() - start)

print('STORE_NAME: ' + ' '.join(f'{elapsed:.3f}s' for elapsed in sorted(times)))


# measure STORE_FAST
def f():
    times = []
    for _ in range(5):
        start = time.time()
        for i in range(10**8):
            pass
        times.append(time.time() - start)

    print('STORE_FAST: ' + ' '.join(f'{elapsed:.3f}s' for elapsed in sorted(times)))

f()

$ python fast_vs_name.py
STORE_NAME: 4.536s 4.572s 4.650s 4.742s 4.855s
STORE_FAST: 2.597s 2.608s 2.625s 2.628s 2.645s

Another difference in the implementation of STORE_NAME and STORE_FAST could theoretically affect these results. The case block for the STORE_FAST opcode ends with the FAST_DISPATCH() macro, which means that the VM goes to the next instruction straight away after it executes the STORE_FAST instruction. The case block for the STORE_NAME opcode ends with the DISPATCH() macro, which means that the VM may possible go to the start of the evaluation loop. At the start of the evaluation loop the VM checks whether it has to suspend the bytecode execution, for example, to release the GIL or to handle the signals. I've replaced the DISPATCH() macro with FAST_DISPATCH() in the case block for STORE_NAME, recompiled CPython and got similar results. So, the difference in times should indeed be explained by:

the extra step to get a name; and
the fact that a dictionary is slower than an array.

LOAD_DEREF and STORE_DEREF

There is one case when the compiler doesn't produce the LOAD_FAST and STORE_FAST opcodes for variables local to a function. This happens when a variable is used within a nested function.

def f():
    b = 1
    def g():
        return b

$ python -m dis nested.py
...
Disassembly of <code object f at 0x1027c72f0, file "nested.py", line 1>:
  2           0 LOAD_CONST               1 (1)
              2 STORE_DEREF              0 (b)

  3           4 LOAD_CLOSURE             0 (b)
              6 BUILD_TUPLE              1
              8 LOAD_CONST               2 (<code object g at 0x1027c7240, file "nested.py", line 3>)
             10 LOAD_CONST               3 ('f.<locals>.g')
             12 MAKE_FUNCTION            8 (closure)
             14 STORE_FAST               0 (g)
             16 LOAD_CONST               0 (None)
             18 RETURN_VALUE

Disassembly of <code object g at 0x1027c7240, file "nested.py", line 3>:
  4           0 LOAD_DEREF               0 (b)
              2 RETURN_VALUE

The compiler produces the LOAD_DEREF and STORE_DEREF opcodes for cell and free variables. A cell variable is a local variable referenced in a nested function. In our example, b is a cell variable of the function f, because it's referenced by g. A free variable is a cell variable from the perspective of a nested function. It's a variable not bound in a nested function but bound in the enclosing function or a variable declared nonlocal. In our example, b is a free variable of the function g, because it's not bound in g but bound in f.

The values of cell and free variables are stored in the f_localsplus array after the values of normal local variables. The only difference is that f_localsplus[index_of_cell_or_free_variable] points not to the value directly but to a cell object containing the value:

typedef struct {
    PyObject_HEAD
    PyObject *ob_ref;       /* Content of the cell or NULL when empty */
} PyCellObject;

The STORE_DEREF opcode pops the value from the stack, gets the cell of the variable specified by oparg and assigns ob_ref of that cell to the popped value:

case TARGET(STORE_DEREF): {
    PyObject *v = POP();
    PyObject *cell = freevars[oparg]; // freevars = f->f_localsplus + co->co_nlocals
    PyObject *oldobj = PyCell_GET(cell);
    PyCell_SET(cell, v); // expands to ((PyCellObject *)(cell))->ob_ref = v
    Py_XDECREF(oldobj);
    DISPATCH();
}

The LOAD_DEREF opcode works by pushing the contents of a cell onto the stack:

case TARGET(LOAD_DEREF): {
    PyObject *cell = freevars[oparg];
    PyObject *value = PyCell_GET(cell);
    if (value == NULL) {
      format_exc_unbound(tstate, co, oparg);
      goto error;
    }
    Py_INCREF(value);
    PUSH(value);
    DISPATCH();
}

What's the reason to store values in cells? This is done to connect a free variable with the corresponding cell variable. Their values are stored in different namespaces in different frame objects but in the same cell. The VM passes the cells of an enclosing function to the enclosed function when it creates the enclosed function. The LOAD_CLOSURE opcode pushes a cell onto the stack and the MAKE_FUNCTION opcode creates a function object with that cell for the corresponding free variable. Due to the cell mechanism, when an enclosing function reassigns a cell variable, an enclosed function sees the reassignment:

def f():
    def g():
        print(a)
    a = 'assigned'
    g()
    a = 'reassigned'
    g()

f()

$ python cell_reassign.py 
assigned
reassigned

and vice versa:

def f():
    def g():
        nonlocal a
        a = 'reassigned'
    a = 'assigned'
    print(a)
    g()
    print(a)

f()

$ python free_reassign.py 
assigned
reassigned

Do we really need the cell mechanism to implement such behavior? Couldn't we just use the enclosing namespace to load and store values of free variables? Yes, we could, but consider the following example:

def get_counter(start=0):
    def count():
        nonlocal c
        c += 1
        return c

    c = start - 1
    return count

count = get_counter()
print(count())
print(count())

$ python counter.py 
0
1

Recall that when we call a function, CPython creates a frame object to execute it. This example shows that an enclosed function can outlive the frame object of an enclosing function. The benefit of the cell mechanism is that it allows to avoid keeping the frame object of an enclosing function and all its references in memory.

LOAD_GLOBAL and STORE_GLOBAL

The compiler produces the LOAD_GLOBAL and STORE_GLOBAL opcodes for global variables in functions. The variable is considered to be global in a function if it's declared global or if it's not bound within the function and any enclosing function (i.e. it's neither local nor free). Here's an example:

a = 1
d = 1

def f():
    b = 1
    def g():
        global d
        c = 1
        d = 1
        return a + b + c + d

The c variable is not global to g because it's local to g. The b variable is not global to g because it's free. The a variable is global to g because it's neither local nor free. And the d variable is global to g because it's explicitly declared global.

Here's the implementation of the STORE_GLOBAL opcode:

case TARGET(STORE_GLOBAL): {
    PyObject *name = GETITEM(names, oparg);
    PyObject *v = POP();
    int err;
    err = PyDict_SetItem(f->f_globals, name, v);
    Py_DECREF(v);
    if (err != 0)
        goto error;
    DISPATCH();
}

The f_globals field of a frame object is a dictionary that maps global names to their values. When CPython creates a frame object for a module, it assigns f_globals to the dictionary of the module. We can easily check this:

$ python -q
>>> import sys
>>> globals() is sys.modules['__main__'].__dict__
True

When the VM executes the MAKE_FUNCTION opcode to create a new function object, it assigns the func_globals field of that object to f_globals of the current frame object. When the function gets called, the VM creates a new frame object for it with f_globals set to func_globals.

The implementation of LOAD_GLOBAL is similar to that of LOAD_NAME with two exceptions:

It doesn't look up values in f_locals.
It uses cache to decrease the lookup time.

CPython caches the results in a code object in the co_opcache array. This array stores pointers to the _PyOpcache structs:

typedef struct {
    PyObject *ptr;  /* Cached pointer (borrowed reference) */
    uint64_t globals_ver;  /* ma_version of global dict */
    uint64_t builtins_ver; /* ma_version of builtin dict */
} _PyOpcache_LoadGlobal;

struct _PyOpcache {
    union {
        _PyOpcache_LoadGlobal lg;
    } u;
    char optimized;
};

The ptr field of the _PyOpcache_LoadGlobal struct points to the actual result of LOAD_GLOBAL. The cache is maintained per instruction number. Another array in a code object called co_opcache_map maps each instruction in the bytecode to its index minus one in co_opcache. If an instruction is not LOAD_GLOBAL, it maps the instruction to 0, which means that the instruction is never cached. The size of the cache doesn't exceed 254. If the bytecode contains more than 254 LOAD_GLOBAL instructions, co_opcache_map maps extra instructions to 0 as well.

If the VM finds a value in the cache when it executes LOAD_GLOBAL, it makes sure that the f_global and f_builtins dictionaries haven't been modified since the last time the value was looked up. This is done by comparing globals_ver and builtins_ver with ma_version_tag of the dictionaries. The ma_version_tag field of a dictionary changes each time the dictionary is modified. See PEP 509 for more details.

If the VM doesn't find a value in the cache, it does a normal look up first in f_globals and then in f_builtins. If it eventually finds a value, it remembers current ma_version_tag of both dictionaries and pushes the value onto the stack.

LOAD_NAME and STORE_NAME (and LOAD_CLASSDEREF)

At this point you might wonder why CPython uses the LOAD_NAME and STORE_NAME opcodes at all. The compiler indeed doesn't produce these opcodes when it compiles functions. However, besides function, CPython has two other types of code blocks: modules and class definitions. We haven't talked about class definitions at all, so let's fix it.

First, it's crucial to understand that when we define a class, the VM executes its body. Here's what I mean:

class A:
    print('This code is executed')

$ python create_class.py 
This code is executed

The compiler creates code objects for class definitions just as it creates code objects for modules and functions. What's interesting is that the compiler almost always produces the LOAD_NAME and STORE_NAME opcodes for variables within a class body. There are two rare exceptions to this rule: free variables and variables explicitly declared global.

The VM executes *_NAME opcodes and *_FAST opcodes differently. As a result, variables work differently in a class body than they do in a function:

x = 'global'

class C:
    print(x)
    x = 'local'
    print(x)

$ python class_local.py
global
local

On the first load, the VM loads the value of the x variable from f_globals. Then, it stores the new value in f_locals and, on the second load, loads it from there. If C was a function, we would get UnboundLocalError: local variable 'x' referenced before assignment when we call it, because the compiler would think that the x variable is local to C.

How do the namespaces of classes and functions interplay? When we place a function inside a class, which is a common practice to implement methods, the function doesn't see the names bound in the class' namespace:

class D:
    x = 1
    def method(self):
        print(x)

D().method()

$ python func_in_class.py
...
NameError: name 'x' is not defined

This is because the VM stores the value of x with STORE_NAME when it executes the class definition and tries to load it with LOAD_GLOBAL when it executes the function. However, when we place a class definition inside a function, the cell mechanism works as if we place a function inside a function:

def f():
    x = "I'm a cell variable"
    class B:
        print(x)

f()

$ python class_in_func.py 
I'm a cell variable

There's a difference, though. The compiler produces the LOAD_CLASSDEREF opcode instead of LOAD_DEREF to load the value of x. The documentation of the dis module explains what LOAD_CLASSDEREF does:

Much like LOAD_DEREF but first checks the locals dictionary before consulting the cell. This is used for loading free variables in class bodies.

Why does it check the locals dictionary first? In the case of a function, the compiler knows for sure if a variable is local or not. In the case of a class, the compiler cannot be sure. This is because CPython has metaclasses, and a metaclass may prepare a non-empty locals dictionary for a class by implementing the __prepare__ method.

We can see now why the compiler produces the LOAD_NAME and STORE_NAME opcodes for class definitions but we also saw that it produces these opcodes for variables within the module's namespace, as in the a = b example. They work as expected because module's f_locals and module's f_globals is the same thing:

$ python -q
>>> locals() is globals()
True

You might wonder why CPython doesn't use the LOAD_GLOBAL and STORE_GLOBAL opcodes in this case. Honestly, I don't know the exact reason, if there is any, but I have a guess. CPython provides the built-in compile(), eval() and exec() functions that can be used to dynamically compile and execute Python code. These functions use the LOAD_NAME and STORE_NAME opcodes within the top-level namespace. It makes perfect sense because it allows to execute code dynamically in a class body and get the same effect as if that code was written there:

a = 1

class A:
    b = 2
    exec('print(a + b)', globals(), locals())

$ python exec.py
3

CPython chose to always use the LOAD_NAME and STORE_NAME opcodes for modules. In this way, the bytecode the compiler produces when we run a module in a normal way is the same as when we execute the module with exec().

How the compiler decides which opcode to produce

We learned in part 2 of this series that before the compiler creates a code object for a code block, it builds a symbol table for that block. A symbol table contains information about symbols (i.e. names) used within a code block including their scopes. The compiler decides which load/store opcode to produce for a given name based on its scope and the type of the code block that is currently being compiled. The algorithm can be summarized as follows:

Determine the scope of the variable:
1. If the variable declared global, it's an explicit global variable.
2. If the variable declared nonlocal, it's a free variable.
3. If the variable is bound within the current code block, it's a local variable.
4. If the variable is bound in the enclosing code block that is not a class definition, it's a free variable.
5. Otherwise, it's a implicit global variable.
Update the scope:
1. If the variable is local and and it's free in the enclosed code block, it's a cell variable.
Decide which opcode to produce:
1. If the variable is a cell variable or a free variable, produce *_DEREF opcode; produce the LOAD_CLASSDEREF opcode to load the value if the current code block is a class definition.
2. If the variable is a local variable and the current code block is a function, produce *_FAST opcode.
3. If the variable is an explicit global variable or if it's an implicit global variable and the current code block is a function, produce *_GLOBAL opcode.
4. Otherwise, produce *_NAME opcode.

You don't need to remember these rules. You can always read the source code. Check out Python/symtable.c to see how the compiler determines the scope of a variable, and Python/compile.c to see how it decides which opcode to produce.

Conclusion

The topic of Python variables is much more complicated than it may seem at first. A good portion of the Python documentation is related to variables, including a section on naming and binding and a section on scopes and namespaces. The top questions of the Python FAQ are about variables. I say nothing about questions on Stack Overflow. While the official resources give some idea why Python variables work the way they do, it's still hard to understand and remember all the rules. Fortunately, it's easier to understand how Python variables work by studying the source code of the Python implementation. And that's what we did today.

We've studied a group of opcodes that CPython uses to load and store values of variables. To understand how the VM executes other opcodes that actually compute something, we need to discuss the core of Python – Python object system. This is our plan for the next time.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #4: how Python bytecode is executed

2020-10-30T13:50:00+00:00

We started this series with an overview of the CPython VM. We learned that to run a Python program, CPython first compiles it to bytecode, and we studied how the compiler works in part two. Last time we stepped through the CPython source code starting with the main() function until we reached the evaluation loop, a place where Python bytecode gets executed. The main reason why we spent time studying these things was to prepare for the discussion that we start today. The goal of this discussion is to understand how CPython does what we tell it to do, that is, how it executes the bytecode to which the code we write compiles.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

Starting point

Let's briefly recall what we learned in the previous parts. We tell CPython what to do by writing Python code. The CPython VM, however, understands only Python bytecode. This is the job of the compiler to translate Python code to bytecode. The compiler stores bytecode in a code object, which is a structure that fully describes what a code block, like a module or a function, does. To execute a code object, CPython first creates a state of execution for it called a frame object. Then it passes a frame object to a frame evaluation function to perform the actual computation. The default frame evaluation function is _PyEval_EvalFrameDefault() defined in Python/ceval.c. This function implements the core of the CPython VM. Namely, it implements the logic for the execution of Python bytecode. So, this function is what we're going to study today.

To understand how _PyEval_EvalFrameDefault() works, it is crucial to have an idea of what its input, a frame object, is. A frame object is a Python object defined by the following C struct:

// typedef struct _frame PyFrameObject; in other place
struct _frame {
    PyObject_VAR_HEAD
    struct _frame *f_back;      /* previous frame, or NULL */
    PyCodeObject *f_code;       /* code segment */
    PyObject *f_builtins;       /* builtin symbol table (PyDictObject) */
    PyObject *f_globals;        /* global symbol table (PyDictObject) */
    PyObject *f_locals;         /* local symbol table (any mapping) */
    PyObject **f_valuestack;    /* points after the last local */
    /* Next free slot in f_valuestack.  Frame creation sets to f_valuestack.
       Frame evaluation usually NULLs it, but a frame that yields sets it
       to the current stack top. */
    PyObject **f_stacktop;
    PyObject *f_trace;          /* Trace function */
    char f_trace_lines;         /* Emit per-line trace events? */
    char f_trace_opcodes;       /* Emit per-opcode trace events? */

    /* Borrowed reference to a generator, or NULL */
    PyObject *f_gen;

    int f_lasti;                /* Last instruction if called */
    int f_lineno;               /* Current line number */
    int f_iblock;               /* index in f_blockstack */
    char f_executing;           /* whether the frame is still executing */
    PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */
    PyObject *f_localsplus[1];  /* locals+stack, dynamically sized */
};

The f_code field of a frame object points to a code object. A code object is also a Python object. Here's its definition:

struct PyCodeObject {
    PyObject_HEAD
    int co_argcount;            /* #arguments, except *args */
    int co_posonlyargcount;     /* #positional only arguments */
    int co_kwonlyargcount;      /* #keyword only arguments */
    int co_nlocals;             /* #local variables */
    int co_stacksize;           /* #entries needed for evaluation stack */
    int co_flags;               /* CO_..., see below */
    int co_firstlineno;         /* first source line number */
    PyObject *co_code;          /* instruction opcodes */
    PyObject *co_consts;        /* list (constants used) */
    PyObject *co_names;         /* list of strings (names used) */
    PyObject *co_varnames;      /* tuple of strings (local variable names) */
    PyObject *co_freevars;      /* tuple of strings (free variable names) */
    PyObject *co_cellvars;      /* tuple of strings (cell variable names) */
    /* The rest aren't used in either hash or comparisons, except for co_name,
       used in both. This is done to preserve the name and line number
       for tracebacks and debuggers; otherwise, constant de-duplication
       would collapse identical functions/lambdas defined on different lines.
    */
    Py_ssize_t *co_cell2arg;    /* Maps cell vars which are arguments. */
    PyObject *co_filename;      /* unicode (where it was loaded from) */
    PyObject *co_name;          /* unicode (name, for reference) */
    PyObject *co_lnotab;        /* string (encoding addr<->lineno mapping) See
                                   Objects/lnotab_notes.txt for details. */
    void *co_zombieframe;       /* for optimization only (see frameobject.c) */
    PyObject *co_weakreflist;   /* to support weakrefs to code objects */
    /* Scratch space for extra data relating to the code object.
       Type is a void* to keep the format private in codeobject.c to force
       people to go through the proper APIs. */
    void *co_extra;

    /* Per opcodes just-in-time cache
     *
     * To reduce cache size, we use indirect mapping from opcode index to
     * cache object:
     *   cache = co_opcache[co_opcache_map[next_instr - first_instr] - 1]
     */

    // co_opcache_map is indexed by (next_instr - first_instr).
    //  * 0 means there is no cache for this opcode.
    //  * n > 0 means there is cache in co_opcache[n-1].
    unsigned char *co_opcache_map;
    _PyOpcache *co_opcache;
    int co_opcache_flag;  // used to determine when create a cache.
    unsigned char co_opcache_size;  // length of co_opcache.
};

The most important field of a code object is co_code. It's a pointer to a Python bytes object representing the bytecode. The bytecode is a sequence of two-byte instructions: one byte for an opcode and one byte for an argument.

Don't worry if some members of the above structures are still a mystery to you. We'll see what they are used for as we move forward in our attempt to understand how the CPython VM executes the bytecode.

Overview of the evaluation loop

The problem of executing Python bytecode may seem a no-brainer to you. Indeed, all the VM has to do is to iterate over the instructions and to act according to them. And this is what essentially _PyEval_EvalFrameDefault() does. It contains an infinite for (;;) loop that we refer to as the evaluation loop. Inside that loop there is a giant switch statement over all possible opcodes. Each opcode has a corresponding case block containing the code for executing that opcode. The bytecode is represented by an array of 16-bit unsigned integers, one integer per instruction. The VM keeps track of the next instruction to be executed using the next_instr variable, which is a pointer to the array of instructions. At the start of each iteration of the evaluation loop, the VM calculates the next opcode and its argument by taking the least significant and the most significant byte of the next instruction respectively and increments next_instr. The _PyEval_EvalFrameDefault() function is nearly 3000 lines long, but its essence can be captured by the following simplified version:

PyObject*
_PyEval_EvalFrameDefault(PyThreadState *tstate, PyFrameObject *f, int throwflag)
{
    // ... declarations and initialization of local variables
    // ... macros definitions
    // ... call depth handling
    // ... code for tracing and profiling

    for (;;) {
        // ... check if the bytecode execution must be suspended,
        // e.g. other thread requested the GIL

        // NEXTOPARG() macro
        _Py_CODEUNIT word = *next_instr; // _Py_CODEUNIT is a typedef for uint16_t
        opcode = _Py_OPCODE(word);
        oparg = _Py_OPARG(word);
        next_instr++;

        switch (opcode) {
            case TARGET(NOP) {
                FAST_DISPATCH(); // more on this later
            }

            case TARGET(LOAD_FAST) {
                // ... code for loading local variable
            }

            // ... 117 more cases for every possible opcode
        }

        // ... error handling
    }

    // ... termination
}

To get a more realistic picture, let's discuss some of the omitted pieces in more detail.

reasons to suspend the loop

From time to time, the currently running thread stops executing the bytecode to do something else or to do nothing. This can happen due to one of the four reasons:

There are signals to handle. When you register a function as a signal handler using signal.signal(), CPython stores this function in the array of handlers. The function that will actually be called when a thread receives a signal is signal_handler() (it's passed to the sigaction() library function on Unix-like systems). When called, signal_handler() sets a boolean variable telling that the function in the array of handlers corresponding to the received signal has to be called. Periodically, the main thread of the main interpreter calls the tripped handlers.
There are pending calls to call. Pending calls is a mechanism that allows to schedule a function to be executed from the main thread. This mechanism is exposed by the Python/C API via the Py_AddPendingCall() function.
The asynchronous exception is raised. The asynchronous exception is an exception set in one thread from another. This can be done using the PyThreadState_SetAsyncExc() function provided by the Python/C API.
The currently running thread is requested to drop the GIL. When it sees such a request, it drops the GIL and waits until it acquires the GIL again.

CPython has indicators for each of these events. The variable indicating that there are handlers to call is a member of runtime->ceval, which is a _ceval_runtime_state struct:

struct _ceval_runtime_state {
    /* Request for checking signals. It is shared by all interpreters (see
       bpo-40513). Any thread of any interpreter can receive a signal, but only
       the main thread of the main interpreter can handle signals: see
       _Py_ThreadCanHandleSignals(). */
    _Py_atomic_int signals_pending;
    struct _gil_runtime_state gil;
};

Other indicators are members of interp->ceval, which is a _ceval_state struct:

struct _ceval_state {
    int recursion_limit;
    /* Records whether tracing is on for any thread.  Counts the number
       of threads for which tstate->c_tracefunc is non-NULL, so if the
       value is 0, we know we don't have to check this thread's
       c_tracefunc.  This speeds up the if statement in
       _PyEval_EvalFrameDefault() after fast_next_opcode. */
    int tracing_possible;
    /* This single variable consolidates all requests to break out of
       the fast path in the eval loop. */
    _Py_atomic_int eval_breaker;
    /* Request for dropping the GIL */
    _Py_atomic_int gil_drop_request;
    struct _pending_calls pending;
};

The result of ORing all indicators together is stored in the eval_breaker variable. It tells whether there is any reason for the currently running thread to stop its normal bytecode execution. Each iteration of the evaluation loop starts with the check whether eval_breaker is true. If it is true, the thread checks the indicators to determine what exactly it is asked to do, does that and continues to execute the bytecode.

computed GOTOs

The code for the evaluation loop is full of macros such as TARGET() and DISPATCH(). These are not the means to make the code more compact. They expand to different code depending on whether the certain optimization, referred to as "computed GOTOs" (a.k.a. "threaded code"), is used. They goal of this optimization is to speed up the bytecode execution by writing code in such a way, so that a CPU can use its branch prediction mechanism to predict the next opcode.

After executing any given instruction, the VM does one of the three things:

It returns from the evaluation function. This happens when the VM executes RETURN_VALUE, YIELD_VALUE or YIELD_FROM instruction.
It handles the error and either continues the execution or returns from the evaluation function with the exception set. The error can occur when, for example, the VM executes the BINARY_ADD instruction and the objects to be added do not implement __add__ and __radd__ methods.
It continues the execution. How to make the VM execute the next instruction? The simplest solution would be to end each non-returning case block with the continue statement. The real solution, though, is a little bit more complicated.

To see the problem with the simple continue statement, we need to understand what switch compiles to. An opcode is an integer between 0 and 255. Because the range is dense, the compiler can create a jump table that stores addresses of the case blocks and use opcodes as indices into that table. The modern compilers indeed do that, so the dispatching of cases is effectively implemented as a single indirect jump. This is an efficient way to implement switch. However, placing switch inside the loop and adding continue statements creates two inefficiencies:

The continue statement in the end of a case block adds another jump. Thus, to execute an opcode, the VM has to jump twice: to the start of the loop and then to the next case block.
Since all opcodes are dispatched by a single jump, a CPU has a little chance of predicting the next opcode. The best it can do is to choose the last opcode or, possibly, the most frequent one.

The idea of the optimization is to place a separate dispatch jump in the end of each non-returning case block. First, it saves a jump. Second, a CPU can predict the next opcode as the most probable opcode following the current one.

The optimization can be enabled or disabled. It depends on whether the compiler supports the GCC C extension called "labels as values" or not. The effect of enabling the optimization is that the certain macros, such as TARGET(), DISPATCH() and FAST_DISPATCH(), expand in different way. These macros are used extensively throughout the code of the evaluation loop. Every case expression has a form TARGET(op), where op is a macro for the integer literal representing an opcode. And every non-returning case block ends with DISPATCH() or FAST_DISPATCH() macro. Let's first look at what these macros expand to when the optimization is disabled:

for (;;) {
    // ... check if the bytecode execution must be suspended

fast_next_opcode:
    // NEXTOPARG() macro
    _Py_CODEUNIT word = *next_instr;
    opcode = _Py_OPCODE(word);
    oparg = _Py_OPARG(word);
    next_instr++;

    switch (opcode) {
        // TARGET(NOP) expands to NOP
        case NOP: {
            goto fast_next_opcode; // FAST_DISPATCH() macro
        }

        // ...

        case BINARY_MULTIPLY: {
            // ... code for binary multiplication
            continue; // DISPATCH() macro
        }

        // ...
    }

    // ... error handling
}

The FAST_DISPATCH() macro is used for some opcode when it's undesirable to suspend the evaluation loop after executing that opcode. Otherwise, the implementation is very straightforward.

If the compiler supports "labels as values" extension, we can use the unary && operator on a label to get its address. It has a value of type void *, so we can store it in a pointer:

void *ptr = &&my_label;

We can then go to the label by dereferencing the pointer:

goto *ptr;

This extension allows to implement a jump table in C as an array of label pointers. And that's what CPython does:

static void *opcode_targets[256] = {
    &&_unknown_opcode,
    &&TARGET_POP_TOP,
    &&TARGET_ROT_TWO,
    &&TARGET_ROT_THREE,
    &&TARGET_DUP_TOP,
    &&TARGET_DUP_TOP_TWO,
    &&TARGET_ROT_FOUR,
    &&_unknown_opcode,
    &&_unknown_opcode,
    &&TARGET_NOP,
    &&TARGET_UNARY_POSITIVE,
    &&TARGET_UNARY_NEGATIVE,
    &&TARGET_UNARY_NOT,
    // ... quite a few more
};

Here's what the optimized version of the evaluation loop looks like:

for (;;) {
    // ... check if the bytecode execution must be suspended

fast_next_opcode:
    // NEXTOPARG() macro
    _Py_CODEUNIT word = *next_instr;
    opcode = _Py_OPCODE(word);
    oparg = _Py_OPARG(word);
    next_instr++;

    switch (opcode) {
        // TARGET(NOP) expands to NOP: TARGET_NOP:
        // TARGET_NOP is a label
        case NOP: TARGET_NOP: {
            // FAST_DISPATCH() macro
            // when tracing is disabled
            f->f_lasti = INSTR_OFFSET();
            NEXTOPARG();
            goto *opcode_targets[opcode];
        }

        // ...

        case BINARY_MULTIPLY: TARGET_BINARY_MULTIPLY: {
            // ... code for binary multiplication
            // DISPATCH() macro
            if (!_Py_atomic_load_relaxed(eval_breaker)) {
              FAST_DISPATCH();
            }
            continue;
        }

        // ...
    }

    // ... error handling
}

The extension is supported by the GCC and Clang compilers. So, when you run python, you probably have the optimization enabled. The question, of course, is how it affects the performance. Here, I'll rely on the comment from the source code:

At the time of this writing, the "threaded code" version is up to 15-20% faster than the normal "switch" version, depending on the compiler and the CPU architecture.

This section should give us an idea of how the CPython VM goes from one instruction to the next and what it may do in between. The next logical step is to study in more depth how the VM executes a single instruction. CPython 3.9 has 119 different opcodes. Of course, we're not going to study the implementation of each opcode in this post. Instead, we'll focus on the general principles that the VM uses to execute them.

Value stack

The most important and, fortunately, very simple fact about the CPython VM is that it's stack-based. This means that to compute things, the VM pops (or peeks) values from the stack, performs the computation on them and pushes the result back. Here's some examples:

The UNARY_NEGATIVE opcode pops value from the stack, negates it and pushes the result.
The GET_ITER opcode pops value from the stack, calls iter() on it and pushes the result.
The BINARY_ADD opcode pops value from the stack, peeks another value from the top, adds the first value to the second and replaces the top value with the result.

The value stack resides in a frame object. It's implemented as a part of the array called f_localsplus. The array is split into several parts to store different things, but only the last part is used for the value stack. The start of this part is the bottom of the stack. The f_valuestack field of a frame object points to it. To locate the top of the stack, CPython keeps the stack_pointer local variable, which points to the next slot after the top of the stack. The elements of the f_localsplus array are pointers to Python objects, and pointers to Python objects is what the CPython VM actually works with.

Error handling and block stack

Not all computations performed by the VM are successful. Suppose we try to add a number to a string like 1 + '41'. The compiler produces the BINARY_ADD opcode to add two objects. When the VM executes this opcode, it calls PyNumber_Add() to calculate the result:

case TARGET(BINARY_ADD): {
    PyObject *right = POP();
    PyObject *left = TOP();
    PyObject *sum;
    // ... special case of string addition
    sum = PyNumber_Add(left, right);
    Py_DECREF(left);
    Py_DECREF(right);
    SET_TOP(sum);
    if (sum == NULL)
        goto error;
    DISPATCH();
}

What's important for us now is not how PyNumber_Add() is implemented, but that the call to it results in an error. The error means two things:

PyNumber_Add() returns NULL.
PyNumber_Add() sets the current exception to the TypeError exception. This involves setting tstate->curexc_type, tstate->curexc_value and tstate->curexc_traceback.

NULL is an indicator for an error. The VM sees it and goes to the error label at the end of the evaluation loop. What happens next depends on whether we've set up any exception handlers or not. If we haven't, the VM reaches the break statement and the evaluation function returns NULL with the exception set on the thread state. CPython prints the details of the exception and exits. We get the expected result:

$ python -c "1 + '42'"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

But suppose that we place the same code inside the try clause of the try-finally statement. In this case, the code inside the finally clause is executed as well:

$ python -q
>>> try:
...     1 + '41'
... finally:
...     print('Hey!')
... 
Hey!
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

How can the VM continue the execution after the error has occurred? Let's look at the bytecode produced by the compiler for the try-finally statement:

$ python -m dis try-finally.py

  1           0 SETUP_FINALLY           20 (to 22)

  2           2 LOAD_CONST               0 (1)
              4 LOAD_CONST               1 ('41')
              6 BINARY_ADD
              8 POP_TOP
             10 POP_BLOCK

  4          12 LOAD_NAME                0 (print)
             14 LOAD_CONST               2 ('Hey!')
             16 CALL_FUNCTION            1
             18 POP_TOP
             20 JUMP_FORWARD            10 (to 32)
        >>   22 LOAD_NAME                0 (print)
             24 LOAD_CONST               2 ('Hey!')
             26 CALL_FUNCTION            1
             28 POP_TOP
             30 RERAISE
        >>   32 LOAD_CONST               3 (None)
             34 RETURN_VALUE

Note the SETUP_FINALLY and POP_BLOCK opcodes. The first one sets up the exception handler and the second one removes it. If an error occurs while the VM executes the instructions between them, the execution continues with the instruction at offset 22, which is the start of the finally clause. Otherwise, the finally clause is executed after the try clause. In both cases, the bytecode for the finally clause is almost identical. The only difference is that the handler re-raises the exception set in the try clause.

An exception handler is implemented as a simple C struct called block:

typedef struct {
    int b_type;                 /* what kind of block this is */
    int b_handler;              /* where to jump to find handler */
    int b_level;                /* value stack level to pop to */
} PyTryBlock;

The VM keeps blocks in the block stack. To setup an exception handler means to push a new block onto the block stack. This is what opcodes like SETUP_FINALLY do. The error label points to a piece of code that tries to handle an error using blocks on the block stack. The VM unwinds the block stack until it finds the top-most block of type SETUP_FINALLY. It restores the level of the value stack to the level specified by the b_level field of the block and continues to execute the bytecode with the instruction at offset b_handler. This is basically how CPython implements statements like try-except, try-finally and with.

There is one more thing to say about exception handling. Think of what happens when an error occurs while the VM handles an exception:

$ python -q
>>> try:
...     1 + '41'
... except:
...     1/0
... 
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
TypeError: unsupported operand type(s) for +: 'int' and 'str'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 4, in <module>
ZeroDivisionError: division by zero

As expected, CPython prints the original exception. To implement such behavior, when CPython handles an exception using a SETUP_FINALLY block, it sets up another block of type EXCEPT_HANDLER. If an error occurs when a block of this type is on the block stack, the VM gets the original exception from the value stack and sets it as the current one. CPython used to have different kinds of blocks but now it's only SETUP_FINALLY and EXCEPT_HANDLER.

The block stack is implemented as the f_blockstack array in a frame object. The size of the array is statically defined to 20. So, if you nest more than 20 try clauses, you get SyntaxError: too many statically nested blocks.

Summary

Today we've learned that the CPython VM executes bytecode instructions one by one in an infinite loop. The loop contains a switch statement over all possible opcodes. Each opcode is executed in the corresponding case block. The evaluation function runs in a thread and sometimes that thread suspends the loop to do something else. For example, a thread may need to release the GIL, so that other thread can take it and continue to execute its bytecode. To speed up the bytecode execution, CPython employs an optimization that allows to make use of the CPU's branch prediction mechanism. A comment says that it makes CPython 15-20% faster.

We've also looked at two data structures crucial for the bytecode execution:

the value stack that the VM uses to compute things; and
the block stack that the VM uses to handle exceptions.

The most important conclusion from the post is this: if you want to study the implementation of some aspect of Python, the evaluation loop is a perfect place to start. Want to know what happens when you write x + y? Take a look at the code for the BINARY_ADD opcode. Want to know how the with statement is implemented? See SETUP_WITH. Interested in the exact semantics of a function call? The CALL_FUNCTION opcode is what you're looking for. We'll apply this method next time when study how variables are implemented in CPython.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Update from November 10, 2020: I was pointed out in the comments on HN that the UNARY_NEGATIVE and GET_ITER opcodes peek the value on top of the stack and replace it with the result instead of popping the value and pushing the result. This is indeed so. Semantically, the two approaches are equivalent. The only difference is that the pop/push approach first decrements and then increments stack_pointer. CPython avoids these redundant operations.

Python behind the scenes #3: stepping through the CPython source code

2020-10-11T07:02:00+00:00

In the first and the second parts of this series we explored the ideas behind the execution and the compilation of a Python program. We'll continue to focus on ideas in the next parts but this time we'll make an exception and look at the actual code that brings those ideas to life.

Plan for today

The CPython codebase is around 350,000 lines of C code (excluding header files) and almost 600,000 lines of Python code. Undoubtedly, it would be a daunting task to comprehend all of that at once. Today we'll confine our study to that part of the source code that executes every time we run python. We'll start with the main() function of the python executable and step through the source code until we reach the evaluation loop, a place where Python bytecode gets executed.

Our goal is not to understand every piece of code we'll encounter but to highlight the most interesting parts, study them, and, eventually, get an approximate idea of what happens at the very start of the execution of a Python program.

There are two more notices I should make. First, we won't step into every function. We'll make only a high-level overview of some parts and dive deep into others. Nevertheless, I promise to present functions in the order of execution. Second, with the exception of a few struct definitions, I'll leave the code as is. The only thing I'll allow myself is to add some comments and to rephrase existing ones. Throughout this post, all multi-line /**/ comments are original, and all single-line // comments are mine. With that said, let's begin our journey through the CPython source code.

Getting CPython

Before we can explore the source code, we need to get it. Let's clone the CPython repository:

$ git clone https://github.com/python/cpython/ && cd cpython

The current master branch is the future CPython 3.10. We're interested in the latest stable release, which is CPython 3.9, so let's switch to the 3.9 branch:

$ git checkout 3.9

Inside the root directory we find the following contents:

$ ls -p
CODE_OF_CONDUCT.md      Objects/                config.sub
Doc/                    PC/                     configure
Grammar/                PCbuild/                configure.ac
Include/                Parser/                 install-sh
LICENSE                 Programs/               m4/
Lib/                    Python/                 netlify.toml
Mac/                    README.rst              pyconfig.h.in
Makefile.pre.in         Tools/                  setup.py
Misc/                   aclocal.m4
Modules/                config.guess

Some of the listed subdirectories are of particular importance to us in the course of this series:

Grammar/ contains the grammar files we discussed last time.
Include/ contains header files. They are used both by CPython and by the users of the Python/C API.
Lib/ contains standard library modules written in Python. While some modules, such as argparse and wave, are written in Python entirely, many wrap C code. For example, the Python io module wraps the C _io module.
Modules/ contains standard library modules written in C. While some modules, such as itertools, are intended to be imported directly, others are wrapped by Python modules.
Objects/ contains the implementations of built-in types. If you want to understand how int or list are implemented, this is the ultimate place to go to.
Parser/ contains the old parser, the old parser generator, the new parser and the tokenizer.
Programs/ contains source files that are compiled to executables.
Python/ contains source files for the interpreter itself. This includes the compiler, the evaluation loop, the builtins module and many other interesting things.
Tools/ contains tools useful for building and managing CPython. For example, the new parser generator lives here.

If you don't see a directory for tests, and your heart starts beating faster, relax. It's Lib/test/. Tests can be useful not only for CPython development but also for getting an understanding of how CPython works. For example, to understand what optimizations the peephole optimizer is expected to do, you can look at the tests in Lib/test/test_peepholer.py. And to understand what some piece of code of the peephole optimizer does, you can delete that piece of code, recompile CPython, run

$ ./python.exe -m test test_peepholer

and see which tests fail.

Ideally, all we need to do to compile CPython is to run ./configure and make:

$ ./configure

$ make -j -s

make will produce an executable named python, but don't be surprised to see python.exe on macOS. The .exe extension is used to distinguish the executable from the Python/ directory on the case-insensitive filesystem. Check out the Python Developer's Guide for more information on compiling.

At this point, we can proudly say that we've built our own copy of CPython:

$ ./python.exe
Python 3.9.0+ (heads/3.9-dirty:20bdeedfb4, Oct 10 2020, 16:55:24)
[Clang 10.0.0 (clang-1000.10.44.4)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> 2 ** 16
65536

Let's see what happens when we run it.

main()

The execution of CPython, like execution of any other C program, starts with the main() function in Programs/python.c:

/* Minimal main program -- everything is loaded from the library */

#include "Python.h"

#ifdef MS_WINDOWS
int
wmain(int argc, wchar_t **argv)
{
    return Py_Main(argc, argv);
}
#else
int
main(int argc, char **argv)
{
    return Py_BytesMain(argc, argv);
}
#endif

There isn't much going on there. The only thing worth mentioning is that on Windows CPython uses wmain() instead of main() as an entrypoint to receive argv as UTF-16 encoded strings. The affect of this is that on other platforms CPython performs an extra step of converting a char string to a wchar_t string. The encoding of a char string depends on the locale settings, and the encoding of a wchar_t string depends on the size of wchar_t. For example, if sizeof(wchar_t) == 4, the UCS-4 encoding is used. PEP 383 has more to say on this.

We find Py_Main() and Py_BytesMain() in Modules/main.c. What they essentially do is call pymain_main() with slightly different arguments:

int
Py_Main(int argc, wchar_t **argv)
{
    _PyArgv args = {
        .argc = argc,
        .use_bytes_argv = 0,
        .bytes_argv = NULL,
        .wchar_argv = argv};
    return pymain_main(&args);
}


int
Py_BytesMain(int argc, char **argv)
{
    _PyArgv args = {
        .argc = argc,
        .use_bytes_argv = 1,
        .bytes_argv = argv,
        .wchar_argv = NULL};
    return pymain_main(&args);
}

pymain_main() doesn't seem to do much either:

static int
pymain_main(_PyArgv *args)
{
    PyStatus status = pymain_init(args);
    if (_PyStatus_IS_EXIT(status)) {
        pymain_free();
        return status.exitcode;
    }
    if (_PyStatus_EXCEPTION(status)) {
        pymain_exit_error(status);
    }

    return Py_RunMain();
}

Nevertheless, we should stop on it for a little bit longer. Last time we learned that before a Python program starts executing, CPython does a lot of things to compile it. It turns out that CPython does a lot of things even before it starts compiling a program. These things constitute the initialization of CPython. We mentioned the initialization in part 1 when we said that CPython works in three stages:

initialization
compilation; and
interpretation.

What pymain_main() does is call pymain_init() to perform the initialization and then call Py_RunMain() to proceed with the next stages.

The initialization stage

What does CPython do during the initialization? Let's ponder on this for a moment. At the very least it has to:

find a common language with the OS to handle properly the encodings of arguments, environment variables, standard streams and the file system
parse the command line arguments and read the environment variables to determine the options to run with
initialize the runtime state, the main interpreter state and the main thread state
initialize built-in types and the builtins module
initialize the sys module
set up the import system
create the __main__ module.

Starting with CPython 3.8, all of that done in three distinct phases:

preinitialization
core initialization; and
main initialization.

The phases gradually introduce new capabilities. The preinitialization phase initializes the runtime state, sets up the default memory allocator and performs very basic configuration. There is no sign of Python yet. The core initialization phase initializes the main interpreter state and the main thread state, built-in types and exceptions, the builtins module, the sys module and the import system. At this point, you can use the "core" of Python. However, some things are not available yet. For example, the sys module is only partially initialized, and only the import of built-in and frozen modules is supported. After the main initialization phase, CPython is fully initialized and ready to compile and execute a Python program.

What's the benefit of having distinct initialization phases? In a nutshell, it allows us to tune CPython more easily. For example, one may set a custom memory allocator in the preinitialized state or override the path configuration in the core_initialized state. Such capabilities are important to the users of the Python/C API who extend and embed Python. PEP 432 and PEP 587 explain in greater detail why having multi-phase initialization is a good idea.

Let's go back to the source code. The pymain_init() function mostly deals with the preinitialization and calls Py_InitializeFromConfig() in the end to perform the core and the main phases of the initialization:

static PyStatus
pymain_init(const _PyArgv *args)
{
    PyStatus status;

    // Initialize the runtime state
    status = _PyRuntime_Initialize();
    if (_PyStatus_EXCEPTION(status)) {
        return status;
    }

    // Initialize default preconfig
    PyPreConfig preconfig;
    PyPreConfig_InitPythonConfig(&preconfig);

    // Perfrom preinitialization
    status = _Py_PreInitializeFromPyArgv(&preconfig, args);
    if (_PyStatus_EXCEPTION(status)) {
        return status;
    }
    // Preinitialized. Prepare config for the next initialization phases

    // Initialize default config
    PyConfig config;
    PyConfig_InitPythonConfig(&config);

    // Store the command line arguments in `config->argv`
    if (args->use_bytes_argv) {
        status = PyConfig_SetBytesArgv(&config, args->argc, args->bytes_argv);
    }
    else {
        status = PyConfig_SetArgv(&config, args->argc, args->wchar_argv);
    }
    if (_PyStatus_EXCEPTION(status)) {
        goto done;
    }

    // Perform core and main initialization
    status = Py_InitializeFromConfig(&config);
        if (_PyStatus_EXCEPTION(status)) {
        goto done;
    }
    status = _PyStatus_OK();

done:
    PyConfig_Clear(&config);
    return status;
}

_PyRuntime_Initialize() initializes the runtime state. The runtime state is stored in the global variable called _PyRuntime of type _PyRuntimeState that is defined as follows:

/* Full Python runtime state */

typedef struct pyruntimestate {
    /* Is running Py_PreInitialize()? */
    int preinitializing;

    /* Is Python preinitialized? Set to 1 by Py_PreInitialize() */
    int preinitialized;

    /* Is Python core initialized? Set to 1 by _Py_InitializeCore() */
    int core_initialized;

    /* Is Python fully initialized? Set to 1 by Py_Initialize() */
    int initialized;

    /* Set by Py_FinalizeEx(). Only reset to NULL if Py_Initialize() is called again. */
    _Py_atomic_address _finalizing;

    struct pyinterpreters {
        PyThread_type_lock mutex;
        PyInterpreterState *head;
        PyInterpreterState *main;
        int64_t next_id;
    } interpreters;

    unsigned long main_thread;

    struct _ceval_runtime_state ceval;
    struct _gilstate_runtime_state gilstate;

    PyPreConfig preconfig;

    // ... less interesting stuff for now
} _PyRuntimeState;

The last field preconfig of _PyRuntimeState holds the configuration that is used to preinitialize CPython. It's also used by the next phase to complete the configuration. Here's the extensively commented definition of PyPreConfig:

typedef struct {
    int _config_init;     /* _PyConfigInitEnum value */

    /* Parse Py_PreInitializeFromBytesArgs() arguments?
       See PyConfig.parse_argv */
    int parse_argv;

    /* If greater than 0, enable isolated mode: sys.path contains
       neither the script's directory nor the user's site-packages directory.

       Set to 1 by the -I command line option. If set to -1 (default), inherit
       Py_IsolatedFlag value. */
    int isolated;

    /* If greater than 0: use environment variables.
       Set to 0 by -E command line option. If set to -1 (default), it is
       set to !Py_IgnoreEnvironmentFlag. */
    int use_environment;

    /* Set the LC_CTYPE locale to the user preferred locale? If equals to 0,
       set coerce_c_locale and coerce_c_locale_warn to 0. */
    int configure_locale;

    /* Coerce the LC_CTYPE locale if it's equal to "C"? (PEP 538)

       Set to 0 by PYTHONCOERCECLOCALE=0. Set to 1 by PYTHONCOERCECLOCALE=1.
       Set to 2 if the user preferred LC_CTYPE locale is "C".

       If it is equal to 1, LC_CTYPE locale is read to decide if it should be
       coerced or not (ex: PYTHONCOERCECLOCALE=1). Internally, it is set to 2
       if the LC_CTYPE locale must be coerced.

       Disable by default (set to 0). Set it to -1 to let Python decide if it
       should be enabled or not. */
    int coerce_c_locale;

    /* Emit a warning if the LC_CTYPE locale is coerced?

       Set to 1 by PYTHONCOERCECLOCALE=warn.

       Disable by default (set to 0). Set it to -1 to let Python decide if it
       should be enabled or not. */
    int coerce_c_locale_warn;

#ifdef MS_WINDOWS
    /* If greater than 1, use the "mbcs" encoding instead of the UTF-8
       encoding for the filesystem encoding.

       Set to 1 if the PYTHONLEGACYWINDOWSFSENCODING environment variable is
       set to a non-empty string. If set to -1 (default), inherit
       Py_LegacyWindowsFSEncodingFlag value.

       See PEP 529 for more details. */
    int legacy_windows_fs_encoding;
#endif

    /* Enable UTF-8 mode? (PEP 540)

       Disabled by default (equals to 0).

       Set to 1 by "-X utf8" and "-X utf8=1" command line options.
       Set to 1 by PYTHONUTF8=1 environment variable.

       Set to 0 by "-X utf8=0" and PYTHONUTF8=0.

       If equals to -1, it is set to 1 if the LC_CTYPE locale is "C" or
       "POSIX", otherwise it is set to 0. Inherit Py_UTF8Mode value value. */
    int utf8_mode;

    /* If non-zero, enable the Python Development Mode.

       Set to 1 by the -X dev command line option. Set by the PYTHONDEVMODE
       environment variable. */
    int dev_mode;

    /* Memory allocator: PYTHONMALLOC env var.
       See PyMemAllocatorName for valid values. */
    int allocator;
} PyPreConfig;

After the call to _PyRuntime_Initialize(), the _PyRuntime global variable is initialized to the defaults. Next, PyPreConfig_InitPythonConfig() initializes new default preconfig, and then _Py_PreInitializeFromPyArgv() performs the actual preinitialization. What's the reason to initialize another preconfig if there is already one in _PyRuntime? Remember that many functions that CPython calls are also exposed via the Python/C API. So CPython just uses this API in the way it's designed to be used. Another consequence of this is that when you step through the CPython source code as we do today, you often encounter functions that seem to do more than you expect. For example, _PyRuntime_Initialize() is called several times during the initialization process. Of course, it does nothing on the subsequent calls.

_Py_PreInitializeFromPyArgv() reads command line arguments, environment variables and global configuration variables and, based on that, sets _PyRuntime.preconfig, the current locale and the memory allocator. It reads only those parameters that are relevant to the preinitialization phase. For example, it parses only the -E -I -X arguments.

At this point, the runtime is preinitialized, and pymain_init() starts to prepare config for the next initialization phase. Do not confuse config with preconfig. The former is a structure that holds most of the Python configuration. It's heavily used during the initialization stage and during the execution of a Python program as well. To get an idea of how config is used, I recommend you look over its lengthy definition:

/* --- PyConfig ---------------------------------------------- */

typedef struct {
    int _config_init;     /* _PyConfigInitEnum value */

    int isolated;         /* Isolated mode? see PyPreConfig.isolated */
    int use_environment;  /* Use environment variables? see PyPreConfig.use_environment */
    int dev_mode;         /* Python Development Mode? See PyPreConfig.dev_mode */

    /* Install signal handlers? Yes by default. */
    int install_signal_handlers;

    int use_hash_seed;      /* PYTHONHASHSEED=x */
    unsigned long hash_seed;

    /* Enable faulthandler?
       Set to 1 by -X faulthandler and PYTHONFAULTHANDLER. -1 means unset. */
    int faulthandler;

    /* Enable PEG parser?
       1 by default, set to 0 by -X oldparser and PYTHONOLDPARSER */
    int _use_peg_parser;

    /* Enable tracemalloc?
       Set by -X tracemalloc=N and PYTHONTRACEMALLOC. -1 means unset */
    int tracemalloc;

    int import_time;        /* PYTHONPROFILEIMPORTTIME, -X importtime */
    int show_ref_count;     /* -X showrefcount */
    int dump_refs;          /* PYTHONDUMPREFS */
    int malloc_stats;       /* PYTHONMALLOCSTATS */

    /* Python filesystem encoding and error handler:
       sys.getfilesystemencoding() and sys.getfilesystemencodeerrors().

       Default encoding and error handler:

       * if Py_SetStandardStreamEncoding() has been called: they have the
         highest priority;
       * PYTHONIOENCODING environment variable;
       * The UTF-8 Mode uses UTF-8/surrogateescape;
       * If Python forces the usage of the ASCII encoding (ex: C locale
         or POSIX locale on FreeBSD or HP-UX), use ASCII/surrogateescape;
       * locale encoding: ANSI code page on Windows, UTF-8 on Android and
         VxWorks, LC_CTYPE locale encoding on other platforms;
       * On Windows, "surrogateescape" error handler;
       * "surrogateescape" error handler if the LC_CTYPE locale is "C" or "POSIX";
       * "surrogateescape" error handler if the LC_CTYPE locale has been coerced
         (PEP 538);
       * "strict" error handler.

       Supported error handlers: "strict", "surrogateescape" and
       "surrogatepass". The surrogatepass error handler is only supported
       if Py_DecodeLocale() and Py_EncodeLocale() use directly the UTF-8 codec;
       it's only used on Windows.

       initfsencoding() updates the encoding to the Python codec name.
       For example, "ANSI_X3.4-1968" is replaced with "ascii".

       On Windows, sys._enablelegacywindowsfsencoding() sets the
       encoding/errors to mbcs/replace at runtime.


       See Py_FileSystemDefaultEncoding and Py_FileSystemDefaultEncodeErrors.
       */
    wchar_t *filesystem_encoding;
    wchar_t *filesystem_errors;

    wchar_t *pycache_prefix;  /* PYTHONPYCACHEPREFIX, -X pycache_prefix=PATH */
    int parse_argv;           /* Parse argv command line arguments? */

    /* Command line arguments (sys.argv).

       Set parse_argv to 1 to parse argv as Python command line arguments
       and then strip Python arguments from argv.

       If argv is empty, an empty string is added to ensure that sys.argv
       always exists and is never empty. */
    PyWideStringList argv;

    /* Program name:

       - If Py_SetProgramName() was called, use its value.
       - On macOS, use PYTHONEXECUTABLE environment variable if set.
       - If WITH_NEXT_FRAMEWORK macro is defined, use __PYVENV_LAUNCHER__
         environment variable is set.
       - Use argv[0] if available and non-empty.
       - Use "python" on Windows, or "python3 on other platforms. */
    wchar_t *program_name;

    PyWideStringList xoptions;     /* Command line -X options */

    /* Warnings options: lowest to highest priority. warnings.filters
       is built in the reverse order (highest to lowest priority). */
    PyWideStringList warnoptions;

    /* If equal to zero, disable the import of the module site and the
       site-dependent manipulations of sys.path that it entails. Also disable
       these manipulations if site is explicitly imported later (call
       site.main() if you want them to be triggered).

       Set to 0 by the -S command line option. If set to -1 (default), it is
       set to !Py_NoSiteFlag. */
    int site_import;

    /* Bytes warnings:

       * If equal to 1, issue a warning when comparing bytes or bytearray with
         str or bytes with int.
       * If equal or greater to 2, issue an error.

       Incremented by the -b command line option. If set to -1 (default), inherit
       Py_BytesWarningFlag value. */
    int bytes_warning;

    /* If greater than 0, enable inspect: when a script is passed as first
       argument or the -c option is used, enter interactive mode after
       executing the script or the command, even when sys.stdin does not appear
       to be a terminal.

       Incremented by the -i command line option. Set to 1 if the PYTHONINSPECT
       environment variable is non-empty. If set to -1 (default), inherit
       Py_InspectFlag value. */
    int inspect;

    /* If greater than 0: enable the interactive mode (REPL).

       Incremented by the -i command line option. If set to -1 (default),
       inherit Py_InteractiveFlag value. */
    int interactive;

    /* Optimization level.

       Incremented by the -O command line option. Set by the PYTHONOPTIMIZE
       environment variable. If set to -1 (default), inherit Py_OptimizeFlag
       value. */
    int optimization_level;

    /* If greater than 0, enable the debug mode: turn on parser debugging
       output (for expert only, depending on compilation options).

       Incremented by the -d command line option. Set by the PYTHONDEBUG
       environment variable. If set to -1 (default), inherit Py_DebugFlag
       value. */
    int parser_debug;

    /* If equal to 0, Python won't try to write ``.pyc`` files on the
       import of source modules.

       Set to 0 by the -B command line option and the PYTHONDONTWRITEBYTECODE
       environment variable. If set to -1 (default), it is set to
       !Py_DontWriteBytecodeFlag. */
    int write_bytecode;

    /* If greater than 0, enable the verbose mode: print a message each time a
       module is initialized, showing the place (filename or built-in module)
       from which it is loaded.

       If greater or equal to 2, print a message for each file that is checked
       for when searching for a module. Also provides information on module
       cleanup at exit.

       Incremented by the -v option. Set by the PYTHONVERBOSE environment
       variable. If set to -1 (default), inherit Py_VerboseFlag value. */
    int verbose;

    /* If greater than 0, enable the quiet mode: Don't display the copyright
       and version messages even in interactive mode.

       Incremented by the -q option. If set to -1 (default), inherit
       Py_QuietFlag value. */
    int quiet;

   /* If greater than 0, don't add the user site-packages directory to
      sys.path.

      Set to 0 by the -s and -I command line options , and the PYTHONNOUSERSITE
      environment variable. If set to -1 (default), it is set to
      !Py_NoUserSiteDirectory. */
    int user_site_directory;

    /* If non-zero, configure C standard steams (stdio, stdout,
       stderr):

       - Set O_BINARY mode on Windows.
       - If buffered_stdio is equal to zero, make streams unbuffered.
         Otherwise, enable streams buffering if interactive is non-zero. */
    int configure_c_stdio;

    /* If equal to 0, enable unbuffered mode: force the stdout and stderr
       streams to be unbuffered.

       Set to 0 by the -u option. Set by the PYTHONUNBUFFERED environment
       variable.
       If set to -1 (default), it is set to !Py_UnbufferedStdioFlag. */
    int buffered_stdio;

    /* Encoding of sys.stdin, sys.stdout and sys.stderr.
       Value set from PYTHONIOENCODING environment variable and
       Py_SetStandardStreamEncoding() function.
       See also 'stdio_errors' attribute. */
    wchar_t *stdio_encoding;

    /* Error handler of sys.stdin and sys.stdout.
       Value set from PYTHONIOENCODING environment variable and
       Py_SetStandardStreamEncoding() function.
       See also 'stdio_encoding' attribute. */
    wchar_t *stdio_errors;

#ifdef MS_WINDOWS
    /* If greater than zero, use io.FileIO instead of WindowsConsoleIO for sys
       standard streams.

       Set to 1 if the PYTHONLEGACYWINDOWSSTDIO environment variable is set to
       a non-empty string. If set to -1 (default), inherit
       Py_LegacyWindowsStdioFlag value.

       See PEP 528 for more details. */
    int legacy_windows_stdio;
#endif

    /* Value of the --check-hash-based-pycs command line option:

       - "default" means the 'check_source' flag in hash-based pycs
         determines invalidation
       - "always" causes the interpreter to hash the source file for
         invalidation regardless of value of 'check_source' bit
       - "never" causes the interpreter to always assume hash-based pycs are
         valid

       The default value is "default".

       See PEP 552 "Deterministic pycs" for more details. */
    wchar_t *check_hash_pycs_mode;

    /* --- Path configuration inputs ------------ */

    /* If greater than 0, suppress _PyPathConfig_Calculate() warnings on Unix.
       The parameter has no effect on Windows.

       If set to -1 (default), inherit !Py_FrozenFlag value. */
    int pathconfig_warnings;

    wchar_t *pythonpath_env; /* PYTHONPATH environment variable */
    wchar_t *home;          /* PYTHONHOME environment variable,
                               see also Py_SetPythonHome(). */

    /* --- Path configuration outputs ----------- */

    int module_search_paths_set;  /* If non-zero, use module_search_paths */
    PyWideStringList module_search_paths;  /* sys.path paths. Computed if
                                       module_search_paths_set is equal
                                       to zero. */

    wchar_t *executable;        /* sys.executable */
    wchar_t *base_executable;   /* sys._base_executable */
    wchar_t *prefix;            /* sys.prefix */
    wchar_t *base_prefix;       /* sys.base_prefix */
    wchar_t *exec_prefix;       /* sys.exec_prefix */
    wchar_t *base_exec_prefix;  /* sys.base_exec_prefix */
    wchar_t *platlibdir;        /* sys.platlibdir */

    /* --- Parameter only used by Py_Main() ---------- */

    /* Skip the first line of the source ('run_filename' parameter), allowing use of non-Unix forms of
       "#!cmd".  This is intended for a DOS specific hack only.

       Set by the -x command line option. */
    int skip_source_first_line;

    wchar_t *run_command;   /* -c command line argument */
    wchar_t *run_module;    /* -m command line argument */
    wchar_t *run_filename;  /* Trailing command line argument without -c or -m */

    /* --- Private fields ---------------------------- */

    /* Install importlib? If set to 0, importlib is not initialized at all.
       Needed by freeze_importlib. */
    int _install_importlib;

    /* If equal to 0, stop Python initialization before the "main" phase */
    int _init_main;

    /* If non-zero, disallow threads, subprocesses, and fork.
       Default: 0. */
    int _isolated_interpreter;

    /* Original command line arguments. If _orig_argv is empty and _argv is
       not equal to [''], PyConfig_Read() copies the configuration 'argv' list
       into '_orig_argv' list before modifying 'argv' list (if parse_argv
       is non-zero).

       _PyConfig_Write() initializes Py_GetArgcArgv() to this list. */
    PyWideStringList _orig_argv;
} PyConfig;

The same way as pymain_init() called PyPreConfig_InitPythonConfig() to create default preconfig, it now calls PyConfig_InitPythonConfig() to create default config. It then calls PyConfig_SetBytesArgv() to store command line arguments in config.argv and Py_InitializeFromConfig() to perform the core and main initialization phases. We move further from pymain_init() to Py_InitializeFromConfig():

PyStatus
Py_InitializeFromConfig(const PyConfig *config)
{
    if (config == NULL) {
        return _PyStatus_ERR("initialization config is NULL");
    }

    PyStatus status;

    // Yeah, call once again
    status = _PyRuntime_Initialize();
    if (_PyStatus_EXCEPTION(status)) {
        return status;
    }
    _PyRuntimeState *runtime = &_PyRuntime;

    PyThreadState *tstate = NULL;
    // The core initialization phase
    status = pyinit_core(runtime, config, &tstate);
    if (_PyStatus_EXCEPTION(status)) {
        return status;
    }
    config = _PyInterpreterState_GetConfig(tstate->interp);

    if (config->_init_main) {
        // The main initialization phase
        status = pyinit_main(tstate);
        if (_PyStatus_EXCEPTION(status)) {
            return status;
        }
    }

    return _PyStatus_OK();
}

We can clearly see the separation between the initialization phases. The core phase is done by pyinit_core(), and the main phase is done by pyinit_main(). The pyinit_core() function initializes the "core" of Python. More specifically,

It prepares the configuration: parses command line arguments, reads environment variables, calculates the path configuration, chooses the encodings for the standard streams and the file system and writes all of this to config.
It applies the configuration: configures the standard streams, generates the secret key for hashing, creates the main interpreter state and the main thread state, initializes the GIL and takes it, enables the GC, initializes built-in types and exceptions, initializes the sys module and the builtins module and sets up the import system for built-in and frozen modules.

During the first step, CPython calculates config.module_search_paths, which will be later copied to sys.path. Otherwise, this step is not very interesting, so let's look at pyinit_config() that pyinit_core() calls to perform the second step:

static PyStatus
pyinit_config(_PyRuntimeState *runtime,
              PyThreadState **tstate_p,
              const PyConfig *config)
{
    // Set Py_* global variables from config.
    // Initialize C standard streams (stdin, stdout, stderr).
    // Set secret key for hashing.
    PyStatus status = pycore_init_runtime(runtime, config);
    if (_PyStatus_EXCEPTION(status)) {
        return status;
    }

    PyThreadState *tstate;
    // Create the main interpreter state and the main thread state.
    // Take the GIL.
    status = pycore_create_interpreter(runtime, config, &tstate);
    if (_PyStatus_EXCEPTION(status)) {
        return status;
    }
    *tstate_p = tstate;

    // Init types, exception, sys, builtins, importlib, etc.
    status = pycore_interp_init(tstate);
    if (_PyStatus_EXCEPTION(status)) {
        return status;
    }

    /* Only when we get here is the runtime core fully initialized */
    runtime->core_initialized = 1;
    return _PyStatus_OK();
}

First, pycore_init_runtime() copies some of the config fields to the corresponding global configuration variables. These global variables were used to configure CPython before PyConfig was introduced and continue to be a part of the Python/C API.

Next, pycore_init_runtime() sets the buffering modes for the stdio, stdout and stderr file pointers. On Unix-like systems this is done by calling the setvbuf() library function.

Finally, pycore_init_runtime() generates the secret key for hashing, which is stored in the _Py_HashSecret global variable. The secret key is taken along with the input by the SipHash24 hash function, which CPython uses to compute hashes. The secret key is randomly generated each time CPython starts. The purpose of randomization is to protect a Python application from hash collision DoS attacks. Python and many other languages including PHP, Ruby, JavaScript and C# were once vulnerable to such attacks. An attacker could send a set of strings with the same hash to an application and increase dramatically the CPU time required to put these strings in the dictionary because they all happen to be in the same bucket. The solution is to supply a hash function with the randomly generated key unknown to the attacker. To learn more about the attack, check this presentation. To learn more about the hash algorithm, check PEP 456. If you need to generate the key deterministically in your program, set the PYTHONHASHSEED environment variable to some fixed value.

In part 1 we learned that CPython uses a thread state to store thread-specific data, such as a call stack and an exception state, and an interpreter state to store interpreter-specific data, such as loaded modules and import settings. The pycore_create_interpreter() function creates an interpreter state and a thread state for the main OS thread. We haven't seen what these structures look like yet, so here's the definition of the interpreter state struct:

// The PyInterpreterState typedef is in Include/pystate.h.
struct _is {

    // _PyRuntime.interpreters.head stores the most recently created interpreter
    // `next` allows us to access all the interpreters.
    struct _is *next;
    // `tstate_head` points to the most recently created thread state.
    // Thread states of the same interpreter are linked together.
    struct _ts *tstate_head;

    /* Reference to the _PyRuntime global variable. This field exists
       to not have to pass runtime in addition to tstate to a function.
       Get runtime from tstate: tstate->interp->runtime. */
    struct pyruntimestate *runtime;

    int64_t id;
    // For tracking references to the interpreter
    int64_t id_refcount;
    int requires_idref;
    PyThread_type_lock id_mutex;

    int finalizing;

    struct _ceval_state ceval;
    struct _gc_runtime_state gc;

    PyObject *modules;  // sys.modules points to it
    PyObject *modules_by_index;
    PyObject *sysdict;  // points to sys.__dict__
    PyObject *builtins; // points to builtins.__dict__
    PyObject *importlib;

    // A list of codec search functions
    PyObject *codec_search_path;
    PyObject *codec_search_cache;
    PyObject *codec_error_registry;
    int codecs_initialized;

    struct _Py_unicode_state unicode;

    PyConfig config;

    PyObject *dict;  /* Stores per-interpreter state */

    PyObject *builtins_copy;
    PyObject *import_func;
    /* Initialized to PyEval_EvalFrameDefault(). */
    _PyFrameEvalFunction eval_frame;

    // See `atexit` module
    void (*pyexitfunc)(PyObject *);
    PyObject *pyexitmodule;

    uint64_t tstate_next_unique_id;

    // See `warnings` module
    struct _warnings_runtime_state warnings;

    // A list of audit hooks, see sys.addaudithook
    PyObject *audit_hooks;

#if _PY_NSMALLNEGINTS + _PY_NSMALLPOSINTS > 0
    // Small integers are preallocated in this array so that they can be shared.
    // The default range is [-5, 256].
    PyLongObject* small_ints[_PY_NSMALLNEGINTS + _PY_NSMALLPOSINTS];
#endif

  // ... less interesting stuff for now
};

The important thing to note here is that config belongs to the interpreter state. The configuration that was read before is stored in config of the newly created interpreter state. The thread state struct is defined as follows:

// The PyThreadState typedef is in Include/pystate.h.
struct _ts {

    // Double-linked list is used to access all thread states belonging to the same interpreter
    struct _ts *prev;
    struct _ts *next;
    PyInterpreterState *interp;

    // Reference to the current frame (it can be NULL).
    // The call stack is accesible via frame->f_back.
    PyFrameObject *frame;

    // ... checking if recursion level is too deep

    // ... tracing/profiling

    /* The exception currently being raised */
    PyObject *curexc_type;
    PyObject *curexc_value;
    PyObject *curexc_traceback;

    /* The exception currently being handled, if no coroutines/generators
     * are present. Always last element on the stack referred to be exc_info.
     */
    _PyErr_StackItem exc_state;

    /* Pointer to the top of the stack of the exceptions currently
     * being handled */
    _PyErr_StackItem *exc_info;

    PyObject *dict;  /* Stores per-thread state */

    int gilstate_counter;

    PyObject *async_exc; /* Asynchronous exception to raise */
    unsigned long thread_id; /* Thread id where this tstate was created */

    /* Unique thread state id. */
    uint64_t id;

    // ... less interesting stuff for now
};

Every thread needs to have an access to its thread state. When you spawn a new thread using the threading module, the thread starts executing a given target in the evaluation loop. It can access its thread state because the thread state is passed as an argument to the evaluation function.

After creating a thread state for the main OS thread, pycore_create_interpreter() initializes the GIL that prevents multiple threads from working with Python objects at the same time. Threads wait for the GIL and take the GIL at the start of the evaluation loop.

If you write a C extension and create new threads from C, you need to manually take the GIL in order to work with Python objects. When you take the GIL, CPython associates the current thread with the corresponding thread state by storing a thread state in the thread specific storage (the pthread_setspecific() library function on Unix-like systems). That's the mechanism that allows any thread to access its thread state.

After pycore_create_interpreter() creates the main interpreter state and the main thread state, pyinit_config() calls pycore_interp_init() to finish the core initialization phase. The code of pycore_interp_init() is self-explanatory:

static PyStatus
pycore_interp_init(PyThreadState *tstate)
{
    PyStatus status;
    PyObject *sysmod = NULL;

    status = pycore_init_types(tstate);
    if (_PyStatus_EXCEPTION(status)) {
        goto done;
    }

    status = _PySys_Create(tstate, &sysmod);
    if (_PyStatus_EXCEPTION(status)) {
        goto done;
    }

    status = pycore_init_builtins(tstate);
    if (_PyStatus_EXCEPTION(status)) {
        goto done;
    }

    status = pycore_init_import_warnings(tstate, sysmod);

done:
    // Py_XDECREF() decreases the reference count of an object.
    // If the reference count becomes 0, the object is deallocated.
    Py_XDECREF(sysmod);
    return status;
}

The pycore_init_types() function initializes built-in types. But what does it mean? And what are types really? As you probably know, everything you work with in Python is an object. Numbers, strings, lists, functions, modules, frame objects, user-defined classes and built-in types are all Python objects. A Python object is an instance of the PyObject struct or an instance of any other C struct that "inherits" (we'll what it means in a moment) from PyObject. The PyObject struct has two fields:

typedef struct _object {
    _PyObject_HEAD_EXTRA // for debugging only
    Py_ssize_t ob_refcnt;
    PyTypeObject *ob_type;
} PyObject;

The ob_refcnt field stores a reference count, and the ob_type field points to the type of the object.

Here's an example of a simple Python object, float:

typedef struct {
    PyObject ob_base; // expansion of the PyObject_HEAD macro
    double ob_fval;
} PyFloatObject;

Note how PyFloatObject "inherits" from PyObject. I say "inherits" because the C standard states that a pointer to any struct can be converted to a pointer to its first member and vice versa. This feature allows CPython to have functions that take any Python object as an argument by taking PyObject, thus achieving polymorphism.

The reason why CPython can do something useful with PyObject is because the behavior of a Python object is determined by its type, and PyObject always has a type. A type "knows" how to create the objects of that type, how to calculate their hashes, how to add them, how to call them, how to access their attributes, how to deallocate them and much more. Types are also Python objects represented by the PyTypeObject structure. All types have the same type, which is PyType_Type. And the type of PyType_Type points to PyType_Type itself. If this explanation seems complicated, this example should not:

$ ./python.exe -q
>>> type([])
<class 'list'>
>>> type(type([]))
<class 'type'>
>>> type(type(type([])))
<class 'type'>

The fields of PyTypeObject are very well documented in the Python/C API Reference Manual. I only leave here the definition of the struct underlying PyTypeObject to get an idea of the amount of information that a Python type stores:

// PyTypeObject is a typedef for struct _typeobject
struct _typeobject {
    PyObject_VAR_HEAD // expands to 
                      // PyObject ob_base;
                      // Py_ssize_t ob_size;
    const char *tp_name; /* For printing, in format "<module>.<name>" */
    Py_ssize_t tp_basicsize, tp_itemsize; /* For allocation */

    /* Methods to implement standard operations */

    destructor tp_dealloc;
    Py_ssize_t tp_vectorcall_offset;
    getattrfunc tp_getattr;
    setattrfunc tp_setattr;
    PyAsyncMethods *tp_as_async; /* formerly known as tp_compare (Python 2)
                                    or tp_reserved (Python 3) */
    reprfunc tp_repr;

    /* Method suites for standard classes */
    PyNumberMethods *tp_as_number;
    PySequenceMethods *tp_as_sequence;
    PyMappingMethods *tp_as_mapping;

    /* More standard operations (here for binary compatibility) */

    hashfunc tp_hash;
    ternaryfunc tp_call;
    reprfunc tp_str;
    getattrofunc tp_getattro;
    setattrofunc tp_setattro;

    /* Functions to access object as input/output buffer */
    PyBufferProcs *tp_as_buffer;

    /* Flags to define presence of optional/expanded features */
    unsigned long tp_flags;

    const char *tp_doc; /* Documentation string */

    /* Assigned meaning in release 2.0 */
    /* call function for all accessible objects */
    traverseproc tp_traverse;

    /* delete references to contained objects */
    inquiry tp_clear;

    /* Assigned meaning in release 2.1 */
    /* rich comparisons */
    richcmpfunc tp_richcompare;

    /* weak reference enabler */
    Py_ssize_t tp_weaklistoffset;

    /* Iterators */
    getiterfunc tp_iter;
    iternextfunc tp_iternext;

    /* Attribute descriptor and subclassing stuff */
    struct PyMethodDef *tp_methods;
    struct PyMemberDef *tp_members;
    struct PyGetSetDef *tp_getset;
    struct _typeobject *tp_base;
    PyObject *tp_dict;
    descrgetfunc tp_descr_get;
    descrsetfunc tp_descr_set;
    Py_ssize_t tp_dictoffset;
    initproc tp_init;
    allocfunc tp_alloc;
    newfunc tp_new;
    freefunc tp_free; /* Low-level free-memory routine */
    inquiry tp_is_gc; /* For PyObject_IS_GC */
    PyObject *tp_bases;
    PyObject *tp_mro; /* method resolution order */
    PyObject *tp_cache;
    PyObject *tp_subclasses;
    PyObject *tp_weaklist;
    destructor tp_del;

    /* Type attribute cache version tag. Added in version 2.6 */
    unsigned int tp_version_tag;

    destructor tp_finalize;
    vectorcallfunc tp_vectorcall;
};

Built-in types, such as int and list, are implemented by statically defining instances of PyTypeObject, like so:

PyTypeObject PyList_Type = {
    PyVarObject_HEAD_INIT(&PyType_Type, 0)
    "list",
    sizeof(PyListObject),
    0,
    (destructor)list_dealloc,                   /* tp_dealloc */
    0,                                          /* tp_vectorcall_offset */
    0,                                          /* tp_getattr */
    0,                                          /* tp_setattr */
    0,                                          /* tp_as_async */
    (reprfunc)list_repr,                        /* tp_repr */
    0,                                          /* tp_as_number */
    &list_as_sequence,                          /* tp_as_sequence */
    &list_as_mapping,                           /* tp_as_mapping */
    PyObject_HashNotImplemented,                /* tp_hash */
    0,                                          /* tp_call */
    0,                                          /* tp_str */
    PyObject_GenericGetAttr,                    /* tp_getattro */
    0,                                          /* tp_setattro */
    0,                                          /* tp_as_buffer */
    Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC |
        Py_TPFLAGS_BASETYPE | Py_TPFLAGS_LIST_SUBCLASS, /* tp_flags */
    list___init____doc__,                       /* tp_doc */
    (traverseproc)list_traverse,                /* tp_traverse */
    (inquiry)_list_clear,                       /* tp_clear */
    list_richcompare,                           /* tp_richcompare */
    0,                                          /* tp_weaklistoffset */
    list_iter,                                  /* tp_iter */
    0,                                          /* tp_iternext */
    list_methods,                               /* tp_methods */
    0,                                          /* tp_members */
    0,                                          /* tp_getset */
    0,                                          /* tp_base */
    0,                                          /* tp_dict */
    0,                                          /* tp_descr_get */
    0,                                          /* tp_descr_set */
    0,                                          /* tp_dictoffset */
    (initproc)list___init__,                    /* tp_init */
    PyType_GenericAlloc,                        /* tp_alloc */
    PyType_GenericNew,                          /* tp_new */
    PyObject_GC_Del,                            /* tp_free */
    .tp_vectorcall = list_vectorcall,
};

CPython also needs to initialize every built-in type. This is what we started our discussion of types with. All types require some initialization, for example, to add special methods, such as __call__() and __eq__(), to the type's dictionary and to point them to the corresponding tp_* functions. This common initialization is done by calling PyType_Ready() for each type:

PyStatus
_PyTypes_Init(void)
{
    // The names of the special methods "__hash__", "__call_", etc. are interned by this call
    PyStatus status = _PyTypes_InitSlotDefs();
    if (_PyStatus_EXCEPTION(status)) {
        return status;
    }

#define INIT_TYPE(TYPE, NAME) \
    do { \
        if (PyType_Ready(TYPE) < 0) { \
            return _PyStatus_ERR("Can't initialize " NAME " type"); \
        } \
    } while (0)

    INIT_TYPE(&PyBaseObject_Type, "object");
    INIT_TYPE(&PyType_Type, "type");
    INIT_TYPE(&_PyWeakref_RefType, "weakref");
    INIT_TYPE(&_PyWeakref_CallableProxyType, "callable weakref proxy");
    INIT_TYPE(&_PyWeakref_ProxyType, "weakref proxy");
    INIT_TYPE(&PyLong_Type, "int");
    INIT_TYPE(&PyBool_Type, "bool");
    INIT_TYPE(&PyByteArray_Type, "bytearray");
    INIT_TYPE(&PyBytes_Type, "str");
    INIT_TYPE(&PyList_Type, "list");
    INIT_TYPE(&_PyNone_Type, "None");
    INIT_TYPE(&_PyNotImplemented_Type, "NotImplemented");
    INIT_TYPE(&PyTraceBack_Type, "traceback");
    INIT_TYPE(&PySuper_Type, "super");
    INIT_TYPE(&PyRange_Type, "range");
    INIT_TYPE(&PyDict_Type, "dict");
    INIT_TYPE(&PyDictKeys_Type, "dict keys");
    // ... 50 more types
    return _PyStatus_OK();

#undef INIT_TYPE
}

Some built-in types require additional type-specific initialization. For example, int needs to preallocate small integers in the interp->small_ints array so that they can be reused, and float needs to determine how the current machine represents floating point numbers.

When built-in types are initialized, pycore_interp_init() calls _PySys_Create() to create the sys module. Why is the sys module the first module to be created? It's very important because it contains such things as the command line arguments passed to a program (sys.argv), the list of path entries to search for modules (sys.path), a lot of system-specific and implementation-specific data (sys.version, sys.implementation, sys.thread_info, etc.) and various functions for interacting with the interpreter (sys.addaudithook(), sys.settrace(), etc.). The main reason, though, to create the sys module so early is to initialize sys.modules. It points to the interp->modules dictionary, which is also created by _PySys_Create(), and acts as a cache for imported modules. It's the first place to look up for a module, and it's the place where all loaded modules go to. The import system heavily relies on sys.modules.

After the call to_PySys_Create(), the sys module is only partially initialized. The functions and most of the variables are available, but invocation-specific data, such as sys.argv and sys._xoptions, and the path-related configuration, such as sys.path and sys.exec_prefix, are set during the main initialization phase.

When the sys module is created, pycore_interp_init() calls pycore_init_builtins() to initialize the builtins module. Built-in functions, like abs(), dir() and print(), built-in types, like dict, int and str, built-in exceptions, like Exception and ValueError, and built-in constants, like False, Ellipsis and None, are all members of the builtins module. Built-in functions are a part of the module definition, but other members have to be placed in the module's dictionary explicitly. The pycore_init_builtins() function does that. Later on, frame->f_builtins will be set to this dictionary to look up names so we don't need to import builtins directly.

The last step of the core initialization phase is performed by the pycore_init_import_warnings() function. You probably know that Python has a mechanism to issue warnings, like so:

$ ./python.exe -q
>>> import imp
<stdin>:1: DeprecationWarning: the imp module is deprecated in favour of importlib; ...

Warnings can be ignored, turned into exceptions and displayed in various ways. CPython has filters to do that. Some filters are turned on by default, and the pycore_init_import_warnings() function is what turns them on. Most crucially, though, is that pycore_init_import_warnings() sets up the import system for built-in and frozen modules.

Built-in and frozen modules are two special kinds of modules. What unites them is that they are compiled directly into the python executable. The difference is that built-in modules are written in C, while frozen modules are written in Python. How is that possible to compile a module written in Python into the executable? This is cleverly done by incorporating marshalled code object of the module into the C source code.

An example of a frozen module is _frozen_importlib. It implements the core of the import system. To support the import of built-in and frozen modules, pycore_init_import_warnings() calls init_importlib(), and the very first thing init_importlib() does is import _frozen_importlib. It may seem that CPython has to import _frozen_importlib in order to import _frozen_importlib, but this is not the case. The _frozen_importlib module is a part of the universal API for importing any module. However, if CPython knows that it needs to import a frozen module, it can do so without reliance on _frozen_importlib.

The _frozen_importlib module depends on two other modules. First, it needs the sys module to get an access to sys.modules. Second, it needs the _imp module, which implements low-level import functions, including the functions for creating built-in and frozen modules. The problem is that _frozen_importlib cannot import any modules because the import statement depends on _frozen_importlib itself. The solution is to create the _imp module in init_importlib() and inject it and the sys module in _frozen_importlib by calling _frozen_importlib._install(sys, _imp). This bootstrapping of the import system ends the core initialization phase.

We leave pyinit_core() and enter pyinit_main() responsible for the main initialization phase. This function performs some checks and calls init_interp_main() to do the work that can be summarized as follows:

Get system's realtime and monotonic clocks, ensure that time.time(), time.monotonic() and time.perf_counter() will work correctly.
Finish the initialization of the sys module. This includes setting the path configuration variables, such as sys.path, sys.executable and sys.exec_prefix, and invocation-specific variables, such as sys.argv and sys._xoptions.
Add support for the import of path-based (external) modules. This is done by importing another frozen module called importlib._bootstrap_external. It enables the import of modules based on sys.path. Also, the zipimport frozen module is imported. It enables the import of modules from ZIP archives.
Normalize the names of the encodings for the file system and the standard streams. Set the error handlers for encoding and decoding when dealing with the file system.
Install default signal handlers. These are the handlers that get executed when a process receives a signal like SIGINT. The custom handlers can be set up using the signal module.
Import the io module and initialize sys.stdin, sys.stdout and sys.stderr. This is essentially done by calling io.open() on the file descriptors for the standard streams.
Set builtins.open to io.OpenWrapper so that open() is available as a built-in function.
Create the __main__ module, set __main__.__builtins__ to builtins and __main__.__loader__ to _frozen_importlib.BuiltinImporter.
Import warnings and site modules. The site module adds site-specific directories to sys.path. This is why sys.path normally contains a directory like /usr/local/lib/python3.9/site-packages/.
Set interp->runtime->initialized = 1

The initialization of CPython is completed. The pymain_init() function returns, and we step into Py_RunMain() to see what else CPython does before it enters the evaluation loop.

Running a Python program

The Py_RunMain() function doesn't seem like a place where the action happens:

int
Py_RunMain(void)
{
    int exitcode = 0;

    pymain_run_python(&exitcode);

    if (Py_FinalizeEx() < 0) {
        /* Value unlikely to be confused with a non-error exit status or
           other special meaning */
        exitcode = 120;
    }

    // Free the memory that is not freed by Py_FinalizeEx()
    pymain_free();

    if (_Py_UnhandledKeyboardInterrupt) {
        exitcode = exit_sigint();
    }

    return exitcode;
}

First, Py_RunMain() calls pymain_run_python() to run Python. Second, it calls Py_FinalizeEx() to undo the initialization. The Py_FinalizeEx() functions frees most of the memory that CPython is able to free, and the rest is freed by pymain_free(). Another important reason to finalize CPython is to call the exit functions, including the functions registered with the atexit module.

As you probably know, there are number of ways to run python, namely:

interactively:

$ ./cpython/python.exe
>>> import sys
>>> sys.path[:1]
['']

from stdin:

$ echo "import sys; print(sys.path[:1])" | ./cpython/python.exe
['']

as a command:

$ ./cpython/python.exe -c "import sys; print(sys.path[:1])"
['']

as a script

$ ./cpython/python.exe 03/print_path0.py
['/Users/Victor/Projects/tenthousandmeters/python_behind_the_scenes/03']

as a module:

$ ./cpython/python.exe -m 03.print_path0
['/Users/Victor/Projects/tenthousandmeters/python_behind_the_scenes']

and, less obvious, package as a script (print_path0_package is a directory with __main__.py):

$ ./cpython/python.exe 03/print_path0_package
['/Users/Victor/Projects/tenthousandmeters/python_behind_the_scenes/03/print_path0_package']

I moved one level up from the cpython/ directory to show that different modes of invocation lead to different values of sys.path[0]. What the next function on our way, pymain_run_python(), does is compute the value of sys.path[0], prepend it to sys.path and run Python in the appropriate mode according to config:

static void
pymain_run_python(int *exitcode)
{
    PyInterpreterState *interp = _PyInterpreterState_GET();
    PyConfig *config = (PyConfig*)_PyInterpreterState_GetConfig(interp);

    // Prepend the search path to `sys.path`
    PyObject *main_importer_path = NULL;
    if (config->run_filename != NULL) {
        // Calculate the search path for the case when the filename is a package
        // (ex: directory or ZIP file) which contains __main__.py, store it in `main_importer_path`.
        // Otherwise, left `main_importer_path` unchanged.
        // Handle other cases later.
        if (pymain_get_importer(config->run_filename, &main_importer_path,
                                exitcode)) {
            return;
        }
    }

    if (main_importer_path != NULL) {
        if (pymain_sys_path_add_path0(interp, main_importer_path) < 0) {
            goto error;
        }
    }
    else if (!config->isolated) {
        PyObject *path0 = NULL;
        // Compute the search path that will be prepended to `sys.path` for other cases.
        // If running as script, then it's the directory where the script is located.
        // If running as module (-m), then it's the current working directory.
        // Otherwise, it's an empty string.
        int res = _PyPathConfig_ComputeSysPath0(&config->argv, &path0);
        if (res < 0) {
            goto error;
        }

        if (res > 0) {
            if (pymain_sys_path_add_path0(interp, path0) < 0) {
                Py_DECREF(path0);
                goto error;
            }
            Py_DECREF(path0);
        }
    }

    PyCompilerFlags cf = _PyCompilerFlags_INIT;

    // Print version and platform in the interactive mode
    pymain_header(config);
    // Import `readline` module to provide completion,
    // line editing and history capabilities in the interactive mode
    pymain_import_readline(config);

    // Run Python depending on the mode of invocation (script, -m, -c, etc.)
    if (config->run_command) {
        *exitcode = pymain_run_command(config->run_command, &cf);
    }
    else if (config->run_module) {
        *exitcode = pymain_run_module(config->run_module, 1);
    }
    else if (main_importer_path != NULL) {
        *exitcode = pymain_run_module(L"__main__", 0);
    }
    else if (config->run_filename != NULL) {
        *exitcode = pymain_run_file(config, &cf);
    }
    else {
        *exitcode = pymain_run_stdin(config, &cf);
    }

    // Enter the interactive mode after executing a program.
    // Enabled by `-i` and `PYTHONINSPECT`.
    pymain_repl(config, &cf, exitcode);
    goto done;

error:
    *exitcode = pymain_exit_err_print();

done:
    Py_XDECREF(main_importer_path);
}

We won't follow all paths, but assume that we run a Python program as a script. This leads us to the pymain_run_file() function that checks whether the specified file can be opened, ensures it's not a directory and calls PyRun_AnyFileExFlags(). The PyRun_AnyFileExFlags() function handles a special case when the file is a terminal (isatty(fd) returns 1). If this is the case, it enters the interactive mode:

$ ./python.exe /dev/ttys000
>>> 1 + 1
2

Otherwise, it calls PyRun_SimpleFileExFlags(). You should be familiar with .pyc files that constantly pop up in __pycache__ directories alongside regular Python files. A .pyc file contains a marshaled code object of a module. It's used instead of the original .py file when we import the module so that the compilation stage can be skipped. I guess you knew that, but did you know that it's possible to run .pyc files directly?

$ ./cpython/python.exe 03/__pycache__/print_path0.cpython-39.pyc
['/Users/Victor/Projects/tenthousandmeters/python_behind_the_scenes/03/__pycache__']

The PyRun_SimpleFileExFlags() function implements this logic. It checks whether the file is a .pyc file, whether it's compiled for the current CPython version and, if yes, calls run_pyc_file(). If the file is not a .pyc file, it calls PyRun_FileExFlags(). Most importantly, though, is that PyRun_SimpleFileExFlags() imports the __main__ module and passes its dictionary to PyRun_FileExFlags() as the global and local namespaces to execute the file in.

int
PyRun_SimpleFileExFlags(FILE *fp, const char *filename, int closeit,
                        PyCompilerFlags *flags)
{
    PyObject *m, *d, *v;
    const char *ext;
    int set_file_name = 0, ret = -1;
    size_t len;

    m = PyImport_AddModule("__main__");
    if (m == NULL)
        return -1;
    Py_INCREF(m);
    d = PyModule_GetDict(m);

    if (PyDict_GetItemString(d, "__file__") == NULL) {
        PyObject *f;
        f = PyUnicode_DecodeFSDefault(filename);
        if (f == NULL)
            goto done;
        if (PyDict_SetItemString(d, "__file__", f) < 0) {
            Py_DECREF(f);
            goto done;
        }
        if (PyDict_SetItemString(d, "__cached__", Py_None) < 0) {
            Py_DECREF(f);
            goto done;
        }
        set_file_name = 1;
        Py_DECREF(f);
    }

    // Check if a .pyc file is passed
    len = strlen(filename);
    ext = filename + len - (len > 4 ? 4 : 0);
    if (maybe_pyc_file(fp, filename, ext, closeit)) {
        FILE *pyc_fp;
        /* Try to run a pyc file. First, re-open in binary */
        if (closeit)
            fclose(fp);
        if ((pyc_fp = _Py_fopen(filename, "rb")) == NULL) {
            fprintf(stderr, "python: Can't reopen .pyc file\n");
            goto done;
        }

        if (set_main_loader(d, filename, "SourcelessFileLoader") < 0) {
            fprintf(stderr, "python: failed to set __main__.__loader__\n");
            ret = -1;
            fclose(pyc_fp);
            goto done;
        }
        v = run_pyc_file(pyc_fp, filename, d, d, flags);
    } else {
        /* When running from stdin, leave __main__.__loader__ alone */
        if (strcmp(filename, "<stdin>") != 0 &&
            set_main_loader(d, filename, "SourceFileLoader") < 0) {
            fprintf(stderr, "python: failed to set __main__.__loader__\n");
            ret = -1;
            goto done;
        }
        v = PyRun_FileExFlags(fp, filename, Py_file_input, d, d,
                              closeit, flags);
    }
    flush_io();
    if (v == NULL) {
        Py_CLEAR(m);
        PyErr_Print();
        goto done;
    }
    Py_DECREF(v);
    ret = 0;
  done:
    if (set_file_name) {
        if (PyDict_DelItemString(d, "__file__")) {
            PyErr_Clear();
        }
        if (PyDict_DelItemString(d, "__cached__")) {
            PyErr_Clear();
        }
    }
    Py_XDECREF(m);
    return ret;
}

The PyRun_FileExFlags() function begins the compilation process. It runs the parser, gets the AST of the module and calls run_mod() to run the AST. It also creates a PyArena object, which CPython uses to allocate small objects (smaller or equal to 512 bytes):

PyObject *
PyRun_FileExFlags(FILE *fp, const char *filename_str, int start, PyObject *globals,
                  PyObject *locals, int closeit, PyCompilerFlags *flags)
{
    PyObject *ret = NULL;
    mod_ty mod;
    PyArena *arena = NULL;
    PyObject *filename;
    int use_peg = _PyInterpreterState_GET()->config._use_peg_parser;

    filename = PyUnicode_DecodeFSDefault(filename_str);
    if (filename == NULL)
        goto exit;

    arena = PyArena_New();
    if (arena == NULL)
        goto exit;

    // Run the parser.
    // By default the new PEG parser is used.
    // Pass `-X oldparser` to use the old parser.
    // `mod` stands for module. It's the root node of the AST.
    if (use_peg) {
        mod = PyPegen_ASTFromFileObject(fp, filename, start, NULL, NULL, NULL,
                                        flags, NULL, arena);
    }
    else {
        mod = PyParser_ASTFromFileObject(fp, filename, NULL, start, 0, 0,
                                         flags, NULL, arena);
    }

    if (closeit)
        fclose(fp);
    if (mod == NULL) {
        goto exit;
    }
    // Compile the AST and run.
    ret = run_mod(mod, filename, globals, locals, flags, arena);

exit:
    Py_XDECREF(filename);
    if (arena != NULL)
        PyArena_Free(arena);
    return ret;
}

run_mod() runs the compiler by calling PyAST_CompileObject(), gets the module's code object and calls run_eval_code_obj() to execute the code object. In the interim, it raises the exec event, which is a CPython's way to notify auditing tools when something important happens inside the Python runtime. PEP 578 explains this mechanism.

static PyObject *
run_mod(mod_ty mod, PyObject *filename, PyObject *globals, PyObject *locals,
            PyCompilerFlags *flags, PyArena *arena)
{
    PyThreadState *tstate = _PyThreadState_GET();
    PyCodeObject *co = PyAST_CompileObject(mod, filename, flags, -1, arena);
    if (co == NULL)
        return NULL;

    if (_PySys_Audit(tstate, "exec", "O", co) < 0) {
        Py_DECREF(co);
        return NULL;
    }

    PyObject *v = run_eval_code_obj(tstate, co, globals, locals);
    Py_DECREF(co);
    return v;
}

We already know from part 2 that the compiler works by:

building a symbol table
creating a CFG of basic blocks; and
assembling the CFG into a code object.

This is exactly what PyAST_CompileObject() does, so we won't discuss it.

run_eval_code_obj() begins a chain of trivial function calls that eventually lead us to _PyEval_EvalCode(). I paste all those functions here so that you can see where the parameters of _PyEval_EvalCode() come from:

static PyObject *
run_eval_code_obj(PyThreadState *tstate, PyCodeObject *co, PyObject *globals, PyObject *locals)
{
    PyObject *v;
    // The special case when CPython is embeddded. We can safely ignore it.
    /*
     * We explicitly re-initialize _Py_UnhandledKeyboardInterrupt every eval
     * _just in case_ someone is calling into an embedded Python where they
     * don't care about an uncaught KeyboardInterrupt exception (why didn't they
     * leave config.install_signal_handlers set to 0?!?) but then later call
     * Py_Main() itself (which _checks_ this flag and dies with a signal after
     * its interpreter exits).  We don't want a previous embedded interpreter's
     * uncaught exception to trigger an unexplained signal exit from a future
     * Py_Main() based one.
     */
    _Py_UnhandledKeyboardInterrupt = 0;

    /* Set globals['__builtins__'] if it doesn't exist */
    // In our case, it's been already set to the `builtins` module during the main initialization.
    if (globals != NULL && PyDict_GetItemString(globals, "__builtins__") == NULL) {
        if (PyDict_SetItemString(globals, "__builtins__",
                                 tstate->interp->builtins) < 0) {
            return NULL;
        }
    }

    v = PyEval_EvalCode((PyObject*)co, globals, locals);
    if (!v && _PyErr_Occurred(tstate) == PyExc_KeyboardInterrupt) {
        _Py_UnhandledKeyboardInterrupt = 1;
    }
    return v;
}

PyObject *
PyEval_EvalCode(PyObject *co, PyObject *globals, PyObject *locals)
{
    return PyEval_EvalCodeEx(co,
                      globals, locals,
                      (PyObject **)NULL, 0,
                      (PyObject **)NULL, 0,
                      (PyObject **)NULL, 0,
                      NULL, NULL);
}

PyObject *
PyEval_EvalCodeEx(PyObject *_co, PyObject *globals, PyObject *locals,
                  PyObject *const *args, int argcount,
                  PyObject *const *kws, int kwcount,
                  PyObject *const *defs, int defcount,
                  PyObject *kwdefs, PyObject *closure)
{
    return _PyEval_EvalCodeWithName(_co, globals, locals,
                                    args, argcount,
                                    kws, kws != NULL ? kws + 1 : NULL,
                                    kwcount, 2,
                                    defs, defcount,
                                    kwdefs, closure,
                                    NULL, NULL);
}

PyObject *
_PyEval_EvalCodeWithName(PyObject *_co, PyObject *globals, PyObject *locals,
           PyObject *const *args, Py_ssize_t argcount,
           PyObject *const *kwnames, PyObject *const *kwargs,
           Py_ssize_t kwcount, int kwstep,
           PyObject *const *defs, Py_ssize_t defcount,
           PyObject *kwdefs, PyObject *closure,
           PyObject *name, PyObject *qualname)
{
    PyThreadState *tstate = _PyThreadState_GET();
    return _PyEval_EvalCode(tstate, _co, globals, locals,
               args, argcount,
               kwnames, kwargs,
               kwcount, kwstep,
               defs, defcount,
               kwdefs, closure,
               name, qualname);
}

Recall that a code object describes what a piece of code does but to execute a code object, CPython needs to create a state for it, which is what a frame object is. _PyEval_EvalCode() creates a frame object for a given code object with specified parameters. In our case, most of the parameters are NULL, so little has to be done. Much more work is required when CPython executes, for example, a function's code object with different kinds of arguments passed. As a result, _PyEval_EvalCode() is nearly 300 lines long. We'll see what most of them are for in the next parts. For now, you can skip through _PyEval_EvalCode() to ensure that in the end it calls _PyEval_EvalFrame() to evaluate the created frame object:

PyObject *
_PyEval_EvalCode(PyThreadState *tstate,
           PyObject *_co, PyObject *globals, PyObject *locals,
           PyObject *const *args, Py_ssize_t argcount,
           PyObject *const *kwnames, PyObject *const *kwargs,
           Py_ssize_t kwcount, int kwstep,
           PyObject *const *defs, Py_ssize_t defcount,
           PyObject *kwdefs, PyObject *closure,
           PyObject *name, PyObject *qualname)
{
    assert(is_tstate_valid(tstate));

    PyCodeObject* co = (PyCodeObject*)_co;
    PyFrameObject *f;
    PyObject *retval = NULL;
    PyObject **fastlocals, **freevars;
    PyObject *x, *u;
    const Py_ssize_t total_args = co->co_argcount + co->co_kwonlyargcount;
    Py_ssize_t i, j, n;
    PyObject *kwdict;

    if (globals == NULL) {
        _PyErr_SetString(tstate, PyExc_SystemError,
                         "PyEval_EvalCodeEx: NULL globals");
        return NULL;
    }

    /* Create the frame */
    f = _PyFrame_New_NoTrack(tstate, co, globals, locals);
    if (f == NULL) {
        return NULL;
    }
    fastlocals = f->f_localsplus;
    freevars = f->f_localsplus + co->co_nlocals;

    /* Create a dictionary for keyword parameters (**kwags) */
    if (co->co_flags & CO_VARKEYWORDS) {
        kwdict = PyDict_New();
        if (kwdict == NULL)
            goto fail;
        i = total_args;
        if (co->co_flags & CO_VARARGS) {
            i++;
        }
        SETLOCAL(i, kwdict);
    }
    else {
        kwdict = NULL;
    }

    /* Copy all positional arguments into local variables */
    if (argcount > co->co_argcount) {
        n = co->co_argcount;
    }
    else {
        n = argcount;
    }
    for (j = 0; j < n; j++) {
        x = args[j];
        Py_INCREF(x);
        SETLOCAL(j, x);
    }

    /* Pack other positional arguments into the *args argument */
    if (co->co_flags & CO_VARARGS) {
        u = _PyTuple_FromArray(args + n, argcount - n);
        if (u == NULL) {
            goto fail;
        }
        SETLOCAL(total_args, u);
    }

    /* Handle keyword arguments passed as two strided arrays */
    kwcount *= kwstep;
    for (i = 0; i < kwcount; i += kwstep) {
        PyObject **co_varnames;
        PyObject *keyword = kwnames[i];
        PyObject *value = kwargs[i];
        Py_ssize_t j;

        if (keyword == NULL || !PyUnicode_Check(keyword)) {
            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() keywords must be strings",
                          co->co_name);
            goto fail;
        }

        /* Speed hack: do raw pointer compares. As names are
           normally interned this should almost always hit. */
        co_varnames = ((PyTupleObject *)(co->co_varnames))->ob_item;
        for (j = co->co_posonlyargcount; j < total_args; j++) {
            PyObject *name = co_varnames[j];
            if (name == keyword) {
                goto kw_found;
            }
        }

        /* Slow fallback, just in case */
        for (j = co->co_posonlyargcount; j < total_args; j++) {
            PyObject *name = co_varnames[j];
            int cmp = PyObject_RichCompareBool( keyword, name, Py_EQ);
            if (cmp > 0) {
                goto kw_found;
            }
            else if (cmp < 0) {
                goto fail;
            }
        }

        assert(j >= total_args);
        if (kwdict == NULL) {

            if (co->co_posonlyargcount
                && positional_only_passed_as_keyword(tstate, co,
                                                     kwcount, kwnames))
            {
                goto fail;
            }

            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() got an unexpected keyword argument '%S'",
                          co->co_name, keyword);
            goto fail;
        }

        if (PyDict_SetItem(kwdict, keyword, value) == -1) {
            goto fail;
        }
        continue;

      kw_found:
        if (GETLOCAL(j) != NULL) {
            _PyErr_Format(tstate, PyExc_TypeError,
                          "%U() got multiple values for argument '%S'",
                          co->co_name, keyword);
            goto fail;
        }
        Py_INCREF(value);
        SETLOCAL(j, value);
    }

    /* Check the number of positional arguments */
    if ((argcount > co->co_argcount) && !(co->co_flags & CO_VARARGS)) {
        too_many_positional(tstate, co, argcount, defcount, fastlocals);
        goto fail;
    }

    /* Add missing positional arguments (copy default values from defs) */
    if (argcount < co->co_argcount) {
        Py_ssize_t m = co->co_argcount - defcount;
        Py_ssize_t missing = 0;
        for (i = argcount; i < m; i++) {
            if (GETLOCAL(i) == NULL) {
                missing++;
            }
        }
        if (missing) {
            missing_arguments(tstate, co, missing, defcount, fastlocals);
            goto fail;
        }
        if (n > m)
            i = n - m;
        else
            i = 0;
        for (; i < defcount; i++) {
            if (GETLOCAL(m+i) == NULL) {
                PyObject *def = defs[i];
                Py_INCREF(def);
                SETLOCAL(m+i, def);
            }
        }
    }

    /* Add missing keyword arguments (copy default values from kwdefs) */
    if (co->co_kwonlyargcount > 0) {
        Py_ssize_t missing = 0;
        for (i = co->co_argcount; i < total_args; i++) {
            PyObject *name;
            if (GETLOCAL(i) != NULL)
                continue;
            name = PyTuple_GET_ITEM(co->co_varnames, i);
            if (kwdefs != NULL) {
                PyObject *def = PyDict_GetItemWithError(kwdefs, name);
                if (def) {
                    Py_INCREF(def);
                    SETLOCAL(i, def);
                    continue;
                }
                else if (_PyErr_Occurred(tstate)) {
                    goto fail;
                }
            }
            missing++;
        }
        if (missing) {
            missing_arguments(tstate, co, missing, -1, fastlocals);
            goto fail;
        }
    }

    /* Allocate and initialize storage for cell vars, and copy free
       vars into frame. */
    for (i = 0; i < PyTuple_GET_SIZE(co->co_cellvars); ++i) {
        PyObject *c;
        Py_ssize_t arg;
        /* Possibly account for the cell variable being an argument. */
        if (co->co_cell2arg != NULL &&
            (arg = co->co_cell2arg[i]) != CO_CELL_NOT_AN_ARG) {
            c = PyCell_New(GETLOCAL(arg));
            /* Clear the local copy. */
            SETLOCAL(arg, NULL);
        }
        else {
            c = PyCell_New(NULL);
        }
        if (c == NULL)
            goto fail;
        SETLOCAL(co->co_nlocals + i, c);
    }

    /* Copy closure variables to free variables */
    for (i = 0; i < PyTuple_GET_SIZE(co->co_freevars); ++i) {
        PyObject *o = PyTuple_GET_ITEM(closure, i);
        Py_INCREF(o);
        freevars[PyTuple_GET_SIZE(co->co_cellvars) + i] = o;
    }

    /* Handle generator/coroutine/asynchronous generator */
    if (co->co_flags & (CO_GENERATOR | CO_COROUTINE | CO_ASYNC_GENERATOR)) {
        PyObject *gen;
        int is_coro = co->co_flags & CO_COROUTINE;

        /* Don't need to keep the reference to f_back, it will be set
         * when the generator is resumed. */
        Py_CLEAR(f->f_back);

        /* Create a new generator that owns the ready to run frame
         * and return that as the value. */
        if (is_coro) {
            gen = PyCoro_New(f, name, qualname);
        } else if (co->co_flags & CO_ASYNC_GENERATOR) {
            gen = PyAsyncGen_New(f, name, qualname);
        } else {
            gen = PyGen_NewWithQualName(f, name, qualname);
        }
        if (gen == NULL) {
            return NULL;
        }

        _PyObject_GC_TRACK(f);

        return gen;
    }

    retval = _PyEval_EvalFrame(tstate, f, 0);

fail: /* Jump here from prelude on failure */

    /* decref'ing the frame can cause __del__ methods to get invoked,
       which can call back into Python.  While we're done with the
       current Python frame (f), the associated C stack is still in use,
       so recursion_depth must be boosted for the duration.
    */
    if (Py_REFCNT(f) > 1) {
        Py_DECREF(f);
        _PyObject_GC_TRACK(f);
    }
    else {
        ++tstate->recursion_depth;
        Py_DECREF(f);
        --tstate->recursion_depth;
    }
    return retval;
}

_PyEval_EvalFrame() is a wrapper around interp->eval_frame(), which is the frame evaluation function. It's possible to set interp->eval_frame() to a custom function. We could, for example, add a JIT compiler to CPython by replacing the default evaluation function with the one that stores compiled machine code in a code object and can run such code. PEP 523 introduced this functionality in CPython 3.6.

By default, interp->eval_frame() is set to _PyEval_EvalFrameDefault(). This function, defined in Python/ceval.c, consists of almost 3,000 lines. Today, though, we're only interested in one. Line 1741 begins what we've been waiting for so long: the evaluation loop.

Conclusion

We've discussed a lot today. We started by making an overview of the CPython project, compiled CPython and stepped through its source code, studying the initialization stage along the way. I think this should give us a descent understanding of what CPython does before it starts interpreting the bytecode. What happens after is the subject of the next post.

Meanwhile, to solidify what we learned today and to learn more, I really recommend you find some time to explore the CPython source code on your own. I bet you have many questions after reading this post, so you should have something to look for. Have a good time!

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #2: how the CPython compiler works

2020-09-20T05:48:00+00:00

Today's subject

In the first post of the series we've looked at the CPython VM. We've learned that it works by executing a series of instructions called bytecode. We've also seen that Python bytecode is not sufficient to fully describe what a piece of code does. That's why there exists a notion of a code object. To execute a code block such as a module or a function means to execute a corresponding code object. A code object contains the block's bytecode, the constants and the names of variables used within the block and the block's various properties.

Typically, a Python programmer doesn't write bytecode and doesn't create the code objects but writes a normal Python code. So CPython must be able to create a code object from a source code. This job is done by the CPython compiler. In this part we'll explore how it works.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

What the CPython compiler is

We understood what the responsibilities of the CPython compiler are, but before looking at how it is implemented, let's figure out why we call it a compiler in the first place.

A compiler, in its general sense, is a program that translates a program in one language into an equivalent program in another language. There are many types of compilers, but most of the time by a compiler we mean a static compiler, which translates a program in a high-level language to a machine code. Does the CPython compiler have something in common with this type of a compiler? To answer this question, let's take a look at the traditional three-stage design of a static compiler.

The frontend of a compiler transforms a source code into some intermediate representation (IR). The optimizer then takes an IR, optimizes it and passes an optimized IR to the backend that generates machine code. If we choose an IR that is not specific to any source language and any target machine, then we get a key benefit of the three-stage design: for a compiler to support a new source language, only an additional frontend is needed, and to support a new target machine, only an additional backend is needed.

The LLVM toolchain is a great example of a success of this model. There are frontends for C, Rust, Swift and many other programming languages that rely on LLVM to provide more complicated parts of the compiler. LLVM's creator, Chris Lattner, gives a good overview of its architecture.

CPython, however, doesn't need to support multiple source languages and target machines but only a Python code and the CPython VM. Nevertheless, the CPython compiler is an implementation of the three-stage design. To see why, we should examine the stages of a three-stage compiler in more detail.

The picture above represents a model of a classic compiler. Now compare it to the architecture of the CPython compiler in the picture below.

Looks similar, doesn't it? The point here is that the structure of the CPython compiler should be familiar to anyone who studied compilers before. If you didn't, a famous Dragon Book is an excellent introduction to the theory of compiler construction. It's long, but you'll benefit even by reading only the first few chapters.

The comparison we've made requires several comments. First, since version 3.9, CPython uses a new parser by default that outputs an AST (Abstract Syntax Tree) straight away without an intermediate step of building a parse tree. Thus, the model of the CPython compiler is simplified even further. Second, some of the presented phases of the CPython compiler do so little compared to their counterparts of the static compilers that some may say that the CPython compiler is no more than a frontend. We won't take this view of the hardcore compiler writers.

Overview of the compiler's architecture

The diagrams are nice, but they hide many details and can be misleading, so let's spend some time discussing the overall design of the CPython compiler.

The two major components of the CPython compiler are:

the frontend; and
the backend.

The frontend takes a Python code and produces an AST. The backend takes an AST and produces a code object. Throughout the CPython source code, the terms parser and compiler are used for the frontend and the backend respectively. This is yet another meaning of the word compiler. It was probably better to call it something like a code object generator, but we'll stick with the compiler since it doesn't seem to cause much trouble.

The job of the parser is to check whether the input is a syntactically correct Python code. If it's not, then the parser reports an error like the following:

x = y = = 12
        ^
SyntaxError: invalid syntax

If the input is correct, then the parser organizes it according to the rules of the grammar. A grammar defines the syntax of a language. The notion of a formal grammar is so crucial for our discussion that, I think, we should digress a little to recall its formal definition.

According to the classic definition, a grammar is a tuple of four items:

$\Sigma$ – a finite set of terminal symbols, or simply terminals (usually denoted by lowercase letters).
$N$ – a finite set of nonterminal symbols, or simply nonterminals (usually denoted by uppercase letters).
$P$ – a set of production rules. In the case of context-free grammars, which include the Python grammar, a production rule is just a mapping from a nonterminal to any sequence of terminals and nonterminals like $A \to aB$.
$S$ – one distinguished nonterminal.

A grammar defines a language that consists of all sequences of terminals that can be generated by applying production rules. To generate some sequence, one starts with the symbol $S$ and then recursively replaces each nonterminal with a sequence according to production rules until the whole sequence consists of terminals. Using established convention for the notation, it's sufficient to list production rules to specify the grammar. Here's, for example, a simple grammar that generates sequences of alternating ones and zeros:

$S \to 10S \;| \;10$

We'll continue to discuss grammars when we look at the parser in more detail.

Abstract syntax tree

The ultimate goal of the parser is to produce an AST. An AST is a tree data structure that serves as a high-level representation of a source code. Here's an example of a piece of code and a dump of the corresponding AST produced by the standard ast module:

x = 123
f(x)

$ python -m ast example1.py
Module(
   body=[
      Assign(
         targets=[
            Name(id='x', ctx=Store())],
         value=Constant(value=123)),
      Expr(
         value=Call(
            func=Name(id='f', ctx=Load()),
            args=[
               Name(id='x', ctx=Load())],
            keywords=[]))],
   type_ignores=[])

The types of the AST nodes are formally defined using the Zephyr Abstract Syntax Definition Language (ASDL). The ASDL is a simple declarative language that was created to describe tree-like IRs, which is what the AST is. Here's the definitions of the Assign and Expr nodes from Parser/Python.asdl:

stmt = ... | Assign(expr* targets, expr value, string? type_comment) | ...
expr = ... | Call(expr func, expr* args, keyword* keywords) | ...

The ASDL specification should give us an idea of what the Python AST looks like. The parser, however, needs to represent an AST in the C code. Fortunately, it's easy to generate the C structs for the AST nodes from their ASDL descriptions. That's what CPython does, and the result looks like this:

struct _stmt {
    enum _stmt_kind kind;
    union {
        // ... other kinds of statements
        struct {
            asdl_seq *targets;
            expr_ty value;
            string type_comment;
        } Assign;
        // ... other kinds of statements
    } v;
    int lineno;
    int col_offset;
    int end_lineno;
    int end_col_offset;
};

struct _expr {
    enum _expr_kind kind;
    union {
        // ... other kinds of expressions
        struct {
            expr_ty func;
            asdl_seq *args;
            asdl_seq *keywords;
        } Call;
        // ... other kinds of expressions
    } v;
    // ... same as in _stmt
};

An AST is a handy representation to work with. It tells what a program does, hiding all non-essential information such as indentation, punctuation and other Python's syntactic features.

One of the main beneficiaries of the AST representation is the compiler, which can walk an AST and emit bytecode in a relatively straightforward manner. Many Python tools, besides the compiler, use the AST to work with Python code. For example, pytest makes changes to an AST to provide useful information when the assert statement fails, which by itself does nothing but raises an AssertionError if the expression evaluates to False. Another example is Bandit that finds common security issues in Python code by analyzing an AST.

Now, when we've studied the Python AST a little bit, we can look at how the parser builds it from a source code.

From source code to AST

In fact, as I mentioned earlier, starting with version 3.9, CPython has not one but two parsers. The new parser is used by default. It's also possible to use the old parser by passing -X oldparser option. In CPython 3.10, however, the old parser will be completely removed.

The two parsers are very different. We'll focus on the new one, but before that, discuss the old parser as well.

old parser

For a long time Python's syntax was formally defined by the generative grammar. It's a kind of grammar we've talked about earlier. It tells us how to generate sequences belonging to the language. The problem is that a generative grammar doesn't directly correspond to the parsing algorithm that would be able to parse those sequences. Fortunately, smart people have been able to distinguish classes of generative grammars for which the corresponding parser can be built. These include context free, LL(k), LR(k), LALR and many other types of grammars. The Python grammar is LL(1). It's specified using a kind of Extended Backus–Naur Form (EBNF). To get an idea on how it can be used to describe Python's syntax, take a look at the rules for the while statement.

file_input: (NEWLINE | stmt)* ENDMARKER
stmt: simple_stmt | compound_stmt
compound_stmt: ... | while_stmt | ...
while_stmt: 'while' namedexpr_test ':' suite ['else' ':' suite]
suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT
...

CPython extends the traditional notation with features like:

grouping of alternatives: (a | b)
optional parts: [a]
zero or more and one or more repetitions: a* and a+.

We can see why Guido van Rossum chose to use regular expressions. They allow expressing the syntax of a programming language in a more natural (for a programmer) way. Instead of writing $A \to aA | a$ , we can just write $A \to a+$. This choice came with the cost: CPython had to develop a method to support the extended notation.

The parsing of an LL(1) grammar is a solved problem. The solution is a Pushdown Automaton (PDA) that acts as a top-down parser. A PDA operates by simulating the generation of an input string using a stack. To parse some input, it starts with the start symbol on the stack. Then it looks at the first symbol in the input, guesses which rule should be applied to the start symbol and replaces it with the right-hand side of that rule. If a top symbol on the stack is a terminal that matches the next symbol in the input, the PDA pops it and skips the matched symbol. If a top symbol is a nonterminal, the PDA tries to guess the rule to replace it with based on the next symbol in the input. The process repeats until the whole input is scanned or if the PDA can't match a terminal on the stack with the next symbol in the input. The latter case means that the input string cannot be parsed.

CPython couldn't use this method directly because of how the production rules are written, so the new method had to be developed. To support the extended notation, the old parser represents each rule of the grammar with a Deterministic Finite Automaton (DFA), which is famous for being equivalent to a regular expression. The parser itself is a stack-based automaton like PDA, but instead of pushing symbols on the stack, it pushes states of the DFAs. Here's the key data structures used by the old parser:

typedef struct {
    int              s_state;       /* State in current DFA */
    const dfa       *s_dfa;         /* Current DFA */
    struct _node    *s_parent;      /* Where to add next node */
} stackentry;

typedef struct {
    stackentry      *s_top;         /* Top entry */
    stackentry       s_base[MAXSTACK];/* Array of stack entries */
                                    /* NB The stack grows down */
} stack;

typedef struct {
    stack           p_stack;        /* Stack of parser states */
    grammar         *p_grammar;     /* Grammar to use */
                                    // basically, a collection of DFAs
    node            *p_tree;        /* Top of parse tree */
    // ...
} parser_state;

And the comment from Parser/parser.c that summarizes the approach:

A parsing rule is represented as a Deterministic Finite-state Automaton (DFA). A node in a DFA represents a state of the parser; an arc represents a transition. Transitions are either labeled with terminal symbols or with nonterminals. When the parser decides to follow an arc labeled with a nonterminal, it is invoked recursively with the DFA representing the parsing rule for that as its initial state; when that DFA accepts, the parser that invoked it continues. The parse tree constructed by the recursively called parser is inserted as a child in the current parse tree.

The parser builds a parse tree, also known as Concrete Syntax Tree (CST), while parsing an input. In contrast to an AST, a parse tree directly corresponds to the rules applied when deriving an input. All nodes in a parse tree are represented using the same node struct:

typedef struct _node {
    short               n_type;
    char                *n_str;
    int                 n_lineno;
    int                 n_col_offset;
    int                 n_nchildren;
    struct _node        *n_child;
    int                 n_end_lineno;
    int                 n_end_col_offset;
} node;

A parse tree, however, is not what the compiler waits for. It has to be converted to an AST. This work is done in Python/ast.c. The algorithm is to walk a parse tree recursively and translate its nodes to the AST nodes. Hardly anyone finds these nearly 6,000 lines of code exciting.

tokenizer

Python is not a simple language from the syntactic point of view. The Python grammar, though, looks simple and fits in about 200 lines including comments. This is because the symbols of the grammar are tokens and not individual characters. A token is represented by the type, such as NUMBER, NAME, NEWLINE, the value and the position in a source code. CPython distinguishes 63 types of tokens, all of which are listed in Grammar/Tokens. We can see what a tokenized program looks like using the standard tokenize module:

def x_plus(x):
    if x >= 0:
        return x
    return 0

$ python -m tokenize example2.py 
0,0-0,0:            ENCODING       'utf-8'        
1,0-1,3:            NAME           'def'          
1,4-1,10:           NAME           'x_plus'       
1,10-1,11:          OP             '('            
1,11-1,12:          NAME           'x'            
1,12-1,13:          OP             ')'            
1,13-1,14:          OP             ':'            
1,14-1,15:          NEWLINE        '\n'           
2,0-2,4:            INDENT         '    '         
2,4-2,6:            NAME           'if'           
2,7-2,8:            NAME           'x'            
2,9-2,11:           OP             '>='           
2,12-2,13:          NUMBER         '0'            
2,13-2,14:          OP             ':'            
2,14-2,15:          NEWLINE        '\n'           
3,0-3,8:            INDENT         '        '     
3,8-3,14:           NAME           'return'       
3,15-3,16:          NAME           'x'            
3,16-3,17:          NEWLINE        '\n'           
4,4-4,4:            DEDENT         ''             
4,4-4,10:           NAME           'return'       
4,11-4,12:          NUMBER         '0'            
4,12-4,13:          NEWLINE        '\n'           
5,0-5,0:            DEDENT         ''             
5,0-5,0:            ENDMARKER      ''

This is how the program looks to the parser. When the parser needs a token, it requests one from the tokenizer. The tokenizer reads one character at a time from the buffer and tries to match the seen prefix with some type of token. How does the tokenizer work with different encodings? It relies on the io module. First, the tokenizer detects the encoding. If no encoding is specified, it defaults to UTF-8. Then, the tokenizer opens a file with a C call, which is equivalent to Python's open(fd, mode='r', encoding=enc), and reads its contents by calling the readline() function. This function returns a unicode string. The characters the tokenizer reads are just bytes in the UTF-8 representation of that string (or EOF).

We could define what a number or a name is directly in the grammar, though it would become more complex. What we couldn't do is to express the significance of indentation in the grammar without making it context-sensitive and, therefore, not suitable for parsing. The tokenizer makes the work of the parser much easier by providing the INDENT and DEDENT tokens. They mean what the curly braces mean in a language like C. The tokenizer is powerful enough to handle indentation because it has state. The current indentation level is kept on the top of the stack. When the level is increased, it's pushed on the stack. If the level is decreased, all higher levels are popped from the stack.

The old parser is a non-trivial piece of the CPython codebase. The DFAs for the rules of the grammar are generated automatically, but other parts of the parser are written by hand. This is in contrast with the new parser, which seems to be a much more elegant solution to the problem of parsing Python code.

new parser

The new parser comes with the new grammar. This grammar is a Parsing Expression Grammar (PEG). The important thing to understand is that PEG is not just a class of grammars. It's another way to define a grammar. PEGs were introduced by Bryan Ford in 2004 as a tool to describe a programming language and to generate a parser based on the description. A PEG is different from the traditional formal grammar in that its rules map nonterminals to the parsing expressions instead of just sequences of symbols. This is in the spirit of CPython. A parsing expression is defined inductively. If $e$, $e_1$, and $e_2$ are parsing expressions, then so is:

the empty string
any terminal
any nonterminal
$e_1e_2$, a sequence
$e_1/e_2$, prioritized choice
$e*$, zero-or-more repetitions
$!e$, a not-predicate.

PEGs are analytic grammars, which means that they are designed not only to generate languages but to analyze them as well. Ford formalized what it means for a parsing expression $e$ to recognize an input $x$. Basically, any attempt to recognize an input with some parsing expression can either succeed or fail and consume some input or not. For example, applying the parsing expression $a$ to the input $ab$ results in a success and consumes $a$.

This formalization allows converting any PEG to a recursive descent parser. A recursive descent parser associates each nonterminal of a grammar with a parsing function. In the case of a PEG, the body of a parsing function is an implementation of the corresponding parsing expression. If a parsing expression contains nonterminals, their parsing functions are called recursively.

A nonterminal may have multiple production rules. A recursive descent parser has to decide which one was used to derive the input. If a grammar is LL(k), a parser can look at the next k tokens in the input and predict the correct rule. Such a parser is called a predictive parser. If it's not possible to predict, the backtracking method is used. A parser with backtracking tries one rule, and, if fails, backtracks and tries another. This is exactly what the prioritized choice operator in a PEG does. So, a PEG parser is a recursive descent parser with backtracking.

The backtracking method is powerful but can be computationally costly. Consider a simple example. We apply the expression $AB/A$ to the input that succeeds on $A$ but then fails on $B$. According to the the interpretation of the prioritized choice operator, the parser first tries to recognize $A$, succeeds, and then tries to recognize B. It fails on $B$ and tries to recognize $A$ again. Because of such redundant computations, the parse time can be exponential in the size of the input. To remedy this problem, Ford suggested to use a memoization technique, i.e. caching the results of function calls. Using this technique, the parser, known as the packrat parser, is guaranteed to work in linear time at the expense of a higher memory consumption. And this is what CPython's new parser does. It's a packrat parser!

No matter how good the new parser is, the reasons to replace the old parser have to be given. This is what the PEPs are for. PEP 617 -- New PEG parser for CPython gives a background on both the old and the new parser and explains the reasons behind the transition. In a nutshell, the new parser removes the LL(1) restriction on the grammar and should be easier to maintain. Guido van Rossum wrote an excellent series on a PEG parsing, in which he goes into much more detail and shows how to implement a simple PEG parser. We, in our turn, will take a look at its CPython implementation.

You might be surprised to learn that the new grammar file is more than three times bigger than the old one. This is because the new grammar is not just a grammar but a Syntax-Directed Translation Scheme (SDTS). An SDTS is a grammar with actions attached to the rules. An action is a piece of code. A parser executes an action when it applies the corresponding rule to the input and succeeds. CPython uses actions to build an AST while parsing. To see how, let's see what the new grammar looks like. We've already seen the rules of the old grammar for the while statement, so here's their new analogues:

file[mod_ty]: a=[statements] ENDMARKER { _PyPegen_make_module(p, a) }
statements[asdl_seq*]: a=statement+ { _PyPegen_seq_flatten(p, a) }
statement[asdl_seq*]: a=compound_stmt { _PyPegen_singleton_seq(p, a) } | simple_stmt
compound_stmt[stmt_ty]:
    | ...
    | &'while' while_stmt
while_stmt[stmt_ty]:
    | 'while' a=named_expression ':' b=block c=[else_block] { _Py_While(a, b, c, EXTRA) }
...

Each rule starts with the name of a nonterminal. It's followed by the C type of the result that the parsing function returns. The right-hand side is a parsing expression. The code in the curly braces denotes an action. Actions are simple function calls that return AST nodes or their fields.

The new parser is Parser/pegen/parse.c. It's generated automatically by the parser generator. The parser generator is written in Python. It's a program that takes a grammar and generates a PEG parser in C or Python. A grammar is described in the grammar file and represented by the instance of the Grammar class. To create such an instance, there must be a parser for the grammar file. This parser is also generated automatically by the parser generator from the metagrammar. That's why the parser generator can generate a parser in Python. But what parses the metagrammar? Well, it's in the same notation as grammar, so the generated grammar parser is able to parse the metagrammar as well. Of course, the grammar parser had to be bootstrapped, i.e. the first version had to be written by hand. Once that's done, all parsers can be generated automatically.

Like the old parser, the new parser gets tokens from the tokenizer. This is unusual for a PEG parser since it allows unifying tokenization and parsing. But we saw that the tokenizer does a non-trivial job, so the CPython developers decided to make use of it.

On this note, we end our discussion of parsing to see what happens next to an AST.

AST optimization

The diagram of the CPython compiler's architecture shows us the AST optimizer alongside the parser and the compiler. This probably overemphasizes the optimizer's role. The AST optimizer is confined to constant folding and was introduced only in CPython 3.7. Before CPython 3.7, constant folding was done at a later stage by the peephole optimizer. Nonetheless, due to the AST optimizer, we can write things like this:

n = 2 ** 32 # easier to write and to read

and expect it to be calculated at compile time.

An example of a less obvious optimization is the conversion of a list of constants and a set of constants into a tuple and a frozenset respectively. This optimization is performed when a list or a set are used on the right-hand side of the in or not in operators.

From AST to code object

Up until now, we've been studying how CPython creates an AST from a source code, but as we've seen in the first post, the CPython VM knows nothing about the AST and is only able to execute a code object. The conversion of an AST to a code object is a job of the compiler. More specifically, the compiler must return the module's code object containing the module's bytecode along with the code objects for other code blocks in the module such as defined functions and classes.

Sometimes the best way to understand a solution to a problem is to think of one's own. Let's ponder what we would do if we were the compiler. We start with the root node of an AST that represents a module. Children of this node are statements. Let's assume that the first statement is a simple assignment like x = 1. It's represented by the Assign AST node: Assign(targets=[Name(id='x', ctx=Store())], value=Constant(value=1)). To convert this node to a code object we need to create one, store constant 1 in the list of constants of the code object, store the name of the variable x in the list of names used in the code object and emit the LOAD_CONST and STORE_NAME instructions. We could write a function to do that. But even a simple assignment can be tricky. For example, imagine that the same assignment is made inside the body of a function. If x is a local variable, we should emit the STORE_FAST instruction. If x is a global variable, we should emit the STORE_GLOBAL instruction. Finally, if x is referenced by a nested function, we should emit the STORE_DEREF instruction. The problem is to determine what the type of the variable x is. CPython solves this problem by building a symbol table before compiling.

symbol table

A symbol table contains information about code blocks and the symbols used within them. It's represented by a single symtable struct and a collection of _symtable_entry structs, one for each code block in a program. A symbol table entry contains the properties of a code block, including its name, its type (module, class or function) and a dictionary that maps the names of variables used within the block to the flags indicating their scope and usage. Here's the complete definition of the _symtable_entry struct:

typedef struct _symtable_entry {
    PyObject_HEAD
    PyObject *ste_id;        /* int: key in ste_table->st_blocks */
    PyObject *ste_symbols;   /* dict: variable names to flags */
    PyObject *ste_name;      /* string: name of current block */
    PyObject *ste_varnames;  /* list of function parameters */
    PyObject *ste_children;  /* list of child blocks */
    PyObject *ste_directives;/* locations of global and nonlocal statements */
    _Py_block_ty ste_type;   /* module, class, or function */
    int ste_nested;      /* true if block is nested */
    unsigned ste_free : 1;        /* true if block has free variables */
    unsigned ste_child_free : 1;  /* true if a child block has free vars,
                                     including free refs to globals */
    unsigned ste_generator : 1;   /* true if namespace is a generator */
    unsigned ste_coroutine : 1;   /* true if namespace is a coroutine */
    unsigned ste_comprehension : 1; /* true if namespace is a list comprehension */
    unsigned ste_varargs : 1;     /* true if block has varargs */
    unsigned ste_varkeywords : 1; /* true if block has varkeywords */
    unsigned ste_returns_value : 1;  /* true if namespace uses return with
                                        an argument */
    unsigned ste_needs_class_closure : 1; /* for class scopes, true if a
                                             closure over __class__
                                             should be created */
    unsigned ste_comp_iter_target : 1; /* true if visiting comprehension target */
    int ste_comp_iter_expr; /* non-zero if visiting a comprehension range expression */
    int ste_lineno;          /* first line of block */
    int ste_col_offset;      /* offset of first line of block */
    int ste_opt_lineno;      /* lineno of last exec or import * */
    int ste_opt_col_offset;  /* offset of last exec or import * */
    struct symtable *ste_table;
} PySTEntryObject;

CPython uses the term namespace as a synonym for a code block in the context of symbol tables. So, we can say that a symbol table entry is a description of a namespace. The symbol table entries form an hierarchy of all namespaces in a program through the ste_children field, which is a list of child namespaces. We can explore this hierarchy using the standard symtable module:

# example3.py
def func(x):
    lc = [x+i for i in range(10)]
    return lc

>>> from symtable import symtable
>>> f = open('example3.py')
>>> st = symtable(f.read(), 'example3.py', 'exec') # module's symtable entry
>>> dir(st)
[..., 'get_children', 'get_id', 'get_identifiers', 'get_lineno', 'get_name',
 'get_symbols', 'get_type', 'has_children', 'is_nested', 'is_optimized', 'lookup']
>>> st.get_children()
[<Function SymbolTable for func in example3.py>]
>>> func_st = st.get_children()[0] # func's symtable entry
>>> func_st.get_children()
[<Function SymbolTable for listcomp in example3.py>]
>>> lc_st = func_st.get_children()[0] # list comprehension's symtable entry
>>> lc_st.get_symbols()
[<symbol '.0'>, <symbol 'i'>, <symbol 'x'>]
>>> x_sym = lc_st.get_symbols()[2]
>>> dir(x_sym)
[..., 'get_name', 'get_namespace', 'get_namespaces', 'is_annotated',
 'is_assigned', 'is_declared_global', 'is_free', 'is_global', 'is_imported',
 'is_local', 'is_namespace', 'is_nonlocal', 'is_parameter', 'is_referenced']
>>> x_sym.is_local(), x_sym.is_free()
(False, True)

This example shows that every code block has a corresponding symbol table entry. We've accidentally come across the strange .0 symbol inside the namespace of the list comprehension. This namespace doesn't contain the range symbol, which is also strange. This is because a list comprehension is implemented as an anonymous function and range(10) is passed to it as an argument. This argument is referred to as .0. What else does CPython hide from us?

The symbol table entries are constructed in two passes. During the first pass, CPython walks the AST and creates a symbol table entry for each code block it encounters. It also collects information that can be collected on the spot, such as whether a symbol is defined or used in the block. But some information is hard to deduce during the first pass. Consider the example:

def top():
    def nested():
        return x + 1
    x = 10
    ...

When constructing a symbol table entry for the nested() function, we cannot tell whether x is a global variable or a free variable, i.e. defined in the top() function, because we haven't seen an assignment yet.

CPython solves this problem by doing the second pass. At the start of the second pass it's already known where the symbols are defined and used. The missing information is filled by visiting recursively all the symbol table entries starting from the top. The symbols defined in the enclosing scope are passed down to the nested namespace, and the names of free variables in the enclosed scope are passed back.

The symbol table entries are managed using the symtable struct. It's used both to construct the symbol table entries and to access them during the compilation. Let's take a look at its definition:

struct symtable {
    PyObject *st_filename;          /* name of file being compiled,
                                       decoded from the filesystem encoding */
    struct _symtable_entry *st_cur; /* current symbol table entry */
    struct _symtable_entry *st_top; /* symbol table entry for module */
    PyObject *st_blocks;            /* dict: map AST node addresses
                                     *       to symbol table entries */
    PyObject *st_stack;             /* list: stack of namespace info */
    PyObject *st_global;            /* borrowed ref to st_top->ste_symbols */
    int st_nblocks;                 /* number of blocks used. kept for
                                       consistency with the corresponding
                                       compiler structure */
    PyObject *st_private;           /* name of current class or NULL */
    PyFutureFeatures *st_future;    /* module's future features that affect
                                       the symbol table */
    int recursion_depth;            /* current recursion depth */
    int recursion_limit;            /* recursion limit */
};

The most important fields to note are st_stack and st_blocks. The st_stack field is a stack of symbol table entries. During the first pass of the symbol table construction, CPython pushes an entry onto the stack when it enters the corresponding code block and pops an entry from the stack when it exits the corresponding code block. The st_blocks field is a dictionary that the compiler uses to get a symbol table entry for a given AST node. The st_cur and st_top fields are also important but their meanings should be obvious.

To learn more about symbol tables and their construction, I highly recommend you the articles by Eli Bendersky.

basic blocks

A symbol table helps us to translate statements involving variables like x = 1. But a new problem arises if we try to translate a more complex control-flow statement. Consider another cryptic piece of code:

if x == 0 or x > 17:
    y = True
else:
    y = False
...

The corresponding AST subtree has the following structure:

If(
  test=BoolOp(...),
  body=[...],
  orelse=[...]
)

And the compiler translates it to the following bytecode:

1           0 LOAD_NAME                0 (x)
            2 LOAD_CONST               0 (0)
            4 COMPARE_OP               2 (==)
            6 POP_JUMP_IF_TRUE        16
            8 LOAD_NAME                0 (x)
           10 LOAD_CONST               1 (17)
           12 COMPARE_OP               4 (>)
           14 POP_JUMP_IF_FALSE       22

2     >>   16 LOAD_CONST               2 (True)
           18 STORE_NAME               1 (y)
           20 JUMP_FORWARD             4 (to 26)

4     >>   22 LOAD_CONST               3 (False)
           24 STORE_NAME               1 (y)
5     >>   26 ...

The bytecode is linear. The instructions for the test node should come first, and the instructions for the body block should come before those for the orelse block. The problem with the control-flow statements is that they involve jumps, and a jump is often emitted before the instruction it points to. In our example, if the first test succeeds, we would like to jump to the first body instruction straight away, but we don't know where it should be yet. If the second test fails, we have to jump over the body block to the orelse block, but the position of the first orelse instruction will become known only after we translate the body block.

We could solve this problem if we move the instructions for each block into a separate data structure. Then, instead of specifying jump targets as concrete positions in the bytecode, we point to those data structures. Finally, when all blocks are translated and their sizes are know, we calculate arguments for jumps and assemble the blocks into a single sequence of instructions. And that's what the compiler does.

The blocks we're talking about are called basic blocks. They are not specific to CPython, though CPython's notion of a basic block differs from the conventional definition. According to the Dragon book, a basic block is a maximal sequence of instructions such that:

control may enter only the first instruction of the block; and
control will leave the block without halting or branching, except possibly at the last instruction.

CPython drops the second requirement. In other words, no instruction of a basic block except the first can be a target of a jump, but a basic block itself can contain jump instructions. To translate the AST from our example, the compiler creates four basic blocks:

instructions 0-14 for test
instructions 16-20 for body
instructions 22-24 for orelse; and
instructions 26-... for whatever comes after the if statement.

A basic block is represented by the basicblock_ struct that is defined as follows:

typedef struct basicblock_ {
    /* Each basicblock in a compilation unit is linked via b_list in the
       reverse order that the block are allocated.  b_list points to the next
       block, not to be confused with b_next, which is next by control flow. */
    struct basicblock_ *b_list;
    /* number of instructions used */
    int b_iused;
    /* length of instruction array (b_instr) */
    int b_ialloc;
    /* pointer to an array of instructions, initially NULL */
    struct instr *b_instr;
    /* If b_next is non-NULL, it is a pointer to the next
       block reached by normal control flow. */
    struct basicblock_ *b_next;
    /* b_seen is used to perform a DFS of basicblocks. */
    unsigned b_seen : 1;
    /* b_return is true if a RETURN_VALUE opcode is inserted. */
    unsigned b_return : 1;
    /* depth of stack upon entry of block, computed by stackdepth() */
    int b_startdepth;
    /* instruction offset for block, computed by assemble_jump_offsets() */
    int b_offset;
} basicblock;

And here's the definition of the instr struct:

struct instr {
    unsigned i_jabs : 1;
    unsigned i_jrel : 1;
    unsigned char i_opcode;
    int i_oparg;
    struct basicblock_ *i_target; /* target block (if jump instruction) */
    int i_lineno;
};

We can see that the basic blocks are connected not only by jump instructions but also through the b_list and b_next fields. The compiler uses b_list to access all allocated blocks, for example, to free the memory. The b_next field is of more interest to us right now. As the comment says, it points to the next block reached by the normal control flow, which means that it can be used to assemble blocks in the right order. Returning to our example once more, the test block points to the body block, the body block points to the orelse block and the orelse block points to the block after the if statement. Because basic blocks point to each other, they form a graph called a Control Flow Graph (CFG).

frame blocks

There is one more problem to solve: how to understand where to jump to when compiling statements like continue and break? The compiler solves this problem by introducing yet another type of block called frame block. There are different kinds of frame blocks. The WHILE_LOOP frame block, for example, points to two basic blocks: the body block and the block after the while statement. These basic blocks are used when compiling the continue and break statements respectively. Since frame blocks can nest, the compiler keeps track of them using stacks, one stack of frame blocks per code block. Frame blocks are also useful when dealing with statements such as try-except-finally, but we will not dwell on this now. Let's instead have a look at the definition of the fblockinfo struct:

enum fblocktype { WHILE_LOOP, FOR_LOOP, EXCEPT, FINALLY_TRY, FINALLY_END,
                  WITH, ASYNC_WITH, HANDLER_CLEANUP, POP_VALUE };

struct fblockinfo {
    enum fblocktype fb_type;
    basicblock *fb_block;
    /* (optional) type-specific exit or cleanup block */
    basicblock *fb_exit;
    /* (optional) additional information required for unwinding */
    void *fb_datum;
};

We've identified three important problems and we've seen how the compiler solves them. Now, let's put everything together to see how the compiler works from the beginning to the end.

compiler units, compiler and assembler

As we've already figured out, after building a symbol table, the compiler performs two more steps to convert an AST to a code object:

it creates a CFG of basic blocks; and
it assembles a CFG into a code object.

This two-step process is performed for each code block in a program. The compiler starts by building the module's CFG and ends by assembling the module's CFG into the module's code object. In between, it walks the AST by recursively calling the compiler_visit_* and compiler_* functions, where * denotes what is visited or compiled. For example, compiler_visit_stmt delegates the compilation of a given statement to the appropriate compiler_* function, and the compiler_if function knows how to compile the If AST node. If a node introduces new basic blocks, the compiler creates them. If a node begins a code block, the compiler creates a new compilation unit and enters it. A compilation unit is a data structure that captures the compilation state of the code block. It acts as a mutable prototype of the code object and points to a new CFG. The compiler assembles this CFG when it exits a node that began the current code block. The assembled code object is stored in the parent compilation unit. As always, I encourage you to look at the struct definition:

struct compiler_unit {
    PySTEntryObject *u_ste;

    PyObject *u_name;
    PyObject *u_qualname;  /* dot-separated qualified name (lazy) */
    int u_scope_type;

    /* The following fields are dicts that map objects to
       the index of them in co_XXX.      The index is used as
       the argument for opcodes that refer to those collections.
    */
    PyObject *u_consts;    /* all constants */
    PyObject *u_names;     /* all names */
    PyObject *u_varnames;  /* local variables */
    PyObject *u_cellvars;  /* cell variables */
    PyObject *u_freevars;  /* free variables */

    PyObject *u_private;        /* for private name mangling */

    Py_ssize_t u_argcount;        /* number of arguments for block */
    Py_ssize_t u_posonlyargcount;        /* number of positional only arguments for block */
    Py_ssize_t u_kwonlyargcount; /* number of keyword only arguments for block */
    /* Pointer to the most recently allocated block.  By following b_list
       members, you can reach all early allocated blocks. */
    basicblock *u_blocks;
    basicblock *u_curblock; /* pointer to current block */

    int u_nfblocks;
    struct fblockinfo u_fblock[CO_MAXBLOCKS];

    int u_firstlineno; /* the first lineno of the block */
    int u_lineno;          /* the lineno for the current stmt */
    int u_col_offset;      /* the offset of the current stmt */
};

Another data structure that is crucial for the compilation is the compiler struct, which represents the global state of the compilation. Here's its definition:

struct compiler {
    PyObject *c_filename;
    struct symtable *c_st;
    PyFutureFeatures *c_future; /* pointer to module's __future__ */
    PyCompilerFlags *c_flags;

    int c_optimize;              /* optimization level */
    int c_interactive;           /* true if in interactive mode */
    int c_nestlevel;
    int c_do_not_emit_bytecode;  /* The compiler won't emit any bytecode
                                    if this value is different from zero.
                                    This can be used to temporarily visit
                                    nodes without emitting bytecode to
                                    check only errors. */

    PyObject *c_const_cache;     /* Python dict holding all constants,
                                    including names tuple */
    struct compiler_unit *u; /* compiler state for current block */
    PyObject *c_stack;           /* Python list holding compiler_unit ptrs */
    PyArena *c_arena;            /* pointer to memory allocation arena */
};

And the comment preceding the definition that explains what the two most important fields are for:

The u pointer points to the current compilation unit, while units for enclosing blocks are stored in c_stack. The u and c_stack are managed by compiler_enter_scope() and compiler_exit_scope().

To assemble basic blocks into a code object, the compiler first has to fix the jump instructions by replacing pointers with positions in bytecode. On the one side, it's an easy task, since the sizes of all basic blocks are known. On the other side, the size of a basic block can change when we fix a jump. The current solution is to keep fixing jumps in a loop while the sizes change. Here's an honest comment from the source code on this solution:

This is an awful hack that could hurt performance, but on the bright side it should work until we come up with a better solution.

The rest is straightforward. The compiler iterates over basic blocks and emits the instructions. The progress is kept in the assembler struct:

struct assembler {
    PyObject *a_bytecode;  /* string containing bytecode */
    int a_offset;              /* offset into bytecode */
    int a_nblocks;             /* number of reachable blocks */
    basicblock **a_postorder; /* list of blocks in dfs postorder */
    PyObject *a_lnotab;    /* string containing lnotab */
    int a_lnotab_off;      /* offset into lnotab */
    int a_lineno;              /* last lineno of emitted instruction */
    int a_lineno_off;      /* bytecode offset of last lineno */
};

At this point, the current compilation unit and the assembler contain all the data needed to create a code object. Congratulations! We've done it! Almost.

peephole optimizer

The last step in the creation of the code object is to optimize the bytecode. This is a job of the peephole optimizer. Here's some types of optimizations it performs:

The statements like if True: ... and while True: ... generate a sequence of LOAD_CONST trueconst and POP_JUMP_IF_FALSE instructions. The peephole optimizer eliminates such instructions.
The statements like a, = b, lead to the bytecode that builds a tuple and then unpacks it. The peephole optimizer replaces it with a simple assignment.
The peephole optimizer removes unreachable instructions after RETURN.

Essentially, the peephole optimizer removes redundant instructions, thus making bytecode more compact. After the bytecode is optimized, the compiler creates the code object, and the VM is ready to execute it.

Summary

This was a long post, so it's probably a good idea to sum up what we've learned. The architecture of the CPython compiler follows a traditional design. Its two major parts are the frontend and the backend. The frontend is also referred to as the parser. Its job is to convert a source code to an AST. The parser gets tokens from the tokenizer, which is responsible for producing a stream of meaningful language units from the text. Historically, the parsing consisted of several steps, including the generation of a parse tree and the conversion of a parse tree to an AST. In CPython 3.9, the new parser was introduced. It's based on a parsing expression grammar and produces an AST straight away. The backend, also known paradoxically as the compiler, takes an AST and produces a code object. It does this by first building a symbol table and then by creating one more intermediate representation of a program called a control flow graph. The CFG is assembled into a single sequence of instructions, which is then optimized by the peephole optimizer. Eventually, the code object gets created.

At this point, we have enough knowledge to get acquainted with the CPython source code and understand some of the things it does. That's our plan for the next time.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Python behind the scenes #1: how the CPython VM works

2020-08-31T10:21:00+00:00

Introduction

Have you ever wondered what python does when you run one of your programs?

$ python script.py

This article opens a series which seeks to answer this very question. We'll dive into the internals of CPython, Python's most popular implementation. By doing so we'll understand the language itself at a deeper level. That is the primary goal of this series. If you're familiar with Python and comfortable reading C but have no much experience working with CPython's source code, there is a good chance you'll find this writing interesting.

What CPython is and why anyone would want to study it

Let's begin by stating some well-known facts. CPython is a Python interpreter written in C. It's one of the Python implementations, alongside with PyPy, Jython, IronPython and many others. CPython is distinguished in that it is original, most-maintained and the most popular one.

CPython implements Python, but what is Python? One may simply answer – Python is a programming language. The answer becomes much more nuanced when the same question is put properly: what defines what Python is? Python, unlike languages like C, doesn't have a formal specification. The thing that comes closest to it is the Python Language Reference which starts with the following words:

While I am trying to be as precise as possible, I chose to use English rather than formal specifications for everything except syntax and lexical analysis. This should make the document more understandable to the average reader, but will leave room for ambiguities. Consequently, if you were coming from Mars and tried to re-implement Python from this document alone, you might have to guess things and in fact you would probably end up implementing quite a different language. On the other hand, if you are using Python and wonder what the precise rules about a particular area of the language are, you should definitely be able to find them here.

So Python is not defined by its language reference only. It would be also wrong to say that Python is defined by its reference implementation, CPython, since there are some implementation details that are not a part of the language. The garbage collector that relies on a reference counting is one example. Since there is no single source of truth, we may say that Python is defined partly by the Python Language Reference and partly by its main implementation, CPython.

Such a reasoning may seem pedantic, but I think it is crucial to clarify the key role of the subject we're going to study. You might still wonder, though, why we should study it. Besides plain curiosity, I see the following reasons:

Having a full picture gives a deeper understanding of the language. It's much more easier to grasp some peculiarity of Python if you're aware of its implementation details.
Implementation details matter in practice. How objects are stored, how the garbage collector works and how multiple threads are coordinated are subjects of high importance when one wants to understand the applicability of the language and its limitations, estimate the performance or detect inefficiencies.
CPython provides Python/C API which allows to extend Python with C and embed Python inside C. To use this API effectively a programmer needs a good understanding of how CPython works.

What it takes to understand how CPython works

CPython was designed to be easy to maintain. A newcomer can certainly expect to be able to read the source code and understand what it does. However, it may take some time. By writing this series I hope to help you to shorten it.

How this series is laid out

I chose to take a top-down approach. In this part we'll explore the core concepts of the CPython virtual machine (VM). Next, we'll see how CPython compiles a program into something that the VM can execute. After that, we'll get familiar with the source code and step through the execution of a program studying main parts of the interpreter on the way. Eventually, we'll be able to pick out different aspects of the language one by one and see how they're implemented. This is by no way a strict plan but my approximate idea.

Note: In this post I'm referring to CPython 3.9. Some implementation details will certainly change as CPython evolves. I'll try to keep track of important changes and add update notes.

The big picture

An execution of a Python program roughly consists of three stages:

Initialization
Compilation
Interpretation

During the initialization stage, CPython initializes data structures required to run Python. It also prepares such things as built-in types, configures and loads built-in modules, sets up the import system and does many other things. This is a very important stage that is often overlooked by the CPython's explorers because of its service nature.

Next comes the compilation stage. CPython is an interpreter, not a compiler in a sense that it doesn't produce machine code. Interpreters, however, usually translate source code into some intermediate representation before executing it. So does CPython. This translation phase does the same things a typical compiler does: parses a source code and builds an AST (Abstract Syntax Tree), generates bytecode from an AST and even performs some bytecode optimizations.

Before looking at the next stage, we need to understand what bytecode is. Bytecode is a series of instructions. Each instruction consists of two bytes: one for an opcode and one for an argument. Consider an example:

def g(x):
    return x + 3

CPython translates the body of the function g() to the following sequence of bytes: [124, 0, 100, 1, 23, 0, 83, 0]. If we run the standard dis module to disassemble it, here's what we'll get:

$ python -m dis example1.py
...
2           0 LOAD_FAST            0 (x)
            2 LOAD_CONST           1 (3)
            4 BINARY_ADD
            6 RETURN_VALUE

The LOAD_FAST opcode corresponds to the byte 124 and has the argument 0. The LOAD_CONST opcode corresponds to the byte 100 and has the argument 1. The BINARY_ADD and RETURN_VALUE instructions are always encoded as (23, 0) and (83, 0) respectively since they don't need an argument.

At the heart of CPython is a virtual machine that executes bytecode. By looking at the previous example you might guess how it works. CPython's VM is stack-based. It means that it executes instructions using the stack to store and retrieve data. The LOAD_FAST instruction pushes a local variable onto the stack. LOAD_CONST pushes a constant. BINARY_ADD pops two objects from the stack, adds them up and pushes the result back. Finally, RETURN_VALUE pops whatever is on the stack and returns the result to its caller.

The bytecode execution happens in a giant evaluation loop that runs while there are instructions to execute. It stops to yield a value or if an error occurred.

Such a brief overview gives rise to a lot of questions:

What do the arguments to the LOAD_FAST and LOAD_CONST opcodes mean? Are they indices? What do they index?
Does the VM place values or references to the objects on the stack?
How does CPython know that x is a local variable?
What if an argument is too big to fit into a single byte?
Is the instruction for adding two numbers the same as for concatenating two strings? If yes, then how does the VM differentiate between these operations?

In order to answer these and other intriguing questions we need to look at the core concepts of the CPython VM.

Code objects, function objects, frames

code object

We saw what the bytecode of a simple function looks like. But a typical Python program is more complicated. How does the VM execute a module that contains function definitions and makes function calls?

Consider the program:

def f(x):
    return x + 1

print(f(1))

What does its bytecode look like? To answer this question, let's analyze what the program does. It defines the function f(), calls f() with 1 as an argument and prints the result of the call. Whatever the function f() does, it's not a part of the module's bytecode. We can assure ourselves by running the disassembler.

$ python -m dis example2.py

1           0 LOAD_CONST               0 (<code object f at 0x10bffd1e0, file "example.py", line 1>)
            2 LOAD_CONST               1 ('f')
            4 MAKE_FUNCTION            0
            6 STORE_NAME               0 (f)

4           8 LOAD_NAME                1 (print)
           10 LOAD_NAME                0 (f)
           12 LOAD_CONST               2 (1)
           14 CALL_FUNCTION            1
           16 CALL_FUNCTION            1
           18 POP_TOP
           20 LOAD_CONST               3 (None)
           22 RETURN_VALUE
...

On line 1 we define the function f() by making the function from something called code object and binding the name f to it. We don't see the bytecode of the function f() that returns an incremented argument.

The pieces of code that are executed as a single unit like a module or a function body are called code blocks. CPython stores information about what a code block does in a structure called a code object. It contains the bytecode and such things as lists of names of variables used within the block. To run a module or to call a function means to start evaluating a corresponding code object.

function object

A function, however, is not merely a code object. It must include additional information such as the function name, docstring, default arguments and values of variables defined in the enclosing scope. This information, together with a code object, is stored inside a function object. The MAKE_FUNCTION instruction is used to create it. The definition of the function object structure in the CPython source code is preceded by the following comment:

Function objects and code objects should not be confused with each other:

Function objects are created by the execution of the 'def' statement. They reference a code object in their __code__ attribute, which is a purely syntactic object, i.e. nothing more than a compiled version of some source code lines. There is one code object per source code "fragment", but each code object can be referenced by zero or many function objects depending only on how many times the 'def' statement in the source was executed so far.

How can it be that several function objects reference a single code object? Here is an example:

def make_add_x(x):
    def add_x(y):
        return x + y
    return add_x

add_4 = make_add_x(4)
add_5 = make_add_x(5)

The bytecode of the make_add_x() function contains the MAKE_FUNCTION instruction. The functions add_4() and add_5() are the result of calling this instruction with the same code object as an argument. But there is one argument that differs – the value of x. Each function gets its own by the mechanism of cell variables that allows us to create closures like add_4() and add_5().

Before we move to the next concept, take a look at the definitions of the code and function objects to get a better idea of what they are.

struct PyCodeObject {
    PyObject_HEAD
    int co_argcount;            /* #arguments, except *args */
    int co_posonlyargcount;     /* #positional only arguments */
    int co_kwonlyargcount;      /* #keyword only arguments */
    int co_nlocals;             /* #local variables */
    int co_stacksize;           /* #entries needed for evaluation stack */
    int co_flags;               /* CO_..., see below */
    int co_firstlineno;         /* first source line number */
    PyObject *co_code;          /* instruction opcodes */
    PyObject *co_consts;        /* list (constants used) */
    PyObject *co_names;         /* list of strings (names used) */
    PyObject *co_varnames;      /* tuple of strings (local variable names) */
    PyObject *co_freevars;      /* tuple of strings (free variable names) */
    PyObject *co_cellvars;      /* tuple of strings (cell variable names) */

    Py_ssize_t *co_cell2arg;    /* Maps cell vars which are arguments. */
    PyObject *co_filename;      /* unicode (where it was loaded from) */
    PyObject *co_name;          /* unicode (name, for reference) */
        /* ... more members ... */
};

typedef struct {
    PyObject_HEAD
    PyObject *func_code;        /* A code object, the __code__ attribute */
    PyObject *func_globals;     /* A dictionary (other mappings won't do) */
    PyObject *func_defaults;    /* NULL or a tuple */
    PyObject *func_kwdefaults;  /* NULL or a dict */
    PyObject *func_closure;     /* NULL or a tuple of cell objects */
    PyObject *func_doc;         /* The __doc__ attribute, can be anything */
    PyObject *func_name;        /* The __name__ attribute, a string object */
    PyObject *func_dict;        /* The __dict__ attribute, a dict or NULL */
    PyObject *func_weakreflist; /* List of weak references */
    PyObject *func_module;      /* The __module__ attribute, can be anything */
    PyObject *func_annotations; /* Annotations, a dict or NULL */
    PyObject *func_qualname;    /* The qualified name */
    vectorcallfunc vectorcall;
} PyFunctionObject;

frame object

When the VM executes a code object, it has to keep track of the values of variables and the constantly changing value stack. It also needs to remember where it stopped executing the current code object to execute another and where to go on return. CPython stores this information inside a frame object, or simply a frame. A frame provides a state in which a code object can be executed. Since we're getting more accustomed with the source code, I leave the definition of the frame object here as well:

struct _frame {
    PyObject_VAR_HEAD
    struct _frame *f_back;      /* previous frame, or NULL */
    PyCodeObject *f_code;       /* code segment */
    PyObject *f_builtins;       /* builtin symbol table (PyDictObject) */
    PyObject *f_globals;        /* global symbol table (PyDictObject) */
    PyObject *f_locals;         /* local symbol table (any mapping) */
    PyObject **f_valuestack;    /* points after the last local */

    PyObject **f_stacktop;          /* Next free slot in f_valuestack.  ... */
    PyObject *f_trace;          /* Trace function */
    char f_trace_lines;         /* Emit per-line trace events? */
    char f_trace_opcodes;       /* Emit per-opcode trace events? */

    /* Borrowed reference to a generator, or NULL */
    PyObject *f_gen;

    int f_lasti;                /* Last instruction if called */
    /* ... */
    int f_lineno;               /* Current line number */
    int f_iblock;               /* index in f_blockstack */
    char f_executing;           /* whether the frame is still executing */
    PyTryBlock f_blockstack[CO_MAXBLOCKS]; /* for try and loop blocks */
    PyObject *f_localsplus[1];  /* locals+stack, dynamically sized */
};

The first frame is created to execute the module's code object. CPython creates a new frame whenever it needs to execute another code object. Each frame has a reference to the previous frame. Thus, frames form a stack of frames, also known as the call stack, with the current frame sitting on top. When a function is called, a new frame is pushed onto the stack. On return from the currently executing frame, CPython continues the execution of the previous frame by remembering its last processed instruction. In some sense, the CPython VM does nothing but constructs and executes the frames. However, as we'll soon see, this summary, to put it mildly, hides some details.

Threads, interpreters, runtime

We've already looked at the three important concepts:

a code object
a function object; and
a frame object.

CPython has three more:

a thread state
an interpreter state; and
a runtime state.

thread state

A thread state is a data structure that contains thread-specific data including the call stack, the exception state and the debugging settings. It should not be confused with an OS thread. They're closely connected, though. Consider what happens when you use the standard threading module to run a function in a separate thread:

from threading import Thread

def f():
    """Perform an I/O-bound task"""
    pass

t = Thread(target=f)
t.start()
t.join()

t.start() actually creates a new OS thread by calling the OS function (pthread_create() on UNIX-like systems and _beginthreadex() on Windows). The newly created thread invokes the function from the _thread module that is responsible for calling the target. This function receives not only the target and the target's arguments but also a new thread state to be used within a new OS thread. An OS thread enters the evaluation loop with its own thread state, thus always having it at hand.

We may remember here the famous GIL (Global Interpreter Lock) that prevents multiple threads to be in the evaluation loop at the same time. The major reason for that is to protect the state of CPython from corruption without introducing more fine-grained locks. The Python/C API Reference explains the GIL clearly:

The Python interpreter is not fully thread-safe. In order to support multi-threaded Python programs, there’s a global lock, called the global interpreter lock or GIL, that must be held by the current thread before it can safely access Python objects. Without the lock, even the simplest operations could cause problems in a multi-threaded program: for example, when two threads simultaneously increment the reference count of the same object, the reference count could end up being incremented only once instead of twice.

To manage multiple threads, there needs to be a higher-level data structure than a thread state.

interpreter and runtime states

In fact, there are two of them: an interpreter state and the runtime state. The need for both may not seem immediately obvious. However, an execution of any program has at least one instance of each and there are good reasons for that.

An interpreter state is a group of threads along with the data specific to this group. Threads share such things as loaded modules (sys.modules), builtins (builtins.__dict__) and the import system (importlib).

The runtime state is a global variable. It stores data specific to a process. This includes the state of CPython (e.g. is it initialized or not?) and the GIL mechanism.

Usually, all threads of a process belong to the same interpreter. There are, however, rare cases when one may want to create a subinterpreter to isolate a group of threads. mod_wsgi, which uses distinct interpreters to run WSGI applications, is one example. The most obvious effect of isolation is that each group of threads gets its own version of all modules including __main__, which is a global namespace.

CPython doesn't provide an easy way to create new interpreters analogous to the threading module. This feature is supported only via Python/C API, but this may change someday.

Architecture summary

Let's make a quick summary of the CPython's architecture to see how everything fits together. The interpreter can be viewed as a layered structure. The following sums up what the layers are:

Runtime: the global state of a process; this includes the GIL and the memory allocation mechanism.
Interpreter: a group of threads and some data they share such as imported modules.
Thread: data specific to a single OS thread; this includes the call stack.
Frame: an element of the call stack; a frame contains a code object and provides a state to execute it.
Evaluation loop: a place where a frame object gets executed.

The layers are represented by the corresponding data structures, which we've already seen. In some cases they are not equivalent, though. For example, the mechanism of memory allocation is implemented using global variables. It's not a part of the runtime state but certainly a part of the runtime layer.

Conclusion

In this part we've outlined what python does to execute a Python program. We've seen that it works in three stages:

initializes CPython
compiles the source code to the module's code object; and
executes the bytecode of the code object.

The part of the interpreter that is responsible for the bytecode execution is called a virtual machine. The CPython VM has several particularly important concepts: code objects, frame objects, thread states, interpreter states and the runtime. These data structures form the core of the CPython's architecture.

We haven't covered a lot of things. We avoided digging into the source code. The initialization and compilation stages were completely out of our scope. Instead, we started with the broad overview of the VM. In this way, I think, we can better see the responsibilities of each stage. Now we know what CPython compiles source code to – to the code object. Next time we'll see how it does that.

If you have any questions, comments or suggestions, feel free to contact me at victor@tenthousandmeters.com

Update from September 4, 2020: I've made a list of resources that I've used to learn about CPython internals.

Ten thousand meters

I made a website with best learning resources on IT topics

Python behind the scenes #13: the GIL and its effects on Python multithreading

OS threads, Python threads, and the GIL

The effects of the GIL

The convoy effect

Fixing the convoy effect

A proper GIL

Can't we remove the GIL?

The future of the GIL and Python concurrency

P.S.

The implementation details of the GIL *

Python behind the scenes #12: how async/await works in Python

It's all about concurrency

A sequential server

OS threads

I/O multiplexing and event loops

Generator functions and generators

Generators as coroutines

yield from

async/await

How generators and coroutines are implemented *

asyncio

Conclusion

P.S.

Python behind the scenes #11: how the Python import system works

Our plan

Modules and module objects

Different kinds of modules

Submodules and packages

Regular packages

Namespace packages

Importing from modules

Wildcard imports

Relative imports

Running programs as modules

Desugaring the import statement

Simple imports

Importing submodules

from <> import <>

from <> import *

Relative imports

Inside __import__()

Resolving relative names

The import process

Finders and loaders

BuiltinImporter and FrozenImporter

PathFinder

FileFinder

SourceFileLoader

Summary of the import process

What's in sys.path?

Conclusion

Python behind the scenes #10: how Python dictionaries work

What is a dictionary

Designing a simple hash table

Hash functions

Non-cryptographic hash functions

Universal hashing

SipHash

Collision resolution methods

Python dictionaries

Overview

Compact dictionaries

Shared keys

String interning

A note on sets

Conclusion

Python behind the scenes #9: how Python strings work

The scope of this post

The essence of text encoding

The road to Unicode

Unicode basics

A brief history of Python strings

Meet modern Python strings

How other Python implementations represent strings

How strings work in other languages

C

Go

Rust

Inside import()

slots