Embedding Python and Multithreading

Just found this post on comp.lang.python about an article detailing the hoops one needs to jump through when it comes to embedding python and you have a multithreaded application.

Note that all that still means that you can't have one Python interpreter running in multiple threads. If I understood things as they are, Python currently needs a 1:1 relationship between Python interpreters and threads. That is, you can have more than one Python interpreter per process but only one per thread. You can have more than one interpreter per process if you really want, but these threads then compete against the same GIL.

This is due to the fact that the standard CPython implementation uses a so-called global interpreter lock (GIL) to serialize access to the internal Python API. There's one GIL per interpreter.

Also note that there's hidden pitfall here: The interpreter periodically (every n byte code instructions) yields execution and releases the GIL. That means if other threads pend on the GIL they might get runnable, which is perhaps not what one might expect.

Calling C extension is another common way to release the GIL, such that a expensive calculation in does not block other threads.