Take the 2-minute tour ×

I am writing a C extension for Python, which should release the Global Interpreter Lock while it operates on data. I think I have understood the mechanism of the GIL fairly well, but one question remains: Can I access data in a Python object while the thread does not own the GIL? For example, I want to read data from a (big) NumPy array in the C function while I still want to allow other threads to do other things on the other CPU cores. The C function should

  • release the GIL with Py_BEGIN_ALLOW_THREADS
  • read and work on the data without using Python functions
  • even write data to previously constructed NumPy arrays
  • reacquire the GIL with Py_END_ALLOW_THREADS

Is this safe? Of course, other threads are not supposed to change the variables which the C function uses. But maybe there is one hidden source for errors: could the Python interpreter move an object, eg. by some sort of garbage collection, while the C function works on it in a separate thread?

To illustrate the question with a minimal example, consider the (minimal but complete) code below. Compile it (on Linux) with

gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -fPIC -I/usr/lib/pymodules/python2.7/numpy/core/include -I/usr/include/python2.7 -c gilexample.c -o gilexample.o
gcc -pthread -shared gilexample.o -o gilexample.so

and test it in Python with

import gilexample

Is the code between Py_BEGIN_ALLOW_THREADS and Py_END_ALLOW_THREADS safe? It accesses the contents of a Python object, and I do not want to duplicate the (possibly large) array in memory.

#include <Python.h>
#include <numpy/arrayobject.h>

// The relevant function
static PyObject * sum(PyObject * const self, PyObject * const args) {
  PyObject * X;
  PyArg_ParseTuple(args, "O", &X);
  PyObject const * const X_double = PyArray_FROM_OTF(X, NPY_DOUBLE, NPY_ALIGNED);
  npy_intp const size = PyArray_SIZE(X_double);
  double * const data = (double *) PyArray_DATA(X_double);
  double sum = 0;


  npy_intp i;
    sum += data[i];


  return PyFloat_FromDouble(sum);

// Python interface code
// List the C methods that this extension provides.
static PyMethodDef gilexampleMethods[] = {
  {"sum", sum, METH_VARARGS},
  {NULL, NULL, 0, NULL}     /* Sentinel - marks the end of this structure */

// Tell Python about these methods.
PyMODINIT_FUNC initgilexample(void)  {
  (void) Py_InitModule("gilexample", gilexampleMethods);
  import_array();  // Must be present for NumPy.
share|improve this question
I did things like this in the past, and I found that the easiest way to make this work is using ctypes to call your C functions. Give your C functions a pure C interface without any reference to Python or NumPy, and write trivial wrappers in Python that accept NumPy arrays and translate them to the appropriate C parameters. I gave an example on how to do this in this answer. –  Sven Marnach Jan 11 '12 at 21:19
@Sven: Do you know whether ctypes makes a working copy of the array in memory? (1) Yes. In this case, I don't want it since I am dealing with large input arrays. (2) No. Then my question whether you can lift the GIL remains valid. However, in case (2), the ctypes behavior would be a hint that lifting the GIL is probably not problematic, also in code which does not use ctypes. Does anyone know whether (1) or (2) holds? –  Daniel Jan 11 '12 at 22:06
No, ctypes does not make a copy of the arrays. And it releases the GIL for you, so you don't have to care about it. The advantage of using ctypes is the simplicity -- you have to extract all necessary meta-information from the NumPy array while still in Python, and the GIL is released at just the right moment. I used this approach for concurrently accessing the data in NumPy arrays from multiple threads. (Note that concurrent write access to the same memory is never save.) –  Sven Marnach Jan 11 '12 at 23:01
Thanks, Sven, this is the most helpful comment for me so far, even if it digresses a little from the original question to ctypes. If I could mark a comment as the ‘accepted answer’, this would be it. I'd like to know whether ctypes does more internally to protect the memory than what the explicit little example above does, but for a practical solution, it's sufficient for now to know that there exists a good way with ctypes. –  Daniel Jan 12 '12 at 0:32
I wrote this as a comment rather than an answer because it digresses. ctypes doesn't do anything to protect the memory. The code for passing the NumPy array as a parameter to a C function is actually in NumPy, not in ctypes -- ctypes is part of the standard library and isn't aware of NumPy. All the code does is extract the pointer to the underlying data and pass it on to ctypes, so ctypes doesn't even know the size of the data. I'll write an more to-the-point answer shortly. –  Sven Marnach Jan 12 '12 at 14:00
add comment

2 Answers

Is this safe?

Strictly, no. I think you should move the calls to PyArray_SIZE and PyArray_DATA outside the GIL-less block; if you do that, you'll be operating on C data only. You might also want to increment the reference count on the object before going into the GIL-less block and decrement it afterwards.

After your edits, it should be safe. Don't forget to decrement the reference count afterwards.

share|improve this answer
Thanks for the comment about PyArray_SIZE and PyArray_DATA. That was a mistake. I edited my question and moved the commands outside the block where the GIL is released. –  Daniel Jan 11 '12 at 18:56
Now: is the code in its revised version safe? Can you elaborate why incrementing the reference count would change things? –  Daniel Jan 11 '12 at 18:58
@Daniel: the reference count might be needed (though I'm not entirely sure in this case) because no other thread must deallocate the array. –  larsmans Jan 11 '12 at 19:12
I checked the reference counts: PyArray_FROM_OTF increases the reference count already, so there is no need to do it manually. However, this brings to my attention that a Py_DECREF is in order at the end of my example code; otherwise there would be a memory leak. Thanks, larsmans! I corrected my original posting. So, as an answer to your original comment: if I incremented the reference counter manually, this would be unnecessary and would not change anything, since the reference counter is at least 1 and thus the object is safe from deallocation. –  Daniel Jan 11 '12 at 19:33
add comment


No you cannot.

share|improve this answer
Does that mean that I always have to copy the contents of a NumPy array, even if I only want to read the data in a thread? I hope that there is a way around it! Any suggestion? –  Daniel Jan 11 '12 at 18:52
add comment

Your Answer


