<p>If a (close) approximation of the median is OK for your purposes, you should consider computing a <em>median of medians</em>, which is a divide and conquer strategy that can be executed in parallel. In principle, <em>MoM</em> has <code>O(n)</code> complexity for serial execution, approaching <code>O(1)</code> for parallel execution on massively parallel systems.</p>

<p>See <a href="https://en.wikipedia.org/wiki/Median_of_medians" rel="nofollow noreferrer">this Wiki entry</a> for a description and pseudo-code. See also <a href="https://stackoverflow.com/questions/10806303/python-implementation-of-median-of-medians-algorithm">this question on Stack Overflow</a> and discussion of the code, and <a href="https://arxiv.org/abs/1104.2732v1" rel="nofollow noreferrer">this ArXiv paper</a> for a GPU implementation.</p>
How to make the calculation of the median faster in Python
<p>I would like to calculate the median line by line in a dataframe of more than 500,000 rows. For the moment I'm using <code>np.median</code> because numpy is optimized to run on a single core.  It's still very slow and I'd like to find a way to parallel the calculation</p>

<p>Specifically, I have <code>N</code> tables of size <code>13 x 500,000</code> and for each table I want to add the columns Q1, Q3 and median so that for each row the median column contains the median of the row. So I have to calculate <code>N * 500,000</code> median values.</p>

<p>I tried with <code>numexpr</code> but it doesn't seem possible.</p>

<p><strong>EDIT :</strong>
In fact I also need Q1 and Q3 so I can't use the statistics module which doesn't allow to calculate quartiles. Here is how I calculate the median for the moment </p>

<pre><code>    q = np.transpose(np.percentile(data[row_array], [25,50,75], axis = 1))
    data['Q1_' + family] = q[:,0]
    data['MEDIAN_' + family] = q[:,1]
    data['Q3_' + family] = q[:,2]
</code></pre>

<p><strong>EDIT 2</strong> I solved my problem by using the median of median algorithm as proposed below </p>