<p>When I run locally, in my laptop (using python 3.10 and pandas 1.3.5), the following code, I get 0.031s approximately (ball parking it):</p>
<pre><code>profile_data = (
                profiles_df[data_cols]
                .loc[profile_ids]
                .rename(columns=new_cols)
                .to_dict(&quot;records&quot;)
            ) 
</code></pre>
<p>where data_cols and new_cols are two lists of strings, and profiles_df is a dataframe with mostly string data.
However, when I run in it in a pod, using the same python and pandas versions, I get it run in 0.1s approx. The pod has still ample secondary memory (a few GBs) and never reaches its limit, nor does it reach the CPU limits (1 out of 1.5)</p>
<ol>
<li>Is there a way to optimize the above code?</li>
<li>What could be causing this difference in performance?</li>
</ol>

<p><code>df.rename()</code> introduces some overhead, but you can skip that step by constructing the result directly with the new column names:</p>
<pre class="lang-py prettyprint-override"><code>profile_data = [{new_cols[col]: profiles_df.loc[ix, col] 
                 for col in new_cols} 
                for ix in profile_ids]
</code></pre>
<p>I do not know the answer to your second question.</p>
