title: Fast dataframe normalization in Python/Pandas
body: <p>In Python/Pandas, I am trying to normalize all numeric columns of a dataframe to counts per million using the following code:</p>
  
  <pre><code>import pandas as pd
  def normalize(df):
      numerics = ['int16', 'int32', 'int64', 'float16', 'float32', 'float64']
      numeric_cols = [col for col in df.columns if df[col].dtype in numerics]
      sums = df.sum(axis=0, skipna = True)
      for col in numeric_cols:
          df.insert(df.columns.get_loc(col)+1, col+'_norm', ((df[col]/sums[col])*1000000)+1)
      return df
  dataset = pd.read_csv(csv_file, index_col=['Guide1', 'Guide2'], sep=',')
  dataset = normalize(dataset)
  </code></pre>
  
  <p>This is a short example input:</p>
  
  <pre><code>Guide1,Guide2,Gene1,Gene2,NHT1,NHT2,hart_essential_1,hart_essential_2,lib,RPE1_n1,RPE1_n2,RPE1_n3
  Gene_1-KO-3,Non-Human-Target-150-KO-26,Gene_1,Non-Human-Target-150,False,True,False,False,12426.0,10634.0,8701.0,8084.0
  Gene_2-KO-3,Non-Human-Target-150-KO-26,Gene_2,Non-Human-Target-150,False,True,False,False,12300.0,12383.0,6252.0,5388.0
  Gene_1-KO-3,Gene_4-KO-2,Gene_1,Gene_4,False,False,False,False,11685.0,10006.0,10621.0,7002.0
  Gene_1-KO-3,Gene_5-KO-2,Gene_1,Gene_5,False,False,False,False,11347.0,6726.0,3927.0,3943.0
  Gene_1-KO-3,Gene_6-KO-1,Gene_1,Gene_6,False,False,False,False,11250.0,12469.0,3552.0,3334.0    
  </code></pre>
  
  <p>The code works as intended and normalizes all numeric columns of that dataframe, but it seems to work very slow, also with other normalization methods. I am regularly working with dataframes that contain up to several millions of rows. For ~1 million rows and only 4 numeric columns, normalization takes almost 10 minutes on my mobile computer with an Intel Core i7-6600U @ 2.6GHz, although I precompute the sums for every column.
  Is there a way to speed this up, e.g., parallelization or some neat Pandas trick?</p>
  
  <p>Any help and/or pointers are much appreciated!</p>