title: Python - Pandas dataframe concatenation Balloons in memory
body: <p>My original dataset is a large list of JSON objects for an adverse reaction to a drug(s). In each JSON object we can have several drugs that give an adverse reaction in the form of rxcui ids.</p>
  
  <p>I have taken the list of JSON objects and extracted the important data we need such as if the person died, and the rxcui and flattened them to a JSON object that is at max 2 levels deep. We would have something like this:</p>
  
  <pre><code>{
    "serious": 1,
    "drug": [
      "DrugA",
      "DrugB",
      "DrugC"
    ],
    "rxcui": [
      100,
      200,
      300
    ]
  }
  </code></pre>
  
  <p>As we can I have to figure out how I can make this in to an array I can feed to a ML algorithm. So My idea is to use one-hot encoding.</p>
  
  <p>This is why I am using the countVectorizer so I can vectorize all of these sub lists</p>
  
  <p>I am trying to concatenate several pandas dataframes (some are sparse dataframes and some are regular) which are onehot encodings of some data. I have checked all of the files (I also pickled them on to the hard disk) and none are larger than 81MB. But as soon as I start to concatenating them they blowup to over 29 GB. How is that possible?</p>
  
  <p>All of my df look like this:</p>
  
  <pre><code>Label0  Label1  Label2  Label3...  Label999
  1       1       0       0     ...  0
  1       1       0       1     ...  1
  .
  .
  .
  </code></pre>
  
  <p>I run concat like this:</p>
  
  <pre><code>x = pandas.concat([x, drugcharacterization, occurcountry, reactionmeddrapt, reactionmeddraversionpt, reactionoutcome, rxcui],axis=1, copy=False)
  </code></pre>
  
  <p>I can also fit all of the sub dataframes I am trying to concatenate easily in memory. Any reason why it blows up once I do this?</p>
  
  <p><strong>edit:</strong>
  Here is how I am getting my dataframes. As we can see I cant create a sparse matrix of one of them, it gives me an error:</p>
  
  <blockquote>
  <pre><code>raise ValueError("empty vocabulary; perhaps the documents only contain stop words")
  </code></pre>
  </blockquote>
  
  <pre><code>import pandas
  from sklearn.feature_extraction.text import CountVectorizer    
  rr = pandas.DataFrame()
  for col in categorical_labels:
      print col
      try:
          vect = CountVectorizer()
          X = vect.fit_transform(z[col].astype(str).map(lambda x: ' '.join(x) if isinstance(x, list) else x))
          r = pandas.SparseDataFrame(X, columns=vect.get_feature_names(), index=z.index, default_fill_value=0).add_prefix(col + ' = ')
          r.to_pickle(col + '_subarr.pkl')
      except:
          r = z[col].astype(str).str.strip('[]').str.get_dummies(', ').add_prefix(col + ' = ')
          r.to_pickle(col + '_subarr.pkl')
  
      rr = pandas.concat([rr,r], axis=1)
  </code></pre>
  
  <p><strong>and here are their indecies:</strong></p>
  
  <pre><code>drugcharacterization.index
  Out[13]: RangeIndex(start=0, stop=234372, step=1)
  
  occurcountry.index
  Out[14]: RangeIndex(start=0, stop=234372, step=1)
  
  reactionmeddrapt.index
  Out[15]: RangeIndex(start=0, stop=234372, step=1)
  
  reactionmeddraversionpt.index
  Out[16]: RangeIndex(start=0, stop=234372, step=1)
  
  reactionoutcome.index
  Out[17]: RangeIndex(start=0, stop=234372, step=1)
  
  rxcui.index
  Out[18]: RangeIndex(start=0, stop=234372, step=1)
  </code></pre>