title: Association analysis in Python - memory error on groupby
body: <p>I am doing an association analysis in a problem like 'Market Basket' (find products that are sold together). I have a dataset of 12 months with 150Mb of data, approx 7mi registers (Invoices and the products, among other data). I think it doesn't worth going to Hadoop.. I'm following this tutorial (<a href="http://pbpython.com/market-basket-analysis.html" rel="nofollow noreferrer">http://pbpython.com/market-basket-analysis.html</a>), but I'm getting Memory Error in the df.unstack() part.</p>
  
  <pre><code>data = pd.DataFrame()
  list_ = []
  for date in files:
      df = pd.read_csv('VND_%s.csv' % date,
                      sep=';',
                      skiprows=0,
                      names=names,
                      index_col=None)
      list_.append(df)
  data = pd.concat(list_)
  
  basket = (data.groupby(['INVOICE', 'MATERIAL'])['TOT_QUANTITY']
            .sum().unstack().reset_index().fillna(0))
  
  def encode_units(x):
      if x &lt;= 0:
          return 0
      if x &gt;= 1:
          return 1
  
  basket_sets = basket.applymap(encode_units)
  frequent_itemsets = apriori(basket_sets, min_support=0.05, use_colnames=True)
  rules = association_rules(frequent_itemsets, metric="lift", min_threshold=1)
  
  print(rules.head())
  </code></pre>
  
  <p>And if you find something that could be improved please tell me. I'd appreciate.</p>