title: Convert Array of DataFrames to Single DataFrame
body: <p><strong>TLDR:</strong> I don't know how to take an array of DataFrames and build a MultiIndex around it.</p>
  
  <p><strong>TLDR 2:</strong> From my research, it is recommended to deal with the return result than to try to work off of a global/single/shared DataFrame in <code>Pool().Map()</code>. If somebody has a way to share it, I'm all for it.</p>
  
  <hr>
  
  <p>I am trying to merge an array of <a href="http://pandas.pydata.org/pandas-docs/stable/advanced.html" rel="nofollow">MultiIndex</a> Pandas DataFrames that have been returned by a <a href="https://docs.python.org/2/library/multiprocessing.html" rel="nofollow">Pool()</a>.<a href="https://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map" rel="nofollow">Map()</a> function.</p>
  
  <pre><code>p = Pool()
  results = p.map(run_experiment, experiment_collection)
  </code></pre>
  
  <p><code>Pool().map()</code> returns as an array. Let's say the process returns the array with 1000 <code>DataFrame</code> with a first index of <code>[0:5]</code> and second <code>[0:50]</code>.</p>
  
  <p>What I want is to create a final output that is a single DataFrame that would separate each experiment so <code>[0:1000]</code> / <code>[0:5]</code> /<code>[0:50]</code>.</p>
  
  <hr>
  
  <p>I know how to create the MultiIndex using <code>np.zeros_like</code> and then fill the DataFrame, but I don't know how to take an array of DataFrames and build a MultiIndex around it.</p>
  
  <pre><code>rounds = range(0,1000)
  levels = [... some set of levels ...]
  labels = [... some set of labels ...]
  iterables = [rounds, labels, levels]
  
  names = ['round', 'label', 'values']
  index = pd.MultiIndex.from_product(iterables, names=names)
  
  index_names = [... some set of index names...]
  empty_df = pd.DataFrame(
                np.zeros_like(np.random.randn(5, 50000)),
                index=index_names,
                columns=index
               )
               .sort_index()
               .sort_index(axis=1)
  </code></pre>
  
  <hr>
  
  <p>In my first example, <code>results</code> is an array of MultiIndex DataFrame of <code>levels</code> / <code>labels</code>. What I am trying to do is create a final DataFrame that replaces the top level (an array holding all these DataFrames) as just a new index.</p>
  
  <p>When I try</p>
  
  <pre><code>p = Pool()
  results = pd.DataFrame(
                p.map(run_experiment, experiment_collection),
                index=index_names,
                columns=index
               )
               .sort_index()
               .sort_index(axis=1)
  </code></pre>
  
  <p>I am getting <code>ValueError: Shape of passed values is (1, 1000), indices imply (shape of intended index)</code>, which makes sense because it is an array of 1000 DataFrames.</p>
  
  <p>If I concatenate (which feels like the better way to go)</p>
  
  <pre><code>results = pd.concat(p.map(run_experiment, experiment_collection))
  </code></pre>
  
  <p>I get a DataFrame with <code>levels</code> / <code>labels</code>, but no <code>round</code>.</p>
  
  <pre><code>iterables = [rounds, labels, levels]
  </code></pre>
  
  <hr>
  
  <pre><code>pd.concat(objs, axis=0, join='outer', join_axes=None, ignore_index=False,
         keys=None, levels=None, names=None, verify_integrity=False)
  </code></pre>
  
  <p>I'm not sure what of the options (keys, levels, names) I should be manipulating here to get my <code>rounds</code> back into the DataFrame.</p>
  
  <pre><code>results = pd.concat(
                p.map(run_experiment, experiment_collection),
                levels=iterables,
                names=names,
                axis=1
            )
  </code></pre>
  
  <p>Gets me really close to the format I want, but no <code>round</code>.</p>
  
  <hr>
  
  <p>I can get more specific if I need to, but not entirely sure what else would be helpful in getting to the answer.</p>