title: ValueError with pandas apply function returning output of variable shape
body: <p>I have a pandas dataframe with three columns structured like this:</p>
  <pre><code>Sample    Start  End
  &lt;string&gt;  &lt;int&gt;  &lt;int&gt;
  </code></pre>
  <p>The values in &quot;Start&quot; and &quot;End&quot; are intervals of positions on a larger string (e.g. from position 9000 to 11000). My goal is to subdivide the larger string into windows of 10000 positions, and count how many of those are contained in intervals from my dataframe.</p>
  <p>For example, window 0:10000 would contain 1000 positions and window 10000:20000 would contain the other 1000 positions from interval 9000:11000.</p>
  <p>To do this, I am first running a function to split these intervals into windows, such that if this is the input:</p>
  <pre><code>Sample    Start  End
  A         2500   5000
  A         9000   11000
  A         18000  19500
  </code></pre>
  <p>Then this is the output:</p>
  <pre><code>
  Sample    Start  End    W_start  W_end
  A         2500   5000   0        10000
  A         9000   10000  0        10000
  A         10000  11000  10000    20000
  A         18000  19500  10000    20000
  </code></pre>
  <p>This is the function I'm doing it with, where <code>df_sub</code> is a line of the dataframe and <code>w_size</code> is the window size (10000):</p>
  <pre class="lang-py prettyprint-override"><code>def split_into_windows(df_sub, w_size):
  
      start, end = df_sub.Start, df_sub.End
      w_start = start - (start % w_size)
      w_end = w_start + w_size
  
      if (w_start &lt;= start &lt;= w_end) and (w_start &lt;= end &lt;= w_end):
          df_out = df_sub
  
      elif (w_start &lt;= start &lt;= w_end) and (end &gt; w_end):
          out = []
          df_tmp = df_sub.copy()
          df_tmp.End = w_end
          out.append(df_tmp.copy())
  
          while (end &gt; w_end):
              w_start += w_size
              w_end += w_size
              df_tmp.Start = max(start, w_start)
              df_tmp.End = min(end, w_end)
              out.append(df_tmp.copy())
  
          df_out = pd.DataFrame(out)
  
      return df_out
  </code></pre>
  <p>I'm calling the function with <code>apply()</code>:</p>
  <pre><code>df = df.apply(split_into_windows, axis=1, args=(w_size,))
  </code></pre>
  <p>But I'm getting this error:</p>
  <pre><code>ValueError: Buffer has wrong number of dimensions (expected 1, got 2)
  </code></pre>
  <p>Looking online I found that this issue seems to be related with pandas merge but I am not using pandas merge. I believe it may be related to the fact that some lines produce a single output series, and some others produce a small dataframe (the split ones).</p>
  <p>See here:</p>
  <pre><code>Sample       A
  Start     6928
  End       9422
  
    Sample  Start    End
  0      A   9939  10000
  1      A  10000  11090
  </code></pre>
  <p>Any tips on how to fix this?</p>
  <p>Minimal dataset to reproduce: <a href="https://file.io/iZ3fguCFlRbq" rel="nofollow noreferrer">https://file.io/iZ3fguCFlRbq</a></p>
  <p>EDIT #1:</p>
  <p>I tried changing a line in the function to have a coherent output (i.e. returning dataframes only):</p>
  <pre><code>df_out = df_sub.to_frame().T
  </code></pre>
  <p>And now the <code>apply()</code> round &quot;works&quot;, as in throws no errors, but the output looks like this:</p>
  <pre><code>0  Sample  Start   End
  0  A       0       6915
  1  Sample  Start   End
  0  A       6928    9422
  2  Sample  Start   End
  0  A       9939    10000
  ...
  
  &lt;class 'pandas.core.series.Series'&gt;
  
  </code></pre>
  <p>EDIT #2:</p>
  <p>I cannot use <code>.iterrows()</code>, it takes too long (estimate: weeks) with the size of dataframe I'm operating with.</p>
  <p>EDIT #3:</p>
  <p>Using <code>multiprocessing</code> like this made me get through the day but it is still a suboptimal solution, compared to what I could achieve with a functioning <code>apply()</code> call and a pandas parallel application such as <code>pandarallel</code> or <code>swifter</code>. Still looking for any tip :)</p>
  <pre><code>pool = mp.Pool(processes=48)
  q = mp.Manager().Queue()
  
  start = time.time()
  for index, row in df_test.iterrows():
      pool.apply_async(split_into_windows, args=(row, w_size, q))
  
  pool.close()
  pool.join()
  
  out = []
  while q.empty() == False:
      out.append(q.get())
  
  df = pd.DataFrame(out)
  </code></pre>