title: Pickling error in passing Pandas DataFrame to method call in multi processing
body: <p>I was scratching my head over past few days why my method call in multiprocessing was failing, and after drilling down further I realized that it was due to some DataFrames being passed as arguments to method which wasn't getting pickled. So as alternative to that, I parsed the DataFrame to csv in my parent method, passed the name of csv file to child method where I read csv to get the data. But this is pretty inefficient method, so let me know where I did wrong. The following are my codes.</p>
  
  <p>DataFrames: 2 No.</p>
  
  <p>DataFrame: a</p>
  
  <pre><code>FIELD_NAME                    object
  DEST_LOC_NBR                 float64
  APPT_NBR                     float64
  SEQ_NBR                      float64
  FIELD_NBR                    float64
  CREATE_TS             datetime64[ns]
  CREATE_USERID_V               object
  BEFORE_FIELD_VALUE            object
  AFTER_FIELD_VALUE             object
  CHNG_ORIGIN_I                float64
  APPT_ STAT_ NBR              float64
  DESCRPTN                      object
  LOC_NBR_x                    float64
  LOCN_ABBR_x                   object
  LOC_NBR_y                     object
  LOCN_ABBR_y                   object
  FRGT_TYP_NBR_x               float64
  FRGT_TYP_DESC_x               object
  FRGT_TYP_NBR_y                object
  FRGT_TYP_DESC_y               object
  </code></pre>
  
  <p>DataFrame: d1</p>
  
  <pre><code>APPT_NBR                             float64
  APPT_MERGE_F                          object
  APPT_STAT_NBR                         object
  DEST_LOC_NBR                         float64
  APPT_TYP_NBR                         float64
  LOC_NBR                              float64
  FRGT_TYP_NBR                         float64
  APPT_FRT_TYP_AFTER_ARVL_TS            object
  APPT_EMPTY_F                          object
  APPT_ACTL_ARVL_TS             datetime64[ns]
  APPT_SCHD_ARVL_TS             datetime64[ns]
  APPT_NBR_OF_CNTRS                    float64
  APPT_NBR_OF_GOHS                     float64
  CREATE_TS                     datetime64[ns]
  LAST_UPD_TS                           object
  VERSION_NBR                          float64
  APPT_CLOSE_TS                         object
  APPT_OPEN_TS                          object
  APPT_UNCNFRMD_TS                      object
  APPT_ORG_SCH_AR_TS                    object
  APPT_CNCL_TS                          object
  APPT_SUSP_TS                          object
  APPT_ARCHV_TS                         object
  CARRIER                               object
  VND_NBR                              float64
  YARD_AREA_NBR                        float64
  CREATE_USERID_V                       object
  APPT_ACTL_ARVL_D                      object
  APPT_SCHD_ARVL_D                      object
  dtype: object
  </code></pre>
  
  <p>I tried even manually pickling on parent method and unpickle in child like I do for csv files, but the same error raises.</p>
  
  <p>I first create random file names as this is Multi-threading process, hence file name can't be same:</p>
  
  <pre><code>fname = ''.join(random.SystemRandom().choice(string.ascii_uppercase + string.digits) for _ in range(5))
  fnames = dict()
  fnames['a'] = '{}_a.pkl'.format(fname)
  fnames['d1'] = '{}_d1.pkl'.format(fname)
  fnames['file'] = fname
  </code></pre>
  
  <p>And, I pickle dataframes using the following:</p>
  
  <pre><code>d1.to_pickle(fnames['d1'])
  a.to_pickle(fnames['a'])
  </code></pre>
  
  <p>And, the following is my method call:</p>
  
  <pre><code>p = multiprocessing.Process(target=ParallelLoopTest, args=(dd, fnames, return_list, final_col_dates, d, final_col_dates_mod, iter, self.DC, self.start_hour_of_day))
  </code></pre>
  
  <p>While my method ParallelLoopTest definition is as follows:</p>
  
  <pre><code>def ParallelLoopTest(dd, fnames, days_out_vars, final_col_dates, d, final_col_dates_mod, iter, DC, start_hour_of_day, store):
  
      a = pd.read_pickle(fnames['a'])
      d1 = pd.read_pickle(fnames['d1'])
  
      df = pd.read_pickle('{}_df.pkl'.format(fnames['file']))
  </code></pre>
  
  <p>And, I face the following error:</p>
  
  <pre><code>Traceback (most recent call last):
    File "C:\Users\dkanhar\Anaconda3\lib\multiprocessing\process.py", line 249, in _bootstrap
      self.run()
    File "C:\Users\dkanhar\Anaconda3\lib\multiprocessing\process.py", line 93, in run
      self._target(*self._args, **self._kwargs)
    File "E:\Projects\Predictive Inbound Cartoon Estimation-MLO\Python\dataprep\DataPrep.py", line 497, in ParallelLoopTest
      a = pd.read_pickle(fnames['a'])
    File "C:\Users\dkanhar\Anaconda3\lib\site-packages\pandas\io\pickle.py", line 63, in read_pickle
      return try_read(path, encoding='latin1')
    File "C:\Users\dkanhar\Anaconda3\lib\site-packages\pandas\io\pickle.py", line 57, in try_read
      return pc.load(fh, encoding=encoding, compat=True)
    File "C:\Users\dkanhar\Anaconda3\lib\site-packages\pandas\compat\pickle_compat.py", line 118, in load
      return up.load()
    File "C:\Users\dkanhar\Anaconda3\lib\pickle.py", line 1039, in load
      dispatch[key[0]](self)
    File "C:\Users\dkanhar\Anaconda3\lib\site-packages\pandas\compat\pickle_compat.py", line 73, in load_newobj
      obj = cls.__new__(cls, *args)
  TypeError: function takes at most 0 arguments (1 given)
  </code></pre>
  
  <p>So, as you can see, Pandas fail to unpickle the DatFrame, no matter what I try. I tried picking manually using <strong>pickly.dump() and pickle.load()</strong> but even that failed with similar error. (<em>TypeError, Function takes 0 arguments (1 given)</em>)</p>
  
  <p>I also feel this is due to DataFrame problem, as I created a random DataFrame using:</p>
  
  <pre><code>df = pd.DataFrame(np.random.randint(0, 100, size=(100000, 4)), columns=list('ABCD'))
  </code></pre>
  
  <p>Pickling it, and unpickling on other end, and that worked.</p>
  
  <p>So, what must be issue with DataFrame? Possible scenarios where DataFrame can't be pickled? </p>
  
  <p>Note that its around 200mb pickle file for Dataframe <strong>a</strong> and 25mb for pickled DataFrame <strong>d1</strong>.</p>
  
  <p>Also, is there way to post head of whole DataFrame? It has around 50 columns which wont be printed, and hence cant be added here, but in meanwhile I'm adding dtypes of both DataFrames.</p>
  
  <p>Any help would be really helpful.</p>
  
  <p>PS: The following link is for my previous post which described the error while Passed DataFRame directly to Method call instead of picking and unpickling:
  <a href="https://stackoverflow.com/questions/39993436/similar-errors-in-multiprocessing-mismatch-number-of-arguments-to-function?noredirect=1#comment67449576_39993436">Similar errors in MultiProcessing. Mismatch number of arguments to function</a></p>
  
  <p><strong>Update</strong>:
  The following is link to Google spreedsheet with head n = 50 for both the DataFrames in concern.</p>
  
  <p><a href="https://docs.google.com/spreadsheets/d/1bGkpmV0__aPVUtc0HSeuRQpufu1T4pmKB4YxsQsPO50/edit?usp=sharing" rel="nofollow noreferrer">https://docs.google.com/spreadsheets/d/1bGkpmV0__aPVUtc0HSeuRQpufu1T4pmKB4YxsQsPO50/edit?usp=sharing</a></p>