title: Looking for a solution to speed up `pyspark.sql.GroupedData.applyInPandas` processing on a large dataset
body: <p>I'm working with a dataset stored in S3 bucket (parquet files) consisting of a total of <code>~165 million</code> records (with ~30 columns). Now, the requirement is to first <code>groupby</code> a certain ID column then generate <code>250+ features</code> for each of these grouped records based on the data. Building these features is quite complex using multiple Pandas functionality along with 10+ supporting functions. The groupby function should generate <code>~5-6 million records</code>, hence the final output should be <code>6M x 250</code> shaped dataframe.</p>
  <p>Now, I've tested the code on a smaller sample and it works fine. The issue is, when I'm implementing it on the entire dataset, it takes a very long time - the progress bar in Spark display doesn't change even after 4+ hours of running. I'm running this in AWS EMR Notebook connected to a Cluster (1 m5.xlarge Master &amp; 2 m5.xlarge Core Nodes).
  I've tried with <code>1 m5.4xlarge Master &amp; 2 m5.4xlarge Core Nodes</code>, <code>1 m5.xlarge Master &amp; 8 m5.xlarge Core Nodes</code> combinations among others. None of them have shown any progress.
  I've tried running it in Pandas in-memory in my local machine for ~650k records, the progress was ~3.5 iterations/sec which came to be an ETA of <code>~647 hours</code>.</p>
  <p>So, the question is - can anyone share any better solution to reduce the time consumption and speed up the processing ? Should another cluster type be used for this use-case ? Should this be refactored or should Pandas dataframe usage be removed or any other pointer would be really helpful.</p>
  <p>Thanks much in advance!</p>