title: DASK Memory Per Worker Guide
body: <p>I'm currently working on refactoring some legacy analytics into Python/DASK to show the efficacy of this as a solution going forward.</p>
  
  <p>I'm trying to set up a demo scenario and am having problems with memory and would like some advice.</p>
  
  <p>My scenario; I have my data split into 52 gzip compressed parquet files on S3, each one, uncompressed in memory is around 100MB, giving a total dataset size of ~5.5GB and exactly 100,000,000 rows.</p>
  
  <p>My scheduler is on a T2.Medium (4GB/2vCPUs) as are my 4 workers.
  Each worker is being run with 1 process, 1 thread and a memory limit of 4GB, I.e. <code>dask-worker MYADDRESS --nprocs 1 --nthreads=1 --memory-limit=4GB</code>.</p>
  
  <p>Now, I'm pulling the parquet files and immediately repartitioning on a column in such a way that I end up with roughly 480 partitions each of ~11MB.</p>
  
  <p>Then I'm using map_partitions to do the main body of work.
  This works fine for small datasets, however for the 100 mil dataset, my workers keep crashing due to not having enough memory.
  What am I doing wrong here?
  For implementation specific info, the function i'm passing to map_partitions can sometimes need roughly 1GB, due to what is essentially a cross join on the partition dataframe.</p>
  
  <p>Am I not understanding something to do with DASK's architecture? Between my scheduler and my 4 workers there is 20GB of memory to work with, yet this is proving to be not enough.
  From what I've read from the DASK documentation is that, so long as each partition, and what you do with that partition, fits in the memory of the worker then you should be ok?</p>
  
  <p>Is 4GB just not enough? Does it need way more to handle scheduler/inter process communication oerheads?</p>
  
  <p>Thanks for reading.</p>