{"action":"opened","issue":{"url":"https://api.github.com/repos/pangeo-data/pangeo/issues/182","repository_url":"https://api.github.com/repos/pangeo-data/pangeo","labels_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/182/labels{/name}","comments_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/182/comments","events_url":"https://api.github.com/repos/pangeo-data/pangeo/issues/182/events","html_url":"https://github.com/pangeo-data/pangeo/issues/182","id":308644835,"number":182,"title":"dask scaling performance on pangeo.pydata.org","user":{"login":"rabernat","id":1197350,"avatar_url":"https://avatars1.githubusercontent.com/u/1197350?v=4","gravatar_id":"","url":"https://api.github.com/users/rabernat","html_url":"https://github.com/rabernat","followers_url":"https://api.github.com/users/rabernat/followers","following_url":"https://api.github.com/users/rabernat/following{/other_user}","gists_url":"https://api.github.com/users/rabernat/gists{/gist_id}","starred_url":"https://api.github.com/users/rabernat/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rabernat/subscriptions","organizations_url":"https://api.github.com/users/rabernat/orgs","repos_url":"https://api.github.com/users/rabernat/repos","events_url":"https://api.github.com/users/rabernat/events{/privacy}","received_events_url":"https://api.github.com/users/rabernat/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-03-26T16:30:29Z","updated_at":"2018-03-26T16:30:29Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"I am pushing ahead with my use case, which involves computing reductions on ~10TB size datasets stored in zarr on gcs. I am observing some erratic behavior from the cluster when the number of chunks in the array gets large (> 10000). Below is my attempt to conduct some simple scaling analysis of cluster performance. I am using a large cluster (100 workers).\r\n\r\nI create my cluster like this\r\n```python\r\nfrom dask.distributed import Client, progress\r\nfrom dask_kubernetes import KubeCluster\r\ncluster = KubeCluster(n_workers=100)\r\nclient = Client(cluster)\r\n```\r\n\r\nHere is the script I use to test raw dask scaling (bypassing any gcs issues).\r\n```python\r\nfrom time import time\r\nimport dask.array as da\r\nimport numpy as np\r\nimport xarray as xr\r\nfrom matplotlib import pyplot as plt\r\n\r\ndef time_dask_ops(n, retries=10):\r\n    \"\"\"Time three parts of dask array processing.\r\n      - create\r\n      - persist\r\n      - compute\r\n    \"\"\"\r\n    \r\n    # create a random dask array with size comparable to a climate dataset\r\n    shape = (n, 50, 2700, 3600)\r\n    chunksize = (1, 1, 2700, 3600)\r\n    ar = da.random.random(shape, chunks=chunksize)\r\n    \r\n    # take the mean along final three dimensions to create a timeseries\r\n    tic = time()\r\n    ts = ar.mean(axis=(1,2,3))\r\n    create_time = time() - tic\r\n    \r\n    tic = time()\r\n    ts_persist = ts.persist(retries=retries)\r\n    persist_time = time() - tic\r\n    \r\n    tic = time()\r\n    ts_compute = ts_persist.compute(retries=retries)\r\n    compute_time = time() - tic\r\n    \r\n    del ts_compute\r\n    del ts_persist\r\n    \r\n    return create_time, persist_time, compute_time\r\n\r\n# log spacing of the array size\r\nnt_values = 2**np.arange(0,10)\r\n# the expensive step\r\nresults = [time_dask_ops(nt) for nt in nt_values]\r\n\r\n# visualize results\r\nnum_tasks = nt_values * 50\r\nds_results = xr.DataArray(results, dims=['num_tasks', 'step'],\r\n                          coords={'num_tasks': num_tasks,\r\n                                 'step': np.array(['create', 'persist', 'compute'])})\r\nfig, ax = plt.subplots(figsize=(12,10))\r\nds_results.plot.line(x='num_tasks', ax=ax, marker='o')\r\nax.plot(num_tasks, num_tasks/100, 'k--', label='linear')\r\nax.set_yscale('log')\r\nax.set_xscale('log')\r\nax.set_ylabel('time [seconds]')\r\nax.grid()\r\n```\r\n\r\n![image](https://user-images.githubusercontent.com/1197350/37919201-3a883696-30f1-11e8-97c8-966b8dee8c3d.png)\r\n\r\nThe scaling of `compute` is slightly sub-linear but doesn't seem too bad.\r\n\r\nOnce the arrays start getting big (n=128, 256, 512), I start seeing errors:\r\n```\r\nFuture exception was never retrieved\r\nfuture: <Future finished exception=WebSocketClosedError()>\r\nTraceback (most recent call last):\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/websocket.py\", line 808, in wrapper\r\n    yield fut\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/gen.py\", line 1099, in run\r\n    value = future.result()\r\ntornado.iostream.StreamClosedError: Stream is closed\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/gen.py\", line 1107, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/websocket.py\", line 810, in wrapper\r\n    raise WebSocketClosedError()\r\ntornado.websocket.WebSocketClosedError\r\nFuture exception was never retrieved\r\nfuture: <Future finished exception=WebSocketClosedError()>\r\nTraceback (most recent call last):\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/websocket.py\", line 808, in wrapper\r\n    yield fut\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/gen.py\", line 1099, in run\r\n    value = future.result()\r\ntornado.iostream.StreamClosedError: Stream is closed\r\n\r\nDuring handling of the above exception, another exception occurred:\r\n\r\nTraceback (most recent call last):\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/gen.py\", line 1107, in run\r\n    yielded = self.gen.throw(*exc_info)\r\n  File \"/opt/conda/lib/python3.6/site-packages/tornado/websocket.py\", line 810, in wrapper\r\n    raise WebSocketClosedError()\r\ntornado.websocket.WebSocketClosedError\r\ndistributed.core - WARNING - Event loop was unresponsive in Scheduler for 7.32s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.\r\ndistributed.core - WARNING - Event loop was unresponsive in Scheduler for 6.85s.  This is often caused by long-running GIL-holding functions or moving large chunks of data. This can cause timeouts and instability.\r\ndistributed.utils_perf - WARNING - full garbage collections took 10% CPU time recently (threshold: 10%)\r\ndistributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)\r\ndistributed.utils_perf - WARNING - full garbage collections took 11% CPU time recently (threshold: 10%)\r\n```\r\n\r\nThese appear to be non-fatal.\r\n\r\nMy next step is to do the same thing with gcs-backed zarr data.\r\n"}}
{"actor":{"display_login":"rabernat"}}