{"action":"created","issue":{"url":"https://api.github.com/repos/bokeh/datashader/issues/633","repository_url":"https://api.github.com/repos/bokeh/datashader","labels_url":"https://api.github.com/repos/bokeh/datashader/issues/633/labels{/name}","comments_url":"https://api.github.com/repos/bokeh/datashader/issues/633/comments","events_url":"https://api.github.com/repos/bokeh/datashader/issues/633/events","html_url":"https://github.com/bokeh/datashader/issues/633","id":350501782,"node_id":"MDU6SXNzdWUzNTA1MDE3ODI=","number":633,"title":"Large amount of time spent on determining datashape","user":{"login":"philippjfr","id":1550771,"node_id":"MDQ6VXNlcjE1NTA3NzE=","avatar_url":"https://avatars2.githubusercontent.com/u/1550771?v=4","gravatar_id":"","url":"https://api.github.com/users/philippjfr","html_url":"https://github.com/philippjfr","followers_url":"https://api.github.com/users/philippjfr/followers","following_url":"https://api.github.com/users/philippjfr/following{/other_user}","gists_url":"https://api.github.com/users/philippjfr/gists{/gist_id}","starred_url":"https://api.github.com/users/philippjfr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/philippjfr/subscriptions","organizations_url":"https://api.github.com/users/philippjfr/orgs","repos_url":"https://api.github.com/users/philippjfr/repos","events_url":"https://api.github.com/users/philippjfr/events{/privacy}","received_events_url":"https://api.github.com/users/philippjfr/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-08-14T16:22:44Z","updated_at":"2018-08-14T20:37:38Z","closed_at":null,"author_association":"COLLABORATOR","body":"When running aggregation on a large dataset a fairly large amount of time is spent in the ``dshape_from_dask`` function (somewhere in the region of 500-600ms). This is a major overhead when running the aggregation many times in a row and even dominates the actual aggregation time in many cases. Here is the output of a profiler running canvas.points on the census dataset:\r\n\r\n```\r\n         77200 function calls (75103 primitive calls) in 1.021 seconds\r\n\r\n   Ordered by: cumulative time\r\n\r\n   ncalls  tottime  percall  cumtime  percall filename:lineno(function)\r\n        1    0.000    0.000    1.021    1.021 {built-in method builtins.exec}\r\n        1    0.000    0.000    1.021    1.021 <string>:2(<module>)\r\n        1    0.000    0.000    1.021    1.021 spaces.py:1093(__getitem__)\r\n        1    0.000    0.000    1.020    1.020 spaces.py:890(_execute_callback)\r\n        1    0.000    0.000    1.020    1.020 spaces.py:540(__call__)\r\n        1    0.000    0.000    1.020    1.020 __init__.py:430(dynamic_operation)\r\n        1    0.000    0.000    1.020    1.020 __init__.py:415(_process)\r\n        1    0.000    0.000    1.020    1.020 operation.py:137(process_element)\r\n      2/1    0.000    0.000    1.020    1.020 operation.py:113(_apply)\r\n        1    0.000    0.000    1.019    1.019 datashader.py:782(_process)\r\n     18/7    0.000    0.000    1.018    0.145 dimension.py:712(map)\r\n        1    0.000    0.000    1.017    1.017 operation.py:146(__call__)\r\n        1    0.000    0.000    0.998    0.998 datashader.py:416(_process)\r\n        1    0.000    0.000    0.964    0.964 core.py:144(points)\r\n        1    0.000    0.000    0.964    0.964 core.py:492(bypixel)\r\n        5    0.000    0.000    0.897    0.179 local.py:405(get_async)\r\n        3    0.000    0.000    0.897    0.299 threaded.py:33(get)\r\n       24    0.000    0.000    0.894    0.037 local.py:150(queue_get)\r\n       24    0.000    0.000    0.894    0.037 queue.py:147(get)\r\n       12    0.000    0.000    0.893    0.074 threading.py:263(wait)\r\n       96    0.893    0.009    0.893    0.009 {method 'acquire' of '_thread.lock' objects}\r\n        1    0.066    0.066    0.491    0.491 utils.py:368(dshape_from_dask) <-----\r\n      2/1    0.000    0.000    0.472    0.472 utils.py:51(__call__)\r\n        1    0.000    0.000    0.471    0.471 dask.py:20(dask_pipeline)\r\n```\r\n\r\nYou can see that almost half the time is spent in the ``dshape_from_dask`` function, and as you aggregate a smaller region this time dominates even more.\r\n\r\nIt would be great if this could be sped up in general but it might be sufficient to allow passing in the datashape in cases where you are doing a lot of aggregation on the same datasource (e.g. when generating a tileset) and want to avoid this 0.5 second penalty each time.\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/bokeh/datashader/issues/comments/413007713","html_url":"https://github.com/bokeh/datashader/issues/633#issuecomment-413007713","issue_url":"https://api.github.com/repos/bokeh/datashader/issues/633","id":413007713,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzAwNzcxMw==","user":{"login":"jbednar","id":1695496,"node_id":"MDQ6VXNlcjE2OTU0OTY=","avatar_url":"https://avatars0.githubusercontent.com/u/1695496?v=4","gravatar_id":"","url":"https://api.github.com/users/jbednar","html_url":"https://github.com/jbednar","followers_url":"https://api.github.com/users/jbednar/followers","following_url":"https://api.github.com/users/jbednar/following{/other_user}","gists_url":"https://api.github.com/users/jbednar/gists{/gist_id}","starred_url":"https://api.github.com/users/jbednar/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jbednar/subscriptions","organizations_url":"https://api.github.com/users/jbednar/orgs","repos_url":"https://api.github.com/users/jbednar/repos","events_url":"https://api.github.com/users/jbednar/events{/privacy}","received_events_url":"https://api.github.com/users/jbednar/received_events","type":"User","site_admin":false},"created_at":"2018-08-14T20:37:38Z","updated_at":"2018-08-14T20:37:38Z","author_association":"COLLABORATOR","body":"Sorry -- just edited my message above to clarify -- I mean to suggest avoiding doing the test at all, always doing what's inside the \"if\"."}}
{"actor":{"display_login":"jbednar"}}