{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/xarray/issues/912","repository_url":"https://api.github.com/repos/pydata/xarray","labels_url":"https://api.github.com/repos/pydata/xarray/issues/912/labels{/name}","comments_url":"https://api.github.com/repos/pydata/xarray/issues/912/comments","events_url":"https://api.github.com/repos/pydata/xarray/issues/912/events","html_url":"https://github.com/pydata/xarray/issues/912","id":166593563,"number":912,"title":"Speed up operations with xarray dataset","user":{"login":"saulomeirelles","id":7504461,"avatar_url":"https://avatars.githubusercontent.com/u/7504461?v=3","gravatar_id":"","url":"https://api.github.com/users/saulomeirelles","html_url":"https://github.com/saulomeirelles","followers_url":"https://api.github.com/users/saulomeirelles/followers","following_url":"https://api.github.com/users/saulomeirelles/following{/other_user}","gists_url":"https://api.github.com/users/saulomeirelles/gists{/gist_id}","starred_url":"https://api.github.com/users/saulomeirelles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saulomeirelles/subscriptions","organizations_url":"https://api.github.com/users/saulomeirelles/orgs","repos_url":"https://api.github.com/users/saulomeirelles/repos","events_url":"https://api.github.com/users/saulomeirelles/events{/privacy}","received_events_url":"https://api.github.com/users/saulomeirelles/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-07-20T14:21:40Z","updated_at":"2016-07-20T18:44:53Z","closed_at":null,"body":"Hi all,\r\n\r\nI've been recently having hard times to manipulate a `xarray` `dataset`. Not sure if I am making some awkward mistake, but it is taking an unacceptable amount of time to perform simple operations. \r\n\r\nHere is a piece of my code:\r\n\r\n```\r\nncfile   = glob('*conc_size_12m.nc')\r\nds       = xray.open_dataset(ncfile[0])\r\nds\r\n```\r\n\r\n```\r\n<xarray.Dataset>\r\nDimensions:          (burst: 2485, duration: 2400, z: 160)\r\nCoordinates:\r\n    zdist            (z) float64 0.01014 0.02027 0.03041 0.04054 0.05068 ...\r\n    burst_nr         (burst) float64 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 ...\r\n    time             (duration, burst) datetime64[ns] 2014-09-16T07:00:00 ...\r\n  * burst            (burst) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 ...\r\n  * duration         (duration) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 ...\r\n  * z                (z) int64 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 ...\r\nData variables:\r\n    conc_profs       (duration, z, burst) float32 3.99138e-05 4.23636e-05 ...\r\n    burst_duration   (duration) float64 0.0 0.1246 0.2493 0.3739 0.4985 ...\r\n    grainSize_profs  (duration, z, burst) float32 200.0 200.0 200.0 200.0 ...\r\n```\r\n\r\n`ds.nbytes * (2 ** -30)`\r\n\r\n`7.15415246784687`\r\n\r\n`%time conc_avg = ds.conc_profs.chunk(2400).mean(('z','duration'))`\r\n\r\n```\r\nCPU times: user 12 ms, sys: 0 ns, total: 12 ms\r\nWall time: 9.84 ms\r\n```\r\n`%time conc_avg.load()` \r\n\r\n`%time conc_avg = ds.conc_profs.isel(burst=0).mean(('z','duration'))`\r\n\r\n```\r\nCPU times: user 708 ms, sys: 2.87 s, total: 3.58 s\r\nWall time: 1min 56s\r\n```\r\n\r\nIf I work with chunks, it is impossible to load back the array in a reasonable amount of time (I waited for more than 30 min).\r\n\r\nLooping over the dimension `burst`, it takes about 2 minutes per loop which is also quite unreasonable.\r\n\r\nI was wondering if the problem could stem from the creation of my `dataset` which I saved into this 7+GB netCDF file. Could that be the case?\r\n\r\nI am working in a Linux Inter Core i5 which is supposed to handle these manipulations with no hicups. I use the IOOS environment to run `xarray` (vr '0.7.1').\r\n\r\nCan someone provide me some advice on how to optimize my script? \r\n\r\nI am happy to supply with more details if needed.\r\n\r\nCheers,"},"comment":{"url":"https://api.github.com/repos/pydata/xarray/issues/comments/234043292","html_url":"https://github.com/pydata/xarray/issues/912#issuecomment-234043292","issue_url":"https://api.github.com/repos/pydata/xarray/issues/912","id":234043292,"user":{"login":"saulomeirelles","id":7504461,"avatar_url":"https://avatars.githubusercontent.com/u/7504461?v=3","gravatar_id":"","url":"https://api.github.com/users/saulomeirelles","html_url":"https://github.com/saulomeirelles","followers_url":"https://api.github.com/users/saulomeirelles/followers","following_url":"https://api.github.com/users/saulomeirelles/following{/other_user}","gists_url":"https://api.github.com/users/saulomeirelles/gists{/gist_id}","starred_url":"https://api.github.com/users/saulomeirelles/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/saulomeirelles/subscriptions","organizations_url":"https://api.github.com/users/saulomeirelles/orgs","repos_url":"https://api.github.com/users/saulomeirelles/repos","events_url":"https://api.github.com/users/saulomeirelles/events{/privacy}","received_events_url":"https://api.github.com/users/saulomeirelles/received_events","type":"User","site_admin":false},"created_at":"2016-07-20T18:44:53Z","updated_at":"2016-07-20T18:44:53Z","body":"No, not really. I got no error message whatsoever. Is there any test I can\ndo to tackle this?\n\nSent from Smartphone. Please forgive typos.\n\nOn Jul 20, 2016 8:41 PM, \"Stephan Hoyer\" <notifications@github.com> wrote:\n\n> I decided to wait for .load() to do the job but the kernel dies after a\n> while.\n>\n> Are you running out of memory? Can you tell what's going on? This is a\n> little surprising to me.\n>\n> —\n> You are receiving this because you authored the thread.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/pydata/xarray/issues/912#issuecomment-234042142>, or mute\n> the thread\n> <https://github.com/notifications/unsubscribe-auth/AHKCTXaBpbA0ieSdI2I_hIUjVBxuKaNeks5qXmvPgaJpZM4JQ0_D>\n> .\n>\n"}}
{"actor":{"display_login":"saulomeirelles"}}