{"action":"opened","issue":{"url":"https://api.github.com/repos/yt-project/yt/issues/2383","repository_url":"https://api.github.com/repos/yt-project/yt","labels_url":"https://api.github.com/repos/yt-project/yt/issues/2383/labels{/name}","comments_url":"https://api.github.com/repos/yt-project/yt/issues/2383/comments","events_url":"https://api.github.com/repos/yt-project/yt/issues/2383/events","html_url":"https://github.com/yt-project/yt/issues/2383","id":534581492,"node_id":"MDU6SXNzdWU1MzQ1ODE0OTI=","number":2383,"title":"Slow data retrieval for some YTRay objects","user":{"login":"zhafen","id":8739972,"node_id":"MDQ6VXNlcjg3Mzk5NzI=","avatar_url":"https://avatars2.githubusercontent.com/u/8739972?v=4","gravatar_id":"","url":"https://api.github.com/users/zhafen","html_url":"https://github.com/zhafen","followers_url":"https://api.github.com/users/zhafen/followers","following_url":"https://api.github.com/users/zhafen/following{/other_user}","gists_url":"https://api.github.com/users/zhafen/gists{/gist_id}","starred_url":"https://api.github.com/users/zhafen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zhafen/subscriptions","organizations_url":"https://api.github.com/users/zhafen/orgs","repos_url":"https://api.github.com/users/zhafen/repos","events_url":"https://api.github.com/users/zhafen/events{/privacy}","received_events_url":"https://api.github.com/users/zhafen/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2019-12-08T18:33:47Z","updated_at":"2019-12-08T18:33:47Z","closed_at":null,"author_association":"NONE","body":"<!--To help us understand and resolve your issue, please fill out the form to\r\nthe best of your ability.-->\r\n<!--You can feel free to delete the sections that do not apply.-->\r\n\r\n### Bug report\r\n\r\n**Bug summary**\r\n\r\nyt appears to be very slow at reading data from h5py files for use in `YTRay` objects, especially for particle-based simulations with snapshot size >~ 7 GB.\r\n\r\n**Code for reproduction**\r\n\r\n[A notebook for reproducing the issue is available here, including ewah file, results of cProfile, and data download links.](https://github.com/yt-project/yt/files/3936681/ray_profiling.zip)\r\n\r\n\r\n**Actual outcome**\r\nRay generation is ~30 s per field, for a total of ~10 min when enough fields are retrieved.\r\n\r\n**Expected outcome**\r\nUsing h5py to open the full data directly and perform simple calculations takes <5 seconds, so at minimum a factor of ~6 faster. If caching is possible then the speed-up could be much greater.\r\n\r\n**Notes**\r\n* Most of the time is spent in `yt/utilities/io_handler.py:158(_read_particle_selection)`.\r\n* The simulation capable of creating this problem has high dynamic range: Particle mass resolution ~7000 Msun, for a total gas mass of a few times 1e12 Msun (i.e. out to a few times the virial radius of a L* galaxy).\r\n* The raw data is a Lagrangian cosmological zoom-in simulation with data separated into four files. Total file size ~8GB.\r\n* Intersecting the box edge doesn't significantly change computation time. (Note that this is true for the YT ray, but it will take twice as long for the Trident ray because Trident will break up rays that go beyond the box into smaller rays, each of which is equally expensive.)\r\n\r\n**Version Information**\r\n  * Operating System: Linux\r\n  * Python Version: 3.7.1\r\n  * yt version: 4.0.dev0 changeset 947ebdc22c48\r\n  * Other Libraries (if applicable): h5py 2.8.0, Trident 1.3.dev1\r\n\r\n**Installation Method**\r\n```bash\r\n$ # install yt\r\n$ git clone https://github.com/yt-project/yt --branch=yt-4.0\r\n$ cd yt\r\n$ pip install -e .\r\n$ cd ..\r\n$ # install trident\r\n$ git clone https://github.com/trident-project/trident\r\n$ cd trident\r\n$ pip install -e .\r\n```\r\n\r\n"}}
{"actor":{"display_login":"zhafen"}}