{"action":"created","issue":{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/238","repository_url":"https://api.github.com/repos/MDAnalysis/mdanalysis","labels_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/238/labels{/name}","comments_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/238/comments","events_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/238/events","html_url":"https://github.com/MDAnalysis/mdanalysis/issues/238","id":66678890,"node_id":"MDU6SXNzdWU2NjY3ODg5MA==","number":238,"title":"Trajectory reads by chunk","user":{"login":"dotsdl","id":1326811,"node_id":"MDQ6VXNlcjEzMjY4MTE=","avatar_url":"https://avatars3.githubusercontent.com/u/1326811?v=4","gravatar_id":"","url":"https://api.github.com/users/dotsdl","html_url":"https://github.com/dotsdl","followers_url":"https://api.github.com/users/dotsdl/followers","following_url":"https://api.github.com/users/dotsdl/following{/other_user}","gists_url":"https://api.github.com/users/dotsdl/gists{/gist_id}","starred_url":"https://api.github.com/users/dotsdl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dotsdl/subscriptions","organizations_url":"https://api.github.com/users/dotsdl/orgs","repos_url":"https://api.github.com/users/dotsdl/repos","events_url":"https://api.github.com/users/dotsdl/events{/privacy}","received_events_url":"https://api.github.com/users/dotsdl/received_events","type":"User","site_admin":false},"labels":[{"id":221183363,"node_id":"MDU6TGFiZWwyMjExODMzNjM=","url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels/Component-Readers","name":"Component-Readers","color":"0052cc","default":false},{"id":195620629,"node_id":"MDU6TGFiZWwxOTU2MjA2Mjk=","url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels/enhancement","name":"enhancement","color":"84b6eb","default":true},{"id":574417238,"node_id":"MDU6TGFiZWw1NzQ0MTcyMzg=","url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels/parallelization","name":"parallelization","color":"10aaee","default":false},{"id":195620938,"node_id":"MDU6TGFiZWwxOTU2MjA5Mzg=","url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels/performance","name":"performance","color":"f7c6c7","default":false},{"id":195620632,"node_id":"MDU6TGFiZWwxOTU2MjA2MzI=","url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels/question","name":"question","color":"cc317c","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2015-04-06T18:54:39Z","updated_at":"2018-08-31T15:33:10Z","closed_at":null,"author_association":"MEMBER","body":"I don't know if this is already the case for some trajectory readers, but it would surely bring better performance if the trajectory iterator read portions of a trajectory by chunks (more than just one frame) at a time. Is this feasable to implement?\n"},"comment":{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/comments/417701692","html_url":"https://github.com/MDAnalysis/mdanalysis/issues/238#issuecomment-417701692","issue_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/238","id":417701692,"node_id":"MDEyOklzc3VlQ29tbWVudDQxNzcwMTY5Mg==","user":{"login":"orbeckst","id":237980,"node_id":"MDQ6VXNlcjIzNzk4MA==","avatar_url":"https://avatars0.githubusercontent.com/u/237980?v=4","gravatar_id":"","url":"https://api.github.com/users/orbeckst","html_url":"https://github.com/orbeckst","followers_url":"https://api.github.com/users/orbeckst/followers","following_url":"https://api.github.com/users/orbeckst/following{/other_user}","gists_url":"https://api.github.com/users/orbeckst/gists{/gist_id}","starred_url":"https://api.github.com/users/orbeckst/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orbeckst/subscriptions","organizations_url":"https://api.github.com/users/orbeckst/orgs","repos_url":"https://api.github.com/users/orbeckst/repos","events_url":"https://api.github.com/users/orbeckst/events{/privacy}","received_events_url":"https://api.github.com/users/orbeckst/received_events","type":"User","site_admin":false},"created_at":"2018-08-31T15:33:10Z","updated_at":"2018-08-31T15:33:10Z","author_association":"MEMBER","body":"Maybe. If we ever get chunk reading to work, it would be worthwhile to benchmark. The scripts for the draft above will be in https://github.com/hpcanalytics/supplement-hpc-py-parallel-mdanalysis.\r\n\r\nThe things that really helped were (1) used a HDF5 format and parallel (MPI) reads of HDF5 or (2) chop the trajectory in multiple chunks and have each MPI rank read its own file. The ChainReader kind of works here but has the disadvantage that the ChainReader in each rank still opens and closes _all_ file chunks (it has to build its virtual frame map) – at least that's my understanding, correct me if I am wrong. If you have 100 ranks and 100 chunks then that's 10,000 file open operations more or less at the same time, and Lustre does not seem to like this."}}
{"actor":{"display_login":"orbeckst"}}