{"action":"created","issue":{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618","labels_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618/labels{/name}","comments_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618/comments","events_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618/events","html_url":"https://github.com/MDAnalysis/mdanalysis/pull/618","id":126423459,"number":618,"title":"Parallel trajectory analysis (revisited)","user":{"login":"mattihappy","id":12560118,"avatar_url":"https://avatars.githubusercontent.com/u/12560118?v=3","gravatar_id":"","url":"https://api.github.com/users/mattihappy","html_url":"https://github.com/mattihappy","followers_url":"https://api.github.com/users/mattihappy/followers","following_url":"https://api.github.com/users/mattihappy/following{/other_user}","gists_url":"https://api.github.com/users/mattihappy/gists{/gist_id}","starred_url":"https://api.github.com/users/mattihappy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattihappy/subscriptions","organizations_url":"https://api.github.com/users/mattihappy/orgs","repos_url":"https://api.github.com/users/mattihappy/repos","events_url":"https://api.github.com/users/mattihappy/events{/privacy}","received_events_url":"https://api.github.com/users/mattihappy/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels/Component-Analysis","name":"Component-Analysis","color":"0088ee"},{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels/performance","name":"performance","color":"f7c6c7"},{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels/Work%20in%20progress","name":"Work in progress","color":"e11d21"}],"state":"open","locked":false,"assignee":{"login":"richardjgowers","id":9249543,"avatar_url":"https://avatars.githubusercontent.com/u/9249543?v=3","gravatar_id":"","url":"https://api.github.com/users/richardjgowers","html_url":"https://github.com/richardjgowers","followers_url":"https://api.github.com/users/richardjgowers/followers","following_url":"https://api.github.com/users/richardjgowers/following{/other_user}","gists_url":"https://api.github.com/users/richardjgowers/gists{/gist_id}","starred_url":"https://api.github.com/users/richardjgowers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/richardjgowers/subscriptions","organizations_url":"https://api.github.com/users/richardjgowers/orgs","repos_url":"https://api.github.com/users/richardjgowers/repos","events_url":"https://api.github.com/users/richardjgowers/events{/privacy}","received_events_url":"https://api.github.com/users/richardjgowers/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/milestones/7","html_url":"https://github.com/MDAnalysis/mdanalysis/milestones/0.14","labels_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/milestones/7/labels","id":1454545,"number":7,"title":"0.14","description":"","creator":{"login":"orbeckst","id":237980,"avatar_url":"https://avatars.githubusercontent.com/u/237980?v=3","gravatar_id":"","url":"https://api.github.com/users/orbeckst","html_url":"https://github.com/orbeckst","followers_url":"https://api.github.com/users/orbeckst/followers","following_url":"https://api.github.com/users/orbeckst/following{/other_user}","gists_url":"https://api.github.com/users/orbeckst/gists{/gist_id}","starred_url":"https://api.github.com/users/orbeckst/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/orbeckst/subscriptions","organizations_url":"https://api.github.com/users/orbeckst/orgs","repos_url":"https://api.github.com/users/orbeckst/repos","events_url":"https://api.github.com/users/orbeckst/events{/privacy}","received_events_url":"https://api.github.com/users/orbeckst/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":8,"state":"open","created_at":"2015-12-10T17:34:28Z","updated_at":"2016-01-20T14:27:57Z","due_on":"2016-01-31T00:00:00Z","closed_at":null},"comments":13,"created_at":"2016-01-13T14:00:11Z","updated_at":"2016-01-21T11:03:26Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/618","html_url":"https://github.com/MDAnalysis/mdanalysis/pull/618","diff_url":"https://github.com/MDAnalysis/mdanalysis/pull/618.diff","patch_url":"https://github.com/MDAnalysis/mdanalysis/pull/618.patch"},"body":"With reference to #617 \r\nMDAnalysis Discussion --> [link ](https://groups.google.com/forum/#!topic/mdnalysis-discussion/MaX70t5RZ7Y)\r\nBranch --> [link](https://github.com/mattihappy/mdanalysis/tree/feature-ParallelAnalysis)\r\n\r\n# Aim\r\nSpeed up trajectory analysis by spreading the computation on multiple threads. Slices of the trajectory are assigned to each thread, which then proceeds running a batch of AnalysisBase istances.\r\n\r\n### Current syntax (may change)\r\nIn order to run a parallel analysis, one needs to setup a Universe object and a list of `AnalysisBase` objects. Such list is then passed to a istance of the `ParallelProcessor` class and run calling the method `parallel_run()`.\r\n\r\n```python\r\nimport MDAnalysis as mda\r\nfrom MDAnalysisTests.datafiles import DCD, PSF\r\nimport parallel_jobs  as pj\r\nimport electromagnetism # some analysis module\r\n\r\nuniverse = mda.Universe(PSF, DCD)\r\nselection_string = 'all'\r\n\r\njobs = [electromagnetism.TotalDipole(selection=selection_string)]\r\nprocess = pj.ParallelProcessor(jobs, universe, threads=4, start=1, stop=-1, step=1)\r\nprocess.parallel_run()\r\n```\r\n\r\nOtherwise, one can just invoke the `run()` method with the optional arguments `parallel=True` and `threads=<nthreads>`.\r\n\r\n```python\r\nimport MDAnalysis as mda\r\nfrom MDAnalysisTests.datafiles import DCD, PSF\r\nimport parallel_jobs  as pj\r\nimport electromagnetism # some analysis module\r\n\r\nuniverse = mda.Universe(PSF, DCD)\r\nselection_string = 'all'\r\n\r\nel = electromagnetism.TotalDipole(universe=universe, selection=selection_string)]\r\nel.run(parallel=True, nthreads=4)\r\n```\r\n \r\n### Design thoughts\r\nGiven four threads and three different analysis objects, and a trajectory with 100 configurations, the workload is distributed as follows\r\n\r\nThread    | Task(s) | Frames\r\n------------ | ------------- | ------------- | \r\nThread 1 | Analysis1, Analysis2, Analysis3 |  0 to 24\r\nThread 2 | Analysis1, Analysis2, Analysis3 | 25 to 49\r\nThread 3 | Analysis1, Analysis2, Analysis3 | 50 to 74\r\nThread 4 | Analysis1, Analysis2, Analysis3 | 75 to 99\r\n\r\nThis should allow to minimize the number of reads from disk, as each analysis job in the same thread will gather relevant properties from arrays cached in the memory ~~but benchmarking is still needed~~. I tried to run some benchmarks on the 250k atoms system below but did not find any performance improvement between two single parallel runs and one batch parallel run.\r\n\r\n### To do\r\n- [x] Implement a `run(parallel=True)` method in the `AnalysisBase` class so that single analysis jobs can be parallelized without using the `ParallelProcessor` class.\r\n- [ ] Add documentation to modules, classes and methods once the design of the class is stable.\r\n- [x] Print progressbar to stderr so that stdout can be redirected to file\r\n- [x] Create a progressbar with percentage of completion, configuration counter and a ETA estimate.\r\n\r\n### Benchmarks\r\nBenchmark performed on a system of N=250 k atoms (294 configurations, size ~1 GB).\r\n\r\n![scaling](https://cloud.githubusercontent.com/assets/12560118/12295747/5397f0e8-ba03-11e5-9252-179e3981564a.png)\r\n\r\nEfficiency computed as the ratio between core speed (cfg s<sup>-1</sup> threads<sup>-1</sup>) and the speed of a serial analysis.\r\n\r\n### Compatibility issues\r\nDue to the implementation of the parallelization, `AnalysisBase` needs to take a Universe object as argument."},"comment":{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/comments/173538672","html_url":"https://github.com/MDAnalysis/mdanalysis/pull/618#issuecomment-173538672","issue_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618","id":173538672,"user":{"login":"richardjgowers","id":9249543,"avatar_url":"https://avatars.githubusercontent.com/u/9249543?v=3","gravatar_id":"","url":"https://api.github.com/users/richardjgowers","html_url":"https://github.com/richardjgowers","followers_url":"https://api.github.com/users/richardjgowers/followers","following_url":"https://api.github.com/users/richardjgowers/following{/other_user}","gists_url":"https://api.github.com/users/richardjgowers/gists{/gist_id}","starred_url":"https://api.github.com/users/richardjgowers/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/richardjgowers/subscriptions","organizations_url":"https://api.github.com/users/richardjgowers/orgs","repos_url":"https://api.github.com/users/richardjgowers/repos","events_url":"https://api.github.com/users/richardjgowers/events{/privacy}","received_events_url":"https://api.github.com/users/richardjgowers/received_events","type":"User","site_admin":false},"created_at":"2016-01-21T11:03:26Z","updated_at":"2016-01-21T11:03:26Z","body":"This is looking pretty close to being mergeable, I think everything's stable enough for you to write docstrings now.\r\n\r\nThe big problem with it that we've not discussed is it only works with pristine Universes.. so for example\r\n``` python\r\nu = mda.Universe(PSF, DCD)\r\n\r\nu.atoms[:100].names = 'ABCD'\r\n\r\na = AnalysisThing(selstr='name ABCD')\r\na.run(parallel=True)\r\n```\r\nSo the problem here is that when the other treads create a Universe they won't have the modified name `ABCD`.  But I think as long as we put enough warnings on everything and make it clear this is still a little experimental, this should be fine.\r\n\r\nI think we'll be able to fix the above problem soon however with #363 \r\n\r\n"}}