{"action":"opened","number":618,"pull_request":{"url":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/618","id":55900459,"html_url":"https://github.com/MDAnalysis/mdanalysis/pull/618","diff_url":"https://github.com/MDAnalysis/mdanalysis/pull/618.diff","patch_url":"https://github.com/MDAnalysis/mdanalysis/pull/618.patch","issue_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618","number":618,"state":"open","locked":false,"title":"Parallel trajectory analysis (revisited)","user":{"login":"mattihappy","id":12560118,"avatar_url":"https://avatars.githubusercontent.com/u/12560118?v=3","gravatar_id":"","url":"https://api.github.com/users/mattihappy","html_url":"https://github.com/mattihappy","followers_url":"https://api.github.com/users/mattihappy/followers","following_url":"https://api.github.com/users/mattihappy/following{/other_user}","gists_url":"https://api.github.com/users/mattihappy/gists{/gist_id}","starred_url":"https://api.github.com/users/mattihappy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattihappy/subscriptions","organizations_url":"https://api.github.com/users/mattihappy/orgs","repos_url":"https://api.github.com/users/mattihappy/repos","events_url":"https://api.github.com/users/mattihappy/events{/privacy}","received_events_url":"https://api.github.com/users/mattihappy/received_events","type":"User","site_admin":false},"body":"With reference to #617 \r\nMDAnalysis Discussion --> [link ](https://groups.google.com/forum/#!topic/mdnalysis-discussion/MaX70t5RZ7Y)\r\nBranch --> [link](https://github.com/mattihappy/mdanalysis/tree/feature-ParallelAnalysis)\r\n\r\n# Aim\r\nSpeed up trajectory analysis by spreading the computation on multiple threads. Slices of the trajectory are assigned to each thread, which then proceeds running a batch of AnalysisBase istances.\r\n\r\n### Current syntax (may change)\r\nIn order to run a parallel analysis, one needs to setup a Universe object and a list of `AnalysisBase` objects. Such list is then passed to a istance of the `ParallelProcessor` class and run calling the method `parallel_run()`.\r\n\r\n```python\r\nimport MDAnalysis as mda\r\nfrom MDAnalysisTests.datafiles import DCD, PSF\r\nimport parallel_jobs  as pj\r\nimport electromagnetism # some analysis module\r\n\r\nuniverse = mda.Universe(PSF, DCD)\r\nselection_string = 'all'\r\n\r\njobs = [electromagnetism.TotalDipole(selection=selection_string)]\r\nprocess = pj.ParallelProcessor(jobs, universe, threads=4, start=1, stop=-1, step=1)\r\nprocess.parallel_run()\r\n```\r\n\r\n### Design thoughts\r\nGiven four threads and three different analysis objects, and a trajectory with 100 configurations, the workload is distributed as follows\r\n\r\nThread    | Task(s) | Frames\r\n------------ | ------------- | ------------- | \r\nThread 1 | Analysis1, Analysis2, Analysis3 |  0 to 24\r\nThread 2 | Analysis1, Analysis2, Analysis3 | 25 to 49\r\nThread 3 | Analysis1, Analysis2, Analysis3 | 50 to 74\r\nThread 4 | Analysis1, Analysis2, Analysis3 | 75 to 99\r\n\r\nThis should allow to minimize the number of reads from disk, as each analysis job in the same thread will gather relevant properties from arrays cached in the memory, but benchmarking is still needed.\r\n\r\n### To do\r\n- [ ] Implement a `parallel_run()` method in the `AnalysisBase` class so that single analysis jobs can be parallelized without using the `ParallelProcessor` class.\r\n- [ ] Add documentation to modules, classes and methods once the design of the class is stable.\r\n- [ ] Print progress counter to stderr so that stdout can be redirected to file\r\n- [ ] Make per core progress counter optional (breaks if the required number of lines is higher than those available on terminal) and output to stderr instead of stout\r\n\r\n### Benchmarks\r\nBenchmark performed on a system of N=250 k atoms (294 configurations, size ~1 GB).\r\n\r\n![scaling](https://cloud.githubusercontent.com/assets/12560118/12295747/5397f0e8-ba03-11e5-9252-179e3981564a.png)\r\n\r\nEfficiency computed as the ratio between core speed (cfg s<sup>-1</sup> threads<sup>-1</sup>) and the speed of a serial analysis.\r\n\r\n### Compatibility issues\r\nDue to the implementation of the parallelization, `AnalysisBase` needs to take a Universe object as argument.","created_at":"2016-01-13T14:00:11Z","updated_at":"2016-01-13T14:00:11Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/618/commits","review_comments_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/618/comments","review_comment_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/comments{/number}","comments_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618/comments","statuses_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/statuses/5fddd1e0fd1bddfe182e04a98994c8200967c18d","head":{"label":"mattihappy:feature-ParallelAnalysis","ref":"feature-ParallelAnalysis","sha":"5fddd1e0fd1bddfe182e04a98994c8200967c18d","user":{"login":"mattihappy","id":12560118,"avatar_url":"https://avatars.githubusercontent.com/u/12560118?v=3","gravatar_id":"","url":"https://api.github.com/users/mattihappy","html_url":"https://github.com/mattihappy","followers_url":"https://api.github.com/users/mattihappy/followers","following_url":"https://api.github.com/users/mattihappy/following{/other_user}","gists_url":"https://api.github.com/users/mattihappy/gists{/gist_id}","starred_url":"https://api.github.com/users/mattihappy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattihappy/subscriptions","organizations_url":"https://api.github.com/users/mattihappy/orgs","repos_url":"https://api.github.com/users/mattihappy/repos","events_url":"https://api.github.com/users/mattihappy/events{/privacy}","received_events_url":"https://api.github.com/users/mattihappy/received_events","type":"User","site_admin":false},"repo":{"id":49460666,"name":"mdanalysis","full_name":"mattihappy/mdanalysis","owner":{"login":"mattihappy","id":12560118,"avatar_url":"https://avatars.githubusercontent.com/u/12560118?v=3","gravatar_id":"","url":"https://api.github.com/users/mattihappy","html_url":"https://github.com/mattihappy","followers_url":"https://api.github.com/users/mattihappy/followers","following_url":"https://api.github.com/users/mattihappy/following{/other_user}","gists_url":"https://api.github.com/users/mattihappy/gists{/gist_id}","starred_url":"https://api.github.com/users/mattihappy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mattihappy/subscriptions","organizations_url":"https://api.github.com/users/mattihappy/orgs","repos_url":"https://api.github.com/users/mattihappy/repos","events_url":"https://api.github.com/users/mattihappy/events{/privacy}","received_events_url":"https://api.github.com/users/mattihappy/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mattihappy/mdanalysis","description":"MDAnalysis is a Python library to analyze molecular dynamics trajectories.","fork":true,"url":"https://api.github.com/repos/mattihappy/mdanalysis","forks_url":"https://api.github.com/repos/mattihappy/mdanalysis/forks","keys_url":"https://api.github.com/repos/mattihappy/mdanalysis/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mattihappy/mdanalysis/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mattihappy/mdanalysis/teams","hooks_url":"https://api.github.com/repos/mattihappy/mdanalysis/hooks","issue_events_url":"https://api.github.com/repos/mattihappy/mdanalysis/issues/events{/number}","events_url":"https://api.github.com/repos/mattihappy/mdanalysis/events","assignees_url":"https://api.github.com/repos/mattihappy/mdanalysis/assignees{/user}","branches_url":"https://api.github.com/repos/mattihappy/mdanalysis/branches{/branch}","tags_url":"https://api.github.com/repos/mattihappy/mdanalysis/tags","blobs_url":"https://api.github.com/repos/mattihappy/mdanalysis/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mattihappy/mdanalysis/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mattihappy/mdanalysis/git/refs{/sha}","trees_url":"https://api.github.com/repos/mattihappy/mdanalysis/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mattihappy/mdanalysis/statuses/{sha}","languages_url":"https://api.github.com/repos/mattihappy/mdanalysis/languages","stargazers_url":"https://api.github.com/repos/mattihappy/mdanalysis/stargazers","contributors_url":"https://api.github.com/repos/mattihappy/mdanalysis/contributors","subscribers_url":"https://api.github.com/repos/mattihappy/mdanalysis/subscribers","subscription_url":"https://api.github.com/repos/mattihappy/mdanalysis/subscription","commits_url":"https://api.github.com/repos/mattihappy/mdanalysis/commits{/sha}","git_commits_url":"https://api.github.com/repos/mattihappy/mdanalysis/git/commits{/sha}","comments_url":"https://api.github.com/repos/mattihappy/mdanalysis/comments{/number}","issue_comment_url":"https://api.github.com/repos/mattihappy/mdanalysis/issues/comments{/number}","contents_url":"https://api.github.com/repos/mattihappy/mdanalysis/contents/{+path}","compare_url":"https://api.github.com/repos/mattihappy/mdanalysis/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mattihappy/mdanalysis/merges","archive_url":"https://api.github.com/repos/mattihappy/mdanalysis/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mattihappy/mdanalysis/downloads","issues_url":"https://api.github.com/repos/mattihappy/mdanalysis/issues{/number}","pulls_url":"https://api.github.com/repos/mattihappy/mdanalysis/pulls{/number}","milestones_url":"https://api.github.com/repos/mattihappy/mdanalysis/milestones{/number}","notifications_url":"https://api.github.com/repos/mattihappy/mdanalysis/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mattihappy/mdanalysis/labels{/name}","releases_url":"https://api.github.com/repos/mattihappy/mdanalysis/releases{/id}","created_at":"2016-01-11T23:14:48Z","updated_at":"2016-01-11T23:14:55Z","pushed_at":"2016-01-13T12:37:58Z","git_url":"git://github.com/mattihappy/mdanalysis.git","ssh_url":"git@github.com:mattihappy/mdanalysis.git","clone_url":"https://github.com/mattihappy/mdanalysis.git","svn_url":"https://github.com/mattihappy/mdanalysis","homepage":"http://www.mdanalysis.org","size":218216,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"open_issues_count":0,"forks":0,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"MDAnalysis:develop","ref":"develop","sha":"8bb02d40e4a344f1ec1495e1ccad1dd9790bf095","user":{"login":"MDAnalysis","id":11445951,"avatar_url":"https://avatars.githubusercontent.com/u/11445951?v=3","gravatar_id":"","url":"https://api.github.com/users/MDAnalysis","html_url":"https://github.com/MDAnalysis","followers_url":"https://api.github.com/users/MDAnalysis/followers","following_url":"https://api.github.com/users/MDAnalysis/following{/other_user}","gists_url":"https://api.github.com/users/MDAnalysis/gists{/gist_id}","starred_url":"https://api.github.com/users/MDAnalysis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MDAnalysis/subscriptions","organizations_url":"https://api.github.com/users/MDAnalysis/orgs","repos_url":"https://api.github.com/users/MDAnalysis/repos","events_url":"https://api.github.com/users/MDAnalysis/events{/privacy}","received_events_url":"https://api.github.com/users/MDAnalysis/received_events","type":"Organization","site_admin":false},"repo":{"id":33389183,"name":"mdanalysis","full_name":"MDAnalysis/mdanalysis","owner":{"login":"MDAnalysis","id":11445951,"avatar_url":"https://avatars.githubusercontent.com/u/11445951?v=3","gravatar_id":"","url":"https://api.github.com/users/MDAnalysis","html_url":"https://github.com/MDAnalysis","followers_url":"https://api.github.com/users/MDAnalysis/followers","following_url":"https://api.github.com/users/MDAnalysis/following{/other_user}","gists_url":"https://api.github.com/users/MDAnalysis/gists{/gist_id}","starred_url":"https://api.github.com/users/MDAnalysis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MDAnalysis/subscriptions","organizations_url":"https://api.github.com/users/MDAnalysis/orgs","repos_url":"https://api.github.com/users/MDAnalysis/repos","events_url":"https://api.github.com/users/MDAnalysis/events{/privacy}","received_events_url":"https://api.github.com/users/MDAnalysis/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/MDAnalysis/mdanalysis","description":"MDAnalysis is a Python library to analyze molecular dynamics trajectories.","fork":false,"url":"https://api.github.com/repos/MDAnalysis/mdanalysis","forks_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/forks","keys_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/keys{/key_id}","collaborators_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/teams","hooks_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/hooks","issue_events_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/events{/number}","events_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/events","assignees_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/assignees{/user}","branches_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/branches{/branch}","tags_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/tags","blobs_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/git/refs{/sha}","trees_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/git/trees{/sha}","statuses_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/statuses/{sha}","languages_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/languages","stargazers_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/stargazers","contributors_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/contributors","subscribers_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/subscribers","subscription_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/subscription","commits_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/commits{/sha}","git_commits_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/git/commits{/sha}","comments_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/comments{/number}","issue_comment_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/comments{/number}","contents_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/contents/{+path}","compare_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/compare/{base}...{head}","merges_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/merges","archive_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/downloads","issues_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues{/number}","pulls_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls{/number}","milestones_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/milestones{/number}","notifications_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/labels{/name}","releases_url":"https://api.github.com/repos/MDAnalysis/mdanalysis/releases{/id}","created_at":"2015-04-04T00:39:41Z","updated_at":"2016-01-11T08:05:58Z","pushed_at":"2016-01-13T12:29:53Z","git_url":"git://github.com/MDAnalysis/mdanalysis.git","ssh_url":"git@github.com:MDAnalysis/mdanalysis.git","clone_url":"https://github.com/MDAnalysis/mdanalysis.git","svn_url":"https://github.com/MDAnalysis/mdanalysis","homepage":"http://www.mdanalysis.org","size":221169,"stargazers_count":48,"watchers_count":48,"language":"Python","has_issues":true,"has_downloads":false,"has_wiki":true,"has_pages":true,"forks_count":30,"mirror_url":null,"open_issues_count":80,"forks":30,"open_issues":80,"watchers":48,"default_branch":"develop"}},"_links":{"self":{"href":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/618"},"html":{"href":"https://github.com/MDAnalysis/mdanalysis/pull/618"},"issue":{"href":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618"},"comments":{"href":"https://api.github.com/repos/MDAnalysis/mdanalysis/issues/618/comments"},"review_comments":{"href":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/618/comments"},"review_comment":{"href":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/MDAnalysis/mdanalysis/pulls/618/commits"},"statuses":{"href":"https://api.github.com/repos/MDAnalysis/mdanalysis/statuses/5fddd1e0fd1bddfe182e04a98994c8200967c18d"}},"merged":false,"mergeable":null,"mergeable_state":"unknown","merged_by":null,"comments":0,"review_comments":0,"commits":3,"additions":315,"deletions":29,"changed_files":4}}