{"action":"created","comment":{"url":"https://api.github.com/repos/dask/dask-jobqueue/pulls/comments/199177882","pull_request_review_id":133234714,"id":199177882,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDE5OTE3Nzg4Mg==","diff_hunk":"@@ -2,74 +2,162 @@\n Dask-Jobqueue\n =============\n \n-*Easy deployment of Dask Distributed on job queuing systems like\n-PBS, Slurm, and SGE.*\n+*Easily deploy Dask on job queuing systems like PBS, Slurm, MOAB, and SGE.*\n \n-Motivation\n-----------\n \n-1. While ``dask.distributed`` offers a flexible library for distributed computing\n-   in Python, it is not always easy to deploy on systems that use job queuing\n-   systems. Dask-jobqueue provides a Pythonic interface for deploying and\n-   managing Dask clusters.\n-2. In practice, deploying distributed requires customization, both for the\n-   machine(s) that it will deployed on and for the specific application it will\n-   be deployed for. Dask-jobqueue provides users with an intuitive interface for\n-   customizing dask clusters.\n+The Dask-jobqueue project makes it easy to deploy Dask on common job queuing\n+systems typically found in high performance supercomputers, academic research\n+institutions, and other clusters.  It provides a convenient interface that is\n+accessible from interactive systems like Jupyter notebooks, or batch jobs.\n \n Example\n -------\n \n .. code-block:: python\n \n    from dask_jobqueue import PBSCluster\n+   cluster = PBSCluster()\n+   cluster.scale(10)         # Ask for ten worker nodes\n \n-   cluster = PBSCluster(processes=6, threads=4, memory=\"16GB\")\n-   cluster.start_workers(10)\n+   # wait for jobs to arrive, depending on the queue, this may take some time\n \n    from dask.distributed import Client\n-   client = Client(cluster)\n+   client = Client(cluster)  # Connect Dask to the worker nodes\n \n-See :doc:`Examples <examples>` for more real-world examples.\n+   import dask.array as da\n+   x = ...                   # Dask commands now use these distributed resources\n \n \n Adaptivity\n ----------\n \n-This can also adapt the cluster size dynamically based on current load.\n-This helps to scale up the cluster when necessary but scale it down and save\n-resources when not actively computing.\n+Dask jobqueue can also adapt the cluster size dynamically based on current\n+load.  This helps to scale up the cluster when necessary but scale it down and\n+save resources when not actively computing.\n \n .. code-block:: python\n \n    cluster.adapt(minimum=1, maximum=100)\n \n+\n+Configuration\n+-------------\n+\n+Dask-jobqueue should be configured to your cluster so that it knows how many\n+resources to request of each node and how to break up those resources.  You can","path":"docs/index.rst","position":65,"original_position":65,"commit_id":"525b901ec2b145397b6e2d2febd5929869bd7d5c","original_commit_id":"525b901ec2b145397b6e2d2febd5929869bd7d5c","user":{"login":"guillaumeeb","id":17138587,"node_id":"MDQ6VXNlcjE3MTM4NTg3","avatar_url":"https://avatars0.githubusercontent.com/u/17138587?v=4","gravatar_id":"","url":"https://api.github.com/users/guillaumeeb","html_url":"https://github.com/guillaumeeb","followers_url":"https://api.github.com/users/guillaumeeb/followers","following_url":"https://api.github.com/users/guillaumeeb/following{/other_user}","gists_url":"https://api.github.com/users/guillaumeeb/gists{/gist_id}","starred_url":"https://api.github.com/users/guillaumeeb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/guillaumeeb/subscriptions","organizations_url":"https://api.github.com/users/guillaumeeb/orgs","repos_url":"https://api.github.com/users/guillaumeeb/repos","events_url":"https://api.github.com/users/guillaumeeb/events{/privacy}","received_events_url":"https://api.github.com/users/guillaumeeb/received_events","type":"User","site_admin":false},"body":"Again the node vs job point of view. Not sure what we should say. At least mention the term job somewhere?","created_at":"2018-06-29T14:30:32Z","updated_at":"2018-06-29T14:36:32Z","html_url":"https://github.com/dask/dask-jobqueue/pull/86#discussion_r199177882","pull_request_url":"https://api.github.com/repos/dask/dask-jobqueue/pulls/86","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/dask/dask-jobqueue/pulls/comments/199177882"},"html":{"href":"https://github.com/dask/dask-jobqueue/pull/86#discussion_r199177882"},"pull_request":{"href":"https://api.github.com/repos/dask/dask-jobqueue/pulls/86"}}},"pull_request":{"url":"https://api.github.com/repos/dask/dask-jobqueue/pulls/86","id":198099781,"node_id":"MDExOlB1bGxSZXF1ZXN0MTk4MDk5Nzgx","html_url":"https://github.com/dask/dask-jobqueue/pull/86","diff_url":"https://github.com/dask/dask-jobqueue/pull/86.diff","patch_url":"https://github.com/dask/dask-jobqueue/pull/86.patch","issue_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/86","number":86,"state":"open","locked":false,"title":"Add cores and total_memory parameters","user":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"body":"These overlap with (and might possibly replace) the threads and memory\r\nparameters.  I believe that they are more consistent with typcial use.\r\n\r\nNow we ask people to specify the cores and total memory of a node.  This\r\nshould remain relatively static over time.  Then we allow them to tune\r\nprocesses as they like independently of other parameters.\r\n\r\n### Example\r\n\r\n```python\r\n# Before\r\ncluster = PBSCluster(threads=4, processes=9, memory='10GB')\r\ncluster = PBSCluster(threads=12, processes=3, memory='30GB')\r\n```\r\n```python\r\n# After\r\ncluster = PBSCluster(cores=36, total_memory='90GB', processes=9)\r\ncluster = PBSCluster(cores=36, total_memory='90GB', processes=3)\r\n# or just\r\ncluster = PBSCluster(cores=36, total_memory='90GB')\r\n```\r\n\r\n\r\nI've found that novices are more comfortable answering the questions on the bottom rather than the questions on the top.  Answering the questions on the top means that you already need to understand how dask-worker manages its keywords.  ","created_at":"2018-06-28T17:27:05Z","updated_at":"2018-06-29T14:36:32Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8dd0a0d557e710ffbc567413aac77beab9369af5","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/dask/dask-jobqueue/pulls/86/commits","review_comments_url":"https://api.github.com/repos/dask/dask-jobqueue/pulls/86/comments","review_comment_url":"https://api.github.com/repos/dask/dask-jobqueue/pulls/comments{/number}","comments_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/86/comments","statuses_url":"https://api.github.com/repos/dask/dask-jobqueue/statuses/525b901ec2b145397b6e2d2febd5929869bd7d5c","head":{"label":"mrocklin:cores-total-memory","ref":"cores-total-memory","sha":"525b901ec2b145397b6e2d2febd5929869bd7d5c","user":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"repo":{"id":126092311,"node_id":"MDEwOlJlcG9zaXRvcnkxMjYwOTIzMTE=","name":"dask-jobqueue","full_name":"mrocklin/dask-jobqueue","owner":{"login":"mrocklin","id":306380,"node_id":"MDQ6VXNlcjMwNjM4MA==","avatar_url":"https://avatars3.githubusercontent.com/u/306380?v=4","gravatar_id":"","url":"https://api.github.com/users/mrocklin","html_url":"https://github.com/mrocklin","followers_url":"https://api.github.com/users/mrocklin/followers","following_url":"https://api.github.com/users/mrocklin/following{/other_user}","gists_url":"https://api.github.com/users/mrocklin/gists{/gist_id}","starred_url":"https://api.github.com/users/mrocklin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mrocklin/subscriptions","organizations_url":"https://api.github.com/users/mrocklin/orgs","repos_url":"https://api.github.com/users/mrocklin/repos","events_url":"https://api.github.com/users/mrocklin/events{/privacy}","received_events_url":"https://api.github.com/users/mrocklin/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/mrocklin/dask-jobqueue","description":"Deploy Dask on job schedulers like PBS and SLURM","fork":true,"url":"https://api.github.com/repos/mrocklin/dask-jobqueue","forks_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/forks","keys_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/teams","hooks_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/hooks","issue_events_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/issues/events{/number}","events_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/events","assignees_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/assignees{/user}","branches_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/branches{/branch}","tags_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/tags","blobs_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/git/refs{/sha}","trees_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/statuses/{sha}","languages_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/languages","stargazers_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/stargazers","contributors_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/contributors","subscribers_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/subscribers","subscription_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/subscription","commits_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/commits{/sha}","git_commits_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/git/commits{/sha}","comments_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/comments{/number}","issue_comment_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/issues/comments{/number}","contents_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/contents/{+path}","compare_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/merges","archive_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/downloads","issues_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/issues{/number}","pulls_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/pulls{/number}","milestones_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/milestones{/number}","notifications_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/labels{/name}","releases_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/releases{/id}","deployments_url":"https://api.github.com/repos/mrocklin/dask-jobqueue/deployments","created_at":"2018-03-20T22:46:49Z","updated_at":"2018-03-20T22:46:51Z","pushed_at":"2018-06-28T18:11:08Z","git_url":"git://github.com/mrocklin/dask-jobqueue.git","ssh_url":"git@github.com:mrocklin/dask-jobqueue.git","clone_url":"https://github.com/mrocklin/dask-jobqueue.git","svn_url":"https://github.com/mrocklin/dask-jobqueue","homepage":null,"size":143,"stargazers_count":0,"watchers_count":0,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"dask:master","ref":"master","sha":"d1cd730cd3da7b3b21023596a89516ca957b85d4","user":{"login":"dask","id":17131925,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE3MTMxOTI1","avatar_url":"https://avatars1.githubusercontent.com/u/17131925?v=4","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"repo":{"id":123497963,"node_id":"MDEwOlJlcG9zaXRvcnkxMjM0OTc5NjM=","name":"dask-jobqueue","full_name":"dask/dask-jobqueue","owner":{"login":"dask","id":17131925,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE3MTMxOTI1","avatar_url":"https://avatars1.githubusercontent.com/u/17131925?v=4","gravatar_id":"","url":"https://api.github.com/users/dask","html_url":"https://github.com/dask","followers_url":"https://api.github.com/users/dask/followers","following_url":"https://api.github.com/users/dask/following{/other_user}","gists_url":"https://api.github.com/users/dask/gists{/gist_id}","starred_url":"https://api.github.com/users/dask/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dask/subscriptions","organizations_url":"https://api.github.com/users/dask/orgs","repos_url":"https://api.github.com/users/dask/repos","events_url":"https://api.github.com/users/dask/events{/privacy}","received_events_url":"https://api.github.com/users/dask/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/dask/dask-jobqueue","description":"Deploy Dask on job schedulers like PBS, SLURM, and SGE","fork":false,"url":"https://api.github.com/repos/dask/dask-jobqueue","forks_url":"https://api.github.com/repos/dask/dask-jobqueue/forks","keys_url":"https://api.github.com/repos/dask/dask-jobqueue/keys{/key_id}","collaborators_url":"https://api.github.com/repos/dask/dask-jobqueue/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/dask/dask-jobqueue/teams","hooks_url":"https://api.github.com/repos/dask/dask-jobqueue/hooks","issue_events_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/events{/number}","events_url":"https://api.github.com/repos/dask/dask-jobqueue/events","assignees_url":"https://api.github.com/repos/dask/dask-jobqueue/assignees{/user}","branches_url":"https://api.github.com/repos/dask/dask-jobqueue/branches{/branch}","tags_url":"https://api.github.com/repos/dask/dask-jobqueue/tags","blobs_url":"https://api.github.com/repos/dask/dask-jobqueue/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/dask/dask-jobqueue/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/dask/dask-jobqueue/git/refs{/sha}","trees_url":"https://api.github.com/repos/dask/dask-jobqueue/git/trees{/sha}","statuses_url":"https://api.github.com/repos/dask/dask-jobqueue/statuses/{sha}","languages_url":"https://api.github.com/repos/dask/dask-jobqueue/languages","stargazers_url":"https://api.github.com/repos/dask/dask-jobqueue/stargazers","contributors_url":"https://api.github.com/repos/dask/dask-jobqueue/contributors","subscribers_url":"https://api.github.com/repos/dask/dask-jobqueue/subscribers","subscription_url":"https://api.github.com/repos/dask/dask-jobqueue/subscription","commits_url":"https://api.github.com/repos/dask/dask-jobqueue/commits{/sha}","git_commits_url":"https://api.github.com/repos/dask/dask-jobqueue/git/commits{/sha}","comments_url":"https://api.github.com/repos/dask/dask-jobqueue/comments{/number}","issue_comment_url":"https://api.github.com/repos/dask/dask-jobqueue/issues/comments{/number}","contents_url":"https://api.github.com/repos/dask/dask-jobqueue/contents/{+path}","compare_url":"https://api.github.com/repos/dask/dask-jobqueue/compare/{base}...{head}","merges_url":"https://api.github.com/repos/dask/dask-jobqueue/merges","archive_url":"https://api.github.com/repos/dask/dask-jobqueue/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/dask/dask-jobqueue/downloads","issues_url":"https://api.github.com/repos/dask/dask-jobqueue/issues{/number}","pulls_url":"https://api.github.com/repos/dask/dask-jobqueue/pulls{/number}","milestones_url":"https://api.github.com/repos/dask/dask-jobqueue/milestones{/number}","notifications_url":"https://api.github.com/repos/dask/dask-jobqueue/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/dask/dask-jobqueue/labels{/name}","releases_url":"https://api.github.com/repos/dask/dask-jobqueue/releases{/id}","deployments_url":"https://api.github.com/repos/dask/dask-jobqueue/deployments","created_at":"2018-03-01T22:07:33Z","updated_at":"2018-06-29T02:38:55Z","pushed_at":"2018-06-29T02:38:53Z","git_url":"git://github.com/dask/dask-jobqueue.git","ssh_url":"git@github.com:dask/dask-jobqueue.git","clone_url":"https://github.com/dask/dask-jobqueue.git","svn_url":"https://github.com/dask/dask-jobqueue","homepage":"https://dask-jobqueue.readthedocs.io","size":133,"stargazers_count":36,"watchers_count":36,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":20,"mirror_url":null,"archived":false,"open_issues_count":22,"license":{"key":"other","name":"Other","spdx_id":null,"url":null,"node_id":"MDc6TGljZW5zZTA="},"forks":20,"open_issues":22,"watchers":36,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/dask/dask-jobqueue/pulls/86"},"html":{"href":"https://github.com/dask/dask-jobqueue/pull/86"},"issue":{"href":"https://api.github.com/repos/dask/dask-jobqueue/issues/86"},"comments":{"href":"https://api.github.com/repos/dask/dask-jobqueue/issues/86/comments"},"review_comments":{"href":"https://api.github.com/repos/dask/dask-jobqueue/pulls/86/comments"},"review_comment":{"href":"https://api.github.com/repos/dask/dask-jobqueue/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/dask/dask-jobqueue/pulls/86/commits"},"statuses":{"href":"https://api.github.com/repos/dask/dask-jobqueue/statuses/525b901ec2b145397b6e2d2febd5929869bd7d5c"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"guillaumeeb"}}