{"action":"created","comment":{"url":"https://api.github.com/repos/citusdata/citus_docs/pulls/comments/172674970","pull_request_review_id":101733897,"id":172674970,"diff_hunk":"@@ -115,7 +115,13 @@ From a throughput standpoint, you can expect data ingest ratios of 250K - 2M row\n \n .. note::\n \n-    To avoid opening too many connections to worker nodes, we recommend running only two COPY commands on a distributed table at a time. In practice, running more than four at a time rarely results in performance benefits. An exception is when all the data in the ingested file has a specific partition key value, which goes into a single shard. COPY will only open connections to shards when necessary.\n+  Make sure your benchmarking setup is well configured so you can observe optimal COPY performance. Follow these tips:\n+\n+  * We recommend a large batch size (~ 50000-100000). You can benchmark with multiple files (1, 10, 1000, 10000 etc), each of that batch size.\n+  * Use parallel ingestion. Increase the number of threads/ingestors to 2, 4, 8, 16 and run benchmarks.\n+  * Use a compute-optimized coordinator, 8-16 vcpus. For the workers choose memory-optimized boxes with a decent number of vcpus (4-8).","path":"performance/scaling_data_ingestion.rst","position":9,"original_position":9,"commit_id":"74009651d7efefbbe182a2b00de2c1c14c424245","original_commit_id":"74009651d7efefbbe182a2b00de2c1c14c424245","user":{"login":"begriffs","id":911911,"avatar_url":"https://avatars2.githubusercontent.com/u/911911?v=4","gravatar_id":"","url":"https://api.github.com/users/begriffs","html_url":"https://github.com/begriffs","followers_url":"https://api.github.com/users/begriffs/followers","following_url":"https://api.github.com/users/begriffs/following{/other_user}","gists_url":"https://api.github.com/users/begriffs/gists{/gist_id}","starred_url":"https://api.github.com/users/begriffs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/begriffs/subscriptions","organizations_url":"https://api.github.com/users/begriffs/orgs","repos_url":"https://api.github.com/users/begriffs/repos","events_url":"https://api.github.com/users/begriffs/events{/privacy}","received_events_url":"https://api.github.com/users/begriffs/received_events","type":"User","site_admin":false},"body":"Just leave it at compute-optimized for coordinator and memory-optimized for workers?","created_at":"2018-03-06T22:00:40Z","updated_at":"2018-03-06T22:00:40Z","html_url":"https://github.com/citusdata/citus_docs/pull/625#discussion_r172674970","pull_request_url":"https://api.github.com/repos/citusdata/citus_docs/pulls/625","author_association":"COLLABORATOR","_links":{"self":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/comments/172674970"},"html":{"href":"https://github.com/citusdata/citus_docs/pull/625#discussion_r172674970"},"pull_request":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/625"}},"in_reply_to_id":172673907},"pull_request":{"url":"https://api.github.com/repos/citusdata/citus_docs/pulls/625","id":173004171,"html_url":"https://github.com/citusdata/citus_docs/pull/625","diff_url":"https://github.com/citusdata/citus_docs/pull/625.diff","patch_url":"https://github.com/citusdata/citus_docs/pull/625.patch","issue_url":"https://api.github.com/repos/citusdata/citus_docs/issues/625","number":625,"state":"open","locked":false,"title":"Include tips for seeing good COPY performance","user":{"login":"begriffs","id":911911,"avatar_url":"https://avatars2.githubusercontent.com/u/911911?v=4","gravatar_id":"","url":"https://api.github.com/users/begriffs","html_url":"https://github.com/begriffs","followers_url":"https://api.github.com/users/begriffs/followers","following_url":"https://api.github.com/users/begriffs/following{/other_user}","gists_url":"https://api.github.com/users/begriffs/gists{/gist_id}","starred_url":"https://api.github.com/users/begriffs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/begriffs/subscriptions","organizations_url":"https://api.github.com/users/begriffs/orgs","repos_url":"https://api.github.com/users/begriffs/repos","events_url":"https://api.github.com/users/begriffs/events{/privacy}","received_events_url":"https://api.github.com/users/begriffs/received_events","type":"User","site_admin":false},"body":"Fixes #599 ","created_at":"2018-03-05T22:22:56Z","updated_at":"2018-03-06T22:00:40Z","closed_at":null,"merged_at":null,"merge_commit_sha":"819442832e644b56c10a0c1aeb09e38f7d3da859","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":377923169,"url":"https://api.github.com/repos/citusdata/citus_docs/labels/in%20progress","name":"in progress","color":"ededed","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/citusdata/citus_docs/pulls/625/commits","review_comments_url":"https://api.github.com/repos/citusdata/citus_docs/pulls/625/comments","review_comment_url":"https://api.github.com/repos/citusdata/citus_docs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/citusdata/citus_docs/issues/625/comments","statuses_url":"https://api.github.com/repos/citusdata/citus_docs/statuses/74009651d7efefbbe182a2b00de2c1c14c424245","head":{"label":"citusdata:benchmark-tips","ref":"benchmark-tips","sha":"74009651d7efefbbe182a2b00de2c1c14c424245","user":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars2.githubusercontent.com/u/2545424?v=4","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"repo":{"id":56884660,"name":"citus_docs","full_name":"citusdata/citus_docs","owner":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars2.githubusercontent.com/u/2545424?v=4","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/citusdata/citus_docs","description":"Citus Documentation","fork":false,"url":"https://api.github.com/repos/citusdata/citus_docs","forks_url":"https://api.github.com/repos/citusdata/citus_docs/forks","keys_url":"https://api.github.com/repos/citusdata/citus_docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/citusdata/citus_docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/citusdata/citus_docs/teams","hooks_url":"https://api.github.com/repos/citusdata/citus_docs/hooks","issue_events_url":"https://api.github.com/repos/citusdata/citus_docs/issues/events{/number}","events_url":"https://api.github.com/repos/citusdata/citus_docs/events","assignees_url":"https://api.github.com/repos/citusdata/citus_docs/assignees{/user}","branches_url":"https://api.github.com/repos/citusdata/citus_docs/branches{/branch}","tags_url":"https://api.github.com/repos/citusdata/citus_docs/tags","blobs_url":"https://api.github.com/repos/citusdata/citus_docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/citusdata/citus_docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/citusdata/citus_docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/citusdata/citus_docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/citusdata/citus_docs/statuses/{sha}","languages_url":"https://api.github.com/repos/citusdata/citus_docs/languages","stargazers_url":"https://api.github.com/repos/citusdata/citus_docs/stargazers","contributors_url":"https://api.github.com/repos/citusdata/citus_docs/contributors","subscribers_url":"https://api.github.com/repos/citusdata/citus_docs/subscribers","subscription_url":"https://api.github.com/repos/citusdata/citus_docs/subscription","commits_url":"https://api.github.com/repos/citusdata/citus_docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/citusdata/citus_docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/citusdata/citus_docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/citusdata/citus_docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/citusdata/citus_docs/contents/{+path}","compare_url":"https://api.github.com/repos/citusdata/citus_docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/citusdata/citus_docs/merges","archive_url":"https://api.github.com/repos/citusdata/citus_docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/citusdata/citus_docs/downloads","issues_url":"https://api.github.com/repos/citusdata/citus_docs/issues{/number}","pulls_url":"https://api.github.com/repos/citusdata/citus_docs/pulls{/number}","milestones_url":"https://api.github.com/repos/citusdata/citus_docs/milestones{/number}","notifications_url":"https://api.github.com/repos/citusdata/citus_docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/citusdata/citus_docs/labels{/name}","releases_url":"https://api.github.com/repos/citusdata/citus_docs/releases{/id}","deployments_url":"https://api.github.com/repos/citusdata/citus_docs/deployments","created_at":"2016-04-22T20:54:52Z","updated_at":"2018-01-22T18:58:57Z","pushed_at":"2018-03-06T00:49:05Z","git_url":"git://github.com/citusdata/citus_docs.git","ssh_url":"git@github.com:citusdata/citus_docs.git","clone_url":"https://github.com/citusdata/citus_docs.git","svn_url":"https://github.com/citusdata/citus_docs","homepage":null,"size":11091,"stargazers_count":16,"watchers_count":16,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":16,"mirror_url":null,"archived":false,"open_issues_count":169,"license":null,"forks":16,"open_issues":169,"watchers":16,"default_branch":"master"}},"base":{"label":"citusdata:master","ref":"master","sha":"f90bf97e5776a09cba867bdb259f91eff0abc33a","user":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars2.githubusercontent.com/u/2545424?v=4","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"repo":{"id":56884660,"name":"citus_docs","full_name":"citusdata/citus_docs","owner":{"login":"citusdata","id":2545424,"avatar_url":"https://avatars2.githubusercontent.com/u/2545424?v=4","gravatar_id":"","url":"https://api.github.com/users/citusdata","html_url":"https://github.com/citusdata","followers_url":"https://api.github.com/users/citusdata/followers","following_url":"https://api.github.com/users/citusdata/following{/other_user}","gists_url":"https://api.github.com/users/citusdata/gists{/gist_id}","starred_url":"https://api.github.com/users/citusdata/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/citusdata/subscriptions","organizations_url":"https://api.github.com/users/citusdata/orgs","repos_url":"https://api.github.com/users/citusdata/repos","events_url":"https://api.github.com/users/citusdata/events{/privacy}","received_events_url":"https://api.github.com/users/citusdata/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/citusdata/citus_docs","description":"Citus Documentation","fork":false,"url":"https://api.github.com/repos/citusdata/citus_docs","forks_url":"https://api.github.com/repos/citusdata/citus_docs/forks","keys_url":"https://api.github.com/repos/citusdata/citus_docs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/citusdata/citus_docs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/citusdata/citus_docs/teams","hooks_url":"https://api.github.com/repos/citusdata/citus_docs/hooks","issue_events_url":"https://api.github.com/repos/citusdata/citus_docs/issues/events{/number}","events_url":"https://api.github.com/repos/citusdata/citus_docs/events","assignees_url":"https://api.github.com/repos/citusdata/citus_docs/assignees{/user}","branches_url":"https://api.github.com/repos/citusdata/citus_docs/branches{/branch}","tags_url":"https://api.github.com/repos/citusdata/citus_docs/tags","blobs_url":"https://api.github.com/repos/citusdata/citus_docs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/citusdata/citus_docs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/citusdata/citus_docs/git/refs{/sha}","trees_url":"https://api.github.com/repos/citusdata/citus_docs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/citusdata/citus_docs/statuses/{sha}","languages_url":"https://api.github.com/repos/citusdata/citus_docs/languages","stargazers_url":"https://api.github.com/repos/citusdata/citus_docs/stargazers","contributors_url":"https://api.github.com/repos/citusdata/citus_docs/contributors","subscribers_url":"https://api.github.com/repos/citusdata/citus_docs/subscribers","subscription_url":"https://api.github.com/repos/citusdata/citus_docs/subscription","commits_url":"https://api.github.com/repos/citusdata/citus_docs/commits{/sha}","git_commits_url":"https://api.github.com/repos/citusdata/citus_docs/git/commits{/sha}","comments_url":"https://api.github.com/repos/citusdata/citus_docs/comments{/number}","issue_comment_url":"https://api.github.com/repos/citusdata/citus_docs/issues/comments{/number}","contents_url":"https://api.github.com/repos/citusdata/citus_docs/contents/{+path}","compare_url":"https://api.github.com/repos/citusdata/citus_docs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/citusdata/citus_docs/merges","archive_url":"https://api.github.com/repos/citusdata/citus_docs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/citusdata/citus_docs/downloads","issues_url":"https://api.github.com/repos/citusdata/citus_docs/issues{/number}","pulls_url":"https://api.github.com/repos/citusdata/citus_docs/pulls{/number}","milestones_url":"https://api.github.com/repos/citusdata/citus_docs/milestones{/number}","notifications_url":"https://api.github.com/repos/citusdata/citus_docs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/citusdata/citus_docs/labels{/name}","releases_url":"https://api.github.com/repos/citusdata/citus_docs/releases{/id}","deployments_url":"https://api.github.com/repos/citusdata/citus_docs/deployments","created_at":"2016-04-22T20:54:52Z","updated_at":"2018-01-22T18:58:57Z","pushed_at":"2018-03-06T00:49:05Z","git_url":"git://github.com/citusdata/citus_docs.git","ssh_url":"git@github.com:citusdata/citus_docs.git","clone_url":"https://github.com/citusdata/citus_docs.git","svn_url":"https://github.com/citusdata/citus_docs","homepage":null,"size":11091,"stargazers_count":16,"watchers_count":16,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":16,"mirror_url":null,"archived":false,"open_issues_count":169,"license":null,"forks":16,"open_issues":169,"watchers":16,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/625"},"html":{"href":"https://github.com/citusdata/citus_docs/pull/625"},"issue":{"href":"https://api.github.com/repos/citusdata/citus_docs/issues/625"},"comments":{"href":"https://api.github.com/repos/citusdata/citus_docs/issues/625/comments"},"review_comments":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/625/comments"},"review_comment":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/citusdata/citus_docs/pulls/625/commits"},"statuses":{"href":"https://api.github.com/repos/citusdata/citus_docs/statuses/74009651d7efefbbe182a2b00de2c1c14c424245"}},"author_association":"COLLABORATOR"}}
{"actor":{"display_login":"begriffs"}}