{"action":"created","comment":{"url":"https://api.github.com/repos/neo4j/neo4j/pulls/comments/76933050","id":76933050,"diff_hunk":"@@ -156,52 +160,40 @@ public void doImport( Input input ) throws IOException\n             // Stage 1 -- nodes, properties, labels\n             NodeStage nodeStage = new NodeStage( config, writeMonitor,\n                     nodes, idMapper, idGenerator, neoStore, inputCache, neoStore.getLabelScanStore(),\n-                    storeUpdateMonitor, memoryUsageStats );\n-\n-            // Stage 2 -- calculate dense node threshold\n-            CalculateDenseNodesStage calculateDenseNodesStage = new CalculateDenseNodesStage( config,\n-                    relationships, nodeRelationshipCache, idMapper, badCollector, inputCache, neoStore );\n-\n-            // Execute stages 1 and 2 in parallel or sequentially?\n+                    storeUpdateMonitor, nodeRelationshipCache, memoryUsageStats );\n+            executeStages( nodeStage );\n             if ( idMapper.needsPreparation() )\n-            {   // The id mapper of choice needs preparation in order to get ids from it,\n-                // So we need to execute the node stage first as it fills the id mapper and prepares it in the end,\n-                // before executing any stage that needs ids from the id mapper, for example calc dense node stage.\n-                executeStages( nodeStage );\n+            {\n                 executeStages( new IdMapperPreparationStage( config, idMapper, cachedNodes,\n                         badCollector, memoryUsageStats ) );\n                 PrimitiveLongIterator duplicateNodeIds = badCollector.leftOverDuplicateNodesIds();\n                 if ( duplicateNodeIds.hasNext() )\n                 {\n                     executeStages( new DeleteDuplicateNodesStage( config, duplicateNodeIds, neoStore ) );\n                 }\n-                executeStages( calculateDenseNodesStage );\n-            }\n-            else\n-            {   // The id mapper of choice doesn't need any preparation, so we can go ahead and execute\n-                // the node and calc dense node stages in parallel.\n-                executeStages( nodeStage, calculateDenseNodesStage );\n             }\n-            // At this point we know how many nodes we have, so we tell the cache that instead of having the\n-            // cache keeping track of that in a the face of concurrent updates.\n-            nodeRelationshipCache.setHighNodeId( neoStore.getNodeStore().getHighId() );\n+\n+            // Stage 2 -- calculate dense node threshold\n+            CalculateDenseNodesStage calculateDenseNodesStage = new CalculateDenseNodesStage(\n+                    withBatchSize( config, config.batchSize()*10 ),\n+                    relationships, nodeRelationshipCache, idMapper, badCollector, inputCache, neoStore );\n+            executeStages( calculateDenseNodesStage );\n \n             importRelationships( nodeRelationshipCache, storeUpdateMonitor, neoStore, writeMonitor,\n                     idMapper, cachedRelationships, inputCache,\n                     calculateDenseNodesStage.getRelationshipTypes( Long.MAX_VALUE ),\n-                    // Is batch size a good measure for considering a group of relationships a minority?\n-                    calculateDenseNodesStage.getRelationshipTypes( config.batchSize() ) );\n+                    calculateDenseNodesStage.getRelationshipTypes( 100 ) );","path":"community/kernel/src/main/java/org/neo4j/unsafe/impl/batchimport/ParallelBatchImporter.java","position":63,"original_position":63,"commit_id":"0f457f490db4219af9264b5b789caa42104b75af","original_commit_id":"0f457f490db4219af9264b5b789caa42104b75af","user":{"login":"tinwelint","id":187292,"avatar_url":"https://avatars.githubusercontent.com/u/187292?v=3","gravatar_id":"","url":"https://api.github.com/users/tinwelint","html_url":"https://github.com/tinwelint","followers_url":"https://api.github.com/users/tinwelint/followers","following_url":"https://api.github.com/users/tinwelint/following{/other_user}","gists_url":"https://api.github.com/users/tinwelint/gists{/gist_id}","starred_url":"https://api.github.com/users/tinwelint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tinwelint/subscriptions","organizations_url":"https://api.github.com/users/tinwelint/orgs","repos_url":"https://api.github.com/users/tinwelint/repos","events_url":"https://api.github.com/users/tinwelint/events{/privacy}","received_events_url":"https://api.github.com/users/tinwelint/received_events","type":"User","site_admin":false},"body":"This whole \"batch insert\" thing for the last couple of is a bit odd, it was introduced because we had a test highlighting this (creates 500 relationships, each of its own type) which took a long time to run after the initial per-type. Although there have been optimizations to the stages since so that the overhead of running the relationship import loop for small groups of relationships isn't as expensive anymore. Hmm, perhaps we should simply remove this thing. Will measure.","created_at":"2016-08-31T06:54:29Z","updated_at":"2016-08-31T06:54:29Z","html_url":"https://github.com/neo4j/neo4j/pull/7790#discussion_r76933050","pull_request_url":"https://api.github.com/repos/neo4j/neo4j/pulls/7790","_links":{"self":{"href":"https://api.github.com/repos/neo4j/neo4j/pulls/comments/76933050"},"html":{"href":"https://github.com/neo4j/neo4j/pull/7790#discussion_r76933050"},"pull_request":{"href":"https://api.github.com/repos/neo4j/neo4j/pulls/7790"}}},"pull_request":{"url":"https://api.github.com/repos/neo4j/neo4j/pulls/7790","id":82503355,"html_url":"https://github.com/neo4j/neo4j/pull/7790","diff_url":"https://github.com/neo4j/neo4j/pull/7790.diff","patch_url":"https://github.com/neo4j/neo4j/pull/7790.patch","issue_url":"https://api.github.com/repos/neo4j/neo4j/issues/7790","number":7790,"state":"open","locked":false,"title":"Revamped parallelism of parts of batch importer","user":{"login":"tinwelint","id":187292,"avatar_url":"https://avatars.githubusercontent.com/u/187292?v=3","gravatar_id":"","url":"https://api.github.com/users/tinwelint","html_url":"https://github.com/tinwelint","followers_url":"https://api.github.com/users/tinwelint/followers","following_url":"https://api.github.com/users/tinwelint/following{/other_user}","gists_url":"https://api.github.com/users/tinwelint/gists{/gist_id}","starred_url":"https://api.github.com/users/tinwelint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tinwelint/subscriptions","organizations_url":"https://api.github.com/users/tinwelint/orgs","repos_url":"https://api.github.com/users/tinwelint/repos","events_url":"https://api.github.com/users/tinwelint/events{/privacy}","received_events_url":"https://api.github.com/users/tinwelint/received_events","type":"User","site_admin":false},"body":"The main story here is composed of a couple of things:\r\n\r\n- a new ForkedProcessorStep which does parallelization inside each\r\n  batch, executed one by one. This to avoid difficulties parallelizing\r\n  some steps which has a costly section which isn't parallelizable.\r\n  With this new step items in a batch can be striped such that each\r\n  forked processor knows which parts to process.\r\n- better mechanical sympathy where most stages are optimized to work\r\n  with batch sizes matching pages in the page cache of the store they\r\n  (mainly) work with.\r\n- more static node->relationship caches, initialised after knowing how\r\n  many nodes have been imported. The static caches are faster than\r\n  the dynamically growing.\r\n\r\nThe forked processor simplifies a couple of stages, there are now\r\nno artificial additional steps for splitting or otherwise modify\r\nbatches to be better parallelizable. Also the whole stage scales\r\nbetter with added processors because the old way of parallelizing\r\nthose stages often involved a step which was single-threaded and\r\nacted as a divider-of-work. Such a step would often become the\r\nbottleneck in the end anyway.\r\n\r\nAbout mechanical sympathy the main problem previously was that reader\r\nand writer of stages which read from and wrote to the same store actually\r\ncontended on each other. Given the smaller batch size, there were\r\nmultiple batches of read records for any given page. Later in the stage\r\nwhere store was updated would often update the same page and so the\r\nreader (still reading that page) would need to do mych more retry-\r\nreading and so slow the whole stage down. Now with the aligned\r\nbatch sizes the reader doesn't contend with the writers in the\r\npage cache.\r\n\r\nAdditionally the main store updating step have been split into steps\r\nupdating entities and properties separately, this to have the\r\nentity updating able to go even faster.\r\n\r\nThe net result of this change as a whole should be that more often\r\nthe disk is the only main bottleneck. On test machines and development\r\nlaptops a 2x-3x performance improvement of the importer have\r\nbeen observed.\r\n\r\nchangelog: [3.0, kernel, import-tool] Faster import tool due to ability to better utilize all available resources and higher I/O saturation.","created_at":"2016-08-24T08:45:56Z","updated_at":"2016-08-31T06:54:29Z","closed_at":null,"merged_at":null,"merge_commit_sha":"12bb7c4aef5651ad12f12981647a60379c660927","assignee":{"login":"MishaDemianenko","id":12656787,"avatar_url":"https://avatars.githubusercontent.com/u/12656787?v=3","gravatar_id":"","url":"https://api.github.com/users/MishaDemianenko","html_url":"https://github.com/MishaDemianenko","followers_url":"https://api.github.com/users/MishaDemianenko/followers","following_url":"https://api.github.com/users/MishaDemianenko/following{/other_user}","gists_url":"https://api.github.com/users/MishaDemianenko/gists{/gist_id}","starred_url":"https://api.github.com/users/MishaDemianenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MishaDemianenko/subscriptions","organizations_url":"https://api.github.com/users/MishaDemianenko/orgs","repos_url":"https://api.github.com/users/MishaDemianenko/repos","events_url":"https://api.github.com/users/MishaDemianenko/events{/privacy}","received_events_url":"https://api.github.com/users/MishaDemianenko/received_events","type":"User","site_admin":false},"assignees":[{"login":"MishaDemianenko","id":12656787,"avatar_url":"https://avatars.githubusercontent.com/u/12656787?v=3","gravatar_id":"","url":"https://api.github.com/users/MishaDemianenko","html_url":"https://github.com/MishaDemianenko","followers_url":"https://api.github.com/users/MishaDemianenko/followers","following_url":"https://api.github.com/users/MishaDemianenko/following{/other_user}","gists_url":"https://api.github.com/users/MishaDemianenko/gists{/gist_id}","starred_url":"https://api.github.com/users/MishaDemianenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MishaDemianenko/subscriptions","organizations_url":"https://api.github.com/users/MishaDemianenko/orgs","repos_url":"https://api.github.com/users/MishaDemianenko/repos","events_url":"https://api.github.com/users/MishaDemianenko/events{/privacy}","received_events_url":"https://api.github.com/users/MishaDemianenko/received_events","type":"User","site_admin":false}],"milestone":null,"commits_url":"https://api.github.com/repos/neo4j/neo4j/pulls/7790/commits","review_comments_url":"https://api.github.com/repos/neo4j/neo4j/pulls/7790/comments","review_comment_url":"https://api.github.com/repos/neo4j/neo4j/pulls/comments{/number}","comments_url":"https://api.github.com/repos/neo4j/neo4j/issues/7790/comments","statuses_url":"https://api.github.com/repos/neo4j/neo4j/statuses/0f457f490db4219af9264b5b789caa42104b75af","head":{"label":"tinwelint:3.0-pbi-optimizations","ref":"3.0-pbi-optimizations","sha":"0f457f490db4219af9264b5b789caa42104b75af","user":{"login":"tinwelint","id":187292,"avatar_url":"https://avatars.githubusercontent.com/u/187292?v=3","gravatar_id":"","url":"https://api.github.com/users/tinwelint","html_url":"https://github.com/tinwelint","followers_url":"https://api.github.com/users/tinwelint/followers","following_url":"https://api.github.com/users/tinwelint/following{/other_user}","gists_url":"https://api.github.com/users/tinwelint/gists{/gist_id}","starred_url":"https://api.github.com/users/tinwelint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tinwelint/subscriptions","organizations_url":"https://api.github.com/users/tinwelint/orgs","repos_url":"https://api.github.com/users/tinwelint/repos","events_url":"https://api.github.com/users/tinwelint/events{/privacy}","received_events_url":"https://api.github.com/users/tinwelint/received_events","type":"User","site_admin":false},"repo":{"id":6666980,"name":"neo4j","full_name":"tinwelint/neo4j","owner":{"login":"tinwelint","id":187292,"avatar_url":"https://avatars.githubusercontent.com/u/187292?v=3","gravatar_id":"","url":"https://api.github.com/users/tinwelint","html_url":"https://github.com/tinwelint","followers_url":"https://api.github.com/users/tinwelint/followers","following_url":"https://api.github.com/users/tinwelint/following{/other_user}","gists_url":"https://api.github.com/users/tinwelint/gists{/gist_id}","starred_url":"https://api.github.com/users/tinwelint/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tinwelint/subscriptions","organizations_url":"https://api.github.com/users/tinwelint/orgs","repos_url":"https://api.github.com/users/tinwelint/repos","events_url":"https://api.github.com/users/tinwelint/events{/privacy}","received_events_url":"https://api.github.com/users/tinwelint/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/tinwelint/neo4j","description":"","fork":true,"url":"https://api.github.com/repos/tinwelint/neo4j","forks_url":"https://api.github.com/repos/tinwelint/neo4j/forks","keys_url":"https://api.github.com/repos/tinwelint/neo4j/keys{/key_id}","collaborators_url":"https://api.github.com/repos/tinwelint/neo4j/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/tinwelint/neo4j/teams","hooks_url":"https://api.github.com/repos/tinwelint/neo4j/hooks","issue_events_url":"https://api.github.com/repos/tinwelint/neo4j/issues/events{/number}","events_url":"https://api.github.com/repos/tinwelint/neo4j/events","assignees_url":"https://api.github.com/repos/tinwelint/neo4j/assignees{/user}","branches_url":"https://api.github.com/repos/tinwelint/neo4j/branches{/branch}","tags_url":"https://api.github.com/repos/tinwelint/neo4j/tags","blobs_url":"https://api.github.com/repos/tinwelint/neo4j/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/tinwelint/neo4j/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/tinwelint/neo4j/git/refs{/sha}","trees_url":"https://api.github.com/repos/tinwelint/neo4j/git/trees{/sha}","statuses_url":"https://api.github.com/repos/tinwelint/neo4j/statuses/{sha}","languages_url":"https://api.github.com/repos/tinwelint/neo4j/languages","stargazers_url":"https://api.github.com/repos/tinwelint/neo4j/stargazers","contributors_url":"https://api.github.com/repos/tinwelint/neo4j/contributors","subscribers_url":"https://api.github.com/repos/tinwelint/neo4j/subscribers","subscription_url":"https://api.github.com/repos/tinwelint/neo4j/subscription","commits_url":"https://api.github.com/repos/tinwelint/neo4j/commits{/sha}","git_commits_url":"https://api.github.com/repos/tinwelint/neo4j/git/commits{/sha}","comments_url":"https://api.github.com/repos/tinwelint/neo4j/comments{/number}","issue_comment_url":"https://api.github.com/repos/tinwelint/neo4j/issues/comments{/number}","contents_url":"https://api.github.com/repos/tinwelint/neo4j/contents/{+path}","compare_url":"https://api.github.com/repos/tinwelint/neo4j/compare/{base}...{head}","merges_url":"https://api.github.com/repos/tinwelint/neo4j/merges","archive_url":"https://api.github.com/repos/tinwelint/neo4j/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/tinwelint/neo4j/downloads","issues_url":"https://api.github.com/repos/tinwelint/neo4j/issues{/number}","pulls_url":"https://api.github.com/repos/tinwelint/neo4j/pulls{/number}","milestones_url":"https://api.github.com/repos/tinwelint/neo4j/milestones{/number}","notifications_url":"https://api.github.com/repos/tinwelint/neo4j/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/tinwelint/neo4j/labels{/name}","releases_url":"https://api.github.com/repos/tinwelint/neo4j/releases{/id}","deployments_url":"https://api.github.com/repos/tinwelint/neo4j/deployments","created_at":"2012-11-13T07:24:49Z","updated_at":"2014-12-16T07:20:14Z","pushed_at":"2016-08-29T20:05:46Z","git_url":"git://github.com/tinwelint/neo4j.git","ssh_url":"git@github.com:tinwelint/neo4j.git","clone_url":"https://github.com/tinwelint/neo4j.git","svn_url":"https://github.com/tinwelint/neo4j","homepage":null,"size":308992,"stargazers_count":0,"watchers_count":0,"language":"Java","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"neo4j:3.0","ref":"3.0","sha":"128719ac490f5e8933358eeeb83ff0c48f30969f","user":{"login":"neo4j","id":201120,"avatar_url":"https://avatars.githubusercontent.com/u/201120?v=3","gravatar_id":"","url":"https://api.github.com/users/neo4j","html_url":"https://github.com/neo4j","followers_url":"https://api.github.com/users/neo4j/followers","following_url":"https://api.github.com/users/neo4j/following{/other_user}","gists_url":"https://api.github.com/users/neo4j/gists{/gist_id}","starred_url":"https://api.github.com/users/neo4j/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/neo4j/subscriptions","organizations_url":"https://api.github.com/users/neo4j/orgs","repos_url":"https://api.github.com/users/neo4j/repos","events_url":"https://api.github.com/users/neo4j/events{/privacy}","received_events_url":"https://api.github.com/users/neo4j/received_events","type":"Organization","site_admin":false},"repo":{"id":6650539,"name":"neo4j","full_name":"neo4j/neo4j","owner":{"login":"neo4j","id":201120,"avatar_url":"https://avatars.githubusercontent.com/u/201120?v=3","gravatar_id":"","url":"https://api.github.com/users/neo4j","html_url":"https://github.com/neo4j","followers_url":"https://api.github.com/users/neo4j/followers","following_url":"https://api.github.com/users/neo4j/following{/other_user}","gists_url":"https://api.github.com/users/neo4j/gists{/gist_id}","starred_url":"https://api.github.com/users/neo4j/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/neo4j/subscriptions","organizations_url":"https://api.github.com/users/neo4j/orgs","repos_url":"https://api.github.com/users/neo4j/repos","events_url":"https://api.github.com/users/neo4j/events{/privacy}","received_events_url":"https://api.github.com/users/neo4j/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/neo4j/neo4j","description":"Graphs for Everyone","fork":false,"url":"https://api.github.com/repos/neo4j/neo4j","forks_url":"https://api.github.com/repos/neo4j/neo4j/forks","keys_url":"https://api.github.com/repos/neo4j/neo4j/keys{/key_id}","collaborators_url":"https://api.github.com/repos/neo4j/neo4j/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/neo4j/neo4j/teams","hooks_url":"https://api.github.com/repos/neo4j/neo4j/hooks","issue_events_url":"https://api.github.com/repos/neo4j/neo4j/issues/events{/number}","events_url":"https://api.github.com/repos/neo4j/neo4j/events","assignees_url":"https://api.github.com/repos/neo4j/neo4j/assignees{/user}","branches_url":"https://api.github.com/repos/neo4j/neo4j/branches{/branch}","tags_url":"https://api.github.com/repos/neo4j/neo4j/tags","blobs_url":"https://api.github.com/repos/neo4j/neo4j/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/neo4j/neo4j/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/neo4j/neo4j/git/refs{/sha}","trees_url":"https://api.github.com/repos/neo4j/neo4j/git/trees{/sha}","statuses_url":"https://api.github.com/repos/neo4j/neo4j/statuses/{sha}","languages_url":"https://api.github.com/repos/neo4j/neo4j/languages","stargazers_url":"https://api.github.com/repos/neo4j/neo4j/stargazers","contributors_url":"https://api.github.com/repos/neo4j/neo4j/contributors","subscribers_url":"https://api.github.com/repos/neo4j/neo4j/subscribers","subscription_url":"https://api.github.com/repos/neo4j/neo4j/subscription","commits_url":"https://api.github.com/repos/neo4j/neo4j/commits{/sha}","git_commits_url":"https://api.github.com/repos/neo4j/neo4j/git/commits{/sha}","comments_url":"https://api.github.com/repos/neo4j/neo4j/comments{/number}","issue_comment_url":"https://api.github.com/repos/neo4j/neo4j/issues/comments{/number}","contents_url":"https://api.github.com/repos/neo4j/neo4j/contents/{+path}","compare_url":"https://api.github.com/repos/neo4j/neo4j/compare/{base}...{head}","merges_url":"https://api.github.com/repos/neo4j/neo4j/merges","archive_url":"https://api.github.com/repos/neo4j/neo4j/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/neo4j/neo4j/downloads","issues_url":"https://api.github.com/repos/neo4j/neo4j/issues{/number}","pulls_url":"https://api.github.com/repos/neo4j/neo4j/pulls{/number}","milestones_url":"https://api.github.com/repos/neo4j/neo4j/milestones{/number}","notifications_url":"https://api.github.com/repos/neo4j/neo4j/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/neo4j/neo4j/labels{/name}","releases_url":"https://api.github.com/repos/neo4j/neo4j/releases{/id}","deployments_url":"https://api.github.com/repos/neo4j/neo4j/deployments","created_at":"2012-11-12T08:46:15Z","updated_at":"2016-08-31T01:58:26Z","pushed_at":"2016-08-31T06:34:33Z","git_url":"git://github.com/neo4j/neo4j.git","ssh_url":"git@github.com:neo4j/neo4j.git","clone_url":"https://github.com/neo4j/neo4j.git","svn_url":"https://github.com/neo4j/neo4j","homepage":"http://neo4j.com","size":370235,"stargazers_count":2934,"watchers_count":2934,"language":"Java","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":970,"mirror_url":null,"open_issues_count":408,"forks":970,"open_issues":408,"watchers":2934,"default_branch":"3.1"}},"_links":{"self":{"href":"https://api.github.com/repos/neo4j/neo4j/pulls/7790"},"html":{"href":"https://github.com/neo4j/neo4j/pull/7790"},"issue":{"href":"https://api.github.com/repos/neo4j/neo4j/issues/7790"},"comments":{"href":"https://api.github.com/repos/neo4j/neo4j/issues/7790/comments"},"review_comments":{"href":"https://api.github.com/repos/neo4j/neo4j/pulls/7790/comments"},"review_comment":{"href":"https://api.github.com/repos/neo4j/neo4j/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/neo4j/neo4j/pulls/7790/commits"},"statuses":{"href":"https://api.github.com/repos/neo4j/neo4j/statuses/0f457f490db4219af9264b5b789caa42104b75af"}}}}
{"actor":{"display_login":"tinwelint"}}