{"push_id":1327065944,"size":1000,"distinct_size":2678,"ref":"refs/heads/3.1","head":"7b890cea4f0be50e697d6ba272edb6b74b84e373","before":"de0d924136f077ba345d65fdb30c186cd2579caa","commits":[{"sha":"bf0c0f810b3f561dbf997df80e43ec23139db352","author":{"name":"Mattias Persson","email":"96e4113efc379fb8ae6068ec263bc1ff29b17f7f@neotechnology.com"},"message":"Revamped parallelism of parts of batch importer\n\nThe main story here is composed of two things:\n\n- a new ForkedProcessorStep which does parallelization inside each\n  batch, executed one by one. This to avoid difficulties parallelizing\n  some steps which has a costly section which isn't parallelizable.\n  With this new step items in a batch can be striped such that each\n  forked processor knows which parts to process.\n- better mechanical sympathy where most stages are optimized to work\n  with batch sizes matching pages in the page cache of the store they\n  (mainly) work with.\n\nThe forked processor simplifies a couple of stages, there are now\nno artificial additional steps for splitting or otherwise modify\nbatches to be better parallelizable. Also the whole stage scales\nbetter with added processors because the old way of parallelizing\nthose stages often involved a step which was single-threaded and\nacted as a divider-of-work. Such a step would often become the\nbottleneck in the end anyway.\n\nAbout mechanical sympathy the main problem previously was that reader\nand writer of stages which read from and wrote to the same store actually\ncontended on each other. Given the smaller batch size, there were\nmultiple batches of read records for any given page. Later in the stage\nwhere store was updated would often update the same page and so the\nreader (still reading that page) would need to do mych more retry-\nreading and so slow the whole stage down. Now with the aligned\nbatch sizes the reader doesn't contend with the writers in the\npage cache.\n\nAdditionally the main store updating step have been split into steps\nupdating entities and properties separately, this to have the\nentity updating able to go even faster.\n\nThe net result of this change as a whole should be that more often\nthe disk is the only main bottleneck. On test machines and development\nlaptops a 2x-3x performance improvement of the importer have\nbeen observed.","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/bf0c0f810b3f561dbf997df80e43ec23139db352"},{"sha":"8c8bab882e9e3b5cfcb71b30e64d1cb5db967f18","author":{"name":"Mattias Persson","email":"96e4113efc379fb8ae6068ec263bc1ff29b17f7f@neotechnology.com"},"message":"More aggressive assignment of processors earlier\n\nwhich works better on machines with a lot of processors. Previously\nthe slow assignment would delay the full potential of the import since\nall system resources wouldn't be fully utilized until a certain amount\nof time into the import.\n\nAlso lowered the moving average size since now batch sizes often are\nbigger (aligning with page size). This has the effect of snappier\nand more changing average stats, but also a more responsive dynamic processor\nassignments which means that a bottleneck step will be detected sooner\nand assigned more processing power.","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/8c8bab882e9e3b5cfcb71b30e64d1cb5db967f18"},{"sha":"1762664ecd5bdd5904c7b9dbfe5abe3dbd10f6ea","author":{"name":"Mattias Persson","email":"96e4113efc379fb8ae6068ec263bc1ff29b17f7f@neotechnology.com"},"message":"Includes more information in overview print\n\nof the import tool:\n- number of processors in overview\n- peak memory usage (from caching) during the import","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/1762664ecd5bdd5904c7b9dbfe5abe3dbd10f6ea"},{"sha":"0f457f490db4219af9264b5b789caa42104b75af","author":{"name":"Mattias Persson","email":"96e4113efc379fb8ae6068ec263bc1ff29b17f7f@neotechnology.com"},"message":"Configurable page size in importer","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/0f457f490db4219af9264b5b789caa42104b75af"},{"sha":"ff8c082fa32d1061ec108ab0cd0caac3c159d7d3","author":{"name":"Andres Taylor","email":"c6cbfe878b4ee67646e7630734db67f536518d54@neotechnology.com"},"message":"Merge pull request #7763 from pontusmelke/3.0-gh-7634\n\nEscape when listing constraints containing `:`","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/ff8c082fa32d1061ec108ab0cd0caac3c159d7d3"},{"sha":"939013a3ed789dca2dd47c1a2274ab2f8f171775","author":{"name":"Andres Taylor","email":"c6cbfe878b4ee67646e7630734db67f536518d54@neotechnology.com"},"message":"Merge remote-tracking branch 'upstream/3.0' into 3.1","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/939013a3ed789dca2dd47c1a2274ab2f8f171775"},{"sha":"b353365f238187617b6b4f4f5203fec4dcdef46a","author":{"name":"Andres Taylor","email":"c6cbfe878b4ee67646e7630734db67f536518d54@neotechnology.com"},"message":"Merge pull request #7796 from pontusmelke/3.0-delete-filter\n\nAllow relationship types of deleted relationships","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/b353365f238187617b6b4f4f5203fec4dcdef46a"},{"sha":"401ab3656c3490150e28957ee541c9b91edbe531","author":{"name":"Andres Taylor","email":"c6cbfe878b4ee67646e7630734db67f536518d54@neotechnology.com"},"message":"Merge remote-tracking branch 'upstream/3.0' into 3.1","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/401ab3656c3490150e28957ee541c9b91edbe531"},{"sha":"5be716ef3a40046fdf9f667a1b5b6ff97c0e585f","author":{"name":"jjaderberg","email":"b36961393197c2619b052405b64371892b3fe442@gmail.com"},"message":"Merge pull request #7819 from mariascharin/2.2-fix-import-tool\n\nChanged ACTS_IN to ACTED_IN","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/5be716ef3a40046fdf9f667a1b5b6ff97c0e585f"},{"sha":"48bb9cc7fe6a5ed0e6f1b9938b4b45c3bcf34c01","author":{"name":"Mats Rydberg","email":"d92aa46dd55fa35d86891adb8bbe1cd74242d83d@neotechnology.com"},"message":"Merge pull request #7824 from henriknyman/3.1-fix-group-to-role-mapping-parser\n\nFix parsing of group-to-role mapping to support AD group names","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/48bb9cc7fe6a5ed0e6f1b9938b4b45c3bcf34c01"},{"sha":"7d7c49866f7c0e7fc5dd445cd79017243ebeab03","author":{"name":"Andres Taylor","email":"c6cbfe878b4ee67646e7630734db67f536518d54@neotechnology.com"},"message":"Turn off TCK tests until openCypher has been updated","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/7d7c49866f7c0e7fc5dd445cd79017243ebeab03"},{"sha":"f7a908026ba193b13a68fbdcfdc3422ca8887789","author":{"name":"Jonatan Jäderberg","email":"b36961393197c2619b052405b64371892b3fe442@neotechnology.com"},"message":"Merge remote-tracking branch 'upstream/2.2' into 2.3","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/f7a908026ba193b13a68fbdcfdc3422ca8887789"},{"sha":"8589ff38083253d547f47e339c34dda644268165","author":{"name":"Jonatan Jäderberg","email":"b36961393197c2619b052405b64371892b3fe442@neotechnology.com"},"message":"Merge remote-tracking branch 'upstream/2.3' into 3.0","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/8589ff38083253d547f47e339c34dda644268165"},{"sha":"9e1953306a5b8faab31f5599c1215addd4292c5b","author":{"name":"Craig Taverner","email":"16a7a05fad8a991a88499eabd88a68550b596989@amanzi.com"},"message":"Support returned point types as params to later queries","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/9e1953306a5b8faab31f5599c1215addd4292c5b"},{"sha":"f0bf4976cd4538b6682bca12d4ed75b93986e62b","author":{"name":"Craig Taverner","email":"16a7a05fad8a991a88499eabd88a68550b596989@amanzi.com"},"message":"Support points created as geometry type","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/f0bf4976cd4538b6682bca12d4ed75b93986e62b"},{"sha":"0320781f2ac1c080a159d669d30a7e8349aad418","author":{"name":"Craig Taverner","email":"16a7a05fad8a991a88499eabd88a68550b596989@amanzi.com"},"message":"Support use of public types returned from procedures\n\nIn particular we support procedures that return the public Point type and this\ncan then be used in the distance function which accepts the internal Point type.\nThe conversion is done with a similar mapper principle to that introduced for\ninternal to public type conversions in 079e1bd6e6aec2440527b94a3bbc4482256eb58d.","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/0320781f2ac1c080a159d669d30a7e8349aad418"},{"sha":"2ae7fc05a0088c16f3594ec5b660541f01438ff0","author":{"name":"Craig Taverner","email":"16a7a05fad8a991a88499eabd88a68550b596989@amanzi.com"},"message":"Nicer scala collection mapping for type conversion","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/2ae7fc05a0088c16f3594ec5b660541f01438ff0"},{"sha":"8358609913df177186beabad2fe8df618ebc7fca","author":{"name":"Jonatan Jäderberg","email":"b36961393197c2619b052405b64371892b3fe442@neotechnology.com"},"message":"Merge remote-tracking branch 'upstream/3.0' into 3.1","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/8358609913df177186beabad2fe8df618ebc7fca"},{"sha":"b31d13ad61af4e60d587742ddbae7e4db67d14ba","author":{"name":"Craig Taverner","email":"16a7a05fad8a991a88499eabd88a68550b596989@amanzi.com"},"message":"Refactored due to PR comments\n\n* Cleaner API for passing converters around\n* Micro optimizations in distance function\n* Removed asymmetry in Point/Geometry type conversion\n* Support 'point as geometry' in distance function","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/b31d13ad61af4e60d587742ddbae7e4db67d14ba"},{"sha":"514b863314ca85c7a052ba1eb2c94644ee2c9b9c","author":{"name":"Mattias Persson","email":"96e4113efc379fb8ae6068ec263bc1ff29b17f7f@neotechnology.com"},"message":"Added more javadocs on import stages","distinct":true,"url":"https://api.github.com/repos/fbiville/neo4j/commits/514b863314ca85c7a052ba1eb2c94644ee2c9b9c"}]}
{"actor":{"display_login":"fbiville"}}