{"push_id":2437079790,"size":14,"distinct_size":14,"ref":"refs/heads/master","head":"dfc9e721d8b540193f9a99641edaf577f1b8c8f7","before":"afe95a7738c86ddcbd1c0368c2ca3983b06ce0f0","commits":[{"sha":"5288235ca3d8ad8437952aa77fd5b89075f21018","author":{"name":"Jim Ferenczi","email":"63c02f77e9b55c1b0416b1dcd7c38c9829c78407@elastic.co"},"message":"Optimize the composite aggregation for match_all and range queries (#28745)\n\nThis change refactors the composite aggregation to add an execution mode that visits documents in the order of the values\r\npresent in the leading source of the composite definition. This mode does not need to visit all documents since it can early terminate\r\nthe collection when the leading source value is greater than the lowest value in the queue.\r\nInstead of collecting the documents in the order of their doc_id, this mode uses the inverted lists (or the bkd tree for numerics) to collect documents\r\nin the order of the values present in the leading source.\r\nFor instance the following aggregation:\r\n\r\n```\r\n\"composite\" : {\r\n  \"sources\" : [\r\n    { \"value1\": { \"terms\" : { \"field\": \"timestamp\", \"order\": \"asc\" } } }\r\n  ],\r\n  \"size\": 10\r\n}\r\n```\r\n... can use the field `timestamp` to collect the documents with the 10 lowest values for the field instead of visiting all documents.\r\nFor composite aggregation with more than one source the execution can early terminate as soon as one of the 10 lowest values produces enough\r\ncomposite buckets. For instance if visiting the first two lowest timestamp created 10 composite buckets we can early terminate the collection since it\r\nis guaranteed that the third lowest timestamp cannot create a composite key that compares lower than the one already visited.\r\n\r\nThis mode can execute iff:\r\n * The leading source in the composite definition uses an indexed field of type `date` (works also with `date_histogram` source), `integer`, `long` or `keyword`.\r\n * The query is a match_all query or a range query over the field that is used as the leading source in the composite definition.\r\n * The sort order of the leading source is the natural order (ascending since postings and numerics are sorted in ascending order only).\r\n\r\nIf these conditions are not met this aggregation visits each document like any other agg.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/5288235ca3d8ad8437952aa77fd5b89075f21018"},{"sha":"318b0af953118bfc2233d7639b200a4bbf88a1fb","author":{"name":"Christoph Büscher","email":"f4f660ca44b35a6a0b9b10d223238ba3bbdde327@posteo.de"},"message":"Remove execute mode bit from source files\n\nSome source files seem to have the execute bit (a+x) set, which doesn't\nreally seem to hurt but is a bit odd. This change removes those, making\nthe permissions similar to other source files in the repository.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/318b0af953118bfc2233d7639b200a4bbf88a1fb"},{"sha":"a9392f6d429f08e750a2e43bed4b224a9b0c70e7","author":{"name":"Christoph Büscher","email":"f4f660ca44b35a6a0b9b10d223238ba3bbdde327@posteo.de"},"message":"Add file permissions checks to precommit task\n\nThis adds a check for source files that have the execute bit set to the\nprecommit task.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/a9392f6d429f08e750a2e43bed4b224a9b0c70e7"},{"sha":"f5d4550e93e4ae84f42b9512a3a3f7344cd62be3","author":{"name":"Boaz Leskes","email":"daaa071c1d6d35b7f0b1ebbbcc270afab5432337@gmail.com"},"message":"Fold EngineDiskUtils into Store, for better lock semantics (#29156)\n\n#28245 has introduced the utility class`EngineDiskUtils` with a set of methods to prepare/change\r\ntranslog and lucene commit points. That util class bundled everything that's needed to create and\r\nempty shard, bootstrap a shard from a lucene index that was just restored etc. \r\n\r\nIn order to safely do these manipulations, the util methods acquired the IndexWriter's lock. That\r\nwould sometime fail due to concurrent shard store fetching or other short activities that require the\r\nfiles not to be changed while they read from them. \r\n\r\nSince there is no way to wait on the index writer lock, the `Store` class has other locks to make\r\nsure that once we try to acquire the IW lock, it will succeed. To side step this waiting problem, this\r\nPR folds `EngineDiskUtils` into `Store`. Sadly this comes with a price - the store class doesn't and\r\nshouldn't know about the translog. As such the logic is slightly less tight and callers have to do the\r\ntranslog manipulations on their own.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/f5d4550e93e4ae84f42b9512a3a3f7344cd62be3"},{"sha":"dd77d7fd0a6bac9acccab9a89850a27780ef46ae","author":{"name":"Jim Ferenczi","email":"7dde0213a0de057f9233beae75f504ebf7ec7c9a@apache.org"},"message":"#28745: remove extra option in the composite rest tests\n\n`allow_partial_search_results` is not needed for these tests.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/dd77d7fd0a6bac9acccab9a89850a27780ef46ae"},{"sha":"bca264699a600e228042d859cf77b690ef55b3e5","author":{"name":"Boaz Leskes","email":"daaa071c1d6d35b7f0b1ebbbcc270afab5432337@gmail.com"},"message":"remove testUnassignedShardAndEmptyNodesInRoutingTable\n\ntestUnassignedShardAndEmptyNodesInRoutingTable and that test is as old as time and does a very bogus thing.\nit is an IT test which extracts the GatewayAllocator from the node and tells it to allocated unassigned\nshards, while giving it a conjured cluster state with no nodes in it (it uses the DiscoveryNodes.EMPTY_NODES.\nThis is never a cluster state we want to reroute on (we always have at least master node in it).\nI'm going to just delete the test as I don't think it adds much value.\n\nCloses #21463","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/bca264699a600e228042d859cf77b690ef55b3e5"},{"sha":"87957603c030ad0d9d4c755d8ad5920ceac909cb","author":{"name":"Nhat Nguyen","email":"22d5b82ea05e5b6c582c59db5f62ecb93b919f71@elastic.co"},"message":"Prune only gc deletes below local checkpoint (#28790)\n\nOnce a document is deleted and Lucene is refreshed, we will not be able \r\nto look up the `version/seq#` associated with that delete in Lucene. As\r\nconflicting operations can still be indexed, we need another mechanism\r\nto remember these deletes. Therefore deletes should still be stored in\r\nthe Version Map, even after Lucene is refreshed. Obviously, we can't\r\nremember all deletes forever so a trimming mechanism is needed.\r\nCurrently, we remember deletes for at least 1 minute (the default GC\r\ndeletes cycle) and clean them periodically. This is, at the moment, the\r\nbest we can do on the primary for user facing APIs but this arbitrary\r\ntime limit is problematic for replicas. Furthermore, we can't rely on\r\nthe primary and replicas doing the trimming in a synchronized manner,\r\nand failing to do so results in the replica and primary making different\r\ndecisions. \r\n\r\nThe following scenario can cause inconsistency between\r\nprimary and replica.\r\n\r\n1. Primary index doc (index, id=1, v2)\r\n2. Network packet issue causes index operation to back off and wait\r\n3. Primary deletes doc (delete, id=1, v3)\r\n4. Replica processes delete (delete, id=1, v3)\r\n5. 1+ minute passes (GC deletes runs replica)\r\n6. Indexing op is finally sent to the replica which no processes it \r\n   because it forgot about the delete.\r\n\r\nWe can reply on sequence-numbers to prevent this issue. If we prune only \r\ndeletes whose seqno at most the local checkpoint, a replica will\r\ncorrectly remember what it needs. The correctness is explained as\r\nfollows:\r\n\r\nSuppose o1 and o2 are two operations on the same document with seq#(o1) \r\n< seq#(o2), and o2 arrives before o1 on the replica. o2 is processed\r\nnormally since it arrives first; when o1 arrives it should be discarded:\r\n \r\n1. If seq#(o1) <= LCP, then it will be not be added to Lucene, as it was\r\n  already previously added.\r\n\r\n2. If seq#(o1)  > LCP, then it depends on the nature of o2:\r\n  - If o2 is a delete then its seq# is recorded in the VersionMap,\r\n    since seq#(o2) > seq#(o1) > LCP, so a lookup can find it and\r\n    determine that o1 is stale.\r\n  \r\n  - If o2 is an indexing then its seq# is either in Lucene (if\r\n    refreshed) or the VersionMap (if not refreshed yet), so a \r\n    real-time lookup can find it and determine that o1 is stale.\r\n\r\nIn this PR, we prefer to deploy a single trimming strategy, which \r\nsatisfies both requirements, on primary and replicas because:\r\n\r\n- It's simpler - no need to distinguish if an engine is running at\r\nprimary mode or replica mode or being promoted.\r\n\r\n- If a replica subsequently is promoted, user experience is fully\r\nmaintained as that replica remembers deletes for the last GC cycle.\r\n\r\nHowever, the version map may consume less memory if we deploy two \r\ndifferent trimming strategies for primary and replicas.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/87957603c030ad0d9d4c755d8ad5920ceac909cb"},{"sha":"7bf909194222f851a5e10ba8012e85624c2c124f","author":{"name":"Andy Bristol","email":"a800037b8faa484b37d8419c30ec0cb11131283b@elastic.co"},"message":"[test] packaging: gradle tasks for groovy tests (#29046)\n\nThe vagrant test plugin adds tasks for the groovy packaging tests,\r\nwhich run after the bats packaging test tasks.Rename the 'bats'\r\nconfiguration to 'packaging' and remove the option to inherit\r\narchives from this configuration.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/7bf909194222f851a5e10ba8012e85624c2c124f"},{"sha":"0ac89a32cc71742256a0a3471aa19656bcaac901","author":{"name":"Nhat Nguyen","email":"22d5b82ea05e5b6c582c59db5f62ecb93b919f71@elastic.co"},"message":"Do not optimize append-only if seen normal op with higher seqno (#28787)\n\nWhen processing an append-only operation, primary knows that operations \r\ncan only conflict with another instance of the same operation. This is\r\ntrue as the id was freshly generated. However this property doesn't hold\r\nfor replicas. As soon as an auto-generated ID was indexed into the\r\nprimary, it can be exposed to a search and users can issue a follow up\r\noperation on it. In extremely rare cases, the follow up operation can be\r\narrived and processed on a replica before the original append-only\r\nrequest. In this case we can't simply proceed with the append-only\r\nrequest and blindly add it to the index without consulting the version\r\nmap. \r\n\r\nThe following scenario can cause difference between primary and\r\nreplica.\r\n\r\n1. Primary indexes an auto-gen-id doc. (id=X, v=1, s#=20)\r\n2. A refresh cycle happens on primary\r\n3. The new doc is picked up and modified - say by a delete by query\r\n   request - Primary gets a delete doc (id=X, v=2, s#=30)\r\n4. Delete doc is processed first on the replica (id=X, v=2, s#=30)\r\n5. Indexing operation arrives on the replica, since it's an auto-gen-id\r\n   request and the retry marker is lower, we put it into lucene without \r\n   any check. Replica has a doc the primary doesn't have.\r\n\r\nTo deal with a potential conflict between an append-only operation and a \r\nnormal operation on replicas, we need to rely on sequence numbers. This\r\ncommit maintains the max seqno of non-append-only operations on replica\r\nthen only apply optimization for an append-only operation only if its\r\nseq# is higher than the seq# of all non-append-only.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/0ac89a32cc71742256a0a3471aa19656bcaac901"},{"sha":"d2baf4b19136a57bd9205a556f753a499e4d357b","author":{"name":"Andrew Banchich","email":"c04a397266f1c35ec96a47a7fec414957fa6320c@gmail.com"},"message":"[Docs] Spelling and grammar changes to reindex.asciidoc (#29232)","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/d2baf4b19136a57bd9205a556f753a499e4d357b"},{"sha":"8d6832c5ee5298e1befc372ed0b999dfeeeac279","author":{"name":"Christoph Büscher","email":"f4f660ca44b35a6a0b9b10d223238ba3bbdde327@posteo.de"},"message":"Make SearchStats implement Writeable (#29258)\n\nMoves another class over from Streamable to Writeable. By this,\r\nalso some constructors can be removed or made private.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/8d6832c5ee5298e1befc372ed0b999dfeeeac279"},{"sha":"d1d3edf156fe074f26fe78c73d764ec1ebd60ff2","author":{"name":"Nhat Nguyen","email":"22d5b82ea05e5b6c582c59db5f62ecb93b919f71@elastic.co"},"message":"TEST: Use different translog dir for a new engine\n\nIn #testPruneOnlyDeletesAtMostLocalCheckpoint, we create a new engine\nbut mistakenly use the same translog directory of  the existing engine.\nThis prevents translog files from cleaning up when closing the engines.\n\nERROR   0.12s J2 | InternalEngineTests.testPruneOnlyDeletesAtMostLocalCheckpoint <<< FAILURES!\n   > Throwable #1: java.io.IOException: could not remove the following files (in the order of attempts):\n   >    translog-primary-060/translog-2.tlog:  java.io.IOException: access denied:\n\nThis commit makes sure to use a separate directory for each engine in\nthis tes.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/d1d3edf156fe074f26fe78c73d764ec1ebd60ff2"},{"sha":"13f9e922f358897caf9f49fa2c9495a96382f48c","author":{"name":"Luca Cavanna","email":"fcdf11cc3de751e1fde3dc9c882ec9c523a33c97@users.noreply.github.com"},"message":"REST client: hosts marked dead for the first time should not be immediately retried (#29230)\n\nThis was the plan from day one but due to a silly bug nodes were immediately retried after they were marked as dead for the first time. From the second time on, the expected backoff was applied.","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/13f9e922f358897caf9f49fa2c9495a96382f48c"},{"sha":"dfc9e721d8b540193f9a99641edaf577f1b8c8f7","author":{"name":"Nhat Nguyen","email":"22d5b82ea05e5b6c582c59db5f62ecb93b919f71@elastic.co"},"message":"TEST: Increase timeout for testPrimaryReplicaResyncFailed\n\nThe default timeout (eg. 10 seconds) may not be enough for CI to\nre-allocate shards after the partion is healed. This commit increases\nthe timeout to 30 seconds and enables logging in order to have more\ndetailed information in case this test failed again.\n\nCloses #29060","distinct":true,"url":"https://api.github.com/repos/bizybot/elasticsearch/commits/dfc9e721d8b540193f9a99641edaf577f1b8c8f7"}]}
{"actor":{"display_login":"bizybot"}}