{"push_id":1386197469,"size":16,"distinct_size":16,"ref":"refs/heads/impala-4352-save-primary-keys","head":"ae5299bd2f739e146cdd659413cbe2fb61adae16","before":"c620744af2a0af98a5c9b1ac2c457a8e06af72ee","commits":[{"sha":"1d8cdb02c67fad9279a899626f3cd56bce0b0eb3","author":{"name":"Alex Behm","email":"1d65425ff20ce21b68f426794449955d15dd30a4@cloudera.com"},"message":"IMPALA-4309: Introduce Expr rewrite phase and supporting classes.\n\nIntroduces a new phase for rewriting Exprs after analysis and\nbefore subquery rewriting. The transformed Exprs replace the\noriginal ones in analyzed statements. If Exprs were changed,\nthe whole statement is reset() and re-analyzed, similar to how\nsubqueries are rewritten. If both Exprs and subqueries are\nrewritten there is only one re-analysis of the changed statement.\n\nThe following new classes work together to perform transformations:\n1. ExprRewriteRule\n- base class for Expr transformation rules\n2. ExprRewriter\n- drives the transformation of Exprs using a list of\n  ExprRewriteRules\n\nStatements that have exprs to be rewritten need to implement\na new method rewriteExprs() that accepts an ExprRewriter.\n\nAs an example, this patch adds a rule for converting\nBetweenPredicates into their equivalent CompoundPredicates.\nThe BetweenPredicate has been notoriously buggy due to a lack\nof such a separate rewrite phase and is now cleaned up.\n\nTesting:\n1. Added a new test for checking that the rewrite framework\n   covers all relevant statements, clauses and can properly\n   handle nested statements and subqueries.\n2. Added a new test for ExprRewriteRules and implemented\n   tests for the BetweenPredicate rewrite.\n2. There are many existing tests for BetweePredicates and\n   they all exercise the new rewrite rule/phase.\n3. Ran a private core/hdfs run and it passed.\n\nChange-Id: I2279dc984bcf7742db4fa3b1aa67283ecbb05e6e\nReviewed-on: http://gerrit.cloudera.org:8080/4746\nReviewed-by: Alex Behm <alex.behm@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/1d8cdb02c67fad9279a899626f3cd56bce0b0eb3"},{"sha":"9755ea63b50d23f323c84edb4307ce603ded4fe1","author":{"name":"Tim Armstrong","email":"fe64fd088b711818286897e35b3f900c59dcb393@cloudera.com"},"message":"IMPALA-4409: respect lock order in QueryExecState::CancelInternal()\n\nThe code previously violated the (partially documented) lock order\nin ImpalaServer. An example of a possible cycle in the dependency\ngraph is:\n\n* SetQueryInFlight() holds SessionState::lock_ and waits for\n  'query_expiration_lock_'\n* ExpireQueries() holds 'query_expiration_lock_' and waits for\n  'query_exec_state_map_lock_'\n* GetQueryExecState() holds 'query_exec_state_map_lock_' and\n  waits for QueryExecState::lock_\n* QES::Cancel() holds QueryExecState::lock_\n  and waits for SessionState::lock\n\nIt's not clear how likely the above scenario is, but it's hard to rule\nit out.\n\nWe have not seen this hang in the wild but have seen similar ones.\n\nTesting:\nRan local stress test on 3-node minicluster with TPC-H 20 and 50%\nof queries being cancelled.\n\nChange-Id: I785fea0163a90d0633fb6ed77ec7c6882ab5c110\nReviewed-on: http://gerrit.cloudera.org:8080/4896\nReviewed-by: Tim Armstrong <tarmstrong@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/9755ea63b50d23f323c84edb4307ce603ded4fe1"},{"sha":"7862e809f608bd34cd0a318d0a91c70a6d95923d","author":{"name":"Matthew Jacobs","email":"8c7569cb953aa24dccb950531aaf8f3c7c1f0047@cloudera.com"},"message":"IMPALA-4411: Kudu inserts violate lock ordering and could deadlock\n\nThis fixes a potential hang because the code took\nQueryExecState::lock_ before SessionState::lock, which is\nthe wrong order. A hang has not yet been observed.\n\nThe code that was taking the session lock doesn't actually\nneed to be holding the QueryExecState lock_, so this is easy\nto fix by moving the relevant code.\n\nTesting: Running an exhaustive test job, and stress tests\nmanually.\n\nChange-Id: I2aa36fce78525a80a6b880e1b668105006bc1425\nReviewed-on: http://gerrit.cloudera.org:8080/4895\nReviewed-by: Matthew Jacobs <mj@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/7862e809f608bd34cd0a318d0a91c70a6d95923d"},{"sha":"40bce41765fd218cec623824c5bad21b1c207c6b","author":{"name":"Taras Bobrovytsky","email":"f5b2dc9a51b15aed0d1bab4083365869dd513a0b@cloudera.com"},"message":"Fix TPCH and TPCDS Kudu loading templates\n\nThe templates (used by the stress test) for loading the TCPH and TPCDS\ndata into Kudu had a missing \"stored as kudu\" statement.\n\nChange-Id: Ibe84e1831cc0722bd0381ec76f385ae2a02a6841\nReviewed-on: http://gerrit.cloudera.org:8080/4939\nReviewed-by: Dimitris Tsirogiannis <dtsirogiannis@cloudera.com>\nTested-by: Taras Bobrovytsky <tbobrovytsky@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/40bce41765fd218cec623824c5bad21b1c207c6b"},{"sha":"a9a00858da2fca89de0efc54393a97c68c033ea6","author":{"name":"Matthew Jacobs","email":"8c7569cb953aa24dccb950531aaf8f3c7c1f0047@cloudera.com"},"message":"IMPALA-4384: NPE when cols list has trailing comma\n\nThe changes to introduce new PRIMARY KEY syntax allowed for\na table definition to have a list of cols, optionally\nfollowed by a PRIMARY KEY clause. However, this rule\nwas broken and would allow for the list of columns followed\nby a trailing comma.\n\n[localhost:21000] > create table foo (id int, x int,);\nQuery: create table foo (id int, x int,)\nERROR: AnalysisException: null\nCAUSED BY: NullPointerException: null\n\nChange-Id: Ibc394dd7e687b501ae7887a51bed2f441d562760\nReviewed-on: http://gerrit.cloudera.org:8080/4869\nReviewed-by: Matthew Jacobs <mj@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/a9a00858da2fca89de0efc54393a97c68c033ea6"},{"sha":"f397d756005a766ed1561ada0e43948051d164cd","author":{"name":"Jim Apple","email":"36e08e51b9117732d42c4c43c222698b0be76927@cloudera.com"},"message":"IMPALA-3853: More RAT cleaning.\n\nApache RAT is a tool to audit code repositories for the ASF copyright\nrules. Our wrapper script around it found a few more things; this\npatch fixes those things.\n\nChange-Id: I01367ea26feaf6a3e2cf4ac04f1c6a63f6e66195\nReviewed-on: http://gerrit.cloudera.org:8080/4904\nReviewed-by: Jim Apple <jbapple@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/f397d756005a766ed1561ada0e43948051d164cd"},{"sha":"9af59bfe2bb2747e71de45cb1fb96b7da3811ad1","author":{"name":"Laszlo Gaal","email":"054ac76481e51fa850f31e74ec97cabc9fe9ea3a@cloudera.com"},"message":"IMPALA-4153: Fix count(*) on all blank('') columns - test\n\nThis change adds test coverage for the fixes committed for\nIMPALA-2399 in commit 9ed3b685a109559fad8dd94f6c03af93e164305a.\nIt uses the table nulltable in the workload functional-query\nto verify the materialization and counting of NULL and empty-\nvalued columns. The test can be run on any supported storage\nand compression combination.\n\nChange-Id: I23923f95f43d67977ee1520a1fc09ce297548b3f\nReviewed-on: http://gerrit.cloudera.org:8080/4755\nTested-by: Internal Jenkins\nReviewed-by: Jim Apple <jbapple@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/9af59bfe2bb2747e71de45cb1fb96b7da3811ad1"},{"sha":"1af0aa4ad16b90bbec7c59fc75abe95d83b99c8d","author":{"name":"Sailesh Mukil","email":"878eaa34c439eddee78b063b5443e5722b0c42a6@cloudera.com"},"message":"IMPALA-4412: Per operator timing in profile summary is incorrect when mt_dop > 0\n\nWith mt_dop > 0, we may incorrectly populate the\nper_fragment_instance_idx for each FInstanceExecParams in\nSimpleScheduler::CreateScanInstances().\n\nThis causes the averages in the exec summary to be wrong resulting in\nmisleading Runtime Profiles.\n\nThis patch makes sure that each FInstanceExecParams has the correct\ninstance index associated with it, thereby fixing the exec summaries\nin the runtime profiles.\n\nChange-Id: Id8634acba05698281e698ab8ddf92db4ae38af88\nReviewed-on: http://gerrit.cloudera.org:8080/4933\nReviewed-by: Sailesh Mukil <sailesh@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/1af0aa4ad16b90bbec7c59fc75abe95d83b99c8d"},{"sha":"c03cfe51fc6640a4b5d0587582123b56b7575888","author":{"name":"Bharath Vissapragada","email":"1bcc0fda38316e6f1200de8c30c1cf5cbf0fffdc@cloudera.com"},"message":"IMPALA-3983/IMPALA-3974: Delete function jar resources after load\n\nThe Catalog copies the UDF jar files to the local file system to\nload the Java UDF classes for validation purposes. However we\ndo not clean them up after the UDF load and hence on a deployment\nwith large number of functions registered, these jar can accumulate\nover a period of time and  can fill up the tmp space. We fix it by\ndeleting the jar resource once the function is loaded.\n\nAlso, this patch switches to --local_library_dir for copying these\ntemporary jars instead of using the path from java.io.tmpdir.\n\nChange-Id: I5f9dedb5b342415380c83e61a72eb497371a8199\nReviewed-on: http://gerrit.cloudera.org:8080/4617\nReviewed-by: Bharath Vissapragada <bharathv@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/c03cfe51fc6640a4b5d0587582123b56b7575888"},{"sha":"14891fe00461cc3af579a143811bc59fa6a1b1eb","author":{"name":"Jim Apple","email":"36e08e51b9117732d42c4c43c222698b0be76927@cloudera.com"},"message":"IMPALA-3676: Use clang as a static analysis tool\n\nThis patch adds a script to run clang-tidy over the whole code\nbase. It is a first step towards running clang-tidy over patches as a\ntool to help users spot bugs before code review.\n\nBecause of the number of clang-tidy checks, this patch only addresses\nsome of them. In particular, only checks starting with 'clang' are\nconsidered. Many of them which are flaky or not part of our style are\nexcluded from the analysis. This patch also exlcudes some checks which\nare part of our current style but which would be too laborious to fix\nover the entire codebase, like using nullptr rather than NULL.\n\nThis patch also fixes a number of small bugs found by clang-tidy.\n\nFinally, this patch adds the class AlignedNew, the purpose of which is\nto provide correct alignment on heap-allocated data. The global new\noperator only guarantees 16-byte alignment. A class that includes a\nmember variable that must be aligned on a k-byte boundary for k>16 can\ninherit from AlignedNew<k> to ensure correct alignment on the heap,\nquieting clang's -Wover-aligned warning. (Static and stack allocation\nare required by the standard to respect the alignment of the type and\nits member variables, so no extra code is needed for allocation in\nthose places.)\n\nChange-Id: I4ed168488cb30ddeccd0087f3840541d858f9c06\nReviewed-on: http://gerrit.cloudera.org:8080/4758\nReviewed-by: Jim Apple <jbapple@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/14891fe00461cc3af579a143811bc59fa6a1b1eb"},{"sha":"c5f49ec9bbc1b191545b6edd2848acb632b45973","author":{"name":"Alex Behm","email":"1d65425ff20ce21b68f426794449955d15dd30a4@cloudera.com"},"message":"IMPALA-4423: Correct but conservative implementation of Subquery.equals().\n\nThe underlying problem was for trivial/constant [NOT] EXISTS subqueries\nwe substituted out Subqueries with bool literals using an ExprSubstitutionMap,\nbut the Subquery.equals() function was not implemented properly, so we ended\nup matching Subqueries to the wrong entry in the ExprSubstitutionMap.\nThis could ultimately lead to wrong plans and results.\n\nTesting: Corrected an existing test and modified an existing test for\nextra coverage.\n\nChange-Id: I5562d98ce36507aa5e253323e184fd42b54f27ed\nReviewed-on: http://gerrit.cloudera.org:8080/4923\nReviewed-by: Alex Behm <alex.behm@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/c5f49ec9bbc1b191545b6edd2848acb632b45973"},{"sha":"32294220c4a0a7a159a70cf8ef313622b7190303","author":{"name":"Matthew Jacobs","email":"8c7569cb953aa24dccb950531aaf8f3c7c1f0047@cloudera.com"},"message":"IMPALA-4379: Fix and test Kudu table type checking, follow up\n\nThe first fix for IMPALA-4379 went in before all comments\nwere addressed. First commit: 9b507b6.\n\nThis addresses some follow-up comments about how to handling\nALTER TABLE setting the storage_handler table property,\nwhich doesn't really make sense to ever allow.\n\nChange-Id: I93d04a04483af598b392c28874363e3b0202e1f3\nReviewed-on: http://gerrit.cloudera.org:8080/4894\nReviewed-by: Matthew Jacobs <mj@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/32294220c4a0a7a159a70cf8ef313622b7190303"},{"sha":"a807b3adba9124c935678e4ad18bbdb795ee1101","author":{"name":"Sailesh Mukil","email":"878eaa34c439eddee78b063b5443e5722b0c42a6@cloudera.com"},"message":"IMPALA-4406: Add cryptography export control notice\n\nThe ASF guidelines on crypto requires projects that use any form of\nasymmetric cryptography to include an export control notice to notify\nthe users of the same.\n\nRefer: https://www.apache.org/dev/crypto.html\n\nChange-Id: I527a5c07a13c207c33544887d3f20501a0294c01\nReviewed-on: http://gerrit.cloudera.org:8080/4940\nReviewed-by: Sailesh Mukil <sailesh@cloudera.com>\nTested-by: Sailesh Mukil <sailesh@cloudera.com>","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/a807b3adba9124c935678e4ad18bbdb795ee1101"},{"sha":"381a0f54be6ea1d16c4d65fc7c2747edd388fe8e","author":{"name":"David Knupp","email":"50f605843009ddca7ef3d7ea40aa3c28abef025f@cloudera.com"},"message":"IMPALA-4421: Send custom cluster & process failure test results to logs/\n\nImproperly nested quotes in tests/run-custom-cluster-tests.sh caused a\ndirectory named \"${RESULTS_DIR}\" to be created in the tests directory,\nwhich in turned interfered with run-tests.py.\n\nAdditonally noticed that run-process-failure-tests.sh was creating a\nresults directory within tests/. We shoud not be polluting the tests/\ndir with random test artifacts, so changed the result directory path\nfor those tests as well.\n\nTested by running both scripts, and ensuring that results wound up in\nlogs/, and nothing was created in tests/.\n\nChange-Id: I475f61b4ba8a693324bbefed8819d029674b66dd\nReviewed-on: http://gerrit.cloudera.org:8080/4918\nReviewed-by: Jim Apple <jbapple@cloudera.com>\nTested-by: Internal Jenkins","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/381a0f54be6ea1d16c4d65fc7c2747edd388fe8e"},{"sha":"533e59f7d0fe7d9e6cfc3e6ff00696dcbf41f1a6","author":{"name":"Michael Brown","email":"51def41f3b867ba43c7f77708ded5b71ff83e30b@cloudera.com"},"message":"IMPALA-4352: test infra: store Impala/Kudu primary keys in object model\n\nTest infrastructure, including the random query generator and the data\nmigrator, needs to know the primary keys of Impala/Kudu tables. This\ntest infrastructure keeps Python object models of the tables and\ncolumns. This patch adds the ability to read from source Impala/Kudu\ntables via SHOW CREATE TABLE and store primary keys as proper\nattributes. The patch also adds tests that ensure the test\ninfrastructure is always able to read and store the primary keys. This\nhelps find breakages sooner rather than later. For example, if a\nregression to \"SHOW CREATE TABLE\" or the test infrastructure makes us no\nlonger able to parse primary keys, GVO or other CI will find the\nbreakage faster than running the query generator.\n\nI also fixed some flake8 issues in files I touched. There were several\nfiles that had a lot of white space warnings, and I wanted to keep the\npatch from getting too large.\n\nChange-Id: Ib654b6cd0e8c2a172ffb7330497be4d4a751e6e5","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/533e59f7d0fe7d9e6cfc3e6ff00696dcbf41f1a6"},{"sha":"ae5299bd2f739e146cdd659413cbe2fb61adae16","author":{"name":"Michael Brown","email":"51def41f3b867ba43c7f77708ded5b71ff83e30b@cloudera.com"},"message":"add failing test for pk ordering based on show create table\n\nChange-Id: Icff62b129463c6faaa9aa54c053c748093255801","distinct":true,"url":"https://api.github.com/repos/mikesbrown/incubator-impala/commits/ae5299bd2f739e146cdd659413cbe2fb61adae16"}]}
{"actor":{"display_login":"mikesbrown"}}