{"push_id":1037518951,"size":9,"distinct_size":9,"ref":"refs/heads/infer-isnotnull","head":"2f43d41cceab4232f38ce65a68cb2ff7e0faa542","before":"6421ba71ffa351e0f97ffa186257eb7f71d115d8","commits":[{"sha":"70a6f0bb57ca2248444157e2707fbcc3cb04e3bc","author":{"email":"c9d35a384bc579d1c6c8c055f2a2b5c04e11b521@databricks.com","name":"Reynold Xin"},"message":"[SPARK-14149] Log exceptions in tryOrIOException\n\n## What changes were proposed in this pull request?\nWe ran into a problem today debugging some class loading problem during deserialization, and JVM was masking the underlying exception which made it very difficult to debug. We can however log the exceptions using try/catch ourselves in serialization/deserialization. The good thing is that all these methods are already using Utils.tryOrIOException, so we can just put the try catch and logging in a single place.\n\n## How was this patch tested?\nA logging change with a manual test.\n\nAuthor: Reynold Xin <rxin@databricks.com>\n\nCloses #11951 from rxin/SPARK-14149.","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/70a6f0bb57ca2248444157e2707fbcc3cb04e3bc"},{"sha":"e9b6e7d8577cd721a433130f29e8b112d98768b9","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-13456][SQL][FOLLOW-UP] lazily generate the outer pointer for case class defined in REPL\n\n## What changes were proposed in this pull request?\n\nIn https://github.com/apache/spark/pull/11410, we missed a corner case: define the inner class and use it in `Dataset` at the same time by using paste mode. For this case, the inner class and the `Dataset` are inside same line object, when we build the `Dataset`, we try to get outer pointer from line object, and it will fail because the line object is not initialized yet.\n\nhttps://issues.apache.org/jira/browse/SPARK-13456?focusedCommentId=15209174&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15209174 is an example for this corner case.\n\nThis PR make the process of getting outer pointer from line object lazy, so that we can successfully build the `Dataset` and finish initializing the line object.\n\n## How was this patch tested?\n\nnew test in repl suite.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #11931 from cloud-fan/repl.","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/e9b6e7d8577cd721a433130f29e8b112d98768b9"},{"sha":"55a605763dfcd544d0c8bdd6a148bdb0a7589fe9","author":{"email":"6be69bb5cf6cb8dd624e7f174d5aba3c49eb4343@us.ibm.com","name":"Holden Karau"},"message":"[SPARK-13887][PYTHON][TRIVIAL][BUILD] Make lint-python script fail fast\n\n## What changes were proposed in this pull request?\n\nChange lint python script to stop on first error rather than building them up so its clearer why we failed (requested by rxin). Also while in the file, remove the commented out code.\n\n## How was this patch tested?\n\nManually ran lint-python script with & without pep8 errors locally and verified expected results.\n\nAuthor: Holden Karau <holden@us.ibm.com>\n\nCloses #11898 from holdenk/SPARK-13887-pylint-fast-fail.","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/55a605763dfcd544d0c8bdd6a148bdb0a7589fe9"},{"sha":"6603d9f7e283cf8199cfddfeea30d9db39669726","author":{"email":"c437f8b7af73b0a10ecac061712ff9de6d41089e@databricks.com","name":"Davies Liu"},"message":"[SPARK-13919] [SQL] fix column pruning through filter\n\n## What changes were proposed in this pull request?\n\nThis PR fix the conflict between ColumnPruning and PushPredicatesThroughProject, because ColumnPruning will try to insert a Project before Filter, but PushPredicatesThroughProject will move the Filter before Project.This is fixed by remove the Project before Filter, if the Project only do column pruning.\n\nThe RuleExecutor will fail the test if reached max iterations.\n\nCloses #11745\n\n## How was this patch tested?\n\nExisting tests.\n\nThis is a test case still failing, disabled for now, will be fixed by https://issues.apache.org/jira/browse/SPARK-14137\n\nAuthor: Davies Liu <davies@databricks.com>\n\nCloses #11828 from davies/fail_rule.","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/6603d9f7e283cf8199cfddfeea30d9db39669726"},{"sha":"43b15e01c46ea1971569f74c9201a55de39e8917","author":{"email":"563f5923eae9a92c8a3f34a26b67fa348a303ed1@databricks.com","name":"Wenchen Fan"},"message":"[SPARK-14061][SQL] implement CreateMap\n\n## What changes were proposed in this pull request?\n\nAs we have `CreateArray` and `CreateStruct`, we should also have `CreateMap`.  This PR adds the `CreateMap` expression, and the DataFrame API, and python API.\n\n## How was this patch tested?\n\nvarious new tests.\n\nAuthor: Wenchen Fan <wenchen@databricks.com>\n\nCloses #11879 from cloud-fan/create_map.","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/43b15e01c46ea1971569f74c9201a55de39e8917"},{"sha":"b8c19eeeb7a49f1212a5459e260706c0f71fe015","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"Infer IsNotNull constraints from non-nullable attributes","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/b8c19eeeb7a49f1212a5459e260706c0f71fe015"},{"sha":"9c53707567c23f0b385ed35b36466958362bd96c","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"unit test","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/9c53707567c23f0b385ed35b36466958362bd96c"},{"sha":"5a449a34bb20924d35944a71f5c21574b3616759","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"CR","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/5a449a34bb20924d35944a71f5c21574b3616759"},{"sha":"2f43d41cceab4232f38ce65a68cb2ff7e0faa542","author":{"email":"da4a4efb907a17fabb3e092841511e7f24e226cf@databricks.com","name":"Sameer Agarwal"},"message":"add back union20","distinct":true,"url":"https://api.github.com/repos/sameeragarwal/spark/commits/2f43d41cceab4232f38ce65a68cb2ff7e0faa542"}]}