{"action":"created","issue":{"url":"https://api.github.com/repos/prestodb/presto/issues/11549","repository_url":"https://api.github.com/repos/prestodb/presto","labels_url":"https://api.github.com/repos/prestodb/presto/issues/11549/labels{/name}","comments_url":"https://api.github.com/repos/prestodb/presto/issues/11549/comments","events_url":"https://api.github.com/repos/prestodb/presto/issues/11549/events","html_url":"https://github.com/prestodb/presto/pull/11549","id":362586960,"node_id":"MDExOlB1bGxSZXF1ZXN0MjE3MjYyOTQ4","number":11549,"title":"[Hotfix]nullsCount in columnStatistic should marked as not present in…","user":{"login":"VicoWu","id":5207388,"node_id":"MDQ6VXNlcjUyMDczODg=","avatar_url":"https://avatars0.githubusercontent.com/u/5207388?v=4","gravatar_id":"","url":"https://api.github.com/users/VicoWu","html_url":"https://github.com/VicoWu","followers_url":"https://api.github.com/users/VicoWu/followers","following_url":"https://api.github.com/users/VicoWu/following{/other_user}","gists_url":"https://api.github.com/users/VicoWu/gists{/gist_id}","starred_url":"https://api.github.com/users/VicoWu/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/VicoWu/subscriptions","organizations_url":"https://api.github.com/users/VicoWu/orgs","repos_url":"https://api.github.com/users/VicoWu/repos","events_url":"https://api.github.com/users/VicoWu/events{/privacy}","received_events_url":"https://api.github.com/users/VicoWu/received_events","type":"User","site_admin":false},"labels":[{"id":196992743,"node_id":"MDU6TGFiZWwxOTY5OTI3NDM=","url":"https://api.github.com/repos/prestodb/presto/labels/CLA%20Signed","name":"CLA Signed","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":20,"created_at":"2018-09-21T12:07:53Z","updated_at":"2018-10-02T20:18:33Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/prestodb/presto/pulls/11549","html_url":"https://github.com/prestodb/presto/pull/11549","diff_url":"https://github.com/prestodb/presto/pull/11549.diff","patch_url":"https://github.com/prestodb/presto/pull/11549.patch"},"body":"My presto is running on my Cloudera 5.7.6 hive meastore.\r\nWhen upgrading to 0.210, Many queries previously run successfully failed with following error:\r\n```\r\njava.lang.IllegalArgumentException: Nulls fraction should be within [0, 1] or NaN, got: -0.16666666666666666\r\n\tat com.google.common.base.Preconditions.checkArgument(Preconditions.java:210)\r\n\tat com.facebook.presto.cost.SymbolStatsEstimate.<init>(SymbolStatsEstimate.java:54)\r\n\tat com.facebook.presto.cost.SymbolStatsEstimate$Builder.build(SymbolStatsEstimate.java:225)\r\n\tat com.facebook.presto.cost.TableScanStatsRule.toSymbolStatistics(TableScanStatsRule.java:92)\r\n\tat com.facebook.presto.cost.TableScanStatsRule.lambda$doCalculate$0(TableScanStatsRule.java:73)\r\n\tat java.util.Optional.map(Optional.java:215)\r\n\tat com.facebook.presto.cost.TableScanStatsRule.doCalculate(TableScanStatsRule.java:73)\r\n\tat com.facebook.presto.cost.TableScanStatsRule.doCalculate(TableScanStatsRule.java:41)\r\n\tat com.facebook.presto.cost.SimpleStatsRule.calculate(SimpleStatsRule.java:39)\r\n\tat com.facebook.presto.cost.ComposableStatsCalculator.calculateStats(ComposableStatsCalculator.java:80)\r\n\tat com.facebook.presto.cost.ComposableStatsCalculator.calculateStats(ComposableStatsCalculator.java:70)\r\n\tat com.facebook.presto.cost.CachingStatsProvider.getStats(CachingStatsProvider.java:70)\r\n\tat com.facebook.presto.cost.CostCalculatorUsingExchanges$CostEstimator.getStats(CostCalculatorUsingExchanges.java:243)\r\n\tat com.facebook.presto.cost.CostCalculatorUsingExchanges$CostEstimator.visitTableScan(CostCalculatorUsingExchanges.java:141)\r\n\tat com.facebook.presto.cost.CostCalculatorUsingExchanges$CostEstimator.visitTableScan(CostCalculatorUsingExchanges.java:99)\r\n\tat com.facebook.presto.sql.planner.plan.TableScanNode.accept(TableScanNode.java:128)\r\n\tat com.facebook.presto.cost.CostCalculatorUsingExchanges.calculateCost(CostCalculatorUsingExchanges.java:96)\r\n\tat com.facebook.presto.cost.CostCalculatorWithEstimatedExchanges.calculateCost(CostCalculatorWithEstimatedExchanges.java:73)\r\n\tat com.facebook.presto.cost.CachingCostProvider.calculateCumulativeCost(CachingCostProvider.java:103)\r\n\tat com.facebook.presto.cost.CachingCostProvider.getGroupCost(CachingCostProvider.java:95)\r\n\tat com.facebook.presto.cost.CachingCostProvider.getCumulativeCost(CachingCostProvider.java:72)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.ReorderJoins$JoinEnumerator.createJoinEnumerationResult(ReorderJoins.java:391)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.ReorderJoins$JoinEnumerator.getJoinSource(ReorderJoins.java:342)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.ReorderJoins$JoinEnumerator.createJoin(ReorderJoins.java:258)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.ReorderJoins$JoinEnumerator.createJoinAccordingToPartitioning(ReorderJoins.java:229)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.ReorderJoins$JoinEnumerator.chooseJoinOrder(ReorderJoins.java:173)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.ReorderJoins$JoinEnumerator.access$000(ReorderJoins.java:135)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.ReorderJoins.apply(ReorderJoins.java:127)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.ReorderJoins.apply(ReorderJoins.java:88)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.transform(IterativeOptimizer.java:165)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreNode(IterativeOptimizer.java:138)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:103)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:185)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:105)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:185)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:105)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.optimize(IterativeOptimizer.java:94)\r\n\tat com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:150)\r\n\tat com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:139)\r\n\tat com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:360)\r\n\tat com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:345)\r\n\tat com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:289)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n```\r\n\r\nThis is because hive metastore use -1 to mark the column nullCounts as unavailable. \r\n\r\nAfter I upgrading to master branch, the error still exists and the error become this:\r\n```\r\ncom.facebook.presto.spi.PrestoException: Corrupted partition statistics (Table: default.plus_higher_category Partition: [<UNPARTITIONED>] Column: plus_higher_category_name): nullsCount must be greater than or equal to zero: -1\r\n\tat com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.checkStatistics(MetastoreHiveStatisticsProvider.java:350)\r\n\tat com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.lambda$validateColumnStatistics$14(MetastoreHiveStatisticsProvider.java:222)\r\n\tat java.util.OptionalLong.ifPresent(OptionalLong.java:142)\r\n\tat com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.validateColumnStatistics(MetastoreHiveStatisticsProvider.java:221)\r\n\tat com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.lambda$null$10(MetastoreHiveStatisticsProvider.java:211)\r\n\tat com.google.common.collect.RegularImmutableMap.forEach(RegularImmutableMap.java:187)\r\n\tat com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.lambda$validatePartitionStatistics$11(MetastoreHiveStatisticsProvider.java:211)\r\n\tat com.google.common.collect.SingletonImmutableBiMap.forEach(SingletonImmutableBiMap.java:65)\r\n\tat com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.validatePartitionStatistics(MetastoreHiveStatisticsProvider.java:204)\r\n\tat com.facebook.presto.hive.statistics.MetastoreHiveStatisticsProvider.getTableStatistics(MetastoreHiveStatisticsProvider.java:142)\r\n\tat com.facebook.presto.hive.HiveMetadata.getTableStatistics(HiveMetadata.java:536)\r\n\tat com.facebook.presto.spi.connector.classloader.ClassLoaderSafeConnectorMetadata.getTableStatistics(ClassLoaderSafeConnectorMetadata.java:201)\r\n\tat com.facebook.presto.metadata.MetadataManager.getTableStatistics(MetadataManager.java:401)\r\n\tat com.facebook.presto.cost.TableScanStatsRule.doCalculate(TableScanStatsRule.java:61)\r\n\tat com.facebook.presto.cost.TableScanStatsRule.doCalculate(TableScanStatsRule.java:36)\r\n\tat com.facebook.presto.cost.SimpleStatsRule.calculate(SimpleStatsRule.java:39)\r\n\tat com.facebook.presto.cost.ComposableStatsCalculator.calculateStats(ComposableStatsCalculator.java:80)\r\n\tat com.facebook.presto.cost.ComposableStatsCalculator.calculateStats(ComposableStatsCalculator.java:70)\r\n\tat com.facebook.presto.cost.CachingStatsProvider.getGroupStats(CachingStatsProvider.java:91)\r\n\tat com.facebook.presto.cost.CachingStatsProvider.getStats(CachingStatsProvider.java:68)\r\n\tat com.facebook.presto.cost.CostCalculatorWithEstimatedExchanges$ExchangeCostEstimator.getStats(CostCalculatorWithEstimatedExchanges.java:198)\r\n\tat com.facebook.presto.cost.CostCalculatorWithEstimatedExchanges$ExchangeCostEstimator.calculateJoinCost(CostCalculatorWithEstimatedExchanges.java:153)\r\n\tat com.facebook.presto.cost.CostCalculatorWithEstimatedExchanges$ExchangeCostEstimator.visitJoin(CostCalculatorWithEstimatedExchanges.java:133)\r\n\tat com.facebook.presto.cost.CostCalculatorWithEstimatedExchanges$ExchangeCostEstimator.visitJoin(CostCalculatorWithEstimatedExchanges.java:76)\r\n\tat com.facebook.presto.sql.planner.plan.JoinNode.accept(JoinNode.java:289)\r\n\tat com.facebook.presto.cost.CostCalculatorWithEstimatedExchanges.calculateCost(CostCalculatorWithEstimatedExchanges.java:72)\r\n\tat com.facebook.presto.cost.CachingCostProvider.calculateCumulativeCost(CachingCostProvider.java:103)\r\n\tat com.facebook.presto.cost.CachingCostProvider.getCumulativeCost(CachingCostProvider.java:80)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.DetermineJoinDistributionType.getJoinNodeWithCost(DetermineJoinDistributionType.java:133)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.DetermineJoinDistributionType.getCostBasedJoin(DetermineJoinDistributionType.java:77)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.DetermineJoinDistributionType.apply(DetermineJoinDistributionType.java:66)\r\n\tat com.facebook.presto.sql.planner.iterative.rule.DetermineJoinDistributionType.apply(DetermineJoinDistributionType.java:43)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.transform(IterativeOptimizer.java:165)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreNode(IterativeOptimizer.java:138)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:103)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:185)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:105)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreChildren(IterativeOptimizer.java:185)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.exploreGroup(IterativeOptimizer.java:105)\r\n\tat com.facebook.presto.sql.planner.iterative.IterativeOptimizer.optimize(IterativeOptimizer.java:94)\r\n\tat com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:150)\r\n\tat com.facebook.presto.sql.planner.LogicalPlanner.plan(LogicalPlanner.java:139)\r\n\tat com.facebook.presto.execution.SqlQueryExecution.doAnalyzeQuery(SqlQueryExecution.java:362)\r\n\tat com.facebook.presto.execution.SqlQueryExecution.analyzeQuery(SqlQueryExecution.java:347)\r\n\tat com.facebook.presto.execution.SqlQueryExecution.start(SqlQueryExecution.java:291)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n```\r\nThe safe solution is to mark the nullsCount as unavailable when hive return -1 for it, instead of just failing user's query!\r\n**Anytime we should not fail users' query when statistic information is not present or looks illegal! Illegal statistics information should be marked as absent **  "},"comment":{"url":"https://api.github.com/repos/prestodb/presto/issues/comments/426415291","html_url":"https://github.com/prestodb/presto/pull/11549#issuecomment-426415291","issue_url":"https://api.github.com/repos/prestodb/presto/issues/11549","id":426415291,"node_id":"MDEyOklzc3VlQ29tbWVudDQyNjQxNTI5MQ==","user":{"login":"findepi","id":144328,"node_id":"MDQ6VXNlcjE0NDMyOA==","avatar_url":"https://avatars0.githubusercontent.com/u/144328?v=4","gravatar_id":"","url":"https://api.github.com/users/findepi","html_url":"https://github.com/findepi","followers_url":"https://api.github.com/users/findepi/followers","following_url":"https://api.github.com/users/findepi/following{/other_user}","gists_url":"https://api.github.com/users/findepi/gists{/gist_id}","starred_url":"https://api.github.com/users/findepi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/findepi/subscriptions","organizations_url":"https://api.github.com/users/findepi/orgs","repos_url":"https://api.github.com/users/findepi/repos","events_url":"https://api.github.com/users/findepi/events{/privacy}","received_events_url":"https://api.github.com/users/findepi/received_events","type":"User","site_admin":false},"created_at":"2018-10-02T20:18:33Z","updated_at":"2018-10-02T20:18:33Z","author_association":"MEMBER","body":"FWIW: \r\n\r\n(i'm linking to today's `master` head commit, so that line markers remain accurate)\r\n\r\nhttps://github.com/apache/impala/blob/ea2809f5ddcf48f3f41dcd12743e8a17b4ea8cd7/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java#L128-L130\r\n\r\nhttps://github.com/apache/impala/blob/ea2809f5ddcf48f3f41dcd12743e8a17b4ea8cd7/fe/src/main/java/org/apache/impala/analysis/ComputeStatsStmt.java#L253-L263\r\n\r\n(I don't understand why counting NULLs in Impala should make stats 2x slower.. I always thought NDVs is what is hard to compute.)"}}
{"actor":{"display_login":"findepi"}}