{"action":"closed","issue":{"url":"https://api.github.com/repos/prestodb/presto/issues/5734","repository_url":"https://api.github.com/repos/prestodb/presto","labels_url":"https://api.github.com/repos/prestodb/presto/issues/5734/labels{/name}","comments_url":"https://api.github.com/repos/prestodb/presto/issues/5734/comments","events_url":"https://api.github.com/repos/prestodb/presto/issues/5734/events","html_url":"https://github.com/prestodb/presto/issues/5734","id":166962644,"number":5734,"title":"“delete” will failed “select” ","user":{"login":"linqerli","id":17487596,"avatar_url":"https://avatars.githubusercontent.com/u/17487596?v=3","gravatar_id":"","url":"https://api.github.com/users/linqerli","html_url":"https://github.com/linqerli","followers_url":"https://api.github.com/users/linqerli/followers","following_url":"https://api.github.com/users/linqerli/following{/other_user}","gists_url":"https://api.github.com/users/linqerli/gists{/gist_id}","starred_url":"https://api.github.com/users/linqerli/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/linqerli/subscriptions","organizations_url":"https://api.github.com/users/linqerli/orgs","repos_url":"https://api.github.com/users/linqerli/repos","events_url":"https://api.github.com/users/linqerli/events{/privacy}","received_events_url":"https://api.github.com/users/linqerli/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-07-22T03:17:33Z","updated_at":"2016-11-09T21:48:29Z","closed_at":"2016-11-09T21:48:29Z","body":"https://groups.google.com/forum/#!topic/presto-users/V5oB6CXVB7A\n\nI see Even if hive deletes the hdfs files a day later, “Select” will still failed in presto . because presto will call hive metastore many times to get partitions name or object during the process of execution and analysis of a \"select\" sql;\n1. SQL1: \"Select \\* from A \" start running , and get all partition names (p1,p2,p3...p10) from hive metastore in coordinator which will be stored in HiveTableLayoutHandle.partitions .\n2. SQL2: \"Delete A where dt=p1\", comming and finished. p1 will be removed from metastore and hdfs.\n3. SQL1: \"Select \\* from A \" , get org.apache.hadoop.hive.metastore.api.partition objects of p1,p2,....p10 from hive metastore by HiveSplitManager.getPartitionMetadata ,\n   now presto will throw exception because hive metastore return null for p1 (if partitionCache in CachingHiveMetastore is expired. or it has no data because of frist time， or cache is disabled by hive.metastore-cache-ttl=0s, which also is configured  in some of your clusters );\n   \n   plese check code of HiveSplitManager.getPartitionMetadata\n   \n   Iterable<List<HivePartition>> partitionNameBatches = partitionExponentially(hivePartitions, minPartitionBatchSize, maxPartitionBatchSize);\n       Iterable<List<HivePartitionMetadata>> partitionBatches = transform(partitionNameBatches, partitionBatch -> {\n           Optional<Map<String, Partition>> batch = metastore.getPartitionsByNames(\n                   tableName.getSchemaName(),\n                   tableName.getTableName(),\n                   Lists.transform(partitionBatch, HivePartition::getPartitionId));\n   \n   if (!batch.isPresent()) {\n               throw new PrestoException(HIVE_METASTORE_ERROR, \"Partition metadata not available\");\n   }\n   .................\n   if (partitionBatch.size() != partitions.size()) {\n               throw new PrestoException(INTERNAL_ERROR, format(\"Expected %s partitions but found %s\", partitionBatch.size(), partitions.size()));\n   }\n"}}
{"actor":{"display_login":"martint"}}