{"action":"created","issue":{"url":"https://api.github.com/repos/uber/hudi/issues/544","repository_url":"https://api.github.com/repos/uber/hudi","labels_url":"https://api.github.com/repos/uber/hudi/issues/544/labels{/name}","comments_url":"https://api.github.com/repos/uber/hudi/issues/544/comments","events_url":"https://api.github.com/repos/uber/hudi/issues/544/events","html_url":"https://github.com/uber/hudi/issues/544","id":395138737,"node_id":"MDU6SXNzdWUzOTUxMzg3Mzc=","number":544,"title":"Hive sync error with could not find any data file written for commit","user":{"login":"louisliu318","id":9956334,"node_id":"MDQ6VXNlcjk5NTYzMzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/9956334?v=4","gravatar_id":"","url":"https://api.github.com/users/louisliu318","html_url":"https://github.com/louisliu318","followers_url":"https://api.github.com/users/louisliu318/followers","following_url":"https://api.github.com/users/louisliu318/following{/other_user}","gists_url":"https://api.github.com/users/louisliu318/gists{/gist_id}","starred_url":"https://api.github.com/users/louisliu318/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/louisliu318/subscriptions","organizations_url":"https://api.github.com/users/louisliu318/orgs","repos_url":"https://api.github.com/users/louisliu318/repos","events_url":"https://api.github.com/users/louisliu318/events{/privacy}","received_events_url":"https://api.github.com/users/louisliu318/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2019-01-02T02:59:57Z","updated_at":"2019-01-04T07:21:16Z","closed_at":null,"author_association":"NONE","body":"Here is my use case:\r\nI am using spark streaming to write data being received from Kafka to hoodie table, then I sync to hive table. And I using the non-partitioned hive table. I had set the `KeyGenerator ` with `NonpartitionedKeyGenerator.class`. When I synced to hive, the following error occurred. What's the reason causing this.  Below is my error ：\r\n```\r\n2019-01-02 10:04:02,511 ERROR scheduler.JobScheduler (Logging.scala:logError(91)) - Error running job streaming job 1546394640000 ms.0\r\njava.lang.IllegalArgumentException: Could not find any data file written for commit [20190102100400__commit__COMPLETED], could not get schema for dataset hdfs://nns-off/databus/hudi/tables/databus_realtime_databus_realtime_databus_sub_hd_t_hudi_sub_hd, Metadata :HoodieCommitMetadata{partitionToWriteStats={}, compacted=false, extraMetadataMap={ROLLING_STAT={\r\n  \"partitionToRollingStats\" : {\r\n    \"\" : {\r\n      \"9e163bc3-c14f-4c46-937a-67134b26f7e2\" : {\r\n        \"fileId\" : \"9e163bc3-c14f-4c46-937a-67134b26f7e2\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434448\r\n      },\r\n      \"0ba4d519-6e8d-42f6-b27e-f027b45f5b06\" : {\r\n        \"fileId\" : \"0ba4d519-6e8d-42f6-b27e-f027b45f5b06\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434436\r\n      },\r\n      \"a4d2eaf3-6027-4093-9621-b40cc2ebcb8b\" : {\r\n        \"fileId\" : \"a4d2eaf3-6027-4093-9621-b40cc2ebcb8b\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434436\r\n      },\r\n      \"0571876c-78a6-4263-8976-c687ad2e0ba9\" : {\r\n        \"fileId\" : \"0571876c-78a6-4263-8976-c687ad2e0ba9\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434435\r\n      },\r\n      \"321509ba-814e-4d9d-a135-dfe958490ec8\" : {\r\n        \"fileId\" : \"321509ba-814e-4d9d-a135-dfe958490ec8\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434435\r\n      },\r\n      \"8c36a009-0042-40f5-abb3-91f891cec775\" : {\r\n        \"fileId\" : \"8c36a009-0042-40f5-abb3-91f891cec775\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434450\r\n      },\r\n      \"15adf256-c64d-4dd9-9d45-4d911b5763ba\" : {\r\n        \"fileId\" : \"15adf256-c64d-4dd9-9d45-4d911b5763ba\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434435\r\n      },\r\n      \"17ef27e0-dedc-4b8b-9997-da7bc65a61d1\" : {\r\n        \"fileId\" : \"17ef27e0-dedc-4b8b-9997-da7bc65a61d1\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434436\r\n      },\r\n      \"4139fed7-ce24-4e16-ba2f-9dd30b5a7100\" : {\r\n        \"fileId\" : \"4139fed7-ce24-4e16-ba2f-9dd30b5a7100\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434436\r\n      },\r\n      \"2284b3dc-4137-44b0-a71f-de5fbd14a7ff\" : {\r\n        \"fileId\" : \"2284b3dc-4137-44b0-a71f-de5fbd14a7ff\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434436\r\n      },\r\n      \"d10310dd-7b4a-4669-9523-e7e9ac01ff17\" : {\r\n        \"fileId\" : \"d10310dd-7b4a-4669-9523-e7e9ac01ff17\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434436\r\n      },\r\n      \"68592d6b-82f7-4ea6-8cce-8b21a07d4b1a\" : {\r\n        \"fileId\" : \"68592d6b-82f7-4ea6-8cce-8b21a07d4b1a\",\r\n        \"inserts\" : 1,\r\n        \"upserts\" : 0,\r\n        \"deletes\" : 0,\r\n        \"totalInputWriteBytesToDisk\" : 0,\r\n        \"totalInputWriteBytesOnDisk\" : 434441\r\n      }\r\n    }\r\n  },\r\n  \"actionType\" : \"commit\"\r\n}}}\r\n\tat com.uber.hoodie.hive.HoodieHiveClient.lambda$getDataSchema$1(HoodieHiveClient.java:317)\r\n\tat java.util.Optional.orElseThrow(Optional.java:290)\r\n\tat com.uber.hoodie.hive.HoodieHiveClient.getDataSchema(HoodieHiveClient.java:315)\r\n\tat com.uber.hoodie.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:94)\r\n\tat com.uber.hoodie.hive.HiveSyncTool.syncHoodieTable(HiveSyncTool.java:68)\r\n\tat com.lianjia.dtarch.databus.streaming.hudi.service.HudiService.syncToHive(HudiService.java:79)\r\n\tat com.lianjia.dtarch.databus.streaming.hudi.service.HudiService.writeWithCompactAndSync(HudiService.java:58)\r\n\tat com.lianjia.dtarch.databus.streaming.hudi.KfkHudiConsumer.lambda$saveToHudi$c06d719c$1(KfkHudiConsumer.java:161)\r\n\tat org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)\r\n\tat org.apache.spark.streaming.api.java.JavaDStreamLike$$anonfun$foreachRDD$1.apply(JavaDStreamLike.scala:272)\r\n\tat org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)\r\n\tat org.apache.spark.streaming.dstream.DStream$$anonfun$foreachRDD$1$$anonfun$apply$mcV$sp$3.apply(DStream.scala:628)\r\n\tat org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ForEachDStream.scala:51)\r\n\tat org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)\r\n\tat org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1$$anonfun$apply$mcV$sp$1.apply(ForEachDStream.scala:51)\r\n\tat org.apache.spark.streaming.dstream.DStream.createRDDWithLocalProperties(DStream.scala:416)\r\n\tat org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply$mcV$sp(ForEachDStream.scala:50)\r\n\tat org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)\r\n\tat org.apache.spark.streaming.dstream.ForEachDStream$$anonfun$1.apply(ForEachDStream.scala:50)\r\n\tat scala.util.Try$.apply(Try.scala:192)\r\n\tat org.apache.spark.streaming.scheduler.Job.run(Job.scala:39)\r\n\tat org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply$mcV$sp(JobScheduler.scala:257)\r\n\tat org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)\r\n\tat org.apache.spark.streaming.scheduler.JobScheduler$JobHandler$$anonfun$run$1.apply(JobScheduler.scala:257)\r\n\tat scala.util.DynamicVariable.withValue(DynamicVariable.scala:58)\r\n\tat org.apache.spark.streaming.scheduler.JobScheduler$JobHandler.run(JobScheduler.scala:256)\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n```"},"comment":{"url":"https://api.github.com/repos/uber/hudi/issues/comments/451369229","html_url":"https://github.com/uber/hudi/issues/544#issuecomment-451369229","issue_url":"https://api.github.com/repos/uber/hudi/issues/544","id":451369229,"node_id":"MDEyOklzc3VlQ29tbWVudDQ1MTM2OTIyOQ==","user":{"login":"louisliu318","id":9956334,"node_id":"MDQ6VXNlcjk5NTYzMzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/9956334?v=4","gravatar_id":"","url":"https://api.github.com/users/louisliu318","html_url":"https://github.com/louisliu318","followers_url":"https://api.github.com/users/louisliu318/followers","following_url":"https://api.github.com/users/louisliu318/following{/other_user}","gists_url":"https://api.github.com/users/louisliu318/gists{/gist_id}","starred_url":"https://api.github.com/users/louisliu318/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/louisliu318/subscriptions","organizations_url":"https://api.github.com/users/louisliu318/orgs","repos_url":"https://api.github.com/users/louisliu318/repos","events_url":"https://api.github.com/users/louisliu318/events{/privacy}","received_events_url":"https://api.github.com/users/louisliu318/received_events","type":"User","site_admin":false},"created_at":"2019-01-04T07:21:16Z","updated_at":"2019-01-04T07:21:16Z","author_association":"NONE","body":"@n3nash Sorry, I forgot to check that.  Before writing data, first check if the data is empty. If it's empty, skip; else write it. Then the problem went away."}}
{"actor":{"display_login":"louisliu318"}}