{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/5420","repository_url":"https://api.github.com/repos/druid-io/druid","labels_url":"https://api.github.com/repos/druid-io/druid/issues/5420/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/5420/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/5420/events","html_url":"https://github.com/druid-io/druid/issues/5420","id":299787120,"number":5420,"title":"Publishing segments with Kafka Index service failed","user":{"login":"herlenashavi","id":10608906,"avatar_url":"https://avatars2.githubusercontent.com/u/10608906?v=4","gravatar_id":"","url":"https://api.github.com/users/herlenashavi","html_url":"https://github.com/herlenashavi","followers_url":"https://api.github.com/users/herlenashavi/followers","following_url":"https://api.github.com/users/herlenashavi/following{/other_user}","gists_url":"https://api.github.com/users/herlenashavi/gists{/gist_id}","starred_url":"https://api.github.com/users/herlenashavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/herlenashavi/subscriptions","organizations_url":"https://api.github.com/users/herlenashavi/orgs","repos_url":"https://api.github.com/users/herlenashavi/repos","events_url":"https://api.github.com/users/herlenashavi/events{/privacy}","received_events_url":"https://api.github.com/users/herlenashavi/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-23T17:20:53Z","updated_at":"2018-02-24T16:16:53Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\n\r\nI'm using Druid's Kafka indexing service to ingest data from Kafka into Druid. After running for about 10 hours I see some ingestion tasks failing with the following error.\r\n\r\nDruid Version: 0.11.0\r\nHDFS Version: 2.7.4 (Should this be 2.7.3 since the hadoop client being used in Druid 0.11.0 is 2.7.3?)\r\n\r\n```\r\nio.druid.java.util.common.ISE: Transaction failure publishing segments, aborting\r\n\tat io.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:589) ~[?:?]\r\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:436) [druid-indexing-service-0.11.0.jar:0.11.0]\r\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:408) [druid-indexing-service-0.11.0.jar:0.11.0]\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_151]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]\r\n\tat java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]\r\n2018-02-23T13:10:06,384 INFO [task-runner-0-priority-0] io.druid.indexing.overlord.TaskRunnerUtils - Task [index_kafka_table_dea2181c87cb11b_cpmmkfgp] status changed to [FAILED].\r\n2018-02-23T13:10:06,387 INFO [task-runner-0-priority-0] io.druid.indexing.worker.executor.ExecutorLifecycle - Task completed with status: {\r\n  \"id\" : \"index_kafka_table_dea2181c87cb11b_cpmmkfgp\",\r\n  \"status\" : \"FAILED\",\r\n  \"duration\" : 623346\r\n}\r\n```\r\n\r\nI see a whole bunch of HDFS errors reported in the task logs about failure to rename temp index file.\r\n\r\n```\r\n2018-02-23T13:10:03,648 INFO [appenderator_merge_0] io.druid.storage.hdfs.HdfsDataSegmentPusher - Unable to rename temp Index file[hdfs://192.200.101.101:54310/druid/segments/table/6913119fe3764213bef1b26fa4258380/115_descriptor.json] to final segment path [hdfs://192.200.101.101:54310/druid/segments/table/20180223T123000.000Z_20180223T124000.000Z/2018-02-23T12_30_01.070Z/115_descriptor.json]. It is already pushed by a replica task.\r\n2018-02-23T13:10:03,650 WARN [appenderator_merge_0] org.apache.hadoop.fs.HadoopFsWrapper - Failed to rename [hdfs://192.200.101.101:54310/druid/segments/table/6913119fe3764213bef1b26fa4258380/115_index.zip] to [hdfs://192.200.101.101:54310/druid/segments/table/20180223T123000.000Z_20180223T124000.000Z/2018-02-23T12_30_01.070Z/115_index.zip].\r\norg.apache.hadoop.fs.FileAlreadyExistsException: rename destination /druid/segments/table/20180223T123000.000Z_20180223T124000.000Z/2018-02-23T12_30_01.070Z/115_index.zip already exists\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.validateOverwrite(FSDirRenameOp.java:539)\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.unprotectedRenameTo(FSDirRenameOp.java:382)\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.renameTo(FSDirRenameOp.java:299)\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.renameToInt(FSDirRenameOp.java:244)\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3638)\r\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename2(NameNodeRpcServer.java:914)\r\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename2(ClientNamenodeProtocolServerSideTranslatorPB.java:599)\r\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\r\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)\r\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\r\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)\r\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\r\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)\r\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)\r\n\r\n\tat sun.reflect.GeneratedConstructorAccessor78.newInstance(Unknown Source) ~[?:?]\r\n\tat sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:1.8.0_151]\r\n\tat java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_151]\r\n\tat org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) ~[?:?]\r\n\tat org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:73) ~[?:?]\r\n\tat org.apache.hadoop.hdfs.DFSClient.rename(DFSClient.java:1991) ~[?:?]\r\n\tat org.apache.hadoop.hdfs.DistributedFileSystem.rename(DistributedFileSystem.java:659) ~[?:?]\r\n\tat org.apache.hadoop.fs.HadoopFsWrapper.rename(HadoopFsWrapper.java:51) ~[?:?]\r\n\tat io.druid.storage.hdfs.HdfsDataSegmentPusher.copyFilesWithChecks(HdfsDataSegmentPusher.java:167) ~[?:?]\r\n\tat io.druid.storage.hdfs.HdfsDataSegmentPusher.push(HdfsDataSegmentPusher.java:149) ~[?:?]\r\n\tat io.druid.segment.realtime.appenderator.AppenderatorImpl.lambda$mergeAndPush$0(AppenderatorImpl.java:610) ~[druid-server-0.11.0.jar:0.11.0]\r\n\tat io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:63) [java-util-0.11.0.jar:0.11.0]\r\n\tat io.druid.java.util.common.RetryUtils.retry(RetryUtils.java:81) [java-util-0.11.0.jar:0.11.0]\r\n\tat io.druid.segment.realtime.appenderator.AppenderatorImpl.mergeAndPush(AppenderatorImpl.java:609) [druid-server-0.11.0.jar:0.11.0]\r\n\tat io.druid.segment.realtime.appenderator.AppenderatorImpl.access$600(AppenderatorImpl.java:100) [druid-server-0.11.0.jar:0.11.0]\r\n\tat io.druid.segment.realtime.appenderator.AppenderatorImpl$3.apply(AppenderatorImpl.java:496) [druid-server-0.11.0.jar:0.11.0]\r\n\tat io.druid.segment.realtime.appenderator.AppenderatorImpl$3.apply(AppenderatorImpl.java:484) [druid-server-0.11.0.jar:0.11.0]\r\n\tat com.google.common.util.concurrent.Futures$1.apply(Futures.java:713) [guava-16.0.1.jar:?]\r\n\tat com.google.common.util.concurrent.Futures$ChainingListenableFuture.run(Futures.java:861) [guava-16.0.1.jar:?]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_151]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_151]\r\n\tat java.lang.Thread.run(Thread.java:748) [?:1.8.0_151]\r\nCaused by: org.apache.hadoop.ipc.RemoteException: rename destination /druid/segments/table/20180223T123000.000Z_20180223T124000.000Z/2018-02-23T12_30_01.070Z/115_index.zip already exists\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.validateOverwrite(FSDirRenameOp.java:539)\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.unprotectedRenameTo(FSDirRenameOp.java:382)\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.renameTo(FSDirRenameOp.java:299)\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSDirRenameOp.renameToInt(FSDirRenameOp.java:244)\r\n\tat org.apache.hadoop.hdfs.server.namenode.FSNamesystem.renameTo(FSNamesystem.java:3638)\r\n\tat org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.rename2(NameNodeRpcServer.java:914)\r\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.rename2(ClientNamenodeProtocolServerSideTranslatorPB.java:599)\r\n\tat org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)\r\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)\r\n\tat org.apache.hadoop.ipc.RPC$Server.call(RPC.java:982)\r\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2217)\r\n\tat org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2213)\r\n\tat java.security.AccessController.doPrivileged(Native Method)\r\n\tat javax.security.auth.Subject.doAs(Subject.java:422)\r\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1746)\r\n\tat org.apache.hadoop.ipc.Server$Handler.run(Server.java:2213)\r\n\r\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1475) ~[?:?]\r\n\tat org.apache.hadoop.ipc.Client.call(Client.java:1412) ~[?:?]\r\n\tat org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229) ~[?:?]\r\n\tat com.sun.proxy.$Proxy64.rename2(Unknown Source) ~[?:?]\r\n\tat org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.rename2(ClientNamenodeProtocolTranslatorPB.java:513) ~[?:?]\r\n\tat sun.reflect.GeneratedMethodAccessor129.invoke(Unknown Source) ~[?:?]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_151]\r\n\tat java.lang.reflect.Method.invoke(Method.java:498) ~[?:1.8.0_151]\r\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191) ~[?:?]\r\n\tat org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102) ~[?:?]\r\n\tat com.sun.proxy.$Proxy65.rename2(Unknown Source) ~[?:?]\r\n\tat org.apache.hadoop.hdfs.DFSCl\r\n```\r\n\r\nHas anyone else hit this issue? I see this was a bug in a previous version of Druid which was fixed #3786. Is this an issue with the Hadoop version(2.7.4) I'm using?"},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/368239244","html_url":"https://github.com/druid-io/druid/issues/5420#issuecomment-368239244","issue_url":"https://api.github.com/repos/druid-io/druid/issues/5420","id":368239244,"user":{"login":"herlenashavi","id":10608906,"avatar_url":"https://avatars2.githubusercontent.com/u/10608906?v=4","gravatar_id":"","url":"https://api.github.com/users/herlenashavi","html_url":"https://github.com/herlenashavi","followers_url":"https://api.github.com/users/herlenashavi/followers","following_url":"https://api.github.com/users/herlenashavi/following{/other_user}","gists_url":"https://api.github.com/users/herlenashavi/gists{/gist_id}","starred_url":"https://api.github.com/users/herlenashavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/herlenashavi/subscriptions","organizations_url":"https://api.github.com/users/herlenashavi/orgs","repos_url":"https://api.github.com/users/herlenashavi/repos","events_url":"https://api.github.com/users/herlenashavi/events{/privacy}","received_events_url":"https://api.github.com/users/herlenashavi/received_events","type":"User","site_admin":false},"created_at":"2018-02-24T16:16:53Z","updated_at":"2018-02-24T16:16:53Z","author_association":"NONE","body":"Just an update, I tried testing this with Hadoop version 2.7.1 and HadoopFSWrapper compiled with hadoop 2.3.0. Still get the same error. While using Druid 0.11.0, which version of Hadoop client (2.3.0 or 2.4.0) is recommended with HDFS version 2.7.1?"}}
{"actor":{"display_login":"herlenashavi"}}