{"action":"opened","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/3396","repository_url":"https://api.github.com/repos/druid-io/druid","labels_url":"https://api.github.com/repos/druid-io/druid/issues/3396/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/3396/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/3396/events","html_url":"https://github.com/druid-io/druid/issues/3396","id":172997190,"number":3396,"title":"Hadoop-based Batch Ingestion failures when multiple local tasks run concurrently","user":{"login":"mark1900","id":11822480,"avatar_url":"https://avatars.githubusercontent.com/u/11822480?v=3","gravatar_id":"","url":"https://api.github.com/users/mark1900","html_url":"https://github.com/mark1900","followers_url":"https://api.github.com/users/mark1900/followers","following_url":"https://api.github.com/users/mark1900/following{/other_user}","gists_url":"https://api.github.com/users/mark1900/gists{/gist_id}","starred_url":"https://api.github.com/users/mark1900/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mark1900/subscriptions","organizations_url":"https://api.github.com/users/mark1900/orgs","repos_url":"https://api.github.com/users/mark1900/repos","events_url":"https://api.github.com/users/mark1900/events{/privacy}","received_events_url":"https://api.github.com/users/mark1900/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-08-24T16:18:18Z","updated_at":"2016-08-24T16:18:18Z","closed_at":null,"body":"The following is the exception I occasionally see when attempting to execute multiple Hadoop Local Jobs with Druid.\r\n\r\n```\r\njava.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135\r\n```\r\n\r\nThis issue seems to be related to the Hadoop LocalJobRunner, which means Druid Batch Ingestion Tasks may fail if multiple jobs are run concurrently.  Not sure if an effective work-around can be implemented in Druid.\r\n\r\n### Full Stack Trace\r\n```\r\njava.lang.RuntimeException: java.lang.reflect.InvocationTargetException\r\n\tat com.google.common.base.Throwables.propagate(Throwables.java:160) ~[guava-16.0.1.jar:?]\r\n\tat io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:138) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexing.common.task.HadoopIndexTask.run(HadoopIndexTask.java:206) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:285) [druid-indexing-service-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexing.overlord.ThreadPoolTaskRunner$ThreadPoolTaskRunnerCallable.call(ThreadPoolTaskRunner.java:265) [druid-indexing-service-0.8.3.jar:0.8.3]\r\n\tat java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_71]\r\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0_71]\r\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0_71]\r\n\tat java.lang.Thread.run(Thread.java:745) [?:1.8.0_71]\r\nCaused by: java.lang.reflect.InvocationTargetException\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_71]\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_71]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]\r\n\tat java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]\r\n\tat io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\t... 7 more\r\nCaused by: java.lang.RuntimeException: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135\r\n\tat io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:211) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_71]\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_71]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]\r\n\tat java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]\r\n\tat io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\t... 7 more\r\nCaused by: java.io.IOException: java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135\r\n\tat org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:143) ~[?:?]\r\n\tat org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163) ~[?:?]\r\n\tat org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731) ~[?:?]\r\n\tat org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240) ~[?:?]\r\n\tat org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]\r\n\tat org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]\r\n\tat java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_71]\r\n\tat javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_71]\r\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[?:?]\r\n\tat org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]\r\n\tat io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:199) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_71]\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_71]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]\r\n\tat java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]\r\n\tat io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\t... 7 more\r\nCaused by: java.util.concurrent.ExecutionException: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135\r\n\tat java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_71]\r\n\tat java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[?:1.8.0_71]\r\n\tat org.apache.hadoop.mapred.LocalDistributedCacheManager.setup(LocalDistributedCacheManager.java:139) ~[?:?]\r\n\tat org.apache.hadoop.mapred.LocalJobRunner$Job.<init>(LocalJobRunner.java:163) ~[?:?]\r\n\tat org.apache.hadoop.mapred.LocalJobRunner.submitJob(LocalJobRunner.java:731) ~[?:?]\r\n\tat org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:240) ~[?:?]\r\n\tat org.apache.hadoop.mapreduce.Job$10.run(Job.java:1290) ~[?:?]\r\n\tat org.apache.hadoop.mapreduce.Job$10.run(Job.java:1287) ~[?:?]\r\n\tat java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_71]\r\n\tat javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_71]\r\n\tat org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657) ~[?:?]\r\n\tat org.apache.hadoop.mapreduce.Job.submit(Job.java:1287) ~[?:?]\r\n\tat io.druid.indexer.IndexGeneratorJob.run(IndexGeneratorJob.java:199) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexer.JobHelper.runJobs(JobHelper.java:321) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexer.HadoopDruidIndexerJob.run(HadoopDruidIndexerJob.java:96) ~[druid-indexing-hadoop-0.8.3.jar:0.8.3]\r\n\tat io.druid.indexing.common.task.HadoopIndexTask$HadoopIndexGeneratorInnerProcessing.runTask(HadoopIndexTask.java:259) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_71]\r\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:1.8.0_71]\r\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:1.8.0_71]\r\n\tat java.lang.reflect.Method.invoke(Method.java:497) ~[?:1.8.0_71]\r\n\tat io.druid.indexing.common.task.HadoopTask.invokeForeignLoader(HadoopTask.java:135) ~[druid-indexing-service-0.8.3.jar:0.8.3]\r\n\t... 7 more\r\nCaused by: java.io.IOException: Rename cannot overwrite non empty destination directory /tmp/hadoop-username/mapred/local/1472019105135\r\n\tat org.apache.hadoop.fs.AbstractFileSystem.renameInternal(AbstractFileSystem.java:735) ~[?:?]\r\n\tat org.apache.hadoop.fs.FilterFs.renameInternal(FilterFs.java:236) ~[?:?]\r\n\tat org.apache.hadoop.fs.AbstractFileSystem.rename(AbstractFileSystem.java:678) ~[?:?]\r\n\tat org.apache.hadoop.fs.FileContext.rename(FileContext.java:958) ~[?:?]\r\n\tat org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:366) ~[?:?]\r\n\tat org.apache.hadoop.yarn.util.FSDownload.call(FSDownload.java:62) ~[?:?]\r\n\t... 4 more\r\n```\r\n\r\n### Related Hadoop issues\r\n* [MAPREDUCE-6766](https://issues.apache.org/jira/browse/MAPREDUCE-6766)\r\n* [MAPREDUCE-6685](https://issues.apache.org/jira/browse/MAPREDUCE-6685)\r\n* [MAPREDUCE-6441](https://issues.apache.org/jira/browse/MAPREDUCE-6441)"}}
{"actor":{"display_login":"mark1900"}}