{"action":"created","issue":{"url":"https://api.github.com/repos/greenplum-db/gpdb/issues/776","repository_url":"https://api.github.com/repos/greenplum-db/gpdb","labels_url":"https://api.github.com/repos/greenplum-db/gpdb/issues/776/labels{/name}","comments_url":"https://api.github.com/repos/greenplum-db/gpdb/issues/776/comments","events_url":"https://api.github.com/repos/greenplum-db/gpdb/issues/776/events","html_url":"https://github.com/greenplum-db/gpdb/pull/776","id":155884402,"number":776,"title":"GPHDFS insert or select query hangs indefinitely","user":{"login":"randomtask1155","id":1808864,"avatar_url":"https://avatars.githubusercontent.com/u/1808864?v=3","gravatar_id":"","url":"https://api.github.com/users/randomtask1155","html_url":"https://github.com/randomtask1155","followers_url":"https://api.github.com/users/randomtask1155/followers","following_url":"https://api.github.com/users/randomtask1155/following{/other_user}","gists_url":"https://api.github.com/users/randomtask1155/gists{/gist_id}","starred_url":"https://api.github.com/users/randomtask1155/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/randomtask1155/subscriptions","organizations_url":"https://api.github.com/users/randomtask1155/orgs","repos_url":"https://api.github.com/users/randomtask1155/repos","events_url":"https://api.github.com/users/randomtask1155/events{/privacy}","received_events_url":"https://api.github.com/users/randomtask1155/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":9,"created_at":"2016-05-20T04:21:44Z","updated_at":"2016-05-26T00:15:40Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/greenplum-db/gpdb/pulls/776","html_url":"https://github.com/greenplum-db/gpdb/pull/776","diff_url":"https://github.com/greenplum-db/gpdb/pull/776.diff","patch_url":"https://github.com/greenplum-db/gpdb/pull/776.patch"},"body":"I was able to reproduce this bug by bypassing loginSecureHadoop() that sets up the principals for the global login context. DFSClient then travels down the code path where hadoop.security.logger sends debug messages to the console resulting in a deadlock.\r\n\r\nto work around this in my repo i simply override the $HADOOP_CONF/log4j.properties by adding options to GP_JAVA_OPT in $GPHOME/lib/hadoop/hadoop_env.sh.  We don't want debug logs going to stdout so if users modify their hadoop log4j we override to the settings to protect from deadlock. \r\n\r\n```\r\nThread 247741: (state = IN_NATIVE)\r\n - java.io.FileOutputStream.writeBytes(byte[], int, int, boolean) @bci=0 (Interpreted frame)\r\n - java.io.FileOutputStream.write(byte[], int, int) @bci=20, line=345 (Interpreted frame)\r\n - java.io.BufferedOutputStream.write(byte[], int, int) @bci=20, line=122 (Interpreted frame)\r\n - java.io.PrintStream.write(byte[], int, int) @bci=16, line=480 (Interpreted frame)\r\n - sun.nio.cs.StreamEncoder.writeBytes() @bci=120, line=221 (Interpreted frame)\r\n - sun.nio.cs.StreamEncoder.implFlushBuffer() @bci=11, line=291 (Interpreted frame)\r\n - sun.nio.cs.StreamEncoder.implFlush() @bci=1, line=295 (Interpreted frame)\r\n - sun.nio.cs.StreamEncoder.flush() @bci=12, line=141 (Interpreted frame)\r\n - java.io.OutputStreamWriter.flush() @bci=4, line=229 (Interpreted frame)\r\n - org.apache.log4j.helpers.QuietWriter.flush() @bci=4, line=59 (Interpreted frame)\r\n - org.apache.log4j.WriterAppender.subAppend(org.apache.log4j.spi.LoggingEvent) @bci=85, line=324 (Interpreted frame)\r\n - org.apache.log4j.WriterAppender.append(org.apache.log4j.spi.LoggingEvent) @bci=10, line=162 (Interpreted frame)\r\n - org.apache.log4j.AppenderSkeleton.doAppend(org.apache.log4j.spi.LoggingEvent) @bci=106, line=251 (Interpreted frame)\r\n - org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(org.apache.log4j.spi.LoggingEvent) @bci=41, line=66 (Interpreted frame)\r\n - org.apache.log4j.Category.callAppenders(org.apache.log4j.spi.LoggingEvent) @bci=26, line=206 (Interpreted frame)\r\n - org.apache.log4j.Category.forcedLog(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=14, line=391 (Interpreted frame)\r\n - org.apache.log4j.Category.log(java.lang.String, org.apache.log4j.Priority, java.lang.Object, java.lang.Throwable) @bci=34, line=856 (Interpreted frame)\r\n - org.apache.commons.logging.impl.Log4JLogger.debug(java.lang.Object, java.lang.Throwable) @bci=12, line=188 (Interpreted frame)\r\n - org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=458, line=139 (Interpreted frame)\r\n - com.sun.proxy.$Proxy10.getDelegationToken(org.apache.hadoop.io.Text) @bci=16 (Interpreted frame)\r\n - org.apache.hadoop.hdfs.DFSClient.getDelegationToken(org.apache.hadoop.io.Text) @bci=26, line=850 (Interpreted frame)\r\n - org.apache.hadoop.hdfs.DistributedFileSystem.getDelegationToken(java.lang.String) @bci=20, line=1328 (Interpreted frame)\r\n - org.apache.hadoop.fs.FileSystem.collectDelegationTokens(java.lang.String, org.apache.hadoop.security.Credentials, java.util.List) @bci=37, line=526 (Interpreted frame)\r\n - org.apache.hadoop.fs.FileSystem.addDelegationTokens(java.lang.String, org.apache.hadoop.security.Credentials) @bci=24, line=504 (Interpreted frame)\r\n - org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(org.apache.hadoop.fs.FileSystem, org.apache.hadoop.security.Credentials, org.apache.hadoop.conf.Configuration) @bci=34, line=121 (Interpreted frame)\r\n - org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodesInternal(org.apache.hadoop.security.Credentials, org.apache.hadoop.fs.Path[], org.apache.hadoop.conf.Configuration) @bci=86, line=100 (Interpreted frame)\r\n - org.apache.hadoop.mapreduce.security.TokenCache.obtainTokensForNamenodes(org.apache.hadoop.security.Credentials, org.apache.hadoop.fs.Path[], org.apache.hadoop.conf.Configuration) @bci=10, line=80 (Interpreted frame)\r\n - org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(org.apache.hadoop.mapreduce.JobContext) @bci=41, line=235 (Interpreted frame)\r\n - org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(org.apache.hadoop.mapreduce.JobContext) @bci=29, line=340 (Interpreted frame)\r\n - com.emc.greenplum.gpdb.hdfsconnector.HDFSReader.assignSplits() @bci=61, line=245 (Interpreted frame)\r\n - com.emc.greenplum.gpdb.hdfsconnector.HDFSReader.doRead() @bci=15, line=157 (Interpreted frame)\r\n - com.emc.greenplum.gpdb.hdfsconnector.HDFSReader.main(java.lang.String[]) @bci=10, line=258 (Interpreted frame)\r\n\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/greenplum-db/gpdb/issues/comments/221744755","html_url":"https://github.com/greenplum-db/gpdb/pull/776#issuecomment-221744755","issue_url":"https://api.github.com/repos/greenplum-db/gpdb/issues/776","id":221744755,"user":{"login":"randomtask1155","id":1808864,"avatar_url":"https://avatars.githubusercontent.com/u/1808864?v=3","gravatar_id":"","url":"https://api.github.com/users/randomtask1155","html_url":"https://github.com/randomtask1155","followers_url":"https://api.github.com/users/randomtask1155/followers","following_url":"https://api.github.com/users/randomtask1155/following{/other_user}","gists_url":"https://api.github.com/users/randomtask1155/gists{/gist_id}","starred_url":"https://api.github.com/users/randomtask1155/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/randomtask1155/subscriptions","organizations_url":"https://api.github.com/users/randomtask1155/orgs","repos_url":"https://api.github.com/users/randomtask1155/repos","events_url":"https://api.github.com/users/randomtask1155/events{/privacy}","received_events_url":"https://api.github.com/users/randomtask1155/received_events","type":"User","site_admin":false},"created_at":"2016-05-26T00:15:40Z","updated_at":"2016-05-26T00:15:40Z","body":"tracker ticket and jira exist here\n\nhttps://jira-pivotal.atlassian.net/browse/MPP-26270\nhttps://www.pivotaltracker.com/n/projects/1382376/stories/116948321\n\nOn Wed, May 25, 2016 at 2:46 PM, Andreas Scherbaum <notifications@github.com\n> wrote:\n\n> Let's fix the immediate issue, and can you create a Tracker ticket to keep\n> track of the log4j/GPDB issue?\n>\n> —\n> You are receiving this because you authored the thread.\n> Reply to this email directly or view it on GitHub\n> <https://github.com/greenplum-db/gpdb/pull/776#issuecomment-221685740>\n>\n"}}
{"actor":{"display_login":"randomtask1155"}}