{"action":"closed","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-output-webhdfs/issues/15","repository_url":"https://api.github.com/repos/logstash-plugins/logstash-output-webhdfs","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-output-webhdfs/issues/15/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-output-webhdfs/issues/15/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-output-webhdfs/issues/15/events","html_url":"https://github.com/logstash-plugins/logstash-output-webhdfs/issues/15","id":133721643,"number":15,"title":"Write errors when running vs HDP 2.3 (==Hadoop 2.7)","user":{"login":"Alex-Ikanow","id":851950,"avatar_url":"https://avatars.githubusercontent.com/u/851950?v=3","gravatar_id":"","url":"https://api.github.com/users/Alex-Ikanow","html_url":"https://github.com/Alex-Ikanow","followers_url":"https://api.github.com/users/Alex-Ikanow/followers","following_url":"https://api.github.com/users/Alex-Ikanow/following{/other_user}","gists_url":"https://api.github.com/users/Alex-Ikanow/gists{/gist_id}","starred_url":"https://api.github.com/users/Alex-Ikanow/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Alex-Ikanow/subscriptions","organizations_url":"https://api.github.com/users/Alex-Ikanow/orgs","repos_url":"https://api.github.com/users/Alex-Ikanow/repos","events_url":"https://api.github.com/users/Alex-Ikanow/events{/privacy}","received_events_url":"https://api.github.com/users/Alex-Ikanow/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":{"login":"dstore-dbap","id":4869673,"avatar_url":"https://avatars.githubusercontent.com/u/4869673?v=3","gravatar_id":"","url":"https://api.github.com/users/dstore-dbap","html_url":"https://github.com/dstore-dbap","followers_url":"https://api.github.com/users/dstore-dbap/followers","following_url":"https://api.github.com/users/dstore-dbap/following{/other_user}","gists_url":"https://api.github.com/users/dstore-dbap/gists{/gist_id}","starred_url":"https://api.github.com/users/dstore-dbap/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dstore-dbap/subscriptions","organizations_url":"https://api.github.com/users/dstore-dbap/orgs","repos_url":"https://api.github.com/users/dstore-dbap/repos","events_url":"https://api.github.com/users/dstore-dbap/events{/privacy}","received_events_url":"https://api.github.com/users/dstore-dbap/received_events","type":"User","site_admin":false},"milestone":null,"comments":15,"created_at":"2016-02-15T14:11:36Z","updated_at":"2016-02-22T14:35:53Z","closed_at":"2016-02-22T14:35:53Z","body":"I'm running a very boring configuration in Logstash 2.1.2 that reads from a local CSV file and writes to webhdfs (connecting to an 8 node HDFS cluster created by a vanilla HDP install)\r\n\r\n```\r\ninput { file { ... } }\r\nfilter { csv { ... } }\r\noutput {\r\n  if [sourceKey] == \"<<hdfs_output_key>>\" {\r\n \t    webhdfs {\r\n\t        path => '/app/aleph2/data/aleph2_testing/56bc4f59e4b026e56f0a3cc5/ttm/test/bucket1/managed_bucket/import/ready/ls_input_%{+yyyy.MM.dd.hh}.json'\r\n\t        host => '<<webhdfs.server.name>>'\r\n\t        port => '50075'\r\n\t        user => 'tomcat'\r\n\t        codec => 'json'\r\n\t        flush_size => 33554432\r\n\t        idle_flush_time => 10\r\n\t    }\r\n   }\r\n}\r\n```\r\n\r\nOn some ~10MB file with ~200K lines\r\n\r\nI get a non-stop stream of the following errors\r\n\r\n```\r\n{:timestamp=>\"2016-02-11T19:51:31.812000-0500\", :message=>\"webhdfs write caused an exception: Failed to connect to host <<webhdfs.server.name>>:50075, Connection reset by peer - Connection reset by peer. Maybe you should increase retry_interval or reduce number of workers. Retrying...\", :level=>:warn}\r\n...\r\n{:timestamp=>\"2016-02-11T19:51:33.808000-0500\", :message=>\"Max write retries reached. Events will be discarded. Exception: Failed to connect to host <<webhdfs.server.name>>:50075, Connection reset by peer - Connection reset by peer\", :level=>:error}\r\n```\r\n\r\nA random number of the events are discarded each run, ~25-50%\r\n\r\nLooking at the server logs, it's filled with things like the following:\r\n\r\n```\r\n2016-02-11 08:54:24,863 WARN  hdfs.StateChange (FSNamesystem.java:appendFileInternal(2690)) - DIR* NameSystem.append: Failed to APPEND_FILE /app/aleph2/data/aleph2_testing/56bc4f59e4b026e56f0a3cc5/ttm/test/bucket1/managed_bucket/import/ready/ls_input_2016.02.11.01.json for DFSClient_NONMAPREDUCE_154673148_72 on 10.12.16.101 because this file lease is currently owned by DFSClient_NONMAPREDUCE_1522645774_64 on 10.12.16.101\r\n```\r\n\r\nWhich appears to be caused by multiple threads accessing the same file according to Google. And there's a related but cryptic comment in the the source code here (sic):\r\n\r\n```\r\n  def write_data(path, data)\r\n    # Retry max_retry times. This can solve problems like leases being hold by another process. Sadly this is no\r\n    # KNOWN_ERROR in rubys webhdfs client.\r\n```\r\n\r\nSo:\r\n* Is this actually not an issue, ie or am I just doing something stupid?\r\n* If it is a legit issue, Is there a workaround? The error says `reduce number of workers`, but that presumably means the global number of worker threads, which would cripple the performance of the other logstash configs\r\n\r\nThanks in advance of any insight/help!"}}