{"action":"created","issue":{"url":"https://api.github.com/repos/logstash-plugins/logstash-codec-netflow/issues/134","repository_url":"https://api.github.com/repos/logstash-plugins/logstash-codec-netflow","labels_url":"https://api.github.com/repos/logstash-plugins/logstash-codec-netflow/issues/134/labels{/name}","comments_url":"https://api.github.com/repos/logstash-plugins/logstash-codec-netflow/issues/134/comments","events_url":"https://api.github.com/repos/logstash-plugins/logstash-codec-netflow/issues/134/events","html_url":"https://github.com/logstash-plugins/logstash-codec-netflow/issues/134","id":318697515,"number":134,"title":"Exception in inputworker... revisited.","user":{"login":"robcowart","id":10326954,"avatar_url":"https://avatars1.githubusercontent.com/u/10326954?v=4","gravatar_id":"","url":"https://api.github.com/users/robcowart","html_url":"https://github.com/robcowart","followers_url":"https://api.github.com/users/robcowart/followers","following_url":"https://api.github.com/users/robcowart/following{/other_user}","gists_url":"https://api.github.com/users/robcowart/gists{/gist_id}","starred_url":"https://api.github.com/users/robcowart/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/robcowart/subscriptions","organizations_url":"https://api.github.com/users/robcowart/orgs","repos_url":"https://api.github.com/users/robcowart/repos","events_url":"https://api.github.com/users/robcowart/events{/privacy}","received_events_url":"https://api.github.com/users/robcowart/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-29T07:45:25Z","updated_at":"2018-04-29T13:11:12Z","closed_at":null,"author_association":"NONE","body":"Logstash 6.2.4\r\nlogstash-codec-netflow-3.12.0\r\nUbuntu 16.04 LTS\r\nJDK 1.8.0_161-b12\r\n\r\nBack in September I opened #90 which @andrewvc fixed with 62e5bfddcf7f271c118a606d2df062a20d5c28fc\r\n\r\nI have recently been working with a couple of customers who have high volumes of IPFIX data, and these issues are reappearing.\r\n\r\nWhen I look at the code it appears that the changes made by @andrewvc were made only to the `decode_netflow9` method, and not `decode_ipfix`.\r\n\r\nIn particular I am looking at...\r\n\r\nline 249-254 from decode_netflow9:\r\n```\r\ntemplate = @decode_mutex_netflow.synchronize { @netflow_templates[key] }\r\n\r\nif !template\r\n  @logger.warn(\"Can't (yet) decode flowset id #{record.flowset_id} from source id #{flowset.source_id}, because no template to decode it with has been received. This message will usually go away after 1 minute.\")\r\n  return events\r\nend\r\n```\r\nlines 347-352 from decode_ipfix: (this is the way Netflow was before @andrewvc changes.)\r\n```\r\nif @ipfix_templates[key] != nil\r\n  template = @ipfix_templates[key]\r\nelse\r\n  @logger.warn(\"Can't (yet) decode flowset id #{record.flowset_id} from observation domain id #{flowset.observation_domain_id}, because no template to decode it with has been received. This message will usually go away after 1 minute.\")\r\n  return events\r\nend\r\n```\r\nThis section of code is also referenced in the backtrace (see below). I am guessing that changing this section of code in `decode_ipfix` to function as it now does in `decode_netflow9` will fix this problem. Ruby isn't really my language so I wanted to point this out so that it can be confirmed and implemented by the experts.\r\n\r\nFor completeness here are examples of the errors I am seeing in the logs:\r\n```\r\n[2018-04-27T22:57:47,252][ERROR][logstash.inputs.udp      ]\r\nException in inputworker {\r\n  \"exception\"=>#<NoMethodError: undefined method `unpack' for nil:NilClass>,\r\n  \"backtrace\"=>[\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.12.0/lib/logstash/codecs/netflow/util.rb:63:in `get'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/primitive.rb:112:in `sensible_default'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/base_primitive.rb:147:in `_value'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/base_primitive.rb:86:in `snapshot'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/struct.rb:109:in `block in snapshot'\",\r\n    \"org/jruby/RubyArray.java:1734:in `each'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/struct.rb:107:in `snapshot'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/base.rb:234:in `=='\",\r\n    \"uri:classloader:/jruby/kernel/basicobject.rb:4:in `!='\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.12.0/lib/logstash/codecs/netflow.rb:347:in `decode_ipfix'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.12.0/lib/logstash/codecs/netflow.rb:131:in `block in decode'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/array.rb:208:in `block in each'\",\r\n    \"org/jruby/RubyArray.java:1734:in `each'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/array.rb:208:in `each'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.12.0/lib/logstash/codecs/netflow.rb:130:in `decode'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.2/lib/logstash/inputs/udp.rb:142:in `inputworker'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.2/lib/logstash/inputs/udp.rb:111:in `block in udp_listener'\"\r\n  ]\r\n}\r\n\r\n[2018-04-27T23:25:36,111][ERROR][logstash.inputs.udp      ]\r\nException in inputworker {\r\n  \"exception\"=>#<NoMethodError: undefined method `has_parameter?' for nil:NilClass>,\r\n  \"backtrace\"=>[\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/struct.rb:267:in `include_obj?'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/struct.rb:109:in `block in snapshot'\",\r\n    \"org/jruby/RubyArray.java:1734:in `each'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/struct.rb:107:in `snapshot'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/base.rb:234:in `=='\",\r\n    \"uri:classloader:/jruby/kernel/basicobject.rb:4:in `!='\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.12.0/lib/logstash/codecs/netflow.rb:347:in `decode_ipfix'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.12.0/lib/logstash/codecs/netflow.rb:131:in `block in decode'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/array.rb:208:in `block in each'\",\r\n    \"org/jruby/RubyArray.java:1734:in `each'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/bindata-2.4.3/lib/bindata/array.rb:208:in `each'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-codec-netflow-3.12.0/lib/logstash/codecs/netflow.rb:130:in `decode'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.2/lib/logstash/inputs/udp.rb:142:in `inputworker'\",\r\n    \"/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-input-udp-3.3.2/lib/logstash/inputs/udp.rb:111:in `block in udp_listener'\"\r\n  ]\r\n}\r\n```"},"comment":{"url":"https://api.github.com/repos/logstash-plugins/logstash-codec-netflow/issues/comments/385250506","html_url":"https://github.com/logstash-plugins/logstash-codec-netflow/issues/134#issuecomment-385250506","issue_url":"https://api.github.com/repos/logstash-plugins/logstash-codec-netflow/issues/134","id":385250506,"user":{"login":"jorritfolmer","id":11555154,"avatar_url":"https://avatars3.githubusercontent.com/u/11555154?v=4","gravatar_id":"","url":"https://api.github.com/users/jorritfolmer","html_url":"https://github.com/jorritfolmer","followers_url":"https://api.github.com/users/jorritfolmer/followers","following_url":"https://api.github.com/users/jorritfolmer/following{/other_user}","gists_url":"https://api.github.com/users/jorritfolmer/gists{/gist_id}","starred_url":"https://api.github.com/users/jorritfolmer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jorritfolmer/subscriptions","organizations_url":"https://api.github.com/users/jorritfolmer/orgs","repos_url":"https://api.github.com/users/jorritfolmer/repos","events_url":"https://api.github.com/users/jorritfolmer/events{/privacy}","received_events_url":"https://api.github.com/users/jorritfolmer/received_events","type":"User","site_admin":false},"created_at":"2018-04-29T13:11:12Z","updated_at":"2018-04-29T13:11:12Z","author_association":"MEMBER","body":"Ah yes good catch!"}}
{"actor":{"display_login":"jorritfolmer"}}