{"action":"created","issue":{"url":"https://api.github.com/repos/floragunncom/search-guard/issues/548","repository_url":"https://api.github.com/repos/floragunncom/search-guard","labels_url":"https://api.github.com/repos/floragunncom/search-guard/issues/548/labels{/name}","comments_url":"https://api.github.com/repos/floragunncom/search-guard/issues/548/comments","events_url":"https://api.github.com/repos/floragunncom/search-guard/issues/548/events","html_url":"https://github.com/floragunncom/search-guard/issues/548","id":355886623,"node_id":"MDU6SXNzdWUzNTU4ODY2MjM=","number":548,"title":"Rolling upgrade to ES 6.4 causes fatal error on upgraded nodes","user":{"login":"mikn","id":132216,"node_id":"MDQ6VXNlcjEzMjIxNg==","avatar_url":"https://avatars1.githubusercontent.com/u/132216?v=4","gravatar_id":"","url":"https://api.github.com/users/mikn","html_url":"https://github.com/mikn","followers_url":"https://api.github.com/users/mikn/followers","following_url":"https://api.github.com/users/mikn/following{/other_user}","gists_url":"https://api.github.com/users/mikn/gists{/gist_id}","starred_url":"https://api.github.com/users/mikn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikn/subscriptions","organizations_url":"https://api.github.com/users/mikn/orgs","repos_url":"https://api.github.com/users/mikn/repos","events_url":"https://api.github.com/users/mikn/events{/privacy}","received_events_url":"https://api.github.com/users/mikn/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-08-31T08:27:28Z","updated_at":"2018-09-04T01:46:02Z","closed_at":null,"author_association":"NONE","body":"<!--\r\nPlease do not ask questions here. \r\nFor that refer to https://groups.google.com/forum/#!forum/search-guard\r\n\r\nGithub is a place only for reporting bugs or feature requests.\r\nIf you report a bug please include always the following informations:\r\n\r\n* Search Guard and Elasticsearch version\r\n* JVM version and operating system version\r\n* Number of nodes in your cluster\r\n* Description of the bug\r\n\r\nThese informations are optional but are very valueable and\r\ntypically would lead to get bugs fixed faster:\r\n\r\n* Steps to reproduce\r\n* Logfiles on DEBUG level\r\n-->\r\nSG version: 23.0\r\nES version: 6.4.0\r\nJVM version: OpenJDK 1.8.171 (Alpine 3.8, kernel 4.9)\r\nNodes: 3 data, 3 master, 2 client\r\n\r\nWhen updating the cluster from 6.3.2 to 6.4.0 we get a fatal error on transport level for Elasticsearch during the upgrade process (we have not tried finishing the rollout with this error).\r\n\r\nRelevant log:\r\n```\r\n[2018-08-31T06:03:41,634][INFO ][o.e.n.Node               ] [log-store-elasticsearch-client-98cc586b9-pjfm7] version[6.4.0], pid[1], build[oss/tar/595516e/2018-08-17T23:18:47.308994Z], OS[Linux/4.9.0-6-amd64/amd64], JVM[Oracle Corporation/OpenJDK 64-Bit Server VM/1.8.0_171/25.171-b11]\r\n[2018-08-31T06:03:41,634][INFO ][o.e.n.Node               ] [log-store-elasticsearch-client-98cc586b9-pjfm7] JVM arguments [-XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Des.cgroups.hierarchy.override=/, -Djava.net.preferIPv4Stack=true, -Xms2g, -Xmx2g, -Des.path.home=/usr/share/elasticsearch, -Des.path.conf=/usr/share/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar]\r\n[2018-08-31T06:03:44,669][INFO ][c.f.s.SearchGuardPlugin  ] ES Config path is /usr/share/elasticsearch/config\r\n[2018-08-31T06:03:46,833][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] OpenSSL OpenSSL 1.0.2o  27 Mar 2018 (268443903) available\r\n[2018-08-31T06:03:47,240][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Config directory is /usr/share/elasticsearch/config/, from there the key- and truststore files are resolved relatively\r\n[2018-08-31T06:03:47,643][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Client Provider : OPENSSL\r\n[2018-08-31T06:03:47,649][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS Transport Server Provider : OPENSSL\r\n[2018-08-31T06:03:47,649][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] TLS HTTP Provider             : OPENSSL\r\n[2018-08-31T06:03:47,649][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for transport layer : [TLSv1.2, TLSv1.1]\r\n[2018-08-31T06:03:47,649][INFO ][c.f.s.s.DefaultSearchGuardKeyStore] Enabled TLS protocols for HTTP layer      : [TLSv1.2, TLSv1.1]\r\n[...]\r\n[2018-08-31T06:04:01,734][INFO ][c.f.s.SearchGuardPlugin  ] 0 Search Guard modules loaded so far: []\r\n[2018-08-31T06:04:01,736][INFO ][o.e.n.Node               ] [log-store-elasticsearch-client-98cc586b9-pjfm7] initialized\r\n[2018-08-31T06:04:01,736][INFO ][o.e.n.Node               ] [log-store-elasticsearch-client-98cc586b9-pjfm7] starting ...\r\n[2018-08-31T06:04:02,049][INFO ][o.e.t.TransportService   ] [log-store-elasticsearch-client-98cc586b9-pjfm7] publish_address {100.96.27.66:9300}, bound_addresses {100.96.27.66:9300}\r\n[2018-08-31T06:04:02,060][INFO ][o.e.b.BootstrapChecks    ] [log-store-elasticsearch-client-98cc586b9-pjfm7] bound or publishing to a non-loopback address, enforcing bootstrap checks\r\n[2018-08-31T06:04:02,072][INFO ][c.f.s.c.IndexBaseConfigurationRepository] Check if searchguard index exists ...\r\n[2018-08-31T06:04:02,375][ERROR][o.e.t.n.Netty4Utils      ] fatal error on the network layer\r\n\tat org.elasticsearch.transport.netty4.Netty4Utils.maybeDie(Netty4Utils.java:182)\r\n\tat org.elasticsearch.transport.netty4.Netty4MessageChannelHandler.exceptionCaught(Netty4MessageChannelHandler.java:73)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256)\r\n\tat io.netty.channel.ChannelInboundHandlerAdapter.exceptionCaught(ChannelInboundHandlerAdapter.java:131)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256)\r\n\tat io.netty.handler.logging.LoggingHandler.exceptionCaught(LoggingHandler.java:175)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:264)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:256)\r\n\tat io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:998)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:285)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:850)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:364)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340)\r\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348)\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935)\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545)\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499)\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n[2018-08-31T06:04:02,437][WARN ][c.f.s.s.t.SearchGuardSSLNettyTransport] [log-store-elasticsearch-client-98cc586b9-pjfm7] exception caught on transport layer [NettyTcpChannel{localAddress=0.0.0.0/0.0.0.0:41624, remoteAddress=log-store-elasticsearch-discovery.services.svc.cluster.local/100.65.68.170:9300}], closing connection\r\nio.netty.handler.codec.DecoderException: java.lang.AbstractMethodError\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]\r\n\tat java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]\r\nCaused by: java.lang.AbstractMethodError\r\n\tat io.netty.internal.tcnative.SSL.doHandshake(Native Method) ~[netty-tcnative-openssl-1.0.2o-static-2.0.11.Final-alpine-linux-x86_64.jar:2.0.11.Final]\r\n\tat io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshake(ReferenceCountedOpenSslEngine.java:1594) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:648) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat javax.net.ssl.SSLEngine.wrap(SSLEngine.java:509) ~[?:1.8.0_171]\r\n\tat io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:947) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:762) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:731) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1149) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1133) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[netty-handler-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]\r\n\t... 15 more\r\n[2018-08-31T06:04:02,379][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [log-store-elasticsearch-client-98cc586b9-pjfm7] fatal error in thread [Thread-5], exiting\r\njava.lang.AbstractMethodError: null\r\n\tat io.netty.internal.tcnative.SSL.doHandshake(Native Method) ~[?:?]\r\n\tat io.netty.handler.ssl.ReferenceCountedOpenSslEngine.handshake(ReferenceCountedOpenSslEngine.java:1594) ~[?:?]\r\n\tat io.netty.handler.ssl.ReferenceCountedOpenSslEngine.wrap(ReferenceCountedOpenSslEngine.java:648) ~[?:?]\r\n\tat javax.net.ssl.SSLEngine.wrap(SSLEngine.java:509) ~[?:1.8.0_171]\r\n\tat io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:947) ~[?:?]\r\n\tat io.netty.handler.ssl.SslHandler.wrap(SslHandler.java:762) ~[?:?]\r\n\tat io.netty.handler.ssl.SslHandler.wrapAndFlush(SslHandler.java:731) ~[?:?]\r\n\tat io.netty.handler.ssl.SslHandler.handleUnwrapThrowable(SslHandler.java:1149) ~[?:?]\r\n\tat io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1133) ~[?:?]\r\n\tat io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[?:?]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[?:?]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[?:?]\r\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) ~[?:?]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[?:?]\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[?:?]\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) ~[?:?]\r\n\tat io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) ~[?:?]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) ~[?:?]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) ~[?:?]\r\n\tat io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) ~[?:?]\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) ~[?:?]\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) ~[?:?]\r\n\tat java.lang.Thread.run(Thread.java:748) [?:1.8.0_171]\r\n```"},"comment":{"url":"https://api.github.com/repos/floragunncom/search-guard/issues/comments/418217449","html_url":"https://github.com/floragunncom/search-guard/issues/548#issuecomment-418217449","issue_url":"https://api.github.com/repos/floragunncom/search-guard/issues/548","id":418217449,"node_id":"MDEyOklzc3VlQ29tbWVudDQxODIxNzQ0OQ==","user":{"login":"mikn","id":132216,"node_id":"MDQ6VXNlcjEzMjIxNg==","avatar_url":"https://avatars1.githubusercontent.com/u/132216?v=4","gravatar_id":"","url":"https://api.github.com/users/mikn","html_url":"https://github.com/mikn","followers_url":"https://api.github.com/users/mikn/followers","following_url":"https://api.github.com/users/mikn/following{/other_user}","gists_url":"https://api.github.com/users/mikn/gists{/gist_id}","starred_url":"https://api.github.com/users/mikn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mikn/subscriptions","organizations_url":"https://api.github.com/users/mikn/orgs","repos_url":"https://api.github.com/users/mikn/repos","events_url":"https://api.github.com/users/mikn/events{/privacy}","received_events_url":"https://api.github.com/users/mikn/received_events","type":"User","site_admin":false},"created_at":"2018-09-04T01:46:02Z","updated_at":"2018-09-04T01:46:02Z","author_association":"NONE","body":"Oh, interesting. It works with this version! Thank you :)"}}
{"actor":{"display_login":"mikn"}}