{"action":"created","issue":{"url":"https://api.github.com/repos/eclipse/hono/issues/735","repository_url":"https://api.github.com/repos/eclipse/hono","labels_url":"https://api.github.com/repos/eclipse/hono/issues/735/labels{/name}","comments_url":"https://api.github.com/repos/eclipse/hono/issues/735/comments","events_url":"https://api.github.com/repos/eclipse/hono/issues/735/events","html_url":"https://github.com/eclipse/hono/issues/735","id":345140290,"node_id":"MDU6SXNzdWUzNDUxNDAyOTA=","number":735,"title":"Half dead HTTP adapter instances","user":{"login":"ctron","id":202474,"node_id":"MDQ6VXNlcjIwMjQ3NA==","avatar_url":"https://avatars0.githubusercontent.com/u/202474?v=4","gravatar_id":"","url":"https://api.github.com/users/ctron","html_url":"https://github.com/ctron","followers_url":"https://api.github.com/users/ctron/followers","following_url":"https://api.github.com/users/ctron/following{/other_user}","gists_url":"https://api.github.com/users/ctron/gists{/gist_id}","starred_url":"https://api.github.com/users/ctron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ctron/subscriptions","organizations_url":"https://api.github.com/users/ctron/orgs","repos_url":"https://api.github.com/users/ctron/repos","events_url":"https://api.github.com/users/ctron/events{/privacy}","received_events_url":"https://api.github.com/users/ctron/received_events","type":"User","site_admin":false},"labels":[{"id":525773933,"node_id":"MDU6TGFiZWw1MjU3NzM5MzM=","url":"https://api.github.com/repos/eclipse/hono/labels/HTTP%20Adapter","name":"HTTP Adapter","color":"0052cc","default":false},{"id":323160606,"node_id":"MDU6TGFiZWwzMjMxNjA2MDY=","url":"https://api.github.com/repos/eclipse/hono/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"open","locked":false,"assignee":{"login":"sophokles73","id":5682135,"node_id":"MDQ6VXNlcjU2ODIxMzU=","avatar_url":"https://avatars1.githubusercontent.com/u/5682135?v=4","gravatar_id":"","url":"https://api.github.com/users/sophokles73","html_url":"https://github.com/sophokles73","followers_url":"https://api.github.com/users/sophokles73/followers","following_url":"https://api.github.com/users/sophokles73/following{/other_user}","gists_url":"https://api.github.com/users/sophokles73/gists{/gist_id}","starred_url":"https://api.github.com/users/sophokles73/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sophokles73/subscriptions","organizations_url":"https://api.github.com/users/sophokles73/orgs","repos_url":"https://api.github.com/users/sophokles73/repos","events_url":"https://api.github.com/users/sophokles73/events{/privacy}","received_events_url":"https://api.github.com/users/sophokles73/received_events","type":"User","site_admin":false},"assignees":[{"login":"sophokles73","id":5682135,"node_id":"MDQ6VXNlcjU2ODIxMzU=","avatar_url":"https://avatars1.githubusercontent.com/u/5682135?v=4","gravatar_id":"","url":"https://api.github.com/users/sophokles73","html_url":"https://github.com/sophokles73","followers_url":"https://api.github.com/users/sophokles73/followers","following_url":"https://api.github.com/users/sophokles73/following{/other_user}","gists_url":"https://api.github.com/users/sophokles73/gists{/gist_id}","starred_url":"https://api.github.com/users/sophokles73/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sophokles73/subscriptions","organizations_url":"https://api.github.com/users/sophokles73/orgs","repos_url":"https://api.github.com/users/sophokles73/repos","events_url":"https://api.github.com/users/sophokles73/events{/privacy}","received_events_url":"https://api.github.com/users/sophokles73/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/eclipse/hono/milestones/16","html_url":"https://github.com/eclipse/hono/milestone/16","labels_url":"https://api.github.com/repos/eclipse/hono/milestones/16/labels","id":3291897,"node_id":"MDk6TWlsZXN0b25lMzI5MTg5Nw==","number":16,"title":"0.7","description":"https://projects.eclipse.org/projects/iot.hono/releases/0.7","creator":{"login":"sophokles73","id":5682135,"node_id":"MDQ6VXNlcjU2ODIxMzU=","avatar_url":"https://avatars1.githubusercontent.com/u/5682135?v=4","gravatar_id":"","url":"https://api.github.com/users/sophokles73","html_url":"https://github.com/sophokles73","followers_url":"https://api.github.com/users/sophokles73/followers","following_url":"https://api.github.com/users/sophokles73/following{/other_user}","gists_url":"https://api.github.com/users/sophokles73/gists{/gist_id}","starred_url":"https://api.github.com/users/sophokles73/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sophokles73/subscriptions","organizations_url":"https://api.github.com/users/sophokles73/orgs","repos_url":"https://api.github.com/users/sophokles73/repos","events_url":"https://api.github.com/users/sophokles73/events{/privacy}","received_events_url":"https://api.github.com/users/sophokles73/received_events","type":"User","site_admin":false},"open_issues":18,"closed_issues":60,"state":"open","created_at":"2018-04-24T07:19:05Z","updated_at":"2018-07-27T15:54:22Z","due_on":null,"closed_at":null},"comments":3,"created_at":"2018-07-27T08:40:29Z","updated_at":"2018-07-27T16:20:33Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Every now and then it happens to me that an HTTP protocol adapter instance is half dead. It is being reported as healthy, yet does not send any messages to the downstream Qpid router. HTTP requests return 503.\r\n\r\nThe following two exceptions keep repeating over and over:\r\n\r\n~~~\r\n08:21:33.966 [vert.x-eventloop-thread-0] ERROR i.v.e.w.impl.RoutingContextImplBase - Unexpected exception in route\r\njava.lang.IllegalStateException: Response is closed\r\n\tat io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:548)\r\n\tat io.vertx.core.http.impl.HttpServerResponseImpl.end0(HttpServerResponseImpl.java:401)\r\n\tat io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:319)\r\n\tat io.vertx.core.http.impl.HttpServerResponseImpl.end(HttpServerResponseImpl.java:308)\r\n\tat io.vertx.ext.web.impl.RoutingContextImplBase.unhandledFailure(RoutingContextImplBase.java:166)\r\n\tat io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:132)\r\n\tat io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)\r\n\tat io.opentracing.contrib.vertx.ext.web.TracingHandler.handlerFailure(TracingHandler.java:90)\r\n\tat io.opentracing.contrib.vertx.ext.web.TracingHandler.handle(TracingHandler.java:44)\r\n\tat io.opentracing.contrib.vertx.ext.web.TracingHandler.handle(TracingHandler.java:25)\r\n\tat io.vertx.ext.web.impl.RouteImpl.handleFailure(RouteImpl.java:223)\r\n\tat io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:117)\r\n\tat io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)\r\n\tat io.vertx.ext.web.impl.RoutingContextImpl.doFail(RoutingContextImpl.java:465)\r\n\tat io.vertx.ext.web.impl.RoutingContextImpl.fail(RoutingContextImpl.java:166)\r\n\tat org.eclipse.hono.service.auth.device.HonoAuthHandler.processException(HonoAuthHandler.java:233)\r\n\tat org.eclipse.hono.service.auth.device.HonoAuthHandler.lambda$null$1(HonoAuthHandler.java:189)\r\n\tat org.eclipse.hono.service.auth.device.CredentialsApiAuthProvider.lambda$authenticate$3(CredentialsApiAuthProvider.java:127)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.Future.lambda$map$2(Future.java:308)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:149)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:18)\r\n\tat io.vertx.core.impl.FailedFuture.setHandler(FailedFuture.java:48)\r\n\tat io.vertx.core.Future.lambda$recover$4(Future.java:392)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:149)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:18)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.Future.lambda$map$2(Future.java:308)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:149)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:18)\r\n\tat org.eclipse.hono.client.impl.AbstractRequestResponseClient.cancelRequest(AbstractRequestResponseClient.java:416)\r\n\tat org.eclipse.hono.client.impl.AbstractRequestResponseClient.lambda$null$3(AbstractRequestResponseClient.java:729)\r\n\tat io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:885)\r\n\tat io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:844)\r\n\tat io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)\r\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)\r\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)\r\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n~~~\r\n\r\n~~~\r\n08:21:33.968 [vert.x-eventloop-thread-0] ERROR i.v.e.w.impl.RoutingContextImplBase - Unexpected exception in route\r\njava.lang.IllegalStateException: Response is closed\r\n\tat io.vertx.core.http.impl.HttpServerResponseImpl.checkValid(HttpServerResponseImpl.java:548)\r\n\tat io.vertx.core.http.impl.HttpServerResponseImpl.putHeader(HttpServerResponseImpl.java:189)\r\n\tat org.eclipse.hono.service.http.HttpUtils.setResponseBody(HttpUtils.java:318)\r\n\tat org.eclipse.hono.service.http.DefaultFailureHandler.sendError(DefaultFailureHandler.java:110)\r\n\tat org.eclipse.hono.service.http.DefaultFailureHandler.handle(DefaultFailureHandler.java:60)\r\n\tat org.eclipse.hono.service.http.DefaultFailureHandler.handle(DefaultFailureHandler.java:40)\r\n\tat io.vertx.ext.web.impl.RouteImpl.handleFailure(RouteImpl.java:223)\r\n\tat io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:117)\r\n\tat io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)\r\n\tat io.opentracing.contrib.vertx.ext.web.TracingHandler.handlerFailure(TracingHandler.java:90)\r\n\tat io.opentracing.contrib.vertx.ext.web.TracingHandler.handle(TracingHandler.java:44)\r\n\tat io.opentracing.contrib.vertx.ext.web.TracingHandler.handle(TracingHandler.java:25)\r\n\tat io.vertx.ext.web.impl.RouteImpl.handleFailure(RouteImpl.java:223)\r\n\tat io.vertx.ext.web.impl.RoutingContextImplBase.iterateNext(RoutingContextImplBase.java:117)\r\n\tat io.vertx.ext.web.impl.RoutingContextImpl.next(RoutingContextImpl.java:133)\r\n\tat io.vertx.ext.web.impl.RoutingContextImpl.doFail(RoutingContextImpl.java:465)\r\n\tat io.vertx.ext.web.impl.RoutingContextImpl.fail(RoutingContextImpl.java:166)\r\n\tat org.eclipse.hono.service.auth.device.HonoAuthHandler.processException(HonoAuthHandler.java:233)\r\n\tat org.eclipse.hono.service.auth.device.HonoAuthHandler.lambda$null$1(HonoAuthHandler.java:189)\r\n\tat org.eclipse.hono.service.auth.device.CredentialsApiAuthProvider.lambda$authenticate$3(CredentialsApiAuthProvider.java:127)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.Future.lambda$map$2(Future.java:308)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:149)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:18)\r\n\tat io.vertx.core.impl.FailedFuture.setHandler(FailedFuture.java:48)\r\n\tat io.vertx.core.Future.lambda$recover$4(Future.java:392)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:149)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:18)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.Future.lambda$map$2(Future.java:308)\r\n\tat io.vertx.core.impl.FutureImpl.tryFail(FutureImpl.java:165)\r\n\tat io.vertx.core.impl.FutureImpl.fail(FutureImpl.java:97)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:149)\r\n\tat io.vertx.core.impl.FutureImpl.handle(FutureImpl.java:18)\r\n\tat org.eclipse.hono.client.impl.AbstractRequestResponseClient.cancelRequest(AbstractRequestResponseClient.java:416)\r\n\tat org.eclipse.hono.client.impl.AbstractRequestResponseClient.lambda$null$3(AbstractRequestResponseClient.java:729)\r\n\tat io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:885)\r\n\tat io.vertx.core.impl.VertxImpl$InternalTimerHandler.handle(VertxImpl.java:844)\r\n\tat io.vertx.core.impl.ContextImpl.lambda$wrapTask$2(ContextImpl.java:339)\r\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)\r\n\tat io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:309)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886)\r\n\tat io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\n~~~\r\n\r\nThe full log is here: [hono-adapter-http-vertx.log](https://github.com/eclipse/hono/files/2235097/hono-adapter-http-vertx.log)\r\n\r\nHowever I am not sure if this repeating exception is the cause of just a side effect. In any case I think that the exception caused by calling into `putHeader` on a closed response should be handled or avoided in the first place."},"comment":{"url":"https://api.github.com/repos/eclipse/hono/issues/comments/408468595","html_url":"https://github.com/eclipse/hono/issues/735#issuecomment-408468595","issue_url":"https://api.github.com/repos/eclipse/hono/issues/735","id":408468595,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODQ2ODU5NQ==","user":{"login":"ctron","id":202474,"node_id":"MDQ6VXNlcjIwMjQ3NA==","avatar_url":"https://avatars0.githubusercontent.com/u/202474?v=4","gravatar_id":"","url":"https://api.github.com/users/ctron","html_url":"https://github.com/ctron","followers_url":"https://api.github.com/users/ctron/followers","following_url":"https://api.github.com/users/ctron/following{/other_user}","gists_url":"https://api.github.com/users/ctron/gists{/gist_id}","starred_url":"https://api.github.com/users/ctron/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ctron/subscriptions","organizations_url":"https://api.github.com/users/ctron/orgs","repos_url":"https://api.github.com/users/ctron/repos","events_url":"https://api.github.com/users/ctron/events{/privacy}","received_events_url":"https://api.github.com/users/ctron/received_events","type":"User","site_admin":false},"created_at":"2018-07-27T16:20:33Z","updated_at":"2018-07-27T16:20:33Z","author_association":"CONTRIBUTOR","body":"I any case, the issue with the half-dead instances still is a problem."}}
{"actor":{"display_login":"ctron"}}