{"action":"closed","issue":{"url":"https://api.github.com/repos/aws/aws-sdk-java-v2/issues/1466","repository_url":"https://api.github.com/repos/aws/aws-sdk-java-v2","labels_url":"https://api.github.com/repos/aws/aws-sdk-java-v2/issues/1466/labels{/name}","comments_url":"https://api.github.com/repos/aws/aws-sdk-java-v2/issues/1466/comments","events_url":"https://api.github.com/repos/aws/aws-sdk-java-v2/issues/1466/events","html_url":"https://github.com/aws/aws-sdk-java-v2/issues/1466","id":504862630,"node_id":"MDU6SXNzdWU1MDQ4NjI2MzA=","number":1466,"title":"SqsAsyncClient is failing unpredictably during streaming","user":{"login":"jaylynstoesz","id":11892230,"node_id":"MDQ6VXNlcjExODkyMjMw","avatar_url":"https://avatars2.githubusercontent.com/u/11892230?v=4","gravatar_id":"","url":"https://api.github.com/users/jaylynstoesz","html_url":"https://github.com/jaylynstoesz","followers_url":"https://api.github.com/users/jaylynstoesz/followers","following_url":"https://api.github.com/users/jaylynstoesz/following{/other_user}","gists_url":"https://api.github.com/users/jaylynstoesz/gists{/gist_id}","starred_url":"https://api.github.com/users/jaylynstoesz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaylynstoesz/subscriptions","organizations_url":"https://api.github.com/users/jaylynstoesz/orgs","repos_url":"https://api.github.com/users/jaylynstoesz/repos","events_url":"https://api.github.com/users/jaylynstoesz/events{/privacy}","received_events_url":"https://api.github.com/users/jaylynstoesz/received_events","type":"User","site_admin":false},"labels":[{"id":1326720996,"node_id":"MDU6TGFiZWwxMzI2NzIwOTk2","url":"https://api.github.com/repos/aws/aws-sdk-java-v2/labels/closing-soon-if-no-response","name":"closing-soon-if-no-response","color":"000000","default":false},{"id":636336644,"node_id":"MDU6TGFiZWw2MzYzMzY2NDQ=","url":"https://api.github.com/repos/aws/aws-sdk-java-v2/labels/investigating","name":"investigating","color":"0000ff","default":false},{"id":1326721012,"node_id":"MDU6TGFiZWwxMzI2NzIxMDEy","url":"https://api.github.com/repos/aws/aws-sdk-java-v2/labels/needs-response","name":"needs-response","color":"ffe79b","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2019-10-09T20:01:03Z","updated_at":"2019-11-05T23:50:38Z","closed_at":"2019-11-05T23:50:37Z","author_association":"NONE","body":"I'm reading from SQS via [Alpakka](https://doc.akka.io/docs/alpakka/current/sqs.html) using the async client. It appears to randomly stop receiving messages from the queue, anywhere from a few minutes to a few hours apart. It appears to happen more frequently when volume is higher, though not necessarily during peak times.\r\n\r\n## Expected Behavior\r\nMy service continuously polls messages from SQS, processes them, and deletes them from the queue. Invalid messages are sent to a dead letter queue. In the attached screenshot of the queue monitoring dashboard, I would expect the **NumberOfMessagesReceived** graph to match the **NumberOfMessagesSent** graph.\r\n\r\n## Current Behavior\r\nThe SQS client randomly throws `java.io.IOException: Server failed to send complete response` and stops polling messages.\r\n![Screen Shot 2019-10-09 at 1 29 35 PM](https://user-images.githubusercontent.com/11892230/66514369-31084700-ea9a-11e9-9e51-ae1d917a3422.png)\r\n\r\n## Steps to Reproduce (for bugs)\r\n```\r\nimport akka.actor.ActorSystem\r\nimport akka.stream.ActorMaterializer\r\nimport akka.stream.alpakka.sqs.javadsl.SqsSource\r\nimport akka.stream.alpakka.sqs.SqsSourceSettings\r\nimport akka.stream.javadsl.Sink\r\nimport software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient\r\nimport software.amazon.awssdk.regions.Region\r\nimport software.amazon.awssdk.services.sqs.model.Message\r\nimport software.amazon.awssdk.services.sqs.SqsAsyncClient\r\n\r\n//...\r\n\r\nfun <T> subscribe() {\r\n    /* Initialize Akka */\r\n    val actorSystem = ActorSystem.create()\r\n\r\n    /* Provision async SQS client */\r\n    val sqsAsyncClient = SqsAsyncClient\r\n        .builder()\r\n        .httpClientBuilder(NettyNioAsyncHttpClient.builder())\r\n        .region(Region.US_EAST_1)\r\n        .build()\r\n\r\n    /* Provision Alpakka SQS module (Akka Streams) */\r\n    val source = SqsSource.create(\r\n        \"https://sqs.us-east-1.amazonaws.com/000000000000/queue-name\",\r\n        SqsSourceSettings.Defaults(),\r\n        sqsAsyncClient\r\n    )\r\n\r\n    /* Akka receives between 1k - 1.5k messages per minute from SQS */\r\n    source\r\n        .map { message: Message -> message.messageId() }\r\n        .runWith(Sink.ignore(), ActorMaterializer.create(actorSystem))\r\n        .handle { _, error ->\r\n            /* Unpredictably returns java.io.IOException: Server failed to send complete response */\r\n            error?.printStackTrace()\r\n        }\r\n}\r\n```\r\n\r\nError stack trace:\r\n```\r\njava.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException\r\n\tat software.amazon.awssdk.utils.CompletableFutureUtils.errorAsCompletionException(CompletableFutureUtils.java:61)\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncExecutionFailureExceptionReportingStage.lambda$execute$0(AsyncExecutionFailureExceptionReportingStage.java:51)\r\n\tat java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)\r\n\tat java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)\r\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)\r\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)\r\n\tat software.amazon.awssdk.utils.CompletableFutureUtils.lambda$forwardExceptionTo$0(CompletableFutureUtils.java:75)\r\n\tat java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)\r\n\tat java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)\r\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)\r\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryErrorIfNeeded(AsyncRetryableStage.java:175)\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:126)\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:107)\r\n\tat java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)\r\n\tat java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)\r\n\tat java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)\r\n\tat java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.onError(MakeAsyncHttpRequestStage.java:249)\r\n\tat software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.lambda$notifyIfResponseNotCompleted$2(ResponseHandler.java:397)\r\n\tat software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.runAndLogError(ResponseHandler.java:180)\r\n\tat software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.notifyIfResponseNotCompleted(ResponseHandler.java:397)\r\n\tat software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.channelInactive(ResponseHandler.java:148)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)\r\n\tat io.netty.handler.logging.LoggingHandler.channelInactive(LoggingHandler.java:167)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)\r\n\tat io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)\t\r\nat io.netty.channel.CombinedChannelDuplexHandler.channelInactive(CombinedChannelDuplexHandler.java:223)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInputClosed(ByteToMessageDecoder.java:390)\r\n\tat io.netty.handler.codec.ByteToMessageDecoder.channelInactive(ByteToMessageDecoder.java:355)\r\n\tat io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1078)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)\r\n\tat io.netty.channel.ChannelInboundHandlerAdapter.channelInactive(ChannelInboundHandlerAdapter.java:81)\r\n\tat io.netty.handler.timeout.IdleStateHandler.channelInactive(IdleStateHandler.java:278)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:236)\r\n\tat io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1403)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:257)\r\n\tat io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:243)\r\n\tat io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:912)\r\n\tat io.netty.channel.AbstractChannel$AbstractUnsafe$8.run(AbstractChannel.java:827)\r\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:405)\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:500)\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:906)\r\n\tat io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)\r\n\tat java.lang.Thread.run(Thread.java:748)\r\nCaused by: software.amazon.awssdk.core.exception.SdkClientException\r\n\tat software.amazon.awssdk.core.exception.SdkClientException$BuilderImpl.build(SdkClientException.java:97)\r\n\tat software.amazon.awssdk.core.internal.util.ThrowableUtils.asSdkException(ThrowableUtils.java:98)\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.retryIfNeeded(AsyncRetryableStage.java:125)\r\n\t... 52 more\r\nCaused by: java.io.IOException: Server failed to send complete response\r\n\tat software.amazon.awssdk.http.nio.netty.internal.ResponseHandler.notifyIfResponseNotCompleted(ResponseHandler.java:396)\r\n\t... 43 more\r\n```\r\n\r\n## Context\r\nThis service is designed to stream business-critical accounting events in order to respond to charges against customer accounts in real time. So, if the service stops polling SQS messages, this prevents us from responding as events occur and could incur significant costs to the business.\r\n\r\n## Your Environment\r\nKotlin: 1.3.41\r\nNetty & SQS Java SDK: 2.9.15\r\n\r\nCC: @debora-ito re: https://github.com/aws/aws-sdk-java-v2/issues/1122#issuecomment-540141440"}}
{"actor":{"display_login":"zoewangg"}}