{"action":"created","issue":{"url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/issues/391","repository_url":"https://api.github.com/repos/awslabs/amazon-kinesis-client","labels_url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/issues/391/labels{/name}","comments_url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/issues/391/comments","events_url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/issues/391/events","html_url":"https://github.com/awslabs/amazon-kinesis-client/issues/391","id":356886735,"node_id":"MDU6SXNzdWUzNTY4ODY3MzU=","number":391,"title":"Lots of error messages like: ` Cancelling subscription, and marking self as failed` with `Invalid StartingSequenceNumber` since upgrading to 2.0","user":{"login":"ryangardner","id":235349,"node_id":"MDQ6VXNlcjIzNTM0OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/235349?v=4","gravatar_id":"","url":"https://api.github.com/users/ryangardner","html_url":"https://github.com/ryangardner","followers_url":"https://api.github.com/users/ryangardner/followers","following_url":"https://api.github.com/users/ryangardner/following{/other_user}","gists_url":"https://api.github.com/users/ryangardner/gists{/gist_id}","starred_url":"https://api.github.com/users/ryangardner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ryangardner/subscriptions","organizations_url":"https://api.github.com/users/ryangardner/orgs","repos_url":"https://api.github.com/users/ryangardner/repos","events_url":"https://api.github.com/users/ryangardner/events{/privacy}","received_events_url":"https://api.github.com/users/ryangardner/received_events","type":"User","site_admin":false},"labels":[{"id":524381356,"node_id":"MDU6TGFiZWw1MjQzODEzNTY=","url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/labels/investigation","name":"investigation","color":"006b75","default":false},{"id":1012925061,"node_id":"MDU6TGFiZWwxMDEyOTI1MDYx","url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/labels/v2.x","name":"v2.x","color":"8cb728","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/milestones/23","html_url":"https://github.com/awslabs/amazon-kinesis-client/milestone/23","labels_url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/milestones/23/labels","id":3645776,"node_id":"MDk6TWlsZXN0b25lMzY0NTc3Ng==","number":23,"title":"v2.0.3","description":"","creator":{"login":"pfifer","id":18317277,"node_id":"MDQ6VXNlcjE4MzE3Mjc3","avatar_url":"https://avatars2.githubusercontent.com/u/18317277?v=4","gravatar_id":"","url":"https://api.github.com/users/pfifer","html_url":"https://github.com/pfifer","followers_url":"https://api.github.com/users/pfifer/followers","following_url":"https://api.github.com/users/pfifer/following{/other_user}","gists_url":"https://api.github.com/users/pfifer/gists{/gist_id}","starred_url":"https://api.github.com/users/pfifer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pfifer/subscriptions","organizations_url":"https://api.github.com/users/pfifer/orgs","repos_url":"https://api.github.com/users/pfifer/repos","events_url":"https://api.github.com/users/pfifer/events{/privacy}","received_events_url":"https://api.github.com/users/pfifer/received_events","type":"User","site_admin":false},"open_issues":1,"closed_issues":13,"state":"closed","created_at":"2018-09-11T17:38:12Z","updated_at":"2018-10-08T23:31:44Z","due_on":null,"closed_at":"2018-10-08T23:31:44Z"},"comments":96,"created_at":"2018-09-04T16:28:19Z","updated_at":"2018-10-11T16:28:34Z","closed_at":null,"author_association":"NONE","body":"```2018-09-04 15:58:00.229  WARN 7 --- [dProcessor-0028] s.a.kinesis.lifecycle.ShardConsumer      : shardId-000000000309: onError().  Cancelling subscription, and marking self as failed.\r\njava.util.concurrent.CompletionException: software.amazon.awssdk.services.kinesis.model.InvalidArgumentException: Invalid StartingSequenceNumber. It encodes shardId-000000000234, while it was used in a call to a shard with shardId-000000000309 (Service: Kinesis, Status Code: 400, Request ID: eab0b990-9136-1aad-beb8-145829c423c5)\r\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:331) ~[na:na]\r\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:346) ~[na:na]\r\n\tat java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:632) ~[na:na]\r\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]\r\n\tat java.base/java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2088) ~[na:na]\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.handle(AsyncRetryableStage.java:155) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.AsyncRetryableStage$RetryExecutor.lambda$execute$0(AsyncRetryableStage.java:121) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat java.base/java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:930) ~[na:na]\r\n\tat java.base/java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:907) ~[na:na]\r\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[na:na]\r\n\tat java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[na:na]\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$Completable.lambda$complete$0(MakeAsyncHttpRequestStage.java:200) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1135) [na:na]\r\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [na:na]\r\n\tat java.base/java.lang.Thread.run(Thread.java:844) [na:na]\r\nCaused by: software.amazon.awssdk.services.kinesis.model.InvalidArgumentException: Invalid StartingSequenceNumber. It encodes shardId-000000000234, while it was used in a call to a shard with shardId-000000000309 (Service: Kinesis, Status Code: 400, Request ID: eab0b990-9136-1aad-beb8-145829c423c5)\r\n\tat software.amazon.awssdk.services.kinesis.model.InvalidArgumentException$BuilderImpl.build(InvalidArgumentException.java:104) ~[kinesis-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.services.kinesis.model.InvalidArgumentException$BuilderImpl.build(InvalidArgumentException.java:64) ~[kinesis-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.core.internal.http.response.SdkErrorResponseHandler.handle(SdkErrorResponseHandler.java:46) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.core.internal.http.response.SdkErrorResponseHandler.handle(SdkErrorResponseHandler.java:30) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.core.internal.http.async.SyncResponseHandlerAdapter.complete(SyncResponseHandlerAdapter.java:92) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.core.client.handler.BaseAsyncClientHandler$InterceptorCallingHttpResponseHandler.complete(BaseAsyncClientHandler.java:225) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.handleResponse(MakeAsyncHttpRequestStage.java:185) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.complete(MakeAsyncHttpRequestStage.java:171) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.core.internal.http.pipeline.stages.MakeAsyncHttpRequestStage$ResponseHandler.complete(MakeAsyncHttpRequestStage.java:122) ~[sdk-core-2.0.1.jar!/:na]\r\n\tat software.amazon.awssdk.http.nio.netty.internal.ResponseHandler$PublisherAdapter$1.onComplete(ResponseHandler.java:255) ~[netty-nio-client-2.0.1.jar!/:na]\r\n\tat com.typesafe.netty.HandlerPublisher.publishMessage(HandlerPublisher.java:362) ~[netty-reactive-streams-2.0.0.jar!/:na]\r\n\tat com.typesafe.netty.HandlerPublisher.flushBuffer(HandlerPublisher.java:304) ~[netty-reactive-streams-2.0.0.jar!/:na]\r\n\tat com.typesafe.netty.HandlerPublisher.receivedDemand(HandlerPublisher.java:258) ~[netty-reactive-streams-2.0.0.jar!/:na]\r\n\tat com.typesafe.netty.HandlerPublisher.access$200(HandlerPublisher.java:41) ~[netty-reactive-streams-2.0.0.jar!/:na]\r\n\tat com.typesafe.netty.HandlerPublisher$ChannelSubscription$1.run(HandlerPublisher.java:452) ~[netty-reactive-streams-2.0.0.jar!/:na]\r\n\tat io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) ~[netty-common-4.1.23.Final.jar!/:4.1.23.Final]\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404) ~[netty-common-4.1.23.Final.jar!/:4.1.23.Final]\r\n\tat io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) ~[netty-transport-4.1.23.Final.jar!/:4.1.23.Final]\r\n\tat io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:886) ~[netty-common-4.1.23.Final.jar!/:4.1.23.Final]\r\n\t... 1 common frames omitted\r\n```\r\n\r\nIn the last three hours, I've had these errors / warnings with the following combinations of encoded shardId's and the shard ID it claims it was used with:\r\n\r\n(all shardId's are with the format `shardId-000000000301` - so I'm omitting the first part)\r\n\r\n| Encoded Shard ID  | Used-with ShardID | count |\r\n| ------------- | ------------- | ------ |\r\n| 301  | 349  | 2507 |\r\n| 190  | 205  | 6385 |\r\n| 222 | 196 | 2589 |\r\n| 302 | 238 | 2582 |\r\n| 270 | 258 | 2503 | \r\n| 236 | 198 | 2512 |\r\n| 253 | 229 | 356 | \r\n| 266 | 342 | 7336 | \r\n| 234 | 309 | 4710 | \r\n| 260 | 229 | 2634 | \r\n| 333 | 278 | 2519 | \r\n| 308 | 318 | 2669 |\r\n\r\nMy stream has not been resharded recently and has 80 shards. \r\n\r\n*Question*: Is this something that happens normally as part of the KCL client behavior, or does this indicate something I should be concerned about? (And If it is considered to be normal behavior, can the logs be changed to something other than warn-level and the full stacktrace not be logged?)\r\n\r\nIf this is something I should be worried about - what might be causing this?"},"comment":{"url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/issues/comments/429023658","html_url":"https://github.com/awslabs/amazon-kinesis-client/issues/391#issuecomment-429023658","issue_url":"https://api.github.com/repos/awslabs/amazon-kinesis-client/issues/391","id":429023658,"node_id":"MDEyOklzc3VlQ29tbWVudDQyOTAyMzY1OA==","user":{"login":"vanhoale","id":28834039,"node_id":"MDQ6VXNlcjI4ODM0MDM5","avatar_url":"https://avatars2.githubusercontent.com/u/28834039?v=4","gravatar_id":"","url":"https://api.github.com/users/vanhoale","html_url":"https://github.com/vanhoale","followers_url":"https://api.github.com/users/vanhoale/followers","following_url":"https://api.github.com/users/vanhoale/following{/other_user}","gists_url":"https://api.github.com/users/vanhoale/gists{/gist_id}","starred_url":"https://api.github.com/users/vanhoale/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vanhoale/subscriptions","organizations_url":"https://api.github.com/users/vanhoale/orgs","repos_url":"https://api.github.com/users/vanhoale/repos","events_url":"https://api.github.com/users/vanhoale/events{/privacy}","received_events_url":"https://api.github.com/users/vanhoale/received_events","type":"User","site_admin":false},"created_at":"2018-10-11T16:28:34Z","updated_at":"2018-10-11T16:28:34Z","author_association":"NONE","body":"I'm still seeing this message with version 2.0.3:\r\n\r\n{\"level\":\"ERROR\",\"logger_name\":\"software.amazon.kinesis.lifecycle.ShardConsumer\",\"message\":\"shardId-000000000000: Last request was dispatched at 2018-10-11T16:24:46.671Z, but no response as of 2018-10-11T16:25:21.902Z (PT35.231S).  Cancelling subscription, and restarting.\",\"logger_timestamp\":\"2018-10-11T16:25:21.902Z\"}\r\n"}}
{"actor":{"display_login":"vanhoale"}}