{"action":"created","issue":{"url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/1331","repository_url":"https://api.github.com/repos/aws/aws-sdk-ruby","labels_url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/1331/labels{/name}","comments_url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/1331/comments","events_url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/1331/events","html_url":"https://github.com/aws/aws-sdk-ruby/issues/1331","id":186833846,"number":1331,"title":"RuntimeError: can't add a new key into hash during iteration","user":{"login":"wjordan","id":56541,"avatar_url":"https://avatars.githubusercontent.com/u/56541?v=3","gravatar_id":"","url":"https://api.github.com/users/wjordan","html_url":"https://github.com/wjordan","followers_url":"https://api.github.com/users/wjordan/followers","following_url":"https://api.github.com/users/wjordan/following{/other_user}","gists_url":"https://api.github.com/users/wjordan/gists{/gist_id}","starred_url":"https://api.github.com/users/wjordan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wjordan/subscriptions","organizations_url":"https://api.github.com/users/wjordan/orgs","repos_url":"https://api.github.com/users/wjordan/repos","events_url":"https://api.github.com/users/wjordan/events{/privacy}","received_events_url":"https://api.github.com/users/wjordan/received_events","type":"User","site_admin":false},"labels":[{"id":39555037,"url":"https://api.github.com/repos/aws/aws-sdk-ruby/labels/information%20requested","name":"information requested","color":"eb6420","default":false},{"id":174154732,"url":"https://api.github.com/repos/aws/aws-sdk-ruby/labels/Version%202","name":"Version 2","color":"207de5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-11-02T15:29:11Z","updated_at":"2016-11-17T17:00:28Z","closed_at":null,"body":"On our production application we are receiving intermittent stack traces on S3 requests- the errors arrive in bursts of a couple hundred over an hour or two, then nothing for a few days, then another burst (6 over the last 30 days). I suspect this might be some sort of concurrency issue related to network-error retries somewhere deep in the `seahorse` stack, but could use some help figuring out what exactly is going on and how to fix.\r\n\r\n Using [`aws-sdk-core` 2.6.1](https://github.com/aws/aws-sdk-ruby/tree/v2.6.1/aws-sdk-core/lib).\r\n\r\nRelevant part of the stack trace (`net_http`):\r\n\r\n- [lib/seahorse/client/net_http/connection_pool.rb:40 :in `block in initialize`](https://github.com/aws/aws-sdk-ruby/blob/v2.6.1/aws-sdk-core/lib/seahorse/client/net_http/connection_pool.rb#L40)\r\n- [lib/seahorse/client/net_http/connection_pool.rb:89 :in `yield`](https://github.com/aws/aws-sdk-ruby/blob/v2.6.1/aws-sdk-core/lib/seahorse/client/net_http/connection_pool.rb#L89)\r\n- [lib/seahorse/client/net_http/connection_pool.rb:89 :in `block in session_for`](https://github.com/aws/aws-sdk-ruby/blob/v2.6.1/aws-sdk-core/lib/seahorse/client/net_http/connection_pool.rb#L89)\r\n- [lib/seahorse/client/net_http/connection_pool.rb:87 :in `synchronize`](https://github.com/aws/aws-sdk-ruby/blob/v2.6.1/aws-sdk-core/lib/seahorse/client/net_http/connection_pool.rb#L87)\r\n- [lib/seahorse/client/net_http/connection_pool.rb:87 :in `session_for`](https://github.com/aws/aws-sdk-ruby/blob/v2.6.1/aws-sdk-core/lib/seahorse/client/net_http/connection_pool.rb#L87)\r\n- [lib/seahorse/client/net_http/handler.rb:116 :in `session`](https://github.com/aws/aws-sdk-ruby/blob/v2.6.1/aws-sdk-core/lib/seahorse/client/net_http/handler.rb#L116)\r\n- [lib/seahorse/client/net_http/handler.rb:68 :in `transmit`](https://github.com/aws/aws-sdk-ruby/blob/v2.6.1/aws-sdk-core/lib/seahorse/client/net_http/handler.rb#L68)\r\n- [lib/seahorse/client/net_http/handler.rb:42 :in `call`](https://github.com/aws/aws-sdk-ruby/blob/v2.6.1/aws-sdk-core/lib/seahorse/client/net_http/handler.rb#L42)\r\n\r\nFull `aws-sdk` stack trace with all handlers:\r\n\r\n```\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:40 :in `block in initialize`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:89 :in `yield`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:89 :in `block in session_for`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:87 :in `synchronize`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/connection_pool.rb:87 :in `session_for`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/handler.rb:116 :in `session`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/handler.rb:68 :in `transmit`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/net_http/handler.rb:42 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/plugins/content_length.rb:12 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_request_signer.rb:88 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_request_signer.rb:23 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/xml/error_handler.rb:8 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/helpful_socket_errors.rb:10 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_request_signer.rb:65 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_redirects.rb:15 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/retry_errors.rb:87 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_dualstack.rb:31 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_accelerate.rb:42 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_md5s.rb:31 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_expect_100_continue.rb:21 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_bucket_name_restrictions.rb:12 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_bucket_dns.rb:31 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/rest/handler.rb:7 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/user_agent.rb:12 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/plugins/endpoint.rb:41 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_url_encoded_keys.rb:39 :in `manage_keys`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_url_encoded_keys.rb:31 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/param_validator.rb:21 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/plugins/raise_response_errors.rb:14 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_sse_cpk.rb:19 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_dualstack.rb:23 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/s3_accelerate.rb:33 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/param_converter.rb:20 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/aws-sdk-core/plugins/response_paging.rb:26 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/plugins/response_target.rb:21 :in `call`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/request.rb:70 :in `send_request`\r\n[GEM_ROOT]/gems/aws-sdk-core-2.6.1/lib/seahorse/client/base.rb:207 :in `block (2 levels) in define_operation_methods`\r\n[...Application code stacktrace...]\r\n```\r\n\r\nReproducing the issue with debug logging would be challenging (since the issue only occurs intermittently in production), so I'm hoping to track this down by inspecting the codebase directly."},"comment":{"url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/comments/261304198","html_url":"https://github.com/aws/aws-sdk-ruby/issues/1331#issuecomment-261304198","issue_url":"https://api.github.com/repos/aws/aws-sdk-ruby/issues/1331","id":261304198,"user":{"login":"wjordan","id":56541,"avatar_url":"https://avatars.githubusercontent.com/u/56541?v=3","gravatar_id":"","url":"https://api.github.com/users/wjordan","html_url":"https://github.com/wjordan","followers_url":"https://api.github.com/users/wjordan/followers","following_url":"https://api.github.com/users/wjordan/following{/other_user}","gists_url":"https://api.github.com/users/wjordan/gists{/gist_id}","starred_url":"https://api.github.com/users/wjordan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wjordan/subscriptions","organizations_url":"https://api.github.com/users/wjordan/orgs","repos_url":"https://api.github.com/users/wjordan/repos","events_url":"https://api.github.com/users/wjordan/events{/privacy}","received_events_url":"https://api.github.com/users/wjordan/received_events","type":"User","site_admin":false},"created_at":"2016-11-17T17:00:28Z","updated_at":"2016-11-17T17:00:28Z","body":"Thanks for looking into this! I will update as soon as a new release is cut.\r\n\r\nUnfortunately since the issue is very intermittent (most recently we saw ~2 weeks between occurrences on our production traffic- we logged 692 errors on 11/4, then 139 errors on 11/17), It will be difficult to let you know if it definitively resolves the issue. However I will be sure to update here if the error ever occurs again after updating."}}
{"actor":{"display_login":"wjordan"}}