{"action":"created","issue":{"url":"https://api.github.com/repos/DataDog/datadogpy/issues/142","repository_url":"https://api.github.com/repos/DataDog/datadogpy","labels_url":"https://api.github.com/repos/DataDog/datadogpy/issues/142/labels{/name}","comments_url":"https://api.github.com/repos/DataDog/datadogpy/issues/142/comments","events_url":"https://api.github.com/repos/DataDog/datadogpy/issues/142/events","html_url":"https://github.com/DataDog/datadogpy/issues/142","id":164592136,"node_id":"MDU6SXNzdWUxNjQ1OTIxMzY=","number":142,"title":"dictionary changed size during iteration for self._metrics in ThreadStats","user":{"login":"leozc","id":1513416,"node_id":"MDQ6VXNlcjE1MTM0MTY=","avatar_url":"https://avatars1.githubusercontent.com/u/1513416?v=4","gravatar_id":"","url":"https://api.github.com/users/leozc","html_url":"https://github.com/leozc","followers_url":"https://api.github.com/users/leozc/followers","following_url":"https://api.github.com/users/leozc/following{/other_user}","gists_url":"https://api.github.com/users/leozc/gists{/gist_id}","starred_url":"https://api.github.com/users/leozc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/leozc/subscriptions","organizations_url":"https://api.github.com/users/leozc/orgs","repos_url":"https://api.github.com/users/leozc/repos","events_url":"https://api.github.com/users/leozc/events{/privacy}","received_events_url":"https://api.github.com/users/leozc/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/DataDog/datadogpy/milestones/7","html_url":"https://github.com/DataDog/datadogpy/milestone/7","labels_url":"https://api.github.com/repos/DataDog/datadogpy/milestones/7/labels","id":1788493,"node_id":"MDk6TWlsZXN0b25lMTc4ODQ5Mw==","number":7,"title":"0.13.0","description":"","creator":{"login":"yannmh","id":5708089,"node_id":"MDQ6VXNlcjU3MDgwODk=","avatar_url":"https://avatars1.githubusercontent.com/u/5708089?v=4","gravatar_id":"","url":"https://api.github.com/users/yannmh","html_url":"https://github.com/yannmh","followers_url":"https://api.github.com/users/yannmh/followers","following_url":"https://api.github.com/users/yannmh/following{/other_user}","gists_url":"https://api.github.com/users/yannmh/gists{/gist_id}","starred_url":"https://api.github.com/users/yannmh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/yannmh/subscriptions","organizations_url":"https://api.github.com/users/yannmh/orgs","repos_url":"https://api.github.com/users/yannmh/repos","events_url":"https://api.github.com/users/yannmh/events{/privacy}","received_events_url":"https://api.github.com/users/yannmh/received_events","type":"User","site_admin":false},"open_issues":0,"closed_issues":9,"state":"closed","created_at":"2016-05-25T21:19:33Z","updated_at":"2016-08-25T14:01:34Z","due_on":null,"closed_at":"2016-08-25T14:01:34Z"},"comments":2,"created_at":"2016-07-08T18:44:29Z","updated_at":"2018-12-08T01:06:21Z","closed_at":"2016-08-24T19:51:18Z","author_association":"NONE","body":"While using **ThreadStats** client under uwsgi:\n\n> Error flushing metrics and events\n> dictionary changed size during iteration\n> on \n> Line: datadog/threadstats/metrics.py in flush at line 142\n> past_intervals = [i for i in self._metrics if i < interval] \n\nThe root cause is while iterating, new items are being added to self._metrics \n"},"comment":{"url":"https://api.github.com/repos/DataDog/datadogpy/issues/comments/445416038","html_url":"https://github.com/DataDog/datadogpy/issues/142#issuecomment-445416038","issue_url":"https://api.github.com/repos/DataDog/datadogpy/issues/142","id":445416038,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTQxNjAzOA==","user":{"login":"overset","id":34342,"node_id":"MDQ6VXNlcjM0MzQy","avatar_url":"https://avatars2.githubusercontent.com/u/34342?v=4","gravatar_id":"","url":"https://api.github.com/users/overset","html_url":"https://github.com/overset","followers_url":"https://api.github.com/users/overset/followers","following_url":"https://api.github.com/users/overset/following{/other_user}","gists_url":"https://api.github.com/users/overset/gists{/gist_id}","starred_url":"https://api.github.com/users/overset/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/overset/subscriptions","organizations_url":"https://api.github.com/users/overset/orgs","repos_url":"https://api.github.com/users/overset/repos","events_url":"https://api.github.com/users/overset/events{/privacy}","received_events_url":"https://api.github.com/users/overset/received_events","type":"User","site_admin":false},"created_at":"2018-12-08T01:06:21Z","updated_at":"2018-12-08T01:06:21Z","author_association":"NONE","body":"This is a regression in python3 that is susceptible to the keys() view object changing which is not threadsafe. In python3 with latest `datadogpy` we're still getting:\r\n\r\n```\r\nRuntimeError: dictionary changed size during iteration\r\n  File \"datadog/threadstats/base.py\", line 307, in flush\r\n    metrics, dists = self._get_aggregate_metrics_and_dists(timestamp or time())\r\n  File \"datadog/threadstats/base.py\", line 348, in _get_aggregate_metrics_and_dists\r\n    rolled_up_metrics = self._metric_aggregator.flush(flush_time)\r\n  File \"datadog/threadstats/metrics.py\", line 173, in flush\r\n    past_intervals = [i for i in self._metrics.keys() if i < interval]\r\n  File \"datadog/threadstats/metrics.py\", line 173, in <listcomp>\r\n    past_intervals = [i for i in self._metrics.keys() if i < interval]\r\n```\r\n\r\nThis is related to the python3 [view object](https://docs.python.org/3.7/library/stdtypes.html?highlight=keys#dict-views) that `.keys()` returns:\r\n\r\n> The objects returned by dict.keys(), dict.values() and dict.items() are view objects. They provide a dynamic view on the dictionary’s entries, which means that when the dictionary changes, the view reflects these changes.\r\n\r\npy2.7 https://docs.python.org/2/library/stdtypes.html?highlight=keys#dict.keys\r\n```\r\nkeys()\r\nReturn a copy of the dictionary’s list of keys. See the note for dict.items().\r\n```\r\npy3.7 https://docs.python.org/3.7/library/stdtypes.html?highlight=keys#dict.keys\r\n```\r\nkeys()\r\nReturn a new view of the dictionary’s keys. See the documentation of view objects.\r\n```\r\n\r\nHere's a quick script to repro similar logic as in https://github.com/DataDog/datadogpy/blob/master/datadog/threadstats/metrics.py#L173 that fails with python3 and works with python2:\r\n\r\n```python\r\nimport threading\r\n\r\ndef prune(data):\r\n    data.pop(1)\r\n\r\ndef print_half(data):\r\n    print(len([i for i in data.keys() if i > len(data) / 2]))\r\n\r\ndata = {i: i for i in range(100000)}\r\nthreading.Thread(target=print_half, args=(data, )).start()\r\nthreading.Thread(target=prune, args=(data, )).start()\r\n```\r\n\r\nIt seems that https://github.com/DataDog/datadogpy/pull/151 only works for python2 and https://github.com/DataDog/datadogpy/pull/143 works for both python2 and python3."}}
{"actor":{"display_login":"overset"}}