{"action":"created","issue":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/19459","repository_url":"https://api.github.com/repos/pandas-dev/pandas","labels_url":"https://api.github.com/repos/pandas-dev/pandas/issues/19459/labels{/name}","comments_url":"https://api.github.com/repos/pandas-dev/pandas/issues/19459/comments","events_url":"https://api.github.com/repos/pandas-dev/pandas/issues/19459/events","html_url":"https://github.com/pandas-dev/pandas/issues/19459","id":292729359,"number":19459,"title":"Dataframe w/ DatetimeIndex memory leak","user":{"login":"ace-e4s","id":18338679,"avatar_url":"https://avatars2.githubusercontent.com/u/18338679?v=4","gravatar_id":"","url":"https://api.github.com/users/ace-e4s","html_url":"https://github.com/ace-e4s","followers_url":"https://api.github.com/users/ace-e4s/followers","following_url":"https://api.github.com/users/ace-e4s/following{/other_user}","gists_url":"https://api.github.com/users/ace-e4s/gists{/gist_id}","starred_url":"https://api.github.com/users/ace-e4s/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ace-e4s/subscriptions","organizations_url":"https://api.github.com/users/ace-e4s/orgs","repos_url":"https://api.github.com/users/ace-e4s/repos","events_url":"https://api.github.com/users/ace-e4s/events{/privacy}","received_events_url":"https://api.github.com/users/ace-e4s/received_events","type":"User","site_admin":false},"labels":[{"id":8935311,"url":"https://api.github.com/repos/pandas-dev/pandas/labels/Performance","name":"Performance","color":"a10c02","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/pandas-dev/pandas/milestones/33","html_url":"https://github.com/pandas-dev/pandas/milestone/33","labels_url":"https://api.github.com/repos/pandas-dev/pandas/milestones/33/labels","id":997544,"number":33,"title":"No action","description":"A milestone for closed or open issues for which there is no action needed (eg not a bug, just a question / discussion, nothing to resolve, wontfix, etc).\r\n\r\n","creator":{"login":"jorisvandenbossche","id":1020496,"avatar_url":"https://avatars2.githubusercontent.com/u/1020496?v=4","gravatar_id":"","url":"https://api.github.com/users/jorisvandenbossche","html_url":"https://github.com/jorisvandenbossche","followers_url":"https://api.github.com/users/jorisvandenbossche/followers","following_url":"https://api.github.com/users/jorisvandenbossche/following{/other_user}","gists_url":"https://api.github.com/users/jorisvandenbossche/gists{/gist_id}","starred_url":"https://api.github.com/users/jorisvandenbossche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jorisvandenbossche/subscriptions","organizations_url":"https://api.github.com/users/jorisvandenbossche/orgs","repos_url":"https://api.github.com/users/jorisvandenbossche/repos","events_url":"https://api.github.com/users/jorisvandenbossche/events{/privacy}","received_events_url":"https://api.github.com/users/jorisvandenbossche/received_events","type":"User","site_admin":false},"open_issues":5,"closed_issues":1234,"state":"open","created_at":"2015-02-26T19:29:05Z","updated_at":"2018-02-01T11:18:22Z","due_on":"2019-09-30T07:00:00Z","closed_at":null},"comments":8,"created_at":"2018-01-30T10:26:51Z","updated_at":"2018-02-02T22:04:44Z","closed_at":"2018-01-30T11:20:35Z","author_association":"NONE","body":"#### Code Sample, a copy-pastable example if possible\r\n\r\n```python\r\nimport gc\r\nimport numpy as np\r\nimport pandas as pd\r\nfrom memory_profiler import profile\r\n\r\n\r\n@profile\r\ndef leaky_af():\r\n    channels = ['Sonya Blade', 'Johnny Cage', 'Raiden', 'Scorpion', 'Kano']\r\n\r\n    data_df = pd.DataFrame()  # w/ default index\r\n    for channel in channels:\r\n        data = pd.Series(data=np.random.random(50000),\r\n                         index=pd.DatetimeIndex(np.arange(0, 50000)))\r\n        data_df[channel] = data\r\n    return\r\n\r\n\r\n@profile\r\ndef tight():\r\n    channels = ['Sonya Blade', 'Johny Cage', 'Raiden', 'Scorpion', 'Kano']\r\n\r\n    data_df = pd.DataFrame(index=pd.DatetimeIndex([]))  # w/ DatetimeIndex\r\n    for channel in channels:\r\n        data = pd.Series(data=np.random.random(50000),\r\n                         index=pd.DatetimeIndex(np.arange(0, 50000)))\r\n        data_df[channel] = data\r\n    return\r\n\r\ndef main():\r\n    \"\"\"\r\n    leaky_af or tight is ran in a loop. The functions are self-contained, accepts nothing,\r\n    returns nothing. Yet, memory usage increases (if leaky_af is used) proportional to the\r\n    length of the loop.\r\n    \"\"\"\r\n    for i in range(50):\r\n        leaky_af()  # replace with tight(), and memory usage stabilizes\r\n        gc.collect()\r\n\r\nif __name__ == '__main__':\r\n    main()\r\n\r\n```\r\n#### Problem description\r\n\r\nMemory seems to leak when adding new Series with DatetimeIndex columns to an empty DataFrame (which was initialized with default index). The DataFrame is created in a self-contained functions which\r\naccepts nothing and returns nothing. So all objects inside that function have no references outside its scope and should be released from memory. Still, memory usage increases steadily when the functions is called repeatedly in a loop. \r\n\r\nHowever, if the DataFrame is initialized with an empty DatetimeIndex index, the leakage stops.\r\n\r\nNote, I'm aware that leaky_af may trigger some copying when appending series with different index type, and that's fine. However, that copy should also be destroyed when leaky_af() ends (and returns nothing). \r\n\r\n##### main() called with tight()\r\n![pandas tight](https://user-images.githubusercontent.com/18338679/35561333-39e99430-05b0-11e8-8fb3-bd06f46a3ddd.png)\r\n\r\n##### main() called with leaky_af()\r\n![pandas leaky_af](https://user-images.githubusercontent.com/18338679/35561335-3ce0feda-05b0-11e8-95c0-6c58504e8e85.png)\r\n\r\n#### Expected Output\r\n\r\n#### Output of ``pd.show_versions()``\r\n\r\npandas 0.22.0\r\npython 2.7.14 (not tested in python 3)\r\n\r\n</details>\r\n"},"comment":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/comments/362721127","html_url":"https://github.com/pandas-dev/pandas/issues/19459#issuecomment-362721127","issue_url":"https://api.github.com/repos/pandas-dev/pandas/issues/19459","id":362721127,"user":{"login":"TomAugspurger","id":1312546,"avatar_url":"https://avatars3.githubusercontent.com/u/1312546?v=4","gravatar_id":"","url":"https://api.github.com/users/TomAugspurger","html_url":"https://github.com/TomAugspurger","followers_url":"https://api.github.com/users/TomAugspurger/followers","following_url":"https://api.github.com/users/TomAugspurger/following{/other_user}","gists_url":"https://api.github.com/users/TomAugspurger/gists{/gist_id}","starred_url":"https://api.github.com/users/TomAugspurger/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/TomAugspurger/subscriptions","organizations_url":"https://api.github.com/users/TomAugspurger/orgs","repos_url":"https://api.github.com/users/TomAugspurger/repos","events_url":"https://api.github.com/users/TomAugspurger/events{/privacy}","received_events_url":"https://api.github.com/users/TomAugspurger/received_events","type":"User","site_admin":false},"created_at":"2018-02-02T22:04:43Z","updated_at":"2018-02-02T22:04:43Z","author_association":"CONTRIBUTOR","body":"That seems most likely.\n\nSince it's python 2 only, I suspect we would gladly take a patch fixing the\nbehavior you're seeing, but I personally don't have any plans to look into\nit :)\n\nOn Fri, Feb 2, 2018 at 10:01 PM, ace-e4s <notifications@github.com> wrote:\n\n> @TomAugspurger <https://github.com/tomaugspurger> I tried to run it with\n> Python 3.6 and got similar results as you; memory usage is comparable\n> between leaky_af and tight, but tight runs much faster.\n>\n> May this be a case of circular referencing somewhere, and Python 2 garbage\n> collector is failing to collect? (If I'm not wrong, Python 3.4 improved\n> significantly in that respect.)\n> Python 3.6 leaky_af()\n>\n> [image: pandas leaky_af - py36]\n> <https://user-images.githubusercontent.com/18338679/35756803-ebec0c4e-086c-11e8-918f-63d9c8c7e930.png>\n>\n> —\n> You are receiving this because you were mentioned.\n> Reply to this email directly, view it on GitHub\n> <https://github.com/pandas-dev/pandas/issues/19459#issuecomment-362720381>,\n> or mute the thread\n> <https://github.com/notifications/unsubscribe-auth/ABQHIuFGVQ3rUqvB20csaD21rJ5IjaZZks5tQ4W0gaJpZM4RyGGZ>\n> .\n>\n"}}
{"actor":{"display_login":"TomAugspurger"}}