{"action":"created","issue":{"url":"https://api.github.com/repos/openvenues/pypostal/issues/10","repository_url":"https://api.github.com/repos/openvenues/pypostal","labels_url":"https://api.github.com/repos/openvenues/pypostal/issues/10/labels{/name}","comments_url":"https://api.github.com/repos/openvenues/pypostal/issues/10/comments","events_url":"https://api.github.com/repos/openvenues/pypostal/issues/10/events","html_url":"https://github.com/openvenues/pypostal/issues/10","id":164058791,"number":10,"title":"parse_address leaks memory on Python2.7 when passing language and/or country","user":{"login":"a9rolf-nb","id":6683058,"avatar_url":"https://avatars.githubusercontent.com/u/6683058?v=3","gravatar_id":"","url":"https://api.github.com/users/a9rolf-nb","html_url":"https://github.com/a9rolf-nb","followers_url":"https://api.github.com/users/a9rolf-nb/followers","following_url":"https://api.github.com/users/a9rolf-nb/following{/other_user}","gists_url":"https://api.github.com/users/a9rolf-nb/gists{/gist_id}","starred_url":"https://api.github.com/users/a9rolf-nb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a9rolf-nb/subscriptions","organizations_url":"https://api.github.com/users/a9rolf-nb/orgs","repos_url":"https://api.github.com/users/a9rolf-nb/repos","events_url":"https://api.github.com/users/a9rolf-nb/events{/privacy}","received_events_url":"https://api.github.com/users/a9rolf-nb/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-07-06T11:54:05Z","updated_at":"2016-07-13T07:34:58Z","closed_at":null,"body":"Tested with Python2.7 and Python3.4.\r\nPassing any string value for either ``language`` or ``country`` in a call to postal.parser.parse_address leaks memory on Python2.7. Unicode vs string does not change it. Garbage collection has no impact. Neither has regular reloading of the Python code modules. I've tried reloading the _postal shared library, but apparently the Python reload machinery does not actually release the .so once loaded, so I gave up on this.\r\n\r\nKnown workarounds:\r\nuse Python3\r\nOnly pass a value, never pass non-empty language, never pass non-empty country\r\n\r\nWe're talking around 40 bytes lost per passed country/language argument, per invocation, starting from two-character str. Empty strings do not leak. Single-character strings do not leak (assuming internal Python optimizations kick in to reuse single-char str instances).\r\n\r\nObserved on this system:\r\n```\r\n$ uname -a\r\nLinux ron-VirtualBox 3.13.0-91-generic #138-Ubuntu SMP Fri Jun 24 17:00:34 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux\r\n$ cat /proc/version\r\nLinux version 3.13.0-91-generic (buildd@lgw01-21) (gcc version 4.8.4 (Ubuntu 4.8.4-2ubuntu1~14.04.3) ) #138-Ubuntu SMP Fri Jun 24 17:00:34 UTC 2016\r\n$ python2.7 --version && python3 --version\r\nPython 2.7.6\r\nPython 3.4.3\r\n$ (pip2.7 freeze && pip3 freeze) | grep postal\r\npostal==0.3\r\npostal==0.3\r\n```\r\nNote: the 2.7.6 is Ubuntu's standard Python version. I have seen the same behavior with Python2.7.11 compiled from source (with `--enable-unicode=ucs4`) on the same machine, and also on a Centos box.\r\nLibpostal (the C library) is installed system global, so both Python versions' postal packages definitely layer onto the same libpostal.so.\r\n\r\n\r\nReproduction script (supports both Python2 and Python3)\r\n```\r\n\"\"\"Demonstrate leaking memory on calling postal.parser.parse_address with\r\n`language` and/or `country` argument(s).\r\nLeak only observable on Python 2.7. Python 3.4 memory usage completely stable.\r\nModule reloading and garbage collection have zero impact.\r\nUsing unicode or str for country/language arguments has zero impact.\r\nLength of country/language strings has impact: longer strings leak more memory.\r\n\"\"\"\r\n\r\n\r\nfrom __future__ import print_function\r\n\r\nimport resource\r\nimport gc\r\nimport imp\r\n\r\nimport postal.parser\r\n\r\ntry:\r\n    # Py3k\r\n    _ = xrange\r\nexcept NameError:\r\n    xrange = range\r\n\r\ndef reload_all_postal_python_modules():\r\n    # reload everything implicitly pulled in by importing postal.parse\r\n    # >>> import sys\r\n    # >>> import postal.parse\r\n    # >>> [name for (name, mod) in sys.modules.items() if 'post' in name and mod]\r\n    # ['postal', 'postal.utils.encoding', 'postal._parser', 'postal.parser', 'postal.utils']\r\n    import postal.parser\r\n    import postal\r\n    import postal.utils\r\n    import postal.utils.encoding\r\n    postal.utils.encoding = imp.reload(postal.utils.encoding)\r\n    postal.utils = imp.reload(postal.utils)\r\n    globals()['postal'].parser = imp.reload(postal.parser)\r\n    globals()['postal'] = imp.reload(postal)\r\n\r\ndef format_maxrss():\r\n    return \"%dkiB\" % (resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, )\r\n\r\ndef run(per_spin=50000, spins=20):\r\n    parse_invocations = 0\r\n    print(\"maxrss=%s at startup\" % (format_maxrss(), ))\r\n    for spin in xrange(spins):\r\n        for invocation in xrange(per_spin):\r\n            # memory continues to grow linearly per invocation\r\n            # ONLY IF either country or language or both are passed.\r\n            # Length of passed value for country / language directly affects rate of memory usage growth.\r\n            # * language='idontknow' => ~47MB per million invocations\r\n            # * country='idontknow' => ~47MB per million invocations\r\n            # * language='idontknowbutletsmakethisalittlelongernow' => ~80MB per million invocations\r\n            # * country='idontknow', language='idontknow' => ~95MB per million invocations\r\n            _ = postal.parser.parse_address(\"Hello\", language='idontknow')\r\n        parse_invocations += per_spin\r\n        print(\"maxrss=%s after spin %d (%d calls to postal.parser.parse_address)\" % (format_maxrss(), spin, parse_invocations))\r\n        \r\n        # reloading postal Python modules regularly does not influence memory usage at all\r\n        reload_all_postal_python_modules()\r\n        # garbage collection does not influence memory usage at all\r\n        gc.collect()\r\n\r\nif __name__ == '__main__':\r\n    run()\r\n\r\n```\r\n\r\nExample output:\r\n```\r\n$ python2.7 pypostal_memory_leak_demo.py \r\nmaxrss=961376kiB at startup\r\nmaxrss=963748kiB after spin 0 (50000 calls to postal.parser.parse_address)\r\nmaxrss=966008kiB after spin 1 (100000 calls to postal.parser.parse_address)\r\nmaxrss=968380kiB after spin 2 (150000 calls to postal.parser.parse_address)\r\nmaxrss=971024kiB after spin 3 (200000 calls to postal.parser.parse_address)\r\nmaxrss=973140kiB after spin 4 (250000 calls to postal.parser.parse_address)\r\nmaxrss=975784kiB after spin 5 (300000 calls to postal.parser.parse_address)\r\nmaxrss=977904kiB after spin 6 (350000 calls to postal.parser.parse_address)\r\nmaxrss=980548kiB after spin 7 (400000 calls to postal.parser.parse_address)\r\nmaxrss=982664kiB after spin 8 (450000 calls to postal.parser.parse_address)\r\nmaxrss=985308kiB after spin 9 (500000 calls to postal.parser.parse_address)\r\nmaxrss=987688kiB after spin 10 (550000 calls to postal.parser.parse_address)\r\nmaxrss=990072kiB after spin 11 (600000 calls to postal.parser.parse_address)\r\nmaxrss=992188kiB after spin 12 (650000 calls to postal.parser.parse_address)\r\nmaxrss=994832kiB after spin 13 (700000 calls to postal.parser.parse_address)\r\nmaxrss=997212kiB after spin 14 (750000 calls to postal.parser.parse_address)\r\nmaxrss=999596kiB after spin 15 (800000 calls to postal.parser.parse_address)\r\nmaxrss=1001712kiB after spin 16 (850000 calls to postal.parser.parse_address)\r\nmaxrss=1004356kiB after spin 17 (900000 calls to postal.parser.parse_address)\r\nmaxrss=1006736kiB after spin 18 (950000 calls to postal.parser.parse_address)\r\nmaxrss=1009120kiB after spin 19 (1000000 calls to postal.parser.parse_address)\r\n$ python3 pypostal_memory_leak_demo.py \r\nmaxrss=962880kiB at startup\r\nmaxrss=962880kiB after spin 0 (50000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 1 (100000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 2 (150000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 3 (200000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 4 (250000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 5 (300000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 6 (350000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 7 (400000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 8 (450000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 9 (500000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 10 (550000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 11 (600000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 12 (650000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 13 (700000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 14 (750000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 15 (800000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 16 (850000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 17 (900000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 18 (950000 calls to postal.parser.parse_address)\r\nmaxrss=962880kiB after spin 19 (1000000 calls to postal.parser.parse_address)\r\n```\r\nHTH\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/openvenues/pypostal/issues/comments/232278837","html_url":"https://github.com/openvenues/pypostal/issues/10#issuecomment-232278837","issue_url":"https://api.github.com/repos/openvenues/pypostal/issues/10","id":232278837,"user":{"login":"thatdatabaseguy","id":238455,"avatar_url":"https://avatars.githubusercontent.com/u/238455?v=3","gravatar_id":"","url":"https://api.github.com/users/thatdatabaseguy","html_url":"https://github.com/thatdatabaseguy","followers_url":"https://api.github.com/users/thatdatabaseguy/followers","following_url":"https://api.github.com/users/thatdatabaseguy/following{/other_user}","gists_url":"https://api.github.com/users/thatdatabaseguy/gists{/gist_id}","starred_url":"https://api.github.com/users/thatdatabaseguy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thatdatabaseguy/subscriptions","organizations_url":"https://api.github.com/users/thatdatabaseguy/orgs","repos_url":"https://api.github.com/users/thatdatabaseguy/repos","events_url":"https://api.github.com/users/thatdatabaseguy/events{/privacy}","received_events_url":"https://api.github.com/users/thatdatabaseguy/received_events","type":"User","site_admin":false},"created_at":"2016-07-13T07:34:57Z","updated_at":"2016-07-13T07:34:57Z","body":"Totally right about documentation. For expand_address the option is actually language code rather than country code (for Germany I suppose it doesn't matter since the two are identical). If the language is known a priori it's a good idea to pass it to expand_address. That should yield better/fewer results, as otherwise a language classifier has to be used to predict the language. For statistically close languages, if the classifier said e.g. \"there's a 90% probability that it's German and a 10% probability it's Dutch\" then it's possible that both German and Dutch expansions would be used."}}
{"actor":{"display_login":"thatdatabaseguy"}}