{"action":"created","issue":{"url":"https://api.github.com/repos/mapbox/rasterio/issues/494","repository_url":"https://api.github.com/repos/mapbox/rasterio","labels_url":"https://api.github.com/repos/mapbox/rasterio/issues/494/labels{/name}","comments_url":"https://api.github.com/repos/mapbox/rasterio/issues/494/comments","events_url":"https://api.github.com/repos/mapbox/rasterio/issues/494/events","html_url":"https://github.com/mapbox/rasterio/issues/494","id":110680160,"number":494,"title":"memory leak in reproject()?","user":{"login":"weatherfrog","id":8851056,"avatar_url":"https://avatars.githubusercontent.com/u/8851056?v=3","gravatar_id":"","url":"https://api.github.com/users/weatherfrog","html_url":"https://github.com/weatherfrog","followers_url":"https://api.github.com/users/weatherfrog/followers","following_url":"https://api.github.com/users/weatherfrog/following{/other_user}","gists_url":"https://api.github.com/users/weatherfrog/gists{/gist_id}","starred_url":"https://api.github.com/users/weatherfrog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/weatherfrog/subscriptions","organizations_url":"https://api.github.com/users/weatherfrog/orgs","repos_url":"https://api.github.com/users/weatherfrog/repos","events_url":"https://api.github.com/users/weatherfrog/events{/privacy}","received_events_url":"https://api.github.com/users/weatherfrog/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/mapbox/rasterio/labels/bug","name":"bug","color":"fc2929"}],"state":"closed","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/mapbox/rasterio/milestones/42","html_url":"https://github.com/mapbox/rasterio/milestones/0.32","labels_url":"https://api.github.com/repos/mapbox/rasterio/milestones/42/labels","id":1550069,"number":42,"title":"0.32","description":null,"creator":{"login":"sgillies","id":33697,"avatar_url":"https://avatars.githubusercontent.com/u/33697?v=3","gravatar_id":"","url":"https://api.github.com/users/sgillies","html_url":"https://github.com/sgillies","followers_url":"https://api.github.com/users/sgillies/followers","following_url":"https://api.github.com/users/sgillies/following{/other_user}","gists_url":"https://api.github.com/users/sgillies/gists{/gist_id}","starred_url":"https://api.github.com/users/sgillies/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sgillies/subscriptions","organizations_url":"https://api.github.com/users/sgillies/orgs","repos_url":"https://api.github.com/users/sgillies/repos","events_url":"https://api.github.com/users/sgillies/events{/privacy}","received_events_url":"https://api.github.com/users/sgillies/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":21,"state":"open","created_at":"2016-01-28T22:09:29Z","updated_at":"2016-03-16T21:54:40Z","due_on":null,"closed_at":null},"comments":14,"created_at":"2015-10-09T15:21:01Z","updated_at":"2016-03-17T08:46:51Z","closed_at":"2016-03-16T21:54:40Z","body":"Is it possible that rasterio is leaking memory? The following plot shows increasing memory consumption of a script that iteratively calls ``rasterio.reproject()``:\r\n![memusage](https://cloud.githubusercontent.com/assets/8851056/10397544/8b9b6728-6ea8-11e5-8078-5ba4e04449ad.png)\r\n\r\nTested with rasterio 0.28.0, numpy 1.10.0 on Python 3.4/2.7 on 64-bit Ubuntu 14.04.\r\n\r\nHere's the code that produces the above figure:\r\n```python\r\n\"\"\"\r\nDemonstrates a potential memory leak in rasterio\r\n\"\"\"\r\n\r\nimport os\r\nfrom itertools import product\r\n\r\nimport psutil\r\nimport numpy as np\r\nimport rasterio\r\nfrom rasterio import transform\r\nfrom rasterio.warp import reproject, RESAMPLING\r\nfrom matplotlib import pyplot as plt\r\n\r\n\r\ndef main():\r\n    \"\"\"\r\n    reproject an image to different sizes, plot memory consumption\r\n    \"\"\"\r\n    mem_usage = []\r\n    widths = heights = range(100, 900, 50)\r\n    for width, height in product(widths, heights):\r\n        mem = get_memory_usage()\r\n        mem_usage.append(mem)\r\n        print(\"### memory usage: {:.2f} kB\".format(mem))\r\n        print(\"reprojecting image to {} x {}\".format(width, height))\r\n        call_reproject(width, height)\r\n\r\n    x = np.arange(0, len(mem_usage))\r\n    plt.plot(x, mem_usage)\r\n    plt.xlabel(\"# reprojections\")\r\n    plt.ylabel(\"memory consumption in kB\")\r\n    image_path = \"/tmp/memusage.png\"\r\n    print(\"saved plot to {}\".format(image_path))\r\n    plt.savefig(image_path, dpi=70)\r\n\r\n\r\ndef get_memory_usage():\r\n    \"\"\"\r\n    Return memory usage of current process in kB\r\n    \"\"\"\r\n    process = psutil.Process(os.getpid())\r\n    mem = process.memory_info()[0] / float(2 ** 20)\r\n\r\n    return mem\r\n\r\n\r\ndef call_reproject(width, height):\r\n    \"\"\"\r\n    rasterio.reproject a random source image to size width x height\r\n    \"\"\"\r\n    # source image\r\n    img_shape = (4, 1000, 1200)\r\n    source = np.random.randint(1, 100, size=img_shape).astype(np.uint8)\r\n    # bounds of source\r\n    west, east, north, south = (-20009678, 20037508,\r\n                                11935768, -11935768)\r\n\r\n    # fix some arbitrary destination bounds\r\n    low_x, high_x, low_y, high_y = (-20009678, 20037508,\r\n                                    -11753184, 11753184)\r\n    with rasterio.drivers():\r\n        _, rows, cols = source.shape\r\n        src_transform = transform.from_bounds(west=west, south=south,\r\n                                              east=east, north=north,\r\n                                              width=cols, height=rows)\r\n        src_crs = dst_crs = {'init': 'epsg:3857'}\r\n        dst_transform = transform.from_bounds(west=low_x, south=low_y,\r\n                                              east=high_x, north=high_y,\r\n                                              width=width, height=height)\r\n        dst_shape = (source.shape[0], height, width)\r\n        destination = np.empty(dst_shape, np.uint8)\r\n        reproject(source, destination,\r\n                  src_transform=src_transform, src_crs=src_crs,\r\n                  dst_transform=dst_transform, dst_crs=dst_crs,\r\n                  resampling=RESAMPLING.nearest)\r\n\r\n\r\nif __name__ == \"__main__\":\r\n    main()\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/mapbox/rasterio/issues/comments/197767854","html_url":"https://github.com/mapbox/rasterio/issues/494#issuecomment-197767854","issue_url":"https://api.github.com/repos/mapbox/rasterio/issues/494","id":197767854,"user":{"login":"weatherfrog","id":8851056,"avatar_url":"https://avatars.githubusercontent.com/u/8851056?v=3","gravatar_id":"","url":"https://api.github.com/users/weatherfrog","html_url":"https://github.com/weatherfrog","followers_url":"https://api.github.com/users/weatherfrog/followers","following_url":"https://api.github.com/users/weatherfrog/following{/other_user}","gists_url":"https://api.github.com/users/weatherfrog/gists{/gist_id}","starred_url":"https://api.github.com/users/weatherfrog/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/weatherfrog/subscriptions","organizations_url":"https://api.github.com/users/weatherfrog/orgs","repos_url":"https://api.github.com/users/weatherfrog/repos","events_url":"https://api.github.com/users/weatherfrog/events{/privacy}","received_events_url":"https://api.github.com/users/weatherfrog/received_events","type":"User","site_admin":false},"created_at":"2016-03-17T08:46:51Z","updated_at":"2016-03-17T08:46:51Z","body":"I can confirm: chart looks a lot better. The leak seems to be fixed in master branch. Thanks a lot!\r\n![memusage](https://cloud.githubusercontent.com/assets/8851056/13840791/04155930-ec25-11e5-815a-5ca931363c99.png)\r\n"}}