{"action":"created","issue":{"url":"https://api.github.com/repos/dw/py-lmdb/issues/115","repository_url":"https://api.github.com/repos/dw/py-lmdb","labels_url":"https://api.github.com/repos/dw/py-lmdb/issues/115/labels{/name}","comments_url":"https://api.github.com/repos/dw/py-lmdb/issues/115/comments","events_url":"https://api.github.com/repos/dw/py-lmdb/issues/115/events","html_url":"https://github.com/dw/py-lmdb/issues/115","id":154512624,"number":115,"title":"Sequentially reading LMDB","user":{"login":"achalddave","id":1168881,"avatar_url":"https://avatars.githubusercontent.com/u/1168881?v=3","gravatar_id":"","url":"https://api.github.com/users/achalddave","html_url":"https://github.com/achalddave","followers_url":"https://api.github.com/users/achalddave/followers","following_url":"https://api.github.com/users/achalddave/following{/other_user}","gists_url":"https://api.github.com/users/achalddave/gists{/gist_id}","starred_url":"https://api.github.com/users/achalddave/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/achalddave/subscriptions","organizations_url":"https://api.github.com/users/achalddave/orgs","repos_url":"https://api.github.com/users/achalddave/repos","events_url":"https://api.github.com/users/achalddave/events{/privacy}","received_events_url":"https://api.github.com/users/achalddave/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":6,"created_at":"2016-05-12T15:31:11Z","updated_at":"2016-05-13T14:05:20Z","closed_at":null,"body":"Maybe I'm not using this the right way, but it seems that sequentially reading an LMDB using a single transaction causes memory usage to constantly grow.\r\n\r\nHere's a small script that demonstrates this:\r\n```python\r\nimport lmdb\r\n\r\nDB_PATH=\"./database.lmdb\"\r\n\r\nnum_entries = 0\r\nwith lmdb.open(DB_PATH, map_size=500e9).begin(\r\n        write=False).cursor() as cursor:\r\n    cursor_iterator = cursor.iternext()\r\n    for _, _ in cursor_iterator:\r\n        num_entries += 1\r\nprint num_entries\r\n```\r\n\r\n(I'm aware that it's possible to count the number of entries by using `stat()` - this is just an easy way to show the problem.)\r\n\r\nIs this expected? I assume breaking up the transactions would solve this issue, but that makes the code uglier.\r\n\r\nEdit: This is an issue even if the cursor_iterator is created using\r\n```python\r\ncursor_iterator = cursor.iternext(values=False)\r\n```"},"comment":{"url":"https://api.github.com/repos/dw/py-lmdb/issues/comments/219052470","html_url":"https://github.com/dw/py-lmdb/issues/115#issuecomment-219052470","issue_url":"https://api.github.com/repos/dw/py-lmdb/issues/115","id":219052470,"user":{"login":"achalddave","id":1168881,"avatar_url":"https://avatars.githubusercontent.com/u/1168881?v=3","gravatar_id":"","url":"https://api.github.com/users/achalddave","html_url":"https://github.com/achalddave","followers_url":"https://api.github.com/users/achalddave/followers","following_url":"https://api.github.com/users/achalddave/following{/other_user}","gists_url":"https://api.github.com/users/achalddave/gists{/gist_id}","starred_url":"https://api.github.com/users/achalddave/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/achalddave/subscriptions","organizations_url":"https://api.github.com/users/achalddave/orgs","repos_url":"https://api.github.com/users/achalddave/repos","events_url":"https://api.github.com/users/achalddave/events{/privacy}","received_events_url":"https://api.github.com/users/achalddave/received_events","type":"User","site_admin":false},"created_at":"2016-05-13T14:05:19Z","updated_at":"2016-05-13T14:05:19Z","body":"Thanks, that's really helpful! LRU seems slightly troublesome, as with the issue here (where the OS will start claiming other process' pages if I were to read through an LMDB sequentially) http://www.openldap.org/lists/openldap-technical/201503/msg00077.html\r\n\r\nI think for now breaking the read into chunks works, since this forces pages to be flushed. Any other suggestions would be appreciated!"}}