{"action":"created","issue":{"url":"https://api.github.com/repos/apache/couchdb/issues/1209","repository_url":"https://api.github.com/repos/apache/couchdb","labels_url":"https://api.github.com/repos/apache/couchdb/issues/1209/labels{/name}","comments_url":"https://api.github.com/repos/apache/couchdb/issues/1209/comments","events_url":"https://api.github.com/repos/apache/couchdb/issues/1209/events","html_url":"https://github.com/apache/couchdb/issues/1209","id":303152148,"number":1209,"title":"Erlang native query server consumes too much RAM when filters a document with lots of revisions","user":{"login":"mmarinchenko","id":21952330,"avatar_url":"https://avatars0.githubusercontent.com/u/21952330?v=4","gravatar_id":"","url":"https://api.github.com/users/mmarinchenko","html_url":"https://github.com/mmarinchenko","followers_url":"https://api.github.com/users/mmarinchenko/followers","following_url":"https://api.github.com/users/mmarinchenko/following{/other_user}","gists_url":"https://api.github.com/users/mmarinchenko/gists{/gist_id}","starred_url":"https://api.github.com/users/mmarinchenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mmarinchenko/subscriptions","organizations_url":"https://api.github.com/users/mmarinchenko/orgs","repos_url":"https://api.github.com/users/mmarinchenko/repos","events_url":"https://api.github.com/users/mmarinchenko/events{/privacy}","received_events_url":"https://api.github.com/users/mmarinchenko/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-03-07T15:50:49Z","updated_at":"2018-03-27T01:49:10Z","closed_at":null,"author_association":"NONE","body":"Hi all! We have online service which uses CouchDB heavily. It has many databases with frequently updated documents. So we faced a problem.\r\n\r\nDocuments are processed by native query servers with all of its revisions as a one single entity when they are requested with `style=all_docs` parameter. As a result when document with many, many revisions including deleted ones is requested through native Erlang filter CouchDB eats inadequate amount of memory.\r\n\r\nThe reason is as follows (branch `master` is used here just for simplicity, we use CouchDB 1.6 actually):\r\n1. `couch_changes:filter()` reads all revisions of a document into `Docs` and calls `couch_query_servers:filter_docs()`\r\n2. `couch_query_servers:filter_docs()` calls `couch_native_process:prompt()` through `ddoc_prompt()`/`proc_prompt()`\r\n3. And then `couch_native_process:handle_call()` tries to convert the document with all of its revisions `to_binary` at once!\r\n\r\nFilter processes each revision one by one, right? Why trying to handle all revisions in memory? And if you can handle all revisions in memory then why not to handle in memory entire database? :)\r\n\r\nOf course, this problem may be workarounded from the maintenance perspective. Old revisions may be purged from the databases, databases may be compacted and so on. But this is not mean that current behavior is an example of good software design, I guess.\r\n\r\n## Expected Behavior\r\nAssume that the RAM in our galaxy is not infinite, and the document may have many and many and many revisions. Simply put do not load all revisions at once. After all each revision is a document itself.\r\n\r\n## Steps to Reproduce\r\n1. Create database with document which has a lot of revisions (if you have very much amount of RAM you may try to create 10 000 revisions or even greater).\r\n2. Enable Erlang native query server in local.ini and write any simple filter using Erlang (the document from point 1 is not required to pass this filter).\r\n3. Request the _changes feed using created Erlang filter with \"style=all_docs\" parameter. Just to clarify you may emulate 16 parallel queries as follows:\r\n`for i in {1..16}; do curl -u admin:admin \"http://localhost:5984/db/_changes?feed=normal&style=all_docs&filter=app%2FverySimpleErlangFilter\" & done`\r\n4. Run htop and look at how CouchDB eats all your memory!\r\n\r\n## Environment\r\n* Version used: 1.6.0\r\n* Operating System and version: Ubuntu 16.04\r\n"},"comment":{"url":"https://api.github.com/repos/apache/couchdb/issues/comments/376368684","html_url":"https://github.com/apache/couchdb/issues/1209#issuecomment-376368684","issue_url":"https://api.github.com/repos/apache/couchdb/issues/1209","id":376368684,"user":{"login":"wohali","id":112292,"avatar_url":"https://avatars2.githubusercontent.com/u/112292?v=4","gravatar_id":"","url":"https://api.github.com/users/wohali","html_url":"https://github.com/wohali","followers_url":"https://api.github.com/users/wohali/followers","following_url":"https://api.github.com/users/wohali/following{/other_user}","gists_url":"https://api.github.com/users/wohali/gists{/gist_id}","starred_url":"https://api.github.com/users/wohali/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/wohali/subscriptions","organizations_url":"https://api.github.com/users/wohali/orgs","repos_url":"https://api.github.com/users/wohali/repos","events_url":"https://api.github.com/users/wohali/events{/privacy}","received_events_url":"https://api.github.com/users/wohali/received_events","type":"User","site_admin":false},"created_at":"2018-03-27T01:49:10Z","updated_at":"2018-03-27T01:49:10Z","author_association":"MEMBER","body":"I see this is on 1.6.0. Have you tried this against 1.7.1? Or 2.1.1?"}}
{"actor":{"display_login":"wohali"}}