{"action":"created","issue":{"url":"https://api.github.com/repos/medic/medic-webapp/issues/2282","repository_url":"https://api.github.com/repos/medic/medic-webapp","labels_url":"https://api.github.com/repos/medic/medic-webapp/issues/2282/labels{/name}","comments_url":"https://api.github.com/repos/medic/medic-webapp/issues/2282/comments","events_url":"https://api.github.com/repos/medic/medic-webapp/issues/2282/events","html_url":"https://github.com/medic/medic-webapp/issues/2282","id":153705914,"number":2282,"title":"Bootstrapping postgres tends to have different behavior than keeping postgres updated","user":{"login":"btbonval","id":880439,"avatar_url":"https://avatars.githubusercontent.com/u/880439?v=3","gravatar_id":"","url":"https://api.github.com/users/btbonval","html_url":"https://github.com/btbonval","followers_url":"https://api.github.com/users/btbonval/followers","following_url":"https://api.github.com/users/btbonval/following{/other_user}","gists_url":"https://api.github.com/users/btbonval/gists{/gist_id}","starred_url":"https://api.github.com/users/btbonval/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/btbonval/subscriptions","organizations_url":"https://api.github.com/users/btbonval/orgs","repos_url":"https://api.github.com/users/btbonval/repos","events_url":"https://api.github.com/users/btbonval/events{/privacy}","received_events_url":"https://api.github.com/users/btbonval/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/medic/medic-webapp/labels/0%20-%20Backlog","name":"0 - Backlog","color":"d4c5f9"},{"url":"https://api.github.com/repos/medic/medic-webapp/labels/couch2pg","name":"couch2pg","color":"0052cc"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/medic/medic-webapp/milestones/36","html_url":"https://github.com/medic/medic-webapp/milestones/Backlog:%20Deprioritized%20Mobile%20Application%20Issues","labels_url":"https://api.github.com/repos/medic/medic-webapp/milestones/36/labels","id":1294435,"number":36,"title":"Backlog: Deprioritized Mobile Application Issues","description":"Mobile application issues we've chosen to deprioritize.","creator":{"login":"browndav","id":763977,"avatar_url":"https://avatars.githubusercontent.com/u/763977?v=3","gravatar_id":"","url":"https://api.github.com/users/browndav","html_url":"https://github.com/browndav","followers_url":"https://api.github.com/users/browndav/followers","following_url":"https://api.github.com/users/browndav/following{/other_user}","gists_url":"https://api.github.com/users/browndav/gists{/gist_id}","starred_url":"https://api.github.com/users/browndav/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/browndav/subscriptions","organizations_url":"https://api.github.com/users/browndav/orgs","repos_url":"https://api.github.com/users/browndav/repos","events_url":"https://api.github.com/users/browndav/events{/privacy}","received_events_url":"https://api.github.com/users/browndav/received_events","type":"User","site_admin":false},"open_issues":45,"closed_issues":65,"state":"open","created_at":"2015-09-07T23:28:13Z","updated_at":"2016-05-09T05:33:13Z","due_on":"2016-06-01T07:00:00Z","closed_at":null},"comments":2,"created_at":"2016-05-09T05:32:12Z","updated_at":"2016-05-09T21:38:47Z","closed_at":null,"body":"In practice, I found out that bootstrapping postgres from couch tend to operate differently than running the adapter on a populated database. So far, these differences have been acted on by the way in which the adapter is called: the adapter does not make any such distinction.\r\n\r\nTechnically, there doesn't have to be a difference, but practically there probably should be.\r\n\r\nLet's use an example of a CouchDB database with between 50,000 and 300,000 docs (both are actual use cases I've dealt with already). Postgres is being freshly applied, so it has 0 docs in it.\r\n\r\nnode has limited system memory available. The following link talks about some of this, including the `--max-old-space-size` workaround I've had to use for dealing with audit logs. Any large docs, or any large enough set of small docs, can crash node with out of memory error.\r\nhttps://bugs.chromium.org/p/v8/issues/detail?id=847\r\n\r\nThere is a balance between fetching enough data from couch in a single call to not run out of memory, but reduce the overall number of HTTP requests sent to the CouchDB server. We probably don't want a separate GET for each of 50,000 to 300,000 docs. There is a related balance in the interval between HTTP requests sent to CouchDB. If we did call a separate GET for each doc, we'd probably not want to do so in a very short interval of time.\r\n\r\nBoth the maximum number of docs fetched per interval and the interval itself are settings for the adapter. When actually running the adapter, we can get away with 100 docs every 15 minutes pretty safely (both staying on top of new data and not hitting out-of-memory when audit logs or very large docs come in). However, if we used that combination when there are 0 docs in postgres, we're talking about at least 5.2 solid days to get to 50,000 until we're \"caught up\" (which, by five days later, we probably aren't caught up).\r\n\r\nSo for bootstrapping, we might want to crank that up to 4000 docs every 3 minutes. We certainly don't expect that kind of throughput in normal use, so we wouldn't want that to be the normal settings after we've bootstrapped.\r\n\r\nThe problem is that when there are very large docs (audit_logs in 0.4 systems, possibly some kind of doc that might happen in 2.6 sometime in the future), we'll hit an out of memory and the adapter outright crashes when pulling 4000. So for some systems, babysitting the adapter is required while bootstrapping.\r\n\r\nPerhaps there is some way to see how large the next batch of X docs is before downloading them, which can help generate a dynamic doc threshold when bootstrapping (or anytime really). Regardless, it might be worthwhile to consider the bootstrapping case and how we can incorporate it into the adapter (rather than babysitting or running with two different sets of settings)."},"comment":{"url":"https://api.github.com/repos/medic/medic-webapp/issues/comments/217998049","html_url":"https://github.com/medic/medic-webapp/issues/2282#issuecomment-217998049","issue_url":"https://api.github.com/repos/medic/medic-webapp/issues/2282","id":217998049,"user":{"login":"SCdF","id":583851,"avatar_url":"https://avatars.githubusercontent.com/u/583851?v=3","gravatar_id":"","url":"https://api.github.com/users/SCdF","html_url":"https://github.com/SCdF","followers_url":"https://api.github.com/users/SCdF/followers","following_url":"https://api.github.com/users/SCdF/following{/other_user}","gists_url":"https://api.github.com/users/SCdF/gists{/gist_id}","starred_url":"https://api.github.com/users/SCdF/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SCdF/subscriptions","organizations_url":"https://api.github.com/users/SCdF/orgs","repos_url":"https://api.github.com/users/SCdF/repos","events_url":"https://api.github.com/users/SCdF/events{/privacy}","received_events_url":"https://api.github.com/users/SCdF/received_events","type":"User","site_admin":false},"created_at":"2016-05-09T21:38:47Z","updated_at":"2016-05-09T21:38:47Z","body":"So here's what I'm thinking about this, long term:\r\n - Every time PROCESS runs, it asks for the `_changes` feed *without* including the docs, from `lastSeq` to now. This will *hopefully* be small enough for node to handle: if not I'd suggest we either tweak node or look at some other optimisations. For example, current LG prod is an 8.5Mb response with ~73 lines, which I'd hope node would be able to deal with.\r\n - split that into two lists, deletes (where `deleted=true`) and create / updates\r\n - Do all the deletes in one go (presumably that works for postgres)\r\n - In N-sized batches work through the other list, where N is probably the number that you'd worked out in the past.\r\n\r\nIdeally as well we could create a view that we can use over 2.4 and 2.6 that only returns the docs that we need (ie not audit docs, and more ideally only reports etc). In fact (and this is a bit off-topic to this ticket) it would be cool if you knew which docs we need?"}}