{"action":"created","issue":{"url":"https://api.github.com/repos/biopython/biopython/issues/716","repository_url":"https://api.github.com/repos/biopython/biopython","labels_url":"https://api.github.com/repos/biopython/biopython/issues/716/labels{/name}","comments_url":"https://api.github.com/repos/biopython/biopython/issues/716/comments","events_url":"https://api.github.com/repos/biopython/biopython/issues/716/events","html_url":"https://github.com/biopython/biopython/issues/716","id":120548262,"node_id":"MDU6SXNzdWUxMjA1NDgyNjI=","number":716,"title":"internal automatic handling of big requests and large records with Entrez","user":{"login":"noMICROSOFTbuhtz","id":11861496,"node_id":"MDQ6VXNlcjExODYxNDk2","avatar_url":"https://avatars0.githubusercontent.com/u/11861496?v=4","gravatar_id":"","url":"https://api.github.com/users/noMICROSOFTbuhtz","html_url":"https://github.com/noMICROSOFTbuhtz","followers_url":"https://api.github.com/users/noMICROSOFTbuhtz/followers","following_url":"https://api.github.com/users/noMICROSOFTbuhtz/following{/other_user}","gists_url":"https://api.github.com/users/noMICROSOFTbuhtz/gists{/gist_id}","starred_url":"https://api.github.com/users/noMICROSOFTbuhtz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/noMICROSOFTbuhtz/subscriptions","organizations_url":"https://api.github.com/users/noMICROSOFTbuhtz/orgs","repos_url":"https://api.github.com/users/noMICROSOFTbuhtz/repos","events_url":"https://api.github.com/users/noMICROSOFTbuhtz/events{/privacy}","received_events_url":"https://api.github.com/users/noMICROSOFTbuhtz/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2015-12-05T11:24:00Z","updated_at":"2018-06-12T15:41:06Z","closed_at":"2018-06-12T15:41:06Z","author_association":"NONE","body":"(Sorry for my bad english.)\nThis is a feature request about a concept. This would be big work and is not implemented in the next month. I am aware of that of course. It is a place to discusse about the concept. And the persons being technical more involved then me (I am just a _new user_ using `Bio.Entrez` with _pubmed_) are maybe able to open seperate technical more detailed issues to implement that step by step.\n\nThere are two problems with `Entrez` using (in my cases) `efetch()` and `esearch()`.\n1. The user itself have to take care about `retmax` - how much records/PMIDs will come back per request. It means doing more requests with incrementing `retstart` each iteration.\n2. Sometimes there is not enough RAM to hold all recods (e.g. over 10.000 efetch-pubmed-records on a 32bit-4GB-RAM-system) in there. This is currently solved with storing seperated requests as XML files on hard drive.\n\nWhen I asked on the list I get a lot of answers how to solve that. This means a lot of users solve known problems again and again with (nearly) the same solution. That is why I think this could be done by `Bio.Entrez` itself.\n\nMy inspiration about that was because `Entrez` currently handle the only-3-requests-per-second interally, too.\n\nMaybe conceputal solutions (without answers to all detail questions comin up):\n1. `Entrez` should care about the number of user-requests and seperate them in a list of requests. Question here is how large (`retmax`) should one request be? The pubmed-support itself doesn't like to tell discrete numbers here. For `esearch()` they gave me `500` because there _ebot_ use this number, too. But `15000` is working, too. Maybe this should be handled _dynamicly_: Check how long one request take and recalculate this for one record. `(timeAfter - timeBefore) / record['Count']`. This number should be observed over alle seperated requests. If it goes to high, reduce `retmax` for the next requests, ...\n2. I know that currently `esearch()` and `efetch()` returning a handle-object that can be treated like a file-handle. I don't know enough about technical internals to imagnine how my wish could be implemented. I parse the XML comin back: `record = Entrez.read(handle)`. When this becomes to big for RAM there is often used seperated XML files on hard drive. I would wish a `record` object that handle that solution internaly. It means `len(record)` or `record['Count']` should return `10.000` not matter that only `500` of them currently in RAM and the rest `9.500` or in XML files on the hard drive. `for r in record:` should work the same way. There could be class encapuslating this XML-file parsing stuff.\n"},"comment":{"url":"https://api.github.com/repos/biopython/biopython/issues/comments/396636575","html_url":"https://github.com/biopython/biopython/issues/716#issuecomment-396636575","issue_url":"https://api.github.com/repos/biopython/biopython/issues/716","id":396636575,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NjYzNjU3NQ==","user":{"login":"peterjc","id":63959,"node_id":"MDQ6VXNlcjYzOTU5","avatar_url":"https://avatars2.githubusercontent.com/u/63959?v=4","gravatar_id":"","url":"https://api.github.com/users/peterjc","html_url":"https://github.com/peterjc","followers_url":"https://api.github.com/users/peterjc/followers","following_url":"https://api.github.com/users/peterjc/following{/other_user}","gists_url":"https://api.github.com/users/peterjc/gists{/gist_id}","starred_url":"https://api.github.com/users/peterjc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/peterjc/subscriptions","organizations_url":"https://api.github.com/users/peterjc/orgs","repos_url":"https://api.github.com/users/peterjc/repos","events_url":"https://api.github.com/users/peterjc/events{/privacy}","received_events_url":"https://api.github.com/users/peterjc/received_events","type":"User","site_admin":false},"created_at":"2018-06-12T15:41:06Z","updated_at":"2018-06-12T15:41:06Z","author_association":"MEMBER","body":"I'm going to mark this as closed."}}
{"actor":{"display_login":"peterjc"}}