{"action":"created","issue":{"url":"https://api.github.com/repos/kyuupichan/electrumx/issues/348","repository_url":"https://api.github.com/repos/kyuupichan/electrumx","labels_url":"https://api.github.com/repos/kyuupichan/electrumx/issues/348/labels{/name}","comments_url":"https://api.github.com/repos/kyuupichan/electrumx/issues/348/comments","events_url":"https://api.github.com/repos/kyuupichan/electrumx/issues/348/events","html_url":"https://github.com/kyuupichan/electrumx/issues/348","id":288048316,"node_id":"MDU6SXNzdWUyODgwNDgzMTY=","number":348,"title":"Pagination of long transaction list responses","user":{"login":"colatkinson","id":6528368,"node_id":"MDQ6VXNlcjY1MjgzNjg=","avatar_url":"https://avatars2.githubusercontent.com/u/6528368?v=4","gravatar_id":"","url":"https://api.github.com/users/colatkinson","html_url":"https://github.com/colatkinson","followers_url":"https://api.github.com/users/colatkinson/followers","following_url":"https://api.github.com/users/colatkinson/following{/other_user}","gists_url":"https://api.github.com/users/colatkinson/gists{/gist_id}","starred_url":"https://api.github.com/users/colatkinson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colatkinson/subscriptions","organizations_url":"https://api.github.com/users/colatkinson/orgs","repos_url":"https://api.github.com/users/colatkinson/repos","events_url":"https://api.github.com/users/colatkinson/events{/privacy}","received_events_url":"https://api.github.com/users/colatkinson/received_events","type":"User","site_admin":false},"labels":[{"id":457367035,"node_id":"MDU6TGFiZWw0NTczNjcwMzU=","url":"https://api.github.com/repos/kyuupichan/electrumx/labels/enhancement","name":"enhancement","color":"84b6eb","default":true}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":11,"created_at":"2018-01-12T08:56:26Z","updated_at":"2018-08-10T10:47:12Z","closed_at":null,"author_association":"NONE","body":"Currently, if the server generates an extremely long response, it simply returns an error message, such as\r\n\r\n    { message: 'response too large (at least 1021611 bytes)', code: -32600 }\r\n\r\nIf the client receives this message, it will just hang indefinitely while attempting to sync.\r\n\r\n### Proposed Solution\r\n\r\nImplement pagination in relevant methods, using block heights as a range. This has a couple advantages:\r\n\r\n1. Allow handling of very active addresses\r\n2. During sync, allow clients to only fetch transactions *after* where they left off, instead of fetching the entirety of their history. Same goes for getting tx inputs. This could conceivably lead to issues with a reorg, but the client could just fetch transactions starting from `last height - n` where `n` is e.g. 6\r\n\r\n### Affected RPC Methods\r\n\r\n* `blockchain.address.get_history`\r\n* `blockchain.address.get_mempool`\r\n* `blockchain.address.listunspent`\r\n* `blockchain.scripthash.get_history`\r\n* `blockchain.scripthash.listunspent`\r\n\r\nIt seems to me like the simplest solution would be to bump the protocol version number, and add optional pagination parameters to each of these methods. The version bump would allow clients to detect support for this, and thus be able to opt to get only the new txs during sync."},"comment":{"url":"https://api.github.com/repos/kyuupichan/electrumx/issues/comments/412047740","html_url":"https://github.com/kyuupichan/electrumx/issues/348#issuecomment-412047740","issue_url":"https://api.github.com/repos/kyuupichan/electrumx/issues/348","id":412047740,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMjA0Nzc0MA==","user":{"login":"kyuupichan","id":3319176,"node_id":"MDQ6VXNlcjMzMTkxNzY=","avatar_url":"https://avatars1.githubusercontent.com/u/3319176?v=4","gravatar_id":"","url":"https://api.github.com/users/kyuupichan","html_url":"https://github.com/kyuupichan","followers_url":"https://api.github.com/users/kyuupichan/followers","following_url":"https://api.github.com/users/kyuupichan/following{/other_user}","gists_url":"https://api.github.com/users/kyuupichan/gists{/gist_id}","starred_url":"https://api.github.com/users/kyuupichan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kyuupichan/subscriptions","organizations_url":"https://api.github.com/users/kyuupichan/orgs","repos_url":"https://api.github.com/users/kyuupichan/repos","events_url":"https://api.github.com/users/kyuupichan/events{/privacy}","received_events_url":"https://api.github.com/users/kyuupichan/received_events","type":"User","site_admin":false},"created_at":"2018-08-10T10:47:12Z","updated_at":"2018-08-10T10:47:12Z","author_association":"OWNER","body":"@erasmospunk let me think about it.  I think my new history DB format that I'm intending to roll out soon actually stores the total TX count in the main history entry of a hashX if the history \"overflows\" about 10-ish transactions.  If so determining length would be a single read, and another single read for the final tx number, and then a filesystem-lookup for the tx_hash.  I might consider storing the final tx number in large histories in the main entry too...."}}
{"actor":{"display_login":"kyuupichan"}}