{"action":"opened","issue":{"url":"https://api.github.com/repos/bcoin-org/bcoin/issues/589","repository_url":"https://api.github.com/repos/bcoin-org/bcoin","labels_url":"https://api.github.com/repos/bcoin-org/bcoin/issues/589/labels{/name}","comments_url":"https://api.github.com/repos/bcoin-org/bcoin/issues/589/comments","events_url":"https://api.github.com/repos/bcoin-org/bcoin/issues/589/events","html_url":"https://github.com/bcoin-org/bcoin/issues/589","id":353963976,"node_id":"MDU6SXNzdWUzNTM5NjM5NzY=","number":589,"title":"Performance of address index HTTP API queries","user":{"login":"braydonf","id":544777,"node_id":"MDQ6VXNlcjU0NDc3Nw==","avatar_url":"https://avatars3.githubusercontent.com/u/544777?v=4","gravatar_id":"","url":"https://api.github.com/users/braydonf","html_url":"https://github.com/braydonf","followers_url":"https://api.github.com/users/braydonf/followers","following_url":"https://api.github.com/users/braydonf/following{/other_user}","gists_url":"https://api.github.com/users/braydonf/gists{/gist_id}","starred_url":"https://api.github.com/users/braydonf/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/braydonf/subscriptions","organizations_url":"https://api.github.com/users/braydonf/orgs","repos_url":"https://api.github.com/users/braydonf/repos","events_url":"https://api.github.com/users/braydonf/events{/privacy}","received_events_url":"https://api.github.com/users/braydonf/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-08-25T00:15:38Z","updated_at":"2018-08-25T00:15:38Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"**Versions**\r\n\r\nbcoin v1.0.2\r\n\r\n**Description**\r\n\r\nHTTP Endpoints:\r\n- `/coin/address/:address`\r\n- `/tx/address/:address`\r\n\r\nUsing an address with thousands or millions of coins or transactions will overload these APIs and will not respond with a result _(didn't wait to know how long)_, and will max out the CPU for quiet awhile.\r\n\r\n**A Solution**\r\n\r\nTaking a look at the address index layout:\r\n```\r\n *  T[addr-hash][hash] -> dummy (tx by address)\r\n *  C[addr-hash][hash][index] -> dummy (coin by address)\r\n```\r\n\r\nThe txids are sorted lexicographic by the hash. For single address queries, it should be possible to introduce an `?after=<txid>&limit=<number>` query argument to be able to jump forward in the query in subsets. _Note:_ Sorting by hash may not be desirable, however that could be handled with additional indexes, or by adding `[count]` or `[height][index]` with the block height and transaction index to sort it in that order.\r\n\r\nThe exposed API only uses a single address, however the code is written to handle multiple addresses. It is problematic to optimize to query subsets with multiple addresses. A query is made to the address index to find matching txids. However the final order isn't known without querying all of the results for each address. To be able to have subsets, that order needs to be known, and be able to skip to various locations quickly. So the entire set of txids is queried each time, and then filling in only a portion of the txids. As this query is repeated often, it would be best if it were already organized on disk and memory, in that order. This optimization can't be made when there is near endless number of combinations of addresses that isn't known ahead of time. A wallet capable of making these optimizations.\r\n\r\nA solution could be to write the associated methods to only handle a single address and then be able to query subsets for larger results. And document to use a wallet API for any multiple addresses query. There may also be ways to use an address index internally to speed up wallet re-scans."}}
{"actor":{"display_login":"braydonf"}}