{"action":"closed","number":7759,"pull_request":{"url":"https://api.github.com/repos/bitcoin/bitcoin/pulls/7759","id":64374655,"html_url":"https://github.com/bitcoin/bitcoin/pull/7759","diff_url":"https://github.com/bitcoin/bitcoin/pull/7759.diff","patch_url":"https://github.com/bitcoin/bitcoin/pull/7759.patch","issue_url":"https://api.github.com/repos/bitcoin/bitcoin/issues/7759","number":7759,"state":"closed","locked":false,"title":"[WIP] rest: Stream entire utxo set","user":{"login":"laanwj","id":126646,"avatar_url":"https://avatars.githubusercontent.com/u/126646?v=3","gravatar_id":"","url":"https://api.github.com/users/laanwj","html_url":"https://github.com/laanwj","followers_url":"https://api.github.com/users/laanwj/followers","following_url":"https://api.github.com/users/laanwj/following{/other_user}","gists_url":"https://api.github.com/users/laanwj/gists{/gist_id}","starred_url":"https://api.github.com/users/laanwj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laanwj/subscriptions","organizations_url":"https://api.github.com/users/laanwj/orgs","repos_url":"https://api.github.com/users/laanwj/repos","events_url":"https://api.github.com/users/laanwj/events{/privacy}","received_events_url":"https://api.github.com/users/laanwj/received_events","type":"User","site_admin":false},"body":"This is by no means ready, but anyhow this builds on #7756 and\r\n\r\n- Adds a streaming API to the HTTP server. This allows streaming data to the client chunk by chunk, which is useful when not the entire data is available at once or it is huge and wouldn't fit (efficiently) in memory.\r\n\r\n- Allows downloading the entire UTXO set through `/rest/utxoset`. This is a raw dump of all outputs, the state normally hashed by `gettxoutsetinfo`. The dump is performed in the background by making use of leveldb snapshotting, so without keeping cs_main locked.\r\n    - This can be useful for analysis purposes if you don't want to mess with bitcoin core's database\r\n    - Filename (via content-disposition) is `utxoset-<height>-<bestblockhash>.dat`. Also a custom `X-Best-Block` and `X-Block-Height` header is added.\r\n\r\nIt matches:\r\n```bash\r\n$ src/bitcoin-cli -datadir=/store/tmp/testbtc gettxoutsetinfo\r\n{\r\n...\r\n  \"hash_serialized\": \"5017f82bbb82a8199ae0fbaa9e5881a0c82575db89e6edd5b39414b35299363b\",\r\n...\r\n}\r\n$ wget --content-disposition http://127.0.0.1:8332/rest/utxoset \r\n2016-03-28 22:58:32 (44.3 MB/s) - ‘utxoset-404681-0000000000000000034854f5a3ab27cfbc220a42c75061dd13d2067cda71191d.dat’ saved [1291578967]\r\n$ ~/bin/dsha256 utxoset-404681-0000000000000000034854f5a3ab27cfbc220a42c75061dd13d2067cda71191d.dat\r\n5017f82bbb82a8199ae0fbaa9e5881a0c82575db89e6edd5b39414b35299363b utxoset\r\n```\r\n\r\n## TODO\r\n- ~~Rebase after #7756 merged~~\r\n- Sensibly name and split up commits\r\n- Clean up and split up code\r\n- Actually handle errors (you can crash a worker thread right now by disconnecting while downloading)\r\n- The timeout actually runs *while* downloading, causing it to break off after downloading. I don't understand why this is. You can work around it with `-rpcservertimeout=6000` or such.\r\n- ~~UTXO set dump doesn't contain keys (?) I'm not sure this format is actually useful this way (see #7758)~~ (fixed in #7848)\r\n- Other formats, potentially ^^\r\n\r\nNote that the HTTP streaming API could in principle also be used for other large data (say, wallet backups), or even for websocket-like event notification.\r\n","created_at":"2016-03-28T20:55:18Z","updated_at":"2016-10-18T19:14:48Z","closed_at":"2016-10-18T19:14:48Z","merged_at":null,"merge_commit_sha":"3a000fbf882fabe59c28a7902984a7416068b5c4","assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/bitcoin/bitcoin/milestones/16","html_url":"https://github.com/bitcoin/bitcoin/milestones/Future","labels_url":"https://api.github.com/repos/bitcoin/bitcoin/milestones/16/labels","id":710147,"number":16,"title":"Future","description":"Undefined future version","creator":{"login":"laanwj","id":126646,"avatar_url":"https://avatars.githubusercontent.com/u/126646?v=3","gravatar_id":"","url":"https://api.github.com/users/laanwj","html_url":"https://github.com/laanwj","followers_url":"https://api.github.com/users/laanwj/followers","following_url":"https://api.github.com/users/laanwj/following{/other_user}","gists_url":"https://api.github.com/users/laanwj/gists{/gist_id}","starred_url":"https://api.github.com/users/laanwj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laanwj/subscriptions","organizations_url":"https://api.github.com/users/laanwj/orgs","repos_url":"https://api.github.com/users/laanwj/repos","events_url":"https://api.github.com/users/laanwj/events{/privacy}","received_events_url":"https://api.github.com/users/laanwj/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":3,"state":"open","created_at":"2014-07-04T06:53:52Z","updated_at":"2016-10-18T19:14:48Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/bitcoin/bitcoin/pulls/7759/commits","review_comments_url":"https://api.github.com/repos/bitcoin/bitcoin/pulls/7759/comments","review_comment_url":"https://api.github.com/repos/bitcoin/bitcoin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/bitcoin/bitcoin/issues/7759/comments","statuses_url":"https://api.github.com/repos/bitcoin/bitcoin/statuses/5cd59bbe03d7b4763155acc76e16867f2c66dc7c","head":{"label":"laanwj:2016_03_utxo_streaming","ref":"2016_03_utxo_streaming","sha":"5cd59bbe03d7b4763155acc76e16867f2c66dc7c","user":{"login":"laanwj","id":126646,"avatar_url":"https://avatars.githubusercontent.com/u/126646?v=3","gravatar_id":"","url":"https://api.github.com/users/laanwj","html_url":"https://github.com/laanwj","followers_url":"https://api.github.com/users/laanwj/followers","following_url":"https://api.github.com/users/laanwj/following{/other_user}","gists_url":"https://api.github.com/users/laanwj/gists{/gist_id}","starred_url":"https://api.github.com/users/laanwj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laanwj/subscriptions","organizations_url":"https://api.github.com/users/laanwj/orgs","repos_url":"https://api.github.com/users/laanwj/repos","events_url":"https://api.github.com/users/laanwj/events{/privacy}","received_events_url":"https://api.github.com/users/laanwj/received_events","type":"User","site_admin":false},"repo":{"id":1750308,"name":"bitcoin","full_name":"laanwj/bitcoin","owner":{"login":"laanwj","id":126646,"avatar_url":"https://avatars.githubusercontent.com/u/126646?v=3","gravatar_id":"","url":"https://api.github.com/users/laanwj","html_url":"https://github.com/laanwj","followers_url":"https://api.github.com/users/laanwj/followers","following_url":"https://api.github.com/users/laanwj/following{/other_user}","gists_url":"https://api.github.com/users/laanwj/gists{/gist_id}","starred_url":"https://api.github.com/users/laanwj/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/laanwj/subscriptions","organizations_url":"https://api.github.com/users/laanwj/orgs","repos_url":"https://api.github.com/users/laanwj/repos","events_url":"https://api.github.com/users/laanwj/events{/privacy}","received_events_url":"https://api.github.com/users/laanwj/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/laanwj/bitcoin","description":"Laanwj's Bitcoin Core fork - see README.md on master branch","fork":true,"url":"https://api.github.com/repos/laanwj/bitcoin","forks_url":"https://api.github.com/repos/laanwj/bitcoin/forks","keys_url":"https://api.github.com/repos/laanwj/bitcoin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/laanwj/bitcoin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/laanwj/bitcoin/teams","hooks_url":"https://api.github.com/repos/laanwj/bitcoin/hooks","issue_events_url":"https://api.github.com/repos/laanwj/bitcoin/issues/events{/number}","events_url":"https://api.github.com/repos/laanwj/bitcoin/events","assignees_url":"https://api.github.com/repos/laanwj/bitcoin/assignees{/user}","branches_url":"https://api.github.com/repos/laanwj/bitcoin/branches{/branch}","tags_url":"https://api.github.com/repos/laanwj/bitcoin/tags","blobs_url":"https://api.github.com/repos/laanwj/bitcoin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/laanwj/bitcoin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/laanwj/bitcoin/git/refs{/sha}","trees_url":"https://api.github.com/repos/laanwj/bitcoin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/laanwj/bitcoin/statuses/{sha}","languages_url":"https://api.github.com/repos/laanwj/bitcoin/languages","stargazers_url":"https://api.github.com/repos/laanwj/bitcoin/stargazers","contributors_url":"https://api.github.com/repos/laanwj/bitcoin/contributors","subscribers_url":"https://api.github.com/repos/laanwj/bitcoin/subscribers","subscription_url":"https://api.github.com/repos/laanwj/bitcoin/subscription","commits_url":"https://api.github.com/repos/laanwj/bitcoin/commits{/sha}","git_commits_url":"https://api.github.com/repos/laanwj/bitcoin/git/commits{/sha}","comments_url":"https://api.github.com/repos/laanwj/bitcoin/comments{/number}","issue_comment_url":"https://api.github.com/repos/laanwj/bitcoin/issues/comments{/number}","contents_url":"https://api.github.com/repos/laanwj/bitcoin/contents/{+path}","compare_url":"https://api.github.com/repos/laanwj/bitcoin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/laanwj/bitcoin/merges","archive_url":"https://api.github.com/repos/laanwj/bitcoin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/laanwj/bitcoin/downloads","issues_url":"https://api.github.com/repos/laanwj/bitcoin/issues{/number}","pulls_url":"https://api.github.com/repos/laanwj/bitcoin/pulls{/number}","milestones_url":"https://api.github.com/repos/laanwj/bitcoin/milestones{/number}","notifications_url":"https://api.github.com/repos/laanwj/bitcoin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/laanwj/bitcoin/labels{/name}","releases_url":"https://api.github.com/repos/laanwj/bitcoin/releases{/id}","deployments_url":"https://api.github.com/repos/laanwj/bitcoin/deployments","created_at":"2011-05-15T07:09:54Z","updated_at":"2016-10-10T13:21:36Z","pushed_at":"2016-10-18T15:49:48Z","git_url":"git://github.com/laanwj/bitcoin.git","ssh_url":"git@github.com:laanwj/bitcoin.git","clone_url":"https://github.com/laanwj/bitcoin.git","svn_url":"https://github.com/laanwj/bitcoin","homepage":"","size":66576,"stargazers_count":10,"watchers_count":10,"language":null,"has_issues":false,"has_downloads":false,"has_wiki":true,"has_pages":false,"forks_count":4,"mirror_url":null,"open_issues_count":0,"forks":4,"open_issues":0,"watchers":10,"default_branch":"master"}},"base":{"label":"bitcoin:master","ref":"master","sha":"d7615af34e8e19920ed12bfdafb09e0e4b57c7c5","user":{"login":"bitcoin","id":528860,"avatar_url":"https://avatars.githubusercontent.com/u/528860?v=3","gravatar_id":"","url":"https://api.github.com/users/bitcoin","html_url":"https://github.com/bitcoin","followers_url":"https://api.github.com/users/bitcoin/followers","following_url":"https://api.github.com/users/bitcoin/following{/other_user}","gists_url":"https://api.github.com/users/bitcoin/gists{/gist_id}","starred_url":"https://api.github.com/users/bitcoin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bitcoin/subscriptions","organizations_url":"https://api.github.com/users/bitcoin/orgs","repos_url":"https://api.github.com/users/bitcoin/repos","events_url":"https://api.github.com/users/bitcoin/events{/privacy}","received_events_url":"https://api.github.com/users/bitcoin/received_events","type":"Organization","site_admin":false},"repo":{"id":1181927,"name":"bitcoin","full_name":"bitcoin/bitcoin","owner":{"login":"bitcoin","id":528860,"avatar_url":"https://avatars.githubusercontent.com/u/528860?v=3","gravatar_id":"","url":"https://api.github.com/users/bitcoin","html_url":"https://github.com/bitcoin","followers_url":"https://api.github.com/users/bitcoin/followers","following_url":"https://api.github.com/users/bitcoin/following{/other_user}","gists_url":"https://api.github.com/users/bitcoin/gists{/gist_id}","starred_url":"https://api.github.com/users/bitcoin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bitcoin/subscriptions","organizations_url":"https://api.github.com/users/bitcoin/orgs","repos_url":"https://api.github.com/users/bitcoin/repos","events_url":"https://api.github.com/users/bitcoin/events{/privacy}","received_events_url":"https://api.github.com/users/bitcoin/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/bitcoin/bitcoin","description":"Bitcoin Core integration/staging tree","fork":false,"url":"https://api.github.com/repos/bitcoin/bitcoin","forks_url":"https://api.github.com/repos/bitcoin/bitcoin/forks","keys_url":"https://api.github.com/repos/bitcoin/bitcoin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/bitcoin/bitcoin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/bitcoin/bitcoin/teams","hooks_url":"https://api.github.com/repos/bitcoin/bitcoin/hooks","issue_events_url":"https://api.github.com/repos/bitcoin/bitcoin/issues/events{/number}","events_url":"https://api.github.com/repos/bitcoin/bitcoin/events","assignees_url":"https://api.github.com/repos/bitcoin/bitcoin/assignees{/user}","branches_url":"https://api.github.com/repos/bitcoin/bitcoin/branches{/branch}","tags_url":"https://api.github.com/repos/bitcoin/bitcoin/tags","blobs_url":"https://api.github.com/repos/bitcoin/bitcoin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/bitcoin/bitcoin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/bitcoin/bitcoin/git/refs{/sha}","trees_url":"https://api.github.com/repos/bitcoin/bitcoin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/bitcoin/bitcoin/statuses/{sha}","languages_url":"https://api.github.com/repos/bitcoin/bitcoin/languages","stargazers_url":"https://api.github.com/repos/bitcoin/bitcoin/stargazers","contributors_url":"https://api.github.com/repos/bitcoin/bitcoin/contributors","subscribers_url":"https://api.github.com/repos/bitcoin/bitcoin/subscribers","subscription_url":"https://api.github.com/repos/bitcoin/bitcoin/subscription","commits_url":"https://api.github.com/repos/bitcoin/bitcoin/commits{/sha}","git_commits_url":"https://api.github.com/repos/bitcoin/bitcoin/git/commits{/sha}","comments_url":"https://api.github.com/repos/bitcoin/bitcoin/comments{/number}","issue_comment_url":"https://api.github.com/repos/bitcoin/bitcoin/issues/comments{/number}","contents_url":"https://api.github.com/repos/bitcoin/bitcoin/contents/{+path}","compare_url":"https://api.github.com/repos/bitcoin/bitcoin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/bitcoin/bitcoin/merges","archive_url":"https://api.github.com/repos/bitcoin/bitcoin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/bitcoin/bitcoin/downloads","issues_url":"https://api.github.com/repos/bitcoin/bitcoin/issues{/number}","pulls_url":"https://api.github.com/repos/bitcoin/bitcoin/pulls{/number}","milestones_url":"https://api.github.com/repos/bitcoin/bitcoin/milestones{/number}","notifications_url":"https://api.github.com/repos/bitcoin/bitcoin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/bitcoin/bitcoin/labels{/name}","releases_url":"https://api.github.com/repos/bitcoin/bitcoin/releases{/id}","deployments_url":"https://api.github.com/repos/bitcoin/bitcoin/deployments","created_at":"2010-12-19T15:16:43Z","updated_at":"2016-10-18T16:30:05Z","pushed_at":"2016-10-18T19:05:18Z","git_url":"git://github.com/bitcoin/bitcoin.git","ssh_url":"git@github.com:bitcoin/bitcoin.git","clone_url":"https://github.com/bitcoin/bitcoin.git","svn_url":"https://github.com/bitcoin/bitcoin","homepage":"https://bitcoin.org/en/download","size":157225,"stargazers_count":10172,"watchers_count":10172,"language":"C++","has_issues":true,"has_downloads":false,"has_wiki":false,"has_pages":false,"forks_count":6748,"mirror_url":null,"open_issues_count":525,"forks":6748,"open_issues":525,"watchers":10172,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/bitcoin/bitcoin/pulls/7759"},"html":{"href":"https://github.com/bitcoin/bitcoin/pull/7759"},"issue":{"href":"https://api.github.com/repos/bitcoin/bitcoin/issues/7759"},"comments":{"href":"https://api.github.com/repos/bitcoin/bitcoin/issues/7759/comments"},"review_comments":{"href":"https://api.github.com/repos/bitcoin/bitcoin/pulls/7759/comments"},"review_comment":{"href":"https://api.github.com/repos/bitcoin/bitcoin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/bitcoin/bitcoin/pulls/7759/commits"},"statuses":{"href":"https://api.github.com/repos/bitcoin/bitcoin/statuses/5cd59bbe03d7b4763155acc76e16867f2c66dc7c"}},"merged":false,"mergeable":true,"mergeable_state":"clean","merged_by":null,"comments":6,"review_comments":6,"commits":2,"additions":204,"deletions":2,"changed_files":3}}
{"actor":{"display_login":"laanwj"}}