{"action":"created","comment":{"url":"https://api.github.com/repos/mimblewimble/grin/pulls/comments/215182492","pull_request_review_id":152390183,"id":215182492,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxNTE4MjQ5Mg==","diff_hunk":"@@ -236,17 +244,115 @@ fn check_header_known(bh: Hash, ctx: &mut BlockContext) -> Result<(), Error> {\n \n /// Quick in-memory check to fast-reject any block we've already handled\n /// recently. Keeps duplicates from the network in check.\n-fn check_known(bh: Hash, ctx: &mut BlockContext) -> Result<(), Error> {\n+fn check_known(header: &BlockHeader, ctx: &mut BlockContext) -> Result<(), Error> {\n+\tlet bh = header.hash();\n \tif bh == ctx.head.last_block_h || bh == ctx.head.prev_block_h {\n-\t\treturn Err(ErrorKind::Unfit(\"already known\".to_string()).into());\n+\t\treturn Err(ErrorKind::Unfit(\"already known in head\".to_string()).into());\n \t}\n+\n \tlet cache = ctx.block_hashes_cache.read().unwrap();\n \tif cache.contains(&bh) {\n \t\treturn Err(ErrorKind::Unfit(\"already known in cache\".to_string()).into());\n \t}\n-\tif ctx.orphans.contains(&bh) {\n-\t\treturn Err(ErrorKind::Unfit(\"already known in orphans\".to_string()).into());\n+\tOk(())\n+}\n+\n+fn check_known_orphans(header: &BlockHeader, ctx: &mut BlockContext) -> Result<(), Error> {\n+\tif ctx.orphans.contains(&header.hash()) {\n+\t\tErr(ErrorKind::Unfit(\"already known in orphans\".to_string()).into())\n+\t} else {\n+\t\tOk(())\n+\t}\n+}\n+\n+// Check if this block is in the store already.\n+fn check_known_store(header: &BlockHeader, ctx: &mut BlockContext) -> Result<(), Error> {","path":"chain/src/pipe.rs","position":132,"original_position":132,"commit_id":"cb725a0b2ad4187ea569033b68d5fd4dfb83b6ef","original_commit_id":"cb725a0b2ad4187ea569033b68d5fd4dfb83b6ef","user":{"login":"antiochp","id":30642645,"node_id":"MDQ6VXNlcjMwNjQyNjQ1","avatar_url":"https://avatars2.githubusercontent.com/u/30642645?v=4","gravatar_id":"","url":"https://api.github.com/users/antiochp","html_url":"https://github.com/antiochp","followers_url":"https://api.github.com/users/antiochp/followers","following_url":"https://api.github.com/users/antiochp/following{/other_user}","gists_url":"https://api.github.com/users/antiochp/gists{/gist_id}","starred_url":"https://api.github.com/users/antiochp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antiochp/subscriptions","organizations_url":"https://api.github.com/users/antiochp/orgs","repos_url":"https://api.github.com/users/antiochp/repos","events_url":"https://api.github.com/users/antiochp/events{/privacy}","received_events_url":"https://api.github.com/users/antiochp/received_events","type":"User","site_admin":false},"body":"Its the threshold for triggering compaction in `check_compact()` -\r\nhttps://github.com/mimblewimble/grin/blob/86aea875de1c4c6595fa1e1403048cea8134c71f/servers/src/common/adapters.rs#L545-L547\r\n","created_at":"2018-09-05T08:42:16Z","updated_at":"2018-09-05T08:42:16Z","html_url":"https://github.com/mimblewimble/grin/pull/1475#discussion_r215182492","pull_request_url":"https://api.github.com/repos/mimblewimble/grin/pulls/1475","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/mimblewimble/grin/pulls/comments/215182492"},"html":{"href":"https://github.com/mimblewimble/grin/pull/1475#discussion_r215182492"},"pull_request":{"href":"https://api.github.com/repos/mimblewimble/grin/pulls/1475"}},"in_reply_to_id":215113294},"pull_request":{"url":"https://api.github.com/repos/mimblewimble/grin/pulls/1475","id":213085894,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEzMDg1ODk0","html_url":"https://github.com/mimblewimble/grin/pull/1475","diff_url":"https://github.com/mimblewimble/grin/pull/1475.diff","patch_url":"https://github.com/mimblewimble/grin/pull/1475.patch","issue_url":"https://api.github.com/repos/mimblewimble/grin/issues/1475","number":1475,"state":"open","locked":false,"title":"improve \"check known\" steps during process_block()","user":{"login":"antiochp","id":30642645,"node_id":"MDQ6VXNlcjMwNjQyNjQ1","avatar_url":"https://avatars2.githubusercontent.com/u/30642645?v=4","gravatar_id":"","url":"https://api.github.com/users/antiochp","html_url":"https://github.com/antiochp","followers_url":"https://api.github.com/users/antiochp/followers","following_url":"https://api.github.com/users/antiochp/following{/other_user}","gists_url":"https://api.github.com/users/antiochp/gists{/gist_id}","starred_url":"https://api.github.com/users/antiochp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antiochp/subscriptions","organizations_url":"https://api.github.com/users/antiochp/orgs","repos_url":"https://api.github.com/users/antiochp/repos","events_url":"https://api.github.com/users/antiochp/events{/privacy}","received_events_url":"https://api.github.com/users/antiochp/received_events","type":"User","site_admin":false},"body":"Currently we do a fast in-memory check `check_known()` during `process_block`.\r\n\r\nThis checks block hashes against the 200 most recent blocks in `block_hash_cache`.\r\n\r\nWe can do an additional check against the full block index for the case where we see an \"old\" block that is beyond the scope of `block_hash_cache`.\r\nThis will be more expensive than the in-memory check but still considerably faster than full block validation on the duplicate block.\r\nThis PR introduces an additional `check_known_store()` to do this.\r\n\r\nFor even older blocks (on the main chain) there is an additional way we can know if we are processing a \"known\" block - we can look in the txhashset MMRs.\r\nRather than rewinding past the block and reapplying it we can rewind _to_ the block and check the MMR roots and sizes.\r\nThis PR adds a `check_known_mmr()` that accomplishes this.\r\n\r\nWe now have 3 steps during `process_block` where we check if the block being processed is already known (with increasing cost of doing the check) -\r\n\r\n    * check_known() # fast in-memory check for recent blocks (200 recent blocks)\r\n    * check_known_store() # slower (but still fast) lookup against full block index (2,000 recent blocks on fast sync node)\r\n    * check_known_mmr() # leverage our MMRs for old blocks on most work chain (limited to most work chain, no forks supported)\r\n\r\nIf a block is not \"known\" via any of these then we can proceed to validating the block and applying it to the txhashset.\r\n\r\n","created_at":"2018-09-04T19:54:43Z","updated_at":"2018-09-05T08:42:16Z","closed_at":null,"merged_at":null,"merge_commit_sha":"6dad23ff82a9402ab88f1c68fab2f13823d11449","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/mimblewimble/grin/pulls/1475/commits","review_comments_url":"https://api.github.com/repos/mimblewimble/grin/pulls/1475/comments","review_comment_url":"https://api.github.com/repos/mimblewimble/grin/pulls/comments{/number}","comments_url":"https://api.github.com/repos/mimblewimble/grin/issues/1475/comments","statuses_url":"https://api.github.com/repos/mimblewimble/grin/statuses/cb725a0b2ad4187ea569033b68d5fd4dfb83b6ef","head":{"label":"antiochp:fast_block_mmr_compare","ref":"fast_block_mmr_compare","sha":"cb725a0b2ad4187ea569033b68d5fd4dfb83b6ef","user":{"login":"antiochp","id":30642645,"node_id":"MDQ6VXNlcjMwNjQyNjQ1","avatar_url":"https://avatars2.githubusercontent.com/u/30642645?v=4","gravatar_id":"","url":"https://api.github.com/users/antiochp","html_url":"https://github.com/antiochp","followers_url":"https://api.github.com/users/antiochp/followers","following_url":"https://api.github.com/users/antiochp/following{/other_user}","gists_url":"https://api.github.com/users/antiochp/gists{/gist_id}","starred_url":"https://api.github.com/users/antiochp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antiochp/subscriptions","organizations_url":"https://api.github.com/users/antiochp/orgs","repos_url":"https://api.github.com/users/antiochp/repos","events_url":"https://api.github.com/users/antiochp/events{/privacy}","received_events_url":"https://api.github.com/users/antiochp/received_events","type":"User","site_admin":false},"repo":{"id":99061244,"node_id":"MDEwOlJlcG9zaXRvcnk5OTA2MTI0NA==","name":"grin","full_name":"antiochp/grin","owner":{"login":"antiochp","id":30642645,"node_id":"MDQ6VXNlcjMwNjQyNjQ1","avatar_url":"https://avatars2.githubusercontent.com/u/30642645?v=4","gravatar_id":"","url":"https://api.github.com/users/antiochp","html_url":"https://github.com/antiochp","followers_url":"https://api.github.com/users/antiochp/followers","following_url":"https://api.github.com/users/antiochp/following{/other_user}","gists_url":"https://api.github.com/users/antiochp/gists{/gist_id}","starred_url":"https://api.github.com/users/antiochp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/antiochp/subscriptions","organizations_url":"https://api.github.com/users/antiochp/orgs","repos_url":"https://api.github.com/users/antiochp/repos","events_url":"https://api.github.com/users/antiochp/events{/privacy}","received_events_url":"https://api.github.com/users/antiochp/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/antiochp/grin","description":"Minimal implementation of the MimbleWimble protocol.","fork":true,"url":"https://api.github.com/repos/antiochp/grin","forks_url":"https://api.github.com/repos/antiochp/grin/forks","keys_url":"https://api.github.com/repos/antiochp/grin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/antiochp/grin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/antiochp/grin/teams","hooks_url":"https://api.github.com/repos/antiochp/grin/hooks","issue_events_url":"https://api.github.com/repos/antiochp/grin/issues/events{/number}","events_url":"https://api.github.com/repos/antiochp/grin/events","assignees_url":"https://api.github.com/repos/antiochp/grin/assignees{/user}","branches_url":"https://api.github.com/repos/antiochp/grin/branches{/branch}","tags_url":"https://api.github.com/repos/antiochp/grin/tags","blobs_url":"https://api.github.com/repos/antiochp/grin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/antiochp/grin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/antiochp/grin/git/refs{/sha}","trees_url":"https://api.github.com/repos/antiochp/grin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/antiochp/grin/statuses/{sha}","languages_url":"https://api.github.com/repos/antiochp/grin/languages","stargazers_url":"https://api.github.com/repos/antiochp/grin/stargazers","contributors_url":"https://api.github.com/repos/antiochp/grin/contributors","subscribers_url":"https://api.github.com/repos/antiochp/grin/subscribers","subscription_url":"https://api.github.com/repos/antiochp/grin/subscription","commits_url":"https://api.github.com/repos/antiochp/grin/commits{/sha}","git_commits_url":"https://api.github.com/repos/antiochp/grin/git/commits{/sha}","comments_url":"https://api.github.com/repos/antiochp/grin/comments{/number}","issue_comment_url":"https://api.github.com/repos/antiochp/grin/issues/comments{/number}","contents_url":"https://api.github.com/repos/antiochp/grin/contents/{+path}","compare_url":"https://api.github.com/repos/antiochp/grin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/antiochp/grin/merges","archive_url":"https://api.github.com/repos/antiochp/grin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/antiochp/grin/downloads","issues_url":"https://api.github.com/repos/antiochp/grin/issues{/number}","pulls_url":"https://api.github.com/repos/antiochp/grin/pulls{/number}","milestones_url":"https://api.github.com/repos/antiochp/grin/milestones{/number}","notifications_url":"https://api.github.com/repos/antiochp/grin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/antiochp/grin/labels{/name}","releases_url":"https://api.github.com/repos/antiochp/grin/releases{/id}","deployments_url":"https://api.github.com/repos/antiochp/grin/deployments","created_at":"2017-08-02T02:04:29Z","updated_at":"2018-09-03T15:56:09Z","pushed_at":"2018-09-04T21:11:58Z","git_url":"git://github.com/antiochp/grin.git","ssh_url":"git@github.com:antiochp/grin.git","clone_url":"https://github.com/antiochp/grin.git","svn_url":"https://github.com/antiochp/grin","homepage":"","size":10738,"stargazers_count":0,"watchers_count":0,"language":"Rust","has_issues":false,"has_projects":false,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"mimblewimble:master","ref":"master","sha":"1e65467d7db989befbab1843c06615d339597d72","user":{"login":"mimblewimble","id":23646132,"node_id":"MDEyOk9yZ2FuaXphdGlvbjIzNjQ2MTMy","avatar_url":"https://avatars0.githubusercontent.com/u/23646132?v=4","gravatar_id":"","url":"https://api.github.com/users/mimblewimble","html_url":"https://github.com/mimblewimble","followers_url":"https://api.github.com/users/mimblewimble/followers","following_url":"https://api.github.com/users/mimblewimble/following{/other_user}","gists_url":"https://api.github.com/users/mimblewimble/gists{/gist_id}","starred_url":"https://api.github.com/users/mimblewimble/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mimblewimble/subscriptions","organizations_url":"https://api.github.com/users/mimblewimble/orgs","repos_url":"https://api.github.com/users/mimblewimble/repos","events_url":"https://api.github.com/users/mimblewimble/events{/privacy}","received_events_url":"https://api.github.com/users/mimblewimble/received_events","type":"Organization","site_admin":false},"repo":{"id":71514072,"node_id":"MDEwOlJlcG9zaXRvcnk3MTUxNDA3Mg==","name":"grin","full_name":"mimblewimble/grin","owner":{"login":"mimblewimble","id":23646132,"node_id":"MDEyOk9yZ2FuaXphdGlvbjIzNjQ2MTMy","avatar_url":"https://avatars0.githubusercontent.com/u/23646132?v=4","gravatar_id":"","url":"https://api.github.com/users/mimblewimble","html_url":"https://github.com/mimblewimble","followers_url":"https://api.github.com/users/mimblewimble/followers","following_url":"https://api.github.com/users/mimblewimble/following{/other_user}","gists_url":"https://api.github.com/users/mimblewimble/gists{/gist_id}","starred_url":"https://api.github.com/users/mimblewimble/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mimblewimble/subscriptions","organizations_url":"https://api.github.com/users/mimblewimble/orgs","repos_url":"https://api.github.com/users/mimblewimble/repos","events_url":"https://api.github.com/users/mimblewimble/events{/privacy}","received_events_url":"https://api.github.com/users/mimblewimble/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/mimblewimble/grin","description":"Minimal implementation of the MimbleWimble protocol.","fork":false,"url":"https://api.github.com/repos/mimblewimble/grin","forks_url":"https://api.github.com/repos/mimblewimble/grin/forks","keys_url":"https://api.github.com/repos/mimblewimble/grin/keys{/key_id}","collaborators_url":"https://api.github.com/repos/mimblewimble/grin/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/mimblewimble/grin/teams","hooks_url":"https://api.github.com/repos/mimblewimble/grin/hooks","issue_events_url":"https://api.github.com/repos/mimblewimble/grin/issues/events{/number}","events_url":"https://api.github.com/repos/mimblewimble/grin/events","assignees_url":"https://api.github.com/repos/mimblewimble/grin/assignees{/user}","branches_url":"https://api.github.com/repos/mimblewimble/grin/branches{/branch}","tags_url":"https://api.github.com/repos/mimblewimble/grin/tags","blobs_url":"https://api.github.com/repos/mimblewimble/grin/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/mimblewimble/grin/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/mimblewimble/grin/git/refs{/sha}","trees_url":"https://api.github.com/repos/mimblewimble/grin/git/trees{/sha}","statuses_url":"https://api.github.com/repos/mimblewimble/grin/statuses/{sha}","languages_url":"https://api.github.com/repos/mimblewimble/grin/languages","stargazers_url":"https://api.github.com/repos/mimblewimble/grin/stargazers","contributors_url":"https://api.github.com/repos/mimblewimble/grin/contributors","subscribers_url":"https://api.github.com/repos/mimblewimble/grin/subscribers","subscription_url":"https://api.github.com/repos/mimblewimble/grin/subscription","commits_url":"https://api.github.com/repos/mimblewimble/grin/commits{/sha}","git_commits_url":"https://api.github.com/repos/mimblewimble/grin/git/commits{/sha}","comments_url":"https://api.github.com/repos/mimblewimble/grin/comments{/number}","issue_comment_url":"https://api.github.com/repos/mimblewimble/grin/issues/comments{/number}","contents_url":"https://api.github.com/repos/mimblewimble/grin/contents/{+path}","compare_url":"https://api.github.com/repos/mimblewimble/grin/compare/{base}...{head}","merges_url":"https://api.github.com/repos/mimblewimble/grin/merges","archive_url":"https://api.github.com/repos/mimblewimble/grin/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/mimblewimble/grin/downloads","issues_url":"https://api.github.com/repos/mimblewimble/grin/issues{/number}","pulls_url":"https://api.github.com/repos/mimblewimble/grin/pulls{/number}","milestones_url":"https://api.github.com/repos/mimblewimble/grin/milestones{/number}","notifications_url":"https://api.github.com/repos/mimblewimble/grin/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/mimblewimble/grin/labels{/name}","releases_url":"https://api.github.com/repos/mimblewimble/grin/releases{/id}","deployments_url":"https://api.github.com/repos/mimblewimble/grin/deployments","created_at":"2016-10-21T00:02:23Z","updated_at":"2018-09-05T06:35:14Z","pushed_at":"2018-09-05T02:07:48Z","git_url":"git://github.com/mimblewimble/grin.git","ssh_url":"git@github.com:mimblewimble/grin.git","clone_url":"https://github.com/mimblewimble/grin.git","svn_url":"https://github.com/mimblewimble/grin","homepage":"http://grin-tech.org/","size":10582,"stargazers_count":1335,"watchers_count":1335,"language":"Rust","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":245,"mirror_url":null,"archived":false,"open_issues_count":72,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":245,"open_issues":72,"watchers":1335,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/mimblewimble/grin/pulls/1475"},"html":{"href":"https://github.com/mimblewimble/grin/pull/1475"},"issue":{"href":"https://api.github.com/repos/mimblewimble/grin/issues/1475"},"comments":{"href":"https://api.github.com/repos/mimblewimble/grin/issues/1475/comments"},"review_comments":{"href":"https://api.github.com/repos/mimblewimble/grin/pulls/1475/comments"},"review_comment":{"href":"https://api.github.com/repos/mimblewimble/grin/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/mimblewimble/grin/pulls/1475/commits"},"statuses":{"href":"https://api.github.com/repos/mimblewimble/grin/statuses/cb725a0b2ad4187ea569033b68d5fd4dfb83b6ef"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"antiochp"}}