{"action":"created","comment":{"url":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/comments/191441880","pull_request_review_id":124013274,"id":191441880,"diff_hunk":"@@ -7,30 +7,32 @@ defmodule Explorer.Indexer.BlockFetcher do\n \n   require Logger\n \n+  import Explorer.Indexer, only: [debug: 1]\n+\n   alias EthereumJSONRPC\n   alias EthereumJSONRPC.Transactions\n   alias Explorer.{Chain, Indexer}\n-  alias Explorer.Indexer.{AddressFetcher, Sequence}\n+\n+  alias Explorer.Indexer.{\n+    AddressFetcher,\n+    Sequence,\n+    InternalTransactionFetcher\n+  }\n \n   # dialyzer thinks that Logger.debug functions always have no_local_return\n   @dialyzer {:nowarn_function, import_range: 3}\n \n   # These are all the *default* values for options.\n   # DO NOT use them directly in the code.  Get options from `state`.\n \n-  @debug_logs false\n-\n   @blocks_batch_size 10\n   @blocks_concurrency 10\n \n-  @internal_transactions_batch_size 50\n-  @internal_transactions_concurrency 8\n-\n   # milliseconds\n   @block_rate 5_000\n \n   @receipts_batch_size 250\n-  @receipts_concurrency 20\n+  @receipts_concurrency 10","path":"apps/explorer/lib/explorer/indexer/block_fetcher.ex","position":36,"original_position":36,"commit_id":"7d77af44b7caada96065719a5bf09949b55129be","original_commit_id":"7d77af44b7caada96065719a5bf09949b55129be","user":{"login":"josevalim","id":9582,"avatar_url":"https://avatars0.githubusercontent.com/u/9582?v=4","gravatar_id":"","url":"https://api.github.com/users/josevalim","html_url":"https://github.com/josevalim","followers_url":"https://api.github.com/users/josevalim/followers","following_url":"https://api.github.com/users/josevalim/following{/other_user}","gists_url":"https://api.github.com/users/josevalim/gists{/gist_id}","starred_url":"https://api.github.com/users/josevalim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josevalim/subscriptions","organizations_url":"https://api.github.com/users/josevalim/orgs","repos_url":"https://api.github.com/users/josevalim/repos","events_url":"https://api.github.com/users/josevalim/events{/privacy}","received_events_url":"https://api.github.com/users/josevalim/received_events","type":"User","site_admin":false},"body":"Should we compute this at runtime based on the number of schedulers online? See `System.schedulers_online`.","created_at":"2018-05-29T14:18:24Z","updated_at":"2018-05-29T14:18:25Z","html_url":"https://github.com/poanetwork/poa-explorer/pull/218#discussion_r191441880","pull_request_url":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/218","author_association":"NONE","_links":{"self":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/comments/191441880"},"html":{"href":"https://github.com/poanetwork/poa-explorer/pull/218#discussion_r191441880"},"pull_request":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/218"}},"in_reply_to_id":191437098},"pull_request":{"url":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/218","id":190689514,"html_url":"https://github.com/poanetwork/poa-explorer/pull/218","diff_url":"https://github.com/poanetwork/poa-explorer/pull/218.diff","patch_url":"https://github.com/poanetwork/poa-explorer/pull/218.patch","issue_url":"https://api.github.com/repos/poanetwork/poa-explorer/issues/218","number":218,"state":"open","locked":false,"title":"Defer internal transaction fetching with buffered task queue","user":{"login":"chrismccord","id":576796,"avatar_url":"https://avatars3.githubusercontent.com/u/576796?v=4","gravatar_id":"","url":"https://api.github.com/users/chrismccord","html_url":"https://github.com/chrismccord","followers_url":"https://api.github.com/users/chrismccord/followers","following_url":"https://api.github.com/users/chrismccord/following{/other_user}","gists_url":"https://api.github.com/users/chrismccord/gists{/gist_id}","starred_url":"https://api.github.com/users/chrismccord/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chrismccord/subscriptions","organizations_url":"https://api.github.com/users/chrismccord/orgs","repos_url":"https://api.github.com/users/chrismccord/repos","events_url":"https://api.github.com/users/chrismccord/events{/privacy}","received_events_url":"https://api.github.com/users/chrismccord/received_events","type":"User","site_admin":false},"body":"Moves internal transaction fetching out of block fetcher into asynchronous bounded tasks. The `BufferedTask` behaviour was introduced to allow batched tasks to be run with a configured max_concurrency, with both programmatic and automatic retries should a task fail.  Tasks can also populate their initial buffer on startup via `init/2`, which allows catch-up of missed data while on-demand entries are buffered. This allows batched processing of entries in a bounded way without saturating all cpu/io/upstream bandwidth when fetching internal transactions, as well as reliable retry/catchup support without the need of an external store. The behaviour is also general purpose and the `AddressFetcher` was rewritten to make use of it. \r\n\r\n*Note*: @KronicDeth will be taking over the PR and resolving the upstream conflicts around `Chain.Address` and `geth` support. Thanks!","created_at":"2018-05-25T20:40:27Z","updated_at":"2018-05-29T14:18:25Z","closed_at":null,"merged_at":null,"merge_commit_sha":null,"assignee":{"login":"KronicDeth","id":298259,"avatar_url":"https://avatars3.githubusercontent.com/u/298259?v=4","gravatar_id":"","url":"https://api.github.com/users/KronicDeth","html_url":"https://github.com/KronicDeth","followers_url":"https://api.github.com/users/KronicDeth/followers","following_url":"https://api.github.com/users/KronicDeth/following{/other_user}","gists_url":"https://api.github.com/users/KronicDeth/gists{/gist_id}","starred_url":"https://api.github.com/users/KronicDeth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KronicDeth/subscriptions","organizations_url":"https://api.github.com/users/KronicDeth/orgs","repos_url":"https://api.github.com/users/KronicDeth/repos","events_url":"https://api.github.com/users/KronicDeth/events{/privacy}","received_events_url":"https://api.github.com/users/KronicDeth/received_events","type":"User","site_admin":false},"assignees":[{"login":"KronicDeth","id":298259,"avatar_url":"https://avatars3.githubusercontent.com/u/298259?v=4","gravatar_id":"","url":"https://api.github.com/users/KronicDeth","html_url":"https://github.com/KronicDeth","followers_url":"https://api.github.com/users/KronicDeth/followers","following_url":"https://api.github.com/users/KronicDeth/following{/other_user}","gists_url":"https://api.github.com/users/KronicDeth/gists{/gist_id}","starred_url":"https://api.github.com/users/KronicDeth/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/KronicDeth/subscriptions","organizations_url":"https://api.github.com/users/KronicDeth/orgs","repos_url":"https://api.github.com/users/KronicDeth/repos","events_url":"https://api.github.com/users/KronicDeth/events{/privacy}","received_events_url":"https://api.github.com/users/KronicDeth/received_events","type":"User","site_admin":false}],"requested_reviewers":[{"login":"alexgaribay","id":911605,"avatar_url":"https://avatars2.githubusercontent.com/u/911605?v=4","gravatar_id":"","url":"https://api.github.com/users/alexgaribay","html_url":"https://github.com/alexgaribay","followers_url":"https://api.github.com/users/alexgaribay/followers","following_url":"https://api.github.com/users/alexgaribay/following{/other_user}","gists_url":"https://api.github.com/users/alexgaribay/gists{/gist_id}","starred_url":"https://api.github.com/users/alexgaribay/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/alexgaribay/subscriptions","organizations_url":"https://api.github.com/users/alexgaribay/orgs","repos_url":"https://api.github.com/users/alexgaribay/repos","events_url":"https://api.github.com/users/alexgaribay/events{/privacy}","received_events_url":"https://api.github.com/users/alexgaribay/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[{"id":859573185,"url":"https://api.github.com/repos/poanetwork/poa-explorer/labels/in%20progress","name":"in progress","color":"ededed","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/218/commits","review_comments_url":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/218/comments","review_comment_url":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/comments{/number}","comments_url":"https://api.github.com/repos/poanetwork/poa-explorer/issues/218/comments","statuses_url":"https://api.github.com/repos/poanetwork/poa-explorer/statuses/7d77af44b7caada96065719a5bf09949b55129be","head":{"label":"poanetwork:cm-defer-internal-fetch","ref":"cm-defer-internal-fetch","sha":"7d77af44b7caada96065719a5bf09949b55129be","user":{"login":"poanetwork","id":28943015,"avatar_url":"https://avatars2.githubusercontent.com/u/28943015?v=4","gravatar_id":"","url":"https://api.github.com/users/poanetwork","html_url":"https://github.com/poanetwork","followers_url":"https://api.github.com/users/poanetwork/followers","following_url":"https://api.github.com/users/poanetwork/following{/other_user}","gists_url":"https://api.github.com/users/poanetwork/gists{/gist_id}","starred_url":"https://api.github.com/users/poanetwork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/poanetwork/subscriptions","organizations_url":"https://api.github.com/users/poanetwork/orgs","repos_url":"https://api.github.com/users/poanetwork/repos","events_url":"https://api.github.com/users/poanetwork/events{/privacy}","received_events_url":"https://api.github.com/users/poanetwork/received_events","type":"Organization","site_admin":false},"repo":{"id":117294980,"name":"poa-explorer","full_name":"poanetwork/poa-explorer","owner":{"login":"poanetwork","id":28943015,"avatar_url":"https://avatars2.githubusercontent.com/u/28943015?v=4","gravatar_id":"","url":"https://api.github.com/users/poanetwork","html_url":"https://github.com/poanetwork","followers_url":"https://api.github.com/users/poanetwork/followers","following_url":"https://api.github.com/users/poanetwork/following{/other_user}","gists_url":"https://api.github.com/users/poanetwork/gists{/gist_id}","starred_url":"https://api.github.com/users/poanetwork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/poanetwork/subscriptions","organizations_url":"https://api.github.com/users/poanetwork/orgs","repos_url":"https://api.github.com/users/poanetwork/repos","events_url":"https://api.github.com/users/poanetwork/events{/privacy}","received_events_url":"https://api.github.com/users/poanetwork/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/poanetwork/poa-explorer","description":"A tool for inspecting and analyzing EVM based blockchains. Blockchain explorer for Ethereum networks.","fork":false,"url":"https://api.github.com/repos/poanetwork/poa-explorer","forks_url":"https://api.github.com/repos/poanetwork/poa-explorer/forks","keys_url":"https://api.github.com/repos/poanetwork/poa-explorer/keys{/key_id}","collaborators_url":"https://api.github.com/repos/poanetwork/poa-explorer/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/poanetwork/poa-explorer/teams","hooks_url":"https://api.github.com/repos/poanetwork/poa-explorer/hooks","issue_events_url":"https://api.github.com/repos/poanetwork/poa-explorer/issues/events{/number}","events_url":"https://api.github.com/repos/poanetwork/poa-explorer/events","assignees_url":"https://api.github.com/repos/poanetwork/poa-explorer/assignees{/user}","branches_url":"https://api.github.com/repos/poanetwork/poa-explorer/branches{/branch}","tags_url":"https://api.github.com/repos/poanetwork/poa-explorer/tags","blobs_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/refs{/sha}","trees_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/trees{/sha}","statuses_url":"https://api.github.com/repos/poanetwork/poa-explorer/statuses/{sha}","languages_url":"https://api.github.com/repos/poanetwork/poa-explorer/languages","stargazers_url":"https://api.github.com/repos/poanetwork/poa-explorer/stargazers","contributors_url":"https://api.github.com/repos/poanetwork/poa-explorer/contributors","subscribers_url":"https://api.github.com/repos/poanetwork/poa-explorer/subscribers","subscription_url":"https://api.github.com/repos/poanetwork/poa-explorer/subscription","commits_url":"https://api.github.com/repos/poanetwork/poa-explorer/commits{/sha}","git_commits_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/commits{/sha}","comments_url":"https://api.github.com/repos/poanetwork/poa-explorer/comments{/number}","issue_comment_url":"https://api.github.com/repos/poanetwork/poa-explorer/issues/comments{/number}","contents_url":"https://api.github.com/repos/poanetwork/poa-explorer/contents/{+path}","compare_url":"https://api.github.com/repos/poanetwork/poa-explorer/compare/{base}...{head}","merges_url":"https://api.github.com/repos/poanetwork/poa-explorer/merges","archive_url":"https://api.github.com/repos/poanetwork/poa-explorer/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/poanetwork/poa-explorer/downloads","issues_url":"https://api.github.com/repos/poanetwork/poa-explorer/issues{/number}","pulls_url":"https://api.github.com/repos/poanetwork/poa-explorer/pulls{/number}","milestones_url":"https://api.github.com/repos/poanetwork/poa-explorer/milestones{/number}","notifications_url":"https://api.github.com/repos/poanetwork/poa-explorer/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/poanetwork/poa-explorer/labels{/name}","releases_url":"https://api.github.com/repos/poanetwork/poa-explorer/releases{/id}","deployments_url":"https://api.github.com/repos/poanetwork/poa-explorer/deployments","created_at":"2018-01-12T22:45:24Z","updated_at":"2018-05-29T08:29:12Z","pushed_at":"2018-05-29T13:44:47Z","git_url":"git://github.com/poanetwork/poa-explorer.git","ssh_url":"git@github.com:poanetwork/poa-explorer.git","clone_url":"https://github.com/poanetwork/poa-explorer.git","svn_url":"https://github.com/poanetwork/poa-explorer","homepage":"","size":1837,"stargazers_count":43,"watchers_count":43,"language":"Elixir","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":18,"mirror_url":null,"archived":false,"open_issues_count":86,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0"},"forks":18,"open_issues":86,"watchers":43,"default_branch":"master"}},"base":{"label":"poanetwork:master","ref":"master","sha":"27ba115974a5f71ae56af43943c333c1f7c84b83","user":{"login":"poanetwork","id":28943015,"avatar_url":"https://avatars2.githubusercontent.com/u/28943015?v=4","gravatar_id":"","url":"https://api.github.com/users/poanetwork","html_url":"https://github.com/poanetwork","followers_url":"https://api.github.com/users/poanetwork/followers","following_url":"https://api.github.com/users/poanetwork/following{/other_user}","gists_url":"https://api.github.com/users/poanetwork/gists{/gist_id}","starred_url":"https://api.github.com/users/poanetwork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/poanetwork/subscriptions","organizations_url":"https://api.github.com/users/poanetwork/orgs","repos_url":"https://api.github.com/users/poanetwork/repos","events_url":"https://api.github.com/users/poanetwork/events{/privacy}","received_events_url":"https://api.github.com/users/poanetwork/received_events","type":"Organization","site_admin":false},"repo":{"id":117294980,"name":"poa-explorer","full_name":"poanetwork/poa-explorer","owner":{"login":"poanetwork","id":28943015,"avatar_url":"https://avatars2.githubusercontent.com/u/28943015?v=4","gravatar_id":"","url":"https://api.github.com/users/poanetwork","html_url":"https://github.com/poanetwork","followers_url":"https://api.github.com/users/poanetwork/followers","following_url":"https://api.github.com/users/poanetwork/following{/other_user}","gists_url":"https://api.github.com/users/poanetwork/gists{/gist_id}","starred_url":"https://api.github.com/users/poanetwork/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/poanetwork/subscriptions","organizations_url":"https://api.github.com/users/poanetwork/orgs","repos_url":"https://api.github.com/users/poanetwork/repos","events_url":"https://api.github.com/users/poanetwork/events{/privacy}","received_events_url":"https://api.github.com/users/poanetwork/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/poanetwork/poa-explorer","description":"A tool for inspecting and analyzing EVM based blockchains. Blockchain explorer for Ethereum networks.","fork":false,"url":"https://api.github.com/repos/poanetwork/poa-explorer","forks_url":"https://api.github.com/repos/poanetwork/poa-explorer/forks","keys_url":"https://api.github.com/repos/poanetwork/poa-explorer/keys{/key_id}","collaborators_url":"https://api.github.com/repos/poanetwork/poa-explorer/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/poanetwork/poa-explorer/teams","hooks_url":"https://api.github.com/repos/poanetwork/poa-explorer/hooks","issue_events_url":"https://api.github.com/repos/poanetwork/poa-explorer/issues/events{/number}","events_url":"https://api.github.com/repos/poanetwork/poa-explorer/events","assignees_url":"https://api.github.com/repos/poanetwork/poa-explorer/assignees{/user}","branches_url":"https://api.github.com/repos/poanetwork/poa-explorer/branches{/branch}","tags_url":"https://api.github.com/repos/poanetwork/poa-explorer/tags","blobs_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/refs{/sha}","trees_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/trees{/sha}","statuses_url":"https://api.github.com/repos/poanetwork/poa-explorer/statuses/{sha}","languages_url":"https://api.github.com/repos/poanetwork/poa-explorer/languages","stargazers_url":"https://api.github.com/repos/poanetwork/poa-explorer/stargazers","contributors_url":"https://api.github.com/repos/poanetwork/poa-explorer/contributors","subscribers_url":"https://api.github.com/repos/poanetwork/poa-explorer/subscribers","subscription_url":"https://api.github.com/repos/poanetwork/poa-explorer/subscription","commits_url":"https://api.github.com/repos/poanetwork/poa-explorer/commits{/sha}","git_commits_url":"https://api.github.com/repos/poanetwork/poa-explorer/git/commits{/sha}","comments_url":"https://api.github.com/repos/poanetwork/poa-explorer/comments{/number}","issue_comment_url":"https://api.github.com/repos/poanetwork/poa-explorer/issues/comments{/number}","contents_url":"https://api.github.com/repos/poanetwork/poa-explorer/contents/{+path}","compare_url":"https://api.github.com/repos/poanetwork/poa-explorer/compare/{base}...{head}","merges_url":"https://api.github.com/repos/poanetwork/poa-explorer/merges","archive_url":"https://api.github.com/repos/poanetwork/poa-explorer/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/poanetwork/poa-explorer/downloads","issues_url":"https://api.github.com/repos/poanetwork/poa-explorer/issues{/number}","pulls_url":"https://api.github.com/repos/poanetwork/poa-explorer/pulls{/number}","milestones_url":"https://api.github.com/repos/poanetwork/poa-explorer/milestones{/number}","notifications_url":"https://api.github.com/repos/poanetwork/poa-explorer/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/poanetwork/poa-explorer/labels{/name}","releases_url":"https://api.github.com/repos/poanetwork/poa-explorer/releases{/id}","deployments_url":"https://api.github.com/repos/poanetwork/poa-explorer/deployments","created_at":"2018-01-12T22:45:24Z","updated_at":"2018-05-29T08:29:12Z","pushed_at":"2018-05-29T13:44:47Z","git_url":"git://github.com/poanetwork/poa-explorer.git","ssh_url":"git@github.com:poanetwork/poa-explorer.git","clone_url":"https://github.com/poanetwork/poa-explorer.git","svn_url":"https://github.com/poanetwork/poa-explorer","homepage":"","size":1837,"stargazers_count":43,"watchers_count":43,"language":"Elixir","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":18,"mirror_url":null,"archived":false,"open_issues_count":86,"license":{"key":"gpl-3.0","name":"GNU General Public License v3.0","spdx_id":"GPL-3.0","url":"https://api.github.com/licenses/gpl-3.0"},"forks":18,"open_issues":86,"watchers":43,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/218"},"html":{"href":"https://github.com/poanetwork/poa-explorer/pull/218"},"issue":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/issues/218"},"comments":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/issues/218/comments"},"review_comments":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/218/comments"},"review_comment":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/pulls/218/commits"},"statuses":{"href":"https://api.github.com/repos/poanetwork/poa-explorer/statuses/7d77af44b7caada96065719a5bf09949b55129be"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"josevalim"}}