{"action":"created","issue":{"url":"https://api.github.com/repos/poanetwork/blockscout/issues/948","repository_url":"https://api.github.com/repos/poanetwork/blockscout","labels_url":"https://api.github.com/repos/poanetwork/blockscout/issues/948/labels{/name}","comments_url":"https://api.github.com/repos/poanetwork/blockscout/issues/948/comments","events_url":"https://api.github.com/repos/poanetwork/blockscout/issues/948/events","html_url":"https://github.com/poanetwork/blockscout/issues/948","id":371578949,"node_id":"MDU6SXNzdWUzNzE1Nzg5NDk=","number":948,"title":"Investigate token count on the address page","user":{"login":"acravenho","id":17620007,"node_id":"MDQ6VXNlcjE3NjIwMDA3","avatar_url":"https://avatars1.githubusercontent.com/u/17620007?v=4","gravatar_id":"","url":"https://api.github.com/users/acravenho","html_url":"https://github.com/acravenho","followers_url":"https://api.github.com/users/acravenho/followers","following_url":"https://api.github.com/users/acravenho/following{/other_user}","gists_url":"https://api.github.com/users/acravenho/gists{/gist_id}","starred_url":"https://api.github.com/users/acravenho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/acravenho/subscriptions","organizations_url":"https://api.github.com/users/acravenho/orgs","repos_url":"https://api.github.com/users/acravenho/repos","events_url":"https://api.github.com/users/acravenho/events{/privacy}","received_events_url":"https://api.github.com/users/acravenho/received_events","type":"User","site_admin":false},"labels":[{"id":803248120,"node_id":"MDU6TGFiZWw4MDMyNDgxMjA=","url":"https://api.github.com/repos/poanetwork/blockscout/labels/bug%20:bug:","name":"bug :bug:","color":"ee0701","default":false},{"id":997629238,"node_id":"MDU6TGFiZWw5OTc2MjkyMzg=","url":"https://api.github.com/repos/poanetwork/blockscout/labels/priority:%20high","name":"priority: high","color":"ffb4b4","default":false},{"id":871581825,"node_id":"MDU6TGFiZWw4NzE1ODE4MjU=","url":"https://api.github.com/repos/poanetwork/blockscout/labels/team:%20developer","name":"team: developer","color":"c2f8f9","default":false},{"id":865602155,"node_id":"MDU6TGFiZWw4NjU2MDIxNTU=","url":"https://api.github.com/repos/poanetwork/blockscout/labels/tokens","name":"tokens","color":"fce3b5","default":false}],"state":"open","locked":false,"assignee":{"login":"feliperenan","id":27698968,"node_id":"MDQ6VXNlcjI3Njk4OTY4","avatar_url":"https://avatars2.githubusercontent.com/u/27698968?v=4","gravatar_id":"","url":"https://api.github.com/users/feliperenan","html_url":"https://github.com/feliperenan","followers_url":"https://api.github.com/users/feliperenan/followers","following_url":"https://api.github.com/users/feliperenan/following{/other_user}","gists_url":"https://api.github.com/users/feliperenan/gists{/gist_id}","starred_url":"https://api.github.com/users/feliperenan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/feliperenan/subscriptions","organizations_url":"https://api.github.com/users/feliperenan/orgs","repos_url":"https://api.github.com/users/feliperenan/repos","events_url":"https://api.github.com/users/feliperenan/events{/privacy}","received_events_url":"https://api.github.com/users/feliperenan/received_events","type":"User","site_admin":false},"assignees":[{"login":"feliperenan","id":27698968,"node_id":"MDQ6VXNlcjI3Njk4OTY4","avatar_url":"https://avatars2.githubusercontent.com/u/27698968?v=4","gravatar_id":"","url":"https://api.github.com/users/feliperenan","html_url":"https://github.com/feliperenan","followers_url":"https://api.github.com/users/feliperenan/followers","following_url":"https://api.github.com/users/feliperenan/following{/other_user}","gists_url":"https://api.github.com/users/feliperenan/gists{/gist_id}","starred_url":"https://api.github.com/users/feliperenan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/feliperenan/subscriptions","organizations_url":"https://api.github.com/users/feliperenan/orgs","repos_url":"https://api.github.com/users/feliperenan/repos","events_url":"https://api.github.com/users/feliperenan/events{/privacy}","received_events_url":"https://api.github.com/users/feliperenan/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/poanetwork/blockscout/milestones/2","html_url":"https://github.com/poanetwork/blockscout/milestone/2","labels_url":"https://api.github.com/repos/poanetwork/blockscout/milestones/2/labels","id":3321184,"node_id":"MDk6TWlsZXN0b25lMzMyMTE4NA==","number":2,"title":"Kuala Lumpur","description":"BlockScout Kuala Lumpur release.","creator":{"login":"acravenho","id":17620007,"node_id":"MDQ6VXNlcjE3NjIwMDA3","avatar_url":"https://avatars1.githubusercontent.com/u/17620007?v=4","gravatar_id":"","url":"https://api.github.com/users/acravenho","html_url":"https://github.com/acravenho","followers_url":"https://api.github.com/users/acravenho/followers","following_url":"https://api.github.com/users/acravenho/following{/other_user}","gists_url":"https://api.github.com/users/acravenho/gists{/gist_id}","starred_url":"https://api.github.com/users/acravenho/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/acravenho/subscriptions","organizations_url":"https://api.github.com/users/acravenho/orgs","repos_url":"https://api.github.com/users/acravenho/repos","events_url":"https://api.github.com/users/acravenho/events{/privacy}","received_events_url":"https://api.github.com/users/acravenho/received_events","type":"User","site_admin":false},"open_issues":126,"closed_issues":123,"state":"open","created_at":"2018-05-04T18:13:34Z","updated_at":"2018-10-19T18:06:16Z","due_on":"2018-11-01T07:00:00Z","closed_at":null},"comments":0,"created_at":"2018-10-18T15:10:24Z","updated_at":"2018-10-19T19:51:13Z","closed_at":null,"author_association":"MEMBER","body":"The token count on Blockscout is not matching the token counts on Etherscan. While I understand that the chain is still indexing, there are transactions with token transfers coming in but the token balances are not updating and showing them under the address's holdings. \r\n\r\nHere is an example:\r\n\r\nhttps://blockscout.com/eth/mainnet/address/0x9a4a90e2732f3fa4087b0bb4bf85c76d14833df1/transactions\r\n\r\nThere are 10-20 transfers showing on blockscout coming into this address but the holdings are not reflecting these balances. \r\n\r\n1. We need to check that at some point these token balances will be fetched.\r\n2. If failed, what system is in place to check them again?\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/poanetwork/blockscout/issues/comments/431479897","html_url":"https://github.com/poanetwork/blockscout/issues/948#issuecomment-431479897","issue_url":"https://api.github.com/repos/poanetwork/blockscout/issues/948","id":431479897,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTQ3OTg5Nw==","user":{"login":"feliperenan","id":27698968,"node_id":"MDQ6VXNlcjI3Njk4OTY4","avatar_url":"https://avatars2.githubusercontent.com/u/27698968?v=4","gravatar_id":"","url":"https://api.github.com/users/feliperenan","html_url":"https://github.com/feliperenan","followers_url":"https://api.github.com/users/feliperenan/followers","following_url":"https://api.github.com/users/feliperenan/following{/other_user}","gists_url":"https://api.github.com/users/feliperenan/gists{/gist_id}","starred_url":"https://api.github.com/users/feliperenan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/feliperenan/subscriptions","organizations_url":"https://api.github.com/users/feliperenan/orgs","repos_url":"https://api.github.com/users/feliperenan/repos","events_url":"https://api.github.com/users/feliperenan/events{/privacy}","received_events_url":"https://api.github.com/users/feliperenan/received_events","type":"User","site_admin":false},"created_at":"2018-10-19T19:51:13Z","updated_at":"2018-10-19T19:51:13Z","author_association":"CONTRIBUTOR","body":"I guess that the `TokenBalance.Fetcher` has a bottleneck. It is configured to fetch only 1 token balance at a time, but it receives a lot of token balances imported by catchup to be fetched. For instance:\r\n\r\n* Catchup sends 1000 token balances to be fetched\r\n* For each token balance, the `TokenBalance.Fetcher` is going to make a request to the smart contract to fetch the balance and import the new balance (one by one). It's not async since our `batch_size` is configured to be 1.\r\n* Catchup sends more 1000 token balances to be fetched and the `TokenBalance.Fetcher` adds it to the queue.\t\r\n\r\nSo, the `TokenBalance.Fetcher` will accumulate a lot of messages sent by catchup, it receives a lot by second, while is processing one by one. This way, the token balances will always be far behind the token transfers.\r\n\r\nThe possible solution would be increasing the `batch_size` and `init_chunk_size` to process more token balances at a time, then the token balances will be closer to the token transfers, although still behind. I remember that we decreased those numbers because we overloaded the node with a lot of requests, but I think we can increase it again and try to make it configurable to decrease when the indexer overload the node.\r\n\r\nAnswering the questions:\r\n> 1. We need to check that at some point these token balances will be fetched.\r\n\r\nIt is going to be fetched, but it is taking too long.\r\n\r\n> If failed, what system is in place to check them again?\r\n\r\nIf the import process fails, it will retry until everything is imported. If the `TokenBalance.Fetcher` crashes for any reason, when it startes, it will run a query to get all token balances that had not their values fetched to `TokenBalance.Fetcher` queue again. But if we get a timeout from the smart contract interaction, we don't update that token balance until the `TokenBalance.Fetcher` crashes. I think we need to create a mechanism to run that query periodically.\r\n\r\nTo wrap up, we need to do two things:\r\n- Increase the `batch_size` and `init_chunk_size` to process more token balances in `TokenBalance.Fetcher`\r\n- Run the query that finds token balances that weren't fetched periodically."}}
{"actor":{"display_login":"feliperenan"}}