{"action":"created","comment":{"url":"https://api.github.com/repos/ethereum/go-ethereum/pulls/comments/65865128","id":65865128,"diff_hunk":"@@ -515,7 +527,16 @@ func (d *Downloader) cancel() {\n // Terminate interrupts the downloader, canceling all pending operations.\n // The downloader cannot be reused after calling Terminate.\n func (d *Downloader) Terminate() {\n-\tatomic.StoreInt32(&d.interrupt, 1)\n+\t// Close the termination channel (make sure double close is allowed)\n+\td.quitLock.Lock()\n+\tselect {\n+\tcase <-d.quitCh:\n+\tdefault:\n+\t\tclose(d.quitCh)\n+\t}\n+\td.quitLock.Unlock()","path":"eth/downloader/downloader.go","position":91,"original_position":91,"commit_id":"990e010b8b46dafeede0d0a12fd1c39f2fbc577e","original_commit_id":"990e010b8b46dafeede0d0a12fd1c39f2fbc577e","user":{"login":"fjl","id":6915,"avatar_url":"https://avatars.githubusercontent.com/u/6915?v=3","gravatar_id":"","url":"https://api.github.com/users/fjl","html_url":"https://github.com/fjl","followers_url":"https://api.github.com/users/fjl/followers","following_url":"https://api.github.com/users/fjl/following{/other_user}","gists_url":"https://api.github.com/users/fjl/gists{/gist_id}","starred_url":"https://api.github.com/users/fjl/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/fjl/subscriptions","organizations_url":"https://api.github.com/users/fjl/orgs","repos_url":"https://api.github.com/users/fjl/repos","events_url":"https://api.github.com/users/fjl/events{/privacy}","received_events_url":"https://api.github.com/users/fjl/received_events","type":"User","site_admin":false},"body":"Answering my own question: quit is permanent, cancel isn't. But can't we use cancel to exit the qos adaptation goroutine?","created_at":"2016-06-06T09:58:37Z","updated_at":"2016-06-06T09:58:37Z","html_url":"https://github.com/ethereum/go-ethereum/pull/2630#discussion_r65865128","pull_request_url":"https://api.github.com/repos/ethereum/go-ethereum/pulls/2630","_links":{"self":{"href":"https://api.github.com/repos/ethereum/go-ethereum/pulls/comments/65865128"},"html":{"href":"https://github.com/ethereum/go-ethereum/pull/2630#discussion_r65865128"},"pull_request":{"href":"https://api.github.com/repos/ethereum/go-ethereum/pulls/2630"}}},"pull_request":{"url":"https://api.github.com/repos/ethereum/go-ethereum/pulls/2630","id":71744415,"html_url":"https://github.com/ethereum/go-ethereum/pull/2630","diff_url":"https://github.com/ethereum/go-ethereum/pull/2630.diff","patch_url":"https://github.com/ethereum/go-ethereum/pull/2630.patch","issue_url":"https://api.github.com/repos/ethereum/go-ethereum/issues/2630","number":2630,"state":"open","locked":false,"title":"eth/downloader: adaptive quality of service tuning","user":{"login":"karalabe","id":129561,"avatar_url":"https://avatars.githubusercontent.com/u/129561?v=3","gravatar_id":"","url":"https://api.github.com/users/karalabe","html_url":"https://github.com/karalabe","followers_url":"https://api.github.com/users/karalabe/followers","following_url":"https://api.github.com/users/karalabe/following{/other_user}","gists_url":"https://api.github.com/users/karalabe/gists{/gist_id}","starred_url":"https://api.github.com/users/karalabe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karalabe/subscriptions","organizations_url":"https://api.github.com/users/karalabe/orgs","repos_url":"https://api.github.com/users/karalabe/repos","events_url":"https://api.github.com/users/karalabe/events{/privacy}","received_events_url":"https://api.github.com/users/karalabe/received_events","type":"User","site_admin":false},"body":"Currently Geth is tuned for low latency, high bandwidth network scenarios, which the sync mechanism enforces by dropping peers who do not reply fast enough to data queries. The rationale was that by honing in on faster peers (i.e. geographically closer to us), we can achieve much faster sync times.\r\n\r\nEven though we are aggressive in peer selection, the downloader tries to be smart about the queries it issues by estimating the throughput of all connected peers and requesting data in proportion with the time it should take the peer to respond, attempting to stabilize both the network (by not assigning huge tasks that would time out) as well as the data stream (by having all peers respond with the same RTT).\r\n\r\nThis algorithm can ensure that independent of what quality peers we have, we will have a stable download stream... **as long as we have enough bandwidth to cover all our peers as well as a negligible latency compared to the employed timeouts**. However if I have low bandwidth myself, packet RTTs increase proportionally do the download tasks, which can easily hit the timeout thresholds, causing peers to be dropped. Even worse, if I have a high latency connection (EDGE, satellite), the timeouts will almost immediately drop all peers, making sync impossible.\r\n\r\nThe fundamental flaw is that our current codebase has fixed values as the target RTT in which peers are expected to respond to queries (1-1.5 seconds) and request TTLs after which peers are dropped (3-4 seconds); but for low bandwidth or high latency connections, these are just too low to work out. Simply raising these would allow users with bad connectivity to sync, but would at the same time slow down sync for good connectivity users to the speed of their weakest peer, which would be a bottleneck for the entire algorithm. Instead, this PR attempts to dynamically adapt the target RTT and request TTL values in such a way that we can still hone in on good peers, but do that based on our active peer set and not a blind baseline expected performance.\r\n\r\n*Note, hard RTT and TTL values were needed prior to concurrent headers because the master peer which we used to pull headers from was already a bottleneck and we couldn't afford to stall sync with arbitrary performance. With concurrent headers, even if the master peer is slow-ish we can still sync fast, so we can me much more lax with regard to RTT and TTL.*\r\n\r\n### Tuning peers\r\n\r\nTo dynamically adapt the RTT and TTL values, we need to make various measurements on our peers' performance and aggregate them into these thresholds. However if we have a lot of peers (e.g. 10, 25, 100), some of them will be high quality peers, and some will be low quality ones. Aggregating all of them would result in too lax RTT/TTL values, delaying sync due to unnecessary bottlenecks.\r\n\r\nInstead of adapting to all connected peers, we would like to hone in on our true performance, and keep only peers that are reasonably close performance wise. To do this, we will tune the parameters based only on our best N peers. Those upwards from N will be required some threshold of performance from our best peers, or else be dropped during sync. This algorithm ensures that although we adapt our RTT/TTL values to our peers and network connectivity, we still saturate our own connection and not introduce bottlenecks. If a better peer joins, it will automatically result in tighter thresholds, whereas if a top peer drops off, the threshold will be automatically relaxed.\r\n\r\nThe PR chosses `N = 5`. The rationale is that by default geth has `--maxpeers=25`, however only half of that is reserved for outbound connections, so with a NATed/firewalled system, peer count is capped at around 12. With `N == 5` we retain enough peers to do meaningful tuning, but also leave enough slots for bad peers to die off and leave room for better ones. Lastly, it's important to mention that the [BitTorrent protocol](http://www.bittorrent.org/beps/bep_0003.html) also uses 4 (stable) + 1 (optimistic) peers for downloading data at any given time, as higher values cause suboptimal TCP congestion control. Although BitTorrent furthers this logic by choking the remaining peers, it's a complexity we need to decide whether to introduce or not (@fjl suggested we do, maybe in a followup PR we can do an N-tuning M-optimistic K-spare peers split).\r\n\r\n### RTT vs. TTL\r\n\r\nAs discussed, sync performance depends on tuning two parameters of the downloader: the target round trip time (RTT) that ensures we have a stable stream of data to process; and the request time to live (TTL) which ensures some baseline quality of peers. Thinking about it though, the two values are somewhat related: if we can sustain a stable RTT over multiple peers, then we can set TTL to be a constant multiple `c` of this RTT value. As long as RTT doesn't fibrillate, `TTL = c * RTT` is a good filter to weed out under performing peers.\r\n\r\nFibrillation however can be an issue, especially during the beginning of sync. If we have only a couple peers (e.g. 2) that we tuned the RTT to, and suddenly experience a surge in connections (e.g. 4 new ones), then limited bandwidth availability would cause actual round trip times to grow proportionally to the additional download requests (not giving an opportunity for our estimated RTT (and hence related TTL) to adapt fast enough). Hitting the TTL limit would result in both new as well as potentially old, perfectly good peers being dropped. Reducing peer count would drive down the RTT, and reconnecting peers would fibrillate it back up.\r\n\r\nThe moral of the story is, that `TTL = c * RTT` only makes sense, if the estimated RTT is relatively stable and approximates the true RTT closely enough. As we cannot guarantee RTT stability during peer joins, we introduce a *confidence* factor into our estimation. The more peers join (proportionally to our existing ones), the less confident we can be in our estimated RTT value; and the less confident we are, the higher TTL threshold needs to be allowed to cater for splitting our bandwidth among more peers.\r\n\r\nAggregating the above rational, we can set `TTL = c * RTT / conf`, where `conf ∈ (0, 1]`. If we assume a stable RTT (i.e. `conf == 1`), we can pick a meaningful number for the `c` scaling factor. It's really arbitrary, but `c = 3` gives enough wiggle room for occasional hiccups but is not lax enough to hinder performance too much. As for the confidence multiplier, the only idea we can use is that limited bandwidth is split between peers, so more peers **could (not would)** result in proportionally larger RTTs. Whenever a peer joins, we reduce `conf` proportionally to cater for bandwidth splits: `conf = conf * (1 - 1/(peers+1)) = conf * peers / (peers + 1)`.\r\n\r\nE.g. if we had a stable RTT (`conf == 1`) with 2 peers, adding new peers would:\r\n\r\n * +1 peer => `conf = 1 * 2 / 3 = 2/3`\r\n * +1 peer => `conf = 2/3 * 3 / 4 = 2/4`\r\n * +1 peer => `conf = 2/4 * 4 / 5 = 2/5`\r\n\r\nPer the above, the confidence factor correctly follows the number of joining peers, even when processed one by one (+1+1+1 results in the expected value that +3 would produce). Note, we'll cap the minimum of the confidence factor to `0.1` to prevent too wild TTL values. The confidence factor is increased towards `1` after every RTT.\r\n\r\n### Adaptive RTT\r\n\r\nBased on the above discussion, if we can estimate a reasonable target round-trip time and can keep it relatively up to date with peers' capacities but at the same time stable as peers join/leave and requests come/go, we can derive a good enough TTL value. The remaining challenge is how to select a good RTT to target for requests. The requirements are that it should be high enough to allow a meaningful batch of data to be retrieved, but low enough so that import can progress without stuttering too much.\r\n\r\nWe will start out with the maximum allowed RTT value if we have no connections (PR sets it to 20s). This ensures that irrelevant of our local connectivity, we can do some meaningful communication. \r\n\r\nFurther, for each peer we will maintain a **very rough** RTT value (initially these are maxed too), which we'll update whenever the said peer responds to one of our data requests (weighed update). Note, this RTT will vary a ton based on our network saturation, the peers network saturation, latencies, congestion, etc. This is fine, it's just meant as a means to compare the performance of different peers at the current workload (whatever that might be).\r\n\r\nUsing these constantly changing RTT values, every once in a while (more on this later) we select the N best peers to tune with (as mentioned above, 5 currently) and updated our target RTT to the median performance of these best peers (again, weighed update to smooth hiccups). With the target RTT updated, we can calculate a fresh TTL value too.\r\n\r\n### Converging towards the optimum\r\n\r\nAdaptive algorithms have a tendency to converge to some stable state, which may or may not be the optimal solution (depends on the exact scenario). In our case if we have a single peer, two stable scenarios could be: a) download 256 headers in 2 seconds; and b) download 128 headers in 1 second. Both are perfectly valid in the long run, but as block processing is a streaming operation, and there are a lot of steps to go through, it is usually more advisable to prefer shorter cycles over longer ones. To this extent, we need to force the algorithm out of sub-optimal stable points.\r\n\r\nOne such scenario is the selection of a stable, but larger RTT (e.g. 5 sec vs. 2 sec). Having a larger than essential RTT means that streaming the blockchain is more bursty and less flowing; but it also means that we could end up keeping weaker peers that ould have been weeded out by the small RTT. As long as the smaller RTT is equally stable, performance wise it will always be preferable. To prevent locking in to a higher than needed RTT, whenever we fetch some data, we will not use our estimated best RTT, but rather only 90% of it. If the smaller RTT was a bad guess (e.g. our latency is the culprit), we'll just estimate the original RTT the next time too and remain unchanged. However if our guessed RTT turns out to be valid, we just shaved off a bit of your sync time. Repeating this will converge on the minimum RTT that is stable for our actual workload.\r\n\r\nThe second interesting scenario is during throughput estimation. The downloader tries to (did this since forever) estimate how many blocks, receipts, states, etc a peer can give us in a given amount of time, and use that to scale the size of data retrievals to the target RTT. This works well in low latency environments (e.g. 10 blocks 1 sec => 20 blocks 2 secs), but it performs poorly if latency comes into play (e.g. with a 250ms latency, 10 blocks 1 sec => 20 blocks 1.5sec). In high latency environments, the algorithm will constantly under-estimate the true throughput and retrieve less, wasting time because of the latency, not because of actual throughput. To solve this, the PR makes a very subtle change to the calculation of download task sizes. Instead of using `RTT / throughput`, we will actually request `+1` data item than what the correct estimate is. If the limiting factor was bandwidth, we'll get back the original throughput and be done with it. If however the limiting factor was latency, we'll get the same RTT, but at a much higher throughput.\r\n","created_at":"2016-05-27T17:09:40Z","updated_at":"2016-06-06T09:58:37Z","closed_at":null,"merged_at":null,"merge_commit_sha":"a8cfe64df74e7c6484362fd18a843117ef317889","assignee":null,"milestone":{"url":"https://api.github.com/repos/ethereum/go-ethereum/milestones/34","html_url":"https://github.com/ethereum/go-ethereum/milestones/1.4.6","labels_url":"https://api.github.com/repos/ethereum/go-ethereum/milestones/34/labels","id":1782503,"number":34,"title":"1.4.6","description":null,"creator":{"login":"karalabe","id":129561,"avatar_url":"https://avatars.githubusercontent.com/u/129561?v=3","gravatar_id":"","url":"https://api.github.com/users/karalabe","html_url":"https://github.com/karalabe","followers_url":"https://api.github.com/users/karalabe/followers","following_url":"https://api.github.com/users/karalabe/following{/other_user}","gists_url":"https://api.github.com/users/karalabe/gists{/gist_id}","starred_url":"https://api.github.com/users/karalabe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karalabe/subscriptions","organizations_url":"https://api.github.com/users/karalabe/orgs","repos_url":"https://api.github.com/users/karalabe/repos","events_url":"https://api.github.com/users/karalabe/events{/privacy}","received_events_url":"https://api.github.com/users/karalabe/received_events","type":"User","site_admin":false},"open_issues":6,"closed_issues":8,"state":"open","created_at":"2016-05-23T11:11:31Z","updated_at":"2016-06-06T09:42:49Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/ethereum/go-ethereum/pulls/2630/commits","review_comments_url":"https://api.github.com/repos/ethereum/go-ethereum/pulls/2630/comments","review_comment_url":"https://api.github.com/repos/ethereum/go-ethereum/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ethereum/go-ethereum/issues/2630/comments","statuses_url":"https://api.github.com/repos/ethereum/go-ethereum/statuses/990e010b8b46dafeede0d0a12fd1c39f2fbc577e","head":{"label":"karalabe:adaptive-qos-tuning","ref":"adaptive-qos-tuning","sha":"990e010b8b46dafeede0d0a12fd1c39f2fbc577e","user":{"login":"karalabe","id":129561,"avatar_url":"https://avatars.githubusercontent.com/u/129561?v=3","gravatar_id":"","url":"https://api.github.com/users/karalabe","html_url":"https://github.com/karalabe","followers_url":"https://api.github.com/users/karalabe/followers","following_url":"https://api.github.com/users/karalabe/following{/other_user}","gists_url":"https://api.github.com/users/karalabe/gists{/gist_id}","starred_url":"https://api.github.com/users/karalabe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karalabe/subscriptions","organizations_url":"https://api.github.com/users/karalabe/orgs","repos_url":"https://api.github.com/users/karalabe/repos","events_url":"https://api.github.com/users/karalabe/events{/privacy}","received_events_url":"https://api.github.com/users/karalabe/received_events","type":"User","site_admin":false},"repo":{"id":33668567,"name":"go-ethereum","full_name":"karalabe/go-ethereum","owner":{"login":"karalabe","id":129561,"avatar_url":"https://avatars.githubusercontent.com/u/129561?v=3","gravatar_id":"","url":"https://api.github.com/users/karalabe","html_url":"https://github.com/karalabe","followers_url":"https://api.github.com/users/karalabe/followers","following_url":"https://api.github.com/users/karalabe/following{/other_user}","gists_url":"https://api.github.com/users/karalabe/gists{/gist_id}","starred_url":"https://api.github.com/users/karalabe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karalabe/subscriptions","organizations_url":"https://api.github.com/users/karalabe/orgs","repos_url":"https://api.github.com/users/karalabe/repos","events_url":"https://api.github.com/users/karalabe/events{/privacy}","received_events_url":"https://api.github.com/users/karalabe/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/karalabe/go-ethereum","description":"Jeffrey Wilcke's Go implementation of the Ethereum y/w paper","fork":true,"url":"https://api.github.com/repos/karalabe/go-ethereum","forks_url":"https://api.github.com/repos/karalabe/go-ethereum/forks","keys_url":"https://api.github.com/repos/karalabe/go-ethereum/keys{/key_id}","collaborators_url":"https://api.github.com/repos/karalabe/go-ethereum/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/karalabe/go-ethereum/teams","hooks_url":"https://api.github.com/repos/karalabe/go-ethereum/hooks","issue_events_url":"https://api.github.com/repos/karalabe/go-ethereum/issues/events{/number}","events_url":"https://api.github.com/repos/karalabe/go-ethereum/events","assignees_url":"https://api.github.com/repos/karalabe/go-ethereum/assignees{/user}","branches_url":"https://api.github.com/repos/karalabe/go-ethereum/branches{/branch}","tags_url":"https://api.github.com/repos/karalabe/go-ethereum/tags","blobs_url":"https://api.github.com/repos/karalabe/go-ethereum/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/karalabe/go-ethereum/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/karalabe/go-ethereum/git/refs{/sha}","trees_url":"https://api.github.com/repos/karalabe/go-ethereum/git/trees{/sha}","statuses_url":"https://api.github.com/repos/karalabe/go-ethereum/statuses/{sha}","languages_url":"https://api.github.com/repos/karalabe/go-ethereum/languages","stargazers_url":"https://api.github.com/repos/karalabe/go-ethereum/stargazers","contributors_url":"https://api.github.com/repos/karalabe/go-ethereum/contributors","subscribers_url":"https://api.github.com/repos/karalabe/go-ethereum/subscribers","subscription_url":"https://api.github.com/repos/karalabe/go-ethereum/subscription","commits_url":"https://api.github.com/repos/karalabe/go-ethereum/commits{/sha}","git_commits_url":"https://api.github.com/repos/karalabe/go-ethereum/git/commits{/sha}","comments_url":"https://api.github.com/repos/karalabe/go-ethereum/comments{/number}","issue_comment_url":"https://api.github.com/repos/karalabe/go-ethereum/issues/comments{/number}","contents_url":"https://api.github.com/repos/karalabe/go-ethereum/contents/{+path}","compare_url":"https://api.github.com/repos/karalabe/go-ethereum/compare/{base}...{head}","merges_url":"https://api.github.com/repos/karalabe/go-ethereum/merges","archive_url":"https://api.github.com/repos/karalabe/go-ethereum/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/karalabe/go-ethereum/downloads","issues_url":"https://api.github.com/repos/karalabe/go-ethereum/issues{/number}","pulls_url":"https://api.github.com/repos/karalabe/go-ethereum/pulls{/number}","milestones_url":"https://api.github.com/repos/karalabe/go-ethereum/milestones{/number}","notifications_url":"https://api.github.com/repos/karalabe/go-ethereum/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/karalabe/go-ethereum/labels{/name}","releases_url":"https://api.github.com/repos/karalabe/go-ethereum/releases{/id}","deployments_url":"https://api.github.com/repos/karalabe/go-ethereum/deployments","created_at":"2015-04-09T12:51:45Z","updated_at":"2016-02-23T11:27:22Z","pushed_at":"2016-06-06T09:45:37Z","git_url":"git://github.com/karalabe/go-ethereum.git","ssh_url":"git@github.com:karalabe/go-ethereum.git","clone_url":"https://github.com/karalabe/go-ethereum.git","svn_url":"https://github.com/karalabe/go-ethereum","homepage":"","size":58547,"stargazers_count":0,"watchers_count":0,"language":"Go","has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"develop"}},"base":{"label":"ethereum:develop","ref":"develop","sha":"780bdb3e8069c38a779639ee92cc0eccba8a6735","user":{"login":"ethereum","id":6250754,"avatar_url":"https://avatars.githubusercontent.com/u/6250754?v=3","gravatar_id":"","url":"https://api.github.com/users/ethereum","html_url":"https://github.com/ethereum","followers_url":"https://api.github.com/users/ethereum/followers","following_url":"https://api.github.com/users/ethereum/following{/other_user}","gists_url":"https://api.github.com/users/ethereum/gists{/gist_id}","starred_url":"https://api.github.com/users/ethereum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ethereum/subscriptions","organizations_url":"https://api.github.com/users/ethereum/orgs","repos_url":"https://api.github.com/users/ethereum/repos","events_url":"https://api.github.com/users/ethereum/events{/privacy}","received_events_url":"https://api.github.com/users/ethereum/received_events","type":"Organization","site_admin":false},"repo":{"id":15452919,"name":"go-ethereum","full_name":"ethereum/go-ethereum","owner":{"login":"ethereum","id":6250754,"avatar_url":"https://avatars.githubusercontent.com/u/6250754?v=3","gravatar_id":"","url":"https://api.github.com/users/ethereum","html_url":"https://github.com/ethereum","followers_url":"https://api.github.com/users/ethereum/followers","following_url":"https://api.github.com/users/ethereum/following{/other_user}","gists_url":"https://api.github.com/users/ethereum/gists{/gist_id}","starred_url":"https://api.github.com/users/ethereum/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ethereum/subscriptions","organizations_url":"https://api.github.com/users/ethereum/orgs","repos_url":"https://api.github.com/users/ethereum/repos","events_url":"https://api.github.com/users/ethereum/events{/privacy}","received_events_url":"https://api.github.com/users/ethereum/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ethereum/go-ethereum","description":"Official golang implementation of the Ethereum protocol","fork":false,"url":"https://api.github.com/repos/ethereum/go-ethereum","forks_url":"https://api.github.com/repos/ethereum/go-ethereum/forks","keys_url":"https://api.github.com/repos/ethereum/go-ethereum/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ethereum/go-ethereum/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ethereum/go-ethereum/teams","hooks_url":"https://api.github.com/repos/ethereum/go-ethereum/hooks","issue_events_url":"https://api.github.com/repos/ethereum/go-ethereum/issues/events{/number}","events_url":"https://api.github.com/repos/ethereum/go-ethereum/events","assignees_url":"https://api.github.com/repos/ethereum/go-ethereum/assignees{/user}","branches_url":"https://api.github.com/repos/ethereum/go-ethereum/branches{/branch}","tags_url":"https://api.github.com/repos/ethereum/go-ethereum/tags","blobs_url":"https://api.github.com/repos/ethereum/go-ethereum/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ethereum/go-ethereum/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ethereum/go-ethereum/git/refs{/sha}","trees_url":"https://api.github.com/repos/ethereum/go-ethereum/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ethereum/go-ethereum/statuses/{sha}","languages_url":"https://api.github.com/repos/ethereum/go-ethereum/languages","stargazers_url":"https://api.github.com/repos/ethereum/go-ethereum/stargazers","contributors_url":"https://api.github.com/repos/ethereum/go-ethereum/contributors","subscribers_url":"https://api.github.com/repos/ethereum/go-ethereum/subscribers","subscription_url":"https://api.github.com/repos/ethereum/go-ethereum/subscription","commits_url":"https://api.github.com/repos/ethereum/go-ethereum/commits{/sha}","git_commits_url":"https://api.github.com/repos/ethereum/go-ethereum/git/commits{/sha}","comments_url":"https://api.github.com/repos/ethereum/go-ethereum/comments{/number}","issue_comment_url":"https://api.github.com/repos/ethereum/go-ethereum/issues/comments{/number}","contents_url":"https://api.github.com/repos/ethereum/go-ethereum/contents/{+path}","compare_url":"https://api.github.com/repos/ethereum/go-ethereum/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ethereum/go-ethereum/merges","archive_url":"https://api.github.com/repos/ethereum/go-ethereum/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ethereum/go-ethereum/downloads","issues_url":"https://api.github.com/repos/ethereum/go-ethereum/issues{/number}","pulls_url":"https://api.github.com/repos/ethereum/go-ethereum/pulls{/number}","milestones_url":"https://api.github.com/repos/ethereum/go-ethereum/milestones{/number}","notifications_url":"https://api.github.com/repos/ethereum/go-ethereum/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ethereum/go-ethereum/labels{/name}","releases_url":"https://api.github.com/repos/ethereum/go-ethereum/releases{/id}","deployments_url":"https://api.github.com/repos/ethereum/go-ethereum/deployments","created_at":"2013-12-26T13:05:46Z","updated_at":"2016-06-06T02:19:02Z","pushed_at":"2016-06-06T09:45:38Z","git_url":"git://github.com/ethereum/go-ethereum.git","ssh_url":"git@github.com:ethereum/go-ethereum.git","clone_url":"https://github.com/ethereum/go-ethereum.git","svn_url":"https://github.com/ethereum/go-ethereum","homepage":"http://ethereum.github.io/go-ethereum/","size":161812,"stargazers_count":1622,"watchers_count":1622,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":525,"mirror_url":null,"open_issues_count":246,"forks":525,"open_issues":246,"watchers":1622,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ethereum/go-ethereum/pulls/2630"},"html":{"href":"https://github.com/ethereum/go-ethereum/pull/2630"},"issue":{"href":"https://api.github.com/repos/ethereum/go-ethereum/issues/2630"},"comments":{"href":"https://api.github.com/repos/ethereum/go-ethereum/issues/2630/comments"},"review_comments":{"href":"https://api.github.com/repos/ethereum/go-ethereum/pulls/2630/comments"},"review_comment":{"href":"https://api.github.com/repos/ethereum/go-ethereum/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ethereum/go-ethereum/pulls/2630/commits"},"statuses":{"href":"https://api.github.com/repos/ethereum/go-ethereum/statuses/990e010b8b46dafeede0d0a12fd1c39f2fbc577e"}}}}
{"actor":{"display_login":"fjl"}}