{"action":"created","issue":{"url":"https://api.github.com/repos/vmware/vic/issues/2587","repository_url":"https://api.github.com/repos/vmware/vic","labels_url":"https://api.github.com/repos/vmware/vic/issues/2587/labels{/name}","comments_url":"https://api.github.com/repos/vmware/vic/issues/2587/comments","events_url":"https://api.github.com/repos/vmware/vic/issues/2587/events","html_url":"https://github.com/vmware/vic/pull/2587","id":181294430,"number":2587,"title":"Imagec library enabling concurrent downloads","user":{"login":"jzt","id":128130,"avatar_url":"https://avatars.githubusercontent.com/u/128130?v=3","gravatar_id":"","url":"https://api.github.com/users/jzt","html_url":"https://github.com/jzt","followers_url":"https://api.github.com/users/jzt/followers","following_url":"https://api.github.com/users/jzt/following{/other_user}","gists_url":"https://api.github.com/users/jzt/gists{/gist_id}","starred_url":"https://api.github.com/users/jzt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jzt/subscriptions","organizations_url":"https://api.github.com/users/jzt/orgs","repos_url":"https://api.github.com/users/jzt/repos","events_url":"https://api.github.com/users/jzt/events{/privacy}","received_events_url":"https://api.github.com/users/jzt/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/vmware/vic/labels/cla-not-required","name":"cla-not-required","color":"159818"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2016-10-05T23:55:47Z","updated_at":"2016-10-11T17:43:05Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/vmware/vic/pulls/2587","html_url":"https://github.com/vmware/vic/pull/2587","diff_url":"https://github.com/vmware/vic/pull/2587.diff","patch_url":"https://github.com/vmware/vic/pull/2587.patch"},"body":"This change converts imagec into working as a library, while preserving the standalone binary in the process. It also adds support for multiple clients downloading the same images concurrently. There is now a layer cache in the persona that is used primarily for tracking layer downloads. It is refreshed at startup by way of the image cache.\r\n\r\nThe majority of the changes you will see in the diff are resulting from moving files around. Most of the meat in these changes are located in `lib/imagec/download.go`, `lib/apiservers/engine/backends/cache/layer_cache.go`, and `lib/imagec/imagec.go`. \r\n\r\nThis has a ways to go, and feedback is always welcome.  Tagging @hickeng, @caglar10ur, @sflxn, @dougm, @hmahmood, @vburenin.\r\n\r\nKnown issues:\r\n- ~~The biggest issue is an intermittent `unexpected EOF` error will kill the client while an image is downloading. It looks really bad, and is probably a really bad user experience/CI killer, but from the persona on down through the portlayer, the image continues to pull and everything works out in the end. There are two ways that the user can know this: One is by simply running the same command a second time. The persona will attach that client to the currently in-progress download, and the client can watch it complete. The other way is by watching the persona logs and seeing the image pull complete. A call to `docker images` after either case will show that it succeeded. I will file an issue to follow up on that problem once this is merged.~~\r\n- ~~Intermittent errors like `malformed chunked encoding`, `invalid byte in chunk length`, and others. I suspect these are being caused by a race condition between multiple goroutines writing to the client's `Outstream` simultaneously. Again, these do not affect the image pull on the backend, only what the client perceives.  These errors seem to show up when pulling with >=3 clients simultaneously.~~\r\n\r\nEOFs and other weird funkiness have been solved. 👍 thanks @hmahmood! \r\n\r\n- Image IDs are not consistent with Docker's (expected as of 1.12), but whats more, if you do a `docker rmi` on an image, only the leaf layer gets removed. When re-pulling the image, only the leaf layer is pulled (since the remaining layers already exist), and only that layer is used to calculate the imageID. Thus, not only are the image IDs inconsistent with Docker, they are also inconsistent between pulls. I have a fix in the works, see #2530. \r\n\r\nThese issues will be captured as bugs once (if) this PR gets merged.\r\n\r\nFixes #1439, #1154, #1298, #1772, #1250, #1409 \r\n\r\n"},"comment":{"url":"https://api.github.com/repos/vmware/vic/issues/comments/252990376","html_url":"https://github.com/vmware/vic/pull/2587#issuecomment-252990376","issue_url":"https://api.github.com/repos/vmware/vic/issues/2587","id":252990376,"user":{"login":"jzt","id":128130,"avatar_url":"https://avatars.githubusercontent.com/u/128130?v=3","gravatar_id":"","url":"https://api.github.com/users/jzt","html_url":"https://github.com/jzt","followers_url":"https://api.github.com/users/jzt/followers","following_url":"https://api.github.com/users/jzt/following{/other_user}","gists_url":"https://api.github.com/users/jzt/gists{/gist_id}","starred_url":"https://api.github.com/users/jzt/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jzt/subscriptions","organizations_url":"https://api.github.com/users/jzt/orgs","repos_url":"https://api.github.com/users/jzt/repos","events_url":"https://api.github.com/users/jzt/events{/privacy}","received_events_url":"https://api.github.com/users/jzt/received_events","type":"User","site_admin":false},"created_at":"2016-10-11T17:43:05Z","updated_at":"2016-10-11T17:43:05Z","body":"lgtm?"}}
{"actor":{"display_login":"jzt"}}