{"action":"created","issue":{"url":"https://api.github.com/repos/mirage/ocaml-git/issues/90","repository_url":"https://api.github.com/repos/mirage/ocaml-git","labels_url":"https://api.github.com/repos/mirage/ocaml-git/issues/90/labels{/name}","comments_url":"https://api.github.com/repos/mirage/ocaml-git/issues/90/comments","events_url":"https://api.github.com/repos/mirage/ocaml-git/issues/90/events","html_url":"https://github.com/mirage/ocaml-git/issues/90","id":90870860,"node_id":"MDU6SXNzdWU5MDg3MDg2MA==","number":90,"title":"Virtual memory explosion on 1.5.0","user":{"login":"AltGr","id":807966,"node_id":"MDQ6VXNlcjgwNzk2Ng==","avatar_url":"https://avatars0.githubusercontent.com/u/807966?v=4","gravatar_id":"","url":"https://api.github.com/users/AltGr","html_url":"https://github.com/AltGr","followers_url":"https://api.github.com/users/AltGr/followers","following_url":"https://api.github.com/users/AltGr/following{/other_user}","gists_url":"https://api.github.com/users/AltGr/gists{/gist_id}","starred_url":"https://api.github.com/users/AltGr/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AltGr/subscriptions","organizations_url":"https://api.github.com/users/AltGr/orgs","repos_url":"https://api.github.com/users/AltGr/repos","events_url":"https://api.github.com/users/AltGr/events{/privacy}","received_events_url":"https://api.github.com/users/AltGr/received_events","type":"User","site_admin":false},"labels":[{"id":31641850,"node_id":"MDU6TGFiZWwzMTY0MTg1MA==","url":"https://api.github.com/repos/mirage/ocaml-git/labels/bug","name":"bug","color":"fc2929","default":true}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":14,"created_at":"2015-06-25T06:36:56Z","updated_at":"2018-10-27T10:02:54Z","closed_at":"2018-10-27T10:02:45Z","author_association":"CONTRIBUTOR","body":"See https://github.com/AltGr/opam-ci/issues/1 for details. 1.4.1 doesn't exhibit this behaviour (although it seems more greedy on resident memory and quite slower).\n### 1.4.11, Git_unix.FS, fetch\n\n![opam-ci-fetch-141](https://cloud.githubusercontent.com/assets/807966/8348065/87b092c6-1b49-11e5-9019-1ca5c08f7430.png)\n\nThe first peak happens during fetch.\n\nThe [ocp-memprof](http://memprof.typerex.org/users/464527645bfe7266d83e8f3c716f1090/2015-06-25_08-14-40_df91f2a4e258e9ba34c1301b85202560_644/index.html) profile is after GC and more interesting when considering resident (heap) memory (the x axis is GC number, not time, the peak corresponds to the fetch).\n### 1.5.0, Git_unix.FS, fetch\n\n![opam-ci-fetch-150](https://cloud.githubusercontent.com/assets/807966/8348090/c3793dbc-1b49-11e5-8ec1-70546cc0f08b.png)\n\nThe [ocp-memprof](http://memprof.typerex.org/users/464527645bfe7266d83e8f3c716f1090/2015-06-25_08-22-44_a379fbe9dc2da7eb72e59975809f7c82_1587/index.html) profile is also informative. Fetch is'nt changed at all, but the following treatment is faster and takes much lest memory (running once the repo has been fetched to disk is thus much cheaper).\n### 1.4.11, Git.Memory, fetch\n\n![opam-ci-mem-141](https://cloud.githubusercontent.com/assets/807966/8348195/1aa0ba56-1b4b-11e5-93f2-cfdd1f1ea864.png)\n\nFails after the fetch with `Failure(\"Git_memory.write_pack\")`. Same behaviour with 1.5.0.\n### 1.4.11, Git_unix.FS, fetch ~unpack:true\n\n![opam-ci-unpack-141](https://cloud.githubusercontent.com/assets/807966/8348297/bc525886-1b4c-11e5-88c0-fe9d746d49d4.png)\n\nFirst peak is the fetch. Then the mountain is the unpacking process: seems there is stuff that's held until the end and should be cleaned up. The number of open FDs may also be of concern.\nThis also doesn't seem to change much between 1.4.11 and 1.5.0\n\nAlso, after unpacking I couldn't find the commit-hash I was looking for and the program aborted.\n\n---\n\nThe last two may deserve their own issues.\nTo reproduce, you may use the [`testing-git` branch of opam-ci](https://github.com/AltGr/opam-ci/tree/testing-git) which disables normal behaviour and reporting and justs runs a batch fetch. You'll need to pin opam-lib to its latest dev. The function causing the huge virtual memory allocations is [RepoGit.opam_files](https://github.com/AltGr/opam-ci/blob/c0b5ec19365afa328fa3004fbb46da282f12cb9a/opam_ci.ml#L153-L184), which just recursively reads the directory structure at a given commit.\n\nAlso, my tests on 1.4.11 actually include a backport of some of the 1.5.0 branch, most importantly, latest cohttp compatibility. See https://github.com/AltGr/ocaml-git/tree/1.5.0-partial-revert\n\nHope this helps.\n"},"comment":{"url":"https://api.github.com/repos/mirage/ocaml-git/issues/comments/433607521","html_url":"https://github.com/mirage/ocaml-git/issues/90#issuecomment-433607521","issue_url":"https://api.github.com/repos/mirage/ocaml-git/issues/90","id":433607521,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMzYwNzUyMQ==","user":{"login":"samoht","id":103693,"node_id":"MDQ6VXNlcjEwMzY5Mw==","avatar_url":"https://avatars0.githubusercontent.com/u/103693?v=4","gravatar_id":"","url":"https://api.github.com/users/samoht","html_url":"https://github.com/samoht","followers_url":"https://api.github.com/users/samoht/followers","following_url":"https://api.github.com/users/samoht/following{/other_user}","gists_url":"https://api.github.com/users/samoht/gists{/gist_id}","starred_url":"https://api.github.com/users/samoht/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samoht/subscriptions","organizations_url":"https://api.github.com/users/samoht/orgs","repos_url":"https://api.github.com/users/samoht/repos","events_url":"https://api.github.com/users/samoht/events{/privacy}","received_events_url":"https://api.github.com/users/samoht/received_events","type":"User","site_admin":false},"created_at":"2018-10-27T10:02:54Z","updated_at":"2018-10-27T10:02:54Z","author_association":"MEMBER","body":"Should be fixed in ocaml-git 2.0"}}
{"actor":{"display_login":"samoht"}}