{"action":"created","issue":{"url":"https://api.github.com/repos/OceansAus/ACCESS-OM2-1-025-010deg-report/issues/8","repository_url":"https://api.github.com/repos/OceansAus/ACCESS-OM2-1-025-010deg-report","labels_url":"https://api.github.com/repos/OceansAus/ACCESS-OM2-1-025-010deg-report/issues/8/labels{/name}","comments_url":"https://api.github.com/repos/OceansAus/ACCESS-OM2-1-025-010deg-report/issues/8/comments","events_url":"https://api.github.com/repos/OceansAus/ACCESS-OM2-1-025-010deg-report/issues/8/events","html_url":"https://github.com/OceansAus/ACCESS-OM2-1-025-010deg-report/issues/8","id":334769578,"node_id":"MDU6SXNzdWUzMzQ3Njk1Nzg=","number":8,"title":"Re-think where image files are stored - git repo will get impracticably huge","user":{"login":"aekiss","id":31054815,"node_id":"MDQ6VXNlcjMxMDU0ODE1","avatar_url":"https://avatars3.githubusercontent.com/u/31054815?v=4","gravatar_id":"","url":"https://api.github.com/users/aekiss","html_url":"https://github.com/aekiss","followers_url":"https://api.github.com/users/aekiss/followers","following_url":"https://api.github.com/users/aekiss/following{/other_user}","gists_url":"https://api.github.com/users/aekiss/gists{/gist_id}","starred_url":"https://api.github.com/users/aekiss/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aekiss/subscriptions","organizations_url":"https://api.github.com/users/aekiss/orgs","repos_url":"https://api.github.com/users/aekiss/repos","events_url":"https://api.github.com/users/aekiss/events{/privacy}","received_events_url":"https://api.github.com/users/aekiss/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2018-06-22T07:09:08Z","updated_at":"2018-07-04T10:21:34Z","closed_at":null,"author_association":"COLLABORATOR","body":"This git / github - based approach will require a rethink.\r\n\r\nCurrently the git repo stores .pdf files generated by the Jupyter notebooks so that people can compile the .tex file with the latest images when they pull or clone. Until this afternoon the repo also stored the report .pdf to make it easy to look at it without running latex.\r\n\r\nHowever, storing .pdfs in git causes the repository size to blow out because git cannot diff these binary files, so any small change to an image or the report .pdf will result in a new copy of the entire file being saved to the file history in the repo (in contrast to text files, for which only the changes are saved). As a result the repo is already large (over 200Mb) and could grow rapidly as contributions are made. There is a quota of 2Gb on VDI home directories, which we will hit quite quickly (though the notebooks could be run from hh5 if needed).\r\n\r\nSo I think it's clear that the generated .pdf files should not be in the repo. \r\nI've removed ACCESS-OM2-1-025-010deg.pdf as it is readily re-generated. \r\nBut **I don't know what to do about the image files**. They are needed to run latex on the .tex file, but can only be generated on the VDI (and this is really slow). So we need a way to store the latest image files to share, but we can't use standard git to do it. We don't need to track the file history for images, as we are keeping the notebook histories so can re-generate old images if needed.\r\n\r\n**Possible avenues:**\r\n1. [git-lfs](https://git-lfs.github.com) (large file support) - [supported by github](https://help.github.com/categories/managing-large-files/) but with [impracticably small bandwidth limits](https://help.github.com/articles/about-storage-and-bandwidth-usage/) on the free plan and has had [bad press](https://medium.com/@megastep/github-s-large-file-storage-is-no-panacea-for-open-source-quite-the-opposite-12c0e16a9a91)\r\n2. [git-annex](https://git-annex.branchable.com) - but where to store the annexed files? Needs to be somewhere that all authors have read & write access\r\n3. just store the images on a shared dropbox or CloudStor and update by hand or via a script (can we write to dropbox or CloudStor from the VDI?)\r\n4. ditching the whole github idea and use a shared dropbox for everything??\r\n5. anything else?\r\n\r\n**Dead ends:**\r\n- it seems shallow clones won't help - they don't allow push\r\n\r\nI don't want a solution that's onerous or confusing. "},"comment":{"url":"https://api.github.com/repos/OceansAus/ACCESS-OM2-1-025-010deg-report/issues/comments/402436056","html_url":"https://github.com/OceansAus/ACCESS-OM2-1-025-010deg-report/issues/8#issuecomment-402436056","issue_url":"https://api.github.com/repos/OceansAus/ACCESS-OM2-1-025-010deg-report/issues/8","id":402436056,"node_id":"MDEyOklzc3VlQ29tbWVudDQwMjQzNjA1Ng==","user":{"login":"AndyHoggANU","id":26753100,"node_id":"MDQ6VXNlcjI2NzUzMTAw","avatar_url":"https://avatars1.githubusercontent.com/u/26753100?v=4","gravatar_id":"","url":"https://api.github.com/users/AndyHoggANU","html_url":"https://github.com/AndyHoggANU","followers_url":"https://api.github.com/users/AndyHoggANU/followers","following_url":"https://api.github.com/users/AndyHoggANU/following{/other_user}","gists_url":"https://api.github.com/users/AndyHoggANU/gists{/gist_id}","starred_url":"https://api.github.com/users/AndyHoggANU/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AndyHoggANU/subscriptions","organizations_url":"https://api.github.com/users/AndyHoggANU/orgs","repos_url":"https://api.github.com/users/AndyHoggANU/repos","events_url":"https://api.github.com/users/AndyHoggANU/events{/privacy}","received_events_url":"https://api.github.com/users/AndyHoggANU/received_events","type":"User","site_admin":false},"created_at":"2018-07-04T10:21:34Z","updated_at":"2018-07-04T10:21:34Z","author_association":"COLLABORATOR","body":"OK, gave this a crack tonight. New files did copy ... but old ones were retained even if deleted.\r\n\r\nAlso, there were still some permission errors on old files:\r\n```\r\n[amh157@vdi-n22 ACCESS-OM2-1-025-010deg-report]$ ./pushfigs.sh \r\nUploading figures to /g/data3/hh5/tmp/cosima/access-om2-report-figures/versions/2018-07-04_101545Z_amh157_0e48489 ...\r\nsending incremental file list\r\nfigures/\r\nfigures/exptdata.py\r\nfigures/.ipynb_checkpoints/\r\nfigures/BarotropicStreamfunction/\r\nrsync: failed to set permissions on \"/g/data3/hh5/tmp/cosima/access-om2-report-figures/versions/2018-07-04_101545Z_amh157_0e48489/figures/BarotropicStreamfunction/._ACC_transport_2.pdf\": Operation not permitted (1)\r\nrsync: failed to set permissions on \"/g/data3/hh5/tmp/cosima/access-om2-report-figures/versions/2018-07-04_101545Z_amh157_0e48489/figures/BarotropicStreamfunction/._gyre_transport.pdf\": Operation not permitted (1)\r\nfigures/BarotropicStreamfunction/BarotropicStreamfunction.ipynb\r\nfigures/__pycache__/\r\nfigures/bathymetry/\r\nfigures/bathymetry/.ipynb_checkpoints/\r\nfigures/ice_validation/ice_validation.ipynb\r\nrsync: failed to set permissions on \"/g/data3/hh5/tmp/cosima/access-om2-report-figures/versions/2018-07-04_101545Z_amh157_0e48489/figures/ice_validation/ice_volume_categories_cumulative.pdf\": Operation not permitted (1)\r\nfigures/ice_validation/.ipynb_checkpoints/\r\nfigures/overturning_circulation/\r\nfigures/overturning_circulation/GlobalOverturningStreamfunction.ipynb\r\nfigures/overturning_circulation/mean_overturning.pdf\r\nfigures/overturning_circulation/mean_overturning_new.pdf\r\nfigures/overturning_circulation/.ipynb_checkpoints/GlobalOverturningStreamfunction-checkpoint.ipynb\r\n\r\nsent 8599114 bytes  received 189 bytes  17198606.00 bytes/sec\r\ntotal size is 9460242  speedup is 1.10\r\nrsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1039) [sender=3.0.6]\r\nUpload failed.\r\n```\r\nFinally, the link from latest didn't seem to update to the version I sent ... should it have?"}}
{"actor":{"display_login":"AndyHoggANU"}}