{"action":"created","issue":{"url":"https://api.github.com/repos/Rochester-NRT/AlphaGo/issues/64","repository_url":"https://api.github.com/repos/Rochester-NRT/AlphaGo","labels_url":"https://api.github.com/repos/Rochester-NRT/AlphaGo/issues/64/labels{/name}","comments_url":"https://api.github.com/repos/Rochester-NRT/AlphaGo/issues/64/comments","events_url":"https://api.github.com/repos/Rochester-NRT/AlphaGo/issues/64/events","html_url":"https://github.com/Rochester-NRT/AlphaGo/issues/64","id":142225350,"number":64,"title":"Managing our data","user":{"login":"ttrine","id":6808821,"avatar_url":"https://avatars.githubusercontent.com/u/6808821?v=3","gravatar_id":"","url":"https://api.github.com/users/ttrine","html_url":"https://github.com/ttrine","followers_url":"https://api.github.com/users/ttrine/followers","following_url":"https://api.github.com/users/ttrine/following{/other_user}","gists_url":"https://api.github.com/users/ttrine/gists{/gist_id}","starred_url":"https://api.github.com/users/ttrine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ttrine/subscriptions","organizations_url":"https://api.github.com/users/ttrine/orgs","repos_url":"https://api.github.com/users/ttrine/repos","events_url":"https://api.github.com/users/ttrine/events{/privacy}","received_events_url":"https://api.github.com/users/ttrine/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/Rochester-NRT/AlphaGo/labels/discussion","name":"discussion","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":10,"created_at":"2016-03-21T00:17:42Z","updated_at":"2016-03-25T21:01:41Z","closed_at":null,"body":"We need a way to ensure that our data is organized, reliable, and consistent. This will become especially important as we begin training models. Right now I have a folder on an external hard drive that looks like this:\r\n- models\r\n- professional games\r\n    - <.sgf games>\r\n- self play\r\n- training samples\r\n    - 46x19x19\r\n        - <.pkl training samples>\r\n\r\nAnd that's pretty much it. To kick off the discussion, here's what I'm wondering about:\r\n- Where can we host our data?\r\n- Can we version control it?\r\n    - If not, how can we keep things consistent across all contributors?\r\n- Should we organize professional games by their database of origin? Or lump them all together in a flat directory structure?"},"comment":{"url":"https://api.github.com/repos/Rochester-NRT/AlphaGo/issues/comments/201500203","html_url":"https://github.com/Rochester-NRT/AlphaGo/issues/64#issuecomment-201500203","issue_url":"https://api.github.com/repos/Rochester-NRT/AlphaGo/issues/64","id":201500203,"user":{"login":"thouis","id":473043,"avatar_url":"https://avatars.githubusercontent.com/u/473043?v=3","gravatar_id":"","url":"https://api.github.com/users/thouis","html_url":"https://github.com/thouis","followers_url":"https://api.github.com/users/thouis/followers","following_url":"https://api.github.com/users/thouis/following{/other_user}","gists_url":"https://api.github.com/users/thouis/gists{/gist_id}","starred_url":"https://api.github.com/users/thouis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/thouis/subscriptions","organizations_url":"https://api.github.com/users/thouis/orgs","repos_url":"https://api.github.com/users/thouis/repos","events_url":"https://api.github.com/users/thouis/events{/privacy}","received_events_url":"https://api.github.com/users/thouis/received_events","type":"User","site_admin":false},"created_at":"2016-03-25T21:01:41Z","updated_at":"2016-03-25T21:01:41Z","body":"I've temporarily put a subset at \r\nhttps://s3.amazonaws.com/FuegoFeatures/kgs-19-2015-05-new.hdf5\r\nhttps://s3.amazonaws.com/FuegoFeatures/KGS-2015_05-19-554-.tar.bz2\r\n\r\nBefore I try to share the whole thing, it would be good to make sure you can work with it (and there aren't any obvious bugs).\r\n\r\nIt doesn't seem to be compressing very well.  I'm not sure why that is, yet, but will look into it."}}