{"push_id":2412699479,"size":1,"distinct_size":1,"ref":"refs/heads/master","head":"afd23642bc4865e255f76820bf3b033115e17f6b","before":"5570158e9359ecebe409723123a14343906abbb1","commits":[{"sha":"afd23642bc4865e255f76820bf3b033115e17f6b","author":{"name":"Andrea Frittoli","email":"dcde243900f3278566720469892b0c7560e71e53@gmail.com"},"message":"Move dstat files to a central cache\n\nThere's no need for every dataset to re-download dstat files.\nBesides dstat file often expire on the logs server, so it's important\nto preserve a local copy.\n\nMove the current folder structure:\n\n- dataset1\n   \\- raw\n       |- run_uuid1.csv.gz\n       \\- run_uuid1.json.gz\n- dataset2\n   \\- raw\n       |- run_uuid1.csv.gz\n       |- run_uuid1.json.gz\n       |- run_uuid2.csv.gz\n       \\- run_uuid2.json.gz\n\nTo a new folder structure:\n\n- .raw\n    |- run_uuid1.csv.gz\n    \\- run_uuid2.csv.gz\n- .metadata\n    |- run_uuid1.json.gz\n    \\- run_uuid2.json.gz\n- dataset1\n   \\- runs.json.gz (list of UUIDs)\n- dataset2\n   \\- runs.json.gz (list of UUIDs)\n\nThis is not backward compatible, so with the new code existing dataset\nwill fail to load properly unless files are moved around.\nFrom the data folder:\n\n$ mkdir .raw\n$ find . -name '*.csv.gz' -exec cp '{}' .raw \\;\n\nMetadata files can be re-built from the DB, this will happen\nautomatically if the files cannot be found.\n\nThis will also leave .*.gz files lying around in the metadata folders\nbut this is done on purpose so that migration can be checked before\nfiles are deleted. Deletion can be done with:\n\n$ find . -name raw -type d -exec rm -rf '{}' \\;","distinct":true,"url":"https://api.github.com/repos/mtreinish/ciml/commits/afd23642bc4865e255f76820bf3b033115e17f6b"}]}
{"actor":{"display_login":"afrittoli"}}