{"push_id":1441655929,"size":24,"distinct_size":1,"ref":"refs/heads/develop","head":"37579b077f8260d9a3a0bf93a6ac2427622f9cc1","before":"76e4574892ec7210c5b8d6a8ae24d09fb1cebf49","commits":[{"sha":"ff3e8f1c7e2d251fd84e634b560ab97127d75a51","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"minimal validation scaffolding","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/ff3e8f1c7e2d251fd84e634b560ab97127d75a51"},{"sha":"6f173926754109d5defdd07d4c3b5443a09360af","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Merge branch 'develop' into validation-engine","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/6f173926754109d5defdd07d4c3b5443a09360af"},{"sha":"5729f251c0e6a95af7b44b6de44766066bbc292d","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Merge branch 'develop' into validation-engine","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/5729f251c0e6a95af7b44b6de44766066bbc292d"},{"sha":"0c27d53305424a400502cfad41d74182e3dcf8f9","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"cleanup, superficial rename","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/0c27d53305424a400502cfad41d74182e3dcf8f9"},{"sha":"99e1e6464d2ffa089aa066fff97cf50d211462ad","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Adds validator test + initial receipt validator\n\n> validator returns json, not plaintext\n> initial receipt validator that parses it as csv\n> some validation tests that currently all fail :)","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/99e1e6464d2ffa089aa066fff97cf50d211462ad"},{"sha":"0b845b65c7b7262496ec9ae99dc0dc390cca9719","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"tests pass; receipt format validates\n\nAlso added better scaffolding & actually delete test submissions oops","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/0b845b65c7b7262496ec9ae99dc0dc390cca9719"},{"sha":"62c3c0cf659ae4e8d7e3b9a35c9f251a9dedabf4","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Update Dockerfile with java for validator download\n\nSo that the validator can download files, it needs java for the\njava ucsc-download-client.\n\nSo (for now) base the docker on alpine with java 8 jre preinstalled.\nAlso adds accessToken and the java download client to the gitignore, and\nbeginning of the framework for the validator to download stuff.","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/62c3c0cf659ae4e8d7e3b9a35c9f251a9dedabf4"},{"sha":"8e8a9e54917055337866a37f10a9e31bf75f1c77","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"?? add __init__.py to maybe fix travis?","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/8e8a9e54917055337866a37f10a9e31bf75f1c77"},{"sha":"de97b67b3f87d32fdb0df47af7a1cf67df5f815e","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"whitespace nitpicking to placate flake8","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/de97b67b3f87d32fdb0df47af7a1cf67df5f815e"},{"sha":"658569cb42a9e19e703ccfa94672fcd735e79e33","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"rudimentary file download validator\n\npulls the metadata file ID out of the receipt and tries to download that.\nNote: doesn't do any cleanup right now.","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/658569cb42a9e19e703ccfa94672fcd735e79e33"},{"sha":"72d7aa3687a6dc3ff354a701e89e54128423b59b","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Merge branch 'develop' into validation-engine","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/72d7aa3687a6dc3ff354a701e89e54128423b59b"},{"sha":"67fddc8da3be4edf0e73c8cd21d265c3467f22c2","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"fix validator to use receipt not description","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/67fddc8da3be4edf0e73c8cd21d265c3467f22c2"},{"sha":"8bb2ddf73b4616c8dd31e86632961cf0d3292da5","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Validation engine checks for downloaded file\n\n> deletes it after confirming it's openable\n> also added some tests to write","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/8bb2ddf73b4616c8dd31e86632961cf0d3292da5"},{"sha":"f2c125e1eb0a51d82e30b558b53ffc3520b568a5","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Replaces java redwood client for download from aws\n\nAdds redwood_client_lite python module to download from aws\nvia the redwood server, without using the full java redwood client.\nValidation engine now uses this client to download the metadata.json.\n\nJava is therefore no longer needed in the Dockerfile","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/f2c125e1eb0a51d82e30b558b53ffc3520b568a5"},{"sha":"50e4087491bc361001f74eec5164de36e0b47689","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"additional cleanup of java stuff","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/50e4087491bc361001f74eec5164de36e0b47689"},{"sha":"b43abd907ebb0ba807adcba402b733b986cf1640","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Validator also dls file start & checks if nonempty\n\nEach item in receipt is now checked for 2 things:\n    - metadata.json : does it parse as valid json\n    - the file in the line : 1st 64 bytes (per client lite) are\n        downloaded and checked if they are empty","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/b43abd907ebb0ba807adcba402b733b986cf1640"},{"sha":"682776868307aba009cccfdcfe2f28f1770900b5","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"First pass at using uwsgi spooler\n\nNOTE: when using the 'body' key on the uwsgi spooler hash,\nit seg faults all over the place. haven't looked into why.\nneed to rearrange the code a bit so that we don't need to pass\nthe receipt in the key value store","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/682776868307aba009cccfdcfe2f28f1770900b5"},{"sha":"fec1af85abedda076b2fac35ee6fb2982d5aae05","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Per #9 - runs validation async when sub is 'put'\n\nUses uwsgi spooler. To test with `make debug`, need to use the commented-out\ncommand at the bottom.\n\nRearranged validation endpoint a bit to make this possible.\nSo validations run when a receipt is added (PUT) are asynchronous;\nvalidations run from the validation endpoint are synchronous.","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/fec1af85abedda076b2fac35ee6fb2982d5aae05"},{"sha":"1084daa2c472de16d058518b52ac5962deb0e89a","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"for #9 - validation classes to fns, error handling\n\nvalidation_engine.py\n > changes validation classes to functions with a shared signature\n    - changes indentation for a  bunch of them so it looks like a ton of code changed\n    but that's just git being confused\n> error handling - catches redwood server errors propagated from the redwood client lite\n\n> removes testing validations\n\n> allows module to be run standalone : python validation_engine.py receipt.tsv\n\nredwood client lite :\n    now catches errors associated with connecting to the redwood server\n    and reraises them as RedwoodServerError","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/1084daa2c472de16d058518b52ac5962deb0e89a"},{"sha":"bba09230abc320f81730b3f5f95114c9e90f85e8","author":{"name":"ETK","email":"4e6bd588519f3dfeef2a9fce300581d25b0f0b63@users.noreply.github.com"},"message":"Don't run validations in flask mode; add msg to db\n\nHighlights:\n    - new migration: Submission table now has a validation_message (text) field to store results of the validation,\n    since users will need to see why their validations failed.\n\n    - new endpoint /v0/validation - GET requests a validation. POST updates the db with validation results (yep! see details below)\n\nAll changes :\n\n- updates makefile to debug in uwsgi mode by default\n\n- adds validation_message col to Submission\n\n- can't validate synchronously in flask anymore; removed that endpoint\n\n- Removed validation tests (so they don't fail) until I fix them since they no longer run synchronously\n\n-- moved the uwsgi spooler endpoint to a new file, validation_runner.py\n\nnew endpoint: /v0/validation\nGET enqueues a validation\nPOST updates DB with the results of a validation\n\nDetails:\n    The spooler callback is now its own module. It needs to be able to update the database with the validation results when the validation is completed. However I didn't like it having independent access to the DB (without going via the flask app). So, instead, it sends a POST to the validation endpoint with the results of the validation.\n\nSo at the moment, yes, anyone can update a submission and claim that it has validated or failed validation. This is terrible and also pretty useful for testing.","distinct":false,"url":"https://api.github.com/repos/BD2KGenomics/dcc-spinnaker/commits/bba09230abc320f81730b3f5f95114c9e90f85e8"}]}
{"actor":{"display_login":"rcurrie"}}