{"push_id":3022942383,"size":2,"distinct_size":2,"ref":"refs/heads/cs_1103","head":"405ff3b00cc05cce4be741d59b1054b749cac7f9","before":"e210651d4935619d16a8713d6cd9317aa33f48a0","commits":[{"sha":"1f82dc41cd75e79d2557573071797c442dc83885","author":{"name":"Yang Pan","email":"2743866a05ca69db48f3d4d3e220ab533f09c0ae@google.com"},"message":"[code search] add flag to wait till code search job finish (#306)\n\n* add flag to wait till job finish\n\n* wait till -> wait until","distinct":true,"url":"https://api.github.com/repos/jlewi/examples/commits/1f82dc41cd75e79d2557573071797c442dc83885"},{"sha":"405ff3b00cc05cce4be741d59b1054b749cac7f9","author":{"name":"Jeremy Lewi","email":"b3f594e10a9edcf5413cf1190121d45078c62290@lewi.us"},"message":"Fix performance of dataflow preprocessing job.\n\n* Fix #300; Dataflow job for preprocessing is really slow.\n\n  * The problem is we are loading the spacy tokenization model on every\n    invocation of the tokenization function and this is really expensive.\n  * We should be doing this once per module import.\n\n* After fixing this issue; the job completed in approximately 20 minutes using\n  5 workers.\n\n  * We can process all 1.3 million records in ~ 20 minutes (elapsed time) using 5 32 CPU workers and about 1 hour of CPU time altogether.\n\n* Add options to the Dataflow job to read from files as opposed to BigQuery\n  and to skip BigQuery writes. This is useful for testing.\n\n* Add a \"unittest\" that verifies the Dataflow preprocessing job can run\n  successfully using the DirectRunner.\n\n* Update the Docker image and a ksonnet component for a K8s job that\n  can be used to submit the Dataflow job.\n\n* Fix #299; Add logging to the Dataflow preprocessing job to indicate that\n  a Dataflow job was submitted.\n\n* Add an option to the preprocessing Dataflow job to read an entire\n  BigQuery table as the input rather than running a query to get the input.\n  This is useful in the case where the user wants to run a different\n  query to select the repo paths and contents to process and write them\n  to some table to be processed by the Dataflow job.","distinct":true,"url":"https://api.github.com/repos/jlewi/examples/commits/405ff3b00cc05cce4be741d59b1054b749cac7f9"}]}
{"actor":{"display_login":"jlewi"}}