{"push_id":1153401209,"size":44,"distinct_size":1,"ref":"refs/heads/magicharvest","head":"d18f3d8b98a37dda3a39d8c954334d9ce5b422c4","before":"423f3e2b483d9bf5ce3ab4a74de404f4ab225bb5","commits":[{"sha":"8e3dd3e6f04f332d72734d152738f672e9e51bc5","author":{"name":"Henrik Vesterinen","email":"b89c27a5aefd25e94a5ada36abb4d4d0ac08c083@cern.ch"},"message":"dateutils: improvements to date getting\n\n* New files: dateutils.py, test_dateutils.py\n\n* Modified files:  base_spider, elsevier_spider, test_base\n\n* Adds `format_date` and `format_year` functions.\n\n* Moves the date formatting completely to loader. Year can be still\ncalled in the spider with `format_year`.\n\n* The file structure of these new functions should be revised.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/8e3dd3e6f04f332d72734d152738f672e9e51bc5"},{"sha":"0e57cd9c42e56bf966e8317c997760eaf9e9234b","author":{"name":"Henrik Vesterinen","email":"b89c27a5aefd25e94a5ada36abb4d4d0ac08c083@cern.ch"},"message":"elsevier_spider: improve reference year getting\n\n* Adds nicely formatted year strings.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/0e57cd9c42e56bf966e8317c997760eaf9e9234b"},{"sha":"c3742d563e855d80e90435b565123d7404180771","author":{"name":"Henrik Vesterinen","email":"b89c27a5aefd25e94a5ada36abb4d4d0ac08c083@cern.ch"},"message":"base_spider: improvements in testing\n\n* Adds a couple of tests.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/c3742d563e855d80e90435b565123d7404180771"},{"sha":"7d80e49efa6524eda8fcb246724fba8de22dcf02","author":{"name":"Henrik Vesterinen","email":"b89c27a5aefd25e94a5ada36abb4d4d0ac08c083@cern.ch"},"message":"elsevier_spider: improvements\n\n* Moves `range_as_string` to utils.py and adds tests.\n\n* Makes `parse_node` a bit more clear.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/7d80e49efa6524eda8fcb246724fba8de22dcf02"},{"sha":"a227dca8afd7a1c14f680297323e96ac508d6af5","author":{"name":"Jan Åge Lavik","email":"7b77ade7cc4aa89c05d0dfe2e8ef162fdc1b0196@gmail.com"},"message":"Merge pull request #16 from bittirousku/fix_date_published\n\ndateutils: improvements to date getting [WIP]","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/a227dca8afd7a1c14f680297323e96ac508d6af5"},{"sha":"57b2e6b5dd8c4fb793c4d0db2af200ad14f92840","author":{"name":"Henrik Vesterinen","email":"b89c27a5aefd25e94a5ada36abb4d4d0ac08c083@cern.ch"},"message":"spiders: new Brown University spider\n\n* Adds also support for new date format in dateutils.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/57b2e6b5dd8c4fb793c4d0db2af200ad14f92840"},{"sha":"862648718ef07ffcd18f7d3ff7f0b81392d2207e","author":{"name":"Jan Aage Lavik","email":"bb473c4ab1980b7da375d483a8fd22e73f2b5a4e@cern.ch"},"message":"docs: fix sample command\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/862648718ef07ffcd18f7d3ff7f0b81392d2207e"},{"sha":"397a729a98f00e10d0dc0570e4b10e2edcf76a15","author":{"name":"Jan Aage Lavik","email":"bb473c4ab1980b7da375d483a8fd22e73f2b5a4e@cern.ch"},"message":"tests: fake remote request in test_base.py\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/397a729a98f00e10d0dc0570e4b10e2edcf76a15"},{"sha":"a9cc09655e7d7c7ea895a249e6cca2bb960cc4e8","author":{"name":"Jan Åge Lavik","email":"7b77ade7cc4aa89c05d0dfe2e8ef162fdc1b0196@gmail.com"},"message":"Merge pull request #17 from bittirousku/brownharvest\n\nspiders: new Brown University spider","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/a9cc09655e7d7c7ea895a249e6cca2bb960cc4e8"},{"sha":"1e4e69c108f549803cc97bdf87870d7c9595802f","author":{"name":"Jan Åge Lavik","email":"7b77ade7cc4aa89c05d0dfe2e8ef162fdc1b0196@gmail.com"},"message":"Merge pull request #18 from jalavik/base-test-url-fix\n\ntests: fake remote request in test_base.py","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/1e4e69c108f549803cc97bdf87870d7c9595802f"},{"sha":"9f348e5ad57a28a0695fe83ea75d1f88b1f6b97d","author":{"name":"Henrik Vesterinen","email":"b89c27a5aefd25e94a5ada36abb4d4d0ac08c083@cern.ch"},"message":"spiders: new T2K spider\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/9f348e5ad57a28a0695fe83ea75d1f88b1f6b97d"},{"sha":"854d6b7a5f4863191ed31bbcb2238a43febac52d","author":{"name":"Henrik Vesterinen","email":"b89c27a5aefd25e94a5ada36abb4d4d0ac08c083@cern.ch"},"message":"inputs: minor fixes\n\n* Adds exception handling to `remove_attributes_from_tags`.\n\n* Adds '\\r' removal to `clean_whitespace_characters`.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/854d6b7a5f4863191ed31bbcb2238a43febac52d"},{"sha":"fdeee3adff078d747fce1e84ec8167f8ef9f5769","author":{"name":"Henrik Vesterinen","email":"b89c27a5aefd25e94a5ada36abb4d4d0ac08c083@cern.ch"},"message":"t2k_spider: tests working\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/fdeee3adff078d747fce1e84ec8167f8ef9f5769"},{"sha":"943623051f35e84edbf2adb538e07df40d661faa","author":{"name":"Kirsten Sachs","email":"66e2341b7ef1d723a59499f4072eebdd1af9550c@desy.de"},"message":"global: central parsing of authors and language\n\n* Adds parse_authors to inputs, raw information is filled in spiders.\n\n* Adds language translation to inputs and mappings\n\n* Modifies affected spiders accordingly\n\n* Modifies tests.\n\n* Checks that all records have collection 'HEP'\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>\nSigned-off-by: Kirsten Sachs <kirsten.sachs@desy.de>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/943623051f35e84edbf2adb538e07df40d661faa"},{"sha":"a460632c83b505d399c9fd9e7e8363d91f86608a","author":{"name":"Kirsten Sachs","email":"66e2341b7ef1d723a59499f4072eebdd1af9550c@desy.de"},"message":"spiders: new PoS spider\n\n* Adds new spider for PoS harvesting, including tests.\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>\nSigned-off-by: Kirsten Sachs <kirsten.sachs@desy.de>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/a460632c83b505d399c9fd9e7e8363d91f86608a"},{"sha":"f5c7964d2e2c6ea5f3d446419ed271c0d4033a51","author":{"name":"Jan Aage Lavik","email":"bb473c4ab1980b7da375d483a8fd22e73f2b5a4e@cern.ch"},"message":"pos_spider: update PDF scraping\n\n* Updates PDF scraping to be fully dynamic.\n\n* Updates PoS tests and adds a new test file for input\n  processors.\n\n* URLs in `urls.url` are now normalized during item loading.\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/f5c7964d2e2c6ea5f3d446419ed271c0d4033a51"},{"sha":"e5f587dbb44601ab46dfa20eff9907e48ca8e35a","author":{"name":"Jan Åge Lavik","email":"7b77ade7cc4aa89c05d0dfe2e8ef162fdc1b0196@gmail.com"},"message":"Merge pull request #22 from jalavik/pos-fixes","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/e5f587dbb44601ab46dfa20eff9907e48ca8e35a"},{"sha":"788536874606131d02e388bf4a8b387320118b2c","author":{"name":"Jan Aage Lavik","email":"bb473c4ab1980b7da375d483a8fd22e73f2b5a4e@cern.ch"},"message":"pipelines: INSPIRE push update\n\n* Updates INSPIRE push mechanism to use Celery's send_task\n  API instead of flower HTTP API.\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/788536874606131d02e388bf4a8b387320118b2c"},{"sha":"7122fd51901de8f94a767dece85a986d19d7d312","author":{"name":"Jan Aage Lavik","email":"bb473c4ab1980b7da375d483a8fd22e73f2b5a4e@cern.ch"},"message":"pipelines: make valid JSON output\n\n* Fixes an issue where the JSONWriterPipeline was not generating\n  valid JSON. (closes #5)\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/7122fd51901de8f94a767dece85a986d19d7d312"},{"sha":"e81a6b378022a45e8f3d04260ca987a5a3c7ee3a","author":{"name":"Jan Aage Lavik","email":"bb473c4ab1980b7da375d483a8fd22e73f2b5a4e@cern.ch"},"message":"settings: style update","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/e81a6b378022a45e8f3d04260ca987a5a3c7ee3a"}]}
{"actor":{"display_login":"bittirousku"}}