{"push_id":926051013,"size":13,"distinct_size":0,"ref":"refs/heads/master","head":"a41776ccb652aad59373c620ccca8f037450ca6c","before":"bc955d4686d22d1907a3f16776f390cb9d29bb14","commits":[{"sha":"6b30b380a2ce6eca8b9785b129bbb09d62835cb7","author":{"email":"8d55e006a1b29770f275a5c59a7341f17f8e69d5@cern.ch","name":"Jan Aage Lavik"},"message":"inputs: fix affiliation input\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/6b30b380a2ce6eca8b9785b129bbb09d62835cb7"},{"sha":"cb26cc04a27cda3a281c683eddf9835554f05677","author":{"email":"ddbde9b76c26601789359358950ea8529deb176c@cern.ch","name":"Ilias Koutsakis"},"message":"spider: APS spider\n\n* Adds extraction of metadata based on the JSON response of the\n  APS API.\n\n* Adds documentation and tests.\n\n* Adds a couple of specific util functions, and tests for them.\n\n* Adds 2 json files that are APS specific, for testing purposes.\n\nSigned-off-by: Ilias Koutsakis <ilias.koutsakis@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/cb26cc04a27cda3a281c683eddf9835554f05677"},{"sha":"10592b32066b051db6f419499d4f69805445a679","author":{"email":"f39b8b8d43129a101811b941b44fac0079a79762@cern.ch","name":"Henrik Vesterinen"},"message":"spiders: new BASE spider\n\n* Adds indendation to JSON output in pipelines.\n\n* Adds ./eggs directory to .gitignore.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/10592b32066b051db6f419499d4f69805445a679"},{"sha":"59da8b93b6b485642115df441556542f3138e964","author":{"email":"f39b8b8d43129a101811b941b44fac0079a79762@cern.ch","name":"Henrik Vesterinen"},"message":"spiders: new alpha spider\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/59da8b93b6b485642115df441556542f3138e964"},{"sha":"7baf6d12fb4c9546c2dafe36b1075f29a3f1fa70","author":{"email":"8d55e006a1b29770f275a5c59a7341f17f8e69d5@cern.ch","name":"Jan Aage Lavik"},"message":"base_spider: enhancements to flow\n\n* Enhances the harvesting flow of the base spider a bit to\n  be more streamlined and improves the MIME type detection of\n  potential PDFs from discovered URLs.\n\n* Updates to usage of namespaces for xpath expressions in the base\n  spider.\n\n* Moves some generic functions from base spider to utils.\n\n* Changes `url` to `urls` in `HEPRecord`.\n\n* Code style and PEP8 updates.\n\n* Updates to tests.\n\nCo-authored-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/7baf6d12fb4c9546c2dafe36b1075f29a3f1fa70"},{"sha":"6b10483c9615d8f022704f8a96ced113eaec0670","author":{"email":"f39b8b8d43129a101811b941b44fac0079a79762@cern.ch","name":"Henrik Vesterinen"},"message":"spiders: Fix direct link finding and clean some code\n\n* Fixes direct link finding in base_spider and does some cleaning.\n  Works quite well, but it's returning  only 974/1000 results now,\n  so requires more testing and logging. Fixes also testing and author\n  getting. PDF link testing is still not available, however.\n\n* Does some cleaning in alpha_spider.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/6b10483c9615d8f022704f8a96ced113eaec0670"},{"sha":"04fb890ebbe1152bff40c6524067f4cf89b2c396","author":{"email":"8d55e006a1b29770f275a5c59a7341f17f8e69d5@cern.ch","name":"Jan Aage Lavik"},"message":"base_spider: avoid duplicate requests\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/04fb890ebbe1152bff40c6524067f4cf89b2c396"},{"sha":"560e174441135a0985f6389abbbf89077ba8d357","author":{"email":"8d55e006a1b29770f275a5c59a7341f17f8e69d5@cern.ch","name":"Jan Aage Lavik"},"message":"tests: update base spider test\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/560e174441135a0985f6389abbbf89077ba8d357"},{"sha":"3ce51e2ab1edb2fd83c56c78ccf23e531a3ac25f","author":{"email":"8d55e006a1b29770f275a5c59a7341f17f8e69d5@cern.ch","name":"Jan Aage Lavik"},"message":"base_spider: remove unused target_folder\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/3ce51e2ab1edb2fd83c56c78ccf23e531a3ac25f"},{"sha":"4f458255b11cad9e731bc2574bbfc60f33d65f7d","author":{"email":"8d55e006a1b29770f275a5c59a7341f17f8e69d5@cern.ch","name":"Jan Aage Lavik"},"message":"alpha_spider: enable dynamic requests handling\n\nSigned-off-by: Jan Aage Lavik <jan.age.lavik@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/4f458255b11cad9e731bc2574bbfc60f33d65f7d"},{"sha":"251fc6624d27bf414731a3877236897d696b52ec","author":{"email":"f39b8b8d43129a101811b941b44fac0079a79762@cern.ch","name":"Henrik Vesterinen"},"message":"spiders: add urls to alpha_spider and clean some code\n\n* Does some cleaning and formats comments in base_spider.\n\n* Does some cleaning in alpha_spider and adds functionality\n  to get record splash page urls.\n\n* Testing in alpha_spider now refers to correct test file.\n\n* Removes old comments in pipelines regarding valid JSON.\n\n* Adds htm test files to MANIFEST.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/251fc6624d27bf414731a3877236897d696b52ec"},{"sha":"d7c7bc267d5d16226ce1607523c2e4ac3d7605d0","author":{"email":"f39b8b8d43129a101811b941b44fac0079a79762@cern.ch","name":"Henrik Vesterinen"},"message":"spiders: improvements in some functions and in testing\n\n* Adds split_fullname function in utils the ability to deal with\n  multiple types of author strings, with or without comma separation.\n  Also now accepts surnames in the first or last position.\n\n* Adds tests for split_fullname, parse_domain and\n  get_mime_type in test_utils.\n\n* Adds items for collections (THESIS) in alpha_spider and\n  base_spider.\n\n* Removes alpha_spider's own split_fullname function and\n  combines author parsing functions parse_author_data and\n  get_authors.\n\n* Adds more tests to base_spider.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/d7c7bc267d5d16226ce1607523c2e4ac3d7605d0"},{"sha":"a41776ccb652aad59373c620ccca8f037450ca6c","author":{"email":"f39b8b8d43129a101811b941b44fac0079a79762@cern.ch","name":"Henrik Vesterinen"},"message":"alpha_spider: improvements\n\n* Moves has_numbers in alpha_spider to utils.py, moves \"PhD\" testing\n  upward so will be more efficient.\n\n* Adds test in test_utils for situations when Content-Type=None.\n\nSigned-off-by: Henrik Vesterinen <henrik.vesterinen@cern.ch>","distinct":false,"url":"https://api.github.com/repos/bittirousku/hepcrawl/commits/a41776ccb652aad59373c620ccca8f037450ca6c"}]}