{"action":"created","issue":{"url":"https://api.github.com/repos/sciencefair-land/datasource-requests/issues/1","repository_url":"https://api.github.com/repos/sciencefair-land/datasource-requests","labels_url":"https://api.github.com/repos/sciencefair-land/datasource-requests/issues/1/labels{/name}","comments_url":"https://api.github.com/repos/sciencefair-land/datasource-requests/issues/1/comments","events_url":"https://api.github.com/repos/sciencefair-land/datasource-requests/issues/1/events","html_url":"https://github.com/sciencefair-land/datasource-requests/issues/1","id":240506571,"node_id":"MDU6SXNzdWUyNDA1MDY1NzE=","number":1,"title":"PubMed Central","user":{"login":"blahah","id":836040,"node_id":"MDQ6VXNlcjgzNjA0MA==","avatar_url":"https://avatars3.githubusercontent.com/u/836040?v=4","gravatar_id":"","url":"https://api.github.com/users/blahah","html_url":"https://github.com/blahah","followers_url":"https://api.github.com/users/blahah/followers","following_url":"https://api.github.com/users/blahah/following{/other_user}","gists_url":"https://api.github.com/users/blahah/gists{/gist_id}","starred_url":"https://api.github.com/users/blahah/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/blahah/subscriptions","organizations_url":"https://api.github.com/users/blahah/orgs","repos_url":"https://api.github.com/users/blahah/repos","events_url":"https://api.github.com/users/blahah/events{/privacy}","received_events_url":"https://api.github.com/users/blahah/received_events","type":"User","site_admin":false},"labels":[{"id":640542995,"node_id":"MDU6TGFiZWw2NDA1NDI5OTU=","url":"https://api.github.com/repos/sciencefair-land/datasource-requests/labels/articles","name":"articles","color":"0e8a16","default":false},{"id":641688612,"node_id":"MDU6TGFiZWw2NDE2ODg2MTI=","url":"https://api.github.com/repos/sciencefair-land/datasource-requests/labels/in-progress","name":"in-progress","color":"cc317c","default":false},{"id":640543581,"node_id":"MDU6TGFiZWw2NDA1NDM1ODE=","url":"https://api.github.com/repos/sciencefair-land/datasource-requests/labels/many-publications","name":"many-publications","color":"fbca04","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":7,"created_at":"2017-07-04T22:48:00Z","updated_at":"2018-12-07T01:07:59Z","closed_at":null,"author_association":"MEMBER","body":"PubMed Central provides JATS XML for 4.4 million articles. Not all of these are OpenAccess, so we will start with the ones that are definitely OK to distribute.\r\n\r\nI propose to create a live-updating PMC datasource.\r\n\r\nIt will also be broken down into one datasource per journal. This will be achieved by generating a separate `meta` feed for each journal, but sharing the `articles` feed for PMC between all the sources.\r\n\r\nThere are 2017 fully participating journals, 4299 selectively depositing journals, and 414 journals that deposit all NIH-funded works). Here's a list of journals: https://www.ncbi.nlm.nih.gov/pmc/journals/.\r\n\r\n## Methodology (main source)\r\n\r\nTo create and maintain the PMC main feed:\r\n\r\n- create a node module and CLI (at [sciencefair-land/pubmed-central](https://github.com/sciencefair-land/pubmed-central))\r\n- every day, download [this daily-updated CSV file listing all OA PMC articles](ftp://ftp.ncbi.nlm.nih.gov/pub/pmc/oa_file_list.csv)\r\n- take any entries not already in the feed, and:\r\n  - get the full article metadata from the PubMed API using [bionode-ncbi](https://github.com/bionode/bionode-ncbi)\r\n  - download the full article bundle from the FTP site (column 2 in the daily CSV file)\r\n- generate new entires in the full PMC feed\r\n\r\n## Methodology (sub-sources)\r\n\r\nTo allow creation of sub-feeds:\r\n\r\n- make an node module for filtering the main feed (only by journal at first)\r\n- generate a new feed from the filtered metadata\r\n- use the full PMC articles feed as the articles feed for the new meta source\r\n\r\nThis setup should ideally have a web interface for finding sub-feeds, or creating them on the fly."},"comment":{"url":"https://api.github.com/repos/sciencefair-land/datasource-requests/issues/comments/445087115","html_url":"https://github.com/sciencefair-land/datasource-requests/issues/1#issuecomment-445087115","issue_url":"https://api.github.com/repos/sciencefair-land/datasource-requests/issues/1","id":445087115,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0NTA4NzExNQ==","user":{"login":"SirRujak","id":5132268,"node_id":"MDQ6VXNlcjUxMzIyNjg=","avatar_url":"https://avatars2.githubusercontent.com/u/5132268?v=4","gravatar_id":"","url":"https://api.github.com/users/SirRujak","html_url":"https://github.com/SirRujak","followers_url":"https://api.github.com/users/SirRujak/followers","following_url":"https://api.github.com/users/SirRujak/following{/other_user}","gists_url":"https://api.github.com/users/SirRujak/gists{/gist_id}","starred_url":"https://api.github.com/users/SirRujak/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/SirRujak/subscriptions","organizations_url":"https://api.github.com/users/SirRujak/orgs","repos_url":"https://api.github.com/users/SirRujak/repos","events_url":"https://api.github.com/users/SirRujak/events{/privacy}","received_events_url":"https://api.github.com/users/SirRujak/received_events","type":"User","site_admin":false},"created_at":"2018-12-07T01:07:59Z","updated_at":"2018-12-07T01:07:59Z","author_association":"NONE","body":"@blahah I have joined the gitter! I have finally been able to fully import the metadata from my test PMC database. Their database isn't exactly what I would call \"clean\" and I definitely need to make some alterations as to how I find the date for papers. Other than that I am back on track as far as things go so hopefully I will have something for you guys to test if you like before long!\r\n\r\nI'm pretty well just developing on my branch at the moment due to me breaking things all the time, but I am happy to make pull requests whenever anyone wants. Also, as I am going I am putting the latest test versions on the edge and beta releases of the snap so anyone that wanted to could replicate my setup without too much trouble."}}
{"actor":{"display_login":"SirRujak"}}