{"action":"closed","issue":{"url":"https://api.github.com/repos/opentrials/opentrials/issues/302","repository_url":"https://api.github.com/repos/opentrials/opentrials","labels_url":"https://api.github.com/repos/opentrials/opentrials/issues/302/labels{/name}","comments_url":"https://api.github.com/repos/opentrials/opentrials/issues/302/comments","events_url":"https://api.github.com/repos/opentrials/opentrials/issues/302/events","html_url":"https://github.com/opentrials/opentrials/issues/302","id":168286009,"number":302,"title":"Write processor for FDA's Approval History collector","user":{"login":"vitorbaptista","id":76945,"avatar_url":"https://avatars.githubusercontent.com/u/76945?v=3","gravatar_id":"","url":"https://api.github.com/users/vitorbaptista","html_url":"https://github.com/vitorbaptista","followers_url":"https://api.github.com/users/vitorbaptista/followers","following_url":"https://api.github.com/users/vitorbaptista/following{/other_user}","gists_url":"https://api.github.com/users/vitorbaptista/gists{/gist_id}","starred_url":"https://api.github.com/users/vitorbaptista/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vitorbaptista/subscriptions","organizations_url":"https://api.github.com/users/vitorbaptista/orgs","repos_url":"https://api.github.com/users/vitorbaptista/repos","events_url":"https://api.github.com/users/vitorbaptista/events{/privacy}","received_events_url":"https://api.github.com/users/vitorbaptista/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/opentrials/opentrials/labels/FDA","name":"FDA","color":"000000"},{"url":"https://api.github.com/repos/opentrials/opentrials/labels/Processors","name":"Processors","color":"1D2021"}],"state":"closed","locked":false,"assignee":{"login":"vitorbaptista","id":76945,"avatar_url":"https://avatars.githubusercontent.com/u/76945?v=3","gravatar_id":"","url":"https://api.github.com/users/vitorbaptista","html_url":"https://github.com/vitorbaptista","followers_url":"https://api.github.com/users/vitorbaptista/followers","following_url":"https://api.github.com/users/vitorbaptista/following{/other_user}","gists_url":"https://api.github.com/users/vitorbaptista/gists{/gist_id}","starred_url":"https://api.github.com/users/vitorbaptista/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vitorbaptista/subscriptions","organizations_url":"https://api.github.com/users/vitorbaptista/orgs","repos_url":"https://api.github.com/users/vitorbaptista/repos","events_url":"https://api.github.com/users/vitorbaptista/events{/privacy}","received_events_url":"https://api.github.com/users/vitorbaptista/received_events","type":"User","site_admin":false},"assignees":[{"login":"vitorbaptista","id":76945,"avatar_url":"https://avatars.githubusercontent.com/u/76945?v=3","gravatar_id":"","url":"https://api.github.com/users/vitorbaptista","html_url":"https://github.com/vitorbaptista","followers_url":"https://api.github.com/users/vitorbaptista/followers","following_url":"https://api.github.com/users/vitorbaptista/following{/other_user}","gists_url":"https://api.github.com/users/vitorbaptista/gists{/gist_id}","starred_url":"https://api.github.com/users/vitorbaptista/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vitorbaptista/subscriptions","organizations_url":"https://api.github.com/users/vitorbaptista/orgs","repos_url":"https://api.github.com/users/vitorbaptista/repos","events_url":"https://api.github.com/users/vitorbaptista/events{/privacy}","received_events_url":"https://api.github.com/users/vitorbaptista/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/opentrials/opentrials/milestones/6","html_url":"https://github.com/opentrials/opentrials/milestones/OpenTrialsFDA","labels_url":"https://api.github.com/repos/opentrials/opentrials/milestones/6/labels","id":1886167,"number":6,"title":"OpenTrialsFDA","description":"","creator":{"login":"vitorbaptista","id":76945,"avatar_url":"https://avatars.githubusercontent.com/u/76945?v=3","gravatar_id":"","url":"https://api.github.com/users/vitorbaptista","html_url":"https://github.com/vitorbaptista","followers_url":"https://api.github.com/users/vitorbaptista/followers","following_url":"https://api.github.com/users/vitorbaptista/following{/other_user}","gists_url":"https://api.github.com/users/vitorbaptista/gists{/gist_id}","starred_url":"https://api.github.com/users/vitorbaptista/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vitorbaptista/subscriptions","organizations_url":"https://api.github.com/users/vitorbaptista/orgs","repos_url":"https://api.github.com/users/vitorbaptista/repos","events_url":"https://api.github.com/users/vitorbaptista/events{/privacy}","received_events_url":"https://api.github.com/users/vitorbaptista/received_events","type":"User","site_admin":false},"open_issues":2,"closed_issues":2,"state":"open","created_at":"2016-07-15T16:55:20Z","updated_at":"2016-09-06T14:26:38Z","due_on":"2016-11-11T00:00:00Z","closed_at":null},"comments":1,"created_at":"2016-07-29T09:29:37Z","updated_at":"2016-09-06T14:26:38Z","closed_at":"2016-09-06T14:26:38Z","body":"We extracted all drug's approval history from the FDA at #263. This is their current structure:\r\n\r\n```javascript\r\n{\r\n    id: 'NDA020699-000',\r\n    fda_application_num: 'NDA020699',\r\n    supplement_number: 0,\r\n    action_date: '1997-10-20',\r\n    approval_type: 'Approval'\r\n    notes: 'Label is not available',\r\n    documents: [\r\n        {\r\n            name: 'Approval Letter(s)',\r\n            urls: ['http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_apltr.pdf']\r\n        }, {\r\n            name: 'Medical Review(s)',\r\n            urls: [\r\n                'http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_medrp1.pdf',\r\n                'http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_medrp2.pdf'\r\n            ]\r\n        }, {\r\n            name: 'Chemistry Review(s)',\r\n            urls: ['http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_chemr_ea_phrmr.pdf']\r\n        }, {\r\n            name: 'Clinical Pharmacology Review(s)',\r\n            urls: ['http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_clinphrmr_admindoc_.pdf']\r\n        }, {\r\n            name: 'Statistical Review(s)',\r\n            urls: ['http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_statr.pdf']\r\n        }\r\n    ]\r\n}\r\n```\r\n\r\nThe next steps are:\r\n\r\n1. Go over each document collected by `fda_dap` and merge multipart PDFs into one.\r\n> If the document has a `s3_url` already, don't bother merging the PDFs.\r\n>\r\n> The `urls` attribute is an array that contain the link to the PDFs. If there's more than 1 element, it means that the PDF was split in multiple parts. Merge those in the order they appear into a single PDF.\r\n\r\n2. Push all PDFs to S3 (for the documents that don't have a `s3_url` already)\r\n> They should be saved to `http://datastore.opentrials.net/documents/<PDF_HASH>.pdf`\r\n\r\n3. Push all PDFs to DocumentCloud (for the documents that don't have a `documentcloud_id` already)\r\n> Instead of uploading the file ourselves, simply send the document's URL on S3, so DocumentCloud can get it from there\r\n\r\n4. Save the `s3_url` and `documentcloud_id` back into the `fda_dap` table\r\n\r\nAfter this processing is done, the previous entity should become:\r\n\r\n```javascript\r\n{\r\n    id: 'NDA020699-000',\r\n    fda_application_num: 'NDA020699',\r\n    supplement_number: 0,\r\n    action_date: '1997-10-20',\r\n    approval_type: 'Approval'\r\n    notes: 'Label is not available',\r\n    documents: [\r\n        {\r\n            name: 'Approval Letter(s)',\r\n            urls: ['http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_apltr.pdf'],\r\n            s3_url: 'http://datastore.opentrials.net/documents/8843d7f92416211de9ebb963ff4ce28125932878.pdf',\r\n            documentcloud_id: 'b21e1b7c-3261-4680-87a5-8a454dfd8d75',\r\n        }, {\r\n            name: 'Medical Review(s)',\r\n            urls: [\r\n                'http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_medrp1.pdf',\r\n                'http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_medrp2.pdf'\r\n            ],\r\n            s3_url: 'http://datastore.opentrials.net/documents/6791c3961e86d32629b696be18516749d9563e80.pdf',\r\n            documentcloud_id: 'b1553bc0-62eb-4dac-a36b-ff58f7f3c833',\r\n        }, {\r\n            name: 'Chemistry Review(s)',\r\n            urls: ['http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_chemr_ea_phrmr.pdf'],\r\n            s3_url: 'http://datastore.opentrials.net/documents/031d717ad3ac8875e1c4a4d1ca06c6f9fe8cacc8.pdf',\r\n            documentcloud_id: '9e68acf3-eb4d-44cf-a948-7f1563de5ef1',\r\n        }, {\r\n            name: 'Clinical Pharmacology Review(s)',\r\n            urls: ['http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_clinphrmr_admindoc_.pdf'],\r\n            s3_url: 'http://datastore.opentrials.net/documents/91b8744e0f3071888376e24e07ea40f1ab6e2118.pdf',\r\n            documentcloud_id: '5bd75f9f-d419-4b57-be1e-ea4ec1a85482',\r\n        }, {\r\n            name: 'Statistical Review(s)',\r\n            urls: ['http://www.accessdata.fda.gov/drugsatfda_docs/nda/97/020699ap_effexor_statr.pdf'],\r\n            s3_url: 'http://datastore.opentrials.net/documents/d84ca121ec22172690fc73507a19ac0efd8e79ac.pdf',\r\n            documentcloud_id: 'bdfa6553-853c-4ce8-8cd4-4be41e82cbf8',\r\n        }\r\n    ]\r\n}\r\n```\r\n\r\nNote that we're not changing or removing any information from the `fda_dap` table, just adding more."}}
{"actor":{"display_login":"vitorbaptista"}}