{"action":"created","issue":{"url":"https://api.github.com/repos/OHDSI/OncologyWG/issues/6","repository_url":"https://api.github.com/repos/OHDSI/OncologyWG","labels_url":"https://api.github.com/repos/OHDSI/OncologyWG/issues/6/labels{/name}","comments_url":"https://api.github.com/repos/OHDSI/OncologyWG/issues/6/comments","events_url":"https://api.github.com/repos/OHDSI/OncologyWG/issues/6/events","html_url":"https://github.com/OHDSI/OncologyWG/issues/6","id":385696711,"node_id":"MDU6SXNzdWUzODU2OTY3MTE=","number":6,"title":"Ingest NAACCR into the vocabulary tables.","user":{"login":"mgurley","id":82806,"node_id":"MDQ6VXNlcjgyODA2","avatar_url":"https://avatars0.githubusercontent.com/u/82806?v=4","gravatar_id":"","url":"https://api.github.com/users/mgurley","html_url":"https://github.com/mgurley","followers_url":"https://api.github.com/users/mgurley/followers","following_url":"https://api.github.com/users/mgurley/following{/other_user}","gists_url":"https://api.github.com/users/mgurley/gists{/gist_id}","starred_url":"https://api.github.com/users/mgurley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgurley/subscriptions","organizations_url":"https://api.github.com/users/mgurley/orgs","repos_url":"https://api.github.com/users/mgurley/repos","events_url":"https://api.github.com/users/mgurley/events{/privacy}","received_events_url":"https://api.github.com/users/mgurley/received_events","type":"User","site_admin":false},"labels":[{"id":1129947660,"node_id":"MDU6TGFiZWwxMTI5OTQ3NjYw","url":"https://api.github.com/repos/OHDSI/OncologyWG/labels/Vocabulary","name":"Vocabulary","color":"7471ce","default":false}],"state":"open","locked":false,"assignee":{"login":"mgurley","id":82806,"node_id":"MDQ6VXNlcjgyODA2","avatar_url":"https://avatars0.githubusercontent.com/u/82806?v=4","gravatar_id":"","url":"https://api.github.com/users/mgurley","html_url":"https://github.com/mgurley","followers_url":"https://api.github.com/users/mgurley/followers","following_url":"https://api.github.com/users/mgurley/following{/other_user}","gists_url":"https://api.github.com/users/mgurley/gists{/gist_id}","starred_url":"https://api.github.com/users/mgurley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgurley/subscriptions","organizations_url":"https://api.github.com/users/mgurley/orgs","repos_url":"https://api.github.com/users/mgurley/repos","events_url":"https://api.github.com/users/mgurley/events{/privacy}","received_events_url":"https://api.github.com/users/mgurley/received_events","type":"User","site_admin":false},"assignees":[{"login":"mgurley","id":82806,"node_id":"MDQ6VXNlcjgyODA2","avatar_url":"https://avatars0.githubusercontent.com/u/82806?v=4","gravatar_id":"","url":"https://api.github.com/users/mgurley","html_url":"https://github.com/mgurley","followers_url":"https://api.github.com/users/mgurley/followers","following_url":"https://api.github.com/users/mgurley/following{/other_user}","gists_url":"https://api.github.com/users/mgurley/gists{/gist_id}","starred_url":"https://api.github.com/users/mgurley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgurley/subscriptions","organizations_url":"https://api.github.com/users/mgurley/orgs","repos_url":"https://api.github.com/users/mgurley/repos","events_url":"https://api.github.com/users/mgurley/events{/privacy}","received_events_url":"https://api.github.com/users/mgurley/received_events","type":"User","site_admin":false},{"login":"dimshitc","id":8568959,"node_id":"MDQ6VXNlcjg1Njg5NTk=","avatar_url":"https://avatars0.githubusercontent.com/u/8568959?v=4","gravatar_id":"","url":"https://api.github.com/users/dimshitc","html_url":"https://github.com/dimshitc","followers_url":"https://api.github.com/users/dimshitc/followers","following_url":"https://api.github.com/users/dimshitc/following{/other_user}","gists_url":"https://api.github.com/users/dimshitc/gists{/gist_id}","starred_url":"https://api.github.com/users/dimshitc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dimshitc/subscriptions","organizations_url":"https://api.github.com/users/dimshitc/orgs","repos_url":"https://api.github.com/users/dimshitc/repos","events_url":"https://api.github.com/users/dimshitc/events{/privacy}","received_events_url":"https://api.github.com/users/dimshitc/received_events","type":"User","site_admin":false}],"milestone":null,"comments":9,"created_at":"2018-11-29T11:45:19Z","updated_at":"2018-12-20T17:13:12Z","closed_at":null,"author_association":"COLLABORATOR","body":"- Call the SEER API to ingest into the OHDSI vocabulary tables NAACCR data item/data item categorical values.\r\n  - Get an account and API key from here: https://api.seer.cancer.gov/new_account\r\n  - Return all NAACCR item numbers/item names for the latest NAACCR version from this REST resource: https://api.seer.cancer.gov/rest/naaccr/latest.   This will return an array of hashes for reach NAACCR data item with the following JSON object structure: { \"item\": 10, \"name\": \"Record Type\" }\r\n  - For each NAACCR data item, call the REST resource (replacing 10 with each NAACCR data item number): https://api.seer.cancer.gov/rest/naaccr/latest/item/10.  This will return a hash with the following structure (the “documentation” key value can be seen in the separate file ‘documentation.html’): { \"item\": 1390, \"name\": \"RX Summ--Chemo\", \"start_col\": 2243, \"end_col\": 2244, \"alignment\":  \"RIGHT\",\"padding_char\": \"0\", \"documentation\": “documentation.html”  }\r\n    - Parse the HTML value in the “documentation’ key.  Here are CSS selectors that will help you find the right content:  \r\n      - ‘div.content.chap10-para .code-row’  This will find each categorical value for the NACCAR data item.\r\n      - ‘td.code-nbr’ to find the code value\r\n      - ‘td.code-dsc’ to find the code description.\r\n  - Not all NAACCR data items will have its list of categorical values within the ‘documentation’ key.  Some NAACCR data items are non-categorical.  Other NAACCR data items are what is called ‘site-specific’ (SSDI). SSDI means the data item’s categorical values are based on the anatomical site/topography being reported upon.   For example:\r\n    - NAACCR data item #1290 ‘RX SUMM--SURG PRIM SITE’: tracks the site-specific codes for the type of surgery to the primary site performed as part of the first course of treatment. This includes treatment given at all facilities as part of the first course of treatment.   Here are the high-level grouping of site-specific surgery codes.\r\n      - 00\tNone\r\n      - 10-19\tSite-specific code; tumor destruction\r\n      - 20-80\tSite-specific codes; resection\r\n      - 90\tSurgery, NOS\r\n      - 98\tSite specific codes; special\r\n      - 99\tUnknown\r\n  - Return all NAACCR site-specific surgery codes table titles from this REST resource: https://api.seer.cancer.gov/rest/surgery/latest/tables.  This will return an array of strings each string representing a site-specific surgery codes table title.  Each table tile is a short hand for a group of anatomical ICDO site codes.\r\n  - For each NAACCR site-specific surgery codes table title, call the REST resource (replacing ‘Breast’ with each NAACCR site-specific surgery code table title):  https://api.seer.cancer.gov/rest/surgery/latest/table?title=Breast.  This will return a hash with the following structure (the “row” key value can be seen in the separate file ‘row.json’): { \"title\": \"Breast\", \"site_inclusions\": \"C500-C509\", \"hist_exclusions\": \"9727,9733,9741-9742,9764-9809,9832,9840-9931,9945-9946,9950-9967,9975-9992\", \"pre_note\": \"C500-C509<br/>\\n(Except for M-9727, 9733, 9741-9742, 9764-9809, 9832, 9840-9931, 9945-9946, 9950-9967 and 9975-9992)\", \"row\":[ {}, {}]}.  The “site_inclusions” key details the range of ICDO topography/anatomical sites bound to this list of surgery codes.  The “row” key is an array of hashes that represents the site-specific surgery code hierarchy.  Some of the hashes in the “row” key are for line-break formatting and instructional text.  Ignore all entries in the “row” key with a “line_break”: true key/value pair or that have a “code”:”” key/value pair.  The hierarchal structure of the “row” key is based on the order of the entries.  The “level” key can go from 0 to 3. \r\n- Screen scrape the NAACCR EOD data SSDI data item possible values.  A whole set of all SSDI NAACCR data items fall into the category ‘EOD Data’.  See here a list of EOD schemas (anatomical site/histology groupings): https://staging.seer.cancer.gov/eod_public/list/1.4/.  Unfortunately, it does not appear that the SEER API yet supports detailing the list of possible values for these data items.  So we will need to scrape.\r\n  - Visit https://staging.seer.cancer.gov/eod_public/list/1.4/\r\n  - Click each “schema”.\r\n  - Scrape the list of possible ‘Primary Site’ and ‘Histology’ associated with the “schema”.\r\n  - For each row in the table labeled “Data Items”, if the “Metadata” column indicates “SSDI”, click on the link in the “Name” column.\r\n  - On the page for the SSDI NAACCR data item, scrape the NAACCR data item and collect the possible values from the table with the headers “Code” and “Description”.\r\n- Screen scrape the NAACCR TNM data SSDI data item possible values.  A whole set of all SSDI NAACCR data items fall into the category ‘TNM Data’.  See here a list of TNM schemas (anatomical site/histology groupings): https://staging.seer.cancer.gov/tnm/list/1.9/ .  Unfortunately, it does not appear that the SEER API yet supports detailing the list of possible values for these data items.  So we will need to scrape.\r\n  - Visit https://staging.seer.cancer.gov/tnm/list/1.9/\r\n  - Click each “schema”.\r\n  - Scrape the list of possible ‘Primary Site’ and ‘Histology’ associated with the “schema”.\r\n  - For each row in the table labeled “Main Data Items”, click on the link in the “Name” column.\r\n  - On the page for the SSDI NAACCR data item, scrape the NAACCR data item and collect the possible values from the table with the header specific the SSDI data item.  For example, 'Clinical T' and 'Clinical T Display’.\r\n\r\n[row.json](https://drive.google.com/open?id=16k4_jS8HxK1CpLbkrxHjEAfVMyOBXSXU)\r\n[documentation.html](https://drive.google.com/open?id=11Cl2pXYM3e84TWOXKlhht3ssCVjKbEJ_)\r\n\r\n\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/OHDSI/OncologyWG/issues/comments/449070304","html_url":"https://github.com/OHDSI/OncologyWG/issues/6#issuecomment-449070304","issue_url":"https://api.github.com/repos/OHDSI/OncologyWG/issues/6","id":449070304,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0OTA3MDMwNA==","user":{"login":"mgurley","id":82806,"node_id":"MDQ6VXNlcjgyODA2","avatar_url":"https://avatars0.githubusercontent.com/u/82806?v=4","gravatar_id":"","url":"https://api.github.com/users/mgurley","html_url":"https://github.com/mgurley","followers_url":"https://api.github.com/users/mgurley/followers","following_url":"https://api.github.com/users/mgurley/following{/other_user}","gists_url":"https://api.github.com/users/mgurley/gists{/gist_id}","starred_url":"https://api.github.com/users/mgurley/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mgurley/subscriptions","organizations_url":"https://api.github.com/users/mgurley/orgs","repos_url":"https://api.github.com/users/mgurley/repos","events_url":"https://api.github.com/users/mgurley/events{/privacy}","received_events_url":"https://api.github.com/users/mgurley/received_events","type":"User","site_admin":false},"created_at":"2018-12-20T17:13:12Z","updated_at":"2018-12-20T17:13:12Z","author_association":"COLLABORATOR","body":"@dimshitc That is a typo.  Sorry. I will fix. The domain for most NAACCR attributes/questions should be 'Measurement' and for NAACCR values/answers should be 'Meas Value'  I will correct.  The vocabulary_id should be NAACCR.  Some treatment-related NAACCR attributes/questions values/answers will be in the new Treatment domain (I will worki on the Treatment NAACCR items/item answers this weekend)."}}
{"actor":{"display_login":"mgurley"}}