{"action":"created","issue":{"url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/515","repository_url":"https://api.github.com/repos/AlexsLemonade/refinebio","labels_url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/515/labels{/name}","comments_url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/515/comments","events_url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/515/events","html_url":"https://github.com/AlexsLemonade/refinebio/issues/515","id":349540533,"node_id":"MDU6SXNzdWUzNDk1NDA1MzM=","number":515,"title":"Deliver and display selected sample annotations in table","user":{"login":"jaclyn-taroni","id":19534205,"node_id":"MDQ6VXNlcjE5NTM0MjA1","avatar_url":"https://avatars0.githubusercontent.com/u/19534205?v=4","gravatar_id":"","url":"https://api.github.com/users/jaclyn-taroni","html_url":"https://github.com/jaclyn-taroni","followers_url":"https://api.github.com/users/jaclyn-taroni/followers","following_url":"https://api.github.com/users/jaclyn-taroni/following{/other_user}","gists_url":"https://api.github.com/users/jaclyn-taroni/gists{/gist_id}","starred_url":"https://api.github.com/users/jaclyn-taroni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaclyn-taroni/subscriptions","organizations_url":"https://api.github.com/users/jaclyn-taroni/orgs","repos_url":"https://api.github.com/users/jaclyn-taroni/repos","events_url":"https://api.github.com/users/jaclyn-taroni/events{/privacy}","received_events_url":"https://api.github.com/users/jaclyn-taroni/received_events","type":"User","site_admin":false},"labels":[{"id":1013904284,"node_id":"MDU6TGFiZWwxMDEzOTA0Mjg0","url":"https://api.github.com/repos/AlexsLemonade/refinebio/labels/design%20input","name":"design input","color":"5864A6","default":false},{"id":917782526,"node_id":"MDU6TGFiZWw5MTc3ODI1MjY=","url":"https://api.github.com/repos/AlexsLemonade/refinebio/labels/metadata","name":"metadata","color":"fef2c0","default":false},{"id":940060527,"node_id":"MDU6TGFiZWw5NDAwNjA1Mjc=","url":"https://api.github.com/repos/AlexsLemonade/refinebio/labels/sci%20review","name":"sci review","color":"1d76db","default":false},{"id":873907685,"node_id":"MDU6TGFiZWw4NzM5MDc2ODU=","url":"https://api.github.com/repos/AlexsLemonade/refinebio/labels/smasher","name":"smasher","color":"16e20b","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/AlexsLemonade/refinebio/milestones/17","html_url":"https://github.com/AlexsLemonade/refinebio/milestone/17","labels_url":"https://api.github.com/repos/AlexsLemonade/refinebio/milestones/17/labels","id":3536138,"node_id":"MDk6TWlsZXN0b25lMzUzNjEzOA==","number":17,"title":"Tambourine Tesla","description":"Public beta","creator":{"login":"jaclyn-taroni","id":19534205,"node_id":"MDQ6VXNlcjE5NTM0MjA1","avatar_url":"https://avatars0.githubusercontent.com/u/19534205?v=4","gravatar_id":"","url":"https://api.github.com/users/jaclyn-taroni","html_url":"https://github.com/jaclyn-taroni","followers_url":"https://api.github.com/users/jaclyn-taroni/followers","following_url":"https://api.github.com/users/jaclyn-taroni/following{/other_user}","gists_url":"https://api.github.com/users/jaclyn-taroni/gists{/gist_id}","starred_url":"https://api.github.com/users/jaclyn-taroni/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jaclyn-taroni/subscriptions","organizations_url":"https://api.github.com/users/jaclyn-taroni/orgs","repos_url":"https://api.github.com/users/jaclyn-taroni/repos","events_url":"https://api.github.com/users/jaclyn-taroni/events{/privacy}","received_events_url":"https://api.github.com/users/jaclyn-taroni/received_events","type":"User","site_admin":false},"open_issues":21,"closed_issues":18,"state":"open","created_at":"2018-07-31T16:02:33Z","updated_at":"2018-08-16T15:46:48Z","due_on":"2018-09-01T07:00:00Z","closed_at":null},"comments":10,"created_at":"2018-08-10T14:37:03Z","updated_at":"2018-08-16T17:40:08Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"### Context\r\n\r\nJackiecrunch (#466)\r\n\r\n### Problem or idea\r\n\r\nWe are performing some smoothing of the sample metadata and these harmonized values are the ones being displayed in the samples table view. However, we smooth a limited number of fields, so information that may be important to a user, such as the presence or absence of a particular disease manifestation, may not be available in this table. In addition, when return sample annotations to users in the TSV metadata files, it is essentially JSON in a column. This will be of limited utility to users without considerable programming savvy. \r\n\r\nHere's an example of a sample from GEO that has been imported into ArrayExpress:\r\n\r\n```\r\n\"samples\": {\r\n        \"IFNa DC_LB016_IFNa\": {\r\n            \"accession_code\": \"E-GEOD-44719-GSM1089311\",\r\n            \"age\": \"\",\r\n            \"annotations\": [\r\n                {\r\n                    \"detected_platform\": \"illuminaHumanv3\",\r\n                    \"detection_percentage\": 98.44078,\r\n                    \"mapped_percentage\": 100.0\r\n                },\r\n                {\r\n                    \"assay\": {\r\n                        \"name\": \"GSM1089311\"\r\n                    },\r\n                    \"characteristic\": [\r\n                        {\r\n                            \"category\": \"cell population\",\r\n                            \"value\": \"IFNa DC\"\r\n                        },\r\n                        {\r\n                            \"category\": \"donor id\",\r\n                            \"value\": \"LB016\"\r\n                        },\r\n                        {\r\n                            \"category\": \"organism\",\r\n                            \"value\": \"Homo sapiens\"\r\n                        },\r\n                        {\r\n                            \"category\": \"stimulation\",\r\n                            \"value\": \"IFNa\"\r\n                        },\r\n                        {\r\n                            \"category\": \"tissue type\",\r\n                            \"value\": \"whole blood\"\r\n                        }\r\n                    ],\r\n                    \"extract\": {\r\n                        \"name\": \"GSM1089311 extract 1\"\r\n                    },\r\n                    \"file\": [\r\n                        {\r\n                            \"comment\": {\r\n                                \"name\": \"Derived ArrayExpress FTP file\",\r\n                                \"value\": \"ftp://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/GEOD/E-GEOD-44719/E-GEOD-44719.processed.1.zip\"\r\n                            },\r\n                            \"name\": \"GSM1089311_sample_table.txt\",\r\n                            \"type\": \"derived data\",\r\n                            \"url\": \"ftp://ftp.ebi.ac.uk/pub/databases/microarray/data/experiment/GEOD/E-GEOD-44719/E-GEOD-44719.processed.1.zip/GSM1089311_sample_table.txt\"\r\n                        }\r\n                    ],\r\n                    \"labeled-extract\": {\r\n                        \"label\": \"biotin\",\r\n                        \"name\": \"GSM1089311 LE 1\"\r\n                    },\r\n                    \"source\": {\r\n                        \"comment\": [\r\n                            {\r\n                                \"name\": \"Sample_description\",\r\n                                \"value\": \"4447846325_G\"\r\n                            },\r\n                            {\r\n                                \"name\": \"Sample_source_name\",\r\n                                \"value\": \"IFNa DC_IFNa\"\r\n                            },\r\n                            {\r\n                                \"name\": \"Sample_title\",\r\n                                \"value\": \"IFNa DC_LB016_IFNa\"\r\n                            }\r\n                        ],\r\n                        \"name\": \"GSM1089311 1\"\r\n                    },\r\n                    \"variable\": [\r\n                        {\r\n                            \"name\": \"cell population\",\r\n                            \"value\": \"IFNa DC\"\r\n                        },\r\n                        {\r\n                            \"name\": \"donor id\",\r\n                            \"value\": \"LB016\"\r\n                        },\r\n                        {\r\n                            \"name\": \"dose\",\r\n                            \"value\": \"1 mL\"\r\n                        },\r\n                        {\r\n                            \"name\": \"stimulation\",\r\n                            \"value\": \"IFNa\"\r\n                        }\r\n                    ]\r\n                }\r\n            ],\r\n            \"cell_line\": \"\",\r\n            \"compound\": \"\",\r\n            \"disease\": \"\",\r\n            \"disease_stage\": \"\",\r\n            \"genotype\": \"\",\r\n            \"organism\": \"HOMO_SAPIENS\",\r\n            \"platform\": \"Illumina_HumanHT-12_V3.0 (Illumina_HumanHT-12_V3.0)\",\r\n            \"race\": \"\",\r\n            \"sample_id\": \"IFNa DC_LB016_IFNa\",\r\n            \"sex\": \"\",\r\n            \"source_archive_url\": \"https://www.ebi.ac.uk/arrayexpress/json/v3/experiments/E-GEOD-44719/samples\",\r\n            \"specimen_part\": \"whole blood\",\r\n            \"subject\": \"lb016\",\r\n            \"time\": \"\",\r\n            \"title\": \"IFNa DC_LB016_IFNa\",\r\n            \"treatment\": \"\"\r\n        },\r\n        ...\r\n ```\r\n \r\nFor this sample, I would like to see the information contained in `variable` and `characteristic` table form. For example: \r\n```\r\n\"variable\": [\r\n                        {\r\n                            \"name\": \"cell population\",\r\n                            \"value\": \"IFNa DC\"\r\n                        }\r\n            ]\r\n ```\r\n\r\n`cell population` would be the header and the value would be `IFNa DC`. Note that some of the fields in `characteristic` and `variable` are redundant (pointed [here](https://github.com/AlexsLemonade/refinebio/issues/165#issuecomment-376565852)), so retaining only one where there would be duplicates would be quite helpful to users.\r\n\r\nHere's an example from GEO:\r\n\r\n```\r\n\"samples\": {\r\n        \"Bone.Marrow_OA_No_ST01\": {\r\n            \"accession_code\": \"GSM1361046\",\r\n            \"age\": \"\",\r\n            \"annotations\": [\r\n                {\r\n                    \"channel_count\": [\r\n                        \"1\"\r\n                    ],\r\n                    \"characteristics_ch1\": [\r\n                        \"tissue: Bone Marrow\",\r\n                        \"disease: OA\",\r\n                        \"serum: Low Serum\"\r\n                    ],\r\n                    \"contact_address\": [\r\n                        \"Crown Street\"\r\n                    ],\r\n                    \"contact_city\": [\r\n                        \"Liverpool\"\r\n                    ],\r\n                    \"contact_country\": [\r\n                        \"United Kingdom\"\r\n                    ],\r\n                    \"contact_email\": [\r\n                        \"p.antczak@liv.ac.uk\"\r\n                    ],\r\n                    \"contact_institute\": [\r\n                        \"University of Liverpool\"\r\n                    ],\r\n                    \"contact_name\": [\r\n                        \"Philipp,,Antczak\"\r\n                    ],\r\n                    \"contact_state\": [\r\n                        \"Merseyside\"\r\n                    ],\r\n                    \"contact_zip/postal_code\": [\r\n                        \"L69 7ZB\"\r\n                    ],\r\n                    \"data_processing\": [\r\n                        \"Data was processed and normalized using the RMA methodology. Probes that were called absent in 50% of samples have been removed.\"\r\n                    ],\r\n                    \"data_row_count\": [\r\n                        \"21742\"\r\n                    ],\r\n                    \"extract_protocol_ch1\": [\r\n                        \"Fibroblasts were retrieved from culture by trypsin digestion and washed in fibroblast medium then PBS before resuspending in 200l of RLT RNA protection buffer (Quiagen) followed by the manufacturer instructions of the RNeasy extraction kit.\"\r\n                    ],\r\n                    \"geo_accession\": [\r\n                        \"GSM1361046\"\r\n                    ],\r\n                    \"growth_protocol_ch1\": [\r\n                        \"Cells were seeded in duplicate at the required density using 10% FCS. Once attached, the medium was removed and the cells washed in serum free medium before replacing with fibroblast medium, which was identical but contained either 10% or 0.1% FCS.\"\r\n                    ],\r\n                    \"hyb_protocol\": [\r\n                        \"Hybridization of Chips was performed as per manufacturers instructions.\"\r\n                    ],\r\n                    \"label_ch1\": [\r\n                        \"biotin\"\r\n                    ],\r\n                    \"label_protocol_ch1\": [\r\n                        \"50ng of total RNA was amplified using Nugen Ovation Biotin RNA amplification\"\r\n                    ],\r\n                    \"last_update_date\": [\r\n                        \"Jun 22 2015\"\r\n                    ],\r\n                    \"molecule_ch1\": [\r\n                        \"total RNA\"\r\n                    ],\r\n                    \"organism_ch1\": [\r\n                        \"Homo sapiens\"\r\n                    ],\r\n                    \"platform_id\": [\r\n                        \"GPL570\"\r\n                    ],\r\n                    \"scan_protocol\": [\r\n                        \"SData were scanned as per manufactureres instructions on a GeneArray Scanner.\"\r\n                    ],\r\n                    \"series_id\": [\r\n                        \"GSE56409\"\r\n                    ],\r\n                    \"source_name_ch1\": [\r\n                        \"Bone Marrow\"\r\n                    ],\r\n                    \"status\": [\r\n                        \"Public on Jun 22 2015\"\r\n                    ],\r\n                    \"submission_date\": [\r\n                        \"Apr 01 2014\"\r\n                    ],\r\n                    \"supplementary_file\": [\r\n                        \"NONE\"\r\n                    ],\r\n                    \"taxid_ch1\": [\r\n                        \"9606\"\r\n                    ],\r\n                    \"title\": [\r\n                        \"Bone.Marrow_OA_No_ST01\"\r\n                    ],\r\n                    \"type\": [\r\n                        \"RNA\"\r\n                    ]\r\n                }\r\n            ],\r\n            \"cell_line\": \"\",\r\n            \"compound\": \"\",\r\n            \"disease\": \"oa\",\r\n            \"disease_stage\": \"\",\r\n            \"genotype\": \"\",\r\n            \"organism\": \"HOMO_SAPIENS\",\r\n            \"platform\": \"Affymetrix Human Genome U133 Plus 2.0 Array (hgu133plus2)\",\r\n            \"race\": \"\",\r\n            \"sample_id\": \"Bone.Marrow_OA_No_ST01\",\r\n            \"sex\": \"\",\r\n            \"source_archive_url\": \"\",\r\n            \"specimen_part\": \"bone marrow\",\r\n            \"subject\": \"\",\r\n            \"time\": \"\",\r\n            \"title\": \"Bone.Marrow_OA_No_ST01\",\r\n            \"treatment\": \"\"\r\n        }, \r\n        ...\r\n ```\r\n \r\n For this example, I am interested in what is contained in the `characteristics` field:\r\n \r\n ```\r\n  \"characteristics_ch1\": [\r\n                        \"tissue: Bone Marrow\",\r\n                        \"disease: OA\",\r\n                        \"serum: Low Serum\"\r\n                    ]\r\n ```\r\n\r\nWhere, ideally, `tissue`, `disease`, and `serum` would all be column names populated with their respective values. We capture the OA and bone marrow information in our smoothing, but the serum treatment (a key part of the experimental design) is not included.\r\n\r\nI know there has been some hesitance to venture into this because the presence of _nested values_ has been mentioned. At the moment, I am not sure why are we able to smooth some of the fields but not extract the others such that they are easily displayed or delivered in a tabular format, particularly if we are able to identify the specific fields (e.g., `characteristic`) that would need to be extracted.\r\n\r\nI believe this effort would reduce friction for a large portion of our user base.\r\n\r\n### Solution or next step\r\n\r\n~Can we run `SRP056295`, an experiment from SRA, and [`E-MEXP-31`](https://www.ebi.ac.uk/arrayexpress/experiments/E-MEXP-31/), ArrayExpress data that _has not_ been imported from GEO, through our system? I do not currently have SRA or ArrayExpress submission metadata on hand, so I can not make a recommendation about what fields to extract yet. Some more examples from ArrayExpress and GEO would probably be helpful, too. These are the ones I happen to have on hand at the moment.~\r\n\r\nSee comment below about zebrafish experiments.\r\n\r\n### New Issue Checklist\r\n\r\n- [x] The title is short and descriptive.\r\n- [x] You have explained the context that led you to write this issue.\r\n- [x] You have reported a problem or idea.\r\n- [x] You have proposed a solution or next step."},"comment":{"url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/comments/413626852","html_url":"https://github.com/AlexsLemonade/refinebio/issues/515#issuecomment-413626852","issue_url":"https://api.github.com/repos/AlexsLemonade/refinebio/issues/515","id":413626852,"node_id":"MDEyOklzc3VlQ29tbWVudDQxMzYyNjg1Mg==","user":{"login":"dvenprasad","id":15315514,"node_id":"MDQ6VXNlcjE1MzE1NTE0","avatar_url":"https://avatars3.githubusercontent.com/u/15315514?v=4","gravatar_id":"","url":"https://api.github.com/users/dvenprasad","html_url":"https://github.com/dvenprasad","followers_url":"https://api.github.com/users/dvenprasad/followers","following_url":"https://api.github.com/users/dvenprasad/following{/other_user}","gists_url":"https://api.github.com/users/dvenprasad/gists{/gist_id}","starred_url":"https://api.github.com/users/dvenprasad/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dvenprasad/subscriptions","organizations_url":"https://api.github.com/users/dvenprasad/orgs","repos_url":"https://api.github.com/users/dvenprasad/repos","events_url":"https://api.github.com/users/dvenprasad/events{/privacy}","received_events_url":"https://api.github.com/users/dvenprasad/received_events","type":"User","site_admin":false},"created_at":"2018-08-16T17:40:08Z","updated_at":"2018-08-16T17:40:08Z","author_association":"COLLABORATOR","body":"Displaying all the unnormalized fields after processing information was part of the original plan. It seems like the data is too unwieldy to that in its current state. \r\n\r\nBased on the conversation on https://github.com/AlexsLemonade/refinebio-frontend/issues/215, I understand that the json is present on a per sample basis. Would it be possible/make sense to display that on a modal on a per sample basis instead? We can have a column at the end 'Additional Metadata' \r\n\r\nAlso, I poked around a few metadata.json files, it looks like contact info repeats for every sample, would it be possible to \"remove\" it for display purposes? They can still be part of the download file. "}}
{"actor":{"display_login":"dvenprasad"}}