{"action":"created","issue":{"url":"https://api.github.com/repos/BgeeDB/expression-annotations/issues/110","labels_url":"https://api.github.com/repos/BgeeDB/expression-annotations/issues/110/labels{/name}","comments_url":"https://api.github.com/repos/BgeeDB/expression-annotations/issues/110/comments","events_url":"https://api.github.com/repos/BgeeDB/expression-annotations/issues/110/events","html_url":"https://github.com/BgeeDB/expression-annotations/issues/110","id":128594621,"number":110,"title":"Rewiew of GTEx annotations","user":{"login":"aechchiki","id":13200013,"avatar_url":"https://avatars.githubusercontent.com/u/13200013?v=3","gravatar_id":"","url":"https://api.github.com/users/aechchiki","html_url":"https://github.com/aechchiki","followers_url":"https://api.github.com/users/aechchiki/followers","following_url":"https://api.github.com/users/aechchiki/following{/other_user}","gists_url":"https://api.github.com/users/aechchiki/gists{/gist_id}","starred_url":"https://api.github.com/users/aechchiki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aechchiki/subscriptions","organizations_url":"https://api.github.com/users/aechchiki/orgs","repos_url":"https://api.github.com/users/aechchiki/repos","events_url":"https://api.github.com/users/aechchiki/events{/privacy}","received_events_url":"https://api.github.com/users/aechchiki/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/BgeeDB/expression-annotations/labels/GTEx","name":"GTEx","color":"fad8c7"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":11,"created_at":"2016-01-25T17:33:04Z","updated_at":"2016-01-31T23:34:49Z","closed_at":null,"body":"Following discussions with @fbastian @ANiknejad and @mimiblis :\r\n(note: new files referred in this issue can be found at https://github.com/BgeeDB/expression-annotations/tree/master/annotations/gtex/review_of_GTEx_annotations_FILES)\r\n\r\nFile \"GTEX_samples_track.tsv\" contains all the samples in the original GTEx file from RxTx, adding information to explain whether and why a given sample was included/excluded from Bgee (data cleansing details on the drive: https://drive.google.com/open?id=1IuNu3WGTSIhXnJffP_yo7lK2abSgxZQDPJgG1SYF5vI ).  \r\n\r\n\"GTEX_samples_track.tsv\" contains 11983 samples.  4816 samples come from subjects tagged \"yes\" :  corresponding samples were also automatically tagged as \"yes\". 3303 samples come from subjects tagged \"no\" : corresponding samples also automatically tagged as \"no\".  3864 samples coming from subjects tagged \"pending\": among these, 3521 samples were tagged \"yes\" and 343 samples were tagged \"no\". We expect to have a total of 8337 samples (out of the original 11983) that can be included in Bgee annotations. Note:  8337 is the sum of 4816 (samples that were tagged \"yes\" because from \"yes\" subjects) and 3521 (samples that were tagged \"yes\" from \"pending\" subjects). \r\n\r\nHowever, in the current file \"GTEX_selected-samples_annots.tsv\" on the repo, only 8258 samples (instead of the expected 8337)  were annotated. We aim to investigate the origin of this incongruence.  \r\n\r\n- We generated a file from \"GTEX_samples_track.tsv\", containing only samples that are supposed to be annotated in Bgee (aka, containing samples from subjects tagged \"yes\" or samples from subjects tagged  \"pending\" for which the sample was tagged \"yes\"): this file excludes all the other samples (aka, containing samples from subjects tagged \"no\" or samples from subjects tagged  \"pending\" for which the sample was tagged \"no\"). This file is called \"GTEX_samples_track-sampYES.tsv\". \r\n- We extracted the GTEXsample IDs (unique for each sample) from \"GTEX_samples_track-sampYES.tsv\". As expected, we found zero redundancies in this ID list (using awk 'a[$0]++' method). \r\n- We  extracted the GTEXsample IDs from \"GTEX_selected-samples_annots.tsv\". This time, we found 159 redundancies in GTEXsample ID (using awk 'a[$0]++' method). The redundant IDs are listed in \"GTEXID_duplicate_selected-samples_annots.tsv\".\r\n\r\nHere's a first issue detected. We found redundancies in the current  \"GTEX_selected-samples_annots.tsv\".\r\n\r\n- We checked manually for the suspected redundancies,  to verify the presence of uninformative duplicates . Manual check gave credit to what the script suggested. \r\n- We thus generated a file called \"GTEX_selected-samples_annots-NON_REDUNDANT.tsv\" in which we have deleted one instance of the duplicated samples. \"GTEX_selected-samples_annots-NON_REDUNDANT.tsv\" contains 8086 non-redundant samples\r\n- We extracted the GTEXsample IDs from \"GTEX_selected-samples_annots-NON_REDUNDANT.tsv\". As expected, we found zero redundancies in this ID list (using awk 'a[$0]++' method). \r\n\r\n\"GTEX_selected-samples_annots-NON_REDUNDANT.tsv\" is a duplicate of \"GTEX_selected-samples_annots.tsv\" in which the redundancies were deleted. \r\n\r\n- We retrieved a list of the IDs that should be included in Bgee annotation accoridng to \"GTEX_samples_track.tsv\" (now stored in  \"GTEX_samples_track-sampYES.tsv\" ), but were not found in \"GTEX_selected-samples_annots.tsv\" (using awk 'NR==FNR{a[$0]=1;next}!a[$0]'). Output was saved to  \"GTEX_IDs-to-check.txt\" \r\n- \"GTEX_IDs-to-check.txt\"  contains 338 IDs that are not currently in the annotation file, even thought \"GTEX_samples_track.tsv\" file suggested to include them. \r\n\r\nWe suggest to manually check these IDs and decide whether or not they should be included in Bgee annotations. \r\n\r\nThe number 338 seem to correspond, more or less, to the number of samples for which the subject ID has changed since the \"few\" changes in subject assignment to \"yes\"/\"pending\"/\"no\", this could explain why they're not included in the current  \"GTEX_selected-samples_annots.tsv\"...so is maybe the \"GTEX_samples_track.tsv\" not updated to the most recent changes?\r\n In my opinion, though, redundancies should be deleted as in  \"GTEX_selected-samples_annots-NON_REDUNDANT.tsv\". \r\n"},"comment":{"url":"https://api.github.com/repos/BgeeDB/expression-annotations/issues/comments/177660866","html_url":"https://github.com/BgeeDB/expression-annotations/issues/110#issuecomment-177660866","issue_url":"https://api.github.com/repos/BgeeDB/expression-annotations/issues/110","id":177660866,"user":{"login":"aechchiki","id":13200013,"avatar_url":"https://avatars.githubusercontent.com/u/13200013?v=3","gravatar_id":"","url":"https://api.github.com/users/aechchiki","html_url":"https://github.com/aechchiki","followers_url":"https://api.github.com/users/aechchiki/followers","following_url":"https://api.github.com/users/aechchiki/following{/other_user}","gists_url":"https://api.github.com/users/aechchiki/gists{/gist_id}","starred_url":"https://api.github.com/users/aechchiki/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aechchiki/subscriptions","organizations_url":"https://api.github.com/users/aechchiki/orgs","repos_url":"https://api.github.com/users/aechchiki/repos","events_url":"https://api.github.com/users/aechchiki/events{/privacy}","received_events_url":"https://api.github.com/users/aechchiki/received_events","type":"User","site_admin":false},"created_at":"2016-01-31T23:34:49Z","updated_at":"2016-01-31T23:34:49Z","body":"Task 2 (partially) completed. \r\n// I say _partially_ because up to now I defined \"keywords\" not \"key Uberon IDs\" as basis for tag a sample as \"YES\" and \"NO\" when the sample comes from a subject tagged \"PENDING\" //\r\n\r\n- I checked if any subject tagged as \"NO\" in GTEx_subjects-track.tsv was, accidentally, reported in `GTEX_selected-samples_annots-NON_REDUNDANT.tsv` (or `GTEX_selected-samples_annots.tsv`, waiting for @ANiknejad's validation). The answer is YES! Subject _GTEX-PLZ6_ was still in the annotation file (in spite of being tagged \"NO\" for drug abuse). 32 samples coming from this subject were found. They should be discarded. However, this subject (and his samples) were correctly tagged as to be discarded from Bgee in `GTEx_samples_track.tsv`, indicating that this file is up-to-date with the most recent changes.\r\n- I checked if any subject tagged as \"PENDING\" and for which the sample was tagged as \"NO\" (according to  `GTEx_samples_track.tsv`)  was, accidentally, reported in `GTEX_selected-samples_annots-NON_REDUNDANT.tsv` (or `GTEX_selected-samples_annots.tsv`, waiting for @ANiknejad's validation). The answer is YES! A total of 55 samples were found. They should be discarded. \r\n- Two additional samples were discarded even though they were coming from subjects tagged \"YES\" because annotation from GTEx is \"suspicious\" ( @mimiblis or @ANiknejad, I suppose one of you did this xD )\r\n\r\n I generated a file listing all these samples to be discarded from `GTEX_selected-samples_annots-NON_REDUNDANT.tsv` (or `GTEX_selected-samples_annots.tsv`): https://github.com/BgeeDB/expression-annotations/blob/master/annotations/gtex/review_of_GTEx_annotations_FILES/discard_from_selected-samples.tsv"}}