{"action":"created","issue":{"url":"https://api.github.com/repos/biocore/biom-format/issues/701","repository_url":"https://api.github.com/repos/biocore/biom-format","labels_url":"https://api.github.com/repos/biocore/biom-format/issues/701/labels{/name}","comments_url":"https://api.github.com/repos/biocore/biom-format/issues/701/comments","events_url":"https://api.github.com/repos/biocore/biom-format/issues/701/events","html_url":"https://github.com/biocore/biom-format/issues/701","id":162309397,"number":701,"title":"importing usearch biom output ","user":{"login":"danchurch","id":7447094,"avatar_url":"https://avatars.githubusercontent.com/u/7447094?v=3","gravatar_id":"","url":"https://api.github.com/users/danchurch","html_url":"https://github.com/danchurch","followers_url":"https://api.github.com/users/danchurch/followers","following_url":"https://api.github.com/users/danchurch/following{/other_user}","gists_url":"https://api.github.com/users/danchurch/gists{/gist_id}","starred_url":"https://api.github.com/users/danchurch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danchurch/subscriptions","organizations_url":"https://api.github.com/users/danchurch/orgs","repos_url":"https://api.github.com/users/danchurch/repos","events_url":"https://api.github.com/users/danchurch/events{/privacy}","received_events_url":"https://api.github.com/users/danchurch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":18,"created_at":"2016-06-26T02:39:42Z","updated_at":"2016-07-09T01:36:36Z","closed_at":null,"body":"I am doing an ITS region fungal study. I used the usearch v8.1 pipeline, without any qiime scripts, to get me to the point of a biom table with taxonomic assignments. As far as I can tell, the json biom table that was outputted by usearch is not readable by any downstream applications, including biom, either python scripts or in R environment. I believe this may be due to the structure of the taxonomic metadata. So I edited them to try to match examples on the biom website. However, the taxonomic info is still not reading correctly. \r\n\r\nThe original output from usearch 8.1:\r\n\r\n\"\"\"\r\n{\r\n        \"id\":\"otu_95_sub.biom\",\r\n        \"format\": \"Biological Observation Matrix 1.0\",\r\n        \"format_url\": \"http://biom-format.org\",\r\n        \"generated_by\": \"usearch8.11803\",\r\n        \"type\": \"OTU table\",\r\n        \"date\": \"Thu Jun 23 20:38:04 2016\",\r\n        \"matrix_type\": \"sparse\",\r\n        \"matrix_element_type\": \"float\",\r\n        \"shape\": [767176,98],\r\n        \"rows\":[\r\n                {\"id\":\"OTU1575410:M01498:244:000000000-ANT97:1:1101:18025:5338:160\", \"metadata\":{\"taxonomy\":\"d:Fungi(0.9991),p:Ascomycota(0.5394),c:Sordariomycetes(0.3516),o:Xylariales(0.1885),f:Xylariaceae(0.0879),g:Xylaria(0.0105)\"}},\r\n\r\n\r\n\"\"\"\r\n\r\nAfter my scripting, rows look like this:\r\n\r\n\"\"\"\r\n        \"shape\": [767176,98],\r\n        \"rows\":[\r\n                {\"id\":\"OTU1575410\", \"metadata\":{\"taxonomy\": [\"k__Fungi\", \"p__Ascomycota\", \"c__Sordariomycetes\", \"o__Xylariales\", \"f__Xylariaceae\", \"g__Xylaria\"]}},\r\n\r\n\"\"\"\r\n\r\nI believe all brackets have been properly closed. Biom will happily convert to a classical OTU table from this for me when I do not include taxonomic info, but every other biom command generates a nonsense file, not human-readable, and not readable by phyloseq. No error message is returned. \r\n\r\nLet me know if this row format seems erroneous, please. If it does not, I will look elsewhere for the source of the error.  If you  are curious, I am keeping a log of my pipeline on my own github account. \r\n"},"comment":{"url":"https://api.github.com/repos/biocore/biom-format/issues/comments/231507183","html_url":"https://github.com/biocore/biom-format/issues/701#issuecomment-231507183","issue_url":"https://api.github.com/repos/biocore/biom-format/issues/701","id":231507183,"user":{"login":"danchurch","id":7447094,"avatar_url":"https://avatars.githubusercontent.com/u/7447094?v=3","gravatar_id":"","url":"https://api.github.com/users/danchurch","html_url":"https://github.com/danchurch","followers_url":"https://api.github.com/users/danchurch/followers","following_url":"https://api.github.com/users/danchurch/following{/other_user}","gists_url":"https://api.github.com/users/danchurch/gists{/gist_id}","starred_url":"https://api.github.com/users/danchurch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danchurch/subscriptions","organizations_url":"https://api.github.com/users/danchurch/orgs","repos_url":"https://api.github.com/users/danchurch/repos","events_url":"https://api.github.com/users/danchurch/events{/privacy}","received_events_url":"https://api.github.com/users/danchurch/received_events","type":"User","site_admin":false},"created_at":"2016-07-09T01:36:36Z","updated_at":"2016-07-09T01:36:36Z","body":"Okay, I think I've figured out my situation. Two factors were muddying the waters, ones we pretty much identified above, made especially confusing by the fact that I am working on both my own computer and a university computing cluster. The first was that the computing cluster I'm using can't upgrade to R 3.3, so no current version of BioClite/Biomformat/phyloseq. So no ability to handle the HDF5 formats. \r\n\r\nThe second is that my personal machine, which is up-to-date (see above) with the necessary packages and R, can handle the new formats, theoretically, but any attempt to import these biom tables as phyloseq objects requires too much memory and fails. \r\n\r\nSo if I ensure that all biom tables remain in json format, all analyses seem to be running smoothly on our computing cluster with R. 3.2.5, phyloseq_1.14.0,  and (sorry @joey711) biom_0.3.12. \r\n\r\nMy scripting seems to make the file more readable to phyloseq. The unmodified output from usearch v8.1 seems to create an unreadable taxonomic data format, at least for me (to see what these biom files outputted from usearch and after my scripts look like, see first message above). \r\n\r\nWith the raw json biom outputs from usearch, in R 3.2.5/phyloseq_1.14.0/biom_0.3.12:\r\n`> bb <- import_biom('otu_95_sub.biom')`\r\n>Error in i$metadata$taxonomy : $ operator is invalid for atomic vectors\r\n\r\n`> bb`\r\n>Error: object 'bb' not found\r\n\r\nAfter my scripting modifications:\r\n`>cc <- import_biom('otu_95_sub_relab3.biom')`\r\n`> cc`\r\n>phyloseq-class experiment-level object\r\notu_table()   OTU Table:         [ 767176 taxa and 98 samples ]\r\ntax_table()   Taxonomy Table:    [ 767176 taxa by 7 taxonomic ranks ]\r\n\r\nSo I have to be careful to keep all my outputs as json files, until the computing cluster updates, or you gentleman can provide backward compatibility for the HDF5 format and bioconductor packages to R versions < 3.3. Not requesting this, just noting. \r\n\r\nAnyway, thanks for all of your help, @joey711 @wasade. I'll wait for any responses you might want to give, but for me I think this issue is closed..."}}
{"actor":{"display_login":"danchurch"}}