{"action":"created","issue":{"url":"https://api.github.com/repos/joey711/phyloseq/issues/537","repository_url":"https://api.github.com/repos/joey711/phyloseq","labels_url":"https://api.github.com/repos/joey711/phyloseq/issues/537/labels{/name}","comments_url":"https://api.github.com/repos/joey711/phyloseq/issues/537/comments","events_url":"https://api.github.com/repos/joey711/phyloseq/issues/537/events","html_url":"https://github.com/joey711/phyloseq/issues/537","id":110328351,"number":537,"title":"Speedup Tax_Glom","user":{"login":"zachcp","id":1260667,"avatar_url":"https://avatars.githubusercontent.com/u/1260667?v=3","gravatar_id":"","url":"https://api.github.com/users/zachcp","html_url":"https://github.com/zachcp","followers_url":"https://api.github.com/users/zachcp/followers","following_url":"https://api.github.com/users/zachcp/following{/other_user}","gists_url":"https://api.github.com/users/zachcp/gists{/gist_id}","starred_url":"https://api.github.com/users/zachcp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/zachcp/subscriptions","organizations_url":"https://api.github.com/users/zachcp/orgs","repos_url":"https://api.github.com/users/zachcp/repos","events_url":"https://api.github.com/users/zachcp/events{/privacy}","received_events_url":"https://api.github.com/users/zachcp/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2015-10-07T22:08:30Z","updated_at":"2016-03-14T21:45:50Z","closed_at":null,"body":"Hi @joey711,\r\n\r\nIn response to #517 I have a suggestion on how to handle the OTU merging when merging across something large like an entire Taxonomy class. If we add the taxonomy data to the OTUtable, we can do the split-apply-recombine strategy to calculate all of the merged OTUs. \r\n\r\nThe code below shows a better-than 10x speedup over `merge_taxa` for merging the OTU table. it may not end up being quite as fast since the OTUs it returns are not correct. It does NOT output the correct OTU for each taxonomy group. i would expect this to be even faster as the OTU tables get bigger. Is the output simple the most abundant OTU in the group?\r\n\r\n I would appreciate any feedback and if it looks good can prepare a pull request.\r\n\r\n\r\n\r\n\r\nthanks,\r\nzach cp\r\n\r\n\r\n```{r}\r\nlibrary(phyloseq)\r\ndata(\"GlobalPatterns\")\r\n\r\n\r\n#'  Helper Function that will take the OTU Table (as a data.frame()) \r\n#'  to which column representing taxa has been added.\r\n#'  There is some munging to handle rownames and\r\n#'  the tax column but the basic idea is there.\r\ncondenseOTUs <- function(otutable, splitcol) {\r\n  \r\n  #convert tax col to numeric\r\n  # needed to allow colSums to work\r\n  otutable[[splitcol]] <- as.numeric(factor(otutable[[splitcol]], labels=c(1:length(unique(otutable[[splitcol]])))))\r\n  \r\n  # split apply, combine\r\n  splits   <- split(otutable, otutable[[splitcol]])  \r\n  summed   <- Map(colSums, splits)\r\n  summeddf <- Reduce(rbind, summed, init = NULL)\r\n\r\n  #get rownames.\r\n  # this needs to be changed to get most abundant OTU if thats what is used elsewhere\r\n  # rightnow it returns the first OTU in the group\r\n  newrownames <- Map(function(x){rownames(x)[[1]]}, splits)\r\n  \r\n  #add back rowname and remove tax column\r\n  rownames(summeddf) <- newrownames\r\n  summeddf[, !colnames(summeddf) %in% c(splitcol)]\r\n}\r\n\r\n\r\n# setup data. we are going to glob the OTU table based on the Class Taxonomy\r\nGP  <- GlobalPatterns\r\n# the following code directly from tax_glom\r\nCN  <- which( rank_names(GP) %in% \"Class\" )\r\ntax <- as(access(GP, \"tax_table\"), \"matrix\")[, 1:CN, drop=FALSE]\r\ntax <- apply(tax, 1, function(i){paste(i, sep=\";_;\", collapse=\";_;\")})\r\nspCliques <- tapply(names(tax), factor(tax), list)\r\n\r\n# using merge_taxa: current Phyloseq\r\nptm <- proc.time()\r\notab1 <- otu_table(GP)\r\nfor( i in names(spCliques)){\r\n  otab1 <- merge_taxa(otab1, spCliques[[i]])\r\n}\r\nproc.time() - ptm\r\n\r\n# using Map-Reduce/vectorized\r\nptm <- proc.time()\r\notab2 <- data.frame(otu_table(GP))\r\ntaxdf <- data.frame(tax)\r\notab2 <- merge(otab2, taxdf, by = \"row.names\")\r\nrow.names(otab2) <- otab2$Row.names\r\notab2 <- otab2[ , 2:ncol(otab2)]\r\notab2 <- condenseOTUs(otab2,\"tax\")\r\nproc.time() - ptm\r\n\r\n#test equivalents\r\n#good\r\nsum(otab1) == sum(otab2)\r\ndim(otab1) == dim(otab2)\r\nsort(rowSums(otab1)) == sort(rowSums(otab2))\r\nsort(colSums(otab1)) == sort(colSums(otab2))\r\n\r\n#notgood: need to obtain the correct OTUs\r\nsort(rownames(otab1)) == sort(rownames(otab2))\r\nrownames(otab1) %in% rownames(otab2)\r\n\r\n\r\n```\r\n\r\n\r\nI setup two tests using the GP code. Using the current `merge_taxa` i get:\r\n```{r}\r\n# current\r\n proc.time() - ptm\r\n   user  system elapsed \r\n  6.864   0.960   7.820 \r\n#vectorized\r\n> proc.time() - ptm\r\n   user  system elapsed \r\n  0.445   0.101   0.543 \r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/joey711/phyloseq/issues/comments/196535220","html_url":"https://github.com/joey711/phyloseq/issues/537#issuecomment-196535220","issue_url":"https://api.github.com/repos/joey711/phyloseq/issues/537","id":196535220,"user":{"login":"colinbrislawn","id":10355152,"avatar_url":"https://avatars.githubusercontent.com/u/10355152?v=3","gravatar_id":"","url":"https://api.github.com/users/colinbrislawn","html_url":"https://github.com/colinbrislawn","followers_url":"https://api.github.com/users/colinbrislawn/followers","following_url":"https://api.github.com/users/colinbrislawn/following{/other_user}","gists_url":"https://api.github.com/users/colinbrislawn/gists{/gist_id}","starred_url":"https://api.github.com/users/colinbrislawn/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/colinbrislawn/subscriptions","organizations_url":"https://api.github.com/users/colinbrislawn/orgs","repos_url":"https://api.github.com/users/colinbrislawn/repos","events_url":"https://api.github.com/users/colinbrislawn/events{/privacy}","received_events_url":"https://api.github.com/users/colinbrislawn/received_events","type":"User","site_admin":false},"created_at":"2016-03-14T21:45:50Z","updated_at":"2016-03-14T21:45:50Z","body":"Any progress on this? I'd love to see this implemented, as everything else in phyloseq is already so speedy! \r\nThank you for building and improving this software.\r\nColin "}}