{"action":"created","issue":{"url":"https://api.github.com/repos/privefl/bigstatsr/issues/44","repository_url":"https://api.github.com/repos/privefl/bigstatsr","labels_url":"https://api.github.com/repos/privefl/bigstatsr/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/privefl/bigstatsr/issues/44/comments","events_url":"https://api.github.com/repos/privefl/bigstatsr/issues/44/events","html_url":"https://github.com/privefl/bigstatsr/issues/44","id":381903246,"node_id":"MDU6SXNzdWUzODE5MDMyNDY=","number":44,"title":"Low CPU, high RAM usage with big_parallelize","user":{"login":"markabney","id":5862451,"node_id":"MDQ6VXNlcjU4NjI0NTE=","avatar_url":"https://avatars2.githubusercontent.com/u/5862451?v=4","gravatar_id":"","url":"https://api.github.com/users/markabney","html_url":"https://github.com/markabney","followers_url":"https://api.github.com/users/markabney/followers","following_url":"https://api.github.com/users/markabney/following{/other_user}","gists_url":"https://api.github.com/users/markabney/gists{/gist_id}","starred_url":"https://api.github.com/users/markabney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markabney/subscriptions","organizations_url":"https://api.github.com/users/markabney/orgs","repos_url":"https://api.github.com/users/markabney/repos","events_url":"https://api.github.com/users/markabney/events{/privacy}","received_events_url":"https://api.github.com/users/markabney/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-11-17T21:33:44Z","updated_at":"2018-11-28T21:58:21Z","closed_at":null,"author_association":"NONE","body":"I have an FBM with 33k rows and 200k columns. The entries are actually chi square values with different degrees of freedom for different rows. I am converting them to standard normal values by using big_parallelize.\r\nMy machine:\r\nmacOS 10.13.6\r\n128GB RAM\r\nnb_cores() -> 12\r\n\r\nUsing the code below, big_parallelize forks several (12) processes which all stay at at CPU usage of 0.1% to 0.2%. Each process initially gets allocated ~4GB RAM. Over the course of an hour or two RAM usage climbs, particularly for kernel_task, eventually resulting in the processes RAM being mostly compressed memory and all RAM on the machine is used. I'm forced to kill the R process at this point.\r\n\r\n```\r\ninter.fbm <- FBM(n.path.snps, n.host.snps, backingfile=\"../Data/interaction_matrix_new\", create_bk=FALSE)\r\nchi2z.par <- function(X, ind, df, Out.mat) {\r\n    chi2Z <- function(chi, df) {\r\n        qnorm(pchisq(chi, df=df))\r\n    }\r\n    chi2z.col <- function(M, df) {\r\n        apply(M, 2, chi2Z, df=df)\r\n    }\r\n    Out.mat[ , ind] <- chi2z.col(X[ , ind], df=df)\r\n    NULL\r\n}\r\n\r\ndestfile <- \"../Data/z_mat\"\r\nres.mat <- FBM(nrow(inter.fbm), ncol(inter.fbm), backingfile=destfile)\r\nbig_parallelize(inter.fbm, p.FUN=chi2z.par, ncores=nb_cores(), df=row.info$df, Out.mat=res.mat)\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/privefl/bigstatsr/issues/comments/442620510","html_url":"https://github.com/privefl/bigstatsr/issues/44#issuecomment-442620510","issue_url":"https://api.github.com/repos/privefl/bigstatsr/issues/44","id":442620510,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MjYyMDUxMA==","user":{"login":"markabney","id":5862451,"node_id":"MDQ6VXNlcjU4NjI0NTE=","avatar_url":"https://avatars2.githubusercontent.com/u/5862451?v=4","gravatar_id":"","url":"https://api.github.com/users/markabney","html_url":"https://github.com/markabney","followers_url":"https://api.github.com/users/markabney/followers","following_url":"https://api.github.com/users/markabney/following{/other_user}","gists_url":"https://api.github.com/users/markabney/gists{/gist_id}","starred_url":"https://api.github.com/users/markabney/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/markabney/subscriptions","organizations_url":"https://api.github.com/users/markabney/orgs","repos_url":"https://api.github.com/users/markabney/repos","events_url":"https://api.github.com/users/markabney/events{/privacy}","received_events_url":"https://api.github.com/users/markabney/received_events","type":"User","site_admin":false},"created_at":"2018-11-28T21:58:21Z","updated_at":"2018-11-28T21:58:21Z","author_association":"NONE","body":"There's nothing special about the data itself, just that it's a large matrix. You could do the following.\r\n```\r\nnr <- 3e4\r\nnc <- 2e5\r\nset.seed(1)\r\nchi2.fbm[] <- rchisq(nr * nc, 1)\r\n```\r\nThe chi2.fbm is what I use in my code (with real rather than simulated values)."}}
{"actor":{"display_login":"markabney"}}