{"action":"created","issue":{"url":"https://api.github.com/repos/chriscainx/mnnpy/issues/4","repository_url":"https://api.github.com/repos/chriscainx/mnnpy","labels_url":"https://api.github.com/repos/chriscainx/mnnpy/issues/4/labels{/name}","comments_url":"https://api.github.com/repos/chriscainx/mnnpy/issues/4/comments","events_url":"https://api.github.com/repos/chriscainx/mnnpy/issues/4/events","html_url":"https://github.com/chriscainx/mnnpy/issues/4","id":321330763,"number":4,"title":"Error during cosine normalization step","user":{"login":"LucasESBS","id":26789402,"avatar_url":"https://avatars1.githubusercontent.com/u/26789402?v=4","gravatar_id":"","url":"https://api.github.com/users/LucasESBS","html_url":"https://github.com/LucasESBS","followers_url":"https://api.github.com/users/LucasESBS/followers","following_url":"https://api.github.com/users/LucasESBS/following{/other_user}","gists_url":"https://api.github.com/users/LucasESBS/gists{/gist_id}","starred_url":"https://api.github.com/users/LucasESBS/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/LucasESBS/subscriptions","organizations_url":"https://api.github.com/users/LucasESBS/orgs","repos_url":"https://api.github.com/users/LucasESBS/repos","events_url":"https://api.github.com/users/LucasESBS/events{/privacy}","received_events_url":"https://api.github.com/users/LucasESBS/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-08T20:14:05Z","updated_at":"2018-05-09T03:35:24Z","closed_at":null,"author_association":"NONE","body":"Hello,\r\n\r\nFirst, thank you for this awesome implementation of MNN in python , great work! I was looking forward to try it and compare it with Seurat's CCA on my 5 batches, but got the following error when running the mnn_correct function: \r\n```python\r\n---------------------------------------------------------------------------\r\nerror                                     Traceback (most recent call last)\r\n<ipython-input-35-cf9c40cb7f9e> in <module>()\r\n     11 sc.pp.log1p(week11_LV)\r\n     12 \r\n---> 13 corrected = mnnpy.mnn_correct(week11_IVS, week11_LA, week11_RA, week11_RV, week11_LV,var_subset=subset_genes, batch_categories = [\"EF11W4D_IVS\", \"EF11W4D_LA\", \"EF11W4D_RA\",\"EF11W4D_RV\",\"EF11W4D_LV\"])\r\n     14 adata = corrected[0]\r\n\r\n~/anaconda3/lib/python3.6/site-packages/mnnpy/mnn.py in mnn_correct(var_index, var_subset, batch_key, index_unique, batch_categories, k, sigma, cos_norm_in, cos_norm_out, svd_dim, var_adj, compute_angle, mnn_order, svd_mode, do_concatenate, save_raw, n_jobs, *datas, **kwargs)\r\n    124                                 cos_norm_out=cos_norm_out, svd_dim=svd_dim, var_adj=var_adj,\r\n    125                                 compute_angle=compute_angle, mnn_order=mnn_order,\r\n--> 126                                 svd_mode=svd_mode, do_concatenate=do_concatenate, **kwargs)\r\n    127         print('Packing AnnData object...')\r\n    128         if do_concatenate:\r\n\r\n~/anaconda3/lib/python3.6/site-packages/mnnpy/mnn.py in mnn_correct(var_index, var_subset, batch_key, index_unique, batch_categories, k, sigma, cos_norm_in, cos_norm_out, svd_dim, var_adj, compute_angle, mnn_order, svd_mode, do_concatenate, save_raw, n_jobs, *datas, **kwargs)\r\n    155     in_batches, out_batches, var_subset, same_set = transform_input_data(datas, cos_norm_in,\r\n    156                                                                          cos_norm_out, var_index,\r\n--> 157                                                                          var_subset, n_jobs)\r\n    158     if mnn_order is None:\r\n    159         mnn_order = list(range(n_batch))\r\n\r\n~/anaconda3/lib/python3.6/site-packages/mnnpy/utils.py in transform_input_data(datas, cos_norm_in, cos_norm_out, var_index, var_subset, n_jobs)\r\n     59         else:\r\n     60             with Pool(n_jobs) as p_n:\r\n---> 61                 out_scaling = p_n.map(l2_norm, datas)\r\n     62                 out_scaling = [scaling[:, None] for scaling in out_scaling]\r\n     63                 out_batches = p_n.starmap(scale_rows, zip(datas, out_scaling))\r\n\r\n~/anaconda3/lib/python3.6/multiprocessing/pool.py in map(self, func, iterable, chunksize)\r\n    264         in a list that is returned.\r\n    265         '''\r\n--> 266         return self._map_async(func, iterable, mapstar, chunksize).get()\r\n    267 \r\n    268     def starmap(self, func, iterable, chunksize=None):\r\n\r\n~/anaconda3/lib/python3.6/multiprocessing/pool.py in get(self, timeout)\r\n    642             return self._value\r\n    643         else:\r\n--> 644             raise self._value\r\n    645 \r\n    646     def _set(self, i, obj):\r\n\r\n~/anaconda3/lib/python3.6/multiprocessing/pool.py in _handle_tasks(taskqueue, put, outqueue, pool, cache)\r\n    422                         break\r\n    423                     try:\r\n--> 424                         put(task)\r\n    425                     except Exception as e:\r\n    426                         job, idx = task[:2]\r\n\r\n~/anaconda3/lib/python3.6/multiprocessing/connection.py in send(self, obj)\r\n    204         self._check_closed()\r\n    205         self._check_writable()\r\n--> 206         self._send_bytes(_ForkingPickler.dumps(obj))\r\n    207 \r\n    208     def recv_bytes(self, maxlength=None):\r\n\r\n~/anaconda3/lib/python3.6/multiprocessing/connection.py in _send_bytes(self, buf)\r\n    391         n = len(buf)\r\n    392         # For wire compatibility with 3.2 and lower\r\n--> 393         header = struct.pack(\"!i\", n)\r\n    394         if n > 16384:\r\n    395             # The payload is large so Nagle's algorithm won't be triggered\r\n\r\nerror: 'i' format requires -2147483648 <= number <= 2147483647 \r\n```\r\nI used 500 highly variable genes; is that too much ? Can it be for another reason? In general, how do the computational time scales with the number of genes selected ? It would be nice to have an idea of a good number of HVGs to be included to have a decent running time without losing too much information. \r\n\r\nThank you, and once again congrats for your work!"},"comment":{"url":"https://api.github.com/repos/chriscainx/mnnpy/issues/comments/387610098","html_url":"https://github.com/chriscainx/mnnpy/issues/4#issuecomment-387610098","issue_url":"https://api.github.com/repos/chriscainx/mnnpy/issues/4","id":387610098,"user":{"login":"chriscainx","id":8361080,"avatar_url":"https://avatars0.githubusercontent.com/u/8361080?v=4","gravatar_id":"","url":"https://api.github.com/users/chriscainx","html_url":"https://github.com/chriscainx","followers_url":"https://api.github.com/users/chriscainx/followers","following_url":"https://api.github.com/users/chriscainx/following{/other_user}","gists_url":"https://api.github.com/users/chriscainx/gists{/gist_id}","starred_url":"https://api.github.com/users/chriscainx/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chriscainx/subscriptions","organizations_url":"https://api.github.com/users/chriscainx/orgs","repos_url":"https://api.github.com/users/chriscainx/repos","events_url":"https://api.github.com/users/chriscainx/events{/privacy}","received_events_url":"https://api.github.com/users/chriscainx/received_events","type":"User","site_admin":false},"created_at":"2018-05-09T03:35:24Z","updated_at":"2018-05-09T03:35:24Z","author_association":"OWNER","body":"Regarding time consumption, if you choose to input all the genes (g) but subset with hvgs (h), the algorithm will run twice (m*n*g + m*n*h, m=batch 1, n=batch 2), because the whole space needs to be projected using the MNNs calculated using h. Calculating MNN uses only h, but it’s fast. When h is small, the majority of time is consumed by all gene computation, changing h will not affect running time very much.\r\n\r\nIf you want to speed it up, a good plan would be subset the genes to hvgs first, and do no subsetting, since other genes are typically unnecessary in later steps (PCA, cluster) anyway... In this case, the time will be m*n*h. I think this is most efficient.\r\n\r\nActually, what’s the specs of your machine? It’s possible that your problem has something to do with it."}}
{"actor":{"display_login":"chriscainx"}}