{"action":"created","issue":{"url":"https://api.github.com/repos/YosefLab/scVI-dev/issues/93","repository_url":"https://api.github.com/repos/YosefLab/scVI-dev","labels_url":"https://api.github.com/repos/YosefLab/scVI-dev/issues/93/labels{/name}","comments_url":"https://api.github.com/repos/YosefLab/scVI-dev/issues/93/comments","events_url":"https://api.github.com/repos/YosefLab/scVI-dev/issues/93/events","html_url":"https://github.com/YosefLab/scVI-dev/pull/93","id":320369818,"number":93,"title":"Brain large benchmark","user":{"login":"Edouard360","id":15527397,"avatar_url":"https://avatars3.githubusercontent.com/u/15527397?v=4","gravatar_id":"","url":"https://api.github.com/users/Edouard360","html_url":"https://github.com/Edouard360","followers_url":"https://api.github.com/users/Edouard360/followers","following_url":"https://api.github.com/users/Edouard360/following{/other_user}","gists_url":"https://api.github.com/users/Edouard360/gists{/gist_id}","starred_url":"https://api.github.com/users/Edouard360/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Edouard360/subscriptions","organizations_url":"https://api.github.com/users/Edouard360/orgs","repos_url":"https://api.github.com/users/Edouard360/repos","events_url":"https://api.github.com/users/Edouard360/events{/privacy}","received_events_url":"https://api.github.com/users/Edouard360/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":6,"created_at":"2018-05-04T17:48:50Z","updated_at":"2018-05-06T20:31:20Z","closed_at":"2018-05-04T18:27:36Z","author_association":"COLLABORATOR","pull_request":{"url":"https://api.github.com/repos/YosefLab/scVI-dev/pulls/93","html_url":"https://github.com/YosefLab/scVI-dev/pull/93","diff_url":"https://github.com/YosefLab/scVI-dev/pull/93.diff","patch_url":"https://github.com/YosefLab/scVI-dev/pull/93.patch"},"body":"Brain Large preprocessing was until now, a bit different from Romain's. (In particular, batch indices weren't taken into account). \r\n\r\nRomain's exact preprocessing is at c13928e.If we care exactly about having same train/test dataset, we can look at this previous commit. \r\n\r\nOtherwise, the same results are also reproducible taking the subsample size: 60000 in Romain's notebook.\r\n\r\n```\r\ngene_dataset = BrainLargeDataset(subsample_size=60000)\r\nrun_benchmarks(gene_dataset, n_epochs=250, use_batches=False, tt_split=0.8)\r\n```\r\n\r\nThis will give train/test 48000/12000 ~= 50000/10000 in Romain's notebook. The same final likelihood score + imputation score are achieved  on test set : **LL = 1148, MAE = 1.07**\r\n\r\nIt takes 1119 sec for the 250 epochs. I think Romain does only 120 epochs, and indeed, after 40 epochs we reach the final log likelihood so we could at least reduce that training time to ~180 sec = 3min. with early stopping.\r\n\r\nAs for the Cortex Dataset, 250 epochs of training give  **LL = 1197, MAE = 2.18**, which is good for MAE, and a little above for LL test (~ 1180 in Romain's preprint), even using **adapt_encoder** new function. But imo with a little tuning (learning rate, batch size), Cortex result's are easily reproducible.\r\n\r\nIn terms of timing / GPU usage, **nvidia-smi => 40/50%** due to the `data_loader.next` procedure.\r\n"},"comment":{"url":"https://api.github.com/repos/YosefLab/scVI-dev/issues/comments/386910901","html_url":"https://github.com/YosefLab/scVI-dev/pull/93#issuecomment-386910901","issue_url":"https://api.github.com/repos/YosefLab/scVI-dev/issues/93","id":386910901,"user":{"login":"Edouard360","id":15527397,"avatar_url":"https://avatars3.githubusercontent.com/u/15527397?v=4","gravatar_id":"","url":"https://api.github.com/users/Edouard360","html_url":"https://github.com/Edouard360","followers_url":"https://api.github.com/users/Edouard360/followers","following_url":"https://api.github.com/users/Edouard360/following{/other_user}","gists_url":"https://api.github.com/users/Edouard360/gists{/gist_id}","starred_url":"https://api.github.com/users/Edouard360/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Edouard360/subscriptions","organizations_url":"https://api.github.com/users/Edouard360/orgs","repos_url":"https://api.github.com/users/Edouard360/repos","events_url":"https://api.github.com/users/Edouard360/events{/privacy}","received_events_url":"https://api.github.com/users/Edouard360/received_events","type":"User","site_admin":false},"created_at":"2018-05-06T20:31:20Z","updated_at":"2018-05-06T20:31:20Z","author_association":"COLLABORATOR","body":"I will investigate the performance gain from that approach ! Thank you for the idea !"}}
{"actor":{"display_login":"Edouard360"}}