{"action":"created","issue":{"url":"https://api.github.com/repos/msmbuilder/msmbuilder/issues/770","repository_url":"https://api.github.com/repos/msmbuilder/msmbuilder","labels_url":"https://api.github.com/repos/msmbuilder/msmbuilder/issues/770/labels{/name}","comments_url":"https://api.github.com/repos/msmbuilder/msmbuilder/issues/770/comments","events_url":"https://api.github.com/repos/msmbuilder/msmbuilder/issues/770/events","html_url":"https://github.com/msmbuilder/msmbuilder/pull/770","id":143103134,"number":770,"title":"added ability to normalize large datasets in memory","user":{"login":"evanfeinberg","id":1760982,"avatar_url":"https://avatars.githubusercontent.com/u/1760982?v=3","gravatar_id":"","url":"https://api.github.com/users/evanfeinberg","html_url":"https://github.com/evanfeinberg","followers_url":"https://api.github.com/users/evanfeinberg/followers","following_url":"https://api.github.com/users/evanfeinberg/following{/other_user}","gists_url":"https://api.github.com/users/evanfeinberg/gists{/gist_id}","starred_url":"https://api.github.com/users/evanfeinberg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/evanfeinberg/subscriptions","organizations_url":"https://api.github.com/users/evanfeinberg/orgs","repos_url":"https://api.github.com/users/evanfeinberg/repos","events_url":"https://api.github.com/users/evanfeinberg/events{/privacy}","received_events_url":"https://api.github.com/users/evanfeinberg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/msmbuilder/msmbuilder/milestones/6","html_url":"https://github.com/msmbuilder/msmbuilder/milestones/v3.5","labels_url":"https://api.github.com/repos/msmbuilder/msmbuilder/milestones/6/labels","id":1382452,"number":6,"title":"v3.5","description":"","creator":{"login":"mpharrigan","id":4967059,"avatar_url":"https://avatars.githubusercontent.com/u/4967059?v=3","gravatar_id":"","url":"https://api.github.com/users/mpharrigan","html_url":"https://github.com/mpharrigan","followers_url":"https://api.github.com/users/mpharrigan/followers","following_url":"https://api.github.com/users/mpharrigan/following{/other_user}","gists_url":"https://api.github.com/users/mpharrigan/gists{/gist_id}","starred_url":"https://api.github.com/users/mpharrigan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpharrigan/subscriptions","organizations_url":"https://api.github.com/users/mpharrigan/orgs","repos_url":"https://api.github.com/users/mpharrigan/repos","events_url":"https://api.github.com/users/mpharrigan/events{/privacy}","received_events_url":"https://api.github.com/users/mpharrigan/received_events","type":"User","site_admin":false},"open_issues":21,"closed_issues":7,"state":"open","created_at":"2015-10-28T23:21:13Z","updated_at":"2016-04-07T18:34:44Z","due_on":"2016-05-30T07:00:00Z","closed_at":null},"comments":23,"created_at":"2016-03-23T23:11:36Z","updated_at":"2016-04-07T23:47:24Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/msmbuilder/msmbuilder/pulls/770","html_url":"https://github.com/msmbuilder/msmbuilder/pull/770","diff_url":"https://github.com/msmbuilder/msmbuilder/pull/770.diff","patch_url":"https://github.com/msmbuilder/msmbuilder/pull/770.patch"},"body":" This PR adds a \"preprocessing\" section which is inspired by the approach of geometric clustering in `MSMBuilder`: take standard `sklearn` methods and expand them to handle `list`s of `np.array`s. This is more convenient, as nearly all `sklearn` methods assume that `X` is a single `np.array` whereas MD data is mostly stored in lists of arrays. \r\n\r\nLarge featurized [MD] datasets are often large compared to total RAM available on a system. If size of the MD dataset is comparable to the total RAM available, then computing the mean and variance with standard `numpy` functions can be tricky. Since concatenating the data essentially doubles the RAM that needs to be used, the usual way of computing mean and variance can in turn make the user exceed the memory limits of the system.\r\n\r\nTo begin, I implemented a numerically stable method described in https://en.wikipedia.org/wiki/Algorithms_for_calculating_variance#Compensated_variant. In addition to the normal `fit`, `transform`, and `fit_transform` methods, I added `fit_transform_in_place` and `transform_in_place`, so one can pass in un-normalized features and compute normalized features without adding to the memory burden of one's computation. \r\n\r\nSome to-do's: add whitening and other preprocessing capabilities from `sklearn` to `msmbuilder.preprocessing`. \r\n\r\nComments or criticisms are welcome. Thank you to @cxhernandez  in particular for his help with this PR."},"comment":{"url":"https://api.github.com/repos/msmbuilder/msmbuilder/issues/comments/207139793","html_url":"https://github.com/msmbuilder/msmbuilder/pull/770#issuecomment-207139793","issue_url":"https://api.github.com/repos/msmbuilder/msmbuilder/issues/770","id":207139793,"user":{"login":"mpharrigan","id":4967059,"avatar_url":"https://avatars.githubusercontent.com/u/4967059?v=3","gravatar_id":"","url":"https://api.github.com/users/mpharrigan","html_url":"https://github.com/mpharrigan","followers_url":"https://api.github.com/users/mpharrigan/followers","following_url":"https://api.github.com/users/mpharrigan/following{/other_user}","gists_url":"https://api.github.com/users/mpharrigan/gists{/gist_id}","starred_url":"https://api.github.com/users/mpharrigan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mpharrigan/subscriptions","organizations_url":"https://api.github.com/users/mpharrigan/orgs","repos_url":"https://api.github.com/users/mpharrigan/repos","events_url":"https://api.github.com/users/mpharrigan/events{/privacy}","received_events_url":"https://api.github.com/users/mpharrigan/received_events","type":"User","site_admin":false},"created_at":"2016-04-07T23:47:24Z","updated_at":"2016-04-07T23:47:24Z","body":"For the second point 2, perhaps interested parties can look into this\r\n\r\nhttp://scikit-learn.org/stable/modules/preprocessing.html\r\n\r\nhttp://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html#sklearn.preprocessing.StandardScaler\r\n\r\nI'd be more comfortable using a thin wrapper around sklearn. At the very least, we should copy their api design. It's essentially the second part of my second number 1: they use an init arg `copy`. When set to false, it does stuff in place"}}