{"action":"created","issue":{"url":"https://api.github.com/repos/hyperspy/hyperspy/issues/1128","repository_url":"https://api.github.com/repos/hyperspy/hyperspy","labels_url":"https://api.github.com/repos/hyperspy/hyperspy/issues/1128/labels{/name}","comments_url":"https://api.github.com/repos/hyperspy/hyperspy/issues/1128/comments","events_url":"https://api.github.com/repos/hyperspy/hyperspy/issues/1128/events","html_url":"https://github.com/hyperspy/hyperspy/pull/1128","id":165049016,"number":1128,"title":"Robust PCA & Online RPCA algorithms","user":{"login":"tjof2","id":6626247,"avatar_url":"https://avatars.githubusercontent.com/u/6626247?v=3","gravatar_id":"","url":"https://api.github.com/users/tjof2","html_url":"https://github.com/tjof2","followers_url":"https://api.github.com/users/tjof2/followers","following_url":"https://api.github.com/users/tjof2/following{/other_user}","gists_url":"https://api.github.com/users/tjof2/gists{/gist_id}","starred_url":"https://api.github.com/users/tjof2/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tjof2/subscriptions","organizations_url":"https://api.github.com/users/tjof2/orgs","repos_url":"https://api.github.com/users/tjof2/repos","events_url":"https://api.github.com/users/tjof2/events{/privacy}","received_events_url":"https://api.github.com/users/tjof2/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/hyperspy/hyperspy/labels/1.0","name":"1.0","color":"fef2c0"},{"url":"https://api.github.com/repos/hyperspy/hyperspy/labels/New","name":"New","color":"f7c6c7"}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/hyperspy/hyperspy/milestones/8","html_url":"https://github.com/hyperspy/hyperspy/milestones/1.0.0","labels_url":"https://api.github.com/repos/hyperspy/hyperspy/milestones/8/labels","id":856137,"number":8,"title":"1.0.0","description":"","creator":{"login":"francisco-dlp","id":989719,"avatar_url":"https://avatars.githubusercontent.com/u/989719?v=3","gravatar_id":"","url":"https://api.github.com/users/francisco-dlp","html_url":"https://github.com/francisco-dlp","followers_url":"https://api.github.com/users/francisco-dlp/followers","following_url":"https://api.github.com/users/francisco-dlp/following{/other_user}","gists_url":"https://api.github.com/users/francisco-dlp/gists{/gist_id}","starred_url":"https://api.github.com/users/francisco-dlp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/francisco-dlp/subscriptions","organizations_url":"https://api.github.com/users/francisco-dlp/orgs","repos_url":"https://api.github.com/users/francisco-dlp/repos","events_url":"https://api.github.com/users/francisco-dlp/events{/privacy}","received_events_url":"https://api.github.com/users/francisco-dlp/received_events","type":"User","site_admin":false},"open_issues":3,"closed_issues":175,"state":"open","created_at":"2014-11-06T15:54:23Z","updated_at":"2016-07-14T19:56:29Z","due_on":"2016-07-07T23:00:00Z","closed_at":null},"comments":12,"created_at":"2016-07-12T10:49:01Z","updated_at":"2016-07-15T14:23:56Z","closed_at":"2016-07-13T08:44:29Z","pull_request":{"url":"https://api.github.com/repos/hyperspy/hyperspy/pulls/1128","html_url":"https://github.com/hyperspy/hyperspy/pull/1128","diff_url":"https://github.com/hyperspy/hyperspy/pull/1128.diff","patch_url":"https://github.com/hyperspy/hyperspy/pull/1128.patch"},"body":"This PR takes over from #1071, which I broke.\r\n\r\n----\r\n\r\nRobust PCA seems to be making its way into scikit-learn, slowly... (see #972), but with one of the slow, iterative, memory-intensive SVT algorithms. However, I've been using this example of \"online\" robust PCA for a while now for outlier detection and working with missing data, which is much less memory-intensive and actually isn't too slow.\r\n\r\nSince OR-PCA actually returns a low-rank component **L** and a sparse error component **S** (where the observations **X** = **L** + **S**), to fit this into the current HyperSpy decomposition framework, I carry out a subsequent SVD on **L** to return factors and loadings.\r\n\r\nAlthough I've not currently implemented it in a way that allows for it, you can actually run OR-PCA in a truly online manner, i.e. update the estimated factors and loadings with each new sample that arrives, without having to re-run over the whole dataset. However, I think this quickly becomes complicated in the current HyperSpy machine learning framework, so consider this implementation more aimed at outlier detection.\r\n\r\nYou can see an example here: https://github.com/tjof2/hyperspy/blob/my-demos/examples/Online%20Robust%20PCA.ipynb\r\n\r\n- [x] Documentation\r\n- [x] Write simple tests\r\n- [x] Option to feed in an initial mask for the sparse component **S**. \r\n- [x] Add batch method (GoDec) for comparison with online version\r\n- [x] Add new QR initialization method (discussion with @thomasaarholt)\r\n- [x] Add stochastic gradient descent (SGD) method - faster for large datasets (high-dimensional)"},"comment":{"url":"https://api.github.com/repos/hyperspy/hyperspy/issues/comments/232964765","html_url":"https://github.com/hyperspy/hyperspy/pull/1128#issuecomment-232964765","issue_url":"https://api.github.com/repos/hyperspy/hyperspy/issues/1128","id":232964765,"user":{"login":"francisco-dlp","id":989719,"avatar_url":"https://avatars.githubusercontent.com/u/989719?v=3","gravatar_id":"","url":"https://api.github.com/users/francisco-dlp","html_url":"https://github.com/francisco-dlp","followers_url":"https://api.github.com/users/francisco-dlp/followers","following_url":"https://api.github.com/users/francisco-dlp/following{/other_user}","gists_url":"https://api.github.com/users/francisco-dlp/gists{/gist_id}","starred_url":"https://api.github.com/users/francisco-dlp/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/francisco-dlp/subscriptions","organizations_url":"https://api.github.com/users/francisco-dlp/orgs","repos_url":"https://api.github.com/users/francisco-dlp/repos","events_url":"https://api.github.com/users/francisco-dlp/events{/privacy}","received_events_url":"https://api.github.com/users/francisco-dlp/received_events","type":"User","site_admin":false},"created_at":"2016-07-15T14:23:56Z","updated_at":"2016-07-15T14:23:56Z","body":"@thomasaarholt, you definitely have a point. Would you like to open a separate issue about it?"}}
{"actor":{"display_login":"francisco-dlp"}}