{"action":"created","issue":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/767","repository_url":"https://api.github.com/repos/markovmodel/PyEMMA","labels_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/767/labels{/name}","comments_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/767/comments","events_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/767/events","html_url":"https://github.com/markovmodel/PyEMMA/issues/767","id":147245196,"number":767,"title":"kmeans issue","user":{"login":"franknoe","id":6495810,"avatar_url":"https://avatars.githubusercontent.com/u/6495810?v=3","gravatar_id":"","url":"https://api.github.com/users/franknoe","html_url":"https://github.com/franknoe","followers_url":"https://api.github.com/users/franknoe/followers","following_url":"https://api.github.com/users/franknoe/following{/other_user}","gists_url":"https://api.github.com/users/franknoe/gists{/gist_id}","starred_url":"https://api.github.com/users/franknoe/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/franknoe/subscriptions","organizations_url":"https://api.github.com/users/franknoe/orgs","repos_url":"https://api.github.com/users/franknoe/repos","events_url":"https://api.github.com/users/franknoe/events{/privacy}","received_events_url":"https://api.github.com/users/franknoe/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-04-10T17:41:37Z","updated_at":"2016-04-11T09:45:09Z","closed_at":null,"body":"Calling k-means clustering to a large in-memory array (list of ndarrays) results in this error. The immediate error is that the attribute ```self._oom_strategy``` doesn't exist, but I think it shouldn't even go into this code branch, because the data are already in memory and don't need to be loaded.\r\n\r\n```python\r\n---------------------------------------------------------------------------\r\nAttributeError                            Traceback (most recent call last)\r\n<ipython-input-19-0ef322ffb797> in <module>()\r\n      1 # cluster the full state space\r\n----> 2 kmeans = coor.cluster_kmeans(data=X, k=2000)\r\n\r\n/Users/noe/data/software_projects/pyemma/pyemma/coordinates/api.pyc in cluster_kmeans(data, k, max_iter, tolerance, stride, metric, init_strategy, fixed_seed)\r\n   1281     res = KmeansClustering(n_clusters=k, max_iter=max_iter, metric=metric, tolerance=tolerance,\r\n   1282                             init_strategy=init_strategy, fixed_seed=fixed_seed)\r\n-> 1283     return _param_stage(data, res, stride=stride)\r\n   1284 \r\n   1285 \r\n\r\n/Users/noe/data/software_projects/pyemma/pyemma/coordinates/api.pyc in _param_stage(previous_stage, this_stage, stride)\r\n    813     this_stage.data_producer = input_stage\r\n    814     this_stage.chunksize = input_stage.chunksize\r\n--> 815     this_stage.estimate(X=input_stage, stride=stride)\r\n    816     return this_stage\r\n    817 \r\n\r\n/Users/noe/data/software_projects/pyemma/pyemma/coordinates/transform/transformer.pyc in estimate(self, X, **kwargs)\r\n    187         # run estimation\r\n    188         try:\r\n--> 189             super(StreamingTransformer, self).estimate(X, **kwargs)\r\n    190         except NotConvergedWarning as ncw:\r\n    191             self._logger.info(\r\n\r\n/Users/noe/data/software_projects/pyemma/pyemma/_base/estimator.pyc in estimate(self, X, **params)\r\n    341         if params:\r\n    342             self.set_params(**params)\r\n--> 343         self._model = self._estimate(X)\r\n    344         self._estimated = True\r\n    345         return self\r\n\r\n/Users/noe/data/software_projects/pyemma/pyemma/coordinates/clustering/kmeans.pyc in _estimate(self, iterable, **kw)\r\n    133 \r\n    134     def _estimate(self, iterable, **kw):\r\n--> 135         self._init_estimate()\r\n    136 \r\n    137         with iterable.iterator(return_trajindex=True, stride=self.stride) as iter:\r\n\r\n/Users/noe/data/software_projects/pyemma/pyemma/coordinates/clustering/kmeans.pyc in _init_estimate(self)\r\n    216                                     description=\"initialize kmeans++ centers\", stage=0)\r\n    217         self._progress_register(self.max_iter, description=\"kmeans iterations\", stage=1)\r\n--> 218         self._init_in_memory_chunks(total_length)\r\n    219         if self.init_strategy == 'uniform':\r\n    220             # gives random samples from each trajectory such that the cluster centers are distributed percentage-wise\r\n\r\n/Users/noe/data/software_projects/pyemma/pyemma/coordinates/clustering/kmeans.pyc in _init_in_memory_chunks(self, size)\r\n    106                                               order='C', dtype=np.float32)\r\n    107         else:\r\n--> 108             if self._oom_strategy == 'raise':\r\n    109                 self._logger.warning('K-means failed to load all the data (%s required, %s available) into memory. '\r\n    110                                   'Consider using a larger stride or set the oom_strategy to \\'memmap\\' which works '\r\n\r\nAttributeError: 'KmeansClustering' object has no attribute '_oom_strategy'\r\n```"},"comment":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/comments/208259240","html_url":"https://github.com/markovmodel/PyEMMA/issues/767#issuecomment-208259240","issue_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/767","id":208259240,"user":{"login":"marscher","id":170287,"avatar_url":"https://avatars.githubusercontent.com/u/170287?v=3","gravatar_id":"","url":"https://api.github.com/users/marscher","html_url":"https://github.com/marscher","followers_url":"https://api.github.com/users/marscher/followers","following_url":"https://api.github.com/users/marscher/following{/other_user}","gists_url":"https://api.github.com/users/marscher/gists{/gist_id}","starred_url":"https://api.github.com/users/marscher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marscher/subscriptions","organizations_url":"https://api.github.com/users/marscher/orgs","repos_url":"https://api.github.com/users/marscher/repos","events_url":"https://api.github.com/users/marscher/events{/privacy}","received_events_url":"https://api.github.com/users/marscher/received_events","type":"User","site_admin":false},"created_at":"2016-04-11T09:45:09Z","updated_at":"2016-04-11T09:45:09Z","body":"This code branch has not been properly tested (since this test-case would require to eat up all system memory to provoke a MemoryError to be thrown.\r\n\r\nThe kmeans-c-extension is not capable of chunk-processing, so it copies all the data, which is a sub-optimal solution, but fast in case that the data fits into memory. Probably we should reconcile this implementation in favor of a chunk based one (eg. pass an Iterable Python instance to the extension which can drag chunks in the kmeans loops). "}}