{"action":"created","issue":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1305","repository_url":"https://api.github.com/repos/markovmodel/PyEMMA","labels_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1305/labels{/name}","comments_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1305/comments","events_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1305/events","html_url":"https://github.com/markovmodel/PyEMMA/issues/1305","id":325863105,"number":1305,"title":"memory error","user":{"login":"euhruska","id":9983894,"avatar_url":"https://avatars0.githubusercontent.com/u/9983894?v=4","gravatar_id":"","url":"https://api.github.com/users/euhruska","html_url":"https://github.com/euhruska","followers_url":"https://api.github.com/users/euhruska/followers","following_url":"https://api.github.com/users/euhruska/following{/other_user}","gists_url":"https://api.github.com/users/euhruska/gists{/gist_id}","starred_url":"https://api.github.com/users/euhruska/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/euhruska/subscriptions","organizations_url":"https://api.github.com/users/euhruska/orgs","repos_url":"https://api.github.com/users/euhruska/repos","events_url":"https://api.github.com/users/euhruska/events{/privacy}","received_events_url":"https://api.github.com/users/euhruska/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-05-23T20:34:13Z","updated_at":"2018-05-23T22:53:55Z","closed_at":null,"author_association":"NONE","body":"I get a memory error, with featurizer.dimension() 3003, I'm not sure why this happens. It happens quite fast, after about 10s. \r\n```\r\ntica_obj = coor.tica(inp, lag=lag, commute_map=True, kinetic_map=False, var_cutoff=0.8, stride=stride_tica)>>>\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/api.py\", line 1261, in tica\r\n    res.estimate(data, chunksize=cs)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/transform/tica.py\", line 159, in estimate\r\n    return super(TICA, self).estimate(X, **kwargs)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/_base/transformer.py\", line 212, in estimate\r\n    super(StreamingEstimationTransformer, self).estimate(X, **kwargs)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/_base/streaming_estimator.py\", line 45, in estimate\r\n    super(StreamingEstimator, self).estimate(X, **kwargs)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/_base/estimator.py\", line 412, in estimate\r\n    self._model = self._estimate(X)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/transform/tica.py\", line 206, in _estimate\r\n    covar.estimate(iterable, chunksize=self.chunksize, **kw)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/_base/streaming_estimator.py\", line 45, in estimate\r\n    super(StreamingEstimator, self).estimate(X, **kwargs)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/_base/estimator.py\", line 412, in estimate\r\n    self._model = self._estimate(X)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/estimation/covariance.py\", line 214, in _estimate\r\n    for data, weight in zip(it, it_weights):\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/_base/iterable.py\", line 226, in __next__\r\n    return self.next()\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/_base/iterable.py\", line 246, in next\r\n    itraj, data_lagged = self._it.next()\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/_base/datasource.py\", line 1005, in next\r\n    X = self._it_next()\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/_base/datasource.py\", line 985, in _it_next\r\n    X = self._use_cols(self._next_chunk())\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/feature_reader.py\", line 396, in _next_chunk\r\n    res = self._data_source.featurizer.transform(chunk)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/featurization/featurizer.py\", line 915, in transform\r\n    vec = f.transform(traj).astype(np.float32)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/pyemma/coordinates/data/featurization/distances.py\", line 107, in transform\r\n    D = mdtraj.compute_contacts(traj, contacts=self.contacts, scheme=self.scheme, periodic=self.periodic)[0]\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/mdtraj/geometry/contact.py\", line 208, in compute_contacts\r\n    atom_distances = md.compute_distances(traj, atom_pairs, periodic=periodic)\r\n  File \"/scratch1/eh22/conda/envs/py3/lib/python3.6/site-packages/mdtraj/geometry/distance.py\", line 81, in compute_distances\r\n    out = np.empty((xyz.shape[0], pairs.shape[0]), dtype=np.float32)\r\nMemoryError\r\n```\r\npython 3.6.3, pyemma 2.5.2\r\n[version.txt](https://github.com/markovmodel/PyEMMA/files/2032763/version.txt)\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/comments/391525779","html_url":"https://github.com/markovmodel/PyEMMA/issues/1305#issuecomment-391525779","issue_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/1305","id":391525779,"user":{"login":"marscher","id":170287,"avatar_url":"https://avatars0.githubusercontent.com/u/170287?v=4","gravatar_id":"","url":"https://api.github.com/users/marscher","html_url":"https://github.com/marscher","followers_url":"https://api.github.com/users/marscher/followers","following_url":"https://api.github.com/users/marscher/following{/other_user}","gists_url":"https://api.github.com/users/marscher/gists{/gist_id}","starred_url":"https://api.github.com/users/marscher/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/marscher/subscriptions","organizations_url":"https://api.github.com/users/marscher/orgs","repos_url":"https://api.github.com/users/marscher/repos","events_url":"https://api.github.com/users/marscher/events{/privacy}","received_events_url":"https://api.github.com/users/marscher/received_events","type":"User","site_admin":false},"created_at":"2018-05-23T22:53:55Z","updated_at":"2018-05-23T22:53:55Z","author_association":"MEMBER","body":"Frank is right about the memory leak suspicion, but it could also be our code - because running covariances uses an extension as well. Eugen you can run through valgrind, like this:\r\n\r\n`valgrind --leak-check=full  -- python script.py`\r\n\r\nThis will produce a very verbose output at stdout, which should be investigated for the pattern \"definitely lost\". If you spot these have a closer look at the stack (c file names and lines, which indicates the location of the memory leak)."}}
{"actor":{"display_login":"marscher"}}