{"action":"created","issue":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/887","repository_url":"https://api.github.com/repos/markovmodel/PyEMMA","labels_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/887/labels{/name}","comments_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/887/comments","events_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/887/events","html_url":"https://github.com/markovmodel/PyEMMA/issues/887","id":169213321,"number":887,"title":"cluster_mini_batch_kmeans eats all memory before dying","user":{"login":"jchodera","id":3656088,"avatar_url":"https://avatars.githubusercontent.com/u/3656088?v=3","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/markovmodel/PyEMMA/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":25,"created_at":"2016-08-03T19:29:57Z","updated_at":"2016-08-06T20:29:42Z","closed_at":null,"body":"I'm trying to cluster a very large dataset (1.2M frames) with `pyemma.coordinates.cluster_mini_batch_kmeans`, and find that the clustering code eats up all memory over a few hours without ever starting to write the progress bar. Any idea what might be going on?\r\n\r\nInput data is obviously too big to post, but here is the script:\r\n```python\r\n#!/usr/bin/env python\r\n\r\nimport pyemma\r\nimport numpy as np\r\nimport mdtraj\r\nimport time\r\nimport os\r\n\r\n# Source directory\r\nsource_directory = '/cbio/jclab/projects/fah/fah-data/munged3/no-solvent/11406' # CK2\r\n#source_directory = '11406' # CK2\r\n\r\n################################################################################\r\n# Load reference topology\r\n################################################################################\r\n\r\nprint ('loading reference topology...')\r\nreference_pdb_filename = 'protein.pdb'\r\nreference_trajectory = os.path.join(source_directory, 'run0-clone0.h5')\r\ntraj = mdtraj.load(reference_trajectory)\r\ntraj[0].save_pdb(reference_pdb_filename)\r\n\r\n################################################################################\r\n# Initialize featurizer\r\n################################################################################\r\n\r\nprint('Initializing featurizer...')\r\nimport pyemma.coordinates\r\nfeaturizer = pyemma.coordinates.featurizer(reference_pdb_filename)\r\n#featurizer.add_all() # all atoms\r\nfeaturizer.add_selection( featurizer.select_Heavy() ) # add heavy atoms\r\nprint('Featurizer has %d features.' % featurizer.dimension())\r\n\r\n################################################################################\r\n# Define coordinates source\r\n################################################################################\r\n\r\nimport pyemma.coordinates\r\nfrom glob import glob\r\ntrajectory_filenames = glob(os.path.join(source_directory, 'run*-clone*.h5'))\r\ncoordinates_source = pyemma.coordinates.source(trajectory_filenames, features=featurizer)\r\nprint(\"There are %d frames total in %d trajectories.\" % (coordinates_source.n_frames_total(), coordinates_source.number_of_trajectories()))\r\n\r\n################################################################################\r\n# Cluster\r\n################################################################################\r\n\r\nprint('Clustering...')\r\ngenerator_ratio = 250\r\nnframes = coordinates_source.n_frames_total()\r\nnstates = int(nframes / generator_ratio)\r\nstride = 1\r\nmetric = 'minRMSD'\r\ninitial_time = time.time()\r\nclustering = pyemma.coordinates.cluster_mini_batch_kmeans(data=coordinates_source, batch_size=0.1, k=nstates, stride=stride, metric=metric, max_iter=10)\r\nfinal_time = time.time()\r\nelapsed_time = final_time - initial_time\r\nprint('Elapsed time %.3f s' % elapsed_time)\r\n\r\n# Save cluster centers\r\nnp.save('clustercenters', clustering.clustercenters)\r\n\r\n# Save discrete trajectories.\r\ndtrajs = clustering.dtrajs\r\ndtrajs_dir = 'dtrajs'\r\nclustering.save_dtrajs(output_dir=dtrajs_dir, output_format='npy', extension='.npy')\r\n\r\n################################################################################\r\n# Make timescale plots\r\n################################################################################\r\n\r\nimport matplotlib as mpl\r\nmpl.use('Agg') # Don't use display\r\nimport matplotlib.pyplot as plt\r\n\r\nfrom pyemma import msm\r\nfrom pyemma import plots\r\n\r\nlags = [1,2,5,10,20,50]\r\nits = msm.its(dtrajs, lags=lags, errors='bayes')\r\nplots.plot_implied_timescales(its)\r\n\r\nplt.savefig('plot.pdf')\r\n```\r\noutput:\r\n```\r\nloading reference topology...\r\nInitializing featurizer...\r\nFeaturizer has 8382 features.\r\nObtaining file info: 100% (585/585) [##############################] eta 00:00 /There are 1179914 frames total in 585 trajectories.\r\nClustering...\r\n03-08-16 08:57:56 pyemma.coordinates.clustering.kmeans.MiniBatchKmeansClustering[0] INFO     number of threads obtained from env variable 'OMP_NUM_THREADS'=16\r\nTerminated\r\n```"},"comment":{"url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/comments/238047234","html_url":"https://github.com/markovmodel/PyEMMA/issues/887#issuecomment-238047234","issue_url":"https://api.github.com/repos/markovmodel/PyEMMA/issues/887","id":238047234,"user":{"login":"jchodera","id":3656088,"avatar_url":"https://avatars.githubusercontent.com/u/3656088?v=3","gravatar_id":"","url":"https://api.github.com/users/jchodera","html_url":"https://github.com/jchodera","followers_url":"https://api.github.com/users/jchodera/followers","following_url":"https://api.github.com/users/jchodera/following{/other_user}","gists_url":"https://api.github.com/users/jchodera/gists{/gist_id}","starred_url":"https://api.github.com/users/jchodera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jchodera/subscriptions","organizations_url":"https://api.github.com/users/jchodera/orgs","repos_url":"https://api.github.com/users/jchodera/repos","events_url":"https://api.github.com/users/jchodera/events{/privacy}","received_events_url":"https://api.github.com/users/jchodera/received_events","type":"User","site_admin":false},"created_at":"2016-08-06T20:29:42Z","updated_at":"2016-08-06T20:29:42Z","body":"It seems to be running just fine! Memory usage < 15GB. Thanks!"}}
{"actor":{"display_login":"jchodera"}}