{"action":"created","issue":{"url":"https://api.github.com/repos/bayespy/bayespy/issues/83","repository_url":"https://api.github.com/repos/bayespy/bayespy","labels_url":"https://api.github.com/repos/bayespy/bayespy/issues/83/labels{/name}","comments_url":"https://api.github.com/repos/bayespy/bayespy/issues/83/comments","events_url":"https://api.github.com/repos/bayespy/bayespy/issues/83/events","html_url":"https://github.com/bayespy/bayespy/issues/83","id":168196115,"number":83,"title":"Inefficient memory usage in message passing","user":{"login":"dar326","id":18688128,"avatar_url":"https://avatars.githubusercontent.com/u/18688128?v=3","gravatar_id":"","url":"https://api.github.com/users/dar326","html_url":"https://github.com/dar326","followers_url":"https://api.github.com/users/dar326/followers","following_url":"https://api.github.com/users/dar326/following{/other_user}","gists_url":"https://api.github.com/users/dar326/gists{/gist_id}","starred_url":"https://api.github.com/users/dar326/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dar326/subscriptions","organizations_url":"https://api.github.com/users/dar326/orgs","repos_url":"https://api.github.com/users/dar326/repos","events_url":"https://api.github.com/users/dar326/events{/privacy}","received_events_url":"https://api.github.com/users/dar326/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/bayespy/bayespy/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/bayespy/bayespy/labels/enhancement","name":"enhancement","color":"84b6eb"},{"url":"https://api.github.com/repos/bayespy/bayespy/labels/performance","name":"performance","color":"fbca04"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":10,"created_at":"2016-07-28T21:02:45Z","updated_at":"2016-09-28T22:09:37Z","closed_at":null,"body":"Hi,\r\n\r\nI'm attempting to use BayesPy with a somewhat large hierarchical model. My model has a structure including 10 nodes (9 Categorial, 1 Multinomial). Each of the Categorical nodes has an unknown Dirichlet prior (theta) and all but one (X_1) has a parent represented by one of the other Categorical nodes. One of the Categorical nodes (X_9) is the parent of the Multinomial node (Y). The Multinomial node is observed and has 256 possible outcomes. For each of ~4000 possible outcomes of X_9, we have a known distribution over the 256 outcomes of Y. The observations for Y consist of a variable number of trials N_i. For simplicity, I am restricting the number of observations to 1000. So, we have obs_0, obs_1, ..., obs_S with N_0, N_1,...,N_S trials, respectively. Currently, N_i is ~100. These nodes have been implemented as mixture distributions when the node has a parent. The one node without a parent (X_1) is a Categorical node.\r\n\r\nWhen setting up this model with BayesPy using the VB inference engine, all of the data and the model require less than 2 GB of memory. However, there is a huge growth in memory when I observe data and then update the model. In particular, the \"update\" line below (when using memory profiling tools and observing memory usage with the \"top\" command), causes a huge spike in memory usage (maxing out above ~100 GB):\r\n\r\n```\r\nQ = VB(*(thetas + Xs + [Y]), tol=1e-4)\r\nQ.update(repeat=1000, tol=1e-4)\r\n```\r\n\r\nThe number of observations (1000 in this example) is just a small sample of my total number of observations. I can deal with sub-sampling my data using stochastic variational inference to learn my priors. So, in this case, while the memory usage is not ideal, I can run the model.\r\n\r\nHowever, this is just a simplified version of my real problem. The real problem has ~65k possible outcomes for the Multinomial node. Even with a sample of 1000 observations, I run into memory errors which I have tracked to the model update function. Specifically, the traceback includes the following code:\r\n\r\n```\r\nself = <bayespy.inference.vmp.nodes.multinomial.MultinomialDistribution object at 0x7f7c5c78cda0>\r\nu = [array([[[ 0.,  0.,  0., ...,  0.,  0.,  0.]],\r\n\r\n       [[ 0.,  0.,  0., ...,  0.,  0.,  0.]],\r\n\r\n       [[ 0.,  0.,  0.,..., ...,  0.,  0.,  0.]],\r\n\r\n       [[ 0.,  0.,  0., ...,  0.,  0.,  0.]],\r\n\r\n       [[ 0.,  0.,  0., ...,  0.,  0.,  0.]]])]\r\nphi = [array([[[-11.09035492, -11.09035492, -11.09035492, ..., -11.09035492,\r\n         -11.09035492, -11.09035492],\r\n        [...  [-11.09035492, -11.09035492, -11.09035492, ..., -11.09035492,\r\n         -11.09035492, -11.09035492]]], dtype=float32)]\r\ng = array([0]), f = 0, ndims = [1]\r\n\r\n    def compute_logpdf(self, u, phi, g, f, ndims):\r\n        \"\"\" Compute E[log p(X)] given E[u], E[phi], E[g] and\r\n            E[f]. Does not sum over plates.\"\"\"\r\n    \r\n        # TODO/FIXME: Should I take into account what is latent or\r\n        # observed, or what is even totally ignored (by the mask).\r\n        L = g + f\r\n        for (phi_i, u_i, ndims_i) in zip(phi, u, ndims):\r\n            # Axes to sum (dimensions of the variable, not the plates)\r\n            axis_sum = tuple(range(-ndims_i,0))\r\n            # Compute the term\r\n            # TODO/FIXME: Use einsum!\r\n>           L = L + np.sum(phi_i * u_i, axis=axis_sum)\r\nE           MemoryError\r\n```\r\n\r\nSo, it appears that this summation line when computing the log pdf for my Multinomial node is responsible for the large memory usage. I noticed that the comment above the line where the memory error occurs mentions using NumPy's einsum at some point in the future.\r\n\r\nI'm wondering if the plan to use einsum is to reduce memory usage? If so, would you expect such a change to reduce the memory usage in my model or is this model simply too large for BayesPy?\r\n\r\nThanks,\r\nDarryl"},"comment":{"url":"https://api.github.com/repos/bayespy/bayespy/issues/comments/250316353","html_url":"https://github.com/bayespy/bayespy/issues/83#issuecomment-250316353","issue_url":"https://api.github.com/repos/bayespy/bayespy/issues/83","id":250316353,"user":{"login":"dar326","id":18688128,"avatar_url":"https://avatars.githubusercontent.com/u/18688128?v=3","gravatar_id":"","url":"https://api.github.com/users/dar326","html_url":"https://github.com/dar326","followers_url":"https://api.github.com/users/dar326/followers","following_url":"https://api.github.com/users/dar326/following{/other_user}","gists_url":"https://api.github.com/users/dar326/gists{/gist_id}","starred_url":"https://api.github.com/users/dar326/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dar326/subscriptions","organizations_url":"https://api.github.com/users/dar326/orgs","repos_url":"https://api.github.com/users/dar326/repos","events_url":"https://api.github.com/users/dar326/events{/privacy}","received_events_url":"https://api.github.com/users/dar326/received_events","type":"User","site_admin":false},"created_at":"2016-09-28T22:09:37Z","updated_at":"2016-09-28T22:09:37Z","body":"Just tried to install the new branch but getting an error:\r\n\r\n```$ pip install git+https://github.com/bayespy/bayespy.git@backpropagation-messaging\r\nCollecting git+https://github.com/bayespy/bayespy.git@backpropagation-messaging\r\n  Cloning https://github.com/bayespy/bayespy.git (to backpropagation-messaging) to /tmp/pip-qc5xgelt-build\r\n  Could not find a tag or branch 'backpropagation-messaging', assuming commit.\r\nerror: pathspec 'backpropagation-messaging' did not match any file(s) known to git.\r\nCommand \"git checkout -q backpropagation-messaging\" failed with error code 1 in /tmp/pip-qc5xgelt-build```\r\n\r\nAlso, poked around the github repo and don't see any reference to the \"backpropagation-messaging.\" Perhaps the branch wasn't checked in?"}}
{"actor":{"display_login":"dar326"}}