{"action":"created","issue":{"url":"https://api.github.com/repos/GPflow/GPflow/issues/761","repository_url":"https://api.github.com/repos/GPflow/GPflow","labels_url":"https://api.github.com/repos/GPflow/GPflow/issues/761/labels{/name}","comments_url":"https://api.github.com/repos/GPflow/GPflow/issues/761/comments","events_url":"https://api.github.com/repos/GPflow/GPflow/issues/761/events","html_url":"https://github.com/GPflow/GPflow/pull/761","id":323982374,"number":761,"title":"Mitigate ScipyOptimizer graph expansion by initializing optimize_tensor only once","user":{"login":"kozisek","id":8324066,"avatar_url":"https://avatars0.githubusercontent.com/u/8324066?v=4","gravatar_id":"","url":"https://api.github.com/users/kozisek","html_url":"https://github.com/kozisek","followers_url":"https://api.github.com/users/kozisek/followers","following_url":"https://api.github.com/users/kozisek/following{/other_user}","gists_url":"https://api.github.com/users/kozisek/gists{/gist_id}","starred_url":"https://api.github.com/users/kozisek/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/kozisek/subscriptions","organizations_url":"https://api.github.com/users/kozisek/orgs","repos_url":"https://api.github.com/users/kozisek/repos","events_url":"https://api.github.com/users/kozisek/events{/privacy}","received_events_url":"https://api.github.com/users/kozisek/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-05-17T11:28:22Z","updated_at":"2018-05-20T20:06:54Z","closed_at":null,"author_association":"NONE","pull_request":{"url":"https://api.github.com/repos/GPflow/GPflow/pulls/761","html_url":"https://github.com/GPflow/GPflow/pull/761","diff_url":"https://github.com/GPflow/GPflow/pull/761.diff","patch_url":"https://github.com/GPflow/GPflow/pull/761.patch"},"body":"# Mitigate graph expansion by initializing optimize_tensor only once\r\n\r\n## Description\r\nWhile investigating the cause of #756 , we found that the primary suspect of the slow down is expanding computation graph which grows by ~380 operations on every run of `optimizer.minimize()`. The cause of the expansion is a repeated instantiation of the optimize_tensor on every `minimize()` call even if the `ScipyOptimizer` instance is not recreated.\r\nBy creating a new optimize_tensor instance only once (and then on demand) as in the actions framework examples, the graph grows only by 11 operations on each run. While this does not fix the problem described in #756 completely, it mitigates it substantially.\r\n\r\n## Minimal example\r\nUsing a modified example from #756 with a seed for reproducibility\r\n```\r\nimport gpflow\r\nimport numpy as np\r\n\r\nopt = gpflow.train.ScipyOptimizer()\r\nmean = gpflow.mean_functions.Constant()\r\nk = gpflow.kernels.Matern52(1) \r\n\r\nnp.random.seed(42)\r\nx = np.atleast_2d(np.random.rand(100))\r\ny = np.atleast_2d(np.random.rand(100))\r\n\r\nm = gpflow.models.GPR(x, y, kern=k, mean_function=mean)\r\n\r\ndef optimize(iterations):\r\n    for i in range(iterations):\r\n        opt.minimize(m)\r\n    return\r\n\r\n%time optimize(100)\r\n```\r\n\r\nBefore the change, the computation takes ~4 minutes. After the change, it takes ~10 seconds to finish while reaching the same objective.\r\n\r\n### Disclaimer\r\nThe loop above is meant only to showcase the slow down of repeated computations, not as the primary use case of repeated calls of the `optimizer.minimize()` function. The use case of running the `minimize()` function repeatedly can range from reusing the same model with different data to loading a saved model and running further optimization."},"comment":{"url":"https://api.github.com/repos/GPflow/GPflow/issues/comments/390507829","html_url":"https://github.com/GPflow/GPflow/pull/761#issuecomment-390507829","issue_url":"https://api.github.com/repos/GPflow/GPflow/issues/761","id":390507829,"user":{"login":"awav","id":24483645,"avatar_url":"https://avatars0.githubusercontent.com/u/24483645?v=4","gravatar_id":"","url":"https://api.github.com/users/awav","html_url":"https://github.com/awav","followers_url":"https://api.github.com/users/awav/followers","following_url":"https://api.github.com/users/awav/following{/other_user}","gists_url":"https://api.github.com/users/awav/gists{/gist_id}","starred_url":"https://api.github.com/users/awav/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awav/subscriptions","organizations_url":"https://api.github.com/users/awav/orgs","repos_url":"https://api.github.com/users/awav/repos","events_url":"https://api.github.com/users/awav/events{/privacy}","received_events_url":"https://api.github.com/users/awav/received_events","type":"User","site_admin":false},"created_at":"2018-05-20T20:06:53Z","updated_at":"2018-05-20T20:06:53Z","author_association":"MEMBER","body":"@kozisek Hello Jakub, any thoughts on comments above?"}}
{"actor":{"display_login":"awav"}}