{"action":"opened","issue":{"url":"https://api.github.com/repos/GPflow/GPflow/issues/732","repository_url":"https://api.github.com/repos/GPflow/GPflow","labels_url":"https://api.github.com/repos/GPflow/GPflow/issues/732/labels{/name}","comments_url":"https://api.github.com/repos/GPflow/GPflow/issues/732/comments","events_url":"https://api.github.com/repos/GPflow/GPflow/issues/732/events","html_url":"https://github.com/GPflow/GPflow/issues/732","id":318107698,"number":732,"title":"Same code gives Cholesky error on a different machine with CPU","user":{"login":"Joshuaalbert","id":14807032,"avatar_url":"https://avatars2.githubusercontent.com/u/14807032?v=4","gravatar_id":"","url":"https://api.github.com/users/Joshuaalbert","html_url":"https://github.com/Joshuaalbert","followers_url":"https://api.github.com/users/Joshuaalbert/followers","following_url":"https://api.github.com/users/Joshuaalbert/following{/other_user}","gists_url":"https://api.github.com/users/Joshuaalbert/gists{/gist_id}","starred_url":"https://api.github.com/users/Joshuaalbert/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Joshuaalbert/subscriptions","organizations_url":"https://api.github.com/users/Joshuaalbert/orgs","repos_url":"https://api.github.com/users/Joshuaalbert/repos","events_url":"https://api.github.com/users/Joshuaalbert/events{/privacy}","received_events_url":"https://api.github.com/users/Joshuaalbert/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-26T16:31:36Z","updated_at":"2018-04-26T16:31:36Z","closed_at":null,"author_association":"NONE","body":"A model runs perfectly fine on my laptop, but when I run it on the same data on a cluster I get failed matrix inversion every time. Both cases its CPU only TF, same version of TF 1.7, identical python environments actually.\r\n\r\nI'll describe the model, and if requested I can post it though it's lengthy.\r\nIt is SVGP with a custom likelihood that uses a GP to model the noise instead of a scalar variance parameter. The noise GP is constant mean and RBF kernel. I use the reparametrization trick to learn the hyperparrameters of this GP simultaneously. I.e. I draw a sample of the noise GP as minibatching occurs during likelihood calculation. During the sampling of the noise GP, Cholesky decomposition fails on one computer but not the other. All input data is scaled to centered unit variance. I increased jitter to 1e-5. On the one computer everything is stable and I can run the same code on all the different datasets I apply it to.\r\n\r\nComputer where is works: Laptop, Fedora 64, i7 Xeon, CPU TF 1.7, python 3.6, gpflow HEAD\r\n\r\nComputer where is fails: Cluster, Redhat 7, I think Xeon cores, CPU TF 1.7, python 3.6, gpflow HEAD\r\n"}}
{"actor":{"display_login":"Joshuaalbert"}}