{"action":"closed","issue":{"url":"https://api.github.com/repos/dfm/emcee/issues/196","repository_url":"https://api.github.com/repos/dfm/emcee","labels_url":"https://api.github.com/repos/dfm/emcee/issues/196/labels{/name}","comments_url":"https://api.github.com/repos/dfm/emcee/issues/196/comments","events_url":"https://api.github.com/repos/dfm/emcee/issues/196/events","html_url":"https://github.com/dfm/emcee/issues/196","id":179191947,"number":196,"title":"Multiprocessing issue when writing/reading files at each step","user":{"login":"tomevans","id":1647504,"avatar_url":"https://avatars.githubusercontent.com/u/1647504?v=3","gravatar_id":"","url":"https://api.github.com/users/tomevans","html_url":"https://github.com/tomevans","followers_url":"https://api.github.com/users/tomevans/followers","following_url":"https://api.github.com/users/tomevans/following{/other_user}","gists_url":"https://api.github.com/users/tomevans/gists{/gist_id}","starred_url":"https://api.github.com/users/tomevans/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomevans/subscriptions","organizations_url":"https://api.github.com/users/tomevans/orgs","repos_url":"https://api.github.com/users/tomevans/repos","events_url":"https://api.github.com/users/tomevans/events{/privacy}","received_events_url":"https://api.github.com/users/tomevans/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2016-09-26T10:28:09Z","updated_at":"2016-09-27T15:09:07Z","closed_at":"2016-09-27T15:09:06Z","body":"I'm needing to use some 3rd party FORTRAN software to evaluate my log likelihood function at each step. Unfortunately, this software requires a plain text file to be written to disk which the FORTRAN software then reads and takes as input to compute the likelihood. In practice, each walker will require a separate input text file to be written at each step. I'd like to run this in multiprocessing mode (threads>1), but unfortunately it crashes. From what I can tell, there's some issue with the input files not being generated as I'd expect for each walker at the log likelihood evaluation. To illustrate the qualitative nature of the issue, I've added a simple hack of the Quickstart example from the emcee website below, followed by the error message I get when it crashes; it's the same issue I'm getting in my more specific application. Any suggestions on what I'm doing wrong, or if this sort of thing is even possible, would be much appreciated. thanks :)\r\n\r\n######################################################\r\nimport emcee\r\nimport numpy as np\r\nimport os, time\r\ndef lnprob(x, mu, icov):\r\n    tempfile = '{0}.in'.format( int( (1e6)*time.time() ) ) # generates unique input filename\r\n    np.savetxt( tempfile, icov ) # write inputs to disk\r\n    icov_reread = np.loadtxt( tempfile ) # in my application, it's actually a FORTRAN program that reads this input file\r\n    os.remove( tempfile )\r\n    diff = x-mu\r\n    return -np.dot(diff,np.dot(icov_reread,diff))/2.0\r\nndim = 50\r\nmeans = np.random.rand(ndim)\r\ncov = 0.5 - np.random.rand(ndim ** 2).reshape((ndim, ndim))\r\ncov = np.triu(cov)\r\ncov += cov.T - np.diag(cov.diagonal())\r\ncov = np.dot(cov,cov)\r\nicov = np.linalg.inv(cov)\r\nnwalkers = 150\r\np0 = np.random.rand(ndim * nwalkers).reshape((nwalkers, ndim))\r\nsampler = emcee.EnsembleSampler(nwalkers, ndim, lnprob, args=[means, icov], threads=4)\r\npos, prob, state = sampler.run_mcmc(p0, 150)\r\n######################################################\r\n\r\nemcee: Exception while calling your likelihood function:\r\n  params: [ 0.00416683  0.00121443  0.09297952  0.88577573  0.19752657  0.43616513\r\n  0.82632147  0.92474106  0.5263543   0.13451031  0.15140635  0.28707279\r\n  0.83895138  0.13485659  0.67149602  0.0437184   0.52699409  0.30602217\r\n  0.49810951  0.05429097  0.09616479  0.56352244  0.40038342  0.96128477\r\n  0.96139381  0.30692886  0.66910766  0.10697474  0.66191896  0.33147187\r\n  0.46090604  0.22815596  0.97349513  0.0471428   0.8184995   0.85949838\r\n  0.81470446  0.95773865  0.70743459  0.19343325  0.82486592  0.85985209\r\n  0.0808939   0.54152892  0.61528055  0.53683463  0.77863934  0.88026515\r\n  0.97775209  0.36900002]\r\n  args: [array([ 0.58793961,  0.92810062,  0.17459986,  0.20492939,  0.18346653,\r\n        0.53892566,  0.57584874,  0.94185575,  0.24163453,  0.41284317,\r\n        0.48564821,  0.67920761,  0.34323044,  0.2020662 ,  0.04917549,\r\n        0.83358774,  0.20774804,  0.56789084,  0.21685981,  0.29940764,\r\n        0.87879338,  0.93357872,  0.55544829,  0.11536287,  0.08379872,\r\n        0.25626357,  0.65025791,  0.45939602,  0.00942292,  0.91788029,\r\n        0.80075432,  0.14981844,  0.9870799 ,  0.85341932,  0.62013305,\r\n        0.69599445,  0.3301484 ,  0.20774183,  0.10103013,  0.59141142,\r\n        0.83804104,  0.31710734,  0.98339552,  0.56217429,  0.13204708,\r\n        0.51717227,  0.9444871 ,  0.80968058,  0.71109077,  0.07345402]), array([[  5.8661203 ,   2.63731571,  -2.74252383, ...,   0.35754405,\r\n          1.63243834,   0.41132016],\r\n       [  2.63731571,   4.7774109 ,   4.78864275, ...,  -0.99615663,\r\n          6.5050327 ,  -5.25536002],\r\n       [ -2.74252383,   4.78864275,  34.84112813, ...,   0.42214212,\r\n         17.77711414, -25.9003554 ],\r\n       ..., \r\n       [  0.35754405,  -0.99615663,   0.42214212, ...,   2.69706443,\r\n         -2.79463492,   0.27055001],\r\n       [  1.63243834,   6.5050327 ,  17.77711414, ...,  -2.79463492,\r\n         17.00835671, -15.69519896],\r\n       [  0.41132016,  -5.25536002, -25.9003554 , ...,   0.27055001,\r\n        -15.69519896,  21.18178664]])]\r\n  kwargs: {}\r\n  exception:\r\nTraceback (most recent call last):\r\n  File \"/home/tevans/.local/lib/python2.7/site-packages/emcee/ensemble.py\", line 519, in __call__\r\n    return self.f(x, *self.args, **self.kwargs)\r\n  File \"/home/tevans/code/pacode/atmo/run/hat-p-18/EmceeMultithreadTest.py\", line 12, in lnprob\r\n    os.remove( tempfile )\r\nOSError: [Errno 2] No such file or directory: '1474885596433220.in'\r\n"}}
{"actor":{"display_login":"tomevans"}}