{"action":"created","issue":{"url":"https://api.github.com/repos/desihub/desispec/issues/225","repository_url":"https://api.github.com/repos/desihub/desispec","labels_url":"https://api.github.com/repos/desihub/desispec/issues/225/labels{/name}","comments_url":"https://api.github.com/repos/desihub/desispec/issues/225/comments","events_url":"https://api.github.com/repos/desihub/desispec/issues/225/events","html_url":"https://github.com/desihub/desispec/pull/225","id":160964804,"number":225,"title":"Remove subprocess calls from high concurrency pipeline code","user":{"login":"tskisner","id":84221,"avatar_url":"https://avatars.githubusercontent.com/u/84221?v=3","gravatar_id":"","url":"https://api.github.com/users/tskisner","html_url":"https://github.com/tskisner","followers_url":"https://api.github.com/users/tskisner/followers","following_url":"https://api.github.com/users/tskisner/following{/other_user}","gists_url":"https://api.github.com/users/tskisner/gists{/gist_id}","starred_url":"https://api.github.com/users/tskisner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tskisner/subscriptions","organizations_url":"https://api.github.com/users/tskisner/orgs","repos_url":"https://api.github.com/users/tskisner/repos","events_url":"https://api.github.com/users/tskisner/events{/privacy}","received_events_url":"https://api.github.com/users/tskisner/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"sbailey","id":218471,"avatar_url":"https://avatars.githubusercontent.com/u/218471?v=3","gravatar_id":"","url":"https://api.github.com/users/sbailey","html_url":"https://github.com/sbailey","followers_url":"https://api.github.com/users/sbailey/followers","following_url":"https://api.github.com/users/sbailey/following{/other_user}","gists_url":"https://api.github.com/users/sbailey/gists{/gist_id}","starred_url":"https://api.github.com/users/sbailey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbailey/subscriptions","organizations_url":"https://api.github.com/users/sbailey/orgs","repos_url":"https://api.github.com/users/sbailey/repos","events_url":"https://api.github.com/users/sbailey/events{/privacy}","received_events_url":"https://api.github.com/users/sbailey/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2016-06-17T20:04:35Z","updated_at":"2016-06-17T23:26:21Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/desihub/desispec/pulls/225","html_url":"https://github.com/desihub/desispec/pull/225","diff_url":"https://github.com/desihub/desispec/pull/225.diff","patch_url":"https://github.com/desihub/desispec/pull/225.patch"},"body":"This removes subprocess calls from all python code which is on the execution path of production pipeline runs.  In particular, the code now calls specex directly using ctypes without forking a new process.  The only other place where subprocess was used is during the merging of PSF.  That functionality was in a small python script in specex, and this has now been copied directly into desispec.scripts.specex.  We can revisit that choice if we ever want to package the specex python tools into an actual python package.  In that case, we could then just call the function from there.\r\n\r\nA 60-process test job now runs on 3 frames, using multiple groups of processes running specex to process a whole frame.  This job previously failed sporadically with subprocess errors."},"comment":{"url":"https://api.github.com/repos/desihub/desispec/issues/comments/226904070","html_url":"https://github.com/desihub/desispec/pull/225#issuecomment-226904070","issue_url":"https://api.github.com/repos/desihub/desispec/issues/225","id":226904070,"user":{"login":"tskisner","id":84221,"avatar_url":"https://avatars.githubusercontent.com/u/84221?v=3","gravatar_id":"","url":"https://api.github.com/users/tskisner","html_url":"https://github.com/tskisner","followers_url":"https://api.github.com/users/tskisner/followers","following_url":"https://api.github.com/users/tskisner/following{/other_user}","gists_url":"https://api.github.com/users/tskisner/gists{/gist_id}","starred_url":"https://api.github.com/users/tskisner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tskisner/subscriptions","organizations_url":"https://api.github.com/users/tskisner/orgs","repos_url":"https://api.github.com/users/tskisner/repos","events_url":"https://api.github.com/users/tskisner/events{/privacy}","received_events_url":"https://api.github.com/users/tskisner/received_events","type":"User","site_admin":false},"created_at":"2016-06-17T23:26:21Z","updated_at":"2016-06-17T23:26:21Z","body":"I just ran this branch successfully on a small version of the madrone sims.  This was using one spectrograph and extracting one flat and one science exposure with 15 nodes.\r\n\r\nSince my run works, I am guessing your trouble was specific to particular node(s) of edison or with the filesystem.  My python startup was much slower than I have been seeing the last couple weeks."}}
{"actor":{"display_login":"tskisner"}}