{"action":"created","issue":{"url":"https://api.github.com/repos/desihub/specter/issues/40","repository_url":"https://api.github.com/repos/desihub/specter","labels_url":"https://api.github.com/repos/desihub/specter/issues/40/labels{/name}","comments_url":"https://api.github.com/repos/desihub/specter/issues/40/comments","events_url":"https://api.github.com/repos/desihub/specter/issues/40/events","html_url":"https://github.com/desihub/specter/pull/40","id":168228898,"number":40,"title":"pixel model and chi2 metrics for better cosmic identification","user":{"login":"sbailey","id":218471,"avatar_url":"https://avatars.githubusercontent.com/u/218471?v=3","gravatar_id":"","url":"https://api.github.com/users/sbailey","html_url":"https://github.com/sbailey","followers_url":"https://api.github.com/users/sbailey/followers","following_url":"https://api.github.com/users/sbailey/following{/other_user}","gists_url":"https://api.github.com/users/sbailey/gists{/gist_id}","starred_url":"https://api.github.com/users/sbailey/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sbailey/subscriptions","organizations_url":"https://api.github.com/users/sbailey/orgs","repos_url":"https://api.github.com/users/sbailey/repos","events_url":"https://api.github.com/users/sbailey/events{/privacy}","received_events_url":"https://api.github.com/users/sbailey/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/desihub/specter/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"tskisner","id":84221,"avatar_url":"https://avatars.githubusercontent.com/u/84221?v=3","gravatar_id":"","url":"https://api.github.com/users/tskisner","html_url":"https://github.com/tskisner","followers_url":"https://api.github.com/users/tskisner/followers","following_url":"https://api.github.com/users/tskisner/following{/other_user}","gists_url":"https://api.github.com/users/tskisner/gists{/gist_id}","starred_url":"https://api.github.com/users/tskisner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tskisner/subscriptions","organizations_url":"https://api.github.com/users/tskisner/orgs","repos_url":"https://api.github.com/users/tskisner/repos","events_url":"https://api.github.com/users/tskisner/events{/privacy}","received_events_url":"https://api.github.com/users/tskisner/received_events","type":"User","site_admin":false},"assignees":[{"login":"tskisner","id":84221,"avatar_url":"https://avatars.githubusercontent.com/u/84221?v=3","gravatar_id":"","url":"https://api.github.com/users/tskisner","html_url":"https://github.com/tskisner","followers_url":"https://api.github.com/users/tskisner/followers","following_url":"https://api.github.com/users/tskisner/following{/other_user}","gists_url":"https://api.github.com/users/tskisner/gists{/gist_id}","starred_url":"https://api.github.com/users/tskisner/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tskisner/subscriptions","organizations_url":"https://api.github.com/users/tskisner/orgs","repos_url":"https://api.github.com/users/tskisner/repos","events_url":"https://api.github.com/users/tskisner/events{/privacy}","received_events_url":"https://api.github.com/users/tskisner/received_events","type":"User","site_admin":false},{"login":"julienguy","id":5192160,"avatar_url":"https://avatars.githubusercontent.com/u/5192160?v=3","gravatar_id":"","url":"https://api.github.com/users/julienguy","html_url":"https://github.com/julienguy","followers_url":"https://api.github.com/users/julienguy/followers","following_url":"https://api.github.com/users/julienguy/following{/other_user}","gists_url":"https://api.github.com/users/julienguy/gists{/gist_id}","starred_url":"https://api.github.com/users/julienguy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/julienguy/subscriptions","organizations_url":"https://api.github.com/users/julienguy/orgs","repos_url":"https://api.github.com/users/julienguy/repos","events_url":"https://api.github.com/users/julienguy/events{/privacy}","received_events_url":"https://api.github.com/users/julienguy/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-07-29T00:53:45Z","updated_at":"2016-07-29T14:01:20Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/desihub/specter/pulls/40","html_url":"https://github.com/desihub/specter/pull/40","diff_url":"https://github.com/desihub/specter/pull/40.diff","patch_url":"https://github.com/desihub/specter/pull/40.patch"},"body":"This PR adds several new outputs to the ex2d() spectral extractions:\r\n  1. pixel model of the extracted spectra projected back into the CCD pixel space, useful for debugging\r\n  2. the PSF-weighted fraction of input pixels that had ivar=0, useful for identifying ill-constrained fluxes (perhaps to be masked by downstream steps)\r\n  3. the PSF-weighted chi^2 of the pixels that contributed to each extracted flux bin, useful for identifying bad extractions, e.g. arising from unmasked cosmics in the input pixel data\r\n\r\nThese are provided as informational numbers that downstream steps could use for masking.  A future specter improvement might use the pixel-level chi2 to identify and mask poorly fitting pixels (e.g. from cosmics) and then re-extract, potentially \"saving\" some flux bins that otherwise would have been lost as just bad extractions.  But that iterative refit isn't included in this PR.\r\n\r\nThe ipython notebook in [doc/nb/unmasked_cosmics.ipynb](https://github.com/desihub/specter/blob/chi2/doc/nb/unmasked_cosmics.ipynb) documents more details about how this works.  The following figure shows an unmasked cosmic in the DESI oak simulations that was incorrectly identified as H-alpha; the upper plot shows the extracted spectrum and the wavelengths that would be masked by applying a cut on the PSF-weighted pixel-chi2.\r\n\r\n![cosmic-r0-00000011-fiber104-wave6984](https://cloud.githubusercontent.com/assets/218471/17234048/ea3f1d4c-54e9-11e6-93ba-c623f2c02e51.png)\r\n\r\nTesting on the DESI oak-lite simulations (2 spectrographs, 3 exposures) using this with a conservative 10-sigma cut improved the ELG redshifts from 90% efficiency to 94%, and LRGs from 97% to 99%.  Full disclosure: compared to oak (10 spectrographs, 9 exposures) the QSO efficiency went down, but the efficiencies were already so low (50-60%) something else is going on that needs to be debugged independent of this.\r\n\r\nTed and Julien: I had previously reported that this was blowing memory on edison batch jobs.  That turned out to be a leftover of caching the results of every subpatch extraction for debugging.  Removing that, the memory footprint is only modestly larger (1.5 vs. 1.4 GB).\r\n\r\nI'll submit a PR on the desispec side to use these features."},"comment":{"url":"https://api.github.com/repos/desihub/specter/issues/comments/236188153","html_url":"https://github.com/desihub/specter/pull/40#issuecomment-236188153","issue_url":"https://api.github.com/repos/desihub/specter/issues/40","id":236188153,"user":{"login":"julienguy","id":5192160,"avatar_url":"https://avatars.githubusercontent.com/u/5192160?v=3","gravatar_id":"","url":"https://api.github.com/users/julienguy","html_url":"https://github.com/julienguy","followers_url":"https://api.github.com/users/julienguy/followers","following_url":"https://api.github.com/users/julienguy/following{/other_user}","gists_url":"https://api.github.com/users/julienguy/gists{/gist_id}","starred_url":"https://api.github.com/users/julienguy/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/julienguy/subscriptions","organizations_url":"https://api.github.com/users/julienguy/orgs","repos_url":"https://api.github.com/users/julienguy/repos","events_url":"https://api.github.com/users/julienguy/events{/privacy}","received_events_url":"https://api.github.com/users/julienguy/received_events","type":"User","site_admin":false},"created_at":"2016-07-29T14:01:20Z","updated_at":"2016-07-29T14:01:20Z","body":"code tested successfully (along with desispec branch cosmic_masking) on one cosmic ray spike (oak1 frame r0-00000006 fiber 413 wave 6519A). with new code, cosmic shows very large CHI2PIX value in frame, and is adequately flagged in MASK.\r\n\r\n"}}
{"actor":{"display_login":"julienguy"}}