{"action":"created","issue":{"url":"https://api.github.com/repos/Keck-DataReductionPipelines/MosfireDRP/issues/37","labels_url":"https://api.github.com/repos/Keck-DataReductionPipelines/MosfireDRP/issues/37/labels{/name}","comments_url":"https://api.github.com/repos/Keck-DataReductionPipelines/MosfireDRP/issues/37/comments","events_url":"https://api.github.com/repos/Keck-DataReductionPipelines/MosfireDRP/issues/37/events","html_url":"https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/37","id":125525443,"number":37,"title":"Arc line wavelength fitting for both Ne and Ar","user":{"login":"dmkassis","id":9427467,"avatar_url":"https://avatars.githubusercontent.com/u/9427467?v=3","gravatar_id":"","url":"https://api.github.com/users/dmkassis","html_url":"https://github.com/dmkassis","followers_url":"https://api.github.com/users/dmkassis/followers","following_url":"https://api.github.com/users/dmkassis/following{/other_user}","gists_url":"https://api.github.com/users/dmkassis/gists{/gist_id}","starred_url":"https://api.github.com/users/dmkassis/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmkassis/subscriptions","organizations_url":"https://api.github.com/users/dmkassis/orgs","repos_url":"https://api.github.com/users/dmkassis/repos","events_url":"https://api.github.com/users/dmkassis/events{/privacy}","received_events_url":"https://api.github.com/users/dmkassis/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-01-08T01:58:29Z","updated_at":"2016-01-09T00:22:58Z","closed_at":null,"body":"If both Ar and Ne lines are used in the wavelength fitting process, the RMS in the fits increases. The reason for the increased RMS is a 0.5 Angstrom shift between the wavelength solutions for the Ne and Ar lines. Using IRAF, I measured the residuals in the fit between Ar and Ne only (see the image). The image is from my note book and shows the residual plot from a wavelength fit of both Ar and Ne lines combined. There are two sets of residuals from Ne and Ar as indicated.\r\n![img_0309](https://cloud.githubusercontent.com/assets/9427467/12187513/9cde34cc-b550-11e5-9e08-cc2b9c8fa16d.JPG). If you use only Ar or only Ne in the sky+arc solution with the MDRP, the residuals are smaller by about a factor of 10. Because of this offset between arcs of different lamps, it is necessary to find two arc lamp specific offsets first, apply those offsets, and then refit the combined line lists. \r\n\r\nI have been attempting to modify the code to accomplish this, but I have run into a snag. \r\nIn Wavelength.py, there is a function def called \"apply_lambda_sky_and_arc\" which is used to merge the two sets of wavelength solutions from sky and arcs. I have attempted to modify this function. As I understand the code, it does the following:\r\n1) generates a picket fence line list using the sky line solution\r\n2) generates a picket fence line list using the arc line solution\r\n3) Determines and applies a shift to the arc picket fence\r\n4) splices the two picket fences, and this seems odd, by taking all of the short wavelegths of the sky line picket fence and taking all the longer wavelength arc picket fence to make one long list. \r\n5) refitting the spliced picket fence line list where the first half is sky list related and the second half is line list related.\r\n\r\nFrom the original Wavelength.ph apply_lambda_sky_and_arc I think that the relevant code section is the following:\r\n   for i in xrange(len(SkyL)):\r\n        slp = SkyL[i][\"2d\"][\"positions\"].astype(np.int)\r\n        slc = SkyL[i][\"2d\"][\"coeffs\"]\r\n        slm = SkyL[i][\"2d\"][\"lambdaMAD\"]\r\n        alp = ArcL[i][\"2d\"][\"positions\"].astype(np.int)\r\n        alc = ArcL[i][\"2d\"][\"coeffs\"]\r\n        alm = ArcL[i][\"2d\"][\"lambdaMAD\"]\r\n\r\n        info(\"2d wavelengths: Slit %i/%i\" % (i+1, len(SkyL)))\r\n\r\n        prev = 0\r\n        dpixels = []\r\n        for j in xrange(len(slp)):\r\n\r\n            if (slm[j] < 0.2) and (alm[j] < 0.2):\r\n\r\n                coeff_sky = slc[j]\r\n                coeff_arc = alc[j]\r\n                lam_sky = CV.chebval(xx, coeff_sky)\r\n                lam_arc = CV.chebval(xx, coeff_arc)\r\n\r\n                # minimize pixel solution                                                                                   \r\n                best_dpix = find_pixel_offset(lam_sky, coeff_arc, LROIs[i])\r\n                dpixels.append(best_dpix)\r\n\r\n                # Refit the chebyshev                                                                                       \r\n                lambdas = CV.chebval(fitpix - best_dpix, coeff_arc)\r\n                arc_positions = lambdas > np.mean(LROIs[i])\r\n                sky_positions = lambdas <= np.mean(LROIs[i])\r\n\r\n                to_fit = CV.chebval(fitpix[sky_positions], coeff_sky)\r\n                to_fit = np.append(to_fit,\r\n                    CV.chebval(fitpix[arc_positions], coeff_arc))\r\n\r\n                coeffs = CV.chebfit(fitpix, to_fit, options[\"chebyshev-degree\"])\r\n\r\n\r\n                prev = lams[slp[j],:] = CV.chebval(xx, coeffs)\r\n                prevcoeff = coeffs\r\n                SkyL[i][\"2d\"][\"coeffs\"][j,:] = coeffs\r\n            else:\r\n                lams[slp[j],:] = prev\r\n\r\n-----------------\r\nAre we sure that this is correctly implemented? If it is, what is the best way to add in a third line list?\r\n\r\nHere is the direction I was going, but I think this is the wrong way to go about this. I feel the shift should be applied to the original arc line list, and then the function refits for all lines (sky + (Ne shifted_ne)+ (Ar+shifted_ar)).\r\n\r\nFrom my modified code that does not quite work yet:\r\n        for j in xrange(len(slp)):\r\n\r\n            if (slm[j] < 0.2) and (alm[j] < 0.2) and ( (arcnames2 == None) or (alm2[j] < 0.2)):\r\n\r\n                coeff_sky = slc[j]\r\n                coeff_arc = alc[j]\r\n                lam_sky = CV.chebval(xx, coeff_sky)\r\n                lam_arc = CV.chebval(xx, coeff_arc)\r\n\r\n                # so we need to shift both Ar and Ne wavelength solutions                                                   \r\n                # And I needt to keep track of both here.                                                                   \r\n\r\n                # minimize pixel solution                                                                                   \r\n                best_dpix = find_pixel_offset(lam_sky, coeff_arc, LROIs[i])\r\n                dpixels.append(best_dpix)\r\n\r\n                print \"Value of bset_dpix:\"\r\n                print best_dpix\r\n\r\n\r\n                if arcnames2 is not None:\r\n                    coeff_arc2 = alc2[j]\r\n                    lam_arc2 = CV.chebval(xx, coeff_arc2)\r\n                    best_dpix2 = find_pixel_offset(lam_sky, coeff_arc2, LROIs2[i])\r\n                    dpixels2.append(best_dpix2)\r\n                    print \"Value of bset_dpix2:\"\r\n                    print best_dpix2\r\n\r\n                # Refit the chebyshev                                                                                       \r\n                print \"Fitpix - best_dpix\"\r\n                print fitpix - best_dpix\r\n                lambdas = CV.chebval(fitpix - best_dpix, coeff_arc)\r\n                print \"Value for lambdas is:\"\r\n                print lambdas\r\n                arc_positions = lambdas > np.mean(LROIs[i])\r\n                sky_positions = lambdas <= np.mean(LROIs[i])\r\n                print \"Arc positions: \"\r\n                print arc_positions\r\n                print \"sky positions: \"\r\n                print sky_positions\r\n\r\n                if arcnames2 is not None:\r\n                    lambdas2 = CV.chebval(fitpix - best_dpix2, coeff_arc2)\r\n                    arc_positions2 = lambdas > np.mean(LROIs2[i])\r\n                    sky_positions2 = lambdas <= np.mean(LROIs2[i])\r\n\r\n\r\n                to_fit = CV.chebval(fitpix[sky_positions], coeff_sky)\r\n                print \"A- Length of variables are:\"\r\n                print len(fitpix)\r\n                print len(to_fit)\r\n                to_fit = np.append(to_fit, CV.chebval(fitpix[arc_positions], coeff_arc))\r\n\r\n                if arcnames2 is not None:\r\n                    print \"B- Length of variables are:\"\r\n                    print len(fitpix)\r\n                    print len(to_fit)\r\n                    to_fit = np.append(to_fit,\r\n                               CV.chebval(fitpix[arc_positions2], coeff_arc2))\r\n\r\n                print \"C-Length of variables are:\"\r\n                print len(fitpix)\r\n                print len(to_fit)\r\n                print to_fit\r\n\r\n                coeffs = CV.chebfit(fitpix, to_fit, options[\"chebyshev-degree\"])\r\n\r\n\r\n                prev = lams[slp[j],:] = CV.chebval(xx, coeffs)\r\n                prevcoeff = coeffs\r\n                SkyL[i][\"2d\"][\"coeffs\"][j,:] = coeffs\r\n            else:\r\n                lams[slp[j],:] = prev\r\n\r\n-----------------------------\r\nThe above code does not work because of the dichotomy in the split seen here:\r\n                arc_positions = lambdas > np.mean(LROIs[i])\r\n                sky_positions = lambdas <= np.mean(LROIs[i])\r\nIn the current method, this has to be split three ways for sky, Ne, and Ar. I have not yet gone that direction because I am not convinced this is the correct method for combining the line lists now that I have had time to experiment with the code..\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/Keck-DataReductionPipelines/MosfireDRP/issues/comments/170167322","html_url":"https://github.com/Keck-DataReductionPipelines/MosfireDRP/issues/37#issuecomment-170167322","issue_url":"https://api.github.com/repos/Keck-DataReductionPipelines/MosfireDRP/issues/37","id":170167322,"user":{"login":"csteidel","id":9736870,"avatar_url":"https://avatars.githubusercontent.com/u/9736870?v=3","gravatar_id":"","url":"https://api.github.com/users/csteidel","html_url":"https://github.com/csteidel","followers_url":"https://api.github.com/users/csteidel/followers","following_url":"https://api.github.com/users/csteidel/following{/other_user}","gists_url":"https://api.github.com/users/csteidel/gists{/gist_id}","starred_url":"https://api.github.com/users/csteidel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/csteidel/subscriptions","organizations_url":"https://api.github.com/users/csteidel/orgs","repos_url":"https://api.github.com/users/csteidel/repos","events_url":"https://api.github.com/users/csteidel/events{/privacy}","received_events_url":"https://api.github.com/users/csteidel/received_events","type":"User","site_admin":false},"created_at":"2016-01-09T00:22:58Z","updated_at":"2016-01-09T00:22:58Z","body":"Marc- I think it is important (based on my own experimentation with this during lab testing) to co-add images of Ne and Ar to do the fitting. There is enough blending of lines that occurs to throw the line centroids off enough to matter in the fits. The match of features to pixels must be done independently for each, and then the input is the merged *list of identified lines and their corresponding pixel positions* .\r\nIf that does not work without a shift of one with respect to another, then one last thing to check is whether the lamps are \"symmetric\" in the sense that there is no left-right imbalance of the illumination  for Argon vs. Neon.  It is *possible* that some difference in illumination that depends on exactly where the lamps are located and how much of each is masked could cause a small shift (for the same reason that internal lamps would be shifted wrt the night sky lines). Can you remind me what the lamp layout looks like in the front of the instrument?  If (e.g.) one of the lamps were burned out, its possible it could manifest itself in this way....\r\n"}}