{"action":"created","issue":{"url":"https://api.github.com/repos/publiclab/spectral-workbench/issues/335","repository_url":"https://api.github.com/repos/publiclab/spectral-workbench","labels_url":"https://api.github.com/repos/publiclab/spectral-workbench/issues/335/labels{/name}","comments_url":"https://api.github.com/repos/publiclab/spectral-workbench/issues/335/comments","events_url":"https://api.github.com/repos/publiclab/spectral-workbench/issues/335/events","html_url":"https://github.com/publiclab/spectral-workbench/issues/335","id":134639312,"number":335,"title":"set precision throughout spectrum/snapshot saving","user":{"login":"jywarren","id":24359,"avatar_url":"https://avatars.githubusercontent.com/u/24359?v=3","gravatar_id":"","url":"https://api.github.com/users/jywarren","html_url":"https://github.com/jywarren","followers_url":"https://api.github.com/users/jywarren/followers","following_url":"https://api.github.com/users/jywarren/following{/other_user}","gists_url":"https://api.github.com/users/jywarren/gists{/gist_id}","starred_url":"https://api.github.com/users/jywarren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jywarren/subscriptions","organizations_url":"https://api.github.com/users/jywarren/orgs","repos_url":"https://api.github.com/users/jywarren/repos","events_url":"https://api.github.com/users/jywarren/events{/privacy}","received_events_url":"https://api.github.com/users/jywarren/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/publiclab/spectral-workbench/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/publiclab/spectral-workbench/milestones/3","html_url":"https://github.com/publiclab/spectral-workbench/milestones/2.2","labels_url":"https://api.github.com/repos/publiclab/spectral-workbench/milestones/3/labels","id":1347200,"number":3,"title":"2.2","description":"Followup changes, non-essential to 2.0 launch; renamed from 2.1 as we're doing a large-ish followup API change to 2.0, which is called 2.1","creator":{"login":"jywarren","id":24359,"avatar_url":"https://avatars.githubusercontent.com/u/24359?v=3","gravatar_id":"","url":"https://api.github.com/users/jywarren","html_url":"https://github.com/jywarren","followers_url":"https://api.github.com/users/jywarren/followers","following_url":"https://api.github.com/users/jywarren/following{/other_user}","gists_url":"https://api.github.com/users/jywarren/gists{/gist_id}","starred_url":"https://api.github.com/users/jywarren/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jywarren/subscriptions","organizations_url":"https://api.github.com/users/jywarren/orgs","repos_url":"https://api.github.com/users/jywarren/repos","events_url":"https://api.github.com/users/jywarren/events{/privacy}","received_events_url":"https://api.github.com/users/jywarren/received_events","type":"User","site_admin":false},"open_issues":27,"closed_issues":7,"state":"open","created_at":"2015-10-09T14:45:59Z","updated_at":"2016-02-18T17:13:09Z","due_on":null,"closed_at":null},"comments":1,"created_at":"2016-02-18T17:13:09Z","updated_at":"2016-02-18T20:55:16Z","closed_at":null,"body":"@Stoft1 - can you offer insight on what precision we should store in any spectral data we keep? I'd like to just standardize this and get it right. Thanks!"},"comment":{"url":"https://api.github.com/repos/publiclab/spectral-workbench/issues/comments/185915578","html_url":"https://github.com/publiclab/spectral-workbench/issues/335#issuecomment-185915578","issue_url":"https://api.github.com/repos/publiclab/spectral-workbench/issues/335","id":185915578,"user":{"login":"Stoft1","id":11202056,"avatar_url":"https://avatars.githubusercontent.com/u/11202056?v=3","gravatar_id":"","url":"https://api.github.com/users/Stoft1","html_url":"https://github.com/Stoft1","followers_url":"https://api.github.com/users/Stoft1/followers","following_url":"https://api.github.com/users/Stoft1/following{/other_user}","gists_url":"https://api.github.com/users/Stoft1/gists{/gist_id}","starred_url":"https://api.github.com/users/Stoft1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Stoft1/subscriptions","organizations_url":"https://api.github.com/users/Stoft1/orgs","repos_url":"https://api.github.com/users/Stoft1/repos","events_url":"https://api.github.com/users/Stoft1/events{/privacy}","received_events_url":"https://api.github.com/users/Stoft1/received_events","type":"User","site_admin":false},"created_at":"2016-02-18T20:55:16Z","updated_at":"2016-02-18T20:55:16Z","body":"It is common for lab measurements to provide greater reported precision than the actual level of accuracy. I've not been able to extract better than 2nm FWHM measurement values from my prototypes even when using 1024 pix width camera settings. This suggests that the default 640x480 webcam data, at best, is probably closer to 3nm FWHM resolution. Since most spectra are collected at only a little better than 1nm per pix (a calibrated 400-700nm 300nm span is only about half the 640 pix width), reporting anything more precise than 1nm is misleading (and reporting 534.075nm in a data file is VERY misleading).\r\n\r\n1) If you kept the original pixel data, then there is no opportunity for data loss of resolution\r\n2) Fractional wavelength values have no meaning in a >2nm precision world\r\n3) 1nm integer data / plot resolution is legitimate as long as the measurement error is noted\r\n4) 1nm integer data is so much easier to display, plot, handle, etc.\r\n5) Therefore, all you need are 2 units: pixels and nm -- both integer values\r\n\r\nAdmittedly there is some one-time error in scaling from pixels to nm but since there are slightly more pixels than integer nm, you will never be off by more than 0.5nm (and usually much less) within a measurement accuracy of 2-3nm. Also remember, that many spectrometers calibrate with CFL data which does not even resolve the double green Hg line which likely adds several nm of additional error. There is additional error which results from where the pixel line is drawn through the spectral band (as per my other recent notes). I'm still working on that issue in search of better automation to capture a consistent spectra.\r\n\r\nMagnitude is also a consideration. The camera has 8-bit dynamic range while the plots can show relative percentage of max. I'm convinced that there is sufficient noise and measurement repeatability issues such that one cannot claim 1/256 intensity accuracy; just 8-bit resolution. Converting to % could provide data storage of (eg. 89.7%) 1/10% resolution (4x the collected data resolution) and be assured of not losing information but, again, the measurement error is much worse than the numbers suggest. Just keeping the 8-bit values and always just scale from that data to get % displays would be fine. Keeping integer % values would not retain precise measurement values, but would not overtly affect the real measurement error. Still, I'd opt for just retaining the 8-bit data and always just scale for display purposes.\r\n\r\nIn short, you could keep 1) original pixel and 8-bit data and 2) calibrated 1nm resolution CFL-cal'd wavelength data with 8-bit intensity. All else can be easily extracted from that.\r\n\r\nIt could be argued that once a spectra has been calibrated the pixel data need not be kept. The counter argument is that the cal bight have been done wrong so the original pixel info must be kept. The reply argument is 1) commercial instruments don't generally provide raw data (at least, not easily accessible) and that 2) allowing re-cal of data opens the process for additional errors. If keeping raw pixel data is easy and cheap, then it's probably a small issue.\r\n\r\nHope this is helpful.\r\n"}}