{"action":"created","issue":{"url":"https://api.github.com/repos/LSSTDESC/DC2_Repo/issues/52","repository_url":"https://api.github.com/repos/LSSTDESC/DC2_Repo","labels_url":"https://api.github.com/repos/LSSTDESC/DC2_Repo/issues/52/labels{/name}","comments_url":"https://api.github.com/repos/LSSTDESC/DC2_Repo/issues/52/comments","events_url":"https://api.github.com/repos/LSSTDESC/DC2_Repo/issues/52/events","html_url":"https://github.com/LSSTDESC/DC2_Repo/issues/52","id":281642109,"number":52,"title":"DC2 Storage Discussion","user":{"login":"salmanhabib","id":9030901,"avatar_url":"https://avatars0.githubusercontent.com/u/9030901?v=4","gravatar_id":"","url":"https://api.github.com/users/salmanhabib","html_url":"https://github.com/salmanhabib","followers_url":"https://api.github.com/users/salmanhabib/followers","following_url":"https://api.github.com/users/salmanhabib/following{/other_user}","gists_url":"https://api.github.com/users/salmanhabib/gists{/gist_id}","starred_url":"https://api.github.com/users/salmanhabib/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/salmanhabib/subscriptions","organizations_url":"https://api.github.com/users/salmanhabib/orgs","repos_url":"https://api.github.com/users/salmanhabib/repos","events_url":"https://api.github.com/users/salmanhabib/events{/privacy}","received_events_url":"https://api.github.com/users/salmanhabib/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":43,"created_at":"2017-12-13T06:48:39Z","updated_at":"2018-02-12T05:37:01Z","closed_at":null,"author_association":"NONE","body":"It would be good to have a discussion of the DC2 storage issue here. Initial info was provided by Simon Krughoff and later discussions with Jim Chiang and Mike Jarvis were very helpful (I am using a lot of information below provided by Mike).\r\n\r\n**Issue** There is a question of why the storage requirement prefactor is so large: 15X where X is the raw file size. \r\n\r\nThe explanation from Simon (paraphrased) is as follows: \r\n\r\n**Explanation** 2x: int --> double; another 2.5x --> adding mask plane and variance plane. Plus copies of each of the following data products: calexp (calibrated exposure), warp (warped calibrated exposures), coadd (coadded warped calibrated exposures). Calexps and warps are the same size, coadds are smaller than both. Roughly speaking, if one assumes that the coadds plus all the other data: e.g. output catalogs, configs, logs, is similar in size to the calexps and warps, this means another factor of 3. Thus one arrives at: 3*(2.5*(2raw_filesize)) = 15raw_filesize uncompressed. \r\n\r\n**Discussion** The data is intrinsically int32 and the initial factor of 2 comes from python's float64 (_this is incorrect, see Jim's comment below_) but there is no reason to store int32 as float64, so this factor of 2 can be immediately eliminated. The mask plane is integer and sparse so should be reducible to a much smaller size (lossless Rice compression). The variance plane does not require high precision and can be subject to lossy compression (reduction possibly by a factor of 4). Mike was not sure why we need to keep the calibrated exposure and was even more convinced that we do not need to keep the warped calibrated exposure. The coadds are needed.\r\n\r\nThus, it appears that the actual storage requirement is likely to be a multiplier of a \"few\" times the raw file size, where \"few\" is likely 2-3, and the raw file size is whatever it is in int32. This is a very big saving over what we are doing now.\r\n\r\nSimon says that the DM stack has an abstraction layer that determines how data is stored. We could work with the DM folks to modify this or, alternatively, write our own data reduction code via a script that runs after each command line task (suggested by Jim, who believes that this should be straightforward). "},"comment":{"url":"https://api.github.com/repos/LSSTDESC/DC2_Repo/issues/comments/364832850","html_url":"https://github.com/LSSTDESC/DC2_Repo/issues/52#issuecomment-364832850","issue_url":"https://api.github.com/repos/LSSTDESC/DC2_Repo/issues/52","id":364832850,"user":{"login":"salmanhabib","id":9030901,"avatar_url":"https://avatars0.githubusercontent.com/u/9030901?v=4","gravatar_id":"","url":"https://api.github.com/users/salmanhabib","html_url":"https://github.com/salmanhabib","followers_url":"https://api.github.com/users/salmanhabib/followers","following_url":"https://api.github.com/users/salmanhabib/following{/other_user}","gists_url":"https://api.github.com/users/salmanhabib/gists{/gist_id}","starred_url":"https://api.github.com/users/salmanhabib/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/salmanhabib/subscriptions","organizations_url":"https://api.github.com/users/salmanhabib/orgs","repos_url":"https://api.github.com/users/salmanhabib/repos","events_url":"https://api.github.com/users/salmanhabib/events{/privacy}","received_events_url":"https://api.github.com/users/salmanhabib/received_events","type":"User","site_admin":false},"created_at":"2018-02-12T05:37:01Z","updated_at":"2018-02-12T05:37:01Z","author_association":"NONE","body":"HI Everyone, I would like to have a conclusion here so that we can plan what we need to do for Run 1.1.\r\nThe summary should be captured in a short DESC Note. "}}
{"actor":{"display_login":"salmanhabib"}}