{"action":"closed","issue":{"url":"https://api.github.com/repos/reichlab/forecast-repository/issues/84","repository_url":"https://api.github.com/repos/reichlab/forecast-repository","labels_url":"https://api.github.com/repos/reichlab/forecast-repository/issues/84/labels{/name}","comments_url":"https://api.github.com/repos/reichlab/forecast-repository/issues/84/comments","events_url":"https://api.github.com/repos/reichlab/forecast-repository/issues/84/events","html_url":"https://github.com/reichlab/forecast-repository/issues/84","id":339585829,"node_id":"MDU6SXNzdWUzMzk1ODU4Mjk=","number":84,"title":"Consider not storing bin rows with zero values","user":{"login":"matthewcornell","id":1034007,"node_id":"MDQ6VXNlcjEwMzQwMDc=","avatar_url":"https://avatars0.githubusercontent.com/u/1034007?v=4","gravatar_id":"","url":"https://api.github.com/users/matthewcornell","html_url":"https://github.com/matthewcornell","followers_url":"https://api.github.com/users/matthewcornell/followers","following_url":"https://api.github.com/users/matthewcornell/following{/other_user}","gists_url":"https://api.github.com/users/matthewcornell/gists{/gist_id}","starred_url":"https://api.github.com/users/matthewcornell/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/matthewcornell/subscriptions","organizations_url":"https://api.github.com/users/matthewcornell/orgs","repos_url":"https://api.github.com/users/matthewcornell/repos","events_url":"https://api.github.com/users/matthewcornell/events{/privacy}","received_events_url":"https://api.github.com/users/matthewcornell/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":4,"created_at":"2018-07-09T19:57:03Z","updated_at":"2018-09-19T17:54:56Z","closed_at":"2018-09-19T17:54:56Z","author_association":"MEMBER","body":"This issue captures this thread from @nickreich - https://reichlab.slack.com/archives/C57HNDFN0/p1530301398000249\r\n\r\n---- [nick]\r\n@cornell We talked today about making what we think are minor modifications to the forecast format for the impetus forecasts, but that could possibly have implications for a more general set of changes. Do you foresee any problems with these suggested changes?\r\n1. we will change bin boundary definitions for the impetus forecasts. Specifically, we would like to change so that every bin has width 1.\r\n2. to accommodate this size-wise, we'd like to have the option of removing from storage any rows of the `.cdc.csv` files that have `type==\"bin\"` and `value==0`.\r\n\r\n---- [matt]\r\nI looked at a data file -> questions:\r\n\r\n1. I see many bin values are 0, and those that aren't sum to 1.0. Is this what you mean by \"bin width 1?\"\r\n2. Does this filtering happen at the time of import, or is it new manually-activated feature (a project button, say)?\r\n3. Is this an Impetus-specific feature, or would it be useful to other projects?\r\n4. Does this need to be validated at import time?\r\n5. Are there other categories of filtering like this that we may want to implement in the future?\r\n\r\n---- [nick]\r\nBy binwidth=1 I mean that the `bin_end - bin_start` would always =1. right now, in impetus project, many bin sizes are about 10, so we would have almost a 10x increase in storage needs if we were to only implement (1) without implementing (2). (numbering referring to my original post).\r\n\r\nRe: your question 2: I was imagining this would happen prior to importing, but you could also run a check within Zoltar to save storage rows. More generally (responding to Q3), not storing rows that contain zero data might be a reasonable way to save space in all projects.\r\n\r\nQ4: it could be checked at import time, but I don't see it as __necessary__ to have all zero rows removed from imports.\r\n\r\nQ5: not sure.\r\n\r\n---- [matt]\r\nok, this needs to be talked out some more. if we decide to not store those zero rows then recreating the data (say for download) will have more steps, e.g., inferring missing bins\r\n\r\nalso, other functions that depend on all bins being present will need to accommodate the missing rows."}}
{"actor":{"display_login":"matthewcornell"}}