{"action":"created","issue":{"url":"https://api.github.com/repos/semio/ddf_utils/issues/43","repository_url":"https://api.github.com/repos/semio/ddf_utils","labels_url":"https://api.github.com/repos/semio/ddf_utils/issues/43/labels{/name}","comments_url":"https://api.github.com/repos/semio/ddf_utils/issues/43/comments","events_url":"https://api.github.com/repos/semio/ddf_utils/issues/43/events","html_url":"https://github.com/semio/ddf_utils/issues/43","id":193664862,"number":43,"title":"add option to set data type in recipe ingredient","user":{"login":"semio","id":351856,"avatar_url":"https://avatars.githubusercontent.com/u/351856?v=3","gravatar_id":"","url":"https://api.github.com/users/semio","html_url":"https://github.com/semio","followers_url":"https://api.github.com/users/semio/followers","following_url":"https://api.github.com/users/semio/following{/other_user}","gists_url":"https://api.github.com/users/semio/gists{/gist_id}","starred_url":"https://api.github.com/users/semio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/semio/subscriptions","organizations_url":"https://api.github.com/users/semio/orgs","repos_url":"https://api.github.com/users/semio/repos","events_url":"https://api.github.com/users/semio/events{/privacy}","received_events_url":"https://api.github.com/users/semio/received_events","type":"User","site_admin":false},"labels":[{"id":409892936,"url":"https://api.github.com/repos/semio/ddf_utils/labels/recipe","name":"recipe","color":"bfdadc","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":5,"created_at":"2016-12-06T02:13:30Z","updated_at":"2016-12-08T01:23:49Z","closed_at":null,"body":"Problems: \r\n\r\n1. When reading data from datasets, chef don't have options to set data type for columns\r\n2. When writing data into csv, chef don't have options to set data  format for columns \r\n\r\n## input\r\n\r\nFor example, say we have data like this:\r\n\r\n| country_id | time | life_expectancy |\r\n| --- | --- | --- |\r\n| 1 | 1990 | 60 |\r\n| 2 | 1990 | 70 |\r\n| 3 | 1990 | 80 |\r\n\r\nWhen we use chef to read this file, it will guess the data type of each column, so that all of three columns are `int` by default. But if we want to set country_id to `string`, life_expectancy to `float`,  and time to `datatime`, currently there is no way to change the types. To support this feature, we can add options to the ingredient like this:\r\n\r\n```yaml\r\n- id: life_expectancy_datapoints\r\n  dataset: example_dataset\r\n  key: 'country_id,time'\r\n  value: '*'\r\n  options:\r\n    dtype:\r\n      country_id: str\r\n      time: datetime\r\n      life_expectancy: float\r\n```\r\n\r\n## output\r\n\r\nWhen saving data to csv, we may want to set the format for some columns. Like adjusting floating point precision, and use Upper case for bool values.\r\n\r\nTake the above example, if we save the data to disk using chef, then it will be the same as input.\r\n\r\n| country_id | time | life_expectancy |\r\n| --- | --- | --- |\r\n| 1 | 1990 | 60 |\r\n| 2 | 1990 | 70 |\r\n| 3 | 1990 | 80 |\r\n\r\nBut what if we want to have more precision on the `life_expactancy` column like below? There is no way to do this for now.\r\n\r\n| country_id | time | life_expectancy |\r\n| --- | --- | --- |\r\n| 1 | 1990 | 60.0 |\r\n| 2 | 1990 | 70.0 |\r\n| 3 | 1990 | 80.0 |\r\n\r\nTo support this we can add options to serving procedure / section. Or alternatively add a new procedure to convert column format before serving. Not sure which one will be better for now.\r\n\r\n```yaml\r\nserving:\r\n    - id: life_expectancy_datapoints\r\n      options:\r\n         formatter:\r\n           life_expectancy:\r\n             precision: 5\r\n```"},"comment":{"url":"https://api.github.com/repos/semio/ddf_utils/issues/comments/265626092","html_url":"https://github.com/semio/ddf_utils/issues/43#issuecomment-265626092","issue_url":"https://api.github.com/repos/semio/ddf_utils/issues/43","id":265626092,"user":{"login":"semio","id":351856,"avatar_url":"https://avatars.githubusercontent.com/u/351856?v=3","gravatar_id":"","url":"https://api.github.com/users/semio","html_url":"https://github.com/semio","followers_url":"https://api.github.com/users/semio/followers","following_url":"https://api.github.com/users/semio/following{/other_user}","gists_url":"https://api.github.com/users/semio/gists{/gist_id}","starred_url":"https://api.github.com/users/semio/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/semio/subscriptions","organizations_url":"https://api.github.com/users/semio/orgs","repos_url":"https://api.github.com/users/semio/repos","events_url":"https://api.github.com/users/semio/events{/privacy}","received_events_url":"https://api.github.com/users/semio/received_events","type":"User","site_admin":false},"created_at":"2016-12-08T01:23:49Z","updated_at":"2016-12-08T01:23:49Z","body":"> Since chef has only ddf datasets as input, aren't concept types enough for this input case? entity sets, entity domains and strings are strings, measures are numbers, times are.. time.\r\n\r\nAh, yes, we could judge by the concept_type. I wasn't aware of that :)"}}
{"actor":{"display_login":"semio"}}