{"action":"created","issue":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/28095","repository_url":"https://api.github.com/repos/pandas-dev/pandas","labels_url":"https://api.github.com/repos/pandas-dev/pandas/issues/28095/labels{/name}","comments_url":"https://api.github.com/repos/pandas-dev/pandas/issues/28095/comments","events_url":"https://api.github.com/repos/pandas-dev/pandas/issues/28095/events","html_url":"https://github.com/pandas-dev/pandas/issues/28095","id":484088146,"node_id":"MDU6SXNzdWU0ODQwODgxNDY=","number":28095,"title":"ROADMAP: Consistent missing value handling with new NA scalar","user":{"login":"jorisvandenbossche","id":1020496,"node_id":"MDQ6VXNlcjEwMjA0OTY=","avatar_url":"https://avatars2.githubusercontent.com/u/1020496?v=4","gravatar_id":"","url":"https://api.github.com/users/jorisvandenbossche","html_url":"https://github.com/jorisvandenbossche","followers_url":"https://api.github.com/users/jorisvandenbossche/followers","following_url":"https://api.github.com/users/jorisvandenbossche/following{/other_user}","gists_url":"https://api.github.com/users/jorisvandenbossche/gists{/gist_id}","starred_url":"https://api.github.com/users/jorisvandenbossche/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jorisvandenbossche/subscriptions","organizations_url":"https://api.github.com/users/jorisvandenbossche/orgs","repos_url":"https://api.github.com/users/jorisvandenbossche/repos","events_url":"https://api.github.com/users/jorisvandenbossche/events{/privacy}","received_events_url":"https://api.github.com/users/jorisvandenbossche/received_events","type":"User","site_admin":false},"labels":[{"id":35818298,"node_id":"MDU6TGFiZWwzNTgxODI5OA==","url":"https://api.github.com/repos/pandas-dev/pandas/labels/API%20Design","name":"API Design","color":"AD7FA8","default":false},{"id":2822342,"node_id":"MDU6TGFiZWwyODIyMzQy","url":"https://api.github.com/repos/pandas-dev/pandas/labels/Missing-data","name":"Missing-data","color":"d7e102","default":false},{"id":219960758,"node_id":"MDU6TGFiZWwyMTk5NjA3NTg=","url":"https://api.github.com/repos/pandas-dev/pandas/labels/Needs%20Discussion","name":"Needs Discussion","color":"207de5","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":26,"created_at":"2019-08-22T16:10:31Z","updated_at":"2019-10-03T22:44:27Z","closed_at":null,"author_association":"MEMBER","body":"I cleaned up my initial write up on the consistent missing values proposal (https://github.com/pandas-dev/pandas/issues/27825#issuecomment-520583911), and incorporated the items brought up in the last video chat. So I think it is ready for some more detailed discussion.\r\n\r\nThe last version of the full proposal can be found here: https://hackmd.io/@jorisvandenbossche/Sk0wMeAmB\r\n\r\nTL;DR:\r\n\r\n- I propose to introduce a new scalar (singleton) `pd.NA` that can be used as the missing value indicator (when accessing a single value, not necessarily how it is stored under the hood). \r\n- This can be used instead of `np.nan` or `pd.NaT` in new data types (eg nullable integers, potential string dtype)\r\n- Long term, we can see if there is a migration possible to use this consistently for all data types.\r\n\r\ncc @pandas-dev/pandas-core "},"comment":{"url":"https://api.github.com/repos/pandas-dev/pandas/issues/comments/538157468","html_url":"https://github.com/pandas-dev/pandas/issues/28095#issuecomment-538157468","issue_url":"https://api.github.com/repos/pandas-dev/pandas/issues/28095","id":538157468,"node_id":"MDEyOklzc3VlQ29tbWVudDUzODE1NzQ2OA==","user":{"login":"dhirschfeld","id":881019,"node_id":"MDQ6VXNlcjg4MTAxOQ==","avatar_url":"https://avatars1.githubusercontent.com/u/881019?v=4","gravatar_id":"","url":"https://api.github.com/users/dhirschfeld","html_url":"https://github.com/dhirschfeld","followers_url":"https://api.github.com/users/dhirschfeld/followers","following_url":"https://api.github.com/users/dhirschfeld/following{/other_user}","gists_url":"https://api.github.com/users/dhirschfeld/gists{/gist_id}","starred_url":"https://api.github.com/users/dhirschfeld/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dhirschfeld/subscriptions","organizations_url":"https://api.github.com/users/dhirschfeld/orgs","repos_url":"https://api.github.com/users/dhirschfeld/repos","events_url":"https://api.github.com/users/dhirschfeld/events{/privacy}","received_events_url":"https://api.github.com/users/dhirschfeld/received_events","type":"User","site_admin":false},"created_at":"2019-10-03T22:44:27Z","updated_at":"2019-10-03T22:44:27Z","author_association":"CONTRIBUTOR","body":"For me, a data type which can be optional/missing is different to one that can't and I think it would be better to preserve this information for each type - i.e. each scalar type has a corresponding nullable/missing type\r\n```python\r\npd.NA[int32], pd.NA[float64], pd.NA[datetime64], pd.NA[timedelta64], ...\r\n```\r\n\r\nAll specialised types could inherit from a base `pd.NA` so that isinstance checks would work:\r\n```python\r\nisinstance(pd.Na[in32], pd.NA) == True\r\nisinstance(pd.Na[datetime64], pd.NA) == True\r\n...\r\n```\r\n\r\n> Assuming that we then want a \"not a value\" for each arithmetic type (otherwise you have the same problems as now that timestamp - NaT has an unclear resulting type),\r\n\r\nIf you have a corresponding missing type for each scalar type I think it's actually clearer:\r\n```\r\ndatetime64 - pd.NA[datetime64] --> pd.NA[timedelta64]\r\ndatetime64 + pd.NA[timedelta64] --> pd.NA[datetime64]\r\n```"}}
{"actor":{"display_login":"dhirschfeld"}}