{"action":"created","issue":{"url":"https://api.github.com/repos/pydata/xarray/issues/827","repository_url":"https://api.github.com/repos/pydata/xarray","labels_url":"https://api.github.com/repos/pydata/xarray/issues/827/labels{/name}","comments_url":"https://api.github.com/repos/pydata/xarray/issues/827/comments","events_url":"https://api.github.com/repos/pydata/xarray/issues/827/events","html_url":"https://github.com/pydata/xarray/issues/827","id":148876551,"number":827,"title":"Issue with GFS time reference","user":{"login":"caiostringari","id":8363752,"avatar_url":"https://avatars.githubusercontent.com/u/8363752?v=3","gravatar_id":"","url":"https://api.github.com/users/caiostringari","html_url":"https://github.com/caiostringari","followers_url":"https://api.github.com/users/caiostringari/followers","following_url":"https://api.github.com/users/caiostringari/following{/other_user}","gists_url":"https://api.github.com/users/caiostringari/gists{/gist_id}","starred_url":"https://api.github.com/users/caiostringari/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/caiostringari/subscriptions","organizations_url":"https://api.github.com/users/caiostringari/orgs","repos_url":"https://api.github.com/users/caiostringari/repos","events_url":"https://api.github.com/users/caiostringari/events{/privacy}","received_events_url":"https://api.github.com/users/caiostringari/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":3,"created_at":"2016-04-16T18:14:33Z","updated_at":"2016-04-18T00:25:00Z","closed_at":null,"body":"I am currently translating some old ferret code into python. However, when downloading GFS operational data, there was an issue...\r\n\r\nWhen downloaded from ferret, the GFS file has the following time reference (using ncdump -h):\r\n\r\n```\r\ndouble TIME(TIME) ;\r\n\t\tTIME:units = \"days since 0001-01-01 00:00:00\" ;\r\n\t\tTIME:long_name = \"time\" ;\r\n\t\tTIME:time_origin = \"01-JAN-0001 00:00:00\" ;\r\n\t\tTIME:axis = \"T\" ;\r\n\t\tTIME:standard_name = \"time\" ;\r\n```\r\n\r\nWhen using xarray to access the openDAP server and writing to disk using ds.to_netcdf(), the file has this time reference.\r\n\r\n```\r\ndouble time(time) ;\r\n\t\tstring time:grads_dim = \"t\" ;\r\n\t\tstring time:grads_mapping = \"linear\" ;\r\n\t\tstring time:grads_size = \"81\" ;\r\n\t\tstring time:grads_min = \"00z15apr2016\" ;\r\n\t\tstring time:grads_step = \"3hr\" ;\r\n\t\tstring time:long_name = \"time\" ;\r\n\t\tstring time:minimum = \"00z15apr2016\" ;\r\n\t\tstring time:maximum = \"00z25apr2016\" ;\r\n\t\ttime:resolution = 0.125f ;\r\n\t\tstring time:units = \"days since 2001-01-01\" ;\r\n\t\ttime:calendar = \"proleptic_gregorian\" ;\r\n```\r\n\r\nThis is not really an issue while using the data inside python because the dates are translated correct. However, in my work flux, I need this file to be read for other models such as  WW3. For instance, trying to read it from WW3, results in:\r\n\r\n```\r\nProcessing data\r\n --------------------------------------------------\r\n           Time : 0015/03/15 00:00:00 UTC\r\n                  reading ....\r\n                  interpolating ....\r\n                  writing ....\r\n           Time : 0015/03/15 03:00:00 UTC\r\n```\r\n\r\nLooking at the reference time, ferret gives **TIME:time_origin = \"01-JAN-0001 00:00:00\"** while xarray gives **string time:units = \"days since 2001-01-01\"**. Well, there are 2000 years missing...\r\n\r\nI tried to fix it using something like:\r\n\r\n`ds.coords['reference_time'] = pd.Timestamp('1-1-1')`\r\n\r\nBut the reference time didn't really updated. Is there an easy way to fix the reference time to match what is in the NOAA's openDAP server ?\r\n\r\n \r\n"},"comment":{"url":"https://api.github.com/repos/pydata/xarray/issues/comments/211138272","html_url":"https://github.com/pydata/xarray/issues/827#issuecomment-211138272","issue_url":"https://api.github.com/repos/pydata/xarray/issues/827","id":211138272,"user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars.githubusercontent.com/u/1217238?v=3","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"created_at":"2016-04-18T00:25:00Z","updated_at":"2016-04-18T00:25:00Z","body":"Ah, I finally figured out what's going on.\r\n\r\nWe use pandas to cleanup time units in an attempt to always write ISO-8601 compatible reference times. Unfortunately, pandas interprets dates like `'1-1-1'` or `'01-JAN-0001'` as January 1, 2001:\r\n```\r\nIn [21]: pd.Timestamp('1-1-1 00:00:0.0')\r\nOut[21]: Timestamp('2001-01-01 00:00:00')\r\n\r\nIn [25]: pd.Timestamp('01-JAN-0001 00:00:00')\r\nOut[25]: Timestamp('2001-01-01 00:00:00')\r\n```\r\nOne might argue this is a bug in pandas, but nonetheless that's what it does.\r\n\r\nxarray can currently handle datetimes outside the range dates hangled by pandas (roughly 1700-2300), but only if pandas raises an OutOfBoundDatetime error.\r\n\r\nTwo fixes that we need for this:\r\n- use netCDF4's reference time decoding (if available) *before* trying to use pandas in [decode_cf_datetime](https://github.com/pydata/xarray/blob/master/xarray/conventions.py#L115). Note that it is important to only decode only the one reference time if possible using netCDF4, because it's a lot faster to parse dates with vectorized operations with pandas/numpy.\r\n- stop using [_cleanup_netcdf_time_units](https://github.com/pydata/xarray/blob/master/xarray/conventions.py#L225), since apparently it can go wrong.\r\n\r\ncc @jhamman who has some experience with these issues"}}