{"action":"opened","issue":{"url":"https://api.github.com/repos/pydata/xarray/issues/2059","repository_url":"https://api.github.com/repos/pydata/xarray","labels_url":"https://api.github.com/repos/pydata/xarray/issues/2059/labels{/name}","comments_url":"https://api.github.com/repos/pydata/xarray/issues/2059/comments","events_url":"https://api.github.com/repos/pydata/xarray/issues/2059/events","html_url":"https://github.com/pydata/xarray/issues/2059","id":314444743,"number":2059,"title":"How should xarray serialize bytes/unicode strings across Python/netCDF versions?","user":{"login":"shoyer","id":1217238,"avatar_url":"https://avatars2.githubusercontent.com/u/1217238?v=4","gravatar_id":"","url":"https://api.github.com/users/shoyer","html_url":"https://github.com/shoyer","followers_url":"https://api.github.com/users/shoyer/followers","following_url":"https://api.github.com/users/shoyer/following{/other_user}","gists_url":"https://api.github.com/users/shoyer/gists{/gist_id}","starred_url":"https://api.github.com/users/shoyer/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shoyer/subscriptions","organizations_url":"https://api.github.com/users/shoyer/orgs","repos_url":"https://api.github.com/users/shoyer/repos","events_url":"https://api.github.com/users/shoyer/events{/privacy}","received_events_url":"https://api.github.com/users/shoyer/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-15T19:36:55Z","updated_at":"2018-04-15T19:36:55Z","closed_at":null,"author_association":"OWNER","body":"# netCDF string types\r\n\r\nWe have several options for storing strings in netCDF files:\r\n- `NC_CHAR`: netCDF's legacy character type. The closest match is NumPy `'S1'` dtype. In principle, it's supposed to be able to store arbitrary bytes. On HDF5, it uses an UTF-8 encoded string with a fixed-size of 1 (but note that HDF5 does not complain about storing arbitrary bytes).\r\n- `NC_STRING`: netCDF's newer variable length string type. It's only available on netCDF4 (not netCDF3). It corresponds to an HDF5 variable-length string with UTF-8 encoding.\r\n- `NC_CHAR` with an `_Encoding` attribute: xarray and netCDF4-Python support an ad-hoc convention for storing unicode strings in `NC_CHAR` data-types, by adding an attribute `{'_Encoding': 'UTF-8'}`. The data is still stored as fixed width strings, but xarray (and netCDF4-Python) can decode them as unicode.\r\n\r\n`NC_STRING` would seem like a clear win in cases where it's supported, but as @crusaderky points out in https://github.com/pydata/xarray/issues/2040, it actually results in much larger netCDF files in many cases than using character arrays, which are more easily compressed. Nonetheless, we currently default to storing unicode strings in `NC_STRING`, because it's the most portable option -- every tool that handles HDF5 and netCDF4 should be able to read it properly as unicode strings.\r\n\r\n# NumPy/Python string types\r\n\r\nOn the Python side, our options are perhaps even more confusing:\r\n- NumPy's `dtype=np.string_` corresponds to fixed-length bytes. This is the default dtype for strings on Python 2, because on Python 2 strings are the same as bytes.\r\n- NumPy's `dtype=np.unicode_` corresponds to fixed-length unicode. This is the default dtype for strings on Python 3, because on Python 3 strings are the same as unicode.\r\n- Strings are also commonly stored in numpy arrays with `dtype=np.object_`, as arrays of either `bytes` or `unicode` objects. This is a pragmatic choice, because otherwise NumPy has no support for variable length strings. We also use this (like pandas) to mark missing values with `np.nan`.\r\n\r\nLike pandas, we are pretty liberal with converting back and forth between fixed-length (`np.string`/`np.unicode_`) and variable-length (object dtype) representations of strings as necessary. This works pretty well, though converting from object arrays in particular has downsides, since it cannot be done lazily with dask.\r\n\r\n# Current behavior of xarray\r\n\r\nCurrently, xarray uses the same behavior on Python 2/3. The priority was faithfully round-tripping data from a particular version of Python to netCDF and back, which the current serialization behavior achieves:\r\n\r\n| Python version | NetCDF version | NumPy datatype | NetCDF datatype |\r\n| --------- | ---------- | -------------- | ------------ |\r\n| Python 2 | NETCDF3 | np.string_ / str | NC_CHAR |\r\n| Python 2 | NETCDF4 | np.string_ / str | NC_CHAR |\r\n| Python 3 | NETCDF3 | np.string_ / bytes | NC_CHAR |\r\n| Python 3 | NETCDF4 | np.string_ / bytes | NC_CHAR |\r\n| Python 2 | NETCDF3 | np.unicode_ / unicode | NC_CHAR with UTF-8 encoding |\r\n| Python 2 | NETCDF4 | np.unicode_ / unicode | NC_STRING |\r\n| Python 3 | NETCDF3 | np.unicode_ / str | NC_CHAR with UTF-8 encoding |\r\n| Python 3 | NETCDF4 | np.unicode_ / str | NC_STRING |\r\n| Python 2 | NETCDF3 | object bytes/str | NC_CHAR |\r\n| Python 2 | NETCDF4 | object bytes/str | NC_CHAR |\r\n| Python 3 | NETCDF3 | object bytes | NC_CHAR |\r\n| Python 3 | NETCDF4 | object bytes | NC_CHAR |\r\n| Python 2 | NETCDF3 | object unicode | NC_CHAR with UTF-8 encoding |\r\n| Python 2 | NETCDF4 | object unicode | NC_STRING |\r\n| Python 3 | NETCDF3 | object unicode/str | NC_CHAR with UTF-8 encoding |\r\n| Python 3 | NETCDF4 | object unicode/str | NC_STRING |\r\n\r\nThis can also be selected explicitly for most data-types by setting dtype in encoding:\r\n- `'S1'` for NC_CHAR (with or without encoding)\r\n- `str` for NC_STRING (though I'm not 100% sure it works properly currently when given bytes)\r\n\r\nScript for generating table:\r\n<details>\r\n\r\n```python\r\nfrom __future__ import print_function\r\nimport xarray as xr\r\nimport uuid\r\nimport netCDF4\r\nimport numpy as np\r\nimport sys\r\n\r\nfor dtype_name, value in [\r\n    ('np.string_ / ' + type(b'').__name__, np.array([b'abc'])),\r\n    ('np.unicode_ / ' + type(u'').__name__, np.array([u'abc'])),\r\n    ('object bytes/' + type(b'').__name__, np.array([b'abc'], dtype=object)),\r\n    ('object unicode/' + type(u'').__name__, np.array([u'abc'], dtype=object)),\r\n]:\r\n    for format in ['NETCDF3_64BIT', 'NETCDF4']:\r\n        filename = str(uuid.uuid4()) + '.nc'\r\n        xr.Dataset({'data': value}).to_netcdf(filename, format=format)\r\n        with netCDF4.Dataset(filename) as f:\r\n            var = f.variables['data']\r\n            disk_dtype = var.dtype\r\n            has_encoding = hasattr(var, '_Encoding')\r\n        disk_dtype_name = (('NC_CHAR' if disk_dtype == 'S1' else 'NC_STRING') +\r\n                           (' with UTF-8 encoding' if has_encoding else ''))\r\n        print('|', 'Python %i' % sys.version_info[0],\r\n              '|', format[:7],\r\n              '|', dtype_name,\r\n              '|', disk_dtype_name,\r\n              '|')\r\n```\r\n</details>\r\n\r\n# Potential alternatives\r\n\r\nThe main option I'm considering is switching to default to `NC_CHAR` with UTF-8 encoding for np.string_ / str and object bytes/str on Python 2. The current behavior could be explicitly toggled by setting an encoding of `{'_Encoding': None}`.\r\n\r\nThis would imply two changes:\r\n1. Attempting to serialize arbitrary bytes (on Python 2) would start raising an error -- anything that isn't ASCII would require explicitly disabling `_Encoding`.\r\n2. Strings read back from disk on Python 2 would come back as unicode instead of bytes.\r\n\r\nThis implicit conversion would be consistent with Python 2's general handling of bytes/unicode, and facilitate reading netCDF files on Python 3 that were written with Python 2.\r\n\r\nThe counter-argument is that it may not be worth changing this at this late point, given that we will be sunsetting Python 2 support by year's end."}}
{"actor":{"display_login":"shoyer"}}