{"action":"created","issue":{"url":"https://api.github.com/repos/Scille/umongo/issues/44","repository_url":"https://api.github.com/repos/Scille/umongo","labels_url":"https://api.github.com/repos/Scille/umongo/issues/44/labels{/name}","comments_url":"https://api.github.com/repos/Scille/umongo/issues/44/comments","events_url":"https://api.github.com/repos/Scille/umongo/issues/44/events","html_url":"https://github.com/Scille/umongo/issues/44","id":174333518,"number":44,"title":"DateTime and timezone awareness","user":{"login":"lafrech","id":1767010,"avatar_url":"https://avatars.githubusercontent.com/u/1767010?v=3","gravatar_id":"","url":"https://api.github.com/users/lafrech","html_url":"https://github.com/lafrech","followers_url":"https://api.github.com/users/lafrech/followers","following_url":"https://api.github.com/users/lafrech/following{/other_user}","gists_url":"https://api.github.com/users/lafrech/gists{/gist_id}","starred_url":"https://api.github.com/users/lafrech/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lafrech/subscriptions","organizations_url":"https://api.github.com/users/lafrech/orgs","repos_url":"https://api.github.com/users/lafrech/repos","events_url":"https://api.github.com/users/lafrech/events{/privacy}","received_events_url":"https://api.github.com/users/lafrech/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2016-08-31T17:37:14Z","updated_at":"2016-09-03T15:32:55Z","closed_at":null,"body":"I'm pulling my hair with datetime TZ awareness issues.\r\n\r\nI initiate the connection to MongoDB with `tz_aware = False`. I'm no expert about this, and I had never thought much about it before now, but from what I gathered, it seems like a reasonable choice to make. Besides, it's `pymongo`'s default (but flask-PyMongo's hardcodes `tz_aware` to `True`).\r\n\r\nWhen a document is pulled from the DB, its `DateTimeField` attribute's TZ awareness depends only on `MongoClient`'s `tz_aware` parameter (no Marshmallow schema involved):\r\n\r\n`tz_aware=True` -> pymongo provides an aware datetime -> umongo returns an aware datetime\r\n`tz_aware=False` -> pymongo provides a naive datetime -> umongo returns a naive datetime\r\n\r\nThis is one more reason to set `tz_aware = False`, because I'm passing a document/object to a lib that expects a naive timezone. (OAuth2 lib expects `expires` timestamp to be naive and compares it to `datetime.utcnow()` ([see code](https://github.com/lepture/flask-oauthlib/blob/e1bb478b2015239dd12f1d3ee13fb2627a5a3407/flask_oauthlib/provider/oauth2.py#L789)).) I suppose I could alternatively modify my getters to make all `datetime`s naive before returning tokens/grants, but on some use cases, it could get cumbersome.\r\n\r\nMarshmallow, however, returns every `datetime` as TZ aware ([doc](http://marshmallow.readthedocs.io/en/latest/api_reference.html#marshmallow.fields.DateTime)). For this reason, umongo's `DateTimeField`'s `_deserialize` method returns a TZ aware `datetime`. Since I'm using it (via webargs) to parse the inputs to my API, I'm getting TZ aware `datetime`s. Likewise, calling `load()` on a document will use `_deserialize` and result in a TZ aware `datetime`.\r\n\r\nSo if I load a date, it becomes TZ aware. Therefore, to compare it to a date from the database, this one needs to be aware as well.\r\n\r\nShould I understand that umongo is meant to be used with `tz_aware=True`, so that dates fetched from the database can be compared to dates loaded thought Marshmallow schemas?\r\n\r\nCould there be a flag/meta allowing to specify if a `DateTimeField` should return a naive `datetime`?\r\n\r\nI made a quick and dirty patch to `DateTime`'s `_deserialize` to remove the TZ from the returned output.\r\n\r\n```py\r\n    dt = super()._deserialize(value, attr, data)\r\n    return dt.replace(tzinfo=None)\r\n```\r\n\r\nThis seems to work on my use case. However, the day we complete pure Marshmallow schema export, the exported schema I pass to webargs for API input parsing won't have that feature. Unless this is added to Marshmallow as well. (I asked there about it there: https://github.com/marshmallow-code/marshmallow/issues/520.)\r\n\r\nFeedback welcome. Those TZ issues are new to me, so I may be totally misguided."},"comment":{"url":"https://api.github.com/repos/Scille/umongo/issues/comments/244552978","html_url":"https://github.com/Scille/umongo/issues/44#issuecomment-244552978","issue_url":"https://api.github.com/repos/Scille/umongo/issues/44","id":244552978,"user":{"login":"lafrech","id":1767010,"avatar_url":"https://avatars.githubusercontent.com/u/1767010?v=3","gravatar_id":"","url":"https://api.github.com/users/lafrech","html_url":"https://github.com/lafrech","followers_url":"https://api.github.com/users/lafrech/followers","following_url":"https://api.github.com/users/lafrech/following{/other_user}","gists_url":"https://api.github.com/users/lafrech/gists{/gist_id}","starred_url":"https://api.github.com/users/lafrech/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lafrech/subscriptions","organizations_url":"https://api.github.com/users/lafrech/orgs","repos_url":"https://api.github.com/users/lafrech/repos","events_url":"https://api.github.com/users/lafrech/events{/privacy}","received_events_url":"https://api.github.com/users/lafrech/received_events","type":"User","site_admin":false},"created_at":"2016-09-03T15:32:55Z","updated_at":"2016-09-03T15:32:55Z","body":"> Pymongo too low level for this task: it only helps you with the mongo world, data coming from user world are to be deserialized by marshmallow which actually returns timezone aware or naive depending of the input !!!\r\n\r\nYes, I understand this.\r\n\r\nWe need to overload `_deserialize` to enforce naiveness on data from the outside world. I was wondering about the data from the DB. Leaving `tz_aware` `False` in the connection is enough to ensure the dates from the DB are naive, but on second thought, we should probably also ensure this in `_deserialize_from_mongo` to make sure all dates in the OO world are naive (or aware, depending on the use choice) whatever is set in the connection client. Besides, this would work for any driver.\r\n\r\nWhile we're at it, we may also add a parameter to allow the user to choose the format to serialize to the outside world (currently aware by default, due to Marshmallow).\r\n\r\nTo summarize, we could have\r\n\r\n- `load_as_tz_aware` or `deserialize_as_tz_aware`: whether dates should be aware or naive when loaded from Mongo or from outside world. Affects both `_deserialize` and `_deserialize_from_mongo`. There is no need to have two different parameters here, as it wouldn't make sense to make a different choice for them.\r\n\r\n- `dump_as_tz_aware` or `serialize_as_tz_aware`: whether dates should be aware or naive when leaving to the outside world. Affects `_serialize`.\r\n\r\nIn this model, `tz_aware` in MongoClient has no effect, except on data from the aggregation framework (we might want to warn the user about that corner case).\r\n\r\n> Beside a user may want one field timezone naive and another timezone aware, so we need this configuration on a per field configuration option.\r\n\r\nAbsolutely. Those parameters are field parameters.\r\n\r\n> ```py\r\n>        if value and value.tzinfo:\r\n>            value = (value - value.utcoffset()).replace(tzinfo=None)\r\n> ```\r\n\r\nThanks for the example.\r\n\r\nThe test I made to check awareness is a bit more complex. I got it from [this SO post](http://stackoverflow.com/a/27596917/4653485). The difference would be for dates with `tzinfo != None` but `tzinfo.utcoffset()` == None.\r\n\r\n```py\r\nif date.tzinfo is not None and date.tzinfo.utcoffset(date) is not None\r\n```\r\nI'll try to rework my draft."}}
{"actor":{"display_login":"lafrech"}}