{"action":"created","issue":{"url":"https://api.github.com/repos/metaodi/osmapi/issues/57","repository_url":"https://api.github.com/repos/metaodi/osmapi","labels_url":"https://api.github.com/repos/metaodi/osmapi/issues/57/labels{/name}","comments_url":"https://api.github.com/repos/metaodi/osmapi/issues/57/comments","events_url":"https://api.github.com/repos/metaodi/osmapi/issues/57/events","html_url":"https://github.com/metaodi/osmapi/issues/57","id":137938588,"number":57,"title":"Passing unicode into minidom","user":{"login":"MichaelVL","id":6823408,"avatar_url":"https://avatars.githubusercontent.com/u/6823408?v=3","gravatar_id":"","url":"https://api.github.com/users/MichaelVL","html_url":"https://github.com/MichaelVL","followers_url":"https://api.github.com/users/MichaelVL/followers","following_url":"https://api.github.com/users/MichaelVL/following{/other_user}","gists_url":"https://api.github.com/users/MichaelVL/gists{/gist_id}","starred_url":"https://api.github.com/users/MichaelVL/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MichaelVL/subscriptions","organizations_url":"https://api.github.com/users/MichaelVL/orgs","repos_url":"https://api.github.com/users/MichaelVL/repos","events_url":"https://api.github.com/users/MichaelVL/events{/privacy}","received_events_url":"https://api.github.com/users/MichaelVL/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":0,"created_at":"2016-03-02T17:26:53Z","updated_at":"2016-03-02T21:02:51Z","closed_at":null,"body":"This issue concerns the latest 'requests' based osmapi.\r\n\r\nI think the use of requests 'response.text' in '_http_request' and thus passing unicode into minidom is problematic.  I think the proper way to do this is to use the raw data 'request.content' for passing into minidom.  Subsequently minidom will read the encoding from the xml header (e.g. '<?xml version=\"1.0\" encoding=\"UTF-8\"?>') and handle the raw data occording to this encoding information.\r\n\r\nI seem to be able to trigger unicode issues with the following code:\r\n\r\nimport osmapi\r\na = osmapi.OsmApi()\r\nc = a.ChangesetDownload(37393499)\r\n\r\nwhich generates this unicode exception:\r\n\r\nTraceback (most recent call last):\r\n  File \"<stdin>\", line 1, in <module>\r\n  File \"osmapi/OsmApi.py\", line 1324, in ChangesetDownload\r\n    return self.ParseOsc(data)\r\n  File \"osmapi/OsmApi.py\", line 1751, in ParseOsc\r\n    data = xml.dom.minidom.parseString(data)\r\n  File \"/usr/lib/python2.7/xml/dom/minidom.py\", line 1928, in parseString\r\n    return expatbuilder.parseString(string)\r\n  File \"/usr/lib/python2.7/xml/dom/expatbuilder.py\", line 940, in parseString\r\n    return builder.parseString(string)\r\n  File \"/usr/lib/python2.7/xml/dom/expatbuilder.py\", line 223, in parseString\r\n    parser.Parse(string, True)\r\nUnicodeEncodeError: 'ascii' codec can't encode character u'\\xf8' in position 11709: ordinal not in range(128)\r\n\r\nChanging '_http_request' to use 'request.content' corrects this issue."},"comment":{"url":"https://api.github.com/repos/metaodi/osmapi/issues/comments/191428968","html_url":"https://github.com/metaodi/osmapi/issues/57#issuecomment-191428968","issue_url":"https://api.github.com/repos/metaodi/osmapi/issues/57","id":191428968,"user":{"login":"austinhartzheim","id":6235007,"avatar_url":"https://avatars.githubusercontent.com/u/6235007?v=3","gravatar_id":"","url":"https://api.github.com/users/austinhartzheim","html_url":"https://github.com/austinhartzheim","followers_url":"https://api.github.com/users/austinhartzheim/followers","following_url":"https://api.github.com/users/austinhartzheim/following{/other_user}","gists_url":"https://api.github.com/users/austinhartzheim/gists{/gist_id}","starred_url":"https://api.github.com/users/austinhartzheim/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/austinhartzheim/subscriptions","organizations_url":"https://api.github.com/users/austinhartzheim/orgs","repos_url":"https://api.github.com/users/austinhartzheim/repos","events_url":"https://api.github.com/users/austinhartzheim/events{/privacy}","received_events_url":"https://api.github.com/users/austinhartzheim/received_events","type":"User","site_admin":false},"created_at":"2016-03-02T21:02:51Z","updated_at":"2016-03-02T21:02:51Z","body":"The [requests response documentation](https://requests.readthedocs.org/en/master/user/quickstart/#response-content) seems to concur with your suggestion:\r\n\r\n> For example, HTTP and XML have the ability to specify their encoding in their body. In situations like this, you should use `r.content` to find the encoding, and then set `r.encoding`. This will let you use `r.text` with the correct encoding.\r\n\r\nI'm also curious as to whether the OSM API is using a fixed encoding for all responses. If that is the case, we could probably solve this by just setting `r.encoding` instead of having the `xml` module infer it from the binary data. But either solution would likely work."}}