{"action":"created","issue":{"url":"https://api.github.com/repos/pubs/pubs/issues/23","labels_url":"https://api.github.com/repos/pubs/pubs/issues/23/labels{/name}","comments_url":"https://api.github.com/repos/pubs/pubs/issues/23/comments","events_url":"https://api.github.com/repos/pubs/pubs/issues/23/events","html_url":"https://github.com/pubs/pubs/issues/23","id":47024689,"number":23,"title":"bibtexparser does not handle files from bibutils xml2bib","user":{"login":"humm","id":2088743,"avatar_url":"https://avatars.githubusercontent.com/u/2088743?v=3","gravatar_id":"","url":"https://api.github.com/users/humm","html_url":"https://github.com/humm","followers_url":"https://api.github.com/users/humm/followers","following_url":"https://api.github.com/users/humm/following{/other_user}","gists_url":"https://api.github.com/users/humm/gists{/gist_id}","starred_url":"https://api.github.com/users/humm/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/humm/subscriptions","organizations_url":"https://api.github.com/users/humm/orgs","repos_url":"https://api.github.com/users/humm/repos","events_url":"https://api.github.com/users/humm/events{/privacy}","received_events_url":"https://api.github.com/users/humm/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/pubs/pubs/labels/bug","name":"bug","color":"fc2929"},{"url":"https://api.github.com/repos/pubs/pubs/labels/lowpriority","name":"lowpriority","color":"fef2c0"}],"state":"closed","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2014-10-28T13:21:14Z","updated_at":"2016-01-07T04:54:18Z","closed_at":"2015-12-20T21:22:28Z","body":"Might not be our fault.\r\n\r\nxml2bib generate files that have a different first character. `cmp` give that output: \r\n\r\n'''\r\ntest2.bib test3.bib differ: byte 1, line 1 is 357 � 100 @\r\n'''\r\n\r\nTo investigate further."},"comment":{"url":"https://api.github.com/repos/pubs/pubs/issues/comments/169550487","html_url":"https://github.com/pubs/pubs/issues/23#issuecomment-169550487","issue_url":"https://api.github.com/repos/pubs/pubs/issues/23","id":169550487,"user":{"login":"omangin","id":1171989,"avatar_url":"https://avatars.githubusercontent.com/u/1171989?v=3","gravatar_id":"","url":"https://api.github.com/users/omangin","html_url":"https://github.com/omangin","followers_url":"https://api.github.com/users/omangin/followers","following_url":"https://api.github.com/users/omangin/following{/other_user}","gists_url":"https://api.github.com/users/omangin/gists{/gist_id}","starred_url":"https://api.github.com/users/omangin/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/omangin/subscriptions","organizations_url":"https://api.github.com/users/omangin/orgs","repos_url":"https://api.github.com/users/omangin/repos","events_url":"https://api.github.com/users/omangin/events{/privacy}","received_events_url":"https://api.github.com/users/omangin/received_events","type":"User","site_admin":false},"created_at":"2016-01-07T04:54:18Z","updated_at":"2016-01-07T04:54:18Z","body":"Actually if this is a bibtexparser bug, it should be reported upstream instead because bibtexparser is supposed to hande BOM. Actually it does so for `0xEF 0xBB 0xBF` (<https://github.com/sciunto-org/python-bibtexparser/blob/master/bibtexparser/bparser.py#L103-L105>). What is `0xFE 0xFF` supposed to be for? Isn't it for UTF-16?\r\n\r\nHowever, according to <https://docs.python.org/3/howto/unicode.html>:\r\n> The Unicode character U+FEFF is used as a byte-order mark (BOM), and is often written as the first character of a file in order to assist with autodetection of the file’s byte ordering. Some encodings, such as UTF-16, expect a BOM to be present at the start of a file; when such an encoding is used, the BOM will be automatically written as the first character and will be silently dropped when the file is read. There are variants of these encodings, such as ‘utf-16-le’ and ‘utf-16-be’ for little-endian and big-endian encodings, that specify one particular byte ordering and don’t skip the BOM.\r\n\r\nSo actually this might be handled correctly by the `io.open` function. In which case the error is caused by <https://github.com/pubs/pubs/blob/b2ebc67f5e028a5dc1633d94397e7c8bd594bb08/pubs/content.py#L40-L44> that forces the openning in UTF-8.\r\n\r\nCan you `file` your file? What is your default locale? Does the error persist without the fix, if instead you remove the encoding='utf-8' in <https://github.com/pubs/pubs/blame/b2ebc67f5e028a5dc1633d94397e7c8bd594bb08/pubs/content.py#L42>?"}}