{"action":"created","issue":{"url":"https://api.github.com/repos/rk700/PyMuPDF/issues/63","repository_url":"https://api.github.com/repos/rk700/PyMuPDF","labels_url":"https://api.github.com/repos/rk700/PyMuPDF/issues/63/labels{/name}","comments_url":"https://api.github.com/repos/rk700/PyMuPDF/issues/63/comments","events_url":"https://api.github.com/repos/rk700/PyMuPDF/issues/63/events","html_url":"https://github.com/rk700/PyMuPDF/issues/63","id":166505665,"number":63,"title":"Support of Unicode","user":{"login":"JorjMcKie","id":8290722,"avatar_url":"https://avatars.githubusercontent.com/u/8290722?v=3","gravatar_id":"","url":"https://api.github.com/users/JorjMcKie","html_url":"https://github.com/JorjMcKie","followers_url":"https://api.github.com/users/JorjMcKie/followers","following_url":"https://api.github.com/users/JorjMcKie/following{/other_user}","gists_url":"https://api.github.com/users/JorjMcKie/gists{/gist_id}","starred_url":"https://api.github.com/users/JorjMcKie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JorjMcKie/subscriptions","organizations_url":"https://api.github.com/users/JorjMcKie/orgs","repos_url":"https://api.github.com/users/JorjMcKie/repos","events_url":"https://api.github.com/users/JorjMcKie/events{/privacy}","received_events_url":"https://api.github.com/users/JorjMcKie/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/rk700/PyMuPDF/labels/question","name":"question","color":"cc317c"}],"state":"open","locked":false,"assignee":{"login":"rk700","id":2336827,"avatar_url":"https://avatars.githubusercontent.com/u/2336827?v=3","gravatar_id":"","url":"https://api.github.com/users/rk700","html_url":"https://github.com/rk700","followers_url":"https://api.github.com/users/rk700/followers","following_url":"https://api.github.com/users/rk700/following{/other_user}","gists_url":"https://api.github.com/users/rk700/gists{/gist_id}","starred_url":"https://api.github.com/users/rk700/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rk700/subscriptions","organizations_url":"https://api.github.com/users/rk700/orgs","repos_url":"https://api.github.com/users/rk700/repos","events_url":"https://api.github.com/users/rk700/events{/privacy}","received_events_url":"https://api.github.com/users/rk700/received_events","type":"User","site_admin":false},"assignees":[{"login":"rk700","id":2336827,"avatar_url":"https://avatars.githubusercontent.com/u/2336827?v=3","gravatar_id":"","url":"https://api.github.com/users/rk700","html_url":"https://github.com/rk700","followers_url":"https://api.github.com/users/rk700/followers","following_url":"https://api.github.com/users/rk700/following{/other_user}","gists_url":"https://api.github.com/users/rk700/gists{/gist_id}","starred_url":"https://api.github.com/users/rk700/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rk700/subscriptions","organizations_url":"https://api.github.com/users/rk700/orgs","repos_url":"https://api.github.com/users/rk700/repos","events_url":"https://api.github.com/users/rk700/events{/privacy}","received_events_url":"https://api.github.com/users/rk700/received_events","type":"User","site_admin":false},{"login":"JorjMcKie","id":8290722,"avatar_url":"https://avatars.githubusercontent.com/u/8290722?v=3","gravatar_id":"","url":"https://api.github.com/users/JorjMcKie","html_url":"https://github.com/JorjMcKie","followers_url":"https://api.github.com/users/JorjMcKie/followers","following_url":"https://api.github.com/users/JorjMcKie/following{/other_user}","gists_url":"https://api.github.com/users/JorjMcKie/gists{/gist_id}","starred_url":"https://api.github.com/users/JorjMcKie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JorjMcKie/subscriptions","organizations_url":"https://api.github.com/users/JorjMcKie/orgs","repos_url":"https://api.github.com/users/JorjMcKie/repos","events_url":"https://api.github.com/users/JorjMcKie/events{/privacy}","received_events_url":"https://api.github.com/users/JorjMcKie/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2016-07-20T06:29:49Z","updated_at":"2016-07-21T10:32:57Z","closed_at":null,"body":"@rk700 \r\nwhen introducing new methods ``setToC`` and ``setMetadata`` I realized that only a certain PDF character set really is supported by these functions: PDFDocEncoding. This character set is one of two options to be used for this type PDF information data. Other text like that of the actual PDF content is independent from this.\r\nPDFDocEncoding is a superset of ASCII and latin-1, but only supports characters of one byte - see appendix D of the PDF manual.\r\nMy question to you (as the owner and as someone who certainly uses characters from a wider Unicode range on a daily basis ...):\r\n**Should we not extend PyMuPDF's capabilities to the full unicode scope?**\r\nWouldn't a Chinese person likes to see outline item titles and metadata information in Chinese letters instead of those of the Latin alphabet? Currently, we cannot support this.\r\n\r\nApart from **PDFDocEncoding**, PDF does support full unicode, but in the **UTF-16BE** encoding only. This means that text encoded this way has to start with U+FEFF followed by the actual information, encoded in UTF-16BE.\r\nI have tested a little bit in this area, here is my current take:\r\n- Py2 and Py3 handle strings and unicode differently. This is a complication.\r\n- In order to specify a unicode string to PDF, it must apparently be provided as hexadecimal data. So e.g. instead of specifying ``/Author(Jorj McKie)`` in metadata one would have to specify ``/Author<feff004a006f0072006a0020004d0063004b00690065>``.\r\n\r\nWhen implementing this, we need to take care of Python 2/3 differences. And we maybe want to switch over to unicode only when we must ...\r\n\r\nPlease let me know your reaction."},"comment":{"url":"https://api.github.com/repos/rk700/PyMuPDF/issues/comments/234217080","html_url":"https://github.com/rk700/PyMuPDF/issues/63#issuecomment-234217080","issue_url":"https://api.github.com/repos/rk700/PyMuPDF/issues/63","id":234217080,"user":{"login":"JorjMcKie","id":8290722,"avatar_url":"https://avatars.githubusercontent.com/u/8290722?v=3","gravatar_id":"","url":"https://api.github.com/users/JorjMcKie","html_url":"https://github.com/JorjMcKie","followers_url":"https://api.github.com/users/JorjMcKie/followers","following_url":"https://api.github.com/users/JorjMcKie/following{/other_user}","gists_url":"https://api.github.com/users/JorjMcKie/gists{/gist_id}","starred_url":"https://api.github.com/users/JorjMcKie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/JorjMcKie/subscriptions","organizations_url":"https://api.github.com/users/JorjMcKie/orgs","repos_url":"https://api.github.com/users/JorjMcKie/repos","events_url":"https://api.github.com/users/JorjMcKie/events{/privacy}","received_events_url":"https://api.github.com/users/JorjMcKie/received_events","type":"User","site_admin":false},"created_at":"2016-07-21T10:32:57Z","updated_at":"2016-07-21T10:32:57Z","body":"Ok, good. I'll be working on it and come back with something hopefully usefull.\r\nSame considerations apply to metadata updates.\r\nOne more complication with Py3 is that ``x.encode()`` yields ``bytes`` not ``str``, as our SWIG interface expects. So the result of ``x.encode()`` needs to be converted back to some ``str`` before invoking SWIG functions ...\r\nIn essence I am back to handing over a utf-16BE **string** prefixed with 0xfeff, but converted to hexadecimal string characters ``EFFF...`` like in the above ``/Author`` example.\r\nThis is a regrettable waste of space in many cases. However, PDF internally stores strings in that format anyway that contains characters beyond the 8-bit range ..."}}
{"actor":{"display_login":"JorjMcKie"}}