{"action":"created","issue":{"url":"https://api.github.com/repos/gemrb/gemrb/issues/47","repository_url":"https://api.github.com/repos/gemrb/gemrb","labels_url":"https://api.github.com/repos/gemrb/gemrb/issues/47/labels{/name}","comments_url":"https://api.github.com/repos/gemrb/gemrb/issues/47/comments","events_url":"https://api.github.com/repos/gemrb/gemrb/issues/47/events","html_url":"https://github.com/gemrb/gemrb/pull/47","id":152459310,"number":47,"title":"Encoding: decouples TLK and FS path encoding","user":{"login":"MarcelHB","id":238558,"avatar_url":"https://avatars.githubusercontent.com/u/238558?v=3","gravatar_id":"","url":"https://api.github.com/users/MarcelHB","html_url":"https://github.com/MarcelHB","followers_url":"https://api.github.com/users/MarcelHB/followers","following_url":"https://api.github.com/users/MarcelHB/following{/other_user}","gists_url":"https://api.github.com/users/MarcelHB/gists{/gist_id}","starred_url":"https://api.github.com/users/MarcelHB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarcelHB/subscriptions","organizations_url":"https://api.github.com/users/MarcelHB/orgs","repos_url":"https://api.github.com/users/MarcelHB/repos","events_url":"https://api.github.com/users/MarcelHB/events{/privacy}","received_events_url":"https://api.github.com/users/MarcelHB/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":4,"created_at":"2016-05-01T22:35:24Z","updated_at":"2016-05-02T17:35:19Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/gemrb/gemrb/pulls/47","html_url":"https://github.com/gemrb/gemrb/pull/47","diff_url":"https://github.com/gemrb/gemrb/pull/47.diff","patch_url":"https://github.com/gemrb/gemrb/pull/47.patch"},"body":"My setup is as follows: I have a german IWD2 (TLK encoding ISO-8859-1), located on a Linux FS running GemRB, resource listing reveals UTF-8 file paths. The `Sounds` directory consists of 60% of files with names containing umlauts.\r\n\r\nWhen I CGen a new party, the sound files list\r\n\r\n1. displays visual garbage around umlaut-chars,\r\n2. the resource manager is unable to find the selected sound file (please review #46 first). No sound then.\r\n\r\nIn order to overcome 1, it is crucial to replace `TLKEncoding` by `FSPathEncoding` where necessary, since the global encoding makes no sense here. For 2, from the Python interface/IWD2 `CSounds` script, I made everything on the way aware of multi-byteness of the file path to use. Otherwise, it ends up in that beautiful `strnlwrcpy` that disrupts everything. I try not to break anything.\r\n\r\n**But**: I don't not know the situation at large, as e.g. `pl_uppercase`/`pl_lowercase` must have been used for files as well. Looking at the name, I assume that some polish version *might* be broken then, but that's just a guess. This is where I need further information, we could make `FSPathEncoding` configurable I guess."},"comment":{"url":"https://api.github.com/repos/gemrb/gemrb/issues/comments/216304575","html_url":"https://github.com/gemrb/gemrb/pull/47#issuecomment-216304575","issue_url":"https://api.github.com/repos/gemrb/gemrb/issues/47","id":216304575,"user":{"login":"MarcelHB","id":238558,"avatar_url":"https://avatars.githubusercontent.com/u/238558?v=3","gravatar_id":"","url":"https://api.github.com/users/MarcelHB","html_url":"https://github.com/MarcelHB","followers_url":"https://api.github.com/users/MarcelHB/followers","following_url":"https://api.github.com/users/MarcelHB/following{/other_user}","gists_url":"https://api.github.com/users/MarcelHB/gists{/gist_id}","starred_url":"https://api.github.com/users/MarcelHB/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MarcelHB/subscriptions","organizations_url":"https://api.github.com/users/MarcelHB/orgs","repos_url":"https://api.github.com/users/MarcelHB/repos","events_url":"https://api.github.com/users/MarcelHB/events{/privacy}","received_events_url":"https://api.github.com/users/MarcelHB/received_events","type":"User","site_admin":false},"created_at":"2016-05-02T17:35:19Z","updated_at":"2016-05-02T17:35:19Z","body":"Might be `<locale>` vs. `<clocale>` ...\r\n\r\n* Thing about `SetSoundFolder`/`QueryText` is: I expect they are used around scripts that might `QueryText` from some TLK-asset content. And even maybe `SetSoundFolder` by values not originating from reading the file system on demand? I haven't covered 100% of uses yet, so that's the proposal. But I can look at that. Auto-detecting encoding is complete no-go, and since we parsing regular Python values, I don't see easy ways to inject necessary states/flags to circumvent args.\r\n * I don't see ASCII-only paths breaking. I'm not sure about ISO-8859x as how such paths are used in the wild. I have a feeling of endangering pl/cz as they might have ISO encoding while `FSPathEncoding.encoding` is currently set to UTF-8, that's what the wide use of `strnlwrcpy` makes me think.\r\n * Yes, at least anything equiavalent to my setup.\r\n* Haven't found it yet, but could check `String` for such a thing. Since std-things mostly to rely on `std::locale` and no one ever included them before, I guess not. :)\r\n  * AFAIR, for all my DE-locale games, the names were uppercased and displayed correctly (Windows, NTFS), but definetely for IWD2.\r\n* Yes, ack'd. And since all the string-processing went through the mentioned `strnlwrcpy`, I expect sound files names (as far as they were subject to non-ASCII chars, but this is something I don't know) to a) have always been broken so far or b.) be broken by now."}}