{"action":"created","issue":{"url":"https://api.github.com/repos/sabnzbd/sabnzbd/issues/543","repository_url":"https://api.github.com/repos/sabnzbd/sabnzbd","labels_url":"https://api.github.com/repos/sabnzbd/sabnzbd/issues/543/labels{/name}","comments_url":"https://api.github.com/repos/sabnzbd/sabnzbd/issues/543/comments","events_url":"https://api.github.com/repos/sabnzbd/sabnzbd/issues/543/events","html_url":"https://github.com/sabnzbd/sabnzbd/issues/543","id":147851712,"number":543,"title":"Problems in case of non-UTF-8 LANG & special characters in filenames","user":{"login":"sanderjo","id":1273502,"avatar_url":"https://avatars.githubusercontent.com/u/1273502?v=3","gravatar_id":"","url":"https://api.github.com/users/sanderjo","html_url":"https://github.com/sanderjo","followers_url":"https://api.github.com/users/sanderjo/followers","following_url":"https://api.github.com/users/sanderjo/following{/other_user}","gists_url":"https://api.github.com/users/sanderjo/gists{/gist_id}","starred_url":"https://api.github.com/users/sanderjo/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sanderjo/subscriptions","organizations_url":"https://api.github.com/users/sanderjo/orgs","repos_url":"https://api.github.com/users/sanderjo/repos","events_url":"https://api.github.com/users/sanderjo/events{/privacy}","received_events_url":"https://api.github.com/users/sanderjo/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"shypike","id":851163,"avatar_url":"https://avatars.githubusercontent.com/u/851163?v=3","gravatar_id":"","url":"https://api.github.com/users/shypike","html_url":"https://github.com/shypike","followers_url":"https://api.github.com/users/shypike/followers","following_url":"https://api.github.com/users/shypike/following{/other_user}","gists_url":"https://api.github.com/users/shypike/gists{/gist_id}","starred_url":"https://api.github.com/users/shypike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shypike/subscriptions","organizations_url":"https://api.github.com/users/shypike/orgs","repos_url":"https://api.github.com/users/shypike/repos","events_url":"https://api.github.com/users/shypike/events{/privacy}","received_events_url":"https://api.github.com/users/shypike/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/sabnzbd/sabnzbd/milestones/7","html_url":"https://github.com/sabnzbd/sabnzbd/milestones/1.1.0","labels_url":"https://api.github.com/repos/sabnzbd/sabnzbd/milestones/7/labels","id":1515062,"number":7,"title":"1.1.0","description":"New features.\r\n- Proper login screen\r\n- Multipar/phpar support\r\n","creator":{"login":"shypike","id":851163,"avatar_url":"https://avatars.githubusercontent.com/u/851163?v=3","gravatar_id":"","url":"https://api.github.com/users/shypike","html_url":"https://github.com/shypike","followers_url":"https://api.github.com/users/shypike/followers","following_url":"https://api.github.com/users/shypike/following{/other_user}","gists_url":"https://api.github.com/users/shypike/gists{/gist_id}","starred_url":"https://api.github.com/users/shypike/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shypike/subscriptions","organizations_url":"https://api.github.com/users/shypike/orgs","repos_url":"https://api.github.com/users/shypike/repos","events_url":"https://api.github.com/users/shypike/events{/privacy}","received_events_url":"https://api.github.com/users/shypike/received_events","type":"User","site_admin":false},"open_issues":24,"closed_issues":23,"state":"open","created_at":"2016-01-19T12:31:45Z","updated_at":"2016-04-13T10:32:02Z","due_on":"2016-05-30T22:00:00Z","closed_at":null},"comments":10,"created_at":"2016-04-12T19:35:25Z","updated_at":"2016-04-15T18:54:14Z","closed_at":null,"body":"**Short**\r\n\r\nif LANG is something without UTF-8 (check with ```locale``` command), and there are special characters in the filenames unpacked, SABnzbd will give errors.\r\nWorkaround: set LANG to something valid with UTF-8\r\nProposal: Let SABnzbd detect & warn\r\n\r\n**Long**\r\n\r\nSee https://forums.sabnzbd.org/viewtopic.php?f=11&t=19713&start=60#p105492\r\n\r\nSo: if LANG is something without UTF-8, and there are special characters in the filenames unpacked, SABnzbd will give errors like:\r\n```\r\n2016-04-10 15:01:33,800::INFO::[postproc:520] Traceback: \r\nTraceback (most recent call last):\r\n  File \"/usr/local/lib/python2.7/site-packages/sabnzbd/postproc.py\", line 354, in process_job\r\n    unpack_error, newfiles = unpack_magic(nzo, short_path(workdir), short_complete, flag_delete, one_folder, (), (), (), (), ())\r\n  File \"/usr/local/lib/python2.7/site-packages/sabnzbd/newsunpack.py\", line 270, in unpack_magic\r\n    xjoinables, xzips, xrars, xsevens, xts, depth)\r\n  File \"/usr/local/lib/python2.7/site-packages/sabnzbd/newsunpack.py\", line 208, in unpack_magic\r\n    xjoinables, xzips, xrars, xsevens, xts = build_filelists(workdir, workdir_complete)\r\n  File \"/usr/local/lib/python2.7/site-packages/sabnzbd/newsunpack.py\", line 1616, in build_filelists\r\n    for root, dirs, files in os.walk(workdir_complete):\r\n  File \"/usr/local/lib/python2.7/os.py\", line 286, in walk\r\n    if isdir(join(top, name)):\r\n  File \"/usr/local/lib/python2.7/posixpath.py\", line 73, in join\r\n    path += '/' + b\r\nUnicodeDecodeError: 'utf8' codec can't decode byte 0xe4 in position 36: invalid continuation byte\r\n```\r\nor\r\n```\r\nERROR: write error ( /home/sander/Downloads/complete/_UNPACK_Special filenames SQJ/stuff-to-be-uploaded/????? ??? ?? ???? ??.bin)\r\n```\r\n\r\nEasy to reproduce (on Linux and FreeBSD, and maybe OSX):\r\n```\r\nLANG= ./SABnzbd.py\r\n```\r\nand then download a post with special character filenames in it (like https://www.binsearch.info/index.php?q=Special+filenames+SQJ )\r\n\r\nSolution outside SABnzbd:\r\n```\r\nLANG=en_US.utf8 ./SABnzbd.py\r\n```\r\nI don't know the solution inside SABnzbd; In SABnzbd.py there is already this code:\r\n```\r\n# Make sure UTF-8 is default 8bit encoding\r\nif not hasattr(sys, \"setdefaultencoding\"):\r\n    reload(sys)\r\ntry:\r\n    sys.setdefaultencoding('utf-8')\r\nexcept:\r\n    print 'Sorry, you MUST add the SABnzbd folder to the PYTHONPATH environment variable'\r\n    print 'or find another way to force Python to use UTF-8 for string encoding.'\r\n    sys.exit(1)\r\n```\r\nand\r\n```\r\nimport locale\r\nimport __builtin__\r\ntry:\r\n    locale.setlocale(locale.LC_ALL, \"\")\r\n    __builtin__.__dict__['codepage'] = locale.getlocale()[1] or 'cp1252'\r\nexcept:\r\n    # Work-around for Python-ports with bad \"locale\" support\r\n    __builtin__.__dict__['codepage'] = 'cp1252'\r\n```\r\nbut that doesn't seem to work: \r\nWith ```LANG=``` the logging (for which I added a line) says:\r\n```\r\n2016-04-12 21:03:03,048::DEBUG::[SABnzbd:1371] Preferred Encoding is ANSI_X3.4-1968\r\n```\r\n... so the ```sys.setdefaultencoding('utf-8')``` didn't work, did it?\r\n\r\nSo:\r\n1) I don't know the solution within SABnzbd\r\n2) A warning + instruction within SABnzbd (+ in the FAQ) could be useful\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/sabnzbd/sabnzbd/issues/comments/210595618","html_url":"https://github.com/sabnzbd/sabnzbd/issues/543#issuecomment-210595618","issue_url":"https://api.github.com/repos/sabnzbd/sabnzbd/issues/543","id":210595618,"user":{"login":"Safihre","id":5703454,"avatar_url":"https://avatars.githubusercontent.com/u/5703454?v=3","gravatar_id":"","url":"https://api.github.com/users/Safihre","html_url":"https://github.com/Safihre","followers_url":"https://api.github.com/users/Safihre/followers","following_url":"https://api.github.com/users/Safihre/following{/other_user}","gists_url":"https://api.github.com/users/Safihre/gists{/gist_id}","starred_url":"https://api.github.com/users/Safihre/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Safihre/subscriptions","organizations_url":"https://api.github.com/users/Safihre/orgs","repos_url":"https://api.github.com/users/Safihre/repos","events_url":"https://api.github.com/users/Safihre/events{/privacy}","received_events_url":"https://api.github.com/users/Safihre/received_events","type":"User","site_admin":false},"created_at":"2016-04-15T18:54:14Z","updated_at":"2016-04-15T18:54:14Z","body":"Guess it works because Sickbeard doesn't have to call external tools?\r\n\r\nSo what should we do? Can we re-start SABnzbd with that LANG parameter from within Python?\r\nOr should we throw \r\n```\r\nWARNING: System not running with UTF-8 encoding, downloads with non-latin characters in file names will fail\r\n``` \r\nwhen \r\n```\r\n 'utf' not in locale.getpreferredencoding() \r\n```\r\n\r\nFor example? If you never download something with UTF-8 characters, it's not a breaking problem.."}}