{"action":"created","comment":{"url":"https://api.github.com/repos/artefactual/archivematica/pulls/comments/81254972","pull_request_review_id":2249190,"id":81254972,"diff_hunk":"@@ -185,6 +185,21 @@ def copy_files(source_location, destination_location, files, api=None):\n         'files': files,\n         'pipeline': pipeline['resource_uri'],\n     }\n+\n+    # Here we attempt to decode the 'source' attributes of each move-file to\n+    # Unicode prior to passing to Slumber's ``post`` method. Slumber will do\n+    # this anyway and will choke in certain specific cases, specifically where\n+    # the JavaScript of the dashboard has base-64-encoded a Latin-1-encoded\n+    # string.\n+    for file_ in move_files['files']:\n+        try:\n+            file_['source'] = file_['source'].decode('utf8')\n+        except UnicodeDecodeError:\n+            try:\n+                file_['source'] = file_['source'].decode('latin-1')\n+            except UnicodeError:\n+                pass","path":"src/archivematicaCommon/lib/storageService.py","position":17,"original_position":17,"commit_id":"44f21b41ad7d1e9fe17d7ce071a0ee4dee13d992","original_commit_id":"44f21b41ad7d1e9fe17d7ce071a0ee4dee13d992","user":{"login":"Hwesta","id":3166226,"avatar_url":"https://avatars.githubusercontent.com/u/3166226?v=3","gravatar_id":"","url":"https://api.github.com/users/Hwesta","html_url":"https://github.com/Hwesta","followers_url":"https://api.github.com/users/Hwesta/followers","following_url":"https://api.github.com/users/Hwesta/following{/other_user}","gists_url":"https://api.github.com/users/Hwesta/gists{/gist_id}","starred_url":"https://api.github.com/users/Hwesta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Hwesta/subscriptions","organizations_url":"https://api.github.com/users/Hwesta/orgs","repos_url":"https://api.github.com/users/Hwesta/repos","events_url":"https://api.github.com/users/Hwesta/events{/privacy}","received_events_url":"https://api.github.com/users/Hwesta/received_events","type":"User","site_admin":false},"body":"Won't this leave `file_['source']` in the same error state as before this fix?  Or will latin-1 always succeed (if so, what generates the UnicodeError?)?","created_at":"2016-09-29T23:22:25Z","updated_at":"2016-09-29T23:24:03Z","html_url":"https://github.com/artefactual/archivematica/pull/481#discussion_r81254972","pull_request_url":"https://api.github.com/repos/artefactual/archivematica/pulls/481","_links":{"self":{"href":"https://api.github.com/repos/artefactual/archivematica/pulls/comments/81254972"},"html":{"href":"https://github.com/artefactual/archivematica/pull/481#discussion_r81254972"},"pull_request":{"href":"https://api.github.com/repos/artefactual/archivematica/pulls/481"}}},"pull_request":{"url":"https://api.github.com/repos/artefactual/archivematica/pulls/481","id":78936023,"html_url":"https://github.com/artefactual/archivematica/pull/481","diff_url":"https://github.com/artefactual/archivematica/pull/481.diff","patch_url":"https://github.com/artefactual/archivematica/pull/481.patch","issue_url":"https://api.github.com/repos/artefactual/archivematica/issues/481","number":481,"state":"open","locked":false,"title":"Dev/issue 9941 unicode directory names","user":{"login":"jrwdunham","id":2504271,"avatar_url":"https://avatars.githubusercontent.com/u/2504271?v=3","gravatar_id":"","url":"https://api.github.com/users/jrwdunham","html_url":"https://github.com/jrwdunham","followers_url":"https://api.github.com/users/jrwdunham/followers","following_url":"https://api.github.com/users/jrwdunham/following{/other_user}","gists_url":"https://api.github.com/users/jrwdunham/gists{/gist_id}","starred_url":"https://api.github.com/users/jrwdunham/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrwdunham/subscriptions","organizations_url":"https://api.github.com/users/jrwdunham/orgs","repos_url":"https://api.github.com/users/jrwdunham/repos","events_url":"https://api.github.com/users/jrwdunham/events{/privacy}","received_events_url":"https://api.github.com/users/jrwdunham/received_events","type":"User","site_admin":false},"body":"This PR fixes fixes a small bug related to non-ASCII characters in directory \r\nnames. I have the following observations related to the issue of non-ASCII\r\ncharacters in directory names. Observation #3 and the table at the bottom \r\ndescribe the bug that the present PR hopefully fixed.\r\n\r\n   1. Automation-tools' debugger spits out ``UnicodeDecodeError`` exceptions\r\n      when non-ASCII characters are in the directory names. However, if you run\r\n      automation-tools with Python 3, this does not happen. (Maybe we ought to\r\n      put the advice to use Python 3 in the documentation.)\r\n\r\n   2. At some point during its processing, Archivematica removes non-ASCII\r\n      characters from the directory names. Unicode combining characters\r\n      (diacritics) are stripped from their base characters. Other Unicode\r\n      characters (I tested the glottal stop character U+0294) are replaced with\r\n      an underscore. In the transfer METS file (in submissionDocumentation/)\r\n      the diacritics are still viewable and the characters in the higher ranges\r\n      appear to be encoded as HTML entities using their code points in decimal,\r\n      e.g., U+0294 is represented as &#660;\r\n\r\n   3. I noticed a strange discrepancy between the behaviour of AM with respect\r\n      to non-ASCII characters in transfer directory names and whether you use\r\n      the dashboard or the automation tools. With automation tools it always\r\n      works, both for diacritics and for higher-range Unicode characters. With\r\n      the dashboard, higher-range Unicode characters (e.g., U+0294) work, but\r\n      diacritics (e.g., U+0301) do not. The error returned via the JSON is\r\n      \"'utf8' codec can't decode byte 0xe9 in position 16: invalid continuation\r\n      byte\".\r\n\r\n\r\n```\r\n+--------------------------------------+\r\n|                  | filés   | filésʔ  |\r\n+--------------------------------------+\r\n| dashboard        | FAIL    | SUCCESS |\r\n+--------------------------------------+\r\n| automation-tools | SUCCESS | SUCCESS |\r\n+--------------------------------------+\r\n\r\n```\r\n","created_at":"2016-07-26T22:21:30Z","updated_at":"2016-09-29T23:24:03Z","closed_at":null,"merged_at":null,"merge_commit_sha":"55b45d521e075e5ace074c9cf62e4fecaf8b3122","assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/artefactual/archivematica/milestones/4","html_url":"https://github.com/artefactual/archivematica/milestones/1.6.0","labels_url":"https://api.github.com/repos/artefactual/archivematica/milestones/4/labels","id":1317264,"number":4,"title":"1.6.0","description":"","creator":{"login":"Hwesta","id":3166226,"avatar_url":"https://avatars.githubusercontent.com/u/3166226?v=3","gravatar_id":"","url":"https://api.github.com/users/Hwesta","html_url":"https://github.com/Hwesta","followers_url":"https://api.github.com/users/Hwesta/followers","following_url":"https://api.github.com/users/Hwesta/following{/other_user}","gists_url":"https://api.github.com/users/Hwesta/gists{/gist_id}","starred_url":"https://api.github.com/users/Hwesta/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Hwesta/subscriptions","organizations_url":"https://api.github.com/users/Hwesta/orgs","repos_url":"https://api.github.com/users/Hwesta/repos","events_url":"https://api.github.com/users/Hwesta/events{/privacy}","received_events_url":"https://api.github.com/users/Hwesta/received_events","type":"User","site_admin":false},"open_issues":9,"closed_issues":111,"state":"open","created_at":"2015-09-21T20:50:42Z","updated_at":"2016-09-29T20:44:15Z","due_on":null,"closed_at":null},"commits_url":"https://api.github.com/repos/artefactual/archivematica/pulls/481/commits","review_comments_url":"https://api.github.com/repos/artefactual/archivematica/pulls/481/comments","review_comment_url":"https://api.github.com/repos/artefactual/archivematica/pulls/comments{/number}","comments_url":"https://api.github.com/repos/artefactual/archivematica/issues/481/comments","statuses_url":"https://api.github.com/repos/artefactual/archivematica/statuses/44f21b41ad7d1e9fe17d7ce071a0ee4dee13d992","head":{"label":"artefactual:dev/issue-9941-unicode-directory-names","ref":"dev/issue-9941-unicode-directory-names","sha":"44f21b41ad7d1e9fe17d7ce071a0ee4dee13d992","user":{"login":"artefactual","id":1163545,"avatar_url":"https://avatars.githubusercontent.com/u/1163545?v=3","gravatar_id":"","url":"https://api.github.com/users/artefactual","html_url":"https://github.com/artefactual","followers_url":"https://api.github.com/users/artefactual/followers","following_url":"https://api.github.com/users/artefactual/following{/other_user}","gists_url":"https://api.github.com/users/artefactual/gists{/gist_id}","starred_url":"https://api.github.com/users/artefactual/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artefactual/subscriptions","organizations_url":"https://api.github.com/users/artefactual/orgs","repos_url":"https://api.github.com/users/artefactual/repos","events_url":"https://api.github.com/users/artefactual/events{/privacy}","received_events_url":"https://api.github.com/users/artefactual/received_events","type":"Organization","site_admin":false},"repo":{"id":5811958,"name":"archivematica","full_name":"artefactual/archivematica","owner":{"login":"artefactual","id":1163545,"avatar_url":"https://avatars.githubusercontent.com/u/1163545?v=3","gravatar_id":"","url":"https://api.github.com/users/artefactual","html_url":"https://github.com/artefactual","followers_url":"https://api.github.com/users/artefactual/followers","following_url":"https://api.github.com/users/artefactual/following{/other_user}","gists_url":"https://api.github.com/users/artefactual/gists{/gist_id}","starred_url":"https://api.github.com/users/artefactual/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artefactual/subscriptions","organizations_url":"https://api.github.com/users/artefactual/orgs","repos_url":"https://api.github.com/users/artefactual/repos","events_url":"https://api.github.com/users/artefactual/events{/privacy}","received_events_url":"https://api.github.com/users/artefactual/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/artefactual/archivematica","description":"Free and open-source digital preservation system designed to maintain standards-based, long-term access to collections of digital objects.","fork":false,"url":"https://api.github.com/repos/artefactual/archivematica","forks_url":"https://api.github.com/repos/artefactual/archivematica/forks","keys_url":"https://api.github.com/repos/artefactual/archivematica/keys{/key_id}","collaborators_url":"https://api.github.com/repos/artefactual/archivematica/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/artefactual/archivematica/teams","hooks_url":"https://api.github.com/repos/artefactual/archivematica/hooks","issue_events_url":"https://api.github.com/repos/artefactual/archivematica/issues/events{/number}","events_url":"https://api.github.com/repos/artefactual/archivematica/events","assignees_url":"https://api.github.com/repos/artefactual/archivematica/assignees{/user}","branches_url":"https://api.github.com/repos/artefactual/archivematica/branches{/branch}","tags_url":"https://api.github.com/repos/artefactual/archivematica/tags","blobs_url":"https://api.github.com/repos/artefactual/archivematica/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/artefactual/archivematica/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/artefactual/archivematica/git/refs{/sha}","trees_url":"https://api.github.com/repos/artefactual/archivematica/git/trees{/sha}","statuses_url":"https://api.github.com/repos/artefactual/archivematica/statuses/{sha}","languages_url":"https://api.github.com/repos/artefactual/archivematica/languages","stargazers_url":"https://api.github.com/repos/artefactual/archivematica/stargazers","contributors_url":"https://api.github.com/repos/artefactual/archivematica/contributors","subscribers_url":"https://api.github.com/repos/artefactual/archivematica/subscribers","subscription_url":"https://api.github.com/repos/artefactual/archivematica/subscription","commits_url":"https://api.github.com/repos/artefactual/archivematica/commits{/sha}","git_commits_url":"https://api.github.com/repos/artefactual/archivematica/git/commits{/sha}","comments_url":"https://api.github.com/repos/artefactual/archivematica/comments{/number}","issue_comment_url":"https://api.github.com/repos/artefactual/archivematica/issues/comments{/number}","contents_url":"https://api.github.com/repos/artefactual/archivematica/contents/{+path}","compare_url":"https://api.github.com/repos/artefactual/archivematica/compare/{base}...{head}","merges_url":"https://api.github.com/repos/artefactual/archivematica/merges","archive_url":"https://api.github.com/repos/artefactual/archivematica/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/artefactual/archivematica/downloads","issues_url":"https://api.github.com/repos/artefactual/archivematica/issues{/number}","pulls_url":"https://api.github.com/repos/artefactual/archivematica/pulls{/number}","milestones_url":"https://api.github.com/repos/artefactual/archivematica/milestones{/number}","notifications_url":"https://api.github.com/repos/artefactual/archivematica/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/artefactual/archivematica/labels{/name}","releases_url":"https://api.github.com/repos/artefactual/archivematica/releases{/id}","deployments_url":"https://api.github.com/repos/artefactual/archivematica/deployments","created_at":"2012-09-14T16:43:38Z","updated_at":"2016-09-28T19:25:06Z","pushed_at":"2016-09-29T22:58:02Z","git_url":"git://github.com/artefactual/archivematica.git","ssh_url":"git@github.com:artefactual/archivematica.git","clone_url":"https://github.com/artefactual/archivematica.git","svn_url":"https://github.com/artefactual/archivematica","homepage":"http://www.archivematica.org","size":33936,"stargazers_count":93,"watchers_count":93,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":29,"mirror_url":null,"open_issues_count":24,"forks":29,"open_issues":24,"watchers":93,"default_branch":"stable/1.5.x"}},"base":{"label":"artefactual:qa/1.x","ref":"qa/1.x","sha":"254a73d44578f28f4c957da34e35dfcd8a9b0d16","user":{"login":"artefactual","id":1163545,"avatar_url":"https://avatars.githubusercontent.com/u/1163545?v=3","gravatar_id":"","url":"https://api.github.com/users/artefactual","html_url":"https://github.com/artefactual","followers_url":"https://api.github.com/users/artefactual/followers","following_url":"https://api.github.com/users/artefactual/following{/other_user}","gists_url":"https://api.github.com/users/artefactual/gists{/gist_id}","starred_url":"https://api.github.com/users/artefactual/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artefactual/subscriptions","organizations_url":"https://api.github.com/users/artefactual/orgs","repos_url":"https://api.github.com/users/artefactual/repos","events_url":"https://api.github.com/users/artefactual/events{/privacy}","received_events_url":"https://api.github.com/users/artefactual/received_events","type":"Organization","site_admin":false},"repo":{"id":5811958,"name":"archivematica","full_name":"artefactual/archivematica","owner":{"login":"artefactual","id":1163545,"avatar_url":"https://avatars.githubusercontent.com/u/1163545?v=3","gravatar_id":"","url":"https://api.github.com/users/artefactual","html_url":"https://github.com/artefactual","followers_url":"https://api.github.com/users/artefactual/followers","following_url":"https://api.github.com/users/artefactual/following{/other_user}","gists_url":"https://api.github.com/users/artefactual/gists{/gist_id}","starred_url":"https://api.github.com/users/artefactual/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/artefactual/subscriptions","organizations_url":"https://api.github.com/users/artefactual/orgs","repos_url":"https://api.github.com/users/artefactual/repos","events_url":"https://api.github.com/users/artefactual/events{/privacy}","received_events_url":"https://api.github.com/users/artefactual/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/artefactual/archivematica","description":"Free and open-source digital preservation system designed to maintain standards-based, long-term access to collections of digital objects.","fork":false,"url":"https://api.github.com/repos/artefactual/archivematica","forks_url":"https://api.github.com/repos/artefactual/archivematica/forks","keys_url":"https://api.github.com/repos/artefactual/archivematica/keys{/key_id}","collaborators_url":"https://api.github.com/repos/artefactual/archivematica/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/artefactual/archivematica/teams","hooks_url":"https://api.github.com/repos/artefactual/archivematica/hooks","issue_events_url":"https://api.github.com/repos/artefactual/archivematica/issues/events{/number}","events_url":"https://api.github.com/repos/artefactual/archivematica/events","assignees_url":"https://api.github.com/repos/artefactual/archivematica/assignees{/user}","branches_url":"https://api.github.com/repos/artefactual/archivematica/branches{/branch}","tags_url":"https://api.github.com/repos/artefactual/archivematica/tags","blobs_url":"https://api.github.com/repos/artefactual/archivematica/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/artefactual/archivematica/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/artefactual/archivematica/git/refs{/sha}","trees_url":"https://api.github.com/repos/artefactual/archivematica/git/trees{/sha}","statuses_url":"https://api.github.com/repos/artefactual/archivematica/statuses/{sha}","languages_url":"https://api.github.com/repos/artefactual/archivematica/languages","stargazers_url":"https://api.github.com/repos/artefactual/archivematica/stargazers","contributors_url":"https://api.github.com/repos/artefactual/archivematica/contributors","subscribers_url":"https://api.github.com/repos/artefactual/archivematica/subscribers","subscription_url":"https://api.github.com/repos/artefactual/archivematica/subscription","commits_url":"https://api.github.com/repos/artefactual/archivematica/commits{/sha}","git_commits_url":"https://api.github.com/repos/artefactual/archivematica/git/commits{/sha}","comments_url":"https://api.github.com/repos/artefactual/archivematica/comments{/number}","issue_comment_url":"https://api.github.com/repos/artefactual/archivematica/issues/comments{/number}","contents_url":"https://api.github.com/repos/artefactual/archivematica/contents/{+path}","compare_url":"https://api.github.com/repos/artefactual/archivematica/compare/{base}...{head}","merges_url":"https://api.github.com/repos/artefactual/archivematica/merges","archive_url":"https://api.github.com/repos/artefactual/archivematica/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/artefactual/archivematica/downloads","issues_url":"https://api.github.com/repos/artefactual/archivematica/issues{/number}","pulls_url":"https://api.github.com/repos/artefactual/archivematica/pulls{/number}","milestones_url":"https://api.github.com/repos/artefactual/archivematica/milestones{/number}","notifications_url":"https://api.github.com/repos/artefactual/archivematica/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/artefactual/archivematica/labels{/name}","releases_url":"https://api.github.com/repos/artefactual/archivematica/releases{/id}","deployments_url":"https://api.github.com/repos/artefactual/archivematica/deployments","created_at":"2012-09-14T16:43:38Z","updated_at":"2016-09-28T19:25:06Z","pushed_at":"2016-09-29T22:58:02Z","git_url":"git://github.com/artefactual/archivematica.git","ssh_url":"git@github.com:artefactual/archivematica.git","clone_url":"https://github.com/artefactual/archivematica.git","svn_url":"https://github.com/artefactual/archivematica","homepage":"http://www.archivematica.org","size":33936,"stargazers_count":93,"watchers_count":93,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":false,"has_pages":false,"forks_count":29,"mirror_url":null,"open_issues_count":24,"forks":29,"open_issues":24,"watchers":93,"default_branch":"stable/1.5.x"}},"_links":{"self":{"href":"https://api.github.com/repos/artefactual/archivematica/pulls/481"},"html":{"href":"https://github.com/artefactual/archivematica/pull/481"},"issue":{"href":"https://api.github.com/repos/artefactual/archivematica/issues/481"},"comments":{"href":"https://api.github.com/repos/artefactual/archivematica/issues/481/comments"},"review_comments":{"href":"https://api.github.com/repos/artefactual/archivematica/pulls/481/comments"},"review_comment":{"href":"https://api.github.com/repos/artefactual/archivematica/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/artefactual/archivematica/pulls/481/commits"},"statuses":{"href":"https://api.github.com/repos/artefactual/archivematica/statuses/44f21b41ad7d1e9fe17d7ce071a0ee4dee13d992"}}}}
{"actor":{"display_login":"Hwesta"}}