{"action":"created","issue":{"url":"https://api.github.com/repos/github/git-lfs/issues/987","repository_url":"https://api.github.com/repos/github/git-lfs","labels_url":"https://api.github.com/repos/github/git-lfs/issues/987/labels{/name}","comments_url":"https://api.github.com/repos/github/git-lfs/issues/987/comments","events_url":"https://api.github.com/repos/github/git-lfs/issues/987/events","html_url":"https://github.com/github/git-lfs/issues/987","id":132367166,"number":987,"title":"\"Problem accessing\" files with unicode characters in their names","user":{"login":"Scratch-net","id":7219414,"avatar_url":"https://avatars.githubusercontent.com/u/7219414?v=3","gravatar_id":"","url":"https://api.github.com/users/Scratch-net","html_url":"https://github.com/Scratch-net","followers_url":"https://api.github.com/users/Scratch-net/followers","following_url":"https://api.github.com/users/Scratch-net/following{/other_user}","gists_url":"https://api.github.com/users/Scratch-net/gists{/gist_id}","starred_url":"https://api.github.com/users/Scratch-net/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/Scratch-net/subscriptions","organizations_url":"https://api.github.com/users/Scratch-net/orgs","repos_url":"https://api.github.com/users/Scratch-net/repos","events_url":"https://api.github.com/users/Scratch-net/events{/privacy}","received_events_url":"https://api.github.com/users/Scratch-net/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/github/git-lfs/labels/bug","name":"bug","color":"fc2929"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-02-09T09:18:01Z","updated_at":"2016-02-19T18:09:56Z","closed_at":null,"body":"We have some files with unicode characters like \r\n\r\n> PB SCN 16 Odhrán.wav\r\n\r\nand when cloning such a repository on windows git-lfs fails to access them\r\nHere's the log. \\<skipped\\> reffers to some unnecessary path part\r\n\r\n```\r\ngit-lfs/1.1.1 (GitHub; windows amd64; go 1.5.3; git 69d1419)\r\ngit version 2.7.0.windows.1\r\n\r\n$ git-lfs.exe pull\r\n....\r\nProblem accessing \"<skipped>_odhr\\303\\243\\302\\242n.wav\"\r\n...\r\n\r\nGetFileAttributesEx \"<skipped>_tom\\303\\243\\302\\242s.wav\": The filename, directory name, or volume label syntax is incorrect.\r\ngoroutine 66 [running]:\r\ngithub.com/github/git-lfs/lfs.Stack(0x0, 0x0, 0x0)\r\n\tC:/Users/techn/go/src/github.com/github/git-lfs/lfs/errors.go:557 +0x87\r\ngithub.com/github/git-lfs/commands.logPanicToWriter(0x2b941c0, 0xc08244a048, 0x2b94528, 0xc082977920)\r\n\tC:/Users/techn/go/src/github.com/github/git-lfs/commands/commands.go:184 +0xf86\r\ngithub.com/github/git-lfs/commands.logPanic(0x2b94528, 0xc082977920, 0x0, 0x0)\r\n\tC:/Users/techn/go/src/github.com/github/git-lfs/commands/commands.go:148 +0x428\r\ngithub.com/github/git-lfs/commands.handlePanic(0x2b94528, 0xc082977920, 0x0, 0x0)\r\n\tC:/Users/techn/go/src/github.com/github/git-lfs/commands/commands.go:123 +0x55\r\ngithub.com/github/git-lfs/commands.LoggedError(0x2b94528, 0xc082977920, 0x958120, 0x14, 0xc082ac3ea8, 0x1, 0x1)\r\n\tC:/Users/techn/go/src/github.com/github/git-lfs/commands/commands.go:73 +0x89\r\ngithub.com/github/git-lfs/commands.checkoutWithChan(0xc082312cc0)\r\n\tC:/Users/techn/go/src/github.com/github/git-lfs/commands/command_checkout.go:183 +0x304\r\ngithub.com/github/git-lfs/commands.checkoutFromFetchChan.func1(0xc082312cc0, 0xc082004560)\r\n\tC:/Users/techn/go/src/github.com/github/git-lfs/commands/command_checkout.go:78 +0x28\r\ncreated by github.com/github/git-lfs/commands.checkoutFromFetchChan\r\n\tC:/Users/techn/go/src/github.com/github/git-lfs/commands/command_checkout.go:80 +0x440\r\n```"},"comment":{"url":"https://api.github.com/repos/github/git-lfs/issues/comments/186337634","html_url":"https://github.com/github/git-lfs/issues/987#issuecomment-186337634","issue_url":"https://api.github.com/repos/github/git-lfs/issues/987","id":186337634,"user":{"login":"technoweenie","id":21,"avatar_url":"https://avatars.githubusercontent.com/u/21?v=3","gravatar_id":"","url":"https://api.github.com/users/technoweenie","html_url":"https://github.com/technoweenie","followers_url":"https://api.github.com/users/technoweenie/followers","following_url":"https://api.github.com/users/technoweenie/following{/other_user}","gists_url":"https://api.github.com/users/technoweenie/gists{/gist_id}","starred_url":"https://api.github.com/users/technoweenie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/technoweenie/subscriptions","organizations_url":"https://api.github.com/users/technoweenie/orgs","repos_url":"https://api.github.com/users/technoweenie/repos","events_url":"https://api.github.com/users/technoweenie/events{/privacy}","received_events_url":"https://api.github.com/users/technoweenie/received_events","type":"User","site_admin":true},"created_at":"2016-02-19T18:09:56Z","updated_at":"2016-02-19T18:09:56Z","body":"I replicated this on my Windows VM, but couldn't on my Mac laptop. But, I had another issue on my Mac, but only when using `git lfs pull`:\r\n\r\n```\r\n$ git status\r\n## master...origin/master\r\n?? \"\\\"PB SCN 16 Odhr\\\\303\\\\241n.wav\\\"\"\r\n```\r\n\r\nSo even if it worked, it's likely that it would've written the wrong file.\r\n\r\nHere's what I _think_ is going on. If you have Git LFS configured with the default smudge filters, everything works. This is because git is passing the correctly-formatted filename to `git lfs smudge`.\r\n\r\nHowever, if you enable `git lfs install --skip-smudge`, you have to run `git lfs pull` to download the LFS objects. It gets it's filenames through `git ls-tree` in [`lsTreeBlobs()`](https://github.com/github/git-lfs/blob/a2781ff9cd5c38d0662218412910cbd14d296ed7/lfs/scanner.go#L577). Here's what the command output looks like:\r\n\r\n```\r\n$ git ls-tree master -rl\r\n100644 blob d899f6551a51cf19763c5955c7a06a2726f018e9      42\t.gitattributes\r\n100644 blob 4d343e022e11a8618db494dc3c501e80c7e18197     126\t\"PB SCN 16 Odhr\\303\\241n.wav\"\r\n```\r\n\r\nOur Go code is parsing that to the literal filename of `\"PB SCN 16 Odhr\\303\\241n.wav\"`. Git LFS on the mac happily accepts this, downloads the file, and writes `\"PB SCN 16 Odhr\\303\\241n.wav\"`, instead of replacing `PB SCN 16 Odhrán.wav`.\r\n\r\n```bash\r\n$ ls -al\r\ntotal 24\r\n-rw-r--r--   1 rick  staff    4 Feb 19 10:41 \"PB SCN 16 Odhr\\303\\241n.wav\"\r\ndrwxr-xr-x   6 rick  staff  204 Feb 19 10:41 .\r\ndrwxr-xr-x  23 rick  staff  782 Feb 19 10:41 ..\r\ndrwxr-xr-x  14 rick  staff  476 Feb 19 10:42 .git\r\n-rw-r--r--   1 rick  staff   42 Feb 19 10:41 .gitattributes\r\n-rw-r--r--   1 rick  staff  126 Feb 19 10:41 PB SCN 16 Odhrán.wav\r\n```\r\n\r\nIn this output, the original pointer is intact, using 126 bytes. The \"large content\" is literally just `wav\\n`, which you can see in the bad file.\r\n\r\nBased on all this, I'm going to look into the following fixes:\r\n\r\n* Raise if `lsTreeBlobs()` returns a filename that doesn't exist in the working directory. Assuming a clean checkout, this should never happen. We don't want the lfs checkout process to create new files.\r\n* Change `lsTreeBlobs()` so it uses the `-z` option. This produces more raw output with the real filename. It appears all on one line because lines are separated by a null byte.\r\n\r\n```bash\r\n$ git ls-tree master -rlz\r\n100644 blob d899f6551a51cf19763c5955c7a06a2726f018e9      42\t.gitattributes100644 blob 4d343e022e11a8618db494dc3c501e80c7e18197     126\tPB SCN 16 Odhrán.wav\r\n```\r\n\r\nNotice the lack of quotes or unicode escape codes."}}