{"action":"created","issue":{"url":"https://api.github.com/repos/golang/go/issues/28112","repository_url":"https://api.github.com/repos/golang/go","labels_url":"https://api.github.com/repos/golang/go/issues/28112/labels{/name}","comments_url":"https://api.github.com/repos/golang/go/issues/28112/comments","events_url":"https://api.github.com/repos/golang/go/issues/28112/events","html_url":"https://github.com/golang/go/issues/28112","id":368487284,"node_id":"MDU6SXNzdWUzNjg0ODcyODQ=","number":28112,"title":"go/scanner: Scanner.Scan sometimes returns token.ILLEGAL and empty literal string","user":{"login":"dmitshur","id":1924134,"node_id":"MDQ6VXNlcjE5MjQxMzQ=","avatar_url":"https://avatars3.githubusercontent.com/u/1924134?v=4","gravatar_id":"","url":"https://api.github.com/users/dmitshur","html_url":"https://github.com/dmitshur","followers_url":"https://api.github.com/users/dmitshur/followers","following_url":"https://api.github.com/users/dmitshur/following{/other_user}","gists_url":"https://api.github.com/users/dmitshur/gists{/gist_id}","starred_url":"https://api.github.com/users/dmitshur/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dmitshur/subscriptions","organizations_url":"https://api.github.com/users/dmitshur/orgs","repos_url":"https://api.github.com/users/dmitshur/repos","events_url":"https://api.github.com/users/dmitshur/events{/privacy}","received_events_url":"https://api.github.com/users/dmitshur/received_events","type":"User","site_admin":false},"labels":[{"id":373402289,"node_id":"MDU6TGFiZWwzNzM0MDIyODk=","url":"https://api.github.com/repos/golang/go/labels/NeedsInvestigation","name":"NeedsInvestigation","color":"ededed","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":{"url":"https://api.github.com/repos/golang/go/milestones/65","html_url":"https://github.com/golang/go/milestone/65","labels_url":"https://api.github.com/repos/golang/go/milestones/65/labels","id":3072996,"node_id":"MDk6TWlsZXN0b25lMzA3Mjk5Ng==","number":65,"title":"Go1.12","description":"","creator":{"login":"rsc","id":104030,"node_id":"MDQ6VXNlcjEwNDAzMA==","avatar_url":"https://avatars1.githubusercontent.com/u/104030?v=4","gravatar_id":"","url":"https://api.github.com/users/rsc","html_url":"https://github.com/rsc","followers_url":"https://api.github.com/users/rsc/followers","following_url":"https://api.github.com/users/rsc/following{/other_user}","gists_url":"https://api.github.com/users/rsc/gists{/gist_id}","starred_url":"https://api.github.com/users/rsc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rsc/subscriptions","organizations_url":"https://api.github.com/users/rsc/orgs","repos_url":"https://api.github.com/users/rsc/repos","events_url":"https://api.github.com/users/rsc/events{/privacy}","received_events_url":"https://api.github.com/users/rsc/received_events","type":"User","site_admin":false},"open_issues":764,"closed_issues":260,"state":"open","created_at":"2018-01-29T21:17:52Z","updated_at":"2018-10-10T14:58:55Z","due_on":"2019-01-31T08:00:00Z","closed_at":null},"comments":2,"created_at":"2018-10-10T04:13:15Z","updated_at":"2018-10-10T16:53:46Z","closed_at":null,"author_association":"MEMBER","body":"### What version of Go are you using (`go version`)?\r\n\r\n<pre>\r\n$ go version\r\ngo version devel +c870d56f98 Wed Oct 10 02:08:36 2018 +0000 darwin/amd64\r\n</pre>\r\n\r\n### Does this issue reproduce with the latest release?\r\n\r\nYes it does, with Go 1.11.1.\r\n\r\n### What operating system and processor architecture are you using (`go env`)?\r\n\r\n<details><summary><code>go env</code> Output</summary><br><pre>\r\n$ go env\r\nGOARCH=\"amd64\"\r\nGOBIN=\"\"\r\nGOCACHE=\"/Users/dmitri/Library/Caches/go-build\"\r\nGOEXE=\"\"\r\nGOFLAGS=\"\"\r\nGOHOSTARCH=\"amd64\"\r\nGOHOSTOS=\"darwin\"\r\nGOOS=\"darwin\"\r\nGOPATH=\"/Users/dmitri/Dropbox/Work/2013/GoLanding:/Users/dmitri/Dropbox/Work/2013/GoLand\"\r\nGOPROXY=\"\"\r\nGORACE=\"\"\r\nGOROOT=\"/Users/dmitri/Dropbox/Work/2015/golang-contribute/go\"\r\nGOTMPDIR=\"\"\r\nGOTOOLDIR=\"/Users/dmitri/Dropbox/Work/2015/golang-contribute/go/pkg/tool/darwin_amd64\"\r\nGCCGO=\"gccgo\"\r\nCC=\"clang\"\r\nCXX=\"clang++\"\r\nCGO_ENABLED=\"1\"\r\nGOMOD=\"\"\r\nCGO_CFLAGS=\"-g -O2\"\r\nCGO_CPPFLAGS=\"\"\r\nCGO_CXXFLAGS=\"-g -O2\"\r\nCGO_FFLAGS=\"-g -O2\"\r\nCGO_LDFLAGS=\"-g -O2\"\r\nPKG_CONFIG=\"pkg-config\"\r\nGOGCCFLAGS=\"-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/b8/66r1c5856mqds1mrf2tjtq8w0000gn/T/go-build878012089=/tmp/go-build -gno-record-gcc-switches -fno-common\"\r\n</pre></details>\r\n\r\n### What did you do?\r\n\r\nI tried scanning over the following minified Go code containing illegal tokens:\r\n\r\n```\r\na..b\r\n```\r\n\r\nWith the following program:\r\n\r\n```Go\r\npackage main\r\n\r\nimport (\r\n\t\"fmt\"\r\n\t\"go/scanner\"\r\n\t\"go/token\"\r\n)\r\n\r\nfunc main() {\r\n\t// src is the input that we want to tokenize.\r\n\tsrc := []byte(\"a..b\")\r\n\r\n\t// Initialize the scanner.\r\n\tvar s scanner.Scanner\r\n\tfset := token.NewFileSet()                      // positions are relative to fset\r\n\tfile := fset.AddFile(\"\", fset.Base(), len(src)) // register input \"file\"\r\n\ts.Init(file, src, nil /* no error handler */, scanner.ScanComments)\r\n\r\n\t// Repeated calls to Scan yield the token sequence found in the input.\r\n\tfor {\r\n\t\tpos, tok, lit := s.Scan()\r\n\t\tif tok == token.EOF {\r\n\t\t\tbreak\r\n\t\t}\r\n\t\tfmt.Printf(\"%s\\t%s\\t%q\\n\", fset.Position(pos), tok, lit)\r\n\t}\r\n}\r\n```\r\n\r\n(Playground: https://play.golang.org/p/No77Yx0D4Ki.)\r\n\r\n### What did you expect to see?\r\n\r\n[`go/scanner.Scanner.Scan`](https://godoc.org/go/scanner#Scanner.Scan) documentation says:\r\n\r\n> If the returned token is token.ILLEGAL, the literal string is the offending character.\r\n>\r\n> In all other cases, Scan returns an empty literal string.\r\n\r\nPerhaps I'm misunderstanding the documentation, but I expected that if returned token is `token.ILLEGAL`, then the literal string should not be empty. I expected to see the literal string contain the offending character or characters.\r\n\r\n```\r\n1:1\tIDENT\t\"a\"\r\n1:2\tILLEGAL\t\"..\"\r\n1:4\tIDENT\t\"b\"\r\n1:5\t;\t\"\\n\"\r\n```\r\n\r\n### What did you see instead?\r\n\r\nA `token.ILLEGAL` token and empty literal string:\r\n\r\n```\r\n1:1\tIDENT\t\"a\"\r\n1:2\tILLEGAL\t\"\"\r\n1:4\tIDENT\t\"b\"\r\n1:5\t;\t\"\\n\"\r\n```\r\n\r\n/cc @griesemer"},"comment":{"url":"https://api.github.com/repos/golang/go/issues/comments/428648511","html_url":"https://github.com/golang/go/issues/28112#issuecomment-428648511","issue_url":"https://api.github.com/repos/golang/go/issues/28112","id":428648511,"node_id":"MDEyOklzc3VlQ29tbWVudDQyODY0ODUxMQ==","user":{"login":"cznic","id":729580,"node_id":"MDQ6VXNlcjcyOTU4MA==","avatar_url":"https://avatars1.githubusercontent.com/u/729580?v=4","gravatar_id":"","url":"https://api.github.com/users/cznic","html_url":"https://github.com/cznic","followers_url":"https://api.github.com/users/cznic/followers","following_url":"https://api.github.com/users/cznic/following{/other_user}","gists_url":"https://api.github.com/users/cznic/gists{/gist_id}","starred_url":"https://api.github.com/users/cznic/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cznic/subscriptions","organizations_url":"https://api.github.com/users/cznic/orgs","repos_url":"https://api.github.com/users/cznic/repos","events_url":"https://api.github.com/users/cznic/events{/privacy}","received_events_url":"https://api.github.com/users/cznic/received_events","type":"User","site_admin":false},"created_at":"2018-10-10T16:53:46Z","updated_at":"2018-10-10T16:53:46Z","author_association":"CONTRIBUTOR","body":"Heh, used `a .. b` in my test by mistake. Still, I'm not sure if the offending character, when ir does not start a token, should be consumed as it may start a valid token."}}
{"actor":{"display_login":"cznic"}}