{"action":"created","issue":{"url":"https://api.github.com/repos/golang/go/issues/13848","repository_url":"https://api.github.com/repos/golang/go","labels_url":"https://api.github.com/repos/golang/go/issues/13848/labels{/name}","comments_url":"https://api.github.com/repos/golang/go/issues/13848/comments","events_url":"https://api.github.com/repos/golang/go/issues/13848/events","html_url":"https://github.com/golang/go/issues/13848","id":125293834,"number":13848,"title":"strings: optimize WriteTo to use an intermediate buffer for large strings","user":{"login":"dsnet","id":6354026,"avatar_url":"https://avatars.githubusercontent.com/u/6354026?v=3","gravatar_id":"","url":"https://api.github.com/users/dsnet","html_url":"https://github.com/dsnet","followers_url":"https://api.github.com/users/dsnet/followers","following_url":"https://api.github.com/users/dsnet/following{/other_user}","gists_url":"https://api.github.com/users/dsnet/gists{/gist_id}","starred_url":"https://api.github.com/users/dsnet/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dsnet/subscriptions","organizations_url":"https://api.github.com/users/dsnet/orgs","repos_url":"https://api.github.com/users/dsnet/repos","events_url":"https://api.github.com/users/dsnet/events{/privacy}","received_events_url":"https://api.github.com/users/dsnet/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":{"url":"https://api.github.com/repos/golang/go/milestones/31","html_url":"https://github.com/golang/go/milestones/Go1.7","labels_url":"https://api.github.com/repos/golang/go/milestones/31/labels","id":1414133,"number":31,"title":"Go1.7","description":"","creator":{"login":"bradfitz","id":2621,"avatar_url":"https://avatars.githubusercontent.com/u/2621?v=3","gravatar_id":"","url":"https://api.github.com/users/bradfitz","html_url":"https://github.com/bradfitz","followers_url":"https://api.github.com/users/bradfitz/followers","following_url":"https://api.github.com/users/bradfitz/following{/other_user}","gists_url":"https://api.github.com/users/bradfitz/gists{/gist_id}","starred_url":"https://api.github.com/users/bradfitz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bradfitz/subscriptions","organizations_url":"https://api.github.com/users/bradfitz/orgs","repos_url":"https://api.github.com/users/bradfitz/repos","events_url":"https://api.github.com/users/bradfitz/events{/privacy}","received_events_url":"https://api.github.com/users/bradfitz/received_events","type":"User","site_admin":false},"open_issues":217,"closed_issues":31,"state":"open","created_at":"2015-11-16T16:57:07Z","updated_at":"2016-02-20T01:58:16Z","due_on":"2016-07-31T07:00:00Z","closed_at":null},"comments":9,"created_at":"2016-01-07T00:16:40Z","updated_at":"2016-02-20T05:00:45Z","closed_at":null,"body":"Using ```go1.5```\r\n\r\nNOTE: This issue used to be about using an intermediate buffer in io.WriteString, but will instead perform the optimization in strings.Reader.WriteTo instead. The description in this issue still refers to WriteString, but the performance numbers will probably be the same once applied to strings.Reader.WriteTo.\r\n\r\n--\r\n\r\nCurrently WriteString does ```w.Write([]byte(s))``` if ```w``` is not a ```stringWriter```. This causes a memory allocation proportional to len(s). Instead, we should use an intermediate buffer for large strings.\r\n\r\nUsing this test code:\r\n```go\r\nvar (\r\n\tlarge = strings.Repeat(\"the quick brown fox jumped over the lazy dog\", 1024*1024)\r\n\twriterOnly = struct{ io.Writer }{ioutil.Discard}\r\n)\r\n\r\n// WriteString2 is modified from io.WriteString to use an\r\n// intermediate buffer for large strings.\r\nfunc WriteString2(w io.Writer, s string) (n int, err error) {\r\n\tconst chunkSize = 32*1024\r\n\tif sw, ok := w.(interface {WriteString(s string) (n int, err error)}); ok {\r\n\t\treturn sw.WriteString(s)\r\n\t}\r\n\tif len(s) < chunkSize {\r\n\t\treturn w.Write([]byte(s))\r\n\t}\r\n\r\n\tbuf := make([]byte, chunkSize)\r\n\tfor len(s) > 0 {\r\n\t\tcnt := copy(buf, s)\r\n\t\ts = s[cnt:]\r\n\t\tcnt, err = w.Write(buf[:cnt])\r\n\t\tn += cnt\r\n\t\tif err != nil {\r\n\t\t\tbreak\r\n\t\t}\r\n\t}\r\n\treturn n, err\r\n}\r\n\r\nfunc BenchmarkWriteString(b *testing.B) {\r\n\tb.ReportAllocs()\r\n\tfor i := 0; i < b.N; i++ {\r\n\t\tif _, err := io.WriteString(writerOnly, large); err != nil {\r\n\t\t\tb.Error(err)\r\n\t\t}\r\n\t}\r\n}\r\n\r\nfunc BenchmarkWriteString2(b *testing.B) {\r\n\tb.ReportAllocs()\r\n\tfor i := 0; i < b.N; i++ {\r\n\t\tif _, err := WriteString2(writerOnly, large); err != nil {\r\n\t\t\tb.Error(err)\r\n\t\t}\r\n\t}\r\n}\r\n``` \r\n\r\nWe can get the number of bytes allocated to be capped at some maximum:\r\n```\r\nBenchmarkWriteString-4 \t     200\t   6482427 ns/op\t46137405 B/op\t       2 allocs/op\r\nBenchmarkWriteString2-4\t     500\t   2437841 ns/op\t   32784 B/op\t       2 allocs/op\r\n```\r\n\r\nI was pleasantly surprised that the runtime also decreased, but this may be because the small buffer fits entirely in the cache."},"comment":{"url":"https://api.github.com/repos/golang/go/issues/comments/186511005","html_url":"https://github.com/golang/go/issues/13848#issuecomment-186511005","issue_url":"https://api.github.com/repos/golang/go/issues/13848","id":186511005,"user":{"login":"gopherbot","id":8566911,"avatar_url":"https://avatars.githubusercontent.com/u/8566911?v=3","gravatar_id":"","url":"https://api.github.com/users/gopherbot","html_url":"https://github.com/gopherbot","followers_url":"https://api.github.com/users/gopherbot/followers","following_url":"https://api.github.com/users/gopherbot/following{/other_user}","gists_url":"https://api.github.com/users/gopherbot/gists{/gist_id}","starred_url":"https://api.github.com/users/gopherbot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gopherbot/subscriptions","organizations_url":"https://api.github.com/users/gopherbot/orgs","repos_url":"https://api.github.com/users/gopherbot/repos","events_url":"https://api.github.com/users/gopherbot/events{/privacy}","received_events_url":"https://api.github.com/users/gopherbot/received_events","type":"User","site_admin":false},"created_at":"2016-02-20T05:00:45Z","updated_at":"2016-02-20T05:00:45Z","body":"CL https://golang.org/cl/19754 mentions this issue."}}