{"action":"created","issue":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/31641","repository_url":"https://api.github.com/repos/cockroachdb/cockroach","labels_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/31641/labels{/name}","comments_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/31641/comments","events_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/31641/events","html_url":"https://github.com/cockroachdb/cockroach/pull/31641","id":372059653,"node_id":"MDExOlB1bGxSZXF1ZXN0MjI0MzY3MTM1","number":31641,"title":"build: speed up Docker for Mac protobuf compilation","user":{"login":"benesch","id":882976,"node_id":"MDQ6VXNlcjg4Mjk3Ng==","avatar_url":"https://avatars1.githubusercontent.com/u/882976?v=4","gravatar_id":"","url":"https://api.github.com/users/benesch","html_url":"https://github.com/benesch","followers_url":"https://api.github.com/users/benesch/followers","following_url":"https://api.github.com/users/benesch/following{/other_user}","gists_url":"https://api.github.com/users/benesch/gists{/gist_id}","starred_url":"https://api.github.com/users/benesch/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/benesch/subscriptions","organizations_url":"https://api.github.com/users/benesch/orgs","repos_url":"https://api.github.com/users/benesch/repos","events_url":"https://api.github.com/users/benesch/events{/privacy}","received_events_url":"https://api.github.com/users/benesch/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-10-19T18:04:48Z","updated_at":"2018-10-19T18:45:53Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/cockroachdb/cockroach/pulls/31641","html_url":"https://github.com/cockroachdb/cockroach/pull/31641","diff_url":"https://github.com/cockroachdb/cockroach/pull/31641.diff","patch_url":"https://github.com/cockroachdb/cockroach/pull/31641.patch"},"body":"Building Cockroach via Docker for Mac has always been slow because Docker for Mac filesystem sharing mechanism is extremely high latency. But protobuf compilation was unacceptably slow. Here are timings from my MacBook Pro before this change:\r\n\r\n```bash\r\nrm -f bin.docker_amd64/.go_protobuf_sources && build/builder.sh bash -c \"time make bin/.go_protobuf_sources\"\r\n\r\nreal\t11m48.185s\r\nuser\t0m25.080s\r\nsys\t1m35.160s\r\n```\r\n\r\nIt is literally faster to compile all of the C++ and Go dependencies than it is to compile some 60 protobufs. The [repeated sed invocations](https://github.com/cockroachdb/cockroach/blob/7def504233099e4906b04eba2a3bafbe61ead4b4/Makefile#L1058-L1064) were almost entirely to blame. The problem, as it turns out, is that `sed -i` writes out files line-by-line, and every call to `write(2)` involves a high-latency roundtrip to macOS and back. \r\n\r\nThe first commit in this PR upgrades the builder to the latest version of GNU sed (as yet unreleased, so we compile from source), which properly buffers output. This shaves more than 10m off the compile time:\r\n\r\n```\r\nreal\t0m32.263s\r\nuser\t0m7.250s\r\nsys\t0m4.720s\r\n```\r\n\r\nThe next commit shaves another 10s off the compile times by coalescing sed invocations. Sed supports running multiple replacements in the same invocation, which means we only need to make one pass over the files. This brings the compile time down yet another 10s:\r\n\r\n```\r\nreal\t0m21.752s\r\nuser\t0m6.380s\r\nsys\t0m3.550s\r\n```\r\n\r\n20s to compile protobufs is still far from \"ideal\", but at least it's approaching the bounds of \"reasonable.\""},"comment":{"url":"https://api.github.com/repos/cockroachdb/cockroach/issues/comments/431460751","html_url":"https://github.com/cockroachdb/cockroach/pull/31641#issuecomment-431460751","issue_url":"https://api.github.com/repos/cockroachdb/cockroach/issues/31641","id":431460751,"node_id":"MDEyOklzc3VlQ29tbWVudDQzMTQ2MDc1MQ==","user":{"login":"craig[bot]","id":37807150,"node_id":"MDM6Qm90Mzc4MDcxNTA=","avatar_url":"https://avatars0.githubusercontent.com/in/10419?v=4","gravatar_id":"","url":"https://api.github.com/users/craig%5Bbot%5D","html_url":"https://github.com/apps/craig","followers_url":"https://api.github.com/users/craig%5Bbot%5D/followers","following_url":"https://api.github.com/users/craig%5Bbot%5D/following{/other_user}","gists_url":"https://api.github.com/users/craig%5Bbot%5D/gists{/gist_id}","starred_url":"https://api.github.com/users/craig%5Bbot%5D/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/craig%5Bbot%5D/subscriptions","organizations_url":"https://api.github.com/users/craig%5Bbot%5D/orgs","repos_url":"https://api.github.com/users/craig%5Bbot%5D/repos","events_url":"https://api.github.com/users/craig%5Bbot%5D/events{/privacy}","received_events_url":"https://api.github.com/users/craig%5Bbot%5D/received_events","type":"Bot","site_admin":false},"created_at":"2018-10-19T18:45:53Z","updated_at":"2018-10-19T18:45:53Z","author_association":"NONE","body":"# Build succeeded\n  * [GitHub CI (Cockroach)](https://teamcity.cockroachdb.com/viewLog.html?buildId=976357&buildTypeId=Cockroach_UnitTests)"}}
{"actor":{"display_login":"craig"}}