{"action":"created","issue":{"url":"https://api.github.com/repos/golang/go/issues/15658","repository_url":"https://api.github.com/repos/golang/go","labels_url":"https://api.github.com/repos/golang/go/issues/15658/labels{/name}","comments_url":"https://api.github.com/repos/golang/go/issues/15658/comments","events_url":"https://api.github.com/repos/golang/go/issues/15658/events","html_url":"https://github.com/golang/go/issues/15658","id":154383450,"number":15658,"title":"runtime: FreeBSD memory corruption involving finalizers","user":{"login":"derekmarcotte","id":12298521,"avatar_url":"https://avatars.githubusercontent.com/u/12298521?v=3","gravatar_id":"","url":"https://api.github.com/users/derekmarcotte","html_url":"https://github.com/derekmarcotte","followers_url":"https://api.github.com/users/derekmarcotte/followers","following_url":"https://api.github.com/users/derekmarcotte/following{/other_user}","gists_url":"https://api.github.com/users/derekmarcotte/gists{/gist_id}","starred_url":"https://api.github.com/users/derekmarcotte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekmarcotte/subscriptions","organizations_url":"https://api.github.com/users/derekmarcotte/orgs","repos_url":"https://api.github.com/users/derekmarcotte/repos","events_url":"https://api.github.com/users/derekmarcotte/events{/privacy}","received_events_url":"https://api.github.com/users/derekmarcotte/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/golang/go/labels/NeedsInvestigation","name":"NeedsInvestigation","color":"ededed"},{"url":"https://api.github.com/repos/golang/go/labels/OS-FreeBSD","name":"OS-FreeBSD","color":"ededed"}],"state":"open","locked":false,"assignee":{"login":"aclements","id":2688315,"avatar_url":"https://avatars.githubusercontent.com/u/2688315?v=3","gravatar_id":"","url":"https://api.github.com/users/aclements","html_url":"https://github.com/aclements","followers_url":"https://api.github.com/users/aclements/followers","following_url":"https://api.github.com/users/aclements/following{/other_user}","gists_url":"https://api.github.com/users/aclements/gists{/gist_id}","starred_url":"https://api.github.com/users/aclements/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aclements/subscriptions","organizations_url":"https://api.github.com/users/aclements/orgs","repos_url":"https://api.github.com/users/aclements/repos","events_url":"https://api.github.com/users/aclements/events{/privacy}","received_events_url":"https://api.github.com/users/aclements/received_events","type":"User","site_admin":false},"assignees":[{"login":"aclements","id":2688315,"avatar_url":"https://avatars.githubusercontent.com/u/2688315?v=3","gravatar_id":"","url":"https://api.github.com/users/aclements","html_url":"https://github.com/aclements","followers_url":"https://api.github.com/users/aclements/followers","following_url":"https://api.github.com/users/aclements/following{/other_user}","gists_url":"https://api.github.com/users/aclements/gists{/gist_id}","starred_url":"https://api.github.com/users/aclements/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/aclements/subscriptions","organizations_url":"https://api.github.com/users/aclements/orgs","repos_url":"https://api.github.com/users/aclements/repos","events_url":"https://api.github.com/users/aclements/events{/privacy}","received_events_url":"https://api.github.com/users/aclements/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/golang/go/milestones/42","html_url":"https://github.com/golang/go/milestones/Go1.8Maybe","labels_url":"https://api.github.com/repos/golang/go/milestones/42/labels","id":1748849,"number":42,"title":"Go1.8Maybe","description":"","creator":{"login":"bradfitz","id":2621,"avatar_url":"https://avatars.githubusercontent.com/u/2621?v=3","gravatar_id":"","url":"https://api.github.com/users/bradfitz","html_url":"https://github.com/bradfitz","followers_url":"https://api.github.com/users/bradfitz/followers","following_url":"https://api.github.com/users/bradfitz/following{/other_user}","gists_url":"https://api.github.com/users/bradfitz/gists{/gist_id}","starred_url":"https://api.github.com/users/bradfitz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bradfitz/subscriptions","organizations_url":"https://api.github.com/users/bradfitz/orgs","repos_url":"https://api.github.com/users/bradfitz/repos","events_url":"https://api.github.com/users/bradfitz/events{/privacy}","received_events_url":"https://api.github.com/users/bradfitz/received_events","type":"User","site_admin":false},"open_issues":159,"closed_issues":80,"state":"open","created_at":"2016-05-04T23:40:26Z","updated_at":"2016-10-20T00:49:12Z","due_on":"2017-01-31T00:00:00Z","closed_at":null},"comments":46,"created_at":"2016-05-12T02:24:56Z","updated_at":"2016-10-20T12:48:20Z","closed_at":null,"body":"Please answer these questions before submitting your issue. Thanks!\r\n\r\n1. What version of Go are you using (`go version`)?\r\n\r\n`go version go1.6.2 freebsd/amd64`\r\n\r\n2. What operating system and processor architecture are you using (`go env`)?\r\n\r\n```bash\r\nGOARCH=\"amd64\"\r\nGOBIN=\"\"\r\nGOEXE=\"\"\r\nGOHOSTARCH=\"amd64\"\r\nGOHOSTOS=\"freebsd\"\r\nGOOS=\"freebsd\"\r\nGOPATH=\"\"\r\nGORACE=\"\"\r\nGOROOT=\"/usr/local/go\"\r\nGOTOOLDIR=\"/usr/local/go/pkg/tool/freebsd_amd64\"\r\nGO15VENDOREXPERIMENT=\"1\"\r\nCC=\"cc\"\r\nGOGCCFLAGS=\"-fPIC -m64 -pthread -fmessage-length=0\"\r\nCXX=\"clang++\"\r\n```\r\n\r\n3. What did you do?\r\n\r\n```go\r\npackage main\r\n\r\n/* stdlib includes */\r\nimport (\r\n        \"fmt\"\r\n        \"os/exec\"\r\n)\r\n\r\nfunc run(done chan struct{}) {\r\n        cmd := exec.Command(\"true\")\r\n        if err := cmd.Start(); err != nil {\r\n                goto finished\r\n        }\r\n\r\n        cmd.Wait()\r\n\r\nfinished:\r\n        done <- struct{}{}\r\n        return\r\n}\r\n\r\nfunc main() {\r\n        fmt.Println(\"Starting a bunch of goroutines...\")\r\n\r\n        // 8 & 16 are arbitrary\r\n        done := make(chan struct{}, 16)\r\n\r\n        for i := 0; i < 8; i++ {\r\n                go run(done)\r\n        }\r\n\r\n        for {\r\n                select {\r\n                case <-done:\r\n                        go run(done)\r\n                }\r\n        }\r\n}\r\n```\r\n\r\n4. What did you expect to see?\r\n\r\nI expect this strange program to spawn instances of /bin/true in parallel, until I stop it.\r\n\r\n5. What did you see instead?\r\n\r\nVarious types of panics caused by what looks to be corruption within the finalizer lists, caused by what I am assuming is based on race conditions.  These panics can happen as quickly as 2 minutes, or much longer.  10 minutes seems a good round number.\r\n\r\nOccasionally addspecial gets stuck in an infinite loop holding the lock, and the process wedges.  This is illustrated in log 1462933614, with x.next pointing to x.  This appears to be corruption of that data structure.  I have seen processes in this state run for 22 hours.\r\n\r\nI understand there is some trepidation expressed in issue #11485 around the locking of the data structures involved.\r\n\r\nHere are some sample messages:\r\n* `fatal error: runtime.SetFinalizer: finalizer already set`\r\n* `runtime: nonempty check fails b.log[0]= 0 b.log[1]= 0 b.log[2]= 0 b.log[3]= 0\r\nfatal error: workbuf is empty`\r\n\r\n[1462926841-SetFinalizer-ex1.txt](https://github.com/golang/go/files/260514/1462926841-SetFinalizer-ex1.txt)\r\n[1462926969-SetFinalizer-ex2.txt](https://github.com/golang/go/files/260516/1462926969-SetFinalizer-ex2.txt)\r\n[1462933295-nonempty-check-fails.txt](https://github.com/golang/go/files/260517/1462933295-nonempty-check-fails.txt)\r\n[1462933614-wedged.txt](https://github.com/golang/go/files/260515/1462933614-wedged.txt)\r\n\r\nThis was run on an 8-core processor, and a 4-core 8-thread processor with ECC RAM, similar results.\r\n\r\nAdditionally, while this example is an extreme, it also represents the core functionality of a project I've been working on part-time for many months.  I'm happy to provide any further assistance diagnosing this issue - I'm very invested!\r\n\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/golang/go/issues/comments/255096026","html_url":"https://github.com/golang/go/issues/15658#issuecomment-255096026","issue_url":"https://api.github.com/repos/golang/go/issues/15658","id":255096026,"user":{"login":"derekmarcotte","id":12298521,"avatar_url":"https://avatars.githubusercontent.com/u/12298521?v=3","gravatar_id":"","url":"https://api.github.com/users/derekmarcotte","html_url":"https://github.com/derekmarcotte","followers_url":"https://api.github.com/users/derekmarcotte/followers","following_url":"https://api.github.com/users/derekmarcotte/following{/other_user}","gists_url":"https://api.github.com/users/derekmarcotte/gists{/gist_id}","starred_url":"https://api.github.com/users/derekmarcotte/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/derekmarcotte/subscriptions","organizations_url":"https://api.github.com/users/derekmarcotte/orgs","repos_url":"https://api.github.com/users/derekmarcotte/repos","events_url":"https://api.github.com/users/derekmarcotte/events{/privacy}","received_events_url":"https://api.github.com/users/derekmarcotte/received_events","type":"User","site_admin":false},"created_at":"2016-10-20T12:48:20Z","updated_at":"2016-10-20T12:48:20Z","body":"Thanks very much for taking a look.  I'm very grateful for your time.\r\n\r\nAre you able to elaborate more fully about what you mean by \"Unlikely we are going to solve this.\"  Are you referring to the issue in general, or the DragonFly kernel crash?  Is the \"we\" The Go Team?  \r\n\r\nI believe the implication of a kernel problem in FreeBSD, because of a panic in DragonFly, simply doesn't follow.  I can give a lot more detail about this, but I fear it's redundant, unless requested.\r\n\r\nFor what it's worth, this issue is reliably reproducible enough that I've been able to bisect down to a commit that makes sense (https://github.com/golang/go/commit/e6d8bfe218a5f387d7aceddcaee5067a59181838), in my first go.  Additionally, the commits leading up to the implicated commit in the bisect are documentation-only, and don't reproduce the behaviour.  I have very high confidence in the bisect.\r\n\r\nI haven't yet published this work yet, as I've been asking some new questions based on the bisect.  I was hoping to be able to speak intelligently about this when I reported results, but **I feel now the pressure has increased.**  I've been making progress slowly, but I am making progress (this isn't my daytime gig).  I will post a repo later today with the work and the bisec log, so that others can reference it.\r\n\r\nHere's my thoughts:\r\n\r\n- at first glance, the implicated commit seems to change how far down in the exec call tree the software gets in the /doesnotexist case\r\n-- that raised the question, did it ever work?\r\n--- starting with the tag go1 the /bin/true case has goruntime panic'd, and the issue therefore has always existed\r\n\r\nNew lines of thought are:\r\n- This suggests that the conventions around how something is actually being exec'd are incorrect, for at least the FreeBSD case. \r\n- Presumably, other syscalls work fine, so the syscall interface is likely correct. \r\n- Does darwin exhibit the problem?  (I will test, although I don't have access to an 8-thread darwin machine), this shares a lot of go runtime code with FreeBSD in the exec codepath\r\n- How far down the exec call tree does this commit push the test, in comparison with the older code\r\n\r\nI plan to investigate these questions, but again, this isn't my daytime gig.\r\n\r\nAs I mentioned earlier, I'm very happy to work with others on this.  \r\n\r\nI'd be very interested in more details in the environment you were unable to reproduce in.  Specifically, how many CPUs?  Is the gomote a virtualized environment, or do you have direct access to hardware somewhere?  Perhaps we can compare build artefacts?  i.e. does your artefact crash in my environment, and vice versa\r\n\r\nThanks again for taking a look."}}
{"actor":{"display_login":"derekmarcotte"}}