<p dir="auto">Possible dup of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="91151999" data-permission-text="Title is private" data-url="https://github.com/golang/go/issues/11411" data-hovercard-type="issue" data-hovercard-url="/golang/go/issues/11411/hovercard" href="https://github.com/golang/go/issues/11411">#11411</a> and <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="110457827" data-permission-text="Title is private" data-url="https://github.com/golang/go/issues/12879" data-hovercard-type="issue" data-hovercard-url="/golang/go/issues/12879/hovercard" href="https://github.com/golang/go/issues/12879">#12879</a></p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="runtime: free list of span 0x7ea0832a2d40:
0xc87618ea00 -&gt; 0x80c87618fa40 (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x85ecc0, 0x13)
    /home/matthew/src/golang/go1.5.1/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x7ea0832a2d40, 0x18100000100, 0xc80002a801)
    /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x439b12)
    /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
    /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0xc820023500)
    /home/matthew/src/golang/go1.5.1/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /home/matthew/src/golang/go1.5.1/src/runtime/proc1.go:674

&gt; go version
go version go1.5.1 linux/amd64"><pre class="notranslate"><code class="notranslate">runtime: free list of span 0x7ea0832a2d40:
0xc87618ea00 -&gt; 0x80c87618fa40 (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x85ecc0, 0x13)
    /home/matthew/src/golang/go1.5.1/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x7ea0832a2d40, 0x18100000100, 0xc80002a801)
    /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x439b12)
    /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
    /home/matthew/src/golang/go1.5.1/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0xc820023500)
    /home/matthew/src/golang/go1.5.1/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /home/matthew/src/golang/go1.5.1/src/runtime/proc1.go:674

&gt; go version
go version go1.5.1 linux/amd64
</code></pre></div>
<p dir="auto">The software is a distributed database server. At the time, there were 3 servers running, all connected to each other (and all running on the same machine). All the servers are running the exact same binary. Clients would connect, run some tests, disconnect. I was asleep.</p>
<p dir="auto">From the rest of the stack traces, it looks as though the panic happened 22 minutes after the server was started and 14 minutes after the last client disconnect (test had passed). All 3 connected servers would have been idle at this point.</p>
<p dir="auto">Of the 3 servers, one (server1) failed with the above, one survived (server2) until the morning when I found it, and the other (server3) appears to have failed at exactly the same time with:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="fatal error: C malloc failed

goroutine 77 [running]:
runtime.throw(0x8241f0, 0xf)
    /home/matthew/src/golang/go1.5.1/src/runtime/panic.go:527 +0x90 fp=0xc8e6765408 sp=0xc8e67653f0
runtime.cmalloc(0xa, 0x409617)
    /home/matthew/src/golang/go1.5.1/src/runtime/cgocall.go:148 +0x68 fp=0xc8e6765438 sp=0xc8e6765408
net._Cfunc_CString(0xc8202cb800, 0x9, 0xc8e67654e8)
    ??:0 +0x28 fp=0xc8e67654a8 sp=0xc8e6765438
net.cgoLookupIPCNAME(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/cgo_unix.go:108 +0x13c fp=0xc8e67655d0 sp=0xc8e67654a8
net.cgoLookupIP(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/cgo_unix.go:163 +0x56 fp=0xc8e6765628 sp=0xc8e67655d0
net.lookupIP(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/lookup_unix.go:67 +0x94 fp=0xc8e6765698 sp=0xc8e6765628
net.glob.func15(0x8d0300, 0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/hook.go:10 +0x4d fp=0xc8e67656d8 sp=0xc8e6765698
net.lookupIPMerge.func1(0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/lookup.go:68 +0x71 fp=0xc8e6765758 sp=0xc8e67656d8
internal/singleflight.(*Group).doCall(0xc2c570, 0xc8764ae230, 0xc8202cb800, 0x9, 0xc8e6765950)
    /home/matthew/src/golang/go1.5.1/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc8e6765808 sp=0xc8e6765758
internal/singleflight.(*Group).Do(0xc2c570, 0xc8202cb800, 0x9, 0xc8e6765950, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/internal/singleflight/singleflight.go:63 +0x284 fp=0xc8e6765878 sp=0xc8e6765808
net.lookupIPMerge(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/lookup.go:69 +0x9b fp=0xc8e6765988 sp=0xc8e6765878
net.lookupIPDeadline(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/lookup.go:91 +0xde fp=0xc8e6765bc0 sp=0xc8e6765988
net.internetAddrList(0x821bd8, 0x3, 0xc8202cb800, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/matthew/src/golang/go1.5.1/src/net/ipsock.go:252 +0x6ee fp=0xc8e6765d28 sp=0xc8e6765bc0
net.ResolveTCPAddr(0x821bd8, 0x3, 0xc8202cb800, 0xf, 0x5d8abc, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/tcpsock.go:56 +0x11b fp=0xc8e6765de8 sp=0xc8e6765d28
....my code."><pre class="notranslate"><code class="notranslate">fatal error: C malloc failed

goroutine 77 [running]:
runtime.throw(0x8241f0, 0xf)
    /home/matthew/src/golang/go1.5.1/src/runtime/panic.go:527 +0x90 fp=0xc8e6765408 sp=0xc8e67653f0
runtime.cmalloc(0xa, 0x409617)
    /home/matthew/src/golang/go1.5.1/src/runtime/cgocall.go:148 +0x68 fp=0xc8e6765438 sp=0xc8e6765408
net._Cfunc_CString(0xc8202cb800, 0x9, 0xc8e67654e8)
    ??:0 +0x28 fp=0xc8e67654a8 sp=0xc8e6765438
net.cgoLookupIPCNAME(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/cgo_unix.go:108 +0x13c fp=0xc8e67655d0 sp=0xc8e67654a8
net.cgoLookupIP(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/cgo_unix.go:163 +0x56 fp=0xc8e6765628 sp=0xc8e67655d0
net.lookupIP(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/lookup_unix.go:67 +0x94 fp=0xc8e6765698 sp=0xc8e6765628
net.glob.func15(0x8d0300, 0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/hook.go:10 +0x4d fp=0xc8e67656d8 sp=0xc8e6765698
net.lookupIPMerge.func1(0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/lookup.go:68 +0x71 fp=0xc8e6765758 sp=0xc8e67656d8
internal/singleflight.(*Group).doCall(0xc2c570, 0xc8764ae230, 0xc8202cb800, 0x9, 0xc8e6765950)
    /home/matthew/src/golang/go1.5.1/src/internal/singleflight/singleflight.go:93 +0x2c fp=0xc8e6765808 sp=0xc8e6765758
internal/singleflight.(*Group).Do(0xc2c570, 0xc8202cb800, 0x9, 0xc8e6765950, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/internal/singleflight/singleflight.go:63 +0x284 fp=0xc8e6765878 sp=0xc8e6765808
net.lookupIPMerge(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/lookup.go:69 +0x9b fp=0xc8e6765988 sp=0xc8e6765878
net.lookupIPDeadline(0xc8202cb800, 0x9, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/lookup.go:91 +0xde fp=0xc8e6765bc0 sp=0xc8e6765988
net.internetAddrList(0x821bd8, 0x3, 0xc8202cb800, 0xf, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /home/matthew/src/golang/go1.5.1/src/net/ipsock.go:252 +0x6ee fp=0xc8e6765d28 sp=0xc8e6765bc0
net.ResolveTCPAddr(0x821bd8, 0x3, 0xc8202cb800, 0xf, 0x5d8abc, 0x0, 0x0)
    /home/matthew/src/golang/go1.5.1/src/net/tcpsock.go:56 +0x11b fp=0xc8e6765de8 sp=0xc8e6765d28
....my code.
</code></pre></div>
<p dir="auto">There could have been memory pressure at the time, but I find it unlikely give indications are it's some 14 mins after the last test finished (and passed). Syslog does not show any activity by the kernel OOM process killer.</p>
<p dir="auto">This server (server3) would only have been in this code because it was trying to reconnect to server1 after server1 had failed. So this would have been exactly 5 seconds after server1 had failed. With server1 having failed, I can't believe there really could have been any memory pressure in the system.</p>
<p dir="auto">Thus I think this could be the same issue as <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="110457827" data-permission-text="Title is private" data-url="https://github.com/golang/go/issues/12879" data-hovercard-type="issue" data-hovercard-url="/golang/go/issues/12879/hovercard" href="https://github.com/golang/go/issues/12879">#12879</a> in that the server is idle at the time. I do not know if the two crashes are or could be related at all. I shall attempt to see how reproducible this is.</p>
<p dir="auto">MacOS 10.10.5<br>
go version go1.5 darwin/amd64</p>
<p dir="auto">Possible dup of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="91151999" data-permission-text="Title is private" data-url="https://github.com/golang/go/issues/11411" data-hovercard-type="issue" data-hovercard-url="/golang/go/issues/11411/hovercard" href="https://github.com/golang/go/issues/11411">#11411</a>.</p>
<p dir="auto">Please note the two comments marked with <code class="notranslate">//</code>.</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="$ulimit -n 999; godoc -analysis=&quot;type&quot; -http=:6060
// lots of normal looking output removed. some &quot;could not import&quot; and other errors.
2015/10/07 21:14:43 Constructing SSA form...
2015/10/07 21:14:49 SSA construction complete
2015/10/07 21:14:49 Package &quot;github.com/go-gl/glfw/v3.0/glfw&quot; has error (*errors.errorString) without position: cgo failed: [go tool cgo -objdir /var/folders/5y/w2d42zkx7rsc43v1pnj080tm0000gn/T/github.com_go-gl_glfw_v3.0_glfw_C258288875 -- -I /var/folders/5y/w2d42zkx7rsc43v1pnj080tm0000gn/T/github.com_go-gl_glfw_v3.0_glfw_C258288875 clipboard.go context.go error.go glfw.go input.go monitor.go native_darwin.go time.go util.go window.go]: exit status 1
2015/10/07 21:14:49 Computing implements relation...
2015/10/07 21:14:50 Extracting type info...
2015/10/07 21:14:57 Visit instructions...
2015/10/07 21:14:59 Visit instructions complete
2015/10/07 21:14:59 Type analysis complete.
// several hours pass
runtime: free list of span 0x6431dd0:
0xc941e5a020 -&gt; 0xc941e5a7c0 -&gt; 0xc941e5a490 -&gt; 0xc941e5a980 -&gt; 0x206567616b636170 (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x845d60, 0x13)
    /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x6431dd0, 0x1dd00000100, 0xc80006d601)
    /usr/local/go/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x40c72)
    /usr/local/go/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
    /usr/local/go/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0xc82001d500)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:674

goroutine 3 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82002cf58 sp=0xc82002cf50
runtime.gosweepone(0x0)
    /usr/local/go/src/runtime/mgcsweep.go:110 +0x3d fp=0xc82002cf80 sp=0xc82002cf58
runtime.bgsweep(0xc820016070)
    /usr/local/go/src/runtime/mgcsweep.go:54 +0xb6 fp=0xc82002cfb8 sp=0xc82002cf80
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002cfc0 sp=0xc82002cfb8
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:205 +0x53

goroutine 1 [IO wait, 191 minutes]:
net.runtime_pollWait(0x1917ca0, 0x72, 0xc8200680a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82373c680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82373c680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82373c620, 0x0, 0x1961ef0, 0xc974f459c0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820072250, 0x5f680, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820072250, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc8238463c0, 0x1917d60, 0xc820072250, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc8238463c0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0x7fff5fbffc1b, 0x5, 0x1916cc8, 0xc82006cdb0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1967 +0x8f
main.main()
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/cmd/godoc/main.go:314 +0x18fd

goroutine 17 [syscall, 367 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 1425 [chan receive, 367 minutes]:
golang.org/x/tools/godoc.(*Corpus).refreshMetadataLoop(0xc8201a4000)
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/meta.go:140 +0x3a
created by golang.org/x/tools/godoc.(*Corpus).Init
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/corpus.go:145 +0x8a"><pre class="notranslate"><code class="notranslate">$ulimit -n 999; godoc -analysis="type" -http=:6060
// lots of normal looking output removed. some "could not import" and other errors.
2015/10/07 21:14:43 Constructing SSA form...
2015/10/07 21:14:49 SSA construction complete
2015/10/07 21:14:49 Package "github.com/go-gl/glfw/v3.0/glfw" has error (*errors.errorString) without position: cgo failed: [go tool cgo -objdir /var/folders/5y/w2d42zkx7rsc43v1pnj080tm0000gn/T/github.com_go-gl_glfw_v3.0_glfw_C258288875 -- -I /var/folders/5y/w2d42zkx7rsc43v1pnj080tm0000gn/T/github.com_go-gl_glfw_v3.0_glfw_C258288875 clipboard.go context.go error.go glfw.go input.go monitor.go native_darwin.go time.go util.go window.go]: exit status 1
2015/10/07 21:14:49 Computing implements relation...
2015/10/07 21:14:50 Extracting type info...
2015/10/07 21:14:57 Visit instructions...
2015/10/07 21:14:59 Visit instructions complete
2015/10/07 21:14:59 Type analysis complete.
// several hours pass
runtime: free list of span 0x6431dd0:
0xc941e5a020 -&gt; 0xc941e5a7c0 -&gt; 0xc941e5a490 -&gt; 0xc941e5a980 -&gt; 0x206567616b636170 (BAD)
fatal error: free list corrupted

runtime stack:
runtime.throw(0x845d60, 0x13)
    /usr/local/go/src/runtime/panic.go:527 +0x90
runtime.mSpan_Sweep(0x6431dd0, 0x1dd00000100, 0xc80006d601)
    /usr/local/go/src/runtime/mgcsweep.go:186 +0x800
runtime.sweepone(0x40c72)
    /usr/local/go/src/runtime/mgcsweep.go:97 +0x154
runtime.gosweepone.func1()
    /usr/local/go/src/runtime/mgcsweep.go:109 +0x21
runtime.systemstack(0xc82001d500)
    /usr/local/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /usr/local/go/src/runtime/proc1.go:674

goroutine 3 [running]:
runtime.systemstack_switch()
    /usr/local/go/src/runtime/asm_amd64.s:216 fp=0xc82002cf58 sp=0xc82002cf50
runtime.gosweepone(0x0)
    /usr/local/go/src/runtime/mgcsweep.go:110 +0x3d fp=0xc82002cf80 sp=0xc82002cf58
runtime.bgsweep(0xc820016070)
    /usr/local/go/src/runtime/mgcsweep.go:54 +0xb6 fp=0xc82002cfb8 sp=0xc82002cf80
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82002cfc0 sp=0xc82002cfb8
created by runtime.gcenable
    /usr/local/go/src/runtime/mgc.go:205 +0x53

goroutine 1 [IO wait, 191 minutes]:
net.runtime_pollWait(0x1917ca0, 0x72, 0xc8200680a0)
    /usr/local/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82373c680, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc82373c680, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).accept(0xc82373c620, 0x0, 0x1961ef0, 0xc974f459c0)
    /usr/local/go/src/net/fd_unix.go:408 +0x27c
net.(*TCPListener).AcceptTCP(0xc820072250, 0x5f680, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:254 +0x4d
net/http.tcpKeepAliveListener.Accept(0xc820072250, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:2135 +0x41
net/http.(*Server).Serve(0xc8238463c0, 0x1917d60, 0xc820072250, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1887 +0xb3
net/http.(*Server).ListenAndServe(0xc8238463c0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1877 +0x136
net/http.ListenAndServe(0x7fff5fbffc1b, 0x5, 0x1916cc8, 0xc82006cdb0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1967 +0x8f
main.main()
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/cmd/godoc/main.go:314 +0x18fd

goroutine 17 [syscall, 367 minutes, locked to thread]:
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 1425 [chan receive, 367 minutes]:
golang.org/x/tools/godoc.(*Corpus).refreshMetadataLoop(0xc8201a4000)
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/meta.go:140 +0x3a
created by golang.org/x/tools/godoc.(*Corpus).Init
    /var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/gopath/src/golang.org/x/tools/godoc/corpus.go:145 +0x8a
</code></pre></div>