{"action":"closed","issue":{"url":"https://api.github.com/repos/prometheus/pushgateway/issues/96","repository_url":"https://api.github.com/repos/prometheus/pushgateway","labels_url":"https://api.github.com/repos/prometheus/pushgateway/issues/96/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/pushgateway/issues/96/comments","events_url":"https://api.github.com/repos/prometheus/pushgateway/issues/96/events","html_url":"https://github.com/prometheus/pushgateway/issues/96","id":186986275,"number":96,"title":"concurrent map read and map write","user":{"login":"pavelhritonenko","id":2510026,"avatar_url":"https://avatars.githubusercontent.com/u/2510026?v=3","gravatar_id":"","url":"https://api.github.com/users/pavelhritonenko","html_url":"https://github.com/pavelhritonenko","followers_url":"https://api.github.com/users/pavelhritonenko/followers","following_url":"https://api.github.com/users/pavelhritonenko/following{/other_user}","gists_url":"https://api.github.com/users/pavelhritonenko/gists{/gist_id}","starred_url":"https://api.github.com/users/pavelhritonenko/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/pavelhritonenko/subscriptions","organizations_url":"https://api.github.com/users/pavelhritonenko/orgs","repos_url":"https://api.github.com/users/pavelhritonenko/repos","events_url":"https://api.github.com/users/pavelhritonenko/events{/privacy}","received_events_url":"https://api.github.com/users/pavelhritonenko/received_events","type":"User","site_admin":false},"labels":[],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2016-11-03T06:23:43Z","updated_at":"2016-11-03T06:26:35Z","closed_at":"2016-11-03T06:26:35Z","body":"I use pushgateway and once in 2-3 days it fails with panic:\r\n\r\n    fatal error: concurrent map read and map write\r\n    \r\n    goroutine 1195432 [running]:\r\n    runtime.throw(0xa235e0, 0x21)\r\n            /usr/local/go/src/runtime/panic.go:547 +0x97 fp=0xc082b3f550 sp=0xc082b3\r\n    f538\r\n    runtime.mapaccess2(0x8b3a60, 0xc0824e9fb0, 0xc082c57310, 0x20d43d, 0x244000)\r\n            /usr/local/go/src/runtime/hashmap.go:343 +0x61 fp=0xc082b3f598 sp=0xc082\r\n    b3f550\r\n    reflect.mapaccess(0x8b3a60, 0xc0824e9fb0, 0xc082c57310, 0xc0824e9fb0)\r\n            /usr/local/go/src/runtime/hashmap.go:993 +0x3c fp=0xc082b3f5c8 sp=0xc082\r\n    b3f598\r\n    reflect.Value.MapIndex(0x8b3a60, 0xc082c56db8, 0x95, 0x82b200, 0xc082c57310, 0x9\r\n    8, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/reflect/value.go:1041 +0x151 fp=0xc082b3f650 sp=0xc082\r\n    b3f5c8\r\n    encoding/gob.(*Encoder).encodeMap(0xc082082000, 0xc082082038, 0x8b3a60, 0xc082c5\r\n    6db8, 0x95, 0xa936f0, 0xc082003be0, 0x0, 0x0)\r\n            /usr/local/go/src/encoding/gob/encode.go:381 +0x2d2 fp=0xc082b3f760 sp=0\r\n    xc082b3f650\r\n    encoding/gob.encOpFor.func3(0xc0820dcf30, 0xc08304d980, 0x8b3a60, 0xc082c56db8,\r\n    0x95)\r\n            /usr/local/go/src/encoding/gob/encode.go:577 +0x111 fp=0xc082b3f7d0 sp=0\r\n    xc082b3f760\r\n    encoding/gob.(*Encoder).encodeStruct(0xc082082000, 0xc082082038, 0xc082a24d20, 0\r\n    x928560, 0xc082c56db0, 0x99)\r\n            /usr/local/go/src/encoding/gob/encode.go:334 +0x3ed fp=0xc082b3f8b0 sp=0\r\n    xc082b3f7d0\r\n    encoding/gob.encOpFor.func4(0x0, 0xc08304d940, 0x928560, 0xc082c56db0, 0x99)\r\n            /usr/local/go/src/encoding/gob/encode.go:587 +0xba fp=0xc082b3f900 sp=0x\r\n    c082b3f8b0\r\n    encoding/gob.encodeReflectValue(0xc08304d940, 0x928560, 0xc082c56db0, 0x99, 0xc0\r\n    82003c10, 0x0)\r\n            /usr/local/go/src/encoding/gob/encode.go:369 +0x14e fp=0xc082b3f970 sp=0\r\n    xc082b3f900\r\n    encoding/gob.(*Encoder).encodeMap(0xc082082000, 0xc082082038, 0x8b39c0, 0xc0821a\r\n    ccf0, 0x15, 0xa93758, 0xc082003c10, 0x0, 0x0)\r\n            /usr/local/go/src/encoding/gob/encode.go:381 +0x330 fp=0xc082b3fa80 sp=0\r\n    xc082b3f970\r\n    encoding/gob.encOpFor.func3(0xc0826dce70, 0xc08304d900, 0x8b39c0, 0xc0821accf0,\r\n    0x15)\r\n            /usr/local/go/src/encoding/gob/encode.go:577 +0x111 fp=0xc082b3faf0 sp=0\r\n    xc082b3fa80\r\n    encoding/gob.(*Encoder).encodeSingle(0xc082082000, 0xc082082038, 0xc082a24d00, 0\r\n    x8b39c0, 0xc0821accf0, 0x15)\r\n            /usr/local/go/src/encoding/gob/encode.go:307 +0x273 fp=0xc082b3fb88 sp=0\r\n    xc082b3faf0\r\n    encoding/gob.(*Encoder).encode(0xc082082000, 0xc082082038, 0x8b39c0, 0xc0821accf\r\n    0, 0x15, 0xc082394480)\r\n            /usr/local/go/src/encoding/gob/encode.go:709 +0x21d fp=0xc082b3fbe8 sp=0\r\n    xc082b3fb88\r\n    encoding/gob.(*Encoder).EncodeValue(0xc082082000, 0x8b39c0, 0xc0821accf0, 0x15,\r\n    0x0, 0x0)\r\n            /usr/local/go/src/encoding/gob/encoder.go:247 +0x6f4 fp=0xc082b3fd58 sp=\r\n    0xc082b3fbe8\r\n    encoding/gob.(*Encoder).Encode(0xc082082000, 0x8b39c0, 0xc0821accf0, 0x0, 0x0)\r\n            /usr/local/go/src/encoding/gob/encoder.go:174 +0x79 fp=0xc082b3fda8 sp=0\r\n    xc082b3fd58\r\n    github.com/prometheus/pushgateway/storage.(*DiskMetricStore).persist(0xc0821ab3b\r\n    0, 0x0, 0x0)\r\n            /go/src/github.com/prometheus/pushgateway/storage/diskmetricstore.go:249\r\n     +0x224 fp=0xc082b3fea8 sp=0xc082b3fda8\r\n    github.com/prometheus/pushgateway/storage.(*DiskMetricStore).loop.func1.1()\r\n            /go/src/github.com/prometheus/pushgateway/storage/diskmetricstore.go:150\r\n     +0x84 fp=0xc082b3ff90 sp=0xc082b3fea8\r\n    runtime.goexit()\r\n            /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc082b3ff98 sp=0xc08\r\n    2b3ff90\r\n    created by time.goFunc\r\n            /usr/local/go/src/time/sleep.go:129 +0x41\r\n    \r\n    goroutine 1 [IO wait]:\r\n    net.runtime_pollWait(0x379fe8, 0x72, 0xc0821221d8)\r\n            /usr/local/go/src/runtime/netpoll.go:160 +0x67\r\n    net.(*pollDesc).Wait(0xc08248a770, 0x72, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_poll_runtime.go:73 +0x41\r\n    net.(*ioSrv).ExecIO(0xc0821366f8, 0xc08248a660, 0x9aa6b0, 0x8, 0xc082e85800, 0xc\r\n    082d35800, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:183 +0x177\r\n    net.(*netFD).acceptOne(0xc08248a600, 0xc08212c2a0, 0x2, 0x2, 0xc08248a660, 0x0,\r\n    0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:583 +0x26c\r\n    net.(*netFD).accept(0xc08248a600, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:613 +0x173\r\n    net.(*TCPListener).AcceptTCP(0xc082136708, 0xc082be7c10, 0x0, 0x0)\r\n            /usr/local/go/src/net/tcpsock_posix.go:254 +0x54\r\n    net.(*TCPListener).Accept(0xc082136708, 0x0, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/tcpsock_posix.go:264 +0x44\r\n    net/http.(*Server).Serve(0xc08213a380, 0x379068, 0xc082136708, 0x0, 0x0)\r\n            /usr/local/go/src/net/http/server.go:2117 +0x130\r\n    main.main()\r\n            /go/src/github.com/prometheus/pushgateway/main.go:108 +0x1271\r\n    \r\n    goroutine 6 [select]:\r\n    github.com/prometheus/pushgateway/storage.(*DiskMetricStore).loop(0xc0821ab3b0,\r\n    0x45d964b800)\r\n            /go/src/github.com/prometheus/pushgateway/storage/diskmetricstore.go:166\r\n     +0x4a9\r\n    created by github.com/prometheus/pushgateway/storage.NewDiskMetricStore\r\n            /go/src/github.com/prometheus/pushgateway/storage/diskmetricstore.go:79\r\n    +0x5af\r\n    \r\n    goroutine 17 [syscall, 2150 minutes]:\r\n    os/signal.signal_recv(0x0)\r\n            /usr/local/go/src/runtime/sigqueue.go:116 +0x139\r\n    os/signal.loop()\r\n            /usr/local/go/src/os/signal/signal_unix.go:22 +0x1f\r\n    created by os/signal.init.1\r\n            /usr/local/go/src/os/signal/signal_unix.go:28 +0x3e\r\n    \r\n    goroutine 131 [chan receive, 2150 minutes]:\r\n    main.interruptHandler(0x379068, 0xc082136708)\r\n            /go/src/github.com/prometheus/pushgateway/main.go:135 +0x199\r\n    created by main.main\r\n            /go/src/github.com/prometheus/pushgateway/main.go:107 +0x1182\r\n    \r\n    goroutine 149 [IO wait]:\r\n    net.runtime_pollWait(0x379e68, 0x72, 0xc082691400)\r\n            /usr/local/go/src/runtime/netpoll.go:160 +0x67\r\n    net.(*pollDesc).Wait(0xc0824771f0, 0x72, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_poll_runtime.go:73 +0x41\r\n    net.(*ioSrv).ExecIO(0xc0821366f8, 0xc0824770e0, 0x9a1ac0, 0x7, 0xa93fc0, 0x23c,\r\n    0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:183 +0x177\r\n    net.(*netFD).Read(0xc082477080, 0xc082839000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:482 +0x17e\r\n    net.(*conn).Read(0xc082058040, 0xc082839000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/net.go:172 +0xeb\r\n    net/http.(*connReader).Read(0xc082b50240, 0xc082839000, 0x1000, 0x1000, 0x0, 0x0\r\n    , 0x0)\r\n            /usr/local/go/src/net/http/server.go:526 +0x19d\r\n    bufio.(*Reader).fill(0xc082ac0300)\r\n            /usr/local/go/src/bufio/bufio.go:97 +0x1f0\r\n    bufio.(*Reader).Peek(0xc082ac0300, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/bufio/bufio.go:132 +0xd3\r\n    net/http.(*conn).readRequest(0xc0820a0280, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/http/server.go:702 +0x2ed\r\n    net/http.(*conn).serve(0xc0820a0280)\r\n            /usr/local/go/src/net/http/server.go:1425 +0x94e\r\n    created by net/http.(*Server).Serve\r\n            /usr/local/go/src/net/http/server.go:2137 +0x455\r\n    \r\n    goroutine 296 [IO wait]:\r\n    net.runtime_pollWait(0x379f28, 0x72, 0xc082f890e0)\r\n            /usr/local/go/src/runtime/netpoll.go:160 +0x67\r\n    net.(*pollDesc).Wait(0xc08249aa70, 0x72, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_poll_runtime.go:73 +0x41\r\n    net.(*ioSrv).ExecIO(0xc0821366f8, 0xc08249a960, 0x9a1ac0, 0x7, 0xa93fc0, 0x460,\r\n    0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:183 +0x177\r\n    net.(*netFD).Read(0xc08249a900, 0xc08212b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:482 +0x17e\r\n    net.(*conn).Read(0xc082f707a8, 0xc08212b000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/net.go:172 +0xeb\r\n    net/http.(*connReader).Read(0xc0821322c0, 0xc08212b000, 0x1000, 0x1000, 0x0, 0x0\r\n    , 0x0)\r\n            /usr/local/go/src/net/http/server.go:526 +0x19d\r\n    bufio.(*Reader).fill(0xc082cfe180)\r\n            /usr/local/go/src/bufio/bufio.go:97 +0x1f0\r\n    bufio.(*Reader).Peek(0xc082cfe180, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/bufio/bufio.go:132 +0xd3\r\n    net/http.(*conn).readRequest(0xc082bf2180, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/http/server.go:702 +0x2ed\r\n    net/http.(*conn).serve(0xc082bf2180)\r\n            /usr/local/go/src/net/http/server.go:1425 +0x94e\r\n    created by net/http.(*Server).Serve\r\n            /usr/local/go/src/net/http/server.go:2137 +0x455\r\n    \r\n    goroutine 1071460 [IO wait]:\r\n    net.runtime_pollWait(0x379c28, 0x72, 0xc08277b5a0)\r\n            /usr/local/go/src/runtime/netpoll.go:160 +0x67\r\n    net.(*pollDesc).Wait(0xc082d35af0, 0x72, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_poll_runtime.go:73 +0x41\r\n    net.(*ioSrv).ExecIO(0xc0821366f8, 0xc082d359e0, 0x9a1ac0, 0x7, 0xa93fc0, 0x768,\r\n    0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:183 +0x177\r\n    net.(*netFD).Read(0xc082d35980, 0xc083108000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:482 +0x17e\r\n    net.(*conn).Read(0xc082f70000, 0xc083108000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/net.go:172 +0xeb\r\n    net/http.(*connReader).Read(0xc082d1e080, 0xc083108000, 0x1000, 0x1000, 0x0, 0x0\r\n    , 0x0)\r\n            /usr/local/go/src/net/http/server.go:526 +0x19d\r\n    bufio.(*Reader).fill(0xc082390000)\r\n            /usr/local/go/src/bufio/bufio.go:97 +0x1f0\r\n    bufio.(*Reader).Peek(0xc082390000, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/bufio/bufio.go:132 +0xd3\r\n    net/http.(*conn).readRequest(0xc0820a0080, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/http/server.go:702 +0x2ed\r\n    net/http.(*conn).serve(0xc0820a0080)\r\n            /usr/local/go/src/net/http/server.go:1425 +0x94e\r\n    created by net/http.(*Server).Serve\r\n            /usr/local/go/src/net/http/server.go:2137 +0x455\r\n    \r\n    goroutine 866710 [IO wait]:\r\n    net.runtime_pollWait(0x379da8, 0x72, 0xc082976de8)\r\n            /usr/local/go/src/runtime/netpoll.go:160 +0x67\r\n    net.(*pollDesc).Wait(0xc0839677f0, 0x72, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_poll_runtime.go:73 +0x41\r\n    net.(*ioSrv).ExecIO(0xc0821366f8, 0xc0839676e0, 0x9a1ac0, 0x7, 0xa93fc0, 0x784,\r\n    0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:183 +0x177\r\n    net.(*netFD).Read(0xc083967680, 0xc082bc8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/fd_windows.go:482 +0x17e\r\n    net.(*conn).Read(0xc082bc4008, 0xc082bc8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/net.go:172 +0xeb\r\n    net/http.(*connReader).Read(0xc08260c060, 0xc082bc8000, 0x1000, 0x1000, 0x0, 0x0\r\n    , 0x0)\r\n            /usr/local/go/src/net/http/server.go:526 +0x19d\r\n    bufio.(*Reader).fill(0xc083958000)\r\n            /usr/local/go/src/bufio/bufio.go:97 +0x1f0\r\n    bufio.(*Reader).Peek(0xc083958000, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/bufio/bufio.go:132 +0xd3\r\n    net/http.(*conn).readRequest(0xc082bf2080, 0x0, 0x0, 0x0)\r\n            /usr/local/go/src/net/http/server.go:702 +0x2ed\r\n    net/http.(*conn).serve(0xc082bf2080)\r\n            /usr/local/go/src/net/http/server.go:1425 +0x94e\r\n    created by net/http.(*Server).Serve\r\n            /usr/local/go/src/net/http/server.go:2137 +0x455"}}
{"actor":{"display_login":"pavelhritonenko"}}