{"action":"created","issue":{"url":"https://api.github.com/repos/grafana/grafana/issues/3969","repository_url":"https://api.github.com/repos/grafana/grafana","labels_url":"https://api.github.com/repos/grafana/grafana/issues/3969/labels{/name}","comments_url":"https://api.github.com/repos/grafana/grafana/issues/3969/comments","events_url":"https://api.github.com/repos/grafana/grafana/issues/3969/events","html_url":"https://github.com/grafana/grafana/issues/3969","id":132428603,"number":3969,"title":"Macaron] PANIC: runtime error: invalid memory address or nil pointer dereference","user":{"login":"miekg","id":353371,"avatar_url":"https://avatars.githubusercontent.com/u/353371?v=3","gravatar_id":"","url":"https://api.github.com/users/miekg","html_url":"https://github.com/miekg","followers_url":"https://api.github.com/users/miekg/followers","following_url":"https://api.github.com/users/miekg/following{/other_user}","gists_url":"https://api.github.com/users/miekg/gists{/gist_id}","starred_url":"https://api.github.com/users/miekg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/miekg/subscriptions","organizations_url":"https://api.github.com/users/miekg/orgs","repos_url":"https://api.github.com/users/miekg/repos","events_url":"https://api.github.com/users/miekg/events{/privacy}","received_events_url":"https://api.github.com/users/miekg/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":5,"created_at":"2016-02-09T14:10:26Z","updated_at":"2016-02-09T16:35:49Z","closed_at":null,"body":"We're seeing a panic when grafana tries to access a mysql database. Just before in the logs we see:\r\n```\r\nMySQL] 2016/02/09 12:39:05 packets.go:32: read tcp 172.29.0.2:50298->xx.xx.xx.xx:3306: read: connection timed out\r\n[MySQL] 2016/02/09 12:39:05 packets.go:118: write tcp 172.29.0.2:50298->xx.xx.xx.xx:3306: write: broken pipe\r\n```\r\nWhich is the lead up to:\r\n```\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/runtime/panic.go:423 (0x4c0fa9)\r\nFeb 09 12:39:05 bla docker[28704]: gopanic: reflectcall(nil, unsafe.Pointer(d.fn), deferArgs(d), uint32(d.siz), uint32(d.siz))\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/runtime/panic.go:42 (0x4bf669)\r\nFeb 09 12:39:05 bla docker[28704]: panicmem: panic(memoryError)\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/runtime/sigpanic_unix.go:24 (0x4d601a)\r\nFeb 09 12:39:05 bla docker[28704]: sigpanic: panicmem()\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/Godeps/_workspace/src/github.com/go-xorm/xorm/session.go:1008 (0x7f4f72)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/pkg/services/sqlstore/user.go:294 (0x580692)\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/runtime/asm_amd64.s:437 (0x4f12ae)\r\nFeb 09 12:39:05 bla docker[28704]: call32: CALLFN(·call32, 32)\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/reflect/value.go:432 (0x5b02ca)\r\nFeb 09 12:39:05 bla docker[28704]: Value.call: call(frametype, fn, args, uint32(frametype.size), uint32(retOffset))\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/reflect/value.go:300 (0x5aef91)\r\nFeb 09 12:39:05 bla docker[28704]: Value.Call: return v.call(\"Call\", in)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/pkg/bus/bus.go:48 (0x67d13e)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/pkg/bus/bus.go:118 (0x67e23b)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/pkg/middleware/auth_proxy.go:21 (0x654583)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/pkg/middleware/middleware.go:45 (0x65946b)\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/runtime/asm_amd64.s:437 (0x4f12ae)\r\nFeb 09 12:39:05 bla docker[28704]: call32: CALLFN(·call32, 32)\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/reflect/value.go:432 (0x5b02ca)\r\nFeb 09 12:39:05 bla docker[28704]: Value.call: call(frametype, fn, args, uint32(frametype.size), uint32(retOffset))\r\nFeb 09 12:39:05 bla docker[28704]: /usr/local/go/src/reflect/value.go:300 (0x5aef91)\r\nFeb 09 12:39:05 bla docker[28704]: Value.Call: return v.call(\"Call\", in)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/Godeps/_workspace/src/github.com/Unknwon/macaron/inject/inject.go:102 (0x8f9a52)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/Godeps/_workspace/src/github.com/Unknwon/macaron/context.go:113 (0x610427)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/Godeps/_workspace/src/github.com/Unknwon/macaron/context.go:104 (0x61030c)\r\nFeb 09 12:39:05 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/Godeps/_workspace/src/github.com/Unknwon/macaron/recovery.go:161 (0x623662)\r\nFeb 09 12:39:06 bla docker[28704]: /usr/local/go/src/runtime/asm_amd64.s:437 (0x4f12ae)\r\nFeb 09 12:39:06 bla docker[28704]: call32: CALLFN(·call32, 32)\r\nFeb 09 12:39:06 bla docker[28704]: /usr/local/go/src/reflect/value.go:432 (0x5b02ca)\r\nFeb 09 12:39:06 bla docker[28704]: Value.call: call(frametype, fn, args, uint32(frametype.size), uint32(retOffset))\r\nFeb 09 12:39:06 bla docker[28704]: /usr/local/go/src/reflect/value.go:300 (0x5aef91)\r\nFeb 09 12:39:06 bla docker[28704]: Value.Call: return v.call(\"Call\", in)\r\nFeb 09 12:39:06 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/Godeps/_workspace/src/github.com/Unknwon/macaron/inject/inject.go:102 (0x8f9a52)\r\nFeb 09 12:39:06 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/Godeps/_workspace/src/github.com/Unknwon/macaron/context.go:113 (0x610427)\r\nFeb 09 12:39:06 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/Godeps/_workspace/src/github.com/Unknwon/macaron/context.go:104 (0x61030c)\r\nFeb 09 12:39:06 bla docker[28704]: /grafana_root/src/github.com/grafana/grafana/pkg/middleware/logger.go:38 (0x65886a)\r\nFeb 09 12:39:06 bla docker[28704]: /usr/local/go/src/runtime/asm_amd64.s:437 (0x4f12ae)\r\nFeb 09 12:39:06 bla docker[28704]: call32: CALLFN(·call32, 32)\r\nFeb 09 12:39:06 bla docker[28704]: /usr/local/go/src/reflect/value.go:432 (0x5b02ca)\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/grafana/grafana/issues/comments/181946259","html_url":"https://github.com/grafana/grafana/issues/3969#issuecomment-181946259","issue_url":"https://api.github.com/repos/grafana/grafana/issues/3969","id":181946259,"user":{"login":"miekg","id":353371,"avatar_url":"https://avatars.githubusercontent.com/u/353371?v=3","gravatar_id":"","url":"https://api.github.com/users/miekg","html_url":"https://github.com/miekg","followers_url":"https://api.github.com/users/miekg/followers","following_url":"https://api.github.com/users/miekg/following{/other_user}","gists_url":"https://api.github.com/users/miekg/gists{/gist_id}","starred_url":"https://api.github.com/users/miekg/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/miekg/subscriptions","organizations_url":"https://api.github.com/users/miekg/orgs","repos_url":"https://api.github.com/users/miekg/repos","events_url":"https://api.github.com/users/miekg/events{/privacy}","received_events_url":"https://api.github.com/users/miekg/received_events","type":"User","site_admin":false},"created_at":"2016-02-09T16:35:49Z","updated_at":"2016-02-09T16:35:49Z","body":"We see it in prod, but it's hard to replicate that configuration to get a self contained test. Upgrading the version the go-sql driver might certainly help. One thing I can't seem to find what the default IdleConnection count is.\r\n\r\nWe're playing with tcp level keep-alives for the time being (and checking if that helps)"}}