{"action":"closed","issue":{"url":"https://api.github.com/repos/grafana/grafana/issues/11814","repository_url":"https://api.github.com/repos/grafana/grafana","labels_url":"https://api.github.com/repos/grafana/grafana/issues/11814/labels{/name}","comments_url":"https://api.github.com/repos/grafana/grafana/issues/11814/comments","events_url":"https://api.github.com/repos/grafana/grafana/issues/11814/events","html_url":"https://github.com/grafana/grafana/issues/11814","id":319935012,"number":11814,"title":"Grafana 5.1.0 Docker Container crashes during datasource provisioning","user":{"login":"sharenz","id":9605611,"avatar_url":"https://avatars2.githubusercontent.com/u/9605611?v=4","gravatar_id":"","url":"https://api.github.com/users/sharenz","html_url":"https://github.com/sharenz","followers_url":"https://api.github.com/users/sharenz/followers","following_url":"https://api.github.com/users/sharenz/following{/other_user}","gists_url":"https://api.github.com/users/sharenz/gists{/gist_id}","starred_url":"https://api.github.com/users/sharenz/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sharenz/subscriptions","organizations_url":"https://api.github.com/users/sharenz/orgs","repos_url":"https://api.github.com/users/sharenz/repos","events_url":"https://api.github.com/users/sharenz/events{/privacy}","received_events_url":"https://api.github.com/users/sharenz/received_events","type":"User","site_admin":false},"labels":[{"id":120793823,"url":"https://api.github.com/repos/grafana/grafana/labels/state:%20needs%20more%20detail","name":"state: needs more detail","color":"1d76db","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-05-03T14:04:23Z","updated_at":"2018-05-03T15:12:41Z","closed_at":"2018-05-03T15:12:41Z","author_association":"NONE","body":"\r\n\r\nPlease include this information:\r\n### What Grafana version are you using?\r\nmsg=\"Starting Grafana\" logger=server version=5.1.0 commit=844bdc53a compiled=2018-04-26T17:09:23+0200\r\n### What datasource are you using?\r\nPrometheus\r\n### What OS are you running grafana on?\r\nhttps://hub.docker.com/r/grafana/grafana/ latest on Debian 9\r\n### What did you do?\r\nStarting Grafana Container with provisioned datasource Prometheus. Note that the system is under quite heavy load and also other containers are starting simultaneously. There are no problems with the other containers.\r\n\r\nThis problem does not occur everytime.\r\n\r\nprometheus.yaml:\r\n```\r\napiVersion: 1\r\n\r\ndatasources:\r\n- name: Prometheus\r\n  type: prometheus\r\n  access: proxy\r\n  url: http://prometheus:9090\r\n  version: 1\r\n  editable: false\r\n```\r\n\r\n### What was the expected result?\r\nGrafana starts\r\n\r\n### What happened instead?\r\nGrafana crashes some times with the following error message:\r\n\r\n```\r\ngrafana_1               | t=2018-05-03T14:16:55+0200 lvl=info msg=\"inserting datasource from configuration \" logger=provisioning.datasources name=Prometheus\r\ngrafana_1               | fatal error: concurrent map read and map write\r\ngrafana_1               | \r\ngrafana_1               | goroutine 848 [running]:\r\ngrafana_1               | runtime.throw(0x13fc946, 0x21)\r\ngrafana_1               | \t/usr/local/go/src/runtime/panic.go:619 +0x81 fp=0xc4201bab48 sp=0xc4201bab28 pc=0x432511\r\ngrafana_1               | runtime.mapaccess1_faststr(0x11ded40, 0xc4201701b0, 0x10b25ba, 0x20, 0x417769)\r\ngrafana_1               | \t/usr/local/go/src/runtime/hashmap_fast.go:181 +0x421 fp=0xc4201babb8 sp=0xc4201bab48 pc=0x411e81\r\ngrafana_1               | github.com/grafana/grafana/pkg/bus.(*InProcBus).Dispatch(0xc420170180, 0x10dff40, 0xc4204b6000, 0xc4201bac01, 0xc4204b6000)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/bus/bus.go:65 +0xbd fp=0xc4201bac78 sp=0xc4201babb8 pc=0x7bbf4d\r\ngrafana_1               | github.com/grafana/grafana/pkg/bus.Dispatch(0x10dff40, 0xc4204b6000, 0xc4201bad10, 0x4a8614)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/bus/bus.go:154 +0x49 fp=0xc4201bacb0 sp=0xc4201bac78 pc=0x7bccd9\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/dashboards.(*dashboardServiceImpl).GetProvisionedDashboardData(0xc42021a740, 0xc4206a6280, 0x7, 0x40800c, 0xc4201bad68, 0xc4201bad70, 0x4a7f01, 0xc420218a80)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/dashboards/dashboard_service.go:52 +0x6f fp=0xc4201bace8 sp=0xc4201bacb0 pc=0x8cfb7f\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/provisioning/dashboards.getProvisionedDashboardByPath(0x1515ec0, 0xc42021a740, 0xc4206a6280, 0x7, 0x150b580, 0xc42021bc10, 0x0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/provisioning/dashboards/file_reader.go:188 +0x52 fp=0xc4201bad60 sp=0xc4201bace8 pc=0xdc86a2\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/provisioning/dashboards.(*fileReader).startWalkingDisk(0xc42018c7c0, 0xc4205be768, 0x40b800)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/provisioning/dashboards/file_reader.go:97 +0x137 fp=0xc4201baec8 sp=0xc4201bad60 pc=0xdc72e7\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/provisioning/dashboards.(*fileReader).ReadAndListen(0xc42018c7c0, 0x1518200, 0xc4201cb880, 0x0, 0xc4202d1320)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/provisioning/dashboards/file_reader.go:59 +0x40 fp=0xc4201bafb8 sp=0xc4201baec8 pc=0xdc6f50\r\ngrafana_1               | runtime.goexit()\r\ngrafana_1               | \t/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc4201bafc0 sp=0xc4201bafb8 pc=0x461f51\r\ngrafana_1               | created by github.com/grafana/grafana/pkg/services/provisioning/dashboards.(*DashboardProvisioner).Provision\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/provisioning/dashboards/dashboard.go:42 +0x9a\r\ngrafana_1               | \r\ngrafana_1               | goroutine 1 [chan receive]:\r\ngrafana_1               | main.main()\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/main.go:96 +0x2ce\r\ngrafana_1               | \r\ngrafana_1               | goroutine 5 [syscall]:\r\ngrafana_1               | os/signal.signal_recv(0x0)\r\ngrafana_1               | \t/usr/local/go/src/runtime/sigqueue.go:139 +0xa6\r\ngrafana_1               | os/signal.loop()\r\ngrafana_1               | \t/usr/local/go/src/os/signal/signal_unix.go:22 +0x22\r\ngrafana_1               | created by os/signal.init.0\r\ngrafana_1               | \t/usr/local/go/src/os/signal/signal_unix.go:28 +0x41\r\ngrafana_1               | \r\ngrafana_1               | goroutine 34 [select]:\r\ngrafana_1               | main.listenToSystemSignals(0xc4201cb9c0, 0xc420040240)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/main.go:110 +0x21a\r\ngrafana_1               | created by main.main\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/main.go:84 +0x28b\r\ngrafana_1               | \r\ngrafana_1               | goroutine 35 [runnable]:\r\ngrafana_1               | github.com/grafana/grafana/pkg/bus.AddCtxHandler(0x13d2230, 0x7, 0x11b41a0, 0x144fb38)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/bus/bus.go:140 +0x58\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/notifications.Init(0xc4201cb8c0, 0xc4202d1ce0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/notifications/notifications.go:34 +0x131\r\ngrafana_1               | main.(*GrafanaServerImpl).Start(0xc4201cb9c0, 0x0, 0x0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/server.go:92 +0x3ca\r\ngrafana_1               | main.main.func2(0xc4201cb9c0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/main.go:88 +0x2f\r\ngrafana_1               | created by main.main\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/main.go:86 +0x2ad\r\ngrafana_1               | \r\ngrafana_1               | goroutine 50 [select, locked to thread]:\r\ngrafana_1               | runtime.gopark(0x1451f38, 0x0, 0x13d018b, 0x6, 0x18, 0x1)\r\ngrafana_1               | \t/usr/local/go/src/runtime/proc.go:291 +0x11a\r\ngrafana_1               | runtime.selectgo(0xc42049df50, 0xc4200e8060)\r\ngrafana_1               | \t/usr/local/go/src/runtime/select.go:392 +0xe50\r\ngrafana_1               | runtime.ensureSigM.func1()\r\ngrafana_1               | \t/usr/local/go/src/runtime/signal_unix.go:549 +0x1f4\r\ngrafana_1               | runtime.goexit()\r\ngrafana_1               | \t/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1\r\ngrafana_1               | \r\ngrafana_1               | goroutine 51 [select]:\r\ngrafana_1               | database/sql.(*DB).connectionOpener(0xc4200b08c0, 0x1518200, 0xc4202e6e80)\r\ngrafana_1               | \t/usr/local/go/src/database/sql/sql.go:935 +0x119\r\ngrafana_1               | created by database/sql.OpenDB\r\ngrafana_1               | \t/usr/local/go/src/database/sql/sql.go:634 +0x178\r\ngrafana_1               | \r\ngrafana_1               | goroutine 52 [select]:\r\ngrafana_1               | database/sql.(*DB).connectionResetter(0xc4200b08c0, 0x1518200, 0xc4202e6e80)\r\ngrafana_1               | \t/usr/local/go/src/database/sql/sql.go:948 +0x12a\r\ngrafana_1               | created by database/sql.OpenDB\r\ngrafana_1               | \t/usr/local/go/src/database/sql/sql.go:635 +0x1ae\r\ngrafana_1               | \r\ngrafana_1               | goroutine 55 [select]:\r\ngrafana_1               | database/sql.(*DB).connectionCleaner(0xc4200b08c0, 0xd18c2e28000)\r\ngrafana_1               | \t/usr/local/go/src/database/sql/sql.go:856 +0x38a\r\ngrafana_1               | created by database/sql.(*DB).startCleanerLocked\r\ngrafana_1               | \t/usr/local/go/src/database/sql/sql.go:843 +0xba\r\ngrafana_1               | \r\ngrafana_1               | goroutine 849 [select]:\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/alerting.(*Ticker).run(0xc42018c840)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/alerting/ticker.go:54 +0x283\r\ngrafana_1               | created by github.com/grafana/grafana/pkg/services/alerting.NewTicker\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/alerting/ticker.go:36 +0x121\r\ngrafana_1               | \r\ngrafana_1               | goroutine 994 [chan receive]:\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/alerting.(*DefaultRuleReader).initReader(0xc42018c880)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/alerting/reader.go:39 +0x4e\r\ngrafana_1               | created by github.com/grafana/grafana/pkg/services/alerting.NewRuleReader\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/alerting/reader.go:30 +0xb6\r\ngrafana_1               | \r\ngrafana_1               | goroutine 995 [runnable]:\r\ngrafana_1               | github.com/grafana/grafana/vendor/github.com/inconshreveable/log15.(*swapHandler).Log(0xc42021a880, 0xc4204ae000, 0x1aa4ff8, 0x1aa5038)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/github.com/inconshreveable/log15/handler_go14.go:13 +0xc6\r\ngrafana_1               | github.com/grafana/grafana/vendor/github.com/inconshreveable/log15.(*logger).write(0xc4202d1b40, 0x13e7ecd, 0x15, 0x3, 0x0, 0x0, 0x0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/github.com/inconshreveable/log15/logger.go:106 +0x24b\r\ngrafana_1               | github.com/grafana/grafana/vendor/github.com/inconshreveable/log15.(*logger).Info(0xc4202d1b40, 0x13e7ecd, 0x15, 0x0, 0x0, 0x0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/github.com/inconshreveable/log15/logger.go:139 +0x66\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/alerting.(*Engine).Run(0xc420076ba0, 0x1518200, 0xc4201cb880, 0x0, 0xc4205bf790)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/alerting/engine.go:43 +0x5e\r\ngrafana_1               | main.(*GrafanaServerImpl).Start.func2(0x8, 0x14520a8)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/server.go:85 +0x3f\r\ngrafana_1               | github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc4201cb8c0, 0xc4202d1c60)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57\r\ngrafana_1               | created by github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup.(*Group).Go\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66\r\ngrafana_1               | \r\ngrafana_1               | goroutine 996 [runnable]:\r\ngrafana_1               | github.com/grafana/grafana/vendor/github.com/go-stack/stack.Caller(0x2, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/github.com/go-stack/stack/stack.go:45 +0xf6\r\ngrafana_1               | github.com/grafana/grafana/vendor/github.com/inconshreveable/log15.(*logger).write(0xc4202d1cc0, 0x13f266f, 0x1b, 0x3, 0x0, 0x0, 0x0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/github.com/inconshreveable/log15/logger.go:111 +0xc9\r\ngrafana_1               | github.com/grafana/grafana/vendor/github.com/inconshreveable/log15.(*logger).Info(0xc4202d1cc0, 0x13f266f, 0x1b, 0x0, 0x0, 0x0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/github.com/inconshreveable/log15/logger.go:139 +0x66\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/cleanup.(*CleanUpService).Run(0xc42021a940, 0x1518200, 0xc4201cb880, 0xc4205bb798, 0xc4205bb790)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/cleanup/cleanup.go:29 +0x5d\r\ngrafana_1               | main.(*GrafanaServerImpl).Start.func3(0xc400000008, 0x14520a8)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/server.go:90 +0x3f\r\ngrafana_1               | github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc4201cb8c0, 0xc4202d1ce0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57\r\ngrafana_1               | created by github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup.(*Group).Go\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66\r\ngrafana_1               | \r\ngrafana_1               | goroutine 990 [select]:\r\ngrafana_1               | github.com/grafana/grafana/pkg/metrics.instrumentationLoop(0xc42028d920, 0x0)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/metrics/metrics.go:334 +0x188\r\ngrafana_1               | created by github.com/grafana/grafana/pkg/metrics.initMetricVars\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/metrics/metrics.go:319 +0xabb\r\ngrafana_1               | \r\ngrafana_1               | goroutine 838 [sleep]:\r\ngrafana_1               | time.Sleep(0x12a05f200)\r\ngrafana_1               | \t/usr/local/go/src/runtime/time.go:102 +0x166\r\ngrafana_1               | github.com/grafana/grafana/pkg/plugins.updateAppDashboards()\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/plugins/dashboards_updater.go:15 +0x34\r\ngrafana_1               | created by github.com/grafana/grafana/pkg/plugins.initPlugins\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/plugins/plugins.go:117 +0x92c\r\ngrafana_1               | \r\ngrafana_1               | goroutine 997 [runnable]:\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/notifications.processMailQueue()\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:29\r\ngrafana_1               | created by github.com/grafana/grafana/pkg/services/notifications.initMailQueue\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/notifications/mailer.go:26 +0x64\r\ngrafana_1               | \r\ngrafana_1               | goroutine 839 [chan receive]:\r\ngrafana_1               | github.com/grafana/grafana/pkg/plugins.(*PluginManager).Run(0xc42006a430, 0x1518200, 0xc4201cb880, 0xc4203d2798, 0xc4203d2790)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/plugins/plugins.go:55 +0x60\r\ngrafana_1               | main.(*GrafanaServerImpl).Start.func1(0xc400000008, 0x14520a8)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/cmd/grafana-server/server.go:70 +0x3f\r\ngrafana_1               | github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup.(*Group).Go.func1(0xc4201cb8c0, 0xc420261a40)\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup/errgroup.go:58 +0x57\r\ngrafana_1               | created by github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup.(*Group).Go\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/vendor/golang.org/x/sync/errgroup/errgroup.go:55 +0x66\r\ngrafana_1               | \r\ngrafana_1               | goroutine 998 [runnable]:\r\ngrafana_1               | github.com/grafana/grafana/pkg/services/notifications.processWebhookQueue()\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/notifications/webhook.go:51\r\ngrafana_1               | created by github.com/grafana/grafana/pkg/services/notifications.initWebhookQueue\r\ngrafana_1               | \t/go/src/github.com/grafana/grafana/pkg/services/notifications/webhook.go:48 +0xb4\r\n```\r\n\r\n \r\n"}}
{"actor":{"display_login":"marefr"}}