{"action":"created","issue":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/4128","repository_url":"https://api.github.com/repos/prometheus/prometheus","labels_url":"https://api.github.com/repos/prometheus/prometheus/issues/4128/labels{/name}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/4128/comments","events_url":"https://api.github.com/repos/prometheus/prometheus/issues/4128/events","html_url":"https://github.com/prometheus/prometheus/issues/4128","id":319194631,"number":4128,"title":"tsdb Panic when reading chunk","user":{"login":"karlhungus","id":64263,"avatar_url":"https://avatars2.githubusercontent.com/u/64263?v=4","gravatar_id":"","url":"https://api.github.com/users/karlhungus","html_url":"https://github.com/karlhungus","followers_url":"https://api.github.com/users/karlhungus/followers","following_url":"https://api.github.com/users/karlhungus/following{/other_user}","gists_url":"https://api.github.com/users/karlhungus/gists{/gist_id}","starred_url":"https://api.github.com/users/karlhungus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/karlhungus/subscriptions","organizations_url":"https://api.github.com/users/karlhungus/orgs","repos_url":"https://api.github.com/users/karlhungus/repos","events_url":"https://api.github.com/users/karlhungus/events{/privacy}","received_events_url":"https://api.github.com/users/karlhungus/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-05-01T13:15:23Z","updated_at":"2018-05-01T14:35:46Z","closed_at":null,"author_association":"NONE","body":"## Bug Report\r\n**What did you do?**\r\nRan prometheus\r\n\r\n**What did you expect to see?**\r\nIt run\r\n\r\n**What did you see instead? Under which circumstances?**\r\n\r\nException on startup: panic: runtime error: index out of range (looks similar to: https://github.com/prometheus/tsdb/issues/251)\r\n\r\n\r\n\r\n**Environment**\r\n\r\n* System information:\r\n```\r\n/prometheus $ uname -srm\r\nLinux 4.15.3-1.el7.elrepo.x86_64 x86_64\r\n```\r\n\r\n* Prometheus version:\r\n```\r\n/prometheus $ prometheus --version\r\nprometheus, version 2.2.1 (branch: HEAD, revision: bc6058c81272a8d938c05e75607371284236aadc)\r\n  build user:       root@149e5b3f0829\r\n  build date:       20180314-14:15:45\r\n  go version:       go1.10\r\n```\r\n\r\n\r\n\r\n\r\n\r\n* Logs:\r\n```\r\nlevel=info ts=2018-05-01T12:35:26.455261083Z caller=main.go:220 msg=\"Starting Prometheus\" version=\"(version=2.2.1, branch=HEAD, revision=bc6058c81272a8d938c05e75607371284236aadc)\"\r\nlevel=info ts=2018-05-01T12:35:26.45538143Z caller=main.go:221 build_context=\"(go=go1.10, user=root@149e5b3f0829, date=20180314-14:15:45)\"\r\nlevel=info ts=2018-05-01T12:35:26.455414873Z caller=main.go:222 host_details=\"(Linux 4.15.3-1.el7.elrepo.x86_64 #1 SMP Mon Feb 12 06:46:25 EST 2018 x86_64 prometheus-59bf4d7cf-sxsk5 (none))\"\r\nlevel=info ts=2018-05-01T12:35:26.455427991Z caller=main.go:223 fd_limits=\"(soft=65536, hard=65536)\"\r\nlevel=info ts=2018-05-01T12:35:26.459927519Z caller=web.go:382 component=web msg=\"Start listening for connections\" address=0.0.0.0:9090\r\nlevel=info ts=2018-05-01T12:35:26.45947935Z caller=main.go:504 msg=\"Starting TSDB ...\"\r\nlevel=error ts=2018-05-01T12:36:48.215666256Z caller=wal.go:275 component=tsdb msg=\"WAL corruption detected; truncating\" err=\"unexpected CRC32 checksum 6cafd16a, want 0\" file=/data/prometheus/wal/033372 pos=75965705\r\nlevel=info ts=2018-05-01T12:36:48.950404399Z caller=main.go:514 msg=\"TSDB started\"\r\nlevel=info ts=2018-05-01T12:36:48.950585155Z caller=main.go:588 msg=\"Loading configuration file\" filename=/etc/prometheus/config.yaml\r\nlevel=info ts=2018-05-01T12:36:48.96267101Z caller=kubernetes.go:191 component=\"discovery manager scrape\" discovery=k8s msg=\"Using pod service account via in-cluster config\"\r\nlevel=info ts=2018-05-01T12:36:48.965331005Z caller=kubernetes.go:191 component=\"discovery manager scrape\" discovery=k8s msg=\"Using pod service account via in-cluster config\"\r\nlevel=info ts=2018-05-01T12:36:48.966139809Z caller=kubernetes.go:191 component=\"discovery manager scrape\" discovery=k8s msg=\"Using pod service account via in-cluster config\"\r\nlevel=info ts=2018-05-01T12:36:48.966845893Z caller=kubernetes.go:191 component=\"discovery manager scrape\" discovery=k8s msg=\"Using pod service account via in-cluster config\"\r\nlevel=info ts=2018-05-01T12:36:48.96989835Z caller=main.go:491 msg=\"Server is ready to receive web requests.\"\r\nlevel=info ts=2018-05-01T12:37:48.996924066Z caller=compact.go:393 component=tsdb msg=\"compact blocks\" count=4 mint=1524960000000 maxt=1524981600000\r\npanic: runtime error: index out of range\r\n\r\ngoroutine 403 [running]:\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunks.(*Reader).Chunk(0xc75cb30c40, 0x13e, 0x1c56c80, 0xc5b252e0f0, 0x0, 0x0)\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/chunks/chunks.go:365 +0x3f8\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*compactionSeriesSet).Next(0xc75bbc0dc0, 0xc763d9c770)\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/compact.go:691 +0x223\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.newCompactionMerger(0x1c47a80, 0xc75bbc0dc0, 0x1c47a80, 0xc75bbc0e60, 0xc7687230a0, 0x1c51880, 0xc7687230a0)\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/compact.go:733 +0x8f\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*LeveledCompactor).populateBlock(0xc420d93220, 0xc75cb30e80, 0x4, 0x4, 0xc763d9c540, 0x1c56cc0, 0xc768766000, 0x1c3e680, 0xc7671fe640, 0x0, ...)\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/compact.go:537 +0x5bf\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*LeveledCompactor).write(0xc420d93220, 0x7ffc588f58e3, 0x10, 0xc763d9c540, 0xc75cb30e80, 0x4, 0x4, 0x0, 0x0)\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/compact.go:441 +0x5dd\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*LeveledCompactor).Compact(0xc420d93220, 0x7ffc588f58e3, 0x10, 0xc75cb30b40, 0x4, 0x4, 0x3023447db51b6301, 0x6431175cbef51e81, 0x0, 0x0)\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/compact.go:339 +0x693\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).compact(0xc420c7a0c0, 0x0, 0x0, 0x0)\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:419 +0x400\r\ngithub.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.(*DB).run(0xc420c7a0c0)\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:279 +0x2cc\r\ncreated by github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb.Open\r\n        /go/src/github.com/prometheus/prometheus/vendor/github.com/prometheus/tsdb/db.go:243 +0x5ba\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/prometheus/prometheus/issues/comments/385685888","html_url":"https://github.com/prometheus/prometheus/issues/4128#issuecomment-385685888","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/4128","id":385685888,"user":{"login":"krasi-georgiev","id":8903888,"avatar_url":"https://avatars0.githubusercontent.com/u/8903888?v=4","gravatar_id":"","url":"https://api.github.com/users/krasi-georgiev","html_url":"https://github.com/krasi-georgiev","followers_url":"https://api.github.com/users/krasi-georgiev/followers","following_url":"https://api.github.com/users/krasi-georgiev/following{/other_user}","gists_url":"https://api.github.com/users/krasi-georgiev/gists{/gist_id}","starred_url":"https://api.github.com/users/krasi-georgiev/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/krasi-georgiev/subscriptions","organizations_url":"https://api.github.com/users/krasi-georgiev/orgs","repos_url":"https://api.github.com/users/krasi-georgiev/repos","events_url":"https://api.github.com/users/krasi-georgiev/events{/privacy}","received_events_url":"https://api.github.com/users/krasi-georgiev/received_events","type":"User","site_admin":false},"created_at":"2018-05-01T14:35:46Z","updated_at":"2018-05-01T14:35:46Z","author_association":"OWNER","body":"had  quick look and it seems some corruption in the chunk so it fails to read the encoding.\r\n\r\nIs this easy to reproduce?"}}
{"actor":{"display_login":"krasi-georgiev"}}