{"action":"created","comment":{"url":"https://api.github.com/repos/prometheus/prometheus/pulls/comments/55494428","id":55494428,"diff_hunk":"@@ -1291,6 +1322,122 @@ func (s *memorySeriesStorage) calculatePersistenceUrgencyScore() float64 {\n \treturn score\n }\n \n+// quarantineSeries registers the provided fingerprint for quarantining. It\n+// always returns immediately. Quarantine requests are processed\n+// asynchronously. If there are too many requests queued, they are simply\n+// dropped.\n+//\n+// Quarantining means that the series file is moved to the orphaned directory,\n+// and all its traces are removed from indices. Call this method if an\n+// unrecoverable error is detected while dealing with a series, and pass in the\n+// encountered error. It will be saved as a hint in the orphaned directory.\n+func (s *memorySeriesStorage) quarantineSeries(fp model.Fingerprint, metric model.Metric, err error) {\n+\treq := quarantineRequest{fp: fp, metric: metric, reason: err}\n+\tselect {\n+\tcase s.quarantineRequests <- req:\n+\t\t// Request submitted.\n+\tdefault:\n+\t\tlog.\n+\t\t\tWith(\"fingerprint\", fp).\n+\t\t\tWith(\"metric\", metric).\n+\t\t\tWith(\"reason\", err).\n+\t\t\tWarn(\"Quarantine queue full. Dropped quarantine request.\")\n+\t\ts.seriesOps.WithLabelValues(droppedQuarantine).Inc()\n+\t}\n+}\n+\n+func (s *memorySeriesStorage) handleQuarantine() {\n+\tfor {\n+\t\tselect {\n+\t\tcase req := <-s.quarantineRequests:\n+\t\t\ts.purgeSeries(req.fp, req.metric, req.reason)\n+\t\t\tlog.\n+\t\t\t\tWith(\"fingerprint\", req.fp).\n+\t\t\t\tWith(\"metric\", req.metric).\n+\t\t\t\tWith(\"reason\", req.reason).\n+\t\t\t\tWarn(\"Series quarantined.\")\n+\t\tcase <-s.quarantineStopping:\n+\t\t\tlog.Info(\"Series quarantining stopped.\")","path":"storage/local/storage.go","position":293,"original_position":293,"commit_id":"eb9caf13be024752f866b91696573bdf964f5448","original_commit_id":"eb9caf13be024752f866b91696573bdf964f5448","user":{"login":"beorn7","id":5609886,"avatar_url":"https://avatars.githubusercontent.com/u/5609886?v=3","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"body":"https://github.com/prometheus/prometheus/issues/1479","created_at":"2016-03-09T09:53:39Z","updated_at":"2016-03-09T09:53:39Z","html_url":"https://github.com/prometheus/prometheus/pull/1448#discussion_r55494428","pull_request_url":"https://api.github.com/repos/prometheus/prometheus/pulls/1448","_links":{"self":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/comments/55494428"},"html":{"href":"https://github.com/prometheus/prometheus/pull/1448#discussion_r55494428"},"pull_request":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/1448"}}},"pull_request":{"url":"https://api.github.com/repos/prometheus/prometheus/pulls/1448","id":61451556,"html_url":"https://github.com/prometheus/prometheus/pull/1448","diff_url":"https://github.com/prometheus/prometheus/pull/1448.diff","patch_url":"https://github.com/prometheus/prometheus/pull/1448.patch","issue_url":"https://api.github.com/repos/prometheus/prometheus/issues/1448","number":1448,"state":"open","locked":false,"title":"Handle errors caused by data corruption more gracefully","user":{"login":"beorn7","id":5609886,"avatar_url":"https://avatars.githubusercontent.com/u/5609886?v=3","gravatar_id":"","url":"https://api.github.com/users/beorn7","html_url":"https://github.com/beorn7","followers_url":"https://api.github.com/users/beorn7/followers","following_url":"https://api.github.com/users/beorn7/following{/other_user}","gists_url":"https://api.github.com/users/beorn7/gists{/gist_id}","starred_url":"https://api.github.com/users/beorn7/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/beorn7/subscriptions","organizations_url":"https://api.github.com/users/beorn7/orgs","repos_url":"https://api.github.com/users/beorn7/repos","events_url":"https://api.github.com/users/beorn7/events{/privacy}","received_events_url":"https://api.github.com/users/beorn7/received_events","type":"User","site_admin":false},"body":"@juliusv The announced 4th level... (although level 1 is almost done, and level 3 is already removed).\r\nThis fixes https://github.com/prometheus/prometheus/issues/877\r\n\r\nThis requires all the panic calls upon unexpected data to be converted\r\ninto errors returned. This pollute the function signatures quite\r\nlot. Well, this is Go...\r\n\r\nThe ideas behind this are the following:\r\n\r\n- panic only if it's a programming error. Data corruptions happen, and\r\n  they are not programming errors.\r\n\r\n- If we detect a data corruption, we \"quarantine\" the series,\r\n  essentially removing it from the database and putting its data into\r\n  a separate directory for forensics.\r\n\r\n- Failure during writing to a series file is not considered corruption\r\n  automatically. It will call setDirty, though, so that a\r\n  crashrecovery upon the next restart will commence and check for\r\n  that.\r\n\r\n- Series quarantining and setDirty calls are logged and counted in\r\n  metrics, but are hidden from the user of the interfaces in\r\n  interface.go, whith the notable exception of Append(). The reasoning\r\n  is that we treat corruption by removing the corrupted series, i.e. a\r\n  query for it will return no results on its next call anyway, so\r\n  return no results right now. In the case of Append(), we want to\r\n  tell the user that no data has been appended, though.\r\n\r\nMinor side effects:\r\n\r\n- Now consistently using filepath.* instead of path.*.\r\n\r\n- Introduced structured logging where I touched it. This makes things\r\n  less consistent, but a complete change to structured logging would\r\n  be out of scope for this PR.","created_at":"2016-03-02T19:17:08Z","updated_at":"2016-03-09T09:53:39Z","closed_at":null,"merged_at":null,"merge_commit_sha":"8ceb059aaf10845306b19b9a580aa3971bd5aeff","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/prometheus/prometheus/pulls/1448/commits","review_comments_url":"https://api.github.com/repos/prometheus/prometheus/pulls/1448/comments","review_comment_url":"https://api.github.com/repos/prometheus/prometheus/pulls/comments{/number}","comments_url":"https://api.github.com/repos/prometheus/prometheus/issues/1448/comments","statuses_url":"https://api.github.com/repos/prometheus/prometheus/statuses/eb9caf13be024752f866b91696573bdf964f5448","head":{"label":"prometheus:beorn7/storage4","ref":"beorn7/storage4","sha":"eb9caf13be024752f866b91696573bdf964f5448","user":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"repo":{"id":6838921,"name":"prometheus","full_name":"prometheus/prometheus","owner":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/prometheus/prometheus","description":"The Prometheus monitoring system and time series database.","fork":false,"url":"https://api.github.com/repos/prometheus/prometheus","forks_url":"https://api.github.com/repos/prometheus/prometheus/forks","keys_url":"https://api.github.com/repos/prometheus/prometheus/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prometheus/prometheus/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prometheus/prometheus/teams","hooks_url":"https://api.github.com/repos/prometheus/prometheus/hooks","issue_events_url":"https://api.github.com/repos/prometheus/prometheus/issues/events{/number}","events_url":"https://api.github.com/repos/prometheus/prometheus/events","assignees_url":"https://api.github.com/repos/prometheus/prometheus/assignees{/user}","branches_url":"https://api.github.com/repos/prometheus/prometheus/branches{/branch}","tags_url":"https://api.github.com/repos/prometheus/prometheus/tags","blobs_url":"https://api.github.com/repos/prometheus/prometheus/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prometheus/prometheus/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prometheus/prometheus/git/refs{/sha}","trees_url":"https://api.github.com/repos/prometheus/prometheus/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prometheus/prometheus/statuses/{sha}","languages_url":"https://api.github.com/repos/prometheus/prometheus/languages","stargazers_url":"https://api.github.com/repos/prometheus/prometheus/stargazers","contributors_url":"https://api.github.com/repos/prometheus/prometheus/contributors","subscribers_url":"https://api.github.com/repos/prometheus/prometheus/subscribers","subscription_url":"https://api.github.com/repos/prometheus/prometheus/subscription","commits_url":"https://api.github.com/repos/prometheus/prometheus/commits{/sha}","git_commits_url":"https://api.github.com/repos/prometheus/prometheus/git/commits{/sha}","comments_url":"https://api.github.com/repos/prometheus/prometheus/comments{/number}","issue_comment_url":"https://api.github.com/repos/prometheus/prometheus/issues/comments{/number}","contents_url":"https://api.github.com/repos/prometheus/prometheus/contents/{+path}","compare_url":"https://api.github.com/repos/prometheus/prometheus/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prometheus/prometheus/merges","archive_url":"https://api.github.com/repos/prometheus/prometheus/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prometheus/prometheus/downloads","issues_url":"https://api.github.com/repos/prometheus/prometheus/issues{/number}","pulls_url":"https://api.github.com/repos/prometheus/prometheus/pulls{/number}","milestones_url":"https://api.github.com/repos/prometheus/prometheus/milestones{/number}","notifications_url":"https://api.github.com/repos/prometheus/prometheus/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prometheus/prometheus/labels{/name}","releases_url":"https://api.github.com/repos/prometheus/prometheus/releases{/id}","deployments_url":"https://api.github.com/repos/prometheus/prometheus/deployments","created_at":"2012-11-24T11:14:12Z","updated_at":"2016-03-09T09:12:09Z","pushed_at":"2016-03-09T05:28:39Z","git_url":"git://github.com/prometheus/prometheus.git","ssh_url":"git@github.com:prometheus/prometheus.git","clone_url":"https://github.com/prometheus/prometheus.git","svn_url":"https://github.com/prometheus/prometheus","homepage":"https://prometheus.io/","size":11399,"stargazers_count":4058,"watchers_count":4058,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":323,"mirror_url":null,"open_issues_count":128,"forks":323,"open_issues":128,"watchers":4058,"default_branch":"master"}},"base":{"label":"prometheus:beorn7/storage3","ref":"beorn7/storage3","sha":"d284864c871ce8f6dd5ef92f3f831616fbe207ac","user":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"repo":{"id":6838921,"name":"prometheus","full_name":"prometheus/prometheus","owner":{"login":"prometheus","id":3380462,"avatar_url":"https://avatars.githubusercontent.com/u/3380462?v=3","gravatar_id":"","url":"https://api.github.com/users/prometheus","html_url":"https://github.com/prometheus","followers_url":"https://api.github.com/users/prometheus/followers","following_url":"https://api.github.com/users/prometheus/following{/other_user}","gists_url":"https://api.github.com/users/prometheus/gists{/gist_id}","starred_url":"https://api.github.com/users/prometheus/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/prometheus/subscriptions","organizations_url":"https://api.github.com/users/prometheus/orgs","repos_url":"https://api.github.com/users/prometheus/repos","events_url":"https://api.github.com/users/prometheus/events{/privacy}","received_events_url":"https://api.github.com/users/prometheus/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/prometheus/prometheus","description":"The Prometheus monitoring system and time series database.","fork":false,"url":"https://api.github.com/repos/prometheus/prometheus","forks_url":"https://api.github.com/repos/prometheus/prometheus/forks","keys_url":"https://api.github.com/repos/prometheus/prometheus/keys{/key_id}","collaborators_url":"https://api.github.com/repos/prometheus/prometheus/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/prometheus/prometheus/teams","hooks_url":"https://api.github.com/repos/prometheus/prometheus/hooks","issue_events_url":"https://api.github.com/repos/prometheus/prometheus/issues/events{/number}","events_url":"https://api.github.com/repos/prometheus/prometheus/events","assignees_url":"https://api.github.com/repos/prometheus/prometheus/assignees{/user}","branches_url":"https://api.github.com/repos/prometheus/prometheus/branches{/branch}","tags_url":"https://api.github.com/repos/prometheus/prometheus/tags","blobs_url":"https://api.github.com/repos/prometheus/prometheus/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/prometheus/prometheus/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/prometheus/prometheus/git/refs{/sha}","trees_url":"https://api.github.com/repos/prometheus/prometheus/git/trees{/sha}","statuses_url":"https://api.github.com/repos/prometheus/prometheus/statuses/{sha}","languages_url":"https://api.github.com/repos/prometheus/prometheus/languages","stargazers_url":"https://api.github.com/repos/prometheus/prometheus/stargazers","contributors_url":"https://api.github.com/repos/prometheus/prometheus/contributors","subscribers_url":"https://api.github.com/repos/prometheus/prometheus/subscribers","subscription_url":"https://api.github.com/repos/prometheus/prometheus/subscription","commits_url":"https://api.github.com/repos/prometheus/prometheus/commits{/sha}","git_commits_url":"https://api.github.com/repos/prometheus/prometheus/git/commits{/sha}","comments_url":"https://api.github.com/repos/prometheus/prometheus/comments{/number}","issue_comment_url":"https://api.github.com/repos/prometheus/prometheus/issues/comments{/number}","contents_url":"https://api.github.com/repos/prometheus/prometheus/contents/{+path}","compare_url":"https://api.github.com/repos/prometheus/prometheus/compare/{base}...{head}","merges_url":"https://api.github.com/repos/prometheus/prometheus/merges","archive_url":"https://api.github.com/repos/prometheus/prometheus/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/prometheus/prometheus/downloads","issues_url":"https://api.github.com/repos/prometheus/prometheus/issues{/number}","pulls_url":"https://api.github.com/repos/prometheus/prometheus/pulls{/number}","milestones_url":"https://api.github.com/repos/prometheus/prometheus/milestones{/number}","notifications_url":"https://api.github.com/repos/prometheus/prometheus/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/prometheus/prometheus/labels{/name}","releases_url":"https://api.github.com/repos/prometheus/prometheus/releases{/id}","deployments_url":"https://api.github.com/repos/prometheus/prometheus/deployments","created_at":"2012-11-24T11:14:12Z","updated_at":"2016-03-09T09:12:09Z","pushed_at":"2016-03-09T05:28:39Z","git_url":"git://github.com/prometheus/prometheus.git","ssh_url":"git@github.com:prometheus/prometheus.git","clone_url":"https://github.com/prometheus/prometheus.git","svn_url":"https://github.com/prometheus/prometheus","homepage":"https://prometheus.io/","size":11399,"stargazers_count":4058,"watchers_count":4058,"language":"Go","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":323,"mirror_url":null,"open_issues_count":128,"forks":323,"open_issues":128,"watchers":4058,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/1448"},"html":{"href":"https://github.com/prometheus/prometheus/pull/1448"},"issue":{"href":"https://api.github.com/repos/prometheus/prometheus/issues/1448"},"comments":{"href":"https://api.github.com/repos/prometheus/prometheus/issues/1448/comments"},"review_comments":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/1448/comments"},"review_comment":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/prometheus/prometheus/pulls/1448/commits"},"statuses":{"href":"https://api.github.com/repos/prometheus/prometheus/statuses/eb9caf13be024752f866b91696573bdf964f5448"}}}}