{"action":"created","issue":{"url":"https://api.github.com/repos/catapult-project/catapult/issues/2293","repository_url":"https://api.github.com/repos/catapult-project/catapult","labels_url":"https://api.github.com/repos/catapult-project/catapult/issues/2293/labels{/name}","comments_url":"https://api.github.com/repos/catapult-project/catapult/issues/2293/comments","events_url":"https://api.github.com/repos/catapult-project/catapult/issues/2293/events","html_url":"https://github.com/catapult-project/catapult/issues/2293","id":151887313,"number":2293,"title":"Perf dashboard failures triaging anomalies, bugs, and bisects","user":{"login":"anniesullie","id":4331931,"avatar_url":"https://avatars.githubusercontent.com/u/4331931?v=3","gravatar_id":"","url":"https://api.github.com/users/anniesullie","html_url":"https://github.com/anniesullie","followers_url":"https://api.github.com/users/anniesullie/followers","following_url":"https://api.github.com/users/anniesullie/following{/other_user}","gists_url":"https://api.github.com/users/anniesullie/gists{/gist_id}","starred_url":"https://api.github.com/users/anniesullie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anniesullie/subscriptions","organizations_url":"https://api.github.com/users/anniesullie/orgs","repos_url":"https://api.github.com/users/anniesullie/repos","events_url":"https://api.github.com/users/anniesullie/events{/privacy}","received_events_url":"https://api.github.com/users/anniesullie/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-04-29T14:56:42Z","updated_at":"2016-04-29T15:08:56Z","closed_at":null,"body":"I'm getting stackdriver alerts on all three of our custom metrics (`TriageAnomalies`, `TriageBugs `, and `UpdateBugWithResults `). Some are sending recovered messages, but this still merits a high-priority investigation.\r\n\r\nAn initial look at http://go/chromeperf-errors shows several new errors with this stack:\r\n```\r\nValueError: math domain error\r\nat _FormatNumber():154 (bisect_report.py:154 in /base/data/home/apps/s~chromeperf/dev-sullivan-69a681ae.392415948149617343/dashboard)\r\nat RevisionRow():123 (bisect_report.py:123 in /base/data/home/apps/s~chromeperf/dev-sullivan-69a681ae.392415948149617343/dashboard)\r\nat _RevisionTable():129 (bisect_report.py:129 in /base/data/home/apps/s~chromeperf/dev-sullivan-69a681ae.392415948149617343/dashboard)\r\nat GetReport():96 (bisect_report.py:96 in /base/data/home/apps/s~chromeperf/dev-sullivan-69a681ae.392415948149617343/dashboard)\r\nat _PostResult():169 (update_bug_with_results.py:169 in /base/data/home/apps/s~chromeperf/dev-sullivan-69a681ae.392415948149617343/dashboard)\r\nat _CheckBisectJob():124 (update_bug_with_results.py:124 in /base/data/home/apps/s~chromeperf/dev-sullivan-69a681ae.392415948149617343/dashboard)\r\nat _CheckJob():110 (update_bug_with_results.py:110 in /base/data/home/apps/s~chromeperf/dev-sullivan-69a681ae.392415948149617343/dashboard)\r\nat get():76 (update_bug_with_results.py:76 in /base/data/home/apps/s~chromeperf/dev-sullivan-69a681ae.392415948149617343/dashboard)\r\n```"},"comment":{"url":"https://api.github.com/repos/catapult-project/catapult/issues/comments/215747642","html_url":"https://github.com/catapult-project/catapult/issues/2293#issuecomment-215747642","issue_url":"https://api.github.com/repos/catapult-project/catapult/issues/2293","id":215747642,"user":{"login":"anniesullie","id":4331931,"avatar_url":"https://avatars.githubusercontent.com/u/4331931?v=3","gravatar_id":"","url":"https://api.github.com/users/anniesullie","html_url":"https://github.com/anniesullie","followers_url":"https://api.github.com/users/anniesullie/followers","following_url":"https://api.github.com/users/anniesullie/following{/other_user}","gists_url":"https://api.github.com/users/anniesullie/gists{/gist_id}","starred_url":"https://api.github.com/users/anniesullie/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/anniesullie/subscriptions","organizations_url":"https://api.github.com/users/anniesullie/orgs","repos_url":"https://api.github.com/users/anniesullie/repos","events_url":"https://api.github.com/users/anniesullie/events{/privacy}","received_events_url":"https://api.github.com/users/anniesullie/received_events","type":"User","site_admin":false},"created_at":"2016-04-29T15:08:56Z","updated_at":"2016-04-29T15:08:56Z","body":"@dave-2 the `_FormatNumber` code in the callstack was modified in https://codereview.chromium.org/1920543002.\r\n\r\n@vuppalapu @ro-berto: We got some \"alert recovered\" emails, although there were no changes to the perf dashboard. Is it possible that whatever try job was causing the formatting problem is no longer being returned by the query here? https://github.com/catapult-project/catapult/blob/master/dashboard/dashboard/update_bug_with_results.py#L83\r\n\r\nIf so, is there any way to narrow down which job may have caused the problem?"}}