{"action":"closed","issue":{"url":"https://api.github.com/repos/elastic/kibana/issues/18313","repository_url":"https://api.github.com/repos/elastic/kibana","labels_url":"https://api.github.com/repos/elastic/kibana/issues/18313/labels{/name}","comments_url":"https://api.github.com/repos/elastic/kibana/issues/18313/comments","events_url":"https://api.github.com/repos/elastic/kibana/issues/18313/events","html_url":"https://github.com/elastic/kibana/issues/18313","id":317437481,"node_id":"MDU6SXNzdWUzMTc0Mzc0ODE=","number":18313,"title":"[ML] Provide a way to show user if ML \"missed\" data due to data ingest delays","user":{"login":"elasticmachine","id":15837671,"node_id":"MDQ6VXNlcjE1ODM3Njcx","avatar_url":"https://avatars3.githubusercontent.com/u/15837671?v=4","gravatar_id":"","url":"https://api.github.com/users/elasticmachine","html_url":"https://github.com/elasticmachine","followers_url":"https://api.github.com/users/elasticmachine/followers","following_url":"https://api.github.com/users/elasticmachine/following{/other_user}","gists_url":"https://api.github.com/users/elasticmachine/gists{/gist_id}","starred_url":"https://api.github.com/users/elasticmachine/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/elasticmachine/subscriptions","organizations_url":"https://api.github.com/users/elasticmachine/orgs","repos_url":"https://api.github.com/users/elasticmachine/repos","events_url":"https://api.github.com/users/elasticmachine/events{/privacy}","received_events_url":"https://api.github.com/users/elasticmachine/received_events","type":"User","site_admin":false},"labels":[{"id":912943303,"node_id":"MDU6TGFiZWw5MTI5NDMzMDM=","url":"https://api.github.com/repos/elastic/kibana/labels/:ml","name":":ml","color":"0e8a16","default":false},{"id":912944483,"node_id":"MDU6TGFiZWw5MTI5NDQ0ODM=","url":"https://api.github.com/repos/elastic/kibana/labels/:ml-results","name":":ml-results","color":"0e8a16","default":false},{"id":1062169267,"node_id":"MDU6TGFiZWwxMDYyMTY5MjY3","url":"https://api.github.com/repos/elastic/kibana/labels/v6.6.0","name":"v6.6.0","color":"ffffff","default":false}],"state":"closed","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2017-10-31T18:33:17Z","updated_at":"2018-12-21T12:36:40Z","closed_at":"2018-12-21T12:36:40Z","author_association":"NONE","body":"*Original comment by @richcollier:*\n\nI recently was working with a customer in Europe. They were struggling to understand why certain obvious spikes in their data weren't being flagged by ML:\r\n\r\n![image](https://user-images.githubusercontent.com/7461287/32241960-e96b2f5a-be47-11e7-8734-a7763a2c3c42.png)\r\n\r\nNotice that the major spike isn't flagged at all, yet the really tiny spike is tagged with an orange anomaly. This is obviously not sensible and I knew something was up.\r\n\r\nIt turns out it was a data ingest delay problem and that this customer needed to double the query_delay to 120s in order for the results to look sensible.\r\n\r\nBut how would the user know that this is happening?\r\n\r\nThe only thing I can think of (as a workaround now) is to use timelion to plot the number of docs in the raw data index and compare that to the anomaly results index and sum up the event_count field (which is a bucket by bucket count of how many records the ML job saw when each bucket was processed). So, for example:\r\n\r\n```\r\n.es(index=farequote,metric='sum:doc_count').label(\"raw data count\"), .es(index='.ml-anomalies-my_farequote_job', timefield='timestamp', metric='sum:event_count').label(\"count ML processed\"), .es(index=farequote).subtract(.es(index='.ml-anomalies-my_farequote_job', timefield='timestamp', metric='sum:event_count')).label(\"difference\")\r\n```\r\n\r\n![image](https://user-images.githubusercontent.com/7461287/32241997-01d143fe-be48-11e7-808e-8d42827a1f33.png)\r\n\r\nIt might be nice if there was something we can do in our UI to annotate that not all records are being seen by the ML job\r\n\r\n"}}
{"actor":{"display_login":"walterra"}}