{"action":"created","issue":{"url":"https://api.github.com/repos/druid-io/druid/issues/5721","repository_url":"https://api.github.com/repos/druid-io/druid","labels_url":"https://api.github.com/repos/druid-io/druid/issues/5721/labels{/name}","comments_url":"https://api.github.com/repos/druid-io/druid/issues/5721/comments","events_url":"https://api.github.com/repos/druid-io/druid/issues/5721/events","html_url":"https://github.com/druid-io/druid/issues/5721","id":318923148,"number":5721,"title":"Ingest: need a way to determine when new data is queryable","user":{"login":"danprince1","id":12734328,"avatar_url":"https://avatars3.githubusercontent.com/u/12734328?v=4","gravatar_id":"","url":"https://api.github.com/users/danprince1","html_url":"https://github.com/danprince1","followers_url":"https://api.github.com/users/danprince1/followers","following_url":"https://api.github.com/users/danprince1/following{/other_user}","gists_url":"https://api.github.com/users/danprince1/gists{/gist_id}","starred_url":"https://api.github.com/users/danprince1/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/danprince1/subscriptions","organizations_url":"https://api.github.com/users/danprince1/orgs","repos_url":"https://api.github.com/users/danprince1/repos","events_url":"https://api.github.com/users/danprince1/events{/privacy}","received_events_url":"https://api.github.com/users/danprince1/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-04-30T15:03:39Z","updated_at":"2018-05-15T21:50:32Z","closed_at":null,"author_association":"NONE","body":"We use druid 0.10, but recently upgraded to 0.11 and still see the same issue.  We have a use case in which we (a) ingest data via HDFS indexing, then (b) wait until the data is available to be queried, then (c) ping a separate system, which then queries the data to do something with it.  The problem is in step b.  At first we tried this:\r\n1. Poll the Overlord status endpoint /druid/indexer/v1/task/<taskid>/status until it returns SUCCESS or FAILED.  This lets us know when the indexing job is complete.\r\n1. Poll the Coordinator loadstatus endpoint /druid/coordinator/v1/loadstatus until it returns 100% for our datasource.\r\n\r\nBut then we discovered that for a **pre-existing** datasource there is a period of time after the indexing job completes in which loadstatus returns 100%, before it starts to return something less than 100%.  So we added another step:\r\n\r\n      1.5.  Poll the Coordinator loadstatus endpoint every second until it returns non-100%.\r\n\r\nThis polling has to be every second because if the datasource is small it takes very little time to distribute through the cluster.  If we poll every 10 seconds, we might miss seeing a non-100% number, and then we have no way of knowing if the new data is available or not.  Even at 1 second it's possible to miss this window.\r\n\r\nIt would be great to have a reliable way of knowing when ingested data is queryable.  Thanks!"},"comment":{"url":"https://api.github.com/repos/druid-io/druid/issues/comments/389324894","html_url":"https://github.com/druid-io/druid/issues/5721#issuecomment-389324894","issue_url":"https://api.github.com/repos/druid-io/druid/issues/5721","id":389324894,"user":{"login":"josephglanville","id":605591,"avatar_url":"https://avatars3.githubusercontent.com/u/605591?v=4","gravatar_id":"","url":"https://api.github.com/users/josephglanville","html_url":"https://github.com/josephglanville","followers_url":"https://api.github.com/users/josephglanville/followers","following_url":"https://api.github.com/users/josephglanville/following{/other_user}","gists_url":"https://api.github.com/users/josephglanville/gists{/gist_id}","starred_url":"https://api.github.com/users/josephglanville/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/josephglanville/subscriptions","organizations_url":"https://api.github.com/users/josephglanville/orgs","repos_url":"https://api.github.com/users/josephglanville/repos","events_url":"https://api.github.com/users/josephglanville/events{/privacy}","received_events_url":"https://api.github.com/users/josephglanville/received_events","type":"User","site_admin":false},"created_at":"2018-05-15T21:50:32Z","updated_at":"2018-05-15T21:50:32Z","author_association":"CONTRIBUTOR","body":"You could use segment metadata queries if you know ahead of time which intervals you are indexing for."}}
{"actor":{"display_login":"josephglanville"}}