{"action":"created","issue":{"url":"https://api.github.com/repos/apache/incubator-druid/issues/6057","repository_url":"https://api.github.com/repos/apache/incubator-druid","labels_url":"https://api.github.com/repos/apache/incubator-druid/issues/6057/labels{/name}","comments_url":"https://api.github.com/repos/apache/incubator-druid/issues/6057/comments","events_url":"https://api.github.com/repos/apache/incubator-druid/issues/6057/events","html_url":"https://github.com/apache/incubator-druid/issues/6057","id":345337109,"node_id":"MDU6SXNzdWUzNDUzMzcxMDk=","number":6057,"title":"Broker sends sequential requests to the historical for union queries","user":{"login":"a2l007","id":4603202,"node_id":"MDQ6VXNlcjQ2MDMyMDI=","avatar_url":"https://avatars0.githubusercontent.com/u/4603202?v=4","gravatar_id":"","url":"https://api.github.com/users/a2l007","html_url":"https://github.com/a2l007","followers_url":"https://api.github.com/users/a2l007/followers","following_url":"https://api.github.com/users/a2l007/following{/other_user}","gists_url":"https://api.github.com/users/a2l007/gists{/gist_id}","starred_url":"https://api.github.com/users/a2l007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a2l007/subscriptions","organizations_url":"https://api.github.com/users/a2l007/orgs","repos_url":"https://api.github.com/users/a2l007/repos","events_url":"https://api.github.com/users/a2l007/events{/privacy}","received_events_url":"https://api.github.com/users/a2l007/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":8,"created_at":"2018-07-27T19:14:01Z","updated_at":"2018-09-20T03:56:34Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Analyzing groupBy queries based on Union datasources, we have seen that for a given query if a historical has segments for the multiple datasources part of this query, the broker generates multiple requests to this historical, but in a serial manner.\r\nFor example, running the following union query:\r\n\r\n```\r\n{\r\n  \"queryType\": \"groupBy\",\r\n  \"dataSource\": {\r\n    \"type\": \"union\",\r\n    \"dataSources\": [\r\n      {\r\n        \"type\": \"table\",\r\n        \"name\": \"tableA\"\r\n      },\r\n      {\r\n        \"type\": \"table\",\r\n        \"name\": \"tableB\"\r\n      },\r\n      {\r\n        \"type\": \"table\",\r\n        \"name\": \"tableC\"\r\n      }\r\n    ]\r\n  },\r\n  \"intervals\": {\r\n    \"type\": \"LegacySegmentSpec\",\r\n    \"intervals\": [\r\n      \"2018-07-23T00:00:00.000Z\\/2018-07-24T00:00:00.000Z\"\r\n    ]\r\n  },\r\n  \"virtualColumns\": [\r\n    \r\n  ],\r\n  \"filter\": null,\r\n  \"granularity\": \"DAY\",\r\n  \"dimensions\": [\r\n    {\r\n      \"type\": \"default\",\r\n      \"dimension\": \"acc_id\"\r\n    }\r\n  ],\r\n  \"aggregations\": [\r\n    {\r\n      \"fieldName\": \"clicks\",\r\n      \"name\": \"clicks\",\r\n      \"type\": \"longSum\"\r\n    }\r\n  ],\r\n  \"postAggregations\": [\r\n    \r\n  ],\r\n  \"having\": null,\r\n  \"limitSpec\": {\r\n    \"type\": \"NoopLimitSpec\"\r\n  },\r\n  \"descending\": false\r\n}\r\n```\r\ngenerates logs such as:\r\n```\r\n2018-07-24T14:50:06,500 INFO [qtp1256384385-231[groupBy_q_test_1]] com.metamx.http.client.pool.ChannelResourceFactory - Generating: https://tier\r\n.historical.foo.com:4443\r\n2018-07-24T14:50:40,366 INFO [qtp1256384385-231[groupBy_q_test_1]] com.metamx.http.client.pool.ChannelResourceFactory - Generating: https://tier\r\n.historical.foo.com:4443\r\n```\r\n\r\nThis historical contains segments for datasources `tableA` and `tableB` and therefore it sends two request to the historical but as it can be seen from the timestamps, there is a delay between both the requests.\r\n [BrokerServerView](https://github.com/apache/incubator-druid/blob/master/server/src/main/java/io/druid/client/BrokerServerView.java#L291) confirms this behaviour that at an instant for a given query, there can be only a single request to a historical. This behaviour clearly causes query execution delays for union queries.\r\nBefore I work on investigating if this can be parallelized, I wanted to check with the community for any comments on whether this is already a known issue with union queries or if it is actually a bug.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/incubator-druid/issues/comments/423032195","html_url":"https://github.com/apache/incubator-druid/issues/6057#issuecomment-423032195","issue_url":"https://api.github.com/repos/apache/incubator-druid/issues/6057","id":423032195,"node_id":"MDEyOklzc3VlQ29tbWVudDQyMzAzMjE5NQ==","user":{"login":"a2l007","id":4603202,"node_id":"MDQ6VXNlcjQ2MDMyMDI=","avatar_url":"https://avatars0.githubusercontent.com/u/4603202?v=4","gravatar_id":"","url":"https://api.github.com/users/a2l007","html_url":"https://github.com/a2l007","followers_url":"https://api.github.com/users/a2l007/followers","following_url":"https://api.github.com/users/a2l007/following{/other_user}","gists_url":"https://api.github.com/users/a2l007/gists{/gist_id}","starred_url":"https://api.github.com/users/a2l007/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/a2l007/subscriptions","organizations_url":"https://api.github.com/users/a2l007/orgs","repos_url":"https://api.github.com/users/a2l007/repos","events_url":"https://api.github.com/users/a2l007/events{/privacy}","received_events_url":"https://api.github.com/users/a2l007/received_events","type":"User","site_admin":false},"created_at":"2018-09-20T03:56:34Z","updated_at":"2018-09-20T03:56:34Z","author_association":"CONTRIBUTOR","body":"Sounds good @jihoonson . I'll test this out. Thanks"}}
{"actor":{"display_login":"a2l007"}}