{"action":"created","issue":{"url":"https://api.github.com/repos/linkedin/dr-elephant/issues/74","repository_url":"https://api.github.com/repos/linkedin/dr-elephant","labels_url":"https://api.github.com/repos/linkedin/dr-elephant/issues/74/labels{/name}","comments_url":"https://api.github.com/repos/linkedin/dr-elephant/issues/74/comments","events_url":"https://api.github.com/repos/linkedin/dr-elephant/issues/74/events","html_url":"https://github.com/linkedin/dr-elephant/pull/74","id":159734924,"number":74,"title":"HttpPooling + gzip for requests to the Job History Server","user":{"login":"rsprabery","id":1026696,"avatar_url":"https://avatars.githubusercontent.com/u/1026696?v=3","gravatar_id":"","url":"https://api.github.com/users/rsprabery","html_url":"https://github.com/rsprabery","followers_url":"https://api.github.com/users/rsprabery/followers","following_url":"https://api.github.com/users/rsprabery/following{/other_user}","gists_url":"https://api.github.com/users/rsprabery/gists{/gist_id}","starred_url":"https://api.github.com/users/rsprabery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/rsprabery/subscriptions","organizations_url":"https://api.github.com/users/rsprabery/orgs","repos_url":"https://api.github.com/users/rsprabery/repos","events_url":"https://api.github.com/users/rsprabery/events{/privacy}","received_events_url":"https://api.github.com/users/rsprabery/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-06-10T22:25:16Z","updated_at":"2016-06-13T06:55:08Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/linkedin/dr-elephant/pulls/74","html_url":"https://github.com/linkedin/dr-elephant/pull/74","diff_url":"https://github.com/linkedin/dr-elephant/pull/74.diff","patch_url":"https://github.com/linkedin/dr-elephant/pull/74.patch"},"body":"Addresses the [scaling issue](https://github.com/linkedin/dr-elephant/issues/73). \r\n\r\nThis patch utilizes Apache's HTTP Client Lib to enable connection pooling for TCP requests that can then be shared between threads and utilized across Job History Server API requests to set an upper limit on the number of TCP connections open to the Job History Server at any one time. We also ensure that gzip is being used by utilizing http client api's that enable this by default.\r\n\r\nWith these changes, the queue of jobs being processed is reduced dramatically with only 96 worker threads as shown below:\r\n\r\n![queue_size](https://cloud.githubusercontent.com/assets/1026696/15980222/ae499736-2f1e-11e6-87a9-48519a0eb17c.png)\r\n\r\n\r\nAfter a few minutes the queue size normalizes to the number of jobs completed in the last [`WAIT_INTERVAL`](https://github.com/linkedin/dr-elephant/blob/master/app/com/linkedin/drelephant/ElephantRunner.java#L48).\r\n\r\nOne must still decide on the ratio of connections to threads. For many applications, the number of connections should be less than the number of worker threads. In the case of Dr.E, each thread really needs it's own connection as there are many requests being made in parallel. There are still speed benefits as the number of TCP connections can be capped, whereas before a separate connection was being created for each request ((4 + task num) * job count = many thousands).\r\n\r\nThe screenshot below shows how threads have to wait for a connection if not configured to have 1 connection per thread (the configuration was 48 worker threads with 24 connections).\r\n\r\n![jvisvm](https://cloud.githubusercontent.com/assets/1026696/15980238/de3e78da-2f1e-11e6-8786-f7da33c47c70.png)\r\n\r\nMoving to a 1:1 ratio for connections to threads moved the bottleneck to `.streamRead()` with virtually no time being spent in other functions. We believe this to be the optimal configuration. Using gzip ensures that the least amount of time possible is spent reading from the stream.\r\n\r\nGzip is enabled by using the [`HttpClientBuilder`](https://hc.apache.org/httpcomponents-client-4.3.x/httpclient/apidocs/org/apache/http/impl/client/HttpClientBuilder.html). This ensures that the \"accept-encoding\" header is set to \"gzip\" and also processes the response body so that [`GZipInputStream`](https://docs.oracle.com/javase/7/docs/api/java/util/zip/GZIPInputStream.html) doesn't have to be used manually. There are other ways to get `HttpClient`'s in Apache's Http Client lib, if those are used the gzip header will have to be re-added and responses manually decompressed.\r\n\r\nLimitations & Future Work:\r\n- Add configuration option for the max number of connections.\r\n  - This should likely just be tied to the number of threads, which could possibly be pulled out into a configuration option (as a separate PR)\r\n- Any suggestions on ensuring compatibility with the Hadoop Authentication Mechanism? I think the first step is to add a test and then extend the class that handles pooling to ensure that the authentication process is also pooled.\r\n  - The benefit of  getting the connection from the `AuthenticatedURL` that ships with the hadoop jars is that authentication is handled automatically. We'll need to add something similar to this patch. I'm currently looking into this, but feedback is much appreciated."},"comment":{"url":"https://api.github.com/repos/linkedin/dr-elephant/issues/comments/225502022","html_url":"https://github.com/linkedin/dr-elephant/pull/74#issuecomment-225502022","issue_url":"https://api.github.com/repos/linkedin/dr-elephant/issues/74","id":225502022,"user":{"login":"akshayrai","id":4448437,"avatar_url":"https://avatars.githubusercontent.com/u/4448437?v=3","gravatar_id":"","url":"https://api.github.com/users/akshayrai","html_url":"https://github.com/akshayrai","followers_url":"https://api.github.com/users/akshayrai/followers","following_url":"https://api.github.com/users/akshayrai/following{/other_user}","gists_url":"https://api.github.com/users/akshayrai/gists{/gist_id}","starred_url":"https://api.github.com/users/akshayrai/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/akshayrai/subscriptions","organizations_url":"https://api.github.com/users/akshayrai/orgs","repos_url":"https://api.github.com/users/akshayrai/repos","events_url":"https://api.github.com/users/akshayrai/events{/privacy}","received_events_url":"https://api.github.com/users/akshayrai/received_events","type":"User","site_admin":false},"created_at":"2016-06-13T06:55:08Z","updated_at":"2016-06-13T06:55:08Z","body":"@rsprabery, thanks for the detailed analysis report and for this PR trying to fix the issue.\r\n\r\nQuestions,\r\n1. What is the rationale behind having 96 worker threads? Doesn't it depend on the load and the number of cpu cores? \r\n2. Only when we start Dr. Elephant for the first time or restart Dr. Elephant after a long shutdown, there is a chance of heavy load on the job history server as Dr. Elephant has to analyze all the unscanned jobs from the resource manager. Once the queue normalizes to the jobs fetched in the last fetch interval, all the threads need not be active. I think we should shutdown unused threads and restart them whenever the load increases!\r\n\r\n\r\nSuggestions,\r\n1. Rather than changing the existing MapreduceFetcher, try including a separate fetcher(duplicate with your changes) and configure it in FetcherConf.xml. We can then deprecate and remove the old fetcher.\r\n2. +1 on making the executors and connections configurable."}}
{"actor":{"display_login":"akshayrai"}}