{"action":"created","issue":{"url":"https://api.github.com/repos/apache/spark/issues/9571","repository_url":"https://api.github.com/repos/apache/spark","labels_url":"https://api.github.com/repos/apache/spark/issues/9571/labels{/name}","comments_url":"https://api.github.com/repos/apache/spark/issues/9571/comments","events_url":"https://api.github.com/repos/apache/spark/issues/9571/events","html_url":"https://github.com/apache/spark/pull/9571","id":115936707,"number":9571,"title":"[SPARK-11373] [CORE] Add metrics to the History Server and FsHistoryProvider","user":{"login":"steveloughran","id":162090,"avatar_url":"https://avatars.githubusercontent.com/u/162090?v=3","gravatar_id":"","url":"https://api.github.com/users/steveloughran","html_url":"https://github.com/steveloughran","followers_url":"https://api.github.com/users/steveloughran/followers","following_url":"https://api.github.com/users/steveloughran/following{/other_user}","gists_url":"https://api.github.com/users/steveloughran/gists{/gist_id}","starred_url":"https://api.github.com/users/steveloughran/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steveloughran/subscriptions","organizations_url":"https://api.github.com/users/steveloughran/orgs","repos_url":"https://api.github.com/users/steveloughran/repos","events_url":"https://api.github.com/users/steveloughran/events{/privacy}","received_events_url":"https://api.github.com/users/steveloughran/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":25,"created_at":"2015-11-09T18:31:03Z","updated_at":"2016-05-13T11:39:07Z","closed_at":null,"pull_request":{"url":"https://api.github.com/repos/apache/spark/pulls/9571","html_url":"https://github.com/apache/spark/pull/9571","diff_url":"https://github.com/apache/spark/pull/9571.diff","patch_url":"https://github.com/apache/spark/pull/9571.patch"},"body":"This adds metrics to the history server, with the `FsHistoryProvider` metering its load, performance and reliability.\r\n\r\nsee [SPARK-11373](https://issues.apache.org/jira/browse/SPARK-11373)\r\n\r\nThe `HistoryServer` sets up the codahale metrics for the Web under metrics/ with metrics/metrics behind metrics, metrics/health any health probes and metrics/threads a thread dump. There's currently no attempt to  hook up JMX, etc. The Web servlets are the ones tests can easily hit and don't need infrastructure, so are the good initial first step.\r\n\r\nIt then passes the metrics and health registries down to the providers in a `ApplicationHistoryBinding` case class, via a new method\r\n\r\n    def start(binding: ApplicationHistoryBinding): Unit\r\n\r\nThe base class has implementation so that all existing providers will still link properly; the base implementation currently checks and fails\r\n the use of a binding case class is also to ensure that if new binding information were added in future, existing implementations would still link.\r\n\r\nThe `FsHistoryProvider` implements the `start()` method, registering two counters and two timers.\r\n\r\n1. Number of update attempts and number of failed updates —and the same for app UI loads.\r\n2. Time for updates and app UI loads.\r\n\r\nPoints of note\r\n\r\n* Why not use Spark's `MetricsSystem`? I did start off with that, but it needs a `SparkContext` to run off, which the server doesn't have. Ideally that would be way to go, as it would support all the spark conf -based metrics setup. Someone who understands the `MetricsSystem` would need to get involved here as would make for a more complex patch. In `FsHistoryProvider` the registry information is all kept in a `Source` subclass for ease of future migration to `MetricsSystem`.\r\n* Why the extra `HealthRegistry`? It's a nice way of allowing providers to indicate (possibly transient) health problems for monitoring tools/clients to hit. For the FS provider it could maybe flag when there hadn't been any successful update for a specified time period. (that could also be indicated by having a counter of \"seconds since last update\" and let monitoring tools monitor the counter value and act on it). Access control problems to the directory is something else which may be considered a liveness problem: it won't get better without human intervention\r\n* The `FsHistoryProvider.start()` method should really take the thread start code from from class constructor's `initialize()` method. This would ensure that incomplete classes don't get called by spawned threads, and makes it possible for test-time subclasses to skip thread startup. I've not attempted to do that in this patch.\r\n* No tests for this yet. Hitting the three metrics servlets in the HistoryServer is the obvious route; the JSON payload of the metrics can be parsed and scanned for relevant counters too. \r\n* Part of the patch for `HistoryServerSuite` removes the call to `HistoryServer.initialize()` the `before` clause. That was a duplicate call, one which hit the re-entrancy tests on the provider & registry. As well as cutting it, `HistoryServer.initialize()` has been made idempotent. That should not be needed -but it will eliminate the problem arising again.\r\n\r\nOnce the SPARK-1537 YARN timeline server history provider is committed, then I'll add metrics support there too. The YARN timeline provider would:\r\n\r\n1. Add timers of REST operations as well as playback load times, which can count network delays as well as JSON deserialization overhead. \r\n2. Add a health check for connectivity too: the timeline server would be unhealthy if connections to the timeline server were either blocking or failing. And again, if there were security/auth problems, they'd be considered non-recoverable.\r\n3. Move thread launch under the `start()` method, with some test subclasses disabling thread launch.\r\n"},"comment":{"url":"https://api.github.com/repos/apache/spark/issues/comments/219020127","html_url":"https://github.com/apache/spark/pull/9571#issuecomment-219020127","issue_url":"https://api.github.com/repos/apache/spark/issues/9571","id":219020127,"user":{"login":"AmplabJenkins","id":4031291,"avatar_url":"https://avatars.githubusercontent.com/u/4031291?v=3","gravatar_id":"","url":"https://api.github.com/users/AmplabJenkins","html_url":"https://github.com/AmplabJenkins","followers_url":"https://api.github.com/users/AmplabJenkins/followers","following_url":"https://api.github.com/users/AmplabJenkins/following{/other_user}","gists_url":"https://api.github.com/users/AmplabJenkins/gists{/gist_id}","starred_url":"https://api.github.com/users/AmplabJenkins/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AmplabJenkins/subscriptions","organizations_url":"https://api.github.com/users/AmplabJenkins/orgs","repos_url":"https://api.github.com/users/AmplabJenkins/repos","events_url":"https://api.github.com/users/AmplabJenkins/events{/privacy}","received_events_url":"https://api.github.com/users/AmplabJenkins/received_events","type":"User","site_admin":false},"created_at":"2016-05-13T11:39:06Z","updated_at":"2016-05-13T11:39:06Z","body":"Merged build finished. Test PASSed."}}