title: How to access spark executor, worker and master level metrics in databricks?
body: <p>My question is specific to databricks. I am trying to access spark metrics via graphite sink in <strong>databricks</strong> by passing below spark configuration and I want the configuration to be passed while creating a cluster.</p>
  
  <pre><code>spark.metrics.conf.*.sink.graphite.class org.apache.spark.metrics.sink.GraphiteSink
  spark.metrics.conf.*.sink.graphite.host myhost
  spark.metrics.conf.*.sink.graphite.port 2003
  spark.metrics.conf.*.sink.graphite.period 10
  spark.metrics.conf.*.sink.graphite.unit seconds
  spark.metrics.conf.*.source.jvm.class org.apache.spark.metrics.source.JvmSource
  </code></pre>
  
  <p>However, the above configuration only fetches driver level metrics. 
  I read in some posts that to get executor level metrics, one has to pass below parameters</p>
  
  <pre><code>spark-submit &lt;other parameters&gt; --files metrics.properties 
  --conf spark.metrics.conf=metrics.properties
  </code></pre>
  
  <p><strong>My question is how can I pass --files parameter in databricks while creating a cluster (as I am not doing spark-submit) or is there any other way to get executor, worker and master level metrics ?</strong></p>
  
  <p>Cluster Json</p>
  
  <pre><code>{
      "num_workers": 0,
      "cluster_name": "mycluster",
      "spark_version": "5.5.x-scala2.11",
      "spark_conf": {
          "spark.metrics.conf.*.sink.graphite.unit": "seconds",
          "spark.metrics.conf.*.sink.graphite.class": "org.apache.spark.metrics.sink.GraphiteSink",
          "spark.metrics.conf.*.sink.graphite.period": "10",
          "spark.databricks.delta.preview.enabled": "true",
          "spark.metrics.conf.*.source.jvm.class": "org.apache.spark.metrics.source.JvmSource",
          "spark.metrics.conf.*.sink.graphite.host": "myhost",
          "spark.metrics.conf.*.sink.graphite.port": "2003"
      },
      "aws_attributes": {
          "first_on_demand": 0,
          "availability": "ON_DEMAND",
          "zone_id": "us-west-2c",
          "spot_bid_price_percent": 100,
          "ebs_volume_count": 0
      },
      "node_type_id": "dev-tier-node",
      "driver_node_type_id": "dev-tier-node",
      "ssh_public_keys": [],
      "custom_tags": {},
      "spark_env_vars": {
          "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
      },
      "autotermination_minutes": 120,
      "enable_elastic_disk": false,
      "cluster_source": "UI",
      "init_scripts": [],
      "cluster_id": "0604-114345-oboe241"
  }
  </code></pre>