title: spark MetricsSystem registerSource failed
body: <p>when I running spark job, I encountered the following problems.</p>
  <pre><code>java.lang.ArrayIndexOutOfBoundsException
          at scala.collection.mutable.ResizableArray$class.ensureSize(ResizableArray.scala:104)
          at scala.collection.mutable.ArrayBuffer.ensureSize(ArrayBuffer.scala:48)
          at scala.collection.mutable.ArrayBuffer.$plus$eq(ArrayBuffer.scala:84)
          at org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:159)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:412)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  <pre><code>java.lang.IndexOutOfBoundsException: 34
          at scala.collection.mutable.ResizableArray$class.apply(ResizableArray.scala:43)
          at scala.collection.mutable.ArrayBuffer.apply(ArrayBuffer.scala:48)
          at scala.collection.IndexedSeqOptimized$class.segmentLength(IndexedSeqOptimized.scala:195)
          at scala.collection.mutable.ArrayBuffer.segmentLength(ArrayBuffer.scala:48)
          at scala.collection.IndexedSeqOptimized$class.indexWhere(IndexedSeqOptimized.scala:204)
          at scala.collection.mutable.ArrayBuffer.indexWhere(ArrayBuffer.scala:48)
          at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:145)
          at scala.collection.AbstractSeq.indexOf(Seq.scala:41)
          at scala.collection.GenSeqLike$class.indexOf(GenSeqLike.scala:129)
          at scala.collection.AbstractSeq.indexOf(Seq.scala:41)
          at scala.collection.mutable.BufferLike$class.$minus$eq(BufferLike.scala:127)
          at scala.collection.mutable.AbstractBuffer.$minus$eq(Buffer.scala:49)
          at org.apache.spark.metrics.MetricsSystem.removeSource(MetricsSystem.scala:169)
          at org.apache.spark.executor.Executor$TaskRunner$$anonfun$1.apply$mcV$sp(Executor.scala:425)
          at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1412)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:421)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  <p>when my task retried more than four times, the job was killed, error msg like this:</p>
  <pre><code>Exception in thread &quot;main&quot; org.apache.spark.SparkException: Job aborted due to stage failure: Task 24 in stage 389.0 failed 4 times, most recent failure: Lost task 24.3 in stage 389.0 (TID 54433, 10.160.140.73, executor 87): java.lang.ArrayIndexOutOfBoundsException
          at scala.collection.mutable.ResizableArray$class.ensureSize(ResizableArray.scala:104)
          at scala.collection.mutable.ArrayBuffer.ensureSize(ArrayBuffer.scala:48)
          at scala.collection.mutable.ArrayBuffer.$plus$eq(ArrayBuffer.scala:84)
          at org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:159)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:412)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  
  Driver stacktrace:
          at org.apache.spark.scheduler.DAGScheduler.org$apache$spark$scheduler$DAGScheduler$$failJobAndIndependentStages(DAGScheduler.scala:1891)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1879)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$abortStage$1.apply(DAGScheduler.scala:1878)
          at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
          at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
          at org.apache.spark.scheduler.DAGScheduler.abortStage(DAGScheduler.scala:1878)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)
          at org.apache.spark.scheduler.DAGScheduler$$anonfun$handleTaskSetFailed$1.apply(DAGScheduler.scala:927)
          at scala.Option.foreach(Option.scala:257)
          at org.apache.spark.scheduler.DAGScheduler.handleTaskSetFailed(DAGScheduler.scala:927)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.doOnReceive(DAGScheduler.scala:2112)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2061)
          at org.apache.spark.scheduler.DAGSchedulerEventProcessLoop.onReceive(DAGScheduler.scala:2050)
          at org.apache.spark.util.EventLoop$$anon$1.run(EventLoop.scala:49)
          at org.apache.spark.scheduler.DAGScheduler.runJob(DAGScheduler.scala:738)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:2064)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:2085)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:2104)
          at org.apache.spark.SparkContext.runJob(SparkContext.scala:2129)
          at org.apache.spark.rdd.RDD$$anonfun$collect$1.apply(RDD.scala:1000)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:151)
          at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:112)
          at org.apache.spark.rdd.RDD.withScope(RDD.scala:385)
          at org.apache.spark.rdd.RDD.collect(RDD.scala:999)
          at org.apache.spark.api.java.JavaRDDLike$class.collect(JavaRDDLike.scala:361)
          at org.apache.spark.api.java.AbstractJavaRDDLike.collect(JavaRDDLike.scala:45)
          at org.qihoo.cluster.XgbostCluster.cluster(XgbostCluster.java:316)
          at org.qihoo.cluster.XgbostCluster.start(XgbostCluster.java:106)
          at org.qihoo.App.main(App.java:80)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
          at java.lang.reflect.Method.invoke(Method.java:497)
          at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
          at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:852)
          at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:161)
          at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:184)
          at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
          at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:927)
          at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:936)
          at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.ArrayIndexOutOfBoundsException
          at scala.collection.mutable.ResizableArray$class.ensureSize(ResizableArray.scala:104)
          at scala.collection.mutable.ArrayBuffer.ensureSize(ArrayBuffer.scala:48)
          at scala.collection.mutable.ArrayBuffer.$plus$eq(ArrayBuffer.scala:84)
          at org.apache.spark.metrics.MetricsSystem.registerSource(MetricsSystem.scala:159)
          at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:412)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
          at java.lang.Thread.run(Thread.java:748)
  </code></pre>
  <p>I think it may be same cause with this jira <a href="https://issues.apache.org/jira/browse/SPARK-34934" rel="nofollow noreferrer">Race condition while registering source in MetricsSystem</a>, but I am not sure.I wonder if I have some way to resolve this problem.
  my spark version is 2.4.5 and i can not change its version.</p>