{"action":"created","issue":{"url":"https://api.github.com/repos/broadinstitute/gatk/issues/4661","repository_url":"https://api.github.com/repos/broadinstitute/gatk","labels_url":"https://api.github.com/repos/broadinstitute/gatk/issues/4661/labels{/name}","comments_url":"https://api.github.com/repos/broadinstitute/gatk/issues/4661/comments","events_url":"https://api.github.com/repos/broadinstitute/gatk/issues/4661/events","html_url":"https://github.com/broadinstitute/gatk/issues/4661","id":314682189,"node_id":"MDU6SXNzdWUzMTQ2ODIxODk=","number":4661,"title":"GenotypeLikelihoodCalculators ArrayIndexOutOfBoundsException with HaplotypeCallerSpark","user":{"login":"chapmanb","id":39391,"node_id":"MDQ6VXNlcjM5Mzkx","avatar_url":"https://avatars1.githubusercontent.com/u/39391?v=4","gravatar_id":"","url":"https://api.github.com/users/chapmanb","html_url":"https://github.com/chapmanb","followers_url":"https://api.github.com/users/chapmanb/followers","following_url":"https://api.github.com/users/chapmanb/following{/other_user}","gists_url":"https://api.github.com/users/chapmanb/gists{/gist_id}","starred_url":"https://api.github.com/users/chapmanb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chapmanb/subscriptions","organizations_url":"https://api.github.com/users/chapmanb/orgs","repos_url":"https://api.github.com/users/chapmanb/repos","events_url":"https://api.github.com/users/chapmanb/events{/privacy}","received_events_url":"https://api.github.com/users/chapmanb/received_events","type":"User","site_admin":false},"labels":[{"id":947454624,"node_id":"MDU6TGFiZWw5NDc0NTQ2MjQ=","url":"https://api.github.com/repos/broadinstitute/gatk/labels/HaplotypeCallerSpark","name":"HaplotypeCallerSpark","color":"5cd15f","default":false},{"id":226129596,"node_id":"MDU6TGFiZWwyMjYxMjk1OTY=","url":"https://api.github.com/repos/broadinstitute/gatk/labels/Spark","name":"Spark","color":"FFD700","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-04-16T14:42:42Z","updated_at":"2018-07-30T13:57:08Z","closed_at":null,"author_association":"NONE","body":"I'm running in intermittent issues when running HaplotypeCallerSpark with GATK 4.0.3.0 and was hoping to generate ideas to debug further. The underlying error is an index error when calculating likelihoods:\r\n```\r\njava.lang.ArrayIndexOutOfBoundsException: 4\r\n\tat org.broadinstitute.hellbender.tools.walkers.genotyper.GenotypeLikelihoodCalculators.calculateGenotypeCountUsingTables(GenotypeLikelihoodCalculators.java:388)\r\n```\r\nI've been unable to generate a reproducible test case. Re-running on the same machine (Amazon m4.4xlarge instances with 16 cores and 64Gb of memory) works. I've seen the error on two different datasets but it happens infrequently as I've also run hundreds using the same setup without any exceptions.\r\n\r\nThe only other thing I spot when looking through the traceback is block issues about the RDDs but I'm not sure if these are a symptom of the failure or a cause:\r\n```\r\n18/04/15 03:55:19 WARN BlockManager: Putting block rdd_18_12 failed due to an exception\r\n18/04/15 03:55:19 WARN BlockManager: Block rdd_18_12 could not be removed as it was not found on disk or in memory\r\n```\r\nHere's the full traceback of the failure:\r\n```\r\n[2018-04-15T03:55Z] ip-10-0-0-57: 18/04/15 03:55:19 WARN BlockManager: Putting block rdd_18_12 failed due to an exception\r\n[2018-04-15T03:55Z] ip-10-0-0-57: 18/04/15 03:55:19 WARN BlockManager: Block rdd_18_12 could not be removed as it was not found on disk or in memory\r\n[2018-04-15T03:55Z] ip-10-0-0-57: 18/04/15 03:55:19 ERROR Executor: Exception in task 12.0 in stage 7.0 (TID 828)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: java.lang.ArrayIndexOutOfBoundsException: 4\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.tools.walkers.genotyper.GenotypeLikelihoodCalculators.calculateGenotypeCountUsingTables(GenotypeLikelihoodCalculators.java:388)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.tools.walkers.genotyper.GenotypeLikelihoodCalculators.getInstance(GenotypeLikelihoodCalculators.java:263)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.utils.variant.GATKVariantContextUtils.makeGenotypeCall(GATKVariantContextUtils.java:282)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.tools.walkers.genotyper.AlleleSubsettingUtils.subsetAlleles(AlleleSubsettingUtils.java:84)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.tools.walkers.genotyper.GenotypingEngine.calculateGenotypes(GenotypingEngine.java:296)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.tools.walkers.genotyper.GenotypingEngine.calculateGenotypes(GenotypingEngine.java:210)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerGenotypingEngine.assignGenotypeLikelihoods(HaplotypeCallerGenotypingEngine.java:150)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.tools.walkers.haplotypecaller.HaplotypeCallerEngine.callRegion(HaplotypeCallerEngine.java:565)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.broadinstitute.hellbender.tools.HaplotypeCallerSpark.lambda$regionToVariants$1(HaplotypeCallerSpark.java:271)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:169)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat scala.collection.convert.Wrappers$JIteratorWrapper.hasNext(Wrappers.scala:42)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.storage.memory.MemoryStore.putIteratorAsValues(MemoryStore.scala:215)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1038)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:1029)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:969)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:1029)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:760)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:334)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.RDD.iterator(RDD.scala:285)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:38)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:323)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.rdd.RDD.iterator(RDD.scala:287)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.scheduler.Task.run(Task.scala:108)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:335)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\r\n[2018-04-15T03:55Z] ip-10-0-0-57: \tat java.lang.Thread.run(Thread.java:745)\r\n```\r\nand in case I am missing anything in how I'm calling HaplotypeCallerSpark, here is the full command line we're using:\r\n```\r\ngatk-launch --java-options '-Xms1000m -Xmx46965m -Djava.io.tmpdir=/mnt/work/cwl/bcbio_validation_workflows/giab-joint/bunny_work/main-giab-joint-2018-04-14-195952.723/root/variantcall/2/variantcall_batch_region/3/bcbiotx/tmpno7wyh' HaplotypeCallerSpark --reference /mnt/work/cwl/bcbio_validation_workflows/giab-joint/biodata/collections/hg38/ucsc/hg38.2bit --annotation MappingQualityRankSumTest --annotation MappingQualityZero --annotation QualByDepth --annotation ReadPosRankSumTest --annotation RMSMappingQuality --annotation BaseQualityRankSumTest --annotation FisherStrand --annotation MappingQuality --annotation DepthPerAlleleBySample --annotation Coverage -I /mnt/work/cwl/bcbio_validation_workflows/giab-joint/bunny_work/main-giab-joint-2018-04-14-195952.723/root/alignment/1/process_alignment/align/NA24385/NA24385-sort.bam -L /mnt/work/cwl/bcbio_validation_workflows/giab-joint/bunny_work/main-giab-joint-2018-04-14-195952.723/root/variantcall/2/variantcall_batch_region/3/gatk-haplotype/chr15/NA24385-chr15_0_101977614-block-regions.bed --interval-set-rule INTERSECTION --spark-master local[16] --conf spark.local.dir=/mnt/work/cwl/bcbio_validation_workflows/giab-joint/bunny_work/main-giab-joint-2018-04-14-195952.723/root/variantcall/2/variantcall_batch_region/3/bcbiotx/tmpno7wyh --annotation ClippingRankSumTest --annotation DepthPerSampleHC --emit-ref-confidence GVCF -GQB 10 -GQB 20 -GQB 30 -GQB 40 -GQB 60 -GQB 80 --output /mnt/work/cwl/bcbio_validation_workflows/giab-joint/bunny_work/main-giab-joint-2018-04-14-195952.723/root/variantcall/2/variantcall_batch_region/3/bcbiotx/tmpno7wyh/NA24385-chr15_0_101977614-block.vcf\r\n```\r\nThanks so much for any clues about how to debug further or avoid the issue."},"comment":{"url":"https://api.github.com/repos/broadinstitute/gatk/issues/comments/408872373","html_url":"https://github.com/broadinstitute/gatk/issues/4661#issuecomment-408872373","issue_url":"https://api.github.com/repos/broadinstitute/gatk/issues/4661","id":408872373,"node_id":"MDEyOklzc3VlQ29tbWVudDQwODg3MjM3Mw==","user":{"login":"tomwhite","id":85085,"node_id":"MDQ6VXNlcjg1MDg1","avatar_url":"https://avatars3.githubusercontent.com/u/85085?v=4","gravatar_id":"","url":"https://api.github.com/users/tomwhite","html_url":"https://github.com/tomwhite","followers_url":"https://api.github.com/users/tomwhite/followers","following_url":"https://api.github.com/users/tomwhite/following{/other_user}","gists_url":"https://api.github.com/users/tomwhite/gists{/gist_id}","starred_url":"https://api.github.com/users/tomwhite/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/tomwhite/subscriptions","organizations_url":"https://api.github.com/users/tomwhite/orgs","repos_url":"https://api.github.com/users/tomwhite/repos","events_url":"https://api.github.com/users/tomwhite/events{/privacy}","received_events_url":"https://api.github.com/users/tomwhite/received_events","type":"User","site_admin":false},"created_at":"2018-07-30T13:57:08Z","updated_at":"2018-07-30T13:57:08Z","author_association":"COLLABORATOR","body":"It could be a synchronization issue. E.g. if thread A is asking for an alleleCount of 3 and and thread B an alleleCount of 4, then thread A could grow the array to 3 after thread B grows it to 4 (meaning the array is grown to size 4 but then set back to size 3), but before thread B reads position 4. This read will then fail.\r\n\r\nBTW I was seeing quite a lot of task failures, around 10%."}}
{"actor":{"display_login":"tomwhite"}}