{"action":"created","issue":{"url":"https://api.github.com/repos/broadinstitute/gatk/issues/4746","repository_url":"https://api.github.com/repos/broadinstitute/gatk","labels_url":"https://api.github.com/repos/broadinstitute/gatk/issues/4746/labels{/name}","comments_url":"https://api.github.com/repos/broadinstitute/gatk/issues/4746/comments","events_url":"https://api.github.com/repos/broadinstitute/gatk/issues/4746/events","html_url":"https://github.com/broadinstitute/gatk/issues/4746","id":321192955,"number":4746,"title":"Make it easier to use native Hadoop libraries in GATK Spark tools running in local mode","user":{"login":"cwhelan","id":709477,"avatar_url":"https://avatars2.githubusercontent.com/u/709477?v=4","gravatar_id":"","url":"https://api.github.com/users/cwhelan","html_url":"https://github.com/cwhelan","followers_url":"https://api.github.com/users/cwhelan/followers","following_url":"https://api.github.com/users/cwhelan/following{/other_user}","gists_url":"https://api.github.com/users/cwhelan/gists{/gist_id}","starred_url":"https://api.github.com/users/cwhelan/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/cwhelan/subscriptions","organizations_url":"https://api.github.com/users/cwhelan/orgs","repos_url":"https://api.github.com/users/cwhelan/repos","events_url":"https://api.github.com/users/cwhelan/events{/privacy}","received_events_url":"https://api.github.com/users/cwhelan/received_events","type":"User","site_admin":false},"labels":[{"id":226129596,"url":"https://api.github.com/repos/broadinstitute/gatk/labels/Spark","name":"Spark","color":"FFD700","default":false},{"id":241205543,"url":"https://api.github.com/repos/broadinstitute/gatk/labels/performance","name":"performance","color":"bfd4f2","default":false}],"state":"open","locked":false,"assignee":{"login":"lbergelson","id":4700332,"avatar_url":"https://avatars3.githubusercontent.com/u/4700332?v=4","gravatar_id":"","url":"https://api.github.com/users/lbergelson","html_url":"https://github.com/lbergelson","followers_url":"https://api.github.com/users/lbergelson/followers","following_url":"https://api.github.com/users/lbergelson/following{/other_user}","gists_url":"https://api.github.com/users/lbergelson/gists{/gist_id}","starred_url":"https://api.github.com/users/lbergelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lbergelson/subscriptions","organizations_url":"https://api.github.com/users/lbergelson/orgs","repos_url":"https://api.github.com/users/lbergelson/repos","events_url":"https://api.github.com/users/lbergelson/events{/privacy}","received_events_url":"https://api.github.com/users/lbergelson/received_events","type":"User","site_admin":false},"assignees":[{"login":"lbergelson","id":4700332,"avatar_url":"https://avatars3.githubusercontent.com/u/4700332?v=4","gravatar_id":"","url":"https://api.github.com/users/lbergelson","html_url":"https://github.com/lbergelson","followers_url":"https://api.github.com/users/lbergelson/followers","following_url":"https://api.github.com/users/lbergelson/following{/other_user}","gists_url":"https://api.github.com/users/lbergelson/gists{/gist_id}","starred_url":"https://api.github.com/users/lbergelson/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lbergelson/subscriptions","organizations_url":"https://api.github.com/users/lbergelson/orgs","repos_url":"https://api.github.com/users/lbergelson/repos","events_url":"https://api.github.com/users/lbergelson/events{/privacy}","received_events_url":"https://api.github.com/users/lbergelson/received_events","type":"User","site_admin":false},{"login":"jamesemery","id":16102845,"avatar_url":"https://avatars2.githubusercontent.com/u/16102845?v=4","gravatar_id":"","url":"https://api.github.com/users/jamesemery","html_url":"https://github.com/jamesemery","followers_url":"https://api.github.com/users/jamesemery/followers","following_url":"https://api.github.com/users/jamesemery/following{/other_user}","gists_url":"https://api.github.com/users/jamesemery/gists{/gist_id}","starred_url":"https://api.github.com/users/jamesemery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamesemery/subscriptions","organizations_url":"https://api.github.com/users/jamesemery/orgs","repos_url":"https://api.github.com/users/jamesemery/repos","events_url":"https://api.github.com/users/jamesemery/events{/privacy}","received_events_url":"https://api.github.com/users/jamesemery/received_events","type":"User","site_admin":false}],"milestone":{"url":"https://api.github.com/repos/broadinstitute/gatk/milestones/20","html_url":"https://github.com/broadinstitute/gatk/milestone/20","labels_url":"https://api.github.com/repos/broadinstitute/gatk/milestones/20/labels","id":2843981,"number":20,"title":"Engine-2Q2018","description":"","creator":{"login":"droazen","id":798637,"avatar_url":"https://avatars0.githubusercontent.com/u/798637?v=4","gravatar_id":"","url":"https://api.github.com/users/droazen","html_url":"https://github.com/droazen","followers_url":"https://api.github.com/users/droazen/followers","following_url":"https://api.github.com/users/droazen/following{/other_user}","gists_url":"https://api.github.com/users/droazen/gists{/gist_id}","starred_url":"https://api.github.com/users/droazen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/droazen/subscriptions","organizations_url":"https://api.github.com/users/droazen/orgs","repos_url":"https://api.github.com/users/droazen/repos","events_url":"https://api.github.com/users/droazen/events{/privacy}","received_events_url":"https://api.github.com/users/droazen/received_events","type":"User","site_admin":false},"open_issues":180,"closed_issues":11,"state":"open","created_at":"2017-10-17T17:04:12Z","updated_at":"2018-05-08T16:05:25Z","due_on":"2018-06-30T07:00:00Z","closed_at":null},"comments":2,"created_at":"2018-05-08T13:45:07Z","updated_at":"2018-05-08T19:45:03Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"In some preliminary testing I've done it looks like using native Hadoop libraries can speed up tools running in Spark local mode. In private Spark tools under development (which travers a WGS BAM and then performing several shuffles) I have seen speedups of up to 40% (~ 46 minutes -> 26 minutes). An initial test of `MarkDuplicatesSpark` using a 30GB bam file gave me a 9% speedup (logs are below). It might be good to investigate making this easier for users (I downloaded Hadoop and built it from source, and then set gatk's java opts to load the native library). Two options might be: 1) distribute native libraries for supported architectures with gatk or 2) make sure gatk docker images include the native libraries and are set to use them.\r\n\r\nLogs for `MarkDuplicatesSpark` without and with native libraries, running on a Broad login server:\r\n\r\nWithout:\r\n\r\n```\r\n$ ${GATK_DIR}/gatk MarkDuplicatesSpark -I CEUTrio.HiSeq.WEx.b37.NA12892.readnamesort.bam -O CEUTrio.HiSeq.WEx37\r\n.NA12892.readnamesort.dupmarked.bam -- --spark-runner LOCAL --spark-master local[8]\r\nUsing GATK wrapper script ${GATK_DIR}/gatk/build/install/gatk/bin/gatk\r\nRunning:\r\n    ${GATK_DIR}/gatk/build/install/gatk/bin/gatk MarkDuplicatesSpark -I CEUTrio.HiSeq.WEx.b37.NA12892.readnamesort.bam -O CEUTrio.HiSeq.WEx.b37.NA12892.readnamesort.dupmarked.bam --spark\r\n-master local[8]\r\n14:40:21.800 WARN  SparkContextFactory - Environment variables HELLBENDER_TEST_PROJECT and HELLBENDER_JSON_SERVICE_ACCOUNT_KEY must be set or the GCS hadoop connector will not be configured properly\r\n14:40:21.889 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:${GATK_DIR}/gatk/build/install/gatk/lib/gkl-0.8.5.jar!/com/intel/gkl/native/libgkl_compression.so\r\n14:40:21.989 INFO  MarkDuplicatesSpark - ------------------------------------------------------------\r\n14:40:21.990 INFO  MarkDuplicatesSpark - The Genome Analysis Toolkit (GATK) v4.0.4.0-7-g46a8661-SNAPSHOT\r\n14:40:21.990 INFO  MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/\r\n14:40:21.991 INFO  MarkDuplicatesSpark - Executing as cwhelan@gsa6.broadinstitute.org on Linux v2.6.32-696.16.1.el6.x86_64 amd64\r\n14:40:21.991 INFO  MarkDuplicatesSpark - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_121-b13\r\n14:40:21.992 INFO  MarkDuplicatesSpark - Start Date/Time: May 7, 2018 2:40:21 PM EDT\r\n14:40:21.992 INFO  MarkDuplicatesSpark - ------------------------------------------------------------\r\n14:40:21.992 INFO  MarkDuplicatesSpark - ------------------------------------------------------------\r\n14:40:21.992 INFO  MarkDuplicatesSpark - HTSJDK Version: 2.14.3\r\n14:40:21.992 INFO  MarkDuplicatesSpark - Picard Version: 2.18.2\r\n14:40:21.993 INFO  MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2\r\n14:40:21.993 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false\r\n14:40:21.993 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true\r\n14:40:21.993 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false\r\n14:40:21.993 INFO  MarkDuplicatesSpark - Deflater: IntelDeflater\r\n14:40:21.993 INFO  MarkDuplicatesSpark - Inflater: IntelInflater\r\n14:40:21.993 INFO  MarkDuplicatesSpark - GCS max retries/reopens: 20\r\n14:40:21.993 INFO  MarkDuplicatesSpark - Using google-cloud-java patch 6d11bef1c81f885c26b2b56c8616b7a705171e4f from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes\r\n14:40:21.994 WARN  MarkDuplicatesSpark -\r\n\r\n   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\r\n\r\n   Warning: MarkDuplicatesSpark is a BETA tool and is not yet ready for use in production\r\n\r\n   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\r\n\r\n\r\n14:40:21.994 INFO  MarkDuplicatesSpark - Initializing engine\r\n14:40:21.994 INFO  MarkDuplicatesSpark - Done initializing engine\r\n14:40:22.338 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable\r\n15:24:12.735 INFO  ReadsSparkSink - Finished sorting the bam file and dumping read shards to disk, proceeding to merge the shards into a single file using the master thread\r\n15:41:27.766 INFO  ReadsSparkSink - Finished merging shards into a single output bam\r\n15:41:34.351 INFO  MarkDuplicatesSpark - Shutting down engine\r\n[May 7, 2018 3:41:34 PM EDT] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 61.21 minutes.\r\nRuntime.totalMemory()=13635682304\r\n```\r\n\r\nWith native libraries (note the lack of the usual warning):\r\n\r\n```\r\n$ ${GATK_DIR}/gatk MarkDuplicatesSpark --java-options \"-Djava.library.path=${HADOOP_DIR}/hadoop-2.6.5-src/hadoop-common-project/hadoop-common/target/hadoop-common-2.6.5/lib/native\" -I CEUTrio.HiSeq.WEx.b37.NA12892.readnamesort.bam -O CEUTrio.HiSeq.WEx.b37.NA12892.readnamesort.dupmarked_native.bam -- --spark-runner LOCAL --spark-master local[8]\r\nUsing GATK wrapper script ${GATK_DIR}/gatk/build/install/gatk/bin/gatk\r\nRunning:\r\n    ${GATK_DIR}/gatk/build/install/gatk/bin/gatk MarkDuplicatesSpark -I CEUTrio.HiSeq.WEx.b37.NA12892.readnamesort.bam -O CEUTrio.HiSeq.WEx.b37.NA12892.readnamesort.dupmarked_native.bam --spark-master local[8]\r\n21:47:47.494 WARN  SparkContextFactory - Environment variables HELLBENDER_TEST_PROJECT and HELLBENDER_JSON_SERVICE_ACCOUNT_KEY must be set or the GCS hadoop connector will not be configured properly\r\n21:47:47.827 INFO  NativeLibraryLoader - Loading libgkl_compression.so from jar:file:${GATK_DIR}/gatk/build/install/gatk/lib/gkl-0.8.5.jar!/com/intel/gkl/native/libgkl_compression.so\r\n21:47:48.268 INFO  MarkDuplicatesSpark - ------------------------------------------------------------\r\n21:47:48.268 INFO  MarkDuplicatesSpark - The Genome Analysis Toolkit (GATK) v4.0.4.0-7-g46a8661-SNAPSHOT\r\n21:47:48.268 INFO  MarkDuplicatesSpark - For support and documentation go to https://software.broadinstitute.org/gatk/\r\n21:47:48.270 INFO  MarkDuplicatesSpark - Executing as cwhelan@gsa6.broadinstitute.org on Linux v2.6.32-696.16.1.el6.x86_64 amd64\r\n21:47:48.270 INFO  MarkDuplicatesSpark - Java runtime: Java HotSpot(TM) 64-Bit Server VM v1.8.0_121-b13\r\n21:47:48.270 INFO  MarkDuplicatesSpark - Start Date/Time: May 7, 2018 9:47:47 PM EDT\r\n21:47:48.270 INFO  MarkDuplicatesSpark - ------------------------------------------------------------\r\n21:47:48.271 INFO  MarkDuplicatesSpark - ------------------------------------------------------------\r\n21:47:48.271 INFO  MarkDuplicatesSpark - HTSJDK Version: 2.14.3\r\n21:47:48.271 INFO  MarkDuplicatesSpark - Picard Version: 2.18.2\r\n21:47:48.271 INFO  MarkDuplicatesSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2\r\n21:47:48.271 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false\r\n21:47:48.272 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true\r\n21:47:48.272 INFO  MarkDuplicatesSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false\r\n21:47:48.272 INFO  MarkDuplicatesSpark - Deflater: IntelDeflater\r\n21:47:48.272 INFO  MarkDuplicatesSpark - Inflater: IntelInflater\r\n21:47:48.272 INFO  MarkDuplicatesSpark - GCS max retries/reopens: 20\r\n21:47:48.272 INFO  MarkDuplicatesSpark - Using google-cloud-java patch 6d11bef1c81f885c26b2b56c8616b7a705171e4f from https://github.com/droazen/google-cloud-java/tree/dr_all_nio_fixes\r\n21:47:48.272 WARN  MarkDuplicatesSpark -\r\n\r\n   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\r\n\r\n   Warning: MarkDuplicatesSpark is a BETA tool and is not yet ready for use in production\r\n\r\n   !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!\r\n\r\n\r\n21:47:48.273 INFO  MarkDuplicatesSpark - Initializing engine\r\n21:47:48.273 INFO  MarkDuplicatesSpark - Done initializing engine\r\n22:29:27.746 INFO  ReadsSparkSink - Finished sorting the bam file and dumping read shards to disk, proceeding to merge the shards into a single file using the master thread\r\n22:43:29.758 INFO  ReadsSparkSink - Finished merging shards into a single output bam\r\n22:43:36.475 INFO  MarkDuplicatesSpark - Shutting down engine\r\n[May 7, 2018 10:43:36 PM EDT] org.broadinstitute.hellbender.tools.spark.transforms.markduplicates.MarkDuplicatesSpark done. Elapsed time: 55.82 minutes.\r\nRuntime.totalMemory()=12430868480\r\n```\r\n\r\nCreated after discussion with @lbergelson "},"comment":{"url":"https://api.github.com/repos/broadinstitute/gatk/issues/comments/387519906","html_url":"https://github.com/broadinstitute/gatk/issues/4746#issuecomment-387519906","issue_url":"https://api.github.com/repos/broadinstitute/gatk/issues/4746","id":387519906,"user":{"login":"jamesemery","id":16102845,"avatar_url":"https://avatars2.githubusercontent.com/u/16102845?v=4","gravatar_id":"","url":"https://api.github.com/users/jamesemery","html_url":"https://github.com/jamesemery","followers_url":"https://api.github.com/users/jamesemery/followers","following_url":"https://api.github.com/users/jamesemery/following{/other_user}","gists_url":"https://api.github.com/users/jamesemery/gists{/gist_id}","starred_url":"https://api.github.com/users/jamesemery/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jamesemery/subscriptions","organizations_url":"https://api.github.com/users/jamesemery/orgs","repos_url":"https://api.github.com/users/jamesemery/repos","events_url":"https://api.github.com/users/jamesemery/events{/privacy}","received_events_url":"https://api.github.com/users/jamesemery/received_events","type":"User","site_admin":false},"created_at":"2018-05-08T19:45:03Z","updated_at":"2018-05-08T19:45:03Z","author_association":"COLLABORATOR","body":"This looks promising, at a minimum we should try setting up the docker with hadoop native libraries so the performance gains can be extended to most use cases. This might also include adding some magic to the gatk launch script inside the docker to detect and run with the correct version of the hadoop libraries.  "}}
{"actor":{"display_login":"jamesemery"}}