{"action":"created","issue":{"url":"https://api.github.com/repos/broadinstitute/picard/issues/647","repository_url":"https://api.github.com/repos/broadinstitute/picard","labels_url":"https://api.github.com/repos/broadinstitute/picard/issues/647/labels{/name}","comments_url":"https://api.github.com/repos/broadinstitute/picard/issues/647/comments","events_url":"https://api.github.com/repos/broadinstitute/picard/issues/647/events","html_url":"https://github.com/broadinstitute/picard/issues/647","id":175788618,"number":647,"title":"Soliciting feedback on plans to merge Picard + GATK4","user":{"login":"vdauwera","id":2748435,"avatar_url":"https://avatars.githubusercontent.com/u/2748435?v=3","gravatar_id":"","url":"https://api.github.com/users/vdauwera","html_url":"https://github.com/vdauwera","followers_url":"https://api.github.com/users/vdauwera/followers","following_url":"https://api.github.com/users/vdauwera/following{/other_user}","gists_url":"https://api.github.com/users/vdauwera/gists{/gist_id}","starred_url":"https://api.github.com/users/vdauwera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdauwera/subscriptions","organizations_url":"https://api.github.com/users/vdauwera/orgs","repos_url":"https://api.github.com/users/vdauwera/repos","events_url":"https://api.github.com/users/vdauwera/events{/privacy}","received_events_url":"https://api.github.com/users/vdauwera/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"vdauwera","id":2748435,"avatar_url":"https://avatars.githubusercontent.com/u/2748435?v=3","gravatar_id":"","url":"https://api.github.com/users/vdauwera","html_url":"https://github.com/vdauwera","followers_url":"https://api.github.com/users/vdauwera/followers","following_url":"https://api.github.com/users/vdauwera/following{/other_user}","gists_url":"https://api.github.com/users/vdauwera/gists{/gist_id}","starred_url":"https://api.github.com/users/vdauwera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdauwera/subscriptions","organizations_url":"https://api.github.com/users/vdauwera/orgs","repos_url":"https://api.github.com/users/vdauwera/repos","events_url":"https://api.github.com/users/vdauwera/events{/privacy}","received_events_url":"https://api.github.com/users/vdauwera/received_events","type":"User","site_admin":false},"assignees":[{"login":"vdauwera","id":2748435,"avatar_url":"https://avatars.githubusercontent.com/u/2748435?v=3","gravatar_id":"","url":"https://api.github.com/users/vdauwera","html_url":"https://github.com/vdauwera","followers_url":"https://api.github.com/users/vdauwera/followers","following_url":"https://api.github.com/users/vdauwera/following{/other_user}","gists_url":"https://api.github.com/users/vdauwera/gists{/gist_id}","starred_url":"https://api.github.com/users/vdauwera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdauwera/subscriptions","organizations_url":"https://api.github.com/users/vdauwera/orgs","repos_url":"https://api.github.com/users/vdauwera/repos","events_url":"https://api.github.com/users/vdauwera/events{/privacy}","received_events_url":"https://api.github.com/users/vdauwera/received_events","type":"User","site_admin":false}],"milestone":null,"comments":5,"created_at":"2016-09-08T15:41:59Z","updated_at":"2016-09-09T17:03:33Z","closed_at":null,"body":"For many years now we’ve been hearing from users of both GATK and Picard about how they’d love to see the two projects **unite into a single \"toolkit-to-rule-them-all\"**, for the sake of **user convenience**, to promote **consistency across tools**, and to **minimize duplication of effort**. \r\n\r\nWith the advent of **GATK 4** this suddenly became **a real possibility**, as the decision was made to start the new GATK codebase from the Picard base classes rather than the old GATK 3.x base classes. This allows for **free-form Picard-style tools and GATK “walkers”** built upon an engine traversal to **peacefully co-exist** within the same framework. Last year, a Picard engineer successfully **ported all Picard tools to the GATK 4 codebase** with only minor changes to the tools themselves. More recently, efforts have been made to **harmonize the build systems** of the two projects, resulting in Picard’s recent move to gradle. \r\n\r\nImportantly, the **core GATK 4 codebase** at https://github.com/broadinstitute/gatk is released entirely under the **BSD 3-clause license**, a big improvement over the confusing licensing situation in GATK 3.x, with its mix of open-source and proprietary licenses within the same repository -- and that is where any Picard tools moved to the GATK 4 codebase would live, **remaining fully open-sourced and free for all**.\r\n\r\nAs **all of the technical pieces are now in place** to allow for a merger of the two projects (with the guarantee that the open-source nature of Picard code will be preserved) we are **soliciting feedback from the Picard developer community** about the prospect of a union with GATK. Would people here be generally in favor of such a move? Are there any strong objections to this idea? Any concerns that should be addressed *before* we head any further down this path?\r\n\r\nTo help start the discussion, **here is the case in favor of a GATK4 + Picard merger** as we see it:\r\n\r\n- It would greatly **cut down on duplication of effort** and re-invention of functionality already existing in one project or the other -- Picard devs could directly use open-source GATK functionality/utilities, and vice versa.\r\n- **Bug fixes and new features** such as CRAM support could **more easily propagate** to all the tools in the GATK+Picard ecosystem, and the tools overall could be made **more consistent and easier to use**.\r\n- Picard and GATK devs could better **share knowledge and give each other feedback**, and with more eyes looking at the code and more shared developer resources available the **overall quality of the tools should increase**.\r\n- **End-users’ lives** would be greatly simplified by having **all of the tools in one place**, with a unified interface, a common set of conventions and a single place to go to for documentation, help and updates. \r\n- Along similar lines, the ability of the GATK/Picard support team to **provide in-depth support for the tools** would be greatly enhanced. Among other things this would speed up plans to provide **developer support and API documentation**. \r\n- Only **minimal changes to the Picard tools** would be required. The biggest change would be a switch to standard POSIX-style command-line arguments (eg., `--argName value` instead of `argName=value`). \r\n- The **work of porting has already largely been done**, and in the process, test coverage of the ported versions of the Picard tools has been improved and bugs have been fixed.\r\n\r\nIn short: we’re stronger together than apart, right? :)\r\n\r\nWe realize that such a move is never trivial, but we're prepared to put resources into making the migration as painless as possible for everyone, and we believe that it would take us all to a better situation than what we have right now. \r\n\r\nWhat do you think? Let the discussion begin!"},"comment":{"url":"https://api.github.com/repos/broadinstitute/picard/issues/comments/245975035","html_url":"https://github.com/broadinstitute/picard/issues/647#issuecomment-245975035","issue_url":"https://api.github.com/repos/broadinstitute/picard/issues/647","id":245975035,"user":{"login":"vdauwera","id":2748435,"avatar_url":"https://avatars.githubusercontent.com/u/2748435?v=3","gravatar_id":"","url":"https://api.github.com/users/vdauwera","html_url":"https://github.com/vdauwera","followers_url":"https://api.github.com/users/vdauwera/followers","following_url":"https://api.github.com/users/vdauwera/following{/other_user}","gists_url":"https://api.github.com/users/vdauwera/gists{/gist_id}","starred_url":"https://api.github.com/users/vdauwera/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/vdauwera/subscriptions","organizations_url":"https://api.github.com/users/vdauwera/orgs","repos_url":"https://api.github.com/users/vdauwera/repos","events_url":"https://api.github.com/users/vdauwera/events{/privacy}","received_events_url":"https://api.github.com/users/vdauwera/received_events","type":"User","site_admin":false},"created_at":"2016-09-09T17:03:32Z","updated_at":"2016-09-09T17:03:32Z","body":"OK, so far we've heard from people on the following concerns; let me know if I'm leaving out anything:\r\n\r\n### Migration pain\r\n\r\nThe idea of maintaining two argument parsing styles is controversial -- we'd love to get more opinions on this. \r\n\r\nFor end-users, we're going to be going through some migration pain anyway to help GATK users transition to GATK4. One thing that should help is that we'll be providing lots of example/working pipelines. We're investing heavily in support on this front. \r\n\r\nFor developers, I think the best thing we can do is produce developer docs, starting with a detailed migration guide. \r\n\r\n### Build time\r\n\r\nSee @droazen's comment above -- building GATK4 is akin to building Picard. We've learned a lot from years of pain with classic GATK, and we're not going back. \r\n\r\n### Licensing / confusion around OSS portion of GATK / GATK4\r\n\r\nDefinitely something we can address with better communication. We haven't talked much about this so far because it was still taking shape, but now a clear picture is emerging. I'll try to write up a full description in the near future that can serve as \"official Broad statement\" (with lawyer backing!). In the meantime, hopefully this will help: \r\n\r\nThe open-source core of GATK4, which I'll call GATK-core from here onwards, encompasses the engine, all the boilerplate, infrastructure, public APIs etc that an external developer would want to develop against, plus Picard tools that have been ported, plus many GATK tools that have been open-sourced (such as the BQSR and VQSR tools). That's the entire codebase you can find at https://github.com/broadinstitute/gatk. In addition to that, there is a separate codebase at https://github.com/broadinstitute/gatk-protected that contains proprietary code for select tools such as HaplotypeCaller and the new CNV tools. \r\n\r\n### Governance model of GATK4-core\r\n\r\nAgreed, this is something we should figure out. Would anyone like to suggest a particular model, or namedrop a project that we should look to for inspiration?\r\n\r\n### Specific changes like metrics formats\r\n\r\nEstablished Picard metrics formats will continue to be supported. Picard outshines GATK on metrics collection, and wherever there is overlap, I expect we'll go with the Picard equivalent. \r\n\r\n"}}
{"actor":{"display_login":"vdauwera"}}