{"action":"closed","number":485,"pull_request":{"url":"https://api.github.com/repos/ga4gh/schemas/pulls/485","id":50203261,"html_url":"https://github.com/ga4gh/schemas/pull/485","diff_url":"https://github.com/ga4gh/schemas/pull/485.diff","patch_url":"https://github.com/ga4gh/schemas/pull/485.patch","issue_url":"https://api.github.com/repos/ga4gh/schemas/issues/485","number":485,"state":"closed","locked":false,"title":"Convert avdl -> proto3","user":{"login":"calbach","id":822298,"avatar_url":"https://avatars.githubusercontent.com/u/822298?v=3","gravatar_id":"","url":"https://api.github.com/users/calbach","html_url":"https://github.com/calbach","followers_url":"https://api.github.com/users/calbach/followers","following_url":"https://api.github.com/users/calbach/following{/other_user}","gists_url":"https://api.github.com/users/calbach/gists{/gist_id}","starred_url":"https://api.github.com/users/calbach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calbach/subscriptions","organizations_url":"https://api.github.com/users/calbach/orgs","repos_url":"https://api.github.com/users/calbach/repos","events_url":"https://api.github.com/users/calbach/events{/privacy}","received_events_url":"https://api.github.com/users/calbach/received_events","type":"User","site_admin":false},"body":"As alluded to in several GA4GH discussions and [issues](https://github.com/ga4gh/schemas/issues/343), this PR converts the GA4GH schemas from Avro to [protocol buffers](https://developers.google.com/protocol-buffers/docs/overview). This is ready to be discussed and reviewed, but a prerequisite for merging will be an agreed plan for corresponding changes to the reference server and compliance suite (in the works, but need an explicit plan for sequencing the PRs).\r\n\r\nI'll highlight a few factors which motivate this proposal:\r\n- Protocol buffers have defined JSON serialization, which was sorely missing in our use of Avro. Libraries for protobuf <-> JSON conversion are defined at http://github.com/google/protobuf. See also the [specification](https://developers.google.com/protocol-buffers/docs/proto3#json).\r\n- Gives the GA4GH a straightforward pathway to support an optimized binary wire format in the future.\r\n- (Slightly subjective) Generally more community adoption / tooling / client library support.\r\n\r\nAt a high level, my aim with this PR was to make the minimal possible functional / documentation / wire JSON / code structure change while still winding up with valid proto3. I have a list of natural follow-up changes given proto as the schema - but I think these make more sense as separate PRs.\r\n\r\nThat being said, there are some changes from Avro -> proto3 which are unavoidable:\r\n- proto3 does not have optional/default semantics, e.g. Avro's `union {string, null} foo = \"\"`\r\n  - In places, this may change the JSON wire format slightly - e.g. null -> \"\" or vice versa.\r\n  - As far as I can tell, the only place this really causes a problem is for `SearchVariantsRequest.callsetIds`, which will be resolved separately by https://github.com/ga4gh/schemas/pull/488.\r\n  - Technically you can still get \"nullable\" semantics by using a wrapper message, at the cost of ugliness and stutter in the API.\r\n  - If you're using the generated proto libraries, this shouldn't make a difference.\r\n- I converted all 'methods' into proto RPC service methods.\r\n  - For now, this would be for documentation purposes only, much as the Avro methods are.\r\n  - In order to support the GET operations, I've added small ‘request’ protos (with a single ID) for those endpoints, in addition to a referenceId on the ListBasesRequest.\r\n- All fields now have [tag numbers](https://developers.google.com/protocol-buffers/docs/proto3#assigning-tags).\r\n- In proto3, enum 0-values should be a sane 'unset' default.\r\n  - The reason behind this is that the 0-value is the default when you initialize the request proto, or if a client sends you nothing. In general, it is better to disambiguate this case.\r\n  - For this reason, added `${ENUM_NAME}_UNSPECIFIED` to all enums. This makes sense in most places, though feels the least natural in Cigar Operation.\r\n- map\\<[]string\\> becomes map\\<google.protobuf.ListValue\\>.\r\n  - This is the best way of generating consistent JSON with the previous approaches, but the ListValue message is actually more flexible than []string (elements could be number, message, etc).\r\n  - For now, the requirement of a `[]string` payload is enforced only by the comment.\r\n  - An alternative would be to relax this documentation to allow all values of `ListValue` or to use the more freeform google.protobuf.Struct, which represents arbitrary JSON.\r\n  - I don't have a strong opinion here, again this first pass is what I believe to be the minimal functional delta.\r\n- JSON wire format change: \"sourceURI\" -> \"sourceUri\"\r\n  - To maintain the old JSON field name, by the proto -> JSON conversion rules we'd have to name the field field \"source_u_r_i\".\r\n- In a few places, nested message definitions to make the hierarchy clearer e.g. CigarUnit { Operation {} operation }. This mainly aesthetic, though it also may impact the generated libraries.\r\n\r\nAlso, for reference, the Python generated classes can be created by the following process:\r\n1. Install protoc >3.0 from https://github.com/google/protobuf/releases\r\n2. `$ cd src/main/resources && protoc -I . --python_out=. proto/{,*/}*/*.proto`","created_at":"2015-11-10T01:31:52Z","updated_at":"2016-01-28T04:44:36Z","closed_at":"2016-01-28T04:44:36Z","merged_at":null,"merge_commit_sha":"74944ce11ef0962e81f506feffdf41986a7d24e5","assignee":null,"milestone":null,"commits_url":"https://api.github.com/repos/ga4gh/schemas/pulls/485/commits","review_comments_url":"https://api.github.com/repos/ga4gh/schemas/pulls/485/comments","review_comment_url":"https://api.github.com/repos/ga4gh/schemas/pulls/comments{/number}","comments_url":"https://api.github.com/repos/ga4gh/schemas/issues/485/comments","statuses_url":"https://api.github.com/repos/ga4gh/schemas/statuses/b501c77a85724380279efd370965234ddd7372ce","head":{"label":"calbach:proto3","ref":"proto3","sha":"b501c77a85724380279efd370965234ddd7372ce","user":{"login":"calbach","id":822298,"avatar_url":"https://avatars.githubusercontent.com/u/822298?v=3","gravatar_id":"","url":"https://api.github.com/users/calbach","html_url":"https://github.com/calbach","followers_url":"https://api.github.com/users/calbach/followers","following_url":"https://api.github.com/users/calbach/following{/other_user}","gists_url":"https://api.github.com/users/calbach/gists{/gist_id}","starred_url":"https://api.github.com/users/calbach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calbach/subscriptions","organizations_url":"https://api.github.com/users/calbach/orgs","repos_url":"https://api.github.com/users/calbach/repos","events_url":"https://api.github.com/users/calbach/events{/privacy}","received_events_url":"https://api.github.com/users/calbach/received_events","type":"User","site_admin":false},"repo":{"id":20236688,"name":"schemas","full_name":"calbach/schemas","owner":{"login":"calbach","id":822298,"avatar_url":"https://avatars.githubusercontent.com/u/822298?v=3","gravatar_id":"","url":"https://api.github.com/users/calbach","html_url":"https://github.com/calbach","followers_url":"https://api.github.com/users/calbach/followers","following_url":"https://api.github.com/users/calbach/following{/other_user}","gists_url":"https://api.github.com/users/calbach/gists{/gist_id}","starred_url":"https://api.github.com/users/calbach/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/calbach/subscriptions","organizations_url":"https://api.github.com/users/calbach/orgs","repos_url":"https://api.github.com/users/calbach/repos","events_url":"https://api.github.com/users/calbach/events{/privacy}","received_events_url":"https://api.github.com/users/calbach/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/calbach/schemas","description":"Work on data models and APIs for Genomic data.","fork":true,"url":"https://api.github.com/repos/calbach/schemas","forks_url":"https://api.github.com/repos/calbach/schemas/forks","keys_url":"https://api.github.com/repos/calbach/schemas/keys{/key_id}","collaborators_url":"https://api.github.com/repos/calbach/schemas/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/calbach/schemas/teams","hooks_url":"https://api.github.com/repos/calbach/schemas/hooks","issue_events_url":"https://api.github.com/repos/calbach/schemas/issues/events{/number}","events_url":"https://api.github.com/repos/calbach/schemas/events","assignees_url":"https://api.github.com/repos/calbach/schemas/assignees{/user}","branches_url":"https://api.github.com/repos/calbach/schemas/branches{/branch}","tags_url":"https://api.github.com/repos/calbach/schemas/tags","blobs_url":"https://api.github.com/repos/calbach/schemas/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/calbach/schemas/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/calbach/schemas/git/refs{/sha}","trees_url":"https://api.github.com/repos/calbach/schemas/git/trees{/sha}","statuses_url":"https://api.github.com/repos/calbach/schemas/statuses/{sha}","languages_url":"https://api.github.com/repos/calbach/schemas/languages","stargazers_url":"https://api.github.com/repos/calbach/schemas/stargazers","contributors_url":"https://api.github.com/repos/calbach/schemas/contributors","subscribers_url":"https://api.github.com/repos/calbach/schemas/subscribers","subscription_url":"https://api.github.com/repos/calbach/schemas/subscription","commits_url":"https://api.github.com/repos/calbach/schemas/commits{/sha}","git_commits_url":"https://api.github.com/repos/calbach/schemas/git/commits{/sha}","comments_url":"https://api.github.com/repos/calbach/schemas/comments{/number}","issue_comment_url":"https://api.github.com/repos/calbach/schemas/issues/comments{/number}","contents_url":"https://api.github.com/repos/calbach/schemas/contents/{+path}","compare_url":"https://api.github.com/repos/calbach/schemas/compare/{base}...{head}","merges_url":"https://api.github.com/repos/calbach/schemas/merges","archive_url":"https://api.github.com/repos/calbach/schemas/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/calbach/schemas/downloads","issues_url":"https://api.github.com/repos/calbach/schemas/issues{/number}","pulls_url":"https://api.github.com/repos/calbach/schemas/pulls{/number}","milestones_url":"https://api.github.com/repos/calbach/schemas/milestones{/number}","notifications_url":"https://api.github.com/repos/calbach/schemas/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/calbach/schemas/labels{/name}","releases_url":"https://api.github.com/repos/calbach/schemas/releases{/id}","deployments_url":"https://api.github.com/repos/calbach/schemas/deployments","created_at":"2014-05-27T21:51:58Z","updated_at":"2014-08-21T21:50:57Z","pushed_at":"2016-01-27T17:58:30Z","git_url":"git://github.com/calbach/schemas.git","ssh_url":"git@github.com:calbach/schemas.git","clone_url":"https://github.com/calbach/schemas.git","svn_url":"https://github.com/calbach/schemas","homepage":"","size":1125,"stargazers_count":0,"watchers_count":0,"language":null,"has_issues":false,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":1,"mirror_url":null,"open_issues_count":0,"forks":1,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"ga4gh:master","ref":"master","sha":"0cc27e05d4a6abb8d027476b38904dcd3cde8e4c","user":{"login":"ga4gh","id":6921777,"avatar_url":"https://avatars.githubusercontent.com/u/6921777?v=3","gravatar_id":"","url":"https://api.github.com/users/ga4gh","html_url":"https://github.com/ga4gh","followers_url":"https://api.github.com/users/ga4gh/followers","following_url":"https://api.github.com/users/ga4gh/following{/other_user}","gists_url":"https://api.github.com/users/ga4gh/gists{/gist_id}","starred_url":"https://api.github.com/users/ga4gh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ga4gh/subscriptions","organizations_url":"https://api.github.com/users/ga4gh/orgs","repos_url":"https://api.github.com/users/ga4gh/repos","events_url":"https://api.github.com/users/ga4gh/events{/privacy}","received_events_url":"https://api.github.com/users/ga4gh/received_events","type":"Organization","site_admin":false},"repo":{"id":17925227,"name":"schemas","full_name":"ga4gh/schemas","owner":{"login":"ga4gh","id":6921777,"avatar_url":"https://avatars.githubusercontent.com/u/6921777?v=3","gravatar_id":"","url":"https://api.github.com/users/ga4gh","html_url":"https://github.com/ga4gh","followers_url":"https://api.github.com/users/ga4gh/followers","following_url":"https://api.github.com/users/ga4gh/following{/other_user}","gists_url":"https://api.github.com/users/ga4gh/gists{/gist_id}","starred_url":"https://api.github.com/users/ga4gh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ga4gh/subscriptions","organizations_url":"https://api.github.com/users/ga4gh/orgs","repos_url":"https://api.github.com/users/ga4gh/repos","events_url":"https://api.github.com/users/ga4gh/events{/privacy}","received_events_url":"https://api.github.com/users/ga4gh/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/ga4gh/schemas","description":"Work on data models and APIs for Genomic data.","fork":false,"url":"https://api.github.com/repos/ga4gh/schemas","forks_url":"https://api.github.com/repos/ga4gh/schemas/forks","keys_url":"https://api.github.com/repos/ga4gh/schemas/keys{/key_id}","collaborators_url":"https://api.github.com/repos/ga4gh/schemas/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/ga4gh/schemas/teams","hooks_url":"https://api.github.com/repos/ga4gh/schemas/hooks","issue_events_url":"https://api.github.com/repos/ga4gh/schemas/issues/events{/number}","events_url":"https://api.github.com/repos/ga4gh/schemas/events","assignees_url":"https://api.github.com/repos/ga4gh/schemas/assignees{/user}","branches_url":"https://api.github.com/repos/ga4gh/schemas/branches{/branch}","tags_url":"https://api.github.com/repos/ga4gh/schemas/tags","blobs_url":"https://api.github.com/repos/ga4gh/schemas/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/ga4gh/schemas/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/ga4gh/schemas/git/refs{/sha}","trees_url":"https://api.github.com/repos/ga4gh/schemas/git/trees{/sha}","statuses_url":"https://api.github.com/repos/ga4gh/schemas/statuses/{sha}","languages_url":"https://api.github.com/repos/ga4gh/schemas/languages","stargazers_url":"https://api.github.com/repos/ga4gh/schemas/stargazers","contributors_url":"https://api.github.com/repos/ga4gh/schemas/contributors","subscribers_url":"https://api.github.com/repos/ga4gh/schemas/subscribers","subscription_url":"https://api.github.com/repos/ga4gh/schemas/subscription","commits_url":"https://api.github.com/repos/ga4gh/schemas/commits{/sha}","git_commits_url":"https://api.github.com/repos/ga4gh/schemas/git/commits{/sha}","comments_url":"https://api.github.com/repos/ga4gh/schemas/comments{/number}","issue_comment_url":"https://api.github.com/repos/ga4gh/schemas/issues/comments{/number}","contents_url":"https://api.github.com/repos/ga4gh/schemas/contents/{+path}","compare_url":"https://api.github.com/repos/ga4gh/schemas/compare/{base}...{head}","merges_url":"https://api.github.com/repos/ga4gh/schemas/merges","archive_url":"https://api.github.com/repos/ga4gh/schemas/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/ga4gh/schemas/downloads","issues_url":"https://api.github.com/repos/ga4gh/schemas/issues{/number}","pulls_url":"https://api.github.com/repos/ga4gh/schemas/pulls{/number}","milestones_url":"https://api.github.com/repos/ga4gh/schemas/milestones{/number}","notifications_url":"https://api.github.com/repos/ga4gh/schemas/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/ga4gh/schemas/labels{/name}","releases_url":"https://api.github.com/repos/ga4gh/schemas/releases{/id}","deployments_url":"https://api.github.com/repos/ga4gh/schemas/deployments","created_at":"2014-03-20T00:13:40Z","updated_at":"2016-01-25T21:24:03Z","pushed_at":"2016-01-28T04:43:27Z","git_url":"git://github.com/ga4gh/schemas.git","ssh_url":"git@github.com:ga4gh/schemas.git","clone_url":"https://github.com/ga4gh/schemas.git","svn_url":"https://github.com/ga4gh/schemas","homepage":"http://ga4gh.org/#/api","size":13327,"stargazers_count":170,"watchers_count":170,"language":"Python","has_issues":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":89,"mirror_url":null,"open_issues_count":143,"forks":89,"open_issues":143,"watchers":170,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/ga4gh/schemas/pulls/485"},"html":{"href":"https://github.com/ga4gh/schemas/pull/485"},"issue":{"href":"https://api.github.com/repos/ga4gh/schemas/issues/485"},"comments":{"href":"https://api.github.com/repos/ga4gh/schemas/issues/485/comments"},"review_comments":{"href":"https://api.github.com/repos/ga4gh/schemas/pulls/485/comments"},"review_comment":{"href":"https://api.github.com/repos/ga4gh/schemas/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/ga4gh/schemas/pulls/485/commits"},"statuses":{"href":"https://api.github.com/repos/ga4gh/schemas/statuses/b501c77a85724380279efd370965234ddd7372ce"}},"merged":false,"mergeable":false,"mergeable_state":"dirty","merged_by":null,"comments":13,"review_comments":7,"commits":4,"additions":1323,"deletions":1845,"changed_files":22}}