{"action":"created","comment":{"url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/comments/215669407","pull_request_review_id":152986396,"id":215669407,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDIxNTY2OTQwNw==","diff_hunk":"@@ -0,0 +1,72 @@\n+# Dealing with malformed files\n+\n+## Field compatibility\n+\n+When loading multiple files, the `INFO` and `FORMAT` fields\n+from all VCF files are merged together to generate a \"representative header\",\n+which is then used to generate a single BigQuery schema. If the same key is\n+defined in multiple files, then its definition must be compatible across files.\n+The compatibility rules are as follows:\n+\n+* Fields are compatible if they have the same `Number` and `Type` fields.\n+  Annotation fields (i.e. those specified by `--annotation_fields`) must also\n+  have the same `Description`.\n+\n+* Fields with different `Type` values are compatible in the following cases:\n+\n+  * `Integer` and `Float` fields are compatible and are converted to `Float`.\n+  * We are adding more compatibility options (e.g. resolving `String` and\n+  `Integer` to `String`). See\n+  [below](#specifying---allow-incompatible-records) for more details.\n+\n+* Fields with different `Number` values are compatible in the following cases:\n+\n+  * \"Repeated\" numbers are compatible with each other. They include:\n+    * `Number=.` (unknown number)\n+    * Any `Number` greater than 1.\n+    * `Number=G` (one value per genotype) and `Number=R` (one value for each\n+      alternate and reference).\n+    * `Number=A` (one value for each alternate) only if running with\n+      `--split_alternate_allele_info_fields False`.\n+\n+  * We are adding more compatibility options (e.g. resolving `Number=1` and\n+    `Number=.` with `Number=.`). See\n+    [below](#specifying---allow-incompatible-records) for more details.\n+\n+\n+## Specifying `--representative_header_file`\n+\n+The headers in the `--representative_header_file <path_to_file>` essentially\n+specify the merged headers from all files being loaded to BigQuery. This file is\n+used to directly generate the BigQuery schema. Note that we only read the\n+header info from the file and ignore VCF records, so the\n+`representative_header_file` can either be a file containing *just* the header\n+fields or can point to an actual VCF file. Providing this file can be useful\n+for:\n+\n+* Speeding up the pipeline especially if a large number of files are provided.\n+  The pipeline will use the provided file to generate the BigQuery schema and\n+  will skip merging headers across files. This is particularly useful if all\n+  files have identical VCF headers.\n+* Providing definitions for missing header fields. See the\n+  [troubleshooting page](./troubleshooting.md) for more details.\n+* Coming soon: resolving incompatible fields across files.\n+\n+## Specifying `--infer-undefined-header-fields`\n+\n+If this flag is set, pipeline will infer `TYPE` and `NUMBER` for undefined\n+fields based on field values seen in VCF files. It will also output a\n+representive header that contains infered definitions as well as definitions\n+from headers. Use this flag if there are fields with missing definition.\n+\n+\n+## Specifying `--allow-incompatible-records`\n+\n+Pipeline will fail by defualt if there is a mismatch between field definition\n+and actual values or if a field has two inconsistent definitions in two\n+different VCF files.\n+By specifying `--allow-incompatible-records`, pipeline will resolve conflicts\n+in header definitons. It will also cast field values to match BigQuery schema if\n+there is a mismatch between field definition and field value. Please see [this","path":"docs/malformed_files.md","position":null,"original_position":70,"commit_id":"5e8cd8028ef76af8810b2a06b4f61460125c2a29","original_commit_id":"c9b17af173ceb42a73340742c6c6ff3994b5a5e5","user":{"login":"nmousavi","id":4221028,"node_id":"MDQ6VXNlcjQyMjEwMjg=","avatar_url":"https://avatars2.githubusercontent.com/u/4221028?v=4","gravatar_id":"","url":"https://api.github.com/users/nmousavi","html_url":"https://github.com/nmousavi","followers_url":"https://api.github.com/users/nmousavi/followers","following_url":"https://api.github.com/users/nmousavi/following{/other_user}","gists_url":"https://api.github.com/users/nmousavi/gists{/gist_id}","starred_url":"https://api.github.com/users/nmousavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmousavi/subscriptions","organizations_url":"https://api.github.com/users/nmousavi/orgs","repos_url":"https://api.github.com/users/nmousavi/repos","events_url":"https://api.github.com/users/nmousavi/events{/privacy}","received_events_url":"https://api.github.com/users/nmousavi/received_events","type":"User","site_admin":false},"body":"Done","created_at":"2018-09-06T15:24:08Z","updated_at":"2018-09-06T15:29:02Z","html_url":"https://github.com/googlegenomics/gcp-variant-transforms/pull/357#discussion_r215669407","pull_request_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/357","author_association":"MEMBER","_links":{"self":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/comments/215669407"},"html":{"href":"https://github.com/googlegenomics/gcp-variant-transforms/pull/357#discussion_r215669407"},"pull_request":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/357"}},"in_reply_to_id":215400167},"pull_request":{"url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/357","id":213374708,"node_id":"MDExOlB1bGxSZXF1ZXN0MjEzMzc0NzA4","html_url":"https://github.com/googlegenomics/gcp-variant-transforms/pull/357","diff_url":"https://github.com/googlegenomics/gcp-variant-transforms/pull/357.diff","patch_url":"https://github.com/googlegenomics/gcp-variant-transforms/pull/357.patch","issue_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/357","number":357,"state":"open","locked":false,"title":"Document flags for robust Variant Transforms.","user":{"login":"nmousavi","id":4221028,"node_id":"MDQ6VXNlcjQyMjEwMjg=","avatar_url":"https://avatars2.githubusercontent.com/u/4221028?v=4","gravatar_id":"","url":"https://api.github.com/users/nmousavi","html_url":"https://github.com/nmousavi","followers_url":"https://api.github.com/users/nmousavi/followers","following_url":"https://api.github.com/users/nmousavi/following{/other_user}","gists_url":"https://api.github.com/users/nmousavi/gists{/gist_id}","starred_url":"https://api.github.com/users/nmousavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmousavi/subscriptions","organizations_url":"https://api.github.com/users/nmousavi/orgs","repos_url":"https://api.github.com/users/nmousavi/repos","events_url":"https://api.github.com/users/nmousavi/events{/privacy}","received_events_url":"https://api.github.com/users/nmousavi/received_events","type":"User","site_admin":false},"body":"Fixes issue #184","created_at":"2018-09-05T17:41:22Z","updated_at":"2018-09-06T15:29:02Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bdbd997c092d9ffed7c3bd01fc9f7f59721fde8e","assignee":null,"assignees":[],"requested_reviewers":[{"login":"allieychen","id":36679499,"node_id":"MDQ6VXNlcjM2Njc5NDk5","avatar_url":"https://avatars0.githubusercontent.com/u/36679499?v=4","gravatar_id":"","url":"https://api.github.com/users/allieychen","html_url":"https://github.com/allieychen","followers_url":"https://api.github.com/users/allieychen/followers","following_url":"https://api.github.com/users/allieychen/following{/other_user}","gists_url":"https://api.github.com/users/allieychen/gists{/gist_id}","starred_url":"https://api.github.com/users/allieychen/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/allieychen/subscriptions","organizations_url":"https://api.github.com/users/allieychen/orgs","repos_url":"https://api.github.com/users/allieychen/repos","events_url":"https://api.github.com/users/allieychen/events{/privacy}","received_events_url":"https://api.github.com/users/allieychen/received_events","type":"User","site_admin":false}],"requested_teams":[],"labels":[],"milestone":null,"commits_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/357/commits","review_comments_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/357/comments","review_comment_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/comments{/number}","comments_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/357/comments","statuses_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/statuses/5e8cd8028ef76af8810b2a06b4f61460125c2a29","head":{"label":"nmousavi:robust-doc","ref":"robust-doc","sha":"5e8cd8028ef76af8810b2a06b4f61460125c2a29","user":{"login":"nmousavi","id":4221028,"node_id":"MDQ6VXNlcjQyMjEwMjg=","avatar_url":"https://avatars2.githubusercontent.com/u/4221028?v=4","gravatar_id":"","url":"https://api.github.com/users/nmousavi","html_url":"https://github.com/nmousavi","followers_url":"https://api.github.com/users/nmousavi/followers","following_url":"https://api.github.com/users/nmousavi/following{/other_user}","gists_url":"https://api.github.com/users/nmousavi/gists{/gist_id}","starred_url":"https://api.github.com/users/nmousavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmousavi/subscriptions","organizations_url":"https://api.github.com/users/nmousavi/orgs","repos_url":"https://api.github.com/users/nmousavi/repos","events_url":"https://api.github.com/users/nmousavi/events{/privacy}","received_events_url":"https://api.github.com/users/nmousavi/received_events","type":"User","site_admin":false},"repo":{"id":114024059,"node_id":"MDEwOlJlcG9zaXRvcnkxMTQwMjQwNTk=","name":"gcp-variant-transforms","full_name":"nmousavi/gcp-variant-transforms","owner":{"login":"nmousavi","id":4221028,"node_id":"MDQ6VXNlcjQyMjEwMjg=","avatar_url":"https://avatars2.githubusercontent.com/u/4221028?v=4","gravatar_id":"","url":"https://api.github.com/users/nmousavi","html_url":"https://github.com/nmousavi","followers_url":"https://api.github.com/users/nmousavi/followers","following_url":"https://api.github.com/users/nmousavi/following{/other_user}","gists_url":"https://api.github.com/users/nmousavi/gists{/gist_id}","starred_url":"https://api.github.com/users/nmousavi/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/nmousavi/subscriptions","organizations_url":"https://api.github.com/users/nmousavi/orgs","repos_url":"https://api.github.com/users/nmousavi/repos","events_url":"https://api.github.com/users/nmousavi/events{/privacy}","received_events_url":"https://api.github.com/users/nmousavi/received_events","type":"User","site_admin":false},"private":false,"html_url":"https://github.com/nmousavi/gcp-variant-transforms","description":"GCP Variant Transforms","fork":true,"url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms","forks_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/forks","keys_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/keys{/key_id}","collaborators_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/teams","hooks_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/hooks","issue_events_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/issues/events{/number}","events_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/events","assignees_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/assignees{/user}","branches_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/branches{/branch}","tags_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/tags","blobs_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/git/refs{/sha}","trees_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/git/trees{/sha}","statuses_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/statuses/{sha}","languages_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/languages","stargazers_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/stargazers","contributors_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/contributors","subscribers_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/subscribers","subscription_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/subscription","commits_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/commits{/sha}","git_commits_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/git/commits{/sha}","comments_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/comments{/number}","issue_comment_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/issues/comments{/number}","contents_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/contents/{+path}","compare_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/compare/{base}...{head}","merges_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/merges","archive_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/downloads","issues_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/issues{/number}","pulls_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/pulls{/number}","milestones_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/milestones{/number}","notifications_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/labels{/name}","releases_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/releases{/id}","deployments_url":"https://api.github.com/repos/nmousavi/gcp-variant-transforms/deployments","created_at":"2017-12-12T18:17:02Z","updated_at":"2018-08-29T15:10:51Z","pushed_at":"2018-09-06T15:28:40Z","git_url":"git://github.com/nmousavi/gcp-variant-transforms.git","ssh_url":"git@github.com:nmousavi/gcp-variant-transforms.git","clone_url":"https://github.com/nmousavi/gcp-variant-transforms.git","svn_url":"https://github.com/nmousavi/gcp-variant-transforms","homepage":null,"size":1741,"stargazers_count":1,"watchers_count":1,"language":"Python","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"open_issues_count":0,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":0,"open_issues":0,"watchers":1,"default_branch":"master"}},"base":{"label":"googlegenomics:master","ref":"master","sha":"64408185087ab37cc975cc8ce43e116d3ae02f45","user":{"login":"googlegenomics","id":7434120,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc0MzQxMjA=","avatar_url":"https://avatars2.githubusercontent.com/u/7434120?v=4","gravatar_id":"","url":"https://api.github.com/users/googlegenomics","html_url":"https://github.com/googlegenomics","followers_url":"https://api.github.com/users/googlegenomics/followers","following_url":"https://api.github.com/users/googlegenomics/following{/other_user}","gists_url":"https://api.github.com/users/googlegenomics/gists{/gist_id}","starred_url":"https://api.github.com/users/googlegenomics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/googlegenomics/subscriptions","organizations_url":"https://api.github.com/users/googlegenomics/orgs","repos_url":"https://api.github.com/users/googlegenomics/repos","events_url":"https://api.github.com/users/googlegenomics/events{/privacy}","received_events_url":"https://api.github.com/users/googlegenomics/received_events","type":"Organization","site_admin":false},"repo":{"id":110025283,"node_id":"MDEwOlJlcG9zaXRvcnkxMTAwMjUyODM=","name":"gcp-variant-transforms","full_name":"googlegenomics/gcp-variant-transforms","owner":{"login":"googlegenomics","id":7434120,"node_id":"MDEyOk9yZ2FuaXphdGlvbjc0MzQxMjA=","avatar_url":"https://avatars2.githubusercontent.com/u/7434120?v=4","gravatar_id":"","url":"https://api.github.com/users/googlegenomics","html_url":"https://github.com/googlegenomics","followers_url":"https://api.github.com/users/googlegenomics/followers","following_url":"https://api.github.com/users/googlegenomics/following{/other_user}","gists_url":"https://api.github.com/users/googlegenomics/gists{/gist_id}","starred_url":"https://api.github.com/users/googlegenomics/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/googlegenomics/subscriptions","organizations_url":"https://api.github.com/users/googlegenomics/orgs","repos_url":"https://api.github.com/users/googlegenomics/repos","events_url":"https://api.github.com/users/googlegenomics/events{/privacy}","received_events_url":"https://api.github.com/users/googlegenomics/received_events","type":"Organization","site_admin":false},"private":false,"html_url":"https://github.com/googlegenomics/gcp-variant-transforms","description":"GCP Variant Transforms","fork":false,"url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms","forks_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/forks","keys_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/keys{/key_id}","collaborators_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/teams","hooks_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/hooks","issue_events_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/events{/number}","events_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/events","assignees_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/assignees{/user}","branches_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/branches{/branch}","tags_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/tags","blobs_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/git/refs{/sha}","trees_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/git/trees{/sha}","statuses_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/statuses/{sha}","languages_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/languages","stargazers_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/stargazers","contributors_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/contributors","subscribers_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/subscribers","subscription_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/subscription","commits_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/commits{/sha}","git_commits_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/git/commits{/sha}","comments_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/comments{/number}","issue_comment_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/comments{/number}","contents_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/contents/{+path}","compare_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/compare/{base}...{head}","merges_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/merges","archive_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/downloads","issues_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues{/number}","pulls_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls{/number}","milestones_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/milestones{/number}","notifications_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/labels{/name}","releases_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/releases{/id}","deployments_url":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/deployments","created_at":"2017-11-08T20:12:08Z","updated_at":"2018-09-05T16:09:44Z","pushed_at":"2018-09-06T15:28:41Z","git_url":"git://github.com/googlegenomics/gcp-variant-transforms.git","ssh_url":"git@github.com:googlegenomics/gcp-variant-transforms.git","clone_url":"https://github.com/googlegenomics/gcp-variant-transforms.git","svn_url":"https://github.com/googlegenomics/gcp-variant-transforms","homepage":null,"size":1727,"stargazers_count":41,"watchers_count":41,"language":"Python","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":15,"mirror_url":null,"archived":false,"open_issues_count":66,"license":{"key":"apache-2.0","name":"Apache License 2.0","spdx_id":"Apache-2.0","url":"https://api.github.com/licenses/apache-2.0","node_id":"MDc6TGljZW5zZTI="},"forks":15,"open_issues":66,"watchers":41,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/357"},"html":{"href":"https://github.com/googlegenomics/gcp-variant-transforms/pull/357"},"issue":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/357"},"comments":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/issues/357/comments"},"review_comments":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/357/comments"},"review_comment":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/pulls/357/commits"},"statuses":{"href":"https://api.github.com/repos/googlegenomics/gcp-variant-transforms/statuses/5e8cd8028ef76af8810b2a06b4f61460125c2a29"}},"author_association":"MEMBER"}}
{"actor":{"display_login":"nmousavi"}}