{"action":"created","comment":{"url":"https://api.github.com/repos/samtools/hts-specs/pulls/comments/336907471","pull_request_review_id":304419040,"id":336907471,"node_id":"MDI0OlB1bGxSZXF1ZXN0UmV2aWV3Q29tbWVudDMzNjkwNzQ3MQ==","diff_hunk":"@@ -453,6 +455,106 @@ \\subsubsection{Color space}\n Color read quality on the original strand of the read. Same encoding as {\\sf QUAL}; same length as {\\tt CS}.\n \\end{description}\n \n+\\subsection{Base modifications}\n+\n+Base basications, including base methylation, are represented as a\n+series of edits from the primary unmodified sequence stored in the\n+main SAM {\\sf SEQ} field.  If the modifed base has no natural\n+unmodified form then this should be stored as ``N''.\n+\n+Each modified base listed also has a quality value associated with\n+it.  Given the unmodified base already has a phred likelihood, this\n+base modification quality should be interpreted as the likelihood of\n+this modification being correct, rather than the base being unmodified.\n+\n+\\begin{description}\n+\\item[MM:Z:\\tagregex{([ACGTN][-+][a-z](,[0-9]+)+;)*}]\n+\\hfill\\\\\n+The first character is the unmodified base as seen in the {\\sf SEQ}\n+field, one of {\\tt A}, {\\tt C}, {\\tt G}, {\\tt T} or {\\tt N}, with\n+the exception that {\\tt N} is used to match any base rather than\n+strictly {\\tt N}.  This is followed by either plus or minus\n+indicating the strand the modification was observed on (relative to\n+the recorded strand of {\\sf SEQ} with plus meaning same\n+orientation), and a base modification symbol.  This is then followed\n+by a comma separated list of how many unmodified seq bases of the\n+stated base type to skip, stored as a delta to the last and starting\n+with 0 as the first (or next) base.  Hence this number series is\n+comparable to the numbers in an {\\tt MD} tag.\n+\n+For example {\\tt C+m,5,12,0;} tells us there are three 5-Methylcytosine\n+bases in the original {\\sf SEQ}.  The first 5 {\\tt C} bases are\n+unmodified and the 6th is modified, as are the 19th (12 inbetween the","path":"SAMtags.tex","position":42,"original_position":42,"commit_id":"6b06097e189d05a44eb2b219b0e42b65cc565183","original_commit_id":"6b06097e189d05a44eb2b219b0e42b65cc565183","user":{"login":"jkbonfield","id":2210525,"node_id":"MDQ6VXNlcjIyMTA1MjU=","avatar_url":"https://avatars0.githubusercontent.com/u/2210525?v=4","gravatar_id":"","url":"https://api.github.com/users/jkbonfield","html_url":"https://github.com/jkbonfield","followers_url":"https://api.github.com/users/jkbonfield/followers","following_url":"https://api.github.com/users/jkbonfield/following{/other_user}","gists_url":"https://api.github.com/users/jkbonfield/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbonfield/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbonfield/subscriptions","organizations_url":"https://api.github.com/users/jkbonfield/orgs","repos_url":"https://api.github.com/users/jkbonfield/repos","events_url":"https://api.github.com/users/jkbonfield/events{/privacy}","received_events_url":"https://api.github.com/users/jkbonfield/received_events","type":"User","site_admin":false},"body":"\"Between\" makes most sense. Thanks.","created_at":"2019-10-21T09:05:04Z","updated_at":"2019-10-21T09:05:04Z","html_url":"https://github.com/samtools/hts-specs/pull/418#discussion_r336907471","pull_request_url":"https://api.github.com/repos/samtools/hts-specs/pulls/418","author_association":"CONTRIBUTOR","_links":{"self":{"href":"https://api.github.com/repos/samtools/hts-specs/pulls/comments/336907471"},"html":{"href":"https://github.com/samtools/hts-specs/pull/418#discussion_r336907471"},"pull_request":{"href":"https://api.github.com/repos/samtools/hts-specs/pulls/418"}},"in_reply_to_id":336744530},"pull_request":{"url":"https://api.github.com/repos/samtools/hts-specs/pulls/418","id":286642747,"node_id":"MDExOlB1bGxSZXF1ZXN0Mjg2NjQyNzQ3","html_url":"https://github.com/samtools/hts-specs/pull/418","diff_url":"https://github.com/samtools/hts-specs/pull/418.diff","patch_url":"https://github.com/samtools/hts-specs/pull/418.patch","issue_url":"https://api.github.com/repos/samtools/hts-specs/issues/418","number":418,"state":"open","locked":false,"title":"Describe MM and MP methylation tags.","user":{"login":"jkbonfield","id":2210525,"node_id":"MDQ6VXNlcjIyMTA1MjU=","avatar_url":"https://avatars0.githubusercontent.com/u/2210525?v=4","gravatar_id":"","url":"https://api.github.com/users/jkbonfield","html_url":"https://github.com/jkbonfield","followers_url":"https://api.github.com/users/jkbonfield/followers","following_url":"https://api.github.com/users/jkbonfield/following{/other_user}","gists_url":"https://api.github.com/users/jkbonfield/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbonfield/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbonfield/subscriptions","organizations_url":"https://api.github.com/users/jkbonfield/orgs","repos_url":"https://api.github.com/users/jkbonfield/repos","events_url":"https://api.github.com/users/jkbonfield/events{/privacy}","received_events_url":"https://api.github.com/users/jkbonfield/received_events","type":"User","site_admin":false},"body":"This provides a couple tags for encoding base modifications as a side-channel on top of the unmodified canonical sequence.  It's a slightly updated implementation of the ideas presented in https://github.com/samtools/hts-specs/issues/362\r\n\r\nThe tag names are somewhat arbitrarily chosen tag names; feel free to suggest improvements, but chosen \"M\" for modification and M/P as both are free and adjacent in the sorted list.\r\n\r\nI'm still in two minds as to whether this method is the best place to be.  We could use a simpler form which is more MD style where modifications are recorded as positional deltas to *any* base rather than the appropriate canonical base.  This is larger, but simpler to work with.  How much value do we put in the storage cost vs complexity?\r\n\r\nA pictorial example on a tiny sequence:\r\n\r\n```\r\nCCAGCTATAGGCCCCCC (canonical)\r\nmCAomTATAGomCmCCm (modified)\r\nm...m......m.m..m\r\n...o......o......\r\n\r\nC+m,0,3,6,1,2;G+o,3,6;  (count all bases)\r\n\r\nCCAGCTATAGGCCCCCC (canonical)\r\nmCAomTATAGomCmCCm (modified)\r\nm.  m      m.m..m\r\n   o     .o\r\n\r\nC+m,0,1,0,1,2;G+o,0,1;  (count canonical bases only)\r\n```\r\n\r\n\r\nIf we call the method in this PR ConDelta and the simpler method AllDelta then for two levels of simulated modification on long read data (3% C->m vs 30% C->m with other C/G modification types adjusted in scale too), then our sizes are as follows:\r\n\r\n```\r\n          Size (3% C->m)               Size (30% C->m)\r\n          BAM          CRAM            BAM            CRAM\r\nNo mods   1327060      815861          1327060        815861\r\nConDelta  +25049/1.9%  +9482 /1.2%     +92938 /7.0%   +39775/4.9%\r\nAllDelta  +30912/2.3%  +12386/1.5%     +134178/10.1%  +66382/8.1%\r\n```\r\n\r\nThe delta sizes here are shown as +extra amount.  Ie ConDelta for BAM is 1327060+25049 bytes long.  (This is obviously a small test set just for experimentation.)\r\n\r\nThe /1.9% bit in +25049/1.9% is showing what percentage of the original file that represents - ie 25049/1327060 is ~0.019.\r\n\r\nThus by comparing AllDelta vs ConDelta total file sizes, where see the simpler method will use in total between 0.4% (3% C meth) to 3.0% (30% C meth) extra storage.   Quite small overall, despite the actual chunk of data being used for methylation signal itself being considerably larger.\r\n\r\nNB: the tests above were all done without the MP modification quality value tag, but that doesn't change size and would have minimal impact on total file sizes.\r\n\r\n**So, discuss...**","created_at":"2019-06-10T11:38:42Z","updated_at":"2019-10-21T09:05:04Z","closed_at":null,"merged_at":null,"merge_commit_sha":"bc5c9cdf9a87249e0220cd6885eb0ad52b9ab41b","assignee":null,"assignees":[],"requested_reviewers":[],"requested_teams":[],"labels":[{"id":100671191,"node_id":"MDU6TGFiZWwxMDA2NzExOTE=","url":"https://api.github.com/repos/samtools/hts-specs/labels/sam","name":"sam","color":"0052cc","default":false},{"id":1151603196,"node_id":"MDU6TGFiZWwxMTUxNjAzMTk2","url":"https://api.github.com/repos/samtools/hts-specs/labels/sam-tags","name":"sam-tags","color":"aec6fc","default":false}],"milestone":null,"commits_url":"https://api.github.com/repos/samtools/hts-specs/pulls/418/commits","review_comments_url":"https://api.github.com/repos/samtools/hts-specs/pulls/418/comments","review_comment_url":"https://api.github.com/repos/samtools/hts-specs/pulls/comments{/number}","comments_url":"https://api.github.com/repos/samtools/hts-specs/issues/418/comments","statuses_url":"https://api.github.com/repos/samtools/hts-specs/statuses/6b06097e189d05a44eb2b219b0e42b65cc565183","head":{"label":"jkbonfield:methylation","ref":"methylation","sha":"6b06097e189d05a44eb2b219b0e42b65cc565183","user":{"login":"jkbonfield","id":2210525,"node_id":"MDQ6VXNlcjIyMTA1MjU=","avatar_url":"https://avatars0.githubusercontent.com/u/2210525?v=4","gravatar_id":"","url":"https://api.github.com/users/jkbonfield","html_url":"https://github.com/jkbonfield","followers_url":"https://api.github.com/users/jkbonfield/followers","following_url":"https://api.github.com/users/jkbonfield/following{/other_user}","gists_url":"https://api.github.com/users/jkbonfield/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbonfield/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbonfield/subscriptions","organizations_url":"https://api.github.com/users/jkbonfield/orgs","repos_url":"https://api.github.com/users/jkbonfield/repos","events_url":"https://api.github.com/users/jkbonfield/events{/privacy}","received_events_url":"https://api.github.com/users/jkbonfield/received_events","type":"User","site_admin":false},"repo":{"id":17675494,"node_id":"MDEwOlJlcG9zaXRvcnkxNzY3NTQ5NA==","name":"hts-specs","full_name":"jkbonfield/hts-specs","private":false,"owner":{"login":"jkbonfield","id":2210525,"node_id":"MDQ6VXNlcjIyMTA1MjU=","avatar_url":"https://avatars0.githubusercontent.com/u/2210525?v=4","gravatar_id":"","url":"https://api.github.com/users/jkbonfield","html_url":"https://github.com/jkbonfield","followers_url":"https://api.github.com/users/jkbonfield/followers","following_url":"https://api.github.com/users/jkbonfield/following{/other_user}","gists_url":"https://api.github.com/users/jkbonfield/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbonfield/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbonfield/subscriptions","organizations_url":"https://api.github.com/users/jkbonfield/orgs","repos_url":"https://api.github.com/users/jkbonfield/repos","events_url":"https://api.github.com/users/jkbonfield/events{/privacy}","received_events_url":"https://api.github.com/users/jkbonfield/received_events","type":"User","site_admin":false},"html_url":"https://github.com/jkbonfield/hts-specs","description":"Specifications of SAM/BAM and related high-throughput sequencing file formats","fork":true,"url":"https://api.github.com/repos/jkbonfield/hts-specs","forks_url":"https://api.github.com/repos/jkbonfield/hts-specs/forks","keys_url":"https://api.github.com/repos/jkbonfield/hts-specs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/jkbonfield/hts-specs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/jkbonfield/hts-specs/teams","hooks_url":"https://api.github.com/repos/jkbonfield/hts-specs/hooks","issue_events_url":"https://api.github.com/repos/jkbonfield/hts-specs/issues/events{/number}","events_url":"https://api.github.com/repos/jkbonfield/hts-specs/events","assignees_url":"https://api.github.com/repos/jkbonfield/hts-specs/assignees{/user}","branches_url":"https://api.github.com/repos/jkbonfield/hts-specs/branches{/branch}","tags_url":"https://api.github.com/repos/jkbonfield/hts-specs/tags","blobs_url":"https://api.github.com/repos/jkbonfield/hts-specs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/jkbonfield/hts-specs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/jkbonfield/hts-specs/git/refs{/sha}","trees_url":"https://api.github.com/repos/jkbonfield/hts-specs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/jkbonfield/hts-specs/statuses/{sha}","languages_url":"https://api.github.com/repos/jkbonfield/hts-specs/languages","stargazers_url":"https://api.github.com/repos/jkbonfield/hts-specs/stargazers","contributors_url":"https://api.github.com/repos/jkbonfield/hts-specs/contributors","subscribers_url":"https://api.github.com/repos/jkbonfield/hts-specs/subscribers","subscription_url":"https://api.github.com/repos/jkbonfield/hts-specs/subscription","commits_url":"https://api.github.com/repos/jkbonfield/hts-specs/commits{/sha}","git_commits_url":"https://api.github.com/repos/jkbonfield/hts-specs/git/commits{/sha}","comments_url":"https://api.github.com/repos/jkbonfield/hts-specs/comments{/number}","issue_comment_url":"https://api.github.com/repos/jkbonfield/hts-specs/issues/comments{/number}","contents_url":"https://api.github.com/repos/jkbonfield/hts-specs/contents/{+path}","compare_url":"https://api.github.com/repos/jkbonfield/hts-specs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/jkbonfield/hts-specs/merges","archive_url":"https://api.github.com/repos/jkbonfield/hts-specs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/jkbonfield/hts-specs/downloads","issues_url":"https://api.github.com/repos/jkbonfield/hts-specs/issues{/number}","pulls_url":"https://api.github.com/repos/jkbonfield/hts-specs/pulls{/number}","milestones_url":"https://api.github.com/repos/jkbonfield/hts-specs/milestones{/number}","notifications_url":"https://api.github.com/repos/jkbonfield/hts-specs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/jkbonfield/hts-specs/labels{/name}","releases_url":"https://api.github.com/repos/jkbonfield/hts-specs/releases{/id}","deployments_url":"https://api.github.com/repos/jkbonfield/hts-specs/deployments","created_at":"2014-03-12T17:11:12Z","updated_at":"2017-05-11T11:35:50Z","pushed_at":"2019-09-30T10:25:44Z","git_url":"git://github.com/jkbonfield/hts-specs.git","ssh_url":"git@github.com:jkbonfield/hts-specs.git","clone_url":"https://github.com/jkbonfield/hts-specs.git","svn_url":"https://github.com/jkbonfield/hts-specs","homepage":"","size":19645,"stargazers_count":0,"watchers_count":0,"language":"TeX","has_issues":false,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":false,"forks_count":0,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":0,"license":null,"forks":0,"open_issues":0,"watchers":0,"default_branch":"master"}},"base":{"label":"samtools:master","ref":"master","sha":"267de2880832e8966fc09bb777130604e117e3f3","user":{"login":"samtools","id":1518450,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTg0NTA=","avatar_url":"https://avatars1.githubusercontent.com/u/1518450?v=4","gravatar_id":"","url":"https://api.github.com/users/samtools","html_url":"https://github.com/samtools","followers_url":"https://api.github.com/users/samtools/followers","following_url":"https://api.github.com/users/samtools/following{/other_user}","gists_url":"https://api.github.com/users/samtools/gists{/gist_id}","starred_url":"https://api.github.com/users/samtools/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samtools/subscriptions","organizations_url":"https://api.github.com/users/samtools/orgs","repos_url":"https://api.github.com/users/samtools/repos","events_url":"https://api.github.com/users/samtools/events{/privacy}","received_events_url":"https://api.github.com/users/samtools/received_events","type":"Organization","site_admin":false},"repo":{"id":3666509,"node_id":"MDEwOlJlcG9zaXRvcnkzNjY2NTA5","name":"hts-specs","full_name":"samtools/hts-specs","private":false,"owner":{"login":"samtools","id":1518450,"node_id":"MDEyOk9yZ2FuaXphdGlvbjE1MTg0NTA=","avatar_url":"https://avatars1.githubusercontent.com/u/1518450?v=4","gravatar_id":"","url":"https://api.github.com/users/samtools","html_url":"https://github.com/samtools","followers_url":"https://api.github.com/users/samtools/followers","following_url":"https://api.github.com/users/samtools/following{/other_user}","gists_url":"https://api.github.com/users/samtools/gists{/gist_id}","starred_url":"https://api.github.com/users/samtools/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/samtools/subscriptions","organizations_url":"https://api.github.com/users/samtools/orgs","repos_url":"https://api.github.com/users/samtools/repos","events_url":"https://api.github.com/users/samtools/events{/privacy}","received_events_url":"https://api.github.com/users/samtools/received_events","type":"Organization","site_admin":false},"html_url":"https://github.com/samtools/hts-specs","description":"Specifications of SAM/BAM and related high-throughput sequencing file formats","fork":false,"url":"https://api.github.com/repos/samtools/hts-specs","forks_url":"https://api.github.com/repos/samtools/hts-specs/forks","keys_url":"https://api.github.com/repos/samtools/hts-specs/keys{/key_id}","collaborators_url":"https://api.github.com/repos/samtools/hts-specs/collaborators{/collaborator}","teams_url":"https://api.github.com/repos/samtools/hts-specs/teams","hooks_url":"https://api.github.com/repos/samtools/hts-specs/hooks","issue_events_url":"https://api.github.com/repos/samtools/hts-specs/issues/events{/number}","events_url":"https://api.github.com/repos/samtools/hts-specs/events","assignees_url":"https://api.github.com/repos/samtools/hts-specs/assignees{/user}","branches_url":"https://api.github.com/repos/samtools/hts-specs/branches{/branch}","tags_url":"https://api.github.com/repos/samtools/hts-specs/tags","blobs_url":"https://api.github.com/repos/samtools/hts-specs/git/blobs{/sha}","git_tags_url":"https://api.github.com/repos/samtools/hts-specs/git/tags{/sha}","git_refs_url":"https://api.github.com/repos/samtools/hts-specs/git/refs{/sha}","trees_url":"https://api.github.com/repos/samtools/hts-specs/git/trees{/sha}","statuses_url":"https://api.github.com/repos/samtools/hts-specs/statuses/{sha}","languages_url":"https://api.github.com/repos/samtools/hts-specs/languages","stargazers_url":"https://api.github.com/repos/samtools/hts-specs/stargazers","contributors_url":"https://api.github.com/repos/samtools/hts-specs/contributors","subscribers_url":"https://api.github.com/repos/samtools/hts-specs/subscribers","subscription_url":"https://api.github.com/repos/samtools/hts-specs/subscription","commits_url":"https://api.github.com/repos/samtools/hts-specs/commits{/sha}","git_commits_url":"https://api.github.com/repos/samtools/hts-specs/git/commits{/sha}","comments_url":"https://api.github.com/repos/samtools/hts-specs/comments{/number}","issue_comment_url":"https://api.github.com/repos/samtools/hts-specs/issues/comments{/number}","contents_url":"https://api.github.com/repos/samtools/hts-specs/contents/{+path}","compare_url":"https://api.github.com/repos/samtools/hts-specs/compare/{base}...{head}","merges_url":"https://api.github.com/repos/samtools/hts-specs/merges","archive_url":"https://api.github.com/repos/samtools/hts-specs/{archive_format}{/ref}","downloads_url":"https://api.github.com/repos/samtools/hts-specs/downloads","issues_url":"https://api.github.com/repos/samtools/hts-specs/issues{/number}","pulls_url":"https://api.github.com/repos/samtools/hts-specs/pulls{/number}","milestones_url":"https://api.github.com/repos/samtools/hts-specs/milestones{/number}","notifications_url":"https://api.github.com/repos/samtools/hts-specs/notifications{?since,all,participating}","labels_url":"https://api.github.com/repos/samtools/hts-specs/labels{/name}","releases_url":"https://api.github.com/repos/samtools/hts-specs/releases{/id}","deployments_url":"https://api.github.com/repos/samtools/hts-specs/deployments","created_at":"2012-03-09T01:58:32Z","updated_at":"2019-10-18T12:12:59Z","pushed_at":"2019-10-18T21:28:41Z","git_url":"git://github.com/samtools/hts-specs.git","ssh_url":"git@github.com:samtools/hts-specs.git","clone_url":"https://github.com/samtools/hts-specs.git","svn_url":"https://github.com/samtools/hts-specs","homepage":"http://samtools.github.io/hts-specs/","size":19511,"stargazers_count":335,"watchers_count":335,"language":"TeX","has_issues":true,"has_projects":true,"has_downloads":true,"has_wiki":true,"has_pages":true,"forks_count":129,"mirror_url":null,"archived":false,"disabled":false,"open_issues_count":141,"license":null,"forks":129,"open_issues":141,"watchers":335,"default_branch":"master"}},"_links":{"self":{"href":"https://api.github.com/repos/samtools/hts-specs/pulls/418"},"html":{"href":"https://github.com/samtools/hts-specs/pull/418"},"issue":{"href":"https://api.github.com/repos/samtools/hts-specs/issues/418"},"comments":{"href":"https://api.github.com/repos/samtools/hts-specs/issues/418/comments"},"review_comments":{"href":"https://api.github.com/repos/samtools/hts-specs/pulls/418/comments"},"review_comment":{"href":"https://api.github.com/repos/samtools/hts-specs/pulls/comments{/number}"},"commits":{"href":"https://api.github.com/repos/samtools/hts-specs/pulls/418/commits"},"statuses":{"href":"https://api.github.com/repos/samtools/hts-specs/statuses/6b06097e189d05a44eb2b219b0e42b65cc565183"}},"author_association":"CONTRIBUTOR"}}
{"actor":{"display_login":"jkbonfield"}}