{"action":"created","issue":{"url":"https://api.github.com/repos/samtools/samtools/issues/541","repository_url":"https://api.github.com/repos/samtools/samtools","labels_url":"https://api.github.com/repos/samtools/samtools/issues/541/labels{/name}","comments_url":"https://api.github.com/repos/samtools/samtools/issues/541/comments","events_url":"https://api.github.com/repos/samtools/samtools/issues/541/events","html_url":"https://github.com/samtools/samtools/issues/541","id":135724389,"number":541,"title":"samtools 1.3 segfaults reading a particular CRAM file","user":{"login":"jrandall","id":458361,"avatar_url":"https://avatars.githubusercontent.com/u/458361?v=3","gravatar_id":"","url":"https://api.github.com/users/jrandall","html_url":"https://github.com/jrandall","followers_url":"https://api.github.com/users/jrandall/followers","following_url":"https://api.github.com/users/jrandall/following{/other_user}","gists_url":"https://api.github.com/users/jrandall/gists{/gist_id}","starred_url":"https://api.github.com/users/jrandall/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jrandall/subscriptions","organizations_url":"https://api.github.com/users/jrandall/orgs","repos_url":"https://api.github.com/users/jrandall/repos","events_url":"https://api.github.com/users/jrandall/events{/privacy}","received_events_url":"https://api.github.com/users/jrandall/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":{"login":"jkbonfield","id":2210525,"avatar_url":"https://avatars.githubusercontent.com/u/2210525?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbonfield","html_url":"https://github.com/jkbonfield","followers_url":"https://api.github.com/users/jkbonfield/followers","following_url":"https://api.github.com/users/jkbonfield/following{/other_user}","gists_url":"https://api.github.com/users/jkbonfield/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbonfield/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbonfield/subscriptions","organizations_url":"https://api.github.com/users/jkbonfield/orgs","repos_url":"https://api.github.com/users/jkbonfield/repos","events_url":"https://api.github.com/users/jkbonfield/events{/privacy}","received_events_url":"https://api.github.com/users/jkbonfield/received_events","type":"User","site_admin":false},"milestone":null,"comments":2,"created_at":"2016-02-23T12:29:08Z","updated_at":"2016-05-19T16:17:09Z","closed_at":null,"body":"samtools 1.3 segfaults when reading a particular CRAM file that unfortunately we cannot share. We initially identified this issue when running `samtools stats` but it occurs with `samtools view` as well. \r\n\r\nRunning under GDB:\r\n```\r\n$ gdb --args samtools view -o /dev/null broken.cram\r\nGNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04\r\nCopyright (C) 2012 Free Software Foundation, Inc.\r\nLicense GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\r\nThis is free software: you are free to change and redistribute it.\r\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\r\nand \"show warranty\" for details.\r\nThis GDB was configured as \"x86_64-linux-gnu\".\r\nFor bug reporting instructions, please see:\r\n<http://bugs.launchpad.net/gdb-linaro/>...\r\nReading symbols from /software/hgi/pkglocal/samtools-1.3/bin/samtools...done.\r\n(gdb) run\r\nStarting program: /software/hgi/pkglocal/samtools-1.3/bin/samtools view -o /dev/null broken.cram\r\nwarning: no loadable sections found in added symbol-file system-supplied DSO at 0x7ffff7ffa000\r\n[Thread debugging using libthread_db enabled]\r\nUsing host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\r\n\r\nProgram received signal SIGSEGV, Segmentation fault.\r\n0x00000000004a8032 in rans_uncompress_O1 () at cram/rANS_static.c:675\r\n675                             D[l1].R[m[1]],\r\n```\r\n\r\nThe offending line appears to be: https://github.com/samtools/htslib/blob/develop/cram/rANS_static.c#L675\r\n\r\nI compiled samtools and htslib from the current develop and confirmed this problem is present there as well. I built a version with optimisations turned off in order to see what the problem is:\r\n```\r\n$ gdb --args ./samtools view -o /dev/null /lustre/scratch115/teams/hgi/users/jr17/broken.cram\r\nGNU gdb (Ubuntu/Linaro 7.4-2012.04-0ubuntu2.1) 7.4-2012.04\r\nCopyright (C) 2012 Free Software Foundation, Inc.\r\nLicense GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\r\nThis is free software: you are free to change and redistribute it.\r\nThere is NO WARRANTY, to the extent permitted by law.  Type \"show copying\"\r\nand \"show warranty\" for details.\r\nThis GDB was configured as \"x86_64-linux-gnu\".\r\nFor bug reporting instructions, please see:\r\n<http://bugs.launchpad.net/gdb-linaro/>...\r\nReading symbols from /nfs/humgen01/teams/hgi/software/git-checkouts/samtools-htslib-bcftools/samtools/samtools...done.\r\n(gdb) run\r\nStarting program: /nfs/humgen01/teams/hgi/software/git-checkouts/samtools-htslib-bcftools/samtools/samtools view -o /dev/null /lustre/scratch115/teams/hgi/users/jr17/broken.cram\r\n[Thread debugging using libthread_db enabled]\r\nUsing host libthread_db library \"/lib/x86_64-linux-gnu/libthread_db.so.1\".\r\n\r\nProgram received signal SIGSEGV, Segmentation fault.\r\n0x00000000004cdb9c in rans_uncompress_O1 (in=0x7cc251 \"\\rO\", in_size=20246, out_size=0x7fffffffc708) at cram/rANS_static.c:675\r\n675                             D[l1].R[m[1]],\r\n(gdb) p D\r\n$1 = (ari_decoder *) 0xa91780\r\n(gdb) p l1\r\n$2 = 60\r\n(gdb) p D[l1]\r\n$3 = {fc = {{F = 0, C = 0} <repeats 256 times>}, R = 0x0}\r\n(gdb) p m[1]\r\n$4 = 2972\r\n(gdb) p D[l1].R\r\n$5 = (unsigned char *) 0x0\r\n(gdb) p D[l1].R[m[1]]\r\nCannot access memory at address 0xb9c\r\n```\r\n\r\nI stepped through the loop that builds the 'D' array of ari_decoder and it seems that it only iterated from i=0 to i=39 (meaning that D[60].R was never initialised): https://github.com/samtools/htslib/blob/develop/cram/rANS_static.c#L591-L640\r\n\r\nThis particular CRAM file was created using scramble 1.14.4 with the command line: `scramble -t 6 -I bam -O cram -`\r\n\r\nI'm not sure if there is an issue here with codec support, but in any case a segfault would not be the most appropriate way to report the error. \r\n"},"comment":{"url":"https://api.github.com/repos/samtools/samtools/issues/comments/220375669","html_url":"https://github.com/samtools/samtools/issues/541#issuecomment-220375669","issue_url":"https://api.github.com/repos/samtools/samtools/issues/541","id":220375669,"user":{"login":"jkbonfield","id":2210525,"avatar_url":"https://avatars.githubusercontent.com/u/2210525?v=3","gravatar_id":"","url":"https://api.github.com/users/jkbonfield","html_url":"https://github.com/jkbonfield","followers_url":"https://api.github.com/users/jkbonfield/followers","following_url":"https://api.github.com/users/jkbonfield/following{/other_user}","gists_url":"https://api.github.com/users/jkbonfield/gists{/gist_id}","starred_url":"https://api.github.com/users/jkbonfield/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jkbonfield/subscriptions","organizations_url":"https://api.github.com/users/jkbonfield/orgs","repos_url":"https://api.github.com/users/jkbonfield/repos","events_url":"https://api.github.com/users/jkbonfield/events{/privacy}","received_events_url":"https://api.github.com/users/jkbonfield/received_events","type":"User","site_admin":false},"created_at":"2016-05-19T16:17:09Z","updated_at":"2016-05-19T16:17:09Z","body":"I have a newer rANS_static.c that has D[].R[] as a static array rather than dynamically allocated.  It also has a bounds check on the input and output buffers.  I **think** this means it'll not crash when given corrupted data, but it's hard to check all cases.  However it may well give complete gibberish back in some cases.\r\n\r\nI don't know how we can even validate the data except perhaps that we decoded the correct number of bytes (rather tenuous).  I'm hoping the issue will resolve itself in a subsequent step."}}