{"action":"opened","issue":{"url":"https://api.github.com/repos/PacificBiosciences/pbalign/issues/80","repository_url":"https://api.github.com/repos/PacificBiosciences/pbalign","labels_url":"https://api.github.com/repos/PacificBiosciences/pbalign/issues/80/labels{/name}","comments_url":"https://api.github.com/repos/PacificBiosciences/pbalign/issues/80/comments","events_url":"https://api.github.com/repos/PacificBiosciences/pbalign/issues/80/events","html_url":"https://github.com/PacificBiosciences/pbalign/issues/80","id":293955330,"number":80,"title":"Best practices for running bpalign on large numbers of input files","user":{"login":"steinjosh","id":1790907,"avatar_url":"https://avatars2.githubusercontent.com/u/1790907?v=4","gravatar_id":"","url":"https://api.github.com/users/steinjosh","html_url":"https://github.com/steinjosh","followers_url":"https://api.github.com/users/steinjosh/followers","following_url":"https://api.github.com/users/steinjosh/following{/other_user}","gists_url":"https://api.github.com/users/steinjosh/gists{/gist_id}","starred_url":"https://api.github.com/users/steinjosh/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/steinjosh/subscriptions","organizations_url":"https://api.github.com/users/steinjosh/orgs","repos_url":"https://api.github.com/users/steinjosh/repos","events_url":"https://api.github.com/users/steinjosh/events{/privacy}","received_events_url":"https://api.github.com/users/steinjosh/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":0,"created_at":"2018-02-02T16:57:32Z","updated_at":"2018-02-02T16:57:32Z","closed_at":null,"author_association":"NONE","body":"Hi,\r\nMy goal is to align subreads from 90 cells to a de novo assembly of those data in order to perform polishing using variantCaller (smrtlink/5.0.1).  My data uses the old standard of having three *.bax.h5 files/cell which I converted into 90 corresponding *.subread.bam files using the bax2bam tool.  To proceed I am testing 2 different options:\r\nOption A: package the 90 subread.bam files into a dataset.xml  using \"dataset create --type SubreadSet\". Then run pbalign on the dataset.xml as input. As I understand it, the advantage of this is that a single pbalign alignment can be run and the output align.bam would be properly sorted as a single file ready for input into variantCaller.  A possible disadvantage would be limitations on run time or memory on the cluster I am working on.  I tried this option and got a failed alignment but was unable to interpret the error messages (see below--and please give feedback if you can). It worked fine when tested on a dataset.xml file containing only  one of the subread.bam files but not when all 90 were included in the dataset.xml.\r\nOption B: Run pbalign on each of 90 subread.bam files, then sort and merge all of the resulting align.bam files into a single file using bamtools, before feeding to variantCaller.  If this is the better option, I could use advice on the best way to call bamtools, e.g. \"bamtools merge *input_files.bam | bamtools sort\", etc.\r\n\r\nDo I have the gist of it or is there a third option I should be aware of?\r\n\r\nHere is the error from when trying to run pbalign on a dataset.xml file containing 90 subread.bam files:\r\n```\r\n$ less stderr.192740_4294967294 \r\nBlasrService returned a non-zero exit status. \r\nBlasrService returned a non-zero exit status. \r\nTraceback (most recent call last):\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbcommand/lib/python2.7/site-packages/pbcommand/cli/core.py\", line 137, in _pacbio_main_runner\r\n    return_code = exe_main_func(*args, **kwargs)\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbalign/lib/python2.7/site-packages/pbalign/pbalignrunner.py\", line 283, in args_runner\r\n    return PBAlignRunner(args, output_dataset_type=output_dataset_type).start()\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbcore/lib/python2.7/site-packages/pbcore/util/ToolRunner.py\", line 112, in start\r\n    return self.run()\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbalign/lib/python2.7/site-packages/pbalign/pbalignrunner.py\", line 239, in run\r\n    self._alnService.run()\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbalign/lib/python2.7/site-packages/pbalign/alignservice/align.py\", line 198, in run\r\n    raise RuntimeError(str(e))\r\nRuntimeError: BlasrService returned a non-zero exit status. \r\n```\r\nHere is additional messaging from stdout:\r\n```\r\n$ less stdout.192740_4294967294 \r\n[INFO] 2018-01-31 20:26:20,846Z [pbalign.pbalignrunner _pacbio_main_runner 129] Using pbcommand v0.6.3\r\n[INFO] 2018-01-31 20:26:20,846Z [pbalign.pbalignrunner _pacbio_main_runner 130] completed setting up logger with <function setup_log at 0x2b833eeba578>\r\n[INFO] 2018-01-31 20:26:20,847Z [pbalign.pbalignrunner _pacbio_main_runner 131] log opts {'file_name': None, 'level': 20}\r\n[INFO] 2018-01-31 20:26:20,847Z [root run 226] pbalign version: 0.3.1\r\n[INFO] 2018-01-31 20:26:20,898Z [root run 178] BlasrService: Align reads to references using blasr.\r\n[INFO] 2018-01-31 20:26:20,899Z [root Execute 63] BlasrService: Call \"blasr /local/scratch/steinj/cargold_set.xml /local/scratch/steinj/cargold.contigs.fasta --out /local/scratch/tmpnQXkMr/WnIm5B.bam  --bam  --bestn 10 --minMatch 12  --maxMatch 30  --nproc 40  --minSubreadLength 50 --minAlnLength 50  --minPctSimilarity 70 --minPctAccuracy 70 --hitPolicy randombest  --randomSeed 1 \"\r\n[ERROR] 2018-02-01 22:10:36,360Z [root Execute 67] BlasrService returned a non-zero exit status. \r\n[ERROR] 2018-02-01 22:10:36,481Z [pbalign.pbalignrunner _pacbio_main_runner 142] BlasrService returned a non-zero exit status. \r\nTraceback (most recent call last):\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbcommand/lib/python2.7/site-packages/pbcommand/cli/core.py\", line 137, in _pacbio_main_runner\r\n    return_code = exe_main_func(*args, **kwargs)\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbalign/lib/python2.7/site-packages/pbalign/pbalignrunner.py\", line 283, in args_runner\r\n    return PBAlignRunner(args, output_dataset_type=output_dataset_type).start()\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbcore/lib/python2.7/site-packages/pbcore/util/ToolRunner.py\", line 112, in start\r\n    return self.run()\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbalign/lib/python2.7/site-packages/pbalign/pbalignrunner.py\", line 239, in run\r\n    self._alnService.run()\r\n  File \"/software/smrtanalysis/smrtlink/install/smrtlink-release_5.0.1.9585/bundles/smrttools/install/smrttools-release_5.0.1.9578/private/pacbio/pythonpkgs/pbalign/lib/python2.7/site-packages/pbalign/alignservice/align.py\", line 198, in run\r\n    raise RuntimeError(str(e))\r\nRuntimeError: BlasrService returned a non-zero exit status. \r\n[INFO] 2018-02-01 22:10:36,889Z [pbalign.pbalignrunner _pacbio_main_runner 154] exiting with return code 2 in 92655.64 sec.\r\n```\r\n\r\nAny clue how I could re-run the above (feeding all 90 subread files into pbalign with a dataset.xml input) to avoid what happened here?  The SLURM job itself actually completed after 2 days with ExitCode 0 and it returned the output.bam file but with zero file size.  I had allotted 5 days, on a node with a 7 day limit and 3.2 GB per core.\r\n\r\nThank you for any advice you may have.\r\nBest,\r\nJosh\r\n\r\n"}}
{"actor":{"display_login":"steinjosh"}}