{"action":"created","issue":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen-vm/issues/140","repository_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen-vm","labels_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen-vm/issues/140/labels{/name}","comments_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen-vm/issues/140/comments","events_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen-vm/issues/140/events","html_url":"https://github.com/chapmanb/bcbio-nextgen-vm/issues/140","id":140810103,"number":140,"title":"Reproducing Example: Tumor Paired Variant Calling","user":{"login":"mortunco","id":12833865,"avatar_url":"https://avatars.githubusercontent.com/u/12833865?v=3","gravatar_id":"","url":"https://api.github.com/users/mortunco","html_url":"https://github.com/mortunco","followers_url":"https://api.github.com/users/mortunco/followers","following_url":"https://api.github.com/users/mortunco/following{/other_user}","gists_url":"https://api.github.com/users/mortunco/gists{/gist_id}","starred_url":"https://api.github.com/users/mortunco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mortunco/subscriptions","organizations_url":"https://api.github.com/users/mortunco/orgs","repos_url":"https://api.github.com/users/mortunco/repos","events_url":"https://api.github.com/users/mortunco/events{/privacy}","received_events_url":"https://api.github.com/users/mortunco/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":2,"created_at":"2016-03-14T22:02:53Z","updated_at":"2016-03-15T01:18:48Z","closed_at":null,"body":"Hi, \r\nI would like to reproduce Tumor paired variant calling for better understanding bcbio. I am trying to operate this task on AWS EC2 machines however I am interfering with problems.\r\n\r\nFirst;\r\nI believe I installed it correctly. I followed [the first](https://github.com/chapmanb/bcbio-nextgen-vm) step and it installed bcbio_vm to my amazon AWS EC2 machine.\r\n\r\n1) My first question is, as far I understand tutorial in here [https://bcbio-nextgen.readthedocs.org/en/latest/contents/cloud.html](url) is about running bcbio_vm without login in to virtual machine ( with ssh ). However, I think I should be able to run these commands also by loging in to the server? Please clarify me I am misunderstand.\r\n\r\nSecond;\r\n2) I attached a Elastic Block Store to my AWS instance therefore I allocated my bam files. I am stating this because in the documentation, it was always referred to bucket. Which one is the right one ? Should I change elastic block store with Bucket?\r\n\r\n3)I tried to run the process with the following command. Also I changed the locations of the files to the pwd of their location.\r\n\r\n\r\n\r\nI would prefer working on amazon servers while logged in to the instance. Is this a wrong methodology for working in a such a project? My aim is to setup a pipeline that will call mutations for ~250 samples ( paired timor-normal). I am ready to what it is needed to be done. \r\n\r\nThank you very much for your help and patience.\r\n\r\nTunc.\r\n\r\n\r\nMy error;\r\n```\r\n[ec2-user@ip-172-31-51-202 work]$ bcbio_vm.py run ../config/cancer-dream-syn3.yaml -n 2\r\nMust specify a `--datadir` or save the default location with `saveconfig`.\r\nCould not run analysis.\r\n```\r\n\r\nMy config file;\r\n```\r\n# Cancer tumor/normal calling evaluation using synthetic dataset 3\r\n# from the ICGC-TCGA DREAM challenge:\r\n# https://www.synapse.org/#!Synapse:syn312572/wiki/62018\r\n---\r\ndetails:\r\n- algorithm:\r\n    aligner: bwa\r\n    align_split_size: 5000000\r\n    nomap_split_targets: 100\r\n    mark_duplicates: true\r\n    recalibrate: true\r\n    realign: true\r\n    remove_lcr: true\r\n    platform: illumina\r\n    quality_format: standard\r\n    variantcaller: [mutect, freebayes, vardict, varscan]\r\n    indelcaller: false\r\n    ensemble:\r\n      numpass: 2\r\n    variant_regions: /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/input/NGv3.bed\r\n    # svcaller: [cnvkit, lumpy, delly]\r\n    # coverage_interval: amplicon\r\n  analysis: variant2\r\n  description: syn3-normal\r\n  #files: ../input/synthetic.challenge.set3.normal.bam\r\n  files:\r\n    - /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/input/synthetic_challenge_set3_normal_NGv3_1.fq.gz\r\n    - /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/input/synthetic_challenge_set3_normal_NGv3_2.fq.gz\r\n  genome_build: GRCh37\r\n  metadata:\r\n    batch: syn3\r\n    phenotype: normal\r\n- algorithm:\r\n    aligner: bwa\r\n    align_split_size: 5000000\r\n    nomap_split_targets: 100\r\n    mark_duplicates: true\r\n    recalibrate: true\r\n    realign: true\r\n    remove_lcr: true\r\n    platform: illumina\r\n    quality_format: standard\r\n    variantcaller: [mutect, freebayes, vardict, varscan]\r\n    indelcaller: false\r\n    ensemble:\r\n      numpass: 2\r\n    variant_regions: /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/input/NGv3.bed\r\n    validate: /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/input/synthetic_challenge_set3_tumor_20pctmasked_truth.vcf.gz\r\n    validate_regions: /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/input/synthetic_challenge_set3_tumor_20pctmasked_truth_regions.bed\r\n    # svcaller: [cnvkit, lumpy, delly]\r\n    # coverage_interval: amplicon\r\n  #   svvalidate:\r\n  #     DEL: ../input/synthetic_challenge_set3_tumor_20pctmasked_truth_sv_DEL.bed\r\n  #     DUP: ../input/synthetic_challenge_set3_tumor_20pctmasked_truth_sv_DUP.bed\r\n  #     INS: ../input/synthetic_challenge_set3_tumor_20pctmasked_truth_sv_INS.bed\r\n  #     INV: ../input/synthetic_challenge_set3_tumor_20pctmasked_truth_sv_INV.bed\r\n  analysis: variant2\r\n  description: syn3-tumor\r\n  #files: ../input/synthetic.challenge.set3.tumor.bam\r\n  files:\r\n    - /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/input/synthetic_challenge_set3_tumor_NGv3_1.fq.gz\r\n    - /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/input/synthetic_challenge_set3_tumor_NGv3_2.fq.gz\r\n  genome_build: GRCh37\r\n  metadata:\r\n    batch: syn3\r\n    phenotype: tumor\r\nfc_date: '2014-08-13'\r\nfc_name: dream-syn3\r\nupload:\r\n  dir: /home/ec2-user/dendi/bcbio_sampledata/cancer-dream-syn3/final\r\n```\r\n\r\n\r\nProof that I think I installed right. \r\n```\r\n[ec2-user@ip-172-31-51-202 work]$ bcbio_vm.py  -h\r\nusage: bcbio_vm.py [-h] [--datadir DATADIR]\r\n                   {run,install,upgrade,ipython,ipythonprep,template,cwl,aws,elasticluster,graph,clusterk,runfn,devel,saveconfig}\r\n                   ...\r\n\r\nAutomatic installation for bcbio-nextgen pipelines, with docker.\r\n\r\noptional arguments:\r\n  -h, --help            show this help message and exit\r\n  --datadir DATADIR     Directory with genome data and associated files.\r\n\r\n[sub-commands]:\r\n  {run,install,upgrade,ipython,ipythonprep,template,cwl,aws,elasticluster,graph,clusterk,runfn,devel,saveconfig}\r\n    run                 Run an automated analysis on the local machine.\r\n    install             Install or upgrade bcbio-nextgen docker container and\r\n                        data.\r\n    upgrade             Install or upgrade bcbio-nextgen docker container and\r\n                        data.\r\n    ipython             Run on a cluster using IPython parallel.\r\n    ipythonprep         Prepare a batch script to run bcbio on a scheduler.\r\n    template            Create a bcbio sample.yaml file from a standard\r\n                        template and inputs\r\n    cwl                 Prepare a common workflow language (CWL) input\r\n    aws                 Automate resources for running bcbio on AWS\r\n    elasticluster       Interface to standard elasticluster commands\r\n    graph               Generate system graphs (CPU/memory/network/disk I/O\r\n                        consumption) from bcbio runs\r\n    clusterk            Run on Amazon web services using Clusterk.\r\n    runfn               Run a specific bcbio-nextgen function with provided\r\n                        arguments\r\n    devel               Utilities to help with develping using bcbion inside\r\n                        of docker\r\n    saveconfig          Save standard configuration variables for current\r\n                        user. Avoids need to specify on the command line in\r\n                        future runs.\r\n```"},"comment":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen-vm/issues/comments/196596401","html_url":"https://github.com/chapmanb/bcbio-nextgen-vm/issues/140#issuecomment-196596401","issue_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen-vm/issues/140","id":196596401,"user":{"login":"chapmanb","id":39391,"avatar_url":"https://avatars.githubusercontent.com/u/39391?v=3","gravatar_id":"","url":"https://api.github.com/users/chapmanb","html_url":"https://github.com/chapmanb","followers_url":"https://api.github.com/users/chapmanb/followers","following_url":"https://api.github.com/users/chapmanb/following{/other_user}","gists_url":"https://api.github.com/users/chapmanb/gists{/gist_id}","starred_url":"https://api.github.com/users/chapmanb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chapmanb/subscriptions","organizations_url":"https://api.github.com/users/chapmanb/orgs","repos_url":"https://api.github.com/users/chapmanb/repos","events_url":"https://api.github.com/users/chapmanb/events{/privacy}","received_events_url":"https://api.github.com/users/chapmanb/received_events","type":"User","site_admin":false},"created_at":"2016-03-15T01:18:48Z","updated_at":"2016-03-15T01:18:48Z","body":"Sorry about the confusion on installation. For installing yourself manually on an AWS instance, you need to follow all the bcbio-vm instructions not just the first step. Reading the documentation now that is confusing and I'll try to clarify this more. You can only stop after the first step if you're using the full bcbio approach to managing AWS instances (https://bcbio-nextgen.readthedocs.org/en/latest/contents/cloud.html), since this does the work of setting up bcbio and data dependencies on AWS for you.\r\n\r\nBeyond that your other steps look right. You don't need to use data in S3 buckets, it's only a convenience if you prefer to store them that way, but referencing local files created on EBS volumes works great. Hopefully things run cleanly for you after finalizing the installation. Please let us know if you run into any other problems."}}