{"action":"created","issue":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1382","repository_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen","labels_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1382/labels{/name}","comments_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1382/comments","events_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1382/events","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1382","id":153384834,"number":1382,"title":"Consistent BWA reference files not found error upon re-installing genome files.","user":{"login":"mortunco","id":12833865,"avatar_url":"https://avatars.githubusercontent.com/u/12833865?v=3","gravatar_id":"","url":"https://api.github.com/users/mortunco","html_url":"https://github.com/mortunco","followers_url":"https://api.github.com/users/mortunco/followers","following_url":"https://api.github.com/users/mortunco/following{/other_user}","gists_url":"https://api.github.com/users/mortunco/gists{/gist_id}","starred_url":"https://api.github.com/users/mortunco/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mortunco/subscriptions","organizations_url":"https://api.github.com/users/mortunco/orgs","repos_url":"https://api.github.com/users/mortunco/repos","events_url":"https://api.github.com/users/mortunco/events{/privacy}","received_events_url":"https://api.github.com/users/mortunco/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":1,"created_at":"2016-05-06T06:12:52Z","updated_at":"2016-05-06T10:22:37Z","closed_at":null,"body":"Hi Brad;\r\n\r\nI am keep getting a error related hg38 BWA genome files consistently. I have tried ```bcbio_nextgen.py --upgrade hg38``` 3 times and in all of them I get successful installation output. When I check the bwa hg38 reference directory. I also found some files about bwa which confused me. Could you help me out to get around with this issue ? Am I misssing a point ?\r\n\r\nThank you very much, \r\n\r\nBest regards,\r\n\r\nT. \r\n\r\n\r\n**Error**\r\n```\r\n[tmorova15@lufer work]$ cat deneme1error \r\n[2016-05-05T20:57Z] Resource requests: bwa, sambamba, samtools; memory: 2.00, 2.00; cores: 16, 16, 16\r\n[2016-05-05T20:57Z] Configuring 1 jobs to run, using 16 cores each with 32.1g of memory reserved for each job\r\n[2016-05-05T20:57Z] Timing: organize samples\r\n[2016-05-05T20:57Z] multiprocessing: organize_samples\r\n[2016-05-05T20:57Z] Using input YAML configuration: /mnt/scratch/tmorova15/run/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml\r\n[2016-05-05T20:57Z] Checking sample YAML configuration: /mnt/scratch/tmorova15/run/cancer-syn3-chr6/config/cancer-syn3-chr6.yaml\r\n[2016-05-05T20:57Z] Downloading hg38 bwa from AWS\r\nTraceback (most recent call last):\r\n  File \"/mnt/scratch/tmorova15/bcbio/bin/bcbio_nextgen.py\", line 4, in <module>\r\n    __import__('pkg_resources').run_script('bcbio-nextgen==0.9.7', 'bcbio_nextgen.py')\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/setuptools-20.7.0-py2.7.egg/pkg_resources/__init__.py\", line 719, in run_script\r\n    \r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/setuptools-20.7.0-py2.7.egg/pkg_resources/__init__.py\", line 1504, in run_script\r\n    \r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio_nextgen-0.9.7-py2.7.egg-info/scripts/bcbio_nextgen.py\", line 226, in <module>\r\n    main(**kwargs)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio_nextgen-0.9.7-py2.7.egg-info/scripts/bcbio_nextgen.py\", line 43, in main\r\n    run_main(**kwargs)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 39, in run_main\r\n    fc_dir, run_info_yaml)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 82, in _run_toplevel\r\n    for xs in pipeline(config, run_info_yaml, parallel, dirs, samples):\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/main.py\", line 115, in variant2pipeline\r\n    [x[0][\"description\"] for x in samples]]])\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py\", line 28, in run_parallel\r\n    return run_multicore(fn, items, config, parallel=parallel)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multi.py\", line 86, in run_multicore\r\n    for data in joblib.Parallel(parallel[\"num_jobs\"])(joblib.delayed(fn)(x) for x in items):\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 800, in __call__\r\n    while self.dispatch_one_batch(iterator):\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 658, in dispatch_one_batch\r\n    self._dispatch(tasks)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 566, in _dispatch\r\n    job = ImmediateComputeBatch(batch)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 180, in __init__\r\n    self.results = batch()\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/joblib/parallel.py\", line 72, in __call__\r\n    return [func(*args, **kwargs) for func, args, kwargs in self.items]\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/utils.py\", line 51, in wrapper\r\n    return apply(f, *args, **kwargs)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/distributed/multitasks.py\", line 295, in organize_samples\r\n    return run_info.organize(*args)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/run_info.py\", line 66, in organize\r\n    item = add_reference_resources(item, remote_retriever)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/run_info.py\", line 157, in add_reference_resources\r\n    data[\"reference\"] = genome.get_refs(data[\"genome_build\"], aligner, data[\"dirs\"][\"galaxy\"], data)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/genome.py\", line 207, in get_refs\r\n    galaxy_config, data)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/genome.py\", line 159, in _get_ref_from_galaxy_loc\r\n    cur_ref = download_prepped_genome(genome_build, data, name, need_remap)\r\n  File \"/mnt/scratch/tmorova15/bcbio/anaconda/lib/python2.7/site-packages/bcbio/pipeline/genome.py\", line 277, in download_prepped_genome\r\n    raise ValueError(\"Could not find reference genome file %s %s\" % (genome_build, name))\r\nValueError: Could not find reference genome file hg38 bwa\r\n\r\n```\r\n**BWA folder**\r\n```\r\n/mnt/scratch/tmorova15/bcbio/genomes/Hsapiens/hg38/bwa\r\n[tmorova15@lufer bwa]$ ls\r\nhg38.fa.alt  hg38.fa.amb  hg38.fa.ann  hg38.fa.bwt  hg38.fa.pac  hg38.fa.sa\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/comments/217405387","html_url":"https://github.com/chapmanb/bcbio-nextgen/issues/1382#issuecomment-217405387","issue_url":"https://api.github.com/repos/chapmanb/bcbio-nextgen/issues/1382","id":217405387,"user":{"login":"chapmanb","id":39391,"avatar_url":"https://avatars.githubusercontent.com/u/39391?v=3","gravatar_id":"","url":"https://api.github.com/users/chapmanb","html_url":"https://github.com/chapmanb","followers_url":"https://api.github.com/users/chapmanb/followers","following_url":"https://api.github.com/users/chapmanb/following{/other_user}","gists_url":"https://api.github.com/users/chapmanb/gists{/gist_id}","starred_url":"https://api.github.com/users/chapmanb/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/chapmanb/subscriptions","organizations_url":"https://api.github.com/users/chapmanb/orgs","repos_url":"https://api.github.com/users/chapmanb/repos","events_url":"https://api.github.com/users/chapmanb/events{/privacy}","received_events_url":"https://api.github.com/users/chapmanb/received_events","type":"User","site_admin":false},"created_at":"2016-05-06T10:22:37Z","updated_at":"2016-05-06T10:22:37Z","body":"Tunc;\r\nSorry about the problem. I'm not sure what exactly is going wrong with your installation, but it seems like it's having trouble finding the right configuration directory with file paths when running. My guess is that it might be picking up a different bcbio_system.yaml file when running that is not your installed one. I have a flag that helps debug this in the latest development. If you update to that:\r\n```\r\nbcbio_nextgen.py upgrade -u development\r\n```\r\nand then re-run there should be a logging line about the bcbio system YAML file used for the run. Is that the one in `/mnt/scratch/tmorova15/bcbio/galaxy/bcbio_system.yaml` or somewhere else? If somewhere else it might not be pointing at the installed reference data and thus be unable to find it. Hope this helps some with debugging and happy to try and provide more pointers with this information."}}