{"action":"created","issue":{"url":"https://api.github.com/repos/grailbio/reflow/issues/90","repository_url":"https://api.github.com/repos/grailbio/reflow","labels_url":"https://api.github.com/repos/grailbio/reflow/issues/90/labels{/name}","comments_url":"https://api.github.com/repos/grailbio/reflow/issues/90/comments","events_url":"https://api.github.com/repos/grailbio/reflow/issues/90/events","html_url":"https://github.com/grailbio/reflow/issues/90","id":380988523,"node_id":"MDU6SXNzdWUzODA5ODg1MjM=","number":90,"title":"Getting `bwa index` to work - aka dealing with bioinf package with naming requirements","user":{"login":"olgabot","id":806256,"node_id":"MDQ6VXNlcjgwNjI1Ng==","avatar_url":"https://avatars2.githubusercontent.com/u/806256?v=4","gravatar_id":"","url":"https://api.github.com/users/olgabot","html_url":"https://github.com/olgabot","followers_url":"https://api.github.com/users/olgabot/followers","following_url":"https://api.github.com/users/olgabot/following{/other_user}","gists_url":"https://api.github.com/users/olgabot/gists{/gist_id}","starred_url":"https://api.github.com/users/olgabot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olgabot/subscriptions","organizations_url":"https://api.github.com/users/olgabot/orgs","repos_url":"https://api.github.com/users/olgabot/repos","events_url":"https://api.github.com/users/olgabot/events{/privacy}","received_events_url":"https://api.github.com/users/olgabot/received_events","type":"User","site_admin":false},"labels":[],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":1,"created_at":"2018-11-15T03:50:23Z","updated_at":"2018-11-27T04:19:10Z","closed_at":null,"author_association":"CONTRIBUTOR","body":"Like many bioinformatics packages, `bwa index` relies on creating files based on the input filenames. Here's the stdout for running `bwa index` locally:\r\n\r\n\r\n<details>\r\n\r\n```\r\n(bwa)\r\n Wed 14 Nov - 19:07  ~/scratch/tick-genome/dna/2018-10-10-dovetail \r\n  bwa index ixodes_scapularis_06May2018_kaVVW.fasta\r\n[bwa_index] Pack FASTA... 30.99 sec\r\n[bwa_index] Construct BWT for the packed sequence...\r\n[BWTIncCreate] textLength=3588917780, availableWord=264529096\r\n[BWTIncConstructFromPacked] 10 iterations done. 99999988 characters processed.\r\n[BWTIncConstructFromPacked] 20 iterations done. 199999988 characters processed.\r\n[BWTIncConstructFromPacked] 30 iterations done. 299999988 characters processed.\r\n[BWTIncConstructFromPacked] 40 iterations done. 399999988 characters processed.\r\n[BWTIncConstructFromPacked] 50 iterations done. 499999988 characters processed.\r\n[BWTIncConstructFromPacked] 60 iterations done. 599999988 characters processed.\r\n[BWTIncConstructFromPacked] 70 iterations done. 699999988 characters processed.\r\n[BWTIncConstructFromPacked] 80 iterations done. 799999988 characters processed.\r\n[BWTIncConstructFromPacked] 90 iterations done. 899999988 characters processed.\r\n[BWTIncConstructFromPacked] 100 iterations done. 999999988 characters processed.\r\n[BWTIncConstructFromPacked] 110 iterations done. 1099999988 characters processed.\r\n[BWTIncConstructFromPacked] 120 iterations done. 1199999988 characters processed.\r\n[BWTIncConstructFromPacked] 130 iterations done. 1299999988 characters processed.\r\n[BWTIncConstructFromPacked] 140 iterations done. 1399999988 characters processed.\r\n[BWTIncConstructFromPacked] 150 iterations done. 1499999988 characters processed.\r\n[BWTIncConstructFromPacked] 160 iterations done. 1599999988 characters processed.\r\n[BWTIncConstructFromPacked] 170 iterations done. 1699999988 characters processed.\r\n[BWTIncConstructFromPacked] 180 iterations done. 1799999988 characters processed.\r\n[BWTIncConstructFromPacked] 190 iterations done. 1899999988 characters processed.\r\n[BWTIncConstructFromPacked] 200 iterations done. 1999999988 characters processed.\r\n[BWTIncConstructFromPacked] 210 iterations done. 2099999988 characters processed.\r\n[BWTIncConstructFromPacked] 220 iterations done. 2199999988 characters processed.\r\n[BWTIncConstructFromPacked] 230 iterations done. 2299999988 characters processed.\r\n[BWTIncConstructFromPacked] 240 iterations done. 2399999988 characters processed.\r\n[BWTIncConstructFromPacked] 250 iterations done. 2499999988 characters processed.\r\n[BWTIncConstructFromPacked] 260 iterations done. 2599999988 characters processed.\r\n[BWTIncConstructFromPacked] 270 iterations done. 2699999988 characters processed.\r\n[BWTIncConstructFromPacked] 280 iterations done. 2799999988 characters processed.\r\n[BWTIncConstructFromPacked] 290 iterations done. 2899999988 characters processed.\r\n[BWTIncConstructFromPacked] 300 iterations done. 2995671796 characters processed.\r\n[BWTIncConstructFromPacked] 310 iterations done. 3080783396 characters processed.\r\n[BWTIncConstructFromPacked] 320 iterations done. 3156426804 characters processed.\r\n[BWTIncConstructFromPacked] 330 iterations done. 3223654836 characters processed.\r\n[BWTIncConstructFromPacked] 340 iterations done. 3283403284 characters processed.\r\n[BWTIncConstructFromPacked] 350 iterations done. 3336503828 characters processed.\r\n[BWTIncConstructFromPacked] 360 iterations done. 3383695748 characters processed.\r\n[BWTIncConstructFromPacked] 370 iterations done. 3425636004 characters processed.\r\n[BWTIncConstructFromPacked] 380 iterations done. 3462908564 characters processed.\r\n[BWTIncConstructFromPacked] 390 iterations done. 3496032516 characters processed.\r\n[BWTIncConstructFromPacked] 400 iterations done. 3525469140 characters processed.\r\n4[BWTIncConstructFromPacked] 410 iterations done. 3551628436 characters processed.\r\n[BWTIncConstructFromPacked] 420 iterations done. 3574874836 characters processed.\r\n[bwt_gen] Finished constructing BWT in 427 iterations.\r\n[bwa_index] 1324.65 seconds elapse.\r\n[bwa_index] Update BWT... 14.92 sec\r\n[bwa_index] Pack forward-only FASTA... 18.06 sec\r\n[bwa_index] Construct SA from BWT and Occ... 469.01 sec\r\n[main] Version: 0.7.17-r1188\r\n[main] CMD: bwa index ixodes_scapularis_06May2018_kaVVW.fasta\r\n[main] Real time: 2068.784 sec; CPU: 1857.620 sec\r\n```\r\n</details>\r\n\r\nAfter running `bwa index` locally, there's quite a few files that get output:\r\n\r\n```\r\n(bwa)\r\n Wed 14 Nov - 19:41  ~/scratch/tick-genome/dna/2018-10-10-dovetail \r\n  ll\r\nPermissions Size User Date Modified Name\r\n.rwxr-xr-x  1.8G olga 10 Oct 17:57  ixodes_scapularis_06May2018_kaVVW.fasta\r\n.rwxr-xr-x  8.1M olga 14 Nov 19:33  ixodes_scapularis_06May2018_kaVVW.fasta.amb\r\n.rwxr-xr-x  4.4M olga 14 Nov 19:33  ixodes_scapularis_06May2018_kaVVW.fasta.ann\r\n.rwxr-xr-x  1.8G olga 14 Nov 19:32  ixodes_scapularis_06May2018_kaVVW.fasta.bwt\r\n.rwxr-xr-x  448M olga 14 Nov 19:33  ixodes_scapularis_06May2018_kaVVW.fasta.pac\r\n.rwxr-xr-x  897M olga 14 Nov 19:41  ixodes_scapularis_06May2018_kaVVW.fasta.sa\r\n```\r\n\r\nHere's what I'm trying to do to get `bwa index` to work:\r\n\r\n```\r\nfunc Index(fasta file, id string) = {\r\n\t// Create a bwt index of the fasta\r\n\r\n    d := dirs.Make([id + \".fasta\": fasta])\r\n\r\n    outdir := exec(image := bwa, cpu := 1) (outdir dir) {\"\r\n        cd {{d}}\r\n        bwa index {{id}}.fasta\r\n\t\"}\r\n    d := trace(d)\r\n\r\n     // NOTE: outdir is not being used anywhere, so it wont get evaluated\r\n    outdir := trace(outdir)\r\n    d := trace(d)\r\n\r\n    val (index, _) = dirs.Pick(d, \"*.bwt\")\r\n\r\n    // to force outdir evaluation, remove above line and uncomment below.\r\n\r\n    val (index, _) = outdir ~> dirs.Pick(d, \"*.bwt\")\r\n\r\n    index\r\n}\r\n```\r\n\r\nBut when I inspect either `d` or `outdir`, all they have is the `fasta`!! Where did the output for `bwa index` go??\r\n\r\n```\r\n(kmer-hashing) \r\n Wed 14 Nov - 19:48  ~/code/tick-genome/pre_assembly_qc/full_workflow   origin ☊ olgabot/align-qc ✔ 8☀ \r\n  make tick2                   \r\nsudo reflow run  -local -localdir /home/olga/scratch/reflow-tmp/ -eval \"bottomup\"  ../../reflow/pre-assembly.rf -minlength 135 -output s3://tick-genome/dna/2018-06-28/pre-assembly_v3/ -adapter_list ../nextera_pe_adapters.tsv -references z\r\n[sudo] password for olga: \r\nreflow: run ID: 9444e403\r\n../../reflow/bwa.rf:24:20(bwa.Index.outdir): dir()\r\n../../reflow/bwa.rf:24:20(bwa.Index.outdir): dir()\r\n../../reflow/bwa.rf:21:15(bwa.Index.d): dir(\"dovetail_2018-05-06.fasta\": file(sha256=sha256:3a72c5bb77cb0a133ce77b8055d79878c040bf34de2c7b4f513592d8070ac0ac, size=464355639))\r\n../../reflow/bwa.rf:21:15(bwa.Index.d): dir(\"jcvi_ise6.fasta\": file(sha256=sha256:766bbd89408379f12e644f242511db72854c348f349db59e601d2c42bdcd7d1b, size=1001461460))\r\n../../reflow/bwa.rf:25:15(bwa.Index.d): dir(\"dovetail_2018-05-06.fasta\": file(sha256=sha256:3a72c5bb77cb0a133ce77b8055d79878c040bf34de2c7b4f513592d8070ac0ac, size=464355639))\r\n../../reflow/bwa.rf:25:15(bwa.Index.d): dir(\"jcvi_ise6.fasta\": file(sha256=sha256:766bbd89408379f12e644f242511db72854c348f349db59e601d2c42bdcd7d1b, size=1001461460))\r\nreflow: assoc.Get flow 69a026d0 state FlowLookup {} extern url \"s3://tick-genome/dna/2018-06-28/pre-assembly_v3/\" deps cef9af32: RequestCanceled: request context canceled\r\ncaused by: context canceled\r\nreflow: assoc.Get flow 69a026d0 state FlowLookup {} extern url \"s3://tick-genome/dna/2018-06-28/pre-assembly_v3/\" deps cef9af32: RequestCanceled: request context canceled\r\ncaused by: context canceled\r\ndirs.Pick: no files matched *.bwt\r\nMakefile:18: recipe for target 'tick2' failed\r\nmake: *** [tick2] Error 11\r\n```"},"comment":{"url":"https://api.github.com/repos/grailbio/reflow/issues/comments/441902075","html_url":"https://github.com/grailbio/reflow/issues/90#issuecomment-441902075","issue_url":"https://api.github.com/repos/grailbio/reflow/issues/90","id":441902075,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MTkwMjA3NQ==","user":{"login":"olgabot","id":806256,"node_id":"MDQ6VXNlcjgwNjI1Ng==","avatar_url":"https://avatars2.githubusercontent.com/u/806256?v=4","gravatar_id":"","url":"https://api.github.com/users/olgabot","html_url":"https://github.com/olgabot","followers_url":"https://api.github.com/users/olgabot/followers","following_url":"https://api.github.com/users/olgabot/following{/other_user}","gists_url":"https://api.github.com/users/olgabot/gists{/gist_id}","starred_url":"https://api.github.com/users/olgabot/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/olgabot/subscriptions","organizations_url":"https://api.github.com/users/olgabot/orgs","repos_url":"https://api.github.com/users/olgabot/repos","events_url":"https://api.github.com/users/olgabot/events{/privacy}","received_events_url":"https://api.github.com/users/olgabot/received_events","type":"User","site_admin":false},"created_at":"2018-11-27T04:19:10Z","updated_at":"2018-11-27T04:19:10Z","author_association":"CONTRIBUTOR","body":"I also don't see any of the stdout/stderr normally produced by the `bwa index` command, even in the execlog. Is there a way to find it?"}}
{"actor":{"display_login":"olgabot"}}