{"action":"created","issue":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2437","repository_url":"https://api.github.com/repos/galaxyproject/galaxy","labels_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2437/labels{/name}","comments_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2437/comments","events_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2437/events","html_url":"https://github.com/galaxyproject/galaxy/issues/2437","id":157744321,"number":2437,"title":"Galaxy 16.04 API: Invoking a workflow that includes tools passed as RuntimeValue objects throws exception","user":{"login":"gregvonkuster","id":6947053,"avatar_url":"https://avatars.githubusercontent.com/u/6947053?v=3","gravatar_id":"","url":"https://api.github.com/users/gregvonkuster","html_url":"https://github.com/gregvonkuster","followers_url":"https://api.github.com/users/gregvonkuster/followers","following_url":"https://api.github.com/users/gregvonkuster/following{/other_user}","gists_url":"https://api.github.com/users/gregvonkuster/gists{/gist_id}","starred_url":"https://api.github.com/users/gregvonkuster/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gregvonkuster/subscriptions","organizations_url":"https://api.github.com/users/gregvonkuster/orgs","repos_url":"https://api.github.com/users/gregvonkuster/repos","events_url":"https://api.github.com/users/gregvonkuster/events{/privacy}","received_events_url":"https://api.github.com/users/gregvonkuster/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/area/workflows","name":"area/workflows","color":"006b75"},{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/kind/bug","name":"kind/bug","color":"84b6eb"},{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/major","name":"major","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-05-31T18:56:01Z","updated_at":"2016-06-07T15:20:18Z","closed_at":null,"body":"Tools that include dynamically generated select lists for reference genomes will produce this behavior.  One example is this tool: https://toolshed.g2.bx.psu.edu/view/iuc/meme_fimo/cd54079f0f72\r\n\r\nWhen invoked, a workflow that includes the above tool will ultimately call the collect_input_datasets() method in the DefaultToolAction class within the `~/lib/galaxy/tools/action/__init__.py` code file.  The `visitor()` method's `process_dataset()` method will receive a `galaxy.tools.parameters.basic.RuntimeValue` object as the value of the data parameter, resulting in the following exception being thrown:\r\n\r\n```\r\ngalaxy.tools ERROR 2016-05-31 14:36:55,952 Exception caught while attempting tool execution:\r\nTraceback (most recent call last):\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/__init__.py\", line 1142, in handle_single_execution\r\n    job, out_data = self.execute( trans, incoming=params, history=history, rerun_remap_job_id=rerun_remap_job_id, mapping_over_collection=mapping_over_collection, execution_cache=execution_cache )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/__init__.py\", line 1220, in execute\r\n    return self.tool_action.execute( self, trans, incoming=incoming, set_output_hid=set_output_hid, history=history, **kwargs )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py\", line 221, in execute\r\n    history, inp_data, inp_dataset_collections = self._collect_inputs(tool, trans, incoming, history, current_user_roles)\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py\", line 206, in _collect_inputs\r\n    inp_data = self.collect_input_datasets( tool, incoming, trans, current_user_roles=current_user_roles )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py\", line 154, in collect_input_datasets\r\n    tool.visit_inputs( param_values, visitor )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/__init__.py\", line 1066, in visit_inputs\r\n    visit_input_values( self.inputs, values, callback )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/parameters/__init__.py\", line 110, in visit_input_values\r\n    callback_helper( input, input_values, name_prefix, label_prefix, parent_prefix=parent_prefix, context=context )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/parameters/__init__.py\", line 75, in callback_helper\r\n    new_value = callback( **args )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py\", line 117, in visitor\r\n    input_datasets[ prefix + input.name ] = process_dataset( value )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/actions/__init__.py\", line 66, in process_dataset\r\n    if not data.datatype.matches_any( formats ):\r\nAttributeError: 'RuntimeValue' object has no attribute 'datatype'\r\n```\r\nThe resulting job will fail with the following error:\r\n\r\n```\r\ngalaxy.tools.execute WARNING 2016-05-31 14:36:55,961 There was a failure executing a job for tool [toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.65] - Error executing tool: 'RuntimeValue' object has no attribute 'datatype'\r\ngalaxy.tools.execute DEBUG 2016-05-31 14:36:55,961 Executed 1 job(s) for tool toolshed.g2.bx.psu.edu/repos/devteam/fastqc/fastqc/0.65 request: (43.538 ms)\r\ngalaxy.workflow.run ERROR 2016-05-31 14:36:55,982 Failed to schedule Workflow[id=21,name=paired_001], problem occurred on WorkflowStep[index=3,type=tool].\r\nTraceback (most recent call last):\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/workflow/run.py\", line 169, in invoke\r\n    jobs = self._invoke_step( step )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/workflow/run.py\", line 239, in _invoke_step\r\n    jobs = step.module.execute( self.trans, self.progress, self.workflow_invocation, step )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/workflow/modules.py\", line 1090, in execute\r\n    raise Exception(message)\r\nException: Failed to create 1 out of 1 job(s) for workflow step.\r\ngalaxy.workflow.run ERROR 2016-05-31 14:36:55,984 Failed to execute scheduled workflow.\r\nTraceback (most recent call last):\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/workflow/run.py\", line 82, in __invoke\r\n    outputs = invoker.invoke()\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/workflow/run.py\", line 169, in invoke\r\n    jobs = self._invoke_step( step )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/workflow/run.py\", line 239, in _invoke_step\r\n    jobs = step.module.execute( self.trans, self.progress, self.workflow_invocation, step )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/workflow/modules.py\", line 1090, in execute\r\n    raise Exception(message)\r\nException: Failed to create 1 out of 1 job(s) for workflow step.\r\n```\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/comments/224314814","html_url":"https://github.com/galaxyproject/galaxy/issues/2437#issuecomment-224314814","issue_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2437","id":224314814,"user":{"login":"gregvonkuster","id":6947053,"avatar_url":"https://avatars.githubusercontent.com/u/6947053?v=3","gravatar_id":"","url":"https://api.github.com/users/gregvonkuster","html_url":"https://github.com/gregvonkuster","followers_url":"https://api.github.com/users/gregvonkuster/followers","following_url":"https://api.github.com/users/gregvonkuster/following{/other_user}","gists_url":"https://api.github.com/users/gregvonkuster/gists{/gist_id}","starred_url":"https://api.github.com/users/gregvonkuster/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gregvonkuster/subscriptions","organizations_url":"https://api.github.com/users/gregvonkuster/orgs","repos_url":"https://api.github.com/users/gregvonkuster/repos","events_url":"https://api.github.com/users/gregvonkuster/events{/privacy}","received_events_url":"https://api.github.com/users/gregvonkuster/received_events","type":"User","site_admin":false},"created_at":"2016-06-07T15:20:17Z","updated_at":"2016-06-07T15:20:17Z","body":"@jmchilton @guerler we are definitely getting closer on this - these 2 PRs have enabled significant progress: https://github.com/galaxyproject/galaxy/pull/2458, https://github.com/galaxyproject/galaxy/pull/2456.\r\n\r\nI can now attempt to fix my API calls that worked with 15.10 to now work with 16.04.  I am attempting to use the API to set the inputs for the workflow, but code comments on the bioblend side seem to be a bit different from the Galaxy side, so I'm hoping you can help.\r\n\r\nIf I have a simple workflow with 2 inputs (which are single HDAs, not collections) and the single \"Map with BWA-MEM tool whose api call (~/api/workflows/484ad813eab74746) returns this dictionary:\r\n```\r\n{\r\n    \"annotation\": \"Map with BWA-MEM\", \r\n    \"deleted\": false, \r\n    \"id\": \"484ad813eab74746\", \r\n    \"inputs\": {\r\n        \"0\": {\r\n            \"label\": \"R1\", \r\n            \"uuid\": \"45d551a9-e579-4de7-9dcb-3c9a86c9cd70\", \r\n            \"value\": \"\"\r\n        }, \r\n        \"1\": {\r\n            \"label\": \"R2\", \r\n            \"uuid\": \"3fc8f5fd-5fa6-4eb2-bd3b-8ca8b27afccd\", \r\n            \"value\": \"\"\r\n        }\r\n    }, \r\n    \"latest_workflow_uuid\": \"de7bab89-e179-4c27-97da-aab697c830c8\", \r\n    \"model_class\": \"StoredWorkflow\", \r\n    \"name\": \"paired_001\", \r\n    \"owner\": \"cegr\", \r\n    \"published\": false, \r\n    \"steps\": {\r\n        \"0\": {\r\n            \"annotation\": \"Fastqsanger input 1\", \r\n            \"id\": 0, \r\n            \"input_steps\": {}, \r\n            \"tool_id\": null, \r\n            \"tool_inputs\": {\r\n                \"name\": \"R1\"\r\n            }, \r\n            \"tool_version\": null, \r\n            \"type\": \"data_input\"\r\n        }, \r\n        \"1\": {\r\n            \"annotation\": \"Fastqsanger input 2\", \r\n            \"id\": 1, \r\n            \"input_steps\": {}, \r\n            \"tool_id\": null, \r\n            \"tool_inputs\": {\r\n                \"name\": \"R2\"\r\n            }, \r\n            \"tool_version\": null, \r\n            \"type\": \"data_input\"\r\n        }, \r\n        \"2\": {\r\n            \"annotation\": null, \r\n            \"id\": 2, \r\n            \"input_steps\": {\r\n                \"fastq_input|fastq_input1\": {\r\n                    \"source_step\": 0, \r\n                    \"step_output\": \"output\"\r\n                }, \r\n                \"fastq_input|fastq_input2\": {\r\n                    \"source_step\": 1, \r\n                    \"step_output\": \"output\"\r\n                }\r\n            }, \r\n            \"tool_id\": \"toolshed.g2.bx.psu.edu/repos/devteam/bwa/bwa_mem/0.7.12.1\", \r\n            \"tool_inputs\": {\r\n                \"analysis_type\": \"{\\\"analysis_type_selector\\\": \\\"full\\\", \\\"algorithmic_options\\\": {\\\"algorithmic_options_selector\\\": \\\"do_not_set\\\", \\\"__current_case__\\\": 1}, \\\"io_options\\\": {\\\"a\\\": \\\"false\\\", \\\"C\\\": \\\"false\\\", \\\"io_options_selector\\\": \\\"set\\\", \\\"h\\\": \\\"5\\\", \\\"M\\\": \\\"true\\\", \\\"T\\\": \\\"30\\\", \\\"__current_case__\\\": 0, \\\"V\\\": \\\"false\\\", \\\"Y\\\": \\\"false\\\"}, \\\"__current_case__\\\": 2, \\\"scoring_options\\\": {\\\"scoring_options_selector\\\": \\\"do_not_set\\\", \\\"__current_case__\\\": 1}}\", \r\n                \"fastq_input\": \"{\\\"iset_stats\\\": \\\"\\\", \\\"fastq_input2\\\": {\\\"__class__\\\": \\\"RuntimeValue\\\"}, \\\"__current_case__\\\": 0, \\\"fastq_input_selector\\\": \\\"paired\\\", \\\"fastq_input1\\\": {\\\"__class__\\\": \\\"RuntimeValue\\\"}}\", \r\n                \"reference_source\": \"{\\\"ref_file\\\": \\\"hg19\\\", \\\"reference_source_selector\\\": \\\"cached\\\", \\\"__current_case__\\\": 0}\", \r\n                \"rg\": \"{\\\"rg_selector\\\": \\\"do_not_set\\\", \\\"__current_case__\\\": 3}\"\r\n            }, \r\n            \"tool_version\": \"0.7.12.1\", \r\n            \"type\": \"tool\"\r\n        }\r\n    }, \r\n    \"tags\": [], \r\n    \"url\": \"/api/workflows/484ad813eab74746\"\r\n}\r\n```\r\nThe invoke_workflow function on the bioblend side states this:\r\n```\r\n:param inputs: A mapping of workflow inputs to datasets and dataset collections.\r\nThe datasets source can be a LibraryDatasetDatasetAssociation (``ldda``),\r\n LibraryDataset (``ld``), HistoryDatasetAssociation (``hda``), or\r\nHistoryDatasetCollectionAssociation (``hdca``).\r\n\r\nThe map must be in the following format:\r\n ``{'<input_index>': {'id': <encoded dataset ID>, 'src': '[ldda, ld, hda, hdca]'}}``\r\n(e.g. ``{'2': {'id': '29beef4fadeed09f', 'src': 'hda'}}``)\r\n\r\nThis map may also be indexed by the UUIDs of the workflow steps,\r\nas indicated by the ``uuid`` property of steps returned from the\r\nGalaxy API.\r\n```\r\nIf I assign inputs using input_index as described above using this API call:\r\n```\r\ngi.workflows.invoke_workflow(workflow_id, inputs=inputs, params=params, history_id=history_id)\r\n```\r\nwhere the inputs is a dict that has been populated like this as described in the bioblend comments above:\r\n```\r\ninputs[input_index] = {'src': 'hda', 'id': input_hda_dict['id']}\r\n```\r\nAttempting to invoke the workflow via the API results in the error in the Galaxy paster log:\r\n```\r\ngalaxy.jobs.runners ERROR 2016-06-07 11:03:41,024 (316) Failure preparing job\r\nTraceback (most recent call last):\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/jobs/runners/__init__.py\", line 166, in prepare_job\r\n    job_wrapper.prepare()\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/jobs/__init__.py\", line 885, in prepare\r\n    tool_evaluator.set_compute_environment( compute_environment, get_special=get_special )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/evaluation.py\", line 63, in set_compute_environment\r\n    visit_input_values( self.tool.inputs, incoming, validate_inputs )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/parameters/__init__.py\", line 104, in visit_input_values\r\n    visit_input_values( input.cases[ values[ '__current_case__' ] ].inputs, values, callback, new_name_prefix, label_prefix, parent_prefix=name_prefix, **payload )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/parameters/__init__.py\", line 110, in visit_input_values\r\n    callback_helper( input, input_values, name_prefix, label_prefix, parent_prefix=parent_prefix, context=context )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/parameters/__init__.py\", line 75, in callback_helper\r\n    new_value = callback( **args )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/evaluation.py\", line 61, in validate_inputs\r\n    value = input.from_json( value, request_context, context )\r\n  File \"/gpfs/cyberstar/p/pughhpc/galaxy-cegr/galaxy/lib/galaxy/tools/parameters/basic.py\", line 1889, in from_json\r\n    raise ValueError( \"History does not include a dataset of the required format / build\" )\r\nValueError: History does not include a dataset of the required format / build\r\ngalaxy.jobs ERROR 2016-06-07 11:03:41,114 fail(): Missing output file in working directory: [Errno 2] No such file or directory: '/gpfs/cyberstar/pughhpc/galaxy-cegr/job_working_directory/000/316/galaxy_dataset_557.dat'\r\n```\r\nSo it seems that the inputs dictionary should not be using keys that are input_index values as stated in the bioblend comments above.  The galaxy code in ~/lib/galaxy/workflow/run_request.py seems to confirm this since some relevant code in the build_workflow_run_request() method is:\r\n```\r\n    inputs = payload.get( 'inputs', None )\r\n    inputs_by = payload.get( 'inputs_by', None )\r\n    # New default is to reference steps by index of workflow step\r\n    # which is intrinsic to the workflow and independent of the state\r\n    # of Galaxy at the time of workflow import.\r\n    default_inputs_by = 'step_index|step_uuid'\r\n```\r\nSo from the Galaxy side it seems that the keys in the inputs dictionary should be \"step_index|step_uuid\" rather than the \"input_index\" as stated on the bioblend side.\r\n\r\nIf this is true, then where do I get these \"step_index|step_uuid\" values?  At first glance, it looks like they may be components of the workflow invocation dictionary, which is not yet available at the time I am attempting to invoke the workflow.\r\n\r\nIf I can get this answer, I think I can get the workflow to execute.\r\n\r\nThanks for all of the help on this."}}
{"actor":{"display_login":"gregvonkuster"}}