{"action":"created","issue":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2581","repository_url":"https://api.github.com/repos/galaxyproject/galaxy","labels_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2581/labels{/name}","comments_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2581/comments","events_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2581/events","html_url":"https://github.com/galaxyproject/galaxy/issues/2581","id":163391642,"number":2581,"title":"Galaxy 16.04 problem with tools using dataset collections","user":{"login":"gregvonkuster","id":6947053,"avatar_url":"https://avatars.githubusercontent.com/u/6947053?v=3","gravatar_id":"","url":"https://api.github.com/users/gregvonkuster","html_url":"https://github.com/gregvonkuster","followers_url":"https://api.github.com/users/gregvonkuster/followers","following_url":"https://api.github.com/users/gregvonkuster/following{/other_user}","gists_url":"https://api.github.com/users/gregvonkuster/gists{/gist_id}","starred_url":"https://api.github.com/users/gregvonkuster/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/gregvonkuster/subscriptions","organizations_url":"https://api.github.com/users/gregvonkuster/orgs","repos_url":"https://api.github.com/users/gregvonkuster/repos","events_url":"https://api.github.com/users/gregvonkuster/events{/privacy}","received_events_url":"https://api.github.com/users/gregvonkuster/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/area/dataset-collections","name":"area/dataset-collections","color":"006b75"},{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/kind/bug","name":"kind/bug","color":"84b6eb"},{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/major","name":"major","color":"fef2c0"}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":13,"created_at":"2016-07-01T13:05:23Z","updated_at":"2016-07-13T23:04:43Z","closed_at":null,"body":"I've encountered a problem where a tool will not properly create a dataset collection when the input to the tool is a dataset collection, although the tool will function correctly if the input is not a dataset collection.  This problem occurs regularly in workflows, causing the workflow to stop executing at that step.  I believe this issue may be related to other issues (e.g., https://github.com/galaxyproject/galaxy/issues/2179), so hopefully fixing this will help to close other tickets as well.\r\n\r\nI've spent some time looking into this and hope that the following details will help to get a quick resolution as I believe this issue is the last obstacle to have a fully functional ChIP-exo workflow for Galaxy.\r\n\r\nI've included a tool here  named test_first that can be used to reproduce this issue.\r\n\r\ntest_first.xml\r\n```\r\n<tool id=\"test_first\" name=\"Test first\" version=\"1.0.0\">\r\n    <description></description>\r\n    <requirements>\r\n    </requirements>\r\n    <command>\r\n        <![CDATA[\r\n            mkdir -p output &&\r\n            python $__tool_directory__/test.py\r\n            --input1 \"${input1}\"\r\n        ]]>\r\n    </command>\r\n    <inputs>\r\n        <param name=\"input1\" type=\"data\" format=\"txt\" label=\"Input 1\"/>\r\n    </inputs>\r\n    <outputs>\r\n        <collection name=\"my_first_coll\" type=\"list\" label=\"My first collection\">\r\n            <discover_datasets pattern=\"(?P&lt;name&gt;.*)\" directory=\"output\" visible=\"false\" />\r\n        </collection>\r\n    </outputs>\r\n    <tests>\r\n    </tests>\r\n    <help>\r\n    </help>\r\n    <citations>\r\n    </citations>\r\n</tool>\r\n```\r\ntest.py\r\n```\r\n#!/usr/bin/env python\r\nimport argparse\r\nimport os\r\n\r\nparser = argparse.ArgumentParser()\r\nparser.add_argument('--input1', dest='input1', help='input 1')\r\nargs = parser.parse_args()\r\n\r\ncollection_path = os.path.join(os.getcwd(), 'output')\r\n\r\ni = 0\r\nfor line in open(args.input1, 'r'):\r\n    i += 1\r\n    file_name = os.path.join(collection_path, 'FILE%s.txt' % i)\r\n    fh = open(file_name, 'wb')\r\n    fh.write(line)\r\n    fh.close()\r\n```\r\nWhen executed with this input dataset (which is not a dataset collection)...\r\n<img width=\"213\" alt=\"input\" src=\"https://cloud.githubusercontent.com/assets/6947053/16521039/1c133ecc-3f62-11e6-9d79-b0b7e8a0e88c.png\">\r\n\r\n...the tool properly produces this dataset collection.\r\n<img width=\"210\" alt=\"collection\" src=\"https://cloud.githubusercontent.com/assets/6947053/16521578/d32d3ed4-3f65-11e6-8036-a597967d3fc1.png\">\r\n\r\nIn this scenario, the dataset collection is initially created with 0 elements - here is the entry in the Galaxy paster log.\r\n```\r\ngalaxy.managers.collections DEBUG 2016-07-01 08:05:07,956 Created collection with 0 elements\r\n```\r\nWhen the tool executes, the datasets are added to the dataset collection produced by the tool.  Here are the entries from the Galaxy paster log.\r\n```\r\ngalaxy.tools.parameters.output_collect DEBUG 2016-07-01 08:05:41,184 (11101) Created dynamic collection dataset for path [/gpfs/cyberstar/pughhpc/galaxy-cegr/job_working_directory/011/11101/working/output/FILE1.txt] with element identifier [FILE1.txt] for output [my_first_coll] (102.013 ms)\r\n...\r\ngalaxy.tools.parameters.output_collect DEBUG 2016-07-01 08:05:41,234 (11101) Created dynamic collection dataset for path [/gpfs/cyberstar/pughhpc/galaxy-cegr/job_working_directory/011/11101/working/output/FILE2.txt] with element identifier [FILE2.txt] for output [my_first_coll] (49.324 ms)\r\n...\r\ngalaxy.tools.parameters.output_collect DEBUG 2016-07-01 08:05:41,286 (11101) Created dynamic collection dataset for path [/gpfs/cyberstar/pughhpc/galaxy-cegr/job_working_directory/011/11101/working/output/FILE3.txt] with element identifier [FILE3.txt] for output [my_first_coll] (51.444 ms)\r\n...\r\ngalaxy.tools.parameters.output_collect DEBUG 2016-07-01 08:05:41,377 (11101) Add dynamic collection datsets to history for output [my_first_coll] (91.413 ms)\r\n...\r\ngalaxy.jobs INFO 2016-07-01 08:05:41,508 Collecting metrics for Job 11101\r\ngalaxy.jobs DEBUG 2016-07-01 08:05:41,529 job 11101 ended (finish() executed in (666.016 ms))\r\n```\r\nOur Galaxy history now includes the original input dataset and our new collection.\r\n<img width=\"211\" alt=\"history1\" src=\"https://cloud.githubusercontent.com/assets/6947053/16521700/979b98a6-3f66-11e6-91f7-66fd33db6b71.png\">\r\n\r\nThe error is encountered when we use the dataset collection we just produced as the input when we execute the tool a 2nd time.\r\n<img width=\"483\" alt=\"collection_input\" src=\"https://cloud.githubusercontent.com/assets/6947053/16521744/f3d760b4-3f66-11e6-979c-008401f50a0e.png\">\r\n\r\nIn this scenario, the dataset collection is initially created with 3 elements.  Here is the Galaxy paster log entry.\r\n```\r\ngalaxy.managers.collections DEBUG 2016-07-01 08:37:10,469 Created collection with 3 elements\r\n```\r\nThis occurs due to the __load_element() function in ~/lib/galaxy/managers/collections.py, which includes the following comment:\r\n```\r\n# Previously created collection already found in request, just pass\r\n# through as is.\r\n```\r\nAs the tool executes, the 3 datasets that are supposed to be added to the collection point to the same file.  Here are the Galaxy paster log entries.\r\n```\r\ngalaxy.tools.parameters.output_collect DEBUG 2016-07-01 08:37:38,289 (11108) Created dynamic collection dataset for path [/gpfs/cyberstar/pughhpc/galaxy-cegr/job_working_directory/011/11108/working/output/FILE1.txt] with element identifier [FILE1.txt] for output [my_first_coll] (70.213 ms)\r\n...\r\ngalaxy.tools.parameters.output_collect DEBUG 2016-07-01 08:37:41,341 (11109) Created dynamic collection dataset for path [/gpfs/cyberstar/pughhpc/galaxy-cegr/job_working_directory/011/11109/working/output/FILE1.txt] with element identifier [FILE1.txt] for output [my_first_coll] (49.238 ms)\r\n...\r\ngalaxy.tools.parameters.output_collect DEBUG 2016-07-01 08:37:51,117 (11107) Created dynamic collection dataset for path [/gpfs/cyberstar/pughhpc/galaxy-cegr/job_working_directory/011/11107/working/output/FILE1.txt] with element identifier [FILE1.txt] for output [my_first_coll] (50.875 ms)\r\n```\r\nThe job finishes successfully:\r\n```\r\ngalaxy.jobs INFO 2016-07-01 08:37:51,343 Collecting metrics for Job 11107\r\ngalaxy.jobs DEBUG 2016-07-01 08:37:51,363 job 11107 ended (finish() executed in (471.584 ms))\r\n```\r\nBut our history does not include the properly constructed dataset collection for the output.  Notice, however, that there are some hidden outputs now.\r\n<img width=\"213\" alt=\"history2\" src=\"https://cloud.githubusercontent.com/assets/6947053/16522125/b32d2910-3f69-11e6-98c6-5ecd2f852689.png\">\r\n\r\nLooking at the hidden items shows the 3 datasets that should be included in the collection (although they all point to the same data files on disk), but they seem to be somehow disconnected from the collection itself.\r\n<img width=\"209\" alt=\"history3\" src=\"https://cloud.githubusercontent.com/assets/6947053/16522186/0cd17eda-3f6a-11e6-9038-b61866d44b2f.png\">\r\n\r\nThis issue occurs in a workflow when tool 1 produces a dataset collection which is used as the input to tool 2.  The workflow will stop at tool 2, with a Galaxy paster log entry like this:\r\n```\r\nhandler1.log:galaxy.tools.execute DEBUG 2016-06-25 18:22:43,515 Executed 0 job(s) for tool toolshed.g2.bx.psu.edu/repos/iuc/resize_coordinate_window/resize_coordinate_window/1.0.1 request: (27.811 ms)\r\n```\r\n\r\nPlease let me know if / how I can help out in getting this resolved as it is critical for the ChIP-exo Galaxy flavor.\r\n\r\nThanks in advance.  ;)"},"comment":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/comments/232512233","html_url":"https://github.com/galaxyproject/galaxy/issues/2581#issuecomment-232512233","issue_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/2581","id":232512233,"user":{"login":"shaunmahony","id":166420,"avatar_url":"https://avatars.githubusercontent.com/u/166420?v=3","gravatar_id":"","url":"https://api.github.com/users/shaunmahony","html_url":"https://github.com/shaunmahony","followers_url":"https://api.github.com/users/shaunmahony/followers","following_url":"https://api.github.com/users/shaunmahony/following{/other_user}","gists_url":"https://api.github.com/users/shaunmahony/gists{/gist_id}","starred_url":"https://api.github.com/users/shaunmahony/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/shaunmahony/subscriptions","organizations_url":"https://api.github.com/users/shaunmahony/orgs","repos_url":"https://api.github.com/users/shaunmahony/repos","events_url":"https://api.github.com/users/shaunmahony/events{/privacy}","received_events_url":"https://api.github.com/users/shaunmahony/received_events","type":"User","site_admin":false},"created_at":"2016-07-13T23:04:43Z","updated_at":"2016-07-13T23:04:43Z","body":"+1 on fixing this. Thanks!"}}
{"actor":{"display_login":"shaunmahony"}}