{"action":"created","issue":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1568","repository_url":"https://api.github.com/repos/galaxyproject/galaxy","labels_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1568/labels{/name}","comments_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1568/comments","events_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1568/events","html_url":"https://github.com/galaxyproject/galaxy/issues/1568","id":128220501,"number":1568,"title":"Data library files in remote ftp/s3 location [Enhancement]","user":{"login":"MatthewRalston","id":4308024,"avatar_url":"https://avatars.githubusercontent.com/u/4308024?v=3","gravatar_id":"","url":"https://api.github.com/users/MatthewRalston","html_url":"https://github.com/MatthewRalston","followers_url":"https://api.github.com/users/MatthewRalston/followers","following_url":"https://api.github.com/users/MatthewRalston/following{/other_user}","gists_url":"https://api.github.com/users/MatthewRalston/gists{/gist_id}","starred_url":"https://api.github.com/users/MatthewRalston/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/MatthewRalston/subscriptions","organizations_url":"https://api.github.com/users/MatthewRalston/orgs","repos_url":"https://api.github.com/users/MatthewRalston/repos","events_url":"https://api.github.com/users/MatthewRalston/events{/privacy}","received_events_url":"https://api.github.com/users/MatthewRalston/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/feature-request","name":"feature-request","color":"fef2c0"},{"url":"https://api.github.com/repos/galaxyproject/galaxy/labels/kind/enhancement","name":"kind/enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":null,"milestone":null,"comments":8,"created_at":"2016-01-22T18:54:19Z","updated_at":"2016-02-03T21:11:58Z","closed_at":null,"body":"I have an idea for a [feature for galaxy and bioblend](https://github.com/galaxyproject/bioblend/issues/177): namely I'd like to have access to data libraries with files located in S3. Similar to the ObjectStore, but for data library files instead of datasets. Currently, the data library and bioblend support \"links\" to files, where a data library can simply \"point\" to a file's location on a local disk and users can pull the datasets into their current histories to begin an analysis. It would be interesting to have similar functionality for remote file locations. \r\n\r\nAFAIK the current implementation of the data libraries can retrieve publically available files from S3 or an ftp site with a certain bioblend method, but there is no set of methods to tell galaxy to instead retrieve the files only when needed and to store the url in the meanwhile. Of course there is no guarantee that the files would actually be there, but the same is true for the \"links\" to files mentioned above. Perhaps there is some way to manage this edge case.\r\n\r\nI'd like to begin work on this feature for the galaxy community, but I don't know where to start in the Galaxy codebase. @dannon who is the expert on data libraries so that I can begin working on this feature?"},"comment":{"url":"https://api.github.com/repos/galaxyproject/galaxy/issues/comments/179470589","html_url":"https://github.com/galaxyproject/galaxy/issues/1568#issuecomment-179470589","issue_url":"https://api.github.com/repos/galaxyproject/galaxy/issues/1568","id":179470589,"user":{"login":"dannon","id":155398,"avatar_url":"https://avatars.githubusercontent.com/u/155398?v=3","gravatar_id":"","url":"https://api.github.com/users/dannon","html_url":"https://github.com/dannon","followers_url":"https://api.github.com/users/dannon/followers","following_url":"https://api.github.com/users/dannon/following{/other_user}","gists_url":"https://api.github.com/users/dannon/gists{/gist_id}","starred_url":"https://api.github.com/users/dannon/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/dannon/subscriptions","organizations_url":"https://api.github.com/users/dannon/orgs","repos_url":"https://api.github.com/users/dannon/repos","events_url":"https://api.github.com/users/dannon/events{/privacy}","received_events_url":"https://api.github.com/users/dannon/received_events","type":"User","site_admin":false},"created_at":"2016-02-03T21:11:58Z","updated_at":"2016-02-03T21:11:58Z","body":"@MatthewRalston Did you check out the S3ObjectStore and the caching there?  I think looking at that and maybe tinkering with it a bit will answer a lot of your questions and maybe guide this effort a bit.\r\n\r\nIf library datasets were never directly usable for anything, then I'd say we could just store a link and only fetch once the dataset was created as an HDA, but since they are we need a more generic caching mechanism."}}