{"action":"created","issue":{"url":"https://api.github.com/repos/datalad/datalad/issues/2593","repository_url":"https://api.github.com/repos/datalad/datalad","labels_url":"https://api.github.com/repos/datalad/datalad/issues/2593/labels{/name}","comments_url":"https://api.github.com/repos/datalad/datalad/issues/2593/comments","events_url":"https://api.github.com/repos/datalad/datalad/issues/2593/events","html_url":"https://github.com/datalad/datalad/issues/2593","id":328033328,"node_id":"MDU6SXNzdWUzMjgwMzMzMjg=","number":2593,"title":"Urgent: dataset templates -> `run-procedure` command","user":{"login":"mih","id":136479,"node_id":"MDQ6VXNlcjEzNjQ3OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/136479?v=4","gravatar_id":"","url":"https://api.github.com/users/mih","html_url":"https://github.com/mih","followers_url":"https://api.github.com/users/mih/followers","following_url":"https://api.github.com/users/mih/following{/other_user}","gists_url":"https://api.github.com/users/mih/gists{/gist_id}","starred_url":"https://api.github.com/users/mih/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mih/subscriptions","organizations_url":"https://api.github.com/users/mih/orgs","repos_url":"https://api.github.com/users/mih/repos","events_url":"https://api.github.com/users/mih/events{/privacy}","received_events_url":"https://api.github.com/users/mih/received_events","type":"User","site_admin":false},"labels":[{"id":526935563,"node_id":"MDU6TGFiZWw1MjY5MzU1NjM=","url":"https://api.github.com/repos/datalad/datalad/labels/come%20in%20and%20sing%20along","name":"come in and sing along","color":"fbca04","default":false},{"id":63778592,"node_id":"MDU6TGFiZWw2Mzc3ODU5Mg==","url":"https://api.github.com/repos/datalad/datalad/labels/enhancement","name":"enhancement","color":"c5def5","default":true}],"state":"open","locked":false,"assignee":{"login":"mih","id":136479,"node_id":"MDQ6VXNlcjEzNjQ3OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/136479?v=4","gravatar_id":"","url":"https://api.github.com/users/mih","html_url":"https://github.com/mih","followers_url":"https://api.github.com/users/mih/followers","following_url":"https://api.github.com/users/mih/following{/other_user}","gists_url":"https://api.github.com/users/mih/gists{/gist_id}","starred_url":"https://api.github.com/users/mih/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mih/subscriptions","organizations_url":"https://api.github.com/users/mih/orgs","repos_url":"https://api.github.com/users/mih/repos","events_url":"https://api.github.com/users/mih/events{/privacy}","received_events_url":"https://api.github.com/users/mih/received_events","type":"User","site_admin":false},"assignees":[{"login":"mih","id":136479,"node_id":"MDQ6VXNlcjEzNjQ3OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/136479?v=4","gravatar_id":"","url":"https://api.github.com/users/mih","html_url":"https://github.com/mih","followers_url":"https://api.github.com/users/mih/followers","following_url":"https://api.github.com/users/mih/following{/other_user}","gists_url":"https://api.github.com/users/mih/gists{/gist_id}","starred_url":"https://api.github.com/users/mih/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mih/subscriptions","organizations_url":"https://api.github.com/users/mih/orgs","repos_url":"https://api.github.com/users/mih/repos","events_url":"https://api.github.com/users/mih/events{/privacy}","received_events_url":"https://api.github.com/users/mih/received_events","type":"User","site_admin":false}],"milestone":null,"comments":8,"created_at":"2018-05-31T08:00:09Z","updated_at":"2018-06-03T15:45:46Z","closed_at":null,"author_association":"MEMBER","body":"More and more stuff is creeping into the code base that is concerned with setting up datasets for particular use cases. We have long identified the need to have some kind of dataset templates, to be able to do `datalad create --template howIneedittobe`. But no effort ever got there.\r\n\r\nSo let me propose something simple, with little required effort and high potential:\r\n\r\n- a template is a tarball (but see the end)\r\n- such tarball get's extracted at the location of a just created dataset (think `create_tree`). It is extracted after creation so the flags like `--force` still work as desired.\r\n- after extraction a Python script that is located in the root of the tarball (with a special name, maybe `.configure`) is executed, the sole argument being the absolute path of the dataset  and performs arbitrary (re-)configurations, additions, whatever\r\n- this script can use different interpreters (python for platform independence, bash for simplicity)\r\n- after the script is done the results are commited -- in practice this means we use `run` to perform this step\r\n- such template tarballs are looked up at a configurable set of locations (system, user)\r\n- a tarball doesn't actually need to be a tarball, but can also just be the configure\r\n\r\n- `create` should be able to execute multiple templates sequentially, so we can orthogonalize the scenarios we want to support.\r\n\r\nafterwards anybody can come up with and share there preferred dataset setups.\r\n\r\n### candidates for API removal after this works:\r\n\r\n- `create(native_metadata)`\r\n- `create(text_no_annex)`\r\n- completely discontinue the Python-only plugin interface code and implementations: either proper commands or this new one for any given functionality\r\n\r\n### other related aspects\r\n- entire business of inheriting configuration from a parent dataset\r\n- this is basically an interface for canned procedures. Configuration of datasets is just one flavor of where this functionality is useful. I think what this will lead to is the return of the `--run-before/after` functionality that we briefly enjoyed in the API. This time however, there is no need to go through plugins or something as complicated, because we have `run` now. This will enable us to even implement \"canned procedures\" that require input arguments.\r\n\r\n### conclusion\r\n\r\nSo we end up with a new command `run-procedure` that sorts out, how such a script needs to be executed, and tailors a call to `run` to execute it (analog to `containers-run` but without containtainers). That means we can support extensions of datalad functionality that do not require going through the Python API.\r\n\r\nOne aspect would need a further extension of the `run` API: ATM we require a clean dataset before `run` does anything. But a simple `add` can work with a dirty dataset just fine. Nobody says that a canned procedure must not create datalad commands directly and must rely on `run` to capture all the changes. So we either need to extend the `run` API to support all the running aspects without the result capture (i.e. a mere frontend for the `Runner` class). Or we need to expose and reuse the pieces of `run` that do that and also use them for `run-procedure`. I'd prefer to first solution, as the handling of `--inputs` and `--outputs` is already *very* convenient.\r\n\r\n@kyleam what do you think?"},"comment":{"url":"https://api.github.com/repos/datalad/datalad/issues/comments/394171066","html_url":"https://github.com/datalad/datalad/issues/2593#issuecomment-394171066","issue_url":"https://api.github.com/repos/datalad/datalad/issues/2593","id":394171066,"node_id":"MDEyOklzc3VlQ29tbWVudDM5NDE3MTA2Ng==","user":{"login":"mih","id":136479,"node_id":"MDQ6VXNlcjEzNjQ3OQ==","avatar_url":"https://avatars1.githubusercontent.com/u/136479?v=4","gravatar_id":"","url":"https://api.github.com/users/mih","html_url":"https://github.com/mih","followers_url":"https://api.github.com/users/mih/followers","following_url":"https://api.github.com/users/mih/following{/other_user}","gists_url":"https://api.github.com/users/mih/gists{/gist_id}","starred_url":"https://api.github.com/users/mih/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/mih/subscriptions","organizations_url":"https://api.github.com/users/mih/orgs","repos_url":"https://api.github.com/users/mih/repos","events_url":"https://api.github.com/users/mih/events{/privacy}","received_events_url":"https://api.github.com/users/mih/received_events","type":"User","site_admin":false},"created_at":"2018-06-03T15:45:46Z","updated_at":"2018-06-03T15:45:46Z","author_association":"MEMBER","body":"I like `explicit`!\r\n\r\nI started the implementation here: https://github.com/datalad/datalad/pull/2603"}}
{"actor":{"display_login":"mih"}}