{"action":"created","issue":{"url":"https://api.github.com/repos/datalad/datalad/issues/424","repository_url":"https://api.github.com/repos/datalad/datalad","labels_url":"https://api.github.com/repos/datalad/datalad/issues/424/labels{/name}","comments_url":"https://api.github.com/repos/datalad/datalad/issues/424/comments","events_url":"https://api.github.com/repos/datalad/datalad/issues/424/events","html_url":"https://github.com/datalad/datalad/issues/424","id":153185006,"number":424,"title":"Be more clever wrt to \"install from submodule checkout\"","user":{"login":"hanke","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/hanke","html_url":"https://github.com/hanke","followers_url":"https://api.github.com/users/hanke/followers","following_url":"https://api.github.com/users/hanke/following{/other_user}","gists_url":"https://api.github.com/users/hanke/gists{/gist_id}","starred_url":"https://api.github.com/users/hanke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanke/subscriptions","organizations_url":"https://api.github.com/users/hanke/orgs","repos_url":"https://api.github.com/users/hanke/repos","events_url":"https://api.github.com/users/hanke/events{/privacy}","received_events_url":"https://api.github.com/users/hanke/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/datalad/datalad/labels/do-in-datalad","name":"do-in-datalad","color":"fad8c7"},{"url":"https://api.github.com/repos/datalad/datalad/labels/enhancement","name":"enhancement","color":"84b6eb"}],"state":"open","locked":false,"assignee":{"login":"bpoldrack","id":10498301,"avatar_url":"https://avatars.githubusercontent.com/u/10498301?v=3","gravatar_id":"","url":"https://api.github.com/users/bpoldrack","html_url":"https://github.com/bpoldrack","followers_url":"https://api.github.com/users/bpoldrack/followers","following_url":"https://api.github.com/users/bpoldrack/following{/other_user}","gists_url":"https://api.github.com/users/bpoldrack/gists{/gist_id}","starred_url":"https://api.github.com/users/bpoldrack/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/bpoldrack/subscriptions","organizations_url":"https://api.github.com/users/bpoldrack/orgs","repos_url":"https://api.github.com/users/bpoldrack/repos","events_url":"https://api.github.com/users/bpoldrack/events{/privacy}","received_events_url":"https://api.github.com/users/bpoldrack/received_events","type":"User","site_admin":false},"milestone":{"url":"https://api.github.com/repos/datalad/datalad/milestones/22","html_url":"https://github.com/datalad/datalad/milestones/Publication%20(tracking)","labels_url":"https://api.github.com/repos/datalad/datalad/milestones/22/labels","id":1482873,"number":22,"title":"Publication (tracking)","description":"This milestone is being abused by Michael for his education. It collects issues regarding the publication of collections and handles.","creator":{"login":"hanke","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/hanke","html_url":"https://github.com/hanke","followers_url":"https://api.github.com/users/hanke/followers","following_url":"https://api.github.com/users/hanke/following{/other_user}","gists_url":"https://api.github.com/users/hanke/gists{/gist_id}","starred_url":"https://api.github.com/users/hanke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanke/subscriptions","organizations_url":"https://api.github.com/users/hanke/orgs","repos_url":"https://api.github.com/users/hanke/repos","events_url":"https://api.github.com/users/hanke/events{/privacy}","received_events_url":"https://api.github.com/users/hanke/received_events","type":"User","site_admin":false},"open_issues":8,"closed_issues":5,"state":"open","created_at":"2016-01-04T17:02:49Z","updated_at":"2016-05-11T10:58:36Z","due_on":null,"closed_at":null},"comments":7,"created_at":"2016-05-05T07:48:40Z","updated_at":"2016-05-11T11:02:47Z","closed_at":null,"body":"Apologies for a possible duplication, but I couldn't find the original discussion anymore.\r\n\r\nSynopsis: Git's submodule implementation is not as clever as it could be with respect to figuring out from which location to clone a submodule. Here is a demo of the core issue:\r\n\r\n````\r\n% datalad install -s http://psydata.ovgu.de/studyforrest/templatetransforms/.git tnt\r\n% cd tnt\r\n% grep -A2 '^\\[submodule \"raw/structural\"\\]' .gitmodules\r\n[submodule \"raw/structural\"]\r\n\tpath = raw/structural\r\n\turl = ../structural\r\n````\r\n\r\nThe location of the respective submodule is this: http://psydata.ovgu.de/studyforrest/structural/\r\n\r\nGiven that it isn't a bare repository the actual source URL is: http://psydata.ovgu.de/studyforrest/structural/.git/\r\n\r\nCurrent Git is not able to figure this out.\r\n\r\nAFAICS we can work around this problem, by detecting a failure and make an educated second guess. First the failure:\r\n\r\n````\r\n% datalad install raw/structural\r\n....\r\n2016-05-05 09:22:07,575 [ERROR  ] CommandError: command '['git', 'submodule', 'update', '--init', 'raw/structural']' failed with exitcode 128.\r\n| Failed to run ['git', 'submodule', 'update', '--init', 'raw/structural'] under '/tmp/tnt'. Exit code=128. out=Submodule 'raw/structural' (http://psydata.ovgu.de/studyforrest/templatetransforms/structural) registered for path 'raw/structural'\r\n|  err=Cloning into 'raw/structural'...\r\n| fatal: repository 'http://psydata.ovgu.de/studyforrest/templatetransforms/structural/' not found\r\n| fatal: clone of 'http://psydata.ovgu.de/studyforrest/templatetransforms/structural' into submodule path 'raw/structural' failed\r\n|  [cmd.py:run:296] (CommandError) (main.py:235)\r\n````\r\n\r\nThis leaves us without an initialized submodule, but a misconfigured submodule URL remains:\r\n\r\n````\r\n% git submodule status raw/structural\r\n-54e555acea857acdb2a8b713cc4c7c7b09607b3d raw/structural\r\n% git config submodule.raw/structural.url\r\nhttp://psydata.ovgu.de/studyforrest/templatetransforms/structural\r\n````\r\n\r\nWe can fix this by looking at \r\n\r\n````\r\n% git config remote.origin.url\r\nhttp://psydata.ovgu.de/studyforrest/templatetransforms/.git\r\n% grep -A2 '^\\[submodule \"raw/structural\"\\]' .gitmodules |grep url | cut -d '=' -f 2\r\n ../structural\r\n````\r\n\r\nRemove `.git` from the origin URL, and resolve the relative path from the submodule URL. For increased flexibility we could try first with the resulting http://psydata.ovgu.de/studyforrest/structural and fail again in this case, and subsequently append `/.git/` and try one last time. However, it seems unlikely that git repositories at a close-by location mix different setup (bare, non-bare, ...).\r\n\r\nAnyway, after fixing up the git config setting things start working:\r\n\r\n````\r\n% git config submodule.raw/structural.url http://psydata.ovgu.de/studyforrest/structural/.git/\r\n% mkdir raw/structural\r\n% datalad install raw/structural\r\n2016-05-05 09:40:47,548 [ERROR  ] stderr| Cloning into 'raw/structural'... (cmd.py:351)\r\n1 installed item is available at\r\n<Dataset path=/tmp/tnt/raw/structural>\r\n% git -C raw/structural status\r\nHEAD detached at 54e555a\r\nnothing to commit, working directory clean\r\n````\r\n\r\nFew notes:\r\n\r\n- this workaround doesn't require a modification of any content checked into Git, hence no useless cruft is generated in the history\r\n- appart from an initial failing clone attempt it doesn't waste bandwidth or storage with useless intermediate results\r\n- the `mkdir` above is required as the initial install attempt kills the submodule mount point for some reason. If datalad doesn't find this mountpoint the second install attempt will happily create a new empty submodule -- this is likely a bug on its own\r\n- for reasons not 100% clear to me the trailing `/` of the fixed submodule URL is critical\r\n\r\n"},"comment":{"url":"https://api.github.com/repos/datalad/datalad/issues/comments/218427784","html_url":"https://github.com/datalad/datalad/issues/424#issuecomment-218427784","issue_url":"https://api.github.com/repos/datalad/datalad/issues/424","id":218427784,"user":{"login":"hanke","id":136479,"avatar_url":"https://avatars.githubusercontent.com/u/136479?v=3","gravatar_id":"","url":"https://api.github.com/users/hanke","html_url":"https://github.com/hanke","followers_url":"https://api.github.com/users/hanke/followers","following_url":"https://api.github.com/users/hanke/following{/other_user}","gists_url":"https://api.github.com/users/hanke/gists{/gist_id}","starred_url":"https://api.github.com/users/hanke/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/hanke/subscriptions","organizations_url":"https://api.github.com/users/hanke/orgs","repos_url":"https://api.github.com/users/hanke/repos","events_url":"https://api.github.com/users/hanke/events{/privacy}","received_events_url":"https://api.github.com/users/hanke/received_events","type":"User","site_admin":false},"created_at":"2016-05-11T11:02:47Z","updated_at":"2016-05-11T11:02:47Z","body":"All merged now, closing...."}}