{"action":"created","issue":{"url":"https://api.github.com/repos/ansible/ansible/issues/48678","repository_url":"https://api.github.com/repos/ansible/ansible","labels_url":"https://api.github.com/repos/ansible/ansible/issues/48678/labels{/name}","comments_url":"https://api.github.com/repos/ansible/ansible/issues/48678/comments","events_url":"https://api.github.com/repos/ansible/ansible/issues/48678/events","html_url":"https://github.com/ansible/ansible/issues/48678","id":380820661,"node_id":"MDU6SXNzdWUzODA4MjA2NjE=","number":48678,"title":"git submodule causes ansible-pull to always clone submodules recursively","user":{"login":"ju2wheels","id":396211,"node_id":"MDQ6VXNlcjM5NjIxMQ==","avatar_url":"https://avatars0.githubusercontent.com/u/396211?v=4","gravatar_id":"","url":"https://api.github.com/users/ju2wheels","html_url":"https://github.com/ju2wheels","followers_url":"https://api.github.com/users/ju2wheels/followers","following_url":"https://api.github.com/users/ju2wheels/following{/other_user}","gists_url":"https://api.github.com/users/ju2wheels/gists{/gist_id}","starred_url":"https://api.github.com/users/ju2wheels/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/ju2wheels/subscriptions","organizations_url":"https://api.github.com/users/ju2wheels/orgs","repos_url":"https://api.github.com/users/ju2wheels/repos","events_url":"https://api.github.com/users/ju2wheels/events{/privacy}","received_events_url":"https://api.github.com/users/ju2wheels/received_events","type":"User","site_admin":false},"labels":[{"id":1043397454,"node_id":"MDU6TGFiZWwxMDQzMzk3NDU0","url":"https://api.github.com/repos/ansible/ansible/labels/affects_2.8","name":"affects_2.8","color":"ededed","default":false},{"id":852590921,"node_id":"MDU6TGFiZWw4NTI1OTA5MjE=","url":"https://api.github.com/repos/ansible/ansible/labels/feature","name":"feature","color":"006b75","default":false},{"id":496359346,"node_id":"MDU6TGFiZWw0OTYzNTkzNDY=","url":"https://api.github.com/repos/ansible/ansible/labels/module","name":"module","color":"1d76db","default":false},{"id":636352796,"node_id":"MDU6TGFiZWw2MzYzNTI3OTY=","url":"https://api.github.com/repos/ansible/ansible/labels/support:core","name":"support:core","color":"ededed","default":false},{"id":486238465,"node_id":"MDU6TGFiZWw0ODYyMzg0NjU=","url":"https://api.github.com/repos/ansible/ansible/labels/waiting_on_contributor","name":"waiting_on_contributor","color":"eb6420","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":2,"created_at":"2018-11-14T18:03:16Z","updated_at":"2018-11-20T19:44:49Z","closed_at":null,"author_association":"NONE","body":"##### SUMMARY\r\nThe `git` submodule which is used by `ansible-pull` when cloning the inventory repo is fixed to either initializing submodules recursively or not initializing submodules and doesn't support just initializing the first level of submodules in the repo.\r\n\r\nUse Case:\r\n\r\nI have thousands of IOT devices being managed using `ansible-pull`. Some SCM providers like Bitbucket have API limits and recursive cloning can be a problem here and also a scalability issue.\r\n\r\nIf my `ansible-pull` inventory uses submodules for roles in `roles/` and each role itself uses submodules in its tests directory which mimics the inventory at `tests/roles/` used only for unit testing, then doing recursive submodule initialization is a huge cloning overhead that is ultimately not necessary.\r\n\r\nThe difference in the number of clones in my case when I use 11 role submodules in the inventory repo is that non recursive submodule cloning causes 12 `git` clones, whereas recursive submodule cloning causes 52 `git` clones.\r\n\r\nWhen your IOT devices may be on internet connections of questionable quality, reducing the amount clones helps reduce the network traffic, load on SCM provider or self hosted instances, and reduce `ansible-pull` failures resulting from just trying to retrieve the inventory/roles.\r\n\r\n##### ISSUE TYPE\r\n- Feature Idea\r\n\r\n##### COMPONENT NAME\r\nansible-pull\r\ngit\r\n\r\n##### ADDITIONAL INFORMATION\r\n\r\nMy guess is that the current `git` module's behavior is modeled after `git-clone`'s behavior with its built in `--recurse-submodules` and `--shallow-submodules` options.\r\n\r\nHowever, in actual implementation, the `git` submodule is not using these `git-clone` options though, but instead using `git-submodule-update` [recursive submodue init](https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/source_control/git.py#L1198-L1209) except with `--init` and `--recursive` always fixed.\r\n\r\nProposal:\r\n\r\nExtend the `git` module to add a `submodule_init` parameter (defaults to `yes` for backward compatibility) that controls whether or not submodules are initialized at all, and update the `submodule_update` method to only initialize recursively if the current `recursive` parameter is enabled. In addition, add `submodule_depth` parameter which is also used by `submodule_update` method to expose the depth and allow us to further reduce the clone size for better scalability.\r\n\r\nExtend the `ansible-pull` command to add flags for `--no-recurse-submodules` and `--no-submodule-init` and update the [options passed to git module](https://github.com/ansible/ansible/blob/devel/lib/ansible/cli/pull.py#L188-L206) (the `submodule_depth` parameter can be set the same way as the `depth` parameter currently is, based on whether or not `--full` is provided).\r\n\r\ncc @bcoca @sivel Thoughts on proposal and likelihood they would be accepted if I submitted PRs?"},"comment":{"url":"https://api.github.com/repos/ansible/ansible/issues/comments/440405835","html_url":"https://github.com/ansible/ansible/issues/48678#issuecomment-440405835","issue_url":"https://api.github.com/repos/ansible/ansible/issues/48678","id":440405835,"node_id":"MDEyOklzc3VlQ29tbWVudDQ0MDQwNTgzNQ==","user":{"login":"sivel","id":39340,"node_id":"MDQ6VXNlcjM5MzQw","avatar_url":"https://avatars3.githubusercontent.com/u/39340?v=4","gravatar_id":"","url":"https://api.github.com/users/sivel","html_url":"https://github.com/sivel","followers_url":"https://api.github.com/users/sivel/followers","following_url":"https://api.github.com/users/sivel/following{/other_user}","gists_url":"https://api.github.com/users/sivel/gists{/gist_id}","starred_url":"https://api.github.com/users/sivel/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/sivel/subscriptions","organizations_url":"https://api.github.com/users/sivel/orgs","repos_url":"https://api.github.com/users/sivel/repos","events_url":"https://api.github.com/users/sivel/events{/privacy}","received_events_url":"https://api.github.com/users/sivel/received_events","type":"User","site_admin":false},"created_at":"2018-11-20T19:44:49Z","updated_at":"2018-11-20T19:44:49Z","author_association":"MEMBER","body":"We have discussed this, and have no aversion to accepting it, however we would need to see an implementation.\r\n\r\nIt was discussed that a way to pass generic options to the underlying module would probably be the best option."}}
{"actor":{"display_login":"sivel"}}