{"action":"created","issue":{"url":"https://api.github.com/repos/ansible/ansible/issues/37141","repository_url":"https://api.github.com/repos/ansible/ansible","labels_url":"https://api.github.com/repos/ansible/ansible/issues/37141/labels{/name}","comments_url":"https://api.github.com/repos/ansible/ansible/issues/37141/comments","events_url":"https://api.github.com/repos/ansible/ansible/issues/37141/events","html_url":"https://github.com/ansible/ansible/pull/37141","id":303245207,"number":37141,"title":"Reduce recursion within group methods","user":{"login":"AlanCoding","id":1385596,"avatar_url":"https://avatars3.githubusercontent.com/u/1385596?v=4","gravatar_id":"","url":"https://api.github.com/users/AlanCoding","html_url":"https://github.com/AlanCoding","followers_url":"https://api.github.com/users/AlanCoding/followers","following_url":"https://api.github.com/users/AlanCoding/following{/other_user}","gists_url":"https://api.github.com/users/AlanCoding/gists{/gist_id}","starred_url":"https://api.github.com/users/AlanCoding/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlanCoding/subscriptions","organizations_url":"https://api.github.com/users/AlanCoding/orgs","repos_url":"https://api.github.com/users/AlanCoding/repos","events_url":"https://api.github.com/users/AlanCoding/events{/privacy}","received_events_url":"https://api.github.com/users/AlanCoding/received_events","type":"User","site_admin":false},"labels":[{"id":785271699,"url":"https://api.github.com/repos/ansible/ansible/labels/bug","name":"bug","color":"fbca04","default":true},{"id":612318497,"url":"https://api.github.com/repos/ansible/ansible/labels/inventory","name":"inventory","color":"1d76db","default":false},{"id":636352796,"url":"https://api.github.com/repos/ansible/ansible/labels/support:core","name":"support:core","color":"ededed","default":false},{"id":468273802,"url":"https://api.github.com/repos/ansible/ansible/labels/test","name":"test","color":"0e8a16","default":false}],"state":"open","locked":false,"assignee":null,"assignees":[],"milestone":null,"comments":3,"created_at":"2018-03-07T20:19:35Z","updated_at":"2018-03-20T15:57:35Z","closed_at":null,"author_association":"MEMBER","pull_request":{"url":"https://api.github.com/repos/ansible/ansible/pulls/37141","html_url":"https://github.com/ansible/ansible/pull/37141","diff_url":"https://github.com/ansible/ansible/pull/37141.diff","patch_url":"https://github.com/ansible/ansible/pull/37141.patch"},"body":"##### SUMMARY\r\n\r\nfully Fixes #36431\r\n\r\nmaybe Fixes #31797\r\n\r\nreplaces #37039, #36437\r\n\r\nThis increases the maximum number of hosts / groups that Ansible can handle before the time to parse the inventory becomes so long as to make it unusable.\r\n\r\nMy metric is the number of hosts or groups that will get the runtime within the 10 second range. I have certain reference cases that I'm running from this repo:\r\n\r\nhttps://github.com/AlanCoding/Ansible-inventory-file-examples/\r\n\r\n| improved                                  | old                 | new                   |\r\n|-------------------------------------------|---------------------|-----------------------|\r\n| groups only, dense                        | 24 groups (276 connections)  | 180 groups (16k connections)   |\r\n| hosts + groups, dense, 1 direct h->g membership per host   | 250 hosts                | 10k hosts                  |\r\n| host + groups, dense, max h->g membership | 900 hosts (38k connections) | 2000 hosts (125k connections) |\r\n| **same**                                      |                     |                       |\r\n| groups only, linear (ordering not tested) | 1k-ish hosts             |                       |\r\n| host + groups, linear                     | 1k-ish hosts             |                       |\r\n\r\nIf you want to run those, run this from the examples repo root\r\n\r\n```\r\n# new version can pull off\r\n# 16k connections\r\ntime NUMBER_GROUPS=180 ansible-playbook -i scripts/connections/dag_max.py debugging/hello_world.yml\r\ntime NUMBER_HOSTS=10000 ansible-playbook -i ./scripts/connections/hosts/dag_max.py debugging/hello_world.yml\r\n# 125k connections\r\ntime NUMBER_HOSTS=2000 ALL_GROUPS=true ansible-playbook -i ./scripts/connections/hosts/dag_max.py debugging/hello_world.yml\r\n\r\n# still not improved\r\ntime NUMBER_HOSTS=1000 ansible-playbook -i ./scripts/connections/hosts/dag_linear.py debugging/hello_world.yml\r\ntime NUMBER_GROUPS=900 ansible-playbook -i scripts/connections/dag_linear.py debugging/hello_world.yml\r\n```\r\n\r\n(the reason I created a new PR) My prior attempt was assuming that group-group optimization would translate to host-group optimization. After testing it, I found that was not the case because `populate_ancestors` triggered similarly deep recursion, unexpectedly scaling with indirect host-group membership.\r\n\r\nThis turned into a larger rewrite, and went kind of set-crazy.\r\n\r\nAdded tests because of the scale of all that.\r\n\r\n##### ISSUE TYPE\r\n - Bugfix Pull Request\r\n\r\n##### COMPONENT NAME\r\ninventory\r\n\r\n##### ANSIBLE VERSION\r\n```\r\n2.5 / 2.6\r\n```\r\n\r\n\r\n##### ADDITIONAL INFORMATION\r\nIn `devel` all of the tests pass except for `test_ancestors_recursive_loop_safe`. I know this isn't strictly required if the cycles are detected before it's called, but I think it's nice to have coverage of this for while loops that could be altered to misbehave.\r\n"},"comment":{"url":"https://api.github.com/repos/ansible/ansible/issues/comments/374651979","html_url":"https://github.com/ansible/ansible/pull/37141#issuecomment-374651979","issue_url":"https://api.github.com/repos/ansible/ansible/issues/37141","id":374651979,"user":{"login":"AlanCoding","id":1385596,"avatar_url":"https://avatars3.githubusercontent.com/u/1385596?v=4","gravatar_id":"","url":"https://api.github.com/users/AlanCoding","html_url":"https://github.com/AlanCoding","followers_url":"https://api.github.com/users/AlanCoding/followers","following_url":"https://api.github.com/users/AlanCoding/following{/other_user}","gists_url":"https://api.github.com/users/AlanCoding/gists{/gist_id}","starred_url":"https://api.github.com/users/AlanCoding/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/AlanCoding/subscriptions","organizations_url":"https://api.github.com/users/AlanCoding/orgs","repos_url":"https://api.github.com/users/AlanCoding/repos","events_url":"https://api.github.com/users/AlanCoding/events{/privacy}","received_events_url":"https://api.github.com/users/AlanCoding/received_events","type":"User","site_admin":false},"created_at":"2018-03-20T15:57:35Z","updated_at":"2018-03-20T15:57:35Z","author_association":"MEMBER","body":"If you could give me some basic outline of scenarios, then I would be happy to write tests for them."}}
{"actor":{"display_login":"AlanCoding"}}