{"action":"created","issue":{"url":"https://api.github.com/repos/ansible/ansible/issues/17152","repository_url":"https://api.github.com/repos/ansible/ansible","labels_url":"https://api.github.com/repos/ansible/ansible/issues/17152/labels{/name}","comments_url":"https://api.github.com/repos/ansible/ansible/issues/17152/comments","events_url":"https://api.github.com/repos/ansible/ansible/issues/17152/events","html_url":"https://github.com/ansible/ansible/issues/17152","id":172154256,"number":17152,"title":"Ansible stalls during setup when pushing to large amount of hosts","user":{"login":"digi691","id":1645548,"avatar_url":"https://avatars.githubusercontent.com/u/1645548?v=3","gravatar_id":"","url":"https://api.github.com/users/digi691","html_url":"https://github.com/digi691","followers_url":"https://api.github.com/users/digi691/followers","following_url":"https://api.github.com/users/digi691/following{/other_user}","gists_url":"https://api.github.com/users/digi691/gists{/gist_id}","starred_url":"https://api.github.com/users/digi691/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/digi691/subscriptions","organizations_url":"https://api.github.com/users/digi691/orgs","repos_url":"https://api.github.com/users/digi691/repos","events_url":"https://api.github.com/users/digi691/events{/privacy}","received_events_url":"https://api.github.com/users/digi691/received_events","type":"User","site_admin":false},"labels":[{"url":"https://api.github.com/repos/ansible/ansible/labels/bug_report","name":"bug_report","color":"fbca04"},{"url":"https://api.github.com/repos/ansible/ansible/labels/needs_info","name":"needs_info","color":"eb6420"},{"url":"https://api.github.com/repos/ansible/ansible/labels/performance","name":"performance","color":"555555"}],"state":"open","locked":false,"assignee":{"login":"jimi-c","id":26403,"avatar_url":"https://avatars.githubusercontent.com/u/26403?v=3","gravatar_id":"","url":"https://api.github.com/users/jimi-c","html_url":"https://github.com/jimi-c","followers_url":"https://api.github.com/users/jimi-c/followers","following_url":"https://api.github.com/users/jimi-c/following{/other_user}","gists_url":"https://api.github.com/users/jimi-c/gists{/gist_id}","starred_url":"https://api.github.com/users/jimi-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jimi-c/subscriptions","organizations_url":"https://api.github.com/users/jimi-c/orgs","repos_url":"https://api.github.com/users/jimi-c/repos","events_url":"https://api.github.com/users/jimi-c/events{/privacy}","received_events_url":"https://api.github.com/users/jimi-c/received_events","type":"User","site_admin":false},"assignees":[{"login":"jimi-c","id":26403,"avatar_url":"https://avatars.githubusercontent.com/u/26403?v=3","gravatar_id":"","url":"https://api.github.com/users/jimi-c","html_url":"https://github.com/jimi-c","followers_url":"https://api.github.com/users/jimi-c/followers","following_url":"https://api.github.com/users/jimi-c/following{/other_user}","gists_url":"https://api.github.com/users/jimi-c/gists{/gist_id}","starred_url":"https://api.github.com/users/jimi-c/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jimi-c/subscriptions","organizations_url":"https://api.github.com/users/jimi-c/orgs","repos_url":"https://api.github.com/users/jimi-c/repos","events_url":"https://api.github.com/users/jimi-c/events{/privacy}","received_events_url":"https://api.github.com/users/jimi-c/received_events","type":"User","site_admin":false}],"milestone":null,"comments":2,"created_at":"2016-08-19T15:13:08Z","updated_at":"2016-09-04T18:44:13Z","closed_at":null,"body":"<!---\r\nPlease do not report issues/requests related to Ansible modules here !!\r\n\r\nReport them to the appropriate modules-core or modules-extras project:\r\n  - https://github.com/ansible/ansible-modules-core/issues\r\n  - https://github.com/ansible/ansible-modules-extras/issues\r\n\r\nAlso verify first that your issue/request is not already reported in GitHub\r\n-->\r\n\r\n\r\n##### ISSUE TYPE\r\n<!--- Pick one below and delete the rest: -->\r\n - Bug Report\r\n\r\n##### COMPONENT NAME\r\n<!--- Name of the plugin/task/feature -->\r\n- Performance Improvement (executor, strategy plugin, etc... )\r\n\r\n##### ANSIBLE VERSION\r\n<!--- Paste verbatim output from “ansible --version” between quotes below -->\r\n```\r\nansible 2.2.0 (devel c5b5a20031) last updated 2016/08/18 18:11:18 (GMT -500)\r\n  lib/ansible/modules/core: (detached HEAD 91a839f1e3) last updated 2016/08/18 19:24:24 (GMT -500)\r\n  lib/ansible/modules/extras: (detached HEAD 1aeb9f8a8c) last updated 2016/08/18 19:24:24 (GMT -500)\r\n\r\n```\r\n\r\n##### CONFIGURATION\r\n<!---\r\nMention any settings you have changed/added/removed in ansible.cfg\r\n(or using the ANSIBLE_* environment variables).\r\n-->\r\nstrategy=free & strategy=linear\r\n##### OS / ENVIRONMENT\r\n<!---\r\nMention the OS you are running Ansible from, and the OS you are\r\nmanaging, or say “N/A” for anything that is not platform-specific.\r\n-->\r\nControl: Ubuntu 14.04.3 LTS\r\nManaging: N/A\r\n##### SUMMARY\r\n<!--- Explain the problem briefly -->\r\nWhen pushing Ansible to over 1000 hosts, Ansible will stall when running the built-in setup tasks on the managing hosts.  A linear strategy seems to get further than free, but both end up stalling before getting to any actual task in the playbook.\r\n\r\n##### STEPS TO REPRODUCE\r\n<!---\r\nFor bugs, show exactly how to reproduce the problem.\r\nFor new features, show how the feature would be used.\r\n-->\r\nWhen checking out commit d2b3b2c03e2934b126f5701e5f6e25821e2dbe35 , Ansible will stall when pushing to over 1000 hosts, when checked out to the prior commit 80385a47bda4207a39cc6437458bbe3c3bb34d63 the behavior does not exist.\r\n<!--- Paste example playbooks or commands between quotes below -->\r\n```\r\n<playbook_test.yml>\r\n---\r\n\r\n- hosts: \"{{ hosts|default('all') }}\"\r\n  strategy: \"{{ strategy|default('linear') }}\"\r\n  user: root\r\n  vars_files:\r\n    - var_files/{{ ansible_distribution }}.yml\r\n    - var_files/{{ ansible_distribution }}-{{ ansible_distribution_major_version }}.yml\r\n    - var_files/{{ ansible_distribution }}-{{ ansible_distribution_version }}.yml\r\n  handlers:\r\n    - include: handlers/main.yml\r\n      static: yes\r\n  tasks:\r\n    - name: test\r\n      command: \"echo foo\"\r\n```\r\n\r\n<!--- You can also paste gist.github.com links for larger files -->\r\nhosts file: https://gist.github.com/digi691/77929ae3c1fd3eec3a23fbda30439862\r\n##### EXPECTED RESULTS\r\n<!--- What did you expect to happen when running the steps above? -->\r\nSuccessful run in the 5-6 minute timeframe.\r\n##### ACTUAL RESULTS\r\n<!--- What actually happened? If possible run with extra verbosity (-vvvv) -->\r\nAnsible will stall with strategy free after about 20 TASK [setup] messages have printed to the screen.  strategy linear maybe gets 2-3x more than that before stalling.  Output below is the last few dozen of lines while using -vvvv.\r\n\r\n<!--- Paste verbatim command output between quotes below -->\r\n```\r\n<xxxx-xxxxx> PUT /tmp/root/tmpSvqING TO /root/.ansible/tmp/ansible-tmp-1471484820.62-266171058798943/setup.py\r\n<xxxx-xxxxx> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAu\r\nthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r '[chhq-vudtbrix09]'\r\n<xxxx-xxxxx> ESTABLISH SSH CONNECTION FOR USER: root\r\n<xxxx-xxxxx> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthenti\r\ncation=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r chhq-vudtbrix09 '/bin/sh -c '\"'\"'chmod u+x /root/.ansible/tmp/ansible-tmp-1471484820.62-266171058798943/ /root/.ansible/tmp/ansible-tmp-1471484820.62-266171\r\n058798943/setup.py && sleep 0'\"'\"''\r\n<xxxx-xxxxx> ESTABLISH SSH CONNECTION FOR USER: root\r\n<xxxx-xxxxx> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthenti\r\ncation=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt chhq-vudtbrix09 '/bin/sh -c '\"'\"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1471484820.62-266171058798943/setup.py; rm -rf \"/root/.ansible/tmp/ansib\r\nle-tmp-1471484820.62-266171058798943/\" > /dev/null 2>&1 && sleep 0'\"'\"''\r\n<xxxx-xxxxx> ESTABLISH SSH CONNECTION FOR USER: root\r\n<xxxx-xxxxx> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentica\r\ntion=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r lnhq-vupadm01 '/bin/sh -c '\"'\"'chmod u+x /root/.ansible/tmp/ansible-tmp-1471484830.66-44898110337002/ /root/.ansible/tmp/ansible-tmp-1471484830.66-44898110337\r\n002/setup.py && sleep 0'\"'\"''\r\n<xxxx-xxxxx> ESTABLISH SSH CONNECTION FOR USER: root\r\n<xxxx-xxxxx> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentica\r\ntion=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt lnhq-vupadm01 '/bin/sh -c '\"'\"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1471484830.66-44898110337002/setup.py; rm -rf \"/root/.ansible/tmp/ansible-tm\r\np-1471484830.66-44898110337002/\" > /dev/null 2>&1 && sleep 0'\"'\"''\r\n<xxxx-xxxxx> PUT /tmp/root/tmp7z18Xh TO /root/.ansible/tmp/ansible-tmp-1471484830.89-182835333315991/setup.py\r\n<xxxx-xxxxx> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuth\r\nentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r '[sg1e-vupadm01]'\r\n<xxxx-xxxxx> ESTABLISH SSH CONNECTION FOR USER: root\r\n<xxxx-xxxxx> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentica\r\ntion=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r sg1e-vupadm01 '/bin/sh -c '\"'\"'chmod u+x /root/.ansible/tmp/ansible-tmp-1471484830.89-182835333315991/ /root/.ansible/tmp/ansible-tmp-1471484830.89-1828353333\r\n15991/setup.py && sleep 0'\"'\"''\r\n<xxxx-xxxxxx> ESTABLISH SSH CONNECTION FOR USER: root\r\n<xxxx-xxxxxx> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentica\r\ntion=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt sg1e-vupadm01 '/bin/sh -c '\"'\"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1471484830.89-182835333315991/setup.py; rm -rf \"/root/.ansible/tmp/ansible-t\r\nmp-1471484830.89-182835333315991/\" > /dev/null 2>&1 && sleep 0'\"'\"''\r\n<xxxx-xxxxx> PUT /tmp/root/tmpQ6zoSd TO /root/.ansible/tmp/ansible-tmp-1471484824.79-234080379367754/setup.py\r\n<xxxx-xxxxx> SSH: EXEC sftp -b - -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAu\r\nthentication=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r '[mt1n-vudredis02]'\r\n<mt1n-vudredis02> ESTABLISH SSH CONNECTION FOR USER: root\r\n<xxx-xxxxx> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthenti\r\ncation=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r mt1n-vudredis02 '/bin/sh -c '\"'\"'chmod u+x /root/.ansible/tmp/ansible-tmp-1471484824.79-234080379367754/ /root/.ansible/tmp/ansible-tmp-1471484824.79-234080\r\n379367754/setup.py && sleep 0'\"'\"''\r\n<mt1n-vudredis02> ESTABLISH SSH CONNECTION FOR USER: root\r\n<xxx-xxxxx2> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthenti\r\ncation=no -o User=root -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/ansible-ssh-%h-%p-%r -tt mt1n-vudredis02 '/bin/sh -c '\"'\"'/usr/bin/python /root/.ansible/tmp/ansible-tmp-1471484824.79-234080379367754/setup.py; rm -rf \"/root/.ansible/tmp/ansib\r\nle-tmp-1471484824.79-234080379367754/\" > /dev/null 2>&1 && sleep 0'\"'\"''\r\n\r\n```\r\n"},"comment":{"url":"https://api.github.com/repos/ansible/ansible/issues/comments/244621451","html_url":"https://github.com/ansible/ansible/issues/17152#issuecomment-244621451","issue_url":"https://api.github.com/repos/ansible/ansible/issues/17152","id":244621451,"user":{"login":"digi691","id":1645548,"avatar_url":"https://avatars.githubusercontent.com/u/1645548?v=3","gravatar_id":"","url":"https://api.github.com/users/digi691","html_url":"https://github.com/digi691","followers_url":"https://api.github.com/users/digi691/followers","following_url":"https://api.github.com/users/digi691/following{/other_user}","gists_url":"https://api.github.com/users/digi691/gists{/gist_id}","starred_url":"https://api.github.com/users/digi691/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/digi691/subscriptions","organizations_url":"https://api.github.com/users/digi691/orgs","repos_url":"https://api.github.com/users/digi691/repos","events_url":"https://api.github.com/users/digi691/events{/privacy}","received_events_url":"https://api.github.com/users/digi691/received_events","type":"User","site_admin":false},"created_at":"2016-09-04T18:44:13Z","updated_at":"2016-09-04T18:44:13Z","body":"My testing was done as of commit 06c76034dc203dac52ddf5cb6b6afc4b484a25af with 1384 hosts in my host list.\r\n\r\nStrategy linear mode is now successful, tested with multiple pushes.\r\n`ansible-playbook -f 25 -i hosts -e strategy=linear playbook-test_setup.yml`\r\n\r\nStrategy Free is still unsuccessful although I get a lot further into the play before it stalls, I believe past all of setup, but cannot be sure.  When trying to create the debug output for this comment, the debug stalls a lot sooner than without ANSIBLE_DEBUG=1.  Below are the output from two separate runs, one for when the fork was set to 25 and another where the fork was set to 50.  Although it may not be useful, it seems something else isn't quite right or an indication of another problem.\r\n\r\n`ANSIBLE_DEBUG=1 ansible-playbook -f 25 -i hosts -e strategy=free playbook-test_setup.yml | tee -a debug_stall_9-4-16`\r\nhttps://gist.github.com/digi691/319dff4bf7bb5e6ad8dc4eb8366e1c24\r\n\r\n`ANSIBLE_DEBUG=1 ansible-playbook -f 50 -i hosts -e strategy=free playbook-test_setup.yml | tee -a debug_stall_9-4-16_try2`\r\nhttps://gist.github.com/digi691/e4a751b8ac08a37ceaaa3405f50a6c5e\r\n\r\n@jimi-c ,\r\nI'll pop on IRC on Tuesday morning to see if your around to see if you want further details.\r\n\r\n"}}
{"actor":{"display_login":"digi691"}}