title: AWX deployment at scale
body: <p>We are currently developing an architecture to automate a very big amount of hosts in a multicloud scenario with Ansible and AWX.
  Currently we are in a phase where we need an decisions for how we want to scale our AWX installation.
  But maybe at first some key points to our setup.</p>
  <pre><code>- Distributed customer hosts across multiple regions and cloudproviders
  - Currently no mass jobs planned
  - Playbooks will stay identical over all Regions and cloudproviders
  - Playbooks will be used for initial provisioning and deployment and for day 2 operations like reboot, etc…
  - Central CMDB available
  </code></pre>
  <p>Our questions is now which would be the best design to get a stable and scaling AWX deployment.
  I got 3 scenarios in my mind which could fulfill our needs.</p>
  <p>The first one would be a central K8S based installation which could scale across the Kubernetes cluster.</p>
  <pre><code>- The benefits would be one single point of configuration and less effort during updates/upgrades.  
  - The downsides would be the SPOF and the big blast radius during an outage of AWX or a network connection. Also the load on the database could be big problem in my opinion.  
  </code></pre>
  <p>The second one would be a distributed installation with a central AWX installation and isolated AWX hosts in every region.</p>
  <pre><code>- The benefit would be one single point of configuration and dedicated workers for each Region  
  - The downsides would be the SPOF and the big blast radius during an outage of AWX or a network connection. Also the load on the database could be big problem in my opinion.  
  </code></pre>
  <p>The 3rd one would be an AWX cluster for each region which gets the current configuration pushed from a central  place via the API ( Projects, Templates, etc…)</p>
  <pre><code>- This would eliminate the SPOF and reduce the blast radius. It also would spread the database load over many instances.  
  - The downsides: configuration automation has to be developed and has to be rock stable.  
  </code></pre>
  <p>Has someone experience in running a setup for this use case and maybe can tell if one of the above approaches was working for you?<br />
  Specially the experience with the database setup in regards to single instance performance would be interesting?<br />
  Did someone experience DB problems when the &quot;critical mass&quot; of hosts was reached?</p>