<p>Goal: have a yaml deployment that I simply run <code>kubectl create -f deployment.yaml</code> on and it create nodes with pods spread accross them in the manner I describe in the yaml file.</p>

<p>Goal output. 3 nodes on google cloud with 2 nodes with 2 pods front-end/back-end and a 3rd node with the database running with a persistent volume. </p>

<p>There is no way to test this locally since docker-desktop is a single node system. Google cloud initially put all my pods on one node and they did not schedule because of lack of resources. </p>

<p>So I looked up the docs and asked several slack channels. The consensus is to use pod/node affinity and antiAffinity to make sure the pods I want are on the nodes I want. </p>

<p><a href="https://github.com/marcusjwhelan/docker_js_stack_client" rel="nofollow noreferrer">Client Application github</a></p>

<p><a href="https://github.com/marcusjwhelan/docker_js_stack_api" rel="nofollow noreferrer">API/DB application github</a></p>

<p><a href="https://github.com/marcusjwhelan/docker_js_stack" rel="nofollow noreferrer">Project overhead, k8s+Kustomize</a></p>

<p>Here are the github repos I am testing this with and trying to build what I am describing. I am trying to stay as cloud agnostic as possibly but if I need to be more specific with cloud keywords please leave a comment about it. The whole deployment is spread between the /kubernetes directory in each project. They did work locally before I added the pod/node affinity items. </p>

mjwrazor
<p>What you're looking for here is <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/" rel="nofollow noreferrer">Kubeadm</a>. It's a tool used to bootstrap Kubernetes masters and join new nodes to clusters. <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" rel="nofollow noreferrer">Here</a> is a reference document and <a href="https://godoc.org/k8s.io/kubelet/config/v1beta1#KubeletConfiguration" rel="nofollow noreferrer">here</a> are the API docs for <a href="https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/" rel="nofollow noreferrer">Kubelet</a> and all of its arguments. Using Kubeadm and these arguments, you can build an <a href="https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#InitConfiguration" rel="nofollow noreferrer">initConfiguration</a>, <a href="https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#ClusterConfiguration" rel="nofollow noreferrer">clusterConfiguration</a>, and <a href="https://godoc.org/k8s.io/kubernetes/cmd/kubeadm/app/apis/kubeadm#JoinConfiguration" rel="nofollow noreferrer">nodeConfiguration</a>. </p>

<p>1) After you create these three files, put <code>initConfiguration</code> and <code>clusterConfiguration</code> into a single YAML file. And put <code>nodeConfiguration</code> into its own file. Put the first file on your master(s) and the second file on your node(s). If you're installing with a Multi-Master configuration, keep <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/" rel="nofollow noreferrer">this document</a> in mind when building your configurations. You will also need a <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/high-availability/#create-load-balancer-for-kube-apiserver" rel="nofollow noreferrer">load balancer</a>, <a href="http://dockerlabs.collabnix.com/kubernetes/beginners/Install-and-configure-a-multi-master-Kubernetes-cluster-with-kubeadm.html" rel="nofollow noreferrer">proxy</a>, or clever <a href="https://github.com/coredns/coredns/tree/master/plugin/loadbalance" rel="nofollow noreferrer">DNS trickery</a> to enable this. </p>

<p>2) <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/" rel="nofollow noreferrer">Install the following packages</a>:</p>

<pre><code>docker
containerd
kubelet
kubeadm
</code></pre>

<p>3) <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#before-you-begin" rel="nofollow noreferrer">Disable any Swap partitions, SELinux,</a> and <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/install-kubeadm/#check-required-ports" rel="nofollow noreferrer">configure your firewall</a> appropriately. </p>

<p>3) <a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/" rel="nofollow noreferrer">Bootstrap Kubernetes</a> on your Master(s) with your config file: <code>kubeadm init --config /tmp/master.yml</code>.</p>

<p>4) <a href="https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/" rel="nofollow noreferrer">Add your workers nodes to the cluster</a> with your config file: <code>kubeadm join --config /tmp/node.yml</code>.</p>

<p>5) Install a <a href="https://kubernetes.io/docs/setup/production-environment/tools/kubeadm/create-cluster-kubeadm/#pod-network" rel="nofollow noreferrer">network plugin</a>.</p>

<p>6) Deploy the rest of your software using a <a href="https://kustomize.io/" rel="nofollow noreferrer">Kustomize</a> project. Hell, you could even <a href="https://blog.scottlowe.org/2019/10/16/using-kustomize-with-kubeadm-configuration-files/" rel="nofollow noreferrer">manage your Kubeadm configuration files with Kustomize</a>. I have. You could even throw the network plugin manifest in here. </p>

<p>This Kubernetes solution is fully cloud agnostic and all of Kubernetes is automated but the configuration isn't. I would automate steps 1-5 with something like <a href="https://www.ansible.com/" rel="nofollow noreferrer">Ansible</a>. Where you would declare each step as a task or role in a playbook. It supports a <a href="https://docs.ansible.com/ansible/latest/modules/list_of_all_modules.html" rel="nofollow noreferrer">ton of modules</a>. And there are open source roles on <a href="https://galaxy.ansible.com/" rel="nofollow noreferrer">Galaxy</a> when there aren't.</p>

<p>That leaves your infrastructure. Making this cloud agnostic is impossible. No two vendors have the same API. That's why abstraction layers like Kubernetes exist to begin with. To solve this problem, attempt to be <em>Cloud Aware</em> instead. Use a tool like <a href="https://www.terraform.io/" rel="nofollow noreferrer">Terraform</a> to manage your infrastructure for you. Then, <a href="https://docs.ansible.com/ansible/latest/modules/terraform_module.html" rel="nofollow noreferrer">add your required Terraform commands to your Ansible playbook</a>.</p>

<p>Bam, now you can deploy your entire infrastructure, configuration, and application with a single command. From a <em>single</em> YAML file:</p>

<p><code>ansible-playbook -i inventory.ini site.yml</code> </p>

<p>Well, a <em>single</em> YAML file and a bunch of other <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks_reuse_roles.html" rel="nofollow noreferrer">roles</a> and <a href="https://docs.ansible.com/ansible/latest/modules/template_module.html" rel="nofollow noreferrer">templates</a>, <a href="https://docs.ansible.com/ansible/latest/user_guide/playbooks.html" rel="nofollow noreferrer">playbooks</a>, and <a href="https://www.terraform.io/docs/providers/google/index.html" rel="nofollow noreferrer">Terraform configuration files</a>. ;) </p>

<p>EDIT: Here are some open source projects I've written that leverage these ideas and design principles. You should be able to at least gleam a lot of Ansible logic from them. The first is meant to build a single node cluster on a bare metal server. The latter is meant to build a four node cluster on Proxmox. </p>

<ul>
<li><a href="https://github.com/zimmertr/Bare_Metal_Kubeadm" rel="nofollow noreferrer">Bare_Metal_Kubeadm</a></li>
<li><a href="https://github.com/zimmertr/Bootstrap-Kubernetes-with-QEMU" rel="nofollow noreferrer">Bootstrap-Kubernetes-With-QEMU</a></li>
</ul>

<p>Both work fully with all of the manifests I have written <a href="https://github.com/zimmertr/Kubernetes-Manifests" rel="nofollow noreferrer">here</a>. Which should also provide you with a wealth of Kustomize and Ansible examples. </p>
