<p>I've been working on a <a href="https://github.com/dgp1130/chatter" rel="nofollow noreferrer">small side project</a> to try and learn Kubernetes. I have a relatively simple cluster with two services, an ingress, and working on adding a Redis database now. I'm hosting this cluster in <a href="https://cloud.google.com/kubernetes-engine/" rel="nofollow noreferrer">Google Kubernetes Engine (GKE)</a>, but using <a href="https://kubernetes.io/docs/setup/learning-environment/minikube/" rel="nofollow noreferrer">Minikube</a> to run the cluster locally and try everything out before I commit any changes and push them to the prod environment in GKE.</p>

<p>During this project, I have noticed that GKE seems to have some slight differences in how it wants the configuration vs what works in Minikube. I've seen this previously with ingresses and now with persistent volumes.</p>

<p>For example, to run Redis with a persistent volume in GKE, I can use:</p>

<pre><code>apiVersion: apps/v1
kind: Deployment
metadata:
  name: chatter-db-deployment
  labels:
    app: chatter
spec:
  replicas: 1
  selector:
    matchLabels:
      app: chatter-db-service
  template:
    metadata:
      labels:
        app: chatter-db-service
    spec:
      containers:
      - name: master
        image: redis
        args: [
          "--save", "3600", "1", "300", "100", "60", "10000",
          "--appendonly", "yes",
        ]
        ports:
        - containerPort: 6379
        volumeMounts:
        - name: chatter-db-storage
          mountPath: /data/
      volumes:
      - name: chatter-db-storage
        gcePersistentDisk:
          pdName: chatter-db-disk
          fsType: ext4
</code></pre>

<p>The <code>gcePersistentDisk</code> section at the end refers to a disk I created using <code>gcloud compute disks create</code>. However, this simply won't work in Minikube as I can't create disks that way.</p>

<p>Instead, I need to use:</p>

<pre><code>      volumes:
      - name: chatter-db-storage
        persistentVolumeClaim:
          claimName: chatter-db-claim
</code></pre>

<p>I also need to include separate configuration for a <code>PeristentVolume</code> and a <code>PersistentVolumeClaim</code>.</p>

<p>I can easily get something working in either Minikube <strong>OR</strong> GKE, but I'm not sure what is the best means of getting a config which works for both. Ideally, I want to have a single <code>k8s.yaml</code> file which deploys this app, and <code>kubectl apply -f k8s.yaml</code> should work for both environments, allowing me to test locally with Minikube and then push to GKE when I'm satisfied.</p>

<p>I understand that there are differences between the two environments and that will probably leak into the config to some extent, but there must be an effective means of verifying a config before pushing it? What are the best practices for testing a config? My questions mainly come down to:</p>

<ol>
<li>Is it feasible to have a single Kubernetes config which can work for both GKE and Minikube?</li>
<li>If not, is it feasible to have a <strong>mostly</strong> shared Kubernetes config, which overrides the GKE and Minikube specific pieces?</li>
<li>How do existing projects solve this particular problem?</li>
<li>Is the best method to simply make a separate <code>dev</code> cluster in GKE and test on that, rather than bothering with Minikube at all?</li>
</ol>

Douglas Parker
<p>Yes, you have found some parts of Kubernetes configuration that was not perfect from the beginning. But there are newer solutions.</p>

<h2>Storage abstraction</h2>

<p>The idea in <em>newer</em> Kubernetes releases is that your <strong>application configuration</strong> is a <em>Deployment</em> with Volumes that refers to <em>PersistentVolumeClaim</em> for a <em>StorageClass</em>.</p>

<p>While <a href="https://kubernetes.io/docs/concepts/storage/persistent-volumes/#class" rel="nofollow noreferrer">StorageClass</a> and <em>PersistentVolume</em> belongs more to the <strong>infrastructure configuration</strong>. </p>

<p>See <a href="https://kubernetes.io/docs/tasks/configure-pod-container/configure-persistent-volume-storage/" rel="nofollow noreferrer">Configure a Pod to Use a PersistentVolume for Storage</a> on how to configure a <em>Persistent Volume</em> for Minikube. For GKE you configure a <em>Persistent Volume</em> with <em>GCEPersistentDisk</em> or if you want to deploy your app to AWS you may use a <em>Persistent Volume</em> for <em>AWSElasticBlockStore</em>.</p>

<h2>Ingress and Service abstraction</h2>

<p><code>Service</code> with type <em>LoadBalancer</em> and <em>NodePort</em> in combination with <code>Ingress</code> does not work the same way across cloud providers and <a href="https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/" rel="nofollow noreferrer">Ingress Controllers</a>. In addition, <em>Services Mesh</em> implementations like <a href="https://istio.io/" rel="nofollow noreferrer">Istio</a> have introduced <code>VirtualService</code>. The plan is to improve this situation with <a href="https://www.youtube.com/watch?v=Ne9UJL6irXY" rel="nofollow noreferrer">Ingress v2</a> as how I understand it.</p>
