# Question
Title: Why am I getting error trying to mount a directory onto a file?
I have this prometheus deployment manifest file: to create prometheus deployment

```
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: prometheus
  namespace: monitoring
  labels:
    app: prometheus
spec:
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  selector:
    matchLabels:
      app: prometheus
  template:
    metadata:
      labels:
        app: prometheus
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "9090"
    spec:
      nodeSelector:
        boardType: aarch64
      initContainers:
      - name: prometheus-data-permission-setup
        image: busybox
        command: ["/bin/chmod","-R","777", "/data"]
        volumeMounts:
        - name: prometheus-data
          mountPath: /data
      containers:
      - name: prometheus
        image: rycus86/prometheus:latest
        imagePullPolicy: IfNotPresent
        args:
          - '--storage.tsdb.path=/data'
          - '--config.file=/etc/prometheus/prometheus.yml'
        command:
        - /bin/prometheus
        ports:
        - name: web
          containerPort: 9090
        volumeMounts:
        - name: config-volume
          mountPath: /etc/prometheus/prometheus.yml
          subPath: prometheus.yml
        - name: prometheus-data
          mountPath: /data
          readOnly: false
        resources:
          requests:
            cpu: "250m"
            ephemeral-storage: "4Gi"
          limits:
            memory: "200Mi"
            cpu: "500m"
            ephemeral-storage: "20Gi"
      restartPolicy: Always
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config-volume
        configMap:
          name: prometheus-config
      - name: prometheus-data
        hostPath:
          path: /data
          type: Directory
      serviceAccountName: prometheus

```

but this error that "trying to mount a directory onto a file "

```
Events:
  Type     Reason     Age               From               Message
  ----     ------     ----              ----               -------
  Normal   Scheduled  30s               default-scheduler  Successfully assigned monitoring/prometheus-b568c9967-tmdq7 to rockpro64
  Normal   Pulling    28s               kubelet            Pulling image "busybox"
  Normal   Pulled     26s               kubelet            Successfully pulled image "busybox" in 1.57966103s
  Normal   Created    26s               kubelet            Created container prometheus-data-permission-setup
  Normal   Started    25s               kubelet            Started container prometheus-data-permission-setup
  Warning  Failed     23s               kubelet            Error: failed to start container "prometheus": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/da8d1a99-a6d0-48f5-ad5e-cbc5db136145/volume-subpaths/config-volume/prometheus/0\\\" to rootfs \\\"/var/lib/docker/overlay2/32aead2ba4f70863e9cf643a1bad1fb0832eb43c217898bfe77cc67ab400d702/merged\\\" at \\\"/var/lib/docker/overlay2/32aead2ba4f70863e9cf643a1bad1fb0832eb43c217898bfe77cc67ab400d702/merged/etc/prometheus/prometheus.yml\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
  Warning  Failed     23s               kubelet            Error: failed to start container "prometheus": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/da8d1a99-a6d0-48f5-ad5e-cbc5db136145/volume-subpaths/config-volume/prometheus/0\\\" to rootfs \\\"/var/lib/docker/overlay2/9dcb8fa640d8fc93f56755f5daeb2a350c482f5f3f26faee03edb320d52f53ff/merged\\\" at \\\"/var/lib/docker/overlay2/9dcb8fa640d8fc93f56755f5daeb2a350c482f5f3f26faee03edb320d52f53ff/merged/etc/prometheus/prometheus.yml\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type
  Warning  BackOff    22s               kubelet            Back-off restarting failed container
  Normal   Pulled     7s (x3 over 24s)  kubelet            Container image "rycus86/prometheus:latest" already present on machine
  Normal   Created    7s (x3 over 24s)  kubelet            Created container prometheus
  Warning  Failed     6s                kubelet            Error: failed to start container "prometheus": Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused \"rootfs_linux.go:58: mounting \\\"/var/lib/kubelet/pods/da8d1a99-a6d0-48f5-ad5e-cbc5db136145/volume-subpaths/config-volume/prometheus/0\\\" to rootfs \\\"/var/lib/docker/overlay2/5b263ca359000d143fd5f11fb26218ead46eabd1a1aeb39ce1a7b48302a2382c/merged\\\" at \\\"/var/lib/docker/overlay2/5b263ca359000d143fd5f11fb26218ead46eabd1a1aeb39ce1a7b48302a2382c/merged/etc/prometheus/prometheus.yml\\\" caused \\\"not a directory\\\"\"": unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

```

I have prometheus-config configmap already setup.

```
# kubectl get cm -n monitoring
NAME                DATA   AGE
kube-root-ca.crt    1      35h
prometheus-config   1      41m

```

what might have gone wrong ?

# Answer
`volumeMount` based on `configMap` actually creates the files for the data keys. You don't need the filename in the `mountPath` or the `subPath`

```
$ cat <<EOF | kubectl apply -f -
apiVersion: v1
data:
  settings.conf: |
    test: config
    multi: line
kind: ConfigMap
metadata:
  name: test-config
---
apiVersion: v1
kind: Pod
metadata:
  name: dapi-test-pod
spec:
  containers:
    - name: test-container
      image: k8s.gcr.io/busybox
      command: [ "/bin/sh", "-c", "ls /etc/config/" ]
      volumeMounts:
      - name: config-volume
        mountPath: /etc/config
  restartPolicy: Never
  volumes:
    - name: config-volume
      configMap:
        name: test-config
EOF

$ kubectl logs dapi-test-pod
settings.conf

```

Ref:
> 2 votes

---
Tags: kubernetes
---


# Question
Title: How to check application responding in Kubernetes
I am a newbie in k8s. I have a .net framework application written in C# running in K8s orchestration. I simply want to check in fixed interval (lets say in every 300sec) that my application which is running inside pod is responding or not.
Can anyone suggest how to implement this ?

# Answer
Kubernetes supports TCP, HTTP, and shell-exec probes for an application startup, readiness, and liveness health checks. If your application is a web application, you can use HTTP based liveness healthcheck as shown in the following code snippet:

```
apiVersion: v1
kind: Pod
metadata:
  labels:
    test: liveness
  name: liveness-http
spec:
  containers:
  - name: liveness
    image: k8s.gcr.io/liveness
    args:
    - /server
    livenessProbe:
      httpGet:
        path: /healthz
        port: 8080
      initialDelaySeconds: 3
      periodSeconds: 300

```
> 1 votes

---
Tags: kubernetes, dotnet, healthcheck
---


# Question
Title: When to use Apache Airflow (MWAA) in AWS instead of AWS Batch?
I am trying to get a `compare and contrast` of various job processing services in AWS like AWS Batch, MWAA (Managed Workflow for Apache Airflow).

Both are used for processing jobs. Just trying to understand when we should go for Batch and when we should go for MWAA.

# Answer
> 4 votes
AWS MWAA uses Apache Airflow to create workflows and DAGS (Directed Acyclic Graphs) with Python to orchestrate complex, dependent tasks. AWS Batch is, as the name states, a batch processing service that utilizes docker containers. Batch allows you to manage instance types and container sizes to fine tune costs of workloads. Batch unlike MWAA does not utilize any orchestration service, so apart from the queue that it utilizes, there is no way to control the workflow of tasks.

If you have a job that requires complex workflows, AWS MWAA will reduce the complexity of managing those workflows. If you require large scale containerized processing for image analysis, data conversion, etc., AWS Batch will allow you to easily control your workloads to optimize speed and cost.

It should also be noted that AWS Step Functions is also a very efficient way to orchestrate various types of processing and application tasks. For simple tasks, there is very little overhead and management compared to AWS Batch or MWAA.

---
Tags: amazon-web-services, apache-airflow
---


# Question
Title: How to pin public root key when downloading an image with docker pull (Docker Content Trust)?
How can I execute `docker pull` (with Docker Content Trust enabled) such that it fails if the image doesn't have a valid signature using the private key corresponding to (or subordinate to) the public key that I provide?

I just discovered that, in fact, DCT silently and dumbly downloads and trusts (TOFU) the root key when downloading an image for the first time.

My application for docker is a CI process that will use the `debian:stable-latest` docker image to build my application's releases in ephemeral cloud instances. Because the server used to build my application is ephemeral, DCT literally provides me zero security and it will be vulnerable to a MITM of the docker image payload on *every* execution of my build process.

Therefore, it's critical that I pin the public root key used by Debian to sign their docker images. So the questions are:

1. How do I download the public root key used by Debian to sign their docker images and
2. How do I tell `docker pull` to use the above key and fail if the image doesn't contain a valid signature from that key or a key that's downchain from the supplied root key?

# Answer
> 1 votes
## tl;dr

You can execute the following to pin the public root key for `debian`:

```
sudo su -
mkdir -p /root/.docker/trust/tuf/docker.io/library/debian/metadata
chown -R root:root /root/.docker
chmod -R 0700 /root/.docker
echo '{"signed":{"_type":"Root","consistent_snapshot":false,"expires":"2025-08-07T20:55:22.677722315-07:00","keys":{"5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsslGF2xHOYztrocb2OsRF2zth16v170QiLAyKdce1nQgOJ34FOk679ClPL9/RNnJukf2JfQXSlVV/qcsvxV2dQ=="}},"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84":{"keytype":"ecdsa-x509","keyval":{"private":null,"public":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lRWExkUFFHTGJaOE84UXFlTzVuZlBRekFLQmdncWhrak9QUVFEQWpBak1TRXcKSHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrWldKcFlXNHdIaGNOTVRVd09ERXhNRE0xTlRJeQpXaGNOTWpVd09EQTRNRE0xTlRJeVdqQWpNU0V3SHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrClpXSnBZVzR3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVE1ZGkxcmxPQjBMQmRNS2N0VFQxYmwKUGd6aXYxOUJDdW9tNEFNL3BUdURtdjBnS0E5S1ptNUVjLy9VQmhSODVCYmR0cTk0cXhQM3IwUjhRc3FQV1Y4SQpvelV3TXpBT0JnTlZIUThCQWY4RUJBTUNBS0F3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdNd0RBWURWUjBUCkFRSC9CQUl3QURBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBOUFOZ3dPN2tBdUVIK3U2N25XNlFLWmlMdWd5UVcKaEQ3Vys5WjIza01mTndJaEFJa3RTaW1TdFdRQkFoOG9WOXhjaWNVWWVUN0pyUG82a0RqeHU1YitGZ3MxCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}},"728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAENtpBkDJ2oYaAAVdOkP0A6J0XwUkYGuFRk+q8N4WCPu2VnNIuBJkatPCWdEtHfQ9nNYLeanWgG62/UmJnx3E2Yg=="}},"d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEwfs26T/cpjvNTXVJpK7Wv8oDOnNKL78AT3Y1QD356OIAggwPupX2LQjZU6CVzCjm+pkJIO4clu9Q2n540gKuzQ=="}}},"roles":{"root":{"keyids":["575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84"],"threshold":1},"snapshot":{"keyids":["d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69"],"threshold":1},"targets":{"keyids":["5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a"],"threshold":1},"timestamp":{"keyids":["728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70"],"threshold":1}},"version":1},"signatures":[{"keyid":"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9f
fb1ab5cc4291e07bf84","method":"ecdsa","sig":"3WbX1VXN9E8LRmSG+E4SQlBUNqBNchhwAStWnRWLLyAOoFNBq5xmIgSO3UYYuKyJvL7kbMoONRbn5Vk2p2Wqrg=="}]}' > /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json

export DOCKER_CONTENT_TRUST=1
docker pull debian:stable-slim

```

## Long Answer

> ⚠ Disclaimer: I am not a docker developer. As of 2021, it appears that DCT is broken out-of-the-box and is far from being useful. My answer here is a best-guess, but I have not confirmed with the docker team if this is the "correct" way to pre-load and pin a given publisher's root key into the DCT keyring.
>
> Be advised, proceed with caution, and your comments are very welcome.

It doesn't appear to be documented anywhere, but per this question it's clear that docker puts its DCT metadata (including root public keys) in the following location:

```
$HOME/.docker/trust/tuf/docker.io/library

```

Inside this `library` dir exists one dir per publisher. For the purposes of this answer, I'll use `debian` as our example publisher.

You can see the list of the `debian` docker images published to Docker Hub here:

### Solution

Let's say we want to download the `stable-slim` image from the `debian` publisher on Docker Hub. In this example, we'll also use a fresh install of Debian 10 as the docker host.

```
##
# first, install docker
##
root@disp2716:~# apt-get install docker.io
...
root@disp2716:~#

##
# confirm that there is no docker config dir yet
##

root@disp2716:~# ls -lah /root/.docker
ls: cannot access '/root/.docker': No such file or directory
root@disp2716:~#

##
# add the debian publisher's root DCT key
##

root@disp2716:~# mkdir -p /root/.docker/trust/tuf/docker.io/library/debian/metadata
root@disp2716:~# chown -R root:root /root/.docker
root@disp2716:~# chmod -R 0700 /root/.docker
root@disp2716:~# echo '{"signed":{"_type":"Root","consistent_snapshot":false,"expires":"2025-08-07T20:55:22.677722315-07:00","keys":{"5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEsslGF2xHOYztrocb2OsRF2zth16v170QiLAyKdce1nQgOJ34FOk679ClPL9/RNnJukf2JfQXSlVV/qcsvxV2dQ=="}},"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84":{"keytype":"ecdsa-x509","keyval":{"private":null,"public":"LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lRWExkUFFHTGJaOE84UXFlTzVuZlBRekFLQmdncWhrak9QUVFEQWpBak1TRXcKSHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrWldKcFlXNHdIaGNOTVRVd09ERXhNRE0xTlRJeQpXaGNOTWpVd09EQTRNRE0xTlRJeVdqQWpNU0V3SHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrClpXSnBZVzR3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVE1ZGkxcmxPQjBMQmRNS2N0VFQxYmwKUGd6aXYxOUJDdW9tNEFNL3BUdURtdjBnS0E5S1ptNUVjLy9VQmhSODVCYmR0cTk0cXhQM3IwUjhRc3FQV1Y4SQpvelV3TXpBT0JnTlZIUThCQWY4RUJBTUNBS0F3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdNd0RBWURWUjBUCkFRSC9CQUl3QURBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBOUFOZ3dPN2tBdUVIK3U2N25XNlFLWmlMdWd5UVcKaEQ3Vys5WjIza01mTndJaEFJa3RTaW1TdFdRQkFoOG9WOXhjaWNVWWVUN0pyUG82a0RqeHU1YitGZ3MxCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}},"728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAENtpBkDJ2oYaAAVdOkP0A6J0XwUkYGuFRk+q8N4WCPu2VnNIuBJkatPCWdEtHfQ9nNYLeanWgG62/UmJnx3E2Yg=="}},"d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69":{"keytype":"ecdsa","keyval":{"private":null,"public":"MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAEwfs26T/cpjvNTXVJpK7Wv8oDOnNKL78AT3Y1QD356OIAggwPupX2LQjZU6CVzCjm+pkJIO4clu9Q2n540gKuzQ=="}}},"roles":{"root":{"keyids":["575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84"],"threshold":1},"snapshot":{"keyids":["d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69"],"threshold":1},"targets":{"keyids":["5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a"],"threshold":1},"timestamp":{"keyids":["728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70"],"threshold":1}},"version":1},"signatures":[{"keyid":"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84","method":"ecdsa","sig":"3WbX1VXN9E8LRmSG+E4SQlBUNqBNchhwAStWnRWLLyAOoFNBq5xmIgSO3UYYuKyJvL7kbMoONRbn5Vk2p2Wqrg=="}]}' > /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~#
root@disp2716:~# chown root:root /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~# chmod 0600 /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~#

##
# pull the docker image with DCT verification
##

root@disp2716:~# export DOCKER_CONTENT_TRUST=1
root@disp2716:~# docker pull debian:stable-slim
Pull (1 of 1): debian:stable-slim@sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632
sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632: Pulling from library/debian
6e640006d1cd: Pull complete
Digest: sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632
Status: Downloaded newer image for debian@sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632
Tagging debian@sha256:850a7ee21c49c99b0e5e06df21f898a0e64335ae84eb37d6f71abc1bf28f5632 as debian:stable-slim
root@disp2716:~#

```

### Proof

While there's no way to tell `docker` to fail on TOFU, we can confirm that the above key pinning works by making the public key something else

```
##
# first, move the docker config dir out of the way
##

mv /root/.docker /root/.docker.bak

##
# add the debian publisher's root DCT key (note I just overwrote the first 8
# characters of the actual key with "INVALID/")
##

root@disp2716:~# mkdir -p /root/.docker/trust/tuf/docker.io/library/debian/metadata
root@disp2716:~# chown -R root:root /root/.docker
root@disp2716:~# chmod -R 0700 /root/.docker
root@disp2716:~# echo '{"signed":{"_type":"Root","consistent_snapshot":false,"expires":"2025-08-07T20:55:22.677722315-07:00","keys":{"5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a":{"keytype":"ecdsa","keyval":{"private":null,"public":"INVALID/KoZIzj0CAQYIKoZIzj0DAQcDQgAEsslGF2xHOYztrocb2OsRF2zth16v170QiLAyKdce1nQgOJ34FOk679ClPL9/RNnJukf2JfQXSlVV/qcsvxV2dQ=="}},"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84":{"keytype":"ecdsa-x509","keyval":{"private":null,"public":"INVALID/RUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUJlVENDQVIrZ0F3SUJBZ0lRWExkUFFHTGJaOE84UXFlTzVuZlBRekFLQmdncWhrak9QUVFEQWpBak1TRXcKSHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrWldKcFlXNHdIaGNOTVRVd09ERXhNRE0xTlRJeQpXaGNOTWpVd09EQTRNRE0xTlRJeVdqQWpNU0V3SHdZRFZRUURFeGhrYjJOclpYSXVhVzh2YkdsaWNtRnllUzlrClpXSnBZVzR3V1RBVEJnY3Foa2pPUFFJQkJnZ3Foa2pPUFFNQkJ3TkNBQVE1ZGkxcmxPQjBMQmRNS2N0VFQxYmwKUGd6aXYxOUJDdW9tNEFNL3BUdURtdjBnS0E5S1ptNUVjLy9VQmhSODVCYmR0cTk0cXhQM3IwUjhRc3FQV1Y4SQpvelV3TXpBT0JnTlZIUThCQWY4RUJBTUNBS0F3RXdZRFZSMGxCQXd3Q2dZSUt3WUJCUVVIQXdNd0RBWURWUjBUCkFRSC9CQUl3QURBS0JnZ3Foa2pPUFFRREFnTklBREJGQWlBOUFOZ3dPN2tBdUVIK3U2N25XNlFLWmlMdWd5UVcKaEQ3Vys5WjIza01mTndJaEFJa3RTaW1TdFdRQkFoOG9WOXhjaWNVWWVUN0pyUG82a0RqeHU1YitGZ3MxCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K"}},"728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70":{"keytype":"ecdsa","keyval":{"private":null,"public":"INVALID/KoZIzj0CAQYIKoZIzj0DAQcDQgAENtpBkDJ2oYaAAVdOkP0A6J0XwUkYGuFRk+q8N4WCPu2VnNIuBJkatPCWdEtHfQ9nNYLeanWgG62/UmJnx3E2Yg=="}},"d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69":{"keytype":"ecdsa","keyval":{"private":null,"public":"INVALID/KoZIzj0CAQYIKoZIzj0DAQcDQgAEwfs26T/cpjvNTXVJpK7Wv8oDOnNKL78AT3Y1QD356OIAggwPupX2LQjZU6CVzCjm+pkJIO4clu9Q2n540gKuzQ=="}}},"roles":{"root":{"keyids":["575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84"],"threshold":1},"snapshot":{"keyids":["d48327d85f0490827db7c931eedb58d293e1da5fc425ea0cde3e6c13b397ad69"],"threshold":1},"targets":{"keyids":["5717dcd81d9fb5b73aa15f2d887a6a0de543829ab9b2d411acce9219c2f8ba3a"],"threshold":1},"timestamp":{"keyids":["728c96ff5e9f48d4e66d5a0c3ecabfdd90bee2b5f9f80b950ed9c668db264a70"],"threshold":1}},"version":1},"signatures":[{"keyid":"575d013f89e3cbbb19e0fb06aa33566c22718318e0c9ffb1ab5cc4291e07bf84","method":"ecdsa","sig":"3WbX1VXN9E8LRmSG+E4SQlBUNqBNchhwAStWnRWLLyAOoFNBq5xmIgSO3UYYuKyJvL7kbMoONRbn5Vk2p2Wqrg=="}]}' > /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~#
root@disp2716:~# chown root:root /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~# chmod 0600 /root/.docker/trust/tuf/docker.io/library/debian/metadata/root.json
root@disp2716:~#

##
# pull the docker image with DCT verification
##

root@disp2716:~# export DOCKER_CONTENT_TRUST=1
root@disp2716:~# docker pull debian:stable-slim
could not validate the path to a trusted root: unable to retrieve valid leaf certificates
root@disp2716:~#
root@disp2716:~# echo $?
1
root@disp2716:~#

```

Note that docker exits 1 with an error refusing to pull the `debian:stable-slim` docker image from Docker Hub because it cannot trust its signature

---
Tags: docker, linux, release-management, docker-registry
---


# Question
Title: Where to install Prometheus for Kubernetes monitoring
I'm considering using Prometheus to monitor and analyze different metrics from my AWS EKS Kubernetes cluster running in Fargate, as it seems to be the standard tool during these times. However, something that really impressed me was that installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing. Am I right or just got a wrong idea from my research? Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor? Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster? Is it possible to set up a single Prometheus installation and centralize all k8s clusters metrics on it?

Same question for Grafana's installation... Should it be installed inside or outside the cluster?

On a different topic, Prometheus history retention period is set to 15 days by default (`storage.tsdb.retention.time`). Even though cluster metrics insight should be useful for a short period of time so you can troubleshoot issues, I wonder if there may be scenarios in which a longer history period might help. Any comments on this matter?

Thanks!

# Answer
> 1 votes
This is more of an opinion based question but let me try to answer it.

> installing Prometheus itself in the very same k8s cluster (as another running set of pods) is a normal thing.Am I right or just got a wrong idea from my research?

> Considering good practices, shouldn't the monitoring tool be outside of the system that you actually intend to monitor?

This is a correct practice to have. Monitoring systems are usually setup to look into the object being monitored. (*outside looking in*). But with self healing systems like k8s, installing monitoring within the system can be done with minimum issues. There are a few cases you do need to look into for proper stability.

* Pods get deleted :- Run Prometheus in a HA mode (2 replicas minimum) so as not to lose any data.
* Pods are replaced due to resource crunch:- Add priority classes to prevent pod form being evicted.
* Use Persistent volumes:- Look into volume node affinity conflict. Also configure the data retention period to make optimised use of storage.
* Cluster goes down. See below.

> Also, if I set up multiple Kubernetes clusters, isn't it more cumbersome to have one Prometheus installation per cluster?

Not at all, using a combination of IaC(Terraform/Chef),Config management(Ansible),CI/CD (Spinnaker),GitOps and Helm charts. Configuring/maintaining Prometheus instances across multiple instances will not be an issue.

> Is it possible to set up a single Prometheus installation and centralise all k8s clusters metrics on it?

Prometheus although highly optimised, will not be able to handle all the metrics thrown at it from multiple clusters.Also Prometheus does not scale horizontally on its own, therefore it would need huge resources. Centralising all metrics to one place will cause other issues as well.

* Network Connectivity/Latency:- Running a centralised Prometheus instance receiving data from over the public internet may cause data loss/false alerts due to network drops and latency.
* Cost:- Running tens of exporter all sending data over the internet will cause high internet costs for cloud deployments.
* Cluster managing:- You may face issues with exporter/job configuration of such a large number of exporters.

Its a good choice to run Prometheus per cluster. wherein you can have all the data locally till the time is required.

Grafana does not consume much resources so its safe to run inside the cluster. However it faces a different set of issues:-

* User/Permission Management :- Each local instance would need to be maintained separately which is a huge task.
* Charts Management:- Same as above.
* L1/DevOps will have to keep n number of tabs open to monitor your infra.

Other than this, setting up alerts on all the clusters separately would also be a never-ending task.

My advice would be to setup Prometheus clusters locally (inside the cluster or right next to it) and connect all of them via Thanos or Cortex and set up a HA mode Grafana centrally on top of Thanos along with alerts and external storage.This will alert if the local Prometheus instance goes down and will help with the *outside looking in* factor

> scenarios in which a longer history period might help. Any comments on this matter?

Depends upon the metrics being stored, A quarterly SLI/SLO and incident reports seem like a good use case to store data for longer than 15 days.

---
Tags: kubernetes, monitoring, prometheus, grafana, eks
---


# Question
Title: What is the Prometheus and Grafana ideal setup?
I am just wondering. If I have many environments monitored via Prometheus, what will be the best configuration? The *security* and *efficiency* are already important for on-premise installations but they are much more important and unavoidable when deployed on a cloud provider.

From your experience, what is the most common and useful configuration? I see three main scenarios feel free to propose something else.

Thanks.

**Solution I, Duplicate nothing**

It is easy for the monitoring user because there is only one Grafana with one Datasource. However, from a security point of view, I have to open a port for each service in each environment. I will, of course, configure one reverse proxy in each environment to secure the exporters over Https and Basic authentication.

```
+---------+       +------------+       +- Environment A -------------+
| Grafana |       | Prometheus |       |     +-----------+           |
|         |------>|            |--+--+------>| Service 1 |           |
+---------+       +------------+  |  | |     |           |           |
                                  |  | |     +-----------+           |
                                  |  | |                             |
                                  |  | |     +-----------+           |
                                  |  +------>| Service 2 |           |
                                  |    |     |           |           |
                                  |    |     +-----------+           |
                                  |    +-----------------------------+
                                  |
                                  |    +- Environment B -------------+
                                  |    |     +-----------+           |
                                  +--------->| Service 1 |           |
                                  |    |     |           |           |
                                  |    |     +-----------+           |
                                  |    |                             |
                                  |    |     +-----------+           |
                                  +--------->| Service 2 |           |
                                       |     |           |           |
                                       |     +-----------+           |
                                       +-----------------------------+

```

**Solution II, Duplicate everything**

This looks like a popular solution and may have many benefits from a deployment point of view. I find it quite elegant because we can have on `docker-compose.yml` that set up everything. But while it can be interesting for security segregation, there will be many Grafana (each with one Prometheus data source) and that may not be useful for the monitoring user.

```
+- Environment A ---------------------------------------------------+
| +---------+       +------------+                                  |
| | Grafana |       | Prometheus |                                  |
| |         |------>|            |                                  |
| +---------+       +------------+    +-----------+                 |
|                             |       | Service 1 |                 |
|                             +------>|           |                 |
|                             |       +-----------+                 |
|                             |                                     |
|                             |       +-----------+                 |
|                             |       | Service 2 |                 |
|                             +------>|           |                 |
|                                     +-----------+                 |
+-------------------------------------------------------------------+

```

**Solution III, Duplicate Prometheus**

This is almost the same as **Solution II** except that I keep one Grafana with many Prometheus data sources. This looks neat too but I still have to secure my Prometheus instance behind a reverse proxy.

I am also thinking to create a **Solution III Bis** with the use of a kind of proxy or push-exporter in place of *Prometheus* in each environment.

**a)** The usage of a proxy does not bring much value except having only one data source in Grafana.

**b)** The usage of a push-exporter may remove the external entry point in each environment however I do not know how I will be "informed" when one environment is down (since it will not push anything).

```
                 +- Environment A ----------------------------------+
 +---------+     | +------------+                                   |
 | Grafana |     | | Prometheus |                                   |
 |         |-+---->|            |                                   |
 +---------+ |   | +------------+    +-----------+                  |
             |   |           |       | Service 1 |                  |
             |   |           +------>|           |                  |
             |   |           |       +-----------+                  |
             |   |           |                                      |
             |   |           |       +-----------+                  |
             |   |           |       | Service 2 |                  |
             |   |           +------>|           |                  |
             |   |                   +-----------+                  |
             |   +--------------------------------------------------+
             |
             |   +- Environment B ----------------------------------+
             |   | +------------+                                   |
             |   | | Prometheus |                                   |
             +---->|            |                                   |
                 | +------------+    +-----------+                  |
                 |           |       | Service 1 |                  |
                 |           +------>|           |                  |
                 |           |       +-----------+                  |
                 |           |                                      |
                 |           |       +-----------+                  |
                 |           |       | Service 2 |                  |
                 |           +------>|           |                  |
                 |                   +-----------+                  |
                 +--------------------------------------------------+

```

# Answer
> 3 votes
I don't think there's an ideal solution at all. However, there may be a preferred one, depending on circumstances.

There're several factors which I think one has to consider:

* How cheap/expensinve a dedicated instance is?
* What are security concerns/considerations?
* What is expected volume of data from every environment?

This table can resume them :

```
 |    Factor \ |  Solution I  | Solution II  | Solution III |
 |____________\|______________|______________|______________|
 | Data Volume |     Low      |     High     |     High     |
 |-------------+--------------+--------------+--------------|
 |    Security |      No      |     Yes      |     Yes      |
 |-------------+--------------+--------------+--------------|
 |Instance Cost|     High     |     Low      |    Medium    |
 |_____________|______________|______________|______________|

```

Based on the analysis of one's particular circumstances a preferred solution may be selected.

I would love to know what others think about the subject, though

# Answer
> 1 votes
You covered the obvious security issue, use HTTPS for all your connections.

While I won't try to tell you how to implement your environment, I will share with you the design in which I am rolling out at this time ...

Grafana with Prometheus Federated cluster consuming data from Prometheus Servers deployed in each product. Thus, my data storage needs for these 'child' clusters is minimal and no need to worry about redundancy. My Federated Cluster will be storing data in a PostgreSQL server (agents available to handle the Prometheus TSDB data are available). I could then replicate this PostgreSQL server across regions for DR with a Grafana Interface per region for redundancy. I have chosen the PostgreSQL Server option over disk volumes for ease of replication.

As you have realized, there are several options on how to perform this task.

Background: My use case includes multiple stand-alone SaaS products as options within a bundled solution.

# Answer
> 0 votes
With the use case III, have you looked into Thanos or Cortex ? Both of these will provide:-

* Horizontal scalability (with deduplication)
* Support for secure communication with the central Thanos querier.
* Long term data storage support in S3 buckets and such, if required.
* Single place to manage all your Prometheus instances rather than adding individually into Grafana.

---
Tags: security, monitoring, prometheus, grafana
---


title: Predictive autoscaling on K8s

text:
# Question
Title: Predictive autoscaling on K8s
I am learning about using Machine Learning for provisioning proactive resources on the Kubernetes and I have found some of the interesting articles about it on the internet(for instance for Thoth or Microscaler). I have also read about predictive autoscaling in AWS and GCP however only for VM's, not to use in the K8s clusters.

https://docs.aws.amazon.com/autoscaling/plans/userguide/how-it-works.html https://cloud.google.com/compute/docs/autoscaler/predictive-autoscaling#how\_predictive\_autoscaling\_works

However I would like to understand more in-depth about the above solutions - how do they work internally and what is the possibility of moving that solutions to the K8s. Do You know where can I find informations about it? Or if that is even possible to find those information or they are hidden internally in those companies? Do You have any other scientific articles or solutions which are now available?

Thanks

# Answer
These are presumably proprietary tools for EC2 and GCE respectively. The need for predictive VM autoscaling is pretty strong for Virtual Machines since provisioning time can be measured in the 10's of minutes depending on the configuration.

Conversely, application spin-up time in Kubernetes is typically measured in seconds. Predictive scaling for the Kubernetes HPA might have limited practical benefit.

Now using cloud provider predictive autoscaling to scale out the VMs running a Kubernetes cluster based upon ML could be pretty interesting.
> 1 votes

---
Tags: kubernetes, amazon-web-services, gcp
---


# Question
Title: How to move EC2 instance to another ECS cluster
Assume I have two clusters `devcluster` and `qacluster` on ECS. How to remove an EC2 instance *(ecs-optimized amazon-linux)* from `qacluster` and add to `devcluster`?

# Answer
You need to follow these steps:

To make sure it uses correct cluster, check logs - `docker logs ecs-agent`. It should have a message like:

```
[INFO] Registration completed successfully. I am running as 'arn:aws:ecs:eu-west-1:ACCOUNT_ID:container-instance/CLUSTER_ID' in cluster 'devcluster'

```
> 6 votes

# Answer
The way a cluster becomes aware of the EC2 instances associated to it, is a configuration file used by the ECS agent. You can modify this file, located at `/etc/ecs/ecs.config`, and name a different cluster. Then restart the ECS agent. This will effectively "move" the EC2 instance to the other cluster.

More information about `ecs.config` in the documentation - http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-config.html

After the modification, it is possible to verify by sending an HTTP request to the ECS Agent metadata service and see the new cluster displayed. http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-update.html

The ECS cluster itself also needs to be notified. Can use the Register/Deregister API calls for this, for example using the AWS CLI:

More housekeeping might be required on the ECS agent itself. Like removing cached checkpoint files, etc. https://github.com/aws/amazon-ecs-agent/issues/139#issuecomment-123061463
> 2 votes

# Answer
Updated for ECS optimized AMI image as of May 2021. The sudo start command is not present and the checkpoint file /var/lib/ecs/data/ecs\_agent\_data.json changed to agent.db.

1. Edit `/etc/ecs/ecs.config` as previously described (add ECS\_CLUSTER=devcluster).
2. Rename the agent state file with `mv /var/lib/ecs/data/agent.db /var/lib/ecs/data/agent.db.old`
3. Restart the ECS agent using `sudo systemctl restart ecs`

Check the status of the agent with `systemctl status ecs`, you want it to display *Active: active (running)*.

This was a bit frustrating to figure out because the Amazon online documentation I found still displays the old information. Couldn't have done it without devops!
> 0 votes

---
Tags: amazon-ecs
---


# Question
Title: How to find the total ips allowed by an AWS subnet?
Is there a way to find how many total ip addresses that an AWS subnet can hold?

If my subnet CIDR is a.b.c.d/n, then the total ips allowed in this subnet - are they decided by a, b, c & d as well?

# Answer
> 1 votes
AWS docs show what the reserved IP's in each subnet are: https://docs.aws.amazon.com/vpc/latest/userguide/VPC\_Subnets.html

The gist of it is, the first 4 IP's and the last IP of the subnet are off limits if your subnet starts with the same IP as the VPC. It's probably a good rule of thumb to avoid assuming these IPs are going to be available in any subnet even though potentially the second subnet address reserved for DNS could be available for consumption in another subnet. A better rule of thumb to avoid under-sizing subnets in AWS. Like, never plan on using a subnet smaller than /24. You need them for auto-scaling groups, NAT gateways, VPN's, and networking hooks into other apps like DynamoDB or Lambda.

---
Tags: amazon-web-services
---


# Question
Title: Resources don't seem to show up in Terraform state after import
I'm in the middle of importing a bunch of AWS resources into Terraform by doing the following:

* put a stub resource into a `.tf` file
* `terraform import` the resource (using the same address)
* copy the relevant config out of the output of `terraform show -no-color` into a new `.tf` file

This has worked before, but now I'm doing a bunch of security groups and it doesn't seem to be working. I'll do an import:

```
terraform import aws_security_group.rds-launch-wizard-2 sg-XXXXXXXX

```

This succeeds. I would expect `aws_security_group.rds-launch-wizard-2` to show up in the output of `terraform state list` or `terraform show`, but it doesn't. I feel like maybe I'm missing something easy? Thanks for any help.

# Answer
I figured it out. The stub resource has to have a provider that is configured with the proper AWS region. All these security groups were in us-west-2. You need this:

```
resource "aws_security_group" "rds-launch-wizard-2" { provider = aws.us_west_2 }

```

instead of this:

```
resource "aws_security_group" "rds-launch-wizard-2" {}

```

This assumes you have a provider already configured thus:

```
provider "aws" {
  region  = "us-west-2"
  alias   = "us_west_2"
}

```
> 2 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: Do any services other than EC2s consume ip addresses in AWS?
**SCENARIO:**

I am running out of free ips in my AWS subnets. I would like to talk to my infra team and request them to create new subnets. But they are asking me list of all ips I have consumed so far and to show how each ip is used

1. TOTAL IP CAPACITY:

So I see there are a total of 384 ips available across 8 subnets in 2 VPCs.
2. FREE IPs:

Now I can see from the VPC page that there are a total of 126 free ips available.
3. TOTAL IPs CONSUMED:

So this means 384-126 = 258 ips have been consumed in this account.
4. Total # of EC2s:

I also got a list of all EC2s from the "Tag Editor" page on AWS Console. That list shows a total of 72 EC2s.
5. DOESN'T ADD UP

So 258 ips have been consumed as per #3 above

But there are only 72 EC2s created totally as per #4 above

So who consumed the remaining 258-72 = 186 ips?

Do any other services other than EC2 consume ips?

Where is the disconnect?

# Answer
> 3 votes
There are many AWS services which requires a network interface such as `AWS load balancers`, `Lambda`, `NAT gateway`, `VPC endpoints` etc. For each AWS load balancer, it may use up to 2 IPs and 1 IP for 1 lambda function. I would recommend to check the `network interface` section under EC2 service which will show you all the IPs which are currently being used and attached to EC2, Lambda, load balancer or some other service.

**NOTE:** For each subnet, 5 IP addresses will be reserved by AWS. So, if you have a subnet with /28 subnet mask (16 available IP addresses ), in AWS, you will get only 11 IPs which can be used.

---
Tags: amazon-web-services, amazon-ec2
---


# Question
Title: What are the different types of work in a development team?
I'm currently on my annual re-read of The Phoenix Project and we're going through the different types of work that an Ops Team face. These are:

* Project Work
* Changes
* Internal Projects
* Unplanned Work

This time around I'm not convinced that these types apply directly to development teams (as opposed to Ops work).

We obviously have Project Work (pretty much everything in our backlogs). Unplanned Work is a no-brainer for anyone who's ever supported an incident...

But do we have an equivalent of "Changes" or "Internal Projects"? Internal Projects kind of correlates to Tech Debt...

Have the authors of The Phoenix Project (or other highly regarded figure in the DevOps community) compared the different types of work a development team may face and commented on whether it is different?

# Answer
> 1 votes
Nice question!

Possibly, Changes are further strategic product development beyond current work which is more than required by technical debt? For example, introducing a new framework, or nice abstraction layer. (which could break a lot though)

Internal projects could be custom helper tools to automate something invisible in the backlog. Sometimes you have also to fill in time sheets.

By the way Unplanned work could be also a request to support sales or HR or some unexpected meeting or all hands social skills training event.

There is anyway a sequel to "Phoenix Project" called "Unicorn Project" by same author, focusing on the Dev part of the equation.

I haven't read it myself yet but hope you might find the answer there.

# Answer
> 0 votes
I would say that changes come in order once your project work is done. You build it, you maintain it. In this phase, changes can be requested for small features, compliance or regulatory reasons for example.

Edit: misread dev team for devops team, made changes accordingly.

---
Tags: software-engineering, capacity-planning, the-phoenix-project
---


# Question
Title: Jenkins: Dynamically git clone only the files required for a particular job from a monorepo
We have a huge ~20GB monorepo. For a particular Jenkins job, we do not need to clone the entire repo but only need a few sub-directories. The solution for this is of course git sparse checkout. However this is quiet cumbersome to set up and maintain. You probably have to run the job a few times to make sure you did not miss any directory. Also, in the future if you added a dependency (inside another directory) that was not in the sparse checkout list, you have to remember to add it back.

What if we can improve this process?

1. We should be able to identify all the scripts that a particular Jenkins job needs by looking into all the files that the OS has opened inside that job's working directory.
2. We already have tools like `lsof`, `sysdig`, `opensnoop` which can show us all files that are open inside a directory.
3. On the first ever run of the Jenkins job, we clone the entire repo.
4. Then we start `lsof +D /job/workspace` and monitor which files got opened.
5. At the end of the job we create a data structure like `{JOB_NAME: [list, of, files]}` and save it somewhere.
6. Rerun the job ~5 times to ensure data correctness.
7. On all subsequent runs, the git clone command will parse above file and only `sparse-checkout` the required files.
8. If in the future, the job fails due to a `FileNotFound` error, we can infer that a new dependency was added, add that new dependency and re-run the job.

Wanted to run the viability of this idea with the DevOps Community.

# Answer
> 1 votes
Assuming I'm understanding you correctly, there's a much simpler way to do this. If you know the path to the file(s) that you need within the git repo, you can use `git archive`. This fetches a single file directly from a remote repository, so it always grabs the latest version. I've used this a few times from within a Jenkins Pipeline script. For instance:

```
def cmd = "git archive --remote='${repoUrl}' '${ref}' '${fileName}' | tar -xO"
sh(cmd)

```

In this example, `repoUrl` is the remote URL of the git repo, `$ref` is the ref (tag or branch) to fetch, and `fileName` is the file to fetch from the repository. If you need more than one file, you can list each one as a separate argument to `git archive`.

Note that if you're running this from a Pipeline script, you may need to wrap it in `withCredentials` in order to set up the proper SSH keys or username/password credentials for your git repo.

---
Tags: jenkins, git
---


# Question
Title: ansible playbook cannot apply, giving host unreachable error
I'm completely new to ansible and I'm having a hard time with it. I have an inventory.yml file which is as the following:

**EDIT: changed my inventory.yml now looks as follows** inventory.yml:

```
[master]
master1 ansible_host=192.168.56.2 ansible_port=2222 ansible_user=deploy ansible_ssh_pass=vagrant ansible_become_pass=vagrant apt_key_master=https://packages.cloud.google.com/apt/doc/apt-key.gpg

[worker]
worker1 ansible_host=192.168.56.3 ansible_port=2200 ansible_user=deploy ansible_ssh_pass=vagrant ansible_become_pass=vagrant apt_key_worker=https://packages.cloud.google.com/apt/doc/apt-key.gpg

```

also I have a playbook called kube-repos.yml which is as follows:

**EDIT: changed kube-repos.yml now looks as follows:** kube-repos.yml:

```
- hosts: all
  become: yes
  tasks:

  - name: add Kubernetes apt-key
    apt_key:
      url: "{{ apt_key_master }}"
      state: present

  - name: add Kubernetes' APT repository
    apt_repository:
      repo: deb http://apt.kubernetes.io/ kubernetes-xenial main
      state: present

```

when I apply my ansible playbook using `ansible-playbook kube-repos.yml` I get the following error:

```
fatal: [master1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey).", "unreachable": true}
fatal: [worker1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: Permission denied (publickey).", "unreachable": true}

```

I see my problem is getting proper authentication ansible to ssh into the host vm's. I am able to do `vagrant ssh master` and `vagrant ssh worker` and get into my vm's perfectly fine but when I apply the playbook and if I use the command `ansible all -m ping` I get the UNREACHABLE error. I am trying to setup a simple kubernetes cluster consisting of one master and one worker node.

I am at a loss now because I have read the ansible documentation but still cannot figure out how to correct this error. So I feel like I have no choice but to ask on here. I hope someone is kind enough to help me out and thanks in advance.

**EDIT:** after the changes suggested by @mikequentel I have the following error:

```
fatal: [worker1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 192.168.56.3 port 2200: Connection refused", "unreachable": true}
fatal: [master1]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh: ssh: connect to host 192.168.56.2 port 2222: Connection refused", "unreachable": true}

```

# Answer
Do the following for each Vagrant machine you want Ansible to work with...in this example, using the machine `master1`:

1. Do `vagrant ssh-config > vagrant-ssh`
2. Enter content from `vagrant-ssh` into your `~/.ssh/config`, modifying it as needed, like this example for some machine named `master1`:

```
Host master1
  HostName 127.0.0.1
  User vagrant
  Port 2222
  UserKnownHostsFile /dev/null
  StrictHostKeyChecking no
  PasswordAuthentication no
  IdentityFile /home/myname/.vagrant/machines/default/virtualbox/private_key
  IdentitiesOnly yes
  LogLevel FATAL

```
3. Then, in your hosts (inventory) file, include the login information needed by Ansible, for each machine:

```
master1 ansible_host=master1 ansible_port=2222 ansible_ssh_user=vagrant ansible_ssh_pass=vagrant ansible_become_pass=vagrant

```

or also use IP address for `ansible_host`:

```
master1 ansible_host=192.168.56.2 ansible_port=2222 ansible_ssh_user=vagrant ansible_ssh_pass=vagrant ansible_become_pass=vagrant

```
> 1 votes

---
Tags: kubernetes, ansible-inventory, ssh
---


# Question
Title: How do I create a LAMP server via EC2 Image Builder?
I am trying to set up a resilient way to deploy our monolithic web app on an AWS EC2. My current setup uses Ansible to "manually" install the components for a LAMP server on the latest Amazon-Managed Ubuntu AMI.

I recently discovered **AWS EC2 Image Builder**, which seems like it could be used to create an AMI with Apache, MySQL (client), and PHP.

In my AWS Console, I went through these steps:

1. EC2 Image Builder \>
2. Image pipelines \>
3. Create image pipeline \>
4. Create Recipe \>
5. Image Type: AMI \>
6. Source Image: \[Amazon Linux | Ubuntu | CentOS | Redhat | Suse \] (I tried each one)

After the above 6 steps, when trying to add an (Amazon Managed) Component, I see PHP 7.1 ~ 7.3.. can be added. But I do NOT see what I would think are other basic components: Apache, Ngnix, MySQL..

How do I add Apache2 + MySQL (client) + PHP to an Amazon managed Ubuntu box via EC2 Image Builder?

# Answer
> 1 votes
You need to create build components and define a custom installation process.

Click the 'Create Build Component' button in the 'Components' section of the 'Recipe' page. Scroll down to 'Definition' and select 'Use example' to see how it works.

---
Tags: amazon-ec2, amazon-ami
---


# Question
Title: OIDC token invalid when put into kubeconfig
I can connect to the Kubernetes API from a container running inside the cluster with an OIDC token like so:

```
kubectl —token $token get pod

```

I would now like to prepare a kubeconfig such that other programs that don’t support `--token` can connect as well. I am trying this in the specific context of `docker buildx create --driver kubernetes ...; docker buildx build .` where the 2nd part needs to create a `deployment`. Here's how I have proceeded:

```
tmp=$(mktemp -p $PWD)
kubectl config --kubeconfig $tmp set-cluster cluster \
  --server https://$KUBERNETES_SERVICE_HOST:$KUBERNETES_SERVICE_PORT \
  --certificate-authority /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
kubectl config --kubeconfig $tmp set-credentials user --auth-provider oidc \
  --auth-provider-arg idp-issuer-url=... \
  --auth-provider-arg client-id=... \
  --auth-provider-arg id-token=$token
kubectl config --kubeconfig $tmp set-context context --cluster cluster --user user
kubectl config --kubeconfig $tmp use-context context
export KUBECONFIG=$tmp
kubectl get pod

```

`kubectl get pod` now fails with "Unable to connect to the server: No valid id-token, and cannot refresh without refresh-token". Why does it deem `$token` invalid when it has worked just before with `kubectl get pod --token` (and still does)? How can create a kubeconfig "envelope" around `$token` such that `kubectl get pod` will accept it and won't fail? (I've not specified `refresh-token` because it shouldn't be needed in the case of a valid token and because I don't have it readily available inside the container.)

# Answer
I have come to the (preliminary) conclusion that this cannot be done.

First, I had set the wrong `idp-issuer-url`. I had selected the IDP that normally serves during OIDC authentication to the Kubernetes cluster. But `$token` (or `$CI_JOB_JWT` in GitLab CI/CD parlance) was not generated there but by GitLab instead. Inspection e.g. at jwt.io indicates `"iss": "kubernetes/serviceaccount"`, so `--auth-provider-arg idp-issuer-url=kubernetes/serviceaccount` should be a better choice.

But this inspection also reveals that the JWT does carry a claim `aud`, hence no client-id. `kubectl` in particular (and perhaps OIDC in general) apparently insist on a non-empty client-id (otherwise: "error: Must provide client-id"). So I guess what happens is that `kubectl` checks the issuer (signer) and client-id for `$token` (claims `iss` and `aud` in its JWT payload) before it will use it. The check for client-id must fail, at which point `kubectl` tries a token refresh and that fails again.

If that diagnosis is correct this cannot be done because `kubectl` insists on a client-id whereas GitLab does not provide one in `$token`. I guess the reason for the discrepancy is that GitLab's is a general JWT token, whereas `kubectl` (more specifically, its OIDC autentication provider) expects a OIDC id tokens in particular (unless it is invoked with `--token`, which apparently bypasses kubeconfig and hence the OIDC authentication provider).
> 1 votes

---
Tags: docker, kubernetes, security, kubectl
---


# Question
Title: How to setup custom authentication and authorization in Istio/K8?
As I am new to istio world. I am having difficulty understanding how that technology's auth system work.

Let's say, I have a project that has 8 pods(services). I understand that authentication and authorization are covered in Istio Gateway using jwt. So that every request is verified. But users with different roles ex: \[teacher, student, staff\] need to have an endpoint to signup/signin/logout with username and password. Is there a way that Istio Gateway handles user creation saving it to db and generating jwt? Or do I have to make another Pod(service) dedicated to Auth and generate jwt from this pod? If so, how can I still set up jwt verification right from the gateway?

Thank you for taking time and reading this)

# Answer
> 2 votes
I am not an expert (yet), but from what I understand, Istio allows you to check that someone (the principle of the call) is already authenticated/authorized.

But that actual authentication needs to have been done previously. You can use just about any Identity Provider to get the authentication done. Once done, the JWT from the authentication is added as a header to the call that goes to your cluster via Istio.

The `RequestAuthentication` and `AuthorizationPolicy` Istio resources can be setup to look at the JWT and validate that it is appropriately signed and that the needed claims are present. (And fail the call with a 401 or 403 if they are not.)

---
Tags: docker, kubernetes, istio
---


# Question
Title: unable to access cluster after installing flannel
I've got a cluster running with 1 master + 2 workers and using flannel as CNI.

kubeadm init was called with apiserver address and 10.244.0.0 for flannel's and nodes are joined.

```
kubeadm init --apiserver-advertise-address=192.168.56.3 --pod-network-cidr=10.244.0.0/16

```

> NAME READY STATUS RESTARTS
> coredns-f9fd979d6-d6jkd 1/1 Running 0
> coredns-f9fd979d6-fpzct 1/1 Running 0
> etcd-kubemaster 1/1 Running 0
> kube-apiserver-kubemaster 1/1 Running 0
> kube-controller-manager-kubemaster 1/1 Running 0
> kube-flannel-ds-5gd2w 1/1 Running 0
> kube-flannel-ds-b4qzz 1/1 Running 0
> kube-flannel-ds-m56bd 1/1 Running 0
> kube-proxy-5hw6x 1/1 Running 0
> kube-proxy-9wnpd 1/1 Running 0
> kube-proxy-ffrmh 1/1 Running 0
> kube-scheduler-kubemaster 1/1 Running 0

iptables was cleared, everything allowed and net.bridge.bridge-nf-call-iptables=1

But the cluster cant be reached.

> http://192.168.56.3:32001/

times out.

```
kubectl config view

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://192.168.56.3:6443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: kubernetes-admin
  name: kubernetes-admin@kubernetes
current-context: kubernetes-admin@kubernetes
kind: Config
preferences: {}
users:
- name: kubernetes-admin
  user:
    client-certificate-data: REDACTED
    client-key-data: REDACTED

```

What to do to get access to the cluster's services?

# Answer
> 1 votes
It turns out that flannel itself must be initialized with the k8s interface.

```
kubectl get pod --all-namespaces | grep flannel

kube-system   kube-flannel-ds-2676q                 1/1     Running   0

kubectl logs kube-flannel-ds-5gd2w -n kube-system -f

I0603 05:56:36.014958       1 main.go:520] Determining IP address of default interface
I0603 05:56:36.015399       1 main.go:533] Using interface with name enp0s3 and address 10.0.2.15

```

enp0s3 is incorrect interface.

delete daemonset:

```
kubectl delete ds kube-flannel-ds -n kube-system

```

delete flannel.1 link:

```
ip link delete flannel.1

```

download kube-flannel

curl https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml -O

Add --iface arg to the kube-flannel with the interface that k8s is advertised on

* --iface=enp0s8

Apply it and k8s becomes accessible:

```
kubectl apply -f kube-flannel.yml

kubectl logs kube-flannel-ds-2676q -n kube-system -f
I0603 07:03:14.430280       1 main.go:533] Using interface with name enp0s8 and address 192.168.56.4

```

deleting the links is not necessary itself, flannel recreates it automatically

"flannel.1" already exists with incompatable configuration: vtep (external) interface: 3 vs 2; recreating device

---
Tags: kubernetes
---


# Question
Title: How to connect two Kubernetes Jobs without using a Service?
I have a deployment that consists on one Job that contains two containers. Due to distribution and economical reasons, I want these two containers to be deployed independently. However, they are still interconnected. This means there should be network communication between both of them.

As far as I know, you can not create a Service over a Job. You need it to be at least a simple Pod. However, this adds another element and modifies the equation: I want to split the Job into two Jobs, not into one Job, one Pod and one Service.

So, with this as context, is there a way to connect to Jobs directly without creating a service? They both are going to be on the same cluster, using the same internal DNS, in different machines.

Basically, something like, from Job1, connect to *job1.job1namespace:8080*.

# Answer
> 1 votes
You might be able (assuming you're running at least Kubernetes 1.20) to leverage a beta feature for the `Pod` spec that allows a `Pod` FQDN to be set in DNS.

I am uncertain if this extends to `Job` resources in the minimum version (v1.20), but because it is part of the `Pod` specification, I would assume so between `Job`s. In this example, assuming `setHostnameAsFQDN: true` is set for that definition, a FQDN like `busybox-1.default-subdomain.my-namespace.svc.cluster-domain.example` (for a `name` metadata field called `busybox` for that spec) would be valid, and should be accessible from a different `Job`.

I was able to test that `Job` resources would accept `setHostnameAsFQDN` in the definition, and it seemed to without issue:

```
apiVersion: batch/v1
kind: Job
metadata:
  name: pi
spec:
  template:
    spec:
      containers:
      - name: pi
        image: perl
        command: ["perl",  "-Mbignum=bpi", "-wle", "print bpi(2000)"]
      restartPolicy: Never
      setHostnameAsFQDN: true
  backoffLimit: 4

```

on Kubernetes v1.21.1.

---
Tags: kubernetes
---


# Question
Title: Difference between Docker container with Debian vs VM with Debian?
What is the difference between running a Docker container with Debian (eg. https://hub.docker.com/\_/debian/) and then running a virtual machine with Debian? And why is it still an advantage if any?

I'm curious to know the details since the concept of Docker is to not include the OS layer and therefore make it lightweight...! But how come there is a docker image with the Debian *operating system* (!) when containers do not run an operating system?! How is it that I can still have all the Linux OS features in a Linux container (bash, filesystem, file structure, package manager, cron jobs etc etc) if I don't have that OS?!

Note: I'm using *Docker Desktop for Windows*.

# Answer
With an VM you have a complete machine with all the overhead like filesystem, bash.... With Docker, you use all of that from the "host". Only what is really different is sort of duplicated.

makes sense?

here is an picture: docker vs VM

To make it a bit clearer:

In the docker container are only the extra bins/libs needed for that particular application.

Everything else is provided by the host-os.

That makes the up and down of the app much faster and the overall systemload is much less, since you only have one os running.

With an VM, on the other hand, you have a whole os running. Most of that is not needed.
> 2 votes

---
Tags: docker, containers, linux
---


# Question
Title: How do I modify the environment variables of a Kubernetes cluster?
I created my cluster through `kubeadm`. Before the creation of the cluster I set an environment variable `~/.bashsrc`. The variable I set is `NO_PROXY`. When the cluster is created this variable gets picked up and is applied to all pods.

I now want to add another entry to the `NO_PROXY` but I know that this update won't be picked up automatically by the cluster.

What is the best way to update the environment variables of a kubernetes cluster?

# Answer
Environment vars at the time of kubeadm init are not deployed to all pods. There are no pods at that time anyway, you have just created a kubemaster.

When you want to put NO\_PROXY into pods/deployments you have to do that the usual way with env on the container.

```
   env:
    - name: NODE_NAME
      valueFrom:
        fieldRef:
          fieldPath: spec.nodeName

```
> 1 votes

---
Tags: kubernetes, kubeadm
---


# Question
Title: What are ways to structure a webhook-triggered job that triggers a human-triggerable job?
I hope it's okay to ask for recommendations here. I'm very new to devops, and am trying to get into the headspace of involved technologies and workflows.

I've created a webhook in my Bitbucket project that posts to `http://JENKINS_URL/generic-webhook-trigger/invoke`. Using techniques described here, the corresponding Pipeline build in Jenkins only acts if the event was the pushing of a git tag.

**I would like this Jenkins Pipeline build to trigger *another* Jenkins build that is triggerable by a human -- what are (good/proper) ways to organize this?** In other words: if `A` is a Pipeline build triggered by a generic webhook, I would like `A` to figure out some variable values and trigger `B`, passing it those variables, and I would like for a human to be able to manually trigger `B` using human-supplied variables.

In my naivete, I imagined `A` being a Pipeline build because they offer the option to use a Pipeline script, which is the only way I'm currently aware of that offers a way to access the Post content parameters. I.e. if `A` was a freestyle build with a shell script, I don't know how I'd access the Post content.

However, when `A` is a Pipeline build there is no `Post-build Actions` section in the Jenkins UI -- i.e. I don't see how a Pipeline build can trigger a subsequent build.

From naivete, again, I envisioned a Pipeline build triggering a freestyle build because the Pipeline build allows accessing generic webhook post content, and the freestyle build is human-triggerable, and can be parameterized so that either the human or the calling Pipeline build can supply those parameters.

(I hope I might also direct some eyeballs to a separate, but related, question about passing parameters to parameterized builds: Why is parameter not propagated to post-build parameterized build?)

# Answer
> 1 votes
Make B as a Pipeline job and give it some build parameters. Humans will be able to trigger this build with whatever parameters they want using the "Build With Parameters" button in the web UI.

Then, in your Pipeline script for A, trigger the build using the built-in build step. This will allow job A to trigger B with whatever parameters are necessary:

```
build(
  job: 'Job B',
  parameters: [
    string(name: 'parameterName1', value: "some parameter value"),
    string(name: 'parameterName2', value: "some parameter value"),
    // etc.
  ],
)

```

---
Tags: jenkins, jenkins-pipeline, pipeline
---


# Question
Title: Scale up pods of k8s deployment across spot and dedicated instances
i need to deploy a k8s deployment to be stretched across two kinds of nodes, dedicated and spot nodes.

The end-goal is to have deployment which has some pods on dedicated nodes, but when scaled uses spot nodes.

**Example**:-

A deployment has 2 pods (min) and is scalable up to 5 pods (max).The deployment should behave such that the first 2 pods are always scheduled on dedicated nodes and the rest 3 pods (5-2) are always deployed on spot nodes.

**Use case**:- To improve cost/reliability of my platform.

* **Cost**:- A prefixed set of pods running at idle vs the max pods running at maximum load gives me an idea of how much my pods are scaling. It also ensures that the end of the day when load subsides,the node counts return back to a fixed predictable count.
* **Reliability**:- I want to reduce the possibility of outages when the spot nodes are taken away.Ensuring that atleast 2 pods are running on a dedicated nodes, give me confidence that my application does not face an outages should all the spot nodes get deleted.(It does happen!)

**What I have tried so far:-**

* **Labels and Node Affinity**:- Node affinity ascertains where the pods will be scheduled. But this approach forces me to select *either* dedicated *or* spot nodes. (not a combination of both)
* **Pod Anti Affinity**:- It works somewhat,it deploy one pod/node till the number of pods = number of dedicated nodes, the starts deployment on spot. But this forces me to limit the size of dedicated nodepool. Also one pod/node is not the use case I am looking for.

**Understanding of the problem:-**

* It seems like a common enough use case where atleast some portion of pods are always running but I have not seen a solution to this anywhere.
* It may go against the k8s philosophy where the app should be unaware/not tightly coupled with the infrastructure below but I am not sure if this is the case here.
* Running a free for all nodes (no affinity),can cause unbalanced deployments, where scaled pods of one deployment, may prevent any pods of another deployment from getting dedicated node resources.

Let me know if this is achievable and if not , how i can reduce the risk of app outage by running on all spot nodes.

# Answer
You can follow below approach:

1. Use Cluster Autoscaler which should scale-in or scale-out nodes as and when required.
2. Create nodegroups with multiple instance types so that you always get a spot node.
3. Use labels and anti-affinity rules to schedule applications on any of these available spot nodes.
> 1 votes

---
Tags: kubernetes, architecture
---


# Question
Title: Liveness Probe through C# code giving error
I am trying to implement liveness probe through C# code (.Net core framework). I simply want to run a curl command inside container like this . Below is the code snippet using kubernetesClient:

```
IList<string> command = new List<string>();
V1Probe livnessconfig = null;
command.Add("curl http://localhost:5001/checkhealth/");
V1ExecAction execommand = new V1ExecAction(command);
livnessconfig = new V1Probe { Exec = execommand, InitialDelaySeconds = 10, PeriodSeconds = 10, TimeoutSeconds = 5, FailureThreshold = 3 };

```

**But getting this error in pod description:**

> Liveness probe errored: rpc error: code = Unknown desc = failed to exec in container: failed to start exec "a80d33b5b2046b8e606ed622da7085013a725": OCI runtime exec failed: exec failed: container\_linux.go:380: starting container process caused: exec: "curl http://localhost:5001/checkhealth/": stat curl http://localhost:5001/checkhealth/

Can someone let me know whether this is correct way to provide command to V1ExecAction. Its metadata implementation in K8s.Models showing that V1ExecAction take command in List:

```
    #region Assembly KubernetesClient, Version=3.0.0.0, Culture=neutral, PublicKeyToken=a0f90e8c9af122d

using Newtonsoft.Json;
using System.Collections.Generic;

namespace k8s.Models
{
    //
    // Summary:
    //     ExecAction describes a "run in container" action.
    public class V1ExecAction
    {
        //
        // Summary:
        //     Initializes a new instance of the V1ExecAction class.
        public V1ExecAction();
        //
        // Summary:
        //     Initializes a new instance of the V1ExecAction class.
        //
        // Parameters:
        //   command:
        //     Command is the command line to execute inside the container, the working directory
        //     for the command is root ('/') in the container's filesystem. The command is simply
        //     exec'd, it is not run inside a shell, so traditional shell instructions ('|',
        //     etc) won't work. To use a shell, you need to explicitly call out to that shell.
        //     Exit status of 0 is treated as live/healthy and non-zero is unhealthy.
        public V1ExecAction(IList<string> command = null);

        //
        // Summary:
        //     Gets or sets command is the command line to execute inside the container, the
        //     working directory for the command is root ('/') in the container's filesystem.
        //     The command is simply exec'd, it is not run inside a shell, so traditional shell
        //     instructions ('|', etc) won't work. To use a shell, you need to explicitly call

       //out to that shell. Exit status of 0 is treated as live/healthy and non-zero is unhealthy.
         [JsonProperty(PropertyName = "command")]
      public IList<string> Command { get; set; }
  } }

```

# Answer
> 1 votes
You need to provide commands in list. Something like below:

```
V1ExecAction execommand = new V1ExecAction("curl", "http://localhost:5001/checkhealth/");

```

Since you already an endpoint for healthcheck, you should use `V1HTTPGetAction` instead of `V1ExecAction`. In your case, `V1ExecAction` will always give exit status as `0` and thus liveliness check won't be correct if your app throws any errors or down.

---
Tags: kubernetes, dotnet
---


# Question
Title: What is the utility of an inhouse DevOps if cloud providers already provide professional services?
I've seen this in past roles when I was working a devops role as an employee.

What should be the purpose of an inhouse devops in a company that already pays for something like AWS Professional Services, in other words, the cloud provider has teams of engineers who are deployed on-prem to solve business needs, put together an infrastructure, and at the same time promote various AWS services to the client.

Where does the inhouse devops fit in this picture?

I remember some years ago I was in a remote meeting, solved a task, only to be told by my manager *"okay, we'll give this to our consultant anyway"*. Not only was the remark awkward because the entire team was in the meeting, but I also don't see what the point of doing the task was in the first place if someone else would've been assigned to do it.

Does it make sense to take on a devops role if it's not for a cloud provider?

# Answer
> 2 votes
Some good reasons could be:

* The salary of an inhouse devops is pretty sure less expensive as the consultants.
* The consultants are only assigned during a limited period, to temporary add (wo)manpower, e.g. while putting together the infrastructure.
* The consultants come with specific expertise that the inhouse devops do not (yet) have, whereas part of the job of the consultants is to perform knowledge transfer and/or on the job training for the inhouse devops.
* The consultants are there to do the boring parts of the work, whereas the inhouse devops are assigned to the strategic / long term tasks (so that when the consultants leave, the know how doesn’t leave the company also.

BTW, IMO the best consultants are those who:

* Say their jobdescription is like “*My job is to make sure that you don’t need me anymore ...*”.
* Never touch the keyboard, and instead always coach / help inhouse devops to have them complete any task (it seems so easy when you see somebody else complete some task, but it is way more difficult to complete it yourself).

# Answer
> 2 votes
Coming from the DevOps consulting background, on top of what @pierre.vriens has mentioned,

1. Consultants don't have any skin in the game and should not be in charge of leading your DevOps strategy in the long run. You need to have the in-house expertise to lead major initiatives. Do high-value work yourself and hire consultants to deliver well-scoped and solved problems.
2. My team provided support for Kubernetes and Azure-related DevOps services. Most of the time, what we built was complicated, and our customers were not ready to maintain or support what was delivered. We spent a good amount of time doing knowledge transfers, but it quickly became overwhelming without a good in-house team.
3. I would not recommend hiring DevOps consultants to coach only. I have done coaching and hands-off engagements and I believe that the customers did not get the full worth of their money. You should embed consultants in your team and train your team by doing stuff side-by-side. Part coaching and part delivery of the actual work.
4. In most cases, cloud vendors work with existing consultants in their network and rebrand them as "their" team. You still need a good amount of technical knowledge and leadership to get the most out of any consultant.

# Answer
> 2 votes
Ownership.

It's fine to pay a second party, or even a third party to provide the support that you need but at the end of the day there is a large distance between not owning the hardware and not owning the human effort.

Developer years of effort that are owned by a company are valuable. It exists in the human space and it manifests in knowledge transfer and familiarity with the product. Effectively, engineers are an investment.

There's an old story about an engineer turned consultant who gets called into an ex-employer to fix a problem. After 30 seconds and a rubber mallet the ex-employer received a bill for an exorbitant amount and promptly asks what the cost was. The engineer in this case points to the decades of experience that enabled (him/her/them) to fix the problem in 30 seconds.

On top of this, the history of enterprise software has made many heads in the software space paranoid that prices can increase after being locked in. This is why there are such big pushes for multi-cloud, or cloud-agnostic infrastructures. Avoiding your engineers being owned by your cloud provider is a good way to avoid that, while on the other side of the coin cloud providers have solid reasons to want to lock you in. They buy the hardware after all. They do the provisioning.

On top of all this, you can guarantee that your inhouse devops only works on your product, and won't leave (without notice-- I hope), and won't be working on other products.

I cannot comment on the specific case mentioned in the original question, as I don't know the nature of the relationship between your company and the consultant.

---
Tags: culture, business
---


# Question
Title: What is it called to deploy major changes for limited number of customers first?
Is there a common name for this deployment strategy which is to avoid deploying major updates across all the customers straight away but rather limit the initial number to see their feedback, find any bugs that slipped through testing thus minimizing the impact (and avoiding overwhelming customer service with support tickets), and only then gradually roll the update out for all?

# Answer
> 1 votes
**Canary release/deployment** is the term you are looking for. Usually in a canary deployment, a small percentage (10% for example) of your infrastructure is updated with the latest revisions of your codebase. That percentage is then tested with various automated and manual testing methods. Once testing is complete, the remaining 90% is updated with the same version. If testing fails, then the initial 10% is either rolled back or re-provisioned with the original (pre-updated) version.

It should be noted, that a strategy like this requires that your application have some type of feature flagging or backwards compatibility in place to allow multiple versions to be running simultaneously. Otherwise, when a rollback occurs, a portion of your infra, users, etc. may be left in a broken state.

---
Tags: deployment, testing, scalability
---


# Question
Title: Use a Kustomize image transformation on an Argo CronWorkflow
I am developing an ETL pipeline using Argo Workflows, and in order to handle environment related configurations, I am using Kustomize.

Here is the base/cronworkflow.yaml file

```
apiVersion: argoproj.io/v1alpha1
kind: CronWorkflow
metadata:
  name: cronworkflow
spec:
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 3
  successfulJobsHistoryLimit: 3
  startingDeadlineSeconds: 300
  workflowSpec:
    entrypoint: start
    templates:
    - name: start
      container:
        name: my-start-task
        image: app-image
        command:
          - /bin/sh
          - -xec
          - |
            ./entrypoint.sh
        env:
          - name: AWS_ACCESS_KEY_ID
            valueFrom:
              secretKeyRef:
                name: aws-creds
                key: access-key

```

Kustomize doesn't seem to work well when using `patchesStrategicMerge`, because it can only deal (out of the box at least) only with standard k8s kinds. With 3rd party kinds, it will simply overwrite the entire tree structure instead of trying to merge.

I had to use `patchesJson6902` on the yaml file defining my Argo `CronWorkflow`, but now the `images` transformer doesn't seem to work anymore.

Am I missing something? Is there a convenient way to fix the behavior of `patchesStrategicMerge`? Are argo workflows "type definitions" for Kustomize available somewhere?

# Answer
Kustomize offers a way to "hint" where to find image tags.

Add a file called `kustomconfig.yaml`:

```
images:
- path: spec/templates[]/container/image
  kind: WorkflowTemplate
- path: spec/templates[]/script/image
  kind: WorkflowTemplate

```

Add the following to your kustomization.yaml:

```
configurations:
- kustomconfig.yaml

```

You're done. The images transformer will pick up those fields.
> 8 votes

---
Tags: kubernetes, workflow, kustomize
---


# Question
Title: Nginx Ingress Controller - error creating inClusterConfig
## What I'm trying to do

I'm trying to get the Nginx controller up and running on my kubernetes cluster with the NLB version of the config files.

## The problem

The `ingress-nginx-admission-create` and `ingress-nginx-admission-patch` jobs immediately fail and do not create the `ingress-nginx-admission` secret that the deployment needs to mount a cert and key.

## Details

### logs from the `ingress-nginx-admission-create` job

```
W0603 17:59:17.907729       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0603 17:59:17.907803       1 client_config.go:613] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
{"error":"invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable","level":"fatal","msg":"error building kubernetes config","source":"k8s/k8s.go:21","time":"2021-06-03T17:59:17Z"}

```

### logs from the `ingress-nginx-admission-patch` job

```
0603 17:59:18.990027       1 client_config.go:608] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
W0603 17:59:18.990093      1 client_config.go:613] error creating inClusterConfig, falling back to default config: open /var/run/secrets/kubernetes.io/serviceaccount/token: no such file or directory
{"error":"invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable","level":"fatal","msg":"error building kubernetes config","source":"k8s/k8s.go:21","time":"2021-06-03T17:59:18Z"}

```

Essentially the same as the `create` job.

### describing the deployment's pod

```
Events:
  Type     Reason       Age                   From               Message
  ----     ------       ----                  ----               -------
  Normal   Scheduled    4m56s                 default-scheduler  Successfully assigned ingress-nginx/ingress-nginx-controller-59b98554fb-8bp6t to ip-172-31-47-215.us-west-2.compute.internal
  Warning  FailedMount  47s (x10 over 4m57s)  kubelet            MountVolume.SetUp failed for volume "webhook-cert" : secret "ingress-nginx-admission" not found
  Warning  FailedMount  38s (x2 over 2m54s)   kubelet            Unable to attach or mount volumes: unmounted volumes=[webhook-cert], unattached volumes=[webhook-cert]: timed out waiting for the condition

```

### potentially useful information

I saw on other sources that I should try deleting the ValidatingWebhookConfiguration and recreate the jobs and deployment--that didn't work and I got the same logs/results. I tried creating a secret with empty secrets prior to the jobs running but that resulted in basically the same thing. The deployment was actually able to get going (in a way) but it didn't have the certificate or key from the jobs.

## My question

Why aren't the jobs creating the secret? Is there something I need to modify?

# Answer
Set `automountServiceAccountToken: true` in your pod specification in both of the jobs and the deployment.

Ref: https://discuss.kubernetes.io/t/nginx-ingress-install-failure/5379/4
> 1 votes

---
Tags: kubernetes, amazon-web-services, nginx, ingress
---


# Question
Title: How can I deal with container "patching" in a strict release process?
I am building a new Container and Kubernetes system for my company. Overall it is going well, but container patching has me a bit confused.

We follow a fairly strict release process. New versions are tested in Dev, then promoted to Test (where they get integration tested). While we sometimes have some other environments, the basic flow then promotes to production. All of this is done using the binaries that are built in Dev (no rebuilding once it leaves Dev).

The version of the application then runs in production until that application needs another version (we have a lot of applications).

Currently (with our VM structure) we will perform patching on the VMs that host the application. While some testing is done for more critical apps, mostly the theory is that the application code is not changed, so it will be OK as security patches are applied to the VM. We occasionally have issues doing this, but it is very very rare.

Now enter container images. They are immutable. You don't apply security patches container images. You build new ones.

This causes problems because, the container image has the base image and the application in one "binary". To update the base image, you have to rebuild the whole thing. That means a new binary, which means more testing is needed. (Even if I get the exact same version of the source, it is not the same binary.)

I see this potentially exploding our deployment and testing pipeline.

**Are there strategies for dealing with this?**

# Answer
I have tackled a very similar problem. We also do not want to rebuild when we patch. To solve this problem, when we do a build, we store our artifact in a repository such as jfrog or nexus. Then when it comes time for patching, we fetch the artifact from the repository and put it on top of an updated base image.

How we use base images is also a very important factor. We rely on stable base image tags such as openjdk11:latest and trust the vendor that any updates are safe since it is still jdk11. This of course means selecting a reliable base image. There are several options here and one worth checking is RedHat UBI images.
> 1 votes

---
Tags: kubernetes, containers, cicd
---


# Question
Title: Enable ingress-nginx Prometheus monitoring without Helm
A few weeks ago, I installed `ingress-nginx` in my Kubernetes cluster. Now, I realize that was a mistake because that did not come with Prometheus monitoring enabled automatically, and even worse, it doesn't use Helm. So now my `ingress-nginx-controller` is all set up and working, but I am unable to change settings using Helm to create the `ServiceMonitor` and scrape metrics from it.

I've been searching everywhere I possibly can for this and I can't find any documentation on how to get Prometheus to monitor `ingress-nginx` without using Helm. I understand I need to make a `ServiceMonitor` resource, but I can't figure out how to make one specifically for `ingress-nginx` because every solution I find online pretty much boils down to: "oh just use Helm bro".

I attempted to create my own `ServiceMonitor` resource like so:

```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: ingress-nginx
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-controller

```

It doesn't work, but I have no clue if this is correct or not because again, absolutely no examples online. It seems like everyone is just expected to use Helm to do all the work for you. Does my `ServiceMonitor` need to be in the `kube-prometheus-stack` namespace, the namespace of the service being monitored, or does it not matter?

* My `ingress-nginx-controller` deployment is in the `ingress-nginx` namespace.
* `kube-prometheus-stack` is installed in the `kube-prometheus-stack` namespace.

Can someone point me in the right direction for getting Prometheus to discover the ingress controller without Helm?

Edit: I moved it to the kube-prometheus-stack namespace, and now Prometheus discovers the service monitor, but it shows 0/8 active targets and I still don't see any nginx stats. Do I need to specify the `/metrics` path in the endpoint? I've tried with and without and it doesn't seem to change anything.

Edit: I created a `ClusterIP` service pointing to the `prometheus` port on my `ingress-nginx-controller` and I am able to port forward that and see the `/metrics` route which shows the Prometheus logs. Pointing my `ServiceMonitor` directly at this new `ClusterIP` service doesn't seem to be doing anything either.

Do I need to point the `ServiceMonitor` directly at the `ingress-nginx-controller` deployment? Or does it need to point at a service? I've tried both and didn't see any other results. I can still see `ingress-nginx` in the Prometheus Service Discovery screen, but 0/8 active targets.

Edit: Here's what I have in my resources now:

NGINX Controller (relevant items only):

```
kind: Deployment
apiVersion: apps/v1
metadata:
  name: ingress-nginx-controller
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-controller
    app.kubernetes.io/component: controller
    app.kubernetes.io/instance: ingress-nginx
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/version: 0.44.0
    helm.sh/chart: ingress-nginx-3.23.0
  annotations:
    prometheus.io/port: '10254'
    prometheus.io/scrape: 'true'
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/component: controller
      app.kubernetes.io/instance: ingress-nginx
      app.kubernetes.io/name: ingress-nginx
  template:
    metadata:
      labels:
        app.kubernetes.io/component: controller
        app.kubernetes.io/instance: ingress-nginx
        app.kubernetes.io/name: ingress-nginx
      annotations:
        prometheus.io/port: '10254'
        prometheus.io/scrape: 'true'
    spec:
      ports:
        - name: http
          containerPort: 80
          protocol: TCP
        - name: https
          containerPort: 443
          protocol: TCP
        - name: webhook
          containerPort: 8443
          protocol: TCP
        - name: prometheus
          containerPort: 10254
          protocol: TCP

```

NGINX metrics service:

```
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx-metrics
  namespace: ingress-nginx
  labels:
    app: ingress-nginx-metrics
spec:
  ports:
    - name: prometheus
      protocol: TCP
      port: 10254
      targetPort: prometheus
  selector:
    app: ingress-nginx-controller
  type: ClusterIP

```

NGINX Service Monitor:

```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
  labels:
    app: ingress-nginx-metrics
    release: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      path: /metrics
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx-metrics

```

I cannot port-forward to the metrics service I created, the connection times out.

This is what I'm seeing in Prometheus:

# Answer
> 2 votes
Your ServiceMonitor seems to be correct with the exception that you have defined no path under spec.endpoints.

You also need to define the port named `prometheus` under the ingress-nginx service or deployment.

ServiceMonitors can be debugged this way:-

1. **Check if the ServiceMonitor created is being read by the Prometheus**:- Look at the /targets URL. (There should be a target in 0/0 state at least and use the old Prometheus UI) If not, that means the ServiceMonitor itself is not being picked up by Prometheus.I suggest looking into the following configuration in your kube-prometheus-stack configuration.

```
    serviceMonitorSelectorNilUsesHelmValues: false  # is true by default
    serviceMonitorSelector: {} #labels which help Prom identify which servicemonitors to pick up
    serviceMonitorNamespaceSelector: {} # If this is empty then all servicemonitors in any namespace will be picked up.

```

The default ServiceMonitor has the Helm metadata attached to it which is used by the Prometheus Operator to filter/choose the ServiceMonitors to monitor. Setting `serviceMonitorSelectorNilUsesHelmValues:false` will ignore any such selection.
2. **If the ServiceMonitor is visible in targets but there are no targets.**:- In this case the issue lies between the ServiceMonitor and the pods it is trying to scrape.Check if the ports you mentioned are accessible and the pods fulfill the selectors mentioned (all `selectors.matchLabels` must match).

My advice would be to start another dummy ServiceMonitor by following this and then modifying the ServiceMonitor one step at a time till it starts monitoring the pods

Edit:-

Moving all servicemonitors to the kube-prometheus-stack namespace is not a good practice. Multiple Helm charts generate their own ServiceMonitor objects which are applied in their own namespace.Enabling the prometheus operator to monitor all namespaces for servicemonitors is a better choice.

For nginx side (ports and label would vary based on your implementations, update the same in the yaml below.):-

1. Add the port in the nginx deployment

```
 - containerPort: 10254
   name: prometheus
   protocol: TCP

```
2. Create a service to map to the monitoring port.

```
apiVersion: v1
kind: Service
metadata:
  labels: [] # labels of your choice
  name: {} # correct name
spec:
  clusterIP: ""
  ports:
    - name: prometheus
      port: 9913
      targetPort: prometheus
  selector: <all labels in your pods should match>
  type: ClusterIP

```
3. Create the ServiceMonitor

```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
    annotations: [] # as required.
    name: ingress-nginx-monitoring
spec:
  endpoints:
  - interval: 30s
    path: /metrics
    port: prometheus
  namespaceSelector:
    matchNames:
    - <namespace where nginx is deployed>
  selector:
    matchLabels: [] # all labels in the given in the service above should match.

```

```

```

# Answer
> 0 votes
The service monitor seems right, only thing is that the service monitor should be in the namespace of the Prometheus stack.

Try the below service monitor:

```
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: ingress-nginx
    release: kube-prometheus-stack
  name: ingress-nginx-metrics
  namespace: kube-prometheus-stack
spec:
  endpoints:
    - interval: 30s
      port: prometheus
  namespaceSelector:
    matchNames:
      - ingress-nginx
  selector:
    matchLabels:
      app: ingress-nginx

```

Also when you do, try port-forwarding the Prometheus

```
 kubectl port-forward svc/<SERVICE_NAME_OF_PROMETHEUS> -n kube-prometheus-stack 9090:9090

```

and check in `Targets` and `Service Discovery` options in Prometheus.  You can see the service monitor in `Service Discovery` and exposed metrics in `Targets`.

---
Tags: kubernetes, prometheus, ingress
---


# Question
Title: How can I add an apt-key without using the hardcoded url in an ansible playbook?
I am completely new to ansible this is my second day working with ansible, What I'm ultimately trying to do is configure a kubernetes cluster which has a master node and one worker node. But my problem is I've been asked to add an apt-key without using a hardcoded url. So I have an ansible playbook called kube-repos.yml which is as follows:

```
- hosts: all
  become: yes
  tasks:

  - name: add Kubernetes apt-key
    apt_key:
      url: "https://packages.cloud.google.com/apt/doc/apt-key.gpg"
      state: present

  - name: add Kubernetes' APT repository
    apt_repository:
      repo: deb http://apt.kubernetes.io/ kubernetes-xenial main
      state: present

```

where I add the apt-key I need to use another method to add the apt-key where it needs to be defined per-host. If anyone knows of other ways of adding an apt-key I would really like to hear them so I can look further into alternate ways of adding an apt-key. Thanks in advanced for any help or points in the right direction.

# Answer
Add the key per host or per group in your inventory. Below example is for an all-in-one yaml inventory. See How to build you inventory for all options to organize your vars (I suggest you read this documentation in depth).

**inventory/example.yml**

```
---
all:
  hosts:
    master1.tld.local:
    master2.tld.local:
    worker1.tld.local:
    worker2.tld.local:
      # This will be used only for this node and will
      # override group var as it is more specific
      # see: https://docs.ansible.com/ansible/latest/user_guide/playbooks_variables.html#understanding-variable-precedence
      my_apt_key: https://some.server.com/custom_node_key.gpg
    worker3.tld.local:

  children:
    master:
      vars:
        # This will be applied to all machines in group
        my_apt_key: https://some.server.com/master_key.gpg
      hosts:
        master1.tld.local:
        master2.tld.local:

    worker:
      vars:
        my_apt_key: https://some.server.com/worker_key.gpg
      hosts:
        worker1.tld.local:
        worker2.tld.local:
        worker3.tld.local:

```

Modify your task in your playbook. As a good practice, I'll add a default value so it does not break if the var is not defined. You can drop this if this is not the behavior you want:

**playbook.yml**

```
  - name: add Kubernetes apt-key
    apt_key:
      url: "{{ my_apt_key | default('https://some.server.com/default.gpg) }}"
      state: present

```

Then enjoy:

```
ansible-playbook -i inventory/example.yml playbook.yml

```
> 1 votes

---
Tags: ansible
---


title: Remove resource from Kustomize base?

text:
# Question
Title: Remove resource from Kustomize base?
I have a kustomize base that I'd like to re-use without editing it. Unfortunately, it creates a namespace I don't want to create. I'd like to simply remove that resource from consideration when compiling the manifests and add a resource for mine since I can't patch a namespace to change the name.

How can this be done?

# Answer
> 5 votes
You can remove specified manifest using 'patchesStrategicMerge' as following:

```
patchesStrategicMerge:
- |-
  apiVersion: v1
  kind: Namespace
  metadata:
    name: unwanted-namespace
  $patch: delete

```

---
Tags: kubernetes, kustomize
---