# Question
Title: How do you securely deploy large number of Kubernetes components in isolation?
Precursor: I am not experienced in the design of large-scale infrastructure deployments for infra applications.

The assumptions for the questions:

* I have read that it is a good practice to host Kubernetes components in isolation from each other on the network as the network provides a layer of security control.
* In a large K8S deployment environment, you may have multiple instances of Kubernetes deployment. Each Kubernetes deployment has components including etcd, kube API server, scheduler, controller manager, etc.

If we consider both points above, then the question are:

**Q1a)** How do you scale the Kubernetes administration/control plane? How do you scale from 1 etcd server to 10 etcd servers for example?

**Q1b)** In a large organization where there are different business units, do you deploy one K8S instance (active/passive) for each business unit, or multiple K8S instances serving the entire organization?

**Q2)** For each deployment method described in part (1b), how do you reconcile multiple instances of Kubernetes to get a master view in order to monitor all the instances of containers running on Kubernetes?

# Answer
> Q1a) How do you scale the Kubernetes administration/control plane? How do you scale from 1 etcd server to 10 etcd servers for example?

You won't. etcd needs to be fast, especially with a growing cluster. 3 members is what you'll want to have. Usually, as a result, you would also deploy 3x kube apiserver, controller-manager & scheduler pods.

> Q1b) In a large organization where there are different business units, do you deploy one K8S instance (active/passive) for each business unit, or multiple K8S instances serving the entire organization?

Large organizations would usually go with several clusters. Don't put all your eggs in the same basket. Make sure you can upgrade some of your clusters, without impacting the others. Allowing end-users to implement their own disaster recovery, managing resources in "sibling" clusters -- kind of active/passive, without designating one or the other as passive, rather planning for each of your clusters to double in size over night, should you need to.

Which doesn't mean they would be small clusters. You could easily have 100s of workers (my current customer's largest cluster has between 350 and 400 nodes, cluster-autoscaler adjusting size base on requested resources). But as much as possible, you want to avoid beasts like those: monitoring or logging stack would consume a lot, requiring larger nodes hosting infra components, operations on those components would become more slow and painful, ... better have two small than one large.

Although here: automation would be critical. your team probalby can't afford to micro-manage 40, 80, 200 clusters. You would have to figure out a way to rollout changes to your clusters with little effort. And consistently. Might involve tools such as Ansible, Terraform, Tekton or ArgoCD.

> Q2) How do you reconcile multiple instances of Kubernetes to get a master view in order to monitor all the instances of containers running on Kubernetes?

I wouldn't. The more you grow, the more metrics/data you would collect, the more complicated it would be to monitor everything from one point, the more likely rules evaluation would be late eventually, maybe misses alerts, ...

Better deploy one Prometheus (or more) per cluster, self-monitoring your cluster. Then, pick two or three "ops" clusters, where you would deploy another Prometheus, making sure the Prometheuses in your other clusters work as expected (can server query, alertmanager is present, alertmanager can relay alerts, ... could be pretty much all based on blackbox exporter).

Configure alertmanagers centralizing alerts into a single place (rocketchat, slack channel, stuff like opsgenie).

Going further, you could look at solutions such as Thanos. You may aggregate metrics from several prometheuses .... although you'll need lots of RAM running thanos components. and some reliable s3 bucket aggregating those metrics into one location. It's definitely not something I would recommend for monitoring purposes, but it can be nice for metrology, making cool grafana dashboards, ...
> 0 votes

---
Tags: kubernetes, containers, security
---


# Question
Title: Convert an existing s3 bucket policy into a terraform-managed policy?
I have a large bucket policy historically maintained by hand.

I'd like to move it into CI/terraform (for the policy maintenance), but I don't want TF to own the bucket itself.

Is there any way to have TF "import" the huge existing policy and generate the TF file for it? https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3\_bucket\_policy

# Answer
> 1 votes
Try this, since the bucket is already there with the policy you can run the below command to get the policy inside the tf state

```
$ terraform import aws_s3_bucket_policy.allow_access_from_another_account my-s3-bucket-name

```

Once the policy is there in the TF, run the below code in `main.tf`

So that it will match with the policy in the state and in the plan it will display if it matches with the state or not.

> //resource "aws\_s3\_bucket" "example" { // bucket = "my-tf-test-bucket" //}
>
> ```
>     resource "aws_s3_bucket_policy" "allow_access_from_another_account" {
>       bucket = "enter the bucket ID from the portal"
>       policy = data.aws_iam_policy_document.allow_access_from_another_account.json
>     }
>
>     data "aws_iam_policy_document" "allow_access_from_another_account" {
>       statement {
>         principals {
>           type        = "AWS"
>           identifiers = ["123456789012"]
>         }
>
>         actions = [
>           "s3:GetObject",
>           "s3:ListBucket",
>         ]
>
>         resources = [
>           "arn for aws bucket",
>         ]
>       }
>     }
>
> ```

---
Tags: terraform, s3
---


# Question
Title: Virtual Machine Monitoring KQL yielding empty results
I have enabled Azure Monitoring in one of our virtual machine. Is been 4 days the virtual machine is on, below is the query:

```
Perf
| where ObjectName == "Memory" and
(CounterName == "Available MBytes Memory" or // the name used in Linux records
CounterName == "Available MBytes") // the name used in Windows records
|  summarize avg(CounterValue) by bin(TimeGenerated, 15min), Computer, _ResourceId // bin is used to set the time grain to 15 minutes
| render timechart

```

The output of Perf itself is empty. The same thing goes for `Update`, it show me zero results.

These are the two extensions I have installed on the VM

Also I have created the Data collection rule to send below data to the log analytics

Any suggestions?

# Answer
> 0 votes
I added Azure Monitor logs to the Performance counters and now it is showing the Perf data

---
Tags: azure, monitoring
---


# Question
Title: Where may I need Docker Compose as Kubernetes won the container orchestration war?
I found Docker Compose just release 1.28 on 2021-1-20 but as Kubernetes won the container orchestration war I am not sure where and why I may still need Docker Compose, under with case Docker Compose is a better option?

I have a web app that consists of a php docker, a nodejs docker, a nginx docker. We host it at aliyun cloud service. In 2018 aliyun used docker swarm so we used docker compose but in 2020 we had switched k8s so I was wondering if it is still worth to learn docker compose ?

--- update ----

I found these articles were helpful

1. Why do I need Compose if I already have Kubernetes? a docker blog published in 2018
2. Docker Swarm vs. Kubernetes for Single-Host Implementations
3. https://stackoverflow.com/questions/47536536/whats-the-difference-between-docker-compose-and-kubernetes an answer is what I do currently:

> If you are networking containers within the same host go for docker compose.
>
> If you are networking containers across multiple hosts go for kubernetes.

And that is what docker compose document said https://docs.docker.com/compose/#common-use-cases

# Answer
Docker compose is still a comparatively easy and light-weight tool to run multiple docker containers with service discovery.

Kubernetes is a metaphorical elephant. Great if you need to smash Roman lines and win the battle, but overkill for a minor skirmish, and definitely more trouble than it's worth if you're trying to fight a small battle in the mountains.

If you have a fairly large app with multiple microservices, then Kubernetes is worth every penny.

But when you're trying to run just a couple of containers, especially if it's only on a single host (i.e. your frontend + backend and maybe a local instance of RabbitMQ), docker-compose is a much simpler/faster way to orchestrate it. Kubernetes is like using a cannon to kill a mosquito when a flyswatter will do.

Example use cases where docker-compose works just fine:

* Local development environments running on your machine. Even if you run Kubernetes in production, there's likely a lot of orchestration around getting it to work which would make it too complex for a single machine.
* Simple apps without a lot of components
* Single tenant apps which run on a single host
* Deploying docker-based apps on bare VMs or physical hosts
> 4 votes

# Answer
> why I may still need Docker Compose, under with case Docker Compose is a better option?

Need it for what? Better for what? Docker Compose is mostly a different way to link multiple container-apps compared to Kubernetes and probably slightly more lightweight on a Desktop.

But if you run your apps in Kubernetes in production, you probably want to use as similar as possible solution on Desktop as well or none. In this case Kind or Minikube is popular alternatives for Desktop - but usually you want to only run the app directly from the IDE when developing and only in some cases test it in a Kubernetes networked environment.

> I was wondering if it is still worth to learn docker compose ?

Knowledge is good. At least you would learn when to use Docker Compose and when to use Kubernetes with knowledge about both. But if you are using Kubernetes at work and in production it is probably better use of your time to improve your Kubernetes skills and knowledge.
> 0 votes

# Answer
TL;DR - you'll use compose locally or for single machine deployments... Yes its worth learning, probably now more than ever as Docker is dominating local development tooling.

--

RE: Now that k8s won the orchestration war.............

The idea that K8s "won the orchestration war" is something that makes me cringe when I hear other people saying it. It's propaganda that was percolated through the ranks by the CNCF marketing team so that people would just stop asking "what if we do it this way" and start adopting kubernetes.

The problem is, that even years after this came out, (it started circulating in probably 2018ish - I heard it at kubecon that year) its still absolutely untrue. In fact, I'd say that k8s has hit peak hype and its likely to see a bit of a decline in the coming years...

Why you might ask?? How could the "war winner" be taken from its throne? It's pretty simple really, most organization don't need 99% of what k8s offers and on top of that they simply cannot afford the massive cost of managing a k8s deployment (its a LOT).

Now I'm a bit partial, as I work for Cycle.io \- a LowOps PaaS thats a direct k8s competitor, but I'm also on the front lines of hearing the horror stories from company after company as they look to migrate away from k8s and get back to productive. I don't expect you to take my word for it, but the growing masses are looking for k8s alternatives and you can find info on it all over the web.

I'd point you to this DZone article for example.
> 0 votes

---
Tags: docker, kubernetes, docker-compose
---


# Question
Title: Why are Release and Build pipeline separated?
Most of the time I have seen that Infrastructure build and deployment have separate YAML configuration build and release respectively.

Why do we do that? What If infrastructure build and deployments are contained in a single YAML file?

I want to know both disadvantages and advantages.

# Answer
A build process is likely to be linked to code changes, such as when committing a set of changes into source control, or merging changes into a main branch.

Builds typically take care of anything which should be consistent across all environments, or where there is no meaningful value in repeating per-environment; for example, static code analysis (lint), coding style checks, and anything else which is specific to the code. A build process might also generate some kind of artefact such as a zip/archive, that archive may be published upto an artefact registry somewhere, with version number too.

Deployment happens per-environment, ideally with minimal variation such as environment-specific configuration parameters and the target resources. For example, it may be useful to deploy into an ephemeral environment for development testing, then some other test environments for QA or integration testing, and finally for at least one production environment (possibly more than one if the strategy involves a passive failover environment, which should obviously be identical to the active production environment)

The difference boils down to looking at each task performed in the process and asking whether that task should happen just once when the code is committed and/or merged, or whether the task should happen every time the code or artefact is deployed, and applied to each target environment.
> 0 votes

---
Tags: azure-devops, continuous-deployment, azure-pipelines, infrastructure-as-code
---


# Question
Title: Defining and using bash functions in AWS CodeBuild
I thought I could slightly improve readability of repetitive `sed` calls when inserting secrets into a file with AWS CodeBuild. I defined a bash function:

```
replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }

```

This can be called like so:

```
replaceConfig DB_PASSWORD

```

And this will replace all instances of the text `{DB_PASSWORD}` in config.json with the value held in secret environment variable `DB_PASSWORD`. Unfotunately, this does not work in CodeBuild. Inspecting the log file, I see:

```
[Container] 2020*07*31 15:44:40 Running command replaceConfig DB_USERNAME
*codebuild*output*tmp*script.sh: line 4: replaceConfig: command not found

```

I get the same error defining the bash function with the alternative syntax:

```
function replaceConfig { sed -i 's|{'$1'}|'${!1}'|' config.json; }

```

I have read that buildspec version 0.1 used a separate shell process for each command, which would have caused this problem, but I am using buildspec version 0.2, so these commands should be sharing the same shell.

Of course I can just stick with the repetitive sed calls, but am I correct in understanding that bash functions cannot be used in buildspec commands at all? Is there a better way to get these secrets into a configuration file?

```
version: 0.2

run-as: root

env:
  shell: bash
  variables:
    AWS_REGION: us-east-2
  secrets-manager:
    DB_USERNAME: db-credentials:username
    DB_PASSWORD: db-credentials:password
    DB_HOST: db-credentials:host
....
phases:
  ....
  post_build:
    commands:
      - replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
      - replaceConfig DB_USERNAME
      - replaceConfig DB_PASSWORD
      - replaceConfig DB_HOST
....

```

# Answer
Bash functions (even when exported) are not preserved across `commands` steps. This is in contrast to exported env variables, so it seems AWS is doing some shenanigans behind the scenes to prevent it. I see two basic options to work around that:

1. Combine steps which need the function in a single step. E.g.

```
commands:
  - |
    replaceConfig() { sed -i 's|{'$1'}|'${!1}'|' config.json; }
    replaceConfig DB_USERNAME &&
      replaceConfig DB_PASSWORD &&
      replaceConfig DB_HOST

```
2. Source the file with the functions you need in every step. This may sound laborious, but can be automated via the `BASH_ENV` environment variable. E.g. let's say you put all the functions you need in a `codbuild-helper.bash` file in your project root, then you can do:

```
env:
  shell: bash
  variables:
    AWS_REGION: us-east-2
    BASH_ENV: "$CODEBUILD_SRC_DIR/codebuild-helper.bash"
  secrets-manager:
    DB_USERNAME: db-credentials:username
    DB_PASSWORD: db-credentials:password
    DB_HOST: db-credentials:host
....
phases:
  ....
  post_build:
    commands:
      - replaceConfig DB_USERNAME
      - replaceConfig DB_PASSWORD
      - replaceConfig DB_HOST
....

```

But since this file will be sourced in every step (regardless if it's needed or not) and even in bash scripts called along the way, you should take care to indeed only put function definitions in there.
> 1 votes

---
Tags: amazon-web-services, bash, aws-codepipeline
---


# Question
Title: Sharing volumes between pods on different clusters
we have two independent kubernetes clusters that work like Main and it's high available one. and we have some stateful applications that store data within cluster's persistent volumes. my question is .. can I share a PV between those two clusters, so I can have a final result of two pods running on the two clusters reading/writing on the same volume as if they're running on one same cluster. I'm focusing on sharing data between the pods regardless how the pods would handle the data writing.

# Answer
> 0 votes
Can you share a PV, between clusters: in theory yes. You can mirror one PV object into another cluster, given you have similar storage classes and accesses to your storage backend.

Sharing PV between Pods, within one or more clusters: depends. First, on your storage backend. There may be some locking mechanism that would prevent you from attaching the same volume to more than one client at a time.

In addition of which: if your storage backend provides with block devices, your volumes would involve some file system, that may not support being mounted on two systems at once.

To work around those two points above, we could consider something like NFS, CephFS, EFS (aws), a samba share, ... Next, it depends on your application: what are the risks of those two Pods of yours writing the same file at once, and corrupting your data?

Short answer: in theory yes, you could. Although if possible: try to question this requirement. Sounds unusual, may be something wrong with design/architecture.

---
Tags: kubernetes, storage, kubelet
---


# Question
Title: How do I supply a professional license to a Docker image?
I’m using `Windows Server Core` as part of an image I’m working on. When it's deployed to production, how do I supply a license to it?

Besides that, is this license needed to be bought for each image it's mounted on?

I've already mounted an image without an issue, so is simply having the license enough to avoid trouble?

# Answer
> 2 votes
As per an archived Docker blog post about Windows containers, it mentions the following:

> For production, licensing is at the host level, i.e. each machine or VM which is running Docker. Your Windows license on the host allows you to run any number of Windows Docker containers on that host. With Windows Server 2016 you get the commercially supported version of Docker included in the license costs, with support from Microsoft and Docker, Inc.

This is supported by the fact that Microsoft and Docker are documented partners, which is in turn supported by Microsoft team member Myles Keating in a separate thread on a similar topic who appears to have joined Microsoft shortly after an internship. The archived post is also mentioned in a similar thread in the Docker forums.

As a more general tip, to decide if `Windows Server Core` is really the image you'd like to use, check out the other available Microsoft container images.

---
Tags: docker, containers, windows
---


# Question
Title: Why does stripping executables in Docker add ridiculous layer memory overhead?
On this image, I ran the following command to reduce executable sizes:

```
find -P -O3 /usr/bin/ /usr/local/bin -type f -not -name strip -and -not -name dbus-daemon -execdir strip -v --strip-unneeded '{}' \;

```

And its size jumped up from ~779.53 to ~986.55MB!

As an attempt to bypass this caveat I created an intermediate layer to copy the changes over from, like so:

```
FROM base as stripped

RUN find -P -O3 /usr/bin/ /usr/local/bin -type f -not -name strip -and -not -name dbus-daemon -execdir strip -v --strip-unneeded '{}' \;

FROM base

COPY --from=stripped /usr/bin/ /usr/bin/
COPY --from=stripped /usr/local/bin/ /usr/local/bin/

```

However the resulting image size did not change. Also note that the `base` image has other programs installed on it, so simply using another Debian distribution as the intermediate layer wouldn't cover stripping each program on the base image.

Why is this large size difference happening? Is there a way to strip executables in Docker at all without having this happen?

# Answer
> 3 votes
Each directive in your Dockerfile *adds* another layer to the image. So anything you do -- removing files, stripping binaries, etc -- is only going to *increase* the size of the image.

It looks like you're trying to overcome this issue by using a multi-stage build, but that's not doing you any good: those two `COPY` directives are introducing effectively the same changes introduced by the `find` command in the previous stage.

The way to solve this is by discarding the old layers, generally by creating a new image that reflects the state of the top layer only. This is called "squashing" the image, and there are various ways of doing this. Here's one mechanism that works. For this example, I'm using this Dockerfile (based on your linked example) to build `squashtest:base`:

```
FROM docker.io/parrotsec/core:base-lts-amd64

RUN export DEBIAN_FRONTEND=noninteractive && \
    apt-get -q -y update --no-allow-insecure-repositories \
    && apt-get -y upgrade --with-new-pkgs \
    && apt-get -y install --no-install-recommends \
    aria2=1.35.0-3 \
    apparmor=2.13.6-10 \
    apparmor-utils=2.13.6-10 \
    auditd=1:3.0-2 \
    curl \
    debsums=3.0.2 \
    gawk=1:5.1.0-1 \
    git  \
    iprange=1.0.4+ds-2 \
    jq=1.6-2.1 \
    libdata-validate-domain-perl=0.10-1.1 \
    libdata-validate-ip-perl=0.30-1 \
    libnet-idn-encode-perl=2.500-1+b2 \
    libnet-libidn-perl=0.12.ds-3+b3 \
    libregexp-common-perl=2017060201-1 \
    libtext-trim-perl=1.04-1 \
    libtry-tiny-perl=0.30-1 \
    localepurge=0.7.3.10 \
    locales \
    miller=5.10.0-1 \
    moreutils=0.65-1 \
    p7zip-full=16.02+dfsg-8 \
    pandoc=2.9.2.1-1+b1 \
    preload=0.6.4-5+b1 \
    python3-pip=20.3.4-4+deb11u1 \
    rkhunter=1.4.6-9 \
    symlinks=1.4-4 \
    && apt-get install -y --no-install-recommends --reinstall ca-certificates=\* \
    && apt-get -y autoremove \
    && apt-get -y clean \
    && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* \
    && rm -f /var/cache/ldconfig/aux-cache \
    && find -P -O3 /var/log -depth -type f -print0 | xargs -0 truncate -s 0 \
    && localedef -i en_US -c -f UTF-8 -A /usr/share/locale/locale.alias en_US.UTF-8 \
    && localepurge \
    && symlinks -rd / \
    && apt-get -y purge --auto-remove localepurge symlinks \
    && find -P -O3 /etc/ /usr/ -type d -empty -delete

```
1. Build the base image.

```
docker build -t squashtest:base -f Dockerfile.base .

```

This produces the following:

```
$ docker image ls squashtest:base
REPOSITORY   TAG       IMAGE ID       CREATED             SIZE
squashtest   base      58dff2c40a28   About an hour ago   786MB

```
2. Build a new image `squashtest:stripped` with stripped binaries using this Dockerfile:

```
FROM squashtest:base

RUN find -P -O3 /usr/bin/ /usr/local/bin \
  -type f -not -name strip -and -not -name dbus-daemon \
  -execdir strip -v --strip-unneeded '{}' \; || :

```

Which produces:

```
$ docker image ls squashtest:stripped
REPOSITORY   TAG        IMAGE ID       CREATED             SIZE
squashtest   stripped   42aa25ebc0c7   About an hour ago   997MB

```

At this point, the image consists of the following layers:

```
$ docker image inspect squashtest:stripped | jq '.[0].RootFS'
{
  "Type": "layers",
  "Layers": [
    "sha256:7e203d602b1c20e9cf0b06b3dd3383eb36bc2b25f6e8064d9c81326dfdc67143",
    "sha256:1fc5866a0b6b7a23a246acfd46b4c513b4a188d2db2d8a26191989a4a18c74d3",
    "sha256:cc3a9d1a7f9222eee31b688d887c79745e20389ecfe0fe208349c73cfd172b4a"
  ]
}

```
3. We can collapse these into a single layer like this:

```
docker run --rm squashtest:stripped \
  tar -C / -cf- --exclude=./dev --exclude=./sys \
  --exclude=./proc  . |
  docker import - squashtest:imported

```

This produces:

```
$ docker image ls squashtest:imported
REPOSITORY   TAG        IMAGE ID       CREATED          SIZE
squashtest   imported   6f036f16d477   46 seconds ago   626MB

```

We've saved 160MB off the base image.

There are other ways to squash a Docker image; there are a number of tools on GitHub (, , ) that accomplish something similar. `docker build` has a `--squash` option if you enable experimental features, but that doesn't appear to accomplish much when I try it.

I would argue that for the 160MB we've managed to save here the effort isn't worth it. Unless you're running Docker in an extremely constrained environment, that's going to be nothing but a drop in the bucket (for reference, that's about the size of `/bin/ls`).

In fact, the `strip` operation in your Dockerfile is mostly pointless: distributions generally strip binaries by default; you can verify this by running `file` on all the binaries in `/usr/bin` and `/bin` on `docker.io/parrotsec/core:base-lts-amd64`:

```
$ docker run -it --rm docker.io/parrotsec/core:base-lts-amd64 bash
# apt -y install file
# file /bin/* /usr/bin/* | grep ELF  | grep -v stripped

```

That last command returns zero results: all the binaries have been stripped.

---
Tags: docker, containers, linux
---


title: AWS CLI and a JSON file

text:
# Question
Title: AWS CLI and a JSON file
I'm setting up my AWS CLI. To access to AWS services like EC2, I have to switch roles.

I used the command `aws sts assume-role`, and it spit out JSON with `Credentials` and `AssumedRoleUser`, complete with a session token. That's great!

Now what? I've googled around, but I cannot figure what to do with generated JSON. How do I tell AWS CLI, "Use this JSON to authenticate with this role"?

Here is the (redacted) JSON I have:

```
{
    "Credentials": {
        "AccessKeyId": "XXXXXXXXXXW5YABBKUXQ",
        "SecretAccessKey": "XXXXXXXXXXXXXXXXXXXXB2EFEUmCtuwbJPcMBF02",
        "SessionToken": "XXXXXXXXXXsi6md6PTNHYjHUljav",
        "Expiration": "2021-12-03T04:59:02+00:00"
    },
    "AssumedRoleUser": {
        "AssumedRoleId": "XXXXXXXXXX:dev",
        "Arn": "arn:aws-us-gov:sts::999999999991:assumed-role/PROJADMIN/dev"
    }
}

```

# Answer
I've done this in a bash script and read three of the four elements in the `awscli` output to set in environment variables for a later `awscli` command.

It uses `jq` to extract the individual values from the JSON reply, something like this:

```
# get session credentials via sts assume-role
ar_json=$(
    aws sts assume-role \
      --role-arn "${role_arn}" \
      --duration-seconds ${session_secs} \
      --role-session-name what-im-doing-$$
)

# now extract the items into environment variables for my main command
export AWS_SESSION_TOKEN=$(jq -rM '.Credentials.SessionToken' <<<"${ar_json}")
export AWS_ACCESS_KEY_ID=$(jq -rM '.Credentials.AccessKeyId' <<<"${ar_json}")
export AWS_SECRET_ACCESS_KEY=$(jq -rM '.Credentials.SecretAccessKey' <<<"${ar_json}")

# my main command that needs the role permissions
aws s3 sync "${bucket_url}" "${local_dir}"

```

There's some error checking I've omitted to keep the example simple. The `-r` option tells `jq` to not put quotes around its output, and the `-M` option tells `jq` to not colorize its output.

Putting these credentials into environment variables carries a risk of exposing them to someone on the same host who is using `ps` with options to view process environments. The session credentials may expire quickly enough to not worry about it.

A more secure way to pass the credentials to your `awscli` commands would be through writing them to configuration files, as another answer suggested.
> 0 votes

# Answer
The answer was to ignore everything in the JSON except the "SessionToken". To use the session token, either add this to the credentials file profile with the same AWS keys, or export it as an env var

aws\_session\_token = XXXXXXXXXXXXXXXXXXXXXXXX
> 1 votes

---
Tags: amazon-web-services, aws-cli
---


# Question
Title: CoreDNS is not working after installation of microk8s
I have installed a microk8s on my 64 bit raspbian, and enabled a dns server by

```
microk8s enable dns

```

Coredns is continously cashes:

```
microk8s.kubectl describe pod coredns-64c6478b6c-snkxx --namespace=kube-system

Namespace:            kube-system
Priority:             2000000000
Priority Class Name:  system-cluster-critical
Node:                 raspberrypi4-docker3/192.168.0.129
Start Time:           Fri, 12 Aug 2022 18:28:54 +0200
Labels:               k8s-app=kube-dns
                      pod-template-hash=64c6478b6c
Annotations:          cni.projectcalico.org/podIP: 10.1.174.245/32
                      cni.projectcalico.org/podIPs: 10.1.174.245/32
                      priorityClassName: system-cluster-critical
Status:               Running
IP:                   10.1.174.245
IPs:
  IP:           10.1.174.245
Controlled By:  ReplicaSet/coredns-64c6478b6c
Containers:
  coredns:
    Container ID:  containerd://194f8be3d70a7141148c63d2dd92c24e47e40af6c0fdb6d582885a60d38d8e62
    Image:         coredns/coredns:1.8.0
    Image ID:      docker.io/coredns/coredns@sha256:cc8fb77bc2a0541949d1d9320a641b82fd392b0d3d8145469ca4709ae769980e
    Ports:         53/UDP, 53/TCP, 9153/TCP
    Host Ports:    0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       StartError
      Message:      failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: openat2 /sys/fs/cgroup/kubepods/burstable/podfb1a1616-e10f-4916-9df8-160a92cf4340/194f8be3d70a7141148c63d2dd92c24e47e40af6c0fdb6d582885a60d38d8e62/memory.max: no such file or directory: unknown
      Exit Code:    128
      Started:      Thu, 01 Jan 1970 01:00:00 +0100
      Finished:     Fri, 12 Aug 2022 18:32:26 +0200
    Ready:          False
    Restart Count:  6
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hs9ph (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      coredns
    Optional:  false
  kube-api-access-hs9ph:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 CriticalAddonsOnly op=Exists
                             node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From               Message
  ----     ------     ----                    ----               -------
  Normal   Scheduled  4m42s                   default-scheduler  Successfully assigned kube-system/coredns-64c6478b6c-snkxx to raspberrypi4-docker3
  Normal   Pulled     4m41s (x2 over 4m42s)   kubelet            Container image "coredns/coredns:1.8.0" already present on machine
  Normal   Created    4m41s (x2 over 4m42s)   kubelet            Created container coredns
  Warning  Failed     4m41s (x2 over 4m42s)   kubelet            Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: openat2 /sys/fs/cgroup/kubepods/burstable/podfb1a1616-e10f-4916-9df8-160a92cf4340/coredns/memory.max: no such file or directory: unknown
  Warning  BackOff    4m39s (x2 over 4m40s)   kubelet            Back-off restarting failed container
  Normal   Pulled     2m32s (x4 over 3m51s)   kubelet            Container image "coredns/coredns:1.8.0" already present on machine
  Normal   Created    2m32s (x4 over 3m51s)   kubelet            Created container coredns
  Warning  Failed     2m32s (x4 over 3m51s)   kubelet            Error: failed to create containerd task: failed to create shim: OCI runtime create failed: container_linux.go:380: starting container process caused: process_linux.go:545: container init caused: process_linux.go:508: setting cgroup config for procHooks process caused: openat2 /sys/fs/cgroup/kubepods/burstable/podfb1a1616-e10f-4916-9df8-160a92cf4340/coredns/memory.max: no such file or directory: unknown
  Warning  BackOff    2m30s (x13 over 3m51s)  kubelet            Back-off restarting failed container

```

How can I solve this CrashLoopBackOff problem?

# Answer
I have solved the coredns problem by running

```
sudo sed -i '$ s/$/ cgroup_enable=cpuset cgroup_enable=memory cgroup_memory=1/' /boot/cmdline.txt

```

in my raspbian.
> 0 votes

---
Tags: kubernetes
---


# Question
Title: Rationale for using Docker to containerize applications
I'm trying to get a better understanding of the reasons to use \[and not use\] Docker based on specific use cases.

From my current understanding, Docker helps to isolate applications and their dependencies within containers. This is useful to ensure consistent reproducible builds in varied environments.

However, I'm struggling to understand the rationale of using Docker where the environments are essentially the same, and the applications are relatively simple.

Say I have the following:

* a cloud VM instance (DigitalOcean, Vultr, Linode, etc.) with 1Gb RAM running Ubuntu 20.
* a Node.js Express app (nothing too complicated)

The following issues come to the fore:

1. *Dockerizing* this application will produce an image that is ~100Mb after optimization (without optimization probably 500Mb or higher based on my research). The app could be 50Kb in size, but the Docker container dependencies to run it are significantly higher by a factor of up to 10,000 or above. This seems very unreasonable from an optimization standpoint.
2. I have to push this container image to a hub before I can use Docker to consume it. So that's 500Mb to the hub, and then 500Mb down to my VM instance; total of about 1Gb of bandwidth per build. Multiply this by the number of times the build needs to be updated and you could be approaching terabytes in bandwidth usage.
3. I read in a DigitalOcean tutorial that before I can run my container image, I have to do the following:

```
docker pull ubuntu

```

This pulls an image of Ubuntu. But, I'm already on Ubuntu, so does this mean I'm running a container that's running Ubuntu inside an existing VM that is running Ubuntu? This appears to be needless duplication, but I'd appreciate clarification.

4. The Docker Docs specify that I should have 4Gb RAM. This means I have to use more expensive VM instances even when my application does not necessarily require it.

How exactly does *containerization* \[using Docker or similar\] optimize and enhance the DevOps experience, especially on an ongoing basis?

I'm not quite *getting it* but I'm open to clarification.

# Answer
> 2 votes
> I'm struggling to understand the rationale of using Docker where the environments are essentially the same, and the applications are relatively simple.

In reality, it is highly unlikely that any development environment on any project would ever be anywhere near the same as staging/production.

* Services running in staging/production will nearly always be physically hosted and managed somewhere which is not intended to be operated interactively by a human day-to-day, with an appropriate IT/security profile to match;
* The nature of development work, and even internal build/testing typically requires a different IT profile to that of a production server.
* Developers rarely have control over the underlying infrastructure or the organisation's IT/security policies.

There are many ways in which the IT profile of developer environments, including build agents and even test machines, can deviate from production:

* Users/permissions or other security settings.
* Installed tools, SDKs, runtimes, debug/test tools, OS features/packages, and other dependencies operating with debug/test configurations enabled.
* Environment variables
* Filesystem structure and the content of files in globally shared directories
* Configurations of globally-installed dependencies such as web servers.

Furthermore, consider the nature of physical devices and VMs

* They are stateful and mutable
* Every change to any aspect of a device or VM, including installed software and configuration changes potentially affects its entire state for all processes running on it.
* Physical devices and VMs typically run many processes/services concurrently, it would usually not be considered economical to have a whole server or VM just for a single running process.

What containers provide:

* Isolation from the host device/VM and from an organisation's IT, Network and Infrastructure policies.
* Isolation from each other - for example, consider the issue of requiring multiple versions of globally-installed runtime dependencies or modifications to shared host resources such as environment variables or local files.
* Developers typically have full control over their choices of container images and the networking/orchestration inside the container runtime.
* Images are based on immutable layers, meaning it is not possible for the state of any layer in an image to change, so a published image should always be a good, known, valid starting point.
* The size of a parent image tends to be inconsequential because there's typically no reason to duplicate nor to re-download it unless a new version of that parent image is published.
* A container is its own thin, mutable layer on top of an image, usually negligible in size, and uses the parent image for all dependencies.
* if a container ends up in an invalid state, it may be quickly and cheaply disposed and replaced with a fresh, clean new container almost instantly by recreating another fresh new container layer.
* The cheap, light-weight nature of containers makes it very efficient to run a single process per-container.

---
Tags: docker, containers
---


# Question
Title: Accessing source files in triggered deployment build
How come my deployment pipeline cannot see the terraform files in my respository?

In Azure Devops I have a single repo for a small app, with a simple folder structure

```
main
├── pipelines
    build.yaml, deploy.yaml, etc
├── terraform
    main.tf, etc.
├── src
├── test

```

The Build pipeline builds the app, runs unit tests, and publishes the build as a pipeline artifact:

```
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    artifact: 'drop'
    publishLocation: 'pipeline'

```

The Deploy pipeline can be run manually for any environment, but when a build is successful on the main branch, the Deploy pipeline is automatically triggered to deploy the app to our CI environment:

```
trigger: none
resources:
  pipelines:
    - pipeline: buildPipeline
      source: 'Build-MyApp'
      trigger:
        branches:
        - main
parameters:
- name: environment
  type: string
  default: dev

```

But when the Deploy pipeline attempts to run terraform it cannot access the TF files from the repo. This is the output of `tree $(Pipeline.Workspace) /F` :

```
/home/vsts/work/1
├── TestResults
├── a
├── b
├── buildPipeline
│   └── drop
│       └── MyApp.zip
└── s

```

Should I configure the Deploy pipeline with access to the triggering build's repo and if so, how? Or should I publish the terraform files (and any other IAC resources) as artifacts of the build?

# Answer
I resolved the problem by publishing the terraform files from the Build pipeline:

```
- publish: $(Build.SourcesDirectory)/terraform
  artifact: Terraform

```

I also realised that I could run `terraform validate` and other static analysis in the Build pipeline, and failing the build if this does not pass. So the Deploy pipeline has less validation to do and is just concerned with deployment.
> 0 votes

---
Tags: azure-devops, infrastructure-as-code
---


# Question
Title: Puppet Notice: This node did not match any of the listed definitions
I have these definitions for my puppet nodes:

Nginx node:

```
node 'nginx.XXXXXXXX.org' {
  package {"epel-release":
    ensure => "installed",
  }
  package {"nginx":
    ensure => "installed",
  }
  file {lookup("web_dirs"):
    ensure => "directory",
  }
}

```

Mongodb node:

```
node 'mongodb.XXXXXXXX.org' {
  package { 'mongodb':
    ensure => 'installed',
  }
  service {'mongodb':
    ensure => 'running',
    enable => true,
  }
  file { '/tmp/create_admin.js':
    content => epp('/etc/puppetlabs/code/environments/production/templates/create_admin.js.epp')
  }
  exec { 'Create the admin user':
    command => "/usr/bin/mongo < /tmp/create_admin.js && touch /home/vagrant/db_admin_created",
    creates  => "/home/vagrant/db_admin_created",
  }
  file { '/etc/mongodb.conf':
    source => "/etc/puppetlabs/code/environments/production/files/mongodb.conf",
    notify => Service["mongodb"],
  }
  file { '/tmp/create_app_user.js':
    content => epp("/etc/puppetlabs/code/environments/production/templates/create_app_user.js")
  }
  exec { "Create the application user":
    command => inline_epp("/usr/bin/mongo -u <%= lookup('db_root_username') %> -p <%= lookup('db_root_password') %> --authenticationDatabase admin < /tmp/create_app_user.js && touch /home/vagrant/db_user_created"),
    creates => "/home/vagrant/db_user_created",
  }
  exec { "Removes the JS script files":
    command => "/bin/rm -rf /tmp/create_admin.js /tmp/create_app_user.js",
  }
}

node default {
  notify { 'This node did not match any of the listed definitions':}
}

```

Node node:

```
node 'node.XXXXXXXX.org' {
  package {"epel-release":
    ensure => "installed",
  }
  package {"nodejs":
    ensure => "installed",
  }
  package {"npm":
    ensure => "installed",
  }
  file { lookup('app_dir'):
    ensure   => "directory",
    source   => "/etc/puppetlabs/code/environments/production/files/appfiles/",
    recurse => true
  }
  exec { "Run npm against package.json":
    command => "/usr/bin/npm install",
    cwd     => lookup("app_dir"),
  }
  file {"${lookup('app_dir')}/server.js":
    content => epp("/etc/puppetlabs/code/environments/production/templates/server.js.epp"),
    mode    => "0755"
  }
  file {"/etc/systemd/system/node.service":
    content => epp("/etc/puppetlabs/code/environments/production/templates/node.service.epp"),
  }
  service {"node":
    ensure => "running",
    enable => true,
  }
}

```

I have setup a workflow where the master puppet node called puppet host's a git repo from which these three pull manifest configs and apply them locally. Out of all the three nodes the Nginx node produces the following error when I apply a manifest change:

```
Notice: Compiled catalog for nginx.mshome.net in environment production in 0.03 seconds
Notice: This node did not match any of the listed definitions
Notice: /Stage[main]/Main/Node[default]/Notify[This node did not match any of the listed definitions]/message: defined 'message' as 'This node did not match any of the listed definitions'
Notice: Applied catalog in 0.02 seconds

```

I have looked at my definitions and I can't seem to find what is off about the nginx node, any pointers would be appreciated. Also note these machines were created using vagrant.

# Answer
You should check the certname of your nginx host, and make sure it matches the name passed to the node block.

You can find out the certname by running the following command on the nginx host.

```
puppet config print certname

```

From what you describe, the certname does not match: one ends in ".net", the other ends in ".org"
> 0 votes

---
Tags: puppet
---


# Question
Title: Exporting multi-arch Docker image from local registry to .tar file
I have a multi-arch build that is pushing to a local registry I have running:

```
docker buildx build --platform linux/amd64,linux/arm64 --push -t localhost:5000/myimage:latest

```

I need to export this multi-arch image to a `.tar` file for future use with `docker load`. I cannot for the life of me find a way to do this.

I've tried:

```
docker pull localhost:5000/myimage:latest
docker save -o myimage.tar localhost:5000/myimage:latest

```

And this exports only the image for my architecture (`amd64`). So then I tried:

```
docker pull --platform linux/arm64 --platform linux/amd64 localhost:5000/myimage:latest
docker save -o myimage.tar localhost:5000/myimage:latest

```

And this has the same result, even though I'm explicitly telling it to pull both architectures.

How can I export a multi-arch image from a local registry as a single `.tar` file?

# Answer
> 2 votes
This is answered in the Docker blog post on multi-platform docker builds. For your specific situation, you would perform a build and push for each platform you'd like, then create a manifest file and push that. Then you can perform the save. It would look something like this:

```
$ docker buildx build --platform linux/amd64 --push -t localhost:5000/myimage:amd64 .
$ docker buildx build --platform linux/arm64 --push -t localhost:5000/myimage:arm64 .
$ docker manifest create myimage:latest myimage:amd64 myimage:arm64
$ docker manifest push localhost:5000/myimage:latest
$ docker image save -o myimage.tar localhost:5000/myimage:latest

```

---
Tags: docker, docker-registry
---


# Question
Title: Load balancing Nexus artifact repository
Load balancing Nexus is currently not supported by Sonatype, except by putting a Nexus instance in front of two with smart-proxying enabled or via a newer feature.

Still, I gave it a shot, by sharing the filesystem with GlusterFS for the `/storage`, and trying to enable stickyness in the Apache Load Balancer config and in Nexus config for the UI.

With the UI I am failing miserably, as it seems the Nexus container does not honor the properties for setting the cookie value; using the following configuration at the moment:

For the storage it seems to work, by limiting to `GET` and `HEAD` requests only, still have to try opening to `POST`s. I was also unsure about the possible collateral effects of scheduled jobs, so all were disabled in the "secondary" node.

Has anybody achieved some level of trustworthy configuration for load-balancing nexus? The UI is not actually important to be load-balanced, I would be happy enough with the storage.

# Answer
> 8 votes
Sonatype's Nexus 3 Pro supports High Availability through a couple of mechanisms that are collectively known as Component Fabric:

* Peer-to-peer Repository Managers means there is no one master, also known as a single point of failure. Packages are replicated between the nodes to ensure they are eventually consistent.
* Storage Backends mean you can use high durability storage such as S3.
* Dynamic Nodes enable auto-scaling support to increase capacity when demand is high and decrease it when demand is low to reduce costs.

It's not really in Sonatype's interests to support HA for the community project as it would cannibalize some of the enterprise customers from their paid product.

# Answer
> 0 votes
As of now (2022) Sonatype Nexus3 only supports high availability through what they call "cloud resilient single-node deployment", which translates to a 1-replica k8s deployment. In this scenario it is Kubernetes that provides resiliency as it will replicate the pod in case of issues with the pod, the k8s cluster node, or the entire availability zone/datacenter.

There's no support for active/active redundant instances of Nexus3, even in the Pro version. Currently Nexus is in version 3.41. They say in their websie, thought, that they have plans to implement clustering HA (active/active).

Smart proxying is not a feature of Nexus3 and the link for "Component fabric" is broken. I assume they removed this feature.

Also they say in their website that HA requires a paid version (Nexus Pro and not Nexus OSS) but I think that this is inaccurate, since as long as it's k8s that is providing the resiliency, it works for both OSS and Pro versions.

---
Tags: load-balance, nexus, apache-httpd, n3dr
---


# Question
Title: Does docker engine (not Desktop) support Linux containers on Windows 11?
I've installed docker engine according to the following instructions: https://docs.docker.com/engine/install/binaries/#install-server-and-client-binaries-on-windows

I'm trying to run Linux containers on Windows 11, without Docker Desktop:

```
PS C:\> docker run -d -p 80:80 docker/getting-started
Unable to find image 'docker/getting-started:latest' locally
latest: Pulling from docker/getting-started
docker: no matching manifest for windows/amd64 10.0.22000 in the manifest list entries.
See 'docker run --help'.

```

I believe I'm getting the above error because dockerd is configured for Windows containers:

```
PS C:\> docker info  -f '{{.OSType}}/{{.Architecture}}'
windows/x86_64

```

I've tried to use `DockerCli.exe -SwitchLinuxEngine` however it doesn't seem to have been installed:

```
PS C:\> DockerCli.exe
DockerCli.exe : The term 'DockerCli.exe' is not recognized as the name of a cmdlet ...

```

How can I switch to Linux containers?

# Answer
> 0 votes
Using the following guide, I'm able to use docker on Windows without Docker Desktop. It may not be exactly true to my original question but meets my use case. https://dev.to/bowmanjd/install-docker-on-windows-wsl-without-docker-desktop-34m9

Summarising the guide (in case it disappears from the internet):

*I won't accept my own answer for now, in the hope that someone finds a better way*

---
Tags: docker, linux
---


# Question
Title: known_hosts module reports changed when nothing has changed
Why does my task report say that it has changed when nothing has changed?

```
- hosts: localhost
  become: false
  gather_facts: false
  tasks:
    - name: Remove non existing host key from known_hosts file
      known_hosts:
        name: 192.168.122.230
        key: 192.168.122.230 ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCbi2hyrvpTRKC37NOm46n4zCPBb9r6cKk8BPrN2eppFu/0PJlB4D+nRI5epHzs5+1LhPkppbOGLC2VIRl3phMDQci3RIszhEZx4lAyX/HAkM+2zdNJlH2WWs3cbw804B4iqfCvy/Gch5ZXl4qEfpVqMURCr/XjaMQETzbjbfgOoyYxw8M/5Kq8VQy+DzqxNNzPi4ElcFQztxxrKDFPwuDplFdxw3YK+iQ4JHxlLWSfgtwsFhg7Z7uM8/efP7ocB23i2GmmG67yM/T/8uAld9t73V8icfe9WnRk2WVY69p4TzC3tMl2KmUDVm5AwvH+FNm/67E9t2inWHgKZacdOaOrgJ7SimPz0ILYDKd4hXg4whz3vdp21M/acjX3jA+fiwx6/GDIofKhyWOP3SwaiprqHZb+rWxerIOZx1IeuIRDZBH5Hjz7UlE5yg1xnqPXXzrFMj9rsKp9S5VB3HGGDfuOU7VymhZiTHIAuGM+weV6r2cOjn5HgdqkU6ABuchMAJvzaj9a3E07Rzk6h/lgWfy5VT/yl7DA7sM0/YSqKPJKgxbstoaOAZl35SDxAx978T0xlomIxaJUehRefK+G1GgPeLMmk0QtpX1dMH8bD4qvKGoLQG1qeJ4W4HrnoTsGLCxsN5/ek3rnqCekYOSiJ/q9+sZyhcLN1hwrDrrFK5fRUw==
        state: absent
      register: reg_known_hosts

    - name: Show known_host register
      debug:
        var: reg_known_hosts

```

```
TASK [Remove non existing host key from known_hosts file] ***************************************
changed: [localhost]

TASK [Show known_hosts register] ****************************************************************
ok: [localhost] => {
    "reg_known_hosts": {
        "changed": true,
        "diff": {
            "after": "192.168.122.230 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIzlnSq5ESxLgW0avvPk3j7zLV59hcAPkxrMNdnZMKP2\n",
            "after_header": "/home/sxkx/.ssh/known_hosts",
            "before": "192.168.122.230 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIIzlnSq5ESxLgW0avvPk3j7zLV59hcAPkxrMNdnZMKP2\n",
            "before_header": "/home/sxkx/.ssh/known_hosts"
        },
        "failed": false,
        "gid": 1000,
        "group": "sxkx",
        "hash_host": false,
        "key": "192.168.122.230 ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCbi2hyrvpTRKC37NOm46n4zCPBb9r6cKk8BPrN2eppFu/0PJlB4D+nRI5epHzs5+1LhPkppbOGLC2VIRl3phMDQci3RIszhEZx4lAyX/HAkM+2zdNJlH2WWs3cbw804B4iqfCvy/Gch5ZXl4qEfpVqMURCr/XjaMQETzbjbfgOoyYxw8M/5Kq8VQy+DzqxNNzPi4ElcFQztxxrKDFPwuDplFdxw3YK+iQ4JHxlLWSfgtwsFhg7Z7uM8/efP7ocB23i2GmmG67yM/T/8uAld9t73V8icfe9WnRk2WVY69p4TzC3tMl2KmUDVm5AwvH+FNm/67E9t2inWHgKZacdOaOrgJ7SimPz0ILYDKd4hXg4whz3vdp21M/acjX3jA+fiwx6/GDIofKhyWOP3SwaiprqHZb+rWxerIOZx1IeuIRDZBH5Hjz7UlE5yg1xnqPXXzrFMj9rsKp9S5VB3HGGDfuOU7VymhZiTHIAuGM+weV6r2cOjn5HgdqkU6ABuchMAJvzaj9a3E07Rzk6h/lgWfy5VT/yl7DA7sM0/YSqKPJKgxbstoaOAZl35SDxAx978T0xlomIxaJUehRefK+G1GgPeLMmk0QtpX1dMH8bD4qvKGoLQG1qeJ4W4HrnoTsGLCxsN5/ek3rnqCekYOSiJ/q9+sZyhcLN1hwrDrrFK5fRUw==",
        "mode": "0600",
        "name": "192.168.122.230",
        "owner": "sxkx",
        "path": "/home/sxkx/.ssh/known_hosts",
        "size": 97,
        "state": "file",
        "uid": 1000
    }
}

```

When looking at the register it says in the `diff` property that `before` and `after` are the same, yet it reports that it made changes?

Something else I found out is that if I completely empty my `.ssh/known_hosts` file and run the playbook it will say `ok: [localhost]`.

What I'm thinking is, is that when I specify a key to remove it will look for all the keys belonging to a particular host, making sure it does not contain the key in question but the before/after still holds a key (the ed25519 key) and then marks it as changed? It looks like when removing a key, the `known_hosts` module is expecting that all keys will be removed.

I can work around it by using the `changed_when` property but I rather understand why Ansible is saying it changed when it did not change at all.

This will work around the issue I'm having:

```
- name: Remove non existing host key from known_hosts file
  known_hosts:
    name: 192.168.122.230
    key: 192.168.122.230 ssh-rsa ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCbi2hyrvpTRKC37NOm46n4zCPBb9r6cKk8BPrN2eppFu/0PJlB4D+nRI5epHzs5+1LhPkppbOGLC2VIRl3phMDQci3RIszhEZx4lAyX/HAkM+2zdNJlH2WWs3cbw804B4iqfCvy/Gch5ZXl4qEfpVqMURCr/XjaMQETzbjbfgOoyYxw8M/5Kq8VQy+DzqxNNzPi4ElcFQztxxrKDFPwuDplFdxw3YK+iQ4JHxlLWSfgtwsFhg7Z7uM8/efP7ocB23i2GmmG67yM/T/8uAld9t73V8icfe9WnRk2WVY69p4TzC3tMl2KmUDVm5AwvH+FNm/67E9t2inWHgKZacdOaOrgJ7SimPz0ILYDKd4hXg4whz3vdp21M/acjX3jA+fiwx6/GDIofKhyWOP3SwaiprqHZb+rWxerIOZx1IeuIRDZBH5Hjz7UlE5yg1xnqPXXzrFMj9rsKp9S5VB3HGGDfuOU7VymhZiTHIAuGM+weV6r2cOjn5HgdqkU6ABuchMAJvzaj9a3E07Rzk6h/lgWfy5VT/yl7DA7sM0/YSqKPJKgxbstoaOAZl35SDxAx978T0xlomIxaJUehRefK+G1GgPeLMmk0QtpX1dMH8bD4qvKGoLQG1qeJ4W4HrnoTsGLCxsN5/ek3rnqCekYOSiJ/q9+sZyhcLN1hwrDrrFK5fRUw==
    state: absent
  register: reg_known_hosts
  changed_when: reg_known_hosts.diff.before != reg_known_hosts.diff.after

```

Ansible version: `ansible [core 2.13.2]`

What is going on here?

**UPDATE - 18-Aug-2022**

I will start by saying that I'm new to Ansible and I'm not a Python programmer.

When reading the source code for the `known_hosts` module I found the following.

`Line: 228`

```
def search_for_host_key(module, host, key, path, sshkeygen):
  '''search_for_host_key(module,host,key,path,sshkeygen) -> (found,replace_or_add,found_line)
  Looks up host and keytype in the known_hosts file path; if it's there, looks to see
  if one of those entries matches key. Returns:
  found (Boolean): is host found in path?
  replace_or_add (Boolean): is the key in path different to that supplied by user?
  found_line (int or None): the line where a key of the same type was found
  if found=False, then replace is always False.
  sshkeygen is the path to ssh-keygen, found earlier with get_bin_path
  '''
# ...

```

Looking at the comment at the top of the function I can conclude that:

* `found` will equal `True` because I have a key for that host (key of type `ed25519`).
* `replace_or_add` will equal `True` because the `key` is different from the one found (key of type `ed25519`).
* `found_line` will be `None` because no key of the same type was found.

With that in mind I think we can have a look at the `enforce_state` function.

`Line: 117`

```
def enforce_state(module, params):

  # ...

  # Next, add a new (or replacing) entry
  if replace_or_add or found != (state == "present"):
    try:
      inf = open(path, "r")
    except IOError as e:
      if e.errno == errno.ENOENT:
        inf = None
      else:
        module.fail_json(msg="Failed to read %s: %s" % (path, str(e)))
    try:
      with tempfile.NamedTemporaryFile(mode='w+', dir=os.path.dirname(path), delete=False) as outf:
        if inf is not None:
          for line_number, line in enumerate(inf):
            if found_line == (line_number + 1) and (replace_or_add or state == 'absent'):
              continue  # skip this line to replace its key
            outf.write(line)
          inf.close()
        if state == 'present':
          outf.write(key)
    except (IOError, OSError) as e:
      module.fail_json(msg="Failed to write to file %s: %s" % (path, to_native(e)))
    else:
      module.atomic_move(outf.name, path)

    params['changed'] = True

```

I think the culprit lies within this if block. For what I could make up from the comment in the previous function; `replace_or_add` is `True`, `found` is `True` and not equal to `state == "present"`. Within this if block it will read the `known_hosts` file and loops over the lines and when a line (number) matches the `found_line` it will continue the loop, otherwise it will write that line to (a temporary) file. However no matter what it does within the loop, later on in the if block it will always set `params['changed'] = True`, meaning it will always report `changed` regardless of there actually being changes. A possible solution to this could be a counter to keep track of the number of times the loop was continued and then set the `params` variable property like so: `params['changed'] = True if counter > 0 else False`.

Something else I think is happening (regardless of changes) is that it will perform a write operation. If so then yes it would make some sense (would it?) that `params['changed']` is set to `True` but I would much rather see the module only write out to `.ssh/known_hosts` if something actually changed.

**UPDATE - 19-Aug-2022**

This portion is not related to the state always being `changed` but rather something to do with the `hash_host` option in the `known_hosts` module, either your hostname or IP is lost and not set in the `known_hosts` file.

Let's assume the following entry is in my `known_hosts` file.

```
host.local,192.168.122.230 ssh-rsa <public_key>

```

When hashing the `known_hosts` file with `ssh-keygen -H` it represents the line above like so.

```
|1|<salt>|<hash> ssh-rsa <public_key>
|1|<salt>|<hash> ssh-rsa <public_key>

```

Where the first line is for `host.local` and the second line is for `192.168.122.230`.

When attempting the same thing with the `known_hosts` module in Ansible something different will happen.

```
- name: add host key to known_hosts file
  known_hosts:
    name: host.local
    key: host.local,192.168.122.230 ssh-rsa <public_key>
    state: present

```

Will result in the following entry in the `known_hosts` file.

```
host.local,192.168.122.230 ssh-rsa <public_key>

```

However when you enable hashing things will change.

```
- name: add host key to known_hosts file
  known_hosts:
    hash_host: yes
    name: host.local
    key: host.local,192.168.122.230 ssh-rsa <public_key>
    state: present

```

```
|1|<salt>|<hash> ssh-rsa <public_key>

```

It will have only hashed the entry for `host.local` the IP version of that line is just gone. This is something to keep in mind when you want to use both the hostname and IP to access the target.

**UPDATE: 24-Aug-2022**

Opened an issue over on the Ansible github.

If you find any information to be incorrect please let me know or edit this post.

# Answer
It seems that the issue you've filed, Ansible Issue #78598 was noticed. It was possible to reproduce, got a small fix and an `easyfix` label.

Since further verification and testing is outstanding, you might be able to add the three proposed lines of code into your own `known_host.py` and test it. You could provide after testing an update on the Ansible Issue.
> 1 votes

---
Tags: ansible
---


# Question
Title: Prometheus Alertamanger - how to silence all alerts for a given period during a maintenance?
A work scenario that I can't cover currently is that I want to set a maintenance mode, meaning all alerts received from Prometheus to be ignored, I want to be able to set it through the UI for a given period until maintenance finish. One way to do it is manually to start covering each alert which in Prometheus-operator scenario is not sane, or I would lose a couple of hours. I can change config from secret to route all to null, but that is not practical as it requires multiple actions and is not good for on time or other user usage. So I tried to use regex for key and value. Both set to .\* .\* regex. On preview I get all alerts covered, but when I click Create I get "Bad Request"? I couldn't find examples in internet and on official page which surprised me quite a lot.

# Answer
> 3 votes
If it's the only selector alertmanager (0.24) doesn't allow it to match an empty string.

So use prometheus=~".+" or severity=~".+" instead of prometheus=~".\*", otherwise you get the error "silence invalid: at least one matcher must not match the empty string" from alertmanager.

# Answer
> 6 votes
With alertmanager 0.22 (perhaps even earlier?) the solution using regex is a bit less obvious as there is no longer any checkbox to tick for using a regex.

Instead, you must use `Name="Value"` for an equality check and `Name=~"RegEx.*"` if you want to use a regular expression. So you must use the `=~` operator.

# Answer
> 4 votes
What about Name = "prometheus", Value = ".\*". That is something that worked for me.

An alternative can be Name = "severity", Value = ".\*" also.

---
Tags: prometheus, sre
---


# Question
Title: Is scrum or kanban really useful for SRE teams?
Agile practices like scrum and kanban were primarily designed for software development.

Interrupt and unplanned work is a significant component of what most SRE (Site Reliability Engineering) or DevOps teams do. While it is always useful to use a tracking system like Jira to manage work, do sprint or kanban really work for SRE teams?

The constraints I see are:

* The work is very dynamic in nature, with priorities changing on a daily basis. Because of this, the sprint duration of two weeks seems very aggressive and it adds unnecessary overhead.
* People being on call adds another dimension to the problem. Sometimes, more than one team member might get involved in on call / post-mortem tasks.
* The team doesn't have a single "product" and hence it doesn't yield itself to a common planning process
* Daily standup meetings may not make much sense because of the lack of overlap among tasks
* The team might be working on tasks related to more than one partner teams and hence spanning multiple Jira projects. Since a sprint or kanban board allows only one Jira project, it may not be able to fit in all the work.

From what I hear from many SREs that I have spoken to, sprint planning hasn't worked for them at all. I would like to hear from the community here what their experience with sprint and kanban is.

I asked this question on scrum.org as well:

Can scrum be used effectively by SRE teams?

And here is a blog post that raises concerns about Agile and SRE in general:

Agile is Lava

# Answer
> 3 votes
> Is scrum or kanban really useful for SRE teams?

## **TLDR: No**

## More nuanced & Long-form answer:

*Speaking from many experiences at various software companies*:

Niether *strict* Agile nor *strict* Kanban has ever truly "*worked*" for any Operations team that I've seen. Whether the company leadership labeled them as the "*DevOps Team*", "*SRE Team*", the "*Core Operations Team*", the "*Platform Team*", or any other label for that type of Ops work, those strict type of methodologies were never a perfect fit for all the unplanned and ad-hoc work. No matter how hard the company leadership tried to shoehorn the team into some form of those methodologies.

The reasons were due to all of the bullet points you mentioned above:

> * The work is very dynamic in nature, with priorities changing on a daily basis. Because of this, the sprint duration of two weeks seems very aggressive and it adds unnecessary overhead.

> * People being on call adds another dimension to the problem. Sometimes, more than one team member might get involved in on call / post-mortem tasks.

These ones are precisely the issues why Agile sprints are not useful. The type of work is very dynamic in nature. Planned vs. Unplanned work constantly compete for the Ops/SRE/DevOps/*Whatever-you-label-it* Engineer's time, bumping and shifting priorities as the "sprint" goes on. Most Agile practices such as "*planning poker*", or adding "*points*" to stories/tasks, also fail due to the "**Planning Fallacy**". This is a built-in **cognitive bias that all humans share**. It means that we tend to underestimate the amount of time it will take us to complete a task. Supposedly the Agile methodology has this concept that is alleged to alleviate the issues caused by the Planning Fallacy and the "Mythical Man-Month": The "Story points" system.

Most planning-poker style meetings on Software & Ops teams either used a "T-shirt size" (`Small`, `Medium`, `Large`), or fibonnacci number point system, which had no basis or equivalence to "time", **by design**. This exercise was ***by definition intentionally*** useless to estimate time, although leadership at these companies expected it to do so thanks to their conflation of the Agile methodology's concept of "*cadence*" and the average number of points that were assigned to these tasks in a sprint. These average point estimates changed and evolved over time as team members, tasks, and projects all changed. Unfortunately, the planning fallacy also applies to non-time-based difficulty estimates, because most all humans are overly optimistic about how difficult an unknown problem or task might be. For unknown or unplanned tasks, the actual difficulty can wildly vary and is not actually plannable.

So, all of the above results in lots of carryover of tasks to the next "*sprint*", whatever time period is chosen for a sprint cadence. As things change (*change is the only constant*), the point system estimates change too. However, managers' and company leadership's expectations generally aren't as flexible. This sets the team up for failure, because it's inevitable that at some point things change enough to throw off the sprint estimates.

Reliability & Ops Engineering is **not a "*sprint*", it's a *marathon***.

> * The team doesn't have a single "product" and hence it doesn't yield itself to a common planning process

This is another aspect describing the same dynamicity of the type of work mentioned above.

> * Daily standup meetings may not make much sense because of the lack of overlap among tasks. The team might be working on tasks related to more than one partner teams and hence spanning multiple Jira projects. Since a sprint or kanban board allows only one Jira project, it may not be able to fit in all the work.

This is yet another reason why most Agile methodology concepts do not fit this type of work. Working across teams & projects is very common for Ops teams. Overlap is rare when working across teams unless there is an underlying common infrastructure that these teams all depend on.

Every single company ended up doing some form of loose Kanban in the end. That is to say: A board was used to track tasks and these were organized by priority. The team was encouraged to limit tasks in progress so as to not become burned out, or overwhelmed by context switching. There was no usefulness of tracking cycle time, nor any of Toyota's Kanban metrics.

**These were software companies, not car factories.**

The usefulness of *strict* Kanban was very debatable, due to the fact that it was developed at a car factory. Factories have processes and machinery to fabricate the same type of parts repeatably. Software engineering and Operations are not like a car factory, even when you try to make things repeatable and deployable by putting them into containers. Containers just hide the internal complexity, they do not eliminate it. For these reasons, strict Kanban also is not useful for this type of work.

## In Conclusion:

The SRE & DevOps methodologies themselves, and the type of work involved are at odds with the Agile methodology, which was developed with Software Engineering in mind, not Operations. I've observed many dysfunctional software companies where development teams can chuck things over the fence and call it "*done*", and then tell Ops to deploy them. They then expect Ops to keep all these plates spinning until the end of time. There is no end point unless a product or service is decommissioned. Additionally, those companies made the mistake of not building in reliability into the product from the beginning of the product lifecycle. This is a recipe for Operations Team burnout and overload when all these complex pieces of software start building up and breaking in complex ways. The company leadership did not scale the Ops/SRE/DevOps team appropriately, nor did they allow Ops Engineering or Developer Engineering time for reducing toil.

# Answer
> 9 votes
We don't use Agile for the DevOps group ourselves, but we do integrate with the normal Scrum Teams. When something is needed by the team from DevOps, such as optimizing the build server, the related team puts a PBI in their backlog with a 'DevOps' label. Our lead has a custom dashboard in Jira with all issues labeled 'DevOps'. They work with the Scrum Master to get prioritization and then one of the DevOps Engineers is tasked with being an ad-hoc member of that Scrum Team for the life of that issue. This helps us prioritize work based on our "customer's" priorities, ties our effort to the sprint, and allows us to get "credit" for the work we're doing.

Prior to this, we used Kanban on Jira just for a todo list. Unfortunately, we would sometimes have to stop working on something we had planned to work on because one of the teams needed something. Now, unless it's an emergency, they basically request DevOps resources (people/time) via their backlog and the Scrum Master communicating that need to our Lead.

# Answer
> 1 votes
Agile works very well for this type of chaotic environment. However, for the reasons you highlighted, pure textbook Scrum may not be a great fit. As a Scrum Master who does a fair amount of DevOps I've used Kanban boards in Jira to track work.

**Kanban Advantages**

* Visualization of the work the team is doing.
* Identifies bottle necks for the team.
* Helps keep the team focused on moving work along.
* Allows for work to be added at any point.
* Gives visibility to work being done for other teams that are relying on the SREs.
* Can be generic to cover multiple types of jobs.

While a traditional stand up may not be beneficial, do consider modifying it. A good stand up meeting highlights blockers that a team member may be facing that another team member knows how to solve.

# Answer
> 0 votes
IMO: Yes, Scrum can be used effectively by SRE teams. I have never heard or read that team members can only work on sprint workitems, so it is a misconception that you need to account for all of your time and effort in the scope of a sprint; Also, I feel there is a misconception that all of your work is applicable to sprints. So, convergently, you should manage some work with Scrum and some outside of it.

Scrum, at it's core, is a framework (with artifacts and events) that is flexibly and used for continuous improvement over the nature of work that is accepted by the team (or applicable to such work).

What you need to do is strike a balance between the time and effort you can provide for planned, accepted sprint work -- that you and your team can deliver on -- and the time and effort to tackle unplanned work outside of the sprint.

---

There is a measure of severity and priority to all workitems. If you're considering Scrum, you should accept work up to a particular severity. You should also consider work that can reduce the severity of upcoming work (if at all possible). I would also recommend you start your sprints with about half capacity (which may be difficult to determine on since relating time and effort in a capacity fashion in the beginning is) - go light. Your team's goal should always be to deliver what work your team accepts, so be picky and don't overextend.

---

I think the nature of your work is actually comparable to production bugfix work for development teams. Thus, you may want to consult production maintenance teams on their Scrum adoption and success, and Not necessarily constrain yourself to DevOps Engineers experience in program and project management.

# Answer
> 0 votes
Here is my observation and this is what I follow:

* Use a ticket for any piece of work that takes an hour or more of effort
* Keep the ticket updated as and when you make progress on the work
* Try to set a due date for each work item and try to finish them before that time
* Set priorities to indicate the urgency level of the ticket
* Organize work into Epics, Stories, and Tasks if you are using Jira. Use Stories for planned work and Tasks for unplanned work.
* Make sure only a small number of tickets are in progress at any point in time, to maintain a strong focus
* Keep updating ticket status (Open -\> In Progress, In Progress -\> Blocked/Waiting, In Progress -\> Closed etc.)
* Use a dashboard to track your tickets
* Make sure that no tickets sit in the system for too long
* If possible, influence your team to follow the same process and have a common dashboard to track team progress

No need to worry about Sprint and Kanban. If you want to really be very formal, start using story points to track effort estimates, but don't waste team's time in task estimation meetings.

---
Tags: process, sre, agile
---


# Question
Title: How Can I Escape Backspace Using Set_Fact In Ansible With Jinja Conditional
I cannot figure out how to escape an HTML closing tag, **\<\b\>** in this example, when used inside an Ansible Jinja conditional (ansible 2.9.27 in this case). Using an additional backslash **\<\\b\>** does not work and neither does putting the backslash between single quotes **\<'\\'b\>**. Anyone have an idea how I can make this work?

Here is my entire Playbook with credentials and email addresses redacted:

```
- name: GET ADMIN GROUP USERS AND COMMENT (GECOS)
  ansible.builtin.shell: "lslogins --noheadings -g adm -o USER,GECOS|sed s'/$/<br>/'"
  register: lslogins_output
#
#
- name: FORMAT LIST FOR HTML EMAIL
  ansible.builtin.set_fact:
    email_machine_list:  |-
      {% for host in ansible_play_batch %}
      {% if hostvars[host]['lslogins_output']['stdout'] | length > 0 %}
      <b>{{ hostvars[host]['ansible_hostname']|upper }}:<\b><br>
      {{ hostvars[host]['lslogins_output']['stdout'] }}<br><br>
      {% elif hostvars[host]['lslogins_output']['stdout'] | length == 0  %}
      <b>{{ hostvars[host]['ansible_hostname']|upper }}:<\b><br>
      NO ADMINS FOUND<br><br>
      {% endif %}
      {% endfor %}
  run_once: true
  delegate_to: localhost
#
#
- name: EMAIL USER LIST
  mail:
    to: Me <me@foo.bar>
    from: Ansible <ansible@foo.bar>
    subject: "Ansible Notification"
    subtype: html
    body:  '<!DOCTYPE html>
           <html>
           <body>
           <h3>{{ email_machine_list }}</h3><br><br>
           </body>
           </html>'
    host: mail.foo.bar
    port: 587
    secure: starttls
    password: mypassword
    username: me@foo.bar
  run_once: true
  become: false
  delegate_to: localhost

```

The emails I receive from Ansible appear like this. The \<\b\> should not appear and should be interpreted as a closing HTML tag:

```
HOST1:<\b>
syslog
user1

HOST2:<\b>
syslog
user1

HOST3:<\b>
syslog
user1

```

# Answer
> 1 votes
Put the template into a block. For example,

```
    - set_fact:
        email_machine_list: |-
          {% for host in ansible_play_batch %}
          <b>{{ hostvars[host]['ansible_hostname']|upper }}:<\b><br>
          {% endfor %}

```

---

<sup>

Example of a complete playbook for testing

Given the inventory

```
shell> cat hosts
host_1 ansible_hostname=10.1.0.11
host_2 ansible_hostname=10.1.0.12
host_3 ansible_hostname=10.1.0.13

```

The playbook

```
- hosts: host_1,host_2,host_3
  gather_facts: false
  tasks:
    - set_fact:
        email_machine_list: |-
          {% for host in ansible_play_batch %}
          <b>{{ hostvars[host]['ansible_hostname']|upper }}:<\b><br>
          {% endfor %}
      run_once: true
    - debug:
        var: test
      run_once: true

```

gives (abridged)

```
TASK [debug] *************************************************
ok: [host_1] =>
  email_machine_list: |-
    <b>10.1.0.11:<\b><br>
    <b>10.1.0.12:<\b><br>
    <b>10.1.0.13:<\b><br>

```

Email

Put the *body* of the email into the block as well. For example,

```
    - name: email the list
      mail:
        to: Me <admin@localhost>
        from: Ansible <ansible@localhost>
        subject: "Ansible Notification"
        subtype: html
        body: |-
          <!DOCTYPE html>
          <html>
          <body>
          <h3>{{ email_machine_list }}</h3><br><br>
          </body>
          </html>
        host: localhost
        port: 25
      run_once: true
      delegate_to: localhost

```

The email was delivered as expected. I had to turn off the rendering of HTML to text in the email client.

```
Return-Path: <ansible@localhost>
X-Original-To: admin@localhost
Delivered-To: admin@localhost
Received: from [127.0.0.1] (localhost [127.0.0.1])
 by test.example.org (Postfix) with ESMTPS id 3664C180493
 for <admin@localhost>; Thu,  8 Sep 2022 23:48:00 +0200 (CEST)
Content-Type: multipart/mixed; -charset="utf-8"; boundary="===============4465080980945319992=="
MIME-Version: 1.0
From: Ansible <ansible@localhost>
Date: Thu, 08 Sep 2022 23:48:00 +0200
Subject: Ansible Notification
X-Mailer: Ansible mail module
To: Me <admin@localhost>
Cc:
Message-Id: <20220908214800.3664C180493@test.example.org>

<!DOCTYPE html>
<html>
<body>
<h3><b>10.1.0.11:<\b><br>
<b>10.1.0.12:<\b><br>
<b>10.1.0.13:<\b><br>
</h3><br><br>
</body>
</html>

```

Enabled rendering of HTML to text works as well

```
10.1.0.11:
10.1.0.12:
10.1.0.13:

```
</sup>

---
Tags: ansible, jinja
---


title: Combine Helm charts or leave separate?

text:
# Question
Title: Combine Helm charts or leave separate?
I'm new to Helm and my first task was to recreate a simple pipeline. This pipeline is just:

PostgreSQL -\> Spark, with ELK monitoring the instances.

Someone mentioned I may be able to combine all of these into a single Helm chart; such that when I run the Helm chart it spins up PostgreSQL, Spark, and ELK. Is there any advantage to this? Before I attempt to do this, is it a best practice or would it be better to keep each one separate?

# Answer
> 0 votes
Using Kustomize you can combine multiple helm charts and make tweaks to them on the fly. JFrog has a tutorial for doing this sort of thing.

# Answer
> 0 votes
It’s definitely a best practice in terms of maintaining k8’s manifests because with a single chart you have all the manifests in a single location and never have to worry about forgetting a specific manifest while deploying the application.

In the templates directory, write the configuration needed for all the services (Postgres, Spark and ELK monitoring) in *deployment.yml*, *service.yml*, *ingress.yml* as needed and do a`helm install`

---
Tags: docker, kubernetes, containers, helm
---


# Question
Title: PHP-FPM with Nginx in different containers (share init.d response between)
I have a Docker setup for my project with PHP-FPM, Nginx and many other services required by the application, but inside the PHP app I've to check the `init.d/nginx configtest` for new dynamic Vhosts (which are shared by volumes) generated for a tenancy based service.

The main thing its that I dont want to make a single container with PHP-FPM and Nginx for scalability reasons.

My idea is to use any Linux package for watch any change on a folder (Vhosts folder in my case) and test and then reload the service.

Any suggestion?

Note: I've RabbitMQ in one container which is used by the PHP application but now I need to connect it with the Nginx-based container (I don't know if there's a CLI or linux native client for this).

# Answer
> 0 votes
This could be solved by using docker-gen or nginx-proxy to simplify even more.

So this service will listen to the docker sock file (process) and whenever a container is up with a certain labels for customise the domain etc. It will create a vhost to that Nginx service and do the reload itself.

# Answer
> 1 votes
Here is nginx and php done as an s2i builder image. You can basically have plain old php in a git repo and run s2i to make a runtime image that runs it with nginx and php. You simply docker build that dockerfile and pass the resultant image as the builder image to s2i along with your source code.

You will notice I have archived that project. That is because redhat have security patched image for php at https://access.redhat.com/containers/#/registry.access.redhat.com/rhscl/php-72-rhel7 they happen to use apache httpd but since I don’t see that I use tell s2i “make a runtime container for my php code in git” and it just works and I get frequent security patches it’s way easier than messing around with my own dockerfile to do security patches.

# Answer
> 0 votes
Using Apache v2.4+ with mod\_php disabled, php-fpm enabled, and mpm\_event (instead of default mpm\_prefork) provides better performance (and tuning options, IMO) for PHP/PHP-FPM and multi-threaded processes. Using nginx as the front-end web server in a "selective reverse proxy to apache2/httpd for PHP processes" will still offer the added benefits of the superior static file serving, caching options + simple reverse proxy + load balancing featureset of nginx and add the superior multi threaded performance of Apache.

My "answer" here, would be: I recommend using Apache v2.4+ as a backend for PHP processes with nginx as your "public facing" reverse-proxy. The primary concern seems to be for scalability, so I think that my suggestion would provide greater scalability and reduce resource consumption "at scale", with only a minimal increase in resource consumption at idle.

============================== This is where my curiosity kicks in (aka feeling curious, might delete later, idk):

Sorry if I'm necromancing this thread here, but there is limited material and content on this subject matter of splitting off PHP-FPM into a separate container from the web server.

I've asked about this very topic here, and here in the OP there is a vague mention of "scalability reasons".

Though my "answer" doesn't actually answer the question proposed in the OP... This is me proposing a solution by asking: "is this really the right question in the first place?"

I agree with the ultimate point @simbo1905 is trying to make, here. Just use the same container for the web server + php-fpm.

But...

What is it about breaking off PHP-FPM from the web server (in this case: nginx) increases the scalability of an application? Why not use multiple instances of matching containers with nginx+php-fpm running together? Just as the answer by @simbo1905 suggests with apache2+php-fpm?

I see the points being made about security and scalability. I understand the concept of scaling up instances of php-fpm containers under heavy load, but nginx makes it VERY easy to load balance such load. Why not have the added benefit of additional load balanced resources if you are concerned about resource the resource usage php-fpm being scalable?

I'm not sure what the benefits are of having a separate PHP-FPM instance running are if you could just "spend" the extra tiny kb's of disk space and RAM for a more stable and performant application. If the PHP-FPM process(es) or the nginx process(es) fail, the application fails.

The case for making PHP/PHP-FPM "more secure" by having separated containers seems moot, also, considering the layers of protection to the PHP engine that are locked down by system users/groups with appropriate permissions. If anything, it seems like a larger attack surface for malicious actors to be able to pinpoint the port(s) that PHP-FPM is running on, instead of the "same system default" of connecting to php-fpm via unix socket(s).

All that said, I'm mostly asking for personal curiosity. I'm somewhat new to the nuances and performance tuning of Docker/Kubernetes, so I feel like there might be something I'm missing. In the research I've done, sometime around late 2019, many Devs/DevOps/Engineers seem to have started to lean towards putting EVERY service in its own separate container with PHP-FPM dependent applications. I have not yet wrapped my head around "why". It makes sense to me having a separate SQL/Datastore container. It also makes sense to me to isolate the web server in use cases where PHP/php-fpm or some other web server dependent service isn't required as a core function of the entire application. But in the cases where you need both the web server(s) and the back-end services to work in concert, I have yet to understand why the need to separate them.

---
Tags: docker, nginx, php
---


# Question
Title: How can I pass a secret from a Salt Master to a Minion without using a Pillar?
I am looking to understand how I can reduce duplication of secrets. For example SSL Certificates. I want to have the Salt Master pull the Certificate from a Secure Source (Azure Key Vault) and then 'share' that value with a targeted Minion(s).

I want to restrict Key Vault (Secret store) access to the Salt Master. What I have not been able to find out is how can I send or share that secret securely with the Minion. I am wanting to avoid the old school manner of storing the value in GPG messages.

# Answer
You don't need to store in in a GPG message in order to include it in pillar.

Simply fetch the secret in the pillar template, for example:

```
cert: '{{ salt["sdb.get"]("sdb://myvault/certs/" ~ grains["id"]) }}'

```

If only the master can fetch secrets from the vault, then pillar is the only way to securely deliver them to minions.

Note that if pillar caching is enabled, the result will be stored on-disk on the master (readable by the salt user). However, so are the credentials required to fetch the secret in the first place.
> 0 votes

---
Tags: salt, secret-management
---


# Question
Title: Yaml linting failling when jinja is used
I am using https://yamllint.readthedocs.io/en/stable/index.html to lint my yaml files (used in a saltstack context, and linted by jenkins jobs at every commit), but as reported and explained here, it is not possible to validate a yaml file in which is used jinja code.

How do you guys validate / lint your yaml files ?

Do you see a way to keep using yamllint without failing due to jinja code ?

Yaml structure error are the most occuring errors in my company, people from very different background come to edit pillar, that's why I really need a yaml validating process.

# Answer
> 0 votes
My bad, I was searching using the wrong keyword, I just found out that ignoring a file or dir is quite easy, you just need to add an gitignore style block to your yamllint config file, as :

```
ignore: |
  *globalParamName.sls
  *platform.sls

```

See the official doc here.

# Answer
> 0 votes
One tool I have used in the past is this which does a great job of rendering YAML/JSON and allows you to inspect the resulting output in order to validate.

# Answer
> 0 votes
I just run the code, e.g. Ansible or kubectl and if the yaml is incorrect these tools point out the line that contains the error. I never use a separate tool for validation the yaml.

# Answer
> 0 votes
There is a new tool available with Salt 3005 - `yaml.lint`

```
salt-call yaml.lint salt://example/bad_yaml.sls

```

---
Tags: salt, jinja, yaml
---


# Question
Title: Azure Devops solution for max excution time
Azure Devops has a 60 minutes max execution time for pipelines but sometimes my deployment takes much more than this, the pipeline time out and, since is a time out, no further tasks are run in the pipeline, even the ones that send email/message about the error on the pipeline. I cannot change the timeout (budget issue).

I am thinking in start the deployment process (is async), finishing the pipeline with a status other than \`\` `success`, and using a trigger (I can create a trigger on the system I am deploying to do something when the deploy finish) to call the Azure DevOps API and change the status of the pipeline to success/failed and/or start another pipeline to run the post-deploy steps.

Is this overenginering? Is there a better way to solve this problem?

# Answer
> 2 votes
If you separate your tasks in to different jobs, each job has 60 min independently of the next job when running on a MS hosted agents and you can manually set a longer timeout if you use a self hosted agent for a job.

For example my automated tests take longer then 60 minutes so I use a timeoutInMinutes Property on the job of 360 minutes, I am using a self hosted agent so that is why I am not limited to the 60min, more details on agents https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser

```
- job: RunTests
    timeoutInMinutes: 360
    displayName: Automated tests Job
    variables:
    - group: eStrategyURLs
    - name: Env
      value: 'UAT'
   ....
  steps:
      - checkout: QA-Automation
        path: s/$(qaTestCodeRepoName)

      - template: '..\SharedResources\CypressTests.Template.yml'
        parameters:
          emailList: $(Build.RequestedForEmail)
...

```

---
Tags: azure-devops, cicd
---


# Question
Title: Saltstack howto create a Concatenation CA?
I would like to have the following a bit more clean. I'm trying to create concatenated CA file using Salt. To accomplish this I first have to collect all glusterfs.pem files from all my minions and transfer them to the salt-master's minions cache and then output all glusterfs.pem into a single CA file (glusterfs.ca) on the salt-master to later on enroll them again on all minions.

Actually my solution is working but it is simply out of the salt context and looks crappy to me. I'm new to salt, I never worked with reactors, so and I was hoping to find some help (preferably with an example if possible) on this site.

This is my current situation only using states:

glusterfs/init.sls (only runs on minions)

```
/etc/ssl/glusterfs.key:
  x509.private_key_managed:
    - order: 9
    - bits: 2048
    - name: /etc/ssl/glusterfs.key

/etc/ssl/glusterfs.pem:
  x509.certificate_managed:
    - order: 10
    - signing_private_key: /etc/ssl/glusterfs.key
    - CN: {{ grains.id }}
    - C: UA
    - ST: Unavailable
    - L: Unavailable
    - basicConstraints: "critical CA:true"
    - keyUsage: "critical cRLSign, keyCertSign"
    - subjectKeyIdentifier: hash
    - authorityKeyIdentifier: keyid,issuer:always
    - days_valid: 3650
    - days_remaining: 0

/etc/ssl/glusterfs.ca:
  file:
    - order: 11
    - managed
    - source: salt://glusterfs/glusterfs.ca

```

ca.sls (only runs on salt-master):

```
collect_glusterfs_certs:
  cmd.run:
    - order: 1
    - name: salt -C 'not salt' cp.push /etc/ssl/glusterfs.pem

concatenation_glusterfs_certs:
  cmd.run:
    - order: 2
    - onlyif: 'test ! -e /var/cache/salt/master/minions/{master*,worker*}/files/etc/ssl/glusterfs.pem && test ! -e /srv/salt/glusterfs/glusterfs.ca'
    - name: 'cat /var/cache/salt/master/minions/*/files/etc/ssl/glusterfs.pem > /srv/salt/glusterfs/glusterfs.ca'

```

I basically first pull all glusterfs.pem to the local salt-master's minion cache and afterwards create the concatenated glusterfs.ca file which I later deploy onto all the minions I previously collected the glusterfs.pem file from...

Thanks in advance

# Answer
> 1 votes
It feels like what you have done here is implement a discount/bastardized CA (Certificate Authority). I fear that in doing so, you might have inadvertently created a security threat for yourself.

In a traditional PKI deployment, your client would generate a private key and a Certificate Signing Request (CSR) that is then submitted to your Certificate Authority for signing. Your CA would then sign your key after validating the applicant by some means and issue a Certificate to be installed on the client.

What you may have just done by concatenating every single certificate of each client into a CA is grant each and every client the ability to sign certificates for other machines. Any machine in your Gluster cluster then may have the ability to issue a certificate for an arbitrary key which then would be trusted by your Gluster cluster which could allow for an attacker to use a compromised system or key to allow an arbitrary endpoint to join your cluster.

Instead, you might want to follow this guide on setting up a CA for use by your SaltStack. In fact, it is worth noting that your salt master is itself a CA. You could simply copy the minion's certificate and key for use by gluster, though that has the downside of a compromised cert/key being able to be used to impersonate a minion.

But generally speaking, you probably don't want to grant all of your minions the power to be a CA.

# Answer
> 0 votes
* You should use requisites to define the order of states, not order. Otherwise it's going to continue execution when necessary prerequisites fail.
* To collect files from each minion, use the Salt Mine.
* To concatenate the files, use file.managed with a jinja template.

So, if you really want to generate a self-signed unlimited CA on every minion and have all of them trust each other, your state could look something like this:

```
/etc/ssl/glusterfs.key:
  x509.private_key_managed:
    - bits: 2048
    - prereq:
      - x509: /etc/ssl/glusterfs.pem

/etc/ssl/glusterfs.pem:
  x509.certificate_managed:
    - signing_private_key: /etc/ssl/glusterfs.key
    - CN: {{ grains.id }}
    - C: UA
    - basicConstraints: "critical CA:true"
    - keyUsage: "critical cRLSign, keyCertSign"
    - subjectKeyIdentifier: hash
    - authorityKeyIdentifier: keyid,issuer:always
    - days_valid: 3650
    - days_remaining: 0

push glusterfs cert:
  module.run:
    - mine.send:
      - x509.get_pem_entries
      - /etc/ssl/glusterfs.pem
    - onchanges:
      - x509: /etc/ssl/glusterfs.pem

/etc/ssl/glusterfs.ca:
  file.managed:
    - source: salt://pem_concat.jinja
    - template: jinja

```

```
{% for certs in salt["mine.get"]("*", "x509.get_pem_entries") -%}
{% for cert in certs.values() %}
{{ cert }}
{%- endfor %}
{%- endfor %}

```

That will need applying multiple times, as other minion generate their certs and send them to the mine.

The mine can also define in the minion config instead of as part of the state.

---
Tags: automation, salt
---


title: Export CloudFormation file from CDK

text:
# Question
Title: Export CloudFormation file from CDK
Since I've seen Hashicorp Terraform, I have been wondering why the "plan" it can export has binary format.

Now in the AWS world you have got a declarative configuration language called CloudFormation which can be abstracted from using AWS CDK (Cloud Development Kit) in a common programming language of your choice.

Therefore my question: imagine somebody familiar with CloudFormation would like to see what CDK code will do. Is there a way to export a CloudFormation file using CDK?

# Answer
Yes: in the hello world CDK example, they call it "synthesizing":

`cdk synth`

^^ produces CloudFormation file.
> 0 votes

# Answer
As Tamara had already mentioned, CDK already produces this for you during synthesis. To clarify this further, if you look in the **cdk.out** directory, you'll find a *xxxxx.template.json* file for every stack.

This is your synthesized template that CDK passes onto Cloudformation.

Additionally, you may also benefit from taking a look at cdk diff.

> The cdk diff command compares the current version of a stack defined in your app with the already-deployed version, or with a saved AWS CloudFormation template, and displays a list of changes.

Let me know if you have any further questions!
> 0 votes

---
Tags: amazon-web-services, aws-cloudformation
---