# Question
Title: How can I transfer the Product backlogs, EPICs, features etc.. from one organisation to another in Azure DevOps?
I have one Org structure created in my Azure DevOps for Eg: Org ABC, under that I have lots of backlog items (epics, features, PB's etc..) Now I have created a new Org Structure for which I am the owner for Eg: ORG XYZ. I want to move some of the pending backlog items from previous Org ABC into my new Org XYZ. Can I do that or I need to create those backlog items from scratch in my new Org ? Please help and advise.

# Answer
There's not a push-button solution to this, although it has been a long-standing customer request.

For the work items, you can use the export / import feature.

You may need to do the same with your cusom process if you didn't use one of the out-of-the box process templates in Azure DevOps. You can use the process migrator tool or the REST API to do this, so it's more involved than the work items.
> 0 votes

---
Tags: azure
---


# Question
Title: In simple terms, what is the advantage of a Virtual Kubelet?
I have read about the Virtual Kubelet project.

I can see that it allows you to *delegate* from Kubernetes to an Azure Container Instance (ACI), that it allows you to deploy both unix/windows boxes, that it will enable "billing by the second".

I'm quite new to Kubernetes and I am struggling understand the advantages in simple terms.

# Answer
> 1 votes
I'm taking a look at some old questions people might find, I accept that you either solved this a long time ago or moved on!

"In simple terms..." well, yes. It's hard to talk about this without it heading into abstrations, management planes, and similar mind-fogging terminology. In fact, I may be about to fail in describing it without at least some jargon, but it's limited to *only jargon you'll already know if this tech will be useful to you*.

The simple version is that Virtural Kubelet lets you take advantage of serverless platforms or event-driven platforms acrorss providers (Azure, AWS, GCP).

The idea is that it brings scalability and cost benefits.

---
Tags: kubernetes, azure, virtual-kubelet
---


title: gitlab-runner docker: command not found

text:
# Question
Title: gitlab-runner docker: command not found
I'm trying to build docker container using GitLab runner. (lead dev left company, and now I have no idea how to do it.)

I'll share everything I can.

Build output

as I understand from output runner is assigned correctly, the problem is with docker

***the runner is running as Shell executor***

here is **.gitlab-ci.yml**

```
stages:
  - build
  - deploy

variables:
  STACK_NAME: isnta_api
  VERSION: ${CI_COMMIT_TAG}
  IMAGE_NAME: ${DOCKER_REGISTRY}/${CI_PROJECT_PATH}:${CI_COMMIT_TAG}

before_script:
  - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
  - echo $STACK_NAME
  - echo $CI_REGISTRY
  - echo $IMAGE_NAME

build:
  stage: build
  only:
    - tags
  script:
    - docker build -f Dockerfile -t ${IMAGE_NAME}  .
    - docker push ${IMAGE_NAME}

deploy:
  stage: deploy
  only:
    - tags
  script:
    - docker stack deploy --with-registry-auth --compose-file docker-compose.yml ${STACK_NAME}

```

**docker-compose.yml**

```
version: '3.7'

services:
  app:
    container_name: ${STACK_NAME}
    image: ${IMAGE_NAME}
    environment:
      PORT: 4042
      APP_ENV: production
      INSTAGRAM_API_KEY: de194cf559msh8d940d38fd1ca47p129005jsnc11b4fd77a36
      INSTAGRAM_API_HOST_NAME: instagram-bulk-profile-scrapper.p.rapidapi.com
      INSTAGRAM_API_BASE_URL: https://instagram-bulk-profile-scrapper.p.rapidapi.com/clients/api/ig/
    networks:
      - nxnet
    ports:
      - 4042:4042
    deploy:
      replicas: 1
      update_config:
        order: start-first

networks:
  nxnet:
    external: true

```

and **Dockerfile**

```
FROM node:14-alpine

COPY . /app

WORKDIR /app
RUN apk add --no-cache bash
RUN npm install

RUN npm run build

CMD npm run start:prod

```

Any suggestions or tips would be valuable. Thank you in advance

# Answer
Do you know for sure if Docker is installed on the machine/VM that has the runner on it? Depending on your setup, you can SSH into the machines that host the Gitlab runners and just do `docker --version`.

If it's not on there, then you will need to visit https://docs.docker.com/engine/install/ and add it.

I ran into the same situation, coming into a project that used GitLab runners with no prior experience - at the end of the day, a GitLab runner is just a program that can run on any machine. Depending on how your CI/CD is configured, it may be on a development VM, server, workstation, etc. Good luck!
> 0 votes

# Answer
Your .gitlab-ci.yml and Dockerfile are fine, the problem is on the Gitlab Runner that executes the CI pipeline.

The runner cannot access its Docker Engine, either because it is not installed on the runner or the runner is not configured to run in `privileged` mode, which is necessary to use Docker in your CI.

1. Check if Docker is installed If you have access to the machine which hosts the runner, check if Docker is installed there https://docs.docker.com/engine/install/
2. Active privileged mode in the Gitlab Runner Configuration

in config.toml:

`privileged = true`

https://docs.gitlab.com/runner/configuration/advanced-configuration.html

Also check the Gitlab CI Documentation on how to use Docker here https://docs.gitlab.com/ee/ci/docker/using\_docker\_build.html
> 0 votes

---
Tags: docker, gitlab, gitlab-ci-runner
---


# Question
Title: Terraform error: Provider "registry.terraform.io/hashicorp/google" requires explicit configuration. Add a provider block to the root module
Is there anyway to find out what triggers a message like this,

> Provider "registry.terraform.io/hashicorp/google" requires explicit configuration. Add a provider block to the root module and configure the provider's required arguments as described in the provider documentation.

This is what the error looks like in context. I have the provider though defined.

```
Planning failed. Terraform encountered an error while generating this plan.

╷
│ Error: Invalid provider configuration
│
│ Provider "registry.terraform.io/hashicorp/google" requires explicit configuration. Add a provider block to the root module and configure the provider's
│ required arguments as described in the provider documentation.
│

```

# Answer
This error means you're likely using the default configuration which requires a user-wide `gcloud` login. You simple need to run,

```
gcloud auth application-default login

```
> 1 votes

---
Tags: terraform, gcloud
---


# Question
Title: Running parallel builds from jenkins pipeline job triggers and waits for same job on other project
I have this test pipeline code, it runs 2 build on different nodes on `build_servers` as expected.

```
pipeline {
    agent { label 'myserver' }
    stages {
        stage('1') {
            steps {
                script {
                    def tests = [:]
                    for (f in [ 'x', 'y' ]) {
                        tests["${f}"] = {
                            node('build_servers') {
                                stage('Clean Workspace') {
                                     echo "cleaning up Ws ${f}"
                                     step([$class: 'WsCleanup'])
                                }
                                stage('create Wspace txt file') {
                                        script {
                                            props="Wspace=${workspace}"
                                            dir ( 'wspace') {
                                                    writeFile file: 'params.txt', text: props
                                            }
                                        }
                                }
                                stage('Run other job') {
                                    script {
                                        println('calling other project')
                                        props = readProperties file: "wspace/params.txt"
                                           file_params = props.collect {
                                                string(name: it.key, value: it.value)
                                        }
                                        build job: "other_job2", parameters: file_params, wait: true, propagate: true
                                    }
                                }
                            }
                        }
                    }
                    parallel tests
                }
            }
        }
    }
}

```

I expect it to trigger 2 different builds on `other_job2`, but it submits just one job and both `tests['x'] and tests['y']` waits for the same job to finish, why is that? should it not create 2 different jobs in `other_job2` project, one for `x` and one for `y`?

in the console output, I see build #5 for both \[x\] and \[y\] exactly as shown below:

```
[x] Starting building: TEST » other_job2 #5
[y] Starting building: TEST » other_job2 #5

```

Also, I am printing `${f}` in `echo "cleaning up WS...` line, I expected to see both x and y, but I see only `cleaning up Ws y` printed for both `[x]` and `[y]` as below

```
[x] cleaning up Ws y
    [Pipeline] cleanWs
[x] [WS-CLEANUP] Deleting project workspace...
    [WS-CLEANUP] Deferred wipeout is used...
    [WS-CLEANUP] done
    [Pipeline] echo
[y] cleaning up Ws y
    [Pipeline] cleanWs
[y] [WS-CLEANUP] Deleting project workspace...
    [WS-CLEANUP] Deferred wipeout is used...
    [WS-CLEANUP] done

```

# Answer
> 2 votes
Something needs to be different in other\_job2 for tests\['x'\] and tests\['y'\].

Example(untested),

```
for (f in [ 'x', 'y' ]) { {
    {some code}
    build job: "other_job2", parameters: [file_params, string(name:'dummy', value: "${f}")], wait: true, propagate: true
    // the dummy parameter is for preventing mutation
}

```

More info: https://www.jenkins.io/doc/pipeline/examples/#jobs-in-parallel

# Answer
> 0 votes
If you are using "Generic Webhook Trigger Plugin" you need to update plugin to the latest version (above 1.86.0) and enable "Allow several triggers per build" within Job configurations.

---
Tags: jenkins, jenkins-pipeline, jenkinsfile
---


# Question
Title: Original Helm chart gone - how can I find / get it from the cluster?
I'm connected to a cluster which has a release, from a chart, but it was installed by someone else who I think used a tweaked, locally-stored chart.

I don't have access to their original chart/repo. Is there a way I can retrieve it from the cluster, so I can tweak/amend, and re-install it?

Or, can I find the URL of the original repo that was used? The chart name is `wordpress-0.6.0`, but that's of no use, since it's the same as the "official"/original one.

# Answer
You can use `helm get manifest <RELEASE_NAME>` to fetch the kubernets manifests and you can use `helm get values <RELEASE_NAME>` to get the `values.yaml` from the cluster. But I don't think there is any way to get the chart files back. Sorry.

PS: Please replace `<RELEASE_NAME>` with your release name.
> 3 votes

# Answer
To get the package to connect to the helm kubernetes cluster and run `helm fetch` which will

> pull down the char from a repository and (optionally) unpack it in the local directory.

use `helm fetch --help` to see formating options

Use `helm --help` to see available commands.

`Available Commands:
 completion Generate autocompletions script for the specified shell (bash or zsh)
 create create a new chart with the given name
 delete given a release name, delete the release from Kubernetes
 dependency manage a chart's dependencies
 diff Preview helm upgrade changes as a diff
 fetch download a chart from a repository and (optionally) unpack it in local directory
 get download a named release
 history fetch release history
 home displays the location of HELM_HOME
 init initialize Helm on both client and server
 inspect inspect a chart
 install install a chart archive
 lint examines a chart for possible issues
 list list releases
 package package a chart directory into a chart archive
 plugin add, list, or remove Helm plugins
 repo add, list, remove, update, and index chart repositories
 reset uninstalls Tiller from a cluster
 rollback roll back a release to a previous revision
 search search for a keyword in charts
 serve start a local http web server
 status displays the status of the named release
 template locally render templates
 test test a release
 upgrade upgrade a release
 verify verify that a chart at the given path has been signed and is valid
 version print the client/server version information`
> 0 votes

# Answer
You can download the latest stable or unstable version from the helm repo. First, You need to verify the remote helm repo URL below command.

```
# helm repo list

```

For stable repo URL, you will get output like

```
NAME            URL
stable          https://kubernetes-charts.storage.googleapis.com

```

It refers to **`https://github.com/helm/charts/tree/master/stable`**

Now I will tell you to download the chart from the repo URL.

```
helm fetch stable/wordpress --version 0.6.0

```

For the extracted copy of the chart

```
 helm fetch stable/wordpress --version 0.6.0 --untar

```

**The above command will download the chart in your current directory.**
> 0 votes

# Answer
*Below worked for me in helm-3:*

1. Get the name of the release of which you want to get charts using `helm list`
2. `helm get manifest [release name]`

You can write this to some text file instead of stdout and separate the charts. **( Cheers )**

**Note:** *You will need to create persistedVolumes & PVC's. Those are not exported using above command*
> 0 votes

# Answer
I know this is an old question, but since there aren't yet any correct answers, and it's likely more people will face this issue in the future, here's my 2 cents:

You cannot download an original chart from a Kubernetes cluster, but what you *can* do is get all of the information that would be in said chart.

Helm creates a release secret (or configMap in Helm v2) which contains pretty much all information which was in the original chart. This is encoded, but we can make this readable.

First, go into the namespace where your install is, and run

`kubectl get secrets`

There will be a secret similar to this:

`sh.helm.release.v1.<chart name>.v1`

Provided you have sufficient access rights, you can access the contents of this secret. However, the contents have been Base64 encoded twice, and also gzipped. Hence, in order to create a readable file with all of the information you need, you can run this command:

`kubectl get secret sh.helm.release.v1.<chart name>.v1 -o json | jq .data.release | tr -d '"' | base64 --decode | base64 --decode | gzip -d > output.json`

Let's break this down a little.

`kubectl get secret` gets the secret from the cluster

`sh.helm.release.v1.<chart name>.v1` is the secret name we got by running `kubectl get secrets` before

`-o json` ensures we get a json output

`jq .data.release` selects the part from the json output we actually want

`tr -d '"'` deletes unnecessary `"` characters for further processing

`base64 --decode` decodes the base64 encoded data. we run this twice

`gzip -d` unzips the resulting file

`>output.json` ensures we write the resulting data to a file called `output.json` for easier reading (otherwise you're looking at dozens to hundreds of lines of output on the terminal)

---

This results in a json file something like this:

```

{
    "name": "chartName",
    "info": {
        "first_deployed": "2023-04-04T12:53:46.750501771Z",
        "last_deployed": "2023-04-04T13:16:24.683772+02:00",
        "deleted": "",
        "description": "Upgrade complete",
        "status": "deployed",
        "notes": "this is a beautiful app for beautiful people"
    },
    "chart": {
        "metadata": {
            "name": "chartname",
            "home": "link-to-homepage.com",
            "sources": [
                "https://github.com/link/to/source"
            ],
            "version": "1.0.0",
            "description": "this beautiful chart deploys a beautiful app for beautiful people",
            "maintainers": [
                {
                    "name": "john doe",
                    "email": "john.doe@company.com"
                },
                {
                    "name": "jane doe",
                    "email": "jane.doe@company.com"
                }
            ],
            "icon": "link-to-icon.png",
            "apiVersion": "v2",
            "appVersion": "v1.0.0",
            "annotations": {
                "artifacthub.io/license": "Apache-2.0",
            }
            "dependencies": [
                {
                    "name": "foo",
                    "version": "1.0.2",
                    "repository": "link-to-repo"
                },
            "type": "application"
        },
        "lock": null,
        "templates": [
            {
                "name": "templates/configmap.yaml",
                "data": "abc123"
            },
            {
                "name": "templates/ingress.yaml",
                "data": "abc123="
            }],
        "values": {
            "foo": "bar"
            "boolean": false
}

```

The `dependencies` refer to Helm charts that the release you're looking at depends on.

The `templates:` all specify a `name` and `data`. These names refer to the files that were originally in the Helm chart `templates` folder, and the `data` part is their contents (base64 encoded again).

The `values:` section specifies whatever was in the Helm chart's `values.yaml` file.

You can use these dependencies, templates and values to re-create the original Helm chart.

Note this is quite a labour-intensive way of retrieving the information, and it will cost quite a bit of time. However, it is to my knowledge the only way to retrieve this information from a cluster if the original Helm Chart is completely lost.
> 0 votes

---
Tags: kubernetes, helm, disaster-recovery-plans
---


# Question
Title: How to list all images running in a cluster in CSV format?
I'd like to get a CSV for our security team with all the container images running in our kubernetes cluster right now including the image digests and namespace. How can I do that?

# Answer
> 0 votes
This depends on anchore kai (which you just download and drop in your path. https://github.com/anchore/kai/releases) and jq which you probably already have installed. This also assumes your kubectl is configured and pointed to the cluster with needed credentials to read everything you need to acquire.

```
kai -i \
  |jq --raw-output '.results[] \
    | {"namespace": .namespace, "image": .images[].tag, "digest": .images[].repoDigest} \
    | [.namespace, .image, .digest] \
    | @csv' \
  `

```

The first line is getting the images using kai. The `-i` outputs the inventory to the commandline. This is in a json format. Then I use `jq` to get the results in line 2. In line 3 I re-arrange the results so the namespace is associated with each image in the same array. Then I dump these into a bunch of individual rows in line 4 and finally in line 5 output that to stdout in a CSV format instead of json.

You'll get a count of the images per namespace to stderr and the csv to stdout so you can simply add `> inventory.csv` to the end of the command to save it off to a file. You'll still see the stderr output to the screen so you can follow along.

---
Tags: kubernetes
---


# Question
Title: How to exit "Pending" status pods in K8s?
I have scenario where PODs are ending up into “Pending” state.

The reason is that when I launch a POD, I launch it with node affinity because the requirement is that I want to run the POD on specific node.

Now in case of auto-scanning up/down. Sometime the node gets deleted but the POD created to be scheduled on the Node end-up into “Pending” state.

Is there a way that “Pending” pods gets existed/deleted by themself if the Node is not available itself?

# Answer
The "pending" pods won't get deleted automatically. Pods will get deleted only when the pod object gets deleted. If you want to delete the pods when there are no nodes present, you may need to write a custom solution (something like a cron job which keeps checking the pending pods and deleting them). Also, if the pods are being created by other objects such as statefulsets/deployments etc, you need to delete the object which is responsible for creating the pods. (You can also just scale down the size to 0 in case of statefulsets/deployments)

PS: The pods are in pending state is the default behaviour and whenever a new node is available, the workload will be getting scheduled. This is one of the features of kubernetes which helps in auto healing capability.
> 0 votes

# Answer
By default, if a pod is unable to be scheduled because of a node affinity constraint and the specified node is not available, the pod will remain in a "Pending" state until the node becomes available or the node affinity constraint is removed.

One option to address this issue is to use a pod disruption budget (PDB) to specify the minimum number of available nodes that must be present in the cluster at all times. This can help ensure that there are always enough available nodes to satisfy the node affinity constraints of your pods.

For example, you can use a PDB to specify that at least one node must always be available in the cluster by setting the minAvailable field to 1:

```
apiVersion: policy/v1beta1
kind: PodDisruptionBudget
metadata:
  name: my-pdb
spec:
  minAvailable: 1
  selector:
    matchLabels:
      app: my-app

```

This PDB would prevent any nodes from being deleted if there are pods with node affinity constraints that are scheduled on those nodes.

Alternatively, you can use a deployment or replica set to manage the lifecycle of your pods and specify a node affinity constraint for the pods in the deployment or replica set. If a node becomes unavailable, the deployment or replica set will automatically reschedule the pods on another available node. This can help ensure that your pods are always running and avoid the "Pending" state.

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      app: my-app
  template:
    metadata:
      labels:
        app: my-app
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/hostname
                operator: In
                values:
                - node1
      containers:
      - name: my-container
        image: my-image

```

I hope this helps!
> 0 votes

# Answer
You may delete all pending pods after a while This lists all pending pods

```
kubectl get pods --all-namespaces --field-selector=status.phase=Pending

```

This will delete all pending pods that are older than 1 minute

```
kubectl get pods --all-namespaces --field-selector=status.phase=Pending -o json | jq '.items[] | select((now - (.metadata.creationTimestamp | fromdateiso8601)) > 60) | .metadata.name' | xargs -I{} kubectl delete pod {} --force --grace-period=0

```
> 0 votes

---
Tags: kubernetes
---


# Question
Title: Terraform kubernetes_manifest: forbidden: User "system:anonymous" cannot get path "/apis"
Getting the following error with Terraform

> forbidden: User "system:anonymous" cannot get path "/apis"

In context,

```
│ Error: Invalid configuration for API client
│
│   with acme.dev_env_02.kubernetes_manifest.grafana-agent,
│   on modules/acme-02/prometheus.tf line 1, in resource "kubernetes_manifest" "grafana-agent":
│    1: resource "kubernetes_manifest" "grafana-agent" {
│
│ forbidden: User "system:anonymous" cannot get path "/apis"

```

I'm also getting this error with,

```
resource "kubernetes_manifest" "seccomp" {

```

# Answer
If you read the docs for the resource `kubernetes_manifest` you'll see it says the following,

> This resource requires API access during planning time. **This means the cluster has to be accessible at plan time and thus cannot be created in the same apply operation.** We recommend only using this resource for custom resources or resources not yet fully supported by the provider.

This was my problem. You can't create the resource in the same terraform files that use the `kubernetes_manifest` on. And the reason why, for me, only I was getting this error is because the shared state was in GitLab and it wasn't being pulled down.
> 0 votes

# Answer
The error means that you are not authenticating to the K8s cluster for some reason.

Take a look here:

https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs#example-usage

For better troubleshooting, I recommend enabling auditing on the K8s cluster, so you can see what went wrong.

https://kubernetes.io/docs/tasks/debug/debug-cluster/audit/
> 1 votes

---
Tags: kubernetes, terraform, gcloud
---


# Question
Title: What is the GitLab "Digest" on the container registry next to the images, why is different for two duplicated images
The GitLab container registry has a bunch of Digests, you can see them here,

Normally, I would expect them to represent the image's own digest, however, they don't, when I look at them here locally after pulling these images down I see the following with `podman images`

```
REPOSITORY    TAG               IMAGE ID      CREATED        SIZE
myImage       latest            f07e84515821  3 months ago   1.91 GB
myImage       pre-kaniko        f07e84515821  3 months ago   1.91 GB

```

Shocked, I re-ran `podman pull` on both of these images, and nothing changed, locally they have the same digest after I run `podman pull`..

What I would expect is both of the Digests in the Container registry to match my own image which is "f07e84515821" however, in GitLab both of the digests are different. Even worse, **the sizes are different.** So I'm confused. What is a GitLab Digest?

# Answer
> 2 votes
If podman is doing the same thing as docker, the image ID is the config digest from the image. The registry is showing the manifest digest. There are multiple digests for an image, the registry is a content addressable store and images are effectively a Merkle tree.

The root node for that Merkle tree is the manifest. There are two types, a manifest list (for multi-platform images) and an image manifest (for a single platform). The manifest list, as its name suggests, contains a list of pointers to manifest (by digest). And the image manifest contains pointers to the config blob and an array of pointers to layers. Putting that all together with a docker example, you have:

```
$ docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
f1f26f570256: Pull complete
7f7f30930c6b: Pull complete
2836b727df80: Pull complete
e1eeb0f1c06b: Pull complete
86b2457cc2b0: Pull complete
9862f2ee2e8c: Pull complete
Digest: sha256:2ab30d6ac53580a6db8b657abf0f68d75360ff5cc1670a85acb5bd85ba1b19c0
Status: Downloaded newer image for nginx:latest
docker.io/library/nginx:latest

$ regctl manifest get nginx --format body | sha256sum
2ab30d6ac53580a6db8b657abf0f68d75360ff5cc1670a85acb5bd85ba1b19c0  -

$ regctl manifest get nginx --format '{{jsonPretty .}}'
{
  "manifests": [
    {
      "digest": "sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "amd64",
        "os": "linux"
      },
      "size": 1570
    },
    {
      "digest": "sha256:227794033aebbcf39a8d11a99a2b42e945a3c00cde978804082f9d05153e8192",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v5"
      },
      "size": 1570
    },
    {
      "digest": "sha256:43a1d47f229b9b9f20f0e414ccce8bbe8f3d8e2c11e3e222ef81d7b0fe7f3eb7",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm",
        "os": "linux",
        "variant": "v7"
      },
      "size": 1570
    },
    {
      "digest": "sha256:3be40d1de9db30fdd9004193c2b3af9d31e4a09f43b88f52f1f67860f7db4cb2",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "arm64",
        "os": "linux",
        "variant": "v8"
      },
      "size": 1570
    },
    {
      "digest": "sha256:77df563c3da838275fc3253099a272aa385f1797ab458661c35ad325be4fecb0",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "386",
        "os": "linux"
      },
      "size": 1570
    },
    {
      "digest": "sha256:917c49246f0ec7a1ee82fb7bcf3e9cbe199d851b33cd98f97a19c951b05b8051",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "mips64le",
        "os": "linux"
      },
      "size": 1570
    },
    {
      "digest": "sha256:1ff30bac28af0a75228c65450a41045323a1f21ffd1de5f21733a403f08ca183",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "ppc64le",
        "os": "linux"
      },
      "size": 1570
    },
    {
      "digest": "sha256:30350eec6d1283dda69de3addac1a546a1b7435cd7519bc509b375679b744e12",
      "mediaType": "application\/vnd.docker.distribution.manifest.v2+json",
      "platform": {
        "architecture": "s390x",
        "os": "linux"
      },
      "size": 1570
    }
  ],
  "mediaType": "application\/vnd.docker.distribution.manifest.list.v2+json",
  "schemaVersion": 2
}

$ regctl manifest get nginx@sha256:bfb112db4075460ec042ce13e0b9c3ebd982f93ae0be155496d050bb70006750 --format '{{jsonPretty .}}'
{
  "schemaVersion": 2,
  "mediaType": "application/vnd.docker.distribution.manifest.v2+json",
  "config": {
    "mediaType": "application/vnd.docker.container.image.v1+json",
    "size": 7916,
    "digest": "sha256:080ed0ed8312deca92e9a769b518cdfa20f5278359bd156f3469dd8fa532db6b"
  },
  "layers": [
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 31411405,
      "digest": "sha256:f1f26f5702560b7e591bef5c4d840f76a232bf13fd5aefc4e22077a1ae4440c7"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 25583851,
      "digest": "sha256:7f7f30930c6b1fa9e421ba5d234c3030a838740a22a42899d3df5f87e00ea94f"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 626,
      "digest": "sha256:2836b727df80c28853d6c505a2c3a5959316e48b1cff42d98e70cb905b166c82"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 956,
      "digest": "sha256:e1eeb0f1c06b25695a5b9df587edf4bf12a5af9432696811dd8d5fcfd01d7949"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 772,
      "digest": "sha256:86b2457cc2b0d68200061e3420623c010de5e6fb184e18328a46ef22dbba490a"
    },
    {
      "mediaType": "application/vnd.docker.image.rootfs.diff.tar.gzip",
      "size": 1404,
      "digest": "sha256:9862f2ee2e8cd9dab487d7dc2152a3f76cb503772dfb8e830973264340d6233e"
    }
  ]
}

$ docker image inspect nginx --format '{{.ID}}'
sha256:080ed0ed8312deca92e9a769b518cdfa20f5278359bd156f3469dd8fa532db6b

```

For more details on the structure of these various objects that make up the image, see the OCI image-spec.

# Answer
> 1 votes
When I expand both of these I get a bit more information, the "Manifest Digests" are different.

Not exactly sure why they would both have different manifests.

---
Tags: gitlab, images
---


# Question
Title: Use ansible to create GitHub repository with README
We use Ansible to automatically setup Kubernetes clusters. KPT https://kpt.dev/ is further used to deploy config files to pods in these clusters.

In this process we need to create new GitHub Repositories with Ansible. The problem is KPT doesn't work with en empty repo.

Is there a way to create a GitHub Repository with Ansible and include for example a README file? The file could be anything, doesn't matter for KPT.

I used the community.general.github\_repo module to create a repository which worked great, but I don't seem to find a way to automatically include a README.md.

Any ideas?

# Answer
In GH API endpoint that is used in this module has the optional parameter `auto_init` \- for the creation of empty `README` (https://docs.github.com/en/rest/repos/repos?apiVersion=2022-11-28#create-an-organization-repository)

I see few options:

But probably the quickest way will be to use this module: https://docs.ansible.com/ansible/latest/collections/ansible/builtin/uri\_module.html to call GH API
> 1 votes

---
Tags: ansible, github
---


title: TeamCity run step in docker

text:
# Question
Title: TeamCity run step in docker
I'm trying to set up TeamCity to run tests on a .NET project. I have installed TeamCity and its agent (with access to docker) using docker compose :

```
  teamcity:
    image: jetbrains/teamcity-server
    container_name: teamcity
    volumes:
     - /home/arsene/teamcity/data:/data/teamcity_server/datadir
     - /home/arsene/teamcity/logs:/opt/teamcity/logs
    environment:
      - TEAMCITY_HTTPS_PROXY_ENABLED=true
    labels:
      - traefik.http.routers.teamcity.rule=Host(`myhost`)
      - traefik.http.routers.teamcity.tls=true
      - traefik.http.routers.teamcity.tls.certresolver=le
      - traefik.http.services.teamcity.loadbalancer.server.port=8111

  teamcityagent:
    image: jetbrains/teamcity-agent
    container_name: teamcityagent
    volumes:
     - /home/arsene/teamcity/agent:/data/teamcity_agent/conf
     - /var/run/docker.sock:/var/run/docker.sock
     - /opt/buildagent/work:/opt/buildagent/work
     - /opt/buildagent/temp:/opt/buildagent/temp
    environment:
      - AGENT_NAME=TeamCityRunner
      - SERVER_URL=myhost
      - DOCKER_IN_DOCKER=start
    privileged: true

```

When I configure my project I select the following config :

But my build fail with very few detail about whats wrong.

```
Step 1/2: Test (.NET)
  Running step within Docker container mcr.microsoft.com/dotnet/sdk:5.0
  dotnet test
    Starting: .NET SDK 5.0.408 /usr/bin/dotnet test /opt/buildagent/work/e83eb8da5bf3868c/ImPresent.Tests/ImPresent.Tests.csproj @/opt/buildagent/temp/agentTmp/1.rsp
    in directory: /opt/buildagent/work/e83eb8da5bf3868c
    MSBUILD : error MSB1021: Cannot create an instance of the logger. The given assembly name or codebase was invalid. (0x80131047)
    Switch: TeamCity.MSBuild.Logger.TeamCityMSBuildLogger,/opt/buildagent/plugins/dotnet/tools/msbuild15/TeamCity.MSBuild.Logger.dll;TeamCity;plain
    Process exited with code 1
    Process exited with code 1 (Step: Test (.NET))
  Step Test (.NET) failed

```

Do you have any idea of what could be wrong, or how to check logs the have more details ? I ran my tests in the same container on my desktop, they pass.

# Answer
> 1 votes
Teamcity agent is totally unable to detect it is running in docker and try to mount its own directories instead of the proper volumes into the launched container for the step (observed from a problem I encountered with a command line step to run a script within a docker environment).

You need to use command line step and do something like this (untested with .Net but should give the idea, you may need to install jq into the agent container too):

```
MOUNTS=$( docker inspect teamcityagent | jq '.[0].Mounts[] | select(.Destination | contains("/opt/buildagent/")) | "-v \(.Name):\(.Destination)" ' -r | tr '\n' ' ')
docker pull mcr.microsoft.com/dotnet/sdk:5.0
docker run --rm $MOUNTS -w %teamcity.build.workingDir% mcr.microsoft.com/dotnet/sdk:5.0 /usr/bin/dotnet test %teamcity.build.workingDir%/ImPresent.Tests/ImPresent.Tests.csproj

```

# Answer
> 0 votes
Two things I would try:

1. Add "Build" Step before testing:

Runner Type: `.NET`.

Command: `build`.

Also it may be required to add another build configuration to `Restore` the project before build.
2. Execute this build step in "verbose" mode to get more logging information.

---
Tags: docker, testing, dotnet, teamcity
---


# Question
Title: How are you managing automated deployments with active development on the next release while features are still being tested in QA?
* CI/CD tool: Azure Devops Services
* Branching Strategy: Git Flow
* Build Triggers: Auto build Dev branch on PR merges, Auto build on PR creation
* Release Triggers: Create release on every build
* Deploy Triggers: Deploy to DEV on PR merges, **Deploy to QA once each morning**, deployments to other environments are triggered manually
* Production Release/Deployment: At the end of every sprint
* Testing Strategy: Some manual, some automated. Regression takes a week (this is too long I know, but this is where we are).

When we are feature complete, we will create a release branch. That release branch will be deployed to QA. From that point on we do not want PRs into the DEV branch to be released to QA until after we have deployed to Production.

Currently we simply turn off the deploy trigger for QA and deploy manually as necessary into the QA environment. We turn the daily deploy trigger back on once the PROD release is successful.

This has always seemed a little clunky to me.

* Should we be managing/promoting within environments differently?
* Should we look into approvals?
* Is there something we should look into updating the Deploy agent pool instead?

# Answer
> 2 votes
Since it doesn't sound like you're using continuous deployment or continuous delivery, you may be deploying to your QA environment too frequently. I would recommend only deploying your release branches to QA for final checks before going to production.

I'm going to assume that you have sufficiently robust automated test coverage in various forms, from unit tests through end-to-end tests. A lack of automated test coverage will make everything slower, more error prone, and require even more manual testing as the system becomes more complex. It would also be helpful to have linters and static analysis in place to quickly detect style, performance, and security issues during your build pipelines.

When you deploy to your development environment on a PR merge, that is where any test setup is happening. Run your full suite of automated tests and put the changes in an environment where any manual testing can be dry run outside of a release context. This helps keep both your system and your tests stable and ready for release.

When you decide to release, deploy the release branch to your QA environment. Run your automated tests and any manual tests here. If everything is good, you're good to go to production. After triaging failures, you can make the decision to go to production with any known issues to fix later or resolve the failures and rerun testing to confirm.

During this process, you can continue to make changes for the next release in the development branch. You do need to make sure that any fixes for the release branch do get merged back into the development branch as well, if they are still relevant.

The biggest concern that I have with teams that need to take the approach is how time-consuming manual testing can be if automated testing does not keep up with system complexity. This is especially true if you're trying to fit the release process into a timeboxed iteration. But there are mitigations, such as having an independent test team handle that releases that work outside the timeboxed iteration or having the whole team involved in the testing activities prior to a production release.

---
Tags: azure-devops, continuous-deployment
---


# Question
Title: How can I make an image out of a k8s specific pod container
I have made a pod, and it has only one container, I've been able to connect ot it by using `kubectl exec -it -c <container_name> <pod_name> bash`

And I've installed and done several stuff on it.

Now I wish to create an image out of that container to record all those changes in a single image.

I am using docker as the driver. And typically, if this container would have been just any other regular docker container (not managed by k8s), I would have just made a `docker commit <container_name>`

Is there a way to something similar but considering the container is managed inside a pod of k8s?

# Answer
> 0 votes
> Now I wish to create an image out of that container to record all those changes in a single image.

That is generally the wrong way to go about it: you should create your image using a `Dockerfile`, and then deploy it to Kubernetes to test it. You should treat your running container as "read-only" and only makes changes by updating your Dockerfile (or similar) and re-building the image.

Doing things this way ensures that you are able to re-create the image, and allows you to use version control on the Dockerfile so that you have a record of what changes you made and why.

> Is there a way to something similar but considering the container is managed inside a pod of k8s?

The only way to create an image\* from the running container would be to use `docker commit`. There is no Kubernetes mechanism for performing this action. You'll need to log into the Kubernetes node on which your container is running and run `docker commit` there, and then potentially `docker push` the image to a repository if you intend on running it elsewhere.

---

\*: That's a bit of a simplification. E.g., you could use something like `tar` to export the filesystem, and once you have that there are various way to unpack it into an image.

---
Tags: docker, kubernetes, containers, images
---


# Question
Title: Kubernetes Persistent volume not found start.sh issue
There is an application in DockerHub and the working directory is `ENTRYPOINT /home/django/start.sh`

I want to add a persistent volume for /home/django foler but when I add it, it can't find start.sh in the logs of the pod. How should I integrate this. I am adding the lines I added to the following kubernetes deployment file.

```
        volumeMounts:
        - name: django-persistent-storage
          mountPath: /home/django/
      volumes:
      - name: django-persistent-storage
        persistentVolumeClaim:
          claimName: django-pv-claim

```

Logs:

```
Error: failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/home/django/start.sh": stat /home/django/start.sh: no such file or directory: unknown

```

This side is not very clear to me. I would like to know exactly what I'm missing. And how can I move the app to persistent volume?

# Answer
> 0 votes
If your ENTRYPOINT is `/home/django/start.sh` and you are mounting a volume on `home/django`, that's going to mask the contents of the `/home/django` directory. That means anything included in that directory (and its children) will be inaccessible in the running container. This isn't Kubernetes-specific; this is exactly how the `mount` operation behaves in general.

In general, the solution is to redesign your application to separate the static code from runtime resources, and have the application initialize the runtime directories when it starts up.

If you're not able to modify the application, you may be able to accomplish your goal by using an init container to copy the contents of `/home/django` into your volume. That process works by (a) first spawning an init container with the volume mounted at a different location (e.g., `/target`), (b) having the init container run a script that copies the contents of `/home/django` to `/target`, then (c) spawn your regular pod normally.

---
Tags: kubernetes
---


title: Terragrunt Linter

text:
# Question
Title: Terragrunt Linter
I'm having troubles using HCL files at VSCode. When I open the HCL file the indentations are messy. So I fix it and close the file, when I open again...messy again... Someone know a add-ons to works with HCL files?

The official Terraform from Hachicorp not works, because I'm using Terragrunt Syntax. I tried too the another called Hashicorp HCL, but the bugs continues...

# Answer
Until an add-on will be available, you can run this:

This will recursively search the current working directory for any folders that contain Terragrunt configuration files and run the equivalent of terraform fmt on them.

```
terragrunt hclfmt

```

Or this, to format a sepcific file:

```
terragrunt hclfmt --terragrunt-hclfmt-file /path/to/hcl/file

```

https://terragrunt.gruntwork.io/docs/getting-started/configuration/

**Take it one step further:**

1. Install Run on Save - https://marketplace.visualstudio.com/items?itemName=emeraldwalk.RunOnSave
2. Add the follwoing config to *settings.json*

```
"emeraldwalk.runonsave": {
    "commands": [
        {
            "match": "\\.hcl$",
            "cmd": "terragrunt hclfmt --terragrunt-hclfmt-file ${file}"
        }
    ]
}

```

Now every time you will save `.hcl` file, the `terragrunt hclfmt` command will run and reformat that file.

Note - if you have auto save enabled, it will run automatically every time you make a change to `.hcl` file.
> 4 votes

# Answer
Currently, there is only one extension for Visual Studio Code which highlights Terragrunt - https://marketplace.visualstudio.com/items?itemName=4ops.terraform

It is better than nothing.

---

You can also format Terragrunt configurations before git commit automatically using pre-commit and such config.
> 3 votes

# Answer
Just to point out that there is now an official HashiCorp Visual Studio Code extension for HCL.

I was also looking for support for HCL syntax in Visual Studio Code and many extensions produced conflicts or were not further in development. Luckily I found the proper extension in a GitHub discussion about this same topic, if you are curious this is the thread. More details in the official HashiCorp blog post announcing the extension.

Enjoy and happy coding.
> 0 votes

---
Tags: terraform, linting, terragrunt
---


# Question
Title: Can you use the Secure File API if the container doesn't have bash? Like Alpine?
GitLab offers a Project-level Secure Files, the API has you add this to your `.gitlab-ci.yml`,

```
test:
  variables:
    SECURE_FILES_DOWNLOAD_PATH: './where/files/should/go/'
  script:
    - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash

```

However, I'm using the Terraform image which is Alpine without bash which the installer script requires.

How can you use Secure File API if your container doesn't have bash?

# Answer
> 0 votes
Currently, the only option when your container doesn't have bash is to add it. With Alpine you can do that with `apk add bash`. You can do that with `before_script` like this,

```
test:
  variables:
    SECURE_FILES_DOWNLOAD_PATH: './where/files/should/go/'
  before_script:
    - apk add bash
  script:
    - curl --silent "https://gitlab.com/gitlab-org/incubation-engineering/mobile-devops/download-secure-files/-/raw/main/installer" | bash

```

---
Tags: gitlab, bash, alpine
---


title: List all active promtail targets

text:
# Question
Title: List all active promtail targets
I have spend a significant time reading the manual, but I still cannot figure out how to actually get a list of the targets that promtail is currently tailing.

The metrics endpoint only exposes aggregate numbers.

How can I list all the targets promtail is currently tailing?

# Answer
> 2 votes
Sounds like you want `${PROMTAIL_HTTP_ENDPOINT}/targets`. See the server code

---
Tags: prometheus
---


# Question
Title: Terraform deletes AWS lambda layers instead of keeping the older versions
I am using terraform version 12.26 for AWS deployment. Whenever I try updating a lambda layer, the lambda layer version is incremented, but the older layer is deleted. Even though my current version of layer is for example 8, the older versions from 1 to 7 are automatically deleted.

I am using steps as mentioned here.

```
resource "aws_lambda_layer_version" "lambda_layer" {
  filename   = "lambda_layer_payload.zip"
  layer_name = "lambda_layer_name"
  source_code_hash    = filebase64sha256("lambda_layer_payload.zip")
  compatible_runtimes = ["nodejs12.x"]
}

```

How do I retain my older versions of layers?

# Answer
> 3 votes
Terraform's job is to maintain one remote object per resource block, either by updating that same object over time or by replacing it with a new object that meets new requirements.

If you wish to gradually accumulate new objects over time then Terraform is not the correct tool for the job. Your need would be better served by an application deployment tool, such as the Serverless framework.

# Answer
> 1 votes
You can tell terraform to dont care about older layer versions anymore.
This can be achieved by using the `skip_destroy` parameter and set it to `true`.
Keep in mind that layer versions are then intentionally dangling resources that are not managed anymore by terraform and may incur extra expense in your AWS account. See also the `aws_lambda_layer_version` documentation.

```
resource "aws_lambda_layer_version" "lambda_layer" {
  filename            = "lambda_layer_payload.zip"
  layer_name          = "lambda_layer_name"
  source_code_hash    = filebase64sha256("lambda_layer_payload.zip")
  skip_destroy        = true
  compatible_runtimes = ["nodejs12.x"]
}

```

As you reference a quite old terraform version in your question I want to point out that I'm currently referencing terraform version 1.3.9 and aws provider version 4.55.0 but I think the `skip_destroy` parameter is already a bit longer available.

---
Tags: amazon-web-services, terraform, aws-lambda
---


# Question
Title: How do I figure out what cluster helm installed on Google GKE?
When I run a command like

```
helm install MyApp

```

How do I know what cluster it will install too using Google GKE?

# Answer
> 0 votes
Helm just uses whatever your kube config is set to, you can see that with,

```
kubectl config current-context

```

Or you can view the raw config with

```
kubectl config view

```

And it should be listed as `current-context`

---
Tags: kubernetes, helm, google-kubernetes-engine
---


# Question
Title: How can I change the context of kubectl commands and helm?
Currently helm is installing to the wrong cluster on GKE, how can I install to a different cluster?

# Answer
First make sure the desired cluster is available to you

```
gcloud container clusters list

```

Now you need to reconcile that command (which has the data from Google) with your local kubectl, you can do that with

```
kubectl config get-contexts

```

If you do not see a matching context, you can add one. The following command adds the context and sets it as the current-context, which is weird for a command called "get-credentials" but that's Google.

```
gcloud container clusters get-credentials myCluster --region=us-central1

```

But, if there is already a context in your kubectl that you merely want to switch to you can do that with

```
kubectl config use-context myContext

```

---

For more information see this link
> 1 votes

---
Tags: kubectl, google-kubernetes-engine, gcloud
---


# Question
Title: Why does running a nginx image in docker interactive mode makes nginx not working
I have made a docker container from nginx running in background this way:

`docker run -d --name nginx1 -p 49699:80 nginx`

When I go to `localhost:49699` on a browser with cleared cache it works properly as I get the nginx vanilla index.html page

But then, if I launch another container in interactive mode first, and then exit and launch it again it doesn't works. So what I mean here, is that if I first stop and remove container nginx1 mentioned above, and then I:

`docker run -it --name nginx2 -p 49699:80 nginx bash`

I create a container called "nginx2" and I immediately get into it's bash. The I hit `exit` without changing anything inside the container. And since it was interactive, the container stops after I exited the bash. So I the re-start it with `docker start nginx2` and it's running with the expect port forwarding as marked by `docker ps`.

If I then go to `localhost:4699` it doesn't shows me nginx.

So it seems that if a nginx container was created in interactive mode. Even if we get out and start the process again with `docker start <nginx_container>` then nginx won't actually work. As if the interactive mode the container was created with is a process that runs above the nginx processes running inside the container.

I am just trying to understand what the "interactive" mode does to a container that is supposed to have had another process (like nginx) running and ready for port-forwarding. Does interactive mode kills nginx processes?

# Answer
> 2 votes
Basically, containers can only run a single process. When you run:

```
docker run -d --name nginx1 -p 49699:80 nginx

```

The container will start by running its `CMD`. If we look at the Dockerfile for the official Nginx container we can see that the `CMD` is:

`CMD ["nginx", "-g", "daemon off;"]`

So when you start the container it's going to run that command which will start Nginx. You also have the option of passing in your own command and overriding the `CMD` that is in the Dockerfile. We can see options for that in the Docker documentation for `run`.

```
docker run [OPTIONS] IMAGE [COMMAND] [ARG...]

```

After the `IMAGE` name we can specify an optional `COMMAND` and a list of arguments. When you ran:

```
docker run -it --name nginx2 -p 49699:80 nginx bash

```

You told docker not to run the containers default command but instead run `bash`. Which is why Nginx isn't running.

You could omit the bash and daemon flag like this:

```
docker run --name nginx1 -p 49699:80 nginx

```

Which would run the container in the foreground but you wouldn't be able to run any commands on the container, all you would be able to do is see the Nginx logs in your shell.

Instead I think you are looking for this:

```
docker run -d --name nginx_test -p 49699:80 nginx

```

This would start a container called `nginx_test` in the background using the `-d` daemon flag. I also often use `--rm` which tells Docker to delete the container when it has stopped. After the container has started you can connect to it and run commands using this:

```
docker exec -it nginx_test bash

```

Which will let you connect to your running container and run additional commands.

---
Tags: docker, containers, deployment, nginx, ports
---


# Question
Title: Error: can't get a valid version for repositories <package>. Try changing the version constraint in Chart.yaml
When I run `helm dependency build` I get

> Error: can't get a valid version for repositories ingress-nginx. Try changing the version constraint in Chart.yaml

I've tried all kinds of things the version,

```
1.x.x
~1
^1

```

Everything generates that error. This the relevant part of my Chart.yaml

```
- name: ingress-nginx
    repository: https://kubernetes.github.io/ingress-nginx
    version: 1.x.x

```

# Answer
# App Version vs Chart Version

There are two pieces of software in play with different version numbers,

The helm dependency `version` field is for the version of the chart. Not the upstream software being packaged.
> 0 votes

---
Tags: helm
---


# Question
Title: Cloud infrastructure term for services like AWS Sagemaker, Google Colaboratory, Microsoft Azure ML Studio
There are multiple `* as a service` terms in cloud computing <sup>\[, \]</sup>, e.g.

* Infrastructure as a Service (IaaS)
* Platform as a Service (PaaS)
* Software as a Service (SaaS)

But when it comes to data-related and Jupyter notebook related platforms like,

* AWS Sagemaker,
* Google Colaboratory,
* Microsoft Azure ML Studio

Where do they fall on the "Pizza" spectrum? And is there a canonical term for these "Jupyter as a service"?

# Answer
I believe the term for these is "MLOps".

They are really a specialized kind of platform as a service, so they're comparable to PaaS on the pizza scale.
> 1 votes

---
Tags: amazon-web-services, azure, google-cloud-platform, platform-as-a-service
---


# Question
Title: Azure DevOps Pipeline failure - Program does not contain a static 'Main' method suitable for an entry point
I have been tasked with deploying an email service to an Azure tenant using Windows Containers. In my Pipeline (Azure DevOps), I am receiving the following error when in the build step:

**CSC : error CS5001: Program does not contain a static 'Main' method suitable for an entry point.**

I believe my docker file is not correct. It was autogenerated from VS2022 and it doesn't resemble the docker files I've used in the past. Also, the Repo is TFVC.

Any assistance would be greatly appreciated.

```
FROM mcr.microsoft.com/dotnet/aspnet:6.0 AS base
WORKDIR /app
EXPOSE 80
EXPOSE 443

FROM mcr.microsoft.com/dotnet/sdk:6.0 AS build
WORKDIR /src
#COPY ["FACTS_SendMail/FACTS_SendMail.csproj", "FACTS_SendMail/"]
COPY ["/FACTS_SendMail.csproj", "FACTS_SendMail/"]
RUN dotnet restore "FACTS_SendMail/FACTS_SendMail.csproj"
COPY . .
WORKDIR "/src/FACTS_SendMail"
RUN dotnet build "FACTS_SendMail.csproj" -c Release -o /app/build

FROM build AS publish
RUN dotnet publish "FACTS_SendMail.csproj" -c Release -o /app/publish /p:UseAppHost=false

FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "FACTS_SendMail.dll"]

```

# Answer
Move the WORKDIR "/src/FACTS\_SendMail" line right above the "COPY . ." line. This solved the issue for me. Apparently setting the working directory before helps in copying the restored packages from dotnet restore to the right place. Not sure I understand it fully but fixed it for me.
> 1 votes

---
Tags: docker, azure-devops, containers, azure-pipelines, windows
---


# Question
Title: K8S: How to manage incoming traffic for multiple instances of the same app with isolated namespaces and unique subdomains using an Ingress Controller?
I am new to K8S and need help deploying multiple instances of the same app. Each instance needs to be isolated in its own namespace and must have two exposed services (backend and frontend) accessible via a unique subdomain for each instance.

I want to use an Ingress Controller to redirect traffic to the appropriate services based on the subdomain. However, the problem is that the Ingress Controller cannot access other namespaces, which means it does not know about the other instances of the app.

I thought about putting the Ingress Controller in the kube-system namespace, but I don't think this is a good practice.

Here is an example of the NGINX Ingress Controller that I have started to use (so that you have an idea of what I want to do):

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    nginx.ingress.kubernetes.io/ingress.class: nginx
spec:
  ingressClassName: nginx
  rules:
    - host: sub-01.my-domain.com
      http:
        paths:
          - path: /api/
            pathType: Prefix
            backend:
              service:
                name: sub-01-backend-service
                port:
                  number: 30000
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sub-01-frontend-service
                port:
                  number: 80
    - host: sub-02.my-domain.com
      http:
        paths:
          - path: /api/
            pathType: Prefix
            backend:
              service:
                name: sub-02-backend-service
                port:
                  number: 30000
          - path: /
            pathType: Prefix
            backend:
              service:
                name: sub-02-frontend-service
                port:
                  number: 80

```

I have also heard about ISTIO, but I am not sure how it could solve my problem.

If you have any ideas or suggestions for solving this problem, I am all ears.

Thanks in advance for your help.

# Answer
https://doc.traefik.io/traefik/ seems to fit your usecase.

" Traefik is an open-source Edge Router that makes publishing your services a fun and easy experience. It receives requests on behalf of your system and finds out which components are responsible for handling them. "
> 0 votes

---
Tags: kubernetes, ingress, kubernetes-helm, istio, ingress-nginx
---


# Question
Title: vagrant up command fails to find any box
**The issue:** For any vagrant box, whenever I run the vagrant up command, I get the following error:

This even happened when I tried the centos/7 box instead.

**What I have tried:**

* Tried having VirtualBox window open while running vagrant up command
* Installing the latest VC Redist for my AMD64 (x64) and Also tried (x86) VC Redist.
* Followed instructions in the following video, as a previous video of this channel resolved my CentOS 9 issue: https://www.youtube.com/watch?v=2FeqyzY4Y14
* Modifying the Vagrantfile; I added a line `config.vm.box_download_insecure=true` below `config.vm.box`

However, the issue still persists.

# Answer
> 0 votes
The issue here was Firewall/Antivirus. When I disabled Antivirus Protection and ran the `vagrant up` command again, Vagrant worked fine, and launched the VM based on VagrantFile configurations.

---
Tags: vagrant, virtualization, virtualbox, vm
---


# Question
Title: How to change value of pre-defined variable in GitLab to all lowercase
My environment requires the use of the predefined variable `CI_PROJECT_NAMESPACE` in `.gitlab-ci.yml`; however, this variable is in all caps, which makes my build job fail. I've tried several ways to convert the value of this variable to all lowercase with no success:

```
variables:
  CI_PROJECT_NAMESPACE: $CI_PROJECT_NAMESPACE | tr '[:upper:]' '[:lower:]' # doesn't work

build_image:
  stage: build
  image:
    name: #...
    entrypoint: #...
  before_script:
    - export CI_PROJECT_NAMESPACE=$CI_PROJECT_NAMESPACE | tr '[:upper:]' '[:lower:]' # also doesn't work

```

I have successfully hard-coded this variable for my own use, but I need the script to work for other users as well without requiring them to hard-code it themselves. Thanks in advance for your thoughts!

# Answer
You had it almost right. You just can't pipe variable itself to tr. You have to use echo in command substitution, which you can pipe to tr.

```
before_script:
- export CI_PROJECT_NAMESPACE=$(echo "$CI_PROJECT_NAMESPACE" | tr '[:upper:]' '[:lower:]')

```
> 2 votes

---
Tags: gitlab, variables
---


# Question
Title: Docker MongoDB container endlessly restarts on macOS
The MongoDB docker container infinitely restarts every few seconds on local docker env, even though it used to work.

Env: macOS 13.2.1, docker 4.17.0, Engine: 20.10.23, Compose: 1.29.2, Credential Helper: v0.7.0

I tried to delete the MongoDB database folder and fetch the image again, but the MongoDB log contains only these:

```
> 09:10:08.38 Welcome to the Bitnami mongodb container
> 09:10:08.40 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mongodb
> 09:10:08.42 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mongodb/issues
> 09:10:08.43 Send us your feedback at containers@bitnami.com
> 09:10:08.45
> 09:10:08.46 INFO  ==> ** Starting MongoDB setup **
> 09:10:08.53 INFO ==> Validating settings in MONGODB_* env vars...
> 09:10:08.56 INFO  ==> Initializing MongoDB...
> 09:10:08.63 INFO  ==> Deploying MongoDB from scratch...
> 09:10:08.67 INFO  ==> Custom configuration /opt/bitnami/mongodb/conf/mongodb.conf detected!

```

# Answer
> 0 votes
I found a temporary solution:

1. Uncheck the "Use Rosetta for x86/amd64 emulation on Apple Silicon" option under Settings \> Features in development
2. Restart Docker

The AMD64-based containers function but run very slowly now, and I'm almost sure the same container used to work with the same option checked.

---
Tags: docker, macos
---


# Question
Title: Docker warning about platform not matching the host platform
I have a Dockerfile that works on both mac and windows. But when I create an actual container on mac, I always get a warning that says:

```
PS /Users/me/> docker run -it platformtest /bin/bash

WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
root@0569c6d5ad39:/builds# exit

```

This is despite the fact that when I create the image, I'm specifying amd64. Here's the relevant part of the Dockerfile:

```
FROM --platform=amd64 mcr.microsoft.com/dotnet/sdk:6.0.202-bullseye-slim-amd64 AS deployFunctionApp
RUN apt update && apt-get install vim -y
COPY . /builds
RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash
RUN apt-get install curl && curl -sL https://deb.nodesource.com/setup_12.x | bash -
RUN apt-get install nodejs

```

And the command I use to create the image is:

```
 docker build --target deployFunctionApp -t myimagename .

```

The build happens with no errors or warnings.

Questions

Why do I get this warning eventhough i've specified the platform during the build process? It seems that I can safely ignore it, cuz everything seems to be working. Is there anything that I should be looking out for? Any way to suppress this message?

# Answer
> 0 votes
The warning is just a warning, though Docker states in Known issues for Docker Desktop \> Known issues for Mac with Apple silicon that containers will be slowe, use more memory, and can even crash:

> ... attempts to run Intel-based containers on Apple silicon machines under emulation can crash as qemu sometimes fails to run the container. In addition, filesystem change notification APIs (inotify) do not work under qemu emulation. Even when the containers do run correctly under emulation, they will be slower and use more memory than the native equivalent.

> In summary, running Intel-based containers on Arm-based machines should be regarded as “best effort” only. We recommend running arm64 containers on Apple silicon machines whenever possible, and encouraging container authors to produce arm64, or multi-arch, versions of their containers. We expect this issue to become less common over time,

---
Tags: docker, dockerfile, macos
---


# Question
Title: How can I chain Packer builds within a single template
I would like to define a single Packer template that consists of 3 builds; each one building upon the previous:

```
source "amazon-ebs" "base" {
  source_ami_filter {
    filters {
      ...
    }
}

build {
  name = "build_1"
  source "amazon-ebs.base" {
    ami_name = "build-1-ami"
  }
}

build {
  name = "build_2"
  source "amazon-ebs.base" {
    ami_name = "build-2-ami
    source_ami = build_1.output.ami_id
    source_ami_filter { filters = {} }
  }
}

build {
  name = "build_3"
  source "amazon-ebs.base" {
    ami_name = "base-3-ami
    source_ami = build_2.output.ami_id
    source_ami_filter { filters = {} }
  }
}

```

I know that it is possible to chain builders in some way (https://www.packer.io/guides/packer-on-cicd/pipelineing-builds#chaining-together-several-of-the-same-builders-to-make-save-points). However, that example is using Docker and doing it a little more indirectly.

Is it possible to refer directly to the AMI ID produced by a previous build stage? I know `build_x.output.ami_id` is incorrect but is there a syntax to allow it?

If so, I think I also need to override/unset the `source_ami_filter` from the source because the documentation says when `source_ami` and `source_ami_filter` are used together then `source_ami` has to meet the other criteria of the filter which won't necessarily be the case.

# Answer
After trying in vain to do this same "chaining" myself, I finally believe the claim made in Packer's documentation that you link to:

> Packer templates don't come with a custom "glue" to bind them together. We recommend using your CI system or wrapping scripts to connect the templates into a chain.

The largest problem lies in the fact that Packer builds for AWS AMIs require the source AMI ID in order to do pre-build validations. So if a single template will build two AMIs, neither of them can reference the AMI ID of the other, because that's unknown during pre-build validations.
> 1 votes

---
Tags: packer
---


title: How to get the latest Tomcat version?

text:
# Question
Title: How to get the latest Tomcat version?
If one checks the Tomcat homepage then one could see some versions, but not only the latest. Off course I could parse the website and find the version, but I am looking for a Tomcat API that shows the latest version to prevent that I have to create a script that is error prone.

**Why?**

I would like to automate the update of our internal tomcat packages. At the moment someone is checking the latest version and create the debian package manually, but I want to see that that is automated.

**What has been tried?**

* The github mirror does not show the latest version https://github.com/apache/tomcat/releases
* The official tomcat html could be parsed, but that does not seem to be the easiest solution
* `curl https://api.github.com/repos/apache/tomcat/releases/latest` returns:

```
{
  "message": "Not Found",
  "documentation_url": "https://developer.github.com/v3/repos/releases/#get-the-latest-release"
}

```

# Answer
Get the latest version of Tomcat:

```
$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=1' | jq -r .[].name
9.0.20

```

Get the latest version of Tomcat 9:

```
$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^9 | tail -n 1
9.0.20

```

Get the latest version of Tomcat 8.5:

```
$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^8.5 | tail -n 1
8.5.41

```

Get the latest version of Tomcat 7:

```
$ curl -s 'https://api.github.com/repos/apache/tomcat/tags?per_page=100' | jq -r '.[] | .name' | sort -V | grep ^7 | tail -n 1
7.0.94

```

These all currently match the latest versions listed at http://tomcat.apache.org/
> 4 votes

# Answer
Indeed, releases don't appear to have been registered on GitHub, the `releases` list returned by the REST is empty, most likely explaining the 404 returned for `releases/latest`:

```
curl -i https://api.github.com/repos/apache/tomcat/releases
HTTP/1.1 200 OK
...

[

]

```

There are however tags registered, including some apparently mapping to the 9.0.X versions:

```
curl -i https://api.github.com/repos/apache/tomcat/tags | & grep '"name"'
    "name": "TONCAT_9_0_0_M23",
    "name": "TOMCAT_9_0_5",
    "name": "TOMCAT_9_0_4",
    "name": "TOMCAT_9_0_3",
    "name": "TOMCAT_9_0_2",
    "name": "TOMCAT_9_0_1",
    "name": "TOMCAT_9_0_0",
    "name": "TOMCAT_9_0_0_m27",
    ...
    "name": "TOMCAT_9_0_0_M6",

```

Example of a tag info, which *might* be of use:

```
  {
    "name": "TOMCAT_9_0_5",
    "zipball_url": "https://api.github.com/repos/apache/tomcat/zipball/TOMCAT_9_0_5",
    "tarball_url": "https://api.github.com/repos/apache/tomcat/tarball/TOMCAT_9_0_5",
    "commit": {
      "sha": "e0fdefed3d9e2a4dbfd36a5a79e75e0eadaa201d",
      "url": "https://api.github.com/repos/apache/tomcat/commits/e0fdefed3d9e2a4dbfd36a5a79e75e0eadaa201d"
    }
  },

```

But the info doesn't appear consistent with the info on the Tomcat homepage you referenced:

* the `9.0.5` version isn't listed as released on the homepage
* only `9.X` versions show up, older ones don't

So I'm not sure if this is helpful.
> 0 votes

# Answer
Today `8.5.49` was returned as a 'stable' version, while it should have been `8.5.47`. The following:

```
URL=https://tomcat.apache.org/download-80.cgi
curl -s $URL  | grep "8\.5\.[0-9]\+</a>" | sed -e 's|.*>\(.*\)<.*|\1|g'

```

returns:

```
8.5.47

```
> 0 votes

# Answer
Using Powershell's `Invoke-WebRequest` cmdlet:

```
$Links=$(Invoke-WebRequest -Uri 'https://tomcat.apache.org/download-80.cgi' | Select-Object -ExpandProperty Links)
$($Links | Where-Object -Property href -Match '#8.5.[0-9]+').href.substring(1)

```

Returns:

```
8.5.54

```

This command works on Windows/Linux/MACOS
> 0 votes

# Answer
I created a script to download, unzip, install tomcat 9.0.latest

sample.war and deploy

curl \<\>/sample/

Here is the information that I assume you're wanting, below:

```
# PATHS #
# DO NOT END DIRECTORY PATHS WITH \ PLEASE!
$WORKING_DRIVE="C:"
$DOWNLOADS_DIRECTORY="${WORKING_DRIVE}\DOWNLOADS"
$TOMCAT_VERSION_NUMBER="9"
$TOMCAT_SUB_VERSION_NUMBER="0"
$TOMCAT_DIRECTORY="${WORKING_DRIVE}\Tomcat${TOMCAT_VERSION_NUMBER}${TOMCAT_SUB_VERSION_NUMBER}"

# Script will automatically download the tomcat file if one is not found in folder
$Links=$(Invoke-WebRequest -Uri "https://tomcat.apache.org/download-${TOMCAT_VERSION_NUMBER}${TOMCAT_SUB_VERSION_NUMBER}.cgi" | Select-Object -ExpandProperty Links)
$TOMCAT9_LATEST_VERSION=$($Links | Where-Object -Property href -Match "#${TOMCAT_VERSION_NUMBER}.${TOMCAT_SUB_VERSION_NUMBER}.[0-9]+").href.substring(1)
$TOMCAT9_LATEST_ZIP_FILE="apache-tomcat-${TOMCAT9_LATEST_VERSION}-windows-x64.zip"
$TOMCAT_DOWNLOAD_URL="https://dlcdn.apache.org/tomcat/tomcat-${TOMCAT_VERSION_NUMBER}/v${TOMCAT9_LATEST_VERSION}/bin/${TOMCAT9_LATEST_ZIP_FILE}"
Write-Output "Downloading latest Tomcat ${TOMCAT9_LATEST_VERSION} file:`n${TOMCAT_DOWNLOAD_URL}"
Invoke-WebRequest -URI ${TOMCAT_DOWNLOAD_URL} -OutFile "${DOWNLOADS_DIRECTORY}\${TOMCAT9_LATEST_ZIP_FILE}"

# https://tomcat.apache.org/download-90.cgi
# https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.74/bin/apache-tomcat-9.0.74-windows-x64.zip
# ^ 12MB

```

#################################

# POWERSHELL OUTPUT:

Downloading latest Tomcat 9.0.74 file: https://dlcdn.apache.org/tomcat/tomcat-9/v9.0.74/bin/apache-tomcat-9.0.74-windows-x64.zip
> 0 votes

---
Tags: automation, tomcat, api
---


# Question
Title: How can I tell my Helm Chart to ignore the default ingress-nginx and create a new one?
Whenever I run `helm install` on the chart I'm making I see this,

> Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: IngressClass "nginx" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "acme-1681486075": current value is "acmecorp-nginx"; annotation validation error: key "meta.helm.sh/release-namespace" must equal "acme": current value is "default"

What concerns me here is this part

> Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: IngressClass "nginx" in namespace "" exists and **cannot be imported into the current release**

I don't want to import the cluster default. I want to create and bring my own ingress controller with my helm chart. Why is it still thinking this is a default to be imported?

```
dependencies:
  ...
  - name: "ingress-nginx"
    repository: https://kubernetes.github.io/ingress-nginx
    version: 4.x.x
controller:
  electionID: ingress-controller-leader
  ingressClassResource:
    name: internal-nginx  # default: nginx
    enabled: true
    default: false
    controllerValue: "k8s.io/internal-ingress-nginx"  # default: k8s.io/ingress-nginx

```

What is the syntax in a Chart.yaml to list ingress-nginx as a dependency for a multiple ingress install?

# Answer
> 0 votes
For me the problem here was spelling, believe it or not. I just had to lowercase the `release` name in the chart. I figured this out, by dumping my helm to a yaml using `helm template` and then applying that. When I did this is what I got

> Error from server (Invalid): error when creating "nginx-ingress-install.yaml": ServiceAccount "release-name-FooBar" is invalid: metadata.name: Invalid value: "release-name-FooBar": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is 'a-z0-9?(.a-z0-9?)\*')

I just had to change my release from `FooBar` to `foobar`

---
Tags: kubernetes, helm, ingress, ingress-nginx
---


# Question
Title: Distrubuted storage PVC on k3s using OpenEBS is stuck in a pending status while provisioning - waiting on external provisioning, not sure why
I'm trying to setup distributed raid 1+0 storage on my k3s cluster running of 5 raspberry pi 4s running armbian (jammy). Currently I'm seeing the event `waiting for a volume to be created, either by external provisioner "openebs.io/provisioner-raw-block" or manually created by system administrator` on my test PVC and it has been stuck provisioning in a pending state for over 24 hours now. It seems like the worker was asked to provision... but it isn't? I'm not sure why not.

My master node is `atlas00` and the workers are `02, 04, 05, 06`, & `08`. I have a hdd attached to each worker mounted at `/mnt/eos<worker #/2 -1>` (so `atlas02` has a mounted drive at `/mnt/eos0` and `atlas04` has a mounted drive at `/mnt/eos1` and so on). Below is my PVC and SC description and list of pods for OpenEBS and my nodes. I'm not sure what all would be helpful here to troubleshoot this even. If you have any insight into what might be going on, I'd appreciate your wisdom - thanks!

```
root@atlas00:~# kubectl describe pvc eos-pvc-test
Name:          eos-pvc-test
Namespace:     default
StorageClass:  eos-raid10
Status:        Pending
Volume:
Labels:        <none>
Annotations:   volume.beta.kubernetes.io/storage-provisioner: openebs.io/provisioner-raw-block
               volume.kubernetes.io/storage-provisioner: openebs.io/provisioner-raw-block
Finalizers:    [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode:    Filesystem
Used By:       <none>
Events:
  Type    Reason                Age                     From                         Message
  ----    ------                ----                    ----                         -------
  Normal  ExternalProvisioning  3m16s (x5983 over 24h)  persistentvolume-controller  waiting for a volume to be created, either by external provisioner "openebs.io/provisioner-raw-block" or manually created by system administrator
root@atlas00:~# kubectl describe sc eos-raid10
Name:            eos-raid10
IsDefaultClass:  No
Annotations:     kubectl.kubernetes.io/last-applied-configuration={"apiVersion":"storage.k8s.io/v1","kind":"StorageClass","metadata":{"annotations":{},"name":"eos-raid10"},"parameters":{"cas.openebs.io/config":"- name: StorageType\n  value: \"raid-10\"\n- name: ReplicaAutoCreation\n  value: \"true\"\n","openebs.io/block-device":"[\n  {\"nodeID\": \"atlas01\", \"devicePath\": \"/dev/sda\"},\n  {\"nodeID\": \"atlas02\", \"devicePath\": \"/dev/sda\"},\n  {\"nodeID\": \"atlas03\", \"devicePath\": \"/dev/sda\"},\n  {\"nodeID\": \"atlas04\", \"devicePath\": \"/dev/sda\"}\n]\n","openebs.io/capacity":"10G","openebs.io/cstor-target-ips":"192.168.42.52,192.168.42.54,192.168.42.56,192.168.42.58","openebs.io/cstor-target-path":"/mnt/eos0,/mnt/eos1,/mnt/eos2,/mnt/eos3","openebs.io/jiva-replica-count":"4","openebs.io/jiva-replica-image":"quay.io/openebs/jiva:0.8.0","openebs.io/readwrite":"rw","openebs.io/storage-engine":"jiva","openebs.io/storage-pool":"default","openebs.io/volume-monitor":"true","targetPortal":"192.168.42.52:3260,192.168.42.54:3260,192.168.42.56:3260,192.168.42.58:3260"},"provisioner":"openebs.io/provisioner-raw-block"}

Provisioner:  openebs.io/provisioner-raw-block
Parameters:   cas.openebs.io/config=- name: StorageType
  value: "raid-10"
- name: ReplicaAutoCreation
  value: "true"
,openebs.io/block-device=[
  {"nodeID": "atlas01", "devicePath": "/dev/sda"},
  {"nodeID": "atlas02", "devicePath": "/dev/sda"},
  {"nodeID": "atlas03", "devicePath": "/dev/sda"},
  {"nodeID": "atlas04", "devicePath": "/dev/sda"}
]
,openebs.io/capacity=10G,openebs.io/cstor-target-ips=192.168.42.52,192.168.42.54,192.168.42.56,192.168.42.58,openebs.io/cstor-target-path=/mnt/eos0,/mnt/eos1,/mnt/eos2,/mnt/eos3,openebs.io/jiva-replica-count=4,openebs.io/jiva-replica-image=quay.io/openebs/jiva:0.8.0,openebs.io/readwrite=rw,openebs.io/storage-engine=jiva,openebs.io/storage-pool=default,openebs.io/volume-monitor=true,targetPortal=192.168.42.52:3260,192.168.42.54:3260,192.168.42.56:3260,192.168.42.58:3260
AllowVolumeExpansion:  <unset>
MountOptions:          <none>
ReclaimPolicy:         Delete
VolumeBindingMode:     Immediate
Events:                <none>
root@atlas00:~# kubectl get pods -n openebs
NAME                                              READY   STATUS    RESTARTS       AGE
openebs-ndm-z9vqb                                 1/1     Running   0              25h
openebs-cstor-cspc-operator-86b95cb7b-9gnst       1/1     Running   0              25h
openebs-ndm-ss88z                                 1/1     Running   0              25h
openebs-ndm-6vhwk                                 1/1     Running   0              25h
openebs-ndm-operator-698457995-ppp78              1/1     Running   0              25h
openebs-cstor-cvc-operator-bf7b8d559-lhl8c        1/1     Running   0              25h
openebs-ndm-5kjtk                                 1/1     Running   0              25h
openebs-cstor-csi-node-dwsgj                      2/2     Running   0              25h
openebs-cstor-csi-node-r8tc4                      2/2     Running   0              25h
openebs-ndm-x7cbh                                 1/1     Running   0              25h
openebs-cstor-csi-node-j59q9                      2/2     Running   0              25h
openebs-cstor-csi-node-xv9th                      2/2     Running   0              25h
openebs-cstor-admission-server-7d89bb76bf-bmd2m   1/1     Running   0              25h
openebs-cstor-csi-node-xt9jv                      2/2     Running   0              25h
openebs-localpv-provisioner-7576d8cbcf-6nbl8      1/1     Running   14 (18m ago)   25h
openebs-cstor-csi-controller-0                    6/6     Running   12 (18m ago)   25h
root@atlas00:~# kubectl get nodes
NAME      STATUS   ROLES                  AGE   VERSION
atlas08   Ready    worker                 26h   v1.26.3+k3s1
atlas06   Ready    worker                 26h   v1.26.3+k3s1
atlas04   Ready    worker                 26h   v1.26.3+k3s1
atlas00   Ready    control-plane,master   26h   v1.26.3+k3s1
atlas02   Ready    worker                 26h   v1.26.3+k3s1
root@atlas00:~# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.2 LTS
Release:        22.04
Codename:       jammy

```

# Answer
If users have a strict firewall setup on their Kubernetes nodes, the provisioning of a PV from a storageclass backed by a cStor storage pool may fail. The pool can be created without any issue and even the storage class is created, but the PVs may stay in pending state indefinitely. you may need to check the openebs-provisioner log.

Workaround:

This issue has currently only been observed, if the underlying node uses a network bridge and if the setting net.bridge.bridge-nf-call-iptables=1 in the /etc/sysctl.conf is present. The aforementioned setting is required in some Kubernetes installations, such as the Rancher Kubernetes Engine (RKE).

To avoid this issue, open the port 5656/tcp on the nodes that run the OpenEBS API pod. Alternatively, removing the network bridge might work.

NB: this solution is from the docs and the link is:https://openebs.io/docs/main/troubleshooting/volume-provisioning#persistent-volumes-indefinitely-remain-in-pending-state
> 1 votes

---
Tags: kubernetes, storage, k3s, distributed-systems
---


# Question
Title: Unable to pull image from a private Docker registry with Let's Encrypt certificate in Kubernetes
I have set up a private Docker registry on my Kubernetes cluster (k3s) running on a Raspberry Pi 4. I have secured the registry with a Let's Encrypt certificate, and I can confirm that the certificate is valid when accessing the registry from a web browser. However, when trying to deploy an application in Kubernetes that uses an image from this private registry, I am encountering an error related to the certificate.

```
Failed to pull image "192.168.42.50:5000/dev-backend:latest": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.42.50:5000/dev-backend:latest": failed to resolve reference "192.168.42.50:5000/dev-backend:latest": failed to do request: Head "https://192.168.42.50:5000/v2/dev-backend/manifests/latest": x509: certificate signed by unknown authority

```

I have created a Certificate resource in the "docker" namespace and successfully obtained a certificate from Let's Encrypt. The certificate request and its corresponding certificate can be seen with the following commands:

```
root@atlas00:~# kubectl get certificaterequests -n docker
NAME                        APPROVED   DENIED   READY   ISSUER             REQUESTOR                                         AGE
docker-registry-tls-hhm6j   True                True    letsencrypt-prod   system:serviceaccount:cert-manager:cert-manager   7h17m

root@atlas00:~# kubectl get certificates -n docker
NAME                  READY   SECRET                AGE
docker-registry-tls   True    docker-registry-tls   7h17m

```

The registry is using the Let's Encrypt certificate correctly, and I have even added the Let's Encrypt root certificate (ISRG Root X1) to the trusted certificate store for the container runtime (Docker) on each node of my cluster.

Here is my deployment YAML file:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: dev-backend
  namespace: dev
spec:
  replicas: 1
  selector:
    matchLabels:
      app: dev-backend
  template:
    metadata:
      labels:
        app: dev-backend
    spec:
      containers:
      - name: dev-backend
        image: 192.168.42.50:5000/dev-backend:latest
        ports:
        - containerPort: 80
        volumeMounts:
        - name: dev-backend-storage
          mountPath: /app/data
      volumes:
      - name: dev-backend-storage
        persistentVolumeClaim:
          claimName: dev-backend-pvc
      imagePullSecrets:
      - name: docker-registry-config

```

I have created a Kubernetes secret (docker-registry-config) containing the necessary credentials for accessing the private registry.

Despite these configurations, Kubernetes still reports the certificate as signed by an unknown authority. How can I ensure that Kubernetes trusts the Let's Encrypt certificate and successfully pulls the image from my private registry? Or is there a certificate authority that Kubernetes trusts out of the box? I'd like to avoid using an 'insecure-registry' approach. Thanks.

# Answer
I had two problems. @taleodor was correct though, I needed to connect using my domain name that I had setup my certificate for. I put a copy of this question on stackoverflow as well as it seemed there was a bigger body of kubernetes questions there already. I outlined the other problem here https://stackoverflow.com/questions/76146713/unable-to-pull-image-from-a-private-docker-registry-with-lets-encrypt-certifica?noredirect=1#comment134287667\_76146713 For this problem I had to find this in my app's deployment yaml:

```
image: 192.168.42.50:5000/dev-backend:latest

```

and change it to

```
image: <my domain>:5000/dev-backend:latest

```

And apply it of course. Then I had to make sure that port 5000 was port forwarded to my machine at 192.168.42.50 but that got me to a 401 (bad password error) which is basically a success.

You also want to make sure that the domain is resolving properly from the machines running k3s cluster with `nslookup <your domain>`. If not you probably have a problem with your `A` record.
> 1 votes

---
Tags: docker, kubernetes, docker-registry, k3s, tls
---


# Question
Title: Add more static instance labels in prometheus config
I'm adding serves to my prometheus config like this:

```
  - job_name: cluster1
    static_configs:
      - targets:
        - server1.example.com
        - server2.example.com
        - server3.example.com

```

but now I would like to label the instances with additional static labels, like the server1 and server2 with "database" and server3 with "mail" and maybe server1 additionally with "mysql" and server2 with "postgres".

I would then like to use this labels in grafana to filter on like here:

```
free_space_in_bytes{job=~'$job', server_type='database', database_type='mysql' }

```

Is this achievable within the prometheus config? Filtering on the jobname or instance is indeed possible, but I would like to extend this to more labels.

# Answer
> 1 votes
If I'm understanding your question, the answer comes straight out of the docs for `static_configs:`

> A static\_config allows specifying a list of targets and a common label set for them. It is the canonical way to specify static targets in a scrape configuration.

So,

```
 - job_name: cluster1
    static_configs:
      - targets:
        - server1.example.com
        - server2.example.com
        - server3.example.com
        labels:
          database: mysql
          some_carefully_chosen_property: its_value_for_these_targets

```

If you want to play around with a `prometheus.yml` in your local directory, to check syntax, you can run

```
docker run -it --rm --name prometheus_configcheck \
  -v ${PWD}/prometheus.yml:/prometheus.yml \
  --entrypoint /bin/promtool \
  prom/prometheus:v2.40.6 \
  check config /prometheus.yml

```

if you don't have prometheus installed locally, or

```
promtool check config prometheus.yml

```

if you do, and it came with promtool, which it should have.

**However** Please do not do this in general and certainly not until you fully understand the issue of cardinality, and why high cardinality will overload your system, and is unnecessary in light of PromQL. You can destroy your setup by choosing either a) too many labels or b) labels with a high or non-bounded set of values, since each combination of labels is stored as a separate metric stream. This article explains it well https://grafana.com/blog/2020/08/27/the-concise-guide-to-labels-in-loki/ (it's about Loki, but labels work in the same way in Prometheus).

Also, your examples for your label values indicate that you may not have identified what property (key) you're talking about. e.g. is it `server_type`? Saying that your label is `mysql` doesn't make sense; the only values would then be `true` or `false`. Try to find out what properties really describe your infrastructure, then don't label things with them, and use them in PromQL queries instead, or add them sparingly as labels.

---
Tags: prometheus, grafana
---


# Question
Title: Why would you set `internalTrafficPolicy: Local` on a Kubernetes `Service`?
What is the use case for the service object's internalTrafficPolicy property?

If my understanding is correct, then when set to `Local`, traffic that arrives at a node, from another node, that is destined for the service's IP address will be dropped.

Does this mean that a pod running on node 1 would never able to talk to a such a service backed by a pod running on node 2? Why would one want this to be the case?

I guess if the pods backing a service were managed with a `DaemonSet` then it could be used to prevent traffic for the service ever going over the cluster network as it would always be handled internally to the node, or dropped... are there any other reasons one would ever want to use this property?

# Answer
> 1 votes
> I guess if the pods backing a service were managed with a DaemonSet then it could be used to prevent traffic for the service ever going over the cluster network as it would always be handled internally to the node, or dropped...

Seems like it according to KEP-2086: Service Internal Traffic Policy which states:

> **Motivation**
> Internal traffic routed to a Service has always been randomly distributed to all endpoints. This KEP proposes a new API in Service to address use-cases such as node-local and topology aware routing for internal Service traffic.

and

> **Story 1**
> As a platform owner, I want to create a Service that always directs traffic to a logging daemon or metrics agent on the same node. Traffic should never bounce to a daemon on another node since the logs would then report an incorrect log source.

---
Tags: kubernetes
---


# Question
Title: How widespread, used is CWL, common workflow language?
Trying to get an overview of different languages and software for workflows and data pipelines--and thus 'data engineering'--I stumbled upon Common Workflow Language. I thought, well, how common is this? - Got to ask the community, and I hope DevOps is appropriate.

Does someone know how used this language is? Perhaps compared to Apache Airflow or other tools for workflows.

# Answer
This question is almost certainly going to be closed "off-topic", but since I have some experience with communities using CWL in the past, I'll provide my bit.

First, a correction to your premise: CWL is a *standard*, which makes it different to most other workflow tools. It has several implementations. One of the main benefits is that it's platform-agnostic, making workflows highly-portable.

As you can see from their website, the user community of this standard is heavily biased towards research communities. Implementations of the standards are de-facto default toolchains in several smaller communities, mostly in the bioinformatics domain, where workflows can become quite complex.

There is a strong incentive to publish workflows so that

1. workflows can be attributed correclty
2. reproducibility can be tested.

If you want to use Apache Airflow, there's an implementation of CWL for that
> 1 votes

---
Tags: pipeline, workflow, apache-airflow
---