# Question
Title: How to delete an existing label in a deployment with helm upgrade
I have an existing deployment that has the label `importance: normal` in spec/template/metadata/labels (all the pods spawned from this deployment have that label in them).

The current deployment yaml does NOT have that `importance: normal` label in it. It was in an older version of the deployment yaml.

So everything that was deployed with the original deployment yaml has this importance label in it. After we perform a `helm upgrade --install` and use the latest deployment yaml(which does not have this label) and when we look at the deployment in the cluster is still has this label in it. The deployment does have the changes we made but it does not delete this old label.

I want to be able to remove that label when a helm upgrade is performed.

I tried the following trying to use the --set flag but have had no luck.

Command I tried - which did nothing:

`helm upgrade --install echo service-standard/service-standard --namespace qa --set-string spec.template.metadata.labels.importance="" -f ./helm-chart/values.shared.yaml --wait --timeout 600s`

I also tried using `--reset-values` and that also had no effect.

Here is the snippet of the deployment that I am trying to remove the label from - The label is in the first spec block (not the second) right before `app: echo-selector` :

```
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "8"
  creationTimestamp: "2022-12-14T15:24:04Z"
  generation: 9
  labels:
    app.kubernetes.io/managed-by: Helm
  name: echo-deployment
spec:
  replicas: 2
  revisionHistoryLimit: 5
  template:
    metadata:
      annotations:
        linkerd.io/inject: enabled
      creationTimestamp: null
      labels:
        app: echo-selector
        importance: normal
        version: current
    spec:
      containers:
      - env:
        - name: TEST

```

We do not want to have to delete the deployment and re install it (which would remove the label). Any help or advice is greatly appreciated!!!!

# Answer
> 1 votes
After double-checking that the labels is not really there using `helm template ...`, you maybe try using `helm upgrade --force ...` so it will replace the resource and use the new labels.

---
Tags: containers, helm, kubernetes-helm
---


# Question
Title: helm check if key exist and then evaluate if it's value is true on single line condition
"helm check if key exist and then evaluate if it's value is true on single line condition" In Helm version 3.10 will the following condition below work as intended or I can achieve it in some other way?

```
{{- if and (hasKey .Values.enableJsConfigMap) (eq .Values.enableJsConfigMap.enabled true) }}

```

# Answer
This wouldn't work. The second condition will be evaluated regardless of the first. So if the key is not defined there will be an error. Use `default` instead:

```
{{- if eq (default .Values.enableJsConfigMap false) true }}
# Do something if enableJsConfigMap exists and its value is true
{{- end }}

```
> 3 votes

---
Tags: helm, kubernetes-helm
---


# Question
Title: Docker Engine not seeing installed Nvidia runtime
I am using Ubuntu 22.04, I installed docker and docker desktop.

After installing the Nvidia docker runtime via the Nvidia docker toolkit, Nvidia runtime is not visible after system restart. I updated the Docker Json file to add the run time and it still did not work.

Nvidia runtime was visible after installation but prior to system restart but now it is not seeable under docker info and containers which use it fail to run as it's an unknown runtime.

I tried reinstalling the Nvidia runtime but it won't come up still.

# Answer
You may have to enable Docker to start after system restart using,

```
sudo systemctl enable docker

```
> 0 votes

---
Tags: docker, ubuntu
---


# Question
Title: What is a "single kubectl list object" as mentioned in jsonnet documentation?
The jsonnet documentation \> Using Jsonnet With Kubernetes says:

> Use YAML stream output, multi-file output, or a **single kubectl list object**. This latter option is provided by Kubernetes without requiring any special support from Jsonnet. It allows us to **group several Kubernetes objects into a single object**. This is the most popular option because it does not require intermediate files, and other tools don't always reliably support YAML streams.

YAML streams is putting several resources separated with `---` , multi-file output is self-explanatory, but I never heard of a "kubectl list object".

If I google it I only get  and  and they don't really explain what is a "kubectl list object".

So, What is a **single kubectl list object** in this context?

# Answer
> 1 votes
The folowing answer comes from a conversation on the Kubernetes #kapitan channel with @ademaria and Types(kinds).

"kubectl list object" is a kubectl client-only feature (see ):

> Note that kubectl and other tools sometimes output collections of resources as kind: List. Keep in mind that **kind: List is not part of the Kubernetes API**; it is exposing an implementation detail from **client-side code** in those tools, **used to handle groups of mixed resources**.

you can write

```
apiVersion: v1
kind: List
items:
- apiVersion: v1
  kind: Pod
  ...
- apiVersion: v1
  kind: ConfigMap

```

and you can `kubectl apply -f xxx.yaml` that and kubectl will at the client side convert that before applying it via the Kubernetes API (which does not have any `kind:List`,etc)

So the jsonnet documentation is refering to the fact that you can create a `v1/List` object (**a single kubectl list object**) with all the resources inside and pass that to kubectl, the same way you will pass a yaml stream.

It's list object or list resource but since it's only client-side they emphasize that by saying its a **kubectl list object**

---
Tags: kubernetes, kubectl
---


# Question
Title: Is it possible to template JIRA issues with pre-filled field contents?
In first place I look for default JIRA functionality e.g. without 3rd party plugins to simlify and improve the issue workflow.

Examples:

* Simple: Template for an acceptable bug report
* Probably, less simple: Set specific keywords if issue is created by a specific user e.g. "ops" for issues created by myself because I am currently in an Ops team. :-)

# Answer
I don't think you will be able to get far without any plugins. What you can take a look at is the *Default Values for 'Create Issue' screen* plugin. We were using it in the past, however gave up because defaults could be set only per issue type, and no other conditions. I was in contact with the supplier, though, and they confirmed that defaults per user and other extensions were on their roadmap. Makes sense to check if they are already available.
> 6 votes

# Answer
You should be able to achieve this using the Automation for JIRA if you are a Project Admin.

Less simple one - Maybe something like

1. When new issue is created
2. Check if condition for specific users
3. Edit the issue and set certain fields to desired values
> 0 votes

---
Tags: templating, jira
---


# Question
Title: Trying to Create a Docker Ansible lab and Cant SSH
I'm trying to setup a mini ansible lab with docker containers. I have 1 Rockylinux based container as the ansible controller, and then (for right now) 1 Rockylinux based container as the host.

For ansible to work, it needs to be able to ssh into the host. I've installed opensh and configured for root login and I'm still getting connection refused whenever I try.

Any config that I'm missing?

my Dockerfiles are pretty basic. Here is the file for the ansible server:

```
FROM rockylinux:8.8
RUN dnf update -y && \
dnf install python38 -y && \
dnf install ansible-core -y &&\
dnf install openssh -y && \
dnf install openssh-server -y && \
dnf install vim -y && \
dnf install net-tools -y
EXPOSE 22

```

the "host" server is the same except there isn't ansible-core installed

any reason why I can't ssh from one container to the other?

# Answer
I'm not going to answer why you can't ssh to your container. There might be a dozen reasons which are impossible to guess with as little information provided in your question. Moreover, installing ssh inside a container is usually not a good idea anyway.

In fact, your assumption is wrong:

> For Ansible to work, it needs to be able to ssh into the host

Actually, for Ansible "to work" i.e. to be able to control targets from a controller, you need a way to connect to the target. And even if ssh is the most common, it's not the only one. There's actually a specific connection plugin to connect to docker containers. The only real prerequisite is to have python inside the target.

There's also a dynamic inventory plugin that will collect all hosts running on a specific docker daemon.

If you really want to create a lab running entirely inside docker and with Ansible running in one of the containers, this is what I suggest as a basic scaffold. You can adapt to your exact needs

Project structure:

```
$ tree lab
lab
├── ansible-lab
│   ├── dummy_playbook.yml
│   └── inventories
│       └── default
│           └── hosts_docker.yml
├── compose.yml
└── images
    ├── controller
    │   └── Dockerfile
    └── target
        └── Dockerfile

6 directories, 5 files

```

I used the same base image as in your example. Here is `images/controller/Dockerfile` for the Ansible controller (side note: please refrain from installing Ansible through your package manager and use pip which is now the only officially supported method)

```
FROM rockylinux:8.8

WORKDIR /ansible-lab

RUN \
  dnf update -y && \
  dnf install -y dnf-plugins-core && \
  dnf config-manager --add-repo https://download.docker.com/linux/centos/docker-ce.repo && \
  dnf install -y python3.8 docker-ce-cli bash-completion && \
  python3 -m pip install --upgrade pip && \
  # Note: the docker python lib is required for docker connection and inventory plugins
  pip install ansible docker

```

This is the `images/target/Dockerfile` for the target host(s) we will manage

```
FROM rockylinux:8.8
RUN \
  dnf update -y && \
  dnf install -y python3.8

```

The `compose.yml` file puts everything together:

```
---
services:
  controller:
    image: rocky_ansible_controller:local
    build: images/controller
    container_name: lab-controller
    command: tail -f /dev/null
    volumes:
      # Access our ansible project which can be edited from the host if needed
      - ./ansible-lab:/ansible-lab
      # We mount the host docker socket so we can access docker containers
      # running there from the container with the docker cli
      - /var/run/docker.sock:/var/run/docker.sock

  host:
    image: rocky_ansible_target:local
    build: images/target
    command: tail -f /dev/null

```

Let's build and start our project with a single target host for now:

```
$ cd lab
$ docker compose build
[... lots of build output cut out ...]

$ docker compose up -d
[+] Running 3/3
 ✔ Network lab_default       Created    0.0s
 ✔ Container lab-host-1      Started    1.0s
 ✔ Container lab-controller  Started    1.1s

```

Now here is our basic inventory `ansible-lab/inventories/default/hosts_docker.yml` using the docker-container inventory plugin (see link above- important: it must end with '\_docker.yml' as explained in the documentation)

*Note that the `lab-` prefix for containers is taken from the directory name holding the compose project. If your directory has an other name than `lab`, you'll have to adapt below or to set the project name yourself.*

```
---
plugin: community.docker.docker_containers
docker_host: unix://var/run/docker.sock
connection_type: docker-cli
compose:
  # All host will be accessed through docker
  ansible_connection: docker
  # Don't use platform python but the one specifically installed
  ansible_python_interpreter: "'/usr/bin/python3'"
groups:
  # This creates a lab_hosts group for all hostnames starting with 'lab-host-'
  lab_hosts: inventory_hostname is match('^lab-host-.*$')

```

And here is the basic `ansible-lab/dummy-playbook.yml` which will simply install a new package on the hosts:

```
---
- name: Dummy playbook for lab
  hosts: lab_hosts

  tasks:
    - name: Install a package on host
      ansible.builtin.dnf:
        name: bind-utils
        state: present

```

We can run this playbook using the inventory from the controller (note that the ansible files are mounted directly inside the container working directory, see the compose file)

```
$ docker compose exec controller ansible-playbook -i inventories/default dummy_playbook.yml

PLAY [Dummy playbook for lab] ********************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************
ok: [lab-host-1]

TASK [Install a package on host] *****************************************************************************************************************************************************************************************************
changed: [lab-host-1]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************
lab-host-1                 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

```

Let's start some more target just to show it's working as intended and scales

```
docker compose up -d --scale host=3
[+] Running 4/4
 ✔ Container lab-host-1      Running    0.0s
 ✔ Container lab-host-3      Started    0.5s
 ✔ Container lab-host-2      Started    0.6s
 ✔ Container lab-controller  Running    0.0s

$ docker compose exec controller ansible-playbook -i inventories/default dummy_playbook.yml

PLAY [Dummy playbook for lab] ********************************************************************************************************************************************************************************************************

TASK [Gathering Facts] ***************************************************************************************************************************************************************************************************************
ok: [lab-host-3]
ok: [lab-host-2]
ok: [lab-host-1]

TASK [Install a package on host] *****************************************************************************************************************************************************************************************************
ok: [lab-host-1]
changed: [lab-host-2]
changed: [lab-host-3]

PLAY RECAP ***************************************************************************************************************************************************************************************************************************
lab-host-1                 : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
lab-host-2                 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
lab-host-3                 : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

```

Of course you'll want to do many more things with those containers. But at any time you can simply restart from scratch. Just shut down everything and start over:

```
$ docker compose down
[+] Running 5/5
 ✔ Container lab-host-1      Removed    10.6s
 ✔ Container lab-host-2      Removed    10.5s
 ✔ Container lab-host-3      Removed    10.4s
 ✔ Container lab-controller  Removed    10.5s
 ✔ Network lab_default       Removed    0.5s

```
> 0 votes

---
Tags: docker, ansible
---


# Question
Title: Ansible inventory for restart on worker node
I looking to execute a simple reboot via Ansible using its playbook

```
- name: Worker-node Reboot
  hosts: node1
  gather_facts: no
  remote_user: team
  become: true

  tasks:
    - name: Reboot the machine (Wait for 5 min)
      reboot:
        reboot_timeout: 300

```

How do I specify reboot for a worker node in Ansible's inventory for a Slurm queue when the playbook is executed outside the head-node?

# Answer
> 0 votes
Specifying which node you want to reboot is done using a targetting pattern. This can be a single node or a group of nodes defined in the inventory. In your case, you are targetting only `node1`.

You need to execute the Ansible playbook from somewhere. It's not quite clear from your question whether the Slurm head node is also the Ansible controller (the machine which is executing `ansible-playbook`.

Let's assume that is the case - that the Ansible controller is also the Slurm head node. As such, your playbook as it currently stands will work. Assuming that the head node / Ansible controller has access to the inventory, it targets a single node and reboots it.

Now, let's consider this case:

> ... when the playbook is executed outside the head-node...

In this case, the Ansible controller is no longer the head node, but some other machine. This could be *any* machine, even a machine outside of the Slurm cluster. It could also be one of the worker nodes itself. The only constraints are:

1. It must have an environment that can run `ansible-playbook`
2. It must have access to the inventory
3. It must have permission to connect to the targetted host, and the user connecting must have `sudo` rights (since you use `become`).
4. And of course, you need some way of triggering it...

If these conditions are satisfied, you can use the exact same playbook.

However, since you're only executing a single task, I would suggest not running a playbook at all, but an ad-hoc task. The documentation even has an example for your exact use case:

```
$ ansible node1 -u team -b -a "/sbin/reboot"

```

If you want to use the `reboot` module:

```
ansible node1 -u team -b -m reboot -a "reboot_timeout=300"

```

If your inventory is in a file, you can pass it with the `-i` flag.

---
Tags: ansible, deployment, ansible-inventory
---


# Question
Title: Error MSB4226: Microsoft.WebApplication.targets" was not found
I was wondering if someone has ran into this problem as I have tried everything to fix. I have a DevOps build pipeline in Azure running visual studio 2015 version (which matches my local machine) and I am running into this error

**Error MSB4226: The imported project "C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\Microsoft\VisualStudio\v14.0\WebApplications\Microsoft.WebApplication.targets" was not found**

It builds perfectly fine on my machine. I have no idea why I cannot get this to build fine on DevOps.. and I haven't seen many questions about it. I went ahead and changed the vb.proj file to reflect the following and it still did not work

    <VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">14.0</VisualStudioVersion>

    <VSToolsPath Condition="'$(VSToolsPath)' == ''">$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v$(VisualStudioVersion)</VSToolsPath>

  </PropertyGroup>
  <Import Project="$(MSBuildBinPath)\Microsoft.VisualBasic.targets" />
  <Import Project="$(VSToolsPath)\WebApplications\Microsoft.WebApplication.targets" Condition="'$(VSToolsPath)' != ''" />
  <Import Project="$(MSBuildExtensionsPath32)\Microsoft\VisualStudio\v14.0\WebApplications\Microsoft.WebApplication.targets" Condition="false" />```

# Answer
It is possible that your project is referring to VS 2017 somewhere. Double-check all your solution and proj files for any reference to VS 2017.

Try to enable debug mode and gather more info.
> 0 votes

---
Tags: azure-devops, azure, azure-pipelines
---


# Question
Title: Why do I get different responses either when I use `sudo` before `kubectl` or not?
For example when I use `kubectl get pods` the response is:

```
No resources found in default namespace.

```

But when I use `sudo kubectl get pods` the result is:

```
E1211 12:01:15.409201   50965 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1211 12:01:15.409569   50965 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1211 12:01:15.411086   50965 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1211 12:01:15.412633   50965 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
E1211 12:01:15.414189   50965 memcache.go:238] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp 127.0.0.1:8080: connect: connection refused
The connection to the server localhost:8080 was refused - did you specify the right host or port?

```

PS: From my previous knowledge I think when I use `kubectl` without `sudo`, it refers to `192.168.1.2:6443` but with the `sudo` word it is looking for `localhost:8080`. I don't know why?

# Answer
> 2 votes
I think you haven't configured kubeconfig in root user.

**The sudo command allows you to run programs with the security privileges of another user.**

When you run `sudo kubectl` you're running `kubectl` with root user and most probably you might have not configured kubeconfig for root user.

# Answer
> 0 votes
This is only loosely related to your question, but if `kubectl get pods` gives:

```
No resources found in default namespace.

```

check `kubectl get namespaces` and then re-run the query with `kubectl get pods -n <namespace>`

---
Tags: kubernetes, kubectl, ports
---


# Question
Title: Deleted service connection in Azure DevOps and now cannot select a new one
I deleted what I thought were unused service connections in Azure DevOps, and now I cannot change the connection on existing pipelines. The build pipelines are yaml based but the service connections were set in the UI, so the yaml doesn't contain AzureSubsciption definitions. If I try to run the build pipeline I get this:

If I try to edit the pipeline, I get this:

I have another service connection (GitHub App) that I have made available to all pipelines. Is there any way to fix the existing pipelines without having to re-create them? Here is the current yaml file for the builds:

```
 pool:
  vmImage: 'ubuntu-latest'

variables:
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

steps:
- script: 'wget https://archive.apache.org/dist/maven/maven-3/3.8.6/binaries/apache-maven-3.8.6-bin.zip'

- task: ExtractFiles@1
  inputs:
      archiveFilePatterns: 'apache-maven-3.8.6-bin.zip'
      destinationFolder: '$(build.sourcesdirectory)/maven'

- task: Cache@2
  inputs:
    key: 'maven | "$(Agent.OS)" | **/pom.xml'
    restoreKeys: |
      maven | "$(Agent.OS)"
      maven
    path: $(MAVEN_CACHE_FOLDER)
  displayName: Cache Maven local repo

- task: DownloadSecureFile@1
  inputs:
    secureFile: 'settings.xml'
  displayName: Download settings.xml file

- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
    publishJUnitResults: false
    javaHomeOption: 'JDKVersion'
    mavenVersionOption: 'Path'
    mavenPath: '$(build.sourcesdirectory)/maven/apache-maven-3.8.6'
    mavenAuthenticateFeed: true
    effectivePomSkip: false
    sonarQubeRunAnalysis: false
    goals: 'test'
  displayName: execute munit tests

- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
    publishJUnitResults: false
    javaHomeOption: 'JDKVersion'
    mavenVersionOption: 'Path'
    mavenPath: '$(build.sourcesdirectory)/maven/apache-maven-3.8.6'
    mavenAuthenticateFeed: false
    effectivePomSkip: false
    sonarQubeRunAnalysis: false
    goals: 'package -DskipTests'
  displayName: package solution

- task: PublishPipelineArtifact@1
  inputs:
    targetPath: $(System.DefaultWorkingDirectory)
    artifact: 'drop'
  displayName: Publish deployment artifacts

```

# Answer
1. When you define a task in your YAML pipeline, you can specify the service connection to use within the task's configuration. This is done through the inputs section of the task definition.

task: AzureResourceGroupDeployment@2 inputs: azureSubscription: 'Your Azure Service Connection' ...

2.The service connections that are available in your Azure DevOps project settings are the ones that can be used in your pipeline. Ensure that the required service connection is properly defined in your Azure DevOps project before referencing it in your pipeline YAML.
> 0 votes

---
Tags: azure-devops, azure-pipelines, yaml
---


# Question
Title: How do I refer in Azure Pipelines to an output variable defined in a previous stage for defining stage conditions and new variables?
I'm writing an Azure Pipeline where I use the `targetBranchName` of a pull request to define the test environment I'm going to use. Since `targetBranchName` has no value before execution, I created a "prepare" stage where I capture the value of the variable to use in further stages.

```
- stage: prepare
  jobs:
  - job: variables
    steps:
    - checkout: none
    - powershell: Write-Output "##vso[task.setvariable variable=TARGET;isOutput=true;isReadOnly=true]$(system.pullRequest.targetBranchName)"

```

Then, in a later stage, I want to use the output variable `BRANCH_NAME` to find whether the stage should run and if so to load the appropriate group variables. I also would like to capture the value of the output variable in a job variable so that I can refer to it with a more "convenient" name.

The stage condition seems to be working when tested with manually set values for `TARGET` in the `prepare` job, but I can't actually see the value of `TARGET`, so I added a "print" step and it shows no value for `TARGET`.

I'm not shure about where is my mistake, if in how I refer to the output variable to copy its value (`stage` vs. `stageDependencies`) or if the syntax for recovering the value of a variable in `dependencies`/`stageDependencies` (`$()`, `$[]` or `${{}}`).

```
- stage: deploy_to_cint
  dependsOn: prepare
  condition: and(succeeded('prepare'), in(dependencies.prepare.outputs['variables.environment.TARGET'], 'env1', 'env2'))
  variables:
  - name: TARGET
    value: $(dependencies.prepare.outputs['variables.environment.TARGET'])
  - group: 'EnvironmentCredentials-$(TARGET)'
  jobs:
  - job: debug
    steps:
    - bash: echo This is running for $(TARGET).

```

# Answer
Refer : https://learn.microsoft.com/en-us/azure/devops/pipelines/process/set-variables-scripts?view=azure-devops&tabs=bash#set-an-output-variable-for-use-in-future-stages

Here is a working example :

```
stages:
- stage: A
  jobs:
  - job: A1
    steps:
     - bash: echo "##vso[task.setvariable variable=myStageVal;isOutput=true]this is a stage output variable"
       name: MyOutputVar
- stage: B
  dependsOn: A
  jobs:
  - job: B1
    variables:
      myStageAVar: $[stageDependencies.A.A1.outputs['MyOutputVar.myStageVal']]
    steps:
      - bash: echo $(myStageAVar)

```
> 0 votes

---
Tags: azure-pipelines, yaml, variables
---


# Question
Title: How to prevent user who does not have approval rights to click redeploy if approved task fail in release?
In a release pipeline, we have a post deployment approval in a stage 1 task which triggers the stage 2 task. In the Approval Policies, we have checked the box: "The user requesting a release or deployment should not approve it". Once the stage 1 task has been approved by someone other than the user requesting the release, the stage 2 task is deployed and executes. However, if the stage 2 task fails, how can we prevent the user requesting a release or deployment from clicking the redeploy button? Currently, the redeploy button can be clicked by anyone.

# Answer
> 0 votes
Check if this helps : https://learn.microsoft.com/en-us/azure/devops/pipelines/release/approvals/approvals?view=azure-devops#post-deployment-approvals

Add a post deployment approval.

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: How can I run a script in an Azure WebApp container after deployment as part of an Azure DevOps Pipeline using the classic pipelines interface?
My knowledge of Azure is pretty limited - this is the first project where I've worked with it.

We are deploying Azure webapps in docker containers via an Azure pipeline configured with the classic interface. The pipeline has one agent job that looks like this when I click the "View YAML" button:

```
pool:
  name: Azure Pipelines
steps:
- task: Docker@0
  displayName: 'Build an image'
  inputs:
    azureSubscription: '<omitted>'
    azureContainerRegistry: '{"loginServer":"<omitted>", "id" : "<omitted>"}'
    dockerFile: Dockerfile
    includeSourceTags: true
    includeLatestTag: true

- task: Docker@0
  displayName: 'Push an image'
  inputs:
    azureSubscription: '<omitted>'
    azureContainerRegistry: '{"loginServer":"<omitted>", "id" : "<omitted>"}'
    action: 'Push an image'
    includeSourceTags: true
    includeLatestTag: true

```

The pipeline isn't really "mine" so I don't want to make any crazy changes but I do have access to edit it and can discuss with the third party admin to make changes if need be.

I need to run a post deployment script to clear caches and such in a container (the app will likely run multiple web containers, it doesn't matter which one the script runs in) after the pipeline finishes.

Unfortunately I just cannot figure out how to do that. Options I've considered:

* Run it in the container's startup script: containers will also be spun up when the app scales and we don't want to clear the caches in that case
* Use a container job \- this seems to be *exactly* what I want but unfortunately is not supported in the classic editor. I'd prefer not to rework the devops pipeline entirely since, as mentioned, it's not really "mine"
* Add a "Docker" task of version 0 with the "Run a Docker command" action and set it to run the "docker exec" command. This seems the most promising but unfortunately there isn't any way I can tell to collect the ID or a name of a just-pushed container (which is a required argument to docker exec). I don't think there's an option to set a name that I can use with docker exec either (I see I can set several tags in the build and push steps but on the image only?), but I'm not too familiar with docker exec either so not 100% sure of that.
* Add an Azure CLI task that SSHs in to the container and runs a command. Unfortunately it seems like there's no easy way to directly execute a command over Azure's SSH (e.g. via `az webapp ssh` or `az webapp create-remote-connection`) so I'm not totally sure whether this would work at all and I suspect it would take many hours of debugging and waiting for containers to deploy to even find out.

# Answer
a. In the Azure DevOps portal, go to the "Pipelines" section.

b. Click on "Releases" and then create a new release pipeline.

c. Define your release pipeline stages for deploying your Azure Web App.

d. In the stage where you want to run the post-deployment script, add an "Azure App Service Deployment" task.

e. Configure the task with your Azure subscription and select your Web App.

f. In the "Additional Deployment Options" section, you can specify a "Script" to run post-deployment. Provide the path to your script within your repository.

For example, if your script is in a folder called "scripts" at the root of your repository, you would specify the "Script" field as $(System.DefaultWorkingDirectory)/scripts/your-script.sh for a Bash script.

g. Complete the task configuration and save your pipeline.
> 0 votes

---
Tags: azure-devops, azure, azure-pipelines
---


# Question
Title: In a YAML file how do I checkout a specific branch in different repo and fallback if it doesn't exist using Azure DevOps?
I have a project I need to build in ADO that contains code in two different repos. Usually there is a matching release branch in both, but occasionally not. I would like to get the matching value and automatically fallback to main if there's no matching.

I was told instead to make the other repo a nuget package, or just compile to a dll, but those both seem like poor solutions when VS2022 now supports multi-repo solutions pretty cleanly.

# Answer
```
steps:
- powershell: |
    $releaseBranch = "$(ReleaseBranchVariable)"  # Replace with the name of the release branch variable
    $mainBranch = "main"  # Specify the name of the main branch

    # Check if the release branch exists in the repository
    if (git branch --list --remote origin/$releaseBranch) {
        $branchToUse = $releaseBranch
    } else {
        Write-Host "Release branch not found, falling back to main branch."
        $branchToUse = $mainBranch
    }

    Write-Host "Using branch: $branchToUse"

    # Check out the selected branch
    git checkout $branchToUse
  displayName: 'Select Branch'

```

This script checks if the release branch exists in the repository. If it does, it uses the release branch; otherwise, it falls back to the main branch.
> 0 votes

---
Tags: azure-devops, azure, azure-pipelines
---


# Question
Title: View cross project Area Path during Task creation
I work on multiple projects, and for a common overview of my Work Items I use Cross Project Queries.

Now the problem is, if I create a new Work Item (within that Querie view), per default the Work Item "Area" and "Iteration" is the one from the current project. In the drop-down menu of "Area" and "Iteration" within the task creation window, the other projects are not listed and can't be searched.

Currently, I have to manually switch to the related project, to create the WIT, or "Move to Team project,... " after creating the WIT in the wrong project.

I am the Organization Owner so it should not be a permission problem. I am also part of every team in every project.

Does anybody know if I missed something, or is this feature just not implemented in the Task Creation (only afterwards via three dots, "Move to Team project,... ").

Thanks

# Answer
> 0 votes
This feature is currently unavailable. You will have to use the "Move to Team project" option.

---
Tags: azure-devops
---


# Question
Title: How do I access custom artifact properties in Azure Devops pipelines?
I've been using Azure Pipelines, but I'm stuck trying to find any examples or documentation on this one point.

The `PublishPipelineArtifact@1` documentation lists a `properties` input that you can attach custom properties to... however, I can't find any documentation on how to retrieve or use these properties elsewhere. There isn't anything in the documentation itself, nor in the equivalent download task. These properties don't appear to be reflected in the UI at all either.

Bing GPT seems to think that there should be a generated `metadata.json` file, but from simple tests no such file is created, and when pressed it acknowledges its references don't actually support what it's claiming.

So how do I reference or use these custom properties?

# Answer
> 0 votes
I have not tried this myself, but I think you could use it like:

```
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Build.ArtifactStagingDirectory)'
    artifactName: 'my-artifact'
    publishLocation: 'pipeline'
    properties: |
      {
        "user-MyCustomProperty1": "Value1",
        "user-MyCustomProperty2": "Value2"
      }

```

You could access this in the downstream tasks like:

```
- job: DownloadAndProcessArtifact
  displayName: 'Download and Process Artifact'
  steps:
  - script: echo "Downloading artifact with MyCustomProperty1=Value1"
    displayName: 'Download Artifact 1'
    condition: and(succeeded(), eq(dependencies.PublishPipelineArtifact.outputs['stepName']['user-MyCustomProperty1'], 'Value1'))

  - script: echo "Downloading artifact with MyCustomProperty1=Value2"
    displayName: 'Download Artifact 2'
    condition: and(succeeded(), eq(dependencies.PublishPipelineArtifact.outputs['stepName']['user-MyCustomProperty1'], 'Value2'))

```

# Answer
> 0 votes
You can use the Azure Devops build artifacts REST API to view the artifact properties e.g. https://dev.azure.com/{organization}/{project}/\_apis/build/builds/{buildid}/artifacts?api-version=7.2-preview.5

This returns a list of artifacts associated with the build and also lists custom properties you have added e.g.

```
        "id": 47658199,
        "name": "SmashedManifest-Default",
        "source": "5e604834-27dc-574c-c96d-2ebcf0a82dfd",
        "resource": {
            "type": "PipelineArtifact",
            "data": "651C4A252ABE81ECBA976C3C0043C9033202AF840E979BEF1D9C589AAADDFFGGEE",
            "properties": {
                "user-AssetRetentionType": "drop"

```

In this case, the user-AssetRetentionType was added as a custom property.

HTH

# Answer
> 0 votes
From CHATGPT

To access the metadata.json file and its content, you can use a script task in your pipeline.

* powershell: |

# Specify the path to the artifact

$artifactPath = "$(Pipeline.Workspace)/"

# Construct the path to the metadata file

$metadataPath = Join-Path $artifactPath "metadata.json"

# Check if the metadata file exists

if (Test-Path -Path $metadataPath -PathType Leaf) { \# Read the contents of metadata.json $metadataContent = Get-Content $metadataPath | ConvertFrom-Json Write-Host "Metadata Content: $($metadataContent | ConvertTo-Json -Depth 100)"

```
  # Access specific properties in the metadata
  $customProperty = $metadataContent.MyCustomProperty
  Write-Host "Custom Property: $customProperty"

```

} else { Write-Host "Metadata file not found." } displayName: 'Access Metadata'

Apparently, the PublishPipelineArtifact task in Azure DevOps does not natively provide a way to directly retrieve or use custom properties set with the properties input elsewhere in your pipeline. The properties input is primarily used for associating metadata with the published artifacts, but retrieving and using these properties require some custom scripting and logic.

To retrieve and use these custom properties in your pipeline, you can create a script task (e.g., PowerShell, Bash) that parses the metadata associated with the published artifact.

* powershell: | $artifactMetadata = Get-Content $(Build.ArtifactStagingDirectory)/myArtifact/metadata.json | ConvertFrom-Json $customProperty = $artifactMetadata.MyCustomProperty Write-Host "Custom Property: $customProperty" displayName: 'Retrieve Custom Properties'

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: Why is Azure Repos Git Checkout sometimes returning a Status code 128?
We've migrated from a on premise GitBlit to Azure DevOps last week.

Our builds are now suffering from periodic error while cloning the git repo. See example output. Main error is probaby **returned status code 128**

We are doing quiet a number of builds a day, and each build has multiple checkouts of different git repositories.

Anybody has a clue?

> ---
>
> 04:51:50 ERROR: Error cloning remote repo 'origin' 04:51:50 hudson.plugins.git.GitException: Command "C:\jenkins\_opt\Git\bin\git.exe fetch --tags --force --progress --depth=1 -- https://company@dev.azure.com/company/GLIMS/\_git/glims\_glims-bin +refs/heads/*:refs/remotes/origin/*" returned status code 128: 04:51:50 stdout: 04:51:50 stderr: warning: ----------------- SECURITY WARNING ---------------- 04:51:50 warning: | TLS certificate verification has been disabled! |
>
> ---
>
> error: RPC failed; curl 56 Recv failure: Connection was reset 04:51:50 fetch-pack: unexpected disconnect while reading sideband packet 04:51:50 fatal: early EOF 04:51:50 fatal: fetch-pack: invalid index-pack output
>
> ---

# Answer
> 0 votes
**Enable TLS Certificate Verification:** It's important to enable TLS certificate verification for secure connections. You can do this by configuring Git with the correct SSL certificates or CA certificates. Jenkins might not have the necessary certificates installed or configured.

**Check Your Network Connection:** The "Connection was reset" error could be related to network issues. Ensure that your Jenkins server has a stable and reliable network connection. If this is a recurring problem, consult your network administrator.

**Update Git and Jenkins:** Ensure that you are using an up-to-date version of Git on your Jenkins server. Make sure Jenkins and its plugins are also up-to-date. Sometimes, updating Jenkins and its plugins can resolve compatibility issues that might lead to Git errors.

**Troubleshooting/Debugging:** You can try to enable Git trace/debugging to get more detailed information about what's happening during the fetch operation. You can do this by setting the GIT\_TRACE and GIT\_CURL\_VERBOSE environment variables to "1" or "true."

---
Tags: azure-devops, git
---


# Question
Title: Redefine deployment (Portainer) pvc after node failure
I have a home lab running k3s.

> ```
> NAME         STATUS   ROLES                         AGE    VERSION
> k8s-node-001 Ready    control-plane,master,worker   302d   v1.27.6+k3s1
> k8s-node-002 Ready    worker                        302d   v1.27.6+k3s1
>
> ```

I recently removed a node forcefully, since the disk stopped working. (k8s-node-003 was the failure - now deleted from the cluster) Unfortunately that node was running the pvc for my portainer setup.

I looked for a way to "move" the pvc to another node without much luck. So I deleted it `kubectl delete pvc portainer -n portainer`

I scaled down the portainer deployment with `kubectl scale deployment portainer -n portainer --replicas=0` and tried to scale it back up `kubectl scale deployment portainer -n portainer --replicas=1`

Inspecting the pod under deployment `kubectl describe pod portainer-858644df85-sjdbk -n portainer` I see that

> ```
> ...
>
> Events:
>   Type     Reason            Age   From               Message
>   ----     ------            ----  ----               -------
>   Warning  FailedScheduling  95s   default-scheduler  0/2 nodes are available: persistentvolumeclaim "portainer" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
>   Warning  FailedScheduling  13s   default-scheduler  0/2 nodes are available: persistentvolumeclaim "portainer" not found. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
>
> ```

I verified that the claim is gone

`kubectl get pv -n portainer portainer`

> ```
> Error from server (NotFound): persistentvolumes "portainer" not found
>
> ```

`kubectl get pvc -n portainer portainer`

> ```
> Error from server (NotFound): persistentvolumeclaims "portainer" not found
>
> ```

I guess I should have saved the pvc and then reinstated that using a command like `kubectl get pvc --namespace portainer -o json > portainer-pvc.yaml` and then reinstated it with something like `kubectl apply -f portainer-pvc.yaml -n portainer` but 20/20 hindsight is not useful now...

Is it sufficient to find the portainer pvc definition and applying that?

## UPDATE

I found the command i used to update portainer and ran that

```
helm upgrade --install --create-namespace -n portainer portainer portainer/portainer \
     --set enterpriseEdition.enabled=true \
     --set tls.force=true \
     --set tls.existingSecret=portainer-tls-secret

```

which appears to have reinitialised the pvc

`kubectl get pvc -n portainer portainer`

> ```
> NAME        STATUS    VOLUME   CAPACITY   ACCESS MODES   STORAGECLASS   AGE
> portainer   Pending                                      local-path     4m12s
>
> ```

...but it seems that there is some other reference to the defunct node:

> ```
> Events:
>   Type     Reason            Age    From               Message
>   ----     ------            ----   ----               -------
>   Warning  FailedScheduling  4m24s  default-scheduler  0/2 nodes are available: 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/2 nodes are available: 2 Preemption is not helpful for scheduling..
>
> ```

Is there a way to alter the configuration besides wiping the whole portainer configuration and reaapplying it ?

# Answer
My saviour was

`kubectl edit deploy portainer -n portainer -o yaml`

changed the node name, saved the file and watched everything work
> 0 votes

---
Tags: kubernetes, kubectl, k3s, persistent-volumes
---


# Question
Title: Can we do zero-downtime kubernetes deployment with a single pod?
I've inherited an app with K8s services with the following config

```
spec:
  ...
  replicas: 1
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate

```

For most of these light-weight services, we run a single instance of a pod on GCP. We're expanding into blue-green deploys for some services, but the majority are still run this way.

When we update the GCR image to a new version, the service becomes temporarily unavailable (maybe 10-30 seconds). I'm assuming this is because K8s is killing the old pod while it brings up the new pod. Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod? This would seemingly allow for a deployment with no downtime, and not rely on having two pods running all the time (saving cost). All docs I can find typically start with N pods so they don't have this issue...

---

## UPDATE

So the issue is not related to maxsurge (changing it had no effect). Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready. These are spring apps, so they take like 30 seconds to start, and the service disruption we're seeing is occurring in the 30s window while the app is booting and the pod has already sent traffic to it. So I guess what I really am after is as follows

**how can k8s be configured to bring up a second pod, wait until it passes startup probe/healthcheck, and only then have the first pod route traffic to it and self-terminate?**

# Answer
The difficulties we encountered with a single pod rolling update are already explained here:

https://stackoverflow.com/questions/46369100/kubernetes-rolling-update-killing-off-old-pod-without-bringing-up-new-one

The solution to us was multifaceted. First, we needed to set `MaxUnavailable = 0` and `MaxSurge=2`. We then needed to setup a persistent volume claim, so that the pods could share disk. If each pod has its own disk, then the pods would just hang, as the second pod is never "available" to take over.
> 0 votes

# Answer
> Is there anyway to configure K8s to actually bring up a new pod, direct new traffic to it, and only then shutdown the old pod?

This is the default behavior. MaxSurge is number of *additional* pods that runs during upgrade.

What you also have to check is how the load balancer is handling traffic, does it have capability to detect that you have deployed a new Pod, e.g. is it Kubernetes aware? See Container native load balancing on Google Cloud, it looks like you should have a `ReadinessProbe` on your `Deployment`.

> Actually I realized what's happening is that the pod is getting traffic routed to it before the app is actually completely ready.

This is the default behavior, if you define a ReadinessProbe for your `Deployment`. As you write, you can also add a `StartupProbe` but the `ReadinessProbe` is most important.
> 5 votes

# Answer
I would suggest you go with ReadinessProbe. So, with this, your application can say that "I am ready to receive the traffic" then the current running pod is ready to be terminated.

Note that it is different to LivenessProbe where it means "I am alive", "but not yet ready to serve the traffic".
> 0 votes

---
Tags: kubernetes, gcp
---


# Question
Title: How can I delete Terraform assets and statefiles created in a pipeline using the remote GitLab backend?
Currently I have a detached pipeline which uses the Terraform with the remote Gitlab backend.

How can I after the pipeline has run delete **BOTH** the assets created and the statefile?

# Answer
Currently Terraform can **NOT** delete statefiles stored in remote backends. The feature request for this is 34172.

Now what you have to do is run
> 0 votes

---
Tags: terraform, gitlab
---


# Question
Title: How to ascertain excesive use of my S3 bucket?
I have a S3 bucket that I use to store images for a website I manage. The bill is usually about $1/month and I store about 12.5 GB on the service. It's never really been an issue before, but recently noticed the bill has shot up to ~$10-12/month. After some investigating, data transfer from US East (Northern Virginia) data transfer to Asia Pacific (Singapore) has dramatically shot up, to almost 500GB/month.

I've looked through the bucket and don't see anything unusual. It's largely just user avatars and photos. Is there anyway I can figure out what's been transferred at such a high rate?

Thanks.

# Answer
> 1 votes
I think that your best course of actions is enabling access log on your S3 bucket to see what files are accessed at this rate

Also I had a similar issue some time ago where a S3 bucket that was only storing profil pictures started to cost a lot more that I should in data transfer. Turn's out the issue was with the ORM packages used to interact with the database, a simple login-in of an user would create an update event (to store the new access token), but this would also load in memory the user data and for some reason the url of the profil picture was loaded too (and downloaded from the S3)

Hope it helps !

---
Tags: amazon-web-services, amazon-s3
---


# Question
Title: Making AWS EFS Replication play nice with Terraform for disaster recovery
How do I use EFS block replication for disaster recovery when the replica volume is unknown to terraform? After breaking the replication/mirror I need to import the volume into the plan so it can become the new main with it's own replica.

This seems really cumbersome. Is this really the best way for a DR scenario?

**Use case** We have a Jenkins using an EFS volume and presently use datasync tasks to migrate data to the DR region. However, datasync tasks are like rsync and sluggish as they walk the tree processing files. Whereas, EFS replication is block based and much faster.

1. terraform creates region A volume + access and replication in region B + access
2. At some point region A goes down
3. We delete the replication from A -\> B transforming region B EFS volume to writable and our primary volume
4. We create a replication from this volume back to region A
5. Allow sync
6. Delete replication making region A replica the main volume

**Terraform sync issues** After step 3 we've got a main volume in region B that terraform would need to know about as terraform should be making the replica in region A

After step 6 we've got the same problem in region A

**Questions**

* Can someone point to a good example of an automated end of end process?
* Is this a case of "import the infra into tf state" as an automated part of the DR process?

# Answer
Reading your usecase it seem that the solutions your are looking for is the `refresh state` of terraform

Allowing you to refresh your state from the current architecture running without applying any diff
> 0 votes

---
Tags: terraform, amazon-efs
---


# Question
Title: In Jenkins, how can I disable "Discard old builds" option via pipeline?
I have several jobs where I want to disable the option shown below in the jobs' configuration page. How can I do it via pipeline?

# Answer
We can't do it by freestyle or manually but we can achieve this by writing a pipeline script for all the jobs in one and running it.

***Example code***:

```
pipeline {
  agent any
  stages {
    stage('Disable discard old builds on all jobs') {
      steps {
        // Get all jobs
        def jobs = Jenkins.instance.getAllItems(Job.class)

        // Iterate over all jobs and disable the "Discard old builds" option
        jobs.each { job ->
          // Get the job configuration
          def jobConfig = job.getConfigFile()

          // Disable the "Discard old builds" option
          jobConfig.properties.remove(jobConfig.properties.find { it instanceof BuildDiscarderProperty })

          // Update the job configuration
          jobConfig.save()
        }
      }
    }
  }
}

```

Mention all the jobs accordingly... & The pipeline job will iterate over all jobs in Jenkins and disable the "Discard old builds" option on each job.
> 1 votes

---
Tags: jenkins, jenkins-pipeline, groovy
---


# Question
Title: Kubenetes new Pods/Deployments/Resources stuck in pending state forever without error
I have a local Kubernetes cluster setup using kubeadm. Everything was working fine a few days back but now I've got a strange problem.

Whenever I add new resources be it a deployment or pod, it never gets scheduled and is forever stuck in Pending state without any error. Please let me know if you anyone has any idea how to fix this issue or even debug for that matter.

I'm pasting output of some of the commands I think might be useful, let me know if you need any additional information

• kubectl get cs

```
NAME                 STATUS    MESSAGE                         ERROR
scheduler            Healthy   ok
controller-manager   Healthy   ok
etcd-0               Healthy   {"health":"true","reason":""}

```

• kubectl get nodes -o wide

```
NAME              STATUS   ROLES           AGE    VERSION   INTERNAL-IP   EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION      CONTAINER-RUNTIME
in-dt-dckr-ldr1   Ready    control-plane   386d   v1.25.2   10.64.1.98    <none>        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8
in-dt-dckr-ldr2   Ready    <none>          386d   v1.25.2   10.64.1.190   <none>        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8
in-dt-dckr-wrk1   Ready    worker          386d   v1.25.2   10.64.1.232   <none>        Ubuntu 20.04.5 LTS   5.15.0-58-generic   containerd://1.6.8

```

• kubectl describe node in-dt-dckr-wrk1

```
Name:               in-dt-dckr-wrk1
Roles:              worker
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=in-dt-dckr-wrk1
                    kubernetes.io/os=linux
                    node-role.kubernetes.io/worker=worker
Annotations:        csi.volume.kubernetes.io/nodeid: {"csi.tigera.io":"in-dt-dckr-wrk1"}
                    kubeadm.alpha.kubernetes.io/cri-socket: unix:///var/run/containerd/containerd.sock
                    node.alpha.kubernetes.io/ttl: 0
                    projectcalico.org/IPv4Address: 10.64.1.232/22
                    projectcalico.org/IPv4VXLANTunnelAddr: 192.168.137.192
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Thu, 13 Oct 2022 01:04:07 +0530
Taints:             <none>
Unschedulable:      false
Lease:
  HolderIdentity:  in-dt-dckr-wrk1
  AcquireTime:     <unset>
  RenewTime:       Fri, 03 Nov 2023 08:57:17 +0530
Conditions:
  Type                 Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----                 ------  -----------------                 ------------------                ------                       -------
  NetworkUnavailable   False   Sat, 21 Jan 2023 19:29:25 +0530   Sat, 21 Jan 2023 19:29:25 +0530   CalicoIsUp                   Calico is running on this node
  MemoryPressure       False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure         False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure          False   Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready                True    Fri, 03 Nov 2023 08:53:02 +0530   Sat, 21 Jan 2023 19:28:44 +0530   KubeletReady                 kubelet is posting ready status. AppArmor enabled
Addresses:
  InternalIP:  10.64.1.232
  Hostname:    in-dt-dckr-wrk1
Capacity:
  cpu:                8
  ephemeral-storage:  959786032Ki
  hugepages-2Mi:      0
  memory:             8023576Ki
  pods:               110
Allocatable:
  cpu:                8
  ephemeral-storage:  884538805627
  hugepages-2Mi:      0
  memory:             7921176Ki
  pods:               110
System Info:
  Machine ID:                 9e7f87693fba4ce5b5cffd58e27c4b5e
  System UUID:                4c4c4544-0032-5910-804e-b3c04f595831
  Boot ID:                    8c6f8a8c-0acc-4ac9-86df-36867df44d8f
  Kernel Version:             5.15.0-58-generic
  OS Image:                   Ubuntu 20.04.5 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  containerd://1.6.8
  Kubelet Version:            v1.25.2
  Kube-Proxy Version:         v1.25.2
PodCIDR:                      192.168.1.0/24
PodCIDRs:                     192.168.1.0/24
Non-terminated Pods:          (18 in total)
  Namespace                   Name                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  Age
  ---------                   ----                                        ------------  ----------  ---------------  -------------  ---
  buildninja                  bn-agent1-6f484b8894-qr88p                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         52d
  buildninja                  bn-agent2-67b6859f58-zg6kc                  0 (0%)        0 (0%)      0 (0%)           0 (0%)         52d
  calico-apiserver            calico-apiserver-978f5784c-77gss            0 (0%)        0 (0%)      0 (0%)           0 (0%)         346d
  calico-system               calico-kube-controllers-6fdfb4dfdc-rfbsr    0 (0%)        0 (0%)      0 (0%)           0 (0%)         346d
  calico-system               calico-node-4gxvn                           0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
  calico-system               calico-typha-5cd5cbc888-mcqfh               0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
  calico-system               csi-node-driver-zr5lc                       0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
  db                          mongodb-cc47dc8b8-wwszq                     0 (0%)        0 (0%)      0 (0%)           0 (0%)         178d
  db                          postgres-79f797b96c-vhswc                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         285d
  docker-registry             gci-docker-registry-644799b4b7-q8wbl        0 (0%)        0 (0%)      0 (0%)           0 (0%)         283d
  kube-system                 coredns-5fcc5bdd47-b6w88                    100m (1%)     0 (0%)      70Mi (0%)        170Mi (2%)     283d
  kube-system                 kube-proxy-zjp4z                            0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
  kube-system                 metrics-server-5db9b4b966-8gqgk             100m (1%)     0 (0%)      200Mi (2%)       0 (0%)         386d
  tigera-operator             tigera-operator-6675dc47f4-z9jtx            0 (0%)        0 (0%)      0 (0%)           0 (0%)         386d
  utils                       web-sample-ff4d6596-hmtw6                   0 (0%)        0 (0%)      0 (0%)           0 (0%)         170d
  utils                       wiki-server-86f49bd4c4-4drdr                0 (0%)        0 (0%)      0 (0%)           0 (0%)         182d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests    Limits
  --------           --------    ------
  cpu                200m (2%)   0 (0%)
  memory             270Mi (3%)  170Mi (2%)
  ephemeral-storage  0 (0%)      0 (0%)
  hugepages-2Mi      0 (0%)      0 (0%)
Events:              <none>

```

• kubectl describe po nginx

```
Name:             nginx
Namespace:        default
Priority:         0
Service Account:  default
Node:             <none>
Labels:           run=nginx
Annotations:      <none>
Status:           Pending
IP:
IPs:              <none>
Containers:
  nginx:
    Image:        nginx
    Port:         <none>
    Host Port:    <none>
    Environment:  <none>
    Mounts:
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-lg45r (ro)
Volumes:
  kube-api-access-lg45r:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

```

I found a similar case but no resolutions there too https://stackoverflow.com/questions/55310076/pods-are-in-pending-state

**Update 1** Adding output for journalctl

• sudo journalctl -u kubelet -f

```
-- Logs begin at Fri 2022-04-22 10:32:53 IST. --
Nov 01 14:18:45 IN-DT-DCKR-LDR1 kubelet[3279607]: I1101 14:18:45.802594 3279607 request.go:682] Waited for 1.137588349s due to client-side throttling, not priority and fairness, request: POST:https://10.64.1.98:6443/api/v1/namespaces/kube-system/pods
Nov 01 14:18:45 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:45.845718 3279607 kubelet.go:1712] "Failed creating a mirror pod for" err="pods \"etcd-in-dt-dckr-ldr1\" already exists" pod="kube-system/etcd-in-dt-dckr-ldr1"
Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123742 3279607 configmap.go:197] Couldn't get configMap kube-system/kube-proxy: failed to sync configmap cache: timed out waiting for the condition
Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123782 3279607 configmap.go:197] Couldn't get configMap calico-system/tigera-ca-bundle: failed to sync configmap cache: timed out waiting for the condition
Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123870 3279607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/61a2bf9a-a4cb-49d8-8b63-de7a82841a55-kube-proxy podName:61a2bf9a-a4cb-49d8-8b63-de7a82841a55 nodeName:}" failed. No retries permitted until 2023-11-01 14:18:46.62383261 +0530 IST m=+3.132668643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-proxy" (UniqueName: "kubernetes.io/configmap/61a2bf9a-a4cb-49d8-8b63-de7a82841a55-kube-proxy") pod "kube-proxy-6ghjb" (UID: "61a2bf9a-a4cb-49d8-8b63-de7a82841a55") : failed to sync configmap cache: timed out waiting for the condition
Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.123907 3279607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43736256-5241-46c4-b403-85db6d1d4d52-tigera-ca-bundle podName:43736256-5241-46c4-b403-85db6d1d4d52 nodeName:}" failed. No retries permitted until 2023-11-01 14:18:46.623887546 +0530 IST m=+3.132723570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tigera-ca-bundle" (UniqueName: "kubernetes.io/configmap/43736256-5241-46c4-b403-85db6d1d4d52-tigera-ca-bundle") pod "calico-node-rb5s8" (UID: "43736256-5241-46c4-b403-85db6d1d4d52") : failed to sync configmap cache: timed out waiting for the condition
Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.124885 3279607 configmap.go:197] Couldn't get configMap kube-system/coredns: failed to sync configmap cache: timed out waiting for the conditionNov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.124929 3279607 secret.go:192] Couldn't get secret calico-system/node-certs: failed to sync secret cache: timed out waiting for the condition
Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.125014 3279607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/08f1a4fd-92c5-48fc-a299-636a4f0a469a-config-volume podName:08f1a4fd-92c5-48fc-a299-636a4f0a469a nodeName:}" failed. No retries permitted until 2023-11-01 14:18:46.624972044 +0530 IST m=+3.133808093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "config-volume" (UniqueName: "kubernetes.io/configmap/08f1a4fd-92c5-48fc-a299-636a4f0a469a-config-volume") pod "coredns-5fcc5bdd47-8llkc" (UID: "08f1a4fd-92c5-48fc-a299-636a4f0a469a") : failed to sync configmap cache: timed out waiting for the condition
Nov 01 14:18:46 IN-DT-DCKR-LDR1 kubelet[3279607]: E1101 14:18:46.125058 3279607 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43736256-5241-46c4-b403-85db6d1d4d52-node-certs podName:43736256-5241-46c4-b403-85db6d1d4d52 nodeName:}" failed. No retries permitted until 2023-11-01 14:18:46.625034355 +0530 IST m=+3.133870397 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-certs" (UniqueName: "kubernetes.io/secret/43736256-5241-46c4-b403-85db6d1d4d52-node-certs") pod "calico-node-rb5s8" (UID: "43736256-5241-46c4-b403-85db6d1d4d52") : failed to sync secret cache: timed out waiting for the condition

```

• sudo journalctl -u containerd -f

```
-- Logs begin at Fri 2022-04-22 10:32:53 IST. --
Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time="2023-10-13T00:13:20.769518216+05:30" level=info msg="RemovePodSandbox for \"ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\""
Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time="2023-10-13T00:13:20.769565086+05:30" level=info msg="Forcibly stopping sandbox \"ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\""
Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time="2023-10-13T00:13:20.769696525+05:30" level=info msg="TearDown network for sandbox \"ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\" successfully"
Oct 13 00:13:20 IN-DT-DCKR-LDR1 containerd[828]: time="2023-10-13T00:13:20.852087515+05:30" level=info msg="RemovePodSandbox \"ea81a51660843963fda52520e4d5834bb065e8bc6b48d7350fea5ed462066936\" returns successfully"
Nov 01 12:16:31 IN-DT-DCKR-LDR1 containerd[828]: time="2023-11-01T12:16:31.730407989+05:30" level=info msg="No cni config template is specified, wait for other system components to drop the config."
Nov 01 14:18:43 IN-DT-DCKR-LDR1 containerd[828]: time="2023-11-01T14:18:43.710560884+05:30" level=info msg="No cni config template is specified, wait for other system components to drop the config."
Nov 01 14:43:42 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:43:42.533234     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec "845f8446d19b7636bb87b001b881ce7b1f07dc372e61b542dd316605313e5bff": OCI runtime exec failed: exec failed: unable to start container process: exec: "bash": executable file not found in $PATH: unknown
Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: time="2023-11-01T14:44:30.136995113+05:30" level=info msg="Container exec \"b7fb9daf0322fbdbe4facb31c3d18fdd44d4c6402dc7733dbf5b2c4d659dcb72\" stdin closed"
Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:44:30.288136     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec "89067c1007468b006bc9b8d86b446ebbd04438025dd42cd90828123dc1a688b7": OCI runtime exec failed: exec failed: unable to start container process: exec: "powershell": executable file not found in $PATH: unknown
Nov 01 14:44:30 IN-DT-DCKR-LDR1 containerd[828]: E1101 14:44:30.419019     828 exec.go:87] error executing command in container: failed to exec in container: failed to start exec "370d225ba5a9bc40bdc43040b1eda910fe278bd0bf5174682a94bc63a3be18ce": OCI runtime exec failed: exec failed: unable to start container process: exec: "cmd": executable file not found in $PATH: unknown

```

# Answer
> 0 votes
You will probably find the error in either

```
sudo journalctl -u kubelet -f

```

or

```
sudo journalctl -u containerd -f

```

My guess would be that you have SystemdCgroup set to false in /etc/containerd/config.toml

Set it to true on all nodes and restart containerd

```
SystemdCgroup = true

```

when you dont have the file yet, create it

`containerd config default > /etc/containerd/config.toml`

then set SystemdCgroup to true and restart containerd

---
Tags: kubernetes, kubectl, kubeadm
---


# Question
Title: How to efficiently copy multiple Docker images between two air gapped repositories?
I need to efficiently copy Docker images between two air gapped repositories. The current method of saving images to disk and transferring them as tarballs is time-consuming, so I am looking for a process that will:

1. Eliminate redundant image layers by only copying each layer once, even if it is used by multiple images.
2. Avoid copying layers that already exist in the remote repository.

Is this possible? How can I do it?

# Answer
> 1 votes
Instead of transferring the images as a packaged tar file, I'd recommend using the OCI Image Layout format that represents a repository as a directory on disk. Copying contents into or out of this directory can then leverage layer deduplication in the content addressable store. You'll need to ensure the same disk is reused each time you transfer the content to avoid downloading layers that were previously pulled.

There are several tools that are able to work with images in this format. The ones I'm aware of are ORAS (originally from Microsoft), skopeo (from RedHat), and regclient (my own project).

Here's an example of what this would look like with regclient's regctl command:

```
# copy to disk
regctl image copy ${public_repo}:${tag} ocidir://path/to/disk:${tag}

# transfer the disk across the air-gap

# copy from disk to internal registry
regctl image copy ocidir://path/to/disk:${tag} ${private_repo}:${tag}

```

If you have a lot of images, the regclient/regsync command is probably what you are looking for.

---
Tags: docker, docker-registry
---


# Question
Title: Use an asymmetric cypher to secure Ansible vault
I'm trying to develop a Playbook which must handle sensitive user data (SSH keys, Registry credentials, etc.).

Only a user and the Ansible controller can see the data associated with that same user.

The data will be stored in a Git repository, accessible to all users.

Thus it must be encrypted.

I'm having an issue to securely encrypt the data using Ansible vault.

The problem is that I cannot give the Ansible vault secret to the user, as the secret can decrypt the ciphertext.

The perfect solution would be to use asymmetric encryption, which I didn't find anyway to do.

The only solution can I see at the moment is this:

1. User encrypt its data using a public key
2. User sends ciphertext to Ansible controller
3. Ansible controller decrypts the ciphertext using a private key
4. Ansible controller encrypts the cleartext using ansible vault (symmetric cipher this time)
5. Ansible controller commits the ciphertext to the git repository

But it'll be painful to setup this process. It would be much simpler if the user could commit the ciphertext encrypted with the public key.

* Is it possible to setup Ansible vault to use asymmetric encryption ?
* Is it possible to setup the playbook to automatically decrypt data encrypted by something else than the vault (basically a custom vault) ?
* Is there some support for encryption other than the Ansible vault ?

# Answer
> *Is it possible to setup Ansible vault to use asymmetric encryption?*

To setup or configure unfortunately not. It might be possible to enhance the functionality of `vault.py` and implement such.

Regarding

> *Is it possible to setup the playbook to automatically decrypt data encrypted by something else than the vault (basically a custom vault)? Is there some support for encryption other than the Ansible vault?*

and since you are like to

> *... handle sensitive user data (SSH keys, Registry credentials, etc.).*

you might take advantage from external Vaults in IdM and Chapter 76-79, "*storing and retrieving secrets*".
> 1 votes

# Answer
Another solution, which would allow you to accomplish the same goals, would be to encrypt your facts files with Mozilla SOPS.

SOPS does a good job of just encrypting parts of the files that you want to have encrypted and you can have multiple people/bots added to the key ring so a build server can decrypt or other people with the key can have it. You can check the secrets into git and it's relatively safe.

You can find more information here https://docs.ansible.com/ansible/latest/collections/community/sops/docsite/guide.html
> 1 votes

---
Tags: ansible, encryption, ansible-vault
---


# Question
Title: How to test production Ansible execution without having to provision target
We use ansible for our day to day operation and so far so good. We have been looking for a way to run local test etc just like chef guys and found molecule that we use with testinfra/vagrant. While I personally found it great for developing roles, I am looking for a way to actually test the playbook(which uses multiple roles , some with pre and post tasks) itself in **PRODUCTION** after it runs.

**EDITS** : Say the playbook configures 5 boxes; creates some mongodb replica set (3 boxes) and deploys some java applications (2 boxes). Basic check after this runs would be to find out if Mongodb members can reach each other, or whether the app boxes can connect and other more specific checks.

Ideally I could use testinfra without the molecule part and I have not really found a way of doing that by also loading the ansible variable file etc.

I have looked into ansible\_spec which from my understanding is from serverspec.

Probably my perception of these tools are wrong but, I am thinking of having the check to be say: the last task in the playbook or call it inside the post\_tasks stage. Not sure how all these will pan out but I would be grateful if you could share your wisdom with me.

Thanks in advance

# Answer
This would be a good case for doing a blue-green deployment, it's twice the cost , but half the hassle of having to deploy into production. Plus you get near instant backups.

If you can't trust what you deploy in dev, it's usually your only bet.

You should be able to effectively get blue-green deployments using parameters passed into ansible vault (or a custom inventory playbook).

For ansible, when you add blue or green environments, you could encrypt the relevant data (ip addresses, passwords, keys) with a vault ID of blue or green

`ansible-vault encrypt --vault-id blue hostvars/monogo-1.yml`

`ansible-vault encrypt --vault-id green hostvars/monogo-1.yml`

and pass that in to do your deployments,

`ansible-playbook --vault-id blue playbooks playbooks/monog-deploy.yml`

that way, the only thing you need to know is whether blue or green is currently live and deploy the opposite. Of course, that could get a little messy - there may be a better way do to that than rely on the vault, but it's the simplest way I can think of where all you need to know is blue or green.

---

I'm guessing I misinterpreted your question a bit, but testing *production* seems like a bad idea. Using ansible to test is similarly a bad idea (so saith my company's architects) and I use inspec to test after an ansible deployment.

IMO, testing your blue-green deployment is still better than testing the production deployed environment. Generally speaking though, I trust that ansible did what it said it was doing and I keep the templates generic enough to know that whatever it lacked, wasn't there to begin with.
> 0 votes

---
Tags: ansible, testing, serverspec
---


# Question
Title: ansible_become has no effect despite the ability to execute sudo on GCP hosts
Say I want to execute the following command on target hosts (deployed on GCP): `ansible all -a "head -1 /etc/shadow"`. The command demands the root access. With the default Ansible configuration, I get the following error:

```
host1 | FAILED | rc=1 >>
head: cannot open '/etc/shadow' for reading: Permission deniednon-zero return code

```

Hence, I extended my `ansible.cfg` to include the following:

```
remote_user = gcp_user
ansible_become=yes
ansible_become_method=sudo
ansible_become_user=root

```

Despite these changes, I can't run the command mentioned above. I still get the same error. I am, however, capable of executing the following command: `ansible all -a "sudo cat /etc/shadow"`

Thus, it seems that `become_user=yes` does not take any effect.

# Answer
> 0 votes
Running ansible with -vvv will tell you which ansible config you're using

For instance

```
me@mycomputer:~$ ansible -vvv -m setup localhost
ansible 2.10.8
  config file = None

```

shows that I'm not using any config file

chances are pretty OK that is what is happening to you too. ansible.cfg should be in the current directory or ~/.ansible (maybe some other places depending on how you installed ansible) you can specify where it is with `ANSIBLE_CONFIG=/path/to/config` in your environment variables

---

The other thing is, the parameters you used in the config file are actually the parameters that go in the playbook, not in the config file

https://docs.ansible.com/ansible/latest/reference\_appendices/config.html

explains where they go in the config file.

When you're running adhoc tasks, you can use them on the commandline (as you wrote) with `-e remote_user=gcp_user -e ansible_become=yes -e ansible_become_method=sudo -e ansible_become_user=root`

https://docs.ansible.com/ansible/latest/reference\_appendices/config.html

---
Tags: ansible, google-cloud-platform, gcp, ssh, ansible-inventory
---


title: Using a different Python version in AWX

text:
# Question
Title: Using a different Python version in AWX
I updated a recent playbook module that says I need to use at least Python version 3.8. I run this playbook on our AWX controller. When I run the cmd module and do a "which python3" I get back version 3.6 in /var/lib/awx/venv/ansible/bin/python3. I tried doing a pre\_task to install python3.9 using the yum module, and I see in the output that it gets installed.

I thought I could use the "var: ansible\_python\_interpreter: *path* but I need to find out where new versions of python get installed. Do you know where python gets installed within the AWX controller when running a playbook to install it?

```
---
- name: Test commands
  hosts: localhost
  connection: local
  gather_facts: false

  pre_tasks:
    - name: install Python
      ansible.builtin.yum:
        name: python3.9
        state: present
      become: yes
      delegate_to: localhost

- name: Configure Avi Controller
  hosts: localhost
  connection: local
  collections:
    - cyberark.pas
    - vmware.alb
  vars:
    ansible_python_interpreter: /var/lib/awx/venv/ansible/bin/python3.9
  tasks:

```

When I run this I get "/var/lib/awx/venv/ansible/bin/python3.9 file or directory does not exist"

# Answer
> 1 votes
AWX is running in a docker container, isn't it?

If you make a venv on your ansible controller with python3.9 it still won't be functional. To get this to work, you will have to add your actual host (which seems insecure ) to your inventory and have a method to log on to it.

The best way around this, that I can think of, would be to upgrade python on your docker container running AWX, probably don't need a venv on the container, but that might be best practice.

---

Also, `hosts: localhost` implies `connection: local` so that part is redundant, unless there's a reason it's not (like a global variable setting `connection: ssh`)

---
Tags: ansible, python, awx
---


# Question
Title: NodeCreationFailure: Instances failed to join the kubernetes cluster
I'm setting up an EKS cluster using https://github.com/ManagedKube/kubernetes-ops

and, based on that code, when I do a `terraform apply` I'm getting a `NodeCreationFailure` error.

Specifically:

```
│ Error: error waiting for EKS Node Group (staging:staging-ng12022040211190144860000000d) to create: unexpected state 'CREATE_FAILED', wanted target 'ACTIVE'. last error: 1 error occurred:
│   * i-0d5c42031eaf061ff, i-0d9a073240fd29c45: NodeCreationFailure: Instances failed to join the kubernetes cluster

```

Admittedly, that's not much code, but does anyone have any pointers on where to start debugging?

# Answer
I incurred the same problem during one of my deployments.

This happens due to the **Networking issue** of the EKS cluster. Assuming that you are working in AWS, follow the following steps to debug this problem:

1. Check if your security groups are correctly configured correctly to your Nodes.
2. If the EKS is present in the public sub-nets, check its connectivity to internet gateway and is able to access the internet.
3. If Eks is present in the private sub-net, check the connectivity with NAT gateway and that NAT gateway is able to access the internet.
> 2 votes

# Answer
I had a similar issue in the past where the nodes couldn't join the cluster. The cause of this was a mismatch in security groups, the nodes and the EKS control plane weren't allowed to talk to each other and therefore timed out.
> 0 votes

# Answer
I was getting the same issue, it was solved by creating another nat gateway and choosing public subnet and attaching that new nat gateway in routable to my private subnet.
> 0 votes

# Answer
AWS has listed a set of reasons it could make Node groups to fail joining a cluster. Refer, https://repost.aws/knowledge-center/eks-worker-nodes-cluster I was able to understand that one of my security groups rejected the comms between Control Plane and the worker node. This troubleshooting was possible using System Automation - TroubleshootingEKSNode.
> 0 votes

---
Tags: eks
---


title: feature flags vs versioning for APIs

text:
# Question
Title: feature flags vs versioning for APIs
We are looking at moving to a trunk based development branching strategy rather than gitflow that we currently use. I understand that feature flags are crucial to trunk based development. However, is there still a place for versioning APIs if I am using feature flags to control exposure of new features across UI and APIs?

All of our APIs are are internal to our application and are not exposed directly to a third party. Even in a case like changes in schema for response/requests - those could be treated as a new feature and handled via a feature flag to enable dark deployments etc. So, what would be a good case to still use API versioning if I have feature flags going?

Will appreciate any thoughts from your experience or any good articles you have come across related to this topic.

Thank you - VK

# Answer
> 2 votes
The two approaches whilst somewhat overlapping solve two different problems:

* Feature flags allow you to **decouple the action of deployment from the action of release**. i.e. you can push a change to an API that adds functionality to support a new feature, then when your product owner wants to release it all they need to do is toggle the feature on.
* API versioning enables you to make changes to your API without breaking compatibility with older clients.

In both cases, you need to ensure that old feature flags and old API versions are eventually removed from the codebase or it will become very complicated very quickly.

The deciding factor comes down to whether or not you have control over all of your clients if you can guarantee that you can make changes to both your clients and the API in such a way that they will remain compatible, or incompatible clients can be removed from your ecosystem very quickly you probably don't need API Versioning.

# Answer
> 0 votes
* **Feature flags** are preferred when toggle needs to be done for shorter duration (for internal teams this can be a good approach).
* **API versionning** is best suitable for long term different features needs to be provided (for external facing apps).

---
Tags: continuous-integration, cicd, versioning, feature-toggles, feature-flags
---


title: Skip terraform resource if it exists

text:
# Question
Title: Skip terraform resource if it exists
I'm Getting an error creating a secret because it was created manually.

```
Error: error creating Secrets Manager Secret: ResourceExistsException: The operation failed because the secret <SECRET> already exists.

```

Is there any way to tell Terraform to skip creating a resource if it already exists ?

# Answer
> 2 votes
after terraform 0.12 we have ways to control it using lifecycle we can simply skip resource by setting up lifecycle

```
resource "aws_instance" "web" {
    ami = "foo"
    lifecycle {
        ignore_changes = ["ami"]
    }
}

```

Here is the link of bug https://github.com/hashicorp/terraform/pull/2525

# Answer
> 1 votes
No there is no way to tell Terraform to skip creating a resource if it already exists.

Is there a reason why you want to skip resource creation? Why not import the existing resource into Terraform state file so it doesn't try to create the resource that already exists?

You could provide more background information for this.

# Answer
> 0 votes
One hack we can do is for every named resource you plan to create, perhaps do a check for the resource using data block and then use count in the resource block in order to create or skip.

With this it will increase the execution time but recurring already exists errors could be mitigated.

---
Tags: terraform
---


# Question
Title: Queue jenkins job task in azure pipeline won't complete
I have set up an azure devops pipeline with a single task - to queue a jenkins build job. The jenkins job is a parameterised pipeline build. The job queues and runs as expected in Jenkins. I would expect the azure pipeline task to complete once the jenkins job is finished and to reflect the success status of the jenkins build, but what is currently happening is that the azure task never completes and simply runs to timeout.

I have selected both the;

"Capture console output and wait for completion"

and

"Capture pipeline output and wait for pipeline completion"

options in the configuration for the task, as far as I am aware that should do exactly as I expect.

I don't have the TFS jenkins plugin installed and would rather avoid doing so if it's not strictly necessary.

I can see that the teamBuildRequestCallback(teamBuildPostData) call from the task is generating a 404 response but I'm not sure why that's the case.

Any insight on this would be greatly appreciated.

# Answer
Had the same issue, worked for me after I changed the plugin version from 1 to 2
> 0 votes

---
Tags: jenkins, azure-devops, jenkins-pipeline, pipeline
---


# Question
Title: Prometheus and Azure WebApps: How to monitor multiple instances with prometheus
I have an application that has two instances running and each request runs on a different machine.

I am monitoring an event with a counter in my application and each instance runs it's own counter using prometheus-net. It would be great if prometheus-net was using a caching mechanism other that in-memory, maybe a redis, so whenever prometheus scrap /metrics, it gets the same data each time.

Unfortunately, each scrap request from prometheus runs on a different machine and it's like I have two counters.

I need a way to tell to prometheus to scrap two urls and I will find a way to combine them.

Unfortunately, the way to access directly the web app instance is by using parameters.

The urls looks like that:

```
https://example.com/metrics?instance=dba246887a8c3c07e1a9cc331d7372c43937e6acd56073246ffdf0e95597480c
https://example.com/metrics?instance=be007691832f4002a86cf0148768a3b039304e9b1fa7ae91a032b0beddc152b2

```

That way I cannot use the following code since I cannot add full url here.

```
 static_configs:
      - targets: ['example.com']

```

I tried the following but it didn't created two 'instances' at the label `instance`

```
- job_name: 'example'
    scrape_interval: 30s
    params:
      -instance: ['372c43937e6acd56073246ffdf0e95597480cdba24688cc331d77a8c3c07e1a9','be007691832f4002a86cf0148768a3b039304e9b1fa7ae91a032b0beddc152b2']
    static_configs:
      - targets: ['example.com']
    metrics_path: /metrics
    scheme: https

```

# Answer
You can do this similarly to how it's done in blackbox\_exporter: list identifying information of your targets in the `target` block, and then use relabeling to replace scraping address with address of your actual exporter and to place targets into parameters of scraping request.

```
scrape_configs:
  - job_name: 'myjob'
    static_configs:
      - targets:  # Targets to probe using parameters
        - 372c43937e6acd56073246ffdf0e95597480cdba24688cc331d77a8c3c07e1a9
        - be007691832f4002a86cf0148768a3b039304e9b1fa7ae91a032b0beddc152b2
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_instance
      - source_labels: [__param_instance]
        target_label: instance
      - target_label: __address__
        replacement: example.com  # The exporter's real address

```

please notice here: `__param_instance` is a label that will be used during scraping as URL parameter with name `instance`.

Here replacement config does:

1. puts ids from target list into parameter `instance`,
2. puts content of that parameter into final label `instance` (otherwise content of `__address__` will be used),
3. puts address of actual target into label `__address__` to be used bu scraping.
> 1 votes

---
Tags: prometheus
---


# Question
Title: why the crictl command did not list the images
when I tried to use crictl to list all images like this in Debian 12:

```
root@k8sslave01:/etc# crictl images
IMAGE               TAG                 IMAGE ID            SIZE

```

did not show any images. the ctr list works fine:

```
root@k8sslave01:/etc# ctr images list
REF                                                           TYPE                                                      DIGEST                                                                  SIZE      PLATFORMS                                                                    LABELS
registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6 application/vnd.docker.distribution.manifest.list.v2+json sha256:3d380ca8864549e74af4b29c10f9cb0956236dfb01c40ca076fb6c37253234db 294.7 KiB linux/amd64,linux/arm/v7,linux/arm64,linux/ppc64le,linux/s390x,windows/amd64 -
registry.k8s.io/pause:3.6                                     application/vnd.docker.distribution.manifest.v2+json      sha256:8a9b8e25874aac2f58bc7d523e55c4a98f03653d109f3acc77cd064544f906f4 290.9 KiB linux/amd64

```

Am I missing something? what should I do to fixed this issue? this is the /etc/crictl.yaml config:

```
root@k8sslave01:/etc# cat /etc/crictl.yaml
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
#debug: true
debug: false

```

I have already tried to import the images like this:

```
root@k8sslave01:~# ctr image import /root/pause.tar --all-platforms
unpacking registry.k8s.io/pause:3.6 (sha256:8a9b8e25874aac2f58bc7d523e55c4a98f03653d109f3acc77cd064544f906f4)...done

```

still could not show by crictl command.

# Answer
> 1 votes
Finally I found the crictl default namespaces in k8s.io, when import images, we should specify the namespaces like this:

```
ctr -n=k8s.io image import /root/pause.tar

```

---
Tags: kubernetes, containerd
---


# Question
Title: is it possible to copy image from one server to another server
On the server A, I install many kubernetes images, because I could not access the dockerhub for some network issue. I want to copy some images from server A to server B. is it possible to copy the images smoothly? I have searched from google but did not found some useful imformation. It is better to copy the image from server A to server B batch than one by one. Now I am using containerd, not using docker.

# Answer
You can pull image from official or mirror then save it like this:

```
docker save registry.k8s.io/pause:3.6>pause.tar

```

copy the pause.tar to other sever and import it like this:

```
ctr -n=k8s.io  image import /root/pause.tar

```
> 0 votes

---
Tags: kubernetes
---


# Question
Title: Stopped the process after executing npm start / macOS homebrew
I was in the process of creating function components. When I noticed a problem was after I hit save and my terminal never updated me if it was a successful compile... I'll admit I panicked... a little. Then started running npm start and been stuck ever since. enter image description here  enter image description here

# Answer
> 0 votes
I found out what it was: I performed an `audit --force`.

I didn't understand at the time. Here is an article I found, maybe it can help the next person. https://overreacted.io/npm-audit-broken-by-design/

---
Tags: npm
---


# Question
Title: nginx container isn't reachable from outside
I have to run a container running nginx as a reverse proxy using podman.

Dockerfile is as follows:

```
FROM nginx

COPY nginx/* /etc/nginx/

```

Where basically I have a default.conf file under conf.d which is like this:

```
server {

    listen       80;
    server_name  my-server;

    client_max_body_size 200M;

    location / {
        root /usr/share/nginx/html;
    }

    location /api {
        proxy_set_header X-FORWARDED-FOR $remote_addr;
        proxy_pass http://backend:3300/api;
    }

    error_page  404              /;

    # redirect server error pages to the static page /50x.html
    #
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }

}

```

where backend is an alias for another container with a rest server running on the same network that contains the proxy.

I setup the network with the command:

```
podman create custom-network

```

then I run the following command:

```
podman run -d --name=frontend -p 80:80 --network=custom-network:alias=frontend -v ./www:/usr/share/nginx/html -v ./logs:/var/log/nginx localhost/my-www:latest

```

At this point everything is up and running and I can open my application from the browser in the same machine, but it is unreachable from another computer on the same network.

I already turned off windows firewall on every network, but still nothing.

Is there a podman command or a configuration tweak that make nginx container reachable from outside?

# Answer
I ran into this same issue on windows and since you mentioned windows firewall I am assuming you are also running it.

I noticed that you did not mention any portmapping.

Get the ip of your podman vm. `podman machine ssh` and then an `ifconfig`. Then you just map your port like below to your vm ip and your requests from outside should work.

`netsh interface portproxy add v4tov4 listenport=<hostport> listenaddress=0.0.0.0 connectport=<containerPORT> connectaddress=172.26.29.96`
> 1 votes

---
Tags: nginx, podman
---