title: Creating container image without docker

text:
# Question
Title: Creating container image without docker
Is it possible to build a container image as a simple user account on a system which does not have docker or podman installed? In other words, is there a user-space tool/application which can build an image that could run on a different system which supports OCI containers, such as AKS, etc.

# Answer
> 3 votes
An image is based on the OCI image-spec which consists of filesystem layers packaged as tar files, a config json, and a manifest. All of these are referenced with a content addressable digest. If you can create these tar and json files, then you can create an image without a runtime.

There are various tools that do this for specific cases. I think most are shipped as containers and designed to run in CI pipelines, but you can probably get standalone binaries. However, I don't think any of these work with a Dockerfile, so if you need something that supports that syntax, you'll need a container runtime. There are rootless runtimes, but they typically have prerequisites that need to be performed on the host by root in advance, and there's a performance hit from using them.

# Answer
> 0 votes
Podman and Buildah are userspace tools. They don't require root access. They can also build `oci` and `docker` images,

> Recognized formats include oci (OCI image-spec v1.0, the default) and docker (version 2, using schema format 2 for the manifest).

Note while Buildah can build from a dockerfile, you really shouldn't ever create dockerfiles. The normal method of operation with buildah involves creating a shell script, which provides you amongst other things a path into the off-line mount point, shell expansion, and conditionals, and the ability to run things in the container before committing (great for initialization).

Here is a quick example without dockerfile, `build_image.sh`

```
#!/bin/bash

ctr=$(buildah from alpine:3)
mnt=$(buildah mount "$ctr")

mkdir "$mnt/rootleveldir"
cp ./localfile "$mnt/rootleveldir"

buildah run -v "$PWD:$PWD" -- sh $PWD/init.sh

buildah config --entrypoint "/rootleveldir" --cmd "run.sh";

buildah commit                 \
  --author "Evan Carroll"      \
  --rm                         \
  "$ctr" "localhost:myimage";

```

Create the image without root with `buildah unshare ./build_image.sh`

Podman can commit any online container to an image, and buildah can commit any offline container to an image.

---
Tags: docker, kubernetes, containers
---


# Question
Title: ImagePullBackOff err while pulling docker image in different project in Google Cloud
I am trying to pull a private image from Artifact Registry repo in Google Cloud from a kubernetes cluster running in a different Google Cloud project using kubectl.

```
kubernetes version 1.20.15-gke.1000

```

The service account for the kubernetes has already been given artifactregistry.reader and storageobject.viewer permissions as the image is in a different project from the kubernetes service account

I apply the below yaml to the kubectl command.

kubectl apply -f proxy\_with\_workload\_identity.yaml

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: test-app
spec:
  selector:
    matchLabels:
      app: app-project
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 1
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: app-project
    spec:
      containers:
        - env:
            - name: DB_USER
              valueFrom:
                secretKeyRef:
                  key: username
                  name: db-credentials
            - name: DB_PASS
              valueFrom:
                secretKeyRef:
                  key: password
                  name: db-credentials
            - name: DB_NAME
              value: postgres
          image: "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1"
          name: app-project
          ports:
            - containerPort: 9376
              protocol: TCP
        - command:
            - /cloud_sql_proxy
            - "-instances=demo-dev:us-central1:1-sql-1=tcp:5432"
          image: "gcr.io/cloudsql-docker/gce-proxy:latest"
          name: cloud-sql-proxy
          resources:
            requests:
              cpu: 200m
              memory: 32Mi
          securityContext:
            runAsNonRoot: true
      serviceAccountName: testapp

```

The cloud-sql-proxy image is getting pulled and the container is running , but the image in the private-repository is not getting pulled "us-central1-docker.pkg.dev/myproject/docker-repo/test-app:v1"

when i check the pods i am shown this error:

# Answer
It seems that you need to add a pull-secret in the same namespace (seems that you work with default)

Command:

```
kubectl create secret docker-registry <your secret name like myregistry-docker-secret> --docker-server=<your-registry-server like us-central1-docker.pkg.dev> --docker-username=<your registry username> --docker-password=<your registry password> --docker-email=<your-email>

```

Ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

Then your manifest should look like

```
.....
    spec:
      imagePullSecrets:
        - name: <your secret name like myregistry-docker-secret>
      containers:
        - env:
......

```
> 0 votes

---
Tags: docker, kubernetes, google-cloud-platform, google-kubernetes-engine
---


# Question
Title: How can I create a dev. environnement using Azure DevOps?
I'm not sure if it's the right place to ask this question. So let me know, if it's not.

I'm building a front end app with VueJs. I actually use 2 computer. I use the first computer as a "server" (just a computer that is always on) and and the other one I use it to code.

The server have a VSCode that is always open that always run my front end app. I use Remote Desktop Connection to connect to this server when I need to code/build/run. I use Azure Dev Ops as a Git solution.

But all this setup is not the best way to work. It's kinda annoying to connect to the server and let vscode always open.

What I want to know is : Is it possible to create a better dev. environnement? For example, what I want to do is to be able to work from VSCode on my "dev's computer" and whenever I push something in Azure DevOps, my server re-build and re-run my app.

How should I start ? Can you give me any documentation to help me? I couldn't find anything in the net (maybe because I'm doing something wrong in my research)

# Answer
> 0 votes
I advice you to learn the basis with generic technologies and then use Azure.

## The basic and modern devops flow

* Devops admin configure the webhook or trigger to connect the git server with the ci,cd,devops sever
* Devops admin configure the job with steps required to build the application.
* Developer push some source code (java, php, c#, nodejs, etc)
* Your git platform/server (bitbucket, gitlab, github, azure, etc) detects this event and perform an http post request to your webhook url (preconfigured) sending it a json payload with important information related to detected event.
* You ci,cd,devops,etc server receives the http post request and using a json parser extract some important or common values used in devops automation like : repository name, branch name, commit author, commit message, etc
* Your ci,cd,devops,etc server launchs a preconfigured job. At this point you could start, launch or invoke any technology. (java, php, c#, nodejs, etc) required to build the application

More details here:

## Devops flow with open source

* git server: github, bitbucket, gitlab, etc
* ci,cd,devops server: jenkins, Harness CI,etc. Check this and this

## Devops flow with Azure

Steps

## Docker

I advice you Docker to skip the azure-pipeline.yml creation. With docker you app will be deployable on any server (gcp,aws,digital ocena, azure, bare metal, etc) in the galaxy. With azure-pipeline.yml just on Azure. Check this:

https://stackoverflow.com/questions/60604539/how-to-use-docker-in-the-development-phase-of-a-devops-life-cycle/60780840#60780840

## Kubernetes

If you achieve the previous steps, you could start the use of kubernetes

## Enterprises environments

If your company has the recommended environments like: staging, testing, uat, pre-prod, etc you will need a configuration management. Check this:

https://stackoverflow.com/questions/54220775/how-to-use-variable-substitution-in-frontend-js-applications-like-backend-applic/54271114#54271114

---
Tags: azure-devops, git, node.js
---


# Question
Title: EKS - Get availability zone of pod deployment from pod
In my EKS cluster, I have a few node groups that cover availability zone A and B. The pods running on the nodes are consumers/producers on various Kafka topics (MSK). There isn't any internal communication between the pods.

I have two brokers for the MSK cluster, one in each availability zone that the pods use.

**I am concerned that since the pods can be deployed in either availability zone, I might be unknowingly charged for data that is being transferred between the availability zones as the pod doesn't have a preference to which broker it talks to**

To avoid this, I am thinking of having the pod select the broker server to talk to once it has started, and to do this it needs to know what availability zone the underlying node its been deployed on.

I was imagining that it would be possible to have the deployment pull the zone information like any other spec field e.g. https://docs.microsoft.com/en-us/answers/questions/330925/how-can-a-pod-know-which-availability-zone-it-loca.html but that isn't a valid field.

I have tried hitting the ec2 metadata ip address like the link below but since its a pod with its own IP on the node, I assume this mechanism doesn't work. https://www.quora.com/How-do-I-find-out-in-which-availability-zone-a-AWS-EC2-server-is

None of the field listed https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.24/#podspec-v1-core seem to have any field that could identify this.

Is there a way of doing this? Or is there a better way of networking the pods/node groups?

# Answer
> 1 votes
The EC2 metadata endpoint has been updated to handle this situation but your current configuration may be preventing it. Have a read here for more: https://aws.amazon.com/about-aws/whats-new/2020/08/amazon-eks-supports-ec2-instance-metadata-service-v2/

> Now, newly launched and any updated EKS managed node groups will be configured with a metadata token response hop limit set to 2. For self-managed nodes, CloudFormation templates and eksctl have been updated to launch nodes by default with a hop limit of 2.

Given that you have the hop limit updated, you can use the IMDSv2 endpoint by first getting a token, and passing that through with your request

```
TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/

```

For availability zone:

```
curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/meta-data/placement/availability-zone/

```

---
Tags: kubernetes, amazon-web-services, eks, kafka
---


# Question
Title: error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied
When I run commands under `k3s kubectl`, I get

```
$ k3s kubectl version
WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions
error: error loading config file "/etc/rancher/k3s/k3s.yaml" : open /etc/rancher/k3s/k3s.yaml: permission denied

```

How should I resolve this? Should I change the permissions of `/etc/rancher/k3s/k3s.yaml`

# Answer
> 22 votes
# No, do not change permissions of `/etc/rancher/k3s/k3s.yaml`

First set up your an environmental variable for `KUBECONFIG=~/.kube/config`.

```
export KUBECONFIG=~/.kube/config

```

Then let's generate the file at that location. Your `k3s.yaml` file should **NOT** be world readable.. This is by-design. It should be owned by root and set to `0600`. Instead copy the config locally as described here,

```
mkdir ~/.kube 2> /dev/null
sudo k3s kubectl config view --raw > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"

```

You can add `KUBECONFIG=~/.kube/config` to your `~/.profile` or `~/.bashrc` to make it persist on reboot.

---
Tags: rancher, k3s
---


# Question
Title: How to verify docker sha256 on hub.docker with docker?
Given an image like this one

I want to be able to verify that the image I have matches

`DIGEST:sha256:6889517fd553d36ed5743827cd81c8f94438bd2502db57e9bbfbfac73e746f5b`

How can this be done in docker's command line tool?

# Answer
Use this command

```
docker images --digests

```

You'll get this output

```
REPOSITORY    TAG       DIGEST                                                                    IMAGE ID       CREATED        SIZE
hello-world   latest    sha256:80f31da1ac7b312ba29d65080fddf797dd76acfb870e677f390d5acba9741b17   feb5d9fea6a5   8 months ago   13.3kB

```
> 3 votes

# Answer
For a given image, these are listed in the RepoDigests section of the inspect:

```
$ docker image inspect nginx --format '{{json .RepoDigests}}'
["nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36","localhost:5000/library/nginx@sha256:644a70516a26004c97d0d85c7fe1d0c3a67ea8ab7ddf4aff193d9f301670cf36"]

```

Note that there are multiple digests for an image. One for a manifest list when you have multi-platform images (this contains the digests for platform specific images). Another for a platform specific image, which is what Hocker Hub shows on their website. And that image manifest contains digests for the image config and layers. The image config is the json describing things like the entrypoint, user, and other settings associated with the image, and this is used as the image ID in Docker.

That image ID is going to be unique to an image since it also contains a version of the layer digests (those digests will be different since they're uncompressed and other digests for layers will be on the compressed version). So if you don't have a repo digest, you can compare the image ID to the config digest to see if they match. E.g.

```
$ regctl manifest get localhost:5000/library/alpine --platform linux/arm64
Name:        localhost:5000/library/alpine
MediaType:   application/vnd.docker.distribution.manifest.v2+json
Digest:      sha256:c3c58223e2af75154c4a7852d6924b4cc51a00c821553bbd9b3319481131b2e0
Total Size:  2.694MB

Config:
  Digest:    sha256:6e30ab57aeeef1ebca8ac5a6ea05b5dd39d54990be94e7be18bb969a02d10a3f
  MediaType: application/vnd.docker.container.image.v1+json
  Size:      1487B

Layers:

  Digest:    sha256:b3c136eddcbf2003d3180787cef00f39d46b9fd9e4623178282ad6a8d63ad3b0
  MediaType: application/vnd.docker.image.rootfs.diff.tar.gzip
  Size:      2.694MB

$ docker image inspect alpine --format '{{.ID}}'
sha256:e66264b98777e12192600bf9b4d663655c98a090072e1bab49e233d7531d1294

```

That config digest is only useful for comparison, you still need the manifest digest if you want to pull the image from the registry.
> 1 votes

---
Tags: docker, images
---


# Question
Title: How can I unevenly distribute pods across Instance Groups
What I'd like to do is distribute 10% of my pods to one instance group with the other 90% on the other group.

I would like to experiment with using different AWS instance types (AMD, graviton, etc) but I only want to put a limited number of pods onto these instances. Ideally, I'd like to do it by service.

I've looked into PodTopologySpread but it appears to try to get roughly even distributions within a range. Alternatively, set the pod scheduler to prefer the different instance type and just limit how many nodes I can run but that doesn't work well if multiple services are in the instance group. The final option is to have two deployment objects with different node counts and but the same service selectors. That would work but seems like a work to tweak pod distributions.

How can I distribute a small number of pods to different instance types?

# Answer
> 0 votes
you are going the right way with PodTopologySpreadConstraints(PTSC for short) to have different instance of the same pod in an uneven distribution through all your k8s nodes. On that note, I would remind you that you can pair various PTSC rules to achieve your desired goal. Based on instance type labels as well as per availability zone.

But what is missing, and could be a solution to your problem, is to incrementally send a slice of your traffic to those pods in order to do a comparison in performance or whatever you want to compare.

You should look into what your ingress controller can manage. For instance, NGINX ingress-controller uses canary deployments and Traefik ingress-controller uses Green-blue deployments to send a certain percentage of traffic to those "test" pod and once you have your metrics you can go on and complete the rollout.

---
Tags: kubernetes, amazon-ec2, kops
---


# Question
Title: Unable to connect from my local system to ec2 instance created by terraform script
Following is the source code:

```
variable "ec2_instance_type_name" {
    type    = string
    default = "t2.nano"
}

terraform {
    required_providers {
        aws = {
            source = "hashicorp/aws"
            version = "~> 3.27"
        }
    }
}

provider "aws" {
    alias   = "us"
    region  = "us-east-1"
}

provider "aws" {
    alias   = "eu"
    region  = "eu-west-1"
}

data "aws_ami" "amazon_2" {
    provider = aws.eu
    most_recent = true

    filter {
        name = "name"
        values = ["amzn2-ami-kernel-*-hvm-*-x86_64-gp2"]
    }
    owners = ["amazon"]
}

data "http" "myip" {
    url = "http://ipv4.icanhazip.com"
}

resource "aws_vpc" "docdb_peer" {
    provider                = aws.eu
    cidr_block              = "172.32.0.0/16"
    enable_dns_support      = true
    enable_dns_hostnames    = true
}

resource "aws_internet_gateway" "gw_connect" {
    provider = aws.eu
    vpc_id = aws_vpc.docdb_peer.id
}

resource "aws_security_group" "vpc_sg" {
    provider = aws.eu
    vpc_id       = aws_vpc.docdb_peer.id
    name         = "vpc-connect"
    description  = "VPC Connect"

    ingress {
        cidr_blocks = ["${chomp(data.http.myip.body)}/32"]
        from_port   = 22
        to_port     = 22
        protocol    = "tcp"
    }

    egress {
        from_port   = 0
        to_port     = 0
        protocol    = "-1"
        cidr_blocks = ["0.0.0.0/0"]
    }
}

resource "aws_subnet" "main" {
    provider = aws.eu
    vpc_id                    = aws_vpc.docdb_peer.id
    availability_zone         = "eu-west-1a"
    cidr_block                = "172.32.0.0/20"
    map_public_ip_on_launch   = true
}

resource "aws_instance" "tunnel-ec2" {
    provider = aws.eu
    vpc_security_group_ids = ["${aws_security_group.vpc_sg.id}"]
    subnet_id     = aws_subnet.main.id
    ami           = data.aws_ami.amazon_2.id
    instance_type = var.ec2_instance_type_name
    key_name      = "ireland_ofc_new"
    depends_on = [aws_internet_gateway.gw_connect]
}

```

I try to ssh into the system using the key pair pem file and it just timeout. My other ec2 instance which I manually created works just fine. Please help resolve the issue.

# Answer
The issue was that the default route was missing in the routing table.

```
resource "aws_route" "update" {
    provider               = aws.docdb_peer
    route_table_id         = "${aws_vpc.docdb_peer.default_route_table_id}"
    destination_cidr_block = "0.0.0.0/0"
    gateway_id             = "${aws_internet_gateway.gw_connect.id}"
}

```

Adding this solved the issue.
> 0 votes

---
Tags: terraform, amazon-ec2, aws-vpc, aws-security-group, aws-subnets
---


# Question
Title: Terraform plan does not update AWS Task Definition with last active revision value
I would like your help on a situation that I'm facing with terrafom. My company recently launched a new website whose wordpress is hosted in an AWS ECS container. I had to make some adjustments to the site's infrastructure and I noticed that, when making the terraform plan, I get a message that there were changes outside the terraform, one of which is the revision number of the ECS Task Definition. But looking further down, when terraform plans the ECS Task Definition, instead of keeping the most recent revision number (31), he goes back to the last version that was applied by terraform (19). I suppose that this number is recorded in the TF state. My question is: is there any way (parameter or attribute) to perform the terraform plan and make it plan the ECS Task Definition with the latest active version deployed in the ECS container, in this case 31? The deployment is done through a bitbucket pipeline and the image is stored in the ECR.

This is the steps we run in bitbucket:

```
- step:
      oidc: true
      name: Get task-definition
      image: amazon/aws-cli:2.5.0
      artifacts:
        - task-definition.json
      script:
        - yum install jq -y
        - export AWS_REGION=us-east-1
        - export AWS_ROLE_ARN=arn:aws:iam::my-account:role/ci-ecs
        - export AWS_WEB_IDENTITY_TOKEN_FILE=$(pwd)/web-identity-token
        - echo $BITBUCKET_STEP_OIDC_TOKEN > $(pwd)/web-identity-token
        - TASK_DEFINITION=$(aws ecs describe-task-definition --task-definition site-institucional-prd)
        - ECR_IMAGE_TAG=145497587889.dkr.ecr.us-east-1.amazonaws.com/site-institucional-prd:${BITBUCKET_COMMIT}
        - NEW_TASK_DEFINTIION=$(echo "$TASK_DEFINITION" | jq --arg IMAGE "$ECR_IMAGE_TAG" '.taskDefinition | .containerDefinitions[0].image = $IMAGE | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities) | del(.registeredAt) | del(.registeredBy)')
        - echo $NEW_TASK_DEFINTIION > task-definition.json

  - step:
      oidc: true
      name: Deploy to ECS
      script:
        - pipe: atlassian/aws-ecs-deploy:1.6.1
          variables:
            AWS_DEFAULT_REGION: 'us-east-1'
            AWS_OIDC_ROLE_ARN: 'arn:aws:iam::my-account:role/ci-ecs'
            CLUSTER_NAME: 'emprc-fgt-prd'
            SERVICE_NAME: 'site-institucional-prd-fgt'
            TASK_DEFINITION: 'task-definition.json'

```

I'm using the internal terraform module.

This is the part of my terraform where we provision the ECS Sevice based on the module we have created:

```
################################################################################
# ECS FARGATE
################################################################################

module "wp" {
 source = "git::git@bitbucket.org:my-repo//terraform/aws-ecs/modules/fargate-service"

  environment = var.environment

  enabled = var.enabled

  // variables task definition

  container_name = var.container_name

  enable_execute_command = var.enable_execute_command

  task_container_image = var.task_container_image

  task_definition_cpu    = var.task_definition_cpu
  task_definition_memory = var.task_definition_memory

  task_container_environment = var.task_container_environment

  extra_container_defs = var.extra_container_defs

  // variables service

  vpc_id          = var.vpc_id
  subnet_ids      = var.subnet_ids
  security_groups = var.security_groups
  service_name    = var.service_name

  cluster_id = var.cluster_id

  desired_count = var.desired_count

  capacity_provider_strategy = var.capacity_provider_strategy

  target_groups = var.target_groups

  health_check = var.health_check

  http_header = var.http_header

  lb_arn       = var.lb_arn
  host_header  = var.host_header

  autoscale = var.autoscale

  tags = var.tags
}

```

Here is the result of my terraform plan with only the problem theme (I will mask some values in the plan below):

```
Terraform detected the following changes made outside of Terraform since the
last "terraform apply":

# module.wp.aws_ecs_service.service[0] has been changed
  ~ resource "aws_ecs_service" "service" {
        id                                 = "arn:aws:ecs:us-east-1:my-account:service/emprc-fgt-prd/my-name"
        name                               = "my-name"
        tags                               = {
            "Environment" = "prd"
            "Name"        = "my-name"
            "Owner"       = "Owner"
            "Project"     = "project-name"
            "Provider"    = "Terraform"
        }
      ~ task_definition                    = "arn:aws:ecs:us-east-1:my-account:task-definition/site-institucional-prd:19" -> "arn:aws:ecs:us-east-1:my-account:task-definition/site-institucional-prd:31"
        # (14 unchanged attributes hidden)

Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.
───────────────────────

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  ~ update in-place
 <= read (data resources)

Terraform will perform the following actions:
 # module.wp.aws_ecs_service.service[0] will be updated in-place
  ~ resource "aws_ecs_service" "service" {
        id                                 = "arn:aws:ecs:us-east-1:my-account:service/emprc-fgt-prd/site-institucional-prd-fgt"
        name                               = "my-name"
        tags                               = {
            "Environment" = "prd"
            "Name"        = "my-name"
            "Owner"       = "Owner"
            "Project"     = "project-name"
            "Provider"    = "Terraform"
        }
      ~ task_definition                    = "arn:aws:ecs:us-east-1:my-account:task-definition/site-institucional-prd:31" -> "arn:aws:ecs:us-east-1:my-account:task-definition/site-institucional-prd:19"
        # (14 unchanged attributes hidden)

```

So new version deployment (wordpress) is always done by the bitbucket pipeline once the infrastructure is in place. In theory we do not change the infra, but when we need to change something on it, the task definition revision number is changed by terraform trying to assume the revision in TF state which is out dated. I understand that this is the correct behavior once the deployment is done by the bitbucket pipeline, but I would like to know if there is a way to keep the current revision active in ECS during the plan/apply.

Thank you for your time and support.

# Answer
> 0 votes
From what I see in your terraform plan, `terraform apply` will indeed set the latest task definition version and probably run redeploy of your ECS.

You can keep your state file up-to-date with `terraform plan --refresh-only` to see the possible changes in infrastructure done manually, and if you are ok with it, `terraform apply --refresh-only` will save those changes in your terraform state file.

Here is some more info on that subject.

Hope it helps

---
Tags: amazon-web-services, terraform, amazon-ecs, bitbucket-pipelines
---


# Question
Title: Unable to authenticate with Terraform AWS provider
I am trying to use terraform to spin up a new instance of a server.

Using the AWS provider , I get this error:

```
╷
│ Warning: Resource targeting is in effect
│
│ You are creating a plan with the -target option, which means that the result of this plan may not represent all of the changes requested by the current configuration.
│
│ The -target option is not for routine use, and is provided only for exceptional situations such as recovering from errors or mistakes, or when Terraform specifically suggests to use it as part of an
│ error message.
╵
╷
│ Error: error configuring Terraform AWS Provider: failed to get shared config profile, 2auth
│
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on main.tf line 1, in provider "aws":
│    1: provider "aws" {
│

```

first off , I have no idea where this 2auth profile it is referencing is.

Also I have tried setting my credentials with in both the tf file

```
provider "aws" {
   region = var.aws_region
   shared_credentials_file = "/Users/samueldare/.aws/credentials"
}

```

and as environment variables, but it still doesnt work.

I would appreciate pointers as to what I am doing wrong.

# Answer
> 1 votes
If you have multiple profiles in your credentials file, you need to state which profile you want to use, otherwise it will use `[default]`:

credentials file:

```
[default]
aws_access_key_id = xxxxxxxx
aws_secret_access_key = xxxxxxxx
[profile2]
aws_access_key_id = yyyyyyyy
aws_secret_access_key = yyyyyyyyy

```

terraform code:

```
provider "aws" {
   region = var.aws_region
   shared_credentials_file = "/Users/samueldare/.aws/credentials"
   profile = "profile2"
}

```

As stated in the docs

---
Tags: amazon-web-services, terraform
---


# Question
Title: What is difference between testing in context of CI and CT pipelines?
What is the difference between testing in the context of Continuous Integration pipeline vs testing in context of Continuous Testing pipeline? The definition of testing is a bit blurred to me in the context of CT pipeline.

An example, I understand testing in CI where a Unit Test is executed on a Pull Request - personally I'd defined it as a Continuous Testing since I continuously test incoming code however, this practice is known as CI. Then, what is CT then? Am I only one confusing CI for CT?

# Answer
Continuous Testing is a side effect of Continuous Integration. CI's main purpose is to integrate your development efforts into a production-ready state as often as possible. The production-readiness is granted via continuously testing the recently integrated code. Continuous Integration does not work properly without Continuous Testing, but you could continuously test without making your development code production ready all the time. In my experience the CT term is not really in use anymore.
> 2 votes

---
Tags: continuous-integration
---


# Question
Title: How to keep packages updated and keep in pace with security updates
In a project with multiple vms that run various applications from Wordpress to Django and various other things.

Is there a way or a tool to keep track of the packages that require updates or have security problems not on an app level but on a vm level. Do I have to ssh into each machine and keep track of the packages or is there a smarter way to do that, could a tool like ansible be used for this job?

# Answer
You could use pre-made roles like this one or this one or this one to manage apt for system related security upgrades.
> 0 votes

---
Tags: ansible, security
---


# Question
Title: How to do maintenance work in kubernetes cluster
We just moved to kubernetes, but the engineer that helped launch it is on paternity leave before we had hoped (never trust a baby not to be eager!).

Now we're trying to do maintenance tasks and one off work and having nodes get killed in the middle of things.

I've looked at using kubernetes jobs for this, but that's overkill. We don't want to write manifest files for everything.

We just need long lived shell access to do this and that.

What's the pattern for this so your maintenance task doesn't get killed?

# Answer
We were eventually able to answer this by following the rules for when a node is terminated. According to the FAQ, there are a number of ways that pods can prevent the cluster autoscaler from removing a node. One type of pod is:

> Pods that are not backed by a controller object (so not created by deployment, replica set, job, stateful set etc).

So our solution is to create a pod in that way via a manifest file. This lets us have a pod named `maintenance` that sticks around and isn't killed by the cluster autoscaler:

```
---
apiVersion: v1
kind: Pod
metadata:
  name: maintenance
  namespace: blah
  labels:
    type: maintenance
spec:
  containers:
    - name: web
      image: whatever
      imagePullPolicy: IfNotPresent
      command: [bash]
      stdin: true
      tty: true

```
> 1 votes

---
Tags: kubernetes, sysadmin
---


# Question
Title: Helm Error: INSTALLATION FAILED : manifests contain a resource that already exists
When I run the helm command to install `my-mysql-operator`, I get

```
helm install my-mysql-operator mysql-operator/mysql-operator --namespace mysql-operator --create-namespace

```

I get the following error,

> Error: INSTALLATION FAILED: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "mysql-operator" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "mysql-operator": current value is "mysql"

So so the resource exists, how do I delete it?

# Answer
> 2 votes
You can see everything helm has installed with

```
helm list --all-namespaces

```

Which should return something like this

```
❯ helm list --all-namespaces
NAME                    NAMESPACE       REVISION        UPDATED                                 STATUS          CHART                   APP VERSION
my-mysql-operator       mysql           1               2022-05-26 12:18:28.652497947 -0500 CDT deployed        mysql-operator-2.0.4    8.0.29-2.0.4

```

The problem here is that the resource `my-mysql-operator` is currently installed into the `mysql` namespace. In order to delete it and recreate it you can do,

```
helm uninstall -n mysql my-mysql-operator

```

Then you should be able to run the above command.

---
Tags: helm
---


# Question
Title: How can I find what options I can set with a `helm install` that the chart provides?
Let's say I want to install an arbitrary helm chart like `mysql-operator/mysql-innodbcluster` how can I find what options I set with `--set`?

# Answer
> 1 votes
You can use `show values` to see what options the chart provides,

```
helm show values mysql-operator/mysql-innodbcluster

```

What you'll get is a YAML file, here is a part of the above YAML output,

```
tls:
  useSelfSigned: false
#  caSecretName:
#  serverCertAndPKsecretName:
#  routerCertAndPKsecretName:

```

You an set the option for `useSelfSigned` as `tls.useSelfSigned` like this:

```
helm install mycluster mysql-operator/mysql-innodbcluster --set tls.useSelfSigned=true

```

---
Tags: helm
---


# Question
Title: How to apply k8s manifest without scaling replicas
We have a manifest file that defines a set number of replicas and an autoscaler that scales them up. The problem we have is when we apply the manifest, it immediately scales the replicas down to the number in the manifest. This eventually is fixed by the autoscaler, but in the meantime, it's not great!

Is there a way to run a manifest and ignore the `replica` parameter?

We tried removing it, but then it scaled to the default count of 1.

# Answer
The application boot up might be using resources above your set utilisation target before settling down. What you can do is to either:

* give it a bit more resource request
* increase the threshold of the HPA utilisation target
* increase the autoscaler interval check
> 0 votes

---
Tags: kubernetes, autoscaling
---


# Question
Title: Trigger Jenkins job from Gitlab CI with parameters
There is supposed to be a way to trigger a Jenkins job via GitlabCi, using the respective plugin.

My question is whether there is a way:

a) to trigger a parameterized Jenkins job

b) to pass parameters when triggering the job

# Answer
> 11 votes
Here's the way I do it: no plugin required, just triggering Jenkins `api` from gitlab-ci.

## Gitlab-CI

I will assume you have a gitlab-ci runner installed and configured.

First, you need to have a `.gitlab-ci.yml` file in your project having a basic structure such as:

```
stages:
- my-jenkins-trigger
variables:
 MY_VARIABLE: "EVERYTHING_IS_AWESOME"

my-jenkins-trigger-job:
 stage: my-jenkins-trigger
 script: curl -i -X POST --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/buildWithParameters?MY_JENK_PARAM=${MY_VARIABLE}

```

In the above, I also assume

* You have a Jenkins job somewhere at URL `JENKINS_JOB_URL`
* this Jenkins job has a build parameter called `MY_VARIABLE`
* `JENKINS_USER`, `JENKINS_TOKEN` are defined \[\*\]

## That simple?

Well yes, but no...

That is the rough structure. That script will merely trigger a Jenkins job and forget about it. You need to work a little more to monitor the job and feed its status back in Gitlab-CI, manage security and possibly get some commit info from gitlab to inject into your job.

## Monitoring

In order to have a proper monitoring, I recommand to write a full *trigger + monitor + return value* script \[\** \] (in whatever language available or you're familiar with).

Just start by triggering the job as I stated above.

Then, run a `while` loop (don't forget to put it to `sleep` \[\***\]) on

```
curl --silent --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/lastBuild/api/json | grep result\":null > /dev/null

```

until the result of this command is not `0`.

Once the Jenkins job is finished, you would probably want to fetch the job's console in Gitlab

```
curl -i -X POST --user JENKINS_USER:JENKINS_TOKEN JENKINS_JOB_URL/lastBuild/consoleText

```

Finally you may `curl` once more on `JENKINS_JOB_URL/lastBuild/api/json` but this time you `grep` it on `UNSTABLE`, `SUCCESS` or `FAILURE`.

## Discussion

By following the guidelines above, you can fully orchestrate Jenkins jobs from Gitlab-CI. I've posted a long discussion on why and when should you do this.

I hope this will help you.

---

\[\*\] Your Gitlab project Settings \> CI/CD \> Secret vaiables
\[\** \] Of course I mean by that to **craft** a script nicely with params, functions, nice variable names, meaningful logs... You name it.
\[\***\] I found a `sleep` of 20 seconds worked for me

# Answer
> 3 votes
FWIW, the trigger you referenced originates from Gitlab repository events, not from a GitlabCI execution.

The only possibility (that crosses my mind) of triggering a jenkins job from inside a GitlabCI execution is by having a (custom?) script invoked as part of the GitlabCI execution which remotely activates a Parameterized Trigger Plugin configured for your jenkins job, via a properly-crafted POST request which would include the desired parameters.

See also https://stackoverflow.com/questions/20359810/how-to-trigger-jenkins-builds-remotely-and-to-pass-parameters.

# Answer
> 0 votes
If you want to keep it simple, try the generic webhook trigger plugin.

https://plugins.jenkins.io/generic-webhook-trigger

You can trigger a build by sending an http POST using a JSON body or URL parameters.

Parse the JSON request

```
def req = readJSON text: payload

```

Now you can use it in your pipeline assuming you had a deploy function.

```
deploy(req.environment, req.branch)

```

I must agree with @Tensibai. Two CI/CD systems on the surface does seem overly complex. You might want to consider sticking with one if possible.

# Answer
> 0 votes
I use a similar setup as this answer.

Difference is **I cannot poll the latest job because it may refer to a job I didn't start**. So instead I get the queue ID from the response headers when triggering a job, and match that with the queue ID of most recent job runs.

I did this in Python because it makes things a bit easier, but it could also be done using sh.

GitLab CI config:

```
Test:
  stage: test
  script:
    - run_jenkins_job.py
      --docker-image "${CI_REGISTRY_IMAGE}"
      --display-name "Test ${CI_COMMIT_REF_SLUG}"

```

run\_jenkins\_job.py:

```
#!/usr/bin/env python3
import argparse
import sys
import time

import requests

JENKINS_JOB_URL = "https://jenkins.example.org/job/MYJOB"
JOB_BUILD_TOKEN = "MYJOBTOKEN"

api_req = requests.Session()
api_req.headers.update(
    {
        "Authorization": "Basic BASIC_USER_AUTH",
        "Content-Type": "application/x-www-form-urlencoded",
    }
)

def main():
    parser = argparse.ArgumentParser(description="Run a GitLab MR job on Jenkins and poll for the result.")
    parser.add_argument("--docker-image", required=True, type=str)
    parser.add_argument("--display-name", required=True, type=str)
    args = parser.parse_args(sys.argv[1:])

    queue_id = schedule_build(args)
    exit_code = poll_build(queue_id)
    sys.exit(exit_code)

def schedule_build(args: argparse.Namespace) -> int:
    response = api_req.post(
        f"{JENKINS_JOB_URL}/buildWithParameters?token={JOB_BUILD_TOKEN}",
        data={
            "docker_image": args.docker_image,
            "display_name": args.display_name,
        },
    )
    response.raise_for_status()
    location_header_value = response.headers["Location"]
    # Example location header: "https://jenkins.example.org/queue/item/123/"
    return int(location_header_value.rstrip("/").split("/")[-1])

def poll_build(queue_id: int) -> int:
    job_url_shown = False

    for _ in range(300):
        time.sleep(15)
        response = api_req.get(f"{JENKINS_JOB_URL}/api/json?tree=builds[result,queueId,url]")
        response.raise_for_status()

        for build in response.json()["builds"]:
            if build["queueId"] != queue_id:
                continue
            elif build["result"] is None:
                if not job_url_shown:
                    # Scheduled job first appeared; show URL.
                    print(f"\nRunning: {build['url']}\n")
                    job_url_shown = True
                print(".", end="", flush=True)
            else:
                if build["result"] == "SUCCESS":
                    return 0
                else:
                    return 1

if __name__ == "__main__":
    main()

```

---
Tags: jenkins, continuous-integration, gitlab
---


# Question
Title: How to solve my error about saving the global configuration before it is loaded in Jenkins?
I recently upgraded Jenkins from 2.204.4 to 2.332.1. In doing so, I received the following error.

```
java.lang.IllegalStateException: An attempt to save the global configuration was made before it was loaded
    at jenkins.model.Jenkins.save(Jenkins.java:3519)
    at jenkins.model.Jenkins.saveQuietly(Jenkins.java:3546)
    at jenkins.model.Jenkins.setSecurityRealm(Jenkins.java:2743)
    at jenkins.model.Jenkins$15.run(Jenkins.java:3481)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:175)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1156)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:222)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:121)
    at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
    at java.lang.Thread.run(Unknown Source)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:291)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:49)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1191)
    at jenkins.model.Jenkins.<init>(Jenkins.java:981)
    at hudson.model.Hudson.<init>(Hudson.java:86)
    at hudson.model.Hudson.<init>(Hudson.java:82)
    at hudson.WebAppMain$3.run(WebAppMain.java:247)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:264)

```

I was able to "fix" this by commenting out the and blocks in config.xml, but that had the side effect of wiping out our Active Directory settings as well as all Role Based permissions.

I saved the settings, so I was able to manually set everything back up, but now after a Jenkins reboot, I get the same error.

I noticed in the Jenkins error log it is throwing the following error for all users on startup

```
SEVERE  hudson.model.User#loadFromUserConfigFile: Failed to load E:\jenkins\users\{user}\config.xml
java.nio.channels.ClosedByInterruptException

```

I am rather new to Jenkins, so I am not sure what would be causing this. Has anyone encountered this or know of a fix?

Edit: As mentioned in my comment I discovered the error is due to the role-strategy plugin. I attempted to use the Configuration as Code plugin, thinking that this would load the config.xml settings at the correct time, but that is not the case.

The issue is that there is still info in `<authorizationStrategy>` block in config.xml when Jenkins restarts... is there a way to clear that?

# Answer
> 1 votes
After much investigation, I came across the fact that some plugins, specifically the Matrix Authorization Strategy Plugin and role-strategy-plugin were in a deadlock state where they both would not be checked under "compatible" to update, as they were dependent on each other being updated.

Being fairly knew to Jenkins, and not wanting to break things, I only updated the compatible plugins, however; I missed this deadlock type issue.

Updating both of these plugins fixed the issue.

# Answer
> 1 votes
Adding to @jander69 research,

I performed the below steps and it worked:

1. Stop the jenkins server using `jenkins.exe stop` command.
2. In the jenkins installation folder, open `Plugins` folder and delete below listed plugins and their `.jpi` files.

i. `role-strategy` folder and `role-strategy.jpi`

ii. `matrix-auth` folder and `matrix-auth.jpi`
3. In the same jenkins installation folder, open `config.xml` file in any editor.
4. Delete the section with tag `<authorizationStrategy> ... </authorizationStrategy>` and save `config.xml` file.
5. Start the jenkins server using `jenkins.exe start` command.

Now, Jenkins should load as expected without the `IllegalStateException: An attempt to save the global configuration was made before it was loaded` error.

---
Tags: jenkins
---


# Question
Title: Where does k3s store it's "/var/lib/kubelet/config.yaml" file?
Referencing another answer here, they suggest setting

```
evictionHard:
  imagefs.available: 1%
  memory.available: 100Mi
  nodefs.available: 1%
  nodefs.inodesFree: 1%

```

In the file `/var/lib/kubelet/config.yaml`. However, I do not see that file in my k3s distribution of kubernetes. Where is this file located with k3s?

# Answer
> 6 votes
# `/etc/rancher/k3s/config.yaml`

***Note, `/etc/rancher/k3s/config.yaml` not `/etc/rancher/k3s/k3s.yaml`! You may have to create the file if it doesn't exist.***

K3s doesn't have a dedicated kubelet service. It's integrated into k3s. So it's configured only with the `--kubelet-arg` flag to k3s. But you can add Kubelet config directives to your `config.yaml` (`/etc/rancher/k3s/config.yaml`),

```
kubelet-arg:
  - "kube-reserved=cpu=500m,memory=1Gi,ephemeral-storage=2Gi"
  - "system-reserved=cpu=500m, memory=1Gi,ephemeral-storage=2Gi"
  - "eviction-hard=memory.available<500Mi,nodefs.available<10%"

```

Which would look like,

```
kubelet-arg:
  - "eviction-hard=imagefs.available<case>,memory.available<case>,nodefs.available<case>,nodefs.inodesFree<case>"

```

You can find more information at,

---
Tags: configuration, k3s, kubelet
---


# Question
Title: k3s: Eviction manager: attempting to reclaim" resourceName="ephemeral-storage
When I run `kubectl get pods -w --namespace kubeapps` I see all my pods are pending

```
NAME                                                         READY   STATUS    RESTARTS   AGE
kubeapps-internal-dashboard-584c686cc6-qwjnf                 0/1     Pending   0          19h
kubeapps-585bc584db-j994c                                    0/1     Pending   0          19h
kubeapps-internal-apprepository-controller-5d7685c4f-zsxnk   0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-ddp8z                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-mzbtn              0/1     Pending   0          19h
kubeapps-internal-dashboard-584c686cc6-wxnfd                 0/1     Pending   0          19h
kubeapps-postgresql-0                                        0/1     Pending   0          19h
kubeapps-585bc584db-rgvfq                                    0/1     Pending   0          19h
kubeapps-internal-kubeops-7686579468-tzjwf                   0/1     Pending   0          19h
kubeapps-internal-kubeappsapis-6f65bccf89-rbdrm              0/1     Pending   0          19h

```

When I check `journalctl -u k3s`, I see

```
E0604 11:37:07.861130    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik: object "kube-system"/"chart-content>
E0604 11:37:07.861184    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062023    1301 configmap.go:200] Couldn't get configMap kube-system/chart-content-traefik-crd: object "kube-system"/"chart-con>
E0604 11:37:08.062042    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik: object "kube-system"/"chart-values-t>
E0604 11:37:08.062060    1301 configmap.go:200] Couldn't get configMap kube-system/chart-values-traefik-crd: object "kube-system"/"chart-valu>
E0604 11:37:08.062089    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/84367524-fc49-4db2-bf08-b217>
E0604 11:37:08.062099    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
E0604 11:37:08.062105    1301 nestedpendingoperations.go:335] Operation for "{volumeName:kubernetes.io/configmap/46bd29b8-1c81-4585-a8fb-ca7e>
I0604 11:37:08.079010    1301 eviction_manager.go:338] "Eviction manager: attempting to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.079027    1301 container_gc.go:85] "Attempting to delete unused containers"
I0604 11:37:08.079496    1301 image_gc_manager.go:327] "Attempting to delete unused images"
I0604 11:37:08.082310    1301 eviction_manager.go:349] "Eviction manager: must evict pod(s) to reclaim" resourceName="ephemeral-storage"
I0604 11:37:08.082338    1301 eviction_manager.go:367] "Eviction manager: pods ranked for eviction" pods=[kube-system/coredns-d76bd69b-2svkn >
E0604 11:37:08.082353    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/coredns-d76bd69b-2svk>
E0604 11:37:08.082359    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/metrics-server-7cd5fc>
E0604 11:37:08.082363    1301 eviction_manager.go:560] "Eviction manager: cannot evict a critical pod" pod="kube-system/local-path-provisione>
I0604 11:37:08.082368    1301 eviction_manager.go:390] "Eviction manager: unable to evict any pods from the node"

```

I think this is a result of the eviction manager, because my system is *low* on disk space? Perhaps it's just looking at percent free, the below is the device from `df -h`

I'm also seeing this in the error log,

```
k3s[1301]: I0604 11:52:13.741674    1301 image_gc_manager.go:310] "Disk usage on image filesystem is over the high threshold, trying to free bytes down to the low threshold" usage=98 highThreshold=85 amountToFree=337223212236 lowThreshold=80
k3s[1301]: E0604 11:52:13.742880    1301 kubelet.go:1347] "Image garbage collection failed multiple times in a row" err="failed to garbage collect required amount of images. Wanted to free 337223212236 bytes, but freed 0 bytes"

```

```
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/x1c7--vg-root  1.8T  1.7T   50G  98% /

```

Where are these settings on k3s?

# Answer
This is because of the default settings for the eviction manager (currently 5% hard, and 10% minimum for image reclaim) safeguard a new pod exhausting the space on your hard disk, and the metrics are a bit overly conservative. You can change the eviction manager settings by providing a `--kubelet-arg` that overrides the k3s defaults.. This can be persisted by providing `kubelet-arg` to your k3s `KUBECONFIG` at `/etc/rancher/k3s/config.yaml`.

Like this,

```
kubelet-arg:
  - "eviction-minimum-reclaim=imagefs.available=2%,nodefs.available=2%"
  - "eviction-hard=memory.available<500Mi,nodefs.available<10Gi"

```

---

Note, while currently required, passing options by commandline to k3s is deprecated in upstream Kubernetes and is subject to removal in k3s when/if it gets finally removed.
> 1 votes

---
Tags: rancher, k3s, kubelet
---


# Question
Title: Where can I find the default arguments supplied to the internal Kublet Service in k3s?
If I want to find the default-arguments to the internal Kublet service, where can I find these documented?

# Answer
> 0 votes
I wasn't able to find them documented, but I did find them in the code in `agent_linux.go` (note there are difference options for windows)

You can see them here,

```
argsMap := map[string]string{
    "healthz-bind-address":         bindAddress,
    "read-only-port":               "0",
    "cluster-domain":               cfg.ClusterDomain,
    "kubeconfig":                   cfg.KubeConfigKubelet,
    "eviction-hard":                "imagefs.available<5%,nodefs.available<5%",
    "eviction-minimum-reclaim":     "imagefs.available=10%,nodefs.available=10%",
    "fail-swap-on":                 "false",
    "cgroup-driver":                "cgroupfs",
    "authentication-token-webhook": "true",
    "anonymous-auth":               "false",
    "authorization-mode":           modes.ModeWebhook,
}

```

---
Tags: k3s, kubelet
---


title: Retrieving arn from another module

text:
# Question
Title: Retrieving arn from another module
I'm trying to get the arn of the kms key for use in the S3 bucket `kms_master_key_id`, the code below is how I thought it might work. I can output the arn value to the root but am unsure how to access this in the S3 resource.

main.tf

```
  module "s3_bucket" {
  source = "./modules/S3"
}

  module "kms" {
  source = "./modules/kms"
}

```

outputs.tf

```
output "kms_arn" {
  value = module.kms.kms_arn
}

```

modules/kms/kms.tf

```
resource "aws_kms_key" "key" {
  policy = data.aws_iam_policy_document.cmk.json
}

resource "aws_kms_alias" "key_alias" {
  name          = "alias/kms-key"
  target_key_id = aws_kms_key.key.id
}

```

modules/kms/\_outputs.tf

```
output "kms_arn" {
  value = aws_kms_key.key.arn
}

```

modules/S3/main.tf

```
resource "aws_s3_bucket" "bucket" {

  bucket = "09432804238423098"
  acl    = "private"
  versioning {
    enabled = false
  }
  force_destroy = true
  server_side_encryption_configuration {
   rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = ../aws_kms_key.key.arn
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

```

# Answer
> 0 votes
main.tf

```
  module "s3_bucket" {
  source = "./modules/S3"
  kms_master_key_id = module.kms.kms_arn
}

  module "kms" {
  source = "./modules/kms"
}

```

modules/kms/kms.tf

```
resource "aws_kms_key" "key" {
  policy = data.aws_iam_policy_document.cmk.json
}

resource "aws_kms_alias" "key_alias" {
  name          = "alias/kms-key"
  target_key_id = aws_kms_key.key.id
}

```

modules/kms/\_outputs.tf

```
output "kms_arn" {
  value = aws_kms_key.key.arn
}

```

modules/S3/main.tf

```
resource "aws_s3_bucket" "bucket" {

  bucket = "09432804238423098"
  acl    = "private"
  versioning {
    enabled = false
  }
  force_destroy = true
  server_side_encryption_configuration {
    rule {
      apply_server_side_encryption_by_default {
        kms_master_key_id = var.kms_master_key_id
        sse_algorithm     = "aws:kms"
      }
    }
  }
}

```

modules/S3/variables.tf

```
variable "kms_master_key_id" {
  default = ""
}

```

---
Tags: terraform
---


title: Server-side Gitlab URL rewriting?

text:
# Question
Title: Server-side Gitlab URL rewriting?
Our old BitBucket server had a URL like this,

And, we're wanting to move to GitLab with a DNS switch which has a different URL-convention namely

The `/scm/` is what I'd like to fix. Is there a way to rewrite URLs through configuration of the GitLab server such that I can make

Redirect to

Note, I know I can change the "Custom Git clone URL for HTTP(S)" and *"Replaces the clone URL root"* under GitLab's `admin/application_settings/general`.

# Answer
> 0 votes
GitLab omnibus and self-managed installs ship with NGINX which is exposed and documented here,

You basically edit the `gitlab.rb` file, edit `nginx['custom_gitlab_server_config']` and then rerun `gitlab-ctl reconfigure` to regenerate the nginx confs and restart nginx.

---
Tags: gitlab, dns, url
---


title: Accessing k8s outside of VM

text:
# Question
Title: Accessing k8s outside of VM
I am running k8s cluster for evaluation purposes inside some vagrant box. I have external kubectl access. I was able to get PostgreSQL helm template running on the cluster. But now let say I want to access it outside of the k8s(or VM actually).

There is a problem that I am not very educated in the niceties of Host to Quest communication in VM not to speak about the k8s stack. By empirical research I have found that turning service to NodePort makes it available outside the k8s. But as I said the installed PostgreSQL service is ClusterIP so I want to access it outside aswell without finding a way to turn it to NodePort. So there is couple of noob questions here

1. what is difference between `kubectl proxy` and `kubectl port-forward`?
2. what is different when executing `kubectl proxy` from inside VM running k8s cluster and running the same kubectl command on the host?
3. what is the usual way to make services available outside of the cluster, when using the cluster just for development?
4. is this quote true?
> port forwarding is not needed when using private network and accessing guest from host with guest private ip, port forwarding is used to access guest with localhost/127.0.0.1 the ports are then forwarded

# Answer
> 2 votes
1. > kubectl port-forward forwards connections to a local port to a port on a pod. Compared to kubectl proxy, kubectl port-forward is more generic as it can forward TCP traffic while kubectl proxy can only forward HTTP traffic. Generally speaking, using port forwarding you could get on your ‘localhost’ any services launched in your cluster.

https://stackoverflow.com/questions/58360323/difference-between-kubectl-port-forwarding-and-proxy
2. It does not make a lot of sence to run 'kubectl proxy' from inside VM running k8s cluster. Kubectl is your client tool, so stick with it outside of cluster range.
3. The way you would access production services, for the most part if your service is DB oriented stick with ClusterIP. If you want to externally access some Web/HTTP service expose it as a Load Balancer or an Ingress ( in case u have N of them ). If you want to test the DB you could just run a dummy pod and execute some sql-client commands from there.

# Answer
> 0 votes
To access Kubernetes you typically use `port-forward`, like this (take from KubeApps)

```
$ kubectl port-forward -n kubeapps svc/kubeapps 8080:80
Forwarding from 127.0.0.1:8080 -> 8080
Forwarding from [::1]:8080 -> 8080

```

If you want to make it public so your port-forward listens to all ip addresses, you can use `--address 0.0.0.0` as documented in `kubectl port-forward --help`,

```
$ kubectl port-forward -n kubeapps svc/kubeapps 8080:80 --address 0.0.0.0
Forwarding from 0.0.0.0:8080 -> 8080

```

Now your host is redirecting traffic from 8080 to inside the Kubernetes cluster.

---
Tags: kubernetes, kubectl, vm
---


# Question
Title: How can I get an installation of k3s working with KubeApps?
How can I get an install of KubeApps running on K3s? What are the minimal required steps?

# Answer
# Installing k3s and kubeapps

1. Install k3s, (official instructions)

```
curl -sfL https://get.k3s.io | sh -

```
2. Add `KUBECONFIG` for user. (Devops.StackExchange)

```
export KUBECONFIG=~/.kube/config
mkdir ~/.kube 2> /dev/null
sudo k3s kubectl config view --raw > "$KUBECONFIG"
chmod 600 "$KUBECONFIG"

```
3. Install Helm, (offical instructions)

```
curl https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3 | bash

```
4. Setup Kubeapps, (official instructions)

1. Install Kubeapps

    ```
    helm repo add bitnami https://charts.bitnami.com/bitnami
    helm repo update
    helm install -n kubeapps --create-namespace kubeapps bitnami/kubeapps

    ```
    2. Configure Kubeapps

    ```
    kubectl create --namespace default serviceaccount kubeapps-operator

    kubectl create clusterrolebinding kubeapps-operator --clusterrole=cluster-admin --serviceaccount=default:kubeapps-operator

    cat <<EOF | kubectl apply -f -
    apiVersion: v1
    kind: Secret
    metadata:
      name: kubeapps-operator-token
      namespace: default
      annotations:
        kubernetes.io/service-account.name: kubeapps-operator
    type: kubernetes.io/service-account-token
    EOF

    kubectl get --namespace default secret kubeapps-operator-token -o jsonpath='{.data.token}' -o go-template='{{.data.token | base64decode}}' && echo

    ```
    3. Play (note we have a slight modification with `--address 0.0.0.0`)

    ```
    kubectl port-forward -n kubeapps --address 0.0.0.0 svc/kubeapps 8080:80

    ```

# Removal

```
sudo k3s-uninstall.sh

# Remove helm, and caches
sudo rm -rf /usr/local/bin/helm ~/.cache/helm /root/.cache/helm

# Remove logs
sudo rm -rf /var/log/pods/

# Your own cluster config/keys/kubeconfig
sudo rm -rf ~/.kube

```
> 0 votes

---
Tags: k3s, kubeapps
---


# Question
Title: How do I disable ICMP echo responses within a Docker container?
I want to be able to change the `/proc/sys/net/ipv4/icmp_echo_ignore_all` file from within my Docker container, but I get the

> bash: /proc/sys/net/ipv4/icmp\_echo\_ignore\_all: Read-only file system

error when attempting to do so, and running Docker with the `--privileged` flag is not an option.

Is there any way I can disable (and preferably also reenable) ICMP echo requests from within my container?

# Answer
Here are the steps to change to disable ICMP echo response within Docker container:

1. connect to the container with all privileges

```
    docker exec -it --privileged container_name bash

```
2. modify the `/proc/sys/net/ipv4/icmp_echo_ignore_all` file via the `systl` conmmand, by default value is `0`

See default value :

```
    sysctl net.ipv4.icmp_echo_ignore_all

```

Disable icmp echo response :

```
    sysctl net.ipv4.icmp_echo_ignore_all=1

```
> -2 votes

---
Tags: docker, networking
---


# Question
Title: What is the information returned by docker inspect for .Mounts?
Ie, if I do:

```
$ docker inspect -f '{{ .Mounts }}' base-hostSrcNv

```

I might get:

```
[{bind  /dir/in/container /dir/on/host   true }]

```

The meaning of `/dir/in/container` and `/dir/on/host` is pretty easy to understand. But what's the last boolean value represent?

# Answer
This is the **write** or **read** mode on the container. But you can enter this command for more detail in the return:

```
    docker inspect --format="{{json .Mounts}}" base-hostSrcNv

```

You can also use **jq** (***apt install jq***) tool to have beautiful json output:

```
    docker inspect --format="{{json .Mounts}}" base-hostSrcNv | jq

```
> 1 votes

---
Tags: docker
---


# Question
Title: Resizing a volume in terraform without enable_online_resize?
Is there a way to `terraform apply` and recreate the storage volume rather than setting `enable_online_resize` or running a `terraform destroy` first?

```
openstack_blockstorage_volume_v3.repo: Modifying... [id=87049fb6-1520-41e3-bcde-44f837dd5d1f]
╷
│ Error: Error extending openstack_blockstorage_volume_v3 87049fb6-1520-41e3-bcde-44f837dd5d1f,
│                                       volume is attached to the instance and
│                                       resizing online is disabled,
│                                       see enable_online_resize option
│
│   with openstack_blockstorage_volume_v3.repo,
│   on main.tf line 60, in resource "openstack_blockstorage_volume_v3" "repo":
│   60: resource "openstack_blockstorage_volume_v3" "repo" {

```

I realize I'll lose the data on the volume?

# Answer
You can use `--replace` like this,

```
terraform apply --replace=openstack_blockstorage_volume_v3.repo

```
> 0 votes

---
Tags: terraform
---


# Question
Title: How do you a initialize a new volume without a file system or partition table?
I'm using OpenStack and I am trying to provision a compute resource on it. I want that compute resource to attach to a volume. We have three things in play.

I need something to,

1. Create a partition table on first mount.
2. To create a file system (ext4) on first mount.
3. To initialize the file system with the things I need.

Does the logic to initialize this volume typically hang out in the cloud-init for the machine? If I initialize it manually, I can mount it using `cloud-init`'s directive for `mounts`. But without manually installation, how do we instruct `user_data` to initialize the volume before if necessary before the mount?

What I'm trying to do is store the `.git` file for a monorepo on an external shared mount. What I would like is on first boot of the sandbox for that to be ready. On subsequent boots/refreshes, I would just like them to run `git fetch` so they only pull down what's necessary rather than needing to re-clone.

# Answer
# Cloud-init solution

`cloud-init` accommodates the partitioning with `disk_setup`

```
disk_setup:
  /dev/vdb:
    table_type: gpt
    layout: true

```

And the corresponding filesystem initialization with `fs_setup`

```
fs_setup:
- label: repo
  filesystem: ext4
  device: /dev/vdb1
  partition: auto

```
> 0 votes

---
Tags: terraform, openstack, volume, cloud-init
---


# Question
Title: connect to kind cluster from inside and outside
Consider this:

There's a linux machine running on azure.

It has apache installed and browing to mymachine.com works, webpage is displayed. All ports are opened in the azure portal "networking settings".

Now there's a k8s kind cluster running on the machine too with a rabbitmq deployment as NodePort service port 32254.

```
kubectl get svc

```
* kubernetes ClusterIP 10.96.0.1 443/TCP 3h17m
* ng ClusterIP 10.96.104.61 80/TCP 68m
* rabbitmq NodePort 10.96.8.15 5672:32740/TCP,15672:32254/TCP,15692:31802/TCP 143m
* rabbitmq-nodes ClusterIP None 4369/TCP,25672/TCP 143m

All good, but I'm unable to access the rabbitmq cluster from outside.

The ip address of the control plane is 172.18.0.3, so this command works:

```
curl http://172.18.0.3:32254

```

But connecting from outside does not work:

```
curl http://mymachine.com:32254

```

So how do we forward from the public-ip of a VM in azure to an internal IP inside the VM?

# Answer
you add the server to balancer then create NAT rule to forward the packet to specific port on your VM.

manage-port-forwarding-for-backend-pool-with-azure-load-balancer
> 1 votes

---
Tags: kubernetes, azure, azure-kubernetes-service
---


# Question
Title: How to don't start entrypoint command on "docker-compose up"?
I have multiple Docker containers in a project and I use `docker-compose up -d` to start containers.

This is my `docker-compose.yml` file:

```
version: "3"
services:
  httpd:
    image: 'nginx:stable-alpine'
    ports:
      - '80:80'
    volumes:
      - ./laravel:/var/www/html
      - ./.docker-config/nginx/nginx.conf:/etc/nginx/conf.d/default.conf:ro
    depends_on:
      - php
      - mysql
    networks:
      - backstage

  php:
    build:
      context: ./.docker-config/dockerfiles
      dockerfile: php.dockerfile
    volumes:
      - ./laravel:/var/www/html:delegated
    networks:
      - backstage

  mysql:
    image: mysql:5.7
    env_file:
      - ./.docker-config/mysql/mysql.env
    ports:
      - '33060:3306'
    networks:
      - backstage

  composer:
    build:
      context: ./.docker-config/dockerfiles
      dockerfile: composer.dockerfile
    volumes:
      - ./laravel:/var/www/html
    networks:
      - backstage

  artisan:
    build:
      context: ./.docker-config/dockerfiles
      dockerfile: php.dockerfile
    volumes:
      - ./laravel:/var/www/html
    entrypoint: ["php", "/var/www/html/artisan"]
    depends_on:
      - mysql
    networks:
      - backstage

  npm:
    image: node:14-alpine
    working_dir: /var/www/html
    entrypoint: ["npm"]
    volumes:
      - ./laravel:/var/www/html
    networks:
      - backstage

  phpunit:
    build:
      context: ./.docker-config/dockerfiles
      dockerfile: php.dockerfile
    volumes:
      - ./laravel:/var/www/html
    entrypoint: ["vendor/bin/phpunit"]
    networks:
      - backstage

```

As you see, I defined `entrypoint` for the `phpunit` container, but I don't want to start the phpunit when I run `docker-compose up -d`.

How can I do that?

# Answer
You can scale service that you don't want to run to 0

`docker-compose up --scale phpunit=0 -d` And it will not start container for phpunit service as stated in docs

You can also check compose profiles for more options on excluding certain services in your docker-compose file.
> 2 votes

---
Tags: docker, docker-compose
---


# Question
Title: Azure Function App logs without Application Insights instance?
I'm a bit confused about my options for Function App logging on Azure. Is Application Insights the only mechanism available to observe the output logged through the function application's context? In other words, can logs be tailed or log files be temporarily observed anywhere without draining to an instance of Application Insights?

# Answer
> 1 votes
You can also configure Azure Functions with NLog:

https://github.com/NLog/NLog.Extensions.Logging/wiki/NLog-cloud-logging-with-Azure-function-or-AWS-lambda

Here you can find a sample that shows an Azure Function with NLog and GrayLog

Logging API Requests and Responses to GrayLog over Azure API Management using Azure Functions

nlog.config:

```
<?xml version="1.0" encoding="utf-8" ?>
<nlog xmlns="http://www.nlog-project.org/schemas/NLog.xsd"
      xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
      autoReload="true"
      throwExceptions="false"
      internalLogLevel="Off">
  <extensions>
    <add assembly="NLog.Web.AspNetCore"/>
    <add assembly="NLog.Web.AspNetCore.Targets.Gelf"/>
  </extensions>
  <targets>
    <target xsi:type="Gelf" name="graylog" endpoint="udp://localhost:12201" facility="APM.RequestLogging.Function" SendLastFormatParameter="true" />
  </targets>
  <rules>
    <logger level="Info" writeTo="graylog" />
  </rules>
</nlog>

```

---
Tags: azure, logging
---


title: Secrets in Docker without Swarm

text:
# Question
Title: Secrets in Docker without Swarm
I am using Docker, but not Swarm (or any other orchestration) for a project.

Is there a way to leverage Docker Secrets without Swarm?

# Answer
You can use docker secrets a bit differently using `docker-compose` without having to use swarm. See this for the official documentation.

# Example:

1. Create a simple compose file like so,

```
version: "3.7"

services:

  db:
    image: mariadb:10.5.2
    env_file:
      - ./db.env
    secrets:
      - rootpass
      - dbpass
      - mysqldb
      - mysqluser
    restart: always

```
2. Now add the following in the end

```
secrets:
  rootpass:
    file: /tmp/root_pass
  dbpass:
    file: /tmp/db_pass
  mysqldb:
    file: /tmp/mysql_db
  mysqluser:
    file: /tmp/mysql_user

```
3. Inside those files, keep your password, username, database name etc. in plain text. Then simply deploy the containers `docker-compose up -d`.

It's similar to how you define volumes and networks in a compose file.

Keep in mind that this isn't true secret implementation. Here's the github PR that added this feature, along with the main file if you're interested.
> 3 votes

# Answer
Regarding the documentation in https://hub.docker.com/\_/mariadb?tab=description, your solution can be this:

```
version: "3.9"

services:

  db:
    image: mariadb:10.7
    secrets:
      - rootpass
      - dbpass
      - mysqldb
      - mysqluser
    environment:
      MARIADB_ROOT_PASSWORD_FILE: /run/secrets/rootpass
      MARIADB_DATABASE_FILE: /run/secrets/mysqldb
      MARIADB_USER_FILE: /run/secrets/mysqluser
      MARIADB_PASSWORD_FILE: /run/secrets/dbpass
    restart: unless-stopped
    ports:
      - '3306:3306'

secrets:
  rootpass:
    file: ./cf/root_pass
  dbpass:
    file: ./cf/db_pass
  mysqldb:
    file: ./cf/mysql_db
  mysqluser:
    file: ./cf/mysql_user

```
> 0 votes

---
Tags: docker, secret-management
---


# Question
Title: KubeApps: Invalid GetAvailablePackageSummaries response from the plugin helm.packages: ... Unable to fetch chart categories
I followed the setup instructions for `k3s`, and `kubectl` in this answer. I'm getting this error when I boot into KubeApps

> An error occurred while fetching the catalog: Invalid GetAvailablePackageSummaries response from the plugin helm.packages: rpc error: code = Internal desc = Unable to fetch chart categories: pq: relation "charts" does not exist.

How can I resolve this error?

When I pull the logs, I see,

> I0608 21:01:12.935502 1 root.go:32\] asset-syncer has been configured with: `server.Config{DatabaseURL:"kubeapps-postgresql:5432", DatabaseName:"assets", DatabaseUser:"postgres", DatabasePassword:"E0mn56sa5d", Debug:false, Namespace:"kubeapps", OciRepositories:[]string{}, TlsInsecureSkipVerify:false, FilterRules:"", PassCredentials:false, UserAgent:"asset-syncer/2.4.5 (kubeapps/2.4.5)", UserAgentComment:"kubeapps/2.4.5", GlobalReposNamespace:"kubeapps", KubeappsNamespace:"", AuthorizationHeader:"", DockerConfigJson:""}`

Followed by,

```
Usage:
  asset-syncer sync [REPO NAME] [REPO URL] [REPO TYPE] [flags]

Flags:
  -h, --help                       help for sync
      --oci-repositories strings   List of OCI Repositories in case the type is OCI
      --version                    version for sync

Global Flags:
      --add_dir_header                   If true, adds the file directory to the header of the log messages
      --alsologtostderr                  log to standard error as well as files
      --database-name string             Name of the database to use (default "charts")
      --database-url string              Database URL (default "localhost:5432")
      --database-user string             Database user
      --debug                            verbose logging
      --filter-rules string              JSON blob with the rules to filter assets
      --global-repos-namespace string    Namespace for global repos (default "kubeapps")
      --log_backtrace_at traceLocation   when logging hits line file:N, emit a stack trace (default :0)
      --log_dir string                   If non-empty, write log files in this directory
      --log_file string                  If non-empty, use this log file
      --log_file_max_size uint           Defines the maximum size a log file can grow to. Unit is megabytes. If the value is 0, the maximum file size is unlimited. (default 1800)
      --logtostderr                      log to standard error instead of files (default true)
      --namespace string                 Namespace of the repository being synced
      --one_output                       If true, only write logs to their native severity level (vs also writing to each lower severity level)
      --pass-credentials                 pass credentials to all domains
      --skip_headers                     If true, avoid header prefixes in the log messages
      --skip_log_headers                 If true, avoid headers when opening log files
      --stderrthreshold severity         logs at or above this threshold go to stderr (default 2)
      --tls-insecure-skip-verify         Skip TLS verification
      --user-agent-comment string        UserAgent comment used during outbound requests
  -v, --v Level                          number for the log level verbosity (default 3)
      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging

```

As well as this errors,

> Error: Error: Get `https://charts.bitnami.com/bitnami/index.yaml`: dial tcp: lookup charts.bitnami.com on 10.43.0.10:53: server misbehaving

---

I filed a bug upstream, but I'm not sure if this is a bug or a misconfiguration:

# Answer
> 0 votes
# Upstream Answer -- Networking Problem.

Though I'm getting this on a fresh install I was guided to this which indicates a bigger problem,

```
kubectl -n kubeapps exec deployment/kubeapps -- curl -sI https://charts.bitnami.com/bitnami/index.yaml

```

this returns "command terminated with exit code 6".

I can further identify the problem with `nslookup`, without using Kube Apps by following the \[k3s tutorial on troubleshooting dns\]https://rancher.com/docs/rancher/v2.5/en/troubleshooting/dns/)

```
kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup www.google.com

```

## Follow up

Now that I know what this problem is (with external DNS resolution), I've asked this question for more information Newly installed k3s cluster on fresh OS install can not resolve external domains, when the host can?

---
Tags: k3s, kubeapps
---


# Question
Title: How can I set let's encrypt certificates with Ansible?
I'm trying to get a let's encrypt certificate for my domain with Ansible. I have been reading this tutorial which is a bit outdated and the Ansible documentation.

My playbook is a mix of what I have found in the tutorial mentioned and the documentation.

```
---

- name: "Create required directories in /etc/letsencrypt"
  file:
    path: "/etc/letsencrypt/{{ item }}"
    state: directory
    owner: root
    group: root
    mode: u=rwx,g=x,o=x
  with_items:
    - account
    - certs
    - csrs
    - keys

- name: Generate let's encrypt account key
  openssl_privatekey:
    path: "/etc/letsencrypt/account/account.key"

- name: Generate let's encrypt private key with the default values (4096 bits, RSA)
  openssl_privatekey:
    path: "/etc/letsencrypt/keys/domain.me.key"

- name: Generate an OpenSSL Certificate Signing Request
  community.crypto.openssl_csr:
    path: "/etc/letsencrypt/csrs/domain.me.csr"
    privatekey_path: "/etc/letsencrypt/keys/domain.me.key"
    common_name: www.domain.me

# Create challenge
- name: Create a challenge for domain.me using an account key file.
  acme_certificate:
    acme_directory: "https://acme-v02.api.letsencrypt.org/directory"
    acme_version: 2
    account_key_src: "/etc/letsencrypt/account/account.key"
    account_email: "email@mail.com"
    terms_agreed: yes
    challenge: "http-01"
    src: "/etc/letsencrypt/csrs/domain.me.csr"
    dest: "/etc/letsencrypt/certs/domain.me.crt"
    fullchain_dest: "/etc/letsencrypt/certs/domain.me-fullchain.crt"
  register: acme_challenge_domain_me

- name: "Create .well-known/acme-challenge directory"
  file:
    path: "project/dir/path/.well-known/acme-challenge"
    state: directory
    owner: root
    group: root
    mode: u=rwx,g=rx,o=rx

- name: "Implement http-01 challenge files"
  copy:
    content: "{{ acme_challenge_domain_me['challenge_data'][item]['http-01']['resource_value'] }}"
    dest: "project/dir/path/{{ acme_challenge_domain_me['challenge_data'][item]['http-01']['resource'] }}"
  with_items:
    - "domain.me"
    - "www.domain.me"
  when: acme_challenge_domain_me is changed and domain_name|string in acme_challenge_domain_me['challenge_data']

- name: Let the challenge be validated and retrieve the cert and intermediate certificate
  acme_certificate:
    acme_directory: "https://acme-v02.api.letsencrypt.org/directory"
    acme_version: 2
    account_key_src: "/etc/letsencrypt/account/account.key"
    account_email: "email@mail.com"
    challenge: "http-01"
    src: "/etc/letsencrypt/csrs/domain.me.csr"
    cert: "/etc/letsencrypt/certs/domain.me.crt"
    fullchain: "/etc/letsencrypt/certs/domain.me-fullchain.crt"
    chain: "{/etc/letsencrypt/certs/domain.me-intermediate.crt"
    remaining_days: "60"
    data: "{{ acme_challenge_domain_me }}"
  when: acme_challenge_domain_me is changed

```

When I run the playbook, I'm getting this error:

`fatal: [web_server]: FAILED! =>`

```
{
  "changed": false,
  "msg": "Failed to validate challenge for dns:www.domain.me: Status is \"invalid\". Challenge http-01: Error urn:ietf:params:acme:error:connection: \"xxx.xxx.x.ip: Fetching http://www.domain.me/.well-known/acme-challenge/NRkTQSpAVbWtjFNq206YES55lEoHHinHUn9cjR7vm7k: Connection refused\".",
  "other": {
    "authorization": {
      "challenges": [
        {
          "error": {
            "detail": "xxx.xxx.x.ip: Fetching http://www.domain.me/.well-known/acme-challenge/NRkTQSpAVbWtjFNq206YES55lEoHHinHUn9cjR7vm7k: Connection refused",
            "status": 400,
            "type": "urn:ietf:params:acme:error:connection"
          },
          "status": "invalid",
          "token": "NRkTQSpAVbWtjFNq206YES55lEoHHinHUn9cjR7vm7k",
          "type": "http-01",
          "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/103702154687/UdA36w",
          "validated": "2022-04-30T16:01:32Z",
          "validationRecord": [
            {
              "addressUsed": "xxx.xxx.x.ip",
              "addressesResolved": ["xxx.xxx.x.ip"],
              "hostname": "www.domain.me",
              "port": "80",
              "url": "http://www.domain.me/.well-known/acme-challenge/NRkTQSpAVbWtjFNq206YES55lEoHHinHUn9cjR7vm7k"
            }
          ]
        }
      ],
      "expires": "2022-05-07T15:57:28Z",
      "identifier": {
        "type": "dns",
        "value": "www.domain.me"
      },
      "status": "invalid",
      "uri": "https://acme-v02.api.letsencrypt.org/acme/authz-v3/103702154687"},
      "identifier": "dns:www.domain.me"
    }
  }

```

Command UFW status gives:

```
To                         Action      From
--                         ------      ----
OpenSSH                    ALLOW       Anywhere
80                         ALLOW       Anywhere
5432/tcp                   ALLOW       Anywhere
443/tcp                    ALLOW       Anywhere
OpenSSH (v6)               ALLOW       Anywhere (v6)
80 (v6)                    ALLOW       Anywhere (v6)
5432/tcp (v6)              ALLOW       Anywhere (v6)
443/tcp (v6)               ALLOW       Anywhere (v6)

```

The nginx configuration is :

```
upstream project {
    server unix:///tmp/project.sock;
}

server {
    listen 443 ssl;
    server_name www.domain.me;
    ssl_certificate /etc/letsencrypt/certs/domain.me.crt;
    ssl_certificate_key /etc/letsencrypt/keys/domain.me.key;

    listen 80;
    server_name domain.me www.domain.me;
    charset utf-8;
    client_max_body_size 4M;
    return 302 https://$server_name$request_uri;

    # Serving static files directly from Nginx without passing through uwsgi
    location /app/static/ {
        alias /home/admin/project/app/static/;
    }

    location / {
        # kill cache
        add_header Last-Modified $date_gmt;
        add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
        if_modified_since off;
        expires off;
        etag off;
        uwsgi_pass project;
        include /home/admin/project/uwsgi_params;
    }

    #location /404 {
    #   uwsgi_pass project;
    #   include /home/admin/project/uwsgi_params;
    #}
}

```

Could you help me understand where the problem is coming from and how to solve it?

I'm not sure if my mistakes are coming from the playbook, Nginx settings, or somewhere else, so apologize if the question isn't perfectly targeted. It's my first time doing this, so please include details and explanations to help me understand.

Thank you.

# Answer
There were a couple mistakes in my tasks, but this one works:

```
---

# Create the directories required by letsencrypt to store keys and certificates.
- name: "Create required directories in /etc/letsencrypt"
  become: yes
  file:
    path: "/etc/letsencrypt/{{ item }}"
    state: directory
    owner: root
    group: root
    mode: u=rwx,g=x,o=x
  with_items:
    - account
    - certs
    - csrs
    - keys

# https://docs.ansible.com/ansible/2.9/modules/acme_certificate_module.html#acme-certificate-module
- name: Generate let's encrypt account key
  become: yes
  openssl_privatekey:
    path: "{{ letsencrypt_account_key }}"

# https://docs.ansible.com/ansible/latest/collections/community/crypto/openssl_privatekey_module.html#openssl-privatekey-module
- name: Generate let's encrypt private key with the default values (4096 bits, RSA)
  become: yes
  openssl_privatekey:
    path: "{{letsencrypt_keys_dir}}/{{ domain_name }}.key"

# https://docs.ansible.com/ansible/latest/collections/community/crypto/openssl_csr_module.html#openssl-csr-module
- name: Generate an OpenSSL Certificate Signing Request
  become: yes
  community.crypto.openssl_csr:
    path: "{{letsencrypt_csrs_dir}}/{{ domain_name }}.csr"
    privatekey_path: "{{letsencrypt_keys_dir}}/{{ domain_name }}.key"
    common_name: "{{domain_name}}"

# Create letsencrypt challenge.
- name: Create a challenge for {{domain_name}} using a account key file.
  become: yes
  community.crypto.acme_certificate:
    acme_directory: "{{acme_directory}}"
    acme_version: "{{acme_version}}"
    account_email: "{{acme_email}}"
    terms_agreed: yes
    account_key_src: "{{letsencrypt_account_key}}"
    csr: "{{letsencrypt_csrs_dir}}/{{domain_name}}.csr"
    dest: "{{letsencrypt_certs_dir}}/{{domain_name}}.crt"
    remaining_days: "{{remaining_days}}"
  register: acme_challenge

# Create the directory to hold the validation token.
- name: "Create .well-known/acme-challenge directory"
  become: yes
  file:
    path: "{{project_path}}/.well-known/acme-challenge"
    state: directory
    owner: root
    group: root
    mode: u=rwx,g=rx,o=rx

# Copy the necessary files for the http-01 challenge.
- name: "Implement http-01 challenge files"
  become: yes
  copy:
    dest: "{{project_path}}/{{ acme_challenge['challenge_data'][item]['http-01']['resource'] }}"
    content: "{{ acme_challenge['challenge_data'][item]['http-01']['resource_value'] }}"
  with_items:
    - "{{ domain_name }}"
  when: acme_challenge is changed and domain_name|string in acme_challenge['challenge_data']

# Execute letsencrypt challenge.
- name: Let the challenge be validated and retrieve the cert and intermediate certificate
  become: yes
  community.crypto.acme_certificate:
    account_key_src: "{{letsencrypt_account_key}}"
    csr: "{{letsencrypt_csrs_dir}}/{{domain_name}}.csr"
    cert: "{{letsencrypt_certs_dir}}/{{domain_name}}.crt"
    acme_directory: "{{acme_directory}}"
    acme_version: "{{acme_version}}"
    account_email: "{{acme_email}}"
    challenge: "{{acme_challenge_type}}"
    fullchain: "{{letsencrypt_certs_dir}}/{{domain_name}}-fullchain.crt"
    chain: "{{letsencrypt_certs_dir}}/{{domain_name}}-intermediate.crt"
    remaining_days: "{{remaining_days}}"
    data: "{{ acme_challenge }}"
  when: acme_challenge is changed

```

The corresponding Nginx config file is:

```
upstream {{project_name}} {
    server unix:{{socket_location}};
}

server {
    listen 80;
    location /.well-known/acme-challenge/ {
        root {{project_path}};
    }
    server_name {{domain_name}} www.{{domain_name}} {{web_server_ip}};
    return 301 https://{{domain_name}}$request_uri;
}

server {

    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/certs/{{domain_name}}-fullchain.crt;
    ssl_certificate_key /etc/letsencrypt/keys/{{domain_name}}.key;

    charset utf-8;
    client_max_body_size 4M;

    # Serving static files directly from Nginx without passing through uwsgi
    location /app/static/ {
        alias {{project_path}}/app/static/;
    }

    location /.well-known/acme-challenge/ {
        root {{project_path}};
    }

    location / {
        # kill cache
        add_header Last-Modified $date_gmt;
        add_header Cache-Control 'no-store, no-cache, must-revalidate, proxy-revalidate, max-age=0';
        if_modified_since off;
        expires off;
        etag off;
        uwsgi_pass {{project_name}};
        include {{project_path}}/uwsgi_params;
    }
}

```
> 1 votes

---
Tags: ansible, nginx, letsencrypt
---


# Question
Title: Ansible: How to run ad-hoc command with multiple environnements?
Given the following architecture:

```
├── ansible.cfg
├── hosts
│   ├── production
│   └── staging
├── production
│   ├── group_vars
│   │   ├── all.yml
│   │   ├── mygroup.yml
│   │   └── mygroup2.yml
│   ├── host_vars
│   │   ├── mhost1.yml
│   │   └── mhost2.yml
│   └── production_playbook.yml
└── staging
    ├── group_vars
    │   ├── all.yml
    │   ├── mygroup.yml
    │   └── mygroup2.yml
    ├── host_vars
    │   ├── mhost1.yml
    │   └── mhost2.yml
    └── staging_playbook.yml

```

The content of ansible.cfg is:

```
[defaults]

inventory=hosts

```

The content of the hosts/production and hosts/staging file is the same:

```
[all]
[mygroup]
mhost1

[mygroup2]
mhost2

```

staging/group\_vars/all.yml, mygroup.yml, mygroup2.yml contains all:

```
ansible_user: root

```

staging/host\_vars/mhost1.yml and mhost2.yml contains both (with their respective ip):

```
ansible_host: xxx.xxx.xxx.xx

```

staging\_playbook.yml contains:

```
---
- hosts: all
  tasks:
    - name: ping all in staging
      ping:

- hosts: mhost1
  tasks:
    - name: ping mhost1 in staging
      ping:

- hosts: mhost2
  tasks:
    - name: ping mhost2 in staging
      ping:

```

In the production environment, the production\_playbook.yml is similar:

```
---
- hosts: all
  tasks:
    - name: ping all in production
      ping:

- hosts: mhost1
  tasks:
    - name: ping mhost1 in production
      ping:

- hosts: mhost2
  tasks:
    - name: ping mhost2 in production
      ping:

```

The only differences are in production/host\_vars where I have different IP addresses.

If I run:

ansible-playbook staging/staging\_playbook.yml

or

ansible-playbook production/production\_playbook.yml

it all works fine so I guess the architecture is correct.

Now, my question is: how can I target a specific host in a specific environment with an Ansible ad-hoc command?

for example:

```
ansible mhost1 -i hosts/staging -m ping

```

which is not working and gives the output:

```
mhost1 | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: ssh: Could not resolve hostname mhost1: nodename nor servname provided, or not known",
    "unreachable": true
}

```

EDIT:

I found out that if I move my inventory to their respective environments like:

```
├── ansible.cfg
├── production
│   ├── group_vars
│   │   ├── all.yml
│   │   ├── mygroup.yml
│   │   └── mygroup2.yml
│   ├── host_vars
│   │   ├── mhost1.yml
│   │   └── mhost2.yml
│   ├── hosts
│   └── production_playbook.yml
└── staging
    ├── group_vars
    │   ├── all.yml
    │   ├── mygroup.yml
    │   └── mygroup2.yml
    ├── host_vars
    │   ├── mhost1.yml
    │   └── mhost2.yml
    ├── hosts
    └── staging_playbook.yml

```

and remove from ansible.cfg:

```
inventory=hosts

```

I can execute my ad-hocs command however to run the playbook I have to specify the inventory like:

ansible-playbook staging/staging\_playbook.yml -i staging/hosts

The architecture I found allowing me to execute my playbooks per environment without having to specify an inventory while executing the command and allowing me to execute ad-hoc commands on a specific host in a specific environnement is this one:

```
├── ansible.cfg
├── hosts
│   ├── production
│   └── staging
├── production
│   ├── group_vars
│   │   ├── all.yml
│   │   ├── mygroup.yml
│   │   └── mygroup2.yml
│   ├── host_vars
│   │   ├── mhost1.yml
│   │   └── mhost2.yml
│   ├── hosts
│   └── production_playbook.yml
└── staging
    ├── group_vars
    │   ├── all.yml
    │   ├── mygroup.yml
    │   └── mygroup2.yml
    ├── host_vars
    │   ├── mhost1.yml
    │   └── mhost2.yml
    ├── hosts
    └── staging_playbook.yml

```

This seems weird since I have an inventory per environment and an inventory outside containing the same code. What is the proper way to achieve the same thing?

# Answer
One of the options would be to change the current directory. This simplifies the structure and lets you keep common configuration and inventory for production and staging. For example, to run the commands,

* either change the working directory for all commands

```
shell> cd staging
shell> ansible-playbook playbook.yml
shell> ansible mhost1 -m ping

```
* , or change the working directory for each command

```
shell> (cd staging; ansible-playbook playbook.yml)
shell> (cd staging; ansible mhost1 -m ping)

```

Make the changes below

* Use inventory file from the current directory. Put into the ansible.cfg

```
inventory=$PWD/hosts

```
* Link *ansible.cfg* to both *production/ansible.cfg* and *staging/ansible.cfg*
* Link *hosts* to both *production/hosts* and *staging/hosts*
* You can rename both *production\_playbook.yml* and *staging\_playbook.yml* to *playbook.yml*

```
├── ansible.cfg
├── hosts
├── production
│   ├── group_vars
│   │   ├── all.yml
│   │   ├── mygroup.yml
│   │   └── mygroup2.yml
│   ├── host_vars
│   │   ├── mhost1.yml
│   │   └── mhost2.yml
|   ├── ansible.cfg -> ../ansible.cfg
│   ├── hosts -> ../hosts
│   └── playbook.yml
└── staging
    ├── group_vars
    │   ├── all.yml
    │   ├── mygroup.yml
    │   └── mygroup2.yml
    ├── host_vars
    │   ├── mhost1.yml
    │   └── mhost2.yml
    ├── ansible.cfg -> ../ansible.cfg
    ├── hosts -> ../hosts
    └── playbook.yml

```
<sup>

**WARNING**

In a critical environment separate *production* and *staging* physically.

</sup>
> 2 votes

---
Tags: ansible, ansible-inventory
---


# Question
Title: Does /etc/rancher/k3s/registries.yaml affect `k3s ctr` and `k3s crictl`?
`k3s ctr` launches the contianerd CLI, and `k3s crictl` is the CRI cli. I'm told you configure k3s authentication to the image registries using `/etc/rancher/k3s/registries.yaml`. Does this yaml file configure containerd, and CRI such that these commands no longer require `--creds`?

```
k3s crictl pull     --creds "evancarroll:$TOKEN" docker.io/alpine:3
k3s ctr images pull --creds "evancarroll:$TOKEN" docker.io/library/alpine:3

```

And can just be

```
k3s crictl pull     docker.io/alpine:3
k3s ctr images pull docker.io/library/alpine:3

```

# Answer
This is the answer I got from Rancher Developer Brandon D in their slack

> `ctr` and `crictl` both interact with containerd, via different apis. ctr uses the containerd native api, and crictl uses the CRI api. The `registries.yaml` config is used by containerd itself, and will be honored no matter what it is that does the pull or how.
> 0 votes

---
Tags: configuration, authentication, k3s, containerd
---


# Question
Title: How do I get k3s to authenticate with Docker Hub?
In `jornalctl -u k3s`, I see a bunch of errors like this

> 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

When I check the associate pods, I see

> Error from server (BadRequest): container "foo" in pod "bar" is waiting to start: trying and failing to pull image

How can I set up k3s to use an authenticate login to DockerHub? I don't see a `docker login`, and I've tried probably 20 different things none of which worked. I can confirm this username and password works with `--creds` to either `crictl` or `ctr`. But it doesn't work when I try to put it in a config file. I've tried to playing with

* `/etc/rancher/k3s/registries.yaml` and restarting.
* that seems to generate a new `/var/lib/rancher/k3s/agent/etc/containerd/config.toml`, I've tried playing with that file too directly

None of this seems to make any difference. I've tried testing with `k3s kubectl run` and also with `k3s crt` and `k3s crictl` without providing `--creds`, and none of these change the error that's returned.

My token has full access at https://hub.docker.com/settings/security

# Answer
> 2 votes
1. Update your `/etc/rancher/k3s/registries.yaml` to
```
configs:
  registry-1.docker.io:
    auth:
      username: evancarroll
      password: TOKENHIDDEN

```
2. Restart k3s, `sudo systemctl force-reload k3s`.

You can confirm the changes were accepted by checking that your key exists in `/var/lib/rancher/k3s/agent/etc/containerd/config.toml`.

---
Tags: dockerhub, k3s
---


# Question
Title: How can you validate that registries.yaml is set properly?
I've set up my `/etc/rancher/k3s/registries.yaml`, how can I confirm it works without doing

```
k3s kubectl run c1 --image docker.io/perl:5.30
k3s kubectl logs --follow=true c1
sleep 1
k3s kubectl delete pod c1

```

Is there a more efficient way to test registry authentication for images than having to use k3s run?

# Answer
> 0 votes
The `/etc/rancher/k3s/registries.yaml` is read by `k3s ctr` and `k3s crictl`. You can validate it works properly with a command like this,

```
sudo k3s crictl pull     docker.io/alpine:3
sudo k3s ctr images pull docker.io/library/alpine:3

```

---
Tags: authentication, k3s
---


# Question
Title: CodeDeploy: deployment failed because a specified file already exists at this location
I build an AMI from a 'prototype' EC2 instance I created to confirm everything is working. I create a Launch Configuration and Auto Scaling Group based on the AMI. I associate the Auto Scaling Group with CodeDeploy as a Blue/Green Deployment behind an Application Load Balancer.

When I run the CodePipeline, all goes well until it gets to the CodeDeploy phase and it tries to install the application in the specified directory in appspec.yml:

The deployment failed because a specified file already exists at this location: /var/www/html/mysite/test.rb

My expectation was that CodeDeploy will only update the files in the deploy directory. Instead, it breaks and says the files already exist. From research online, I discovered that when you deploy, CodeDeploy will look for a file (on the instance) with information on previously deployed files for the application/deployment-group in question. If it finds the file it then it uses it to cleanup (delete) the existing files, preparing for copying of the new revision files.

But if the cleanup file doesn't exist, CodeDeploy won't continue running. Therefore the files will not be cleaned up, and the deployment will fail since CodeDeploy does not overwrite files. it wont be run.

Therefore the files will not be cleaned up, and the deployment will fail since CodeDeploy does not overwrite files.

It seems as though the only option in these scenarios is to have a hook (eg BeforeInstall) that calls a script to delete the applications files (if they exist) on the instance. Is this really true? The only option I have is to delete the files in a beforeInstall hook?

I notice if you copy a deployment in the Console, there is an option to 'overwrite the content'. But this option is not available in the Deployment Group configuration. It only exists when you copy the deployment.

# Answer
> 1 votes
You probably can just add a flag `--file-exists-behavior OVERWRITE`  to your `aws deploy create-deployment` command.

Ref: https://docs.aws.amazon.com/cli/latest/reference/deploy/create-deployment.html

# Answer
> 0 votes
Add "file\_exists\_behavior: OVERWRITE" in appspec.yml.

This is my sample appspec.yml

```
version: 0.0
os: linux
files:
  - source: /
    destination: /var/www/html/
    overwrite: true
file_exists_behavior: OVERWRITE

```

---
Tags: amazon-web-services, aws-codedeploy
---