# Question
Title: Jenkins - docker login doesn't seem to persist: docker pull won't work but docker-compose can pull without problems
I am setting up a new Jenkins as our old one is based on a deprecated base image.

Dockerfile is:

```
FROM jenkins/jenkins:lts-centos7

USER root

RUN yum -y install docker \
    && yum clean all

USER jenkins

```

When I `docker exec ...` into the container and log into our Sonatype Nexus to pull Docker images from it, the successful login does not seem to be heeded by Jenkins afterwards:

```
[root@909fb3a3d52c .docker]# docker login https://our.nexus.internal:<endpoint>
Username (jenkins): jenkins
Password:
Login Succeeded
[root@909fb3a3d52c .docker]# pwd
/root/.docker
[root@909fb3a3d52c .docker]# cat config.json
{
    "auths": {
        "https://our.nexus.internal:<endpoint>": {
            "auth": "<CORRECT base64 hash>"
        }
    }

```

Login has succeeded, the auth credentials are written to /root/.docker/config.json and when I base64-decode them they are correct "jenkins:".

However:

```
[root@909fb3a3d52c .docker]# docker pull https://our.nexus.internal:<endpoint>/myImage:myTag
Error response from daemon: Get https://our.nexus.internal:<endpoint>/v2/myImage/manifests/myTag: no basic auth credentials

```

This will also happen during execution of a Jenkins pipeline (obviously), where job log reports:

```
docker pull https://our.nexus.internal:<endpoint>/myImage:myTag

Error response from daemon: Get Error response from daemon: Get https://our.nexus.internal:<endpoint>/v2/myImage/manifests/myTag: no basic auth credentials

script returned exit code 1

```

I assume this to not be a question about nexus but about how the credentials from docker login are stored and used.

Sidenote: We have a different Jenkins currently operational which is based on jenkinsci/blueocean:1.22.0 which does NOT display this behaviour. It can docker login fine and docker pull fine.

**edit1:**

Running `docker-compose` does work while a manual `docker pull...` will not:

```
[root@3eeee032ff08 /]# docker login https://nexus:port
Username: jenkins
Password:
Login Succeeded

[root@3eeee032ff08 /]# docker pull nexus:port/company/image:myTag
Error response from daemon: Get https://nexus:port/v2/company/image/manifests/myTag: no basic auth credentials

[root@3eeee032ff08 /]# cat docker-compose.yml
version: '3.3'

services:
jenkins:
    user: root
    image: nexus:port/company/image:myTag

[root@3eeee032ff08 /]# docker-compose up

Creating network "default_default" with the default driver
Pulling jenkins (nexus:port/company/image:myTag)...
myTag: Pulling from company/image
f34b00c7da20: Pull complete
3f316190de62: Pull complete
480967924aca: Pull complete
619ac94672e0: Pull complete
cb1c6713a236: Pull complete
55766a17b1c6: Pull complete
8f6e881b2ef2: Pull complete
f118a721eee8: Pull complete
fae3c8e02b83: Pull complete
093ceb59408d: Pull complete
e3bd3cbf474d: Pull complete
41a6f4e5de6d: Pull complete
aa1aece4c754: Pull complete
fc897d8ca22f: Pull complete
Digest: sha256:3fd74169f648784fe960add9c0138e3802a91e82c9b5fdbd734faaeaaef696f9
Status: Downloaded newer image for nexus:port/company/image:myTag

```

**edit2:**

When not basing off of `jenkins/jenkins:lts-centos7` but off of `jenkins/jenkins:latest`, which is a Debian base, then installing Docker on it using the official docs, login and pulling works:

```
worp@local$ docker run --rm -ti -u root -v /var/run/docker.sock:/var/run/docker.sock jenkins/jenkins:debian-docker bash

root@8aec8c792751:/# cat /etc/*-release
PRETTY_NAME="Debian GNU/Linux 9 (stretch)"
NAME="Debian GNU/Linux"
VERSION_ID="9"
VERSION="9 (stretch)"
VERSION_CODENAME=stretch
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@8aec8c792751:/# docker login https://nexus:port
Username: admin
Password:
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

root@8aec8c792751:/# cat /root/.docker/config.json
{
    "auths": {
        "nexus:port": {
            "auth": "<base64hash>"
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/19.03.12 (linux)"
    }
}

root@8aec8c792751:/# docker pull nexus:port/company/image:myTag
myTag: Pulling from company/image
Digest: sha256:3fd74169f648784fe960add9c0138e3802a91e82c9b5fdbd734faaeaaef696f9
Status: Image is up to date for nexus:port/company/image:myTag
nexus:port/company/image:myTag

```

This is getting weirder by the minute. I hope I am overlooking something extremely obvious so I feel very stupid afterwards.

# Answer
> 1 votes
I was able to produce a correct behaviour starting from `centos:centos7` base image and installing Docker only (no Jenkins yet) via the steps provided in the official Docker docu: https://docs.docker.com/engine/install/centos/

Seen here:

```
[root@d8d441ae6a7a /]# history
    2  yum remove docker                   docker-client                   docker-client-latest                   docker-common                   docker-latest                   docker-latest-logrotate                   docker-logrotate                   docker-engine
    3  yum install -y yum-utils
    4  yum-config-manager     --add-repo     https://download.docker.com/linux/centos/docker-ce.repo
    5  yum install docker-ce docker-ce-cli containerd.io
    6  systemctl start docker
    7  docker ps
    8  docker run hello-world
    9  docker login https://nexus:port
10  docker login https://nexus:port
11  docker pull nexus:port/company/image:tag

# docker pull nexus:port/company/image:tag
...
Status: Downloaded newer image for nexus:port/company/image:tag

```

While I still have the faulty behaviour when I install docker via `yum install -y docker`. So it's pretty clear to assume that I'm not facing an issue with the Jenkins Docker images specifically but rather face centos problems with their Docker package.

I guess there is no need here for an answer. I will go on to talk to the centos people.

Cheers and thanks for your consideration everyone!

# Answer
> 1 votes
Based on your output, all of your docker commands are running as root when in general they should be running as the jenkins user. You can try switching the user to jenkins and running the same docker login and pull commands.

I've used the following in my Jenkins pipeline scripts to log into a Nexus 3 Docker repository (https://docs.cloudbees.com/docs/admin-resources/latest/plugins/docker-workflow):

```
docker.withRegistry('https://docker.mycorp.com/', 'docker-login') {
  git '…'
  docker.build('myapp').push('latest')
}

```

# Answer
> 0 votes
The same problem occured for me after I have updated docker to the version 23.0.4 on a jenkins node.

In my case the problem was that the jenkins node wasn't able to store passes properly using the old way without credentials store, which is just storing passes as a base64 encoded string in `~/.docker/config.yaml`. Even though it was working fine before updating docker.

The only solution that worked for me was configuring credentials store on that Jenkins node and performing some additional workarounds.

I have configured credentials store using following instruction: https://leimao.github.io/blog/Docker-Login-Encrypted-Credentials/

And the additional workarounds were:

1. Exporting GPG\_TTY env: `export GPG_TTY=$(tty)`. This has to be exported every time the jenkins user login to docker so I have added this line to `.bashrc` file of my jenkins user. Without this env the Ubuntu Keyring wasn't detecting that my jenkins user is trying to access the `gpg` key and login to docker was failing.
2. Setting empty password for `gpg` key created when configuring Credentials Store. I had to make it empty because my Jenkins user wasn't able to pass a password for Ubuntu keyring as the keyring was using a popup to ask for the password.

---
Tags: docker, jenkins, docker-compose
---


title: How do I run Terraform with AWS MFA?

text:
# Question
Title: How do I run Terraform with AWS MFA?
I'm using Terraform with the terraform-provider-aws provider to manage my AWS infrastructure. I'm trying to enable `mfa_delete` on an S3 bucket, but when I try to apply the change I get this error:

```
1 error occurred:
        * aws_s3_bucket.logs: 1 error occurred:
        * aws_s3_bucket.logs: Error putting S3 versioning: AccessDenied: Mfa Authentication must be used for this request
        status code: 403, request id: <redacted>, host id: <redacted>

```

How can I run Terraform in a mode that uses multi-factor authentication? My IAM account has a Virtual MFA device attached, but that MFA isn't used when I'm terraforming because terraform uses the API Access Key associated with my account.

# Answer
> 4 votes
The solution is to specify an `assume_role` statement:

```
provider "aws" {
  profile = "default"
  assume_role {
    role_arn = "arn:aws:iam::[ACCOUNT_ID]:role/terraform-test-role"
  }
}

```

# Answer
> 1 votes
I was looking into a similar use case and found this github issue in the terraform public repo: https://github.com/terraform-providers/terraform-provider-aws/issues/629

As quoted in that issue: "Indeed it does seem like the Terraform AWS provider would need to do something special to make this work. As I noted in my other comment, we don't consider Terraform to be an interactive tool in general and so introducing extra interactive input in the apply phase is not ideal."

https://github.com/terraform-providers/terraform-provider-aws/issues/629#issuecomment-408937315

# Answer
> 1 votes
The error message here is telling you that you are not allowed to enable MFADelete without having an authenticated token with mfa. Meaning the executing principal must have logged in with MFA.

essentially the API call on the cli would be something like this

```
aws --profile my_profile s3api put-bucket-versioning --bucket bucket-name --versioning configuration 'MFADelete=Enabled,Status=Enabled' --mfa 'arn:aws:iam::<account-id>:mfa/root-account-mfa-device <mfacode>

```

Note that only the root account MFA can do this (the bucket owner) not a standard user. As detailed here

A simple terraform config assuming its executing under the credentials the previous profile specified would be:

```
resource "aws_s3_bucket" "bucket" {
  bucket = "some_test_bucket"
  acl    = "private"

  versioning {
    enabled = true
    mfa_delete = true
  }
}

```

Now so far so good, however, it should be noted that `mfa_delete = true` cannot toggle that setting, it is there to reflect the enabling of that in the state file only as mentioned in the docs

The correct way to set the mfa\_delete to enabled is via the API yourself. As mentioned above as the provider cannot change it but has the state there to ensure its not going to change the state after it is set on a subsequent run (out of band terraform changes to enable with the API).

This answer assumes AWS provider v2.65.0 or higher

# Answer
> 0 votes
Assuming that

* your IAM user can generate an mfa session token via `GetSessionToken`, and
* you have a way of getting this token and applying it in your **~/.aws/credentials** file (this can be done using helper scripts, like https://github.com/broamski/aws-mfa)

you don't need to specify anything special in the terraform configuration. You just need to make sure your aws cli profile is configured correctly as described above AND there is a valid token when you run terraform.

# Answer
> 0 votes
I used the AWS cli to do `sts assume-role` and then used `jq` to decompose the response and stored the values in environment variables, which `terraform` then uses. https://registry.terraform.io/providers/hashicorp/aws/latest/docs#environment-variables

My setup:

1. A user
2. A profile in `~/.aws/credentials` which has my access/secret
3. A role which can only use used by the user and requires MFA

```
    session_token=$(aws sts assume-role \
        --profile ${profile} \
        --role-arn "${role_arn}" \
        --role-session-name "${session_name}" \
        --serial-number "${mfa_serial}" \
        --duration-seconds $session_duration \
        --token-code "${token_code}" \
        --query Credentials)

    export AWS_REGION=${region}
    export AWS_ACCESS_KEY_ID="$(echo $session_token | jq -r .AccessKeyId)"
    export AWS_SECRET_ACCESS_KEY="$(echo $session_token | jq -r .SecretAccessKey)"

    terraform plan

```

---
Tags: amazon-web-services, terraform, authentication
---


# Question
Title: How do you add user secrets to a deployment without having them to source control?
I am developing applications that require user secrets to function. These credentials should not be stored in source control due to security concerns. Instead, they need to be manually created by the user or generated via a deploy script, with the values kept separate from the source code.

I am looking for the best approach to securely manage and access these user secrets in a deployment environment. What are the recommended options and best practices for handling such sensitive information, ensuring it remains secure and accessible to the applications when needed?

Specifically, I would like to understand:

1. What are the common strategies or tools used to store and manage user secrets securely outside of source control?
2. How can these secrets be securely accessed by the applications during deployment or runtime?
3. Are there any recommended techniques or patterns to automate the process of secret management and deployment while minimizing the risk of exposing sensitive information?

Any guidance or insights on the best practices, tools, and techniques for handling user secrets in a deployment scenario would be greatly appreciated.

# Answer
The current practice for this is to use a secret vault. There are flavours of this, such as Azure Key Vault, HashCorp Vault or AWS Secrets Manager.

You would call out to these vaults to get secrets on demand.

Your deployment tool might have a feature for sensitive varibles (i.e. they shouldn't be retrievable or viewable), but you can call out to a general valut to get secrets, as vaults offer some additional features that make it easier to *do the right thing* with secrets. There's usually an API to access the secrets and you can do it both during a deployment and at runtime (though there may be service limits).

To sum it all up...

1. The common strategy for secrets is to use secure vaults - you don't want them in config files or in source control
2. Each vault will have best practices for access, so when you pick a solution follow their guidance - you can normally lock access by more than authorization (for example, IP restrictions)
3. One of the key patterns on top of secrets vaults is to rotate secrets frequently (and automatically)

This is like a micro-topic all of its own as you can increase your security by locking things down, applying least privilege, and monitoring access to secrets to spot suspicious activity. I hope this helps as an introduction.
> 0 votes

---
Tags: security
---


title: Number EC2 instances required for tasks

text:
# Question
Title: Number EC2 instances required for tasks
Let's say I want to place my tasks on EC2 instances of type `m6i.large`, which has 2 vCPU and 8GB of memory.

I have 10 ECS tasks, each requiring 512 mCPU and 512MB of memory. How many EC2 instances do I need to run them?

```
Total CPU    = 10 x 512 / 1024 = 5 vCPU
Total Memory = 10 x 512 / 1024 = 5 GB

```

Do they require 3 instances of `m6i.large` to meet the CPU demands? Or can they share the same instance (and hopefully not burst at the same time to overload the machine)?

# Answer
> 0 votes
You're not asking about auto scaling groups, but I think you probably should be in that paradigm. If you're using an EC2 instance for doing two tasks where you could use the EC2 instance for one task - or even an EKS workload, you can save dough.

If you're just talking about operations on a single instance, nothing really changes in the cloud. You need to keep your load (see top commensurate to your number of cores. I have a hard time trusting the mCPU number you gave - I don't really know what that means (this is probably just me not knowing a thing).

I think you will get the most bang for your buck running t2a.smalls (1 CPU, 2GB's of RAM) with one process on them each. Reserve ten of them and let ASG pick up the bursty work.

---
Tags: amazon-web-services, amazon-ec2, amazon-ecs
---


# Question
Title: In Jenkins, when creating a Windows build node, how is its "Path" assigned?
Jenkins v2.375.1

I created a Windows build node and I did a "Launch agent via SSH" to launch it, and it's now online. When I go to its "System Information" page, I see a "Path" assigned to it. Where/how is it assigned? Better yet, how can I override it? I did set a "PATH" enironment variable in its Configuration page, but that value is NOT equal to the "Path" I see in the "System Information" page, shown below

Any clues? TIA

# Answer
This can be done in the node's `Configuration -> Launch method -> Advanced` section. See screen shot
> 0 votes

---
Tags: jenkins
---


# Question
Title: Is there a way to pass secrets in cloud-init using Terraform?
Cloud-init has the ability to compliment Terraform in configuring the Virtual Machine it just created. My problem is that cloud-init as uploaded through `user-data` means that the script especially if multi-part is available on the OpenStack metadata server.

This means *any* user on that virtual machine can get that user-data by simply running

```
curl http://169.254.169.254/latest/user-data

```

Is there anyway to pass secrets in a cloud-init file?

---

This is a follow up to *"Is there a way to install a private key for a user with cloud-init?"*. There I ask a question which could conceivably be answered with a provisioner that doesn't use cloud-init at all. This question is more broad.

# Answer
> This means any user on that virtual machine can get that user-data by simply running
>
> > curl http://169.254.169.254/latest/user-data

Correct

> Is there anyway to pass secrets in a cloud-init file?

Some ideas to consider:

* add a firewall rule at the end of initialization to limit access to `169.254.169.254`
* include a private key in the base image that can be used to decrypt encrypted content which is written via `write-files` or `runcmd` / `bootcmd`
* include the secret directly in the base image, or via a drive that is attached during provisioning
> 0 votes

---
Tags: terraform, cloud-init
---


# Question
Title: "An unknown error occurred" when authenticating between GitHub Actions and AWS using OIDC
I was unable to successfully authenticate when attempting to create a GitHub Actions workflow that would authenticate with an AWS IAM Identity Provider (IdP) using Open ID Connect (OIDC), based on this blog: https://aws.amazon.com/blogs/security/use-iam-roles-to-connect-github-actions-to-actions-in-aws/

In GitHub Actions the error appeared as:

```
Not authorized to perform sts:AssumeRoleWithWebIdentity

```

In AWS CloudTrail the error appeared as:

```
    "errorCode": "AccessDenied",
    "errorMessage": "An unknown error occurred",

```

# Answer
> 0 votes
The issue was caused by not removing the space after `repo:` in the IAM Identity Provider Trust relationships' trust policy line that I had copied from the blog

```
                    "token.actions.githubusercontent.com:sub": "repo: <aws-samples/EXAMPLEREPO>:ref:refs/heads/<ExampleBranch>",

```

---
Tags: amazon-web-services, aws-iam, github-actions
---


# Question
Title: Docker WWWUSER e "WWWGROUP variable is not set
I just created a project in WLS with Laravel using the command:

```
curl -s https://laravel.build/example-app | bash

```

When trying to use the command docker-compose exec app bash it returns the this error:

```
WARN[0000] The "WWWUSER" variable is not set. Defaulting to a blank string.
WARN[0000] The "WWWGROUP" variable is not set. Defaulting to a blank string.
service "app" is not running container #1
Looking in Docker, sail-8.2/app is running normally.

```

Also when trying to use commands using sail returns this:

```
Command 'sail' not found, but can be installed with:
sudo apt install bsdgames

```

# Answer
You last issue is the easiest to answer because it's part of the setup for sail that you likely just missed. When you initially ran your installation, the last terminal message would have been something like:

> Thank you! We hope you build something incredible. Dive in with: cd example-app && ./vendor/bin/sail up

As you can see, the `sail` you want is located in the vendor files that were installed when composer was run. Adding the following to your computer's config file will bridge the gap.

```
alias sail='[ -f sail ] && sh sail || sh vendor/bin/sail'

```

The issue about `service "app" is not running container #1` arises because Laravel renames the container service in the `docker-compose.yml` file. You can see the service is named `laravel.test`. If you were to run, for example:

```
docker compose exec laravel.test php artisan optimize

```

Then your container's routes and config should return as cleared. Notice, too, that `docker compose` is different from `docker-compose`, which will now throw a notice that the hypenated version is being deprecated.

However, the reason why you'll want to configure your shell alias in your computer's config file (*.zshrc* or *.bashrc*) is that you can replace the first 5 sections with `sail` to run the command on your container, as opposed to your local machine, i.e.

```
sail artisan optimize

```

This is very convenient if, for example, you need your container to run on a different version of PHP than your local machine. Using the `sail` alias, or even the full path `./vendor/bin/sail`, to run your Composer and Laravel commands will target the container's version of php, not your local machine's.

With respect to the `WWWGROUP` and `WWWUSER` I don't think you need to concern yourself with this message. Sail seems to take care of setting the webuser and group in the image.
> 0 votes

---
Tags: docker
---


# Question
Title: Bypassing OAuth in Traefik router that routes to other network device
In Traefik2, I've got `router.toml` that contains the following lines:

```
[http.routers]
    [http.routers.sonarr-bypass]
        rules = "Host(`sonarr.DOMAIN.com`) && Headers(`traefik-auth-bypass-key`, `thekey`)"
        middleware = ["chain-no-auth"]
        service = "sonarr"
        [http.routers.sonarr-bypass.tls]
    [http.routers.sonarr]
        rule = "Host(`sonarr.DOMAIN.com`)"
        middlewares = ["chain-oauth"]
        service = "sonarr"
        [http.routers.sonarr.tls]

[http.services]
    [http.services.sonarr.loadBalancer]
        [[http.services.sonarr.loadBalancer.servers]]
            url = "http://192.168.50.51:8989"

```

The `chain-no-auth` and `chain-oauth` middlewares work elsewhere, like in my `docker-compose.yml`, but the above sends me to my OAuth server regardless of my headers when accessing the URL `sonarr.DOMAIN.com`.

Am I missing something? Perhaps I need to set the priority?

# Answer
I put `rules` instead of `rule` since there was more than one condition. That was it. All working now.
> 0 votes

---
Tags: docker, authentication, traefik
---


title: Keeping Docker logs only in memory

text:
# Question
Title: Keeping Docker logs only in memory
There are various log drivers available for Docker but it looks like all of them persist the logs to disk.

To reduce disk usage I would like to keep the logs only in memory (and limited to a certain size).

I could use log driver `none` but I believe it immediately discards all log messages and sometimes it is still helpful to look at the currently produced logs without having to restart the container to switch to a different log driver.

Is this possible?

# Answer
One solution could be, I never tried, is to mount a memory filesystem (tmpfs) and let docker log to that directory with syslog

create a memory filesystem:

https://docs.oracle.com/cd/E18752\_01/html/817-5093/fscreate-99040.html

let docker log with syslog:

https://www.simulmedia.com/blog/centralized-docker-logging-with-rsyslog
> 1 votes

---
Tags: docker, logging
---


# Question
Title: ctop does not work with latest version of docker: Get "http://unix.sock/info": dial unix /var/run/docker.sock: connect: no such file or directory
I am very fond of ctop, a command-line manager for docker containers. I recently upgraded my docker to v23.0.5 on mac OSX Ventura 13.4 (Intel chip). After doing so, and running ctop, I get the daemon erroring out with this message:

> Get "http://unix.sock/info": dial unix /var/run/docker.sock: connect: no such file or directory

I checked and it is indeed true that there is no `docker.sock` there with the new version of docker. Checking another machine running docker v20, docker.sock is right there in that location. I figured maybe with the new version, that file has a new home. I did manage to find a docker.sock at `Users/sethlutske/.docker/run/docker.sock`, so I set the environment variable `export DOCKER_HOST=/Users/sethlutske/.docker/run/docker.sock` as recommended in the ctop docs on customization, but then running `ctop` gives me another error:

> Get "tcp:///Users/sethlutske/.docker/run/docker.sock/info": unsupported protocol scheme "tcp"

What has changed in the version of docker? ctop isn't exactly maintained any more, but ctop really isn't the issue considering it allows customization of where to find the docker socket. I'm trying to understand why the docker docs say that the docker socket should be at this location, but its somewhere else, and seems to be different than what ctop expects?

# Answer
> 5 votes
This is probably because Docker Desktop version 4.18+ doesn't create the default Docker socket `/var/run/docker.sock` anymore.

The reasoning is on the official docs: https://docs.docker.com/desktop/mac/permission-requirements/

If you want to have the default socket back, you should open your Docker Desktop, go to `Settings -> Advanced -> Allow the default Docker socket to be used` and press `Apply & restart`.

---
Tags: docker
---


# Question
Title: Gitlab runner tons of errors when trying to use shell executor
I installed the runners using the `docker` executor, but then later realized I actually needed the `shell` executor for layer caching... However now I just keep running into so many errors trying to get the runner to work...

```
sudo gitlab-runner list
Runtime platform                                    arch=amd64 os=linux pid=17362 revision=79704081 version=16.0.1
Listing configured runners                          ConfigFile=/etc/gitlab-runner/config.toml
affiliate-processor-1                               Executor=shell Token=******************* URL=https://gitlab.com
affiliate-processor-1                               Executor=shell Token=******************* URL=https://gitlab.com
gitlab-runner-final                                 Executor=shell Token=******************* URL=https://gitlab.com

```

And the journal logs:

```
sudo journalctl -u gitlab-runner --follow
-- Logs begin at Sat 2023-05-27 18:43:05 UTC. --
Jun 05 12:29:01 affiliate-processor-1 gitlab-runner[17177]: Running in system-mode.
Jun 05 12:29:01 affiliate-processor-1 gitlab-runner[17177]:
Jun 05 12:29:01 affiliate-processor-1 gitlab-runner[17177]: Configuration loaded                                builds=0
Jun 05 12:29:01 affiliate-processor-1 gitlab-runner[17177]: listen_address not defined, metrics & debug endpoints disabled  builds=0
Jun 05 12:29:01 affiliate-processor-1 gitlab-runner[17177]: [session_server].listen_address not defined, session endpoints disabled  builds=0
Jun 05 12:29:01 affiliate-processor-1 gitlab-runner[17177]: Initializing executor providers                     builds=0
Jun 05 12:29:01 affiliate-processor-1 gitlab-runner[17177]: ERROR: Checking for jobs... forbidden               runner=m5Q9_Nc_6
Jun 05 12:29:03 affiliate-processor-1 gitlab-runner[17177]: ERROR: Checking for jobs... forbidden               runner=m5Q9_Nc_6
Jun 05 12:29:04 affiliate-processor-1 gitlab-runner[17177]: ERROR: Checking for jobs... forbidden               runner=m5Q9_Nc_6
Jun 05 12:29:04 affiliate-processor-1 gitlab-runner[17177]: ERROR: Runner "https://gitlab.com************************" is unhealthy and will be disabled for 1h0m0s seconds!  unhealthy_requests=3 unhealthy_requests_limit=3

```

How can I fix this?

Some more issues, `systemctl status gitlab-runner.service` shows:

```
/usr/bin/gitlab-runner run --working-directory /home/chris --config /etc/gitlab-runner/config.toml --service gitlab-runner --user gitlab-runner

```

Why is it using my PERSONAL home directory as the working directory??? Surely it rather uses some system directory instead or the gitlab-runner home directory?!?!

---

OKAY I have uninstalled the old runners which seemed to be the cause of the issue... But now I have these pipeline errors:

```
mkdir: cannot create directory ‘/home/chris/builds/onty8Erw6’: Permission denied

```

?!?!?!? WHY IS IT USING THE `/home/chris` directory?!?!

# Answer
> 0 votes
I had to just set the following in `/etc/gitlab-runner/config.toml`

```
[[runners]]
  ...
  builds_dir = "/home/gitlab-runner/build/"

```

This parameter does not change the value in `/etc/systemd/system/gitlab-runner.service` **until** a build is triggered (it seems)...

---
Tags: gitlab, gitlab-ci-runner
---


# Question
Title: What is best practice for passing files and directories as command line arguments to a containerized program?
I am working to containerize a program. The program requires a configuration file and directories for its operation, which are passed as command line arguments, e.g. something like this:

```
$ myprog -c /path/to/config.conf -i /directory/with/input_files -o /directory/with/output_files

```

I know that I will need mount docker volumes to the containerized version of this program so that it can access the config file and directories. But, I would like to maintain the command-line interface unchanged (as far as possible) to hide the complexity of using docker volumes, etc. from users. How do I do this?

Will I need to create a wrapper script that maps the values of the command-line arguments (`-c`, `-i`, `-o`) to mount the corresponding paths as docker volumes? If so, what is the best practice for packaging it (i.e. to distribute the script with the container) and making this work cross-platform (i.e. so that it runs on Windows, MacOS, Linux)?

# Answer
I would suggest the following approach for the files within your `docker-compose.yml`:

```
volumes:
    - ./config:/path/to
    - ./input:/directory/with/input_files
    - ./output:/directory/with/output_files

```

*Do note* that I mount the entire folder in this example, despite you only mentioning one config file. This is on purpose - reasons for that are well explained within this video. Resources on how to define `docker-compose.yml` in full can be found here. Your final approach could look something like this:

1. Push Dockerfile to registry of your choice
2. Provide user the `docker-compose.yml` for example via Git
3. User creates folder with the necessary sub-directories (`config`, `input` and `output` in this example)
4. User supplies valid config file in `config` folder
5. User executes `docker-compose up -d`
6. Success :)
> 0 votes

---
Tags: docker, containers
---


# Question
Title: jenkins - gitparam jobdsl - branchFilter
I want to fill branchFilter automatically on Jenkins using JobDSL plugin and language.

This is the url of Jenkins Job DSL plugin:

https://jenkinsci.github.io/job-dsl-plugin/#path/javaposse.jobdsl.dsl.jobs.FreeStyleJob.parameters-gitParam

I want to select all branches but master. To do that I have written the following lines:

```
  //This Build is parametrized
  parameters {
      gitParam('branch') { // https://jenkinsci.github.io/job-dsl-plugin/#method/javaposse.jobdsl.dsl.helpers.BuildParametersContext.gitParam
          description('Git Branch')
          type('BRANCH')
          branchFilter("^(?!.*master).*$")
      }
  }

```

But when compiling it I get error that such method is not available.

> Processing provided DSL script ERROR: (script, line 28) No signature of method: javaposse.jobdsl.dsl.helpers.GitParamContext.branchFilter() is applicable for argument types: (java.lang.String) values: \[\*\] Possible solutions: tagFilter(java.lang.String)

Is there any workaround to do this automatically on the script or we have to fulfill that field after job has been created?

# Answer
> 1 votes
This is how I did:

```
gitParameter {
            // The name of the parameter.
            name('SELECTED_BRANCH')
            // The type of the list of parameters: Tag - list of all commit tags in repository - returns Tag Name Branch - list of all branch in repository - returns Branch Name Revision - list of all revision sha1 in repository followed by its author and date - returns Tag SHA1
            type('PT_BRANCH')
            // In my case this is a variable pointing to the gitlab hook variable
            defaultValue('$gitlabAfter')
            // A description that will be shown to the user later.
            description('Select a git the configuration you want to deploy')
            // Name of branch to look in.
            branch('')
            // Regex used to filter displayed branches.
            // Whatever is captured in the group will be the name used as the value of the SELECTED_BRANCH
            branchFilter('origin/(config\\/.*)')
            // This parameter is used to get tag from git.
            tagFilter('*')
            // Select how to sort the downloaded parameters.
            sortMode('NONE')
            // Which value is selected, after loaded parameters.
            selectedValue('NONE')
            // If in the task are defined multiple repositories, this option specifies which the repository is taken into account on getting data.
            useRepository("<my git url>.*")
            // When this option is enabled will show a text field.
            quickFilterEnabled(false)
            // Specify the number of items the list will display.
            // listSize(String value)
        }

```

As strange as it seems, some parameters shouldn't really be necessary, but the plugin requires them anyway, so that's why the config is verbose.

Also, pay attention that this one is `gitParameter` and not `gitParam`, yes, both exist, and they take different parameters.

For more information about the properties you can go here:

https://plugins.jenkins.io/git-parameter/

# Answer
> 0 votes
```
    parameters {
        gitParameter {
            name('selected')
            type('BRANCH')
            defaultValue('master')
            description('')
            branch('')
            branchFilter('origin')
            tagFilter('*')
            sortMode('NONE')
            selectedValue('NONE')
            useRepository("")
            quickFilterEnabled(false)
        }

```

Using this syntax it’s able to create a configuration inside the Jenkins job, but once I trigger that job , this configuration is getting removed automatically is there any solution to retain the configuration for every builds

---
Tags: jenkins, job-dsl
---


# Question
Title: Windows curl command corrupts zip files on upload to SFTP server
Per our IT department, I have to use a curl command to upload files to our SFTP server.

I use a Jenkins pipeline to build our artifacts on Windows build nodes, which are `*.zip` files, and this is the `curl` command I use

```
C:\> curl -V
curl 7.03.1 (Windows) libcurl/7.03.1 Schannel

C:\> curl -T ./build_artifact.zip -u 'user:password'  ftp://ftp.company.com/devops/test/build_artifact.zip --ftp-ssl-reqd --ftp-create-dirs -k

```

The file uploads correctly.

But when I download the file on my Mac and I unzip it, I get

```
$ unzip build_artifact.zip
Archive:  build_artifact.zip
  End-of-central-directory signature not found.  Either this file is not
  a zipfile, or it constitutes one disk of a multi-part archive.  In the
  latter case the central directory and zipfile comment will be found on
  the last disk(s) of this archive.
unzip:  cannot find zipfile directory in one of build_artifact.zip or
        build_artifact.zip.zip, and cannot find build_artifact.zip.ZIP, period.

```

Note if I execute the same `curl` command on my Mac to upload to the SFTP site, I can readily unzip the artifact when I subsequently download it.

Any clues on why this happens, and how to fix it?

# Answer
Updating to the latest Git bash, which included `curl v8.x` fixed the issue.
> -1 votes

# Answer
This could be the common known issue based on the upload transfer of binary files using the `curl` command on Windows.

By default, `curl` on a Windows device uses ASCII transfer mode which can cause issues with binary files like `.zip` archives. Whereas on a Mac device, the `curl` command uses binary transfer mode as default.

To avoid this, override the ASCII transfer method by implicitly specifying the transfer mode as binary, by using the `--ftp-binary` option.

See amended command below:

```
$ curl -T ./build_artifact.zip -u 'user:password' ftp://ftp.company.com/devops/test/build_artifact.zip --ftp-ssl-reqd --ftp-create-dirs -k --ftp-binary

```
> 1 votes

---
Tags: windows, curl
---


# Question
Title: Is terraform good option for data management
I have complex stack which consists of infrastructure, application and config parts, which are deployed to datacenters around the world. We have custom inhouse configuration system, based on database which is replicated elsewhere.

We have loads of configuration, which depends on our infrastructure - i.e. database connection strings, application specific configuration etc. Right now all this config is managed by hand, and drifts a lot. It's hard to say which config is still required and which can be decommissioned, it's hard to ensure exactly the same changes go from test environment to production, and that all environments were updated.

I wonder how can I improve this situation. I would like to have single source of truth in form of git repository, which can then be deployed to multiple datacenters (with some templating of variable substitution).

When I read this requirements, I think terraform with some kind of sql CRUD provider would be perfect tool:

* stored in git
* infra is already defined there, so pushing stuff like database connection strings is easy
* can review changes
* no manually configuring data in the system
* easy replication across environments

System based on database has to stay (too much code depends on that). Of course there is other data in the database, which changes a lot or is customer specific, and would not be managed by this setup.

Does it make sense to use terraform in such way?

Other tools I've checked:

* Ansible
* Salt
* Chef
* Puppet

All these tools were built with similar setup, however I see following downsides:

1. Would have to push infrastructure identifiers into these tools (which is repetitive and error-prone)
2. These are complex tools, which would have to be deployed alongside terraform
3. These tools do not track state and changes, so it's not possible to see what I'm changing (i.e. terraform plan)
4. When infrastructure is provisioned there is separate bootstrapping process to register in configuration tool

# Answer
> 2 votes
Terraform from high level is called IaC tool (Infrastructure as code) tool; So it comes in most handy when you use it in its primary function, **providing infrastructure**.

The part of configuration should be handled by **configuration management** tools; as you mentioned like ansible, chef, puppet ect..

By my experience, in one product, we are using terraform and ansible, hand to hand; it works perfectly fine, and not just it does the job, it is easy to read, understand and maintain/change/organize. Mixing IaC and Configuration management tools in general is so good to maintain your whole platform.

However, In some other product I worked on before, when they originally designed the product, they tried to use terraform as you say in every aspect, building and configuring, since you theoretically **can execute whatever script with terraform plugins** like null\_resource; so it is doable, but on the contrary of native configuration management tools: it is hard to understand, maintain and change.

Of course all of the above depends on the complexity of each system and its design and components, but my advise, **if you want configuration management, go for configuration management tools**.

---
Tags: terraform, configuration-management, databases
---


# Question
Title: ansible: difference between a variable and a fact
While I'm using Ansible for quite some time, I'm not sure I really understand the differences between a variable and a fact.

Would it be any difference doing

```
- set_fact:
    nginx_ssl: /etc/nginx/ssl
    nginx_conf_file: /etc/nginx/nginx.conf

```

or

```
  vars:
    nginx_ssl: /etc/nginx/ssl
    nginx_conf_file: /etc/nginx/nginx.conf

```

Could someone explain me, possibly with an example where it really make a difference ?

# Answer
Ansible **facts** are data collected about the (target) systems on which Ansible takes actions. They are variables, but set by Ansible (in a way like ***system defined variables***). They are collected during `Gathering Facts` stage of a playbook run, and it is controlled by the `gather_facts` setting. Ansible calls this variables discovered from systems. That said, it is possible to set custom facts also.

Some examples are:

* `ansible_hostname` \- the FQDN of the target system
* `ansible_os_family` \- the Operating System family of target system (RedHat, Debian, etc.)

The other **variables** are the ones we can set as per our requirement (in a way like ***user defined variables***).

Some examples are:

* `my_fav_fruits: [ 'orange', 'apple', 'banana' ]` \- yours might differ.
* `config_dir: '/etc/my_app/conf.d'` \- for my application configuration files.

---

**Update:**

*Updating answer to make it relevant to the edit(s) made in question.*

As @Bruce Becker's answer rightly pointed out, there is a difference in the precedence of variables set with `set_fact`. Also variables that need to be set at "run time" can be set this way. Without further explanation, taking your example variables, if I create a play like below:

```
- hosts: localhost
  vars:
    nginx_ssl: '/etc/nginx/ssl'
    nginx_conf_file: '/etc/nginx/nginx.conf'

  tasks:
  - name: set nginx path to /opt when running on Debian
    set_fact:
      nginx_ssl: '/opt/nginx/ssl'
      nginx_conf_file: '/opt/nginx/nginx.conf'
    when: ansible_distribution == 'Debian'
  - debug:
      msg: 'ssl: {{ nginx_ssl }} and conf: {{ nginx_conf_file }}'

```

Then the `set_fact` variables will take precedence (on Debian) and output will be:

```
"msg": "ssl: /opt/nginx/ssl and conf: /opt/nginx/nginx.conf"

```

On other distributions, it will be what was declared in `vars:`.
> 6 votes

# Answer
Not to contradict anything written in @Ceshadri\_C's answer, but to focus more on the difference between facts and variables - perhaps the variable precedence page could be of greater insight.

**The difference between facts and variables set is defined by their precedence**, *i.e.* all variables are important, but some variables are more important than others. In your case, *the same variable* set as a fact (priority 19) would take precedence declared as a variable in a task (priority 17) or play (priority 12).
> 3 votes

# Answer
Facts are also made available to subsequent plays. Variables are not.
> 1 votes

---
Tags: ansible, variables
---


# Question
Title: Prevent docker to write inside container
I have an extracted container deployed on a shared storage, and want to launch this container locally. However, when I launch this container to run an application, it creates some temporary files inside the container. I do not want to write anything inside the container and want these files to be generated locally (since it may create overhead on the shared storage). How can this be done? Thanks

# Answer
You could run the container with its root file system as read only:

`docker run --read-only`

and then ensure that a rw docker volume or tmpfs is mounted at each location the container process tries to write.

Depending on how well tested the app is for running in a ro filesystem, this may or may not work as expected.
> 1 votes

---
Tags: docker, dockerfile
---


# Question
Title: show running docker containers nicely formatted
Default layout of output from `docker ps` is somewhat unwieldy ... how to get a better presentation of all locally running docker containers from terminal on linux ?

For context below is output from `docker ps` as viewed from terminal for human consumption

# Answer
> 0 votes
```
docker ps -a |  grep -v Exited | cut -c1-$COLUMNS

```

Depending on desired columns this is a good starting point to obtain a nice display of all running containers for human consumption ... it auto handles when you resize terminal window using env var COLUMNS

```
CONTAINER ID   IMAGE                                               COMMAND                  CREATED             STATUS
0677a58056f3   loud_clamav/av                                      "/bootstrap.sh"          24 minutes ago      Up 24 minutes (he
ee374532c1bb   localhost:5000/hygge/loudweb-enduser                "/usr/bin/supervisor…"   24 minutes ago      Up 24 minutes
5546cd73ebcc   localhost:5000/hygge/loudweb-admin                  "/usr/bin/supervisor…"   24 minutes ago      Up 24 minutes
5e7c55a950b1   mongo                                               "docker-entrypoint.s…"   24 minutes ago      Up 24 minutes
505909fa4490   mcr.microsoft.com/mssql/server:2017-latest-ubuntu   "/opt/mssql/bin/nonr…"   30 minutes ago      Up 30 minutes
a1ea1d2882f5   localhost:5000/hygge/loudweb-dind/2204280434        "/usr/bin/supervisor…"   About an hour ago   Up 24 minutes
4713b632a08f   tvial/docker-mailserver:latest                      "supervisord -c /etc…"   43 hours ago        Up 24 minutes
010da9fc7fd5   registry:2.7.0                                      "/entrypoint.sh /etc…"   43 hours ago        Up About an hou
o

```

the big benefit of above command is dynamically handles current size of your terminal where you executed the command ( notice the use of COLUMNS env var)

naturally you can create an alias as a shortcut to run this command just save below in your file `~/.bashrc` which allows you to just type in `dpss`

```
alias dpss='docker ps -a |  grep -v Exited | cut -c1-$COLUMNS ' #  truncate width shows driven by current terminal size ... to avoid line wrapping

```

# Answer
> 0 votes
As a possible alternative to using shell pipes, check out the `--filter` and `--format` options of `docker ps`. (`--no-trunc` is also handy sometimes.)

For example, the output of `docker ps -a | grep -v Exited | cut -c1-$COLUMNS` can be approximated by the following command:

`docker ps -a --filter 'status=running' --format 'table {{.ID}}\t{{.Image}}\t{{.Command}}\t{{.RunningFor}}\t{{.Status}}'`

('approximated' because filter expressions annoyingly don't support NOT-operations and `status=running` is not exactly the same as `NOT exited`. Plus you don't get the neat auto-truncation of columns.)

Downside is that you have to grapple with a new DSL (Go templates) that might not be as familiar as shell pipelines. On the plus side, `--format` allows you to shape the output in any way that you see fit, e.g.

```
$ docker ps --format '{{.Image}}: {{.Command}}'
netdata/netdata: "/usr/sbin/run.sh"

```

---
Tags: docker, linux
---


# Question
Title: & (ampersand) in Terraform string or/and AWS IAM policy
I coudn't find meaning of `&` sign in Terraform or/and AWS IAM policy. There is an example in `Data Source: aws_iam_policy_document` (docs) where you can see `Basic Example` (docs, github source):

```
data "aws_iam_policy_document" "example" {
  (..)

  statement {
    actions = [
      "s3:*",
    ]

    resources = [
      "arn:aws:s3:::${var.s3_bucket_name}/home/&{aws:username}",
      "arn:aws:s3:::${var.s3_bucket_name}/home/&{aws:username}/*",
    ]
  }
}

```

I cound't find the meaning of `&{aws:username}`. I'd try to look up in AWS IAM documentation but there is only explanation how to use variables with dollar (`$`):

> *The variable is marked using a $ prefix followed by a pair of curly braces ({ }).*

Source

Question: What does `&` character mean in above example?

# Answer
> 1 votes
The note callout at the top of the `aws_iam_policy_document` documentation explains this syntax:

> **Note:** AWS's IAM policy document syntax allows for replacement of policy variables within a statement using `${...}`-style notation, which conflicts with Terraform's interpolation syntax. In order to use AWS policy variables with this data source, use `&{...}` notation for interpolations that should be processed by AWS rather than by Terraform.

In other words, this is a special syntax implemented as a special case inside the `hashicorp/aws` Terraform provider to make it easier to use the IAM policy interpolation syntax without conflicting with Terraform's own template interpolation syntax. The provider will automatically replace `&{` with `${` while it is rendering the final policy document.

---
Tags: amazon-web-services, terraform
---


# Question
Title: What is the meaning of the podCIDR field in the Node spec in kubenretes?
I've set up a Kubernetes cluster using `kubeadm` using the command:

```
$ sudo kubeadm init --pod-network-cidr=192.168.0.0/16 --kubernetes-version=1.24.00

```

My cluster has 2 worker nodes, and when I run for example:

```
$ kubectl get node worker1 -oyaml

```

I can see that there's a field in the node definition `.spec.podCIDR`,which is set to `192.168.1.0/24`.

But when running a pod on this node, I can see that it got an IP of `192.168.192.5`, which is not inside the range specified in the Node definition above.

So I'm trying to understand what is the meaning of this field? The Kubernetes docs don't say too much:

> podCIDR (string) PodCIDR represents the pod IP range assigned to the node.

Can someone calrify?

# Answer
A Kubernetes cluster runs pods, and the pods need to interact with each other for application needs. This can be possible if the pods are on the same network.

Kubernetes uses **container networking interface** to maintain pod networks and assign IP addresses to pods. There are many CNI providers available, and a few of them are Flannel, Calico, Cilium, Weavenet, and Canal. While setting up the cluster the cluster administrator must also set up the CNI provider.

While configuring the CNI, the administrator provides a pool of IP addresses which can be assigned to pods. This pool of ip addresses are called pod cidr.
> 1 votes

# Answer
The podCIDR field in the Node spec in Kubernetes specifies the range of IP addresses that will be allocated to the pods running on the node.

This field is used by the Kubernetes cluster to determine how to assign IP addresses to each pod.

The podCIDR field must be unique for each node in the cluster.
> 0 votes

# Answer
some CNI implementations do not respect node.spec.podCIDR.

cilium is one example: https://docs.cilium.io/en/v1.12/concepts/networking/ipam/cluster-pool/

> The difference is that instead of Kubernetes assigning the per-node PodCIDRs via the Kubernetes v1.Node resource, the Cilium operator will manage the per-node PodCIDRs via the v2.CiliumNode resource.
> 0 votes

---
Tags: kubernetes, networking
---


# Question
Title: How to add an aditional ServiceMonitor for prometheus-operator?
I installed prometheus-operator using kube-prometheus-stack and so far everything has worked fine, but I need to add one more service to monitoring. This is a Windows Server that is not running on the Kubernetes cluster.

I can't add the target to prometheus.yml because the storage is read-only. I saw that I could use ServiceMonitor for this purpose, but even following the steps in the official documentation, the exporter is not detected.

# Answer
> 1 votes
I don't know if you managed to get this working, but I had the exact same issue.

Running a K8s cluster with `kube-prometheus-stack` installed via Helm, and a CRD prometheus-operator. I also needed to monitor an external Windows SQL VM outside the cluster, in the same network CIDR, with the K8s cluster & SQL VM being in separate subnets.

I ended up installing the Windows-Exporter on the SQL VM. How I got Prometheus to pick up and scrape this external target is as follows:

1. I defined a ServiceMonitor.
2. I defined a *headless* Service (it has no label selector specified)
3. I then defined a custom Endpoints object that the headless Service uses.

**Note**: Because we define a headless Service we need to manually create/manage the underlying `Endpoints`, as Services create those by default. Also, labelling your resources correctly are super important.

Here are the relevant YAML files:

1. ServiceMonitor (SM) yaml:

```
  apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  metadata:
    name: sql-server-metrics # This name is usually important
    namespace: monitoring
    labels:
      # The below meta labels are NB!
      # It is used to match the SVC to SM and the EP to the SVC.
      kubernetes.io/service-name: sql-server-metrics
      app.kubernetes.io/name: sql-server-metrics
  spec:
    endpoints:
      - port: sql
        interval: 30s
        path: /metrics
        targetPort: 9182
    # The below selector tells SM to pick up all SVC's with these meta labels
    selector:
      matchLabels:
        kubernetes.io/service-name: sql-server-metrics
        app.kubernetes.io/name: sql-server-metrics

```
2. Service (SVC) yaml:

```
  apiVersion: v1
  kind: Service
  metadata:
    name: sql-server-metrics
    namespace: monitoring
    labels:
      kubernetes.io/service-name: sql-server-metrics
      app.kubernetes.io/name: sql-server-metrics
  spec:
    # This can be one of 4 types, check the Service docs for K8s
    type: ExternalName
    # This can be the IP or FQDN/Partial-FQDN
    externalName: {IP_OR_HOSTNAME}
    ports:
      - name: sql
        protocol: TCP
        port: 9182
        targetPort: 9182

```
3. Endpoints (EP) yaml:

```
  apiVersion: v1
  kind: Endpoints
  metadata:
    name: sql-server-metrics
    namespace: monitoring
    labels:
      # This is needed to match the selector-less External Service to this Endpoint
      kubernetes.io/service-name: sql-server-metrics
      app.kubernetes.io/name: sql-server-metrics
  subsets:
    - addresses:
      - ip: {VM_IP}
        hostname: {VM_HOSTNAME}
      ports:
        - name: sql
          port: 9182
          protocol: TCP

```

I do hope the above response helps you or anyone in the future, please do feel free to mark this as the answer or to ask for any clarity and I will respond ASAP.

Chow!

---
Tags: kubernetes, prometheus
---


# Question
Title: Using a terraform template to produce a cloud-init yaml: template variables do not respect yaml context/whitespace
I am using Terraform. Terraform has a notion templatefile which can be used to take variables that Terraform has access to (locally supplied or retrieved externally) and to apply them to a Jinja2 template. This specific example is very close to the one in the documentation.

People use these templates to generate cloud-init files with Jinja2 templates. Here is a snipped of the template,

```
ssh_keys:
    ecdsa_private: "${ssh_ecdsa_key_private}"

```

Then these variables get set in Terraform, like this,

```
data "template_file" "user_data" {
  template = templatefile(
    "cloud/cloud.cfg",
    {
      ssh_ecdsa_key_private  = file("./ssh/acme-bitbucket-id_ecdsa")
    }
  )

```

This is however problematic for me presumably because cloud-init is yaml.

```
  in "<unicode string>", line 22, column 1:
    "
    ^
could not find expected ':'
  in "<unicode string>", line 24, column 8:
          "-----BEGIN OPENSSH PRIVATE KEY-----
           ^"
2022-03-18 00:20:32,611 - util.py[WARNING]: Failed at merging in cloud config part from part-001
2022-03-18 00:20:32,611 - util.py[DEBUG]: Failed at merging in cloud config part from part-001
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 140, in handle_part
    self._merge_part(payload, headers)
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 116, in _merge_part
    (payload_yaml, my_mergers) = self._extract_mergers(payload, headers)
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/cloud_config.py", line 95, in _extract_mergers
    mergers_yaml = mergers.dict_extract_mergers(payload_yaml)
  File "/usr/lib/python3/dist-packages/cloudinit/mergers/__init__.py", line 79, in dict_extract_mergers
    raw_mergers = config.pop('merge_how', None)
AttributeError: 'NoneType' object has no attribute 'pop'
2022-03-18 00:20:32,616 - __init__.py[DEBUG]: Calling handler CloudConfigPartHandler: [['text/cloud-config', 'text/cloud-config-jsonp']] (__end__, None, 3) with frequency once-per-instance

```

I believe what's happening here is the file isn't being indented properly. Let's say `./ssh/acme-bitbucket-id_ecdsa` looked like this,

```
-----BEGIN OPENSSH PRIVATE KEY-----
FOO
BAR
BAZ
-----END OPENSSH PRIVATE KEY-----

```

I think that's getting processed into the template like this,

```
ssh_keys:
    ecdsa_private: -----BEGIN OPENSSH PRIVATE KEY-----
FOO
BAR
BAZ
-----END OPENSSH PRIVATE KEY-----

```

What is the way to stop properly produce a `cloud-init` file with Terraform?

# Answer
Here is how I got around it, I declared these as HDL object,

```
data "template_file" "user_data" {
  template = templatefile(
    "cloud/cloud.cfg",
    {
      ssh_keys = {
        ssh_ecdsa_key_private  = file("./ssh/acme-bitbucket-id_ecdsa")
        ssh_ecdsa_key_public   = file("./ssh/acme-bitbucket-id_ecdsa.pub")
      }
    }
  )
}

```

And then I pulled it into the yaml file using `yamlencode`.

There is a bit of documentation that covers this, over at *"Generating JSON or YAML from a template"*
> 0 votes

# Answer
By the way if you want to use yaml for your cloud-init template you can achieve what you want using `indent` function

```
ssh_keys:
  rsa_private: |
    ${indent(4, "${private_key}")}

```

Make sure to adjust the indented spaces as per your template.
> 1 votes

---
Tags: terraform, cloud-init
---


# Question
Title: How to wait for gitlab downstream pipelines to finish before going to next stage?
I'm not sure this is the way to proceed, but I'm starting using downstream pipelines.
I have 3 projects :

* Project A : web-frontend-1
* Project B : web-frontend-2
* Project C : global-e2e-testing

I'm setting up a CI pipeline on Project C which should do :

* Run main pipeline of Project A, with var TESTING=true (that is building docker images)
* Run main pipeline of Project B, with var TESTING=true (that is building docker images)
* Deploy on testing environment (it's pulling docker images, and relaunching project)
* Run e2e tests on testing environment

Here is my Project C `.gitlab-ci.yml` (upstream pipeline) :

```
stages:
- build-testing
- deploy
- test

build-project-a-testing:
  stage: build-testing
  trigger:
    project: workspace/project-a
  variables:
    TESTING: true

build-project-b-testing:
  stage: build-testing
  trigger:
    project: workspace/project-b
  variables:
    TESTING: true

deploy-testing:
  stage: deploy
  tags: ["testing-runner"]
  script:
    - echo "deploying..."

playwright-firefox:
  stage: test
  before_script:
    - npm i
    - npx playwright install
  script:
    - npx playwright test --project=firefox --workers=1
  tags:
    - run-playwright
  artifacts:
    name: playwright-report-firefox
    paths:
      - playwright-report
      - results.xml
    reports:
      junit: results.xml
    expire_in: 1 week
    allow_failure: true

```

The issue I have is that the downstream pipeline are launched but without waiting them for ending.
So deployment job, would deploy an old container image, and testing is not relevant.

Is there a way to wait for downstream pipeline to finish ?
I can "reverse" the order of which project is downstream / upstream, but since I have two projects to build, the issue would be the same.

# Answer
> 1 votes
The strategy keyword is the solution :

From the documentation :

> ```
> trigger_job:
>   trigger:
>     include: path/to/child-pipeline.yml
>     strategy: depend
>
> ```
>
> In this example, jobs from subsequent stages wait for the triggered pipeline to successfully complete before starting.

---
Tags: gitlab, cicd, gitlab-ci-runner
---


# Question
Title: What are possible backup status in velero?
I am using velero for k8s cluster backup. I was looking for possible backup status in velero. I couldn't find any proper documentation on this. Can someone please help

# Answer
> 1 votes
As I see in Velero code (version: v.1.5.3), the backups can have the following phases:

* New
* FailedValidation
* InProgress
* Completed
* PartiallyFailed
* Failed
* Deleting

see: https://github.com/vmware-tanzu/velero/blob/v1.5.3/pkg/apis/velero/v1/backup.go#L195

Restores have these phases:

* New
* FailedValidation
* InProgress
* Completed
* PartiallyFailed
* Failed

see: https://github.com/vmware-tanzu/velero/blob/v1.5.3/pkg/apis/velero/v1/restore.go#L184

# Answer
> 0 votes
In the latest version the files for finding the phases have changed. Also new phases have bee n added:

## Backup Phases

* New
* FailedValidation
* InProgress
* WaitingForPluginOperations
* WaitingForPluginOperationsPartiallyFailed
* Finalizing
* FinalizingPartiallyFailed
* Completed
* PartiallyFailed
* Failed
* Deleting

see: https://github.com/vmware-tanzu/velero/blob/433daa18bd76548d14d38fea3af6d3b64f0bc9a5/pkg/apis/velero/v1/backup\_types.go#L276

## Restore Phases

* New
* FailedValidation
* InProgress
* WaitingForPluginOperations
* WaitingForPluginOperationsPartiallyFailed
* Completed
* PartiallyFailed
* Failed

see: https://github.com/vmware-tanzu/velero/blob/433daa18bd76548d14d38fea3af6d3b64f0bc9a5/pkg/apis/velero/v1/restore\_types.go#L229

---
Tags: docker, kubernetes, backup
---


# Question
Title: Unable to see manually created playbooks in AWX
I have a strange issue that has me racking my brain. I have installed AWX 22.3.0 onto Ubuntu 22.04.1-Ubuntu at least three times now. Let me make sure I get everything to those reading this.

I have followed many whitepapers, directions, and videos, and for the life of me, I can't figure out why I can't get any playbooks using,.yaml or.yml in the `/var/lib/awx/projects/` to appear within "Base Project Path". I continue to receive the message below:

```
There are no available playbook directories in /var/lib/awx/projects. Either that directory is empty or all of its contents are already assigned to other projects. Create a new directory there and make sure the playbook files can be read by the "awx" system user, or have AWX directly retrieve your playbooks from source control using the Source Control Type option above.

```

I'm using K3S Kubernetes, and my pods are all active and working.

I have edited my "awx-web" to make sure it has the `/var/lib/awx/projects` path as the

```
kubectl exec -ti deploy/awx-web -c awx-web -- /bin/bash

```

env

```
export PROJECTS_ROOT="/new/path/to/projects"

```

All of it looks good. I deleted and recreated the pod after the change as well. But the ROOT path was already there originally, as I expected based on how I installed it.

Hopefully, that is enough for people to help me with my problem. I'm just at a loss and don't know what else it could be at this point. I mean, do I need to open firewall ports? I just don't know.

Also, the permissions are below after I did a `mkdir -P` and manually created a `/var/lib/awx/projects/playbooks` directory. Then I created a YAML file.

Maybe it could be permission as well; who knows?

# Answer
For those that are interested, I have yet to find a resolution for this problem. I simply created a github account, and that works for Ansible automation.
> 0 votes

---
Tags: kubernetes, ubuntu, awx
---


# Question
Title: EKS: Usable Memory goes lower and lower on the node to 5% then jumps up to 80% available
I was able to correlate the "usable memory" going down with events in kubernetes:

```
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Tue, 06 Jun 2023 19:31:34 +0000   Thu, 11 May 2023 15:24:32 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available

```

The time of the `LastTransitionTime` coincides with the lowest memory and the transition to a lot of available memory.

Also, I think that during the times of low available memory I get a lot of OOM errors and processes are getting killed with exit code 137.

Is there something misconfigured in this Kubernetes cluster? I think the available memory should not ever go that low and it should start evicting pods earlier. Also I don't understand why the whole memory becomes available - this means that we are evicting ALL pods?

# Answer
From the get-go, my educated guess would be memory leak your workloads. It seems that his happens pretty regular - every 12 hours, do you have some scheduled/cron job?

Alternative problem could be overcommitted resources.

First of all check `resources.limits` on workloads you are running on the EKS cluster. Make sure `resources.limits.memory` is set on every single container. You can play around with limit ranges to set defaults for a namespace; this way you "protect" your nodes to some extent (alternative is to use resource quotas.

I see you are using Datadog, you can leverage Kubernetes metrics to find spiking workload - `kubernetes.memory.working_set` (reference):

> Current working set in bytes - this is what the OOM killer is watching for

The description of metric you are currently using (reference):

> The `system.mem.usable` metric represents free memory plus cached memory plus buffers (on Linux, it reflects “MemAvailable” attribute from `/proc/meminfo` whenever possible).

On a "cluster" side of things you may enable "resource reservation", which may help with keeping system daemons (like `kubelet`) alive. Also read docs on node pressure evictions and thresholds.
> 0 votes

---
Tags: kubernetes, eks
---


title: Auto tune kubernetes requests/limits?

text:
# Question
Title: Auto tune kubernetes requests/limits?
I know of ways to set up monitoring (e.g. prometheus/alert-manger) to notify me if kubernetes pod requests/limits are too far off actual container usage.

Are there any systems built to help you actually act on this information in some automated fashion though? This may sound odd as I realize automatically raising a limit makes it... not a limit?

Having said that, it still sounds useful to be able to automate tuning across a large k8s environment. This may be especially true during non-prod/staging when you're doing load testing and trying to figure out what the reasonable limits are in the first place.

# Answer
> 1 votes
Automating this kind of behaviour is a quite risky since you have ResourceQuotas usually set up as a constraint for a namespace and from there on the pods must be aligned with it. You might end up with a pod non being scheduled at all.

# Answer
> 0 votes
Great question, especially mentioning fine-tuning on DEV / STG during performance testing. You may try using Vertical Pod Autoscaler to at least get good baseline and go from there.

You can just use the `recommender` component (`spec.updatePolicy.updateMode: Off`) and not touch controllers at all:

> Recommender - it monitors the current and past resource consumption and, based on it, provides recommended values for the containers' cpu and memory requests.

---

Keep in mind that VPA can not work with HPA enabled, and also keep in mind that each "adjustment" causes new ReplicaSet to be created (more limitations are documented in the docs).

---
Tags: kubernetes
---


# Question
Title: What is the relationship between deployment and container level readiness probes?
I have a helm chart that defines my container's `readinessProbe` as:

```
readinessProbe:
    httpGet:
      path: /health/readiness
      port: http

```

And when I deploy it I can browse to http://myservice.mydomain/istioBasePath/health/readiness and I get a response of `Healthy`.

But if I do a `kubectl describe pod` on one of the pods that comprise this service, I see this in the details for my container:

```
Readiness:  http-get http://:15020/app-health/my-servce-name-is-shown-here/readyz

```

Which is not at all what I setup in the deployment for my container. (If nothing else, it has a `z` in the name.)

Additionally, when I add the IP address of the container to that URL and run a `curl` from within the cluster, it does not respond with any text. Running `curl` with the `-I` option shows:

```
HTTP/1.1 200 OK
Date: Wed, 17 May 2023 23:26:11 GMT

```

(Again, no `Healthy` text is returned.)

Something is different here, but I am not quite sure what it is. The URL for my "Deployment Defined" readiness probe works, but it returns the text `Healthy`. The probe defined on the container (that I did not setup, and thought I was overriding with my deployment), just has a Http status result.

I thought I had only one probe going on, but clearly there is a difference.

Can someone explain the difference to me?

# Answer
> 1 votes
You are most likely using Istio which you did not mention in the question. By default (from version 1.5.2) probe rewrites are enabled. This is required because of mTLS - for more details read the documentation.

> The health check requests to the liveness-http service are sent by Kubelet. This becomes a problem when mutual TLS is enabled, because the Kubelet does not have an Istio issued certificate. Therefore the health check requests will fail.
>
> TCP probe checks need special handling, because Istio redirects all incoming traffic into the sidecar, and so all TCP ports appear open. The Kubelet simply checks if some process is listening on the specified port, and so the probe will always succeed as long as the sidecar is running.
>
> Istio solves both these problems by rewriting the application PodSpec readiness/liveness probe, so that the probe request is sent to the sidecar agent. For HTTP and gRPC requests, the sidecar agent redirects the request to the application and strips the response body, only returning the response code. For TCP probes, the sidecar agent will then do the port check while avoiding the traffic redirection.

---
Tags: kubernetes, healthcheck
---


# Question
Title: Shorten a reused expression in a helm template
I am writing a helm template for external secrets.

The secrets are in AWS secrets manager, and they're prefixed with "dev" or "prod".

Namespaces are used to differentiate environments, but there are many more than "dev" or "prod" -- "prod" and "prod-phi" should use "prod" secrets, while everything else should use "dev" secrets.

To accomplish that, I have a pretty long expression, which you can see prefixing a secret name here:

```
    - secretKey: RdsDataUsername
      remoteRef:
        key: {{ or (eq .Release.Namespace "prod") (eq .Release.Namespace "prod-phi") | ternary "prod" "dev" }}/rds/data/primary
        property: username

```

I am repeating that expression "`{{ or (eq .Release.Namespace "prod") (eq .Release.Namespace "prod-phi") | ternary "prod" "dev" }}`" many times throughout this file, and it's affecting readability.

Is there a way to set a variable one time in the file using this expression, then reference that instead?

Something like:

```
variables:
  prefix: {{ or (eq .Release.Namespace "prod") (eq .Release.Namespace "prod-phi") | ternary "prod" "dev" }}

spec:
  data:
    - secretKey: RdsDataUsername
      remoteRef:
        key: {{ $prefix }}/rds/data/primary
        property: username

```

# Answer
> 1 votes
You have couple options two that come to my mind:

* use variable
* use helper function

### Variable

On top of the file declare a variable and then just use it:

```
{{- $prefix := (or (eq .Release.Namespace "prod") (eq .Release.Namespace "prod-phi") | ternary "prod" "dev") -}}
...
spec:
  data:
    - secretKey: RdsDataUsername
      remoteRef:
        key: {{ $prefix }}/rds/data/primary

```

### Helper function

make a file `_helpers.tpl`

```
{{/*
Environment prefix.
*/}}
{{- define "CHART-NAME.prefix" -}}
{{- (or (eq .Release.Namespace "prod") (eq .Release.Namespace "prod-phi") | ternary "prod" "dev") -}}
{{- end -}}

```

and use it:

```
spec:
  data:
    - secretKey: RdsDataUsername
      remoteRef:
        key: {{ include "CHART-NAME.prefix" . }}/rds/data/primary

```

---

> **Note**: If you are using the prefix in multiple files, you can use both "methods" in combination.
>
> ```
> {{- $prefix := include "CHART-NAME.prefix" . -}}
> ...
> spec:
>   data:
>     - secretKey: RdsDataUsername
>       remoteRef:
>         key: {{ $prefix }}/rds/data/primary
>
> ```

---
Tags: kubernetes, helm, kubernetes-helm
---


# Question
Title: Why does AWS wrap roles with instance profiles?
In AWS EC2, a machine is assigned privileges using instance profiles. Specifically:

* the desired permissions are articulated as an IAM policy,
* that policy is attached to an IAM role,
* that role is assigned to an Instance Profile,
* the Instance Profile is assigned to the new EC2 instance (often first via a Launch Template where it is accompanied with other configuration settings),
* then processes running on the instance can (using AWS SDKs) obtain credentials for the role, and use those to authorise the actions (via requests to AWS API service endpoints) permitted by the policy.

What is the purpose of the Instance Profile layer? For example, is it solely just a wrapper for a role? In other words, what is the intent behind exposing this indirection layer to the client? Particularly, why isn't the role assigned directly to the instance (in the Launch Template or the RunInstance API)?

# Answer
> 1 votes
The data structures involved with EC2 and IAM are almost as old as EC2 itself. I don't know the definitive answer because I didn't work at AWS when these were built, but it was most probably a kludge and maybe future-proofing.

Many of the newer services that are built today, for example Lambda, don't need this kludge anymore and can link IAM Roles directly to the Lambda function.

Since for the most part, AWS try and keep existing APIs and not break them. This EC2 IAM Profile thing is unlikely to change.

---
Tags: amazon-web-services, amazon-ec2, aws-iam
---


# Question
Title: Where should I look for the x509 certificate Kubelet is using to communicate with the kube-apiserver?
I have a cluster that is setup using the tool `kops` version 1.21.5. I'm aware that's quite old. The nodes have been up for quite some time now (eg; ~450 days)

I'm receiving the following alert "A client certificate used to authenticate to kubernetes apiserver is expiring in less than 7.0 days" which I believe ships by default with prometheus operator.

I've had a look on the nodes and they are running `kubelet` with the following flags

```
/usr/local/bin/kubelet --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook ... --kubeconfig=/var/lib/kubelet/kubeconfig ...--tls-cert-file=/srv/kubernetes/kubelet-server.crt --tls-private-key-file=/srv/kubernetes/kubelet-server.key

```

I believe this means they should be attempting to authenticate with the cert/key from either those files or the one stored in the `kubeconfig` (a bit unsure on this point).

However when I check those certificates using `openssl x509 --noout -text` they all seem to have sensible values in the `notAfter`

I've had a node go down, whereby it'll start complaining about 'Unauthorized' when trying to access the kube-apiserver. When this happened, I was able to confirm that there is definitely an x509 certificate being sent to the api-server that is past its 'notAfter' date, based on the logs from the kube-apiserver pod. The 'notAfter' date in the logs is not the date on the .crt/stored cert in the kubeconfig.

On the node in /var/log/syslog;

```
"Error updating node status, will retry" err="error getting node \"<my node name>": Unauthorized"

```

In the kubectl logs from the relevant `kube-apiserver`

```
"Unable to authenticate the request" err="[x509: certificate has expired or is not yet valid: current time 2023-06-18T02:18:16Z is after 2023-06-16T14:46:43Z, verifying certificate

```

I've tried to `find` everything ending in .crt and didn't see anything obvious. There were the two from above, a bunch of `mozilla` ones which at a guess come from Ubuntu by default and a lot in /docker/overlay2 which I guess are something from the containers.

Terminating the nodes in AWS seems to result in a new node being created which has a valid expiry that doesn't trip the warning anymore. So something must be getting recreated somewhere.

I'd really like to know where this cert is, but everything I search for online has pointed me to either those files or `/etc/kubernetes/pki` (which doesn't seem to exist on my workers)

# Answer
> 0 votes
The files in `/etc/kubernetes/pki` are found on the API server, not on the nodes. And as you've noted the `--tls-cert-file=/srv/kubernetes/kubelet-server.crt` parameter of your Kubelet specifies which certificate file it is using.

One possible problem you might have is time drift. In order to verify certificates, both server and client need to have synchronized clocks. You can try and sync your node with NTP and perhaps this will resolve the issue.

Another potential problem is that the Certificate Authority, one of the files in `/etc/kubernetes/pki` on the API server, is not the same CA that signed your kubelet's certificate. When a Kubelet is created, it receives a certificate signed by this specific Kubernetes API server, and that is how they authenticate with each other. But if for example you try and authenticate this Kubelet with a different service, it will not recognize it. While a Kubelet is running, new refreshed certificates are issued from time to time and applied on the Kubelet side. You can inspect the Kubelet certificate and the CA certificate to make sure they match. You noted that the certificate is not expired, but it might have been revoked, this can also be seen on the side of the API server in its revocation list.

---
Tags: kubernetes, kops
---


# Question
Title: How can I take away RAM from a running Kubernetes pod?
I've got pods that use a greatly varying amount of memory over their lifetime.

If I understand correctly, in K8S, if a pod gets more memory (growing from request to limit), it doesn't relinquish it back to the common pool, even after it no longer needs it.

So I ended up in a situation where I've got many pods, all of which have lots of RAM that they don't need for 95% of their lifetime.

So, is there any way to get memory back from greedy pods when they're inactive? Without killing them I mean, as that would present its own complications.

# Answer
**TLDR**: If you are not getting OOM kills, you are fine. Leave it as is, kernel will reclaim memory when it is required/on demand.

What you ask is not "possible" from user point of view (beyond killing the process). The memory is claimed/used and can be freed only by process itself or kernel (memory reclaiming / OOM-killer etcetera).

In Kubernetes world, resource management and resource allocation is done by cgroups. Essentially what happens is resources are sliced up. Many times the sum of all limits on a particular node/worker go over its physical capacity, this is called overcommitting. Overcommitting is double-edged sword, on one hand you can fit more workloads on a node/worker but once there is higher traffic/load (and application requires more resources) it can be OOMkilled or throttled to stand still.

I'd like to add that memory management is very complex topic, it is described in depth in the https://docs.kernel.org/admin-guide/mm/index.html.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: How does continuous integration relate to continuous delivery / deployment?
Here is a quote from the current content of :

> ... process of merging developer's working code copies to a shared codebase frequently to prevent or minimize integration problems.

OK, I get that. But then there is also and , and that's where I *continuously* get a bit lost:

* How does *continuous integration* relate to *continuous delivery* and/or *continuous deployment*, assuming that somewhere along the line(s) via ***`integration`*** you end up ***`delivering`*** in a target environment where everything will be ***`deployed`***.
* What's the difference between *continuous delivery* and *continuous deployment*?

Back in the days, before DevOps was called DevOps, we used terminology which might possibly help to understand these new DevOps terms, such as:

* *promote* to (or *demote* from) some pre-prod target, optionally combined with some type of regeneration process (compiles, binds, etc) to package all related components together in executable-like things. That's what should be similar/close to *continuous integration*, or not?
* *distribute* to some target environment, using something like FTP (if standard copies cannot bridge the gap), but do not yet activate it in the target. That's what should be similar/close to *continuous delivery*, or not?
* *install* (or *activate*) in some target environment, combined with things like binds, stop/start operations, etc. That's what should be similar/close to *continuous deployment*, or not?

# Answer
Continuous delivery and continuous deployment both take continuous integration one step further, by adding a 'deployment to production' step to the process. The difference between continuous delivery and deployment is that for delivery this step is done manually and for deployment is it automatic.

*Difference between Continuous Integration, Continuous Delivery and Continuous Deployment. Picture copied from codeproject.com*

Whether you do continuous delivery or continuous deployment is very much an implementation choice. If you do continuous deployment, changes in code will be deployed automatically after the acceptance tests are passed. This may or may not be desirable for your product. With continuous delivery, people can make a choice whether a particular code change is deployed or not (and possibly where exactly it is deployed).

Since the difference between continuous delivery and deployment is small and many people are unaware of the exact difference, the two terms are sometimes used interchangeably.
> 27 votes

# Answer
Continuous delivery and continuous deployment (CD) are more or less the same thing\*. Every time a change is considered 'good to go' (tested/verified) it should be release immediately. You can do this as many times a day as there is work completed.

Continuous integration (CI) only refers to merging code together often to ensure that feature branches don't veer too far away from your main 'master' branch, and also so that you know very quickly if there are any problems with the code from an integration perspective - i.e. Did you break any functionality while changing stuff.

As far as how they relate to each other, CI helps greatly to verify code so that it can be released quickly (CD). You can still achieve CD without CI (and vice versa), but you will find that often it makes life much easier to integrate your code early and often to find problems quicker, which in turn will allow you to fix said problems quicker, and finally deliver your features quicker!

\*Edit: here is an article discussing the difference. https://puppet.com/blog/continuous-delivery-vs-continuous-deployment-what-s-diff Continuous delivery does not always mean actually deploying to production at all times, rather it means constant deployment into a production-like environment, with confidence that those changes could go to production at any time once the business is ready. In practice, most people conflate these terms.
> 4 votes

# Answer
A certain version of a software product must first complete its integration stage before it can be delivered or deployed.

For *continous* delivery/deployment *continuous* integration is a must. Otherwise, if the integration completed events are too far apart to qualify for the "continuous" attribute then so are the possible deliveries/deployments (only a subset of the integrated versions typically qualify for deliveries/deployment).

Update: My answer only underlines the dependency (relationship) between CI and (both) CDs, the terminology is pretty well covered by THelper's answer.

The only comment I'd have to that is about the (overloaded) use of `deployment`. Deployments in non-production environments are a real thing. They can even happen often-enough - done, for example, as part of various testing stages during continuous delivery. But that doesn't make such deployments `continuous deployments`. Continuous deployments refers specifically to deployments in the production environment.
> 3 votes

# Answer
Basically, continuous integration is a part of both continuous delivery and continuous deployment, except that releases happen automatically. You can also think of continuous delivery as a logical next step of continuous integration and it works across all environments. Continuous integration also helps with artifact verification so it can be deployed quicker. Although it is not possible to have continuous deployment without continuous integration and it is so much easier to catch bugs with continuous integration. All of these “continuous things” are ultimately about removing unnecessary actions in development workflow. Most importantly, CI/CD are important from both technical and business perspective. Companies who fail to adopt these DevOps principles risk going the way of the dinosaurs. In today’s fast paced IT environment it is either DevOps or die.
> 1 votes

# Answer
CI (Continuous Integration): The most important reason for why we want to use CI is, that we want to avoid unnecessary complicated conflicts caused by delaying the merge of our work with the work of other team members and therefore also fix issues rather sooner than later, because the more we postpone discovery and resolution the more expensive and hard it is. We do that by frequently merging changes into a common branch (for example the development branch). There is an automated component within Continuous Integration for the repeatable tasks, which is the pipelines that are running, which lint, test, build and possibly e2e test the software, but CI is not a fully automated process in regards to how we deploy to production. Most of the time this stems from the fact that the tests that the programmers write are not trusted enough and do not give enough confidence for whatever reason.

CD (Continuous Delivery): Continuous Delivery is a fully automated process. It can only be a fully automated process if we fully trust the tests that we write. Therefore I’d argue you cannot have Continuous Delivery without E2E testing.The decision whether or not to release the new software version is made manually, but this is just a button click and a business decision not a technical decision, because we have a production ready release after the pipeline passes. CI is a part of Continuous Delivery or in other words CI is a subset of Continuous Delivery. With Continuous Delivery everything is automated except the decision making process whether or not we deploy the release to production.

CD (Continuous Deployment): Same as Continuous Delivery, but we do not have that button, ie. everytime the pipeline on the release branch does pass, it gets pushed to production.

The aim of all three is the same, it is to avoid bugs and ship more frequently via improving collaborative workflows and automation.
> 0 votes

---
Tags: continuous-integration, terminology, continuous-delivery, continuous-deployment
---


# Question
Title: How to get Docker container stdout in Jenkins logs?
(Moved this question from Stack Overflow after getting no traction there; perhaps I'll have more luck here by getting consideration by more DevOps-specific experts)

As part of my Jenkins pipeline, I run a Docker container with Jenkins' `docker.withRun()`.
For the purpose of demonstration, the Docker container entrypoint is a trivial bash script that echoes an environment variable:

```
#! /bin/bash
# runme

if [ -z "$STR" ]
then
  echo 'STR is undefined'
else
  echo "STR is $STR"
fi

```

```
# Dockerfile

FROM debian:10.3 as base

COPY ./runme /usr/local/bin

ENTRYPOINT runme

```

Building and running locally:

```
$ docker build --no-cache -t tmp .
[+] Building 0.3s (7/7) FINISHED
 => [internal] load build definition from Dockerfile                                          0.0s
 => => transferring dockerfile: 109B                                                          0.0s
 => [internal] load .dockerignore                                                             0.0s
 => => transferring context: 2B                                                               0.0s
 => [internal] load metadata for docker.io/library/debian:10.3                                0.0s
 => CACHED [1/2] FROM docker.io/library/debian:10.3                                           0.0s
 => [internal] load build context                                                             0.0s
 => => transferring context: 26B                                                              0.0s
 => [2/2] COPY ./runme /usr/local/bin                                                         0.1s
 => exporting to image                                                                        0.0s
 => => exporting layers                                                                       0.0s
 => => writing image sha256:f5b6fdb3035c12a061c1221f570f126f5a7479e7e3ab07bba0a9694b5b04ab52  0.0s
 => => naming to docker.io/library/tmp                                                        0.0s
$ docker run -t tmp
STR is undefined
$ docker run -e STR=foo -t tmp
STR is foo
$

```

I try to run this same Docker container from a Jenkins pipeline:

```
pipeline {
  agent any

  options { timestamps{} }

  stages {
    stage('docker run') {
      steps {
        script {
          def img = 'tmp'
          def docker_run_args = '-t -e STR=foo'
          def ms = docker.image(img)

          ms.withRun(docker_run_args) {}
        }
      }
    }
  }
}

```

...I can see from the Jenkins job's console logs that the container is running, but its stdout is not included in the console logs:

```
[Pipeline] stage
[Pipeline] { (docker run)
[Pipeline] script
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
09:49:40  + docker run -d -t -e STAGE=foo tmp
[Pipeline] withEnv
[Pipeline] {
[Pipeline] sh
09:49:45  + docker stop e5020f0e5416f34221d3cbc275fa44ccc05f9af03b138e4ec67d47b1d90a2d3a
09:49:45  e5020f0e5416f34221d3cbc275fa44ccc05f9af03b138e4ec67d47b1d90a2d3a
09:49:45  + docker rm -f e5020f0e5416f34221d3cbc275fa44ccc05f9af03b138e4ec67d47b1d90a2d3a
09:49:45  e5020f0e5416f34221d3cbc275fa44ccc05f9af03b138e4ec67d47b1d90a2d3a
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // script
[Pipeline] }
[Pipeline] // stage

```

As you can see from the above, the string "`STR is ...`" is not included in the Jenkins job's console logs, but appears in my terminal's stdout just fine when I run the container myself.
I tried using `docker.run()` instead of `docker.withRun()`, but it didn't change the behavior.

**Is there a way I can get the Docker container's stdout included in the Jenkins job's console logs?**

# Answer
You need to run that docker in interactive mode.

```
docker run --rm -it -e STR=foo tmp

```
* `-i` \- interactive mode
* `--rm` \- to delete the container once finished the process (`tmp`)
> 1 votes

# Answer
You can get the Docker stdout into a variable as follows and then log it at will.

```
script {
  myVar = sh(script: 'docker run -e STR=foo -t tmp', returnStdout: true).trim()
  println "got ${myVar}"
}

```
> 0 votes

---
Tags: docker, jenkins, jenkins-plugins
---


# Question
Title: In salt what causes the WARNING 'version' argument will be ignored for multiple package targets
For some of my `pkg.installed` states I get this warning

> \[INFO \] Executing state pkg.installed for \[install haproxy package\]
>
> \[WARNING \] 'version' argument will be ignored for multiple package targets

I am not specifying a `'version'` argument though.

My state looks like this:

```
"install haproxy package":
  pkg.installed:
    - pkgs:
      - haproxy

```

Why am I getting this warning and how do I get rid of it?

# Answer
This turned out to be a bug in salt version 3006.1. See bug report https://github.com/saltstack/salt/issues/64563.
> 0 votes

---
Tags: salt
---