# Question
Title: Jenkins Pipeline - conditional execution with branch and 1 other parameter (manual)
We are deploying our application using Jenkins pipeline like this -

```
pipeline {
  agent any
  stages {
    stage('Build For Production') {
        when { branch 'development' }
        steps {
             sh './bin/build.sh'
        }
    }
    stage('Build For Production') {
        when { branch 'master' }
        steps {
             sh './bin/copy_needed_auth.sh'
             sh './bin/build.sh'
        }
    }
  }
}

```

When a developer pushes code on bitbucket, The application is deployed automatically. Using branch, we set our deployment strategy.

```
when { branch 'master' }

```

But we need to set a manual chacking for deploying on production (master branch) like - when a developer will merge code in the master branch, he will also set some tag or something like that so that Jenkins pipeline will check branch + other manual logic to deploy in production.

we are doing like this -

```
when {
    branch 'master'
    tag: 'release-*'
}

```

But it's not working. Is there any other strategy to do that?

# Answer
> 1 votes
If I got it right, you could try to tag by using environment variables. Maybe something defined in the environment {} directive within the stage. E. g.

`$PROD = xyz_prod`

Then,

`when { branch 'master' tag: $PROD }`

Not sure why your approach failed, but building tags on a script using variables could do the trick.

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: Run Docker Containers without having a logged in User on Windows
Is it possible to run Docker containers without being logged in as a user on Windows? I have a few containers I am auto-deploying on a server, but I am currently stuck leaving a generic service account logged in so that the containers run and the docker daemon stays running.

My current setup works, but I feel like there must be a better way to deploy/run than to always have an account logged in.

**Setup**

* Windows Server 2012 and 2016
* No docker-desktop because of nested virtualization scenario requirement
* Linux not an option due to business requirements

# Answer
Docker can be set to run as a service, that should bypass the need to log in with a service account:

1. `dockerd.exe --register-service`
2. `net start Docker`
3. `sc config Docker start=auto`

Dockerd.exe
> 1 votes

---
Tags: docker, windows
---


# Question
Title: How to communicate between AWS ECS containers
## Problem

How do ECS containers communicate between each others?

## Situation

I use docker compose ecs integration.
In AWS docker context, `docker compose up` did a deploy correctly.
But reverse proxy doesn't work correctly, maybe because of **communication with containers**.

My `docker-compose.yml`

```
version: '3'
services:
  nginx:
    image: {ACCOUNT}.dkr.ecr.{REGION}.amazonaws.com/{MY_IMAGE}
    ports:
      - "80:80"

  nodeapp:
    image: {ACCOUNT}.dkr.ecr.{REGION}.amazonaws.com/{MY_REACT_IMAGE}
    working_dir: /src
    command: sh -c "yarn install && yarn build && yarn start"
    ports:
      - "3000:3000"

```

MY nginx.conf

```
server {
    listen 80;
    server_name localhost;
    location / {
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_set_header X-NginX-Proxy true;
        proxy_pass http://localhost:3000/;
        # proxy_pass http://nodeapp:3000/; // works only for local
        # proxy_pass http://nodeapp.{MY CLUSTER}.local:3000/; // doesn't work

    }
}

```

## My understandings

1. In the same ECS service, containers can communicate with each other by **localhost**.
2. In the diffrent ECS service, containers can communicate with each other by **service discovery**.

Are these correct? And, how do I do that?

#### 1: The same ECS service

I tried to do that at first. But, `docker compose up`(ecs integration) created different services(for *nginx service* and *application(react) service*. How do I configure that?

#### 2: The different service

In nginx conf, **proxy\_pass http:nodeapp.{MY\_CLUSTER}.local:3000** doesn't work correctly.

---

How do ECS containers communicate between each others?
Let me know the simplest way.
Thanks in advance.

# Answer
Understanding 1 is not correct, localhost is only for internal containers not other can access on localhost, if they share same network name can be used, nodeapp:3000 will work on local.

For different services we have to call them by local IP or a domain name if not in same network.
> 2 votes

---
Tags: docker, docker-compose, containers, fargate, ecs
---


title: Drone is unable to clone GitHub Project

text:
# Question
Title: Drone is unable to clone GitHub Project
I'm trying to set up drone, but when running a build it fails at the clone step. I'm pretty sure this is a DNS issue and that I just configured the runner wrong, but I just can't figure it out.

This is my docker compose file (I have removed some information):

```
  drone:
    image: drone/drone:2.0.4
    container_name: drone
    hostname: drone
    restart: unless-stopped
    depends_on:
      - postgres
    environment:
      - DRONE_GITHUB_SERVER=https://github.com
      - DRONE_GITHUB_CLIENT_ID=
      - DRONE_GITHUB_CLIENT_SECRET=
      - DRONE_RPC_SECRET=
      - DRONE_SERVER_HOST=
      - DRONE_SERVER_PROTO=https
      - DRONE_DATABASE_DRIVER=postgres
      - DRONE_DATABASE_DATASOURCE=postgres://@postgres:5432/drone?sslmode=disable
    networks:
      - traefik-network
    volumes:
      - drone-data:/data

  drone-runner-1:
    image: drone/drone-runner-docker:1.6.3
    container_name: drone-runner-1
    hostname: drone-runner-1
    restart: unless-stopped
    environment:
      - DRONE_RPC_HOST=
      - DRONE_RPC_PROTO=https
      - DRONE_RPC_SECRET=
      - DRONE_RUNNER_CAPACITY=2
      - DRONE_RUNNER_NAME=${HOSTNAME}
      - DRONE_RUNNER_NETWORKS=traefik
    volumes:
      - "/var/run/docker.sock:/var/run/docker.sock:rw"

networks:
  traefik-network:
    external: true
    name: traefik

```

I made sure that I can ping github.com from the traefik network, so that shouldn't be an issue, right? But when I try to ping github from the runner container I get the following output:

```
root@Ubuntu-2004-focal-64-minimal ~ # docker exec -it drone-runner-1 nslookup github.com
Server:         127.0.0.11
Address:        127.0.0.11:53

Non-authoritative answer:
*** Can't find github.com: No answer

Non-authoritative answer:
Name:   github.com
Address: 140.82.121.3

```

Also, here is the output of the clone step:

```
Initialized empty Git repository in /drone/src/.git/
   + git fetch origin +refs/heads/master:
   fatal: unable to access 'https://github.com/[REPO]': Could not resolve host: github.com

```

# Answer
> 1 votes
I guess there is an issue with your overlay network which has been created by the docker command. The host machine able to ping the GitHub means we are blocking somewhere in the docker network. you can do `tcpdump` over the docker bridge to track the blocker.

you can do some changes on your docker-compose.yaml to test it as below as a workaround. Add privileged on the container just access the host DNS.

```
drone:
    image: drone/drone:2.0.4
    privileged: true
    container_name: drone
    hostname: drone

```

---
Tags: docker, dns
---


# Question
Title: How to safe restart Jenkins from OS command line
There is an existing question here on how to safely restart Jenkins:

How to safe restart Jenkins?

However, all the answers are either HTTP(s) options or using the jenkins-cli.jar CLI route.

I want to be able to safely restart the Jenkins environment from a script running on the Master host (a Debian Linux system, the script is running from 'cron' as root)

The Jenkins GUI is using LDAP via the company Active Directory to manage logins, so creating an extra user for this is non trivial (and I don't want to have an extra user open to HTTP Jenkins access if I don't absolutely have to), and using the original default "admin" user isn't possible because Jenkins only allows login from one "security realm".

The scripts are able to uncleanly restart Jenkins (because they can restart the processes), but surely it is possible to signal a clean restart somehow.

One of the answers in the linked question suggests "/etc/init.d/jenkins safeRestart", but that isn't a valid command.

# Answer
> 1 votes
Assuming Jenkins is installed via a package which installs a systemd unit file:

```
systemctl restart jenkins

```

---
Tags: jenkins
---


# Question
Title: Automating deployment of microservices from start to finish
I'm currently part of a team that trains multiple pods of new hires. Each pod creates a client simulated project that involves the following steps:

1. Creating kubernetes namespaces for each pod for deployment
2. Using spring initializer to create a Java project for each microservice (each pod's project has the same dependencies/configs but the only difference is each pod has a different name)
3. Initializing each project in GitLab with repositories for each microservice
4. Setting up webhooks for Jenkins integration (adding a Jenkinsfile, Dockerfile, gradle.properties and build.gradle with the appropriate project K8 namespace)

Since we go through the entire microservice spin-up process manually each time a new pod comes through, we're looking to automate as much of this as possible.

Right now our approach is to create scripts that will automate kubernetes namespace creation and customization for the Dockerfile and build.gradle. However, does anyone have experience working with the GitLab API to automate webhook configuration and Jenkins pipeline config? Any advice is welcome!

# Answer
> 1 votes
> Since we go through the entire microservice spin-up process manually each time a new pod comes through, we're looking to automate as much of this as possible.

This is typically what a *Platform Team* does. Its good to provide a *"Self Serve Portal"* \- e.g. so that developers can just intiate these things themself - whenever they have a need for it.

A project that tries to be an open source "*developer portal*" is Backstage, created by Spotify. Even if you don't choose to use Backstage, it can serve as inspiration. See e.g. Backstage: Spotify's platform developer experience

---
Tags: docker, jenkins, kubernetes, gitlab, microservices
---


# Question
Title: Unable to use AWS ECS docker context in bitbucket pipelines deployment step
I'm trying to add a deployment step to bitbucket pipelines that works successfully when I simulate locally but fails when executing the steps in bitbucket.

I have created a simple docker image that contains docker and compose-cli as per https://raw.githubusercontent.com/docker/compose-cli/main/scripts/install/install\_linux.sh

When I run my "build-env" docker image locally as per bitbuckets debug instructions https://support.atlassian.com/bitbucket-cloud/docs/debug-pipelines-locally-with-docker/ \- everything works as expected and I can connect to the AWS ECS docker context and use the "docker compose" commands

However, when I push the changes to bitbucket and execute the build step in bitbucket pipelines the build fails with "compose" command not found.

I have inspected env vars and double checked the correct docker CLI is installed and ensured the correct image is being pulled.

Either the ecs ccontext creation or connection is failing or the wrong docker binary is being called? Or something else i'm not aware of.

The pipeline yaml

```
image: build-env:latest

definitions:
  steps:
    - step: &deploy
        name: Deploy this build
        deployment: test
          - docker context create ecs --from-env deploy
          - docker context use deploy
          - docker version
          - docker compose ps

```

The dockerfile for the build-env

```
FROM ubuntu:20.04

RUN apt-get update

RUN apt-get install -y python3-pip python3-dev \
  && cd /usr/local/bin \
  && ln -s /usr/bin/python3 python \
  && pip3 install --upgrade pip awscli

RUN apt-get install -y curl docker.io

RUN curl -L https://raw.githubusercontent.com/docker/compose-cli/main/scripts/install/install_linux.sh | sh

CMD ["python3"]

```

The bitbucket output

```
docker context create ecs --from-env deploy<1s

+ docker context create ecs --from-env deploy
Successfully created ecs context "deploy"
docker context use deploy<1s

+ docker context use deploy
deploy

docker version<1s

+ docker version
Cannot connect to the Docker daemon at tcp://localhost:2375. Is the docker daemon running?
Client:
 Cloud integration: 1.0.17
 Version:           20.10.7
 API version:       1.41
 Go version:        go1.13.8
 Git commit:        20.10.7-0ubuntu1~20.04.1
 Built:             Wed Aug  4 22:52:25 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

docker compose ps<1s

+ docker compose ps
**docker: 'compose' is not a docker command.**

See 'docker --help'
Build teardown<1s

```

Very strange that it works locally but fails on bitbuckets servers... Does anyone know where I'm going wrong???

The end goal is to use "docker compose up" within the AWS context to trigger the ecs cluster updates. It works perfectly everywhere except for inside bitbuckets environment.

Thanks in advance....

**Update** I have managed to narrow the issue down. Seems the failure is occurring at "docker context use deploy"

The context is being created and I can inspect and list it, but attempting to use it always fails as seen in the bitbucket pipeline output below.

```
docker context use deploy<1s

+ docker context use deploy
deploy
docker context inspect<1s

+ docker context inspect
[
    {
        "Name": "default",
        "Metadata": {
            "StackOrchestrator": "swarm"
        },
        "Endpoints": {
            "docker": {
                "Host": "tcp://localhost:2375",
                "SkipTLSVerify": false
            }
        },
        "TLSMaterial": {},
        "Storage": {
            "MetadataPath": "\u003cIN MEMORY\u003e",
            "TLSPath": "\u003cIN MEMORY\u003e"
        }
    }
]
docker context ls<1s

+ docker context ls
NAME                TYPE                DESCRIPTION                               DOCKER ENDPOINT        KUBERNETES ENDPOINT   ORCHESTRATOR
default *           moby                Current DOCKER_HOST based configuration   tcp://localhost:2375                         swarm
deploy              ecs                 credentials read from environment

```

# Answer
> 2 votes
I found the answer in this github issue

https://github.com/docker/cli/issues/1809

> if DOCKER\_HOST environment variable is set, we ignore the current context setting.

Bitbucket sets this environment variable to the default context meaning I was unable to swap docker context.

Once I added the command *unset DOCKER\_HOST* the context switch succeeded and docker compose worked as expected.

---
Tags: docker, docker-compose, deployment, continuous-deployment, bitbucket-pipelines
---


# Question
Title: Export AWS RDS mysql Database to S3 Bucket using xtrabackup or any other method
I have an AWS RDS MySQL Database. The size of the Database is around of 1.5 GB. We are using MySQL 5.7.32.

We tried with SQL dump but in dump data is missing in Dump.

We found out that we can use xtrabackup. We are using xtrabackup version 2.4.21 based on MySQL server 5.7.32 Linux (x86\_64).

We used below command

```
sudo ./xtrabackup --backup --user=user --host=project.eu-central-1.rds.amazonaws.com --password=password --databases=database--port=3306

```

Error was like

```
xtrabackup: recognized server arguments:
xtrabackup: recognized client arguments: --backup=1 --user=user --host=project.eu-central-1.rds.amazonaws.com --password=* --databases=database --port=3306
210811 05:31:47  version_check Connecting to MySQL server with DSN 'dbi:mysql:;mysql_read_default_group=xtrabackup;host=project.eu-central-1.rds.amazonaws.com;port=3306' as 'user'  (using password: YES).
210811 05:31:47  version_check Connected to MySQL server
210811 05:31:47  version_check Executing a version check against the server...
210811 05:31:48  version_check Done.
210811 05:31:48 Connecting to MySQL server host: project.eu-central-1.rds.amazonaws.com, user: user, password: set, port: 3306, socket: not set
Using server version 5.7.12-log
./xtrabackup version 2.4.21 based on MySQL server 5.7.32 Linux (x86_64) (revision id: 5988af5)
xtrabackup: uses posix_fadvise().
xtrabackup: Can't change dir to '/rdsdbdata/db/' (Errcode: 2 - No such file or directory)
xtrabackup: cannot my_setwd /rdsdbdata/db/

```

# Answer
RDS already has an 'Export to S3' button. Alternatively you can use AWS Data Migration Service (DMS) to export the data to S3 in real-time.

And as you point out, regular `mydumper` will also work because RDS is a regular MySQL-compatible database.
> 0 votes

# Answer
We need to trace why data missing, we have taken backup many time from rds without any issues. We have more than 80gb data. 2ndly xtrabackup needs access to mysql data directory so can't be used with rds.

Just mysql dump can do the job. Also rds offers S3 export per table in few format like parquet etc
> 0 votes

# Answer
Xtrabackup forum come to the rescue and cleared our doubt that it works only when we have physical access to the Database and suggested to use mydumper

mydumper - GitHub - maxbube/mydumper: Official mydumper project 1

It is working perfectly.
> 0 votes

---
Tags: amazon-web-services, mysql, amazon-rds, backup, disaster-recovery-plans
---


# Question
Title: 404 status code in response from Cloudfront for a single page react application
we have a single page app in ReactJS that is being served by CloudFront but the response status continues to be 404 for any nested pages (i.e pages other than the home page).

How can this be resolved? Attached snapshot

# Answer
> 1 votes
you can catch an error like 404 and redirect it to 200 and your index.html with custom error pages in cloudfront like shown here https://www.youtube.com/watch?v=gBkysF8\_-Es

---
Tags: amazon-web-services, amazon-s3, amazon-cloudfront
---


# Question
Title: How to setup basic auth for Prometheus deployed on K8s cluster using yamls?
How to setup basic auth for Prometheus deployed on K8s cluster using yamls??

I was able to achieve this easily when Prometheus was deployed on a host locally using tar file. But when it is deployed as a pod in K8s cluster, tried almost everything on the internet but no luck.

Any kind of help would be really appreciated!

Thanks!

# Answer
You need to add the required annotations to the ingress as required.For nginx, you can follow this
> 1 votes

---
Tags: kubernetes, prometheus
---


# Question
Title: How prometheus scrapes individual pods actuator endpoints when they are exposed via a single service in kubernetes?
I have created a deployment with 2 replica sets. But the Service exposes the 8080 and the management endpoint using NodePort. How should I set up the pods deployment if Prometheus deployment should get 2 targets instead of one (the k8s service port mapping)

I have a deployment.yaml creating 2 replicas:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: rps-app # Deployment manages the pods with these labels
  name: rps-app-deployment
spec:
  replicas: 2
  selector:
    matchLabels: # This replicaSet manages the pods with these labels
      app: rps-app
  minReadySeconds: 10
  strategy:
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: rps-app
      name: rps-app-deployment
    spec:
      hostname: rps-app
      restartPolicy: Always
      containers:
        - image: edekargaurav88/rps2:1.4
          name: rps-app
          imagePullPolicy: Always
          resources:
            limits:
              cpu: "1"
              memory: 512M
            requests:
              cpu: "0.5"
              memory: 256M
          ports:
            - containerPort: 8080
              #hostPort: 8080
              name: http
            - containerPort: 8091
              #hostPort: 8091
              name: metrics
          env:
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD
              value: postgres
            - name: POSTGRES_URL
              value: jdbc:postgresql://192.168.10.220:5432/postgres
            - name: SPRING_PROFILES_ACTIVE
              value: prod,liquibase
          livenessProbe:
            httpGet:
              path: /actuator/health
              port: 8091
              scheme: HTTP
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 6
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: 8091
              scheme: HTTP
            initialDelaySeconds: 30
            periodSeconds: 10
            timeoutSeconds: 5

```

My Service yaml:

```
apiVersion: v1
kind: Service
metadata:
  name: rps-app
spec:
  allocateLoadBalancerNodePorts: true # default
  selector:
    app: rps-app
  type: NodePort
  ports:
    - protocol: TCP
      port: 8080
      targetPort: 8080
      name: app
    - protocol: TCP
      port: 8091
      targetPort: 8091
      name: health

```

# Answer
> 3 votes
No changes need to be made to your service deployment. As for Prometheus, if you are deploying a Prometheus Operator, A simple ServiceMonitor will work for you. ServiceMonitors select the services to monitor based on selectors and then identify Endpoints to monitor under the Service. Use the following to setup a ServiceMonitor

---
Tags: kubernetes, prometheus
---


# Question
Title: Dynamic environments per client, which is the best approach, if any?
I don't know how to explain this problem better, but I'll try to explain it clearly:

* Where I work our customers are companies, with lots of users
* We offer a SaaS solution for them to manage stuff
* For every customer, there are around 50 right now, we create a vhost in one of our machines, a database in some of our database VMs and we configure the dotenv files for this application, potentially creating more VMs or database machines (nowadays this happens less frequently)
* The SaaS allows us to configure the same application to serve multiple clients by doing a multi-tenant setup with multiple environment files

The problem: The whole process of creating vhost, configuring stuff, etc. is manual. We know how to automate this setup with ansible + terraform by creating application VMs and infrastructure for every customer or creating a HA environment with every application dotenv on a shared application, and we even have the option to migrate this to k8s. However, the most valuable thing to us would be to be able to add, to whichever solution we choose, extra customers based on API calls or some sort of automation.

If you're on k8s world, I think this would be equivalent to add a new ConfigMap to every customer based on some sort of automation, alongside deployments for these customers.

# Answer
> 1 votes
From what I understood is that you want add deploy a configMap and a Deployment based on some API calls every time you need to add a new customer?

Basically you need a CD Pipeline that triggers a deployment based on new customer added. The trigger can be an API Call or a webhook from your software where you add a new customer. It should support sending an API call with details required in those ConfigMaps and Deployments. Now with the details of API Call you'd need to pass those parameters to a Values.yaml of Helm Chart which then goes and deploys to your selected cluster.

If you don't have a CD, you can use Azure Devops or ArgoCD. Both of these support trigger based on API Calls. Then setup the pipeline with connection to your cluster. In Azure Devops, you'd have to create a service connection and in ArgoCD register your cluster. Also, Helm Chart is the easiest way to deploy k8s manifests to it'd be preferable if you did it via that. ArgoCD would limit you to k8s only. You can go with Azure Devops for support for all. I'd have said Jenkins but managing that is a pain in the back. If that is not an issue you can try that too.

# Answer
> 0 votes
Look into a workflow engine like Argo Workflows.

I have seen similar setups. Consider all the steps required to onboard a new client as a workflow.

A workflow engine would then read this particular set of steps (a workflow-template) and performs the tasks as required (a run/workflow).

Since you mentioned going into k8s you might look into this or a non-k8s workflow engine as required. While the workflows itself might run on k8s containers, the steps can be for any environment.

I used Argo Workflows to setup the templates. A dedicated setup for the Devops/SRE teams should fit your use case perfectly. You can also automate other step-wise processes and deploy them too.

Advantages:-

* Argo Workflows provides a UI/CLI approach to trigger workflows. Additionally it has a Events/Webhooks support for triggering workflows as well.
* Support for multiple artifact-repositories, data sources to read/write your client configurations.
* RBAC/SSO support which can help you permission your workflows and who can run them.
* System where you can control the execution steps/environment and the user just has to provide the inputs to your workflows and have permissions to run the same. This reduced a lot of manual intervention and troubleshooting due to steps going wrong during each try and saved us a lot of time.
* Logs are easily available along with the outputs of the workflows as artifacts.
* Workflows can be versioned.The workflows are yaml files which can be versioned and released, hence maintained as code.
* Workflows can be deployed using the CD tools of your choice.(ArgoCD).

Disadvantages:-

* Maybe an overkill for just one workflow.Try deploying multiple templates.

---
Tags: kubernetes, ansible, terraform, automation, distributed-systems
---


# Question
Title: Get a list of remote VPCs on AWS using Terraform?
I'd like to get a list of all remote VPCs from AWS with terraform. I've read the documentation and found two data sources:

1. **AWS\_DEFAULT\_VPC**
2. **AWS\_VPC**

The first one will give you the default VPC and the second one accepts a VPC id and toss out other information such as subnets...

How can I get a list of all VPCs existing in ceratin AWS region (US-West-2 for instance) ?

# Answer
> 1 votes
You'll have to use the output of the list in a for loop to get the subnet ids for\_each, but it would look something like this:

```
data "aws_vpcs" "foo" {}

output "vpcs" {
  value = data.aws_vpcs.foo.ids
}

```

# Answer
> 0 votes
You can use Terraform data sources.

Terraform official link : https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/vpc

##### Example:

```
variable "vpc_id" {}

data "aws_vpc" "selected" {
  id = var.vpc_id
}

resource "aws_subnet" "example" {
  vpc_id            = data.aws_vpc.selected.id
  availability_zone = "us-west-2a"
  cidr_block        = cidrsubnet(data.aws_vpc.selected.cidr_block, 4, 1)
}

```

---
Tags: amazon-web-services, terraform, aws-vpc
---


# Question
Title: Do I have to include gitlab variables in my terraform code in order for my credentials to be recognized?
I am trying to set up a pipeline, which is supposed to deploy terraform code into an aws account. The code works fine locally, and I have setup and imported the state bucket and dynamodb table which enables state locking.

However, when I run the code in a gitlab ci, I get the following error message:

> Error: error configuring S3 Backend: no valid credential sources for S3 Backend found. │ │ Please see https://www.terraform.io/docs/language/settings/backends/s3.html │ for more information about providing credentials. │ │ Error: NoCredentialProviders: no valid providers in chain. Deprecated. │ For verbose messaging see aws.Config.CredentialsChainVerboseErrors

I export the aws credentials as variables in the before\_script like so:

```
before_script:
  - rm -rf .terraform
  - terraform --version
  - export AWS_ACCESS_KEY_ID
  - export AWS_ROLE_ARN
  - export AWS_DEFAULT_REGION
  - export AWS_ROLE_ARN

```

Is there another step that I am missing, or do I maybe have to include the credential variables in the terraform code?

# Answer
The answer to this question is no. I merely forgot to export the AWS\_SECRET\_ACCESS\_KEY variable in the before\_script section.
> 2 votes

# Answer
In our terraform pipeline we use the aws cli command to assume a role and get the credential variables, the EC2 instance for the pipeline worker is using a profile which can assume the role.

```
creds=$(aws sts assume-role --role-arn ${ROLE_ARN} --role-session-name tf-pipeline | jq -r '.Credentials')
export AWS_ACCESS_KEY_ID=$(echo $creds | jq -r '.AccessKeyId')
export AWS_SECRET_ACCESS_KEY=$(echo $creds | jq -r '.SecretAccessKey')
export AWS_SESSION_TOKEN=$(echo $creds | jq -r '.SessionToken')

```
> 0 votes

# Answer
in the case of using GitLab ci ENV var, if u are testing in branches other than main/master, make sure your variables are not **PROTECTED**
> 0 votes

---
Tags: continuous-integration, terraform, gitlab, gitlab-ci-runner, continuous-delivery
---


# Question
Title: Making aws_lb_listener default_action dynamic
I use Terraform a mostly uniform group of servers - EC2, S3 etc. This works reasonably well, except we need to have rather different actions for the load balancers depending on the specific group, and I can't figure out how to do it.

To be more specific, this is what the common setup looks like

```
  resource "aws_alb_listener" "listener_https" {
     load_balancer_arn = aws_alb.alb.arn
     port = "443"
     protocol = "HTTPS"
     ssl_policy = "ELBSecurityPolicy-FS-1-2-Res-2019-08"
     certificate_arn = data.aws_acm_certificate.main.arn
     default_action {
        target_group_arn = aws_alb_target_group.group.arn
        type = "forward"
     }
 }

```

but in a couple of cases I need the default action to be a redirect, rather than a forward.

I thought using a `dynamic "default_action"` block could help, but I can't seem to find a way to set both the name of the fields and the value (which could mean it's not the correct approach, in which case I'd love to know what the correct one is).

I guess my question could also be generalized to "(how) can I define all the values of a given resource as a single value and then apply them" (equivalent to what I'd do in python by preparing a dict with the values I need and then creating the object with `**dict`)

# Answer
I found a solution - not sure it's the best but it seems to work well enough for my case.

The solution is in three parts: first I added a variable to provide the default that works for most of my cases:

```
variable "default_lb_action" {
  default = {
      "production": {"type": "forward"}
      "staging": {"type": "forward"}
  }
}

```

Then I added a new `local` like this:

```
 locals {
   lb_default_action = {
      type = var.default_lb_action[local.environment].type
      target_group_arn = var.default_lb_action[local.environment].type == "forward" ? lookup(var.default_lb_action[local.environment], "target_group_arn", aws_alb_target_group.group.arn) : null
      redirect = try(var.default_lb_action[local.environment].redirect, [])
    }
  }

```

Please note that I've left the option to explicitly declare a target\_group\_arn, which may or may not be a requirement, and that redirect defaults to an empty list.

The last step is the load balancer listener definition itself, which now looks like this:

```
resource "aws_alb_listener" "listener_https" {
  load_balancer_arn = aws_alb.alb.arn
  port = "443"
  protocol = "HTTPS"
  ssl_policy = "ELBSecurityPolicy-FS-1-2-Res-2019-08"
  certificate_arn = data.aws_acm_certificate.main.arn
  default_action {
      type = local.lb_default_action.type
      target_group_arn = local.lb_default_action.target_group_arn
      dynamic "redirect" {
          for_each = local.lb_default_action.redirect
          iterator = redir
          content {
            host = redir.value["host"]
            path = redir.value["path"]
            port = redir.value["port"]
            protocol = redir.value["protocol"]
            query = redir.value["query"]
            status_code = redir.value["status_code"]
          }
      }
  }
}

```

The important bits here are that the `target_group_arn` can be null (if the `type` is not `"forward"`) and that the `redirect` block may be empty as well.

With the above, I can set a var like this and get the result I needed:

```
default_lb_action = {
  production = {
    type = "redirect"
    redirect = [
      {
        host = "www.example.com"
        path = "/#{path}"
        port = "#{port}"
        protocol = "HTTPS"
        query = "#{query}"
        status_code = "HTTP_301"
      }]
  }
  staging = {"type": "forward"}
}

```

Hope this helps somebody else!
> 0 votes

---
Tags: terraform
---


# Question
Title: How to monitor ELK using metricbeat on kubernetes (v8 compatible and simple solution)
For probably more than a year there's been a deprecation warning on xpack monitoring in elasticsearch, logstash, kibana etc. saying the built-in monitoring is going to disappear.

Since I don't like running deprecated setups I figured it was time to fix it now, should be easy, right? Not really, what is 2 lines of config in ELK 7.x instead requires deploying an extra metricbeat instance. Personally I really don't like this approach since it's a hassle and seems like a waste of resources to deploy another container just to send off some monitoring metrics. But elastic has chosen the path so we can just follow.

I read a bit in some official channels and there seems to be some official solutions in the works that require custom kubernetes ELK CRDs etc. This is not what I want, I just want a simple solution to deploy anywhere as a sidecar.

# Answer
> 2 votes
I propose the following solution although it will require 1 extra metricbeat running per monitored entity, it's simple, not optimized. The approach will also work for other non-kubernetes contexts like docker-compose with a little bit of tweaking.

* Deploy sidecar (extra container per pod) to monitor something (in this example, it's logstash but doing same for elasticsearch or kibana should be trivial, just replacing the monitoring module and possbibly metrics port.
* Set xpack.monitoring.enabled: false in logstash.yml (or elastic/kibana)
* Have metricbeat fetch metrics via localhost on port 9600
* Configure metricbeat with mounted configmaps defined in yaml
* Example assumes no TLS/RBAC to elasticsearch but if you have tls modifying the below should hopefully be straight forward

```
# Configmaps for metricbeat conf files
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-modules
  namespace: logstash
data:
  logstash-xpack.yml: |-
    - module: logstash
      xpack.enabled: true # This is what turns on xpack monitoring metric export
      period: 10s
      hosts: ["localhost:9600"] # Connect to logstash metrics api using localhost inside pod
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: metricbeat-config
  namespace: logstash
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true
      reload.period: 10s
    metricbeat.modules: # Disable system metrics
    - module: system
      enabled: false

    output.elasticsearch:
      hosts: ["http://elasticsearch:9200"] # You probably need to modify elasticsearch URL
    setup.ilm.enabled: true

---
...
Here goes the rest of your yaml for logstash/elasticsearch/kibana
And then the sidecar container:
...
      containers:
      - name: metricbeat-sidecar
        image: docker.elastic.co/beats/metricbeat:7.14.0
        volumeMounts:
        - name: config
          mountPath: /usr/share/metricbeat/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
...
# and then finally in the volumes section:
...
      volumes:
      - name: config
        configMap:
          defaultMode: 0640
          name: metricbeat-config
      - name: modules
        configMap:
          defaultMode: 0640
          name: metricbeat-modules

```

Main sources:

Update: Similarly for kibana:

```
# Configmaps for metricbeat conf files
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-metricbeat-modules
  namespace: elk
data:
  xpack.yml: |-
    - module: kibana
      xpack.enabled: true # This is what turns on reporting
      period: 10s
      hosts: ["localhost:5601"]
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: kibana-metricbeat-config
  namespace: elk
data:
  metricbeat.yml: |-
    metricbeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: true
      reload.period: 10s
    metricbeat.modules: # Disable system metrics
    - module: system
      enabled: false

    output.elasticsearch:
      hosts: ["http://elasticsearch:9200"]
    setup.ilm.enabled: true
...
container:
  - name: metricbeat-sidecar
      image: docker.elastic.co/beats/metricbeat:7.14.0
      volumeMounts:
        - name: config
          mountPath: /usr/share/metricbeat/metricbeat.yml
          readOnly: true
          subPath: metricbeat.yml
        - name: modules
          mountPath: /usr/share/metricbeat/modules.d
          readOnly: true
...
      volumes:
    - name: config
      configMap:
        defaultMode: 0640
        name: kibana-metricbeat-config
    - name: modules
      configMap:
        defaultMode: 0640
        name: kibana-metricbeat-modules

```

---
Tags: kubernetes, monitoring, logstash, elasticsearch, elk
---


title: Validating release artifact provinence

text:
# Question
Title: Validating release artifact provinence
While this question could be posed in a vendor agnostic fashion - I'll gear it around Github and Azure to give it some context.

I am planning an open-source project, in a sensitive business domain (banking). As such, I feel it would be important for users who download and install it to be able to verify the provenance of the binary - through some kind of independently tracked "chain of custody" - tracing it back to the actual source code label that the CI process consumed.

How is this currently achieved? If at all? Are there any known mechanisms/practices for this?

What I imagine, is a publicly visible landing page in Azure DevOps, hosted by Microsoft (unable to tampered by anyone, not even the project owner) - which maps the installation binary hash to the source control label thumbprint. This would allow the user to trust that the binary in their possession is guaranteed to be the product of the code they are auditing.

# Answer
You have 2 questions here that likely need two different answers. Luckily, both are solved problems!

1. **How to prove the provenance of code?**. - this is done with what's called Code Signing. Azure DevOps has tools for this - KeyVault and Pipelines can both do it, I suspect there are others.
2. **How to prove that a package contains the code it's supposed to contain?**. The best and most predictable way to do this is likely with an MD5 checksum.
> 1 votes

---
Tags: release-management
---


# Question
Title: What do skipped, rescued, and ignored results mean in ANSIBLE playbook execution?
What do skipped, rescued, and ignored results mean in an Ansible playbook execution? What would be a possible scenario where these results would be different from 0?

```
PLAY RECAP *******************************************************************
localhost: ok=3 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

```

# Answer
> 4 votes
For example, the play below

```
- hosts: localhost
  tasks:

    - name: Task will be SKIPPED
      debug:
        msg: You will never see this message.
      when: false

    - name: Failed command will be IGNORED
      command: /usr/bin/false
      ignore_errors: true

    - name: Failed command will be RESCUED
      block:
        - command: /usr/bin/false
      rescue:
        - debug:
            msg: "{{ ansible_failed_result }}"

```

gives

```
PLAY RECAP *******************************************************************
localhost: ok=2 changed=1 unreachable=0 failed=0 skipped=1 rescued=1 ignored=1

```

For details see

Briefly, an error is ignored when you set `ignore_errors: true`. This can be set on multiple levels. See Playbook Keywords. If you use `block/rescue` you can actively handle the consequences of the error.

---
Tags: ansible, automation, networking
---


# Question
Title: Docker Swarm stop working with redeploy service, require system reboot
I got a Docker Swarm with 3 nodes, and contain around 40-50 stack on it, now its random stop working to redeploy my new changes its just say in `new` mode, and really don't know what going one here.

```
ID             NAME                            IMAGE                                                               NODE                                DESIRED STATE   CURRENT STATE                ERROR
                        PORTS
1wyc57lkgebc   translation-manager_web.1       {images}                                       Running         New 6 minutes ago

6xthicsqmkst    \_ translation-manager_web.1   {images}   docker-swarm-02   Shutdown        Shutdown 6 minutes ago

```

Normal its taking like 20-30sec to redeploy new changes to my Docker Swarm setup.

If I'm running the `service update` command its just stay like this and never ending

```
translation-manager_web
overall progress: 0 out of 1 tasks
1/1: new       [=====>                                             ]

```

Its look like its random witch service / container there hit this error.

I got 50% free disk space, lots of memory and free CPU that's the first I think its wrong.

Swarm node info:

```
ID                            HOSTNAME                                    STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
6n19d0ryoon2zy4s295s928pp     staging-docker-swarm-01           Ready     Active                          20.10.8
8se5pzi4znzde6qtlmu0yt0og     staging-docker-swarm-02           Ready     Active                          20.10.8
p7srx7ojo50rmdxtk6ppriomd     staging-docker-swarm-03           Ready     Active                          20.10.8
yf8ubbk650rrkmrwkyk3no8zb     staging-docker-swarm-04           Ready     Active                          20.10.8
f3acixkd61015h0kqgp61ilav *   staging-docker-swarm-manager-01   Ready     Active         Leader           20.10.8

```

**-- Update: 18/08/2021**

I have added one more worker to my swarm, and 3 new manager to it, and its its still the same issue.

I have 62 Stacks with 109 services running in my swarm right now if the numbers helping.

# Answer
I found out of what happening here. I hit the limit of the IP amount in the ingress network. Default ingress network for Docker Swarm when you create it is 254 IP's its landing in the IP range /24

So what I do its change this IP range to /16 and now get 65k IP's

Every published port you are using multiply with replicates take a single IP so in my case when I got 109 services with avg. replicates its taking 218 IP's out of the default IP range.

So after this change everything are working again in my Docker Swarm setup.

Here is the command to change your ingress network from /24 to /16

```
docker network rm ingress
docker network create \
  --driver overlay \
  --ingress \
  --subnet=10.11.0.0/16 \
  --gateway=10.11.0.2 \
  --opt com.docker.network.driver.mtu=1200 \
  ingress

```
> 0 votes

---
Tags: docker, containers, docker-swarm
---


title: K8s with docker hub

text:
# Question
Title: K8s with docker hub
I have configured jenkins with github.com. Jenkins notices if commit occurs and then jenkins make the docker image and upload the new image to docker hub. How can I make k8s listens to docker hub i.e. k8s notices the change in docker hub and then creates a new deployment. What tool should be used?

# Answer
> 0 votes
You can't on the fly, but you can build a system around Docker hub if you setup a webhooks. I have build a free software for Docker Swarm auto-deployment https://github.com/pnk-sh/docker-octopus-stack

If you use software there "pull" every min. then you get in troubles with Docker Hub when you hit 5000 pull's a day.

Its required you buy a pro-account, free account will only allow you around 250 pulls a day.

# Answer
> 0 votes
You could try to run Jenkins agents in the k8s cluster. Then you could write a script that is capable of update the tag of a deployment. We use gitlab in k8s and in conjunction with helm it is possible to update and deploy the image.

---
Tags: jenkins, kubernetes, dockerhub
---


title: How do I fail a gitlab pipeline job?

text:
# Question
Title: How do I fail a gitlab pipeline job?
I am currently using dependency checker to scan my applications via gitlab

This is how my pipeline looks like.

```
dependency_scanning:
  stage: security_scan
  script:
    # Run dependency check on all modules
    - mvn dependency-check:aggregate -B "-DdbUser=$NVD_DB_USER" "-DdbPassword=$NVD_DB_PASSWORD" "-DconnectionString=$NVD_DB_CONSTRING"
  artifacts:
    paths:
      - ./target/dependency-check-report.html
  only:
    - master

```

And this is my output.

```
Executing "step_script" stage of the job script
00:32
$ mvn dependency-check:aggregate -B "-DdbUser=$NVD_DB_USER" "-DdbPassword=$NVD_DB_PASSWORD" "-DconnectionString=$NVD_DB_CONSTRING"
3304 [INFO] Scanning for projects...
4813 [INFO]
4813 [INFO] --------------------< com.example:pipeline-tester >---------------------
4814 [INFO] Building demo 0.0.1-SNAPSHOT
4814 [INFO] --------------------------------[ jar ]---------------------------------
6018 [INFO]
6019 [INFO] --- dependency-check-maven:6.2.2:aggregate (default-cli) @ pipeline-tester ---
12434 [INFO] Checking for updates
12494 [INFO] Skipping NVD check since last check was within 4 hours.
12497 [INFO] Skipping RetireJS update since last update was within 24 hours.
12499 [INFO] Check for updates complete (64 ms)
12577 [INFO]
Dependency-Check is an open source tool performing a best effort analysis of 3rd party dependencies; false positives and false negatives may exist in the analysis performed by the tool. Use of the tool and the reporting provided constitutes acceptance for use in an AS IS condition, and there are NO warranties, implied or otherwise, with regard to the analysis or its use. Any use of the tool and the reporting provided is at the user’s risk. In no event shall the copyright holder or OWASP be held liable for any damages whatsoever arising out of or in connection with the use of this tool, the analysis performed, or the resulting report.
   About ODC: https://jeremylong.github.io/DependencyCheck/general/internals.html
   False Positives: https://jeremylong.github.io/DependencyCheck/general/suppression.html
 Sponsor: https://github.com/sponsors/jeremylong
12579 [INFO] Analysis Started
14108 [INFO] Finished Archive Analyzer (1 seconds)
14113 [INFO] Finished File Name Analyzer (0 seconds)
16312 [INFO] Finished Jar Analyzer (2 seconds)
16401 [INFO] Finished Dependency Merging Analyzer (0 seconds)
16405 [INFO] Finished Version Filter Analyzer (0 seconds)
16811 [INFO] Finished Hint Analyzer (0 seconds)
22504 [INFO] Created CPE Index (5 seconds)
28005 [INFO] Finished CPE Analyzer (11 seconds)
28010 [INFO] Finished False Positive Analyzer (0 seconds)
28609 [INFO] Finished NVD CVE Analyzer (0 seconds)
29109 [INFO] Finished Sonatype OSS Index Analyzer (0 seconds)
29522 [INFO] Finished Vulnerability Suppression Analyzer (0 seconds)
29606 [INFO] Finished Dependency Bundling Analyzer (0 seconds)
29908 [INFO] Analysis Complete (17 seconds)
30206 [INFO] Writing report to: /builds/cdit/pipeline-tester/target/dependency-check-report.html
31205 [WARNING]
One or more dependencies were identified with known vulnerabilities in demo:
tomcat-embed-core-9.0.46.jar (pkg:maven/org.apache.tomcat.embed/tomcat-embed-core@9.0.46, cpe:2.3:a:apache:tomcat:9.0.46:*:*:*:*:*:*:*, cpe:2.3:a:apache_software_foundation:tomcat:9.0.46:*:*:*:*:*:*:*, cpe:2.3:a:apache_tomcat:apache_tomcat:9.0.46:*:*:*:*:*:*:*) : CVE-2021-33037
See the dependency-check report for more details.
31513 [INFO] ------------------------------------------------------------------------
31514 [INFO] BUILD SUCCESS
31514 [INFO] ------------------------------------------------------------------------
31514 [INFO] Total time:  28.305 s
31514 [INFO] Finished at: 2021-08-12T07:20:28Z
31514 [INFO] ------------------------------------------------------------------------
Saving cache for successful job
00:01
Creating cache master...
.m2/repository: found 3366 matching files and directories
Archive is up to date!
Created cache
Uploading artifacts for successful job
00:00
Uploading artifacts...
./target/dependency-check-report.html: found 1 matching files and directories
Uploading artifacts as "archive" to coordinator... ok  id=335682 responseStatus=201 Created token=Bf8rx3gp
Cleaning up file based variables
00:00
Job succeeded

```

Is it possible to make my pipeline fail even though it is success, in this case when vulnerabilities are present?

# Answer
All you need to do is to configure the mvn dependency plugin to fail on found vulnerabilities.

You can do that with the `failBuildOnCVSS` config flag. E.g. with severity 8 from docs

```
<project>
    ...
    <build>
        ...
        <plugins>
            ...
            <plugin>
              <groupId>org.owasp</groupId>
              <artifactId>dependency-check-maven</artifactId>
              <version>6.2.2</version>
              <configuration>
                  <failBuildOnCVSS>8</failBuildOnCVSS>
              </configuration>
              <executions>
                  <execution>
                      <goals>
                          <goal>check</goal>
                      </goals>
                  </execution>
              </executions>
            </plugin>
            ...
        </plugins>
        ...
    </build>
    ...
</project>

```

By that the mvn job will fail if vulnerabilities with severity of 8 or higher are found and by that the pipeline will fail too.
> 2 votes

---
Tags: gitlab, cicd, pipeline
---


# Question
Title: Where does PGBouncer sit on a Kubernetes Cluster
I'm wanting to deploy some sort of connection pooling within my applications across multiple nodes. Python naturally provides some pooling with the SQLAlchemy implementation; but obviously this probably does not scale as well as 1 central pool.

As far as I see it PGBouncer can Sit within each pod, it can be running as its own pod, or it can be running as a daemonset which ties a PGBouncer instance to be running for each kube node.

How would you / how have you deployed PGBouncer in a similar environment?

This question is equally relevant for twemproxy.

# Answer
> 2 votes
PGBouncer is a connection pooler. So, one instance of PGBouncer should be enough in between the apps and postgres.

---
Tags: kubernetes, postgresql
---


# Question
Title: Gitlab build from multiple repositories - git submodules
I have gitlab repo ZERO (project) which serves as a code base for multiple websites (repo ONE, TWO, THREE..), these are also gitlab repositories. I am using gitlab CI/CD options, I want to be able to build docker image consisting of code base repo ZERO + specific code in specific website repo, i.e. ONE. Base repo ZERO and project repositories ONE, TWO, THREE will be pushed to regularily.

To me it seems usage of `git submodules` might be appropriate, but when doing internet research on this matter, `submodules` is getting hate for being complicated and slow. Are submodules straight solution for this usecase, or is there alternative solution which might suit me better?

# Answer
git submodules couple them tightly but using cicd pipelines you can have option to have different versions based on different branches of each repo. If you have kubernetes cluster ahead you can use argocd to do this for you even graphically.
> 0 votes

---
Tags: docker, git, gitlab
---


# Question
Title: Jenkins Pipeline job does not use Squid Proxy
I have configured a Squid Proxy and it seems to be working when I add below Env variables to the /home/user/.bashrc file to a client machine. Note: SSL Bump is configured, Self Signed Cert is added to the /etc/ssl/certs and the cert is being trusted.

```
export http_proxy=http://172.12.1.100:3128/
export https_proxy=http://172.12.1.100:3128/

```

For example when I run

```
user@squid-client:$ echo $http_proxy
http://172.12.1.100:3128/

user@squid-client:/tmp$ wget https://google.com

--2021-08-17 09:42:14--  https://google.com/
Connecting to 172.12.1.100:3128... connected.
Proxy request sent, awaiting response... 301 Moved Permanently
Location: https://www.google.com/ [following]
--2021-08-17 09:42:14--  https://www.google.com/
Connecting to 172.12.1.100:3128... connected.
Proxy request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.1’

index.html.1                       [ <=>                                                 ]  12.68K  --.-KB/s    in 0s

2021-08-17 09:42:15 (42.6 MB/s) - ‘index.html.1’ saved [12986]

```

Now I connected this Linux instance to my Jenkins Master as a slave (Unix Slave via SSH) and run a simple pipeline script which basically has the same wget command.

But the squid access log nor the Jenkins Console output does not show anything related to the Squid Proxy.

Here is the part of the console output from that Jenkins Job.

```
+ wget google.com
--2021-08-17 09:26:39--  http://google.com/
Resolving google.com (google.com)... 142.251.40.78, 2607:f8b0:4023:1002::64, 2607:f8b0:4023:1002::8a, ...
Connecting to google.com (google.com)|142.251.40.78|:80... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: http://www.google.com/ [following]
--2021-08-17 09:26:39--  http://www.google.com/
Resolving www.google.com (www.google.com)... 142.250.68.164, 2607:f8b0:4023:1006::6a, 2607:f8b0:4023:1006::93, ...
Connecting to www.google.com (www.google.com)|142.250.68.164|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: unspecified [text/html]
Saving to: ‘index.html.4’

```

I executed the "printenv" command via the pipeline and it also does not show $http\_proxy variable.

I want my jobs to download packages, files via this Squid Proxy Server. Any help would be really appreciated.

Note 2: I tried to source /etc/profile (after adding the export http\_proxy lines to that file) and seems source is not supported in whatever the shell used by the Jenkins slave service.

Note 3: Both The Squid Proxy and the Client (Jenkins Slave )runs on Ubuntu 18.04.

# Answer
My `sh """ commands """` step was not using the $http\_proxy environment variable mentioned in the .bashrc or /etc/profile.

What I did was using the Jenkins `Prepare an environment for the run` option and adding my values.

Now a simple pipeline uses the Squid proxy settings without any issue. A wget connects to the Squid proxy without any issue.
> 0 votes

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: How can I detect if an Ansible playbook is executed by ansible-pull rather than ansible-playbook?
I am looking for a `when` condition which is fullfilled when the playbook is executed by ansible-pull.

# Answer
I found two approaches to test if a playbook is applied via `ansible-pull`:

1. Check for `ansible_connection == "local"`. However this doesnt work in all places as local connections might also used regularily by your playbook.
2. Pass an extra variable to the invocation: `ansible-pull --extra-vars "ansible_pull=true"`
> 1 votes

---
Tags: ansible
---


# Question
Title: Is docker absolutely necessary to deploy applications with Linode's Kubernetes?
Is learning and using docker required to deploy applications with Linode's Kubernetes Engine? (LKE)

As fun as docker sounds, I'd like to avoid having to learn that additional complexity if possible.

I have numerous CentOS and Ubuntu servers and custom (php) applications running at Linode, and I'm interested in transferring one of them to LKE.

# Answer
> 4 votes
> I have numerous CentOS and Ubuntu servers and custom (php) applications running at Linode, and I'm interested in transferring one of them to LKE.

If this is your case, the first thing you need to do, to be able to run your application on Kubernetes, is to actually run your application in a **docker container**.

When you have got your application running in a **docker container**, then can you learn about Kubernetes Deployment and Ingress *configurations*. You will run the same **docker container** in your Kubernetes cluster as you can run locally using `docker`.

The first step to get your application running in a docker container is to learn about how `Dockerfiles` are used and written. When your have built a docker container, that is what you are going to use in Kubernetes as well.

# Answer
> 4 votes
**TL;DR:** you don't need it, but you also can't just copy-paste VM to Kubernetes

**TL;DR 2:** learn to walk before you start running

1. Learn what is process and how namespaces work on Linux
2. Learn how to assemble a filesystem image (simple `mount` and `chroot` might help with building the idea)
3. Imagine filesystem layers as git commits (Docker / filesystem `diff`)
4. Learn how to publish and pull a filesystem for Kubernetes
5. Learn basic deployment with simple service on Kubernetes
6. Learn advanced networking and filesystem concepts on Kubernetes

---

You are mistaking "complexity" with "building steps". A "kubernetes engine" how cloud vendors call it nowadays is nothing more than pure Kubernetes + some vendor specific CRUD that uses Kubernetes API (in similar way you'd use them with kubectl).

Now if we have Kubernetes, it's necessary to understand how stuff are actually deployed in there. Kubernetes is an orchestrator for Linux processes running in namespaces (a.k.a. lightweight "vm", though people dislike the "vm" term). Process needs to appear from somewhere, here comes into place a filesystem.

Filesystem is provided to Kubernetes by a so called "image" and that can be created by LXC or Docker or anything else if Kubernetes supports it / has a driver for it. The specs for the format are open (thus even the recent switch from docker to containerd as process/container runner).

And here is where Docker or any other filesystem creating tool (Nix) comes to place. You either learn it or you can create the filesystem even from scratch (based on the containerd specs, if you don't appreciate your free time). Once the filesystem image is created, you publish it somewhere, such as Docker image registry (some vendors call it "container registry", though you don't really store *containers* there) and Kubernetes pulls the filesystem image from it to its local registry (storage).

### Alternatively!

You don't necessarily *need* to even touch Docker nor build a filesystem *ever* if you can find an already available image on Docker Hub or other public registries. Once you have such an image, just use a Kubernetes `ConfigMap` to replace a file in such a filesystem e.g. default `nginx.conf` with your custom one and it'll run just fine. Same for Python, Java, Go, etc. If you can find an image that's usable and you have your project properly packaged, there's no need to even touch Docker. If something breaks in the filesystem though, it's what you might eventually need to reproduce an issue locally.

For you that'd be using e.g. PHP + Apache Docker image, mounting your Apache config files and perhaps even `php.ini` in one `ConfigMap`, mounting the project files as another `ConfigMap` to `/some/php/location/www` and perhaps even adding a volume from the cloud vendor if you want to utilize preserving the content as files instead only in DB (filesystem clears after each container is killed/removed).

Example:

1. Create kubernetes `Deployment` config with `php:<version>-apache` image:tag.
2. Use `ConfigMap` as a file
3. Check how Linode works with Kubernetes PVC with CSI driver
4. Mount your project as tarball or from linode volume

### But!

By using a public image make sure it's not out of date and you check for vulnerabilities yourself if you don't have some system/company process set up already, otherwise you'll be just using an old version of a filesystem anyone can download and try attacking locally i.e. prepare for and then just either ddos you without much of an effort or just hijack your container and do whatever depending on the available binaries in the filesystem, set user permissions and allowed network connectivity to and from the container.

# Answer
> 1 votes
**Kubernetes** itself can run with other containerization technologies than Docker.

Perusing the official **Linode** documentation and FAQ https://www.linode.com/docs/guides/kubernetes-reference/ , they seem uncomitted to the question. In the "Kubernetes" FAQ they mention correctly that Kubernetes can run with several containerization technologies but that Docker is the most common. I did not find a clear, "first class citizen" documentation from them where they describe how to configure another solution. So I assume that at least the managed Kubernetes solution from Linode does only offer Docker as container backend.

That out of the way, you are asking this not for technical reasons, but specifically because you want to avoid the overhead of learning Docker technology just to get one service into a Kubernetes environment.

The **answer** to this is very clear: for all means and purposes, it makes no sense to go to a non-Docker container system with Kubernetes to *avoid learning effort.* Docker is by *far* the most widely used and documented alternative with many tutorials, FAQs, best practices, examples and so on.

The basic Docker usage is very simple. Writing a Dockerfile is so easy that it is more or less trivial. All the difficulty is in finding the correct commands to build and run your application and listing them in a text file with trivial syntax - and *that* would be the same effort for any and all container solutions.

The difficulties of Docker arise when you go to *run* your containers on a professional level. And this is exactly what a managed Kubernetes solution does for you. So again, you will not come in contact with any Docker specific "difficult" tasks (like making sure your containers are always up, getting the storage, networking, CPU quotas and so on to work well and so on). This is all handled by Kubernetes for you, and you will not notice whether it's Docker or another container backend driving your cluster.

---
Tags: docker, kubernetes
---


# Question
Title: Is there a way to push tarred repository to dockerhub without loading on local
I have a tarred docker repository created by using docker save.

I want to push this tar to a registry, say dockerhub. Is there a way we can do that without loading on local docker?

# Answer
> 1 votes
Actually this is not possible as `docker push` command will look for a local image or a repository.
You will have to load your image first from tar file, the push it to your registry.

```
docker load --input repo.tar
docker image tag myrepo registry-host:5000/myrepo/myrepo:v1
docker image push registry-host:5000/myrepo/myrepo:v1

```

---
Tags: docker, docker-registry, dockerhub
---


# Question
Title: How do we get notified when PagerDuty services are disabled?
We have a small team, and we're just getting started with PagerDuty. We have a very noisy service that has a lot of testing goign on around it. At times, we've had to turn the service off to keep pagers from blowing up, but then forget to turn it back on.

Is there any way to get notified when a service has been disabled?

# Answer
3-4 years experience with PD here.

PagerDuty will not monitor itself in the way you want, it's made to push notifications, calls, alerts and etc. As mentioned in the comments you should use maintenance windows, disabling a service means you know what you are doing, that's not something I have ever done except for 1-2 testing purposes for the last 4 years. If you are forgetting to enable the service.. then there is another problem that is not technical related, you should not be forgetting that step! I would suggest you don't disable the service, use maintenance windows, if the service is noisy, why put it in PagerDuty, monitor it when it's fully down, or use some other rule to do this. What I would do:

1. Evaluate what is a critical alert.
2. Should you be notified via pager duty for every single thing?
3. Probably set more dynamic alerts, that the alert will be triggered only in specific hours(work hours). Link to PD documentation
4. Try not to disable the Service, use maintenance windows.
5. Decide what kind of escalation policies do you need/want.

EDIT: Never tested this, but PagerDuty has an API that possibly could be used to get a status of a service, but something has to listen/run that API GET request and send it to another PagerDuty Service, that can be used to monitor what you wanted, kinda. API Service Documentation

EDIT 2: I tested the API, if you use that method, you can use something else to monitor for xx amount of time if the service is disabled, by getting this data out of the request: `"status": "disabled"`. For example any Monitor that can send a GET request and get this data, can be set to "If `status: disabled` for 1 hour, send PagerDuty alert". And you can have a separate PagerDuty Service for example that listens for this monitoring.
> 2 votes

---
Tags: pagerduty
---


title: Creating a Kanban Board in Azure DevOps

text:
# Question
Title: Creating a Kanban Board in Azure DevOps
We are using Azure Boards as a Project Management Tool where we capture customer requirements and then track all the work\_items by creating a user\_stories and splitting them into multiple tasks and so on.. We are using Agile Process and it works fine. However, we have some support based projects where we need a KANBAN Board. How do we create a KANBAN Board in Azure?

# Answer
> 0 votes
I would suggest reading the official documentation(quickstart) from Microsoft, as there are Prerequisites and you may need to add a new team since you are using Agile =\> Link

---
Tags: azure-devops, azure
---


# Question
Title: How to assign static IP to Kubernetes cluster?
Is there any way to assign static IP to Kubernetes cluster, in which any user can able to access its application over static IP only.

for an example, I have set up a cluster over 192.168.0.131 (master), 192.168.0.132(worker1) and 192.168.0.133(worker 2). One dotnet application is running on nodeport 32000, which is accessible over all the cluster as expected.

Now I have to assign one IP which communicate to this cluster and user can get access on that. Also is there any way that we can use this IP for other applications as well which can be differentiate based on different ports not nodeport.

FYI, I am using kubernetes cluster over bare metal.

# Answer
> 1 votes
The use case you described is one that is covered by `ingress-controller`. Ingress controller will listen to an static IP which is mapped to kubernetes service of type `LoadBalancer` and will then forward the traffic to the pods as required.

LoadBalancer Services are used to create services that listen to traffic from outside the cluster.

One you assign a static IP in your network, you can then configure the ingress-controller to listen to the same IP.

Examples of ingress-controllers include nginx-ingress,Traefik etc.

Nginx atleast listens to both TCP/UDP protocols and just need port to k8s service mapping to forward traffic to the proper pod.

If you use Helm,it also provides the configuration to give it the static IP it should listen to. (It maps and creates the LoadBalancer svc object as part of its installation if used).

If not using helm you can create the objects on your own.

MetalLB and considerations on bare metal

---
Tags: kubernetes, linux, ubuntu
---


# Question
Title: Gitlab CI: change pipeline definition dynamically
I need to pass info from stage A to stage B so I can change stage B dynamically.

Details:

* I have a stage (A) where I validate a terraform plan and check if it's going to destroy resources.
* I have another stage (B), which applies the terraform plan, which I want to be triggered manually if stage A detected that resources are going to be destroyed.
* When nothing is being destroyed the terraform apply stage (B) should run automatically

How can I pass this information from stage A to B?

Making an artifact in A and using "rules:exist" in stage B is not possible, issue

And if I understand the docs correctly it's not possible to use a "dotenv" var in the "rules:if" section of stage B.

It seems like Gitlab doesn't want you to change the pipeline definition dynamically. Maybe there is a practical solution that I'm not aware of to solve this usecase?

# Answer
> 3 votes
The reason you can't use `dotenv` variables in the `rules` clause(s) is that `rules` are evaluated before the pipeline even starts executing.

If you have a `rules` clause like

```
rules:
  - if: $CI_COMMIT_TAG === 'v1.0.0'
  when: never
  - when: always

```

Gitlab CI will check the following things:

1. Is the pipeline a `Tag Pipeline` (from the `CI_PIPELINE_SOURCE` variable)? If no, `CI_COMMIT_TAG` won't exist, so the clause doesn't exist, and `when:always` is the only rule. Add the job to the pipeline.
2. If the pipeline is a tag pipeline, is the tag === 'v1.0.0'? Then use `when:never`. Don't add the job to the pipeline.
3. Tag pipeline and the tag !== 'v1.0.0', use `when:never` and add the job to the pipeline.

All these things happen before the pipeline is in the `created` state, so therefore it cannot depend on a variable that is created once the pipeline starts running.

Knowing this, the way I would solve the problem is to mark the job in Stage B as a manual job like it already is, but when you want to run it automatically, call the Gitlab Jobs API from the Stage A job to run the Stage B job.

This will use a couple of API Calls:

1. First, we need to get the jobs for this particular Pipeline:

```
curl --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/pipelines/$CI_PIPELINE_IID/jobs?scope=manual"

```

This will return all the `manual` jobs from the pipeline. If you have more than one, you can retrieve the correct one by looking at the value for `stage` and `name` from the response. Here's an example from the docs:

```
[
  {
    "commit": {
      "author_email": "admin@example.com",
      "author_name": "Administrator",
      "created_at": "2015-12-24T16:51:14.000+01:00",
      "id": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd",
      "message": "Test the CI integration.",
      "short_id": "0ff3ae19",
      "title": "Test the CI integration."
    },
    "coverage": null,
    "allow_failure": false,
    "created_at": "2015-12-24T15:51:21.727Z",
    "started_at": "2015-12-24T17:54:24.729Z",
    "finished_at": "2015-12-24T17:54:24.921Z",
    "duration": 0.192,
    "queued_duration": 0.023,
    "artifacts_expire_at": "2016-01-23T17:54:24.921Z",
    "tag_list": [
      "docker runner", "ubuntu18"
    ],
    "id": 6,
    "name": "rspec:other",
    "pipeline": {
      "id": 6,
      "project_id": 1,
      "ref": "main",
      "sha": "0ff3ae198f8601a285adcf5c0fff204ee6fba5fd",
      "status": "pending"
    },
    "ref": "main",
    "artifacts": [],
    "runner": null,
    "stage": "test",
    "status": "failed",
    "tag": false,
    "web_url": "https://example.com/foo/bar/-/jobs/6",
    "user": {
      "id": 1,
      "name": "Administrator",
      "username": "root",
      "state": "active",
      "avatar_url": "http://www.gravatar.com/avatar/e64c7d89f26bd1972efa854d13d7dd61?s=80&d=identicon",
      "web_url": "http://gitlab.dev/root",
      "created_at": "2015-12-21T13:14:24.077Z",
      "bio": null,
      "location": null,
      "public_email": "",
      "skype": "",
      "linkedin": "",
      "twitter": "",
      "website_url": "",
      "organization": ""
    }
  }
]

```

Once we have the Job ID, we can call the Play Job API:

```
curl --request POST --header "PRIVATE-TOKEN: <your_access_token>" "https://gitlab.example.com/api/v4/projects/$CI_PROJECT_ID/jobs/$ID_OF_STAGE_B_JOB/play"

```

Note: we don't want to use the Predefined Variable `$CI_JOB_ID` for the Play Job API since that will be the ID of the Stage A job, not the Stage B job.

Here's the Pipeline Jobs API, and here's the Play Job API for more information and query parameters.

---
Tags: terraform, gitlab, pipeline
---


# Question
Title: New Runners are locked and the checkbox to unlock them is disabled
I'm running the latest version of on-prem GitLab (version 14.1.0-ee). The last few new GitLab runners I've added show up as locked. I open the runners config in the WebUI and I see that the "Lock to current projects" checkbox for locking them is disabled, but also checked-on, like this:

How do I enable (unlock) these runners?

I see if I assign them to a specific project, then they'll unlock. But if I then remove them from those projects, they lock again.

# Answer
> 2 votes
This ended up being a usability bug and GitLab has made a change to avoid future confusion on this point. The relevant ticket for this is here: https://gitlab.com/gitlab-org/gitlab/-/issues/337041

---
Tags: gitlab, gitlab-ci-runner
---


# Question
Title: Read disk of distroless container in Kubernetes for debugging
I am using a helm chart, which deploys containers based on Distroless. This means, I cannot open a shell on these containers, like I normally would for debugging, since Distroless containers seem to have no shell command.

```
$ kubectl exec --stdin --tty $POD -n $NAMESPACE -- /bin/sh
error: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "ded18296c4c0008c29a165ac761f73e3bbabcb068936b4b6efd2bcd686aa8930": OCI runtime exec failed: exec failed: container_linux.go:380: starting container process caused: exec: "/bin/sh": stat /bin/sh: no such file or directory: unknown

```

I also tried using the `kubectl debug` command, but get the following error:

```
$ kubectl debug -it $POD -n $NAMESPACE --image alpine
error: ephemeral containers are disabled for this cluster (error from server: "the server could not find the requested resource")

```

This is on an Azure Kubernetes cluster, and I could not find an option to enable "ephemeral containers".

For debugging, I would like to read a log file that is written by the container, but not exposed to standard output. How can I do this in Kubernetes?

# Answer
> 1 votes
> which deploys containers based on Distroless

Using Distroless is a good practice! This typically means that you don't have more than the absolute minimum in your container to run your application - it is typically just your application and some necessary libraries. This also means that there are no extra tools and debugging possibilities within the container - which is good in a production environment for security and maintainability reasons.

> This means, I cannot open a shell on these containers, like I normally would for debugging

To be able to use debugging or other tools in a container based on Distroless, you can build the same container with e.g. Ubuntu as base image. To be able to do this, you typically need the `Dockerfile`, so that you can change how the container is built.

---
Tags: kubernetes, azure-kubernetes-service
---


# Question
Title: What challenges did you face once you gave devs access to Jenkins jobs?
I am a DevOps engineer at a medium-sized company. Our team is super busy and we have a lot on our plate. Recently we decided to see where we can improve our productivity. One of our main responsibilities is to provide dev teams with different services they require (usually they open a ticket, sending a slack msg, etc…). As part of our efforts to improve productivity, we recently decided to assemble the most common popular “services” provided to devs and automate them using Jenkins. In addition, we allow devs to access Jenkin’s UI to spin up all kinds of jobs.

On one hand, it does allow devs to act on their own, on the other hand, I get a lot of backfires from different personas...

Devs

* Why did this job fail?
* Devs send me messages on slack that they got lost between all the Jenkins jobs, and they don't know which Jenkins Job is relevant for them. "I forgot the identifier of the machine I created"
* Devs make mistakes and deploy to the wrong environment.
* Why did my dev environment get terminated?
* "Update me if you modify parameters"
* etc...

Team lead

* Let’s add a manual approval for these jobs
* Make sure we have an audit log for every dev operation
* Make sure we devs are not spinning to many machines
* The developer team leader asked BI about these Jenkins Jobs because it became critical for developers' day-to-day.
* etc…

Jenkins seems pretty limited to allow these capabilities and I am afraid to create a tech debt and put too much pressure on Jenkins :)

Do you have any suggestions? How do you manage it within your company?

# Answer
> In addition, we allow devs to access Jenkin’s UI to spin up all kinds of jobs

This would be the first thing that should be changed. The pipelines should only be started automatically instead of manual.

DevOps is a culture and implies that all repetitive work should be automated.

> Why did this job fail?

You should ask them whether they read the log. If the issue in the pipeline is NOT related to their code then the creator of the pipeline should investigate it and make a solid CI.

> Devs send me messages on slack that they got lost between all the Jenkins jobs, and they don't know which Jenkins Job is relevant for them. "I forgot the identifier of the machine I created"

This indicates that the devs see unrelated pipelines. If they create microservices then there would be a dedicated pipeline. If the pipeline is broken then one knows that the issue is related to their code. I had a lot of discussions with devs back in the day. Most of the time they said that it felt superfluous to create another repository for their code, but when I referred to the 12factor app and demonstrated the effect of running a pipeline for every app, they were convinced.

> Devs make mistakes and deploy to the wrong environment.

The one that created the pipelines decides what permissions the user will get. A best practice is to apply the least privilege. In terms of CI this means that the developer will only be capable of reading the log. Once the log is read one should be capable of solving the issue. Sometimes, devs do not run their code locally on their laptop first.

> Update me if you modify parameters

Once the pipeline have been implemented, you should be aware that you should NOT introduce breaking changes. If this is inevitable then you should communicate it well, but you should prevent it at all costs to prevent that the devs will decide not to use the pipelines anymore. It is better to spend some more time to create a solid pipeline, before let the devs use it.

> Make sure we devs are not spinning to many machines

This should be automated as well. You could decided to stop the instances automatically if they are running for more than half an hour. You could introduce this and inform the devs and teamlead about it.

> Make sure we have an audit log for every dev operation

Prevent that the devs login to Jenkins. The pipeline should run automatically and the devs should only be capable of reading the log, see also `Devs make mistakes and deploy to the wrong environment`

> Jenkins seems pretty limited to allow these capabilities and I am afraid to create a tech debt and put too much pressure on Jenkins :) Do you have any suggestions? How do you manage it within your company?

Gitlab with a gitlab-ci.yml for each microservice, Gitlab templates, helm and runners on k8s.
> 1 votes

---
Tags: jenkins, rundeck
---


# Question
Title: Jenkins pipeline retry option triggers two builds at a time, one in execution state other in pending state for a single click
For example I have clicked on retry option at build number 2 in Jenkins pipeline view, where as build number 3 gets start and build number 4 also gets start and awaits at pending state.

Why the build number 4 is getting started parallelly and how to avoid this one.

# Answer
> 1 votes
Have you tried unticking the "allow concurrent builds" option?

# Answer
> 0 votes
@shree, I have also seen this scenario in localhost, later I have found that there is an issue with the google chrome browser, somehow it is not compatible with that, please try it with another Firefox browser and check.

---
Tags: jenkins-pipeline
---


# Question
Title: Restart container if its application hangs
I have a critical container application which task consist of polling sensors and taking consequent action on some external hardware.

If the application of the container crashes it's fine. `restart: always` will take care of this.

However if the application hangs, possibly blocked on IO or another unexpected event, this is where things can go wrong as the external hardware will get stuck in it's state possibly indefinitely.

I'd like my container application to periodically communicate with something and if it stops doing so, the container gets restarted.

Does this kind of mechanism exists?

I'm running the container with Docker.

# Answer
by using `docker stack` you can achieve a health check. you can add the below lines in `docker-compose.yaml`.

```
version: '3.1'

services:
  web:
    image: docker-flask
    ports:
      - '5000:5000'
    healthcheck:
      test: curl --fail -s http://localhost:5000/ || exit 1
      interval: 1m30s
      timeout: 10s
      retries: 3

```

Reference: https://howchoo.com/devops/how-to-add-a-health-check-to-your-docker-container
> 2 votes

# Answer
Create health endpoint inside application and call it from outside and if no response restart docker
> 1 votes

---
Tags: docker, containers
---


title: Postgresql in DevOps skills

text:
# Question
Title: Postgresql in DevOps skills
What are the basic skills and requirements a DevOps candidate should have when working with PostgreSQL? How to monitor it, deploy, etc.?

# Answer
In my opinion an Operations candidate (DevOps is a culture, not a role) should be capable to perform at least the following actions:

* deploy postgres
* backup postgres
* restore postgres
* run some psql commands like select and some innerjoins
* knows what the effect on memory usage would be if a `select *` is issued
* No manual labour, only automation. Deploy the postgres using a pipeline like gitlab
* infrastructure as code
* monitor it, disk usage, memory, cpu
* the difference between a relational and non-relational database
* see also this Q&A
* there are tools like Flyway that are capable of applying version control to databases. This could facilitate to run it in a CI/CD.
> 6 votes

# Answer
To add to 030's answer, and as a devops engineer who has had to do this a few times, learning how to set up replication and failover in an automated fashion is very important.

If you're learning, start by setting up replication by hand on some test servers

https://www.postgresql.org/docs/13/runtime-config-replication.html

Then craft a method for automating the configuration. There are multiple ways to set up replication (they can run in parallel) and restoring a replicated database is pretty simple, but testing it out once and understanding how the file system is set up, what kinds of permissions users need to share data and what kinds of ACLs you need open between nodes to facilitate data transfer is very important.

You'd also want to craft a plan for what to do for promoting a new read/write master in case your primary node goes down.
> 2 votes

---
Tags: postgresql, sql
---


# Question
Title: HAProxy Not Working with Kubernetes NodePort for Backend (Bare Metal)
I have a host running HAProxy already. It's been up and running since before I did anything with Kubernetes. It works flawlessly as a reverse proxy and SSL terminator for various web things in various Docker containers on various other host machines.

Now I have a Kubernetes cluster up and running across some of those other machines. I've created the NodePort Service that exposes port 30080 on each worker node, as follows:

```
apiVersion: v1
kind: Service
metadata:
  name: snginx
  labels:
    app: nginx
spec:
  type: NodePort
  externalTrafficPolicy: Local    #Cluster or Local
  selector:
    app: nginx
  ports:
  - protocol: TCP
    port: 8080
    targetPort: 80
    nodePort: 30080

```

From the machine running HAProxy (which is not part of the cluster), I can curl the NodePort successfully ( curl 10.0.0.20:30080 ), and I get "Welcome to nginx!..." However, if I set that NodePort as a backend in HAProxy, I get a 503 "No server is available", and HAProxy traffic log says:

```
localhost haproxy[18489]: [redactedIP]:49399 [30/Aug/2021:19:24:00.764] http-in~ load/load 66/0/-1/-1/67 503 212 - - SC-- 1/1/0/0/3 0/0 "GET / HTTP/1.1"

```

The haproxy admin log says:

```
Aug 30 20:07:13 localhost haproxy[18839]: Server load/load is DOWN, reason: Layer4 connection problem, info: "General socket error (Permission denied)"

```

However, I've disabled the firewall with

```
sudo systemctl disable --now firewalld

```

and verified the status is not running. Also, SELinux was disabled when I installed the cluster. Also, I can ping 10.0.0.20 just fine.

"load" is the hostname I'm using for testing load balancing (i.e. load.mydomain.com).

Also, if I use static NAT on my physical router to route directly to that NodePort, from outside the building, it works as expected.

What gives? What's the difference between the proxied request and curl?

Thank you.

# Answer
> 1 votes
I figured it out, and SELinux is the difference. That is, SELinux on the HAProxy host (not a cluster node):

"SELinux only allows the web server to make outbound connections to a limited set of ports"

That is, you can't make an outbound http request to any port in the NodePort range (30000-32768) without opening that port on the "client", which is the HAProxy server in this case.

```
sudo semanage port --add --type http_port_t --proto tcp 30080

```

More specifically, I guess, the security context that HAProxy runs under is different than curl.

---
Tags: kubernetes, load-balance, kubectl
---


# Question
Title: A better way to not use dockerhub - permanent config within docker?
I've searched the documentation for docker if this is possible and come up blank, but thought I'd pose the question here before shelving this. Is there a way to just tell Docker to use something other than dockerhub as a first port of call?

We're using Artifactory's virtual repository to merge local and remote repositories into one convenient output, but my main gripe is having to say `docker pull docker-virtual-stage.artifactory.mycompany.com/centos` instead of just `docker pull centos`.

Jfrogs method to get this endpoint seems over-complex \ not easily scaled up.

TL;DR: Is there a better way to not use dockerhub while remaining user friendly?

# Answer
How about setting the registry mirror, and then blocking blocking DNS to docker.io just to make sure it goes to your mirror?

```
#/etc/docker/daemon.json

{
"registry-mirrors": ["https://docker-virtual-stage.artifactory.mycompany.com"]
}

```

As this guy suggested: https://stackoverflow.com/a/64158584/10664600
> 1 votes

---
Tags: docker, artifactory, dockerhub
---


# Question
Title: Failed to start windows service with Start Parameter in golang
I've tried to create a executable as a windows service using golang based on example provided in https://github.com/golang/sys/tree/master/windows/svc/example

I intended to create auto start service with executable specific command line argument 'init'

So whenever service get started executable should perform init operation. Following go code snippet is looks like this.

based on the documentation,

1. I added command line argument along with binarypath as mentioned in this config https://pkg.go.dev/golang.org/x/sys/windows/svc/mgr#Config
2. I added commandline argument in CreateService method. https://pkg.go.dev/golang.org/x/sys/windows/svc/mgr#Mgr.CreateService
3. Tried with Execute args. https://pkg.go.dev/golang.org/x/sys/windows/svc/mgr#Service.Start

But start parameter is not properly added in windows service and executable is nor running as expected.

In created windows service I'm able to see init is added along with executable name, but it is not added in start parameter as well as whenever service is started it is expected to write some log in file, that is also not happening.

Can anyone let me know if this configurations are valid ? If not suggest your thoughts. Thanks.

# Answer
The following is supposed to be a comment, but I do not posses the permission to do so, thus writing it as an answer.

source says 'CreateService installs new service name on the system. The service will be executed by running exepath binary. Use config c to specify service parameters. **Any args will be passed as command-line arguments when the service is started; these arguments are distinct from the arguments passed to Service.Start or via the "Start parameters" field in the service's Properties dialog box.**'
> 0 votes

---
Tags: windows, go
---


# Question
Title: How to create network connection into remote docker stack?
I am struggling to set this up correctly. I am trying to run an app with `docker stack deploy` on a remote machine. The following app works, when I deploy it with `docker swarm deploy` on my windows notebook. But when I run it on the linux cluster. I cannot connect to it. If can help me to identify the missing piece here? When I run the app locally, I get some output from `curl localhost:8000`, however, not at the remote machine. There I get `curl: (7) Failed to connect to localhost port 8000: Connection refused`. I also do not understand why I get a different result (app accessible, not accessible) on local and remote computer, i.e. what configuration is the disrupting element.

I tried pruning all the networks and start again from scratch.

First, I connect to a remote server via ssh:

```
ssh -qy -L 8000:localhost:8000

```

Here, I start a web application with `docker stack`, with this simple `yaml` file `testapp.yml`:

```
    version: '3'
    services:
       db:
         image: mysql:5.7
         volumes:
           - db_data:/var/lib/mysql
         restart: always
         environment:
           MYSQL_ROOT_PASSWORD: somewordpress
           MYSQL_DATABASE: wordpress
           MYSQL_USER: wordpress
           MYSQL_PASSWORD: wordpress

       wordpress:
         depends_on:
           - db
         image: wordpress:latest
         volumes:
           - web_data:/var/www/html
         ports:
           - "8000:80"
         restart: always
         environment:
           WORDPRESS_DB_HOST: db:3306
           WORDPRESS_DB_USER: wordpress
           WORDPRESS_DB_PASSWORD: wordpress
           WORDPRESS_DB_NAME: wordpress

    volumes:
        db_data: {}
        web_data: {}

```

And deploy this with

```
docker swarm deploy -c testapp.yml testapp

```

Now, I get the following output with `sudo docker stack services testapp`:

```
ID             NAME                MODE         REPLICAS   IMAGE              PORTS
93rj1ialspj2   testapp_db          replicated   1/1        mysql:5.7
nl18a7mr4auk   testapp_wordpress   replicated   1/1        wordpress:latest   *:8000->80/tcp

```

However, when I navigate to `localhost:8000`there is

> This site can’t be reached. localhost refused to connect.

And `curl localhost:8000` (on the both systems local and remote) returns

> curl: (7) Failed to connect to localhost port 8000: Connection refused

Running docker 20.10.9 remotely and 20.10.8 locally.

# Answer
> 2 votes
First of all, thanks for the drawing so I understand what you are trying.

The next step is, when you are working with Docker Swarm all your stacks/services can hit the external networks like Redis, SQL and S3 storage so that is nothing you need to take care of.

When you want traffic inside your cluster you need a proxy in the front, I personally are using NGINX as proxy layer and then you need to create a upstream with a stack of IP's to your Docker Swarm Manager nodes, never sending it to your workers.

Docker Swarm working the way you are sending traffic to a single port number eg. 9000 and this port 9000 is binding to your stack/services on port 80 or what you are needed. Every single replicate will now running on port 9000 for externel traffic from the managers and into your cluster.

Besure you are deploying your stacks only for worker nodes except if its controller software you needed, else you can get a unstable cluster based on my own expires.

If your Redis and SQL running in the cluster (i hope not) but if you want it, you can setup a network for thoes stacks and the add the network internel into your orther stacks, but i will recommend to keep Redis, SQL and somthing like that on the own servers

So the following up on my description here is you need the following.

* HTTP / HTTPS Proxy to route traffic into a custom port eg. 9000 for your Docker Swarm Managers
* Expose a port eg. 9000 to your service port eg. 80
* Setup connection inside the service to use the external Redis and SQL

Thats it, hope i have explain it good enough else you can just ask :)

**Expose sample - Docker**

`docker run -p 9000:6379 –name redis_cont -d redis`

allow port 9000 on your host to route into the redis\_cont on port 6379

**Expose sample - docker-compose.yml**

```
version: "3.7"
services:

 redis:
    image: "redis:alpine"
    ports:
     - "9000:6379"

```

Deploy a redis stack into your Docker Swarm and allow port 9000 on your host(Docker Swarm Cluster) to route into the Redis stack service on port 6379

---
Tags: docker, docker-compose
---


# Question
Title: Configuration for Cloudflare endpoint using SSL to Kubernetes (load balancer & flask service)
Here is a summary of what I'm trying to do:

I can get my flask app to respond without HTTPS/SSL. But am receiving a '503 Service Unavailable' when I am trying to use SSL. My understanding is since the load balancer has the Cloudflare origin certificate installed, it will SSL terminate so the flask app does not need modifications for SSL.

Here is my Kubernetes yaml for the load balancer and flask app service:

```
apiVersion: v1
kind: Service
metadata:
  name: flask-service
  labels:
    run: flask-service
  annotations:
    service.beta.kubernetes.io/do-loadbalancer-tls-ports: "443"
    service.beta.kubernetes.io/do-loadbalancer-certificate-id: "68e1d971-1a7c-40d2-8f33-aed797a9535d"
spec:
  selector:
    app: flask
  ports:
  - name: tcp
    protocol: "TCP"
    port: 5000
    targetPort: 5000
  - name: https
    protocol: TCP
    port: 443
    targetPort: 5000
  type: LoadBalancer
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: flask
spec:
  selector:
    matchLabels:
      app: flask
  replicas: 1
  template:
    metadata:
      labels:
        app: flask
    spec:
      containers:
      - name: flask
        imagePullPolicy: Always
        image: gcr.io/xxxxyyyyyy/flask:staging
        ports:
        - containerPort: 5000
        resources:
          limits:
            memory: 8000Mi
          requests:
            memory: 4000Mi

```

I would appreciate any suggestions as I have been looking at this problem for the past few days with no luck.

# Answer
In the company I'm working for, we are just have a lot of same issues with Cloudflare and Docker Swarm. We are hitting a lots of 503 Error and we are running both node, php and python application.

What we discovered is when you are create a single connection between Cloudflare and your Cluster its some buffering and keep a live settings you did not have control over, and this 503 Error keep coming.

We discovered this with Cloudflare service, HA Proxy, Load Balance from Digital ocean and with default NGINX settings.

We dive deep into it and try to search on what happen, first of all cloudflare give us around 1-300ms more ping time for every single request, the secound is NGINX enable proxy buffering and this 2 things to getter is not working pretty well I think.

So what I can recommend and its working for us is setup a NGINX server as proxy layer, disable proxy buffering and then use Letsencrypt (certbot) to generate your own SSL - don't dot it on production, create a test setup first.

That's my best shoot to tell you what's going on here, its sound a lot like the issue we fight with in months before we discovered this issue about the connection drops between client and the Docker Swarm service containers.

If you test my answer of, late me know about its help you.
> 0 votes

---
Tags: kubernetes, ssl, flask, cloudflare
---


# Question
Title: Does Immutable Servers imply no source code updates on the servers?
So once an Immutable Server is in place, nothing should change at all. Or is it only configuration.

Are source code updates acceptable on Immutable Servers?

# Answer
No.

Immutable is exactly what it means, immutable, no change on configuration or code running or system library or whatever, if a change has to be made, create a new image and deploy it, never change it while running.

Source code updates are the least thing to change on a running server, this should not happen on a running server, immutable or not.

You should create a versioned release of your code to deploy on servers, once you have this level of fixed versioning you can step ahead to versioning your server image which include this code release.

Any change, being a code patch or a system patch or a configuration change should trigger a new release to be deployed.
> 7 votes

# Answer
That's not true. Immutable servers simply means that the servers cannot be changed but the can be deleted and created from the updated source code.
> 0 votes

---
Tags: continuous-integration, continuous-delivery, continuous-deployment, immutable-servers
---