# Question
Title: Devops CICD run ReactJS application while testing with jest + puppeteer
**Scenario**

I'm working on getting my end-to-end tests to work in a GitLab pipeline. Currently all the tests run locally but I'm having an issue getting them to run using GitLab's continuous integration feature.

When running the tests locally puppeteer will use a headless version of chrome for end-to-end tests but the ReactJS' CRA server ***has*** to be running for this to work. To get the CRA server running locally I use `npm start` in one terminal and once it is fully running I can run `npm test` in a different terminal. During testing Puppeteer uses headless Chromium to run e2e tests on the site.

**The Issue**

`npm start` starts my CRA server but that process doesn't stop as I'm not sure how to stop it unless I manually quit it. This hangs docker and blocks me from ever starting my tests.

**Question**

How can I start my application with `npm start` and once it starts then run `npm run test:e2e` ? Ultimately I'd like to quit CRA once the tests are completed.

**Current Pipeline yml**

```
image: node:10.19.0
stages:
  - test
  - build
  - deploy

cache:
  # untracked: true
  key:
    files:
      - package-lock.json
  paths:
    - node_modules
#test_async:
#  script:
#    - npm install
#    - node ./specs/start.js ./specs/async.spec.js

before_script:
  - apt-get update && apt-get install -y lftp gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
  - if [ ! -d "node_modules" ]; then npm install; fi

test:
  stage: test
  script:
    - npm start
    - npm run test:e2e

build:
  stage: build
  script:
    - npm run build

# deploy-to-staging:
#   stage: deploy
#   script:
#     - lftp -e "open ftp.mediajackagency.com; user $MJA_FTP_USER $MJA_FTP_PASS; mirror --reverse --verbose build/ /var/www/domains/projects/my-project/build/; bye"
#   environment:
#     name: staging
#     url: http://dev.mediajackagency.com/projects/my-project/build
#   when: manual
#   only:
#     - dev

```

# Answer
I switched to using a production build for testing by use of a local webserver. I leveraging the `[jest-puppeteer][1]` library which allows you to start a webserver with your tests.

**jest-puppeteer.config.js**

```
/**
 * find all flags at the following site @jkr
 * https://jestjs.io/docs/en/cli
 */

module.exports = {
    launch: {
        devtools: true, // allows for use of 'debugger;' in tests
        // executablePath: '/usr/bin/chromium-browser',
        headless: true,
        defaultViewport: {
            width: 1024,
            height: 768,
            //isMobile: true,
            //hasTouch: true,
        },
        ignoreDefaultArgs: ['--disable-extensions'],
        args: [
            '--enable-font-antialiasing',
            '--font-render-hinting=medium',
            '--disable-gpu',
            '--disable-dev-shm-usage',
            '--disable-setuid-sandbox',
            '--no-first-run',
            '--no-sandbox', // GOOD
            '--no-zygote',
            '--single-process', // GOOD

            "--renderer",
            "--no-service-autorun",
            "--no-experiments",
            "--no-default-browser-check",
            "--disable-extensions",
        ]
    },
    server: { // launches webserver just for tests @jkr
        command: 'npm run test:webserver'
        // cd build && ../node_modules/local-web-server/bin/cli.js --port 3000 --spa index.html
    },
    browser: 'chromium',
    browserContext: 'default'
};

```

**.gitlab-ci.yml**

```
image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest

cache:
  # untracked: true
  key: my-project-name
  # key: ${CI_COMMIT_REF_SLUG} # per branch
  # key:
  #   files:
  #     - package-lock.json # only update cache when this file changes (not working) @jkr
  paths:
    - .npm/
    - node_modules
    - build

stages:
  - prepare # can install ci-deps here ? @jkr
  - test # uses test:build specifically @jkr
  - build
  - deploy

# before_install:

before_script:
  - npm ci --cache .npm --prefer-offline #
#   # - apt-get update && apt-get install -y lftp gconf-service libasound2 libatk1.0-0 libatk-bridge2.0-0 libc6 libcairo2 libcups2 libdbus-1-3 libexpat1 libfontconfig1 libgcc1 libgconf-2-4 libgdk-pixbuf2.0-0 libglib2.0-0 libgtk-3-0 libnspr4 libpango-1.0-0 libpangocairo-1.0-0 libstdc++6 libx11-6 libx11-xcb1 libxcb1 libxcomposite1 libxcursor1 libxdamage1 libxext6 libxfixes3 libxi6 libxrandr2 libxrender1 libxss1 libxtst6 ca-certificates fonts-liberation libappindicator1 libnss3 lsb-release xdg-utils wget
#   - if [ ! -d "node_modules" ]; then set NODE_ENV=dev && npm install; fi # make sure dev dependencies are installed todo: 'if' needs to be verified working @jkr
#   # - npm install
#   # - if [ ! -d "build" ]; then npm run build; fi

prepare: # could this job be in a before_script for test ? @jkr
  stage: prepare
  script:
    - npm install
    - npm run build:test
  # cache:
  #   paths:
  #     - node_modules
  #   policy: push

test:
  stage: test
  script:
    - npm run test:ci-deps
    - npm run test:e2e # runs puppeteer tests @jkr

build:
  stage: build
  script:
    - npm run build
  # artifacts:
  #   paths:
  #     - build

deploy-ftp:
  stage: deploy
  script:
    - lftp -e "open ftp.my.domain.com; user $MJA_FTP_USER $MJA_FTP_PASS; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye"
  environment:
    name: staging
    url: http://my.domain.com/my-project/build
  when: manual
  only:
    - dev

```
> 1 votes

---
Tags: continuous-integration, gitlab, gitlab-ci-runner
---


# Question
Title: Cross Account EC2 Access from within Amazon Workspace
I have a requirement to setup a bastion host (Amazon Workspace) to connect to an ec2 instance living on a different account.

As Amazon Workspaces cannot assume a role the way EC2s can, A cross account policy cannot be used to setup a link between them.

Both Amazon Workspace and EC2 instance live in their respectve VPC's in separate accounts. Is there a Cross account policy like solution which I can use to link these two?

P.S. I cannot create a Workspace in the same VPC/Account because the directory service is only available in one account but the EC2 which needs to be connected to is in other. Also, the EC2 cannot go public as well.

# Answer
Use a VPC peering between the two accounts so that your instance from account A will be able to communicate with account B.

This will work if you have different subnet CIDR blocks in the two accounts. If you have the same (i.e. in both VPC, your subnets are in `192.168.0.0/16`), then change the one that has the least impact and proceed with a VPC peering.

Finally, you will have to configure your security group to allow the bastion to communicate with the other instance.

Here are more details for AWS VPC peering: VPC peering
> 1 votes

---
Tags: amazon-web-services, amazon-ec2, aws-cli
---


# Question
Title: Installing ingress controller, services and pods on K8s via Ansible
I've 3 Debian-10 VMs on a bare metal Proxmox host in my network and installed via Ansible the roles `geerlingguy.docker` and `geerlingguy.kubernetes` with complete default settings. The kubeadm on the master node shows me the three nodes. So, I think, the installation is fine.

And what now? I'm new to kubernetes, I know there are configs that need to be applied. But how can I "see" them or "change" them via Kubernetes/Ansible. I don't want go back to the command line here. I know about the 'k8s' module - but the examples are not very helpful. A kubernetes should have an ingress controller and some services. Is there a good example playbook for installing Traefik/Haproxy/Nginx/whatever and maybe a small example hello-world-example for Kubernetes? Is there a "map" of possible values for all the YAML options (I used to learn from examples)

Sorry for the very "open" question. I'm searching for some days now and either my questions to Google are wrong or Google sends me always to the same sites with either very "basic" informations that ends by installing the Kubernetes or very "heavy" information, where the author is far away from easy to read.

To make the question a litte bit more concrete.

* The 3 VM's with IP 192.168.10.10-192.168.10.12 are running with Kubernetes 1.17.9 on Docker 19.03.12
* I've installed Helm v3.2.4, some repos (Traefik, Nginx, Stable-Kubernetes-Charts)
* I'm able to install with Helm some -don't know - are that pods or services?
* I'd like to add a simple hello-world HTTP service, like that from gmolaire (thanks!)
* That service is exposed to one of the three VMs on port 80

When I run that tutorial with step `kubectl create -f ingress.yaml` I'm getting the error

```
Error from server (InternalError): error when creating "ingress.yaml":
Internal error occurred: failed calling webhook "validate.nginx.ingress.kubernetes.io": Post https://ingress-nginx-controller-admission.ingress-nginx.svc:443/extensions/v1beta1/ingresses?timeout=30s: dial tcp 10.105.231.128:443: connect: connection refused

```

This might be, because I cannot run `kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/mandatory.yaml` it ends with a 404.

# Answer
> 3 votes
Since your new in this platform, I would suggest you to explore in the bellow order (assuming you want to use nginx as the load balancer ingress controller):

1- This tutorial is the most complete to get started with k8s. I strongly suggest you to follow it through to understand the jargon and the working components before going to the next steps.

2- Take a look at this example where they go through creating an nginx application in a k8s cluster.

3- Look at this tutorial where the author describes the key concepts and working operation of an ingress controller.

4- Finally, look at this to start setting up your ingress controller from the official doc page. Other resources are available to allow you to setup as much as you'd like.

Extra: Here is the official k8s `kubectl` cheat sheet.

---
Tags: kubernetes, ansible
---


# Question
Title: Why is Docker best practice to minimise the number of layers in an image?
I see here that

> Docker best practices says .. minimise the number of layers in our images

The tips given are to chain multiple lines in the dockerfile with `\` so as to combine many lines of code into single commands. A specific example being:

Acceptable:

```
RUN pip install jupyter

RUN pip install pandas

```

Better:

```
RUN pip install jupyter && \
pip install pandas

```

From this knowledge, I understand what should be done, however, I don't understand why. What are the advantages of minimising the number of layers in a Docker image? (or, conversely, what are the *dis*advantages of *not* doing so?)

# Answer
> 4 votes
you should minimize the Layers of an Image for following Reasons:

* first, you can only have (afaik) 127 Layers in an Image.
* second: yes, you should include severals Steps in a Single run because of the Copy-On-Write functionality:

if you run:

```
RUN dd if=/dev/zero of=output.dat  bs=1M  count=10
RUN rm -rf output.dat

```

your image is still 10MB (plus some bytes for the delete Marker)

instead, you should run:

```
RUN dd if=/dev/zero of=output.dat  bs=1M  count=10 &&\
    rm -rf output.dat

```

then, your image size is not increasing.

surely, this is not the Usual Use-Case, but, for example from a Real-World-Example:

```
# install terraform
ENV TF_VERSION="0.12.16"
RUN curl -LO https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip \
  && unzip terraform_${TF_VERSION}_linux_amd64.zip \
  && chmod +x ./terraform \
  && mv ./terraform /usr/local/bin/terraform \
  && rm terraform_${TF_VERSION}_linux_amd64.zip

```

more informations here: https://docs.docker.com/storage/storagedriver/#the-copy-on-write-cow-strategy

---
Tags: docker, containers, images
---


# Question
Title: How can I use bash functions in makefile?
If I need to use something like NVM in my makefile, I run into a problem, because NVM exports some bash functions that are not available in Make, even if my shell is defined as bash in the makefile.

How do I cause make to "inherit" all the parent shell's functions? Is there a way to make NVM accessible to a makefile, without writing export -f for a zillion little NVM bash functions?

# Answer
One possible issue is that the NVM install script only writes the commands to set itself up to either .bash\_profile OR .bashrc. Depending on your systems configuration, it may only be setup for interactive shells, not ones spawned by other programs (more info). Sourcing the nvm setup script can be a good option, as a previous comment noted. If you have a number of commands to run for 1 make target, look into .ONESHELL. Sourcing $NVM\_DIR/nvm.sh should take care of it.

Another option that invokes bash directly with the interactive flag might work as well.
> 2 votes

---
Tags: bash, make
---


# Question
Title: Issue with cloudformation in target groups and ASG
I am trying to use cloudformation to setup a ASG and I have a target group which I would like to associate with the ASG. The target group is created okay but when I go to attach it to the ASG I get an error message. I'd like to be able to attach this as I want a ALB in front of the ASG

# Answer
> 1 votes
As the error message states, TargetGroupARNs value should be of type List. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-targetgrouparns

---
Tags: amazon-web-services, aws-cloudformation
---


# Question
Title: How do I delete a default AKS node pool when I get an error saying it's the "backend address pool of the basic load balancer"
I'm attempting to replace a node pool in Azure Kubernetes Service with a new pool (that has a larger VM size). The current pool was deployed using an ARM template that uses version 2019-06-01 of the resource manager API. As per the Microsoft documentation I should be able to add a new System node pool and delete the original, default node pool.

I can successfully deploy an additional node pool that has it's mode set to System using this template, that template uses version 2020-03-01 of the resource manager API so that I can set the mode of the new pool.

When I attempt to delete the original, default node pool using `az aks nodepool delete -g gandt-testshared-rg --cluster-name gandt-testshared-aks -n agentpool` I get an error saying `Operation failed with status: 'Bad Request'. Details: The agent pool cannot be deleted because it is used as the backend address pool of the basic load balancer.`.

Why am I getting this error and what can I do to get the backend address pool of the basic load balancer to use the new node pool?

**Update**

I've deleted the cluster and redeployed using the original ARM template then added the 2nd node pool using az cli command `az aks nodepool add -g gandt-testshared-rg --cluster-name gandt-testshared-aks --name poolb2ms01 --node-count 1 --node-vm-size Standard_B2ms --mode System` and got the same result when deleting the default node pool

# Answer
If my assumption is correct that's related to having a loadbalancer that wasn't correctly deleted. Which could be resulted by redeploying a service (of type: load balancer) in a short time before Azure cleans up the previously associated loadbalancer resource.

To my knowledge, that could be fixed with the rolling of Kubernetes 1.17 and the Loadbalancer Finalizer feature, through this pull.

please update the question if that showed up with the new cluster version.
> 1 votes

---
Tags: kubernetes, azure, azure-kubernetes-service
---


# Question
Title: How to pass a String parameter with spaces in Jenkins to environment variable
In jenkins Job, I am taking a String parameter as below:

FoldersList = test1,test2,test3,Folder name1,Folder name2,Folder name3

when I pass this value to environment variable the jenkins job fails with error : Unknown lifecycle phase etc etc since spaces are not allowed in folder names.

I am passing the environment variable like this in goals and options section :

exec:java -Dexec.classpathScope="test" -Dexec.mainClass="com.mypackage.main.TestNGRunner" -Dexec.cleanupDaemonThreads=false "-Dexec.args=-inputexcel=Data.xlsx -jenkinsJobName=testJob -layer=MultiplefolderTest -InpFoldersList="\\"$inpFoldersList"\\" -threadCount=12 -retryCount=3"

If I pass a single string within double quotes it works but the user will enter multiple folder names possibly with spaces. Please can somebody help since I tried different options but none work. Is there a way to manipulate the string and pass it to the environment variable.

# Answer
> 1 votes
If the input string is failing due to folders with spaces not having quotes around them, try adding quotes around every object.

The InpFolderList already adds end quotes:

`InpFoldersList="\"$inpFoldersList"\"`

Add the quotes around every object.

**groovy:**

```
class Example {
   static void main(String[] args) {
        String FolderList = "test1,test2,test3,Folder name1,Folder name2,Folder name3";
        println(FolderList.replaceAll(",","\",\""));
   }
}

```

**Output:** (End quotes are left off)

`test1","test2","test3","Folder name1","Folder name2","Folder name3`

**Test Code:** https://www.jdoodle.com/execute-groovy-online/ (Copy / Paste Code and Execute)

Using above as an example, you could also add the end quotes and have a complete String variable to pass as $InpFoldersList, which would remove confusion if attempting to debug.

**Reference:** http://docs.groovy-lang.org/latest/html/api/org/codehaus/groovy/runtime/StringGroovyMethods.html#replaceAll(java.lang.CharSequence,java.util.regex.Pattern,java.lang.CharSequence)

# Answer
> 0 votes
What about save the env variable within a file and read from it after, you can encrypt/decrypt or use another type of parameter if that is a problem.

---
Tags: jenkins
---


# Question
Title: Snake bites its tail problem automating configuration of Certbot with Ansible
I have a public web-site with Nginx in front of it. We use free Letsencrypt certificates to set up `https`. These are acquired and renewed by Certbot program. I want to implement entire servers configuration with Ansible playbooks. These playbooks have to:

1. Automatically setup everything from scratch on a set of freshly dispatched empty CentOS instances.
2. Be able to setup with a different domain name if necessary
3. Be completely idempotent so I can make small changes and just run them as is.

There are two similar problems I faced trying to achieve it.

The first is I need Nginx already running to fulfill challenges to acquire certificate and then run Nginx with different config to make use of now acquired certificate. When I acquire certificate Nginx has to serve some files for Letsencrypt to check. This incurs several problems:

1. Playbook has to run Nginx twice in one go with different configs. This is tricky to express using Ansible because of descriptive nature of playbooks. Nginx now has different states in just one Playbook execution. I no longer express state but rather instructions.
2. Using several slightly different configs means either a lot of Nginx config code repetitions or complex and confusing `if`s in them.

The second problem is that I wanted to put all my Nginx configuration under VCS so I want playbooks to come with already written Nginx configs. However this not the way recommended by Certbot. Certbot wants you to use a command that *parses* and then *generates* Nginx configs to add a certificate there. This may seem convenient but I this way I cannot be sure the results are going to be completely reproducible. Suppose I acquire cert and generate config, then change something by hand and then I lose my servers and domain name and try to run them from scratch. Then some new problems may occur which otherwise wouldn't have. My Nginx configs are complex with many services on different ports, conditions and other details. I have to control how it looks like.

I considered making several different playbooks for different cases. But this also leads to code repetitions and makes using them more complex for a newcoming employee. And rarely used playbooks or scripts may become out of sync.

I did achieve a solution that passes all these criteria. But I am not happy with it because it turned out rather complex with Nginx configuration happening twice and many `if`s and booleans variables. And it's not how Ansible playbooks are intended to be written. Is there a good way to do all this?

# Answer
Don't read the text written on Certbot site, the real talk is in the documentation.

In order to make less actions, consider grouping configs by domains. Plays for a playbook in order to achieve what you want:

Deploy certbot and nginx with no config files, then for each **domain** on host:

1. Deploy an nginx config for default server with `location /.well-known/acme-challenge { root /etc/letsencrypt/challenges; }`
2. Reload nginx config
3. Check existence of a certificate for the domain in `/etc/letsencrypt/live/domain`
4. If cert is already presented, end the run
5. `certbot certonly -n --web-root=/etc/letsencrypt/challenges --keep -d domain` to make a cert request
6. Check for cert in a retry loop
7. Fix permissions for live certificates
8. Deploy normal nginx configs

You may deploy a stub config per domain instead of using the default server. The playbook could be run asynchronously for each domain.
> 1 votes

---
Tags: ansible, nginx, certbot, letsencrypt
---


# Question
Title: How can I send monitoring notifications to Google Chat in Google Cloud Platform?
I want to send monitoring notifications to Google Chat in Google Cloud Platform. Under Monitoring \> Alerting \> Edit notificaton channels, Google Chat is not listed (only the competitor Slack, interesting...).

If I try to use a Webhook, the message never arrives in Google Chat. This is because the webhook request sent by Google has the wrong format - it does not include a "text" key in the JSON body (see here for the format of the request sent by Google).

**Is it impossible to send monitoring notifications to Google Chat, even though the competitor Slack is available?**

# Answer
> 3 votes
I think the current solution is to implement some script to send notification. I followed this guide and wrote a Python Cloud Function to do the notifications.

---
Tags: google-cloud-platform, google-apps
---


# Question
Title: Jenkins Not pulling the latest commit into master or slave
I have a Jenkins pipeline where after every build I change a file(increase build version in a property file) and checkin back to the git repository. After a build, I commit some differnt code change to the git repo and run the build. One of the following two things happen

1. Jenkins does not pull the latest code change from git to master machine or slave machine.
2. The build increment checkin stage fails when we checkin from Jenkins slave.

The only solution is to wipe up the repository and build it again . Our project is a very big one and doing this every time causes a lot of time-wasting.

What configuration am i missing?

# Answer
1. You can use a plugin called "Workspace Cleanup Plugin" which wipes out the workspace post build. This can be a part of the pipeline and would not require manual cleanup every time when a build should be triggered.
2. For the second issue, I would recommend you to check if you're able to checkin to the same repository with git credentials configured in jenkins settings through the command line in the slave.
> 1 votes

---
Tags: jenkins, jenkins-pipeline, git
---


# Question
Title: Trigger cicd pipeline after merge request accept on master, Gitlab
Trying to run some task after merge request get accept (only on master branch). I am aware of `only: - master -merge_requests` config options, but want to specifically run pipeline only when code get merged in master.

# Answer
> 1 votes
https://docs.gitlab.com/ee/ci/yaml/

https://gitlab.com/gitlab-org/gitlab-ce/issues/31296#note\_34944784

> ```
> integration-testing:
>   stage: test
>   only:
>     - master
>   script:
>     - ./gradlew clean build asciidoctor
>
> ```

Another option is to create a webhook

https://gitlab.scm.webanywhere.co.uk/help/user/project/integrations/webhooks.md

If the "Merge request events" is clicked one could trigger a pipeline if a Pull Request gets updated, created or merged.

# Answer
> 0 votes
More options like this.

```
integration-testing:
  stage: test
  only:
    refs:
      - master
      - /release-.+/

```

Complete documentation can be found at gitlab doc

---
Tags: gitlab, gitlab-ci-runner
---


# Question
Title: Secure RDS (private endpoint) and allow connecting from local machine for specific predefined users
I'm looking for some way to **allow access to RDS (Private endpoint) instances from local machine** for a specific set of people (for the internal team.)

**ways I know:** Create Bastion EC2 and through SSH tunneling connect from local.

as I'm already using k8s from where the instances are reachable. is there any way to do some kind of port forwarding and finally connect RDS instance from local machine?

what I was trying: as I'm already having k8s which has access to those RDS instances, I tried creating a service in that, and through the Loadbalancer, I tried to connect using ssh but failed. (Probably I misunderstood/missing something in the process.)

is there any other secure easy way to do it?

# Answer
> 1 votes
I am not sure it will be helped. I have similar design in my office.

I have some Kubernetes Cluster. all node and master running on local ip from a PVC. And my rds too, only have local ip run on same PVC. Kubernetes Cluster already connect to RDS using local ip.

Then, I build VPN server on bastion server. Have local ip and public ip, run on same PVC with RDS and Kubernetes cluster. So when need connect, I just start my vpn connection on my laptop.

Only vpn/bastion server have IP public. + Load balancer too.

I hope this will give you inspiration.

---
Tags: amazon-rds, eks
---


# Question
Title: Can't hit redis using DNS name in multi-container pod on Kubernetes
Can anyone shed some light on why this might not work? I can access redis just fine using the pod's IP address,

```
$ rdcli -h 10.152.183.65
10.152.183.65:6379>

```

but when I try to use the DNS name, I cannot connect.

```
$ rdcli -h dbbackupd.default.cluster.local
dbbackupd.default.cluster.local:6379> (error) Redis connection to dbbackupd.default.cluster.local:6379 failed - getaddrinfo ENOTFOUND dbbackupd.default.cluster.local dbbackupd.default.cluster.local:6379

```

Inside the pod that is trying to contact redis, I get the right-looking output when using the host command, so name resolution looks good:

```
$ host dbbackupd.default.svc.cluster.local

dbbackupd.default.svc.cluster.local has address 10.152.183.65

```

My theory is that because redis is in a multi-container pod, that dns is not routing it to the container properly. Here is my Service's definition in case that sheds light:

```
apiVersion: v1
kind: Service
metadata:
  name: dbbackupd
  labels:
    app: dbbackupd
spec:
  selector:
    app: dbbackupd
  type: NodePort
  ports:
    - name: redis-mq
      protocol: TCP
      port: 6379
      targetPort: 6379
    - name: sql-comm
      protocol: TCP
      port: 1433
      targetPort: 1433

```

# Answer
You're using two different host names:

```
$ rdcli -h dbbackupd.default.cluster.local
$ host dbbackupd.default.svc.cluster.local

```

The one used with `rdcli` is missing `.svc`.
> 3 votes

---
Tags: kubernetes, containers, dns, redis
---


# Question
Title: What is the best way to set up a CI/CD pipeline on ECS?
There are so many options:

* Docker-compose with ECS cli looks the easiest solution
* Terraform
* CloudFormation (looks complex!)
* Ansible

I am only interested in setting up a basic ECS docker set-up with ELB and easily updating the Docker image version.

We all love technology here, but we're not all super geniuses when it comes to tech. So I'm looking to keep my set-up as simple as possible. We run Jenkins, 2 NodeJS applications, 2 Java applications in ECS and I know it involves IAM, Security Groups, EBS, ELB, ECS Service/Task, ECS Task Definition, but that already gets complex quickly in CloudFormation.

What are good technologies that will allow us to use Docker, keep things simple and don't require us to be very intelligent to understand our own programming code?

# Answer
The question is a bit broad, but here are some suggestions to accomplish what I believe you are asking. If they are simple or not is for you to judge!

If the infrastructure referred to already exists and you would like to manage, update using infrastructure as code (IaC), you could try tools to convert the infrastructure to code. (These are new and at least one is in beta at the time or writing this)

If the infrastructure does not exist, I recommend using infrastructure as code (IaC). AWS was mentioned, as well as the complexity of CloudFormation, with these considerations, Terraform could be a good option.

Finally, if you would like to abstract away the bulk of the infrastructure and just run containers in the cloud.

Once the code and infrastructure option is selected, CI/CD is as easy as adding a buildspec.yaml to the repository and adding the Code Build infrastructure to the Terraform code (AWS) and similar for Cloud Build (GCP).

Hopefully these options help find you the simple solution you are looking for!
> 1 votes

---
Tags: docker, amazon-web-services, ecs
---


# Question
Title: Jenkins Helm Chart - kubernetes-plugin pulling image from a private gcr
I am failing to pull images from a private gcr repo through helm jenkins on a private k8s repo. I am getting:

> rpc error: code = Unknown desc = Error response from daemon: unauthorized: You don't have the needed permissions to perform this operation, and you may have invalid credentials. To authenticate your request, follow the steps in: https://cloud.google.com/container-registry/docs/advanced-authentication.

Is there anyway I can mitigate this error without specifically specifying credentials on pipeline level?

Pulling image from host work and creating a deployment in K8s is pulling successfully, but when pulled within Jenkins it doesn't work.

# Answer
Found the issue, had:

```
kubectl -n jenkins patch serviceaccount jenkins -p '{"imagePullSecrets": [{"name": "gcr-json-key"}]}'

```

... applied on `default` serviceaccount instead of `jenkins`.
> 1 votes

---
Tags: jenkins, kubernetes, docker-registry
---


# Question
Title: How do I make ansible-playbook log its output in a machine readable format like xml or json?
Looking for information on how to output machine readable output form ansible-playbook.

# Answer
This is done using one of the configuration options. For a general overview of configuring Ansible, see the configuration documentation, in particular the callback plugin configuration.

Callback plugins change how Ansible provides feedback on the progress and state of tasks. There are several options for JSON output:

1. JSON output to stdout
2. JSON to syslog

Setting these configuration options can be done either in a configuration file (`ansible.cfg`) or with environment variables (`ANSIBLE_XXX`), depending on what you prefer. To quote the current documentation:

> Changes can be made and used in a configuration file which will be searched for in the following order:

> ```
>    ANSIBLE_CONFIG (environment variable if set)
>    ansible.cfg (in the current directory)
>    ~/.ansible.cfg (in the home directory)
>    /etc/ansible/ansible.cfg
>
> ```

> Ansible will process the above list and use the first file found, all others are ignored.

Assuming you want to send the output to stdout then, your `ansible.cfg` could look like:

```
[defaults]
callback_whitelist=json
stdout_callback=json

```

Using environment variables:

```
 ANSIBLE_CALLBACK_WHITELIST=json ANSIBLE_STDOUT_CALLBACK=json ansible-playbook ...

```
> 3 votes

---
Tags: ansible
---


# Question
Title: Inspec errors using https://github.com/dev-sec/linux-baseline
I am running the following command `inspec exec https://github.com/dev-sec/linux-baseline -t ssh://ubuntu@10.0.1.22 -i ~/.ssh/id_rsa --sudo`

And I am getting failures for

* Check login.defs (4 failed)
    + All these params look like they should pass
* sysctl-29: Disable loading kernel modules
    + I accidentally set `echo "1" > /proc/sys/kernel/modules_disabled` and now I'm unable to set it back :-(
* package-07: Install syslog server package
    + What package should I install?
* os-06: Check for SUID/ SGID blacklist
    + Where can I set this?

```
ubuntu@ip-10-0-1-10:~/.ssh$ inspec exec https://github.com/dev-sec/linux-baseline -t ssh://ubuntu@10.0.1.22 -i ~/.ssh/id_rsa --sudo
/usr/local/rvm/gems/ruby-2.3.1@global/gems/rspec-core-3.8.0/lib/rspec/core/example_group.rb:679: warning: already initialized constant RSpec::Core::ExampleGroup::INSTANCE_VARIABLE_TO_IGNORE
/usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rspec-core-3.8.0/lib/rspec/core/example_group.rb:679: warning: previous definition of INSTANCE_VARIABLE_TO_IGNORE was here
/usr/local/rvm/gems/ruby-2.3.1@global/gems/rspec-core-3.8.0/lib/rspec/core/example_group.rb:722: warning: already initialized constant RSpec::Core::ExampleGroup::WrongScopeError
/usr/local/rvm/rubies/ruby-2.3.1/lib/ruby/gems/2.3.0/gems/rspec-core-3.8.0/lib/rspec/core/example_group.rb:722: warning: previous definition of WrongScopeError was here
verify_host_key: false is deprecated, use :never
[2020-08-15T21:22:01+00:00] WARN: URL target https://github.com/dev-sec/linux-baseline transformed to https://github.com/dev-sec/linux-baseline/archive/master.tar.gz. Consider using the git fetcher
[2020-08-15T21:22:01+00:00] WARN: Attribute 'login_defs_umask' does not have a value. Use --attrs to provide a value for 'login_defs_umask' or specify a default  value with `attribute('login_defs_umask', default: 'somedefault', ...)`.
[2020-08-15T21:22:01+00:00] WARN: Attribute 'login_defs_passmaxdays' does not have a value. Use --attrs to provide a value for 'login_defs_passmaxdays' or specify a default  value with `attribute('login_defs_passmaxdays', default: 'somedefault', ...)`.
[2020-08-15T21:22:01+00:00] WARN: Attribute 'login_defs_passmindays' does not have a value. Use --attrs to provide a value for 'login_defs_passmindays' or specify a default  value with `attribute('login_defs_passmindays', default: 'somedefault', ...)`.
[2020-08-15T21:22:01+00:00] WARN: Attribute 'login_defs_passwarnage' does not have a value. Use --attrs to provide a value for 'login_defs_passwarnage' or specify a default  value with `attribute('login_defs_passwarnage', default: 'somedefault', ...)`.
[2020-08-15T21:22:02+00:00] WARN: Attribute 'blacklist' does not have a value. Use --attrs to provide a value for 'blacklist' or specify a default  value with `attribute('blacklist', default: 'somedefault', ...)`.
[2020-08-15T21:22:02+00:00] WARN: Attribute 'syslog_pkg' does not have a value. Use --attrs to provide a value for 'syslog_pkg' or specify a default  value with `attribute('syslog_pkg', default: 'somedefault', ...)`.
[2020-08-15T21:22:02+00:00] WARN: Attribute 'sysctl_forwarding' does not have a value. Use --attrs to provide a value for 'sysctl_forwarding' or specify a default  value with `attribute('sysctl_forwarding', default: 'somedefault', ...)`.
[2020-08-15T21:22:02+00:00] WARN: Attribute 'kernel_modules_disabled' does not have a value. Use --attrs to provide a value for 'kernel_modules_disabled' or specify a default  value with `attribute('kernel_modules_disabled', default: 'somedefault', ...)`.

Profile: DevSec Linux Security Baseline (linux-baseline)
Version: 2.4.6
Target:  ssh://ubuntu@10.0.1.22:22

  ✔  os-01: Trusted hosts login
     ✔  File /etc/hosts.equiv should not exist
  ✔  os-02: Check owner and permissions for /etc/shadow
     ✔  File /etc/shadow should exist
     ✔  File /etc/shadow should be file
     ✔  File /etc/shadow should be owned by "root"
     ✔  File /etc/shadow should not be executable
     ✔  File /etc/shadow should not be readable by other
     ✔  File /etc/shadow group should eq "shadow"
     ✔  File /etc/shadow should be writable by owner
     ✔  File /etc/shadow should be readable by owner
     ✔  File /etc/shadow should be readable by group
  ✔  os-03: Check owner and permissions for /etc/passwd
     ✔  File /etc/passwd should exist
     ✔  File /etc/passwd should be file
     ✔  File /etc/passwd should be owned by "root"
     ✔  File /etc/passwd should not be executable
     ✔  File /etc/passwd should be writable by owner
     ✔  File /etc/passwd should not be writable by group
     ✔  File /etc/passwd should not be writable by other
     ✔  File /etc/passwd should be readable by owner
     ✔  File /etc/passwd should be readable by group
     ✔  File /etc/passwd should be readable by other
     ✔  File /etc/passwd group should eq "root"
  ✔  os-03b: Check passwords hashes in /etc/passwd
     ✔  /etc/passwd passwords should be in "x" and "*"
  ✔  os-04: Dot in PATH variable
     ✔  Environment variable PATH split should not include ""
     ✔  Environment variable PATH split should not include "."
  ×  os-05: Check login.defs (4 failed)
     ✔  File /etc/login.defs should exist
     ✔  File /etc/login.defs should be file
     ✔  File /etc/login.defs should be owned by "root"
     ✔  File /etc/login.defs should not be executable
     ✔  File /etc/login.defs should be readable by owner
     ✔  File /etc/login.defs should be readable by group
     ✔  File /etc/login.defs should be readable by other
     ✔  File /etc/login.defs group should eq "root"
     ✔  login.defs ENV_SUPATH should include "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
     ✔  login.defs ENV_PATH should include "/usr/local/bin:/usr/bin:/bin"
     ×  login.defs UMASK should include #<Inspec::Attribute::DEFAULT_ATTRIBUTE:0x00000005a1ea00 @name="login_defs_umask">
     can't convert Inspec::Attribute::DEFAULT_ATTRIBUTE to String (Inspec::Attribute::DEFAULT_ATTRIBUTE#to_str gives Inspec::Attribute::DEFAULT_ATTRIBUTE)
     ×  login.defs PASS_MAX_DAYS should eq #<Inspec::Attribute::DEFAULT_ATTRIBUTE:0x00000005a1e3e8 @name="login_defs_passmaxdays">
     can't convert Inspec::Attribute::DEFAULT_ATTRIBUTE to Array (Inspec::Attribute::DEFAULT_ATTRIBUTE#to_ary gives Inspec::Attribute::DEFAULT_ATTRIBUTE)
     ×  login.defs PASS_MIN_DAYS should eq #<Inspec::Attribute::DEFAULT_ATTRIBUTE:0x00000005a1dee8 @name="login_defs_passmindays">
     can't convert Inspec::Attribute::DEFAULT_ATTRIBUTE to Array (Inspec::Attribute::DEFAULT_ATTRIBUTE#to_ary gives Inspec::Attribute::DEFAULT_ATTRIBUTE)
     ×  login.defs PASS_WARN_AGE should eq #<Inspec::Attribute::DEFAULT_ATTRIBUTE:0x00000005a1da38 @name="login_defs_passwarnage">
     can't convert Inspec::Attribute::DEFAULT_ATTRIBUTE to Array (Inspec::Attribute::DEFAULT_ATTRIBUTE#to_ary gives Inspec::Attribute::DEFAULT_ATTRIBUTE)
     ✔  login.defs LOGIN_RETRIES should eq "5"
     ✔  login.defs LOGIN_TIMEOUT should eq "60"
     ✔  login.defs UID_MIN should eq "1000"
     ✔  login.defs GID_MIN should eq "1000"
  ↺  os-05b: Check login.defs - RedHat specific
     ↺  Skipped control due to only_if condition.
  ×  os-06: Check for SUID/ SGID blacklist
     ×  suid_check diff
     can't convert Inspec::Attribute::DEFAULT_ATTRIBUTE to Array (Inspec::Attribute::DEFAULT_ATTRIBUTE#to_ary gives Inspec::Attribute::DEFAULT_ATTRIBUTE)
  ✔  os-07: Unique uid and gid
     ✔  /etc/passwd uids should not contain duplicates
     ✔  /etc/group gids should not contain duplicates
  ✔  os-08: Entropy
     ✔  3092 should >= 1000
  ✔  os-09: Check for .rhosts and .netrc file
     ✔  [] should be empty
  ✔  os-10: CIS: Disable unused filesystems
     ✔  File /etc/modprobe.d/dev-sec.conf content should match "install cramfs /bin/true"
     ✔  File /etc/modprobe.d/dev-sec.conf content should match "install freevxfs /bin/true"
     ✔  File /etc/modprobe.d/dev-sec.conf content should match "install jffs2 /bin/true"
     ✔  File /etc/modprobe.d/dev-sec.conf content should match "install hfs /bin/true"
     ✔  File /etc/modprobe.d/dev-sec.conf content should match "install hfsplus /bin/true"
     ✔  File /etc/modprobe.d/dev-sec.conf content should match "install squashfs /bin/true"
     ✔  File /etc/modprobe.d/dev-sec.conf content should match "install udf /bin/true"
     ✔  File /etc/modprobe.d/dev-sec.conf content should match "install vfat /bin/true"
  ✔  os-11: Protect log-directory
     ✔  File /var/log should be directory
     ✔  File /var/log should be owned by "root"
     ✔  File /var/log group should match /^root|syslog$/
  ✔  package-01: Do not run deprecated inetd or xinetd
     ✔  System Package inetd should not be installed
     ✔  System Package xinetd should not be installed
  ✔  package-02: Do not install Telnet server
     ✔  System Package telnetd should not be installed
  ✔  package-03: Do not install rsh server
     ✔  System Package rsh-server should not be installed
  ✔  package-05: Do not install ypserv server (NIS)
     ✔  System Package ypserv should not be installed
  ✔  package-06: Do not install tftp server
     ✔  System Package tftp-server should not be installed
  ×  package-07: Install syslog server package
     ×  System Package Attribute 'syslog_pkg' does not have a value. Skipping test. should be installed
     expected that `System Package Attribute 'syslog_pkg' does not have a value. Skipping test.` is installed
  ✔  package-08: Install auditd
     ✔  System Package auditd should be installed
     ✔  Audit Daemon Config log_file should cmp == "/var/log/audit/audit.log"
     ✔  Audit Daemon Config log_format should cmp == "raw"
     ✔  Audit Daemon Config flush should match /^incremental|INCREMENTAL|incremental_async|INCREMENTAL_ASYNC$/
     ✔  Audit Daemon Config max_log_file_action should cmp == "keep_logs"
     ✔  Audit Daemon Config space_left should cmp == 75
     ✔  Audit Daemon Config action_mail_acct should cmp == "root"
     ✔  Audit Daemon Config space_left_action should cmp == "SYSLOG"
     ✔  Audit Daemon Config admin_space_left should cmp == 50
     ✔  Audit Daemon Config admin_space_left_action should cmp == "SUSPEND"
     ✔  Audit Daemon Config disk_full_action should cmp == "SUSPEND"
     ✔  Audit Daemon Config disk_error_action should cmp == "SUSPEND"
  ✔  package-09: CIS: Additional process hardening
     ✔  System Package prelink should not be installed
  ↺  sysctl-01: IPv4 Forwarding
     ↺  Skipped control due to only_if condition.
  ✔  sysctl-02: Reverse path filtering
     ✔  Kernel Parameter net.ipv4.conf.all.rp_filter value should eq 1
     ✔  Kernel Parameter net.ipv4.conf.default.rp_filter value should eq 1
  ✔  sysctl-03: ICMP ignore bogus error responses
     ✔  Kernel Parameter net.ipv4.icmp_ignore_bogus_error_responses value should eq 1
  ✔  sysctl-04: ICMP echo ignore broadcasts
     ✔  Kernel Parameter net.ipv4.icmp_echo_ignore_broadcasts value should eq 1
  ✔  sysctl-05: ICMP ratelimit
     ✔  Kernel Parameter net.ipv4.icmp_ratelimit value should eq 100
  ✔  sysctl-06: ICMP ratemask
     ✔  Kernel Parameter net.ipv4.icmp_ratemask value should eq 88089
  ✔  sysctl-07: TCP timestamps
     ✔  Kernel Parameter net.ipv4.tcp_timestamps value should eq 0
  ✔  sysctl-08: ARP ignore
     ✔  Kernel Parameter net.ipv4.conf.all.arp_ignore value should eq 1
  ✔  sysctl-09: ARP announce
     ✔  Kernel Parameter net.ipv4.conf.all.arp_announce value should eq 2
  ✔  sysctl-10: TCP RFC1337 Protect Against TCP Time-Wait
     ✔  Kernel Parameter net.ipv4.tcp_rfc1337 value should eq 1
  ✔  sysctl-11: Protection against SYN flood attacks
     ✔  Kernel Parameter net.ipv4.tcp_syncookies value should eq 1
  ✔  sysctl-12: Shared Media IP Architecture
     ✔  Kernel Parameter net.ipv4.conf.all.shared_media value should eq 1
     ✔  Kernel Parameter net.ipv4.conf.default.shared_media value should eq 1
  ✔  sysctl-13: Disable Source Routing
     ✔  Kernel Parameter net.ipv4.conf.all.accept_source_route value should eq 0
     ✔  Kernel Parameter net.ipv4.conf.default.accept_source_route value should eq 0
  ✔  sysctl-14: Disable acceptance of all IPv4 redirected packets
     ✔  Kernel Parameter net.ipv4.conf.default.accept_redirects value should eq 0
     ✔  Kernel Parameter net.ipv4.conf.all.accept_redirects value should eq 0
  ✔  sysctl-15: Disable acceptance of all secure redirected packets
     ✔  Kernel Parameter net.ipv4.conf.all.secure_redirects value should eq 0
     ✔  Kernel Parameter net.ipv4.conf.default.secure_redirects value should eq 0
  ✔  sysctl-16: Disable sending of redirects packets
     ✔  Kernel Parameter net.ipv4.conf.default.send_redirects value should eq 0
     ✔  Kernel Parameter net.ipv4.conf.all.send_redirects value should eq 0
  ✔  sysctl-17: Disable log martians
     ✔  Kernel Parameter net.ipv4.conf.all.log_martians value should eq 1
     ✔  Kernel Parameter net.ipv4.conf.default.log_martians value should eq 1
  ✔  sysctl-18: Disable IPv6 if it is not needed
     ✔  Kernel Parameter net.ipv6.conf.all.disable_ipv6 value should eq 1
  ↺  sysctl-19: IPv6 Forwarding
     ↺  Skipped control due to only_if condition.
  ✔  sysctl-20: Disable acceptance of all IPv6 redirected packets
     ✔  Kernel Parameter net.ipv6.conf.default.accept_redirects value should eq 0
     ✔  Kernel Parameter net.ipv6.conf.all.accept_redirects value should eq 0
  ✔  sysctl-21: Disable acceptance of IPv6 router solicitations messages
     ✔  Kernel Parameter net.ipv6.conf.default.router_solicitations value should eq 0
  ✔  sysctl-22: Disable Accept Router Preference from router advertisement
     ✔  Kernel Parameter net.ipv6.conf.default.accept_ra_rtr_pref value should eq 0
  ✔  sysctl-23: Disable learning Prefix Information from router advertisement
     ✔  Kernel Parameter net.ipv6.conf.default.accept_ra_pinfo value should eq 0
  ✔  sysctl-24: Disable learning Hop limit from router advertisement
     ✔  Kernel Parameter net.ipv6.conf.default.accept_ra_defrtr value should eq 0
  ✔  sysctl-25: Disable the system`s acceptance of router advertisement
     ✔  Kernel Parameter net.ipv6.conf.all.accept_ra value should eq 0
     ✔  Kernel Parameter net.ipv6.conf.default.accept_ra value should eq 0
  ✔  sysctl-26: Disable IPv6 autoconfiguration
     ✔  Kernel Parameter net.ipv6.conf.default.autoconf value should eq 0
  ✔  sysctl-27: Disable neighbor solicitations to send out per address
     ✔  Kernel Parameter net.ipv6.conf.default.dad_transmits value should eq 0
  ✔  sysctl-28: Assign one global unicast IPv6 addresses to each interface
     ✔  Kernel Parameter net.ipv6.conf.default.max_addresses value should eq 1
  ×  sysctl-29: Disable loading kernel modules
     ×  Kernel Parameter kernel.modules_disabled value should eq #<Inspec::Attribute::DEFAULT_ATTRIBUTE:0x000000052722e8 @name="kernel_modules_disabled">
     can't convert Inspec::Attribute::DEFAULT_ATTRIBUTE to Array (Inspec::Attribute::DEFAULT_ATTRIBUTE#to_ary gives Inspec::Attribute::DEFAULT_ATTRIBUTE)
  ✔  sysctl-30: Magic SysRq
     ✔  Kernel Parameter kernel.sysrq value should eq 0
  ✔  sysctl-31a: Secure Core Dumps - dump settings
     ✔  Kernel Parameter fs.suid_dumpable value should cmp == /(0|2)/
  ✔  sysctl-31b: Secure Core Dumps - dump path
     ✔  Kernel Parameter kernel.core_pattern value should match /^\|?\/.*/
  ✔  sysctl-32: kernel.randomize_va_space
     ✔  Kernel Parameter kernel.randomize_va_space value should eq 2
  ✔  sysctl-33: CPU No execution Flag or Kernel ExecShield
     ✔  /proc/cpuinfo Flags should include NX

Profile Summary: 48 successful controls, 4 control failures, 3 controls skipped
Test Summary: 112 successful, 7 failures, 3 skipped

```

# Answer
A few initial comments for posterity:

1. You can check exactly what the control is asserting by looking at the source code
2. You can see what the remediation should be, by checking Dev-Sec's Ansible role
3. Most of your issues seem to be related to missing or incorrect input variables. See the Inspec documentation on that topic

The specific issues you raise:

## Check login.defs (4 failed)

This comes from the control `os-5`:

```
control 'os-05' do
  impact 1.0
  title 'Check login.defs'
  desc 'Check owner and permissions for login.defs. Also check the configured PATH variable and umask in login.defs'
  describe file('/etc/login.defs') do
    it { should exist }
    it { should be_file }
    it { should be_owned_by 'root' }
    its('group') { should eq 'root' }
    it { should_not be_executable }
    it { should be_readable.by('owner') }
    it { should be_readable.by('group') }
    it { should be_readable.by('other') }
  end
  describe login_defs do
    its('ENV_SUPATH') { should include('/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin') }
    its('ENV_PATH') { should include('/usr/local/bin:/usr/bin:/bin') }
    its('UMASK') { should include(login_defs_umask) }
    its('PASS_MAX_DAYS') { should eq login_defs_passmaxdays }
    its('PASS_MIN_DAYS') { should eq login_defs_passmindays }
    its('PASS_WARN_AGE') { should eq login_defs_passwarnage }
    its('LOGIN_RETRIES') { should eq '5' }
    its('LOGIN_TIMEOUT') { should eq '60' }
    its('UID_MIN') { should eq '1000' }
    its('GID_MIN') { should eq '1000' }
  end
end

```

The specific assertions that are failing are related to:

* umask
* password age (min/max days, warning)

These use attributes (ie, Inspec input values), which are read higher up in the control:

```
login_defs_umask = attribute('login_defs_umask', value: os.redhat? ? '077' : '027', description: 'Default umask to set in login.defs')

login_defs_passmaxdays = attribute('login_defs_passmaxdays', value: '60', description: 'Default password maxdays to set in login.defs')
login_defs_passmindays = attribute('login_defs_passmindays', value: '7', description: 'Default password mindays to set in login.defs')
login_defs_passwarnage = attribute('login_defs_passwarnage', value: '7', description: 'Default password warnage (days) to set in login.defs')

```

The error you are getting:

```
login.defs UMASK should include #<Inspec::Attribute::DEFAULT_ATTRIBUTE:0x00000005a1ea00 @name="login_defs_umask">
     can't convert Inspec::Attribute::DEFAULT_ATTRIBUTE to String (Inspec::Attribute::DEFAULT_ATTRIBUTE#to_str gives Inspec::Attribute::DEFAULT_ATTRIBUTE)

```

should be expected, given the previous warning Inspec told you:

```
WARN: Attribute 'login_defs_umask' does not have a value. Use --attrs to provide a value for 'login_defs_umask' or specify a default  value with `attribute('login_defs_umask', default: 'somedefault', ...)`

```

Either you have an old version of Inspec where the default is not set, or you passed a set of input variables where these values are not set.

## os-06: Check for SUID/ SGID blacklist

This control checks the `suid` of a blacklist. It uses a custom resource `suid_check`. It essentially find's files with certain characteristics:

```
def permissions
    output = inspec.command('find / -perm -4000 -o -perm -2000 -type f ! -path \'/proc/*\' ! -path \'/var/lib/lxd/containers/*\' -print 2>/dev/null | grep -v \'^find:\'')
    output.stdout.split(/\r?\n/)
end

```

Your diff fails because some files match the blacklist, which is again set as an attribute:

```
blacklist = attribute(
  'blacklist',
  value: suid_blacklist.default,
  description: 'blacklist of suid/sgid program on system'
)

```

It's default has a method `default` with a big list of paths. You can pass your own blacklist by setting it in an input file.

## sysctl-29: Disable loading kernel modules

> I accidentally set echo "1" \> /proc/sys/kernel/modules\_disabled and now I'm unable to set it back

The kernel documentation says:

> Once true, modules can be neither loaded nor unloaded, and the toggle cannot be set back to false.

You probably have to reboot the machine.

## package-07: Install syslog server package

The control makes an assertion on a package:

```
describe package(val_syslog_pkg) do

```

where `val_syslog_pkg` is an attribute passed as an input variable:

`val_syslog_pkg = attribute('syslog_pkg', value: 'rsyslog', description: 'syslog package to ensure present (default: rsyslog, alternative: syslog-ng...')`

In Ubuntu, this package name is `syslog-ng`.
> 1 votes

---
Tags: inspec
---


# Question
Title: How to append namespace before metric name in prometheus?
I have static config in prometheus.yml

```
  - job_name: 'myjob'
    static_configs:
      - targets: ['monitoring.api.endpoint.net']

```

And I recieve various metrics for example `cpu_usage`, `disk_write_ops`, etc. Is it possible to put them all into namespace like `compute_cpu_usage` with relabel tool?

# Answer
Found this solution:

```
static_configs:
  - targets: ['monitoring.api.endpoint.net']
metric_relabel_configs:
- source_labels: [__name__]
  regex: '(.*)'
  replacement: 'compute_${1}'
  target_label: __name__

```
> 1 votes

---
Tags: prometheus
---


# Question
Title: How to log native commands run by Ansible Playbook
Ansible internally uses SSH Commands to run on the target machine . Is there a way to log the commands that are being executed ?

# Answer
> 2 votes
Set the log file to use inline or in Ansible config, then run ansible-playbook with verbose option:

```
ANSIBLE_LOG_PATH="ansible-$(date +%Y%m%d%H%M%S).log" \
ansible-playbook -vvvv \
    --diff \
    -i inventory/myinventory \
    myplaybook.yml

```
* `ANSIBLE_LOG_PATH` \- is an environment variable that maps to the `log_path` configuration item. Sets the file to log to. When using an environment variable you can use shell substitution to add the date and time to the file name so the next run doesn't overwrite the log file.
* `-vvvv` \- be very very very verbose. Will show SSH commands executed.
* `--diff` \- bonus, when making a change show what has been done.
* the rest are standard options.

The log will be in a file in the current directory named something like `ansible-20200818155322.log`

# Answer
> 0 votes
Use the register module and store the result in a variable.

```
 - hosts: all
   tasks:
     - name: Ansible register variable basic example
       shell: "find *.txt"
       args:
         chdir: "/Users/mdtutorials2/Documents/Ansible"
       register: find_output

     - debug:
         var: find_output

```

---
Tags: ansible
---


# Question
Title: GitLab pipeline stop at the stage of pulling an image from Docker hub
## Description:

I have created a test project to serve a simple HTML page with `httpd:alpine` image. The following are the parts of the repository:

`index.html`:

```
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Rostami test page</title>
</head>
<body>
    <h1>Hey guys!</h1>
</body>
</html>

```

`Dockerfile`:

```
FROM httpd:alpine

COPY ./ /usr/local/apache2/htdocs/

```

`docker-compose.yml`:

```
version '2.2'

services:
  myhtml:
  image: html_mrt:latest
  ports:
  - "9999:80"
  restart: on-failure
  container_name: HTML_Rostami

```

`.gitlab-ci.yml`:

```
stages:
  - build
  - deploy

build:
  stage: build
  script:
  - docker build -t html_mrt .
  tags:
  - docker

deploy:
  stage: deploy
  script:
  - docker-compose up -d
  - docker rmi $(docker image -q filter "dangling=true")
  tags:
    - docker

```

---

## Issue:

In addition, when I execute `gitlab-runner status` on the GitLab-Runner container, it says:

```
Runtime platform                                    arch=amd64 os=linux pid=79 revision=a998cacd version=13.2.2
gitlab-runner: Service is not running.

```

I've tried to start the service by `gitlab-runner start` but nothing has affected.

---

## Problem:

It's the log of the build stage:

```
Running with gitlab-runner 13.2.2 (a998cacd)
  on Test K-1ZsHyF
Preparing the "docker" executor
00:11

Using Docker executor with image docker ...
Pulling docker image docker ...
Using docker image sha256:81f5749c9058a7284e6acd8e126f2b882765a17b9ead14422b51cde1a110b85c for docker ...
Preparing environment
00:02

Running on runner-k-1zshyf-project-1-concurrent-0 via 806fd175c866...
Getting source from Git repository
00:03

Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/m.rostami/test/.git/
Checking out 82d339a9 as master...
Skipping Git submodules setup
Executing "step_script" stage of the job script
00:04

$ docker build -t html_mrt .
time="2020-08-18T06:36:52Z" level=error msg="Can't add file /builds/m.rostami/test/docker-compose.yml to tar: io: read/write on closed pipe"
time="2020-08-18T06:36:52Z" level=error msg="Can't close tar writer: io: read/write on closed pipe"
error during connect: Post http://docker:2375/v1.40/build?buildargs=%7B%7D&cachefrom=%5B%5D&cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&labels=%7B%7D&memory=0&memswap=0&networkmode=default&rm=1&session=8jgar93piqxu2s7y5yp9w2wtg&shmsize=0&t=html_mrt&target=&ulimits=null&version=1: dial tcp: lookup docker on 192.168.1.1:53: no such host
ERROR: Job failed: exit code 1

```

`192.168.1.1` is the default gateway and nameserver of the host that the Docker daemon is running.

How to fix the problem?

Thanks in advance.

# Answer
> 2 votes
I image you have used the dockerExecutor in gitlab-runner. This means you try to use docker-in-docker. https://docs.gitlab.com/ee/ci/docker/using\_docker\_build.html

I suggest to configure the docker socket binding which might be less secure but very easy to handle.

To register a runner with docker-in-docker support mount the `/var/run/docker.sock:/var/run/docker.sock` e.g.

```
sudo gitlab-runner register -n \
  --url https://gitlab.com/ \
  --registration-token REGISTRATION_TOKEN \
  --executor docker \
  --description "My Docker Runner" \
  --docker-image "docker:19.03.12" \
  --docker-volumes /var/run/docker.sock:/var/run/docker.sock

```

---
Tags: gitlab, gitlab-ci-runner, apache-httpd
---


# Question
Title: Are there serious problems with an EC2 auto-scaling AMI that automatically downloads newest server from git?
I'm converting some servers from manual scaling (with a load balancer) to auto-scaling. I'm currently using an Ansible script to manage upgrades, and I wouldn't want an automatically created instance to not be using the newest version of the software. I created an AMI that will download the newest commit from a chosen branch on first boot.

I've since realized my approach is somewhat like "cowboy coding", since AWS provides a mechanism to replace auto-scaling instances when there should be an update. (If using that, my existing update script would become obsolete, and I guess the update would entail creating a new AMI containing the new server version.)

Are there any serious problems with using the "cowboy" approach? I realize auto-created servers might end up with newer code than the other servers (if someone pushes but does not deploy their code changes). I wonder whether auto-scaling will cause Ansible scripts to fail if servers are terminated while being processed by Ansible. What have I missed?

# Answer
What you're going to do is only calling for inconsistencies between your nodes.

I wouldn't do *any* deployment during the instance boot-up, instead install **AWS CodeDeploy** as part of your boot-up (or bake it into the AMI) and once the instance boots up it contacts the CodeDeploy service and obtains the newest code version.

Likewise if you need to roll-out a newer code version push it to CodeDeploy and it will take care of updating all the servers in your Auto-Scaling Group in a consistent, predictable way.

With ASG you really really want to prevent inconsistencies between the nodes and this is one of the ways to do it.

Hope that helps :)
> 2 votes

---
Tags: amazon-web-services, ansible, autoscaling
---


title: Combining Two Containers

text:
# Question
Title: Combining Two Containers
I'm working on a Windows laptop and running Docker for Windows on it. I want to run an Ubuntu container with a specific version and a MySQL server on it. So the question is, do I have to download MySQL on the Ubuntu container or can I run 2 containers (Ubuntu and MySQL) and combine them? How do I combine these 2 containers?

# Answer
At first, I misread your question, thinking you wanted to run more than one service together, so I had recommended Docker Compose which makes it easy to run multiple services together, with networking, and can even specify the start up order.

But now I see you are just trying to run MySQL, and wondering do you need separate containers for the OS and MySQL: in short, no you don't need two containers. Instead, just get the MySQL image from DockerHub at https://hub.docker.com/\_/mysql?tab=description

The reason you can just use that MySQL Docker image by itself is because it already is installed on a Debian image, which you can see in the Dockerfile at https://github.com/docker-library/mysql/blob/master/8.0/Dockerfile where it shows:

```
FROM debian:buster-slim

```

So, to solve the problem, do:

1. `docker pull mysql`
2. `docker run --name some-mysql -e MYSQL_ROOT_PASSWORD=my-secret-pw -d mysql:tag` as mentioned in the instructions on the Docker Hub page. Additional documentation at https://github.com/docker-library/docs/tree/master/mysql

I do strongly recommend using Docker Compose (`docker-compose` command) to run containers, as it makes it easier to organise and run them.
> 2 votes

---
Tags: docker, windows, mysql, ubuntu
---


# Question
Title: How to create a host on a DigitalOcean team with Terraform?
Teams on Digital Ocean are very recent and there is no documentation about using them with Terraform.

All information I can found is how to work with them manually.

I was hoping that I could say something like:

```
resource "digitalocean_droplet" "host" {
    image = "${var.image}"
    name = "${var.host_name}"
    region = "${var.region}"
    size = "${var.size}"
    private_networking = "${var.private_networking}"
    ssh_keys = ["${var.ssh_fingerprint}"]
    team = "${var.team}"
}

```

Note the `team = "${var.team}"` at the end.

Since Terraform is just a wrapper on the top of the provider APIs I looked at the documentation trying to find some endpoint of field about this but found nothing.

How should I use Digital Ocean teams with Terraform? Or I am stuck with the `main` account if I want to automate things with Terraform?

# Answer
According to the article you linked, Team Resources are shared. This means that all droplets (hosts) are shared across the team once you have a team in place.

source: https://www.digitalocean.com/community/tutorials/how-to-use-teams-on-digitalocean#team-resources

This means that each team member, using his own personal token, can manage all the resources shared with her. Terraform allows each user to use his own credentials when managing resources, and includes a state file (that must be shared as well). Managing shared `tfstate` and managing secrets is addressed in this DevOps.SE question.
> 4 votes

# Answer
In https://www.digitalocean.com/docs/accounts/teams/resources/ , it says "The API does not support teams." and "doctl is a command-line interface for the DigitalOcean API and supports many of the same actions. Like the API, doctl does not support teams."

So I don't think you can do what you want to do, unfortunately. AFAIK, Terraform just uses the API.
> 2 votes

---
Tags: terraform
---


# Question
Title: How to solve 'upstream connect error or disconnect/reset before headers. reset reason: connection termination' in Istio?
I try to setup an aws load balancer (ELB) with SSL follow the instruction at #6566

Certificate was attached on ELB.

However, I got the issue "**upstream connect error or disconnect/reset before headers. reset reason: connection termination**" when trying to access our web on browser.

Our setup without SSL has been worked before.

I use the custom values.yaml to install the istio (helm template):

```
helm template ./istio/install/kubernetes/helm/istio --name istio \
--namespace istio-system --values ./mesh/values.yaml | kubectl apply -f -

```

I have inserted below annotations to the `gateways` tag:

```
istio-ingressgateway:
    serviceAnnotations:
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "arn:aws:acm:ap-southeast-1:xxxxx:certificate/my-crt"
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http"
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"

```

Here is my `gateway.yaml`:

```
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway #default istio ingressgateway
  servers:
  - port:
      number: 80
      name: http-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-gateway
spec:
  gateways:
  - istio-gateway
  hosts:
  - "*"
  http:
  - match:
    - uri:
        prefix: /socket.io/
    route:
    - destination:
        host: api-gateway-ws.default.svc.cluster.local
        port:
          number: 5001
  - match:
    - uri:
        prefix: /
    route:
    - destination:
        host: api-gateway.default.svc.cluster.local
        port:
          number: 5000

```

# Answer
> 7 votes
I encountered the same error but with a different issue. The Service port needed a name added to it. https://github.com/istio/istio/issues/19966. And they need to follow the format (protocol-suffix) https://istio.io/docs/ops/deployment/requirements/

```
ports:
    - name: https # Use http or https
       protocol: TCP
       port: 8080
       targetPort: 8080

```

# Answer
> 4 votes
I have resolved the issue by update the gateway manifest. Not sure why the error happen when adding multiple "match".

```
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
spec:
  selector:
    istio: ingressgateway #default istio ingressgateway
  servers:
  - port:
      number: 80
      name: http-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https-istio-gateway
      protocol: HTTP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: api-gateway
spec:
  gateways:
  - istio-gateway
  hosts:
  - "*"
  http:
  - match:
    - uri:
        prefix: "/socket.io"
    route:
    - destination:
        host: api-gateway-ws.default.svc.cluster.local
        port:
          number: 5001
    websocketUpgrade: true
  - route:
    - destination:
        host: api-gateway.default.svc.cluster.local
        port:
          number: 5000

```

# Answer
> 4 votes
This is most probably there is mismatch of the port that application running in the DOCKER is not the same port exposed in the DOCKER configuration

Docker file `EXPOSE <PORT>` should be same as the port the application is started.

Istio Ingress gateway can bind to the container port but can't communicate the application.

# Answer
> 1 votes
**Split large virtual services and destination rules into multiple resources:** The downside of your kind of configuration is that other configuration (e.g., route rules) for any of the underlying microservices, will need to also be included in this single configuration file. Reference: https://istio.io/docs/ops/best-practices/traffic-management/enter link description here

# Answer
> 0 votes
sometimes it happens when you apply Istio mTLS so first try to disable mTLS in the relevant namespace. (especially if you are using some kind of gRPC in the pod)

```
kubectl get peerauthentication --all-namespaces

```

---
Tags: kubernetes, istio
---


# Question
Title: What is the best approach to deploying to multiple servers using Jenkins?
Trying to figure out the best approach for deploying to multiple servers using Jenkins Blue Ocean pipeline. We will have six servers in development, six in QA, six in Staging, and six in production. Should I put the list of the servers in a config file and store in git or is there a plug-in or something within Jenkins that stores host names? All feedback is greatly appreciate.

# Answer
Have you looked at the Matrix project plugin?

> **Multi-Configuration Projects**
>
> A multi-configuration project is useful for instances where your builds will make many similar build steps, and you would otherwise be duplicating steps.
>
> **Configuration matrix**
>
> The Configuration Matrix allows you to specify what steps to duplicate, and create a multiple-axis graph of the type of builds to create.

Welcome to the Matrix is an excellent reference doc.

Example (from the blog) - Pipeline for multiple platforms and browsers

> ```
> Jenkinsfile
>
> pipeline {
>     agent none
>     stages {
>         stage('BuildAndTest') {
>             matrix {
>                 agent any
>                 axes {
>                     axis {
>                         name 'PLATFORM'
>                         values 'linux', 'windows', 'mac'
>                     }
>                     axis {
>                         name 'BROWSER'
>                         values 'firefox', 'chrome', 'safari', 'edge'
>                     }
>                 }
>                 stages {
>                     stage('Build') {
>                         steps {
>                             echo "Do Build for ${PLATFORM} - ${BROWSER}"
>                         }
>                     }
>                     stage('Test') {
>                         steps {
>                             echo "Do Test for ${PLATFORM} - ${BROWSER}"
>                         }
>                     }
>                 }
>             }
>         }
>     }
> }
>
> ```
> 1 votes

---
Tags: jenkins, jenkins-pipeline, jenkins-plugins, jenkinsfile, jenkins-blueocean
---


# Question
Title: Sending Exceptions to Elastic Stack via REST
I am working with devs wishing to replace Exceptionless, a Real-time exception reporting tool.

They wish to send the exceptions from a web app that they would previously send to Exceptionless to our ELK stack.

I am only experienced with using Beats to ship data to ELK.

In the past I have written custom code to do similar things like this, but we shipped to Splunk, not ELK.

How friendly is the process of sending information to ELK via REST? Are their tips, tricks and/or pitfalls?

Is this a supported, recommended, and good way to do exception logging/reporting/analytics/etc.

# Answer
> 1 votes
It depends on what you're trying to accomplish but IMO if you're interested in reliably getting your exceptions you probably want something between Elasticsearch and your app - a queueing or logging system of some kind like Kafka or Redis.

Send your exceptions there and then drain that to Elasticsearch. Otherwise you run the risk of overwhelming ES's ability to accept writes at the very time you need it most (cascading failure, DDOS etc).

# Answer
> 0 votes
https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-update.html

> ```
> POST test/_update/1
> {
>     "script" : {
>         "source": "if (ctx._source.tags.contains(params.tag)) { ctx._source.tags.remove(ctx._source.tags.indexOf(params.tag)) }",
>         "lang": "painless",
>         "params" : {
>             "tag" : "blue"
>         }
>     }
> }
>
> ```

---
Tags: elastic
---


# Question
Title: is a running container created by each RUN command during multistage build?
I'm new to Docker, sorry if my question sounds dumb. below is a dockerfile from a textbook:

```
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt              #line 2

FROM diamol/base AS test-stage
COPY --from=build-stage /build.txt /build.txt
RUN echo 'Testing...' >> /build.txt              #line 6

FROM diamol/base
COPY --from=test-stage /build.txt /build.txt
CMD cat /build.txt

```

and the author says:

***The RUN instruction executes a command inside a container during the build, and any output from that command is saved in the image layer.***

My question is:

Since the author mentioned RUN instruction executes a command inside a **container** during the build, does it mean that in the first RUN command in line 2, a contianer is created based on the `diamol/base` image, and this container modify the content of the files then commit the change back to `diamol/base` image, then is container is terminated? and same thing happen in line 6,container created to run and then terminate, image gets updated, is my understanding correct?

# Answer
Yes your understanding is correct. The `RUN <commands>` command in the Dockerfile is somewhat equivalent to

```
container = $(docker run base-image) #1
docker exec base-image commands #2
docker commit $container #3

```

Since images are static, we'll need a running instance/process to execute our command. this is why a container is created from the base image. #1

After the execution of the command in #2, the changes are committed back #3, which means a new layer is created on top of the previous layers. This layer will contain the changes introduced by the container in (#2).
> 2 votes

---
Tags: docker
---


# Question
Title: what's the final output image in multistage build?
Below is the dockerfile that creates an image:

```
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt

FROM diamol/base AS test-stage
COPY --from=build-stage /build.txt /build.txt
RUN echo 'Testing...' >> /build.txt

FROM diamol/base
COPY --from=test-stage /build.txt /build.txt
RUN echo 'Done...' >> /build.txt

```

and my understanding is, the final image consists layers of diamol/base and the layer created after the last RUN command as picture below shows:

Is my understanding correct?

# Answer
> 1 votes
The final image will have all the layers from `diamol/base` as well as two additional layers - one for COPY and one for RUN command.

File `/build.txt` will have the following contents:

```
Building...
Testing...
Done...

```

---
Tags: docker
---


# Question
Title: Docker container host machine communication
Hi i’m trying to communicate with mongodb in the host from a flask container…

So using “docker.host.internal” on the URI didn’t work (host machine is Ubuntu)

Any suggestions ?? To solve this

# Answer
You can run your flask container using the host networking mode. This way docker won't isolate the container and will run it the same network namespace as the host, and you container will be able to discover your database using `localhost` as mongodb hostname

```
docker run --network host <flask-image>

```
> 1 votes

# Answer
From the point of view of your MongoDB instance and your Flask app they're running on separate servers, so the steps to make the former visible to the latter are the same as if they were running on two different physical servers.

1. Check if the servers are on the same LAN - they are (docker bridge network).
2. Find if MongoDB is configured to listen on all network interfaces. If not either configure it to do so or to listen on the interface that is connected to the LAN both servers have in common.
3. If you haven't already, find what IP your MongoDB server has on the common LAN and set Flask app to use it to connect to the database.
4. Optionally, if your MongoDB server has a firewall, make sure that whatever port MongoDB is listening on is not blocked.
> 0 votes

# Answer
You can add an alias IP address to the loopback interface on the host to get a consistent address to use from your containers.

Any private address you won't see on the local or container networks, something like 10.8.8.8

Edit `/etc/network/interfaces.d/lo` for a permanent address after reboot

```
auto lo
iface lo inet loopback
iface lo inet static
  address 10.8.8.8/32

```

or to temporarily test

```
ip address add 10.8.8.8/32 dev lo

```
> -1 votes

---
Tags: docker, ubuntu, flask
---


# Question
Title: why use multiple From instructions in multistage build?
I'm new to Docker, sorry if my question sounds dumb. below is a dockerfile from a textbook shows how multistage build works:

```
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt

FROM diamol/base AS test-stage
COPY --from=build-stage /build.txt /build.txt
RUN echo 'Testing...' >> /build.txt

FROM diamol/base
COPY --from=test-stage /build.txt /build.txt
CMD cat /build.txt

```

My questions is:

Why use multiple From instructions? why we don't do sth like:

```
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt

COPY --from=build-stage /build.txt /build.txt
RUN echo 'Testing...' >> /build.txt

COPY --from=test-stage /build.txt /build.txt
CMD cat /build.txt

```

# Answer
> 2 votes
First of all, you don't need the COPY instructions or multiple RUN instructions if you have only one stage:

```
FROM diamol/base AS build-stage
RUN echo 'Building...' > /build.txt && echo 'Testing...' >> /build.txt
CMD cat /build.txt

```

Of course, the textbook example is just that - an example to illustrate the concept.

In reality multistage builds are used to create smaller final images, while keeping the entire build process inside the container.

Generally speaking one may install all development and runtime dependencies in a build stage, then copy just the final distribution package to the final stage, where only runtime dependencies are installed to reduce the final image size.

For example:

```
FROM ubuntu:latest AS base
RUN apt -y install some-library

FROM base AS build
RUN apt -y install some-library-dev
COPY . /src
WORKDIR /src
RUN make && make install

FROM base
COPY --from=build /usr/bin/app /usr/bin/app
CMD /usr/bin/app

```

---
Tags: docker, dockerfile
---


# Question
Title: Unable to build Image as it is asking some input
I am trying to create an Image which will run ftp server on centos. Dockerfile content is as below:

```
FROM centos

RUN \
  yum update -y && \
  yum install vsftpd -y

EXPOSE 21
CMD vsftpd -DFOREGROUN

```

I am running below command :

```
docker build -f webserver .

```

which is asking me input as below:

```
Please select the geographic area in which you live. Subsequent configuration
questions will narrow this down by presenting a list of cities, representing
the time zones in which they are located.

  1. Africa      4. Australia  7. Atlantic  10. Pacific  13. Etc
  2. America     5. Arctic     8. Europe    11. SystemV
  3. Antarctica  6. Asia       9. Indian    12. US
Geographic area: 9

```

once I enter 9 and press enter button, it stuck at the same window. Can you please help me, how should I process this?

# Answer
> 3 votes
The `docker` build process is non-interactive. You have to find a way to make it not ask for anything. The way to do that is by creating the necessary config files and placing them in the correct locations through the `Dockerfile` before installing vsftpd. The questions are there simply for vsftpd to generate a/some configuration file\[s\] for itself. You create one beforehand and place it correctly using the `copy` command before `run yum install`. If vsftpd finds the valid config file\[s\], it won't ask for anything anymore.

# Answer
> 0 votes
The problem is a typo in your build command:

> docker build -f webserver .

what's the webserver file you are using for that build if the script is described in the Dockerfile? BTW I have built your image and there is no interactive menu appeared, so just build it))

`docker build -t webserver .`

---
Tags: docker, dockerfile
---


# Question
Title: How do I use Jenkins to maintain versioning on a Python project?
An independent Jenkins server monitors my develop branch in Team Foundation Server's Git repository and builds a new package when it's updated. The problem I'm having is that it uses the same version, specified in setup.py. I could configure pypi-server to allow overwrites, but prefer to maintain build history.

Theoretically, there could be a build step that checks the version and increments it, but I'm hoping that someone has already done this work and published it somewhere.

Has anyone seen something like this?

# Answer
> 2 votes
If you follow Semantic Versioning you could achieve this as a step in the pipeline using Semantic Release.

This and the original tool it's based on are designed for automated releases as part of a continuous integration and delivery pipeline. From the docs:

> The key point with using this package is to automate your releases and stop worrying about version numbers. Different approaches to automatic releases and publishing with the help of this package can be found below. Using a CI is the recommended approach.

In your case the workflow would be to do all the testing in pull request, and then when they are merged, a step in the `develop` branch would run `semantic-release publish`. Again, referencing the docs, this does the following actions:

> Publish will do a sequence of things:
>
> * Run semantic-release version.
> * Push changes to git.
> * Run build\_command and upload the created files to PyPI.
> * Run semantic-release changelog and post to your vcs provider.
> * Attach the files created by build\_command to GitHub releases.

Part of the benefit of having this included in the pipeline is maintaining a clear and consistent changelog, with the help of a commit message linter like committizen.

---
Tags: jenkins, jenkins-pipeline, continuous-deployment, release-management
---


# Question
Title: Minimum postgresql + spring + react hardware requirements
I have to deploy my postgresql + spring + react fullstack application on premise so cloud is out of the question. It's going to be used by around 15 users at a time. But I have to specify what hardware do I need to run my application. Just to be safe I wanted to ask for 16GB of RAM and 100 GB of storage on ubuntu machine. But my friends says it's way too much.

I have been looking for answer everywhere on the internet but it's like nobody deals with dev ops, everybody just writes applications and that's it. Or they host them on something like heroku.

In summary, what's minimum hardware to run postgresql + spring + react application?

# Answer
> 1 votes
It's hard to tell, this kind of thing comes with experience but I'm sure 4 GB of RAM and 20 GB of disk storage will be enough.

---
Tags: postgresql, spring-boot
---


# Question
Title: Include curl commands in a PostgreSQL Docker environment
Using `COPY table FROM PROGRAM 'curl "www.example.com/data.csv"';` I am able to copy a CSV directly from the web into a psql table.

However, when I am running the same command from inside a docker container I get the following error:

```
COPY sites FROM PROGRAM 'curl "www.example.com/data.csv"';
ERROR:  program "curl "www.example.com/data.csv"" failed
DETAIL:  command not found

```

Obviously curl is not available in the containerized environment. Therefore I changed my Postgres Docker image from `image: postgres:12-alpine` to `image: postgres-12`. Unfortunately I still get the same error.

How can include the `curl` command in a containerized PostgreSQL Docker environment?

This is my docker-compose file:

```
version: '3.7'

services:
  web:
    build:
      context: ./services/web
      dockerfile: Dockerfile.prod
    command: gunicorn --bind 0.0.0.0:5000 manage:app
    volumes:
      - static_volume:/home/app/web/project/static
      - media_volume:/home/app/web/project/media
    expose:
      - 5000
    env_file:
      - ./.env.prod
    depends_on:
      - db
  db:
    image: postgres:12
    volumes:
      - postgres_data:/var/lib/postgresql/data/
    env_file:
      - ./.env.prod.db
  nginx:
    build: ./services/nginx
    volumes:
      - static_volume:/home/app/web/project/static
      - media_volume:/home/app/web/project/media
    ports:
      - 1337:80
    depends_on:
      - web
Include curl commands in a PostgreSQL Docker environment
volumes:
  postgres_data:
  static_volume:
  media_volume:

```

# Answer
> 3 votes
One way to use curl inside the container will be to use a custom Dockerfile to build you postgrsql container. Just use the `image: postgres:12-alpine` as the base image of your Dockerfile and install curl in alpine with the following command : `RUN apk --no-cache add curl`

---
Tags: docker, docker-compose, databases, images, postgresql
---


# Question
Title: AWS ECS Fargate, auto scaling and load balancer
For a ECS service with launch type "fargate", it is possible to configure a load balancer and auto scaling. I don\`t understand, how those two types of scaling work in this case, and how they work together.

**My example**
I define a task, and set the task size to "0.25 vCPU". I then configure the service to use a load balancer, and auto scaling. I set the auto scaling to a maximum number of 4 tasks.

**My questions**

1. I would expect, that the task is executed with 0.25 vCPU. When the task runs out of CPU, it starts another one, which uses another 0.25 vCPU. This is done until the maximum number of tasks is reached, this means a use and costs of 1vCPU in my example. Is this expectation correct?
2. What is the role of the load balancer? As I understand it, it is just needed for routing and https certfificate. Or does it play a role at scaling, too?
3. When I want to scale container based, then I need an own task definition for every container, right? And an own load balancer for every service? Is it possible to use one load balancer for multiple services?

In which AWS docs I can read about those topics?

# Answer
Load balancing will route the requests across tasks while autoscaling will increase or decrease the number of tasks based on the traffic.

starting new tasks when cpu threshold exceed is autoscaling, this can be setup with cloudwatch events.

you can load balance services based on rules and then route them to target groups. read here
> 2 votes

---
Tags: amazon-web-services, amazon-ecs, load-balance, fargate
---


# Question
Title: Curl Error in HTTP2 framing layer when trying to install Jenkins plugin
I am trying to create a custom Jenkins docker image. For that I need to install certain plugins in Jenkins. I am doing so by adding the below lines to the Dockerfile

```
COPY plugins.txt /usr/share/jenkins/ref/

ENV CURL_CONNECTION_TIMEOUT=60
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

```

My plugins.txt looks like this -

```
configuration-as-code

```

I had more plugins in the list but seems like I am unable to install even if there is only one.

The full error message that I receive is -

```
Creating initial locks...
Analyzing war /usr/share/jenkins/jenkins.war...
Registering preinstalled plugins...
Using version-specific update center: https://updates.jenkins.io/2.235...
Downloading plugins...
Downloading plugin: configuration-as-code from https://updates.jenkins.io/2.235/latest/configuration-as-code.hpi
curl: (16) Error in the HTTP2 framing layer
15:34:43 Failure (16) Retrying in 1 seconds...
 > configuration-as-code depends on snakeyaml-api:1.26.3
Downloading plugin: snakeyaml-api from https://updates.jenkins.io/2.235/latest/snakeyaml-api.hpi
curl: (16) Error in the HTTP2 framing layer
15:35:39 Failure (16) Retrying in 1 seconds...
curl: (16) Error in the HTTP2 framing layer
15:35:52 Failure (16) Retrying in 1 seconds...
curl: (16) Error in the HTTP2 framing layer
15:36:05 Failure (16) Retrying in 1 seconds...
15:36:06 Failed in the last attempt (curl -sSfL --connect-timeout 60 --retry 3 --retry-delay 0 --retry-max-time 60 https://updates.jenkins.io/2.235/latest/snakeyaml-api.hpi -o /usr/share/jenkins/ref/plugins/snakeyaml-api.jpi)
Downloading plugin: snakeyaml-api-plugin from https://updates.jenkins.io/2.235/latest/snakeyaml-api-plugin.hpi
curl: (22) The requested URL returned error: 404 Not Found
15:36:08 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404 Not Found
15:36:11 Failure (22) Retrying in 1 seconds...
curl: (22) The requested URL returned error: 404 Not Found
15:36:14 Failure (22) Retrying in 1 seconds...
15:36:15 Failed in the last attempt (curl -sSfL --connect-timeout 60 --retry 3 --retry-delay 0 --retry-max-time 60 https://updates.jenkins.io/2.235/latest/snakeyaml-api-plugin.hpi -o /usr/share/jenkins/ref/plugins/snakeyaml-api-plugin.jpi)
Failed to download plugin: snakeyaml-api or snakeyaml-api-plugin

WAR bundled plugins:

Installed plugins:
configuration-as-code:1.43
snakeyaml-api:1.26.4
Some plugins failed to download! Not downloaded: snakeyaml-api
The command '/bin/sh -c /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt' returned a non-zero code: 1

```

Curl tries to download `snakeyaml-api-plugin` when `snakeyaml-api` fails. The latter is not being downloaded due to the error `curl: (16) Error in the HTTP2 framing layer`

For multiple plugins, I get the HTTP2 framing layer error multiple times. Many a times, the installation gets stuck at a point without any progress

You can look into the shell script `install-plugins.sh` here

Tested on Windows 10 as well as MacOS Mojave Docker version - 19.03.12

# Answer
If HTTP/2 is not working try forcing HTTP 1.1:

```
COPY plugins.txt /usr/share/jenkins/ref/

ENV CURL_OPTIONS -sSfL --http1.1
ENV CURL_CONNECTION_TIMEOUT 60
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

```
> 1 votes

---
Tags: docker, jenkins, curl
---


title: CI/CD pipeline with manual advancement?

text:
# Question
Title: CI/CD pipeline with manual advancement?
I am currently learning about CI/CD pipelines for DevOps. I have some practice implementing them in some basic scenarios, but I've recently encountered a slightly more complex scenario and I'm not sure the best way to integrate CI/CD pipelines into this project.

### The Scenario

The client and the development team have agreed to work in 2 week sprints. Once the sprint has ended, the client's management team wants 1 week to play with the sprint delivery in a sandbox environment (we'll call it `UAT`) before it is released to production (`PROD`). This means that releases to `PROD` happen on "off weeks" from a sprint perspective, if they happen at all for that sprint. If the customer is unhappy with something they'll ask us to not deploy to prod until the issue is resolved. We still continue working in sprints, we just don't release that sprint to `PROD`.

### Question

Clearly the *naive* CI/CD pipeline approach doesn't work here since most code changes aren't expected to make it straight down a pipeline to `PROD`. In light of this, **what is the best approach here for a team that is still interested in utilizing pipelines as much as possible?**

### Some Thoughts

* We could use a single pipeline with manual approval steps prior to the `UAT` portion and then again prior the `PROD` portion. This has the benefit of being a single pipeline, so a single build artifact progresses through the entire pipeline, which seems to be a best practice. The problem is that most things that enter the pipeline are going to get hung up at a manual approval step. Most pipeline runs will then either be stalled or marked as failed.
* We could use 2 pipelines. One is totally CI/CD with `DEV` and `STAGING` environments. Every two weeks, a separate pipeline would be manually started (or started by some other release mechanism associated with SCM like tagging or merging). This would be in charge of rebuilding, retesting, and deploying to `UAT` and `PROD`.

If it's relevant, jenkins would be the pipeline of choice.

# Answer
The "D" in CI/CD means different things to different people, either delivery or deployment. For both groups, it rarely means release everything immediately to production. For the delivery group, the output of CI/CD is an artifact that *can* be deployed. For the deployment group, there are typically gates in pipelines with either automatic or manual checks before deployment happens. Automatic gates include the unit tests, CI tests, but also things like canary releases with monitoring of key metrics.

Manual approvals in Jenkins involve adding an input step to your pipeline. E.g.

```
input([message: "Ready to continue?"])

```

In my experience, having various stalled pipelines waiting for user approval is fairly common. I often include timeouts to clear out stale prompts, and sometimes I configure old prompts to automatically cancel with a more recent pipeline is approved to reduce the workload on users. If you do the latter, you may want to have a separate pipeline for backout/revert jobs since that would otherwise being a new job that gets promoted to production in an emergency and aborts all the other work in progress.

I still like to separate my builds from deploys with separate pipelines. It gives developers better feedback with their CI pipelines of what's working. Then the deployment team picks from a list of automatically queued up deployment jobs that were automatically triggered from those successful CI jobs.
> 2 votes

---
Tags: jenkins-pipeline, build-pipeline
---


# Question
Title: config.yaml is not setting up a new context
I have a kubernetes Config file such as:

```
apiVersion: v1
kind: Config
preferences: {}
contexts:
- context:
    cluster: test-sim-development
    namespace: test-sim
    user: developer
  name: test-sim

```

When I issue a command such as:

```
kubectl config --kubeconfig infra_k8/config.yaml set-context test-sim && kubectl config use-context test-sim

```

I get back the following error output:

```
Context "test-sim" modified.
error: no context exists with the name: "test-sim"

```

Why is it not finding the "test-sim" name? It is clearly referenced according to the kubernetes docs.

# Answer
> 1 votes
it seems you’re trying to change the config by setting a context and you’re not providing the config to the change context command. So do this

```
kubectl config --kubeconfig=infra_k8/config.yaml use-context test-sim

```

The setters are used when you want to add more entries to the config file, so you don’t need that command when changing context.

---
Tags: kubernetes, kubectl
---


# Question
Title: Dedicated/Separate Git Repo as Terraform Remote Backend
I see that there are multiple options for Backends, however git isn’t one of them.

I believe git can be used as a backend for tfstate, in the below steps:

1. Create a dedicated git repo
2. Commit the tfstate files in a specified directory and branch. Only fetch the required files, rather than the whole repo, as this repo could be used to store multiple tfstate files on different directories.
3. For locking and unlocking: Method 1 (git LFS): use ‘git lfs lock

Method 2 (git - where lfs isn’t possible): Use another file suffixed with .lock, which should contain the lock state.

Is there any plan for Terraform to release git as a backend?

I've also thought of ways to use the HTTP backend to upload to a git, no luck yet.

Thoughts on how this can be achieved?

There's already a Github/Gitlab/Bitbucket provider.

Thanks,

Note: Also asked in the Terraform Discuss

# Answer
> 1 votes
Using git for terraform remote state management is an interesting, but non-standard implementation. Storing binary / large files in git, integrating a locking system and a few other concerns come to mind. Using s3, Buckets or other storage options provide the multiple statefile management (versions in buckets, folders in buckets or terraform folder structure / workspaces) and other functionality you mentioned.

If you would like to pursue using git for remote state management, check out the community project terraform-backend-git. There is also a good discussion around it in the terraform issues section: https://github.com/hashicorp/terraform/issues/16616.

If you attempt to use the tool from the community, it would be great to hear about your experience. Good luck!

---
Tags: terraform, git
---


title: Gitlab CI - Change branch dynamically

text:
# Question
Title: Gitlab CI - Change branch dynamically
I have a nodeJS application which I deploy with Gitlab CI. When pushing to branch XYZ, the corresponding script runs, let's assume on stage 'dev'. That's working. So far so good. Is it possible to select a specific branch so that instead of branch XYZ, e.g. Branch FOO can be deployed on 'dev' without having to change the gitlab-ci.yml file every time?

# Answer
> 1 votes
Instead of hard-coding your branch name in `.gitlab-ci.yml` use `$CI_COMMIT_REF_NAME`. There are a bunch of pre-set environment variables you can use to get info about the current job.

The gitlab documentation has a list of them: https://docs.gitlab.com/ee/ci/variables/predefined\_variables.html

# Answer
> 0 votes
each step actually can run on each branch except you limit it with 'only' options. it will be more flexible to you. Please see this documentation section. https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic

---
Tags: gitlab, cicd, gitlab-ci-runner
---


# Question
Title: Automate Application Upgrade in Azure VMs
Sorry if this question is on the wrong Stack Exchange or too vague. I am really new to DevOps and got a big task and don't want to start on the wrong direction.

We have many Linux-based (Redhat) VMs in Azure and we have Splunk installed on them and I am looking for a way to automate Splunk upgrade.

I don't want to upgrade Splunk every-time there is a new version, but I was wondering if I can selectively upload a new version somewhere in Azure and have a Runbook that detects the new version and execute the Runbook to upgrade the Splunk app on each VM?

Upgrading Splunk requires a custom script (to make sure the new version is compatible with our environment) which makes this automation harder to design I suppose.

Is there a better way or option to perform Splunk upgrade? Preferably in a rolling fashion.

Any help would be appreciated.

Thanks.

# Answer
> 1 votes
Do you think this might help you?

https://docs.microsoft.com/en-us/azure/automation/update-management/update-mgmt-overview#:~:text=You%20can%20use%20Update%20Management,and%20in%20other%20cloud%20environments.&text=For%20multiple%20Azure%20VMs%20by,page%20in%20the%20Azure%20portal.

If your VM is disposable and you can replace it with a new VM then you can also use VM Image galleryfor Linux where you can have images for a VM or multiple VM types and you can update it at one place using Azure DevOps and then update the underlying VM image

https://docs.microsoft.com/en-us/azure/virtual-machines/linux/image-builder-gallery-update-image-version

---
Tags: azure-devops, azure, automation
---