# Question
Title: How to override default aws region credentials for Terraform?
I am using Terraform to create aws resources. I practiced with a single region and with my aws cli credentials being used. worked great!

however when I tried to override the region in the Terraform .tf config file it hangs indefinitely.

So how does one either set my aws credentials for the CLI to allow any region, OR how does one set Terraform config files (.tf) to allow multi region infrastructure?

# Answer
I believe you would be using multiple modules to create various resources(EC2,ELB, Etc) in AWS using Terraform. You can create a custom module having output value of each variable you want to configure per your requirement like aws-region, etc.

Then source(import) this custom module into your resource module and use its values using $.

custom\_module

```
output "aws_region" {
  value = "us-xyz-1"
}
output "aws_anyproperty" {
  value = "abc"
}

Add above custom_module as source in your resource module

module "custom_module" {
  source="../../../modules/custom_module"
}

provider "aws" {
  version="~> y.x"
  profile="${module.custom_module.aws_anyproperty}"
  region="${module.custom_module.aws_region}"
}

```
> 1 votes

---
Tags: terraform, amazon-s3, aws-cli, aws-iam
---


# Question
Title: How to keep polling GitHub repository and get commits by it's messages on a regular interval using Jenkins?
I need to know how to write a Jenkins job which polls a GitHub repository continuously and send an email of commits made to that repo.

# Answer
> 1 votes
Considering you have proper credentials in Jenkins and admin right for Git repo, Create a webhook between Jenkins and your Git repo. Webhook will help you push (reverse poll)the commits. You can use jenkins DSL (groovy) to operate on that Git repo by using Git commands. You can use specific Git commands to list Authors of last commits (or any other requirement) and Use email-ext (or any other) plugin in jenkins to be able to send email through DSL.

One line reference to list Author (yeah..it works as it is in any Git repo)-

git log -1 --pretty=format:'%an'

My reference of email to culprits if something goes wrong my jenkins build

step(\[$class: 'Mailer', notifyEveryUnstableBuild: true, recipients: emailextrecipients(\[\[$class: 'CulpritsRecipientProvider'\], \[$class: 'RequesterRecipientProvider'\]\])\])

---
Tags: jenkins, github
---


# Question
Title: How to perform low-level inspecting of a Docker network traffic?
I'm currently in a scenario that requires Docker container C to be able to intercept communication between docker containers A and B, without their knowledge. I am basically attempting to do a standard man in the middle attack on my own containers. A and B are bridged right now. I'm using `tshark` in container C to inspect incoming traffic, but C is not recieving anything, except for initial broadcasting requests.

I've tried different `com.docker.network.bridge` options but none seem to work well. This is what my network looks like at the moment:

```
[
    {
        "Name": "PKKJKO6WJEVH936CQ12H",
        "Id": "5033f8958b845153f94beaa4b3bdbe17879ee331ad9441737687eac70b37fbc4",
        "Scope": "local",
        "Driver": "bridge",
        "IPAM": {
            "Driver": "default",
            "Options": {},
            "Config": [
                {
                    "Subnet": "172.18.0.0/16",
                    "Gateway": "172.18.0.1/16"
                }
            ]
        },
        "Containers": {
            "41411a497edf7a632b042d71879d1c0b5941af0361325d2dbe4fefa9bbb30727": {
                "Name": "insane_goldstine",
                "EndpointID": "5c9ffaa6270e8a1823464fa1cdd9f64868083d32cfc7576a5843b1c76944353f",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "849135f83dca0d7261f03952b1648851dfb4c85a0cda0a7d1f5ab5b24e31a014": {
                "Name": "berserk_ramanujan",
                "EndpointID": "68a1ac542dea9fcfa1e35a209f571fc8c670f8d8ec33cf126c4dcde1bca1729d",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "af80dbfaf424fb4b8ce192a8d2e0e946d97ae1abb731bf3fc75918d31c084625": {
                "Name": "clever_goldwasser",
                "EndpointID": "0b2b3a4d24b63168abfbd0d32138ead9f1e053de60a1ac4487b2af87938e7cbd",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            }
        },
        "Options": {}
    }
]

```

What I believe is happenning is, that Docker bridge behaves like a standard networking bridge - it learns which MAC addresses are where and does not send traffic elsewhere. (At least if this page https://en.wikipedia.org/wiki/Bridging\_(networking) is correct) Can Docker Bridge be configured to behave more like a networking hub? Possibly by keeping it's MAC memory empty? Or by continually erasing it?

So in conclusion, is there any way I configure docker, or container C, for C to listen to the traffic?

# Answer
Turns out the answer is iptables, by adding a simple rule:

```
sudo iptables -I FORWARD -s <source ip> -j TEE --gateway <destination ip>

```

e.g.:

```
sudo iptables -I FORWARD -s 172.19.0.3 -j TEE --gateway 172.19.0.4

```

I can clone the traffic, so if I do this:

```
sudo iptables -I FORWARD -s <Container1 IP> -j TEE --gateway <Wireshark Container IP>
sudo iptables -I FORWARD -s <Container2 IP> -j TEE --gateway <Wireshark Container IP>

```

I will clone all traffic going between Containers1 and 2 to Wireshark container.

(Note that although this solution is a proof of concept, at the time of writing this answer)
> 1 votes

---
Tags: docker, networking
---


# Question
Title: Make container start if docker daemon or host system is restarted
First off, I am new to Docker. I was able to create a container based off an image I had previously pulled, start it, but the container is stopped if the docker daemon or the host system are restarted.

So the question is, what is the next step (after creating a container) to make it persistent so that it is started automatically when the daemon is started or restarted?

Thank you in advance.

# Answer
> 1 votes
You need to add a restart policy to your container.

example:

```
docker run -dit --restart=always hello-world

```

In this example, I am telling docker to always restart the hello-world container no matter what happens (and also run it in the background interactively). If you do a `docker ps`, you will see it keeps restarting. To revert this, you need to update the container restart policy like this:

```
docker update --restart=no <CONTAINER_ID>

```

---
Tags: docker
---


# Question
Title: LAMP in docker must be multiple containers or one?
I newbie in docker and I cant find answer what solution is more "right" for LAMP stack:

1) One "standalone" container that will consist from Apache, PHP, MySQL, etc

2) Special container for each component, build and run project via docker-composer.

Im tried to take a look on foreigh realisations of LAMP in docker, but found both variants. Maybe my question dont have right answer and for each project solution must be individual?

# Answer
> 1 votes
I would go with solution 2: use docker-compose. The idea is that with solution 2 you can easily update mysql for example without having to modify and rebuild your dockerfile. you just have to update the version of the mysql image in the docker compose file.

I found with a quick search on github this repository https://github.com/sprintcube/docker-compose-lamp

Note: for production, use terraform script rather than docker-compose

---
Tags: docker
---


# Question
Title: Is it useful to track dev time with CI/CD?
Estimating is hard, but it is made easier with good record keeping.
Is there a known solution to tracking dev time on a ticket by making use of CI/CD to update JIRA tickets or the like with logged worked time, based on the time difference between first commit and final CD?

# Answer
You can use the rich jira rest API for this kind of automation work. Jira API provides mechanism to update logged hours using **PUT /rest/api/2/issue/{issueIdOrKey}/worklog/{id}**

You can use this rest API endpoint to update the corresponding issue with the difference of first commit timestamp and first successful CD.

Find more about it here.
> 2 votes

---
Tags: cicd, capacity-planning, jira
---


# Question
Title: How can I expose port 3389 using a secure approach
Within our legacy infrastructure we need to keep port 3389 exposed so that we can remote desktop into this server (note that we can't go back and utilize IaC to prevent this).

We currently restrict the IP range that can access this port. Is this enough for security? This article mentions never globally exposing this port: https://www.grc.com/port\_3389.htm

# Answer
> 2 votes
You can add, if it's not already the case, a frontal, rev proxy, to prevent direct access to the TS. This could also add an authentication layer... I am not a lot in Microsoft stuff so this is really a basic advice. If TS allow the use of pubkey instead of passwd this would be a good addition too.

---
Tags: amazon-web-services, security, ports
---


# Question
Title: Docker: subnet clients can't ping other subnet
Two containers are created with this `Dockerfile`:

```
FROM ubuntu:18.04

RUN apt update && apt upgrade -y && apt install -y vim iputils-ping net-tools tcpdump iproute* iptables traceroute

```

The were created with the following commands in a `Ubuntu 18.04` machine:

```
sudo docker network create --internal --subnet=192.168.2.0/24 netR

sudo docker network create --internal --subnet=172.16.1.0/24 netB1

sudo docker run --name R1_ipsec --hostname R1 --privileged --cap-add=NET_ADMIN --net netR --ip="192.168.2.100" -dit image1
sudo docker run --name R2_ipsec --hostname R2 --privileged --cap-add=NET_ADMIN --net netR --ip="192.168.2.200" -dit image1

sudo docker network connect --ip="172.16.1.254" netR R1

sudo docker run --name PC1_ipsec --hostname PC1_ipsec --privileged --cap-add=NET_ADMIN --net netB1 --ip="172.16.1.2" -dit image1

sudo docker exec PC1 route delete default gw 172.16.1.1

sudo docker exec PC1 route add default gw 172.16.1.254 dev eth0

```

The problem: `PC1` can't ping the `R1`'s interface `192.168.1.100`, even if `R1` have `net.ipv4.ip_forward = 1`

Doker version: `Docker version 18.06.1-ce, build e68fc7a`

# Answer
The problem was in this line:

```
sudo docker network create --internal --subnet=192.168.2.0/24 netR

```

The `--internal` option makes the network unreachable from other Docker networks.

The solution was to remove this option.
> 0 votes

---
Tags: docker, networking
---


title: Failed to start Container after reboot

text:
# Question
Title: Failed to start Container after reboot
I am hosting an own registry.
After rebooting the Server my registry container is unable to start.

I used this command to start the registry

```
docker run -d -p 5000:5000 --name registry -v /var/lib/registry/:/var/lib/registry -v /root/certs:/certs -v /root/auth/:/auth -eEGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_ADDR=0.0.0.0:5000 -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key -e "REGISTRY_AUTH=htpasswd" -e "REGISTRY_AUTH=htpasswd" -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -e "REGISTRY_AUTH_HTPASSWD_REALM=Registry Realm" --restart always registry:2.7.1

```

After a reboot i get this message when i try "docker start registry":

> Error response from daemon: OCI runtime create failed: container with id exists: dfb0bef21bdfc8a89b59498befd37f83513e75527c0beb552e0400df2a2b7c7d: unknown Error: failed to start containers: registry

Starting a new container works fine.
How can a fix, it and sould the container not start by itself because of "--restart alway"

```
docker --version

```

Docker version 18.09.6, build 481bc77

Thx for your help.

### Update

Intesting news I have written an init script to to the job, but the problem is exactly the same. The container exists but isn't started. If I try to start it, I get the error message from above.

On the boot screen in get this information.

So the docker daemon seam to be not ready.
Do you have any suggestions why?

# Answer
> 1 votes
Try to check is `registry` really stopped using `docker ps -a` if container runned stop container then start.

Part related to the `--restart always` option, restart always will proceed only if container works at least 10 seconds and docker itself can monitor this service.

See more info at this link restart policy detail

---
Tags: docker, docker-registry
---


# Question
Title: How to integrate firebase functions into devops pipeline
I have a project which has `firebase functions`

From my local I can run `firebase deploy --only functions` and it will build and deploy firebase functions to firebase.

Now I want to enable CICD via Azure Devops. So I need to build the functions in CI and get the build artifact in CD to deploy it to different firebase project (as different environments).

The problem is building and deploying is done in the same time via `firebase deploy`

I googled a lot not able to find a solution

# Answer
I found the solution.

As build artifact you need to publish, `firebase.json`, `.firebaserc` and whole `functions` folder including `package.json` (without `node_modules`)

Then in release using `sudo firebase deploy --token $(firebase-token) --project $(firebase-project) --only functions --force`
> 1 votes

---
Tags: azure-devops, azure, google-cloud-platform, cicd
---


# Question
Title: Reset time vs alerting window size in alerting strategies for SLOs
In https://landing.google.com/sre/workbook/chapters/alerting-on-slos/#pros\_and\_cons\_of\_alerting\_based\_on\_burn\_r I don't understand how the reset time can be 58 min when the alerting window size is one hour.

How can the reset time be shorter than the alerting window size?

# Answer
> 1 votes
Consider a service using the alerting rule from the book:

`job:slo_errors_per_request:ratio_rate1h{job="myjob"} > 36 * 0.001`

The burn rate is the rate at which we are exhausting our error budget. A burn rate of 1 means that we'll have exactly exhausted our error budget at the end of our reporting period. A burn rate of 2 means that we'll have exhausted our budget half way through our reporting period.

For a 99.9% SLO, an error rate of 0.1% constitutes a burn rate of 1. An error rate of 1% means a burn rate of 10, and a 100% error rate means a burn rate of 1000

If we want to alert when we've exhausted 5% of our 30 day error budget in 1 hour, then that means we are burning error budget at a rate of 36, which translates to an error rate of 3.6% for a 99.9% SLO.

Imagine a service with a constant rate of requests, that starts serving 100% errors at t=0 minutes

Here our SLI counts the total number of "events".

SLIs written in terms of some kind of discrete events, where an event can either be good or bad, have many nice properties: They range from 0-100%, where 0 is "nothing is working" and 100% is "nothing is broken"; I find this quite intuitive. By making all of your SLIs behave the same, you simplify tooling, alerting rules, and understanding.

With a request-based system, events are usually individual requests. So 0 errors, 1 request in minute at t=-2. I used 1 request per minute for simplicity. The result is the same if you have 1 request per minute or 1 million; it's the ratio of errors to total requests that matters.

t=-2, errors=0, total=1
t=-1, errors=0, total=1
t=0, errors=1, total=1
t=1, errors=1, total=1
...

If our monitoring system evaluates its rules every minute, we get:

at t=-1, our 1h average error rate is 0
at t=0, the 1h error rate is 1/60 = 1.666%
at t=1, the 1h error rate is 2/60 = 3.333%
at t=2, the 1h error rate is 3/60 = 5%

Our alert fires 2 minutes after the event started.

Let's assume our incident continues until t=60.

at t=60, our 1h error rate is 100%: a burn rate of 1000, for a 99.9% SLO
at t=61, the 1h error rate drops to 59/60 = 98.333%
at t=62, the 1h error rate is 58/60 = 96.666%
at t=118, the 1h error rate is 2/60 = 3.333%

At this point, our 1h error rate is below the alerting threshold, and so the alert is no longer in the firing state, 58 minutes after the end of the event.

at t=120, our 1h error rate finally drops back to 0%, as no errors have been observed in the last hour.

---
Tags: monitoring, cloud, sre
---


title: GCE metadata not defined on Travis CI

text:
# Question
Title: GCE metadata not defined on Travis CI
I have tried initializing terraform code in Travis CI but an error keeps coming up;

```
Failed to load state:
Failed to open state file at
gs://{{project_id}}/terraform/state/default.tfstate:
Get
https://storage.googleapis.com/{{project_id}}/terraform/state/default.tfstate:
metadata: GCE metadata "instance/service-accounts/default/token" not defined

```

I have initialized gcloud(Google Cloud SDK), with a service account ie

```
gcloud auth activate-service-account --key-file {{service-account-path}}

```

and set:

```
export TF_CREDS={{service-account-path}}

```

but the error still comes up.

My terraform provider and backend are in the following format respectively:

```
provider "google" {
  region  = "${var.region}"
  zone    = "${var.zone}"
  project = "${var.project_id}"
}

```

```
terraform {
  backend "gcs" {
    bucket = "{{project_id}}"
    prefix = "terraform/state"
  }
}

```

I removed the default Google Cloud SDK thinking that it was causing a conflict when another which is specific to the usage in the CI/CD process is installed. This, however, did not help.

The .travis.yml looks like:

```
language: node_js
sudo: true
node_js:
  - "12.2.0"
  - "12.0"
  - "11.0"
  - "10.0"
  - "9.10.0"

services:
  - docker

stages:
  - name: test
  - name: test-build
  - name: build
    if: |
      tag =~ /^v\d+\.\d+(\.\d+)?(-\S*)?$/ || \
      tag =~ /^dev\/v\d+\.\d+(\.\d+)?(-\S*)?$/ || \
      tag =~ /^release\/v\d+\.\d+(\.\d+)?(-\S*)?$/ || \
      tag =~ /^prod\/v\d+\.\d+(\.\d+)?(-\S*)?$/
  - name: deploy
    if: |
      tag =~ /^v\d+\.\d+(\.\d+)?(-\S*)?$/ || \
      tag =~ /^dev\/v\d+\.\d+(\.\d+)?(-\S*)?$/ || \
      tag =~ /^release\/v\d+\.\d+(\.\d+)?(-\S*)?$/ || \
      tag =~ /^prod\/v\d+\.\d+(\.\d+)?(-\S*)?$/

install:
  - yarn install

script:
  - yarn list
  # - yarn check
  # - yarn test

jobs:
  include:
    - stage: test-build
      install:
        - yarn install

      script:
        - yarn run build

    - stage: build
      install:
        - docker -v
      before_script:
        - git clone https://$GITHUB_TOKEN@github.com/{{github-user}}/devops_configurations.git ../devops_configurations/
        - cp ../devops_configurations/React/* .
      script:
        - docker login -u $DOCKER_HUB_USER -p $DOCKER_HUB_PASSWORD
        - docker build -t $DOCKER_HUB_USER/test_react:$(echo $TRAVIS_TAG | tr "/" .) -f Dockerfile-multibuild .
        - docker push $DOCKER_HUB_USER/test_react:$(echo $TRAVIS_TAG | tr "/" .)
        - docker rmi $DOCKER_HUB_USER/test_react:$(echo $TRAVIS_TAG | tr "/" .)
        - docker logout

    - stage: deploy
      install:
        # Install Google Cloud SDK
        - if [ ! -d "$HOME/google-cloud-sdk/bin" ]; then
            rm -rf $HOME/google-cloud-sdk;
            sudo rm -rf /usr/lib/google-cloud-sdk/;
            export CLOUDSDK_CORE_DISABLE_PROMPTS=1;
            curl https://sdk.cloud.google.com | bash > /dev/null;
          fi
        # Add gcloud to $PATH
        - source /home/travis/google-cloud-sdk/path.bash.inc
        # Install Kubectl
        - if ! [ -x "$(command -v kubectl)" ]; then
            gcloud components install kubectl --quiet;
          fi
        # Install Terraform
        - if ! [ -x "$(command -v terraform)" ]; then
            TERRAFORM_VERSION=0.12.0;
            URL="https://releases.hashicorp.com/terraform/${TERRAFORM_VERSION}/terraform_${TERRAFORM_VERSION}_linux_amd64.zip";
            wget $URL;
            unzip terraform_${TERRAFORM_VERSION}_linux_amd64.zip -d executables/;
            sudo mv executables/terraform /usr/local/bin/terraform;
            sudo chmod +x /usr/local/bin/terraform;
            rm terraform_${TERRAFORM_VERSION}_linux_amd64.zip;
            rm -rf executables/;
          fi
      before_script:
        - git clone https://$GITHUB_TOKEN@github.com/{{github-user}}/devops_configurations.git ../devops_configurations/
        - cd ../devops_configurations/terraform/
        - mkdir {{service-account-folder}}
        - echo $GCP_MAIN_SERVICE_ACCOUNT | base64 --decode > {{service-account-path}}
        - gcloud auth activate-service-account --key-file {{service-account-path}}
        - export TF_CREDS={{service-account-path}}
      script:
        - terraform init
        - terraform plan -var-file variables.tfvars
          -var="react_docker_image=$DOCKER_HUB_USER/test_react:$(echo $TRAVIS_TAG | tr "/" .)"
          -var="vue_docker_image=$(terraform output terraform_gcp_gke_project_id || echo '$DOCKER_HUB_USER/test_vue:dev.v0.0.1-test')"
          -var="angular_docker_image=$(terraform output terraform_gcp_gke_project_id || echo '$DOCKER_HUB_USER/test_angular:dev.v0.0.1-test')"
          -target kubenetes_deployment.react-deployment
        - terraform apply -var-file variables.tfvars
          -var="react_docker_image=$DOCKER_HUB_USER/test_react:$(echo $TRAVIS_TAG | tr "/" .)"
          -var="vue_docker_image=$(terraform output terraform_gcp_gke_project_id || echo '$DOCKER_HUB_USER/test_vue:dev.v0.0.1-test')"
          -var="angular_docker_image=$(terraform output terraform_gcp_gke_project_id || echo '$DOCKER_HUB_USER/test_angular:dev.v0.0.1-test')"
          -target kubenetes_deployment.react-deployment
      after_script:
        - rm -rf {{service-account-folder}}
        - unset TF_CREDS

```

Any help would be appreciated.

# Answer
Setting

```
export GOOGLE_APPLICATION_CREDENTIALS=${TF_CREDS}

```

solved the issue.
> 5 votes

---
Tags: terraform, google-cloud-platform, travis-ci
---


# Question
Title: Google Cloud search resources in organizations
I am new to Google cloud platform and still learning. I am interested in organizations, where I will have multiple accounts in multiple regions. Is there a way I can query resources based on some configuration parameters across all my accounts? Example - list all Virtual machines which are in US West region, micro size and currently stopped.

# Answer
"Multiple accounts in multiple regions" is an AWS idea which is really antithetical to how things are done in Google Cloud. GCP gives you a lot of tools to slice-and-dice your systems, but none of them work in the "multiple accounts/multiple regions" paradigm.

But if you insist on going that route, your best bet may be to create custom bash scripts which harness `gcloud`, the CLI for GCP. It is a pleasure to use, and could probably do any sort of automation, even across accounts, that you have in mind.
> 1 votes

---
Tags: google-cloud-platform
---


# Question
Title: Why isn't my S3 Cross Region Replication working for some files?
I've followed the documentation to set up cross-region replication to another AWS account. It looked like it was working correctly until I noticed an entire prefix was missing.

Here is my destination bucket policy:

```
{
    "Version": "2008-10-17",
    "Id": "S3-Console-Replication-Policy",
    "Statement": [
        {
            "Sid": "S3ReplicationPolicyStmt1",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::<awsaccount>:root"
            },
            "Action": [
                "s3:GetBucketVersioning",
                "s3:PutBucketVersioning",
                "s3:ReplicateObject",
                "s3:ReplicateDelete",
                "s3:ObjectOwnerOverrideToBucketOwner"
            ],
            "Resource": [
                "arn:aws:s3:::bucket-backup",
                "arn:aws:s3:::bucket-backup/*"
            ]
        }
    ]
}

```

Looking further into it, I have also realized this particular prefix is where we store our public-access data, so maybe that's related somehow?

# Answer
> 2 votes
So I verified it was definitely public access causing the issue somehow; uploading an identical file but with the public read acl on totally broke it.

I worked with AWS support and they noticed I had "Block public access" enabled in the destination bucket. Apparently rather than replicating the object and then making it private, it just fails replication.

tl;dr if you want to replicate objects that are public read, don't block public read at the destination.

---
Tags: amazon-web-services, amazon-s3
---


# Question
Title: What is the difference between RUN and CMD in a Dockerfile?
I am going to change my permission of files to `600`. The following command works perfectly.

```
RUN /bin/bash -c 'chmod 600 /root/.ssh/config'

```

But this one failed

```
CMD /bin/bash -c 'chmod 600 /root/.ssh/config'

```

In the beginning, I thought the shell environment isn't `bash` so caused the error. However, adding `bin/bash` didn't solve the issue.

# Answer
> 3 votes
Note: Don’t confuse RUN with CMD. RUN actually runs a command and commits the result; CMD does not execute anything at build time, but specifies the intended command for the image.

reference

# Answer
> 3 votes
TL;DR: `RUN` performs the step during the image build, `CMD` is the default command when you start a container. If you define `CMD` and then start your container with a different command (e.g. `docker run $my_image /bin/bash`), you'll never see it run since you replaced your `chmod` command with an empty `/bin/bash` command.

---

The detailed explanation:

The **`RUN`** command creates a temporary container based off of the previous step of the build, executes your selected command, and when the command returns it captures the filesystem changes as a new layer of your image. Run is the equivalent of doing a `docker run` \+ `docker commit` to generate your image, but with a reproducible process.

The **`CMD`** command replaces the current value of the image metadata for the `command` value that you can see when you inspect the image. You can only have one value for `CMD` for your image. You can also easily override the value of the image command by passing any string after the image name, e.g. `docker run busybox echo hello` replaces the default `/bin/sh` command with an `echo hello` command for that specific container.

There's also **`ENTRYPOINT`** which defines the entrypoint for your image. When you define an entrypoint, any value for `CMD` gets passed as arguments to the entrypoint when starting your container (a container is only started with a single process as pid 1 and exits when that process exits).

---

**Shell vs Exec syntax:** Note that each of these accepts a command to run in two forms, the shell/string form, and the exec/json form. The shell form is anything that isn't a json list, and the result command is run as `/bin/sh -c "$your_string_here"`. A shell is useful for IO redirection, chaining commands, and variable expansion. But a shell will also intercept signals which can cause issues when the container is stopped. The exec form runs your command directly in the OS without a shell, which is better for signals, but lacking all the other shell features. When you chain an entrypoint and cmd together, you almost always want the exec syntax since `/bin/sh -c` only accepts one argument. It's common to run a shell script but have the last command run be an `exec` which replaces the shell. E.g. a common entrypoint structure is:

Dockerfile:

```
ENTRYPOINT ["/entrypoint.sh"]
CMD ["server", "arg1", "arg2"]

```

entrypoint.sh:

```
#!/bin/sh
# ... steps to setup the host, make configs, fix permissions, copy files, etc ...
# ...
# now run the value of CMD, replacing this shell as pid 1:
exec "$@"

```

---
Tags: dockerfile
---


# Question
Title: How to get dependency from file share in Azure DevOps pipeline
We are trying to set up a CI pipeline for building several Docker containers and deploying them to a container registry using Azure DevOps. Our code, the Docker and configuration files all reside in a Azure git repository and building those containers just works fine.

However, we have to build one container around a tomcat app that we only have as a .war file that is approx. 20Mb in size. We do not want to commit this file to our repository (we are not convinced that using git LFS would solve the problem because of experiences in other projects), but instead keep it on an azure storage account (either blob or file share storage).

My problems start when I want to include a task to download this file into the build environment as a task in the build pipeline. Getting a file that is openly accessible is easy with either a powershell or bash script. However, in order to download from the storage account you first need to authenticate and I can not find a good way to achieve this.

# Answer
> 1 votes
Steps for download are described here: How To Download Azure BLOB Storage Using Azure PowerShell. According to this article, you have to use the connection string from the Access keys page. You can create your custom secret variable (Variables) and save a connection string in your new variable. Then in your ps scripts you can reference to connection string (stored in variable) with "$env:VARIABLE\_NAME".

---
Tags: azure-devops, azure-pipelines
---


title: How to downgrade Jinja for Ansible?

text:
# Question
Title: How to downgrade Jinja for Ansible?
I'm using Ansible 2.8.0 (`ansible --version` installed by `brew` on macOS) which is using Jinja 2.10.

I've tried to downgrade Jinja to 2.8 via `pip3 install jinja2==2.8`, and I can confirm the right version with `pip3 list`, but Ansible is still using Jinja 2.10 (verified by these steps). It seems it's using different Python's `site-packages` folder.

I'd like to downgrade Jinja due to a bug, so I'd like to test Ansible using older version of Jinja.

How do I downgrade Jinja used by Ansible, ideally without downgrading Ansible it-self?

---

```
$  ansible --version
ansible 2.8.0
  config file = /Users/kenorb/.ansible.cfg
  configured module search path = ['/Users/kenorb/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/Cellar/ansible/2.8.0/libexec/lib/python3.7/site-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.7.3 (default, Mar 27 2019, 09:23:15) [Clang 10.0.1 (clang-1001.0.46.3)]
$  pip3 --version
pip 19.0.3 from /usr/local/lib/python3.7/site-packages/pip (python 3.7)
$  brew info ansible
ansible: stable 2.8.0 (bottled), HEAD

```

# Answer
> 1 votes
Since Ansible was installed via Homebrew on macOS, the workaround is to install Ansible using Pip.

```
$ brew remove ansible
$ pip3 install ansible
$  pip3 list | grep -e ansible -e Jinja2
ansible      2.8.0
Jinja2       2.8
$  ansible-playbook check_jinja.yaml -v
TASK [jinja_version]
ok: [localhost] => {"changed": false, "msg": "2.8"}

```

---
Tags: ansible, python, jinja, macos
---


# Question
Title: Artifacts mentioned in secondary-artifacts section not getting uploaded to s3 bucket
I have created a codePipeline project through which my application get build using maven and it's artifacts are uploaded to s3. I'm trying to upload multiple artifacts as a single zip file to s3 bucket through codebuild. Below is my sample buildspec.yml file

```
version: 0.2

phases:
  install:
    runtime-versions:
      java: openjdk8
    commands:
      - apt update -y && apt install maven -y
      - echo Install phase started on `date`
      - echo Install phase ended on `date`
  pre_build:
    commands:
      - echo Pre_build phase started on `date`
      - bash pipelines/aws/scripts/onCheckout.sh
      - echo Pre_build phase ended on `date`
  build:
    commands:
      - echo Build phase started on `date`
      - bash pipelines/aws/scripts/onBuild.sh
      - echo Build phase ended on `date`
  post_build:
    commands:
      - echo Post_build phase started on `date`
      - bash pipelines/aws/scripts/onPackage.sh
      - bash pipelines/aws/scripts/onPublish.sh
      - bash pipelines/aws/scripts/onPromote.sh
      - echo Post_build phase ended on `date`
      - echo $CODEBUILD_SRC_DIR
artifacts:
  files:
    - pipelines/**/*
    - target/spring-petclinic-2.0.0.jar
  base-directory: $CODEBUILD_SRC_DIR
secondary-artifacts:
  artifact1:
    files:
      - pipelines/aws/appspec.yml
    base-directory: $CODEBUILD_SRC_DIR
    discard-paths: yes

```

codebuild-output:

```
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Phase complete: POST_BUILD State: SUCCEEDED
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Phase context status code: Message:
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding base directory path: $CODEBUILD_SRC_DIR
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Assembling file list
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding /codebuild/output/src289881682/src
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding artifact file paths for base directory /codebuild/output/src289881682/src
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Assembling file list
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding pipelines/**/*
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Expanding target/spring-app-2.0.0.jar
https://forums.aws.amazon.com/ 2019/05/24 12:00:27 Found 52 file(s)
https://forums.aws.amazon.com/ 2019/05/24 12:00:28 Phase complete: UPLOAD_ARTIFACTS State: SUCCEEDED
https://forums.aws.amazon.com/ 2019/05/24 12:00:28 Phase context status code: Message:

```

Artifacts mentioned in secondary-artifacts are completely ignored due to unknown reasons, Am I missing any parameters?

# Answer
In the CodeBuild "Artifacts" configuration, there is an "Add Artifact" button.

You need to click that and reference your secondary artifact to get it to upload to S3 as well.
> 3 votes

---
Tags: amazon-web-services, amazon-s3, aws-codepipeline
---


title: Can't get P3 spot instance on AWS

text:
# Question
Title: Can't get P3 spot instance on AWS
I'm new to aws-ec2.

I'd like to make spot request for a `p3.2xlarge` instance in my local region (EU-Ireland) to benchmark V100 Tesla GPUs against my home GPU for deep learning.

I have already requested and have been granted a limit increase for 1 instance of this type in my region.

I fill in the usual spot request application form changing only the instance type to `p3.2xlarge (8 vCPU, 61 GiB EBS only)` and the security group to `SSH` for logging in via my IP.

I can see the instance p3.2xlarge instance type listed among the instance types in the fleet request settings.

`p3.2xlarge 8 vCPUs 61GiB $0.9915/hr 70 %`

`p3.2xlarge 8 vCPUs 61GiB $0.9915/hr 70 %`

`r3.4xlarge 16 vCPUs 122GiB $0.3019/hr 80 %`

`r5d.2xlarge 8 vCPUs 64GiB $0.1512/hr 76 %`

`r5a.4xlarge 16 vCPUs 128GiB $0.3023/hr 70 %`

`r3.2xlarge 8 vCPUs 61GiB $0.159/hr 79 %`

`r4.2xlarge 8 vCPUs 61GiB $0.1903/hr 68 %`

`i3en.2xlarge 8 vCPUs 64GiB $0.3/hr 70 %`

`m4.4xlarge 16 vCPUs 64GiB $0.3056/hr 66 %`

I submit and when the request is fulfilled, I can see the instance type that was assigned is an `r5d.2xlarge` not a `p3.2xlarge`. This instance doesn't even have an Nvidia graphics card and is therefore useless for my comparison.

How do I make a spot request for a `p3.2xlarge` that will be filled by the same instance type as I have requested?

# Answer
*Spot Fleet* aims to meet your capacity requirements across a range of instances. This reduces the risk of any unavailability of a specific instance type or family affecting your workload.

As you are interested in only one specific machine type, you will need to remove the other types from your spot fleet request; or just create a separate spot instance request for the `p3.2xlarge` (a plain spot instance request, not a spot fleet request)
> 1 votes

---
Tags: amazon-web-services, amazon-ec2
---


title: Find appropriate AWS policy

text:
# Question
Title: Find appropriate AWS policy
I have created a user group in AWS ("developers"). Now I want to attach a policy to the group which will grant the users permission to list, create and update Amplify apps. I am struggling to determine the appropriate policy.

I see that the "AdministratorAccess" policy includes access to the *AWS Amplify* service, but I don't want the developers to have administrator level access. What are my options?

Edit: When the specific user accesses the Amplify console the following error is displayed

```
List apps call failed User: arn:aws:iam::XXXXXXXXXX:user/myUser is not authorized to perform: amplify:ListApps on resource: arn:aws:amplify:eu-west-1:XXXXXXXXXX:apps/*

```

# Answer
If you want to create your own IAM policy, you can use the visual editor present in IAM. Using visual editor, you can construct a new policy from scratch using the very intuitive GUI.

Goto **IAM**, then **policies**, then use **visual editor**.
> 1 votes

# Answer
I had to log in as root user, then access the IAM service.

Under policies click "Create Policy", then select "Amplify" under the Select a service.
> 0 votes

# Answer
This link detail how we can create role for providing access to applications in aws and you can grant these role to your group .

https://docs.aws.amazon.com/IAM/latest/UserGuide/id\_roles\_use\_switch-role-ec2.html

Another way you can create your own policy and assign to group .

https://docs.aws.amazon.com/IAM/latest/UserGuide/access\_policies\_create.html
> -1 votes

---
Tags: amazon-web-services, aws-iam
---


title: Deploying a pod per REST request?

text:
# Question
Title: Deploying a pod per REST request?
Is it possible to deploy 1 pod per each REST request? My process starts up in .004s and is about 1GB in size. After the REST request completes the pod will terminate.

Given that I'd like to deploy the pod/container for this on each REST request. Is it possible to do this in \< 200ms? A sort of serverless architecture using kubernetes.

How would you do this in a reliable / resilient way?

# Answer
As per my experience, it is not a good idea because the availability of a pod is not directly related to how fast the container is up, a lot of other things affects the time the pod is ready and the time kubernetes sends traffic to that new pod.

You need to think about how your scalability strategy is trying to adapt to your actual traffic and the different health checks change that readiness time.

In Kubernete you have the livenessProbe and radynessProbe used to check two different thinks on the lifecycle of the pod, because of that you need to review the lifecycle of a pod in kubernetes and if your app can adapt to it.

https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/

In case you are worried about the number of requests you can handle per second my advice is to have a pool of pods ready to response a good amount of it and take metrics to know better how your app responds to a high demanding traffic context and understand how the loadbalancing of kubenetes work and can affect your app, with that try to scale in a more smart way.
> 0 votes

---
Tags: kubernetes
---


title: Ansible equivalent of below sed command

text:
# Question
Title: Ansible equivalent of below sed command
How can i use ansible lineinfile module to simulate working of below sed command.

```
sed -i _bkp '/(ADDRESS = (PROTOCOL = TCP)(HOST = 10.0.0.1)(PORT = 1501))/d;s/(ADDRESS = (PROTOCOL = TCP)(HOST = 10.0.0.2)(PORT = 1501))/(ADDRESS = (PROTOCOL = TCP)(HOST = my-site.com)(PORT = 1501))/;/(BALANCE = yes))/d' myconffile

```

# Answer
> 1 votes
To replace all lines in the file replace module is needed. This is the equivalent (backup differs).

```
  tasks:
    - lineinfile:
        path: myconffile
        backup: yes
        state: absent
        regexp: '\(ADDRESS = \(PROTOCOL = TCP\)\(HOST = 10.0.0.1\)\(PORT = 1501\)\)'
    - replace:
        path: myconffile
        backup: yes
        regexp: '\(ADDRESS = \(PROTOCOL = TCP\)\(HOST = 10.0.0.2\)\(PORT = 1501\)\)'
        replace: '(ADDRESS = (PROTOCOL = TCP)(HOST = my-site.com)(PORT = 1501))'
    - lineinfile:
        path: myconffile
        backup: yes
        state: absent
        regexp: '\(BALANCE = yes\)'

```

The simplified version is below.

```
  tasks:
    - lineinfile:
        path: myconffile
        backup: yes
        state: absent
        regexp: "{{ item }}"
      loop:
        - 'AAA'
        - 'DDD'
    - replace:
        path: myconffile
        backup: yes
        regexp: 'BBB'
        replace: 'CCC'

```

---
Tags: ansible
---


# Question
Title: How to retrigger a Jenkins build after failure using declarative syntax?
I am using declarative pipeline syntax. I want to re trigger my build for 3 times if it fails. I want to add this step in my pipeline may be in post step: Something like below:

```
post {
    failure{
        retrigger //try for 3 times atleast
    }
}

```

# Answer
The other answer is incorrect. There is indeed a builtin to retry arbitrary sections of your job called `retry`.

If you want to retry the whole job three times, you can wrap your entire job in a `retry` block:

```
retry(count: 3) {
  // your job definition here
}

```

However, if it's safe to do so, I would recommend wrapping individual steps or stages instead:

```
stage('my first stage') {
  retry(count: 3) {
    sh('some-command')
    sh('some-other-command')
  }
}

```
> 9 votes

# Answer
There is a retry parameter in the global options when using declarative pipeline :

```
pipeline {
    agent any
    options {
        retry(3)
    }
    stages {
        ...
    }
}

```
> 3 votes

# Answer
There doesn't seem to be a built in pipeline step to do this, and neither does there seem to be a existing plugin that works with pipelines. I'd recommend having a look at the stackoverflow answer: https://stackoverflow.com/a/46852240/1019835

You can implement that code as a shared library, which would allow you to share your solution between projects. Have a look at this tutorial to do that: https://jenkins.io/doc/book/pipeline/shared-libraries/
> -1 votes

---
Tags: jenkins-pipeline, jenkinsfile
---


title: Building a CI/CD pipeline on GCP

text:
# Question
Title: Building a CI/CD pipeline on GCP
I have a containerized web application on GitHub and I would like to build a CI/CD pipeline on GCP consisting of a trigger stage, a build stage and a deployment stage and be able to do continuous integration and deployment. What GCP tools can I use for this task? I have successfully accomplished the same task on AWS with Jenkins, where my GitHub commit triggers a multi-stage pipeline and deploy the app to EC2.

This is what I have been able to do on AWS:

# Answer
> 0 votes
For building use Google Cloud build. For deployment you can you use Google Cloud Deploy. https://cloud.google.com/deployment-manager/docs/ https://cloud.google.com/cloud-build/docs/

---
Tags: docker, continuous-integration, automation, google-cloud-platform, gcp
---


# Question
Title: What is the optimal size of a Jenkins Pipeline Stage?
Should I create single `checkout` stage or multiple stages like `checkout scripts`, `checkout code`, `checkout something else`?

How should I decide on what piece of logic to put into separate stage? Or are stages just a pipeline visualization tool?

# Answer
> 3 votes
Single responsibility principle (SRP)

> Martin defines a responsibility as a reason to change, and concludes that a class or module should have one, and only one, reason to be changed (i.e. rewritten). As an example, consider a module that compiles and prints a report. Imagine such a module can be changed for two reasons. First, the content of the report could change. Second, the format of the report could change. These two things change for very different causes; one substantive, and one cosmetic. The single responsibility principle says that these two aspects of the problem are really two separate responsibilities, and should therefore be in separate classes or modules. It would be a bad design to couple two things that change for different reasons at different times.
>
> The reason it is important to keep a class focused on a single concern is that it makes the class more robust. Continuing with the foregoing example, if there is a change to the report compilation process, there is greater danger that the printing code will break if it is part of the same class.

5-15 lines

> Functions should normally be short, between 5-15 lines is my personal "rule of thumb" when coding in Java or C#. This is a good size for several reasons:
>
> * It fits easily on your screen without scrolling
> * It's about the conceptual size that you can hold in your head
> * It's meaningful enough to require a function in its own right (as a standalone, meaningful chunk of logic)
> * A function smaller than 5 lines is a hint that you are perhaps breaking the code up too much (which makes it harder to read / understand if you need to navigate between functions). Either that or your're forgetting your special cases / error handling!

**Personal view**

I have the SRP in mind when creating Jenkins stages. This implies that I create a separate stage for checkout, build and publish for example. If a stage becomes too large, i.e. longer than 15 lines, then I create a script and run the script in this stage. One of the benefits is readability, possible to apply unit testing on the script and CI agnostic, i.e. migration to another CI would be possible as well.

Example

> ```
> pipeline {
>     agent any
>
>     stages {
>         stage('Build') {
>             steps {
>                 echo 'Building..'
>             }
>         }
>         stage('Test') {
>             steps {
>                 echo 'Testing..'
>             }
>         }
>         stage('Deploy') {
>             steps {
>                 echo 'Deploying....'
>             }
>         }
>     }
> }
>
> ```

# Answer
> 1 votes
In addition to being a pipeline visualization tool, stages also identify portions of the pipeline that could be executed more or less independently from each other, potentially on different workers. Do not underestimate the importance of choosing the right pipeline structure.

Stages can be re-executed if/when needed, for example in case of failures caused by environment issues, without having to re-run the already completed upstream stages.

Stages provide ability for pipeline branching. For example a single build stage can produce artifacts that could be used by multiple downstream test stages than can be executed in parallel, for shorter overall pipeline execution time.

Each stage operation can be supported by pools of workers of different types and sizes.

Stages can be conditionally executed. For example an expensive integration test stage (lengthy and/or with a smaller resource pool than the build stage) could be executed only once in a while rather than for every commit, or only on a release branch and not on the primary development branch.

Since generally the intent is to have the pipeline triggered by each and every commit in the respective branch the size of the pipeline - i.e. the number of stages, their duration and the structure of the pipeline graph (it doesn't necessarily have to be linear) - has a potentially complex relationship with:

* the rate and profile of the commits triggering the pipelines
* the pool(s) of resources available to support the operation of the pipeline stage(s)
* the role of the branch, the associated balance between the quality level considered mandatory and its mapping onto the verification pipeline structure (i.e. which stage failures are considered blocking, requiring backout/rollback of the respective changes vs. those for which it's acceptable to just file issues to be fixed down the road)
* the amount of additional effort/resources necessary to identify the culprits for and fix the blocking failures detected by the pipeline executions, to complete the (continuous) integration/delivery/deployment process

In smaller scale projects things are usually simple, but in larger scale ones these aspects require serious thinking as they can have a huge impact on the overall development process.

I'll use your first question as an example:

* if you have a dozen of repositories which combined take less than a minute to checkout/pull - it's probably OK to keep things simple and just pull them all in one step.
* if you have several thousands repositories which may take 1h+ to pull you should probably just pull in each stage what you need for that stage. Or maybe pull them all in the first stage, cache them and re-use that cache in later stages if that helps avoid repeating lengthy pulls.

Whatever makes sense, in your particular project context.

# Answer
> 0 votes
Yes, in my perspective stages are mostly meant for a visualisation purpose, and we need to maintain the clear demarcation between stages. My suggestion to go with all the SCM related checkouts in a single stage itself .

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: How to scale the control plane of a service mesh?
Scalability in the data plane is easy to achieve by simply having more replicas. The configuration the proxies receive from the control plane makes that possible . However, if the number of services (and their sidecar proxies) get too big, the control plane can become a bottleneck by not being able to send metrics or other configuration data fast enough to the sidecars.

Is there a possibility to horizontally scale the control plane of a service mesh so that the total amount of possible services can be increased? I didn't find anything regarding that.

# Answer
> 1 votes
It's true that when you scale your services is possible to have a bottleneck on the monitoring, but for avoiding that you have to check if your monitoring system is pull or push, in case it is push maybe is not the best option but pull is not the right answer for all situations.

One of the most used monitoring systems today is prometheus, you can se how it works here:

https://prometheus.io/docs/introduction/overview/

But is not the only one and you need to see how it should fit in your stack.

Check the possible integrations with istio

https://istio.io/docs/tasks/telemetry/metrics/querying-metrics/

In case you have already some of that probably need to think on scaling this part instead of the sidecars or istio itself.

---
Tags: containers, monitoring, microservices, istio
---


# Question
Title: Why is IF failing with satl state file from Jenkins?
I invoke a saltstack state file "init.sls" from a Jenkins pipeline as below. If `job == Yes`, it is expected to execute `test_command` and test\_final, else `if job == No`, it must execute `test2_command` and `test_final`

```
{% if {{ job }} == 'Yes' %}
test_command:
  cmd.run:
    - name: echo "TEST COMMAND EXEC"
{% elif {{ job }} == 'No' %}
test2_command:
  cmd.run:
    - name: echo "TEST2 COMMAND EXEC"
{% endif %}

test_final:
  cmd.run:
    - name: echo "FINAL SUCCESS"

```

However it is not successful. Is there a better way to achieve using salt state execution?

# Answer
> 1 votes
The jinja syntax error is caused by the `{{` around your `job` var. Those curly braces are not required as you are already inside jinja block, noted by the `{%`.

So to get your code working just replace your `{{ job }}` by `job`. This should be working.

---
Tags: salt
---


# Question
Title: Using the built binary of a docker image in another docker image
# Problem

I have two repositories A and B. I have a docker image registry, where I can push the docker images made of the repositories A and B.

I need only the binary of A's source code to be included in the docker image of B. So when running the docker image of B, the jar file of A is present and can be used. And in case of updates in repository A, the registry image is subsequently updated.

# Question

I've searched now several threads and the docker documentation. Is there a recommended way to make this happen with docker respectivly dockerfile commands?

So far I tried the following and it worked, but it wasn't so intiutive. Is this the recommended way?

```
FROM myregistry/my-A-img AS A
FROM ubuntu-base-img

# Build commands for B ...
COPY build-B.jar /app

# Get the A image from the registry
COPY --from=A /path/to/build-A.jar /app/lib

# Run B binaries

```

# Answer
> 0 votes
What you need is to use the build pattern or the multistage features of docker

https://docs.docker.com/develop/develop-images/multistage-build/

With that, you can use as many images as you need depending on your build process

---
Tags: docker, dockerfile
---


# Question
Title: Automate Helm's tiller installation with Terraform
I need to be able to generate disposable kubernetes environments using `terraform`. These environments should use `helm` for application deployment and ideally have `tiller` installed during the `terraform` resource provision.

What is the best way of automating this? Should I "copy" the deployment generated by `helm init` and translate it to `terraform` lingo? Is there any other solution more elegant than this?

I tried to explicitly declare the `terraform`'s `helm` provider block, but without any real chart deployment it does not bother into installing the `tiller` service.

# Answer
How I see there is two options (from my exp):

1. Use configuration for provider. https://www.terraform.io/docs/providers/helm/index.html, there is option to install tiller.
2. Use prefabricated modules, for example: https://github.com/sagikazarmark/terraform-tiller https://registry.terraform.io/modules/Vidimensional/tiller/kubernetes/0.0.1
> 1 votes

# Answer
I think this is achieveable with `terraform`s `null_resource` (https://www.terraform.io/docs/providers/null/resource.html) with `local-exec`.

Example (untested):

```
resource "null_resource" "helm_init_command" {

  provisioner "local-exec" {
    command = "helm --kubeconfig=<...> init"

}

```
> 0 votes

---
Tags: kubernetes, terraform, helm
---


# Question
Title: Task to delete users without mount points in /home (using register and when)
I have a list of users that I want to delete (del\_users), but I only want to delete those that haven't mountpoint on your /home directory. My currently code is:

```
- name: Verify if exist mount point on /home
  shell: "mount -l | grep bind | grep -ci {{ item }}"
  ignore_errors: yes
  check_mode: False
  loop: "{{ del_users }}"
  register: del_users_reg

- name: Delete user if doesn't exist mount point on /home
  user:
    name: "{{ item }}"
    state: absent
    remove: yes
  when: item.stdout_lines == 0
  loop: del_users_reg

```

When I execute this task appears the next error:

```
TASK [osbasic : Verify if exist mount point on /home] *******************************************************************************************************************************************************************************************
 [WARNING]: Consider using the mount module rather than running 'mount'.  If you need to use command because mount is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid
of this message.

failed: [LGAUSUX10] (item=user1) => {"changed": true, "cmd": "mount -l | grep bind | grep -ci usuario1", "delta": "0:00:00.025089", "end": "2019-06-11 13:33:40.978563", "item": "user1", "msg": "non-zero return code", "rc": 1, "start": "2019-06-11 13:33:40.953474", "stderr": "", "stderr_lines": [], "stdout": "0", "stdout_lines": ["0"]}

TASK [osbasic : Verify if exist mount point on /home] *******************************************************************************************************************************************************************************************
...ignoring

TASK [osbasic : Delete user if doesn't exist mount point on /home] **********************************************************************************************************************************************************************************

TASK [osbasic : Delete user if doesn't exist mount point on /home] **********************************************************************************************************************************************************************************
fatal: [LGAUSUX10]: FAILED! => {"msg": "Invalid data passed to 'loop', it requires a list, got this instead: del_users_reg. Hint: If you passed a list/dict of just one element, try adding wantlist=True to your lookup invocation or use q/query instead of lookup."}
        to retry, use: --limit @/etc/ansible/osbasic.retry

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
LGAUSUX10                  : ok=9    changed=1    unreachable=0    failed=1

```

How could put a condition for delete users only if the result of first task is 0?

# Answer
> 2 votes
Several problems in your playbook.

1. `loop` is basically expecting a list, not a string containing a variable name. You need to expand your variable with a jinja2 expression =\> `loop: "{{ some_list_variable }}"`
2. It will be hard to give you the correct syntax without seeing the exact data structure of your result var but... Typically, the register from a loop contains a `results` list of hashes, each containing info about the ran command for each loop iteration. This is your base loop for the second task =\> `loop: "{{ del_users_reg.results }}"`

If you still have trouble fixing your playbook, update your question with a debug of the registered var and I will update my answer accordingly.

---
Tags: ansible
---


# Question
Title: Terraform | The "count" object can be used only in "resource" and "data" blocks,
I'm trying to upgrade out Terraform seeing as 0.12 (stable) came out recently. I've ran into some issues with a certain pair of resource blocks. I'm getting the below error message when trying to do terraform plan, if I remove the count index it gives me a different error and says that I need the count index but that gives this error:

```
The "count" object can be used only in "resource" and "data" blocks, and only
when the "count" argument is set.

```

Here is the terraform stripped down to only the relevant parts.

```
resource "azurerm_network_interface" "network_interface" {
    count                             = "${var.vm_server_count}"
    location                          = "${var.location}"
    }
}
Problematic resource block

resource "azurerm_network_interface_backend_address_pool_association" "network_interface" {
  network_interface_id    = "${azurerm_network_interface.network_interface[count.index]}"
  ip_configuration_name   = "example"
  backend_address_pool_id = "${azurerm_lb_backend_address_pool.network_interface.id}"
}

```

What am I missing here? This is Terraform 0.12 stable using Azure as the provider.

# Answer
The `resource "azurerm_network_interface_backend_address_pool_association" "network_interface"` block doesn't have the `count` argument set, so `count.index` is not meaningful in that block.

Given the specific pair of resource types you're using here, I expect your intent was to create one association for each network interface, in which case you'd add the following `count` argument to the second block:

```
  count = length(azurerm_network_interface.network_interface)

```

Although this didn't return an error on Terraform 0.11, it did not behave as I expect you intended: it would've created only one `azurerm_network_interface_backend_address_pool_association` for the *first* network interface, and left the others unassociated.

---

If associating only the first one was your intent instead, then replace the `count.index` in the second block with `0` to be explicit that it's using only the first one:

```
  network_interface_id    = azurerm_network_interface.network_interface[0]

```
> 6 votes

---
Tags: terraform, azure, terraform-module
---


# Question
Title: Terraform: Route53, LB and CDN interdependencies
My team uses Terraform to deploy the necessary infrastructure. Currently, our stack is composed of: AutoScalingGroup fronted by a Load-Balancer, and a Route53 DNS record that points to this LB;

We would like to extend the stack to include a Cloudfront distribution that points to the LB, and make the Route53 DNS record point to the CDN.

**I would like to** be able to:

* spin up Cloudfront and change the DNS record to point to the CDN, without causing any downtime;
* tear down the CDN, change the DNS to point to the load-balancer, without causing any downtime;

**The current configuration** is as follows:

```
resource "aws_lb" "alb" {
  name                       = "${var.alb-name}"
  ...
}

```

```
resource "aws_cloudfront_distribution" "cdn" {
  count = "${var.use_cloudfront != "false" ? 1 : 0}"
  ...
}

```

```
resource "aws_route53_record" "www" {
  count = "${var.domain-name != "" ? 1 : 0}"

  depends_on = ["aws_lb.alb", "aws_cloudfront_distribution.cdn"]

  zone_id = "${data.aws_route53_zone.primary.zone_id}"
  name    = "${var.domain-name}"
  type    = "A"

  alias = {
    // we use concat because 'count' makes the response of the resource a list.
    // link to similar issue: https://stackoverflow.com/questions/45654774/terraform-conditional-resource
    name                   = "${var.use_cloudfront == "true" ? element(concat(aws_cloudfront_distribution.cdn.*.domain_name, list("")), 0) : aws_lb.alb.dns_name}"
    zone_id                = "${var.use_cloudfront == "true" ? element(concat(aws_cloudfront_distribution.cdn.*.hosted_zone_id, list("")), 0) : aws_lb.alb.zone_id}"
    evaluate_target_health = true
  }
}

```

**The current behavior** is as follows:

* the spin-up of the CDN, followed by re-routing the DNS record to point to the CDN, does not lead to any downtime;
* **the tear-down of the CDN, followed by re-routing the DNS record to the load-balancer, does cause downtime;**

I am aware that the `depends_on = ["aws_lb.alb", "aws_cloudfront_distribution.cdn"]` on the Route53 record might not be the best choice to create the desired behavior. However, I am not sure how to reach the desired behavior using terraform.

I've also added the Terraform output to a Gist file: https://gist.github.com/rafaelmarques7/03ec6e576faff81ef36d7fc878e85230

I am wondering if anyone has any suggestions to reach the desired behavior. Let me know if the provided information is insufficient. Thank you for your help, it is appreciated!

# Answer
Modelling these permutations as a single module with lots of parameters is possible as you've seen, but it's often not an ideal way to model things because this sort of careful rollout strategy tends to require control over individual steps, rather than rolling everything all at once.

The Terraform team recommends using module composition instead. That means to split the problem into multiple smaller modules that the calling module can then combine to achieve the desired result.

In your specific case, it looks like there are three different building blocks: a load balancer, an optional Cloudfront distribution, and a DNS record. I'm going to show some examples of module composition using Terraform 0.12 features. If you're still using Terraform 0.11 then you can still follow a similar pattern, but Terraform 0.11's expression handling is more limited so the details would look different.

When breaking down this problem, my first step would be to identify what information needs to flow between these components. In this case:

* The load balancer has some arguments of its own, but it doesn't depend on the other components at all.
* The optional Cloudfront distribution needs to know the hostname of the load balancer.
* The Route53 record needs a record name and zone id to populate an `alias` block; they could be for either the load balancer or the Cloudfront distribution depending on which of the two architectures the calling module is following.

In the full architecture where Cloudfront is used, these modules might be composed like this:

```
module "load_balancer" {
  source = "./modules/load-balancer"

  # ... load-balancer specific arguments
}

module "cloudfront" {
  source = "./modules/cloudfront"

  target = module.load_balancer.dns_name
}

module "dns" {
  source = "./modules/route53-record"

  alias = module.cloudfront.route53_alias
}

```

In the load-balancer-only architecture, we remove the `cloudfront` module and connect the DNS directly to the load balancer:

```
module "load_balancer" {
  source = "./modules/load-balancer"

  # ... load-balancer specific arguments
}

module "dns" {
  source = "./modules/route53-record"

  alias = module.load_balancer.route53_alias
}

```

Terraform 0.12 allows us to define that `alias` variable on the `dns` module with a specific object type, to easily pass the alias settings between modules as a single object value:

```
# (in the route53-record module)
variable "alias" {
  type = object({
    name    = string
    zone_id = string
  })
}

resource "aws_route53_record" "example" {
  # (other arguments as in your example)

  alias {
    name                   = var.alias.name
    zone_id                = var.alias.zone_id
    evaluate_target_health = true
  }
}

```

The idea here is that both of the other modules follow this conventional structure so that the `route53-record` module can be compatible with either of them without caring which one is used. This would then correspond with outputs in each of the other modules following the same structure:

```
# in the load-balancer module
output "route53_alias" {
  value = {
    name    = aws_alb.example.dns_name
    zone_id = aws_alb.example.zone_id
  }
}

```

```
# in the cloudfront module
output "route53_alias" {
  value = {
    name    = aws_cloudfront_distribution.example.domain_name
    zone_id = aws_cloudfront_distribution.example.hosted_zone_id
  }
}

```

With this different design, we can address your original use-case by giving control to the author of the calling module to drive the switch between these two models. In order to avoid downtime when switching from Cloudfront back to the load balancer, we need to repoint the DNS first and then destroy the Cloudfront distribution only when it is no longer used. So we can do that in two steps, first by modifying the first example above to point the DNS record at the load balancer while retaining the Cloudfront distribution:

```
module "load_balancer" {
  source = "./modules/load-balancer"

  # ... load-balancer specific arguments
}

module "cloudfront" {
  source = "./modules/cloudfront"

  target = module.load_balancer.dns_name
}

module "dns" {
  source = "./modules/route53-record"

  alias = module.load_balanccer.route53_alias
}

```

Then, once the Cloudfront distribution has become idle, remove that module altogether at your leisure and apply again to clean it up.

Composition is the primary way we model relationships in Terraform, so it's most natural to use composition when working with modules too, rather than building big modules with lots of conditionals to handle every permutation. Composition allows flexibility in how to make changes to the infrastructure in future so that the maintainer of the top-level module can be the one to decide what tradeoffs to make, such as making smaller changes to avoid downtime.

---

I used modules above because the original question was about modules, but I'd note also that after this decomposition we're left with one module per resource whose name is the resource type, which is one of the "smell tests" in When to write a module?.

If decomposing these into separate modules ends up just creating modules that are thin wrappers around single resource blocks then it's usually better to just eliminate the module altogether and just use the resource types directly.

However, if one or more of the modules still raises the level of abstraction (e.g. by providing hard-coded good values for most of the resource arguments and only exposing a small subset of them for customization) then that module in particular may still be warranted.
> 0 votes

---
Tags: amazon-web-services, terraform, dns, amazon-cloudfront
---


# Question
Title: Add mapping ip and host into /etc/hosts with docker
I am using docker to scale my service. I know there's a command called `--add--host` for adding host into /etc/host

However, I have a bunch of hosts need to be added. It's just too long to launch a docker in this way

```
docker run -it \
  --add-host=testing1.example.com:10.0.0.1 \
  --add-host=testing2.example.com:10.0.0.2 \
  --add-host=testing3.example.com:10.0.0.3 \
  ...

```

Therefore, I was thinking to create a function to read a mapping file and solve the problem.

The file format is like the following

**host-list**

```
14.64.65.3 A
11.64.65.18 B
12.64.65.23 C
13.10.200.63 D
13.10.5.77 E
14.10.5.72 F
15.10.5.81 G
...

```

I am new on shell script and have no idea to start with.

# Answer
> 2 votes
You could try to use docker compose and the docker compose "extra\_hosts" tag, as described on the docker compose specification at https://docs.docker.com/compose/compose-file/.

> extra\_hosts
>
> Add hostname mappings. Use the same values as the docker client --add-host parameter.
>
> extra\_hosts:
>
> * "somehost:162.242.195.82"
> * "otherhost:50.31.209.229"
>
> An entry with the ip address and hostname is created in /etc/hosts for this service, e.g:
>
> 162.242.195.82 somehost
>
> 50.31.209.229 otherhostinside containers

So, this will mean, that you need to use docker compose in addition to docker.

---
Tags: docker, shell-script, bash
---


# Question
Title: What is the advantage of using Prometheus over ingesting metrics using logs?
I want to setup monitoring on our environment. I am wondering on what are the advantages of using something like Prometheus for making metrics vs. just using something like the ELK stack for creating similar metrics using logs?

# Answer
> 0 votes
# Efficiency

If you don't need the file it saves space and trouble to leave it out. Logs require rotation and retention policies which are nice to skip.

# Reliability

Skipping the file avoids issues with the file that could caused lost metrics like a full disk.

# Latency

Skipping the file means that you remove latency from disk I/O. For most metrics you don't care if they show up a few seconds late, but if you're trying to build things to respond to some metric quickly. You might want to put it on Kafka also.

# Answer
> -1 votes
The basic difference between prometheus metrics And logs is that prometheus is a timeseries db while logs - in General sense - are event driven.

Thus using prometheus is suitable for monitoring the whole lifecycle of your endpoints, making aggregations on results And generally evaluating the behaviour of your target before something occurs.

Logs are more suitable if you want reportin when something particular happens, in an order of events And generally tracking down incidents after they've already happened.

---
Tags: monitoring, prometheus, metrics
---


# Question
Title: Running Protractor tests in docker through teamcity
I have a project for which I am using Teamcity to automate deployment, and I would also like to automate testing.

It is an angular frontend, and we have some protractor tests.

I would like to run these tests as one of the steps in our build configuration.

I would like to have these tests run inside a docker container instead of the teamcity agent.

I am able to run other steps using docker containers, so I have teamcity and docker working as expected.

My main problem here is being able to run this within the docker container. I would like to have a container that would be able to run without having to rely on external containers, however it seems that the containers I have found and tried using rely on an external selenium instance. Either I am misunderstanding how this should run, or I would like a self contained container to run the tests and drop dead afterwards.

Bear in mind that these tests can be run locally in our development workstations, with selenium and protractor running locally. I would like to achieve the same through a docker container, without that container having to rely on an external selenium server instance.

I have tried the following docker containers:

When I try to run the tests, they both complain selenium server could not be found.

This is my protractor configuration file:

```
// conf.js
var spec_files = "tests/*.specs.js";
exports.config = {
  framework: 'jasmine',
  seleniumAddress: 'http://localhost:4444/wd/hub',
  specs: [spec_files],
  multiCapabilities: [{
    browserName: 'chrome'
  }],
  baseUrl: 'test',
  restartBrowserBetweenTests: true
}

```

The baseUrl parameter is fed into protractor with the following command:

```
protractor conf.js --baseUrl="http://[Replaced with Environment domain name]:8080"

```

What am I missing here?

# Answer
I have been able to figure out how to get everything to work.

Let's start from the beginning:

**Changes to config file**

1. Remove the selenium server settings (the docker container starts one on every run, and uses that automatically)
2. Add 'useAllAngular2AppRoots' otherwise the tests will all fail

The resulting config file will be something like this (ignore the params section, that is just an object with additional parameters you can inject into your tests):

```
// conf.js
var spec_files = "tests/*.specs.js";
exports.config = {
  framework: 'jasmine',
  specs: [spec_files],
  useAllAngular2AppRoots: true,
  multiCapabilities: [{
    browserName: 'chrome'
  }],
  restartBrowserBetweenTests: true,
  params: {
    sleepTime: 0
  }
}

```

**Setting up the build step in teamcity**

Usually when I want to run a build step I do it in a docker container, and execute command line steps inside that container.

To get this to run I could not do this, as we need to inject a few parameters at the end of the run command, so you need to create a docker command, looking like this:
> 0 votes

---
Tags: docker, testing, teamcity
---


title: Why Docker cant launch container?

text:
# Question
Title: Why Docker cant launch container?
I have docker-compose file with one image (httpd) and one volume that look into ./ folder (It contains only one `index.html` static file). When I start my computer I launch this image with

```
docker-compose up

```

And it works, I see message

```
Starting mysiteloc_app_1        ... done

```

And can access it via localhost.

But when I wait several hours (seriously, I do nothing, for experiment I just wait 3 hours without touching keyboard or mouse) and try to use

```
docker-compose up

```

I get message

```
mysiteloc_redis_1 is up-to-date

```

But if I restart docker daemon all works nice. What the problem? Docker reference say that if i get `up-to date` message I changed volume or docker-compose.yml and need to rebuild, but firstly I dont changed anything, secondly why after reboot it works without rebuild?

P.S: When I write this message I realised that use `httpd:latest` image. Problem may be related with minor httpd updates? But then why I dont need rebuild after restarting daemon?

# Answer
After getting your site up with `docker-compose up` there is No practical reason for running `docker-compose up` again without making any changes, because the container is running.

So docker deamon informs you there is nothing New to do.
> 1 votes

---
Tags: docker, docker-compose
---


title: Why is docker container recreated

text:
# Question
Title: Why is docker container recreated
I'm testing Wordpress docker on an Amazon EC2 machine. yesterday, all file based changes (like plugins, themes, etc) were gone. The Ec2 machine is up for 8 days, other containers are also up for 8 days, but the Wordpress container is like created 20 hours ago.

I'm not asking about volume mapping etc, I know that to really persist file changes in a container we should use volumes, my question is:

1. 1-On what events the container is re-created? We've not used `docker run` when it's bee re-created
2. How can we see a log of all the events happening on the docker (and not in specific containers)?

EDIT: Log from `journalctl -u docker.service`:

```
Jun 11 16:05:44 ip-172-31-39-113 dockerd[906]: time="2019-06-11T16:05:44.328303723Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete
Jun 13 04:05:30 ip-172-31-39-113 dockerd[906]: time="2019-06-13T04:05:30.306067658Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete
Jun 13 13:40:31 ip-172-31-39-113 systemd[1]: Stopping Docker Application Container Engine...
Jun 13 13:40:31 ip-172-31-39-113 dockerd[906]: time="2019-06-13T13:40:31.557328738Z" level=info msg="Processing signal 'terminated'"

```

Only the second line is almost at the time container has been deleted and re-created

EDIT2: output from `docker ps -a`:

```
CONTAINER ID        IMAGE                              COMMAND                  CREATED             STATUS                    PORTS               NAMES
3aebfdef77c8        wordpress:latest                   "docker-entrypoint.s…"   24 hours ago        Exited (0) 15 hours ago                       site-wordpress

```

# Answer
First, see Docker daemon’s log. Refer docker docs

If some request or command executed to the Wordpress container, it might had restarted through docker daemon.
> 1 votes

# Answer
Docker won't recreate the container itself without something like swarm mode enabled. Since you aren't doing this, an external tool is recreating the container through the docker API. This may be something like a `docker-compose` command, someone running a `docker` command directly or via a script, or some other orchestration tool. You can check `docker events` to get an idea of what API calls were made, but you'll need to look outside of docker and figure out where these commands are coming from.
> 1 votes

---
Tags: docker, containers
---


# Question
Title: How to get Github username variable in Azure DevOps build pipeline?
I have an Azure DevOps pipeline build configuration which builds automatically with the master branch. I want to access the **GitHub user** or username who committed last.

If I access variables `$(Build.QueuedBy)` and `$(Build.RequestedFor)` in a build that I triggered manually from the portal, I get the outputs as `Sith` and `Sith` respectively.

But when I pushed code to master and triggered automatically with CI, I get outputs for the variables as `Microsoft.VisualStudio.Services.TFS` and `Microsoft.VisualStudio.Services.TFS` respectively.

I can access all other variables like `$(Build.DefinitionName)` and `$(Build.BuildNumber)`
in all the cases without any issues.

How to get Github username variable as `Sith` instead of `Microsoft.VisualStudio.Services.TFS`?

# Answer
> 0 votes
This is not supported by Azure DevOps pipelines yet. Microsoft will provide it soon. Will update this post once they made it available.

Pretty straight forward way to get last commit user in Bash:
`git log -1 --pretty=format:'%an'`

---
Tags: azure-devops, azure, github, azure-pipelines
---


title: View logrotated log continuously

text:
# Question
Title: View logrotated log continuously
Is there a command to view logrotated logs as a continuous stream with proper ordering?

I need to view all entries in syslog in historical order starting from the oldest entry available. So far I was inspecting logs one by one manually.

```
-rw-r-----   1 syslog    adm             140368 Jun 12 09:41 syslog
-rw-r-----   1 syslog    adm              45498 Jun 12 06:25 syslog.1
-rw-r-----   1 syslog    adm               1874 Jun 11 06:25 syslog.2.gz
-rw-r-----   1 syslog    adm               1758 Jun 10 06:25 syslog.3.gz
-rw-r-----   1 syslog    adm               1615 Jun  9 06:25 syslog.4.gz
-rw-r-----   1 syslog    adm              28494 Jun  8 06:25 syslog.5.gz
-rw-r-----   1 syslog    adm              92359 Jun  7 06:53 syslog.6.gz
-rw-r-----   1 syslog    adm              13849 Jun  6 09:21 syslog.7.gz

```

# Answer
If all the files are there without any extension then yes you can execute the cat command with * (cat *) it will display the content of each and every files underneath this directory, but here in your output it has the zip files (*.gz) too, so you need to execute 2 commands rather than single command...

1. cat syslog syslog.1
2. zcat \*.gz

Hope this will helps.
> 1 votes

---
Tags: logging
---


title: Jenkinsfile equivalent in Azure repos

text:
# Question
Title: Jenkinsfile equivalent in Azure repos
We have our source code in GitHub. We use Jenkins for CI. We had written a Jenkinsfile in the Github repos and used GitHub plugin for Jenkins. The plugin would scan the repos and then Jenkins would execute the code in Jenkinsfile for the builds. This works perfectly.

Now, we are moving our source code from GitHub to Azure repos(https://azure.microsoft.com/en-us/services/devops/repos/) purely for non technical reasons. From what I could tell, looking around, there does not seem to be an equivalent of Git hub plugin for Azure repos in Jenkins. Is this correct. Also is there a plugin in Jenkins for Azure repos.

# Answer
I'm new to Azure DevOps so I may be wrong - I think you are asking about using a Jenkins plugin to scan Azure Repos for Jenkinsfiles.

If you were moving to Azure DevOps, you can store your Azure Pipeline configuration, *azure-pipelines.yml*, in YAML format and this can be stored in the code repository (like a Jenkinsfile).

For further pipeline details see; Create your first pipeline and YAML schema reference.

It looks like you can configure Azure Pipelines to scan your organization repos in Azure Repos and GitHub

The GitHub documentation does warn that this feature is **actively being improved** \- so it may change in the future.

For GitHub you need also configure Azure Pipelines access using the GitHub app:

https://github.com/apps/azure-pipelines/installations/new/permissions
> 1 votes

# Answer
Azure pipelines can be set up to automatically run a CI build from GitHub or a private Git repository sitting within Azure DevOps. Your build or release pipeline in Azure DevOps has Jenkins tasks so you can pass the build job to Jenkins to complete if desired.

> Also is there a plugin in Jenkins for Azure repos.

Yes - This link will help you

Alternatively if Jenkins is the source of your build and you want to pass this to Azure Pipelines then this will help
> 0 votes

---
Tags: jenkins, azure-devops, jenkins-plugins
---


# Question
Title: GitLab CI/CD job without pulling LFS files?
When our GitLab CI/CD job runs, it first pulls a Docker image and then git clones the entire repo. Afterwards, the Build commands are executed in the Docker image, etc.

Is it possible to only clone the **non**-LFS files from the repo? ... Or perhaps clone the HEAD of the master branch from only a specific subdirectory and it's subdirectories, etc.?

# Answer
Yes, full cloning of LFS files can be restricted! By default, GitLab will clone your repo into the CI/CD build directory. To limit the clone from downloading the LFS files, tell it not to do it. You do this by setting a variable in `.gitlab-ci.yml` like this.

```
# Other declarations etc above the specific job
jobname:
  variables:
    GIT_LFS_SKIP_SMUDGE: 1
# More job declarations, etc.

```

The "smudge" process replaces the link/pointers to the LFS files with the actual files. By telling GitLab's CI job to skip the smudge, the LFS files aren't downloaded.

Placing `GIT_LFS_SKIP_SMUDGE: 1` in a variables section at the top may make it apply to all of the defined jobs.
> 5 votes

---
Tags: gitlab, gitlab-ci-runner, git-lfs
---


# Question
Title: Deploying Anchore to Kubernetes Cluster using Helm
Kubernetes version 1.14.2 (Client) 1.14.1 (Server) Anchore Chart anchore-engine.1.0.5 App Version 0.4.0

I'm trying to deploy Anchore via Helm to a Kubernetes cluster, but I'm getting failures in every pod.

Helm is properly configured and I've confirmed this by installing several other releases. I install Anchore with

```
helm install --name anchore-release stable/anchore-engine

```

Checking pods confirms all of the pods install

```
anchore-release-anchore-engine-analyzer-659854799f-qkvdn     0/1     Running   0          108s
anchore-release-anchore-engine-api-5d75bbd9d9-9hnms          0/1     Running   0          108s
anchore-release-anchore-engine-catalog-58d7f8bfcd-7d2jz      0/1     Running   0          108s
anchore-release-anchore-engine-policy-6d7579d494-zcg4g       0/1     Running   0          108s
anchore-release-anchore-engine-simplequeue-d8f4f4dbc-62hst   0/1     Running   0          108s

```

However, none ever enter Ready. Taking a peek at any one of them reveals the issue.

```
     Type     Reason     Age                   From                Message
  ----     ------     ----                  ----                -------
  Normal   Pulled     17m                   kubelet, XXXXXX  Container image "docker.io/anchore/anchore-engine:v0.4.0" already present on machine
  Normal   Created    17m                   kubelet, XXXXXX  Created container anchore-engine-analyzer
  Normal   Started    17m                   kubelet, XXXXXX  Started container anchore-engine-analyzer
  Normal   Scheduled  17m                   default-scheduler   Successfully assigned default/anchore-release-anchore-engine-analyzer-659854799f-9qdpq to XXXXXX
  Warning  Unhealthy  15m (x5 over 15m)     kubelet, XXXXXX  Liveness probe failed: Get http://XXX.XXX.XXX.XXX:8084/health: dial tcp XXX.XXX.XXX.XXX:8084: connect: connection refused
  Warning  Unhealthy  2m50s (x91 over 17m)  kubelet, XXXXXX  Readiness probe failed: Get http://XXX.XXX.XXX.XXX:8084/health: dial tcp XXX.XXX.XXX.XXX:8084: connect: connection refused

```

All of the pod experience liveness/readiness probe failures. In order to examine this closer I moved onto the master node and attempted to query these ports manually. I found that all of the Anchore pods were pingable but had exposed no ports. There is no firewall running anywhere within the cluster, and I installed an Nginx pod to confirm that. The other pod was able to expose and access ports in the 8000 range.

Flannel is serving as the networking manager within the cluster, and due to the Nginx test-deployment properly exposing its ports I'm inclined to believe it's set up properly.

Is anyone familiar with what could be cause the Anchore pods to not expose any of their ports, or another issue which could give the appearance of that? As additional information, here's an example description of one of the pods.

```
Name:               anchore-release-anchore-engine-analyzer-659854799f-9qdpq
Namespace:          default
Priority:           0
PriorityClassName:  <none>
Node:               XXXXXX/XXX.XXX.XXX.XXX
Start Time:         Wed, 05 Jun 2019 17:41:11 -0400
Labels:             app=anchore-release-anchore-engine
                    component=analyzer
                    pod-template-hash=659854799f
Annotations:        <none>
Status:             Running
IP:                 XXX.XXX.XXX.XXX
Controlled By:      ReplicaSet/anchore-release-anchore-engine-analyzer-659854799f
Containers:
  anchore-engine-analyzer:
    Container ID:  docker://977890fe98a69f6399db55a06e26911befa2e94f98a1dc2d102041132961364e
    Image:         docker.io/anchore/anchore-engine:v0.4.0
    Image ID:      docker-pullable://docker.io/anchore/anchore-engine@sha256:1db43d98a0895cb266658b1cee9db275227b585f19590824ece58ad497bb772f
    Port:          8084/TCP
    Host Port:     0/TCP
    Command:
      anchore-manager
    Args:
      service
      start
      analyzer
    State:          Running
      Started:      Wed, 05 Jun 2019 17:41:13 -0400
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:analyzer-api/health delay=120s timeout=10s period=10s #success=1 #failure=6
    Readiness:      http-get http://:analyzer-api/health delay=0s timeout=10s period=10s #success=1 #failure=3
    Environment Variables from:
      anchore-release-anchore-engine  Secret     Optional: false
      anchore-release-anchore-engine  ConfigMap  Optional: false
    Environment:
      ANCHORE_POD_NAME:  anchore-release-anchore-engine-analyzer-659854799f-9qdpq (v1:metadata.name)
    Mounts:
      /analysis_scratch from analyzer-scratch (rw)
      /config/config.yaml from config-volume (rw,path="config.yaml")
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-nzsfd (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      anchore-release-anchore-engine
    Optional:  false
  analyzer-scratch:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  default-token-nzsfd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  default-token-nzsfd
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute for 300s
                 node.kubernetes.io/unreachable:NoExecute for 300s

```

I appreciate any input. Thanks.

# Answer
A co-worker ended up taking over deploying this a few days ago and managed to get it working. I'm not certain what he did differently, but I know discovered very similar issues I was facing with other Helm charts boiled down to the database they depended on not deploying properly which caused the other components to fail during initialization.

Database issues were different between the other services I was deploying, but one was rejecting the mounting of a root NFS share. The other appeared to have a mismatch between the base-chart and its packaged-in database chart where the database name and password configuration names were different. Anchore's database was likely failing due to the root-NFS share issue, which then cascaded to other components.
> 1 votes

---
Tags: kubernetes, helm
---


# Question
Title: How to forcibly remove the resource created by the broken module?
I've created few `dns_cname_record` resources, however Terraform DNS provider started to throw the errors like:

> Error: update server is not set

I've created the bug report for it, as it prevents me from running any Terraform command. The error is basically shown on every Terraform command, including `plan`.

Now I want to clean up and get rid of this DNS module completely, and all references to it. So I've removed all my `dns_cname_record` resources from all `.tf` files. However there are some still resources stored in a state file which I want to get rid of.

```
$  terraform show | grep -w dns_cname_record
# dns_cname_record.foo-example:
resource "dns_cname_record" "foo-example" {
# dns_cname_record.bar-example:
resource "dns_cname_record" "bar-example" {

```

So I've tried to destroy it with force without refreshing state, but it failed as well:

```
$ TF_LOG=trace terraform destroy -target=dns_cname_record.foo-example -force -refresh=false
...
[TRACE] <root>: eval: *terraform.EvalIf
[TRACE] <root>: eval: *terraform.EvalApplyProvisioners
[TRACE] <root>: eval: *terraform.EvalIf
[TRACE] <root>: eval: *terraform.EvalIf
[TRACE] <root>: eval: *terraform.EvalApply
[DEBUG] dns_cname_record.foo-example: applying the planned Delete change
[TRACE] GRPCProvider: ApplyResourceChange
[DEBUG] dns_cname_record.foo-example: apply errored, but we're indicating that via the Error pointer rather than returning it: update server is not set
[TRACE] <root>: eval: *terraform.EvalWriteState
[TRACE] EvalWriteState: writing current state object for dns_cname_record.foo-example
[TRACE] <root>: eval: *terraform.EvalApplyPost
[ERROR] <root>: eval: *terraform.EvalApplyPost, err: update server is not set
[ERROR] <root>: eval: *terraform.EvalSequence, err: update server is not set
[ERROR] <root>: eval: *terraform.EvalOpFilter, err: update server is not set
[TRACE] [walkDestroy] Exiting eval tree: dns_cname_record.foo-example (destroy)
[TRACE] vertex "dns_cname_record.foo-example (destroy)": visit complete
[TRACE] dag/walk: upstream of "dns_cname_record.foo-example (clean up state)" errored, so skipping
[TRACE] dag/walk: upstream of "provider.dns (close)" errored, so skipping
[TRACE] dag/walk: upstream of "meta.count-boundary (EachMode fixup)" errored, so skipping
[TRACE] dag/walk: upstream of "root" errored, so skipping

Error: update server is not set

Releasing state lock. This may take a few moments...
[DEBUG] plugin: plugin process exited: path=~/.terraform/plugins/darwin_amd64/terraform-provider-dns_v2.1.1_x4 pid=76229
[DEBUG] plugin: plugin exited

```

and at the end, it doesn't remove invalid entry, because of the error.

I've also tried to remove `terraform-provider-dns_v2.1.1_x4` binary module file, but `teraform plan` complains about it, and `terraform init` downloads it again, and the problem repeats.

Is there any higher power command than `destroy -force -refresh=false` to get rid of the invalid entries from the state file which breaking local setup? Or I need to edit the state file manually?

# Answer
The workaround is to pull the current state file, edit it and use it as a base line. For example:

```
$ terraform state pull > terraform.tfstate
$ vim terraform.tfstate # Carefully remove invalid entries.
$ python -mjson.tool terraform.tfstate # Validate JSON.

```

Note: In Vim, placing cursor on the opening bracket, hitting `d%` will remove the whole group.

Then plan and apply:

```
$ terraform plan -state=terraform.tfstate -refresh=false
$ terraform apply -state=terraform.tfstate -refresh=false

```

---

Alternatively, pull, edit and push your local state file into remote state:

```
$ terraform state pull > terraform.tfstate
$ vim terraform.tfstate # Remove invalid entries and increase the serial value.
$ terraform state push terraform.tfstate

```

---

For Azure, it's better to use `terraform-provider-azurerm` instead of general DNS provider (`terraform-provider-dns`).
> 6 votes

---
Tags: terraform, terraform-module
---


# Question
Title: Unable to parse YAML: mapping values are not allowed here
Good morning. I'm migrating from Gitlab to CircleCI and try to map out a simple pipeline ("workflow") that basically should look like this:

```
[build] -> [tests] -> [release*]
* only on tags

```

I wasted endless hours literally copy-pasting example configs from the documentation. I defined three jobs:

1. build
2. tests
3. release

And I created a minimal workflow that just echos the job title, includes some requires (to map the pipeline) and have a condition that only triggers releases on tags.

However, I always get a YAML parsing error:

```
#!/bin/sh -eo pipefail
# Unable to parse YAML
# mapping values are not allowed here
#  in 'string', line 25, column 19:
#               requires:
#                       ^
#
# -------
# Warning: This configuration was auto-generated to show you the message above.
# Don't rerun this job. Rerunning will have no effect.
false

Exited with code 1

```

This is my `config.yaml` that currently fails:

```
version: 2
jobs:
  build:
    steps:
      - run:
          name: Compile Binaries
          command: echo "Compile Binaries"
  tests:
    steps:
      - run:
          name: Run All Tests
          command: echo "Run All Tests"
  build:
    steps:
      - run:
          name: Release Artifacts
          command: echo "Release Artifacts"

workflows:
  version: 2
  minimal:
    jobs:
      - build
      - tests
          requires:
            - build
      - release
          requires:
            - tests
          filters:
            tags:
              only: /^v.*$/

```

I also tried online YAML validators but they always say this is valid YAML, so something is wrong with the way Circle expects the config to be. What could it be?

---

Edit: I managed to even minify the example that causes the exact same error:

```
jobs:
  build:
    steps:
      - run:
          name: Release Artifacts
          command: echo "Release Artifacts"

workflows:
  minimal:
    jobs:
      - release
          filters:
            tags:
              only: /^v.*$/

```

Causes:

```
#!/bin/sh -eo pipefail
# Unable to parse YAML
# mapping values are not allowed here
#  in 'string', line 12, column 18:
#               filters:
#                      ^
#
# -------
# Warning: This configuration was auto-generated to show you the message above.
# Don't rerun this job. Rerunning will have no effect.
false

Exited with code 1

```

# Answer
> 4 votes
Running your `config.yml` through yamllint, produced the following:

```
  1:1       warning  missing document start "---"  (document-start)
  13:3      error    duplication of key "build" in mapping  (key-duplicates)
  25:19     error    syntax error: mapping values are not allowed here

```

When I add colons to the entry on lines 25 and 28:

```
workflows:
  version: 2
  minimal:
    jobs:
      - build
      - tests:
          requires:
            - build
      - release:
          requires:
            - tests
          filters:
            tags:
              only: /^v.*$/

```

...the syntax error is no longer reported:

```
1:1       warning  missing document start "---"  (document-start)
13:3      error    duplication of key "build" in mapping (key-duplicates)

```

# Answer
> 4 votes
You simply have an incorrect yaml syntax in your `workflows` `jobs` list.

From what I could very quickly understand on circleCI's documentation, jobs names are accepted as a shorthand if you don't define any parameters. If you have parameters, the list element is a hashmap with top element being the job name. In your specific case, this should give (note the column after the names of jobs with params):

```
workflows:
  version: 2
  minimal:
    jobs:
      - build
      - tests:
          requires:
            - build
      - release:
          requires:
            - tests
          filters:
            tags:
              only: /^v.*$/

```

---
Tags: continuous-integration, configuration, yaml, circleci-2.0
---


# Question
Title: How can we automatically transition Epic Jiras based on the transitions of underlying Jiras?
We are keeping track of our software development and engineering components in Jira by having Epics to contain all of the Jiras required for a deliverable.

We would like to have our Epics transition automatically, based on some simple rules applied to the transitions of the underlying Jiras.

For example, the first underlying Jira that moves to "in-progress", should automatically transition the parent Epic to "in-progress"; and the last Jira that moves to "done" should automatically transition the Epic to "done".

Is there any way to set up triggers to automatically transition Epics based on the transitions of underlying Jiras?

# Answer
I think this link on the Atlassian Community forum may be what you're trying to do.
> 1 votes

---
Tags: jira
---