# Question
Title: Extra none:none images after docker build
I am quite new on docker and tried to build a postgres database image with preset database scheme created on. Thus created below dockerfile

```
#alpine linux version of the postgres image. uses a really small amount of disk
FROM postgres:alpine

#environment database name that postgres will get to create during the startup
ENV POSTGRES_DB=test

#Superuser name that will be created during the container creation
ENV POSTGRES_USER=test

#superuser password to be used to connect to the postgres instance
ENV POSTGRES_PASSWORD=test

#sql file to be run while container getting created
COPY auto-create-scheme.sql /docker-entrypoint-initdb.d/

```

but once the modified image created. I got below output;

```
docker build --no-cache -t test .
  Sending build context to Docker daemon  28.67kB
  Step 1/5 : FROM postgres:alpine
   ---> 47510d0ec468
  Step 2/5 : ENV POSTGRES_DB=test
   ---> Running in 7d0246b1cb20
  Removing intermediate container 7d0246b1cb20
   ---> 8be2b9662e86
  Step 3/5 : ENV POSTGRES_USER=test
   ---> Running in 51982d642f99
  Removing intermediate container 51982d642f99
   ---> 954931dc66a8
  Step 4/5 : ENV POSTGRES_PASSWORD=test
   ---> Running in a0ce506ccf78
  Removing intermediate container a0ce506ccf78
   ---> c97298682dac
  Step 5/5 : COPY create-scheme.sql /docker-entrypoint-initdb.d/
   ---> 95df9ff93fcd
  Successfully built 95df9ff93fcd
  Successfully tagged test:latest

```

following this; if I run docker images -a

I can see those image layers as repo and tag values none.

How can I create my build file to be able ***discard*** these image layers have only the final image ***test***.

```
<none>                           <none>              c97298682dac        5 minutes ago       71.6MB
<none>                           <none>              8be2b9662e86        5 minutes ago       71.6MB
<none>                           <none>              954931dc66a8        5 minutes ago       71.6MB
test                             latest              95df9ff93fcd        5 minutes ago       71.7MB

```

# Answer
> 3 votes
Those images are the docker build cache. You need this to be able to quickly rebuild similar images, and to avoid rebuilding unchanged layers which would result in resending those to remote registry servers and forcing servers to download those unchanged layers again. One important point on these layers:

# They do not consume any significant disk space

You can view the layers of an image by running the `docker image history` command on an image. Here's an example of a rather large image built on my machine:

```
$ docker image history 25f90c8ba1ca
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
25f90c8ba1ca        11 days ago         /bin/sh -c #(nop)  CMD ["/bin/sh" "-c" "echo…   0B
8a190467427e        11 days ago         /bin/sh -c #(nop) COPY dir:36bb9e67a0f2844c2…   298MB
1f6f5474071d        11 days ago         /bin/sh -c #(nop) COPY dir:7c364877951bb2312…   249MB
007510a95d17        11 days ago         /bin/sh -c #(nop) COPY dir:9817b6e3d212385d0…   231MB
ee23b7a42488        11 days ago         /bin/sh -c #(nop) COPY dir:57e817f3d9d977f66…   277MB
e190d1bb3f66        11 days ago         /bin/sh -c #(nop) COPY dir:9ba997e629e74ca64…   205MB
abe0ec5c0ef9        11 days ago         /bin/sh -c #(nop) COPY dir:4dbbd5b2010590e28…   213MB
544af937d8a3        11 days ago         /bin/sh -c #(nop) COPY dir:0ac40edba2c7e1317…   178MB
b78085aba431        11 days ago         /bin/sh -c #(nop) COPY dir:2137a075c552d6783…   135MB
3ff8488b067b        11 days ago         /bin/sh -c #(nop) COPY dir:45da2005488e81e9c…   112MB
19b7cdd9d682        11 days ago         /bin/sh -c #(nop) COPY dir:2b745a4311740ddf7…   65.5MB
eb2cf84859fb        11 days ago         /bin/sh -c #(nop) COPY dir:cd7b639f2eee0e9da…   50.1MB
1ba1c21bb0a3        11 days ago         /bin/sh -c #(nop) COPY dir:594739607d1f2d593…   51.9MB
1da5d6bf03e9        11 days ago         /bin/sh -c #(nop) COPY dir:757daf68a395cfaae…   30MB
f6df8b695822        11 days ago         /bin/sh -c #(nop) COPY dir:449b6c2ff93606741…   18.5MB
72895d213dac        11 days ago         /bin/sh -c #(nop) COPY dir:2290eb8d11f15dbe6…   13.8MB
3b6ceadce77b        11 days ago         /bin/sh -c #(nop) COPY file:fcd71a6bfc87e9dd…   167B
52dd0e701be3        11 days ago         /bin/sh -c #(nop)  ENV TARGET=/usr/share/ngi…   0B
aae476eee77d        3 weeks ago         /bin/sh -c #(nop)  CMD ["nginx" "-g" "daemon…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  STOPSIGNAL [SIGTERM]         0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  EXPOSE 80/tcp                0B
<missing>           3 weeks ago         /bin/sh -c #(nop) COPY file:1d1ac3b9a14c94a7…   1.09kB
<missing>           3 weeks ago         /bin/sh -c #(nop) COPY file:af94db45bb7e4b8f…   643B
<missing>           3 weeks ago         /bin/sh -c GPG_KEYS=B0F4253373F8F6F510D42178…   13.3MB
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV NGINX_VERSION=1.15.5     0B
<missing>           4 weeks ago         /bin/sh -c #(nop)  LABEL maintainer=NGINX Do…   0B
<missing>           6 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           6 weeks ago         /bin/sh -c #(nop) ADD file:25c10b1d1b41d46a1…   4.41MB

$ docker image inspect --format '{{json .RootFS.Layers}}' 25f90c8ba1ca | jq
.
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e",
  "sha256:5fcfba1ba609a8fbfc3eccbc805eb18a78041897ec3f6dc168c9ed871baed950"
]

$ docker image inspect --format '{{json .RootFS.Layers}}' 8a190467427e | jq .
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e",
  "sha256:5fcfba1ba609a8fbfc3eccbc805eb18a78041897ec3f6dc168c9ed871baed950"
]

$ docker image inspect --format '{{json .RootFS.Layers}}' 1f6f5474071d | jq .
[
  "sha256:df64d3292fd6194b7865d7326af5255db6d81e9df29f48adde61a918fbd8c332",
  "sha256:39e8483b9882aa8e95fb00c14e1c5dbdf32e836af4aff4bf862608e0a35bc914",
  "sha256:431a5c7929dd5848b9734d84d5cc5be237dd84e9e5ab1cb84ad6975bdc171c0c",
  "sha256:a83dbde6ba05e8112bd2732375ffbd350d9872d9c9e7e088ba5127118cd6d99e",
  "sha256:65ec7af97662fd3f933dc8b0aefb9f2d9d25c0d78530142bcdab1323c6aee223",
  "sha256:25212046c4c3a7589e45b426085da35d7e5e5a7a834180977a43df533672c360",
  "sha256:508c1d799f8da8accdd8b4e92964c73d6f9b8c4e7381fd0e1241e894d01418d3",
  "sha256:a197e87da4cd4587c8965d81878fbb1d898c8ba7d2a05ff0e9862eb700d26053",
  "sha256:f66588501183b7000c7683808ec4653bb83dc3c07f77d67fc34c78e9d9e5cb33",
  "sha256:de2288e4aa944daa6059184813c549d994f5c2407bfae61c7f8b4e0f950e73b1",
  "sha256:77ae40a90a8d41d53751ee800d4008d7a3706ecae6364560b4606ba0167c2674",
  "sha256:1a4bae64e86919b1f40e61a6022e8f910748566b88c2628893c4e424a60c74a2",
  "sha256:ca61f354107fc14de3c01f5bb192ff8b673b6f84c7089a23d754b23a65751310",
  "sha256:d0da652f2c59d6fb6380c0b6940aaa9f5e5a4fc28ae0b82f1acb5ccf9a1884e1",
  "sha256:5f3a91149be048f4f0a4d2450eab6ff8e630bb416b5c508633014e645f57df06",
  "sha256:0c0762559bcfec46462749615ee06b0998e61673e8118cd1a43e20b38e27825a",
  "sha256:581ad1603f59463febb540052046bbdec82bdfc7c75fd4a94578bdca88b91fec",
  "sha256:ff599edf8583ba1dfd7d9c79c1b718f94641e1fb644a0be29ca5b45dcc1768b4",
  "sha256:6ce4a45f89645a2020cfdae51c83470d9f24424ea16f4bddaf1244aa08f2945e"
]

```

In each of those inspect commands you should see that each new image is just adding on an extra layer until you reach the final image. These layers are the same sha256 checksums, pointing to the same files on the disk. Docker does not copy these files per image, it doesn't even copy them when creating containers (excluding initializing named volumes), the layers are read-only and reused by every descendant image and container via a union filesystem. The disk usage that docker displays is the sum of all of the layers even when other images may share layers, while your actual disk usage is only of the unique layers.

If you rebuild images often on the server with the same build tag, you can prune dangling images with `docker image prune`. Otherwise, simply avoid displaying these build artifacts by running `docker image ls` without the `-a` option.

# Answer
> 0 votes
You can use the parameter `--rm=true` in the `docker build` command. Once the build has been successfully completed, the untagged images will not appear in your host. However, if the build was unsuccessful, they will remain for debugging purposes.

A minor consideration though is that keeping these intermediate images may allow you to build new images faster because not everything would have to be created from scratch.

---
Tags: dockerfile, images
---


# Question
Title: Communication between VPCs and a transit VPC hosting VPN Server without Masking source IP
I am setting up a VPN Server to move all resources behind VPN on AWS, which includes moving multiple VPCs.

The simple setup is:

I have a separate VPC (`VPC1`) running VPN Server and another VPC (`VPC2`) with an EC2 machine.

Let's assume the following CIDRs for VPCs & VPN.

* VPC1: 10.1.0.0/16
* VPC2: 10.2.0.0/16
* VPN: 10.8.0.0/24

Everything is working fine within VPC1.

Now I have enabled VPC Peering between `VPC1` & `VPC2`, but I am not able to access EC2 machine running in `VPC2` unless I mask the VPN CIDR as following.

```
iptables -t nat -A POSTROUTING -s 10.8.0.0/24 -o eth0 -j MASQUERADE

```

But doing so, I lose the source IP of the request. All requests are forwarded as requested by VPN machine.

Let me know if any other information/clarification is required.

# Answer
VPC peering does not support transit traffic.

Unsupported VPC Peering Configurations described several scenarious that are not supported.

This case is not specifically mentioned, but is implicit from the other limitations.

Note also that it is impossible to fully configure the setup you are attempting, because when you route a given destination towards a remote VPC via a peering connection, there is no route table in the target VPC that is applicable to that traffic. Only the instance subnets are available. Route tables apply only to subnets, and the default route table only applies to subnets without an explicitly assigned route table -- not to incoming traffic from a peering connection. There is no way to route this traffic back to your instance.

This is why Hardware VPN, Direct Connect, and NAT devices are not available across peering,

A Transit VPC is not what you have described. These use AWS VPC Hardware VPN connections on the spokes and an EC2-based *router* at the hub to terminate the tunnels... and the transit routing between the tunnels is done within the instance -- not by the VPC network infrastructure.
> 1 votes

---
Tags: amazon-web-services, aws-vpc, vpn
---


# Question
Title: Enabling Github webhooks for multibranch jobs?
I'm setting up a CI scheme for a few projects, on Jenkins, using the multibranch plugin. Everything is working pretty nicely, but the obsessive in me always wants a bit more.

Current situation:
The multibranch thingy is working: picks up and builds new branches and PRs. It does so by polling: Every N minutes it polls Github for new branches and branch updates.

I want:
I would very much like to build new pushes on Github Webhook push events and avoid polling. When not dealing with multibranch, it was as simple as checking the `GitHub hook trigger for GITScm polling` box. In this context, I would have to add a new trigger to the Jenkinsfile. I assume something like:

```
triggers {
    githubWebhook()
}

```

Currently, my Jenkinsfile is something like this:

```
pipeline {
    agent {
        docker {
            image '${REDACTED}/builder:sbt-0.13-jdk-oracle-8'
            args '-v sbt-cache:/root/.sbt/ -v ivy-cache:/root/.ivy2/'
            customWorkspace '/src/'
        }
    }
    options {
        buildDiscarder(logRotator(numToKeepStr: '10'))
        timestamps()
        timeout(time: 1, unit: 'HOURS')
    }
    stages {
        stage('Testing') {
            steps {
                sh "sbt clean coverage test coverageReport coverageAggregate sonar"
            }
        }

        stage('Push snapshot'){
            when {
                not {
                    branch 'master'
                }
            }
            steps {
                echo "[DRYRUN] sbt publishSnapshot"
            }
        }

        stage('Push release') {
            when {
                branch 'master'
            }
            steps {
                echo "[DRYRUN] sbt publish"
            }
        }
    }
    post {
        changed {
            echo "[DRYRUN] changed build"
        }
    }
}

```

Has anyone dealt with something like this? Any other approach?

# Answer
According to the official GitHub Branch Source Plugin documentation, the plugin can automatically configure webhooks for you if you have your GitHub API token configured in Jenkins global settings:

> You can change the build configuration to perform the folder computation at different times. The default setting will scan your GitHub repository for changes if it has not received any change notifications from GitHub within the past day. This scan will catch the repositories you’ve added or removed from the managed GitHub repository. You can always force the Folder Computation to run from the GitHub Organization page.
>
> While the build triggers are often enough, you can set up webhooks to automatically trigger builds when changes are pushed to your GitHub repositories. To do this you must have a GitHub login with a token.
>
> 1. Go to the main configuration settings page, Manage Jenkins \> Configure System
> 2. In the GitHub Plugin Configuration section, add a server with your credentials
> 3. If you need a token, generate one with the Additional Actions \> Convert login to password and token
>
> You can also configure this manually through GitHub itself by registering the URL provided in the help section of the server config.
> 1 votes

# Answer
First, you have to create a new job in Jenkins ( Github organization ) and configure it to scan all repos and all branches, you can do it by following this URL second, press scan organization from Jenkins, now you will see all repos and all braches which contain Jenkinsfile which contains all your stages and all pipeline stages, check also this URL
> 1 votes

---
Tags: jenkins, continuous-integration, github
---


# Question
Title: Ansible: Other option available for telnet check of open ports?
I'm new to Ansible. Here's my task ...

I have 400+ hosts, and I need to verify if 5 different ports are open from their end to our web server.

Individually, I could log in and run:

```
telnet mywebserver.com 443
telnet mywebserver.com 80
telnet mywebserver.com 8443

```

..and so on..

What module or plugin could be used in Ansible so I could automate this, and have it report the results (whether open or closed ports) back to my Ansible server?

# Answer
> 36 votes
You can use the Ansible wait\_for module which checks a specific TCP port is open.

Since in this case, all ports should be open already, we can use a minimal no. of retries, just enough to cover network issues:

```
- name: Check all port numbers are accessible from the current host
  wait_for:
    host: mywebserver.com
    port: "{{ item }}"
    state: started         # Port should be open
    delay: 0               # No wait before first check (sec)
    timeout: 3             # Stop checking after timeout (sec)
  ignore_errors: yes
  with_items:
    - 443
    - 80
    - 80443

```

By default, Ansible will check once every second (configurable in Ansible 2.3 using the `sleep` attribute), so this will check 3 times per port.

Run this in a playbook against your inventory of 400+ hosts - Ansible will check in parallel that all hosts can reach `mywebserver.com` on those ports.

* the parallelism is subject to the forks setting in your `ansible.cfg`.

We use `ignore_errors: yes` here so that any errors are marked in red but do not stop execution.

Open ports are reported as `ok` items in output and closed ports are reported as `failed` (you must use `-vv` flag on `ansible-playbook` to see this output).

### Fine-tuning output

If you want more specific output for the success and failure cases, the code must be more complex, adding a second task:

* `wait_for` task must `register` a variable
* the second task produces output using `debug` based on success/failure condition (e.g. using Jinja2 conditional expression)
* then you need to put both these tasks in an include file (without any `with_items` loop), and write a main playbook task that uses an `include` ... `with_items` to call the include file once per port.

# Answer
> 2 votes
AFAIK there is no built-in module for this purpose, but you can use `shell` \+ `nc`:

```
---
- hosts: all
  tasks:
    - shell: nc -z -w 1 -G 1 my.hostname.com {{ item }} || echo "Port {{ item }} is closed"
      with_items: [80,443,8443]

```

# Answer
> 0 votes
You can use wait\_for module for the same

example quoted from the documentation:

```
- name: Wait 300 seconds for port 8000 of any IP to close active connections, don't start checking for 10 seconds
  wait_for:
    host: 0.0.0.0
    port: 8000
    delay: 10
    state: drained

```

# Answer
> 0 votes
We use our tool dda-serverspec (https://github.com/DomainDrivenArchitecture/dda-serverspec-crate) for such tasks. You may define your expectation

`{:netcat [{:host "mywebserver.com" :port "443"}
 {:host "telnet mywebserver.com" :port "80"}
 {:host "telnet mywebserver.com" :port "8443"}]}`

and test these expectation either against localhost or remote by ssh. For remote tests you've to define a targets:

`{:existing [{:node-name "test-vm1"
:node-ip "35.157.19.218"}
{:node-name "test-vm2"
 :node-ip "18.194.113.138"}]
 :provisioning-user
 {:login "ubuntu"}}`

You may run the test by `java -jar dda-serverspec.jar --targets targets.edn serverspec.edn`

---
Tags: ansible, testing, ports
---


title: Infrastructure as code and TDD

text:
# Question
Title: Infrastructure as code and TDD
Infrastructure as code tells us to use tools that automate your builds. Great. Tools like ansible, chef, puppet, salt stack and others push us towards writing how infrastructure looks like, while resolving the differences.

In Salt Stack those bits are called states. If state does not match the reality, the tool will resolve it for us. In other words - we are writing a test for our infrastructure and if the test fails, the tool will fix it on its own. At least that's the idea.

XP teaches us to use TDD and the question is if it's applicable to infrastructure? The tooling suggests it is.

I can imagine few type of tests that can be very useful.

We write smoke tests that are bundled with the deployed service to ensure that end-to-end the deployed service works and runs as expected. This would be an API call or/and systemctl check to make sure what we just deployed works. A lot of this functionality can be covered in the same states since tools like ansible have states to make sure a service is running.

There is project Molecule that allows running individual roles (as ansible calls its states) against docker or another temporary virtualisation engine. This forces to decouple roles and allows to execute them in isolation from the playbook while working on them. Tests mostly allow mocking the variables that the role is supposed to work with. Other examples seem like a duplication of the ansible engine though (assert a file belongs to a user...).

ThoughtWorks tech radar right now praises tools like inspec, serverspec or goss for validating that the server meets the spec. But we are writing a spec, aren't we?

So is there a point in further testing infrastructure if we are describing infrastructure in states/roles? I could suspect this becomes more required in larger organisations where one team provides the spec and other follows, or if there is a large set of roles maybe you want to run a subset of those and get a speed benefit from tests? I'm struggling to see why you would write a test if you could have a role/state for the same question in mind.

# Answer
In short, I see two categories of tests for your infrastructure: 1) does it have everything you need to run your application and 2) does it not have any superfluous stuff.

First and foremost, you can treat the test suite of your actual software as a kind of "meta test" for your infrastructure. As long as you create the infrastructure from scratch for each test run, and the test suite runs completely on that infrastructure (i.e., doesn't use outside services) the fact that the whole suite is green means that your codified infrastructure is sufficient as well.

Second, especially from the security perspective, you can write tests against your infrastructure. I.e., if one part of your infrastructure is a VM running Linux, you might write a test that does a port scan against that VM, to make sure that there are no unintentional ports open, which may have been installed by an unintended `apt-get install` side-effect. Or you could write tests that checks whether any unexpected files were changed after your proper test suite has completed. Or you could check the `ps` outputs of your VMs or Docker containers for unexpected processes and such, build white-lists etc., and thus get automatic notification if some 3rd party package changed in an undocumented (or unnoticed way) in some upgrade.

These second type of tests are, in a way, similar to what you would do in a classical ops setting anyway, i.e., hardening your servers and checking for intrusions, avoiding full ressource and such.
> 6 votes

# Answer
It looks like everybody here assumes an IAC tool always run as expected, but I can tell (from my own experience) this is not always the case, otherwise unit test would actually be useless.

I remember a picture saying "Ansible playbook ran, everything is fine" with a building burning in the background ...

Running a declarative state and having the server in this actual declared state are 2 different thing from my point of view and experience at least.

A broad and heterogeneous environment, spread across multiple DC, reachable through public network etc... There are multiple reason for which a state could not be applied, either fully or partially.

For all these reasons, there is room for unit test allowing one to get a snapshot of the actual server state, which, again, might differ from the aimed state.

So I'd say yes, unit test are useful even in a IAC managed environment.

**EDIT**

What about the non-regression side of the dev branch of the IaC code base ? so you'd make changes to your code in dev branch and merge it to the prod branch hoping for not breaking everything ? unit test are so valuable, and usually simple to implement I do not see why one would code without this feature.

Reference (in french sorry for that): https://fr.slideshare.net/logilab/testinfra-pyconfr-2017
> 2 votes

# Answer
IMHO it's rather redundant to write TDD tests for items entirely covered by the IaaC state specification. Doing so implies that the effectiveness of the IaaC is questionable - why would you use it if so?

Looking at it from a different prospective IaaC itself (if/when done properly) incorporates capabilities already tested and considered to be functioning reliably. Which is what makes it attractive and which makes writing TDD matching tests redundant.

For example, an IaaC configuration specifying a system with SSH being installed already incorporates reliable checking for SSH being correctly installed and, if not, mechanisms for properly installing it. Which makes a TDD test for checking if SSH is installed redundant. If your IaaC config also specifies sshd to be started and listening on a specific port then TDD tests for sshd running and listening to the respective port would also be redundant.

Note that my answer is not targeting TDD or any other type of testing that checks if your IaaC configuration as a whole fits a certain purpose. That remains valid and can be used in TDD, CI or similar checks during development of that IaaC specification - I believe @AnoE's answer is applicable in such case.
> 1 votes

# Answer
In my experience one of the main differences between Dev and Ops are "heavy run time dependencies". Installing packages heavily depends on repositories, networks or valid keys, or lets say instantiating a new cloud server - it depends on your provider resources.

In terms of server-provisioning even if you did not change your provision code, your image will be valid most of the times but sometimes not. So I think testing is really essential for providing working images.

If you go beond single servers things get even worse ... how you will test reachability in whole network-setups? Including DNS-resolution, routing and firewalling? Even if your IaaC providers API works as expected (I've seen wired issues in this area) I really like TDD in this case.

Because I've not found any testing tools in this area, we wrote one in our spare-time: https://github.com/DomainDrivenArchitecture/dda-serverspec-crate

So I think TDD is really important in the DevOps world!
> 1 votes

---
Tags: deployment, testing, infrastructure-as-code, serverspec, tdd
---


# Question
Title: py2neo AddressError When Connecting to neo4j in Docker
I am having difficulty creating a `Graph` object on Neo4j 3.4.6 using py2neo 4.10 with Python 3.7.0.

I created a Docker container running Neo4j, and I disabled authentication using Dockerfile entry `ENV NEO4J_AUTH=none`. I verified that I can browse to the Neo4j database from the host with http://localhost:7474 and that I was not required to enter a password.

I created a second Docker contain for my web server. I accessed the Bash shell using `Docker exec -it 033f92b042c1 /bin/bash`. I verified that I can ping the Docker image containing the Neo4j installation.

From the second Docker image, I tried to create a `Database` object or a `Graph` object.

```
import neo4j
import py2neo
from py2neo import Graph
graph = Graph("bolt://172.17.0.3:7687")

```

I tried different protocols and `localhost` rather than the IP. In each case, Python throws this:

```
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 88, in __new__
    inst = cls._instances[key]
KeyError: '5280a6d494b601f0256493eab3a08e55'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 305, in __new__
    database = Database(uri, **settings)
  File "/usr/local/lib/python3.7/site-packages/py2neo/database.py", line 92, in __new__
    from py2neo.internal.http import HTTPDriver, HTTPSDriver
  File "/usr/local/lib/python3.7/site-packages/py2neo/internal/http.py", line 26, in <module>
    from neo4j.addressing import SocketAddress
  File "/usr/local/lib/python3.7/site-packages/neo4j/addressing.py", line 26, in <module>
    from neo4j.exceptions import AddressError
ImportError: cannot import name 'AddressError' from 'neo4j.exceptions' (/usr/local/lib/python3.7/site-packages/neo4j/exceptions.py)

```

Am I missing a dependency, or is there another way I should be connecting the Docker images?

# Answer
I had a similar error while using a virtual environment using PyCharm. In my case, I was using an upgraded version of neo4j-driver in my virtual environment compared to my host environment. I downgraded neo4j-driver from 1.7 to 1.6.2.

Make sure you have neo4j-driver installed in general.
> 1 votes

---
Tags: docker
---


# Question
Title: Amazon EKS: using spot instances for worker nodes
When using AWS EKS, is it possible to set up the worker nodes on spot instances? how? Anything I special I should pay attention to, in such a setup?

# Answer
It can be done by modifying the Cloudformation Template (which is mentioned in this document) in the `LaunchConfiguration` section to specify a spot price.

```
NodeLaunchConfig:
  Type: AWS::AutoScaling::LaunchConfiguration
  Properties:
    SpotPrice: "20" # <=== Here
    AssociatePublicIpAddress: 'true'
    IamInstanceProfile: !Ref NodeInstanceProfile
    ImageId: !Ref NodeImageId
    InstanceType: !Ref NodeInstanceType
    KeyName: !Ref KeyName
    SecurityGroups:
    - !Ref NodeSecurityGroup
    BlockDeviceMappings:
      - DeviceName: /dev/xvda
        Ebs:
          VolumeSize: !Ref NodeVolumeSize
          VolumeType: gp2
          DeleteOnTermination: true
    UserData:
      Fn::Base64:
        !Sub |
          #!/bin/bash
          set -o xtrace
          /etc/eks/bootstrap.sh ${ClusterName} ${BootstrapArguments}
          /opt/aws/bin/cfn-signal --exit-code $? \
                   --stack  ${AWS::StackName} \
                   --resource NodeGroup  \
                   --region ${AWS::Region}

```
> 0 votes

---
Tags: amazon-web-services, kubernetes
---


# Question
Title: How to make the Jenkins agent build then start the server?
I want the Jenkins agent to **build** and **start** the server.

I do this by specifying how to build and start the server in the *Build* step of the project configuration in **Bash**

```
echo "Installing dependencies"
npm install
echo "Starting server"
./startscript.sh 3000 & #this script will start the server with specified port

```

**Why the ampsersand in `/startscript.sh 3000 &`?**

I noticed that without the ampersand `&` in the last command, the server is started as expected (I can access it from browser). However, the Jenkins build hangs and cannot finish (because the script starts the server and won't return until the server is terminated).

So I tried to fix this with adding the ampersand to make the script process runs in background. With this, Jenkins job successfully ends the build.

**But adding the ampersand still doesn't fix** The server process is still terminated (it actually started because I checked the log and saw that it was started). I think that this might be because when Jenkins finishes building, it exits the shell and sends the **SIGHUP** to attached background jobs.

**Adding nohub still doesn't fix**

Running the last command with `nohup ./startscript.sh 3000 &` still doesn't fix the bug.

**Question**

How to make the agent build then start the server.

**Note**

The console output of Jenkins builds shows both the command and the output of it (like the behaviour of the -x flag of the shell). And the last command was shown like this:

```
+ nohup ./startscript.sh 3000
Finished: SUCCESS

```

It doesn't show the ampersand `&`... So I don't know if the ampersand was actually correctly interpreted during the run.

# Answer
> 3 votes
You are running into a conflict between jenkins nature of making sure everything is done before a job is complete and your desire for some long-running things to be asynchronous. There is no right answer here, both are reasonable desires, but you and jenkins are pulling in different directions. It may be ok for some parts of the jenkins job to not be completely done but they are good enough for the job. This is a decision you get to make.

If getting this working inside of jenkins is too much trouble there are work arounds. You can make your own daemon that looks for job completions (via files or database entries) and then starts something else. Obviously this means you lose the UX features of jenkins that let everybody know where something is. But you could make this work inside of jenkins by having different jobs also. The quick job finishes and then the longer job takes over. Since each is its own job it can have timeout requirements that make sense for it.

---
Tags: jenkins, builds, bash
---


# Question
Title: Is there an established metric to describe optimal/needed ressources for a specific computational task?
Let's say I have a piece of software which does a computational task that is pretty demanding to either CPU/GPU/RAM/Disk-IO or a combination of those. Then it would be optimal if one has some metric to describe what the optimal configuration of a machine would be to run this software.

Guess that metric should include: 1) "minimal requirements" and more importantly 2) the "optimal ratio" of the above parameters, so that there is no idling CPU/RAM/GPU so that I get the best computation per money/electricity/computation time.

This is especially true, if I want to run your job on AWS/GoogleCloud/OtherProvider or on an HPC environment where you can pick those parameters.

I am aware that this is probably a pretty hard problem, since it is not easy to compare different kinds of CPUs (Cache, clock speed, etc.) or different kinds of RAM.

But still, does there exist and "standard" or approach for this? If yes, what are the methods to establish such a metric?

PS: Is this the right place to post this question?

# Answer
> 1 votes
As far as I'm aware, there is no universal standard amongst cloud providers. One reason for this is because most cloud providers like to abstract how their backend infrastructure, that their instances run on, is managed. How your instance is ran could change minute by minute depending on the cloud provider. The user doesn't care as long as they are getting the desired computing power that they want. Cloud providers also may not want to divulge their exact infrastructure metrics for proprietary reasons as well.

Despite these reasons, some cloud providers offer a standard amongst their own virtual instances. These standards are relative to the computing power of each instance type that the provider offers.

For example, Amazon Web Services uses EC2 Computing Units (ECU) as a metric to compare the approximate computing power in relation to each instance. Comparing two instances types such as an m5.large and an m5.xlarge for example, the m5.large has an ECU of 8 vs the m5.xlarge which has a ECU of 16. The m5.xlarge has roughly double the compute power and therefore double the cost.

---
Tags: amazon-web-services, google-cloud-platform, metrics, performance
---


title: How can I use Docker Hub in Terraform?

text:
# Question
Title: How can I use Docker Hub in Terraform?
The instructions in Terraform Docker Provider give the example:

```
# Configure the Docker provider
provider "docker" {
  host = "tcp://127.0.0.1:2376/"
}

# Create a container
resource "docker_container" "foo" {
  image = "${docker_image.ubuntu.latest}"
  name  = "foo"
}

resource "docker_image" "ubuntu" {
  name = "ubuntu:latest"
}

```

I'm assuming this is pointing to an on-prem docker source. How do I used Docker Hub directly?

# Answer
This can be done with docker\_registry\_image:

```
data "docker_registry_image" "ubuntu" {
  name = "ubuntu:latest"
}

resource "docker_image" "ubuntu" {
  name          = "${data.docker_registry_image.ubuntu.name}"
  pull_triggers = ["${data.docker_registry_image.ubuntu.sha256_digest}"]
}

```
> 1 votes

---
Tags: docker, terraform
---


title: Route webhook to all replicas

text:
# Question
Title: Route webhook to all replicas
Normally a request gets routed to one of the replicas. However, if each instance need to take some action when a webhook request comes in, how can I route the request to all replicas?

Obviously only one instance can reply, so I am skeptical if this is possible.

Alternatively, is there a way for when one of the instances receives the request, to communicate it to the others?

# Answer
This is something you'd have to build. Kubernetes doesn't natively offer this as a pattern.

There are a few approached you can take, but a common on is to use a queue that has this functionality. I'd suggest starting with Nats because it's a CNCF project that is easy to run natively in Kubernetes. When one of your replicas receives the request, it can post a message to the queue that can deliver the message to every subscriber.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Kubernetes - Internal only API Endpoints
Hypothetical question.

Say we have a Java application running in a Kubernetes pod (single docker container) that has 'public' endpoints and 'internal' endpoints.

The `/public` endpoints should be accessible from public internet and exposed via the Ingress.

The `/internal` endpoints should only be accessible to other pods in the Kubernetes cluster.

Is there any way we can limit access to these `/internal` endpoints in 'vanilla' Kubernetes? One way I guess would to be declare the public endpoints in the ingress but them seems messy.

I also believe tools like Istio allow policy based access?

# Answer
> 2 votes
I'm not sure I understand why listing the public endpoints in the ingress is messy. The common solution is to list the ingress endpoints in the ingress controller.

As you mentioned, there are alternatives such as Istio, Consul and other network policy solutions. There's also the Open Policy Agent that can help with this, if you don't want to write YAML.

---
Tags: kubernetes, istio
---


# Question
Title: Jmeter simulating traffic and avoiding 429
I am using Jmeter to investigate the performance and capacity needs of a few web services. Jmeter sends traffic to the web-facing url (served by Nginx), with a rising rate, to see where the service fails. At the moment, this traffic is generated using a single machine, in the CI/CD pipeline.

Before I can really stress the resources, the web server is responding with HTTP 429

Is there a way to set data in the header or something to force each connection to be seen as new, instead of being banned by the server?

Edit : specified that the http server is nginx

# Answer
> 2 votes
Given that this is symptomatic of rate limiting in Nginx in this case, the solution is to configure the rate limiting feature of Nginx to allow this

A whitelist could be created in the configuration to remove ratelimiting from the url generating the traffic.

To follow the Nginx documentation, this could be specified as whitelisting an IP.

The current server setup is :

```
limit_req_zone $binary_remote_addr zone=flood:10m rate=12r/s;
limit_req_zone $binary_remote_addr zone=bot:10m rate=200r/m;
limit_req_status 429;
limit_conn_zone $binary_remote_addr zone=connperip:10m;
limit_conn_status 429;

```

This sets up the rate limit on connections. A whitelist can be created as suggested above, by simulating a boolean condition with a key value map:

```
geo $limit {
    default 1;
    10.0.0.0/8 0;
    192.168.0.0/24 0;
}

map $limit $limit_key {
    0 "";
    1 $binary_remote_addr;
}

```

This removes the rate limit (sets it to 0) if the remote address falls into the whitelist:

```
limit_req_zone $limit_key zone=req_zone:10m rate=5r/s;

server {
    location / {
        limit_req zone=req_zone burst=10 nodelay;

```

---
Tags: testing, resilience-testing, http
---


title: OpenShift configuration management

text:
# Question
Title: OpenShift configuration management
As far as I can tell, reading the OpenShift documentation, including https://github.com/openshift/training/, the officially published "standard" way to manage OpenShift configuration (about their objects like services, build config etc.) is to simply use `oc` or the web console and "do it". They do not seem to document/suggest a standard way to keep this configuration, i.e. the YAML or JSON representation that you see in the web console or in `oc edit` in, say, a git repository.

Would you say that most projects (that you witnessed or know about) work this way, i.e., just keeping the config inside OpenShift? Or are there established standards, to have this outside of OpenShift itself?

I noticed that there is a 3rd party OpenShift ansible module on github, but am not quite sure how proven/established that is.

# Answer
> 3 votes
AFAIK neither OpenShift nor Kubernetes itself allow you to externalize services configuration. You should have a bunch of yml/json files somewhere and control their deployment/configuration using CLI tools.

But there is Rancher which in version 2.0 allow us to import external Kubernetes clusters and manage them as Rancher's own environments.

Description of how this could be done with Rancher you can find here.

**FYI:** Rancher 2.0 is slowly moving to Beta so it might take time for you to use it in your activities.

# Answer
> 2 votes
In my experience Openshift is open in terms of supporting good Kubernetes tools that the community comes up with. You can adopt the Kubernetes tooling that fits your needs. We use Helm to manage the configuration objects (aka yaml) of our applications running in our Openshift cluster. Helm can manages both genetic kubernetes objects as well as openshift specific extensions. Helm is introduced in the official openshift blogs. **Update**: we are rewriting our configuration management using the mighty helmfile which makes it easier to manage multiple helm releases.

This answer gives an overview of how we store all the Helm (or Helmfile) configuration in git and push what is in git out to openshift to give us “infrastructure as code”.

Also you might be missing an important feature of `oc` which is templates. Yet Helm is newer and has a better template framework built into it. When we used the `oc` template system but I had to write my own wrapper logic to deal with conditional logic to add or remove optional features from our application templates. All that custom logic got removed when we upgraded to Helm. Helm also has a rollback to previous release feature.

I believe that the openshift team started adding things like oc templates to fill a gap in the ecosystem at one time. That gap is now filled by Helm which is evolving faster (e.g., Helmfile that also suppers Kustomize). The Kubernetes Operator framework is another example of something from upstream Kubernetes that Openshift now supports. IMHO this isn’t a surprise to RedHat and Openshift they are not competing with any other opensource project they are selling supper for an opensource ecosystem and the strong it is the better it is for them.

The majority of our config can be yaml templated with helm charts. The only thing that seems hard to template are `ImageStreamTag` objects as they contain a lot of specific information such as the sha256 of each image layer. Since the tags we want to run change all the time it as our software changes it feels natural to define the "shapes" of our environments using yaml managed by Helmfile yet run `oc` to pull a particular image stream tag into a project (aka namespace).

---
Tags: configuration-management, openshift
---


# Question
Title: How to know which user created the GCE Instance
We have several billing enabled projects in our GCP account and each of them has several Compute Engine Instances. I want to remove non-critical instances just to reduce the unnecessary billing. But before I do that, I need to know which instance is being used for what and for that I need to know which user created it in the first place.

How do I get to know the creator of each instance either by using `Google Cloud Shell` or `Google Cloud Console`?

# Answer
Note: I'm not a GCE user yet, the answer is based solely on documentation.

You could be Viewing Audit Logs in the `Google Cloud Console`, more specifically the Admin Activity logs:

> Admin Activity logs contain log entries for API calls or other administrative actions that modify the configuration or metadata of resources. For example, the logs record when VM instances and App Engine applications are created and when permissions are changed. To view the logs, you must have the IAM roles **Logging/Logs Viewer** or **Project/Viewer**.

You would be looking for the User identities in audit logs:

> Audit logs record the identity of the user performing logged actions. The identity is held in the `AuthenticationInfo` field of AuditLog objects.

Ideally you'd be looking for the GCE instance creation events, but some/all of those may be too old for the logs retention policy. Then maybe search for VM instance start or other relevant events.

You could also check and maybe cross-reference the audit logs info with that from Usage Reports or exported billing info.

If this is a regular, on-going activity you may want to setup an automated log exporting and processing pipeline.
> 7 votes

# Answer
Not sure if you sorted this out, but I had to do something similar to get who started the instance so I can badger them into stopping the instance if they are not using it. I put together a Logging query:

`resource.type = gce_instance AND (jsonPayload.event_subtype = compute.instances.start OR jsonPayload.event_subtype = compute.instances.insert ) AND jsonPayload.event_type = GCE_OPERATION_DONE AND timestamp >= "2018-10-29T14:28:34-07:00" AND jsonPayload.actor.user!="" AND jsonPayload.resource.name=my-sweet-instance-name`

And here is the nodejs function I put together to get it:

```
const Logging   = require( '@google-cloud/logging' );
const moment    = require( 'moment' );

const logging   = new Logging( );

var getStartInfo = function( instanceName, querySince, cb ) {

    var tstart = ( querySince ? querySince : moment( ).subtract( 48, 'hours' ).format( ) ); //

    var theLogFilter = 'resource.type = gce_instance AND ' +
        '(jsonPayload.event_subtype = compute.instances.start OR jsonPayload.event_subtype = compute.instances.insert ) AND ' +
        'jsonPayload.event_type = GCE_OPERATION_DONE AND ' +
        'timestamp >= "' + tstart + '" AND ' +
        'jsonPayload.actor.user!="" AND ' +
        'jsonPayload.resource.name=' + instanceName;

    logging.getEntries( {
        filter: theLogFilter,
        autoPaginate: false
    }, ( err, entries, nextQuery, apiResponse ) => {

        if ( err ) {
            console.log( "ERROR: " + err );
            cb( err );
            return;
        }

        var item, startedBy, startTime, runningTime, mostRecentStart;

        //console.log( 'Entries: ' + JSON.stringify( entries ) );
        // Mabye if none found, we try again with a longer querySince?
        if ( entries.length == 0 ) {

            console.log( "\nNo log entries found for instance '" + instanceName + "'. Filter:" );
            console.log( theLogFilter );
            cb( "No entries found" );
            return;
        }

        // Are these sorted by time?
        for ( var i = 0; i < entries.length; i++ ) {

            startedBy = entries[ i ].metadata.jsonPayload.fields.actor.structValue.fields.user.stringValue;
            startTime = entries[ i ].metadata.jsonPayload.fields.event_timestamp_us.stringValue / 1000; // This is nano seconds since epoch

        }

        if ( cb )
            cb( null, { "startedBy": startedBy, "startTime": moment( startTime ).format() } );

    } );

}

```

Hopefully that helps someone because it was a good bit of work to put together.
> 2 votes

---
Tags: google-cloud-platform, gcp, google-compute-engine
---


# Question
Title: How to setup artifact filters to differentiate for deployment
We would like to deploy only the artefacts that are built from our dev branch to the dev environment.

and master branch to be deployed to staging(stage) and production(PRD) environments

As a tool, we are using VSTS

https://i.stack.imgur.com/KtT3B.jpg

How should we coordinate this process if we can not implement artefact filters in VSTS?

I am considering to create a separate container registry for master builds

https://i.stack.imgur.com/Dzcve.jpg

Wanted to learn the best practice before i implement this

# Answer
Azure DevOps supports artifact filters for each environment. Below is screenshot of how to enable the feature.
> 4 votes

---
Tags: azure-devops
---


# Question
Title: How to make one branch public while the others remain private on GitLab?
I am currently working on a LaTeX template. I chose gitlab to host the template.

Now I want to give some colleagues access to my template, but only to the master branch. I use the other branches to write my personal documents based on the template.

Is it possible to set the master branch to public while the other branches remain private?

# Answer
I'm not an expert in GitLab but I would assume it to be similar to other solutions like it (bitbucket, github).

A simple way to achieve this would be to clone the repo in another private repo instead of doing a separate branch. In effect this would work similar to having your local repo and the gitlab repo. Merging a branch and merging with another repo is not much different with git
> 2 votes

# Answer
Git does not have branch specific permissions.

For branch specific permissions, you need a server-side authorization layer like Gitolite — obviously, this requires you to be managing your own Git server.
> 0 votes

---
Tags: git, gitlab, configuration
---


title: Hashicorp vault not writing out KV

text:
# Question
Title: Hashicorp vault not writing out KV
I am trying to set up a Hashicorp Vault server and have ran into some more road blocks. I can not get it so that I can write/read data either locally or through a domain name (I know the risks of have it available to the web and for this/testing it is fine).

**Error message:**

```
Error writing data to secret/keys: Put http://localhost/v1/secret/keys: net/http: HTTP/1.x transport connection broken: malformed HTTP response "\x15\x03\x01\x00\x02\x02"

```

**Here is my config files:**

/etc/vault.hcl

```
backend "file" {
        path = "/var/lib/vault"
        address = "127.0.0.1:8200"
        scheme = "http"
        redirect_addr = "http://mydomain.com:8200"
        VAULT_ADDR = "http://mydomain.com:8200"
}

listener "tcp" {
        address = "http://mydomain.com"
        tls_disable = 0
        tls_cert_file = "/etc/letsencrypt/live/mydomain/fullchain.pem"
        tls_key_file = "/etc/letsencrypt/live/mydomain/privkey.pem"

}

```

---

/etc/systemd/system/vault.service

```
[Unit]
Description=a tool for managing secrets
Documentation=https://vaultproject.io/docs/
After=network.target
ConditionFileNotEmpty=/etc/vault.hcl

[Service]
User=vault
Group=vault
ExecStart=/usr/local/bin/vault server -config=/etc/vault.hcl
ExecReload=/usr/local/bin/kill --signal HUP $MAINPID
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
SecureBits=keep-caps
NoNewPrivileges=yes
KillSignal=SIGINT

[Install]
WantedBy=multi-user.target

```

**Some other configuration:**

```
VAULT_ADDR=http://mydomain.com

```

Some additional info:

I used the command `sudo vault operator unseal -tls-skip-verify` because I could not get it to work over HTTPS

Command I used to configure the server `vault server -config=/etc/vault.hcl`

# Answer
You have your VAULT\_ADDR set to HTTP, but vault is listening on TLS - the error you are receiving is the HTTP request receiving a TLS handshake.

Try changing VAULT\_ADDR to https://mydomain.com.
> 1 votes

# Answer
The set of errors are same that we have observed **during TLS integration in Vault**.

As stated by @Colin, **change VAULT\_ADDR to HTTPS**, VAULT\_ADDR=https://mydomain.com

Along with that **change listener tcp address in vault.hcl** as follows:

```
listener "tcp" {
  address = "0.0.0.0:8200"
  tls_disable = 0
  tls_cert_file = "***"
  tls_key_file = "***"
}

```

Hope this works for you as well!
> 0 votes

---
Tags: hashicorp-vault, installation
---


title: Undeploy or deactivate an AWS Lambda

text:
# Question
Title: Undeploy or deactivate an AWS Lambda
Is there a way to undeploy or deactivate an AWS Lambda or must I either implement it myself or delete and later redeploy it to achieve the desired state? What I would like is similar to just "undeploying it" so that it can be deployed again later.

# Answer
you can change the trigger working process and rewrite and re schedule the process. Which the way you want. it will work.

Hope it helps.
> 3 votes

---
Tags: aws-lambda
---


title: Automatic periodic conditional checks

text:
# Question
Title: Automatic periodic conditional checks
I need a simple tool that will carry out some conditional checks periodically and give me a red/green status that I can use as an indication that something is wrong.

**These are examples of checks that I would like to monitor automatically:**

* Does a row exist in a database that meets a certain criteria?
* Does a file, with a maximum age, exist on some network share?
* Does the response-body of a HTTP-request match a specific regex?

So basically, it would free me from checking these various indicators manually and just have a red/green dashboard that I could monitor.

I'm thinking something like this has got to exist in a ready-made solution.

# Answer
> 9 votes
All in all any monitoring tool would do for this case, monit, nagios, shinken, icinga, centreon, or even a crontab in bash could do...

# Answer
> 2 votes
Another option is to use Sensu. There are several plugins including a http one, that is able to check whether a response contains a certain string:

> check-http.rb -u http://my.site.com/health -q 'OK'

# Answer
> 2 votes
Try Sparky. This is a lightweight but powerful alternative to linux crontab. It comes with nice UI to see cronjob reports and statues. You can also run tasks remotely over ssh or through docker. The typical tasks you've mentioned are covered by existing DSL ( written on Perl6 ), for example:

"Does a file, with a maximum age, exist on some network share?"

```
bash "find file.txt -mmin -10", %( expect_stdout => "file.txt" );

```

"Does the response-body of a HTTP-request match a specific regex?"

```
http-ok 'http://sparrowhub.org', %( has-content => 'SparrowHub' );

```

And yet if it's not enough it's easy to create extensions.

PS Sparky only supports linux for now ...

---
Tags: monitoring, logging, operations
---


# Question
Title: What is the recommended way to monitor differences between multiple Kubernetes namespaces/clusters?
Given a Kubernetes cluster with two namespaces; what is the recommended way to monitor and sync differences in configmaps, deployments, ingresses etc. between the namespaces?

# Answer
**Edit**: The deploy script that is described in this answer is very much looks like Helmfile and we are now in the process of upgrading to Helmfile so that we can retire our scripts that run multiple Helm releases as that’s that Helmfile does well.

This is a bit of an alternative and indirect answer. Use infrastructure-as-code best practices to avoid environments getting out of sync. I believe people will find it a helpful alternative suggestion so a valid answer.

Prevention is better than cure. Don't let the namespaces get out of sync by treating your configuration as code and manage it via git. Helm is a package manager for Kubernetes configuration. It can handle arbitrary configuration objects such as Openshift extensions. This let's you setup a deployment pipeline and a git workflow that ensures your environments stay in sync. (Update: use Helmfile to manage multiple Helm releases).

By way of an example in our setup we have a Helm chart that just installs configmaps and another that just installs secrets. The actual values to install are passed in as a parameter like `my-app-secret-values.yaml`. So we have different yaml values all under git and a deploy script that iterates over them applying the helm chart that loads the values and updates the corresponding configmap or secret. Helm reads, compares, and only writes when something has changed. So the deploy script will result in only changing a secret or config map that has changed in git. All the configuration is under git and managed like code. Syncing configuration between environments can then be managed just like promotion of code. As a bonus helm creates a backup every version of every object it changes stored inside of Kubernetes. If let's you list the prior versions and rollback to a previous configuration. Also as everything is deployed from git all the history of all settings in all namespaces can be seen in git.

We also install and run out main applications as traditional Helm charts that are promoted through environments. Those helm charts reference the configmaps and secrets that are separately deployed when we want to adjust the settings in a given environment.
> 2 votes

# Answer
We are currently investigating https://github.com/roboll/helmfile to easily sync Helm charts across namespaces.
> 2 votes

# Answer
Heptio has open sourced a tool called Theseus to help with this. There's nothing built in to Kubernetes to do this, but check out the Theseus utility, you should be able to write a script to do this.
> 1 votes

---
Tags: kubernetes, orchestration
---


title: Billing my clients for CDN usage

text:
# Question
Title: Billing my clients for CDN usage
My clients have the ability to download pictures I uploaded to a CDN network. The CDN network is a PaaS service I found online.

I want to be able to distinguish between clients who download those pictures, so I can bill them by their usage.

Does anyone know such CDN service, that allows to bill (or at least measure) the clients of their (the CDN's) users?

# Answer
> 3 votes
I did something similar with Akamai a few years ago that might work for you and I have a Plan B in case this doesn't fit your needs.

# Plan A

* assign each client a unique hostname. So you might have `client0001.example.com` and so on.
* optionally, you can let the client create a `CNAME` within their domain. I was doing this for ad creatives so I wanted blame for content to go the originators of the content. Since it looks to end users like they were serving it from their own domain and my org was just in the middle of a CNAME chain that most people wouldn't notice I'm guessing the grief went where it belonged.
* then within the CDN have the host headers map to your client ID's
* get regular reports from the CDN for hits/bandwidth used per client ID
* bill accordingly

# Plan B

If you don't want to do all of that mucking with DNS or if your provider won't do the sort of mapping described there's still a way to make this work:

* give each client their own directory like `/cli0001/`, `/cli0002/`
* write your own reporting that downloads the raw logs from the CDN provider and then pulls the client ID's out of each directory name and totals it all up for you.

# Billing plans

There are different ways you can go about billing folks, but common choices are:

* create tiers so that the first megabyte is $X/byte and the next gigabyte is a little cheaper per byte and so on
* give everyone a quota for a fixed fee or call it "free" within a broader service. Then charge for going over the quota at a reasonable rate.

---
Tags: billing, cdn
---


# Question
Title: Dynamically change config files in c# Application based on environments(dev,QA,Prod) through Bamboo
I have a windows service written in .net c#. We have three different environments say dev, test and prod. I also have 3 different config files for 3 environments say devAppConfig , TestAppConfig and ProdAppConfig.

Earlier, we used to deploy manual, so we used to replace the config files and deploy the binaries.Now the deployments are to be automated for that we are using Bamboo.

Now my question is how do I dynamically change the AppConfig for different environment deployments.

I have 3 different stages in Bamboo naming DevDeploy , TestDeploy, ProdDeploy. When I run these stages, it has to change the config file and do the deployment, but I'm not sure how.

Can anyone guide me in the right direction for my issue?

Note: I could have added a prebuild event to copy the appropriate config file before building the code. But the main aim here is to able to use the same binaries across all the environments without building it every time.There has to be some setting somewhere (not sure where) to determine which environment it is being deployed and use the appropriate config file.

# Answer
In your deployment project configuration for each environment you could add a task prior to the actual deployment which would take care of setting up the right configuration file.

It could be a script simply copying/replacing the configuration file just as you currently do manually. The script could be invoked with some pre-configured argument to identify the environment or it could detect the environment via some environment variable configured for each the deployment, see Additional deployment environment options
> 1 votes

---
Tags: continuous-integration, configuration, builds, bamboo
---


# Question
Title: Where can I find a Dockerfile + required files to create a Flask/MySQL forum/webapp with users?
What I mean, is that I can simply run a command or download a .zip file that results in having a Dockerfile with the whole development environment baked in and includes not just Flask and MySQL set up, but some sort of sample nontrivial web application that already has tables made for users, posts, and that standard sort of content, along with the Python etc. code to go along with it. This would be like cloning a demo Python/Flask app from git, except it comes with a Dockerfile that configures the database, creates database users and tables, etc. A Dockerfile for other developers. Advice?

# Answer
This question seems to be in two parts:

1. How to deploy a flask+database app in Docker?
2. Is there a non-trivial application to use as example?

The second part is difficult to answer, it will depend on what you consider a good example. However, an example app with the features that you want can be deployed in several ways, and will work the same if it is in Docker or otherwise.

The Docker Labs flask example contains a basic example of how to deploy a Flask app in a Docker container.

Something more appropriate to what you are looking for though is the combination of web app and backing database. For this you likely want Docker-Compose \- specifically the example of Docker Compose for a Django and postgresql app
> 1 votes

---
Tags: docker, python, flask
---


# Question
Title: How to run Dockerized graphical applications on a headless server with packages required for display remotely on clients as part of a separate image?
In this **first section** I detail some of the terms I have used:

**Dockerized graphical applications**: A Docker container run from a Docker image which runs an application which (traditionally) uses x11 (or similar) to display the application on the host machine. More details in second section.

**packages required for display remotely on clients**: For example, a VNC server which allows a user of a local client (e.g. a laptop with desktop Ubuntu 18.04 installed) to navigate through the browser to an particular host-name and port.

---

---

In this **second section** I give an example of the usual way to run a Dockerized application, using gedit as an example:

Example Dockerfile:

```
FROM ubuntu:18.04
RUN apt-get update && apt-get install -y gedit
CMD gedit

```

Example Docker build command:

```
docker build -t gedit:latest .

```

Example Docker run command:

Once-off (can potentially be a harmful command, use with caution):

```
xhost +local:docker

```

Subsequently:

```
docker run --rm --name=gedit -e DISPLAY=$DISPLAY -v /tmp/.X11-unix/:/tmp/.X11-unix gedit:latest

```

Please note: There are some security concerns running `xhost +local:docker` so to reverse this run `xhost -local:docker`

Sub-question / curiosity: The above installs x11, etc into the image. Is there any way to have a separate image with all of those dependencies and have multiple containers share folders in order to satisfy the x11 dependencies?

---

---

In this **third section** I detail an example in which my question is relevant:

I want to keep the Dockerized app as explained in section 2, which runs well locally on a client's laptop but have it as part of a cluster of docker containers (using docker-compose for example) so that the image built from the dockerfile in section 2 is unmodified.

I want to include an additional container which has all the VNC and x11 packages required so that the client can run the application remotely.

**Which mechanisms allow the above to take place?**

# Answer
> 2 votes
You can't run a program running on x11 without a x11 buffer somewhere, be it vnc, xvfb or whatever. You will always need x11 libraries on the system running the program or it couldn't work at all. That's akin to trying to run a C program without libc on the system.

Using system libraries from a different container with a volume is a bad idea, as it create a hard dependency on both containers using the same libraries, and is likely to crash somewhere when both systems try to acquire a lock on the library to load it.

I assume with a lot of tweaks it should be doable, but the added complexity is a real footgun and create more dependencies than should exist.

---
Tags: docker
---


# Question
Title: Error while following Docker get-started part 3
I am learning Docker by following the get started on its official page.

I got stuck at part 3 when I tried to run

`docker service ps getstartedlab_web`

the containers fail to start. Checking the logs reveals this

`'app.py': [Errno 2] No such file or directory`

I saw that this issue has been reported elsewhere and the reason was because of the conflict between the `volume` in docker-compose.yml and `workdir` in dockerfile.

But my docker-compose.yml doesn't have that `volume` property, so what's the problem?

**Update** I understand the error message. But I don't know why such an error message was shown. I was able to push/pull and start the container alone with `docker run` (without docker-compose). The problem only arises when I tried to use docker-compose to scale it.

# Answer
So what's the problem?

> 'app.py': \[Errno 2\] No such file or directory

I think you already answered your question. The troubleshooting part, you suspect something, but you did not test it to see if it is really what you think.

Create two more files, requirements.txt and app.py, and put them in the same folder with the Dockerfile. This completes our app, which as you can see is quite simple. When the above `Dockerfile` is built into an image, `app.py` and `requirements.txt` is present because of that Dockerfile’s `COPY` command, and the output from `app.py` is accessible over HTTP thanks to the `EXPOSE` command. Get Started, Part 2: Containers
> 2 votes

---
Tags: docker, docker-compose
---


# Question
Title: What is the difference between Sysadmin and DevOps Engineer?
When applying for a job, usually you can find two types of similar jobs: Sysadmin Engineer and DevOps Engineer.

Both of them deal with server configuration and ensure the reliable operation of computer systems. It can be hard to tell the difference between the two. What are the main differences between them?

# Answer
Mainly DevOps is not a role (when used as such it's more a buzzword than a real role).

DevOps is roughly an organization pattern aiming at breaking the silo between developers and sysadmins.
The main goal is to build teams with devs and sysadmins (along with testers usually) responsible for a product (application) from its definition, architecture decisions up to the maintenance in run of this product.
Each member of the team will be part of the decision on the whole life-cycle of the product, a dev will do some sysadmin tasks in production, and a sysadmin will participate in the design phase of the product to avoid caveats from the infrastructure perspective for example.

At ideal, a sysadmin would also be part of the development team for the product, in real world sysadmin code more on the configuration around the product and monitoring solutions, but being able to voice concerns to other members of the team avoid a lot of misunderstanding on the deployment process.
> 66 votes

# Answer
## Short Version

DevOps a combination of organisational culture, Agile/Lean ways of working and software automation that when applied to Systems Administration and Operations allows these functions to operate with the same level of Agility as Agile or Lean Development Teams.

## Long Version

The ideas behind DevOps came out of the Systems Administration, Operations and Agile communities, specifically, Patrick Debois gave a presentation at Agile2008 entitled 'Agile Infrastructure' where he highlighted the disparity between the way the three functions within an organisation operate:

1. **Agile Development Teams** \- Agile teams writing code.
2. **Systems Administrations Teams** \- Building infrastructure to run the software.
3. **Operations Teams** \- Supporting applications and infrastructure in Production/Live.

Debois' proposal was to unify the three ways of working together, specifically moving Systems Administration teams and Operations teams from a Waterfall Model to an Agile Model. To that end, Debois setup DevOpsDays 2009 in Ghent, Belgium inadvertently coining the phrase **DevOps**.

The idea of DevOps resonated with the Authors of the VisibleOps series of books: Gene Kim, George Spafford and Kevin Behr; who went on to write The Phoenix Project and The DevOps Handbook. Both books explore how Agile and Lean can positively impact Systems Administration and Operations teams.
> 23 votes

# Answer
As a *DevOps* Engineer coming from an Operations background, you will have moved from building and deploying servers and software *manually* to **scripting** the installation of software onto your servers with the likes of BASH, PowerShell, Python etc. After a while, you would realise how cool scripting is and start to explore more sophisticated ways to **automate deployment**.

Eventually, you would have settled on a *Chef, Puppet, Ansible* or other **configuration management** tool to help manage the state of your fleet of systems. As your skills with the automation of application deployment and system management matured, along with your tools, you have more recently moved into the realm of '**Infrastructure as Code**' and use it to not only automate the deployment of software but the infrastructure and environments required to drive the software during the business's shift to the Cloud.

Now you are cooking with gas. Over time you have been introduced to the benefits of using developer centric tooling such as **source control** to manage the modules, recipes and templates that make up your arsenal of deployment and management tools.

When you shifted into the *DevOps* team you were exposed to the software development life cycle and the concept of **continuous integration**. Boy those developers were releasing changes quickly and to keep up you found yourself working more closely with the devs! You experienced the urgency placed on the development team to change things ALL-THE-TIME which grates against the old operational paradigm of "*if it ain't broke, don't fix it*". No more bragging about system uptime anymore, you are into *disposable* infrastructure.

You noticed that the move to *DevOps* was more than working with the *devs*, or using *new tools* and *techniques*, but there was a distinct **cultural** shift in the team, one that permeated through the organisation at large. You were working as a close-knit team with *shared responsibilities*, *shared tooling* and *shared goals*.

You took your skills in automated deployment and massaged them into the "**CICD**" pipeline being orchestrated by a "**continuous integration server**" like *Jenkins*, *Bamboo* or *Code Pipeline*. Now, when the developers push new code, your scripts, tools and templates stand up new environments on demand, trigger testing frameworks to do their thing and tear down the pre-production environments after the green lights are lit on the release, adhering to the ideas of "**continuous delivery**".

As the new code snakes its way through the CICD stages, you, the developers and the business gain confidence that the update will not break when released to production. There is some way to go before the team gets to "**continuous deployment**", you still need to settle on the finer points of automating the **blue/green** deployment capability, and the decision is mostly a business one. For the time being you are content that the number of calls at 3am have subsided and the sev-1's and sev-2's dwindle.

Even if you do get a sev-1, you are not pulling all-nighters any longer with the managers breathing down your back - you can easily release the previous version through the CICD pipeline and get the system online again in short order. The business has noticed that the **stability** of the IT systems has improved despite the *velocity of changes*.

You marvel at the way you manage the resources needed to drive the software in your business, especially when you think back to how it used to be and amount of blood you left behind on rails in the datacenter...
> 13 votes

# Answer
**Sysadmin vs. DevOps (personal view)**

Some companies talk about Dev, Ops and Test. If something needs to be tested then they say: "Test should do that". If something needs to be developed, Dev will do that and if software needs to be deployed, Ops will do that.

In my opinion and what I have experienced at several companies is that this results in a "throw it over the wall" mindset that results in friction between people and teams. Personally, I sometimes feel that people work individually and say this is what I did, I have nothing to do instead of working as a team.

According to me DevOps means that everybody in a team is responsible and busy with development, testing and operations. There is no I in a team and no separate departments. Everybody should release. Of course there are specialties, but in my opinion everybody should be able to do at least 25% of some work in every area. E.g. if someone was a Developer back in the day then one should be able to change some configuration management code, e.g. chef and deploy software.

**References**

Sysadmin

According to Wikipedia:

> A system administrator, or sysadmin, is a person who is responsible for the upkeep, configuration, and reliable operation of computer systems; especially multi-user computers, such as servers.
>
> The system administrator seeks to ensure that the uptime, performance, resources, and security of the computers he or she manages meet the needs of the users, without exceeding the budget.
>
> To meet these needs, a system administrator may acquire, install, or upgrade computer components and software; provide routine automation; maintain security policies; troubleshoot; train or supervise staff; or offer technical support for projects.

DevOps

According to Wikipedia:

> DevOps (a clipped compound of "development" and "operations") is a software development and delivery process that emphasizes communication and collaboration between product management, software development, and operations professionals. It supports this by automating and monitoring the process of software integration, testing, deployment, and infrastructure changes by establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.

DevOps

DevOps Toolchain
> 6 votes

# Answer
A system administrator is responsible for maintaining and configuring servers and their responsibility is to ensure the user with the performance, uptime and security that they are looking for. Defining the role of a DevOps engineer is a little more difficult since there is no formal career path and DevOps itself can have many forms.

A DevOps engineer can be, for example, a developer who is interested in network and deployment operations or a system administrator who has a passion for coding and scripting. The transition from system administrator to DevOps engineer is not very difficult, in fact, this article does a very good job describing the process.

Many people would even argue that this transition from system administrator to DevOps engineer is essential since the position of system administrator will become obsolete in the future. Even though there are plenty of legacy servers that need maintenance and system administrators possess a lot of “tribal knowledge” the sysadmin position will become more scarce in the future.
> 2 votes

# Answer
There is going to be servers which you do not hear running in data centers. Everything is going to be software. Storage, network, systems, security, data centers; SDN, firewalls, NFV, storage, servers, etc. Sysadmins without a software development background, SDLC experience (I don't even mean scripting Perl, Powershell, etc) will probably vanish. Distributed, scalable and virtualized environments, which is mostly cloud, grow horizontal not vertically.

---

I dare to say Sysadmins grow vertical, DevOps (or OpsDev) grow horizontal. You can see the same pattern how microservices evolved from monoliths. I would rather choose DevOps engineer from software team not from operations/system team.

Because operations/system team just runs what software team builds.

* Sysadmins do not build/compile Linux/FreeBSD/windows kernel etc. just as software engineers build/compile applications.
* Sysadmins do not go through the software development life cycle (SDLC).
* Sysadmins are not part of the production pipeline (CI/CD process).
* Sysadmin start working after CI/Continuous Delivery/Deployment ends.

And if you break and assign Deployment/Delivery, it could be a broken pipeline
the software team is the creator system/operations team are the runners/caretakers mostly.

It sounds like there is no server/system to administer, no sysadmin need.

Serverless computing is a cloud-computing execution model in which the cloud provider acts as the server, dynamically managing the allocation of machine resources. Pricing is based on the actual amount of resources consumed by an application, rather than on pre-purchased units of capacity Serverless computing

Someone from the software team already knows how to build, maintain even how to code (SRE vs DevOps/OpsDev).

---

I wonder why it is called DevOps but not OpsDev? is it related to the direction between the two?

**\*In the middle of nowhere, I did not start writing about software-defined storage, this is in reaction to a now deleted comment about it ***

### About software-defined storage

* Software-defined storage (SDS) is a marketing term for computer data storage software for policy-based provisioning and management of data storage independent of the underlying hardware. Software-defined storage
* EMC announced its first ever open source product: Project CoprHD. CoprHD is a Software Defined Storage automation and management controller, and EMC’s recent decision to open source it is at the crux of our strategy for delivering greater value to global businesses as we enter into an area of growth and extreme change. As the world’s leader in storage and information management, it behooves EMC to lead the way on Software Defined Storage (SDS).
* CoprHD is an open source software-defined storage controller and API platform. It enables policy-based management and cloud automation of storage resources for the block, object and file storage providers CoprHD
> -1 votes

---
Tags: terminology
---


# Question
Title: How to use Jenkins github user credentials to access github api in my script?
I have a Jenkins job `foo` that works perfectly well. It is authenticated with `jenkins_foo_user`. It is able to clone the repo and checkout the right branch.

In my codebase, I have another script - `query_github.py` a script used to query Github API to get information such as commits and pull request. To be able to use this API I need an access token.

I want to be able to use the jenkins user access token to authenticate and access the github credentials.

How do I do this?

Note: Using the Github Enterprise version

# Answer
You want the Credentials Plugin.

The administrator of the Jenkins instance can add the plugin, then add the credential (access token) that the job needs. Then, in the job, use the credential as a variable.

See the documentation for more details.
> 0 votes

---
Tags: jenkins, git, jenkins-plugins, github, authentication
---


# Question
Title: Identifying a benchmark to compare overall performance of VM virtualization vs hardware
Which performance index is most suitable to find out real performance of your VM limited by sharing and quotas?

For example, if you have a physical server with 4 cores and 16 GB RAM, its performance limit might be higher than if you have same on AWS or in some VMWare product.

How to compare to find out the gap? Is there an overall index or better to use some reference applications' performance as benchmark? (tough, though, if you still going to develop that application)

# Answer
You should do a comparison of commercial virtualization products on the same hardware resources, OR benchmarking commercial virtualization products individually, there are ways to do it, you can even script it.

For example, VMmark is a free tool used to measure the performance and scalability of virtualization platforms VMmark

Since your question is very generic, you should clarify real performance, overall performance. It might mean "real performance in virtualization".
> 2 votes

# Answer
IMHO an absolute benchmark can be rather useless/misleading, I'd always consider the specific context of the application intended to be executed on the platforms being benchmarked and its performance requirements and use only benchmarks relevant to that application context.

A few examples:

* if the application doesn't make a lot of use of the CPU and/or its memory, comparing the number of cores and/or the amount of RAM or benchmarking CPU/RAM use would not be very relevant
* if the application accesses data on AWS S3 using an AWS platform may provide better performance than using an on-premises bare metal or VMware platform even if the AWS platform's absolute I/O performance is below that of the on-premise platform

Side note: by always starting with the application context in mind potentially more performant solutions other than bare metal/VM/Infrastructure as a Service (IaaS) can be selected, for example Platform as a Service (PaaS) or serverless/Function as a Service (FaaS) solutions. It all depends on the application.
> 1 votes

---
Tags: cloud, metrics, performance, distributed-systems
---


# Question
Title: How to Remove a Kubernetes (GKE) Cluster node-pool without down-time
I am using Google Kubernetes Engine. I have a multiple Kubernetes cluster node-pools. I want to delete one of them.

Is there a way that I can first migrate all the pods on one node-pool to another node-pool through the same kind of rolling update methods found in other features?

I thought it would do this automatically for me. But when I deleted the node-pool I got downtime, luckily this was not a production environment.

I would like to know if there is a way to automatically migrate and move all pods from one node-pool to another node-pool and slowly roll the change when a node-pool is removed?

Thanks

# Answer
> 3 votes
If you cordon and drain the nodepool before deleting it, then you can avoid downtime. I use the following script (shamelessly taken from the lazyweb elsewhere and adapted to my needs):

```
oldpool=pool-1
oldnodes=$(k get no --selector='cloud.google.com/gke-nodepool='$oldpool -o json | jq .items[].metadata.name -r | xargs)

kubectl cordon --selector='cloud.google.com/gke-nodepool='$oldpool

for n in $oldnodes; do
  echo "draining $n"
  kubectl drain --delete-local-data --ignore-daemonsets $n
  echo "----------------------------"
done;

```

# Answer
> 1 votes
1). Inside the node pool(lets call it ***old\_node\_pool*** ) that you want to delete, First Cordon all the nodes one after the other.

```
 kubectl cordon <name_of_node_1>
 kubectl cordon <name_of_node_2>
           ........

```

2). Drain all the nodes of the ***old\_node\_pool***.

```
 kubectl drain <name_of_node_1>
 kubectl drain <name_of_node_2>
           ........

```

3). Wait to see all the pods are up and running on the new node pool.

4). Delete the old node pool.

---
Tags: kubernetes, google-cloud-platform, google-compute-engine
---


# Question
Title: What tools do you use to monitor applications and business-specific data? (apart from Jenkins)
Currently, we use Jenkins to run small node.js scripts in order to check if applications are doing what they're supposed to. Like, we have some routines which should run in an interval and we need to check if these routines have run and have done everything we expect.

Why Jenkins? Mostly because we can just login and look at the dashboard and see the health status of all our jobs at a glance; we can set up notifications; and because we don't know other tools to execute our scripts in a "monitoring" way.

But we know we're misusing Jenkins and the problems we have faced (like an attack we suffered which used our Jenkins instance to attack a credit card company and the constant instability) are making us look for another tool (or tools).

But all I could find was performance monitoring tools. What tools do you guys use to monitor things other than performance?

# Answer
> 4 votes
We use Datadog, with metrics pushed from important operational processes using the `statsd` protocol ( http://docs.datadoghq.com/guides/metrics/ ), so we aren't too deeply tied to Datadog on the application side.

In Datadog's interface we can then configure monitors to alert on any number of things, including anomaly detection if defining thresholds is impossible.

So for instance, we have many tasks that run on a hourly basis, they'll post a datapoint to a given metric (`app.task.task_name`), and we will then monitor that this metric is consistently posting at one data point per hour. If the rate changes, we get alerted.

# Answer
> 1 votes
I use Sparky to monitoring **internal** applications statuses. VSTS builds success, ssl certificates, source code consistency and so on. It fits ideally as I have a **one dashboard** to give me an essential information on most critical parts of the systems split across different projects and applications.

# Answer
> 0 votes
Where do you have your infrastructure? I've found that AWS Cloudwatch is pretty good in most simple cases. Zabbix is a good open source tool too. Of course there's Solarwinds NPM but that's quite expensive.

# Answer
> 0 votes
I could maybe suggest looking into Prometheus:

https://prometheus.io/docs/instrumenting/clientlibs/

You can pull metrics from your app and put alerting on it, add grafana dashboards, etc.

# Answer
> 0 votes
I am currently using TotalCloud which integrates with AWS and CloudWatch. It is used for infrastructure monitoring and management in a single pane of glass.

---
Tags: jenkins, monitoring
---


# Question
Title: Availability/Reproducibility of Ubuntu Minimal as Docker image
While for Debian you find on Docker Hub "slim" distributions, it seems not the case with Ubuntu.

Now there is a "minimal Ubuntu for public clouds" distro from Canonical, but I can find just cloud VM images; is there a Dockerfile for that, as well?

# Answer
The answer to the question "is there a Dockerfile for that, as well?"

https://github.com/tianon/docker-brew-ubuntu-core/tree/dist-amd64

Note their git branches contain the actual release build based on architecture.
> 2 votes

# Answer
Most Dockerfiles start from a parent image. If you need to completely control the contents of your image, you might need to create a base image instead. A base image either has no `FROM` line in its `Dockerfile`, or has `FROM` scratch.

### Create a full image using tar

In general, start with a working machine that is running the distribution you’d like to package as a parent image, though that is not required for some tools like Debian’s Debootstrap, which you can also use to build Ubuntu images.

There are more example scripts for creating parent images in the Docker GitHub Repo:

```
BusyBox
CentOS / Scientific Linux CERN (SLC) on Debian/Ubuntu or on CentOS/RHEL/SLC/etc.
Debian / Ubuntu

```

### Create a base image

I guess after creating you can export this on docker hub with `Dockerfile`.
> 0 votes

# Answer
You can walk backward in DockerFile. You can uninstall/cleanup "groups" "packages" to slim down. You can create your custom minimal image from that cloud image.It is like just a snapshot of a specific image at some point.

This is the minimal position. This is when the baby is born.

docker pull hello-world

Dockerfile-linux.template

You may want to look at kickstart/preseed if you want deep-diving.It is about being an official image or some image on which you have full control. \]
> 0 votes

---
Tags: linux, dockerfile, cloud
---


# Question
Title: How to change the owner of VOLUME directory in Dockerfile?
I've got the following `Dockerfile`:

```
FROM ubuntu:xenial
RUN useradd -d /home/ubuntu -ms /bin/bash -g root -G sudo -p ubuntu ubuntu
WORKDIR /home/ubuntu
USER ubuntu
VOLUME /opt/myvolume

```

Which I built it:

```
$ docker build -t vol-test .
Sending build context to Docker daemon  2.048kB
Step 1/5 : FROM ubuntu:xenial
 ---> 0b1edfbffd27
Step 2/5 : RUN useradd -d /home/ubuntu -ms /bin/bash -g root -G sudo -p ubuntu ubuntu
 ---> Using cache
 ---> d82e3ecc5fe8
Step 3/5 : WORKDIR /home/ubuntu
 ---> Using cache
 ---> ab1db29ee8bf
Step 4/5 : USER ubuntu
 ---> Using cache
 ---> 129393a35d9e
Step 5/5 : VOLUME /opt/myvolume
 ---> Running in 691a4cbd077e
Removing intermediate container 691a4cbd077e
 ---> 11bc9e9db9d3
Successfully built 11bc9e9db9d3
Successfully tagged vol-test:latest

```

However, when run, the `/opt/myvolume` directory is owned by `root`, not `ubuntu`:

```
$ docker run vol-test id
uid=1000(ubuntu) gid=0(root) groups=0(root),27(sudo)
$ docker run vol-test find /opt/myvolume -ls
    66659      4 drwxr-xr-x   2 root     root         4096 Jul 18 23:02 /opt/myvolume
$ docker run -u ubuntu vol-test find /opt/myvolume -ls
    66940      4 drwxr-xr-x   2 root     root         4096 Jul 18 23:12 /opt/myvolume

```

because it's created during the run.

Is it possible to define or change the default owner of `VOLUME` directory in `Dockerfile`?

I'm running it on macOS and Linux.

# Answer
As stated in the documentation, VOLUME instruction inherits the directory content and permissions existing in the container, so you can workaround the problem with a dockerfile like this:

```
FROM ubuntu:xenial
RUN useradd -d /home/ubuntu -ms /bin/bash -g root -G sudo -p ubuntu ubuntu
RUN mkdir /opt/myvolume  && chown ubuntu /opt/myvolume
WORKDIR /home/ubuntu
VOLUME /opt/myvolume

```

The creation of the directory has to be done as root (to be able to write within /opt).
> 27 votes

# Answer
I had a similar problem, this worked for me:

1. Write Docker file with:

```
# Create app layer:
FROM python:3.4
# Create app user & group "testuser" with IDs:
RUN groupadd -r testuser --gid 1234 && useradd -d /home/testuser -ms /bin/bash -r -g testuser testuser --uid 1234
# Create "testuser" working dir:
WORKDIR /home/testuser
# Make working dir known to Python
ENV PYTHONPATH "${PYTHONPATH}:/home/testuser"
# Create & mount shared storage:
RUN mkdir /var/run/testuser-storage
VOLUME ["/var/run/testuser-storage"]
# Start container as "testuser":
ENV NAME testuser
ENV HOME /home/testuser
USER testuser

```
2. Run these bash commands:

```
# Create the same user & group "testuser" with IDs on host:
getent group testuser > /dev/null || /usr/sbin/groupadd -r testuser --gid 1234
getent passwd testuser > /dev/null || /usr/sbin/useradd -r -g testuser -d /var/lib/testuser -s /bin/nologin testuser --uid 1234
# Create shared storage dirs on host:
mkdir /var/run/testuser-storage
chown -R testuser.testuser /var/run/testuser-storage
# Build and run "testuser" Docker image:
docker build . -t testuser
docker run --net host --name testuser -v /var/run/testuser-storage:/var/run/testuser-storage -d testuser
# Change ownership of shared volume dir to the be the GID of "testuser"
chown -R 1234:1234 /var/run/testuser-storage

```

Note the names, UID's ,GID's must be the same for the Docker user and the host user. The last bash command tells the Docker image that that host user is the same as the Docker shared volume dir user, so that file dir then becomes owned by the "testuser" in the Docker container.
> 5 votes

---
Tags: docker, dockerfile, permissions
---


title: Whitelist Travis CI to Google Cloud SQL

text:
# Question
Title: Whitelist Travis CI to Google Cloud SQL
I'm having trouble working out how exactly I can whitelist my TravisCI runner for testing a python adaptor talking to a Google Cloud SQL service. The Cloud SQL will only accept connections from IPs that have been whitelisted ahead of time, though I'm not sure how reserve one from Travis (if such a thing is even possible). How should I go about this?

# Answer
> 2 votes
You can find the IP addresses of the Travis build machines here. If you add the IP addresses of the Travis infrastructure you use to your whitelisted IPs in Cloud SQL it should work. Keep in mind these IP addresses can change in the future though.

---
Tags: google-cloud-platform, travis-ci
---


# Question
Title: In a modern IaaS microservice based system is Zookeeper still a useful tool?
I've used zookeeper before as a configuration management service to keep track of application settings, feature toggles etc. However, that was back in the day before microservices and AWS/Azure/GCP. Since zookeeper tends to need multiple VMs to operate, is it still a good tool in the modern ecosystem?

Are there better ways to handle feature toggles and centralized configuration settings for multiple environments using a more modern (aka serverless style) system?

# Answer
It's difficult to say whether other tools will be *better* if there is nothing really concrete to measure against. A good, partially biased answer comes from Hashicorp's Consul

This seems to describe why Consul would be inherently better than Zookeeper, from the point of view of ease of use:

> ZooKeeper et al. provide only a primitive K/V store and require that application developers build their own system to provide service discovery. Consul, by contrast, provides an opinionated framework for service discovery and eliminates the guess-work and development effort

... scalability:

> ZooKeeper provides ephemeral nodes which are K/V entries that are removed when a client disconnects. These are more sophisticated than a heartbeat system but still have inherent scalability issues and add client-side complexity. All clients must maintain active connections to the ZooKeeper servers and perform keep-alives.

... and simplicity :

> Consul provides first-class support for service discovery, health checking, K/V storage, and multiple datacenters. To support anything more than simple K/V storage, all these other systems require additional tools and libraries to be built on top. By using client nodes, Consul provides a simple API that only requires thin clients. Additionally, the API can be avoided entirely by using configuration files and the DNS interface to have a complete service discovery solution with no development at all.

It does not address the need for extra resources, but since the architecture is different, I would assume the need for resources is also less:

> Instead of only having server nodes, Consul clients run on every node in the cluster.

Hope this can provide some form of answer.
> 3 votes

---
Tags: configuration-management, zookeeper
---


# Question
Title: AutomatedTest selenium chromedriver not running after update chrome
After update of chrome, I have an issue with E2E automation tests. When I run test, it open chrome website with error: CONNECTION REFUSED after refresh of website test passed. Did you had similar issue? Why I can’ connect to selenium for first time but after refresh it working?

# Answer
This is a good question for stackoverflow as well.

selenium tests often break because chrome is updated automatically and the chromedriver does not, or vice-versa.

Just make sure the chromedriver you're using fits the chrome version that you have.

http://chromedriver.chromium.org/

Note version 2.43 and chrome 70 are not exactly compatible even though it says they are. If you have chrome 70, I suggest chromedriver 2.42.

I use docker which I manually maintain to avoid automated chrome updates, and I use protractor which has webdriver tool where I can tell it to freeze the chromedriver version. So nothing gets updated unless I say so, and it's been working well for a year and a half now.
> 1 votes

---
Tags: selenium
---