# Question
Title: How to enable the npm cache in a Jenkins build slave that runs in docker?
I have a Docker image, let's call it `frontend.image`, that I use for a Jenkins build slave. The Jenkins Docker plugin will spin up a container from this image and build artifacts inside the container. This all works great. In this case, the `frontend.image` is used to build an AngularJs app. Part of building this Angular app is to install npm packages the app requires.

This process, npm install, seems to take a long time, 3 minutes it seems, npm always installs every package every time.

So I added a volume for my slave, it is a host mounted volume, the Docker plugin will use this volume every time it runs the frontend container:

The user that executes the command `npm install` is `jenkins`. npm keeps a cache which you can find with command `npm config get cache` which outputs `/home/jenkins/.npm`

That is why I have the host volume `/slaves/volumes/tsl.frontend:/home/jenkins` mounted to my web container slave.

I build my Angular app using a Jenkins project, builds no problem, many npm packages are installed. If ssh into my Docker host and run cmd `ls /slaves/volumes/tsl.frontend` I see lots of npm packages. This means my host volume mount for the slave worked.

Now I build the Jenkins project again, npm installs every single package again, even though the Docker slave build container is using the volume host mount. I can even confirm by bashing into the slave container with cmd `docker exec -it <some_clever_random_container_id> bash` then cmd `su jenkins` then cmd `npm cache ls` which lists many npm packages that are cached.

So, even with my host mount volume, which has permissions `chmod 777` by the way so there is no permissions issues, I cannot get `npm install` to use the cache.

In my Jenkins build, which spins up the Docker slave container, the first cmd I run is `npm cache ls` and many packages are listed, doesn't this mean my host volume is working as expected and the npm cache index has integrity aka not corrupted?

I have tried the regular `npm install` cmd, which, when I run on my localhost machine installs all the packages the first time and almost no packages the next time. And also the npm cache "hack" `npm --cache-min 9999999 install`, taken from this SO answer as well as cmd `npm --skip-installed --cache-min 9999999 install`

A related question was posted on StackOverflow.

# Answer
> 7 votes
I finally solved this by using Docker image layer caching for the npm install, following this answer

This means I moved the npm install out of the Docker slave image and into the actually frontend image, here is my final Docker file that truly caches the npm install in between builds if package.config has no changes:

```
FROM centos:7
MAINTAINER Brian Ogden

# Not currently being used but may come in handy
ARG ENVIRONMENT
ENV NODE_VERSION 6.11.1

RUN yum -y update && \
    yum clean all && \
    yum -y install http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm \
    yum -y makecache && \
    yum -y install nginx-1.12.0 wget

# Cleanup some default NGINX configuration files we don’t need
RUN rm /etc/nginx/conf.d/default.conf

#############################################
# NodeJs Install
#############################################

#Download NodeJs package
RUN wget -q -O - https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz \
    | tar --strip-components=1 -xzf - -C /usr/local

# https://stackoverflow.com/a/35774741/1258525
# use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
COPY ./package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir /app && cp -a /tmp/node_modules /app/

WORKDIR /app
COPY . /app

RUN npm run build-$ENVIRONMENT

RUN cd /app && cp -a dist/* /usr/share/nginx/html
COPY ./docker/conf/frontend.conf /etc/nginx/conf.d/frontend.conf
COPY ./docker/conf/nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

CMD ["nginx"]

```

# Answer
> 0 votes
Another approach you can do is set up a nexus repository server where you host your npm modules and proxy the external ones. It does not leverage the cache, but because the resources are within your local network or maybe in the same swarm it shouldn't take as long.

---
Tags: docker, jenkins, npm
---


# Question
Title: Amazon AWS - Amazon Connect IAM policies
I was trying to lock down user account just to Amazon Connect, Lambda and S3. I was able to give permissions to S3 and Lambda but I cannot find permissions for Amazon Connect. I have went through the documentation etc. but I cannot find out the proper name of that service so I will be able to write a policy for it. Would you be able to help?

# Answer
There's a generator for policies in IAM (Into IAM -\> Policies -\> Create), choosing connect gives me the following policy:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "connect:DescribeInstance",
                "connect:CreateInstance",
                "connect:ModifyInstance",
                "connect:ListInstances",
                "connect:GetFederationTokens",
                "connect:DestroyInstance",
                "connect:GetFederationToken"
            ],
            "Resource": "*"
        }
    ]
}

```

There's a warning on the action part saying:

* connect:DescribeInstance action requires 7 more actions to provide full permissions
* connect:CreateInstance action requires 13 more actions to provide full permissions
* connect:ModifyInstance action requires 10 more actions to provide full permissions

Each of them says:

To allow an entity to call 'DescribeInstance', grant all of the following required permissions.

* firehose:DescribeDeliveryStream
* firehose:ListDeliveryStreams
* kinesis:DescribeStream
* kinesis:ListStreams
* kms:DescribeKey
* kms:ListAliases
* s3:ListAllMyBuckets

To allow an entity to call 'CreateInstance', grant all of the following required permissions.

* ds:CreateAlias
* ds:DeleteDirectory
* ds:DescribeDirectories
* firehose:DescribeDeliveryStream
* firehose:ListDeliveryStreams
* kinesis:DescribeStream
* kinesis:ListStreams
* kms:CreateGrant
* kms:DescribeKey
* kms:ListAliases
* kms:RetireGrant
* s3:CreateBucket
* s3:ListAllMyBuckets

To allow an entity to call 'ModifyInstance', grant all of the following required permissions.

* firehose:DescribeDeliveryStream
* firehose:ListDeliveryStreams
* kinesis:DescribeStream
* kinesis:ListStreams
* kms:CreateGrant
* kms:DescribeKey
* kms:ListAliases
* kms:RetireGrant
* s3:CreateBucket
* s3:ListAllMyBuckets

Which once compiled gives the following policy when not filtering on any resource:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "connect:DescribeInstance",
                "connect:ModifyInstance",
                "connect:GetFederationTokens",
                "s3:CreateBucket",
                "kinesis:DescribeStream",
                "kms:RetireGrant",
                "connect:DestroyInstance",
                "firehose:DescribeDeliveryStream",
                "kinesis:ListStreams",
                "connect:CreateInstance",
                "s3:ListAllMyBuckets",
                "connect:ListInstances",
                "kms:ListAliases",
                "ds:DescribeDirectories",
                "kms:DescribeKey",
                "firehose:ListDeliveryStreams",
                "ds:CreateAlias",
                "kms:CreateGrant",
                "connect:GetFederationToken",
                "ds:DeleteDirectory"
            ],
            "Resource": "*"
        }
    ]
}

```
> 3 votes

---
Tags: amazon-web-services, amazon-s3, aws-lambda
---


title: openshift fails to deploy jenkins

text:
# Question
Title: openshift fails to deploy jenkins
When I try to deploy jenkins-ephemeral on openshift 1.5, it fails with the following error message

```
error: couldn't get deployment jenkins-3: Get https://172.30.0.1:443/api/v1/namespaces/projectname/replicationcontrollers/jenkins-3: dial tcp 172.30.0.1:443: getsockopt: network is unreachable

```

I have services enabled

```
oc get services
NAME           CLUSTER-IP       EXTERNAL-IP   PORT(S)     AGE
jenkins        172.30.86.254    <none>        80/TCP      59m
jenkins-jnlp   172.30.132.179   <none>        50000/TCP   59m

```

but from the node I can't telnet to port 80 of 172.30.86.254

```
telnet: connect to address 172.30.86.254: Connection refused

```

but the 443 port of the same machine is reachable.

# Answer
This seems to be a proxy issue in your config. It is solved by setting NO\_PROXY as follows:

> Edit the proxy environment variables in the OpenShift Origin control files. Ensure all of the files in the cluster are correct.

```
HTTP_PROXY=http://<user>:<password>@<ip_addr>:<port>/
HTTPS_PROXY=https://<user>:<password>@<ip_addr>:<port>/
NO_PROXY=master.hostname.example.com,10.1.0.0/16,172.30.0.0/16

```

> Supports host names and CIDRs. Must include the SDN network and service IP ranges 10.1.0.0/16,172.30.0.0/16 by default.
> 0 votes

---
Tags: jenkins, kubernetes
---


# Question
Title: How do you change a secret that half of your production applications use right now?
The context of the question is Secrets management.

As the title says, how do you change a secret that half of your production applications use right now?

# Answer
This question ties in heavily with your other question with where the secrets are set and where they retrieved from.

I assume your question is asking about 'canary deployments' where you change the config for only a small portion of your apps, to test things out before deploying everywhere.

If you follow the best practice of using the 12 Factor app's recommendation of 'environment variables', then changing secrets becomes very easy. At the very worst, you could log into the production server, update the ENV VAR, and then restart your app manually. In Kubernetes, this is even easier, as you can edit the ENV variables from the UI, and then restart the pod.

However, you don't want to be doing that manual process if you can avoid it. If your Configuration Management tool sets the ENV VAR, then you can tell it to update a specific group of servers only. If your build server injects the ENV vars, and it supports canary deployments, then you can easily just do a canary deployment like normal.
> 2 votes

# Answer
i wouldn't 'change' the secret at least in the short term but roll out a second secret with the same permission set as the original and roll that out through your configuration management or secrets management. ideally following @boomshadow 's answer. once the new credential is fully rolled out invalidate the old one/s repeat as necessary.
> 0 votes

---
Tags: amazon-web-services, security, microservices, github, encryption
---


# Question
Title: saltstack: archive.extracted template jinja
I have an archive on my master and want to extract it on the minion. For this it is fine I use :

```
  archive.extracted:
    - name:  'c:\PACKAGE'
    - source: salt://myfile.zip
    - archive_format: zip
    - enforce_toplevel: False
    - failhard: True

```

the problem is that I need to apply jinja template on some files of the zip file content.

But there is no template property on archive.extracted and I can't find a way to apply jinja template after it is extracted on minion server...

any idea ?

# Answer
There is no much choices regarding your situation, I see two of them :

1. You ask your dev dpt to add your jinja templating to their builds (which may be hard or even impossible)
2. You can use salt module such as `file.append` or `file.replace` to add / replace your jinja code inside (this is not very clean that's the least we can say)

If you'd share an example that may help us to give some good advice
> 1 votes

---
Tags: salt
---


# Question
Title: Docker swarm replica stays 0/1 and service stays in status new
I have a problem with my docker swarm. I created a new service with the following command:

```
docker-compose config | docker stack deploy -c - serviceNameX --with-registry-auth

```

The docker-compose file looks like that:

```
version: '3.1'
services:
  serviceNameX:
    image: ${DOCKER_REGISTRY}imageNameX:${VERSION}
    deploy:
      resources:
        limits:
          memory: 2gb
    env_file:
      - ../common.env
      - .env

networks:
  default:
    external:
      name: ${DOCKER_NETWORK}

```

fyi: docker-compose config is used because docker stack deploy can't expand environment variables. At least to my knowledge.

docker service ps serviceNameX

```
ID                  NAME                              IMAGE                                                         NODE                DESIRED STATE       CURRENT STATE       ERROR               PORTS
ify7vhl0l2m8        serviceNameX.1   nexus.local/imageNameX:1                                                                           Running             New 2 hours ago

```

docker service ls

```
afnjo97f6nuf        serviceNameX                      replicated          0/1                 nexus.local/imageNameX:1

```

All nodes are Ready and Active

There is no indication or information of an error. Usually docker service ps shows if an error occurs. Is there another useful log?

# Answer
> 2 votes
Try do pull your image manually first.

```
docker pull ${DOCKER_REGISTRY}imageNameX:${VERSION}

```

And then re-run your compose-file

It may be a bug using a private registry, or something simple as you don't have enough disk space to pull the image.

---
Tags: docker, docker-compose, docker-swarm
---


# Question
Title: Automating creation of new VM and pushing code to server
## Automating creating of new VM and pushing code to server

Hi all,

I am writing a integration test for a client-server application. To simulate the production server setup, and to update the code, I set up VMs on my computer. The application runs on a Windows server, so I have to use systems compatible with Windows.

## STEP 1: Set up VM properly:

1. Download the OVA file from MS
2. create a new VM from the OVA file
3. reset the mac id
4. set a static dns ip address and a VM ip address
5. open up the port 22
6. install bitvise ssh server
7. transfer public keys to the VM
8. install python 3.5.2
9. update `pip`
10. install `virtualenv` systemwide

### STEP 2: Deploy code to VM:

1. create a requirements file using pip using: `pip freeze -r requirements`
2. zip the entire repo using powershell using: `powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"`
3. using `pscp` I transfer the files to the VM using: `pscp ./atu.zip ipython_user@10.15.33.31:/C:/Users/ipython_user/Desktop`
4. unzip the files again using powershell using: `powershell.exe -nologo -noprofile -command "& { Add-Type -A 'System.IO.Compression.FileSystem'; [IO.Compression.ZipFile]::ExtractToDirectory('atu.zip', 'atu'); }"`
5. delete the `virtualenv` folder called `env` in the folder using `plink`
6. create a fresh `env` folder using using `virtualenv`: `virtualenv env`
7. using the `requirements.txt` document, I install the requirements again by direcatly using the `pip` installed in the virtualenv above using: `/path/to/env/Scripts/pip.exe install -r /path/to/requirements.txt`
8. run the file with the `python` distribution installed in the virtualenv using: `/path/to/python file_name.py`

Because I was trying to execute relatively complex commands, which in turn had multiple quotes (double and single), spaces etc, I had to spend quite a bit of time just debbuging how to run these commands over the network. I have automated most of the deployment part(*STEP 2*) using a combination of `subprocess.call`, `plink`, and `pscp`. However, I haven't yet automated setting up the VM (*STEP 1*).

I am guessing, the above set of actions are not just something I am doing, and probably others have done this as well.

Currently, I am using low level tools to acheive this. While this works for setting up Windows VM, it wont probably work for Linux based VMs or if I change python versions use a different language etc etc.

I have also searched a bit and came across this book: https://www.safaribooksonline.com/library/view/test-driven-development-with/9781449365141/ch09.html

where the author goes into how to deploy to a linux server. However, it seems that he is essentially doing what I am doing by directly running commands on the linux server, just through the `fabric` wrapper. Because `ssh` doesn't work as well on Windows, I am guessing, I will have to get back to the nitty-grittys of deployment with `fabric` as well.

### Question:

1. Is there a better way to setup the VM, and deploy code to the VMs, while abstracting away the difference between different operating systems, python versions, tools etc?
2. I have also heard of a `git` based workflow. Is that better than directly pushing the code to the server? Why so?

# Answer
Instead of using custom scripts which it seems that you are using, I would consider using open-source tools, similar to what Tensibai has suggested in his comments.

I would further split your two Tasks a bit more differently, such that Step 1 would be to just "*Set up the VM*", and Step 2 would be to "*Provision the VM with required programs and configuration*".

My 2-cents would be to use tools such as Packer and Ansible.

You would use (for example), the VirtualBox Packer builder, to stand up your VM, which would handle points 1-4 of your Step 1, and then use the Ansible-local provisioner, or something to this effect, to run a playbook, which would handle the rest of Step 1, and all of Step 2.

---

The above suggestion would make your life a lot easier, and would be able to abstract different OS's as well as you'd just write a different Packer template and Ansible task. With installing different Python versions / Tools, you'll just need to update the Ansible role / have several different roles with the specific tools.

There are of course, other tools such as Vagrant, Puppet, and Chef, which can handle what you'd like to achieve, so it's up to you find your preferred tool.

---

Mini breakdown of Ansible modules you can use:

Etc. The full list of Ansible Windows Modules can be found here.
> 2 votes

# Answer
I use The Foreman for this. It leverages the fog vsphere module to interact with your vSphere server and also works with Google, Microsoft, and Amazon's clouds and other hypervisor systems as well if needed or you need to migrate at some point in the future.

This software stack is backed by Red Hat and re-branded as Red Hat Satellite 6 if you wish to use their licensed and supported version and is specifically designed for management of the Lifecycle of machines (both physical and virtual) and it can also manage Docker containers.

Upon deployment of your VM, you have the following options for configuring your newly deployed VM:

* Kickstart script
* Puppet
* Ansible
* Chef
* SaltStack

As a python shop, you will most likely be comfortable with either SaltStack or Ansible and would want to be sure to install The Foreman with support for your chosen configuration management software.

This software comes with a REST API that can be leveraged to trigger vm deployments automatically from a build pipeline (Eg, Jenkins, Bamboo, TravisCI, CircleCI, etc) or you could have the VM trigger testing after configuration is complete (or both).
> 2 votes

---
Tags: deployment, git, python, virtualenv
---


# Question
Title: What are best practices for docker tag versioning?
I've recently hooked our CI servers up to build docker images upon git commit.

We have about 8 different containers that get built, each with their own language / frameworks. Some are node and have a package.json, others are python services that contain no semantic version information.

My question is not about how to create tags, its about creating the values for the tag.

How to ensure that each tag has a unique semantic version number for the specific images? Who should be the authority on tracking / incrementing a build version?

# Answer
> 7 votes
I would direct you to my post Coupling docker registry and source control where dmaze answered from the official forums.docker.com. Commit hash and branch name or tags suffices.

> In your Dockerfile, use a LABEL to record the source of the build. That probably includes the commit hash from distributed source control (git, Mercurial), the branch name if relevant, any release tags if present, and possibly details like the timestamp of the last commit. docker history and docker inspect should be able to show these.
>
> When you docker push your images, push them at least twice, with the commit hash and with the branch name as the “version” part (quay.io/mycorp/imagename:123abc7, quay.io/mycorp/imagename:dmaze-test). If release tags are readily available, the CI system should push images with these tags too.

We are currently using a combination of branch name/commit hash. For us that seems to be enough. timestamps while they are useful IMO just add clutter as they don't provide anything the commit hash doesn't.

I agree with **030** regarding:

> who should be the authority on tracking / incrementing a build version

100% is responsibility of the CI to maintain such things, with proper communication between other teams.

# Answer
> 2 votes
> How to ensure that each tag has a unique semantic version number for the specific images?

One could create a tag that consists of multiple elements, e.g. a combination of a timestamp, git commit hash and semantic version. The latter has to be set manually, while the first two could be automated. Such a tag could look as follows:

```
20171015141729-58617f500f7efe236c7ba6a1dfdf37a478b4c878-0.1.4

```

This tag contains the date of build, the commit and the semantic version. If a docker image runs in production and a bug is found then one knows the version of the product, the code that is inside and when the image was built and under what circumstances.

> Who should be the authority on tracking / incrementing a build version?

In my opinion this should be the responsibility of the CI as this is able to automate processes and as the creation of tags could be automated such a tool is the right tool for the job.

# Answer
> 1 votes
I suppose that you use one of the DevOps tools for CI/CD like Jenkins, I suggest the following approach,

If you use something like Jenkins-

* You can configure your job such a way that you can use Jenkins environment variable "BUILD\_ID", which retrieves the build id of the job when triggered to tag it to your image. This way you can version control your docker images. Please check the below example.

ex:- `sudo docker build -t <image_name>:<BUILD_ID>`

So, if you have got tag like mechanism for your SCM, you can check the tag in the respective build ID either in job based builds or in config.xml of the build ID in JENKINS HOME\_FOLDER.

---
Tags: docker, continuous-integration, versioning
---


# Question
Title: Ansible Expect module times out; unable to end play or move to another module
I've noticed that the expect module will always time out and cause an `rc=1` code. It's doing everything that I want it to, I just want to move to another module.

How can I end the expect module and move to the next one?

```
  tasks:
    - name: Disk Unassignment
      expect:
        echo: yes
        timeout: 180
        command: telnet '{{ terms }}' '{{ termp }}'
        responses:
          Escape character is:  "echo"
          Error: "boot_diags"
          Continue with boot?: "yes"
          root logged: "disk remove_ownership all"
          Do you: "yes"
          Volumes: "yes"
    - meta: end_play

```

> TASK \[Disk Unassignment\] \*************************************************************************************************************************************************************************** task path: /home/sansible/ansible/disk-unassign.yaml:13 fatal: \[localhost\]: FAILED! =\> {"changed": true, "cmd": "telnet '10.233.82.7' '2045'", "delta": "0:05:07.775726", "end": "2018-03-22 12:30:52.315879", "msg": "non-zero return code", "rc": 1, "start": "2018-03-22 12:25:44.540153", "stdout": "Trying 10.233.82.7...\r\r\nConnected to 10.233.82.7.\r\r\nEscape character is '^\]'.\r\r\necho\r\nConnected to port 45. \r\necho\r\nInvalid command: \\"echo\\"\r\nAvailable commands: version, update\_flash, netboot, boot\_diags, boot\_backup, boot\_primary, boot\_ontap, sp, flash, bye, set, lsmod, autoboot, go, boot, load, ndp, ping, arp, ifconfig, show, savenv, saveenv, unsetenv, set-defaults, setenv, printenv, help\r\n\r\nError (errcode=-1)\r\nLOADER-A\> boot\_diags\r\nLoading X86\_64/freebsd/image1/kernel:0x200000/10377184 0xbe57e0/6364864 Entry at 0x80294c20\r\nLoading X86\_64/freebsd/image1/platform.ko:0x11f8000/2512792 0x145e798/393448 0x14be880/541680 \r\nStarting program at 0x80294c20\r\nNetApp Data ONTAP 9.1RC2X9\r\n**Invalid config, overriding as ignore\_chassis\_config is set** \r\n\r\nTrying to mount root from msdosfs:/dev/da0s1 \[ro\]...\r\nmd0 attached to /X86\_64/freebsd/image1/rootfs.img\r\nTrying to mount root from ufs:/env/md0.uzip \[\]...\r\nmountroot: waiting for device /env/md0.uzip ...\r\nCopyright (C) 1992-2016 NetApp.\r\nAll rights reserved.\r\nWriting loader environment to the boot device.\r\nLoader environment has been saved to the boot device.\r\n\*******************************\r\n\* *\r\n* Press Ctrl-C for Boot Menu. *\r\n* \*\r\n\******************************* \t\r\nNo SVM keys found.\r\nFirewall rules loaded.\r\nIpspace \\"ACP\\" created\r\nWAFL CPLEDGER is enabled. Checklist = 0x7ff841ff\r\nadd host 127.0.10.1: gateway 127.0.20.1\r\n5\r\n\r\n\tNOTE: You have chosen to boot the diagnostics kernel.\r\n\tUse the 'sldiag' command in order to run diagnostics on the system.\r\n\r\n\r\n\tYou have selected the maintenance boot option:\r\n\tthe system has booted in maintenance mode allowing the\r\n\tfollowing operations to be performed:\r\n\r\n\t? \tacorn \r\n\tacpadmin \taggr \r\n\tcna\_flash \tdisk \r\n\tdisk\_latency \tdisk\_list \r\n\tdisk\_mung \tdisk\_shelf \r\n\tdiskcopy \tdisktest \r\n\tdumpblock \tenvironment \r\n\tfcadmin \tfcstat \r\n\tfctest \tfru\_led \r\n\tha-config \thalt \r\n\thelp \tifconfig \r\n\tkey\_manager \tled\_off \r\n\tled\_on \tnicadmin \r\n\tnv8 \traid\_config \r\n\tsasadmin \tsasstat \r\n\tscsi \tsesdiag \r\n\tsldiag \tstorage \r\n\tstsb \tsysconfig \r\n\tsystemshell \tucadmin \r\n\tversion \tvmservices \r\n\tvol \tvol\_db \r\n\tvsa \txortest \r\n\r\n\tType \\"help \\" for more details.\r\n\r\n\r\n\tIn a High Availablity configuration, you MUST ensure that the \r\n\tpartner node is (and remains) down, or that takeover is manually \r\n\tdisabled on the partner node, because High Availability \r\n\tsoftware is not started or fully enabled in Maintenance mode.\r\n\r\n\tFAILURE TO DO SO CAN RESULT IN YOUR FILESYSTEMS BEING DESTROYED\r\n\r\n\tNOTE: It is okay to use 'show/status' sub-commands such as\r\n\t'disk show or aggr status' in Maintenance mode while the partner is up\r\nContinue with boot? yes\r\nyes\r\n\r\n\r\nMar 22 12:24:22 \[rtpsb-f6280-01-01:acp.need.firmware.update:info\]: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHJ0000000152A6). \r\n\rMar 22 12:24:22 \[rtpsb-f6280-01-01:acp.need.firmware.update:info\]: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHJ0000000152A6). \r\n\r\r\nThu Mar 22 12:24:49 EDT 2018\r\n\*\> Mar 22 12:24:49 \[rtpsb-f6280-01-01:kern.syslog.msg:notice\]: root logged in from SP\r\n \r\n\rdisk remove\_ownership all\r\nAll disks will have their ownership information removed.\r\nFor high availability (HA) configurations, the partner node must be down.\r\nDo you wish to continue? yes\r\nVolumes must be taken offline. Are all impacted volumes offline(y/n)?? yes\r\n\*\> ", "stdout\_lines": \["Trying 10.233.82.7...", "", "Connected to 10.233.82.7.", "", "Escape character is '^\]'.", "", "echo", "Connected to port 45. ", "echo", "Invalid command: \\"echo\\"", "Available commands: version, update\_flash, netboot, boot\_diags, boot\_backup, boot\_primary, boot\_ontap, sp, flash, bye, set, lsmod, autoboot, go, boot, load, ndp, ping, arp, ifconfig, show, savenv, saveenv, unsetenv, set-defaults, setenv, printenv, help", "", "Error (errcode=-1)", "LOADER-A\> boot\_diags", "Loading X86\_64/freebsd/image1/kernel:0x200000/10377184 0xbe57e0/6364864 Entry at 0x80294c20", "Loading X86\_64/freebsd/image1/platform.ko:0x11f8000/2512792 0x145e798/393448 0x14be880/541680 ", "Starting program at 0x80294c20", "NetApp Data ONTAP 9.1RC2X9", "**Invalid config, overriding as ignore\_chassis\_config is set** ", "", "Trying to mount root from msdosfs:/dev/da0s1 \[ro\]...", "md0 attached to /X86\_64/freebsd/image1/rootfs.img", "Trying to mount root from ufs:/env/md0.uzip \[\]...", "mountroot: waiting for device /env/md0.uzip ...", "Copyright (C) 1992-2016 NetApp.", "All rights reserved.", "Writing loader environment to the boot device.", "Loader environment has been saved to the boot device.", "\*******************************", "\* *", "* Press Ctrl-C for Boot Menu. *", "* \*", "\******************************* \t", "No SVM keys found.", "Firewall rules loaded.", "Ipspace \\"ACP\\" created", "WAFL CPLEDGER is enabled. Checklist = 0x7ff841ff", "add host 127.0.10.1: gateway 127.0.20.1", "5", "", "\tNOTE: You have chosen to boot the diagnostics kernel.", "\tUse the 'sldiag' command in order to run diagnostics on the system.", "", "", "\tYou have selected the maintenance boot option:", "\tthe system has booted in maintenance mode allowing the", "\tfollowing operations to be performed:", "", "\t? \tacorn ", "\tacpadmin \taggr ", "\tcna\_flash \tdisk ", "\tdisk\_latency \tdisk\_list ", "\tdisk\_mung \tdisk\_shelf ", "\tdiskcopy \tdisktest ", "\tdumpblock \tenvironment ", "\tfcadmin \tfcstat ", "\tfctest \tfru\_led ", "\tha-config \thalt ", "\thelp \tifconfig ", "\tkey\_manager \tled\_off ", "\tled\_on \tnicadmin ", "\tnv8 \traid\_config ", "\tsasadmin \tsasstat ", "\tscsi \tsesdiag ", "\tsldiag \tstorage ", "\tstsb \tsysconfig ", "\tsystemshell \tucadmin ", "\tversion \tvmservices ", "\tvol \tvol\_db ", "\tvsa \txortest ", "", "\tType \\"help \\" for more details.", "", "", "\tIn a High Availablity configuration, you MUST ensure that the ", "\tpartner node is (and remains) down, or that takeover is manually ", "\tdisabled on the partner node, because High Availability ", "\tsoftware is not started or fully enabled in Maintenance mode.", "", "\tFAILURE TO DO SO CAN RESULT IN YOUR FILESYSTEMS BEING DESTROYED", "", "\tNOTE: It is okay to use 'show/status' sub-commands such as", "\t'disk show or aggr status' in Maintenance mode while the partner is up", "Continue with boot? yes", "yes", "", "", "Mar 22 12:24:22 \[rtpsb-f6280-01-01:acp.need.firmware.update:info\]: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHJ0000000152A6). ", "", "Mar 22 12:24:22 \[rtpsb-f6280-01-01:acp.need.firmware.update:info\]: ACP module (NA) needs a firmware upgrade, (disk shelf serial number: SHJ0000000152A6). ", "", "", "Thu Mar 22 12:24:49 EDT 2018", "*\> Mar 22 12:24:49 \[rtpsb-f6280-01-01:kern.syslog.msg:notice\]: root logged in from SP", " ", "", "disk remove\_ownership all", "All disks will have their ownership information removed.", "For high availability (HA) configurations, the partner node must be down.", "Do you wish to continue? yes", "Volumes must be taken offline. Are all impacted volumes offline(y/n)?? yes", "*\> "\]} to retry, use: --limit @/home/sansible/ansible/disk-unassign.retry
>
> PLAY RECAP \***************************************************************************************************************************************************************************************** localhost : ok=0 changed=0 unreachable=0 failed=1

# Answer
> 4 votes
Tensibai was correct, the telnet session in the `expect` module had to end.

Holding down the key combination `Ctrl`+`]` to exit wasn't going to work, so an optional escape character was chosen in the telnet command like so:

```
telnet -e ! 10.224.223.10

```

---
Tags: ansible
---


# Question
Title: Deploy to server after merge to master branch
I currently have continuous deployment set up with a Travis CI job. All pushes are sent to the staging server and merges to the master to the production server. The problem is that the deployment to the production server is done before the merge is accepted. The continuous integration tests have to all pass, but the deployment occurs before manual code review is done.

The question is, what is a lightweight method to ensure that a deployment to the production server only takes place after the merge request has been accepted?

As a reference, the following code is used for deployment:

```
#!/bin/bash

# exit with nonzero exit code if anything fails
set -e

# Add the SSH login key
chmod 600 veleda-deploy-key
mv veleda-deploy-key ~/.ssh/id_rsa

# Register the Veleda staging and production server SSH keys
echo '|1|BqdQKtUnA/AtCT/p2M7wgMq3wlY=|lH39cRtAE64wd6EG3ry2J9ewXic= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBH3antqwy3D4NVVfHQX3SQc/g4wl/SAVC9w9QEry7hhQmB0SJIprwNAq8Hy2DzVCS7kTj/q7fCiiL7oAznrax+0=' >> $HOME/.ssh/known_hosts
echo '|1|+Z7oOsZ+zdL6u8o8VSWp+bRzd2g=|XMw2HyJIHoekOYlJYw1n75plL2E= ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCJ/fa/mr577/qCuRXqUNccfmhpUtmi46LSyE7nDbOgxv8kZFs7yQ/sh6TM5npR+ZIbe9I0qmdvA+cE1QfvN21E=' >> $HOME/.ssh/known_hosts

# Select the remote to push to depending on the branch
if [ $TRAVIS_BRANCH != 'production' ] ; then
    echo "Pushing to staging"
    export REMOTE=staging.veleda.io
else
    echo "Pushing to production"
    export REMOTE=veleda.io
fi

# Push to the remote server
git remote add deploy "deploy@$REMOTE:/home/deploy/repo/"
git push -f deploy $TRAVIS_BRANCH

# Unpack and update the Docker services
ssh -t deploy@"$REMOTE" "\
    mkdir -p veleda &&
    git --work-tree=./veleda --git-dir=./repo checkout -f $TRAVIS_BRANCH &&
    cd veleda &&
    ./start.sh"

```

# Answer
One could use conditional builds https://docs.travis-ci.com/user/conditional-builds-stages-jobs/

If code is merged into master one could decide to deploy code to production, but I personally prefer a human intervention by a Product Owner.

> ```
> jobs:
>   include:
>       if: branch = master
>
> ```

or

> ```
> stages:
>   - name: deploy
>     # require the branch name to be master
>     if: branch = master
>
> ```

At other projects I preferred to deploy only tags to enforce the git flow (tags should be created otherwise no deployment to production).

> ```
> stages:
>   - name: deploy
>     # require the tag name to match a regular expression
>     if: tag =~ ^v1
>
> ```

One could also try (I did not try it yet) whether the `if` works in combination with script sections:

https://github.com/030/ansible-firefox/blob/master/.travis.yml

Last remark: I would avoid creating such scripts (the ones you included in the question) as most CI tools contain condition checks.
> 4 votes

# Answer
The updated deploy script in the question supports the following workflow (and requires this server setup):

1. Create a feature branch and push commits to it
2. Each push deploys the code to the staging server
3. Review the code with merge request (feature branch --\> master)
4. After merging to master, create a new merge request (master --\> production)

Due to Github merge request errata, I rebase when merging to master but normal merges when merging to production. The reason behind this is that Github creates new commits (different SHA1) when rebasing.

> The rebase and merge behavior on GitHub deviates slightly from git rebase. Rebase and merge on GitHub will always update the committer information and create new commit SHAs, whereas git rebase outside of GitHub does not change the committer information when the rebase happens on top of an ancestor commit.

The main difference with this workflow to 030's answer is that a *production* branch is used instead of tags.

For bonus points use *staging* SSL certificates from Let's Encrypt on your staging server. To enable it while using a reverse proxy in Docker, set `LETSENCRYPT_TEST=true`
> 2 votes

---
Tags: continuous-deployment, github, travis-ci
---


# Question
Title: Ansible translation: Installing Apache + Certbot in Ubuntu
I currently have a doubt for my last line of code to make it work in Ansible, the code is the following:

```
sudo apt-get update

sudo apt-get upgrade

sudo apt-get install apache2 python-letsencrypt-apache

sudo letsencrypt --apache

sudo apt-get install python-software-properties

sudo apt-get install software-properties-common

sudo add-apt-repository ppa:certbot/certbot

sudo apt-get install python-certbot-apache

sudo certbot --webroot-path /var/www/html -d www.web.tk

```

Now I would like to convert it to Ansible:

```
- apt_repository:
    repo: 'ppa:certbot/certbot'

- apt:
    name: "{{ item }}"
    update_cache: yes
  with_items:
    - all items above

```

But I have a problem as the line `sudo letsencrypt --apache` requires user interaction and I would like to give it the website and email from Ansible inventory file

# Answer
If you really want to create the role yourself, you have to use `--agree-tos --noninteractive` to disable user interaction.

See https://certbot.eff.org/docs/using.html for more details
> 1 votes

---
Tags: ansible, certbot, apache-httpd
---


# Question
Title: Why flask is returning a default login page instead mine, custom one
I have an issue with my app, since I was adding the feature "users roles" I am getting a strange page content on login page looks like default login page instead of my custom login page and I cannot explain why is happening because I did not change the urls and when I am removing the "users roles code" is working perfect. Could you explain how is possible?

This is the code what is creating the issue:

```
import os
import base64
from io import BytesIO
from flask import Flask, render_template, request, redirect, url_for, flash, session, \
    abort
from werkzeug.security import generate_password_hash, check_password_hash
from flask_sqlalchemy import SQLAlchemy
from flask_login import LoginManager, UserMixin, login_user, logout_user, \
    current_user
from flask_bootstrap import Bootstrap
from flask_wtf import FlaskForm
from wtforms import StringField, PasswordField, SubmitField, SelectField
from wtforms.validators import Required, Length, EqualTo
import onetimepass
import pyqrcode
from datetime import datetime, date, time, timedelta
from sqlalchemy import Table, Column, Integer, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.ext.declarative import declarative_base
from flask_security import Security, SQLAlchemyUserDatastore, \
    UserMixin, RoleMixin, login_required

# create application instance
app = Flask(__name__)
app.config.from_object('config')

# initialize extensions
bootstrap = Bootstrap(app)
db = SQLAlchemy(app)
lm = LoginManager(app)
Base = declarative_base()

# Define models
roles_users = db.Table('roles_users',
        db.Column('users_id', db.Integer(), db.ForeignKey('users.id')),
        db.Column('role_id', db.Integer(), db.ForeignKey('role.id')))

class Role(db.Model, RoleMixin):
    id = db.Column(db.Integer(), primary_key=True)
    name = db.Column(db.String(80), unique=True)
    description = db.Column(db.String(255))

class User(UserMixin, db.Model):
    """User model."""
    __tablename__ = 'users'
    id = db.Column(db.Integer, unique=True, nullable=False, primary_key=True)
    username = db.Column(db.String(64), unique=True, index=True)
    password_hash = db.Column(db.String(128))
    otp_secret = db.Column(db.String(16))
    access = db.Column(db.String(16))
    roles = db.relationship('Role', secondary=roles_users,
                            backref=db.backref('users', lazy='dynamic'))

    def __init__(self, **kwargs):
        super(User, self).__init__(**kwargs)
        if self.otp_secret is None:
            # generate a random secret
            self.otp_secret = base64.b32encode(os.urandom(10)).decode('utf-8')

    @property
    def password(self):
        raise AttributeError('password is not a readable attribute')

    @password.setter
    def password(self, password):
        self.password_hash = generate_password_hash(password)

    def verify_password(self, password):
        return check_password_hash(self.password_hash, password)

    def get_totp_uri(self):
        return 'otpauth://totp/2FA-Demo:{0}?secret={1}&issuer=2FA-Demo' \
            .format(self.username, self.otp_secret)

    def verify_totp(self, token):
        return onetimepass.valid_totp(token, self.otp_secret)

################################################################################
user_datastore = SQLAlchemyUserDatastore(db, User, Role)
security = Security(app, user_datastore)
###############################################################################

@lm.user_loader
def load_user(user_id):
    """User loader callback for Flask-Login."""
    return User.query.get(int(user_id))

class RegisterForm(FlaskForm):
    """Registration form."""
    access = SelectField(validators=[Required()],
            choices=[("dm", "demo"),
            ("dm2", "demo2")])
    username = StringField('Username', validators=[Required(), Length(1, 9)])
    password = PasswordField('Password', validators=[Required(), Length(4, 4)])
    password_again = PasswordField('Password again',
                                   validators=[Required(), EqualTo('password')])
    submit = SubmitField('Register')

class LoginForm(FlaskForm):
    """Login form."""
    username = StringField('Username', validators=[Required(), Length(1, 9)])
    password = PasswordField('Password', validators=[Required(), Length(4, 4)])
    token = PasswordField('Token', validators=[Required(), Length(6, 6)])
    submit = SubmitField('Login')

@app.route('/')
def index():
    return render_template('index.html')

@app.route('/register', methods=['GET', 'POST'])
def register():
    """User registration route."""
    if current_user.is_authenticated:
        # if user is logged in we get out of here
        return redirect(url_for('index'))
    form = RegisterForm()
    if form.validate_on_submit():
        user = User.query.filter_by(username=form.username.data).first()
        if user is not None:
            flash('Username already exists.')
            return redirect(url_for('register'))
        # add new user to the database
        user = User(username=form.username.data, password=form.password.data, access=form.access.data)
        db.session.add(user)
        db.session.commit()

        # redirect to the two-factor auth page, passing username in session
        session['username'] = user.username
        return redirect(url_for('two_factor_setup'))
    return render_template('auth/register.html', form=form)

@app.route('/twofactor')
def two_factor_setup():
    if 'username' not in session:
        return redirect(url_for('index'))
    user = User.query.filter_by(username=session['username']).first()
    if user is None:
        return redirect(url_for('index'))
    # since this page contains the sensitive qrcode, make sure the browser
    # does not cache it
    return render_template('auth/2fa.html'), 200, {
        'Cache-Control': 'no-cache, no-store, must-revalidate',
        'Pragma': 'no-cache',
        'Expires': '0'}

@app.route('/qrcode')
def qrcode():
    if 'username' not in session:
        abort(404)
    user = User.query.filter_by(username=session['username']).first()
    if user is None:
        abort(404)

    # for added security, remove username from session
    del session['username']

    # render qrcode for FreeTOTP
    url = pyqrcode.create(user.get_totp_uri())
    stream = BytesIO()
    url.svg(stream, scale=3)
    return stream.getvalue(), 200, {
        'Content-Type': 'image/svg+xml',
        'Cache-Control': 'no-cache, no-store, must-revalidate',
        'Pragma': 'no-cache',
        'Expires': '0'}

@app.route('/login', methods=['GET', 'POST'])
def login():
    """User login route."""
    if current_user.is_authenticated:
        # if user is logged in we get out of here
        return redirect(url_for('index'))
    form = LoginForm()
    if form.validate_on_submit():
        user = User.query.filter_by(username=form.username.data).first()
        if user is None or not user.verify_password(form.password.data) or \
                not user.verify_totp(form.token.data):
            flash('Invalid username, password or token.')
            return redirect(url_for('login'))

        # log user in
        login_user(user)
        flash('You are now logged in!')
        return redirect(url_for('cm_e_11a'))
    return render_template('auth/login.html', form=form)

@app.route('/logout')
def logout():
    """User logout route."""
    logout_user()
    return redirect(url_for('index'))

```

And this is the Login page what is returning

instead of mine what should contain username, password & pin

And the login page has this source code:

```
<h1>Login</h1>
<form action="/login" method="POST" name="login_user_form">
  <input id="next" name="next" type="hidden" value="">
<input id="csrf_token" name="csrf_token" type="hidden" value="IjEyNWY4ZjliNmIxZTJjNmM3YmM4YzFlM2U0NWZlYjJlMDEzYThiNWEi.DZnvqw.key_C7NmtQogY7CSpb14sfVDUOQ">

  <p>
    <label for="email">Email Address</label> <input id="email" name="email" type="text" value="">

  </p>

  <p>
    <label for="password">Password</label> <input id="password" name="password" type="password" value="">

  </p>

  <p>
    <label for="remember">Remember Me</label> <input id="remember" name="remember" type="checkbox" value="y">

  </p>

  <p><input id="next" name="next" type="hidden" value=""></p>

  <p><input id="submit" name="submit" type="submit" value="Login"></p>

</form>

```

# Answer
> 1 votes
I found the solution!!!

Views Flask-Security is packaged with a default template for each view it presents to a user. Templates are located within a sub-folder named security. The following is a list of view templates:

```
security/forgot_password.html
security/login_user.html
security/register_user.html
security/reset_password.html
security/change_password.html
security/send_confirmation.html
security/send_login.html

```

Overriding these templates is simple:

* Create a folder named security within your application’s templates folder
* Create a template with the same name for the template you wish to override

---
Tags: python
---


title: Jenkins pipeline + Docker integration

text:
# Question
Title: Jenkins pipeline + Docker integration
I have a CI server which has Jenkins installed in the host it self, and I want to run builds and tests inside Docker containers. Until here it works like it should, but I also want to deploy containers using docker-compose and this should be done in the host, but when I run commands inside the Jenkinsfile, they are ran inside the Docker container used for builds and tests.

```
pipeline {
agent {
    dockerfile {
        filename "Dockerfile"
    }
}

stages {
    stage("Run unit tests") {
        steps {
            # This is ran inside the Docker container, as expected
            sh "cp environments/.env.build .env"
            sh "vendor/bin/phpunit"
        }
    }

    stage('Deploy to test') {
        when {
            branch "develop"
        }

        steps {
            # This is still ran inside the Docker container, not the host
            sh 'docker-compose up -d'
        }
    }

    stage("Archive") {
        steps {
            # This is ran inside the Docker container, as expected
            sh "zip -r dist.zip ."
        }
    }
}

post {
    always {
        archiveArtifacts artifacts: "dist.zip"
        junit "reports/phpunit.report.xml"
    }
}
}

```

# Answer
> 0 votes
One way to solve this it to use multiple pipelines. The first pipeline runs the test in docker. When that has been completed, one could call another job

https://jenkins.io/doc/pipeline/steps/pipeline-build-step/

https://support.cloudbees.com/hc/en-us/articles/221400287-How-to-pass-parameter-to-downstream-job-in-Pipeline-job-

> ```
> node {
>     paramAValue = "paramAValue"
>     paramBValue = "paramBValue"
>     build job: 'downstream-freestyle', parameters: [[$class:  'StringParameterValue', name: 'ParamA', value: paramAValue], [$class:
>        'StringParameterValue', name: 'ParamB', value: paramBValue]]
> }
>
> ```

that runs docker-compose on Jenkins itself.

---
Tags: docker, jenkins
---


title: Combine Docker Swarm and Kubernetes

text:
# Question
Title: Combine Docker Swarm and Kubernetes
My company is trying to play a little catch-up in the DevOps space. I have been doing a lot of research on containerization of applications and the orchestration systems that go with it. I came across an article (one that I wish I had saved) where they were talking about combining Swarm with Kubernetes to get better functionality. In this article they did not define what they gained by doing that.

I wondered what benefits would this provide? Is adding the extra layer of complexity really going to give you much return?

**EDIT:** I am looking for technical pro's/con's. KISS is a good motto but doesn't hold up in a debate with your CEO or Board.

I am almost positive that we will be selecting Docker for our containers and Swarm for an orchestration. I, however, would like to see Kubernetes in our space so the proposition that you could merge the technologies together for a more robust solution intrigues me.Thanks for any insight.

# Answer
**Update:** Docker just released support for Kubernetes as scheduler, which changes the situation and makes Kubernetes just an alternative scheduler to Docker Swarm.

**TL;DR: DON'T DO IT.** Engineers always try to create these dog-pigs. Every unnecessary technology you bring will bring another whole set of faults. If you can pick one, then pick one and be happy you don't have to do both. If you like to play with Kubernetes, just get a private account on Google Cloud and play with it as much as you like. But don't make everyone at your company suffer through unnecessary complications.

Those are two parallel and mostly equivalent technologies. If your business had legitimate business reason to deploy in multiple Cloud providers for reliability for example and wanted to deploy into both AWS ECS (Elastic Container Service - based on Docker) and Google GKE (Container Engine - based on Kubernetes) and you were asking how do you build a pipeline, which would build your software and package in containers for deployment in both, that would be something else, but doing it just because you want to play with a new technology is very irresponsible.
> 11 votes

# Answer
One reason to with Kubernetes as a scheduler if you use or would consider Azure as a cloud provider is their relatively new AKS service (managed kubernetes). In this case you would not combine kubernetes with docker swarm though.

This to me is clear indication as to where the community is going. I would not want to learn something that I would later need to throw in the trash can.
> 0 votes

---
Tags: docker, containers, orchestration, kubernetes, docker-swarm
---


title: sshagent and jenkinsfile

text:
# Question
Title: sshagent and jenkinsfile
I am new to Jenkinsfile. I have previously configured my Jenkins job from GUI, and we are now upgrading our server to use Jenkinsfile. My job contains a section which needs to SSH to another server. To do this, I previously used the sshagent plugin on the GUI to specify my keys. However, on the jenkinsfile mechanism this does not work.

First things first: Aside from using jenkinsfile, we have also trasitioned to building in **docker**. I read in some places that docker might conflict with sshagent plugin, so I am trying to remove docker by starting a clean job with "agent any" and trying to test.

So my new job looks like this:

```
    @NonCPS  // Necessary to allow .each to work.
def changelist() {
    def changes = ""
    currentBuild.changeSets.each { set ->
        set.each { entry ->
            changes += "${entry.author.id} "
        }
    }
    changes
}

pipeline {
    agent none
    stages {
            stage('test') {
            agent any
            steps {
                sshagent ( ['a-jenkins-credential']) {
    sh '''
ssh -vv myuser@myserver echo testing connection || true
ssh-add -L
echo done running remote windows test
'''
  }
            }
        }

}
}

```

On the same server, but **using the jenkins GUI, this works flawlessly**, but on the jenkinsfile pipeline this produces:

```
FATAL: [ssh-agent] Could not find specified credentials

```

and proceeds run and fail on the ssh command. The funny thing is, that this job uses the exact same key to succesfully pull from GIT, so it's not like it's not installed on the server.

# Answer
Found the problem:

I was using the human readable key name from the GUI. Need to use the key's UUId ID instead (this is specified next to the name in credentials screen)

```
steps {
            sshagent ( ['THIS-SHOULD-HAVE-BEEN-A-UNIQUE-ID-INSTEAD-OF-A-NAME']) {

```
> 10 votes

---
Tags: jenkins, jenkins-pipeline, jenkins-plugins
---


# Question
Title: Jenkins returns 502 gateway error and (104: Connection reset by peer)
So my server got maintenance schedule from cloud provider, after maintenance process, the server rebooted, then my jenkins cannot be accessed from the web anymore, I am using nginx and checking the logs I found this.

> /var/log/nginx/error.log

```
2018/03/26 16:11:00 [error] 21689#21689: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 139.139.139.139, server: jenkins.domain.com, request: "GET / HTTP/1.1", upstream: "http://127.0.0.1:8080/", host: "jenkins.domain.com"

2018/03/26 16:11:00 [error] 21689#21689: *1 recv() failed (104: Connection reset by peer) while reading response header from upstream, client: 139.139.139.139, server: jenkins.domain.com, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:8080/favicon.ico", host: "jenkins.domain.com", referrer: "http://jenkins.domain.com/"

```

here is my nginx conf file:

```
upstream app_server {
server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  root /var/www/jenkins.domain.com;

  location ~ /\.well-known/acme-challenge {
        default_type "text/plain";
  }

  location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;

        if (!-f $request_filename) {
            proxy_pass http://app_server;
            break;
        }
    }
}

```

What is the cause of this and maybe what can I do to fix this? Really need your help guys.

Thanks

# Answer
> 2 votes
If the system recently rebooted and your proxy server is giving 502/503s, it's most likely that the backend service failed to start.

Using whatever tools are most appropriate for your OS/distro (e.g. `systemctl`, `service`/`chkconfig`, `stop`/`start`/`restart`/etc.), do the following things:

* Verify that the backend (in this case Jenkins) is configured to start on boot. Enable it if not.
* Start the backend.
* If the backend fails to start, check the application logs to find the source of the error. For Jenkins on Linux, this would be `/var/log/jenkins/jenkins.log`.

---
Tags: jenkins, nginx
---


title: How to deploy Envoy EDS/SDS

text:
# Question
Title: How to deploy Envoy EDS/SDS
This is a micro services deployment question. How would you deploy Envoy SDS(service discovery service) so other envoy proxies can find the SDS server hosts, in order to discover other services to build the service mesh. Should I put it behind a load balancer with a DNS name( single point of failure) or just run the SDS locally in every machine so other micro services can access it? Or is there a better way of deployment that SDS cluster can be dynamically added into envoy config without a single point of failure?

# Answer
> 2 votes
Putting it behind a DNS name with a load balancer across multiple SDS servers is a good setup for reasonable availability. If SDS is down, Envoy will simple not get updated, so it's generally not the most critical failure -- new hosts and services simply won't get added to the cluster/endpoint model in Envoy.

If you want higher availability, you set up multiple clusters. If you add multiple entries to your bootstrap config, Envoy will fail over between them. You can either specify multiple DNS names or multiple IPs.

# Answer
> 1 votes
I ended up listing the SDS/EDS directly under the private DNS zone. However this means it is required to have a static port that the SDS listens to on any box its placed, and every other service will have to have it passed as a known constant. So backend micro services will get a set of ips for available SDS servers, in order to retrieve app service endpoint information. This does not include a load balancer. It can use DNS supported load balancing features. SDS uses a script to heartbeat itself to the DNS. It was implemented in AWS so SDS is running in ECS containers, automatically placed by ECS cluster management service, and that heartbeat script in the SDS server uses AWS metadata service to obtain and report its ip address to DNS. This method means there is very less downtime, since DNS servers hardly ever fail. But we dont get advanced load balancing. Also better than having an SDS in each ec2 host, since it will cost less resources, and will not run into the possibility of all micro services in a particular box failing since its local sds server failed.

---
Tags: microservices
---


# Question
Title: Why should you do port mapping for docker containers?
I have been using docker for a few months and I am just a developer not a DevOps or a networking person. However, I came across a `docker-compose` which maps an external port to an internal port. Something like in this compose file where it says:

```
ports:
  - 8080:80

```

My question is WHY? Why do we need to map an external port to internal port. Some of the explanation that I came across was that if you want to not expose a particular port to the user but use it internally. That also begs the same question: what would be a real life example of someone wanting to do this and why?

# Answer
## TL;DR

There are a number of use cases for port mapping, but for DevOps at scale the primary reason is generally to enable mapping well-known service ports to available ports on the host. This matters when you're running large numbers of containers that use the same port by default, and you don't want to manually assign or track alternative port numbers.

## A *Very* Short Port Primer

As a general rule, a port can only map to a single service or process on each host (multiplexing ports and multi-port services are something of an exception). Only 65,536 ports are available for services to bind, with the lowest 1,024 generally reserved for binding by the root user. Services also typically bind to well-known ports such as 22, 53, or 5432 to make the service easy to find. All of these issues matter, but it's often the last issue that typically concerns Docker hosts most.

## Mapping Container Ports

Imagine that you have multiple PostgreSQL containers on a single host. By default, each wants to bind to port 5432 as its default. While you could certainly modify each container or run-command to bind the container's service to a unique host port, this quickly becomes a hassle at scale.

Instead, Docker and other container managers make it easy to *map* ports between the host OS and the container. For example:

```
# launch three PostgreSQL instances
for i in {1..3}; do
    docker run --rm -d -P postgres:alpine
done

# show port mappings for each container
docker container ls -q --filter="ancestor=postgres:alpine" |
    xargs -n1 docker port
5432/tcp -> 0.0.0.0:32773
5432/tcp -> 0.0.0.0:32772
5432/tcp -> 0.0.0.0:32771

```

This shows that you have three instances of PostgreSQL, all happily listening on the default port of 5432 *inside* their containers. However, each instance is listening on a different port (32771, 32772, and 32773) on the Docker host!

At scale, you would typically use DNS, autodiscovery, linking, or container networking to help clients and applications find the right PostgreSQL instance to connect with. With just a few instances running, parsing `docker ps` may be sufficient for your needs. Your specific use case may vary.
> 9 votes

# Answer
When you instantiate a new container it has its own networking, isolated from the host. So you can NOT access the container directly from its port.

> ports: \- 8080:80

this command will specify that you are fowarding your host port to the container port. In your local host in the port 8080 will call the port 80 of your container.

There are some ways to do it, for example using traefik to execute the reverse proxy inside the containers, you can address all containers to a network, exposing only the port of traefik, so you can access all containers executing the router rule that u have specified.

The simple answer is:

'Cause it's the easiest way to access a service from a container.
> 3 votes

---
Tags: docker, docker-compose, networking
---


# Question
Title: Jenkins, Jenkins SSH Plug-In, EC2, "ubuntu" user and "www-data"
I am trying to install Jenkins to achieve a Github -\> EC2 instance folder pipeline. So far everything works well when I target an empty folder... however, when I target my production folder owned by `www-data` the SSH copy of the files fail for permission, as Jenkins uses the `ubuntu` user, the only one I have an SSH login for.

How do I solve this issue?

# Answer
> 0 votes
The ssh plugin gives you both options, Either send build artifacts to remote server or run a command on the remote server. The name of the plugin is "send build artifacts over ssh".. use that

---
Tags: jenkins, jenkins-pipeline, jenkins-plugins
---


# Question
Title: Why won't InSpec admit that port 5432 is listening inside a PostgreSQL container?
# Background

I'm trying to test a PostgreSQL container with InSpec 2.1.10, but I cannot get a simple port check to return true, even though Docker, netstat, and psql all report that the port is properly mapped. While it's possible this is a bug in InSpec, it seems more likely that the problem exists between the keyboard and the chair. What have I missed?

# Repeatable Tests

## My Spec

```
# port_test.rb
describe port(5432) do
  it { should be_listening }
end

```

## My Testing Commands

```
container=$(docker run -d -P --rm postgres:alpine)
inspec exec port_test.rb -t docker://$container

```

## My Output

> ```
> Profile: tests from test.rb (tests from test.rb)
> Version: (not specified)
> Target:  docker://8caceaa7e2eefa4a2bf50a58ea1ddf9c927c08d3397e3d9ad5b60d3519971380
>
>   Port 5432
>      ×  should be listening
>      expected `Port 5432.listening?` to return true, got false
>
> Test Summary: 0 successful, 1 failure, 0 skipped
>
> ```

## But It Should Work!

Everything but InSpec reports that things are working as expected.

```
$ docker port $container
5432/tcp -> 0.0.0.0:32770

# from inside container using netstat
$ docker exec $container netstat -lntp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -
tcp        0      0 :::5432                 :::*                    LISTEN      -

# on macOS Docker host; flags are the macOS equivalent to `-lntp`
$ netstat -anv | fgrep $(docker port $container | cut -d: -f2)
tcp6       0      0  ::1.32770              *.*                    LISTEN      131072 131072    709      0
tcp4       0      0  *.32770                *.*                    LISTEN      131072 131072    709      0

# connecting to database in container from host
$ psql -V \
       -h localhost \
       -p $(docker port $container | cut -d: -f2)
psql (PostgreSQL) 10.3

```

# Answer
You did everything right, it is the correct InSpec test and use with Docker!

You encountered a bug in InSpec that is related to InSpec's netstat output parsing combination with Alpine's netstat output.

See InSpec's output for `postgres` container instead of `postgres:alpine`:

```
$ inspec exec docker_test.rb -t docker://1c8162517c22                                                               1d15h master[a5ff79d3]

Profile: tests from docker_test.rb (tests from docker_test.rb)
Version: (not specified)
Target:  docker://1c8162517c229ff8c87ab3e8c909f46b4370d7c63a65c0f973a5c10fdebda800

  Port 5432
     ✔  should be listening

Test Summary: 1 successful, 0 failures, 0 skipped

```

In your case, the problem is that Apline's netstat reports a different output then other Linux eg. Ubuntu, CentOS:

```
# netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       User       Inode      PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      999        108791     -
tcp6       0      0 :::5432                 :::*                    LISTEN      999        108792     -

```

On Alpine, the inode output is missing:

```
netstat -tulpen
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:5432            0.0.0.0:*               LISTEN      -
tcp        0      0 :::5432                 :::*                    LISTEN      -

```

That is why InSpec is not able to parse the netstat output. It is clearly a bug in InSpec. As a mitigation, I recommend to install the `iproute2` package for alpine containers via `apk add iproute2`. InSpec prefers `ss` over `netstat`. Please report this bug to InSpec team.
> 3 votes

---
Tags: docker, ports, inspec, postgresql, alpine
---


# Question
Title: docker micro-service cannot connect to bitcoin docker
I have a micro-service `watchblockchain` that makes requests on `bitcoin` blockchain. When I use both services, everything works just fine. Then I want to put those services in docker containers. `watchblockchain` cannot communicate with the `bitcoin` docker. I use docker image: kylemanna/bitcoind/

Here is my `bitcoin.conf` file:

```
disablewallet=1
printtoconsole=1
rpcauth=jfjobidon:123...xyz
testnet=1
txindex=1
server=1
rpcport=8332

```

I can open a console in the `bitcoin` container and verify that the blocks are downloading and I can make requests on the blockchain ex: bitcoin-cli getblockcount

In `watchblockchain`, I have this connection config:

```
var clientBTC = new bitcoin.Client({
    host: 'bitcoin',
    port:  8332,
    user: 'jfjobidon',
    pass: 'Lw.....='
});

```

And I mapped `bitcoin` to 127.0.0.1 in my hosts file

My `docker-compose.yml`:

```
version: "3"
services:
  watchblockchain:
    build: ./watchBlockchain
    depends_on:
      - redis
    deploy:
      replicas: 1
      restart_policy:
        condition: on-failure
  redis:
    image: redis:4.0
    command: redis-server --appendonly yes
    ports:
      - "6379:6379"
    deploy:
      placement:
        constraints: [node.role == manager]
    command: redis-server --appendonly yes

```

When I launch: docker-compose up, I have the following error:

> Error: connect ECONNREFUSED 127.0.0.1:8332

I think it is a network communication problem.

How can I get my `watchblockchain` service to communication with the `bitcoin` docker ?

# Answer
> 0 votes
the containers `hosts` file will differ from your hosts `hosts` file.

see it kind of like a VM.

Also 127.0.0.1 is the wrong address, you might need to configure the actual ip/hostname your host has and where the service is listening on inside the watchblockchain container.

So Problems:

* Your docker container knows nothing about the host `bitcoin`. an entry in your docker hosts `/etc/hosts` won't change that.
* Maybe your bitcoin service only binds to localhost `127.0.0.1` \- this will also keep any docker container from connecting to it. Maybe you can configure it so that it binds to the docker bridge IP. So only all the docker containers are able to connect to it.
* It would be easier if your bitcoind would also be a docker container and was configured inside the `docker-compose.yml`. That way you could resolve the host using the container name. But still your bitcoind should be configured to bind on every interface.

Hope this helps ;)

---
Tags: docker, docker-compose
---


# Question
Title: How many builds does jenkins need to run before it has estimated times?
We just updated our Jenkins to v2.207.1 and our build estimation times on pipeline jobs have stopped showing up. Rather than a blue bar that when hovered over shows an estimate, we are now seeing the blue-and-white striped barber pole and an estimation of N/A.

I believe this behavior might be only on pipeline jobs - we have so few freestyle jobs that I haven't seen them run yet.

What logic does Jenkins use to calculate those estimations? How many builds will we need before those times fill back in?

# Answer
> 5 votes
I checked out the Jenkins source code, and I believe Jenkins uses the last 3 *successful* or *unstable* builds for its estimation. If it out of the last 6 builds it can't find 3 *successful* or *unstable* builds, it uses 1 or more of the last *completed* (not aborted) builds. It then takes the total duration time of the "candidates" and simply divides it by the number of candidates (i.e. mean of the durations). Here is the code I based this off of that can be found in the Job.java file in the Jenkins repo:

```
    /**
     * Returns candidate build for calculating the estimated duration of the current run.
     *
     * Returns the 3 last successful (stable or unstable) builds, if there are any.
     * Failing to find 3 of those, it will return up to 3 last unsuccessful builds.
     *
     * In any case it will not go more than 6 builds into the past to avoid costly build loading.
     */
    @SuppressWarnings("unchecked")
    protected List<RunT> getEstimatedDurationCandidates() {
        List<RunT> candidates = new ArrayList<RunT>(3);
        RunT lastSuccessful = getLastSuccessfulBuild();
        int lastSuccessfulNumber = -1;
        if (lastSuccessful != null) {
            candidates.add(lastSuccessful);
            lastSuccessfulNumber = lastSuccessful.getNumber();
        }

        int i = 0;
        RunT r = getLastBuild();
        List<RunT> fallbackCandidates = new ArrayList<RunT>(3);
        while (r != null && candidates.size() < 3 && i < 6) {
            if (!r.isBuilding() && r.getResult() != null && r.getNumber() != lastSuccessfulNumber) {
                Result result = r.getResult();
                if (result.isBetterOrEqualTo(Result.UNSTABLE)) {
                    candidates.add(r);
                } else if (result.isCompleteBuild()) {
                    fallbackCandidates.add(r);
                }
            }
            i++;
            r = r.getPreviousBuild();
        }

        while (candidates.size() < 3) {
            if (fallbackCandidates.isEmpty())
                break;
            RunT run = fallbackCandidates.remove(0);
            candidates.add(run);
        }

        return candidates;
    }

    public long getEstimatedDuration() {
        List<RunT> builds = getEstimatedDurationCandidates();

        if(builds.isEmpty())     return -1;

        long totalDuration = 0;
        for (RunT b : builds) {
            totalDuration += b.getDuration();
        }
        if(totalDuration==0) return -1;

        return Math.round((double)totalDuration / builds.size());
    }

```

---
Tags: jenkins
---


# Question
Title: Ansible Playbook Localhost Invocation Working Directory
I want to run an ansible playbook locally (host is localhost, like `-i "localhost,"`) and inside the Playbook obtain the working directory from which the `ansible-playbook` command was invoked.

However, when I read out the current working directory in a playbook task, I get the directory of the playbook itself.

Is there any variable I can refer to to obtain the working directory on the controlling host (which is localhost) in my case?

I can inject via a variable `-e` but I would prefer to have a fall-back that if that variable is not set I can obtain it from the (\[local\]host) environment.

# Answer
You can use `env` lookup plugins. Plugins are always evaluated in the context of "parent" ansible process on the control host.

```
- debug:
    msg: "{{ lookup('env', 'PWD') }}"

```

More lookup plugins can be found here: http://docs.ansible.com/ansible/latest/user\_guide/playbooks\_lookups.html
> 17 votes

# Answer
I'd use the shell module to run a simple pwd then use register to capture that

```
name: current_working_directory
    shell: pwd
    register: current_working_directory

```

you can then reuse this anywhere

```
{{  current_working_directory.stdout }}

```
> -1 votes

---
Tags: ansible
---


# Question
Title: Automated Integration Testing of an API in Azure Functions using VSTS
I am building a HTTP API as an Azure Function App in C# and I would like to run automated integration tests (calls to the API) as part of the CI/CD.

Is this possible to do using VSTS?

I have searched but only found how to do unit testing (not integration testing) as part of the deployment.

# Answer
There is no difference on vsts for your test (unit, integrated or functional). You just set the right dll name with your tests:

or select Test plan if you have a test plan with associated test recordings
> 1 votes

---
Tags: testing, azure, azure-devops, api
---


# Question
Title: Multiple builds from one branch but different CSS values
Looking for advice on how best to configure my workflow.

I have an application, that we currently customise for clients (white label) but our OLD devs have made the deployment of this very messy, multiple branches for each client. So merging is a nightmare and some clients are missing features etc.

The only difference between the clients version of our app, is some minor CSS changes and a couple of images.

We currently build with Jenkins and host with AWS Elastic Beanstalk.

What my question is, once we have merged and corrected the issues with the old developers. Am I able to trigger a single or multiple builds, from 1 branch and replace all the CSS values in my app and a couple of images? So we can just push to our master branch and build our production app and also all our clients white labels

# Answer
If I correct understand Your problem, You want to build app with diffrent ccs and images.

Jenkins suports maven and simplest way is to use maven profiles plus maven-war-plugin eg.:

```
<build>
<plugins>
    <plugin>
        <groupId>org.apache.maven.plugins</groupId>
        <artifactId>maven-war-plugin</artifactId>
        <configuration>
            <webResources>
                <webResource>
                    <directory>src/main/profile/webapp/${images.folder}</directory>
                    <targetPath>images</targetPath>
                </webResource>
            </webResources>
        </configuration>
    </plugin>
</plugins>

```

```
<profiles>
<profile>
    <id>prod_client_1</id>
    <properties>
        <images.folder>prod_client_1</images.folder>
    </properties>
</profile>
<profile>
    <id>prod_client_2</id>
    <properties>
        <images.folder>prod_client_2</images.folder>
    </properties>
</profile>

```

For other resources filtering maven-resources-plugin

At least should be helpful this approach sharing resources across projects
> 0 votes

---
Tags: jenkins, amazon-web-services
---


# Question
Title: How can I push multiple containers, created with docker-compose, to a registry
I am using Jenkins pipeline and run a shell command to bring up my linked containers for testing. Once the testing is complete, I want to push the containers to a registry (I would prefer AWS ERC, but at this point I will take any working examples). What is the proper way to do this? Should each image go to a different repository? How can I get all of the containers created by the compose file pushed up? I have been attempting to get this done with the docker-compose push command, but I can't seem to get it working properly or find any decent examples.

Jenkinsfile

```
node('ec2-build-slave') {
    stage('Docker Build') {
        /* This builds the actual image; synonymous to
        * docker build on the command line */

        sh("docker network create aa_backend_network")
        sh("docker-compose up -d --build")
        //docker.build("$IMAGE")
    }
    stage ('Docker Push'){
        // /* This pushes the previously built image to the Amazon
        // * ECS Container Registry */
        sh("eval \$(aws ecr get-login --region us-west-2 --no-include-email | sed 's|https://||')")
        // docker.withRegistry(ECR_URL, ECR_CRED) {
        // docker.image(IMAGE).push()
        // }

        sh("docker-compose build")
        sh("docker-compose push")
    }
}
catch (exc) {
    currentBuild.result = "FAILED"
    throw exc
}
finally {
    // Success or failure, always send notifications
    notifyBuild(currentBuild.result)
    sh("docker-compose down -v")
    sh("docker network prune -f")
}
}
}

```

docker-compose

```
version: "2"
services:
  db_setup:
    networks:
      - default
    build: .
    depends_on:
      - db
    container_name: db_setup
    image: db_setup
    command: python asteri_analytics_db.py
    working_dir: /asteri_analytics_db/asteri_analytics_db
  db:
    networks:
      - default
      - outside
    image: mongo:3.6.3
    container_name: mongo_db
    restart: always
    environment:
      MONGO_DATA_DIR: /data/db
      MONGO_LOG_DIR: /dev/null
      MONGO_INITDB_ROOT_USERNAME: ###edited####
      MONGO_INITDB_ROOT_PASSWORD: ###edited####
    ports:
      - 27017:27017

networks:
  outside:
    external:
      name: aa_backend_network

```

# Answer
> 3 votes
Just make sure the images you build all have the right names in terms of registry prefix and so on. If not you need to rename them accordingly.

Otherwise this task is pretty easy. To push all Images you build using a docker-compose file just use `docker-compose push`.

Make sure you are authorised to push to the registry (logged in etc.)

---
Tags: docker, jenkins, jenkins-pipeline, docker-compose, amazon-ecs
---


# Question
Title: Is Windows in general always slower than linux?
There are multiple distribution in the company. Building the same project results in different build times (see table 1).

```
| win10laptop32gbram | win10laptopOld | win7laptop | win7server | linuxvm | mac |
|--------------------|----------------|------------|------------|---------|-----|
| 3                  | 3              | 2          | 1.5        | 1       | 1.9 |

```

Table 1: Ratio of build speed between different Operating System and hardware.

Do other people from other organizations experience this as well?

# Answer
*TL;DR depends on the use case with up to 12 times between best/worst*

Comprehensive testing has been done over at Phoronix (28.03.18) and the results are very mixed. The tests looked at:

* Clear Linux 21510
* Debian 9.4
* OpenSUSE 42.3 Leap
* Ubuntu 16.04 LTS
* Windows 10 Pro Build 16299
* Debian 9 On WSL
* OpenSUSE 42.3 On WSL
* Ubuntu 16.04 LTS On WSL

The results are all over the show. In a few tests, the WSL (Windows Subsystem for Linux) is faster then native Windows (GraphicsMagick) and the Intel optimized distribution, Clear Linux, sometimes has the worst results (Netperf UDP request response). While in others Windows is twice as fast as all Linux distributions (CacheBench). All in all, deciding on **performance is not conclusive**. Choose the OS that provides the tooling you require.

Nonetheless, the article itself does provide a winner given that only first place finishes are taken into account:

> ... Clear Linux was the fastest of the operating systems tested with coming in first 40% of the time, Debian 9.4 in second with first place finishes 22.5% of the time, and Windows 10 Pro itself was the fastest 12.5% of the time.

When only taking Java into account:

> In the Java tests, to no real surprise, the performance tended to be about the same across all tested operating systems.

And finally, in a compile test (Golang), which is the test that is most related to the use case in the question:

> The build performance on Windows 10 and WSL were slower than the bare metal Linux performance.
> 7 votes

---
Tags: builds
---


# Question
Title: Octopus Deployment - only deploy some specific folders
We are using Octopus for our deployment to different environments. But we found that it always packages the complete solution and deploys it to the next level. Is it possible to put some restrictions - like do not copy and deploy the config folders. In our case we just need to deploy some specific folders and not the complete packaged solution. Is this possible with Octopus.

# Answer
Yes.. we can control the packages using nuspec file which will be added to your solution and you will be specifying the list of folders/files you dont wanna deploy. And you will use that nuspec to create the nuget package for deployments
> 1 votes

# Answer
As far as I know, I think Octopus doesn't packages anything. The package should come from your CI Tool (Jenkins or TeamCity maybe?)

Having said that, I would *check on your previous step* in the CI/CD chain to see how are you packaging your solution, and filter those folders there, before packaging.

That will also help you:

* Reducing the package size
* Increase speed in the deployment process
* Avoid storing unnecessary files

**High-level suggested CI/CD Chain summary**

CI Tool

```
1. Compile/Test (if you're working with a compiled language)
2. Package only required folders
3. Push to Octopus

```

Octopus (CD Tool)

```
1. Store and manage final packages

```
> 1 votes

---
Tags: deployment, continuous-deployment, dotnet, octopus-deploy
---


title: Docker image delivery lead time SLA

text:
# Question
Title: Docker image delivery lead time SLA
What is the best practice either reference value in terms of an SLA spec to expect the infrastructure to:

* deliver a Docker image say for 100MB and 1GB size as reference?
* unpack the above?
* instantiate a container of the above?

Sure you can say "how important is that" but somewhere it gets irrational. SLA offers rational approach.

# Answer
Service Level Agreement

https://en.wikipedia.org/wiki/Service-level\_agreement

https://www.atlassian.com/it-unplugged/best-practices-and-trends/stop-hating-on-slas

https://www.cio.com/article/2438284/outsourcing/outsourcing-sla-definitions-and-solutions.html

What will happen if the docker registry is down? Are all departments blocked? What is the impact?

If all production images reside in a docker registry and someone wants to release a newer version to production, but the registry is down, is it still possible to release the product?

SLA is about defining processes, checking the impact of each of them and the more important the higher the SLA.

One could this tool to choose a certain SLA for a certain procedure:

https://uptime.is/

A SLA of 99.999 https://uptime.is/99.999 means a yearly downtime of 5m 15.6s, while https://uptime.is/99.9 means 8h 45m 57.0s.

One could set a SLA of 99.999 for the docker processes, but is a downtime of 5m 15.6s really required or is 8h 45m 57.0s in a year acceptable? If one hosts an ecommerce website and the webshop is down a couple of hours this will cause a lot of harm for the company as customers could not buy products and will probably buy products somewhere else.

For the docker processes that are used by developers and for releases to production one could argue that the first one could be 99.9 and the latter 99.999.
> 1 votes

---
Tags: docker, performance, infrastructure, sla
---


# Question
Title: Run docker inside a VirtualBox of Windows Server 2016 (GUI) on Mac OS
I have downloaded the Windows Server 2016 iso from Microsoft for evaluation.

I then installed it via VirtualBox on my Macbook Pro running the latest Mac OS (High Sierra).

I downloaded Docker for Windows but when I try to load it, it either fails with an error saying it cannot write to database, or it fails because it says virtualization / Hyper-V has to be enabled in BIOS.

I have Hyper-V selected in the VirtualBox settings so that should work (and restarted the virtual machine).

However it still fails.

Is it possible to run Docker for Windows on a Windows Virtual Machine, all hosted on Mac OS?

# Answer
According to me this is not possible as the hyper-v should be enabled on the host system. If this is enabled in a VirtualBox, but not on the host system, then this will not work according to what I have experienced.
> 1 votes

---
Tags: docker, windows
---


# Question
Title: Build isolation - Docker container based builds in Bamboo 6.4.0
Finally, it is possible in Bamboo to let a build run in a Docker container without dealing with agent configuration.

The system just asks for a image.

Now, what about the underlying infrastructure, what happens there? Will then Bamboo server host just spawn containers locally (which limits horizontal scalability)?

# Answer
> 3 votes
Yes, you are correct. Bamboo will spawn the build containers on the agent that the build is running on (either Local, Remote, or, Elastic - depending on your configuration).

In order for this to work the agent will require Docker to be installed and running.

---
Tags: docker, bamboo
---


# Question
Title: Docker managers are losing quorum on AWS
I have deployed Docker EE on AWS by choosing the package directly from the AWS Marketplace: Docker EE for AWS (Standard/Advanced) - BYOL. The cluster has been successfully launched and deployed on AWS.

After deploying several stacks on the UCP something really strange is happening, the managers are not able to see each other, they are losing the quorum inside the Swarm and the `docker info` command says that they are out of Swarm.

The AutoScalling group from AWS starts launching new managers inside the Swarm and the new launched managers are terminated in a couple of minutes because they are not Swarm's participants as well. Actually the AutoScalling group goes in a infinity loop by launching new managers and terminating the old ones.

The worst thing is that I am loosing the data from the Docker Services and there is no way of recovering.

Any ideas?

# Answer
It seems like the AutoScaling Group from AWS are not that synchronized with the Docker Swarm configuration and the default timeouts in the AutoScaling are not accepted in the Docker Swarm managers.

Actually the Health Check is not configured good by default in the EE CloudFormation template, even if the managers are running and stable the Health Check fails and the AutoScalling Group tries to launch new managers. New launched managers are not able to become cluster participants, right after their running state.
> 1 votes

---
Tags: docker, amazon-web-services, docker-swarm
---


# Question
Title: Automatic production deployment for a Node.js app on RHEL 7?
I am looking for some way to have a maximally simple(no Jenkins or anything like that) automatic production deployment for a Node.js application on a RHEL 7 system. The caveat is:

At the organization where I work, we have a very restricted IT environment, that prohibits access to any ports other than 80 and 443 from anywhere, and only allows 22(ssh) from within some rooms in the office building. To upload files a tunnel is needed which is slow and is also not available in some places. And SSHing to the server to run commands is a huge pain for multiple reasons too. And no - I can't do anything about it.

In light of this the 'trigger' has to initiate from the server itself. The server should somehow decide to pull changes from a certain branch, ideally when a special commit message is seen, and then run some pre-configured scripts to deploy it. The tests at that point are already all done, the app has been at the staging and beta server and so the only things needed is to copy directory, update dependencies, compile frontend assets and restart the systemd service that runs the app (all that I already have in the script that Jenkins executes on staging server after running all the tests).

I would imagine the sever would need to poll, e.g. every minute for changes to the repo, and yes, that can be done with a cron job, but I am not sure how, and if there are some tools that make it easier, without creating a resource hog that Jenkins does (which is what we use on our staging and beta servers), then I would like to know about it (I wasn't able to find anything yet, maybe I am not using the right search terms).

Any advice?

Update-1: I am aware of Puppet and Chef, but if at all possible, would prefer avoiding proprietary solutions if possible for now because a - our project is not big enough yet for the decision makers to put real resources in it, and b - even if those have free versions available, they are most likely cumbersome and do a lot of things, whereas we want as simple a solution as possible, because that solution might be part of the package if we sell the project elsewhere (naturally we don't want to also "re-sell" Puppet or Chef). Some kind of Open Source solution, or simply a method of doing it using simple shell scripts would be best.

Update-2: As Tensibai mentioned - Chef is open source and fits our purposes, but it's a huge platform with a lot of features, which is precisely what we are trying not to use in production server (otherwise we'd just use Jenkins like we do in staging).

# Answer
> 1 votes
All this time after I think I am finally able to answer this question. The solution is Ansible \- just like @James Shewey mentioned. It's **agentless** and quite **light** python framework, that can be made to work through jump hosts and perform automated tasks through SSH. This means it requires no server setup or manual operations of any kind from our customer to deploy the application.

Without anything else it's a one-command deployment solution, which is almost as good as a fully automated one and it has the benefit of being readable and usable by people not familiar with code.

Puppet offers similar functionality, but it has an agent which means if packaged with the application, it does require manual server setup from the customer. Also Puppet is ruby and I try to stay clear of that thing as much as possible. I've learned the hard way just how bad ruby is for anything other than prototyping.

# Answer
> 1 votes
Ok, let's assume you'll pull the latest version of branch prod from a git repo, here's what it would take with chef on a basic illustrative purpose:

```
git "/opt/my_application" do
  source "https://<git_host>/<you_repo_url>/<repo.git>"
  revision "production"
  action :sync
  notifies :run,"execute[app_config]"
end

execute "app_config" do
  command "npm install"
  cwd "/opt/my_application"
  action :nothing
end

```

For more advanced use cases you can have a look at the poise application cookbooks, it includes a javascript plugin for node.js application.

chef is fully open source, either client or server side (out of the fancy UI) and on all OSes including windows.

---
Tags: git, linux, shell-script, node.js
---