# Question
Title: When exactly and what for should VOLUME in a Dockerfile be used?
When writing a Dockerfile, I do have the option of specifying volumes, such as:

```
VOLUME /var/www

```

But even without this declaration I am able to mount volumes to this directory and other places, e.g. by using `-v /foo:/var/www` or -v `./httpd/conf.d:/usr/local/apache2/conf/conf.d` for bind mounts when running docker.

So:

1. Is the specification purely informational?
2. What are the (dis)advantages of (not) specifying volumes?
3. Why and in which cases should I use `VOLUME` and when shouldn't I?
4. What about files that may optionally be mounted? Do they count as volumes in that case, too?

# Answer
> 3 votes
**Short version**

`VOLUME` instruction and `-v dest` are used to create unnamed/anonymous volumes, `-v scr:dest` option is for mapped volume.

**Long Version**

Dockerfile's `VOLUME` does not allow you to specify a host path.
On the host-side, the volumes are created with a very long ID-like name, these volumes are often referred to as unnamed/anonymous volumes.

Given this Dockerfile:

```
FROM php7:latest
VOLUME /var/www

```

Build it:

```
docker build -t myTest

```

Run it :

```
docker run --rm -it myTest

```

Inside the container, run `ls` and you'll notice the directory exists; `/var/www`

Running the container also creates a directory on the host-side.

While having the container running, execute `docker volume ls` on the host machine and you'll see something like this

```
DRIVER    VOLUME NAME
local     c984..e4fc

```

Back in the *container*, execute `touch /var/www/myFile`

This file is now available on the host machine, in one of the unnamed volumes.

```
ls /var/lib/docker/volumes/c984..e4fc/_data

```

Similarly, you can try to delete this file on the host and it will be deleted in the container as well.

Now run a new container, but specify a volume using `-v`:

```
docker run --rm -it -v /myVolume myTest

```

This *adds* a second volume and the whole system ends up having two unnamed volumes. On the host-side, the new second volume is anonymous and resides together with the other volume in `/var/lib/docker/volumes/`.

It was stated earlier that the `Dockerfile` can not map to a host path which sort of pose a problem for us when trying to bring files in from the host to the container during runtime. A different `-v` syntax solves this problem.

Imagine I have a subfolder in my project directory `./src` that I wish to sync to `/src` inside the container. This command does the trick:

```
docker run -it -v $(pwd)/src:/src myTest

```

Both sides of the `:` character expects an absolute path. Left side being an absolute path on the host machine, right side being an absolute path inside the container.

We run this command:

```
docker run -v $(pwd)/src:/src myTest

```

**Is the specification purely informational?**

No, it does mount a volume, it's similar to `-v /var/www` you just don't specify a mount point in the host machine, so docker will take care of it.

**What are the (dis)advantages of (not) specifying volumes?**
**Why and in which cases should I use VOLUME and when shouldn't I?**

It's probably a best practice to never use `VOLUME`.
The first reason we have already identified: We can not specify the host path.
The second reason is people might forget to use the `--rm` option when running the container, you will end up with a couple of unused volumes and it might be a daunting task to figure out which of all anonymous volumes are safe to remove

**What about files that may optionally be mounted? Do they count as volumes in that case, too?**

In this case you don't need volumes; mapping a single file does not make sense, I would use `ADD` or `COPY` inside the dockerfile.

---
Tags: docker, dockerfile
---


title: Why HashiCorp vault returning null?

text:
# Question
Title: Why HashiCorp vault returning null?
I have a HashiCorp vault setup and it seems to return `null` all the time except when I ssh into the VM and then I set the values again followed by checking the values directly on the vault server.

Any ideas what might be happening?

I am a bit lost on this and not even sure what logs or config files are helpful. This is one of my first experiences with vault and so far it seems finicky at best.

# Answer
If you are unsure what happened, try deploying it in AWS via AWS vault quickstarts. They deploy vault end-to-end, so now you can compare it to yours and find out what was the problem.
> 1 votes

---
Tags: hashicorp-vault
---


# Question
Title: What are best practices for running Hashicorp's Vault in terms of high availability?
The Production Hardening document indicates how to harden Vault for production purposes, but what if Vault is run on for example Digital Ocean and the platform is unavailable or the disk gets corrupted?

# Answer
> 1 votes
According to this documentation two vault servers have to be run. One of them has to be active and the other passive. In order to run such a setup both vault servers need to be unsealed.

# Answer
> 1 votes
Well, the first step is to use a backend which supports HA. Consul, for example, supports HA. If you are not an enterprise customer, you may have to run a shell script via cron to upload consul snapshots to s3 or where ever you want to store them. Later, you can restore all your secrets. And obviously, the snapshots are encrypted.

---
Tags: hashicorp-vault, high-availability
---


# Question
Title: How granular in purpose should FaaS functions be?
For clarification, when I say "FaaS" I am referring to serverless offerings such as AWS Lambda and Google Cloud Functions.

I have a specific case in mind that I'll describe but I would very much like to hear about any guiding principles in this area as well.

I have a Google Cloud Function that is currently doing some image processing. The flow looks like this:

1. JSON file containing several base64 encoded images is uploaded to storage bucket
2. Cloud Fucntion downloads file
3. Converts and merges all base64 strings into image file
4. Uploads image file to storage bucket
5. Calls Google Cloud Vision API once file is uploaded

This works, but it has me wondering: how much should a single function *do*? Most discussions I see regarding single purpose vs. monolithic functions are centered around monolithic functions having logic branches e.g. "if-else". In the above case, this is an example of doing a lot, but, it's conceptually a single unit of "work".

Were I to refactor this into true single purpose functions, I imagine I'd have two functions -- one for downloading and merging images and the other for calling the Vision API. This splits what I consider "image processing" into two sub-units. I have no problem doing this but I'm curious to know whats a good rule-of-thumb for "splitting" FaaS function?

Edit: if anyone is wondering why the image file gets uploaded back into a storage bucket and then the Vision API is called, this due to a limitation of that API where inputs can only be storage bucket URIs and not local files

# Answer
In my point of view, you can have a future gain separating the function into multiple parts because it will allow to use another combining system/upload/whatever as input to the processing part and another system to process the images later.

My personal rule of thumb for separation is the main concern:

* Step 1 is giving the input, it's already somethhing it itself.
* Steps 3 and 4 are, basically, take an input, do the work andgive an output. It waits base64 strings as input and output an image file to a bucket, it should be autonomous in its processing
* Step 5 is an api call, it waits image as input and "whatever vision API does" as result/output, that's a task in itself.

I did omit step 2 as it's the trigger of step 3 and 4.

The main interest into this separation of duty is the possibility to work with contracts between the functions (mostly bullet 2 and 3): 2 is expected to output images in a bucket, 3 expects to find images in a bucket, you can then replace 2 or 3 with whatever you want as long as this contract is fulfilled.

Now for when to do this split and is it necessary, if you don't expect things to change in a near future and there's no hard limit approaching, like the time to do the whole work is approaching the max execution time, then there's no need to over anticipate problems in my humble opinion, the whole step 3 to 5 can be seen as a single task unit as you said and this make sense.

My take would be to measure the time within the function to see if there's an interest (for parallel processing for exemple) in taking the combine and transform to image to its own unit.

In brief to answer the question title: there's no silver bullet or hard rule for the granularity, it's always a matter of weighting the benefits toward the cost of slicing the process.
> 2 votes

---
Tags: aws-lambda, serverless
---


# Question
Title: Automated tag versioning on unchanged docker images
I am starting to use Continuous Integration on my microservices solution (monorepo approach), and I am referencing this post for versioning approach.

But, let's say I make some change in one of the microservice and commit that change, but all the others are exactly the same. How the process would know that those microservices (docker images) weren't changed so don't need to increase the build number? I feel like I would end-up with many docker images with different build numbers but same content.

Am I misunderstanding the concept? Sorry, this whole microservices and docker thing are pretty new to me.

# Answer
> 1 votes
How it should look like:

1. Commit changes in some microservice and push it to VCS origin repo
2. CI system sees new commit
3. CI starts build and tests
4. If everything's good, it pushes image to docker registry with tags

So, the answer is: CI should know, what service has changes.

# Answer
> 0 votes
@chupasaurus gave an answer based on a multi-repo solution.

Based on this answer, I came up with the following jenkins pipeline approach. I've decided to build and test the whole solution, but you can use the same approach from "Push Image" stage to build and test only the changed services.

```
pipeline {
agent any
environment {
    CHANGES = getChangeLog()
}
stages {
    stage('Build') {
        steps {
            bat 'dotnet clean solutionFile.sln'
            bat 'dotnet build solutionFile.sln'
        }
    }
    stage('Test') {
        steps {
            bat 'dotnet test solutionFile.sln --no-build --logger:trx;LogFileName=results.trx'
            // parse tests
        }
    }
    stage('Push Image') {
        stage('serviceA') {
            when {
                allOf {
                        expression {
                            hasChanged = powershell(returnStdout: true, script: '"$env:CHANGES" -like "*serviceA*").trim() // check if any file path contains "serviceA"
                            return  hasChanged == "True"
                        }
                        branch 'master'
                }
            }
            steps {
                // set tag or use latest by default from docker-compose
                bat 'docker-compose build serviceA.api'
                bat 'docker-compose push registry/serviceA.api'
            }
        }
    }
    stage('Deploy Image') {
        stage('serviceA') {
            // deploy image code
        }
    }
}

def getChangeLog(){
    def changes = ""
    build = currentBuild
    while(build != null && build.result != 'SUCCESS') {
        changes += "In ${build.id}:\n"
        for (changeLog in build.changeSets) {
            for(entry in changeLog.items) {
                for(file in entry.affectedFiles) {
                    changes += "${file.path}\n"
                }
            }
        }
        build = build.previousBuild
    }
    echo changes
    return changes
}

```

This code isn't fully tested, I just wanted to give a practical approach.

---
Tags: docker, containers, microservices
---


title: Ultimate docker permission solution

text:
# Question
Title: Ultimate docker permission solution
I'm using docker some time and I like the conception very much. But I found a thing is a big problem for me.

I'm talking about permissions. While I'm not worrying about production, setting up my docker development environment becomes hell, because of file ownerships and permissions. For example:

* www-data user from php container cannot create files in my directories (www-data uid is 82 while my uid is 1000) I was trying to change www-data id to 1000 but I share this environment with macOS users, who typically have uid of 500
* I cannot remove files that have been created within container - that makes it hard for me to jump between git branches sometimes (`sudo rm somefile`).

I spend hours, no, days looking for ultimate solution. Tried `gosu` in containers, mounting `/etc/shadow` as a volume etc. The problem is, I was unable to find a solution that can be considered multi-platform and proper (mounting local system files readonly just doesn't look good).

Thank you!

# Answer
I'm a little confused about the exact use case. Are you asking about writing to volumes that are mounted inside the container?

You can change what the process in the docker container runs as by using --user on your run commands.

```
docker run --user 1000 --ti centos/7 /bin/bash

```

This answer may help you: https://stackoverflow.com/questions/41100333/difference-between-docker-run-user-and-group-add-parameters

The manual explains this as well: https://docs.docker.com/engine/reference/run/#user
> 1 votes

# Answer
For the easy solution, use named volumes when possible. They will initialize to the contents of the image, include file owners and permissions.

When you need host volumes, my preferred solution for developer machines is to run the entrypoint as root and drop permissions to the application user after adjusting uid/gid values of the users inside the container to match the the id's on volumes that have been mounted. For example, here's a repo where I've done this for a Jenkins container that needs to run as the jenkins user but with gid access to the mounted docker socket: https://github.com/sudo-bmitch/jenkins-docker

The key steps to the solution:

* Start the container as root on developer machines (in production you can put some logic in the entrypoint to know that it's not root and make the file permissions on the server match those of the container).
* Configure the entrypoint to lookup the uid/gid of a selected host volume.
* Compare the volume uid/gid to the container user's uid/gid. If they do not match, update the user's uid/gid inside the container and adjust file permissions on files outside of the volume. Be careful to handle the scenario where the uid/gid already exists inside the container.
* Use gosu to switch from root to your container user and run the desired application.
> 1 votes

---
Tags: docker, linux
---


title: Kubernetes Deployments & Rollbacks

text:
# Question
Title: Kubernetes Deployments & Rollbacks
At the moment I'm performing Kubernetes deployments by applying `deployment.yaml` then updating the container/pod image versions with `kubectl set-image` as outlined in the docs. I'm doing this using Jenkins.

If I understand correctly, each time I apply `kubectl set-image` it updates the deployment in situ creating a new ReplicaSet meaning that I don't need to create a new `deployment.yaml` for each version bump.

Is this best practice or should I be creating a new `deployment.yaml` every time we bump version?

My next question concerns rollbacks, if I run `kubectl rollout history deployment/name`, I see three revisions but no 'cause'. I think this is because I've not specified `--record` on the initial deployment, and does that record each time we do a `set-version` too?

# Answer
I think it might be a good idea to update deployment.yaml for each version/release and commit to git so that later, any other person may follow through and even revert back to an earlier release/version.
> 2 votes

# Answer
When we started to use Kubernetes at my current job we just re-applied the deployment manifest, updating the DATE on a annotation, this worked fine for some time, then we switched to use HELM https://helm.sh/

My recommendation is to switch to Helm since it manages deployments in a better way, even config maps are versioned between releases and also you can roll-back as you want.

Makes life easier when integration with CI/CD pipeline cause everything is parameterized/configurable.
> 0 votes

---
Tags: kubernetes, deployment
---


# Question
Title: Preserving ownership of a folder after mounting it
I have dockerized an application which accesses a folder "Folder1" inside the container. The folder is owned by a user "user1". When I bring the container up, I need to mount this folder over my host machine. Upon doing so, the ownership of "Folder1" inside the container changes to root as the corresponding mapped folder is under root's ownership on the host machine.

This change of ownership makes the folder inaccessible to my application inside the container. Any idea how do I fix this?

# Answer
> 1 votes
At least in the \*nix world the ownership (i.e. uid and gid) of a file/folder on a mounted filesystem doesn't change, it is the actual ownership from the host exporting that filesystem (if ownership is supported by that filesystem).

Note that if the container has different user/group names matching the uid/gid of the file/folder they will appear under the container's names in `ls -l`, not as the host's names (or as numbers if no container names match the respective IDs).

So if on the host machine the folder is owned by root, it will also be owned by root in the container (in other words it's nothing "special" about it being owned by root vs other users). The only way to allow the container app to write to the filesystem is to change the ownership accordingly on the host system (or give it `other` write permissions).

Maybe worth noting that if the host's filesystem itself doesn't support ownership, it can still be "faked" inside the container using the uid and gid mount options (in the \*nix world). See Ownership of a mount point.

---
Tags: docker
---


title: Where to state CFT dependencies

text:
# Question
Title: Where to state CFT dependencies
With AWS, why can't I just state in the cloudformation template which dependencies I need? Right now I need to add the dependencies automatically in an accompanying script during our gitlab pipeline. It would seem more convenient to do like Google Appengine, and state the dependencies in the top-level template. Here's an example adding dependencies from my google appengine app.yaml, just stating what I need and it seems AWS doesn't have something similar:

```
libraries:
- name: webapp2
  version: latest
- name: jinja2
  version: latest

```

# Answer
> 1 votes
This is a matter of implementation details and it's actually coming from the AWS Lambda product's implementation (which the doc you linked refers to), not from the Cloud Formation itself.

Presently the AWS Lambda implementation *requires* you to include the actual dependent packages in the deployment package (per the doc you referenced). Hence the CFT doesn't really need a way to specify such dependencies - it'd be useless as it can't be passed to Lambda.

Should Lambda infra end up supporting a way to just list the dependent packages, Cloud Formation *might* be enhanced to support specifying the list right in the template and passing it somehow to Lambda. But this is just speculation, it all depends on the AWS feature roadmap :)

Side note: the Google App Engine (GAE) example could be seen as a bit questionable as it's specific for just a small subset of that product's capabilities (applicable only to the GAE-supplied libraries for the python 2.7 standard environment). Maybe a better reference would be to the `requirements.txt` or `package.json` (or other language-specific) files used to specify external/third party package dependencies for GAE environments, as well as other Google Cloud Platform (GCP) products, including Cloud Functions (CFs) (which would be the GCP product closest matching AWS Lambda). See `Specifying dependencies` for CFs in Python, NodeJS 6 and NodeJS 8.

---
Tags: amazon-web-services, aws-cloudformation
---


# Question
Title: Kubernetes - unable to login to Dashboard
I have installed Kubernetes using AWS Minikube scripts - https://github.com/scholzj/aws-minikube

After installation I am trying to run

```
kubectl proxy

```

But I am getting this

> https:kubernetes-dashboard:\\" is forbidden: User \\"system:anonymous\\" cannot get services/proxy in the namespace \\"kube-system\\"

I am not even able to list pods because I do not have valid tokens with `system.master` role. I have no idea how I can create a user or token and authenticate when there is no `admin` user present on the system yet.

Can you please explain to me?

# Answer
how we can extract the certificates from the kubeconfig file:

1. Locate your kubeconfig or config file which you use to run kubectl commands. If you have used my Vagrant file above, you can find it on /etc/kubernetes/admin.conf
2. You need to export a single file (.p12) with the following two certificates: the client-certificate-data, and the client-key-data. My example runs the command on /home/vagrant. If you run this command on macOS, be sure to change the base64 -d to base64 -D

```
$ grep 'client-certificate-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.crt

$ grep 'client-key-data' ~/.kube/config | head -n 1 | awk '{print $2}' | base64 -d >> kubecfg.key

$ openssl pkcs12 -export -clcerts -inkey kubecfg.key -in kubecfg.crt -out kubecfg.p12

```
3. Import the kubecfg.p12 certificate, reopen your browser, and visit the Kubernetes Dashboard URL. Accept any warning and you should see the authentication page. You can skip the login and check you are not able to perform any task.
4. The following steps have been copied from the Kubernetes Dashboard wiki page (Creating-sample-user)

```
    1- Create service account

        cat <<EOF | kubectl create -f -
        apiVersion: v1
        kind: ServiceAccount
        metadata:
          name: admin-user
          namespace: kube-system
        EOF
        >>

    2- Create ClusterRoleBinding

        cat <<EOF | kubectl create -f -
        apiVersion: rbac.authorization.k8s.io/v1
        kind: ClusterRoleBinding
        metadata:
          name: admin-user
        roleRef:
          apiGroup: rbac.authorization.k8s.io
          kind: ClusterRole
          name: cluster-admin
        subjects:
        - kind: ServiceAccount
          name: admin-user
          namespace: kube-system
        EOF
        >>

    3- Get the Bearer Token. Once you run the following command, copy the token value which you will use on the following step.

        $ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')

    4- Come back to your browser and choose token on the login page. You will need to paste the token value you have copied on the previous step.
5- Click “SIGN IN” and you should be able to see your Kubernetes Dashboard fully operational.

```
> 2 votes

---
Tags: kubernetes
---


# Question
Title: How do I prevent service A from being placed in the same node as service B in Docker Swarm?
From the docs, I could only find information about placement restrictions based on **node** attributes, likes labels and its role. However, I couldn't find how to configure placement based on the **services already running** on a given node.

How can I do that?

# Answer
> 2 votes
At present the answer is "no", swarm mode does not support affinity or anti-affinity between a service and other containers/services running on a node. However, depending on your goal there are other options.

One option is to reserve resources like CPU and memory for your container. This will ensure two resource intensive applications do not schedule themselves on the same node if there are insufficient resources.

The other option, as you've mentioned, is to use labels and constraints. You can give the same label to multiple nodes, which is often done to specify a type of node, or assign nodes to a specific team/project/environment.

Both of these options will reduce your fault tolerance, particularly if you assign a label to one or a small handful of nodes. If those nodes with the label go down, docker will be unable to reschedule your service on another node.

---
Tags: docker, dockerfile, docker-swarm
---


# Question
Title: dotnet not found on travis for an F# project
I'm new to both travis and this website, so please bear with me ;).

I've been trying to set it up to work with my GH project. The repo is visible on travis side, but I get the following error while building:

`./fake.sh: line 25: dotnet: command not found`

Not sure why - most likely I've done something wrong with my .travis file. Here it is:

```
{
  "os": "linux",
  "dist": "trusty",
  "mono": "5.10.1",
  "sudo": "required",
  "group": "stable",
  "dotnet": "2.1.403",
  "script": [
  -  "./fake.sh build -t Nuget"
  ],
  "install": [
  -  "export FrameworkPathOverride=$(dirname $(which mono))/../lib/mono/4.5/"
  ],
  "language": "fsharp",
  "before_install": [
  - "chmod +x fake.sh"
  ]
}

```

The file "fake.sh" is my build script. It uses the F# FAKE library, but the target I invoke contains nothing, but invocation of "dotnet pack". It works on my local machine.

EDIT: The fake.sh is the standard, unchanged F# FAKE file that is included in "dotnet new fake" template. Here's its content:

```
#!/usr/bin/env bash

set -eu
set -o pipefail

# liberated from https://stackoverflow.com/a/18443300/433393
realpath() {
  OURPWD=$PWD
  cd "$(dirname "$1")"
  LINK=$(readlink "$(basename "$1")")
  while [ "$LINK" ]; do
    cd "$(dirname "$LINK")"
    LINK=$(readlink "$(basename "$1")")
  done
  REALPATH="$PWD/$(basename "$1")"
  cd "$OURPWD"
  echo "$REALPATH"
}

TOOL_PATH=$(realpath .fake)
FAKE="$TOOL_PATH"/fake

if ! [ -e "$FAKE" ]
then
  dotnet tool install fake-cli --tool-path "$TOOL_PATH"
fi
"$FAKE" "$@"

```

All it does is it makes sure all the build requirements are there in place. Then it runs the "build.fsx" script:

```
open Fake.Core
open Fake.DotNet
open Fake.IO
open Fake.IO.FileSystemOperators
open Fake.IO.Globbing.Operators
open Fake.Core.TargetOperators

let nugetDir = "../nuget"

Target.create "Clean" (fun _ ->
    !! "src/**/bin"
    ++ "src/**/obj"
    ++ nugetDir
    |> Shell.cleanDirs
)

Target.create "Build" (fun _ ->
    !! "src/**/*.*proj"
    |> Seq.iter (DotNet.build id)
)

Target.create "Nuget" (fun _ ->
    !! "src/**/*.*proj"
    |> Seq.iter (DotNet.pack (fun o -> { o with OutputPath = Some nugetDir }))
)

"Clean"
  ==> "Build"

"Clean"
  ==> "Nuget"

Target.runOrDefault "Build"

```

# Answer
> 2 votes
OK - found the solution myself. It's a kind of tricky one. Even though my project is written in F#, I still have to set

`"language": "csharp"`

in my .travis.yml file. That's a bit counter-intuitive, but I must admit that it's written in the doc, so can't really blame anyone.

---
Tags: travis-ci, dotnet
---


# Question
Title: What is the behavior if docker compose fails?
It seems to me that all the containers attempted to be launched are removed if Docker Compose fails for one container being launched.

Is this standard behavior, or is this just an artifact of my case?

# Answer
Well it depends on whether the services (containers) within the docker-compose.yml depends on each other. If for example, service B depends on service A, and A fails to start up, so will B.

It can also be the case that a service logically depends on another service (without actually declaring it in your docker-compose.yml), and it not starting up causes a chain of failures. For example, say you have a web service and a db service. If the web service attempts to connect to the db service but the db service couldn't start up (config broken for example), then the web service will probably throw an exception and fail.

But no, just because one service fails, doesn't mean all other services will just be stopped. You can have two independent services, where one fails and stops, while the other is still running.
> 3 votes

---
Tags: docker, docker-compose
---


# Question
Title: What are some good Kubernetes training courses?
I'm looking for Kubernetes training online or in person. What are some good training courses to take?

# Answer
To understand kubernetes internal, it's good to have good understanding of containers, docker, etc.

Here, is the list of useful docker and kubernetes resources:

1) Play with Docker Classroom https://training.play-with-docker.com/

2) Docker Interactive Tutorials https://www.katacoda.com/

3) Play with Docker (PWD) https://labs.play-with-docker.com/ Play with Kubernetes(PWK) https://labs.play-with-k8s.com/

4) Introduction to Kubernetes course on edX also his company, CloudYuga is a certified Kubernetes service provider and training partner

5) Blog by Docker Captain, https://collabnix.com/ Also, search kubenetes blogs on Medium which are very useful.

6) Kubernetes The Hard Way github repo by Kelsey hightower updated with latest kubernetes 1.12.0 version.

Hope this helps.
> 1 votes

---
Tags: kubernetes, learning
---


# Question
Title: How do I handle multiple AWS accounts in Terraform?
I've read up on Terraform and using AWS S3 and DynamoDB for sharing state and locking. My problem is that we have multiple AWS accounts, by which I mean multiple sign-in locations like:

Is it better to have separate S3 and DynamoDB instances in each account, or should we have a single location that's used across all of them?

# Answer
The answer is highly dependent on what you put inside it, if the states are only account related I'd go with a separate S2/dynamodb per account, if the states are cross-account then you'd need a central place to record them.
> 1 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: How can I get the first part of the API Deployment URL from inside cloud formation template?
https://u1i7vf9mnl.execute-api.us-west-1.amazonaws.com/v1

How can I get the u1i7vf9mnl portion of the URL from inside of the cloudformation template as a Ref or Fn::GetAtt or some other way?

# Answer
Did you try the variable `ServerlessRestApi`? It works for us like this:

```
Outputs:
    EncryptURL:
      Description: "Encrypt endpoint URL for Stage environment"
      Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Stage/encrypt"

```
> 2 votes

---
Tags: api, aws-cloudformation
---


# Question
Title: What processes or tools enable Segregation of Duties when engineers both deploy and run code?
In highly regulated environments, such as the Financial Services sector, Segregation of Duties is an essential mechanism for avoiding collision amongst individuals with development responsibilities and production privileges.

Traditionally this has meant that **Developers** develop code and then hand it over to **Operations** however in many DevOps Operating Models the segregation between Development and Operations is, at a minimum, blurred:

* In Google's Site Reliability Engineering, or SRE, practice there is a separate SRE function within Google, however, Developers are brought in to backstop the SREs in times of high operational load.
* In the "You Build It, You Run It" model there is no separate operations function.

After spending months drilling down to the root causes of a Segregation of Duties mechanism it seems it predominantly exists to satisfy Sarbanes Oxley Section 404: Management Assessment of Internal Controls:

> (a) Rules Required. The Commission shall prescribe rules requiring each annual report required by section 13(a) or 15(d) of the Securities Exchange Act of 1934 to contain an internal control report, which shall--
>
> (1) state the responsibility of management for establishing and maintaining an adequate internal control structure and procedures for financial reporting; and
>
> (2) contain an assessment, as of the end of the most recent fiscal year of the issuer, of the effectiveness of the internal control structure and procedures of the issuer for financial reporting.
>
> (b) Internal Control Evaluation and Reporting. On the internal control assessment required by subsection (a), each registered public accounting firm that prepares or issues the audit report for the issuer shall attest to, and report on, the assessment made by the management of the issuer. An attestation made under this subsection shall be made in accordance with standards for attestation engagements issued or adopted by the Board. Any such attestation shall not be the subject of a separate engagement.

Based upon the comments it is important to call out a couple of **assumptions** I am making:

* I am predominantly considering mass-market financial services, i.e. transaction volumes are high but relatively low value. This would be as opposed to commercial financial services which has a different transaction value profile.
* A financial institution's online offering will be made up of many components that have differing risk considerations:
    + **Move Money** \- Moving money between accounts or transfers between accounts of different owners. An operation that has to consider Anti-Money Laundering, Fraud Protection, and Embargo'ed countries to name a few.
    + **Customer Acquisition** \- Less "Risky" as it has low transaction volumes compared to Move Money but still needs consideration.
    + **Internet Banking** \- Covers a wide range of services with varying levels of risk, Move Money would be considered part of this.
* Conceivably a different approach could be taken for each depending on the risk, however, in the interests of keeping it simple, I am working towards a solution that would apply to some of the riskiest operations.

**TL;DR**: It is the responsibility of the management to ensure that adequate internal controls are in place that comply with the Securities and Exchange Commission's regulations.

Sarbanes Oxley 404 is normally satisfied through the completion of a Top-Down Risk Assessment, part of which will assess the risk of collusion and puts forward mitigation strategies.

Within a company which employs DevOps practice and culture, where Developers routinely have access to both source control and production how can Segregation of Duties be achieved, or more generally how can the risk of collusion be mitigated.

# Answer
Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how this is typically done/addressed in a mainframe environment, where the "engineers" (as in your question title) are actually groups of people were dozens (possibly hundreds) of people are involved. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).

---

### 1. Architecture

---

Here are the highlights of how I'd answer your question:

* All code (and related artifacts like executables, etc) is stored in files which, all together is what we call the ***library structure***.
* For each environment on each (possibly remote) target system, there is a server (a "started task" in mainframe speak), which takes care of ALL (repeat: ALL) updates to anything in the library structure. There are a few exceptions (like security people, or space management team), but apart from that, nobody (repeat: nobody) has authorization to apply updates to any file within that library structure. In other words: ***the server gets exclusive update authority to the entire library structure***. Attention: OPS-people will go bonkers if you walk in to limit their access (at first they are going to resist ...), so make sure you're covered by upper management (CxO) to impose those access rules ...
* The actual software changes my consist of a single component (a tiny code fix in the middle of the night ...), or it may also be hundreds or thousands of sources, executables, or whatever other artifacts (during a release weekend). To make them manageable, things that should be moved (more or less) together, at the same time, are bundled together in what's called a ***software change package***.

With the above in place, any kind of update to be applied by the server to the library structure, will only be possible via a well defined workflow, which we call the ***lifecycle of a software change package*** (SDLC if you prefer). To actually execute the various steps in that workflow, this is what it takes to make it happen:

* Only the server will execute the required (and preconfigured) steps.
* The server will only do a specific step (= update something somewhere in the library structure), after the required approvals (from human beings) have been gathered to perform such step.
* The approvals can only be given by users who have a **role** that allows them (= permission) to issue such approvals.

---

### 2. Roles and permissions

---

The server will ensure that the user trying to make something happen (like 'approve something') will only be able to do so, if the user's permissions are appropriate. That part is easy. But you don't want to use the SCM system to administer all those permissions for all the users involved, that's what belongs in your security system (not the SCM system!), so that you can adapt your workflow (in your SCM system) to go check those permissions whenever appropriate. The steps below provide some more details on that.

**Step 1: Configure the permissions (in the security system)**

* Define **security entities** in your security system, with well defined names for those entities. A few samples (add as many similar ones to fit your own needs):

+ `PrmUnit`, used for getting permission to request a *Promote* to say *Unit*-testing.
    + `PrmQA`, used for getting permission to request a *Promote* to say *Qa*-testing (let's assume this is the highest level of testing).
    + `PrdEnduser`, used by end-users involved in some level of testing, to indicate that they are satisfied by the results produced by some kind of testing. And because of that, those end-users agree with the change moving forward in the library structure.
    + `PrdRelmgnt`, used by release managers to authorize an *Activation* in production (= the last/highest level in the library structure).
* Define **groups of users** in your security system. A few samples (add as many similar ones to fit your own needs):

+ `GrpDevs`, which (say) corresponds to your developers (probably more then just 1).
    + `GrpEnduser`, which (say) corresponds to your end-users (at least 1, preferably with more similar users).
    + `GrpRelMgnt`, which (say) corresponds to your release managers (at least 1, preferably a few more users).
* **Grant permissions**, also using your security system, to allow access to selected "*security entities*" for selected "*groups of users*". To continue the example above, here is what seems appropriate (adapt to fit your own needs):

+ Group `GrpDevs` gets access to (only!) security entity `PrmUnit`.
    + Group `GrpEnduser` gets access to (only!) security entity `PrdEnduser`.
    + Group `GrpRelMgnt` gets access to (both!) security entity `PrmQA` and `PrdRelmgnt`.

**Step 2: Configure the workflow (in the SCM system)**

After the permissions are configured in your security system (as in Step 1), all that's left to do in your SCM system is to configure how the various steps in the lifecyle match with the related security entities in your security system. That is, only those users who have the appropriate access to the required security entity, are allowed to request the server to perform the corresponding step in the workflow.

Here are some examples of how you'd configure your SCM system to make some magic happen:

* If a user has access to `PrmUnit`, then such user is allowed to request a *Promote* to *Unit*-testing. Obviously, the users in group `GrpDevs` are the users authorized for this (note: not, e.g, the users in group `GrpRelMgnt`).
* If a user has access to `PrmQA`, then such user is allowed to request a *Promote* to *QA*-testing. Obviously, the users in group `GrpRelMgnt` are the users authorized for this (note: not, e.g, the users in group `GrpDevs`, or in group `GrpEnduser`).
* If a user has access to `PrdEnduser`, then such user is allowed to authorize the change moving forward in the library structure (which is typically a prereq for users in group `GrpRelMgnt` to even be able to review a change). Obviously, the users in group `GrpEnduser` are the (only) users authorized for this.
* If a user has access to `PrdRelmgnt`, then such user is allowed to authorize an *Activation* in production (= the last/highest level in the library structure).

---

### 3. Expect the unexpected, and be ready for it

---

The above is just a blueprint, which hopefully helps to understand how in the end it is the server that takes care of the segregation of duties ... provided you have the CxO cover you to impose some access rules that not everybody will like.

To complete the picture as explained above, the server creates an audit trail (logging) of anything that's happening in the system. So that at any point in time, it is always possible to answer questions like

> ***What happened when and why, and which authorized user actually approved it ... upfront?***

But, probably the toughest part is to have adequate reporting tools available (and know how to use them). At least to (easily) satisfy requests from IT auditors (their questions can be very challenging). But also to point to relevant log records in your SCM system to answer all sorts of "What happened"-questions in crisis situations where (part of) production is down.

---

PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.
> 8 votes

# Answer
Answer based on my knowledge of French "Internal controls" regulation, kind of equivalent to the SEC regulations you point to, I assume linking here to a French legal text would not be really useful and I know of no good translation of it.

In an ideal 'You build it, you run it' model, everyone in the team will be accountable for the change. The risk assessment can't be enforced by a separation of duties and the only way I know to keep conforming with regulation is to have a periodic short cycle audit of transactions along with an unalterable action tracking to get back to the person who did the release.
This mean all logs of transaction and actions are pushed to a restricted area the team doesn't have access to, a change in the what is logged "should" be catch by functional tests the team has no access to and at worse will be catch by the audit and tracked to its author.

This is not applicable to all products, at time of writing in France any company allowed to emit money (mainly banks), has to ensure every transaction will be recorded and thus can't take the risk of missing a transaction.
On the other hand, they have no legal obligation to track any commercial offer or risk evaluation when someone ask for a loan, and thus the products handling this customer selection and computing the fees which will be in the offer are easier to fit in a post-release audit model.

Main idea is that the release model has to be tweaked according to risk assessment obligations.

A related resource is the ISO27001 norm.
> 6 votes

# Answer
IMHO, Developers & Operations could be represented by nothing more than just two git repositories **for the same codebase**, with distinct permissions model each, so that teams will not interfer in the work of each other, at all.

Let's call them **Dev.mygithub.co** & **ops.mygithub.co**, just as an example.

The idea is, that Developers are free to create/branch/merge to their heart's content -git is providing full traceability and that is what matters here- meanwhile, at the moments that regulatory framework implies review effort a **Pull Request** can be raised, for the merging to happen in a controlled manner.

Taking that concept to its next level, a **develop** branch can be propagated towards the remote Ops' **production** via yet another **Pull Request** act. That last part has to happen by operations' hands and eyes, since they have a responsibility to vet it into production and they choose the level of review.

Such scheme allows *infinite flexibility, full traceability, ability to catch issues early on via a variety of processes, separation of concerns and a very reasonable user experience in the process*!

N.B. The model described above may be followed even if Ops & Dev overlap totally!
> 0 votes

# Answer
in order, topmost is most expensive:

* distinct dev and ops sites & methods to carry over work from one to the other =\> many sites
* distinct dev and ops systems & methods as above =\> 1 joint site
* distinct dev and ops git/vcs repositories & related methods =\> 1 system
* distinct dev and ops git/vcs branches (protected) & related methods =\> 1 repo, many branches

Depending on what you do, some solutions are better than others; for example, if you have the need to serve two teams with distinct roles within them and each having ownership within and provide full traceability, you are hovering over first three.

In short, anything that enforces that one guy or gal cannot take the ball all alone and run with it, as s/he crosses a distinct explicit boundary among dev & ops.

Now, depending on the subsequent level of risk, that boundary might be enforced or not. Choices, choices.
> 0 votes

---
Tags: sre, financial-services, operating-model
---


# Question
Title: How does someone share a drive to be available for my containers?
I have been trying to setup my ASP.Net Core project with docker support, but it seems like I am failing to share my `C:\` drive?

Following that link I have encountered these documentations stating that I should..

> allow connections to 10.0.75.1 on port 445 (the Windows host) from 10.0.75.2 (the virtual machine)

.. which I did, but I nevertheless ended up getting the same error message. So I have tried to follow some further instructions and tried to reinstall the File and Print sharing service on the Hyper-V virtual network card. Wich has lead me to another (Windows related) error;

*Apparently*, the latest Windows 10 release does not support/allow uninstalling `File and Printer Sharing for Microsoft Networks` but instead throws the `0x80071779 error`.

Thus there does not seem to be any "solution" to now share a drive on docker without any "hacky" solutions, no? Have I maybe missed something?

# Answer
> 0 votes
Interestingly I have solved that error by pressing on "Reset credentials" in `Docker -> Settings -> Shared Drives` and typing in the exact same account information I had before (admin account).

---
Tags: docker, windows
---


title: What is warm(ing) up?

text:
# Question
Title: What is warm(ing) up?
There are a few questions that mention something about ***warm up***, such as:

What is this ***warm up*** thing? Is it something similar to "being ready when some request comes in (e.g. to preload things in memory, to prevent some sort of I/O operations)"? If not what could be a good comparison to explain it to a novice?

# Answer
All container services and Function-As-a-Service services of the big cloud providers, such as Amazon ECS, Lambda and Azure cloud functions run on actual virtual machines of these providers. Normally they only take a small number of milliseconds to start up, but when your container or function has not run for a certain amount of time, they will take a couple of seconds to actually start up the virtualization environment for your container or function. These startup times improve all the time, but you still get better startup time results by calling your functions or a request in your container in regular intervals.

A typical interval could be every minute, so that you can be sure that the virtual environment the container or function runs on has likely not been shut down, as it just was in use in the past 60 seconds.

In relation to an auto scaling group, I would say that "warm up" is a bit of a mis-nomer. Either there are enough instances provisioned to handle the traffic you expect, or you may need to make a 'scheduled change' to the number of instances, but there is actually nothing to warm the instances up.

One edge-case would be if instances have disk contents restored from backup. On AWS, there is some delay in getting the contents on these disk from snapshot and the disk can be warmed up by reading all blocks on it
> 2 votes

# Answer
In the Google App Engine PaaS context, standard environment apps using automatic scaling can use warmup requests to keep request latency/response time down even during traffic peaks requiring new instances to be started. From Warmup requests:

> Warmup requests are a specific type of loading request that load application code into an instance ahead of time, before any live requests are made. Manual or basic scaling instances do not receive an `/_ah/warmup` request.
>
> To learn more about how to use warmup requests, see Configuring warmup requests.

Without warmup requests configured instances would be started only in response to actual/live requests which are already routed to the new instance (which isn't yet ready). Meaning that the response time for those requests would be increased with the actual instance startup time.

With warmup requests configured the live requests keep being handled by instances already running while the new instances are being started (by the warmup requests). Only after the new instance is up and running (indicated by successful handling the warmup request) the scheduler will start routing live requests to it.
> 2 votes

---
Tags: terminology
---


title: Fully enabled End to End CI/CD Solution

text:
# Question
Title: Fully enabled End to End CI/CD Solution
I've been working in the field of DevOps for some time now and have learned about some tools like Jenkins, Docker, Kubernetes, Puppet, Artifactory, etc.. Although I know where each of these tools could be used and what purpose each of these serves, I really wanted to see how a fully implemented CI/CD looks like.

I understand that the solution would definitely change with the change in the requirements, technology stack, team size, etc. and that there is not a perfect solution for everything but, I haven't seen a fully implemented DevOps lifecycle ever.

I'm not sure if asking for resources is prohibited here (Feel free to close the question if it is) but could you please redirect me to some article, blog, white paper or suggest me a book where I could learn and get a feel of what fully implemented CI/CD pipeline looks like.

# Answer
> 2 votes
> could you please redirect me to some article, blog, white paper or suggest me a book where I could learn and get a feel of what fully implemented CI/CD looks like.

Sure, look how Google does it.

> I haven't seen a fully implemented DevOps lifecycle ever.

There seems to be a misunderstanding here. "DevOps" is a lot of things; but a "full" CI/CD (meaning each individual developer commit turning up in production fully automated shortly after the commit - continuous *deployment* (not only *delivery*)) is by far not the norm. In my experience with a few larger local and national customers, where production or business critical things are controlled by the software, and there usually are large amounts of critical data involved, it *always* stops before the production stage.

Full CI/CD is complicated. Yes, you can cobble the tooling together pretty quickly, but the prevention, detecting and backing out of unwanted commits is very hard. If you look at the above article about Google you will see that huge effort went into getting this to run (I seem to remember only their core DevOps team is about 100 persons, doing nothing than tooling; and then they invest a lot of work into peer reviews, knowledge transfer, etc. etc.). Whole ecosystems of components, procedures and mindsets were created specifically for this. The fact that we mere mortals get to play with things like Kubernetes is a *result* from that, their breadcrumbs if you will.

Even only the one fact that they are doing Trunk-based development would be too complicated (regarding processes, necessary quality requirements etc.) for many more normal, smaller teams.

# Answer
> 1 votes
We currently use the Jenkins as our CI/CD server and we use docker in every stage of our CI/CD.

Every time a change is pushed to a SCM, it is automatically built and deployed to production(cloud).

The Jenkins library feature helps to keep all generic steps in a centralized library and reused by every pipeline.

You can draw some inspiration from SAP S/4HANA cloud Sdk out of the box CI/CD. https://blogs.sap.com/2017/09/20/continuous-integration-and-delivery/

---
Tags: continuous-integration, continuous-deployment, continuous-delivery
---


# Question
Title: Docker for Windows 7 with Visual Studio 2017 Integration
My objective is to have a development environment with docker and Kubernetes integrated with Visual Studio 2017 so that I can run the app on the docker container from directly within Visual Studio. I'm on a Windows 7 OS.

My research has taken me thus far:

1. Docker for Windows only supports Windows 10+. As an alternative, I can use Docker Toolbox which setups a Virtual Machine via Virtual Box and runs a Linux VM (boot2docker). Docker actually runs in this Linux VM and shares its executable with Windows 7 via environment variables.
2. Kitematic comes with Docker Toolbox and is essentially a GUI tool for managing my docker images.

Here are the questions, I'm trying to get answered and the issues I'm experiencing:

1. When I launch Kitematic, it says "Couldn't find a native setup" for docker. When I try to install Docker for Windows, clicking on the executable does nothing. **Is it impossible to install Docker natively on Windows 7 without a Virtual Machine in the middle**?
2. In using the Boot2Docker VM, it is a Linux distro and consequently, I can only use .NET core and ASP.NET core. **If I can't get a native Docker instance, is it possible to have a Boot2Docker Windows machine so I can utilize all of the .NET framework on it?**
3. I was finally able to build and run the docker image from Git Bash utilizing the Boot2Docker docker machine, but when I try to integrate with Visual Studio I get this error: `C:\Users\JoeTest\.nuget\packages\microsoft.visualstudio.azure.containers.tools.targets\1.0.1916590\build\Container.targets(222,5): error CTC1003: Visual Studio container tools require Docker to be running.`

I've found various solutions that range from PowerShell to ensuring the C:/Users/ folder was shared between the Virtual Box image and Windows 7 host, but nothing seems to work for the VS Studio integration. Here are the steps to reproduce:

1. Created project in Visual Studio 2017 for ASP.NET core on Windows 7
2. Installed and Validated Docker Toolbox, e.g. can execute docker run hello-world
3. Closed the project in VS Studio and setup the following in Powershell:

`docker-machine env
$Env:DOCKER_TLS_VERIFY = "1"
$Env:DOCKER_HOST = "tcp://192.168.99.100:2376"
$Env:DOCKER_CERT_PATH = "C\Users\JoeTest\.docker\machine\machines\default"
$Env:DOCKER_MACHINE_NAME = "default"
$Env:COMPOSE_CONVERT_WINDOWS_PATHS = "true"
/# Run this command to configure your shell:
/# & "C:\Program Files\Docker Toolbox\docker-machine.exe" env | Invoke-Expression
PS C:\Users\JoeTest> & "C:\Program Files\Docker Toolbox\docker-machine.exe" env | Invoke-Expression`

`invoke-item C:\Users\JoeTest\source\repos\VSEWebAppSample\VSEWebAppSample.sln`

This opens the Visual Studio Project. 4. Then, I type Ctrl + F5 and get the following output:

```
1>------ Build started: Project: VSEWebAppSample, Configuration: Debug Any CPU ------
1>VSEWebAppSample -> C:\Users\JoeTest\source\repos\VSEWebAppSample\VSEWebAppSample\bin\Debug\netcoreapp2.1\VSEWebAppSample.dll
1>VSEWebAppSample -> C:\Users\JoeTest\source\repos\VSEWebAppSample\VSEWebAppSample\bin\Debug\netcoreapp2.1\VSEWebAppSample.Views.dll
1>C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NonInteractive -NoProfile -WindowStyle Hidden -ExecutionPolicy RemoteSigned -File "C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1" -Version vs2017u5 -RuntimeID debian.8-x64 -InstallPath "C:\Users\JoeTest\vsdbg\vs2017u5"
1>Info: Using vsdbg version '15.7.20425.2'
1>Info: Using Runtime ID 'linux-x64'
1>Info: C:\Users\JoeTest\vsdbg\vs2017u5 exists, deleting.
1>Add-Type : Cannot add type. The assembly 'System.IO.Compression.FileSystem' cou
1>ld not be found.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:47 char:13
1>+     Add-Type <<<<  -assembly "System.IO.Compression.FileSystem"
1>    + CategoryInfo          : ObjectNotFound: (System.IO.Compression.FileSyste
1>   m:String) [Add-Type], Exception
1>    + FullyQualifiedErrorId : ASSEMBLY_NOT_FOUND,Microsoft.PowerShell.Commands
1>   .AddTypeCommand
1>
1>Add-Type : Cannot add type. One or more required assemblies are missing.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:47 char:13
1>+     Add-Type <<<<  -assembly "System.IO.Compression.FileSystem"
1>    + CategoryInfo          : InvalidData: (:) [Add-Type], InvalidOperationExc
1>   eption
1>    + FullyQualifiedErrorId : ASSEMBLY_LOAD_ERRORS,Microsoft.PowerShell.Comman
1>   ds.AddTypeCommand
1>
1>Add-Type : Cannot add type. The assembly 'System.IO.Compression' could not be f
1>ound.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:48 char:13
1>+     Add-Type <<<<  -assembly "System.IO.Compression"
1>    + CategoryInfo          : ObjectNotFound: (System.IO.Compression:String) [
1>   Add-Type], Exception
1>    + FullyQualifiedErrorId : ASSEMBLY_NOT_FOUND,Microsoft.PowerShell.Commands
1>   .AddTypeCommand
1>
1>Add-Type : Cannot add type. One or more required assemblies are missing.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:48 char:13
1>+     Add-Type <<<<  -assembly "System.IO.Compression"
1>    + CategoryInfo          : InvalidData: (:) [Add-Type], InvalidOperationExc
1>   eption
1>    + FullyQualifiedErrorId : ASSEMBLY_LOAD_ERRORS,Microsoft.PowerShell.Comman
1>   ds.AddTypeCommand
1>
1>New-Object : Cannot find type [System.IO.Compression.ZipArchive]: make sure the
1> assembly containing this type is loaded.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:64 char:29
1>+     $zipArchive = New-Object <<<<  System.IO.Compression.ZipArchive -Argument
1>List $zipStream
1>    + CategoryInfo          : InvalidType: (:) [New-Object], PSArgumentExcepti
1>   on
1>    + FullyQualifiedErrorId : TypeNotFound,Microsoft.PowerShell.Commands.NewOb
1>   jectCommand
1>
1>Unable to find type [System.IO.Compression.ZipFileExtensions]: make sure that t
1>he assembly containing this type is loaded.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:65 char:46
1>+     [System.IO.Compression.ZipFileExtensions] <<<< ::ExtractToDirectory($zipA
1>rchive, $targetLocation)
1>    + CategoryInfo          : InvalidOperation: (System.IO.Compression.ZipFile
1>   Extensions:String) [], RuntimeException
1>    + FullyQualifiedErrorId : TypeNotFound
1>
1>You cannot call a method on a null-valued expression.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:66 char:24
1>+     $zipArchive.Dispose <<<< ()
1>    + CategoryInfo          : InvalidOperation: (Dispose:String) [], RuntimeEx
1>   ception
1>    + FullyQualifiedErrorId : InvokeMethodOnNull
1>
1>Out-File : Could not find a part of the path 'C:\Users\JoeTest\vsdbg\vs2017u5\s
1>uccess_rid.txt'.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:97 char:26
1>+     $runtimeId | Out-File <<<<  -Encoding utf8 $SuccessRidFile
1>    + CategoryInfo          : OpenError: (:) [Out-File], DirectoryNotFoundExce
1>   ption
1>    + FullyQualifiedErrorId : FileOpenFailure,Microsoft.PowerShell.Commands.Ou
1>   tFileCommand
1>
1>Out-File : Could not find a part of the path 'C:\Users\JoeTest\vsdbg\vs2017u5\s
1>uccess_version.txt'.
1>At C:\Users\JoeTest\AppData\Local\Temp\GetVsDbg.ps1:100 char:24
1>+     $version | Out-File <<<<  -Encoding utf8 $SuccessVersionFile
1>    + CategoryInfo          : OpenError: (:) [Out-File], DirectoryNotFoundExce
1>   ption
1>    + FullyQualifiedErrorId : FileOpenFailure,Microsoft.PowerShell.Commands.Ou
1>   tFileCommand
1>
1>Info: Successfully installed vsdbg at 'C:\Users\JoeTest\vsdbg\vs2017u5'
1>docker build -f "C:\Users\JoeTest\source\repos\VSEWebAppSample\VSEWebAppSample\Dockerfile" -t vsewebappsample:dev --target base "C:\Users\JoeTest\source\repos\VSEWebAppSample"
1>Sending build context to Docker daemon  2.471MB
1>
1>Step 1/4 : FROM microsoft/dotnet:2.1-aspnetcore-runtime AS base
1> ---> 40d759655ea3
1>Step 2/4 : WORKDIR /app
1> ---> Using cache
1> ---> f82084d08a44
1>Step 3/4 : EXPOSE 49718
1> ---> Using cache
1> ---> 22d65a07b772
1>Step 4/4 : EXPOSE 44315
1> ---> Using cache
1> ---> 1a6aecd0e9f1
1>Successfully built 1a6aecd0e9f1
1>Successfully tagged vsewebappsample:dev
1> C:\Users\JoeTest\.nuget\packages\microsoft.visualstudio.azure.containers.tools.targets\1.0.1916590\build\Container.targets(222,5): error CTC1003: Visual Studio container tools require Docker to be running.

```

One article I found said I needed to update my PowerShell version but I went to this link and was confused which option I should download. Any help is much appreciated.

# Answer
If you are on win 7, you cannot use the current "docker for windows", which only works for win10 pro and server 2016.

Your only solution is to use the Docker Toolbox https://docs.docker.com/toolbox/overview/. Incidentally you must use it with Oracle's VirtualBox. But you don't need any virtual machine to make this work.
> 1 votes

---
Tags: docker, docker-machine
---


# Question
Title: Understanding and tweaking AWS policies for job functions
There are quite many possibilities to configure AWS policies; Amazon has therefore provided policies for job functions.

Now is it anyhow possible to sneak into JSON code of these policies to learn in detauls what they do and use them as foundation for own policies along their lines?

# Answer
> 1 votes
Sure you can have a look at the JSON policy documents. On that page that you link to (AWS Managed Policies for Job Functions) click the Policy you're interested in *while being logged to your AWS account* and it will take you to the policy definition.

E.g. if you click to **AdministratorAccess** it will open the *IAM Policies* page and show you the JSON:

Hope that helps :)

# Answer
> 0 votes
Job function related policies are visible among all others and their JSON can be shown there as well.

https://console.aws.amazon.com/iam/home#/policies

---
Tags: amazon-web-services, security
---


# Question
Title: Deploying lambda conceiving sensitive variables using serverless
I want to deploy AWS lambda written in NODEJS. And I have sensitive variables that can't be disclosed in AWS lambda variables section. Is there a way to deploy lambda with an encrypted variable which no one can see and can't be decrypted from the machine which was used to deploy the lambda. Only some other user who has the key should be able to decrypt the variable.

# Answer
As far as I'm aware. There is no way to encrypt the environment variable value in the variables section of the lambda function. And to be fair, even if there was, it wouldn't provide any benefit in terms of keeping the variable secret. If the variable could be hidden/encrypted in the variables sections to be decrypted later, a user could simply execute the lambda function and print the results of the decrypted variable to stdout.

One approach could be to use aws secrets manager to store and retrieve your secrets. You can then assign your lambda function a role to allow access to the secret store, and use AWS SDK to retrieve your value.
> 1 votes

---
Tags: serverless
---


# Question
Title: Are there any truly simple artifact repositories?
I'm looking for a binary-artifact repository, but everything I see is super complicated, to set up, use, and manage. All I'm looking for is:

* upload repos as tgz/zip with metadata (name, date, tag)
* query and download by SHA and/or name
* a local tool (python-based?) to help create repos from a subdir with the proper metadata and signature

Even a web-based portal is almost more than what I need (sure, it would be nice.) Doesn't seem like it should require all the infrastructure and complexity of archiva, artifactory, etc. Or maybe some of these tools have a simple mode and I just can't find it?

# Answer
> 1 votes
I didn't find anything that was exactly what I was looking for; simple zip/tar and even dpkg didn't have the traceability I was hoping for in a binary-artifact system, so I wrote my own: see https://github.com/garyo/binary-artifact (MIT licensed).

It's basically a glorified tar/zip creator that creates a manifest with a content signature and a well-defined file name that includes the content sig, so the artifact can be specified in a build file (e.g. Ansible/Puppet/Chef/SCons) in a repeatable, traceable way.

This way the artifacts can live anywhere the build system can find them (Google drive, Dropbox, local NAS, whatever), download them knowing exactly which version is being downloaded (thanks to the specific name), and unpack and use them as usual. This tool makes no assumptions about where the artifacts get stored.

# Answer
> 1 votes
You can always fall back to scp, sftp or the like, i.e. command-line based upload/download. Tags can be done with symlinks, and you don't get advanced metadata, but still.

# Answer
> 0 votes
I generaly work with a CDN, you can upload files and compressed files, and many CDNs has api , like it, but exist many other not conventional solutions, by example insert code or script in a fake image using base 64 and upload it to server public , or use facebook pixel to add script or file. Hope this help you.

---
Tags: artifacts, repository
---


# Question
Title: Tomcat deploy war with Ant - error writing to server
I am using Ant to deploy a war to a Tomcat server, but it gives me the following error:

```
    BUILD FAILED
    /home/deploy/Documents/Work/Deploy_Wars/xrepo/deployment/build.xml:37: The following error occurred while executing this line:
    /home/deploy/Documents/Work/Deploy_Wars/xrepo/deployment/ant_scripts/deploy_wars.xml:36: The following error occurred while executing this line:
    /home/deploy/Documents/Work/Deploy_Wars/xrepo/deployment/ant_scripts/deploy_wars.xml:42: java.io.IOException: Error writing to server
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:699)
        at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:711)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1585)
        at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492)
        at org.apache.catalina.ant.AbstractCatalinaTask.execute(AbstractCatalinaTask.java:223)
        at org.apache.catalina.ant.DeployTask.execute(DeployTask.java:201)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.apache.tools.ant.dispatch.DispatchUtils.execute(DispatchUtils.java:106)
        at org.apache.tools.ant.Task.perform(Task.java:350)
        at org.apache.tools.ant.Target.execute(Target.java:448)
        at org.apache.tools.ant.Target.performTasks(Target.java:469)
        at org.apache.tools.ant.Project.executeSortedTargets(Project.java:1399)
        at org.apache.tools.ant.helper.SingleCheckExecutor.executeTargets(SingleCheckExecutor.java:36)
        at org.apache.tools.ant.Project.executeTargets(Project.java:1260)
        at org.apache.tools.ant.taskdefs.Ant.execute(Ant.java:446)
        at org.apache.tools.ant.taskdefs.CallTarget.execute(CallTarget.java:105)
        at org.apache.tools.ant.UnknownElement.execute(UnknownElement.java:292)
        at sun.reflect.GeneratedMethodAccessor4.invoke(Unknown Source)

```

Here is the Ant target that I use for this deployment (and here is the line where the exception is thrown):

```
    <target name="deploy-debug" description="Install DEBUG">
        <deploy url="http://debug.webiste.ro:8180/manager" username="admin" password="password"
                path="/debug-war" war="wars/debug-war.war"/>
    </target>

```

I am using Tomcat 9.0.10 and Ant 1.10.3 on Ubuntu 18.04.

I saw some people suggesting that the tomcat user is not properly defined in tomcat\_users.xml. I have modified it, but it still gives me the exception when I try to deploy. I can succsefully login on the Tomcat manager app in the browser. Here is my tomcat\_users.xml:

```
    <tomcat-users xmlns="http://tomcat.apache.org/xml"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
          xsi:schemaLocation="http://tomcat.apache.org/xml tomcat-users.xsd"
          version="1.0">
          <role rolename="tomcat"/>
          <role rolename="manager-script"/>
          <role rolename="manager-gui"/>
          <role rolename="admin-gui"/>
          <role rolename="admin-script"/>
          <user username="admin" password="password" roles="manager-gui,manager-script,admin-gui,admin-script"/>
    </tomcat-users>

```

The war file that I am trying to deploy has 80 Mb which is bigger than the default maximum upload value of Tomcat (50 Mb). I have tried increasing the maximum upload value to 100 Mb but it still gives me the Exception.

Any ideas on this problem?

# Answer
> 1 votes
It could be disk space, or hot deploy. For hot deploy, you can put the following parameters:

```
...
<target name="tomcat-stop">
    <exec executable="${server.home}/bin/catalina.bat">
        <arg value="stop"/>
    </exec>
</target>

<target name="tomcat-start">
    <exec executable="${server.home}/bin/startup.bat">
        <arg value="start"/>
    </exec>
</target>
...
<target name="all" depends="tomcat-stop,clean,init,compile,junit-slow,make_war,deploy,tomcat-start"></target>

```

The answer is at this link: https://stackoverflow.com/a/32482795/8442153

---
Tags: deployment, tomcat, ant
---


title: vagrant box tests in gitlab-ci

text:
# Question
Title: vagrant box tests in gitlab-ci
I have a vagrant box, provisioned with Ansible that needs testing on github pull requests. Is it possible to setup CI test suite for the mirror in gitlab-ci without extra expenses on droplets? Or maybe this is something that could be possible with upcoming `automation` on github? Big fan of `travis` but looks like they not going to support virtualbox deployment in their containers anytime soon.

# Answer
You can do via docker, since it's a test. Here is an example repository, both docker and vagrant. https://github.com/jonashackt/gitlab-ci-stack https://blog.codecentric.de/en/2018/05/gitlab-ci-pipeline/
> 2 votes

---
Tags: ansible, gitlab, github, vagrant, virtualbox
---


# Question
Title: Open source private cloud VM provisioning for container workload
There is a new generation of tooling developed that tries to integrate different IaaS providers (Azure/AWS/GCE/DigitalOcean) into one "infrastructure codebase". It's dubbed "Cloud Native" and it's supposed to result in "one tool to rule them all". There's Terraform from HashiCorp, InfraKit from Docker, Juju from Canonical, independent Foreman, Pulumi and others.

It's easy to get lost. Which of these tools is best suited to IaaS on premises? Let's suppose you have a bunch of metal servers and you want to provision some VMs there. Let's suppose you don't want VirtualBox/VMWare/ESX/Xen, you want KVM. And on top of that an obvious choice is libvirt. To manage multiple hypervisors we need to connect to the libvirt daemon remotely using connection URI. For KVM we can use qemu+ssh. Let's suppose we set those up using PXE boot. Now we have a bunch of `qemu+ssh://` links along with ssh access.

Now we need a central component on top of that to manage resources and some tool that will provision virtual hosts according to some declarative specification.

On top of those VMs I want to run containers, so automatic setup of container orchestration cluster is desired. In addition, some way of attaching storage is desired. At least directly attached block devices or fs mounts.

Any suggestions?

Note: Found a similar question on SO, but for Xen.

Clarification: "best suited" needs metrics. So these are the metrics I'm looking to optimize:

* Zero closed source LOC
* Number of tools if not "All-in-one"
* UI actions to setup 3 master + 5 workers k8s/swarm cluster
    + number of commands (for CLI tool)
    + number of clicks (for Web based/GUI tool)
    + length/size of the config file
* UI actions to install the tool
* UI actions to maintain configuration (preferably none, self-healing infrastructure)

# Answer
> 3 votes
"best suited" is the question. How do you measure this? what are the metrics and why do you choose them? I think you are looking for some kind of "tools combination", earlier tested on production by someone else's if preferred. What you want to do is try some opensource/free private cloud on-premises and run IaC on this private environment. You can test tools forever. How do you compare those tools? I believe all of them works fine as long as there is not any inconsistency problem. I think there is a missing/main part in your search when it comes to IaC. Because of few options for that missing angle as well. I think it is better to create a chart for all of us as you are going this process in order to benefit people here. Do you have a working environment? If so could you please detail all the tools you used to create this environment?

Some vendors run open source projects. How about them?

The Cloud Native Computing Foundation seeks to drive adoption of this paradigm by fostering and sustaining an ecosystem of open source, vendor-neutral projects.CNCF Cloud Native Definition v1.0

I started to wonder if this is cloud-native then what is it that is not cloud-native?

I think this is what you are looking for. The distinction between provider and provisioner is blurred. A workflow automation tool should be doing what cloud orchestrator with GUI does. Terraform makes a distinction between provider and provisioner but Cloudify removes them as I see it. For an architect designing the whole workflow with GUI might be useful to reach the desired state. Writing desired state for which you do DRY tests already beforehand or showing with graphs. There is something you know already where it goes since you already tested, seeing it flowing in graphics workflow makes sense? Then go for it. To me it is painting the history I do not see the benefit.

There are many ways to break down the types of automation, whether it’s imperative versus declarative, or orchestration versus desired configuration state.

Most orchestrators are fully CLI based, or offer very limited GUI operations. A distinct advantage here is that Cloudify enables you to one-click your way through architectural relationship diagrams. See: Terraform vs. Cloudify.

### Providers

A provider is responsible for understanding API interactions and exposing resources. Providers generally are an IaaS (e.g. AWS, GCP, Microsoft Azure, OpenStack), PaaS (e.g. Heroku), or SaaS services (e.g. Terraform Enterprise, DNSimple, CloudFlare). See: Providers.

### Provisioners

Provisioners are used to execute scripts on a local or remote machine as part of resource creation or destruction. Provisioners can be used to bootstrap a resource, cleanup before destroy, run configuration management, etc. See: Provisioners.

# Answer
> 1 votes
Here's what I've tried:

## Terraform + libvirt

There's unofficial libvirt provider for terraform. It works, but there are a few gotchas:

## Docker machine

Docker machine has unofficial kvm driver.

---
Tags: infrastructure-as-code, virtualization, provisioning, infrastructure
---


# Question
Title: Parallel stages running on same agent in Jenkins declarative pipeline
I've been using a Jenkins pipeline with a parallel step for testing like this:

```
pipeline {
    agent { label 'php' }
    stages {
        stage('build') {
            steps {
                ...
            }
        }
        stage('test') {
            parallel {
                stage('unit tests') {
                    steps {
                        ...
                    }
                }
                stage('integration tests') {
                    steps {
                        ...
                    }
                }
            }
        }
        stage('deploy') {
            steps {
                ...
            }
        }
    }
}

```

The pipeline runs fine on a single agent, but now I want to add a step with manual input, which should not keep the agent running. The agents are cloud servers that get started and stopped on demand.

To do so, I assigned "none" agent to the pipeline and the actual agent to all stages but the one requesting input:

```
pipeline {
    agent none
    stages {
        stage('build') {
            agent { label 'php' }
            steps {
                ...
            }
        }
        stage('test') {
            parallel {
                stage('unit tests') {
                    agent { label 'php' }
                    steps {
                        ...
                    }
                }
                stage('integration tests') {
                    agent { label 'php' }
                    steps {
                        ...
                    }
                }
            }
        }
        stage('deploy staging') {
            agent { label 'php' }
            steps {
                ...
            }
        }
        stage('confirm release') {
            steps {
                input { message 'release?' }
            }
        }
        stage('deploy production') {
            agent { label 'php' }
            steps {
                ...
            }
        }
    }
}

```

Artifacts that are needed in a following stage are carried over with `stash` and `unstash`. This works, but with two nuisances:

1. Stashing and unstashing between each single stage causes overhead. I would rather have all stages until the manual input executed in a single workspace. But `agent` is only allowed in stages that contain steps.
2. Each parallel stage blocks a single agent. So if there is only one free agent, there is no parallel execution, unlike before.

I can live with (1) if it's not possible otherwise, but parallel execution on one agent is important for me. Is it possible somehow in my current setup? If not, what are my options to achieve something similar?

# Answer
I modified your Jenkins file as shown below.

```
pipeline {
agent { label 'agent01'}
stages {
    stage('build') {
        steps {
            echo "build"
            sh "ls -lrt && touch build.txt"
        }
    }
    stage('test') {
        parallel {
            stage('unit tests') {
                steps {
                    echo "unit tests"
                    sh "ls -lrt && touch ut.txt"
                    sleep(time:10,unit:"SECONDS")
                }
            }
            stage('integration tests') {
                steps {
                    echo "integration tests"
                    sh "ls -lrt && touch it.txt"
                    sleep(time:10,unit:"SECONDS")
                }
            }
        }
    }
    stage('deploy staging') {
        agent { label 'master' }
        steps {
            echo "deploy staging"
            sh "ls -lrt && touch dep.txt"
        }
    }
    stage('confirm release') {
        agent { label 'master' }
        steps {
            echo "RELEASE?"
            sh "ls -lrt && touch release.txt"
        }
    }
    stage('deploy production') {
        steps {
            echo "deploy production"
            sh "ls -lrt && touch produ.txt"
        }
    }
}
}

```
1. You need not stash and unstash files since it runs on same node and workspace.
2. Use a general node for all the stages except for the one that needs a user input.

The output is

```
    Running in Durability level: MAX_SURVIVABILITY
[Pipeline] node
Running on agent01 in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] stage
[Pipeline] { (build)
[Pipeline] echo
build
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
+ touch build.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (test)
[Pipeline] parallel
[Pipeline]     [unit tests] { (Branch: unit tests)
[Pipeline]     [integration tests] { (Branch: integration tests)
[Pipeline]     [unit tests] stage
[Pipeline]     [unit tests] { (unit tests)
[Pipeline]     [integration tests] stage
[Pipeline]     [integration tests] { (integration tests)
[Pipeline]     [unit tests] echo
[unit tests] unit tests
[Pipeline]     [unit tests] sh
[unit tests]     [sandbox-pipeline] Running shell script
[Pipeline]     [integration tests] echo
[integration tests] integration tests
[Pipeline]     [integration tests] sh
[unit tests] + ls -lrt
[unit tests] total 0
[unit tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
[unit tests] + touch ut.txt
[integration tests]     [sandbox-pipeline] Running shell script
[Pipeline]     [unit tests] sleep
[integration tests] + ls -lrt
[integration tests] total 0
[integration tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
[integration tests] -rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 ut.txt
[integration tests] + touch it.txt
[unit tests] Sleeping for 10 sec
[Pipeline]     [integration tests] sleep
[integration tests] Sleeping for 10 sec
[Pipeline]     [unit tests] }
[Pipeline]     [unit tests] // stage
[Pipeline]     [unit tests] }
[Pipeline]     [integration tests] }
[Pipeline]     [integration tests] // stage
[Pipeline]     [integration tests] }
[Pipeline] // parallel
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy staging)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] echo
deploy staging
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
+ touch dep.txt
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (confirm release)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/sandbox-pipeline
[Pipeline] {
[Pipeline] echo
RELEASE?
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
-rw-r--r-- 1 jenkins 115 0 Oct 19 13:58 dep.txt
+ touch release.txt
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (deploy production)
[Pipeline] echo
deploy production
[Pipeline] sh
[sandbox-pipeline] Running shell script
+ ls -lrt
total 0
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 build.txt
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 ut.txt
-rw-r--r-- 1 jenkins docker 0 Oct 19 13:53 it.txt
+ touch produ.txt
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
Finished: SUCCESS

```
> 1 votes

---
Tags: jenkins, jenkins-pipeline, deployment
---


title: Cannot pull ansible docker image

text:
# Question
Title: Cannot pull ansible docker image
```
$ docker pull ansible/ansible
Using default tag: latest
Error response from daemon: manifest for ansible/ansible:latest not found

```

I try and get started with ansible but I get the above error message trying to pull the Docker image. Is it broken or am I doing something wrong?

# Answer
> 4 votes
If run you `docker pull <image>`, you are implicitly asking for a tag named `latest`. There is no such tag available from the Ansible repository. You can see a list of available tags at https://hub.docker.com/r/ansible/ansible/tags/.

For example, if you wanted the `fedora27py3` version of the image, you would run:

```
docker pull ansible/ansible:fedora27py3

```

I note that the most recent of thoses images is 7 months old. If you're experimenting iwth Ansible you might want to just install it into a Python virtual environment.

---
Tags: ansible
---


# Question
Title: Zero Downtime Deployment with Database Backward Compatibility
**Question**

Does anyone have experience with Database Backward Compatibility to achieve Zero Downtime Deployment? Are there good links or references that I can read?

**Reason**

I am working with a monolithic application where the code base is tightly coupled with the database. Because of this, when we want to do deployment (weekly), we have to work after business hours.

**Goal**

What I am trying to achieve is the ability to implement blue-green deployment where the old code is compatible with the new version of the database - and vice versa.

# Answer
> 4 votes
There is plenty of information about this. The basic idea goes like so -

Create a script that does the database migration, but make sure that it only adds tables and fields - never removes, renames, or changes the type of existing tables or fields.

The new version of the application will start using the new tables/fields when it is an existing field from the previous version that needed a change - the app must continue the previous version as it used to, and start using the new version as well.

In some cases, before the newly created field can be used it needs to be populated based on the previous field - the application can do this when it first starts.

If some functionality depends on the new field, and it is not yet ready, opening it to users can be hidden behind a feature flag that is toggled when ready to expose the new functionality.

When yet another version is released, the application can finally stop writing to the old field you were using and start using the new field exclusively.

The above mostly explains what happens when fields need to be renamed or type changed. For new tables or fields, it is much simpler since the existing version is not using these.

---
Tags: databases
---


# Question
Title: post jenkins build console output to slack
I know how to notify build status using the slack notification plugin. I can see`project - #buildnumber Success after 1 min 18 sec (Open)` in my slack channel. But I want the console output to be sent to slack as well (upon failure) instead of a link. I am using free style project since I need some scripts to build my project. The purpose is that I want engineers to know the result of their build without granting them Jenkins access.

## edit:

if console log can be uploaded as a text, would be also good. Other forms of 'output' would be welcome as well.

## new edit:

I found the answer by myself.

# Answer
I figured it out by myself, there is no plugin needed.

`build.sh &> /dev/stdout | tee jenkins.txt` and then `curl -F file=@jenkins.txt -F channels=yourchannel -H "Authorization: Bearer your token" https://slack.com/api/files.upload`

There is no '#' needed for slack channel and channel should not be private.

And if you use `buils.sh | tee jenkins.log` then you can't get the log from stderr.
> 5 votes

# Answer
You probably want the console output to be posted as an attachment, not a message, to slack. The post to slack plugin might be what you are looking for.

The other thing you could do is create a part of the freestyle job that strips out what you want to a text file, and add that as part of the slack message attachment, in the slack plugin configuration.
> 2 votes

# Answer
Posting the full stack trace tends to clog up the channel. Here are a few things you can do:

* Send build status along with the Jenkins Job link to the Channel so Devs can come back to the link for debugging
* Send stack trace as a file
* Send build status as and link to channel and Stack trace info to the user who committed the code for build

Either way, Post to Slack plugin along with other plugins will help you address this need.
> 1 votes

---
Tags: jenkins, jenkins-plugins, slack
---


# Question
Title: Using Red Hat Docker image as base image
I am considering using https://access.redhat.com/containers/#/registry.access.redhat.com/rhel7 as a base image for my Dockerfile instead of getting some random image in DockerHub. The main reason is the vetting process the images undergo.

Once any additional dependencies are installed, the resulting image will be pushed to a private Docker Hub repository. Is this a valid process or is there a disadvantage that I've missed?

# Answer
As long as you have access to the `FROM` image registry, you are totally free to choose where to pull from and where to push to.
> 0 votes

---
Tags: docker, dockerfile, dockerhub
---


# Question
Title: Interference of Docker CMD with su: works with su-exec but not with su?
In this bash script under Linux Alpine they have (line 8):

```
exec su-exec "$ZOO_USER" "$0" "$@"

```

As far as I know, `su-exec` is however an Apache httpd dependecy \- can I avoid it installing it under Ubuntu and use some more plain alternative?

I have tested so far:

```
 su -c "$0 $@" $ZOO_USER

```

But then the call through Docker `CMD` shows that I might have some sort of escaping/quotes error,

```
CMD ["zkServer.sh", "start-foreground"]

```

resulting in the following output; obviously there is a misplacement for the username field in the command sequence.

```
No passwd entry for user 'start-foreground'

```

But what is then the difference to the original (su-exec), which works fine? And how to get it right with `su`?

====

UPD: su-exec is here a third-parthy Alpine Linux package https://github.com/ncopa/su-exec

# Answer
# wrong argument order to `su`

`$ZOO_USER` needs to be the first argument to `su` with the `"$@"` coming afterwards.

```
su -c "$0" "$ZOO_USER" "$@"

```

It is ok for the options to come before the username. It is a good idea to keep them in double quotes so that there aren't issues if someone accidentally puts spaces into it.

Your error message reinforces that this is an order of arguments problem.

# even better

Based on comments this would be even better:

```
su -c "$0" "$ZOO_USER" -- "$@"

```

## su man page excerpt

> Additional arguments may be provided after the username, in which case they are supplied to the user's login shell. In particular, an argument of `-c` will cause the next argument to be treated as a command by most command interpreters. The command will be executed by the shell specified in `/etc/passwd` for the target user.
>
> You can use the `--` argument to separate `su` options from the arguments supplied to the shell.

## busybox docs excerpt:

In a comment the OP asked about why this wouldn't work in Alpine Linux. Since alpine is based on `busybox` I looked up the busybox docs. According to the docs busybox doesn't support sending additional arguments to `su`:

```
su
su [OPTIONS] [-] [username]

Change user id or become root

Options:

        -p, -m  Preserve environment
        -c CMD  Command to pass to 'sh -c'
        -s SH   Shell to use instead of default shell

```
> 6 votes

# Answer
The su-exec program in this script is actually the Alpine version created by Natanael Copa.

It is installed from the OP's Dockerfile on line 6.

The Alpine su-exec program is a substitute for gosu. Both programs are used to enable execution of a command as a specific user with the same environment variables and permissions as would normally be available to the user.

Thus you can effectively "step down" from root inside the container to run a specific command as a user and not as root.

So the alternative solution in Ubuntu would be to install gosu itself, which has no dependencies, and in the `docker-entrypoint.sh` script use:

```
gosu "$ZOO_USER" "$@"

```

(as per @alexey-shrub's comment below)
> 3 votes

---
Tags: docker, bash
---


title: Jenkins pipeline nested parallel

text:
# Question
Title: Jenkins pipeline nested parallel
Jenkins pipeline, declarative.

Simple question, how do I create a parallel stages inside a stage which is parallel by itself?

```
pipeline {
agent { label 'master' }
options {
    timestamps()
    timeout(time: 15, unit: 'MINUTES')
    ansiColor('xterm')
}

stages {
        stage('Main parallel') {
            parallel {
                stage('Start containers') {
                agent { label 'my-agent' }
                    stages {
                        stage('Start Linux containers') {
                            steps {
                                script {
                                    dir (""){
                                        println "running"
                                    }
                                }
                            }
                        }

                    }
                }
                stage('Build UI') {
                    stages {
                        stage('Cleanup main workspace'){
                            steps {
                                    dir("") {
                                        cleanWs()
                                    }
                            }
                        }
                        stage('Git checkout') {
                            steps {
                                dir("") {
                                        checkout()
                                }
                            }
                        }

                        stages {
                            stage('Gradle build') {
                                parallel{
                                    stage('gradle1') {
                                        steps {
                                            dir("") {
                                                script {
                                                            buildInfo = rtGradle.run switches: gradleParams , tasks: 'clean deploy'

                                                        }
                                                }
                                            }
                                        }
                                    }
                                    stage('gradle2') {
                                        steps {
                                            dir("") {
                                                script {
                                                        buildInfo2 = rtGradle2.run switches: gradleParams , tasks: 'unitTests'

                                                    }
                                                }
                                            }
                                        }
                                    }
                            }
                        }
                }
            }
        }
    }
}

```

I am sure it is possible, but how?

# Answer
> 5 votes
Sure, you can do this with Scripted Pipelines. I've never tried to do it with Declarative so I'm not sure whether it's possible to do with Declarative or, if it is, if it's easy or straightforward.

Here's an example of what this might look like with a Scripted Pipeline. I haven't tested it so there may be syntax errors or other problems.

```
def parallelTopLevelSteps = [:]

parallelTopLevelSteps['A and B'] = {
  def parallelNestedSteps = [:]

  parallelNestedSteps['step A'] = { echo('A') }
  parallelNestedSteps['step B'] = { echo('B') }

  parallel(parallelNestedSteps)
}

parallelTopLevelSteps['C and D'] = {
  def parallelNestedSteps = [:]

  parallelNestedSteps['step C'] = { echo('C') }
  parallelNestedSteps['step D'] = { echo('D') }

  parallel(parallelNestedSteps)
}

parallel(parallelTopLevelSteps)

```

---
Tags: jenkins-pipeline
---