# Question
Title: How to install docker in Raspbian Virtual Machine
I first installed docker in raspberry pi using: https://iotbytes.wordpress.com/setting-up-docker-on-raspberry-pi-and-running-hello-world-container/ which successfully did.

I tried doing the same with Raspbian VM but using `curl -sSL https://get.docker.com | sh` says **"Unable to locate package docker -ce"** Please help me with the detailed procedure. I am using Oracle Virtual Box (Linux 4.9.0-8-amd64 #1 SMP Debian 4.9.110-3+deb9u6 (2018-10-08))

# Answer
The Docker website doesn't know about your operating system and has no docker installation for you.

Try the one packaged by raspbian:

```
sudo apt install docker.io

```

The name is different in some distributions, you can use:

```
apt-cache search docker

```

To see if you can find one. If you can't find any, then switch to an older (or newer?) version of Raspbian.

Or you can search for instructions on how to compile it from source code.
> 1 votes

---
Tags: docker, virtualization
---


title: How to make PM2 as a autostart service?

text:
# Question
Title: How to make PM2 as a autostart service?
I want to make my PM2 as a startup service, because for example when my server loses power or something happens to it, it does not automatically restart the PM2 service.

# Answer
When configuring PM2 is finished and it's working properly, you must run two commands to make your PM2 as a autostart service and enjoy that. 1) Run 'pm2 save': It's save all of your PM2 configure to a file.

```
admin@server:~$ sudo pm2 save

```

2) Run 'pm2 startup': This command creates a system service and runs your last change saved with the first step.

```
admin@server:~$ sudo pm2 startup

```

So, Run the first command when you changed the configuration to store the last changes.
> 2 votes

---
Tags: linux
---


# Question
Title: What is the parent linux image of golang:1.7
On doing

```
$ docker history --no-trunc golang:1.7

```

it gives the output:

```
/bin/sh -c #(nop) COPY file:f6191f2c86edc9343569339f101facba47e886e33e29d70da6916ca6b1101a53 in /usr/local/bin/                                                                                2.48kB
/bin/sh -c #(nop) WORKDIR /go                                                                                0B
/bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/bin" && chmod -R 777 "$GOPATH"                                                                                0B
/bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/local/go/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin                                                                                0B
/bin/sh -c #(nop)  ENV GOPATH=/go                                                                                0B
/bin/sh -c set -eux;   dpkgArch="$(dpkg --print-architecture)";  case "${dpkgArch##*-}" in   amd64) goRelArch='linux-amd64'; goRelSha256='ad5808bf42b014c22dd7646458f631385003049ded0bb6af2efc7f1f79fa29ea' ;;   armhf) goRelArch='linux-armv6l'; goRelSha256='fc5c40fb1f76d0978504b94cd06b5ea6e0e216ba1d494060d081e022540900f8' ;;   i386) goRelArch='linux-386'; goRelSha256='99f79d4e0f966f492794963ecbf4b08c16a9a268f2c09053a5ce10b343ee4082' ;;   ppc64el) goRelArch='linux-ppc64le'; goRelSha256='8b5b602958396f165a3547a1308ab91ae3f2ad8ecb56063571a37aadc2df2332' ;;   s390x) goRelArch='linux-s390x'; goRelSha256='d692643d1ac4f4dea8fb6d949ffa750e974e63ff0ee6ca2a7c38fc7c90da8b5b' ;;   *) goRelArch='src'; goRelSha256='1a67a4e688673fdff7ba41e73482b0e59ac5bd0f7acf703bc6d50cc775c5baba';    echo >&2; echo >&2 "warning: current architecture ($dpkgArch) does not have a corresponding Go binary release; will be building from source"; echo >&2 ;;  esac;   url="https://golang.org/dl/go${GOLANG_VERSION}.${goRelArch}.tar.gz";  wget -O go.tgz "$url";  echo "${goRelSha256} *go.tgz" | sha256sum -c -;  tar -C /usr/local -xzf go.tgz;  rm go.tgz;   if [ "$goRelArch" = 'src' ]; then   echo >&2;   echo >&2 'error: UNIMPLEMENTED';   echo >&2 'TODO install golang-any from jessie-backports for GOROOT_BOOTSTRAP (and uninstall after build)';   echo >&2;   exit 1;  fi;   export PATH="/usr/local/go/bin:$PATH";  go version   247MB
/bin/sh -c #(nop)  ENV GOLANG_VERSION=1.7.6                                                                                0B
/bin/sh -c apt-get update && apt-get install -y --no-install-recommends   g++   gcc   libc6-dev   make   pkg-config  && rm -rf /var/lib/apt/lists/*                                                                                139MB
/bin/sh -c apt-get update && apt-get install -y --no-install-recommends   bzr   git   mercurial   openssh-client   subversion     procps  && rm -rf /var/lib/apt/lists/*                                                                                123MB
/bin/sh -c set -ex;  if ! command -v gpg > /dev/null; then   apt-get update;   apt-get install -y --no-install-recommends    gnupg2    dirmngr   ;   rm -rf /var/lib/apt/lists/*;  fi                                                                                0B
/bin/sh -c apt-get update && apt-get install -y --no-install-recommends   ca-certificates   curl   wget  && rm -rf /var/lib/apt/lists/*                                                                                44.6MB
/bin/sh -c #(nop)  CMD ["bash"]                                                                                0B
/bin/sh -c #(nop) ADD file:a023a99f7d01868b164d63bfaf8aabc7f271659c69939c3854f041f5a3217428 in /                                                                                123MB

```

I see them using apt-get so it is probably some kind of debian distribution.

What is the base distribution of this container? Is it ubuntu/alpine/... something else?

What does this mean?

```
/bin/sh -c #(nop) ADD file:a023a99f7d01868b164d63bfaf8aabc7f271659c69939c3854f041f5a3217428 in /                                                                                123MB

```

Thank you.

# Answer
The base image is `buildpack-deps:jessie-scm` based on the Dockerfile here. So the linux distro is Debian Jessie.

The `ADD file: ...` line shows that the first layer in the image was created by adding some files from outside the image (similar to "COPY" shown in the other layers)

From Dockerfile reference:

> The ADD instruction copies new files, directories or remote file URLs from and adds them to the filesystem of the image at the path
> 3 votes

---
Tags: docker
---


# Question
Title: jar file is not displaying in docker container
I have an sbt project and i am creating its Jar using the `sbt publishLocal` command and i want a docker image for that i am doing something like this

```
FROM hseeberger/scala-sbt:11.0.2_2.12.8_1.2.8

MAINTAINER Sara Waheed <sarawaheed3191@gmail.com>

WORKDIR /myproject

ADD . /myproject

# Define default command.
CMD ["sbt", "publishLocal"]

```

and i am building it `docker build -t myproject:v1` and run it `docker run myproject:v1` i have seen in the logs

```
[info] Main Java API documentation successful.
[info] Packaging /stripe-payment-account/target/myproject-1.0.1-javadoc.jar ...
[info] Done packaging.
[info] :: delivering :: com.test#myproject;1.0.1 :: 1.0.1 :: release :: Tue May 07 10:26:13 UTC 2019
[info]  delivering ivy file to /stripe-payment-account/target/ivy-1.0.1.xml
[info]  published myproject to /root/.ivy2/local/com.test/myproject/1.0.1/poms/myproject.pom
[info]  published myproject to /root/.ivy2/local/com.test/myproject/1.0.1/jars/myproject.jar
[info]  published myproject to /root/.ivy2/local/com.test/myproject/1.0.1/srcs/myproject-sources.jar
[info]  published myproject to /root/.ivy2/local/com.test/myproject/1.0.1/docs/myproject-javadoc.jar
[info]  published ivy to /root/.ivy2/local/com.test/myproject/1.0.1/ivys/ivy.xml
[success] Total time: 77 s, completed May 7, 2019, 10:26:13 AM

```

and when i try to look the file contents of this container via

```
docker exec -t -i containerid /bin/bash

```

and look into

```
cd ///root/.ivy2

```

and ls -a only shows

```
.  ..  .sbt.ivy.lock  cache

```

the local folder is missing here but when i run the container it shows that it created the jar in `/root/.ivy2/local/com.test/myproject/1.0.1/docs/myproject-javadoc.jar`

why its not showing the local folder in `.ivy2` folder what i am doing wrong here?

# Answer
> 1 votes
You are using docker wrongly in this case IMHO and I have a serious doubt that your `docker exec` command is targeting the correct container because, from what I see (and my very limited knowledge of sbt/scala), it should be dead by the time the jar is published. Let me explain.

`sbt publishLocal` is not a long living command. It will exit when the jar is published to the local `.ivy` store on disk. So when you launch a container from your image in foreground with `docker run --name mytestcontainer myproject:v1` (I added a name for easier debugging purpose on your side...), I am pretty sure you see all your above logs and that the command exits back to your shell.

At this point, if you issue a `docker ps`, your `mytestcontainer` will not show up, unless with `docker ps -a`where it will be listed in the exited container. If you try to start it again (`docker start mytestcontainer`), it will exit as soon as the jar is re-compiled/published. So there is no way to target an `exec` on this container since it is not running. The only way to have a shell in a running container out of this image is to override the start command in `docker run` but this will bypass your jar publication.

Now, what you first want is to publish the jar as part of the image, not to compile/publish it everytime you start a container out of the image.

```
FROM hseeberger/scala-sbt:11.0.2_2.12.8_1.2.8
MAINTAINER Sara Waheed <sarawaheed3191@gmail.com>

WORKDIR /myproject
ADD . /myproject

RUN sbt publishLocal

```

In this case, the jar will be published during the build

```
docker build -t myproject:v1

```

You can then run a container out of your image dropping to jshell as intended by the base image you are using:

```
docker run -it --rm --name testcontainer myproject:v1
May 12, 2019 8:31:33 AM java.util.prefs.FileSystemPreferences$1 run
INFO: Created user preferences directory.
|  Welcome to JShell -- Version 11.0.2
|  For an introduction type: /help intro

jshell>

```

Or if you want a more classic bash shell, you can override the start command:

```
docker run -it --rm --name testcontainer myproject:v1 /bin/bash

```

To go further, if your intent is to deliver your application in a ready to use image, you should have a look at Docker multistage builds. The idea is to build your jar in an image containing all the needed build tools and to copy the produced artifacts to an other image intended only for run that you will finally distribute. Here is the basic idea from your example. Image/command used are solely for illustration purpose and the process is highly perfectible (e.g. harcoded version....):

```
FROM hseeberger/scala-sbt:11.0.2_2.12.8_1.2.8 as build
WORKDIR /myproject
ADD . /myproject
RUN sbt publishLocal

FROM openjdk:8-alpine
WORKDIR /myproject
COPY --from=build /root/.ivy2/local/com.test/myproject/1.0.1/jars/myproject.jar .
CMD['java', '-jar', 'myproject.jar']

```

# Answer
> 0 votes
Not knowing anything about the java appication, but just going on what you can do with docker: The workdir for the parent image is /root. You can try to omit the WORKDIR statement. The build script might be working in the current directory, which would not be /root, but /workdir. Or you can add WORKDIR /root again before CMD.

Also don't forget some basic checks, eg.

```
df -h

```

to check if your server has enough free space (granted, it would have given an error message.) Also remove old containers:

```
docker ps -a
docker rm containername
docker images
docker rmi imagename

```

If you add the WORKDIR root at the end, and the jar is created at

```
 /root/.ivy2/local/default/root_2.12/0.1.0-SNAPSHOT/jars/root_2.12.jar

```

instead of, say:

```
/root/.ivy2/local/com.test/myproject/1.0.1/jars/myproject.jar

```

You can try to find the scripts that generate that path with grep:

```
egrep -r '(jars|root|default)' *

```

This will recursively list all the files where those terms appear. You can scan them for clues so as to what configuration files or environment variables to use. Also check the configuration section of the documentation for the jar generating application. (It's not software I am familiar with.) If you find that there is an environment variable that informs the location, lets say for example PROJECT, then you can include it in the Dockerfile:

```
ENV PROJECT myproject.test.com

```

Or if you want to dynamically pass it to the container when you run it:

```
docker run -E PROJECT=myproject.test.com myproject:v1

```

More likely however, is that it is inside some sort of configuration file that you need to copy to the container. Because "docker build" copies all the files in the current directory, you can just create the file in the current directory. Or if you need to manually put it somewhere else:

```
PUT configfile /some/path

```

Or

```
RUN mv configfile /some/path

```

Or you can create it dynamically inside the container. If it's just one line that needs to be in a file in the container:

```
RUN bash -c "echo project: myproject.test.com > /some/path/configfile"

```

As a plan B you can always just create a softlink inside the container, if it will solve your problem:

```
mkdir -p /root/.ivy2/local/com.test/myproject/1.0.1
ln -s /root/.ivy2/local/default/root_2.12/0.1.0-SNAPSHOT/jars /root/.ivy2/local/com.test/myproject/1.0.1/

```

You can also add this as a RUN command before the CMD in the Dockerfile:

```
RUN mkdir -p /root/.ivy2/local/com.test/myproject/1.0.1 && /root/.ivy2/local/default/root_2.12/0.1.0-SNAPSHOT/jars /root/.ivy2/local/com.test/myproject/1.0.1/

```

---
Tags: docker, dockerfile
---


# Question
Title: Is there a way to acquire metadata about an Amazon RDS database instance using only SQL?
For example, if I have a SQL Server instance in RDS, and I have only a connection string to connect to the server, is there a sproc or some pre-configured tables I can access via SQL on the instance that would report the instance size string? (db.r5.xlarge)

# Answer
> 1 votes
I don't have a SQL Server on AWS to check, but you could see if it's in xp\_msver, or figure it out from `ProcessorCount`, `ProcessorType` and `PhysicalMemory` in that sproc.

---
Tags: amazon-rds, sql
---


# Question
Title: How to keep previous containers on docker-compose up?
I am running a CI/CD pipeline that builds my service and ends with `docker-compose up -d` which brings up the service with a new docker image tag (`my-service:tag1`).

But when I look at `docker ps -a` the previous container (that was running image `service:tag0`) does not appear anymore, and I cannot access its logs.

Why does docker-compose remove the previous container?

Is there a way to see the logs of the previous containers?

# Answer
`docker-compose up` will remove containers when:

* The service name is the same, but has a change, in which case the old container is replaced with the new state of the service
* You include the `--remove-orphans` option when running `docker-compose`

Otherwise, if you change the name of the service, and do not specify `--remove-orphans`, you will see "orphaned" containers still running.
> 3 votes

---
Tags: docker, docker-compose, continuous-deployment, cicd
---


# Question
Title: What options are available to get a value that is associated to a key in yaml?
I often used jq to lookup a value in json. How to do this in yaml? I do not want to use something like sed as too much characters have to be removed then.

# Answer
One could use go-yq if one wants to lookup a value by applying jq style, i.e. if firefox\_version then one could look it up by adding a dot:

> ```
> docker run -v ${PWD}:/ansible-firefox utrecht/go-yq:2.1.0 \
>        .firefox_version /ansible-firefox/defaults/main.yml
>
> ```
> 0 votes

---
Tags: yaml
---


# Question
Title: What are best and comprehensive practices to consider when running docker in production?
Finally, you are so much in love with Docker that you want to move your online business-critical production systems with sensitive customer data to a Docker Swarm. Some might even already have done so. The other organization can't afford it by a policy forbidding production processes running in root mode.

What could be a checklist of building blocks to consider for a Docker production environment? One does not need all of them, but all of them should be important to be assessed.

*Disclaimer: I know there is a SE policy to avoid "large endless lists" but I think this checklist cannot be very big... and endless noway.*

So - what are these buildings blocks?

1. If not already deployed, consider running a Linux host system with advanced security settings - hardened kernel, SELinux etc.
2. Consider using a tiny Docker base image, like alpine, busybox or even scratch e.g. start with an empty base image
3. Use USER setting other than root
4. Carefully assess to further reduce the already shrinked set of kernel capabilities granted to container
5. Consider having only one executable binary per container to launch your process, ideally statically linked
6. Those who want to break your system to get a shell access might wonder if they found out your container has all shells disabled
7. Mount read-only volumes where only possible

**Question: what else?**

# Answer
**The host on which the containers are running**

Run the docker security bench on every node that runs docker containers https://github.com/docker/docker-bench-security

Running the following command on a node that runs docker containers:

```
docker run -it --net host --pid host --cap-add audit_control \
    -e DOCKER_CONTENT_TRUST=$DOCKER_CONTENT_TRUST \
    -v /var/lib:/var/lib \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v /usr/lib/systemd:/usr/lib/systemd \
    -v /etc:/etc --label docker_bench_security \
    docker/docker-bench-security

```

returns a list of checks:

```
[INFO] 1 - Host Configuration

[WARN] 1.1  - Ensure a separate partition for containers has been created

[NOTE] 4.2  - Ensure that containers use trusted base images

[PASS] 4.6  - Ensure HEALTHCHECK instructions have been added to the container image

```

Quote from the repository README:

> The Docker Bench for Security is a script that checks for dozens of common best-practices around deploying Docker containers in production. The tests are all automated, and are inspired by the CIS Docker Community Edition Benchmark v1.1.0.

Some of the issues that are reported by the security bench could be solved by reading the official docker security article and comparing it with the bullets that are defined in the question the following things are important as well:

* protect the docker daemon socket by implementing ssl
* content trust using notary and `DOCKER_CONTENT_TRUST` variable
> 24 votes

# Answer
**Docker images itself**

An additional option is to use Clair.

> Clair is an open source project for the static analysis of vulnerabilities in application containers (currently including appc and docker).
>
> In regular intervals, Clair ingests vulnerability metadata from a configured set of sources and stores it in the database.
>
> Clients use the Clair API to index their container images; this creates a list of features present in the image and stores them in the database.
>
> Clients use the Clair API to query the database for vulnerabilities of a particular image; correlating vulnerabilities and features is done for each request, avoiding the need to rescan images.
>
> When updates to vulnerability metadata occur, a notification can be sent to alert systems that a change has occured.
>
> Our goal is to enable a more transparent view of the security of container-based infrastructure. Thus, the project was named Clair after the French term which translates to clear, bright, transparent.
> 8 votes

# Answer
In addition to the points in this thread; the following would be my recommendation:

* Get control over Docker PID1 with dumb-init
* Do not run docker in production without a container orchestration system
    + Take your pick from Kubernetes, Mesos, Swarm etc.
* Use gosu for user control inside a docker image
* Follow the 12 factor app paradigm, if you are running stateful apps in containers, change it up.
    + If you really need to run stateful apps (mysql, zookeeper, elasticsearch) in containers, leverage orchestrator paradigms like Kubernetes Statefulsets
* Do robust secret / config management with tools like hashicorp vault / consul
* Ship the same container built by the devs to prod through a CI pipeline that takes it through staging, integration-tests thoroughly.
* Create notifications around CVEs and patches, trigger builds on patch-notify
* Have extensive logging to get insight into the running container, you do not want to give the devs SSH access to either the host or the containers
    + recommendation: fluentd
* Have both container and host metrics
    + recommendation: prometheus+node-exporter
> 8 votes

# Answer
**Docker is still in development.**

As with every other software in-dev bugs will happen, insecure features might be added, there could be architectural flaws that lead to security breaches. Do not underestimate this! Your system might be completely safe today, but with the patch from next week someone finds a bug, writes an exploit, and suddenly your system is wide open.

Unless you must, don't update to the latest version. Use the latest well-tested version instead.

**Docker is no virtualization**

If someone escapes from a Docker container, that attacker is on the real machine immediately. There is no second gate like virtualization that will prevent a breach.

Treat a Docker container like any other program. Run with lowest possible user rights, block all network traffic that isn't required, virtualize the entire Docker host if performance allows.

**Docker is no protection**

Whatever code is run inside Docker containers is run without question from Docker. Any attacker can simply install his software inside the container, and Docker would run that like any other code.

Aside from the things you mentioned in the question, consider using metrics and alerts to get notified if any Docker image is doing strange things. Is there a sudden, ongoing CPU spike? Is the program suddenly scanning network ports? Is there suspicious disk access? You should get a notification if any of that happens. There are many tools available to measure these things, you should use them.
> 7 votes

# Answer
If you are filling up your docker entrypoint with `sed` commands, consider this practice:

* Use a tool such as confd to manage your docker images configuration files and keep them updated

Confd will read data from many supported key-value stores and render configuration templates dynamically.
> 2 votes

# Answer
Reading the official docker security article and comparing it with the bullets that are defined in the question the following things are important as well:

* protect the docker daemon socket by implementing ssl
* content trust using notary and `DOCKER_CONTENT_TRUST` variable
> 0 votes

# Answer
One could use A2D to bake an app into a docker image while taking certain things into account, e.g. non-root, permissions, location of the app:

> ```
> docker run -v $PWD:/projectName utrecht/a2d:1.0.0 \
>        -projectName someProjectName -dockerfile /projectName/Dockerfile
>
> ```

returns:

> ```
> FROM golang:1.12.4-alpine as builder
> COPY . ./someProjectName/
> WORKDIR someProjectName
> RUN adduser -D -g '' someProjectName && \
>     apk add git && \
>     CGO_ENABLED=0 go build && \
>     cp someProjectName /someProjectName && \
>     chmod 100 /someProjectName
>
> FROM scratch
> COPY --from=builder /etc/group /etc/group
> COPY --from=builder /etc/passwd /etc/passwd
> COPY --from=builder --chown=someProjectName:someProjectName /someProjectName /usr/local/someProjectName
> COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
> USER someProjectName
> ENTRYPOINT ["/usr/local/someProjectName"]
>
> ```
> 0 votes

---
Tags: docker, security, devsecops, a2d
---


# Question
Title: Automatically write new bash history to change-log file
Every time I ssh into a server I might issue some bash commands and then sign off (hup signal?). Is there a way to write the shell commands that I issued on the server to some changelog file?

I want to remove basic commands like ls/cd, and duplicate commands, as well. It's basically as if every command I issue in bash is like this:

```
eval "$(my_command | tee "$HOME/change.log")"

```

but I don't know how to set this up, anybody know a good way to do this?

It would be really nice to eliminate commands like ls and cd, but also show the pwd of every command that was issued:

```
command  (/home/ubuntu)
command1 (/root/.ssh)
command2 (/home/ubuntu/teros)
command1 (/usr/local)
command1 (/usr/local/lib)
command3 (/etc)

```

# Answer
> 2 votes
Try using GNU Screen or Tmux. These are called "screen multiplexers" but you'd get a number of benefits

1. The commands you run will continue running even if you disconnect, or you have network related connection drops. Both software allow "re-attaching" to the running session
2. Command outputs will persist in memory. You can start a long running command, log off, then connect back some time later and see the output of the command.
3. Save the whole shell session to a file. You can have the log of all the commands you issued, as well as the output you got, in plain text format.

# Answer
> 0 votes
For the second part, what happens if you `export PS1` in `~/.bashrc` or `~/ .bash_profile`

```
export PS1='$(whoami)@$(hostname):$(pwd)'

```

---
Tags: bash, ssh
---


# Question
Title: How do I get Kubernetes to work when I get an error "the server could not find the requested resource"?
I cannot create a kubernetes pod. I just deployed Kubernetes to Ubuntu 16.04.

I have this as a.yml:

```
apiVersion: v1
kind: ReplicationController
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    app: nginx
  template:
    metadata:
      name: nginx
      labels:
        app: nginx
    spec:
      containers:
      - name: nginx
        image: nginx
        ports:
        - containerPort: 80

```

I run this: `kubectl create -f a.yml`

I get this:

> Error from server (NotFound): the server could not find the requested resource

I expected the above command to create a pod.

I ran this: `kubectl cluster-info dump`

I got this:

```
{
    "selfLink": "/api/v1/nodes",
    "resourceVersion": "185",
    "Items": [
        {
            "name": "ip-10-10-10-10",
            "selfLink": "/api/v1/nodes/ip-10-10-10-10",
            "uid": "cdc51051-ea07-11e7-b7ae-025c31abfac0",
            "resourceVersion": "185",
            "creationTimestamp": "2017-12-26T06:41:29Z",
            "labels": {
                "kubernetes.io/hostname": "ip-10-10-10-10"
            },
            "Spec": {
                "PodCIDR": "",
                "ExternalID": "ip-10-10-10-10",
                "ProviderID": "",
                "Unschedulable": false,
                "Taints": null,
                "ConfigSource": null
            },
            "Status": {
                "Capacity": {
                    "cpu": "0",
                    "memory": "0",
                    "pods": "40"
                },
                "Allocatable": {
                    "cpu": "0",
                    "memory": "0",
                    "pods": "40"
                },
                "Phase": "",
                "Conditions": [
                    {
                        "Type": "Ready",
                        "Status": "True",
                        "LastHeartbeatTime": "2017-12-26T07:03:01Z",
                        "LastTransitionTime": "2017-12-26T06:41:29Z",
                        "Reason": "kubelet is posting ready status",
                        "Message": ""
                    }
                ],
                "Addresses": [
                    {
                        "Type": "LegacyHostIP",
                        "Address": "10.10.10.10"
                    }
                ],
                "DaemonEndpoints": {
                    "KubeletEndpoint": {
                        "Port": 0
                    }
                },
                "NodeInfo": {
                    "MachineID": "",
                    "SystemUUID": "",
                    "BootID": "",
                    "KernelVersion": "",
                    "OSImage": "",
                    "ContainerRuntimeVersion": "",
                    "KubeletVersion": "",
                    "KubeProxyVersion": "",
                    "OperatingSystem": "",
                    "Architecture": ""
                },
                "Images": null,
                "VolumesInUse": null,
                "VolumesAttached": null
            }
        }
    ] } {
    "selfLink": "/api/v1/namespaces/kube-system/events",
    "resourceVersion": "185",
    "Items": [] } {
    "selfLink": "/api/v1/namespaces/kube-system/replicationcontrollers",
    "resourceVersion": "185",
    "Items": [] } {
    "selfLink": "/api/v1/namespaces/kube-system/services",
    "resourceVersion": "185",
    "Items": [] } Error from server (NotFound): the server could not find the requested resource (get daemonsets.extensions)

```

What am I doing wrong?

Update 12/26/17

The command `kubectl get ns` produces

> the server doesn't have a resource type "namespaces"

$ `kubectl version`

> Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.0", GitCommit:"925c127ec6b946659ad0fd596fa959be43f0cc05", GitTreeState:"clean", BuildDate:"2017-12-15T21:07:38Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"0", GitVersion:"v1.0.3", GitCommit:"61c6ac5f350253a4dc002aee97b7db7ff01ee4ca", GitTreeState:"clean", BuildDate:"", GoVersion:"", Compiler:"", Platform:""}

Update 1/15/18 How do I get some namespaces to be returned?

# Answer
As per https://kubernetes.io/docs/tasks/tools/install-kubectl/#before-you-begin:

> You must use a kubectl version that is within one minor version difference of your cluster. For example, a v1.2 client should work with v1.1, v1.2, and v1.3 master. Using the latest version of kubectl helps avoid unforeseen issues.

So you need to either downgrade your kubectl version or upgrade your cluster version.
> 7 votes

# Answer
It looks like your k8s cluster has not been deployed well. I have verified that it works to deploy the nginx by using the template as defined in the question.

In contrast to your cluster, the cluster I am using has several k8s images to host the dashboard etc.

```
        "Images": [
            {
                "Names": [
                    "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.6.3"
                ],
                "SizeBytes": 138972432
            },
            {
                "Names": [
                    "gcr.io/google_containers/kubernetes-dashboard-amd64@sha256:71a0de5c6a21cb0c2fbcad71a4fef47acd3e61cd78109822d35e1742f9d8140d",
                    "gcr.io/google_containers/kubernetes-dashboard-amd64:v1.8.0"
                ],
                "SizeBytes": 119155776
            },

```

Also in order to deploy images, at least some namespaces should be returned.

```
$ kubectl get ns
NAME          STATUS    AGE
default       Active    37m
kube-public   Active    37m
kube-system   Active    37m

```
> 2 votes

---
Tags: kubernetes
---


# Question
Title: How to solve "E405: Remote package tasks should have a retry" in Ansible?
> E405 v4.0.0 Remote package tasks should have a retry Package operations are unreliable as they require network communication and the availability of remote servers. To mitigate the potential problems, retries should be used via register: my\_result and until: my\_result | success

This shows up when running `ansible-lint` as the following error:

```
[405] Remote package tasks should have a retry
plays/sample.yml:31
Task/Handler: Install list of packages

```

Given the sample code:

```
- name: Install list of packages
  apt:
    name: [jq, vim, curl, unzip, dnsutils]
    state: present

```

How to solve this?

# Answer
`ansible-lint` warning `405` can be quite irritating as in certain cases, i.e. when running locally retrying is very unlikely to make any difference. Actually resolving it is normally simply a case of adding the following:

```
register: task_result
until: task_result is success
retries: 10
delay: 2

```

When this is added to task Ansible will retry ten times with a delay of two seconds between each retry until the `returncode` is `0`.
> 8 votes

# Answer
```
pip install --upgrade ansible-lint>=4.1.0

```

Rule 405 has been dropped following the discussion in https://github.com/ansible/ansible-lint/issues/456
> 0 votes

---
Tags: ansible
---


# Question
Title: What package manager does the gitlab docker CI image use?
I'm trying to install additional software but `apt-get` is not available.

So in my `.gitlab-ci.yml` file I have:

`image: docker:stable`

What is the image based off from?

# Answer
https://hub.docker.com/\_/docker/

Seems it's just a reference to the above... Which uses APK as the package manager
> 2 votes

---
Tags: docker, gitlab, gitlab-ci-runner
---


# Question
Title: Azure - Connection String in Key Vault vs Application Settings
For Azure Functions and WebJobs, is there any benefit to putting connection strings as Secrets in Key Vault instead of putting them directly in Application Settings (and referencing them using ConfigurationManager.ConnectionStrings)? Are Azure Key Vault Secrets primarily meant for VMs and such rather than Azure Functions and WebJobs?

It seems like it's just adding an extra step in both development (update the Secret in Key Vault and the Secret Identifier in Application Settings) and an extra step in runtime (an additional retrieval from Key Vault), with the only benefit being that the Secret in the Application Settings is an identifier instead of the actual Secret. I don't see the security benefit here, whereas there are detriments.

# Answer
The benefits as I see them are the general reasons to use Azure Key Vault

* Secrets are centrally stored with options for authorization, auditing, etc.
* Azure can be scripted to update the secrets on a timer - so in case a connection string gets in to the wrong hands, it is only valid for a day (or a week, or whatever)
* In case the secret is shared between multiple applications, you only have to update it in one place (not the greatest benefit, but still nice)
> 9 votes

# Answer
I have the same opinion as you, I see no point in using the Key Vault in such scenarios.

Instead I utilize Azure App Configuration https://docs.microsoft.com/en-us/azure/azure-app-configuration/overview , where I simply store all application settings. And the only string I store together within the actual app service/ func/ web job, would be the connection string for accessing the Azure App Configuration.

In my Startup where the configuration settings is applied I either read from ..settings.json, and if not present (which it should not be if running on Azure), I look for the connection string to the Azure App Configuration.

If ..settings.json is found, it means that I'm running locally.

..settings.json is encrypted if it is not in gitignore file.

It looks something like this:

```
var appConfigEndpoint = Environment.GetEnvironmentVariable("AppConfigEndpoint");
            var appConfigLabel = Environment.GetEnvironmentVariable("AppConfigLabel");

            if (!string.IsNullOrEmpty(appConfigEndpoint))
            {
                config.AddAzureAppConfiguration(options =>
                {
                    options.Connect(appConfigEndpoint);
                    options.Use(keyFilter: "*", labelFilter: appConfigLabel);
                });
            }
            else
            {
                config.AddJsonFile("appsettings.test.json", optional: false);
            }

```
> 3 votes

# Answer
The key vault retrieval can be a performance problem. Reading from key vault takes several seconds. The Azure SLA only says it will take less than 5 seconds 99.9% of the time. It does not guarantee the average time, but from my experience and hearsay, the average is about 2 seconds.

Unless your web app is "always on", it will be unloaded if idle for 5-20 minutes. That means, unless you have a lot of traffic, your site response times will frequently suck.
> 0 votes

# Answer
We are planning to use below

**Secrets** \- Azure Key-Vault

* Master Key rotation for Always Encrypted columns
* Certificate Store (can easily write a provider in .NET)
* Sensitive items like **connection strings** (hiding other resource info), passwords etc.

**Configurations** \- Azure App Configurations (still in preview)

* Watch is a wonderful method for service discovery especially if you need to update flags or values for operational uses.
* No need to re-initiate a container or instance.
> 0 votes

---
Tags: azure
---


# Question
Title: How to use stored certificate in Jenkins declarative pipeline?
I have a stored certificate in Jenkins, and I need to use it in a pipelinebuild job, but the problem is it appears the Pipeline can't use certificates:

trying to use:

```
    environment {
        KEY = credentials('mycert')
    }

```

results in:

```
ERROR: No suitable binding handler could be found for type com.cloudbees.plugins.credentials.impl.CertificateCredentialsImpl. Supported types are StandardUsernamePasswordCredentials,FileCredentials,StringCredentials,SSHUserPrivateKey.

```

How do I get around this and use that stored cert in a pipeline? There has to be a way otherwise why would storing certs as credentials even be possible.. would be useless to store something without ever being able to use it.

# Answer
Some credential types cannot be bound directly in an environment section. From the docs :

> If you need to set credentials in a Pipeline for anything other than secret text, usernames and passwords, or secret files - i.e SSH keys or certificates, then use Jenkins' Snippet Generator feature, which you can access through Jenkins' classic UI.

So the generator would give you something looking like this:

```
withCredentials([certificate(aliasVariable: '', credentialsId: 'myCert', keystoreVariable: 'CERT', passwordVariable: '')]) {
    // some block
}

```
> 2 votes

# Answer
I usually set the credential NAME into an environment variable

```
environment {
    CERT = 'mycert'
}

```

Then I can use that name where ever I need it

```
withCredentials([usernamePassword(credentialsId: "${env.CERT}",
                                  passwordVariable: 'CERT_PASSWORD',
                                  usernameVariable: 'CERT_USER')]) {
    // Do things needing environment variables
}

```

or

```
checkout(poll: false,
         scm: [
               $class: 'GitSCM',
               userRemoteConfigs: [[credentialsId: "${env.CERT}",
                                   url: 'https://code.example.org/somerepo']]
               ])

```
> 1 votes

---
Tags: jenkins, jenkins-pipeline
---


title: Jenkins pipeline lock resources

text:
# Question
Title: Jenkins pipeline lock resources
I need to run commands on my pipeline during a certain stage, on devices which connected to my Jenkins as slaves. But I want them to run only when another one finished. On the other hand, I want them to run all previous stages before they "wait" for another device to finish.

What is the best way to implement this lock? Again, it is the same job, but different slaves.

# Answer
> 4 votes
I found exactly what I was looking for-

\[Jenkins plugin\] (https://wiki.jenkins.io/display/JENKINS/Lockable+Resources+Plugin)

Which let you create a lock between, slaves , different jobs and more..

(in contrast to the plugin description)

# Answer
> 0 votes
There is no perfect way to do this that I know of. To explain why, take this pipeline script as an example:

```
node('my-first-node') {
  stage('my-first-stage') {
    // some steps here
  }
}

node('my-second-node') {
  stage('my-second-stage') {
    // some steps here
  }
}

node('my-first-node') {
  stage('my-third-stage') { // must be run after my-second-stage
    // some steps here
  }
}

```

This will run your stages in the correct order, but there is no guarantee that the node assigned to run my-first-stage will be the same node as the one assigned to run my-third-stage. Furthermore, even if you can be confident that they are the same node (for instance, you only have one node with the "my-first-node" label in your executor pool), there is no guarantee that the assigned workspace folder will be the same for both stages either. And even if you use the workspace step to manually assign a workspace directory, there is no guarantee that the assigned workspace directory will be the one you asked for either.

However, if you don't care that my-first-stage and my-third-stage run in the same workspace directory on the same node and you only care about the order that your steps run in, this works fine.

---
Tags: jenkins-pipeline, groovy
---


# Question
Title: docker swarm init made docker go corrupted
I started learning docker recently and I set up a virtual machine (VirtualBox) with Lubuntu image:

* Linux verion: `Linux 4.18.0-10-generic i686`
* Docker version: `18.06.1-ce`, API-version: `1.38`, Go version: `go1.10.4`, Git commit: `e68fc7a`

So unless I have started acquaintance with swarm everything was going smoothly. However after I executed `docker swarm init` daemon seemed going down and whatever command I'm now trying to execute, it responds with: `Cannot connect to Docker daemon at..`

When I'm trying to run `sudo dockerd` it shows the output where there're some warnings like `failed to load plugin io.containerd.snapshotter.v1.btrfa` and one error saying `error creating cluster object... error="name conflicts with an existing object" module=node node.id=...`. Eventually the output ends with `floating point exception` with some address in memory.

Restarting does not help. The only things that makes docker run is reverting to the snapshot that I luckily made before calling `swarm init`.

So I would appreciate your help or advice how I do fix everything out.

# Answer
> 5 votes
This response is for Kernel: Linux 4.4.132-1075-rockchip-ayufan-ga83beded8524 Architecture: aarch64 aarch64 aarch64 GNU/Linux

I had the same issue when issuing commands to a cluster in swarm mode.

Running `docker info` would result in `Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?`

Running `sudo systemctl start docker` would result in a failed service state and output reading

```
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.

```

Running `systemctl status docker.service` would result in

```
docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Thu 2019-05-16 06:03:46 UTC; 1min 12s ago
     Docs: https://docs.docker.com
  Process: 3874 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (cod
 Main PID: 3874 (code=exited, status=2)

```

Running `journalctl -xe` wouldn't result in any information related to docker

Running `sudo dockerd -D` would result in lots of output with a certain line reading `error="name conflicts with an existing object" module=node node.id=105rworelt1fvw9ttjhnf61aw`

Looking in `/var/lib/docker` there is a folder called `swarm` and doing a grep on all files and folders in the swarm directory for the string `105rworelt1fvw9ttjhnf61aw` came up with two files with the string `105rworelt1fvw9ttjhnf61aw` in them. sudo grep -r "105rworelt1fvw9ttjhnf61aw" /var/lib/docker/swarm/

Those two files were `/var/lib/docker/swarm/.tmp-state.json504201306` and `/var/lib/docker/swarm/state.json`

At this point I wasn't feeling very surgical so I removed all files and folders under the `/var/lib/docker/swarm` and then tried `sudo systemctl start docker` again

```
A bit surgical: sudo rm -ri /var/lib/docker/swarm/

```

Note: Don't delete the `swarm` directory. If you do just create it again.

After this a `docker info` would work and I was able to `docker swarm init` again and deploy a stack without issues

---
Tags: docker, linux
---


# Question
Title: How to run a build from a specific old tag in gitlab ci?
I can build from a specific old tag in teamity. Screenshot:

How to run a build from a specific old tag in gitlab ci?

```
variables:
  MAVEN_OPTS: "-Djava.awt.headless=true -Dmaven.repo.local=.m2/repository"
  MAVEN_CLI_OPTS: "-s maven_settings.xml --batch-mode --errors --fail-at-end --show-version"

stages:
  - build
  - release
  - build-tag

build:
  stage: build
  script:
    - mvn $MAVEN_CLI_OPTS -Dmaven.test.skip=true clean install -PRPM -U

release:
  stage: release
  script:
    - eval $(ssh-agent)
    - ssh-add <(echo "$SSH_PRIVATE_KEY")
    - git config user.name "$PUSH_USER_NAME"
    - git config user.email "$PUSH_USER_EMAIL"
    - git checkout master
    - git reset --hard origin/master
    - mvn $MAVEN_CLI_OPTS clean release:prepare -Dresume=false -DautoVersionSubmodules=true -DdryRun=false -Dmaven.test.skip=true
  when: manual

build-tag:
  stage: build-tag
  script:
    - git checkout master
    - git pull
    - git reset --hard $(git describe --abbrev=0 --tags)
    - mvn $MAVEN_CLI_OPTS -Dmaven.test.skip=true clean install -PRPM -U
  when: manual

```

build-tag i dont see in pipeline

How to run a build from a specific old tag in gitlab ci?

# Answer
Within the project

* go to *CI / CD* -\> *Pipelines*
* press *Run Pipeline*
* under *Create for* choose the branch or tag you want to run the pipeline for
* press *Create Pipeline*

Now to trigger a manual Job you can use the jobs of the recently created pipeline.
> 1 votes

---
Tags: gitlab, maven
---


# Question
Title: Ansible to check version of software on remote hosts before install/upgrade
Goal: Skip the host if version of custom software (non-rpm based) installed at remote host is greater than or equal to software version intended for upgrade. For all intents and purposes, this playbook (underneath) works well.

Questions :

1. How to formulate the code to get rid of \[DEPRECATION WARNING\] during runtime?
2. Is there are better way to apply the same logic?

```
   ansible --version
    ansible 2.7.10
    python version = 2.7.5

```

```
#Remote software version
cat /path/to/version_file.txt
FullVersion=12.52.105.2112

```

```
#Playbook Install/upgrades firmare on remote nodes
#
---

- hosts: "RedHat_7_nodes"
  remote_user: test_user
  connection: ssh
  gather_facts: '{{ gather }}'

# Next Section - Variables

  vars_prompt:

    - name: EXPECTED_REMOTE_SOFTWARE_VERSION
      prompt: What is expected version of Software? (If remote server is already at that level, this playbook will skip it)
      private: no

  tasks:

   - name: Check if the remote Software version is below "{{ EXPECTED_REMOTE_SOFTWARE_VERSION }}"
     shell: grep FullVersion /path/to/version_file.txt  | awk -F = '{print $2}'
     check_mode: no
     register: CURRENT_INSTALLED_VERSION

   - debug:
       msg: "{{ CURRENT_INSTALLED_VERSION }}"
   - meta: end_play
     when: CURRENT_INSTALLED_VERSION.stdout | version(EXPECTED_REMOTE_SOFTWARE_VERSION,'>=')

   - name: Run the binary and upgrade since the installed version is lower than intended software version
     shell: echo "Run the binary and upgrade since the installed version is lower than intended software version "
     check_mode: no

```

```
# Runtime 1. Expected version is lower than installed version. Skip this host. No upgrade.

ansible-playbook WebAgent_Install.yml -i inventory.txt --extra-vars "gather=yes"

What is expected version of Software? (If remote server is already at that level, this playbook will skip it): 12.52.105.2111

PLAY [RedHat_7_nodes] ************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************
ok: [joker7.cartoon.biz]

TASK [Check if the remote Software version is below "12.52.105.2111"] ************************************************************************************************************************
changed: [joker7.cartoon.biz]

TASK [debug] *********************************************************************************************************************************************************************************
ok: [joker7.cartoon.biz] => {
    "msg": {
        "changed": true,
        "cmd": "grep FullVersion /path/to/version_file.txt | awk -F = '{print $2}'",
        "delta": "0:00:00.013802",
        "end": "2019-05-16 14:37:00.635365",
        "failed": false,
        "rc": 0,
        "start": "2019-05-16 14:37:00.621563",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "12.52.105.2112",
        "stdout_lines": [
            "12.52.105.2112"
        ]
    }
}
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|version` use `result is version`. This feature will be removed in version 2.9. Deprecation warnings can
 be disabled by setting deprecation_warnings=False in ansible.cfg.

PLAY RECAP ***********************************************************************************************************************************************************************************
joker7.cartoon.biz         : ok=3    changed=1    unreachable=0    failed=0

```

```
# Runtime - 2. Expected version is higher than installed version. Upgrade.

ansible-playbook WebAgent_Install.yml -i inventory.txt --extra-vars "gather=yes"

What is expected version of Software? (If remote server is already at that level, this playbook will skip it) [12.52.105.2112]: 12.52.105.2113

PLAY [RedHat_7_nodes] ************************************************************************************************************************************************************************

TASK [Gathering Facts] ***********************************************************************************************************************************************************************
ok: [joker7.cartoon.biz]

TASK [Check if the remote Software version is below "12.52.105.2113"] ************************************************************************************************************************
changed: [joker7.cartoon.biz]

TASK [debug] *********************************************************************************************************************************************************************************
ok: [joker7.cartoon.biz] => {
    "msg": {
        "changed": true,
        "cmd": "grep FullVersion /path/to/version_file.txt | awk -F = '{print $2}'",
        "delta": "0:00:00.013155",
        "end": "2019-05-16 14:35:49.364318",
        "failed": false,
        "rc": 0,
        "start": "2019-05-16 14:35:49.351163",
        "stderr": "",
        "stderr_lines": [],
        "stdout": "12.52.105.2112",
        "stdout_lines": [
            "12.52.105.2112"
        ]
    }
}
[DEPRECATION WARNING]: Using tests as filters is deprecated. Instead of using `result|version` use `result is version`. This feature will be removed in version 2.9. Deprecation warnings can
 be disabled by setting deprecation_warnings=False in ansible.cfg.

TASK [Run the binary and upgrade since the installed version is lower than intended software version] ****************************************************************************************
changed: [joker7.cartoon.biz]

PLAY RECAP ***********************************************************************************************************************************************************************************
joker7.cartoon.biz         : ok=4    changed=2    unreachable=0    failed=0

```

# Answer
> 2 votes
> How to formulate the code to get rid of \[DEPRECATION WARNING\] during runtime?

Simply by following the tip in the error message.

```
when: CURRENT_INSTALLED_VERSION.stdout is version(EXPECTED_REMOTE_SOFTWARE_VERSION,'>=')

```

> Is there are better way to apply the same logic?

I don't see anything wrong if it meets your current needs.

Only remark: The first task is not idempotent. It will report change anytime it runs (because shell is supposed by default to change the remote system). Since your command is a simple grep that does not change anything, your can have the task report always ok by adding the option:

```
changed_when: false

```

---
Tags: ansible
---


title: Jenkins Tool vs Slave local tool

text:
# Question
Title: Jenkins Tool vs Slave local tool
What is the difference between using maven as a tool versus having a slave that has maven installed?

I am using Jenkins on OCP and it came with a slave image for maven and not setup as a tool. I now need to install gradle and I am wondering if I should add it as a tool or make a slave with it installed.

# Answer
Installing it as a tool allows Jenkins to manage the binary and ensure it's available for your build while also providing supporting pipeline syntax `withEnv()`. Having an agent with the binary allows you to shell out and run your commands using `sh` without `withEnv()`. It also allows you to decide where these binaries come from depending on your agent type (*static, vm, container, pod*).

Personally, I prefer the latter because I can control the updating of said binaries on the agent in a more automated fashion.
> 1 votes

---
Tags: jenkins, gradle
---


# Question
Title: How to write a dynamic declarative pipeline that contains sequential job inside parallel job
I'm trying to write a declarative pipeline code that accepts a map and create a pipeline. I can able to achieve sequential stages or parallel stages but facing problems while making a pipeline that contains sequential stages inside parallel stages.

The input data would be Map. Each list in the map should run parallel and the items inside the list corresponding to each key should run in sequentially.

example data : \[1:\[11,12\], 2:\[21,22\], 3:\[31,32\]\]

Am using Jenkins 2.164.3

The output should be

# Answer
This was covered in an official blog post when the feature to nest stages inside a parallel block was added.

Here is the example copy-pasted from the blog post. In this example, the Windows and Linux blocks are run in parallel, but the stages within each block are run sequentially:

```
pipeline {
    agent none

    stages {
        stage("build and deploy on Windows and Linux") {
            parallel {
                stage("windows") {
                    agent {
                        label "windows"
                    }
                    stages {
                        stage("build") {
                            steps {
                                bat "run-build.bat"
                            }
                        }
                        stage("deploy") {
                            when {
                                branch "master"
                            }
                            steps {
                                bat "run-deploy.bat"
                            }
                        }
                    }
                }

                stage("linux") {
                    agent {
                        label "linux"
                    }
                    stages {
                        stage("build") {
                            steps {
                                sh "./run-build.sh"
                            }
                        }
                        stage("deploy") {
                             when {
                                 branch "master"
                             }
                             steps {
                                sh "./run-deploy.sh"
                            }
                        }
                    }
                }
            }
        }
    }
}

```

EDIT: I missed that you're trying to do this based on arbitrary input. If you want to do that, you must use scripted Pipelines (or a script nested in a Declarative Pipeline). Plain Declarative Pipelines don't support that kind of runtime-defined Pipelines.
> 2 votes

---
Tags: jenkins-pipeline, jenkinsfile
---


# Question
Title: How to automate the process of patching Jenkins Slave instances in AWS
I've been working on an automated patching solution for some infrastructure in one of our AWS accounts. This AWS happens to host the Jenkins instance we use for CI/CD actions, among other things.

Our instance runs off AWS EC2 instances and are persistent - basically they live long enough that they would be out of compliance if we never patched them.

I have patched Jenkins slaves manually in the past, where the workflow looked something like this:

* Log into Jenkins instance
* Make sure there are no running jobs on the target node
* Mark the target node as offline
* Patch the target node
* Log into Jenkins instance
* Mark the node as online again

This process involves interacting with the Jenkins instance itself. Since I want this solution to be automated, I don't want to have to manually log in and facilitate this process before and after patching. Right now, I have the ability to execute arbitrary bash code from the target node locally and I've been working on a way of using the target node to mark itself offline in some way, and right now my best bet looks like using the environment variables that Jenkins sets while it is running a job to determine whether or not the node is busy.

That will solve the problem of the making sure the node isn't busy, but I have no idea how I will prevent it from accepting new jobs while automation patches the machine. I don't want to use API calls because I believe that would require me to put credentials on all of the Jenkins machines, which I am adverse to doing (Secrets management is a pain). Another possible method could be to prevent the Jenkins master from assigning the slave more tasks by shutting off the slave's SSH, but that sounds like a terrible idea.

Anyone have any ideas as to how I could mark Jenkins slaves offline and online, with my only real tool being local bash execution on the slave itself?

# Answer
If you use the Self-Organizing Swarm Plug-in to connect your build agents to your master, you can spin agents up/down without having to make any changes on your master. Agents will automatically register themselves with the master. You will have to supply credentials to the agents to self-register. I would suggest using SSM Parameter Store to store them and then retrieve them in userdata.

Your agents can then be completely ephemeral, and updates performed in userdata or by updating the AMI.

See this Terraform module for a more detailed example of implementing this.
> 1 votes

---
Tags: jenkins, amazon-web-services, jenkins-pipeline, operations
---


# Question
Title: How to use in Chef different attributes values for different nodes
I have a cookbook that is being used to deploy some software on a chef nodes. I use it when bootstraping a node and then remove it from the run\_list since I need it only to run once. I don't know if it's proper to do thing like that in Chef or if it's an antipattern. But this cookbook needs to have some variables that change on every node. Let's say that it needs to know the name of the main network interface of that node and I need to provide to it some kind of license number. I do that by using attributes, I specify in attributes/default.rb:

```
default['cookbook_name']['interface'] = 'eth0'
default['cookbook_name']['license'] = 'abcxyz'

```

And then I upload this to chef-server and run recipe on a node that is using this attributes. But if the other person wants to deploy the same software using this cookbook on some other node, then he needs to wait until I finish because if he will replace those values in attributes.rb with values of second node and upload it to chef-server then it will overwrite values that I need right now.

What is the proper way to do that? I feel like I am missing something about the work flow of attributes. As far as I see, using data bags will not solve the problem. Is there some way to assign "attribute" only to a specific node? Can I assign an attribute only to a given node?

# Answer
> 0 votes
You may use attributes precedence for that or a wrapper cookbook, I highly encourage you to take some time to check out https://learn.chef.io which cover those uses cases.

This is a kind of antipattern, one shot configuration are not what Chef is for, but for this use case I'd go with a common cookbook doing what is needed and a wrapper cookbook defining the attributes and including the common recipe.

A typical wrapper cookbook would need a few files:

* `metadata.rb` as another cookbook with a line `depends 'common_cookbook'`
* `attributes\default.rb` with the values for this type of server, same syntax as in common\_cookbook
* `recipes\default.rb` with the line `include_recipe 'common_cookbook'`

This pattern is also known as role cookbooks, more details in this blog post

---
Tags: chef
---


title: Chef - Setting Unique Node Attributes

text:
# Question
Title: Chef - Setting Unique Node Attributes
I'm fairly new to Chef and have been writing recipes for Kiosks that we use at work.

I've created a role for the Kiosk that contains a set of attributes, however one of these attributes is a unique ID that is different for each Kiosk node.

This ID is provided by our customer who uses the Kiosk, is abritrary, and can't be generated automatically.

What's best practice for setting this value? I'm using Chef Server and was thinking the best place to do it would be at Bootstrap but not entirely sure how.

# Answer
Yesterday I asked similar question and found this similar question after getting replies in my own post. What people suggested me in my question is editing a node by `knife edit` and then setting an attribute for that particular node manually. This attribute would be assigned only to this node. Or the other way - you could create a wrapper cookbook that would override attributes of your "base" cookbook. For more details check out answers and comments for my question.
> 1 votes

---
Tags: chef
---


# Question
Title: Gitflow with pull request or pull request
My team is currently using gitflow workflow into develop branch.

Here is an overview for what we are doing. We are a sql server data warehouse team using Azure repos and Azure devops to build and deploy our code to SSIS, SSAS, and 30+ databases in our environment. I typically deploy to a integration server and then to a prod server. I have an auto build for our SQL server solution that triggers on checkin to the develop branch. I run manually a build that generate the deployment scripts. All the SSAS and SSIS builds and deployments are triggered on the checkin to the branch.

I am looking at trying to migrate to a workflow that would be a pull request so I don't have a broken build that holds everyone eases deployments. If someone breaks the build in the develop branch then I cannot deploy by the current method.

Prior to moving to the azure devops I was using a release workflow to create the deployment artifacts manually and I would deploy everything manually. This was a completely manual process and time consuming process that was very tedious.

I would like to figure out a path to get to using pull request. Does this mean I would do away with gitflow process? My team is currently using VSCode for creating features and this part works well. I am looking for advice for moving forward with pull request.

# Answer
You can set up Branch Policies on your develop branch which requires a build to pass successfully before the Pull Request can be completed.

I have set this up for my team with great success; we very rarely will have a broken master branch. The PR does not run the main build, but instead runs a separate, slimmed down build that checks that the code compiles and the unit tests pass. The "Gated" build runs on Pull Request creation, the "Full" build is triggered on change in master branch (PR completion).
> 1 votes

---
Tags: git, azure-devops, azure-pipelines
---


# Question
Title: Why is Prometheus not a good choice for data with high cardinality?
I have a background in relational databases and new to Prometheus. I wonder why Prometheus is not a good choice for high cardinality data? Why do I need to use low cardinality data? It's exact opposite from SQL DBs.

What are the technical reason for that?

# Answer
> 13 votes
As far as I know, Prometheus doesn't mind high-cardinality **data**. What Prometheus doesn't like is high-cardinality **labels**.

Let's start with Prometheus official documentation, it gives a good high-level explanation why:

> CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

The important part is really that **unique combinations of key-value** creates **new time series** in Prometheus.

For instance, if you store a gauge-type metric `registration_complete` that records the time a user took to complete his/her registration form, Prometheus won't have any issue with that. You'll have one time series that have hundred of thousand of values: one value for each user that registered (how long it took). You'll be able to graph that metric over time, get the p95, etc.

If you want to add *some* cardinality to it, you could add a label like `region`: US-east, Asia-Pacific, etc. You'll be able to graph all regions and compare them or maybe group them all. The number of regions is probably low (\<10) and it is certainly bound. If on AWS, there's a fixed number of regions and they don't change much over time. Sure, AWS might add, remove or rename a region, but that's not changing by the minute and there's not thousands of them.

So to come back to what Prometheus advise against: you should not create high-cardinality labels. You should not add a `user_id` label to your `registration_complete` metric. If you do so, you'll have hundred of thousands of *different* time series, one for each user!, and they will all have only one data point. That's really the worst case scenario.

In this case, in order to graph the `registration_complete` metric over all labels, Prometheus will have to query all the individual time series (the X thousands of them) and aggregate all of them.

You said you come from a SQL background, so I'll attempt an analogy. Unique combination of key-value label pairs creating new time series in Prometheus amounts to individual tables in SQL. Having a label `user_id` equals to having one table per user\_id.

Note: not all TSDB work the same way and I can't speak for them all.

---
Tags: monitoring, databases, prometheus, sql, data
---


# Question
Title: How do you monitor and alert on thrashing containers?
Are there any proposed best practices for how to monitor for thrashing containers? We're in a situation where we have containers that might try to come up and will crash -\> restart a few dozen times without anyone noticing. However, it's "normal" for containers to be so ephemeral, so tracking this sort of anomaly seems difficult (what if your workload really is just 30 seconds?)

I'm just wondering if there are any "good enough" alerting practices that anyone else has come up with for tracking unhealthy restarts vs healthy restarts?

# Answer
> 3 votes
Regardless if containers are ephemeral or not, there's two things you could consider *not normal*:

* Non-zero exit codes
* Restart count \> 0 at the orchestration layer

How you can alert on those metrics will vary wildly between orchestration layers, providers and monitoring tool. I'll keep the discussion about the *principles*:

## Non-zero exit codes

This one seems obvious enough. If you have an ephemeral workload, make sure it exits cleanly and you'll be able to monitor the health of your containers. Even if your ephemeral workload last 30 seconds and you launch thousand of them (small short-lived workers style), as long as everything exits cleanly, there's nothing to worry about.

## Restart count at the orchestration layer

Most orchestration layers will expose a restart count. For Kubernetes, that's literally the 'Restart Count' metric when describing a pod. Those restarts are the orchestration layer restarting a container because it wasn't supposed to be in a 'stopped' state, but it was. Ephemeral workloads won't have a restart count, because of their very nature (or if they do, the "final" shutdown of the container won't cause a restart when the workload is done).

Whether you want to alert as soon as the restart count crosses a certain threshold or if you want to be more clever about it and calculate a rate (roughly: restart count / pod/service duration) is up to you.

---
Tags: containers, monitoring
---


title: AWS Site crashes every few hours

text:
# Question
Title: AWS Site crashes every few hours
Every few hours, my Laravel website on AWS Elastic Beanstalk crashes and throws the following error:

```
[Sun May 19 13:06:13.436195 2019] [php7:error] [pid 3431] [client]
   PHP Fatal error:  require(): Failed opening required
   '/var/app/current/vendor/composer/../symfony/polyfill-php70/bootstrap.php'
   (include_path='.:/usr/share/pear7:/usr/share/php') in
   /var/app/current/vendor/composer/autoload_real.php on line 66

[Sun May 19 13:06:31.485081 2019] [php7:warn] [pid 3459] [client] PHP
   Warning:
   require(/var/app/current/vendor/composer/../symfony/polyfill-php70/bootstrap.php):
   failed to open stream: No such file or directory in
   /var/app/current/vendor/composer/autoload_real.php on line 66

```

In-order to fix the error, I either go to `/var/www/html` or `/var/app/current` and use `composer.phar install` to fix the packages installed.

Unfortunately, this doesn't cut it. Once I fix the issue, it breaks again a few hours later with the same errors. I tried to fix the packages associated with the deployed zip file. But that's to no avail either.

Is there something I'm misunderstanding about the AWS architecture? I'm new to AWS, any help?

# Answer
I resolved this issue by removing the vendor folder in the Laravel package.

Once the vendor folder is removed, the composer.json file is automatically used to run the installation. All necessary files are then downloaded and installed.

This removes the need to run the 'composer install' command post installation.
> 1 votes

---
Tags: amazon-web-services, amazon-ec2
---


title: KeyCloak, PrivacyIdea and Email Gateway

text:
# Question
Title: KeyCloak, PrivacyIdea and Email Gateway
I am doing a Micro-services application hosted in AWS to be accessed via Amazon API Gateway by end-users from several organisations worldwide. I am hoping to use KeyCloak as User Authentication Server (2FA). Keycloak will generate a JWT token for the authenticated user and pass the JWT token to the AWS API Gateway to authorise access to the micro services.

Keycloak will be hosted in an AWS Public VPC subnet. In addition, I will have PrivacyIdea as OTP code generator and dispatcher in another subnet or maybe the same subnet in the same VPC. I will also have an Email Gateway in same or another VPC subnet.

Privacy Idea will generate the OPC codes, probably using HOTP and dispatch them via the Email gateway to the users mailbox. The user's corporate email address will be setup in PrivacyIdea. These same users will be setup up in Keycloak. My idea is that somehow the PrivacyIdea-generated and emailed OTP codes will be inputed by the users along with their login-ID and password into a Keycloak-generated login form, which will then authenticate the user.

Keycloak as authentication server and the use of Emailed OTP codes is an architectural choice that I have no room to change.

My problem is that Keycloak only directly supports Google Authenticator or FreeOTP mobile SMS or application based OTP; hence my having to introduce PrivacyIdea and an Email Gateway. Can anyone kindly provide me any technical guidance or experience on how the OTP seeding and communication of OTP codes from PrivacyIdea or OTP database sharing can be done.

Thank you.

# Answer
There is no need to push any OTP values into the keycloak database. You can use the new privacyidea keycloak provider.

The user logs in with his username/password and the privacyidea-provider will take care, that the users get the email and ask in a 2nd dialog for the OTP.

See: https://github.com/privacyidea/keycloak-provider
> 1 votes

---
Tags: amazon-web-services
---


# Question
Title: Is there a [ci skip] option in GitLab CI?
I'm debugging the use of a report creator. I forgot a line and would like to edit the code without triggering the build pipeline.

I know that it's possible to write \[ci skip\] in the commit pipeline to pass the pipeline execution in Jenkins CI, but I'm using GitLab CI.

Does the same mechanism exist for GitLab CI? I couldn't find it in the docs.

# Answer
Yes, there is:

> If your commit message contains `[ci skip]` or `[skip ci]`, using any capitalization, the commit will be created but the pipeline will be skipped.
>
> Alternatively, one can pass the `ci.skip` Git push option if using Git 2.10 or newer:
>
> ```
> git push -o ci.skip
>
> ```

From: GitLab CI Pipelines documentation \- "Skip Pipeline"
> 95 votes

---
Tags: continuous-integration, gitlab, build-pipeline
---


# Question
Title: How to have kubernetes (re)build containers on deployment similar to openshifts ImageStream
I am currently investigating how to manage the containers for our new data-processing pipeline.

What I would like to do is to be able to have whatever tool I choose rebuild the docker images when I have a new version ready in git (without me having to come up with a name for the docker image). I have found that OpenShifts ImageStreams does this but OpenShift feels like an intrusive dependency so I am searching for a well supported way of doing this with a more clean kubernetes install.

I have looked at helm but did not find this feature.

So

What is the most mature and widely supported tool that can build the containers automatically during the deployment to kubernetes?

# Answer
You need something that will do basically CI/CD. There are few options (in no particular order):

There are more, but basically, you need a component to watch for changes, something that knows how to build an image, and something else that can deploy it (some of these might be done by the same tools)

Helm will only do the deployment, but nothing else
> 2 votes

---
Tags: kubernetes, automation, builds
---


# Question
Title: Ansible - stop when host is success, exit with IP
Basically, my use case is that I want to run on list of hosts (group from inventory).

It will run one by one, until it finds machine answer my standards, when it find the one with free RAM as I say e.g 40GB free, stop looping host, continue the next role/task with the IP address of this machine.

# Answer
The thing that makes this complicated is that ansible registers variables and facts locally, on the target remote host - and then has a different set of variables and facts once it moves to the next host. So it would require a bit of fiddling to do this exactly the way you want.

One way to approach this would be to just use run\_once, but it doesn't work with `serial: 1` which you are asking for (check one host at a time);

```
---
  - name: test
    hosts: all
    vars:
      mem_to_look_for: 62950
    tasks:
      - name: do things
        debug:
          msg: "hello"
        run_once: true
        when: ansible_memory_mb.real.total==mem_to_look_for

```

If you really want it to run serially, we could borrow some of the knowledge out of this article https://stackoverflow.com/questions/42376123/how-to-terminate-an-ansible-playbook-run-when-it-is-successful-on-one-host and tweak it a little.. and it works great!

```
---
  - name: "test"
    hosts: all
    gather_facts: no
    serial: 1
    vars:
      mem_to_look_for: 62950
    tasks:
      - meta: end_play
        when: "{% for item in hostvars.values() %}{% if item.done|default(False) %}True{% endif %}{% endfor %}"

      - name: gather facts
        setup:

      - name: ensure host has the memory we're looking for
        set_fact:
          done: true
        when: ansible_memory_mb.real.total==mem_to_look_for

      - name: skip this host if it doesn't have the memory we're looking for
        meta: end_play
        when: ansible_memory_mb.real.total!=mem_to_look_for

      - name: run this thing only on one host
        debug:
          msg: "running the thing"

```

I set the playbook to gather\_facts: no because we don't want to continue gathering facts once we found the host we're looking for. It doesn't abort the playbook unfortunately, but just kind of runs a no-op on the remainder of hosts. You could also probably speed things up by tweaking the `setup:` module by specifying which subset of facts you want to collect, but I left it to collect all the facts in this example. https://docs.ansible.com/ansible/latest/modules/setup\_module.html you may also want to just move the host selection to the setup module by adding `failed_when: ansible_memory_mb.real.total!=mem_to_look_for` at the end of the setup module, but then the hosts would be marked as failed. I'm not sure if that's what you're looking for, but it's really up to you on if you want them to be failed or just skipped.

Additionally, if you want it to completely stop checking hosts as soon as its done I think the any\_errors\_fatal property will do that. https://docs.ansible.com/ansible/latest/user\_guide/playbooks\_error\_handling.html#aborting-the-play
> 1 votes

---
Tags: ansible, ansible-inventory
---


# Question
Title: Unable to create container after installing mysql and phpmyadmin | Ubuntu
I have installed docker successfully and try to install phpmyadmin and mysql.

I have did following steps. Using below link I have pulled php version

```
https://hub.docker.com/r/codepulse/php-7.2.0-apache-mysql/

```
1. docker pull codepulse/php-7.2.0-apache-mysql
2. docker pull phpmyadmin/phpmyadmin

After installing it when I execute `docker ps -a` I could not found any container there.

Could any one help to install phpmyadmin and mysql.

# Answer
Pulling the images only make them available for you to use locally. Therefore, to use them within a container you should run the command `docker run` with the appropriate configuration for each services
> 1 votes

---
Tags: docker, mysql
---


# Question
Title: Why many companies prefer Jenkins than Aws CodePipeline?
I have searched few job sites. Many companies are searching engineers who can handle Jenkins as CI platform. I think the Aws CodePipeline is easier than Jenkins.

Anyone can advise me, please. Thank you in advance.

# Answer
Jenkins has been around for longer so it has a larger installed base. It doesn't matter if AWS has something that is cheaper and better. Many places don't want to put the time and energy into upgrading Jenkins. Migrating to a new CI pipeline is even "scarier" -- regardless of whether it would be a wise strategic decision.

Jenkins 1.x made this far worse because of needing extensions for basic functionality. Different departments wanting different plugins that wouldn't play well together led to multiple Jenkins instances. Even if the technical challenge of merging these were relatively minor, the chance of getting departments that could never agree on anything before to suddenly decide that they can migrate to some new thing -- that is missing some feature from a plugin they can't live without -- seems pretty slim.
> 3 votes

# Answer
Maybe cost is a factor? And it's open-source? I would think those two factors alone make it popular amongst startups.

I work in an enterprise environment and we use AWS a lot but still use Jenkins as our main build tool.

Sure there are better, more polished options, but like I said... cost and open-source, and also many people are used to it already.
> 1 votes

---
Tags: jenkins, jenkins-pipeline, aws-codepipeline
---


title: Spot block vs reserved instances

text:
# Question
Title: Spot block vs reserved instances
Let's say we have a continuous process where each two hours we receive some data which has to be analysed by a fleet of workers on AWS EC2 instances which always complete their job within two hours (let's say normally within 90 minutes).

What would be the most cost-effective solution in this case?

1. Use Spot blocks with finite duration of 2 hours?
2. Use fleet of reserved EC2 instances?

Though spot blocks seem to be less expensive, I am concerned about their availability as data has to be processed immediately after receiving within the next 2 hours.

# Answer
> 1 votes
Anything that "needs" to happen in a given time frame is not a candidate for spot instances, including spot blocks.

Spot capacity is, by definition, surplus/idle capacity, subject to availability, so your spot block may simply fail to start, as capacity may not be available. It is less expensive for exactly that reason -- AWS is selling surplus capacity for which no one is currently willing to pay more.

Information about the pool of compute capacity that is currently flagged internally as eligible for participation in the spot market is not publicly accessible, and although it was possible to make some reasonable assumptions in the past about capacity based on signals you could obtain from the real-time pricing, the spot pricing algorithm was modified in 2017 in a way that largely masks those signals (and the change, in my opinion, was a good thing for any customer who was not aggressively gaming the system).

Spot blocks are intended to run for a committed period of time, so they are less likely to be interrupted than other spot instances, but they can initially start up successfully and then subsequently be interrupted at any time.

> In rare situations, Spot blocks may be interrupted due to Amazon EC2 capacity needs. In these cases, we provide a two-minute warning before we terminate an instance, and you are not charged for the terminated instances even if you used them.
>
> https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/spot-requests.html#fixed-duration-spot-instances

---
Tags: amazon-web-services, amazon-ec2
---


# Question
Title: CI/CD pipelines Azure devops automatic merge after deploy release
I have a classic env. setup like following:

I have 2 branches : Develop and Master.

Is there any way in `azure devops` to setup the following **rule:**

1. When a deploy is succeeded on dev environment (defined in the release pipeline of azure devops) **------\>** create automatically a `pull request` to merge **develop** into **Master**.
2. or the other one: if a `Build` of **develop branch** is succeded **-------\>** create **automatically** a `pull request` to merge **develop into Master**.

Any help will be appreciated.

# Answer
> 1 votes
A few questions so I can better understand your use case:

* How often would this deployment/build of yours be run?
* Will there ever be any changes going into **Master** that do not come from **develop**?

If you plan to run this deploy/build which would create a PR to merge to **Master** for each change in **develop**, then I would recommend to delete the **develop** branch and instead develop directly against **Master**. Configure **Master** with Branch Policies enabled that would run a gated build (compile/unit test) before the PR can be merged. This is what my team has been doing for a while now with great success. This change in workflow would prevent the need for having a deploy/build which automatically creates a PR.

If you plan to run this deploy/build less frequently than every change, then I would recommend to use the VSTS APIs and a custom scripting task to automatically create a PR. I believe the scripting task would be fairly similar regardless of whether a deploy runs it or the build runs it. The documentation for VSTS APIs is fairly comprehensive, and I use several of them as part of our pipeline workflows (I use PowerShell). Here is documentation to Create a Pull Request. This would be run as the last task in the deploy/build.

If you plan to merge to **Master** from areas other than the **develop** branch, I just want to call out that the PR which will be automatically created by deploy/build could have conflicts and require manual intervention.

# Answer
> 1 votes
I would look into Azure Functions. Create a gate trigger and enable azure function after a successful release.

References: https://docs.microsoft.com/en-us/azure/devops/repos/git/create-pr-status-server-with-azure-functions?view=azure-devops

---
Tags: git, azure-devops, build-pipeline, azure-pipelines, branch
---


# Question
Title: Azure Devops: Filter Test Projects in Nuget Build Pipeline
I'm working on a nuget build pipeline. My problem is that my integration test project is being packaged and deployed to the azure artifacts, while my unit test projects are not.

The integration test project is a console application. I want it in the project, but I don't want it to deploy into the artifacts.

My Projects in the solution:

* MyProject.Common
* MyProject.Common.UnitTests
* MyProject
* MyProject.UnitTests
* MyProject.IntegrationTests

I am using .net core v2 steps

Restore
Build
Test
Publish
Pack
Push

Test Path to project (works)

> `**/*[Uu]nit[Tt]ests.csproj`

Publish Path to projects (works)

> `!**/*[Tt]ests.csproj`
> `**/*.csproj`

Pack path to project (does not work - filters unit test projects, but not integration test)

> `!**/*[Tt]ests.csproj;**/*.csproj`

# Answer
> 3 votes
I noticed that the description for the filepath properties of the **pack** command is different than the descriptions in the **test** and **publish** commands:

```
Pattern to search for csproj or nuspec files to pack.

You can separate multiple patterns with a semicolon, and you can
make a pattern negative by prefixing it with '-:'. Example:
**/*.csproj;-:**/*.Tests.csproj

```

So perhaps try `-:**/*[Tt]ests.csproj;**/*.csproj` instead. I don't think this explains why the unit tests are excluded and the integrated tests are not, though.

---
Tags: azure-devops, build-pipeline, azure-pipelines, dotnet
---


title: Host a docker registry native

text:
# Question
Title: Host a docker registry native
I want to host my own docker-registry server to be independent from docker-hub.
I want to run it native on my server, but I can only find documentations, which are using running it inside a container.

My question:
Is it possible to install docker registry over a package or do i have to run it in a container?

# Answer
> 0 votes
I take for granted you are referring to the registry server released by docker.com. The documentation clearly states on the very first line:

> Before you can deploy a registry, you need to install Docker on the host

Since the registry is the main product of the docker/distribution opensource project, you could fork and see how you can modify this as a standalone install not requiring docker, but I have no clue if it is possible and that would probably end up being a bad idea.

There are other existing products you can install 'standalone' that will let you manage your own registry. But they all come with some degree of requirements (java, ruby...). I have experience with 2 free products I use daily:

* Sonatype nexus3 oss is a multi format repository manager supporting docker registries (requires java).
* gitlab-ce has support for container registries in each projects if you configure it (ruby required).

The latest one is probably an overkill if you only need a basic registry and not all of the git, wiki, etc. ressources.

I recently came across a fairly recent review article listing other products I have no experience with:

It also mentions as a bonus the use of Openshift as a registry, but even minishift for something as light as possible will require to install docker.

All of these solutions can probably meet your 'native run' requirements, but they are all much more complex to install and maintain than simply poping-up a `registry:2` container.

# Answer
> 0 votes
**DEPRECATED PYTHON**

Originally you could install `docker-registry` via `pip` https://github.com/docker/docker-registry

But this was deprecated with the new generation of `docker-registry` in `golang`

> Notice: The classical python "Docker Registry" is deprecated, in favor of a new golang implementation. This here is kept for historical purpose, and will not receive any significant work/love any more. You should head to the landing page of the new registry or the "Distribution" github project instead.

Here is an old guide that I wouldn't recommend.
**OLD GUIDE (2014):** https://www.ceondo.com/notes/docker-registry/

---

**LINUX SNAPS**

You can also look into Linux Snaps
**LINK:** https://snapcraft.io/docker
**GUIDE:** https://www.allprogrammingtutorials.com/tutorials/installing-docker-on-ubuntu-centos-using-snap.php

```
# Lookup docker snap
snap info docker

# Install from stable channel
sudo snap install docker

# Install from a specific channel for deploying older versions
sudo snap install --channel=<channel-name> docker

```

---

**USING DOCKER**

The fastest way to get a private docker registry server running is to install docker and then just run a detached container

```
sudo apt-get update
sudo apt-get install docker.io
sudo systemctl start docker
sudo systemctl enable docker
docker --version
docker run -d -p 5000:5000 -v /registry/images:/var/lib/registry --restart=always --name registry registry:2

```

You may need to setup persistent volumes to store the library images.

I'm currently using docker installed on a Synology DiskStation to host my own private docker registry container

---
Tags: docker, docker-registry
---


# Question
Title: Interacting with a Trident container (NetApp) via CLI within a Openshift/Kubernetes cluster?
### Background

We're currently using NetApps Trident PV (Persistent Volume) orchestrator to dynamically create volumes on our NetApp ONTAP clusters for Pods running within our Openshift/Kubernetes environment.

**NOTE:** See links in the references section if you're unfamiliar with Trident + NetApp.

### 1. Native Built CLI

Most people will use the pre-built binary but this will only run `tridentctl` on Linux. Previously we'd been using the `tridentctl` command natively on a Linux system to interact with the Trident controller running within our cluster.

However, I'd like a more native method to do this from macOS. Currently NetApp does not ship a prebuilt `tridentctl` binary to do this.

### 2. Using Docker Image

Previously we'd been using the `tridentctl` command natively on a Linux system to interact with the Trident controller running within our cluster.

Here's my Trident Pod:

```
$ oc -n trident get pods
NAME                       READY     STATUS    RESTARTS   AGE
trident-6bdbdbc5dd-l8qtr   2/2       Running   14         6d

```

Remote shell into the Pod:

```
$ oc -n trident rsh trident-6bdbdbc5dd-l8qtr
Defaulting container name to trident-main.
Use 'oc describe pod/trident-6bdbdbc5dd-l8qtr -n trident' to see all of the containers in this pod.
/ #

```

Attempt to run `tridentctl`:

```
/ # tridentctl get backend
Error: could not find the Kubernetes CLI
/ #

```

### 3. Other Methods

I explored my options in vain but did not care for any of these:

* Build Trident from source
* Construct a new Dockerfile based on the Trident Docker image
* Roll my own Docker image with `tridentctl` \+ `kubectl`

### References

# Answer
> 1 votes
There's apparently 2 methods one can use to leverage `tridentctl` to interact with a running Trident Pod in their Openshift/Kubernetes cluster.

### 1. Server string

The `tridentctl` CLI can be instructed to talk to whatever server you want remotely using the `-s` or `--server` argument. In this context you could use the approach of remote shelling into the Trident Pod and then directing it to interact with the local Trident Orchestrator daemon.

***For example***
```
$ oc -n trident rsh trident-6bdbdbc5dd-l8qtr
Defaulting container name to trident-main.
Use 'oc describe pod/trident-6bdbdbc5dd-l8qtr -n trident' to see all of the containers in this pod.
/ #

```

Now to interact:

```
/ # tridentctl -s 127.0.0.1:8000 get backend
+--------------------------+----------------+--------+---------+
|           NAME           | STORAGE DRIVER | ONLINE | VOLUMES |
+--------------------------+----------------+--------+---------+
| ontapnas_192.168.101.101 | ontap-nas      | true   |     389 |
+--------------------------+----------------+--------+---------+
/ #

```

### 2. Creating a Bash Function

To expand on the above method, one could construct a shell function or alias in Bash to streamline this interaction like so:

*Openshift*
```
tridentctl () {
    oc rsh -t -n trident -c trident-main \
        $(oc get pods -n trident -l app=trident.netapp.io -o jsonpath="{.items[].metadata.name}") \
        tridentctl -s 127.0.0.1:8000 $*
}

```
*Kubernetes*
```
tridentctl () {
    kubectl exec -t -n trident \
        $(kubectl get pods -n trident -l app=trident.netapp.io -o jsonpath="{.items[].metadata.name}") \
        -c trident-main -- tridentctl -s 127.0.0.1:8000 $*
}

```

With one of these functions set, you could then run the following command from your macOS system:

```
$ tridentctl get storageclass -o json
{
  "items": [
    {
      "Config": {
        "version": "1",
        "name": "basic",
        "attributes": {
          "backendType": "ontap-nas"
        },
        "storagePools": null,
        "additionalStoragePools": null
      },
      "storage": {
        "ontapnas_192.168.101.101": [
          "NA_01_aggr1",
          "NA_01_aggr2"
        ]
      }
    }
  ]
}

```
***How it works***

The above function merely determines the name of the Pod within the `trident` namespace and then remotely executes the command `tridentctl` using the same method above (from method #1) of connecting to the locally running Trident Orchestrator daemon within the Pod using the `--server` string.

The advantage with one of these approaches is that you simply can use a function/alias to codify it.

***Caveats***

Keep in mind that with the above shell function/alias approach will not work with commands such as this:

```
$ tridentctl create backend -f <backend-file>

```

because the files you'll be referencing with it, `-f <backend-file>`, will be local to the system where you'll be running the function/alias, but the `tridentctl` command will actually be executing within a Pod on another system. This Pod will not have access to your local system's filesystem.

Your only options in this situation will be to either copy the file into the Pod's `/tmp` filesystem and execute it from there or to redirect the contents of the file, `<backend-file>` through a pipe, (`|`), for example:

```
$ cat <backend-file> | tridentctl create backend -f -

```

---
Tags: kubernetes, openshift, storage, persistent-volumes
---


# Question
Title: Jenkins plugin site to ignore certificates
Is there a way that I can tell Jenkins to ignore certificates for the update site.

**Background**

I have an air-gapped Jenkins installation and there is currently proper way to make a proxy repo. Thus I have the following setup: \- I have modified the `hosts` file on the Jenkins host so that `updates.jenkins-ci.org` points to the host of the internal artifact repo \- There is a reverse-proxy in front of artifact repo which handles traffic for `updates.jenkins-ci.org`

However we are only allowed to use https thus there will be a `name mismatch error` as the certificate does not match the hostname of the artifact repo host

Remark: I don't want to generally disable the checks but only for the update site so the `The Skip Certificate Check plugin` is not the solution for me

# Answer
Following your comment confirming you can use your own signed certificates internally.

One solution would be to replace the current certificate on your repo with a new one holding `updates.jenkins-ci.org` as an alias. This response on stackoverflow has the necessary info to get you going.

Meanwhile, the plugin higlighted above by @casey vega in the comments (which I didn't know) seems to be a solution much easier to manage.
> 0 votes

---
Tags: jenkins
---


# Question
Title: What does 1 vcpu mean precisely/exactly?
When we are talking about "1 vcpu" in different cloud solutions:

**Q:** What exactly does it mean?

Ex.: if I will use a 10 vcpu machine for the same cpu intensive process (which only uses CPU) then I would see the same amount of CPU load on them? Would they finish at the same time regarding the same only-CPU-usage task?

Or the definition of vcpu could differ in the different cloud solutions?

# Answer
vcpu = Virtual CPU, that's just the number of core of the instance. Nothing more, nothing less.

For the rest of your questions they're unrelated, this depends on the underlying hypervisor's physical processors, there's no guarantee they use the same type with same frequency.
> 1 votes

# Answer
The definition can definitely vary, for example AWS defines vCPU as:

> AWS: Each vCPU is a *thread of a* CPU core, *except for ...*

Similarly, Azure and Google also tend to match a vCPU to a *hyper-thread* rather than always 1:1 with an actual core.

> Google: For the n1 series of machine types, a vCPU is implemented as a single hardware hyper-thread on one of the available CPU Platforms.

Generally, vCPU is used as a metric for simple comparison of machines within a vendor's range of machines.

It may also used a cost equivalence factor (e.g. within the same machine type family, a 4 vCPU machine would be twice the cost of 2 vCPU machine)

How well a certain load can fully utilise a machine's resources, and how performance between different machine families and vendors, depends on multiple factors. So if it's important to know for sure then it would be best to run some real tests; otherwise you can take the vCPU count as an approximate indicator.
> 1 votes

---
Tags: cloud
---