# Question
Title: Do you use existing automation code (cookbooks/playbooks/dockerfiles/etc) or you write your own from scratch?
I noticed that many cookbooks/playbooks/... are not of particularly high quality, and even if they are they rarely cover all cases. So it should be expanded with my custom logic everytime, while 90% of the \*book remains unused (+ adds complexity/reduces ease of modification/lacks unification between \*books/...).

In particular, configuration management most of the time is "high-level enough" to not write too many boilerplate even if you implement something from scratch. The only exception might be additional "pure modules" for specific use cases if DSL does not provide enough functionality.

I understand this is pretty subjective, but I'm truly surprised I can't find anything related to this (pretty basic) topic. What is your experience, can we say there is a best practice for when to implement your solution from "scratch" or use the existing infrastructure/deployment automation code?

# Answer
My limited experience is indeed that often, you can get a first solution very quickly with pre-made automation code; unfortunately, often there either appear unexpected problems during "first contact", or later during use, especially when you wish to expand the configuration for unusual cases.

So, indeed, I do tend to approach this with an open mind. I will certainly start out with something pre-made, but sometimes my gut feeling tells me to take a look under the hood, and then I'll decide rather spontaneously if I better do it myself, or whether it is a good idea to roll with it...
> 2 votes

---
Tags: configuration-management, infrastructure-as-code
---


# Question
Title: Linking VSTS build with jenkins artifacts
So I have a local jenkins server building artifacts. I trigger this build through VSTS. When browsing through the logs when uploading the artifacts to VSTS from the local jenkins server I found this: `Downloading content from Jenkins server:http://[ip]:[port]//job/[Folder Name]/job/[Build Definition Name]//api/json?tree=lastSuccessfulBuild[id,displayname] with strict SSL:true`

This could pose an issue where the triggered build may get beat out by another build (theoretically). While this is not a high likelihood, I would like to mitigate all possible artifact confusions as that is one of the main points my company is moving to build automation and continuous integration. I did see something related to the VSTS Task in issue 4110 but not sure what is coming of this.

What I'm trying to accomplish is that when Jenkins finishes the build it uploads it to VSTS. Because of the size of the build, when we go to releases, I would like to not download the artifacts back to our local premise, (using deployment groups) but instead copy it internally directly from the Jenkins build server. Does anyone have any suggestions?

# Answer
> 0 votes
So I wrote this powershell script to be ran by the deployment group agents who have access to the server locally.

```
Param(
[string]$JenkinsPatArg
)
$isRunningLocal = $false
$jenkinsUrlBase = "[public ipaddress]"

if(Test-Connection -ComputerName "[private ipaddress/DNS name]" -Quiet)
{
    $isRunningLocal = $true
    $jenkinsUrlBase = "[private ipaddress/dns name]"
}

$buildDefinitionName = $env:RELEASE_ARTIFACTS_GAMEZIP_DEFINITIONNAME
$buildTeamProject = [Uri]::EscapeDataString($env:SYSTEM_TEAMPROJECT)
$targetBuildNumber = $env:RELEASE_ARTIFACTS_[Artifact Alias]_BUILDNUMBER

$Auth = "$($env:username):$($JenkinsPatArg)"

$Bytes = [System.Text.Encoding]::UTF8.GetBytes($Auth)
$Base64bytes = [System.Convert]::ToBase64String($Bytes)
$Headers = @{ "Authorization" = "Basic $Base64bytes"}

Write-Host "Searching for $($targetBuildNumber) in project $($buildTeamProject)->$($buildDefinitionName)"

$correctBuild

Write-Host "Querying: http://$($jenkinsUrlBase):[port number]/job/$($buildTeamProject)/job/$($buildDefinitionName)/api/json?tree=builds[id,url,actions[buildNumber]]"

ForEach($build in (Invoke-RestMethod -Uri "http://$($jenkinsUrlBase):[port number]/job/$($buildTeamProject)/job/$($buildDefinitionName)/api/json?tree=builds[id,url,actions[buildNumber]]" -ContentType json -Headers $Headers).builds)
{

    if($build.actions.buildNumber -eq $targetBuildNumber)
    {
        $correctBuild = $build
        Write-Host $build.id
        Write-Host $build.url
        Write-Host $build.actions.buildNumber
        break

    }
}

if($correctBuild -eq $null)
{
    Write-Error "Build not found.  Must rebuild source to obtain artifact"
    return;
}

if($isRunningLocal){$correctBuild.url = $correctBuild.url.Replace("[public ip address]", "[local ipaddress/DNS name]")}

$targetZip = "$Env:temp/$($buildDefinitionName)_$($correctBuild.id).zip"

(Invoke-RestMethod -Uri "$($correctBuild.url)/artifact/$($buildDefinitionName)/*zip*/$($buildDefinitionName)_$($correctBuild.id).zip" -ContentType Application/zip -Headers $Headers -OutFile $targetZip)

Add-Type -AssemblyName System.IO.Compression.FileSystem;[System.IO.Compression.ZipFile]::ExtractToDirectory($targetZip, $env:targetDirectory);
#Expand-Archive -Path $targetZip -DestinationPath $env:targetDirectory -Force
Remove-Item $targetZip;

```

passing in the a personal API key (created by Jenkins user manager to use by visual studio team services), this script first checks if it is can access the server using local connections, if so set the variables appropriately.

From there we create local handlers for environmental variables. Our release artifact name defined in VSTS is the same as that in our jenkins server. The build number used is created by VSTS and passed to jenkins and stored by the VSTS plugin(Jenkins TFS Plugin download). We get this version number from the environment as the artifact build number. We also ensure that our team project path url is windows URL encoded brief explanation. Next we hold our authentication which is stored as a variable in our VSTS variables. The `$JenkinsPatArg` is marked as secret. As such, we must pass it in as it isn't visible as an environment variable. We then encode the authentication and add it to our request headers.

We then add a debug line to print what target we are looking for and what project we are looking in, just to make sure we are looking for the right thing. We create a variable to hold the build information that we find and do one more debug write stating what are query url is so that we can manually run it if we run into any issues.

Next we then run this url query against the build server and iterate over all the builds returned. The query filters out all the JSON to just the build ID, it's absolute url, and the VSTS build number. We check to see if the build we are examining has the same build number as the artifact we are deploying, if so store it in the `$correctBuild` variable, print the was found for debugging, and exit the loop.

Coming out of the loop we check to see if we found a result. If we did not find a build (due to it being cleaned up most likely) we inform the user that the artifact needs to be rebuild from source. Otherwise we download the artifact zip to a temporary location from the jenkins server and extract it to the target directory, defined as a variable. Lastly we delete the downloaded zip.

This is ran as a powershell script task and due to it's length, can not be an inline script. So to deploy this, i have a devops git repo for my scripts and I just include this as an artifact that is downloaded and ran. This task is preceded by other tasks which stop the current process from running, and delete the current files in place before running this task. After this task is ran, more scripts are ran to run the downloaded artifacts.

I hope this helps someone else trying to coordinate and link VSTS builds with Jenkins artifacts.

---
Tags: jenkins, azure-devops
---


title: How to properly scale Jenkins?

text:
# Question
Title: How to properly scale Jenkins?
In my project we have one AWS server running Jenkins Master + 1 Jenkins slave (2 executors)... and we need more
In order to augment our build power we have three options:

1. **Scale up**: Make AWS instance bigger and add more executors.
2. **Scale up**: Make AWS instance bigger and add another jenkins slave process.
3. **Scale out**: Create another AWS instance with a jenkins slave and connect it to master

We want to do 2. as we are in a big organization and our current Jenkins Master has already access to every place he needs. Option 3. "New server" is complicated as it needs more bureaucratic approvals that will take weeks.

So my questions are:

* **Is there any technical issues in option 2?**. Maybe the executors of each jenkins slave are not aware of the other slave executors?
* In general, **what is the best approach to scale Jenkins?** Scaling up or scaling out?

# Answer
There are no *fundamental* technical issues with running multiple jenkins slaves on the same machine. In fact Running Multiple Slaves on the Same Machine lists several good reasons for doing it:

> While the correct use of executors largely obviates the need for multiple slave instances on the same machine, there are some unique use cases to consider:
>
> * You want more configurability between the configured nodes. Say you have one node set to be used as much as possible, and the other node to be used only when needed.
> * You may have multiple Jenkins master installations building different things, and so this configuration would allow you to have slaves for more than one master on the same box. That's right, with Jenkins you really can serve two masters.
> * You may wish to leverage the easiness of starting/stopping/replacing virtual machines, perhaps in conjunction with Jenkins plugins such as the Libvirt Slaves Plugin.
> * You wish to maximize your hardware investment and utilization, at the same time minimizing operating cost (e.g. utility expenses for running idling slaves).

*In general* scaling out is preferred, primarily because ability to scale up is typically limited by the types/sizes of the available physical resources.

In particular for augmenting build power I'd recommend an analysis of your **actual** build to determine how it uses the machine resources, which/where its bottlenecks are and what scalability limitations it raises to reveal if scaling up even helps.

For example I encountered cases in which the build time for 2 parallel builds on the same machine was longer than the combined build times of the same 2 builds executed sequentially (non-overlapping) on the same machine. In such case I wouldn't even consider scaling up as it would actually *decrease* the overall building capacity.
> 14 votes

# Answer
Use Kubernetes and Helm.

I would recommend using the Jenkins Helm chart. It installs with `helm install stable/jenkins` and automatically scales.

https://github.com/kubernetes/charts/tree/master/stable/jenkins
> 7 votes

# Answer
I think you should do neither ;)

Well kinda. I think you need more executors, maybe your builds are really resource intensive? I would run at least 4 but we run 6 to 8 depending on jobs. I like to match # of cores to exectors. So you might want to scale up your nodes, I think we run a M4 large for our 4-8 executors.

I also think you should scale-out but you should do so smartly. Jenkins has a plugin to automatically scale-out on AWS depending on what is in the build queue. Basically you tell it how many jobs and how long the wait before it stands up a slave and sends the jobs to the new slave. You can also set the max amount of slaves, the min amount etc.
> 4 votes

# Answer
I'd scale out instead of scale up, going for option 3. We have made a setup where we have all Jenkins agents running on an ECS (custom Docker based Jenkins) with an auto-scaling group. We have all our Jenkins masters communicating to the ECS, thus sharing the workload on the ECS, and no need to recreate the Jenkins master in a scale up exercise.
> 2 votes

---
Tags: jenkins, continuous-integration, capacity-planning
---


# Question
Title: Running additional commands on Docker image startup
As far as I know, it's a good practice to use containers from Dockerhub instead of making my own ones. Most of those contain `docker-entrypoint.sh` script that is executed on startup. However, sometimes I need to do something on startup in addition to what is already described in `docker-entrypoint.sh`. Namely, I'm setting up a RabbitMQ container, I want to use a bind mount to save configs, but the config directory has to be owned by `rabbitmq` user. I wish I could execute `chown -R rabbitmq /etc/rabbitmq` on startup in addition to `docker-entrypoint.sh`, but I don't know how to do that.

I understand that I can simply clone the repo with Dockerfile and entrypoint from git, add something to the entrypoint and build docker image manually, but it looks like a crutch. I can also just attach to the container and execute that command manually which also looks like a crutch.

# Answer
> 1 votes
Simply write a `Dockerfile` which uses that image as a base. Then you can replace individual files (like the entrypoint) easily.

Also, instead of replacing, you could rename the original entry script, create your own, which does whatever it needs to do, and then just calls the original.

Somewhat like this:

* Dockerfile:

```
FROM fancy-base-image:1.2.3
RUN mv docker-entrypoint.sh docker-entrypoint-original.sh
ADD docker-entrypoint.sh

```
* docker-entrypoint.sh:

```
#!/bin/bash
echo Do something.
./docker-entrypoint-original.sh
echo Do something else

```

Obviously this is just a fake example, you'll have to adjust for directory and file names and such.

# Answer
> 0 votes
One could fix the issue by creating a pull request

Or one could find another github project that has less or no issues

Or one could adjust or create an own image

> I wish I could execute chown -R rabbitmq /etc/rabbitmq on startup in addition to docker-entrypoint.sh, but I don't know how to do that

Once the docker container runs (one could verify this by issuing `docker ps`), one could run `docker exec -it container /bin/bash` and run commands. Note: I personally prefer to investigate why the owner and group is not set to rabbitmq instead of fixing it in a running container as it is mutable.

---
Tags: docker
---


# Question
Title: How to avoid the flakyness that comes with "gpg --recv-key"?
If you want to verify a signature of a downloaded file with gpg, you first have to import the key. Unfortunately, this operation is extremely slow and flaky in practice.

For example, here I tried it out:

```
$ gpg --keyserver pgpkeys.mit.edu --recv-key A0E98066
gpg: keyserver receive failed: No data

$ gpg --keyserver pgpkeys.mit.edu --recv-key A0E98066
gpg: key B550E09EA0E98066: public key "Yichun Zhang (agentzh) <agentzh@gmail.com>" imported
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:  14  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 14u
gpg: next trustdb check due at 2019-01-13
gpg: Total number processed: 1
gpg:               imported: 1

```

The first run failed. When I repeated, it took very long but eventually succeeded. Especially, in a CI environment, it breaks a lot, that is why I am looking for alternatives.

Question:

* How do you avoid the dependence to the remote keyserver?
* Is there an easy way to export the key as a file and use it later to import it in the CI environment without making a request to the keyserver?

# Answer
Download the key once:

```
gpg --keyserver pgpkeys.mit.edu --recv-key A0E98066
gpg --export A0E98066 > openresty-agentzh-A0E98066.gpg

```

Then later you can import the key without having to access the keyserver:

```
gpg --import openresty-agentzh-A0E98066.gpg

```
> 4 votes

# Answer
Another option, if you trust your repo or package source is to disable gpg key checking. This is especially viable if you are connecting to a repo over SSL as you can verify you are pulling the correct source and there has been no MITM attack, though you lose verification of the integrity of each individual package (in the case the repo was hacked and a rogue package inserted or package is tampered with). To do this set in your repo file:

gpgcheck=0
> 2 votes

---
Tags: continuous-integration
---


# Question
Title: Trigger jenkins pipeline job with Bitbucket hook
I'm missing something obvious here.

Is there no way to trigger a Jenkins scripted pipeline job via Bitbucket hook?

There's a bug out about it but it's been around for years with apparently no fix. Is the only way to make this work really to make an upstream trigger job that kicks off the pipeline? That seems insanely clunky.

# Answer
If you just need generic post-receive hooks, you can just do a regular web hook and hit the build API endpoint. For instance, if the name of your Jenkins server is `jenkins.example.com` and the name of the Pipeline job to trigger is `my_awesome_job`, then you can fire off a POST request to `https://jenkins.example.com/job/my_awesome_job/build`. Depending on your security model, this request may or may not need to be authenticated and authorized. If you do need authentication and authorization, you can use HTTP Basic auth (unfortunately I haven't found a reasonable way to do keypair auth).

If you need to trigger builds on pull requests, then you'll need a plugin. I've been using the pull-request-notifier plugin for Bitbucket to trigger builds from Bitbucket and it's worked pretty well. It works especially well in combination with the Bitbucket Branch Source plugin for Jenkins, which automatically creates Jenkins jobs for every branch and open PR of every single repository in a Bitbucket project. The combination of these two plugins allows me to automatically trigger builds as soon as a PR is opened and prevent merging the PR until the build passes (this is a pretty common workflow so I figure it's worth mentioning even if you didn't explicitly ask about it).
> 4 votes

---
Tags: jenkins, jenkins-pipeline, bitbucket, bcbsn
---


# Question
Title: Should Jenkins apply round robin if nodes have the same label?
In Jenkins it is possible to assign labels to nodes, e.g. somenode to nodes and then one could call somenode in the pipeline and then Jenkins will run the build on some of the nodes.

**Problem**

The assumption was that Jenkins would apply round robin, i.e. the number of builds that is run on node 1 2 3 4 and 5 is equal, but actually the majority, I.e. more than 90 percent is run on node four.

# Answer
> 0 votes
@jayhendren indicated that:

> I think you might find the answer to your question in this SO post: https://stackoverflow.com/q/11024007/2777965

---
Tags: jenkins
---


# Question
Title: How to get the ip range of github-production-release-asset-2e65be.s3.amazonaws.com?
I have denied all outgoing traffic, but need to allow some of them in order to allow the installation of certain software packages. One of such packages resides on github and that means that the ip range of `github-production-release-asset-2e65be.s3.amazonaws.com` has to be allowed, but after adding a couple of ip addresses it turns out that there are multiple ips.

**Attempt to solve the issue**

https://api.github.com/meta was found and returns the ips, but not those of AWS

**Current approach**

As a workaround I have allow outgoing connection to `52.216/16`, but that it is too open

# Answer
> 0 votes
@Tensibai indicated that:

> See: https://docs.aws.amazon.com/general/latest/gr/aws-ip-ranges.html if you really want to go this way, but there's quite large ranges, the 52.216.0.0 is a /15 for us-east-1 and not a /16 btw { "ip\_prefix": "52.216.0.0/15", "region": "us-east-1", "service": "S3" },

---
Tags: firewall
---


# Question
Title: How to solve `no live upstreams while connecting to upstream` when trying to rproxy jenkins using jwilder/nginx?
https://github.com/jwilder/nginx-proxy

```
nginx.1    | 2018/03/04 22:24:54 [error] 102#102: *276 no live upstreams
while connecting to upstream, client: 83.85.143.226, server:
jenkins.releasesoftwaremoreoften.com, request: "GET / HTTP/2.0", upstream:
"http://jenkins.releasesoftwaremoreoften.com/", host:
"jenkins.releasesoftwaremoreoften.com"

```

The docker-compose file looks as follows:

```
version: '3'
services:
  jenkins:
    image: jenkins/jenkins:lts
    environment:
      - VIRTUAL_HOST=jenkins.releasesoftwaremoreoften.com
      - VIRTUAL_PORT=8083
    ports:
      - 127.0.0.1:8083:8080
      - 127.0.0.1:50000:50000
    volumes:
      - /usr/local/jenkins/jenkins_home:/var/jenkins_home
    networks:
      - rproxy
      - jenkins

networks:
  jenkins:
  rproxy:
    external:
      name: rproxy

```

Another attempt to make it work, was changing the VIRTUAL\_HOST to 80, but that did not work as well.

Navigating to http://jenkins.releasesoftwaremoreoften.com/ returns a 502.

# Answer
> 0 votes
After a spending a lot of time I decided not to continue with this approach and decided to use Jenkins helm.

# Answer
> -1 votes
VIRTUAL\_HOST is meant to point to the internal port, in your case it should be 8080 instead of 8083

---
Tags: nginx, rproxy
---


# Question
Title: How to create a docker registry secret without passing the password on the command line?
According to this document

https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

one could create a docker registry by running:

```
kubectl create secret docker-registry regcred \
--docker-server=<your-registry-server> \
--docker-username=<your-name> \
--docker-password=<your-pword> --docker-email=<your-email>

```

How to create a secret without entering the password as plain text?

# Answer
> 0 votes
It is possible to create a secret without passing on the commandline by reading it from file:

> ```
> $ kubectl create secret generic ssh-key-secret \
>   --from-file=ssh-privatekey=/path/to/.ssh/id_rsa \
>   --from-file=ssh-publickey=/path/to/.ssh/id_rsa.pub
>
> ```

See https://kubernetes.io/docs/concepts/configuration/secret/ for more information.

---
Tags: kubernetes
---


# Question
Title: Is it possible to define a minimum code coverage using Pygradle?
Pygradle is used to build and test a Python project. In gradle it is possible to define a minimum code coverage percentage:

> ```
> jacocoTestCoverageVerification {
>     violationRules {
>         rule {
>             limit {
>                 minimum = 0.5
>             }
>         }
>
>         rule {
>             enabled = false
>             element = 'CLASS'
>             includes = ['org.gradle.*']
>
>             limit {
>                 counter = 'LINE'
>                 value = 'TOTALCOUNT'
>                 maximum = 0.3
>             }
>         }
>     }
> }
>
> ```

If this limit is not reached, build will fail. Is it possible to define a minimum code coverage using Pygradle?

When the above snippet is run, the build fails and the following is returned:

```
FAILURE: Build failed with an exception.

* Where:
Build file '/home/user/SomeProject/build.gradle' line: 14

* What went wrong:
A problem occurred evaluating root project 'SomeProject'.
> Could not find method jacocoTestCoverageVerification() for arguments [build_fsdafasdfsa] on root project 'SomeProject' of type org.gradle.api.Project.

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.

* Get more help at https://help.gradle.org

BUILD FAILED in 2s

```

# Answer
> 0 votes
At the moment it looks like it is not possible to define a minimum code coverage as the ticket that was created is still open https://github.com/linkedin/pygradle/issues/184

---
Tags: pygradle
---


# Question
Title: Is it required to move a domain to Google DNS in order to use certbot in kubernetes on Google Cloud platform?
Follow-up to What are trusted providers that offer wildcard certificates and associated costs to be used in an Orchestration Platform like Kubernetes?

With https://github.com/jetstack/cert-manager it would be possible to automatically renew certificates.

At the moment my domain is registered somewhere else than google. Is it required to move the domain to Google DNS in order to use certbot in kubernetes on Google Cloud platform?

At the moment when I add a new subdomain, e.g. abc.somedomain.com I will add a CNAME to my DNS provider. At the moment I have two subdomains and I would like to switch to kubernetes. At the moment the IPs refer to some Virtual Machine instances that reside on another provider.

**Current situation**

What I understand of Certbot is that it is able to create certificates that are valid for three months. I certbot for almost a year right now and I am happy about it. When I create a new subdomain in my DNS provider I create a CNAME and run certbot and a new certificate is created. This works fine.

**Preferred situation**

Now I would like to phase out my VM and replace it with kubernetes. I was able to create a kubernetes cluster and expose the microservice to the internet using an ingress. One the IP was exposed, it is possible to access it.

**Problems**

The problem is that I now would like to get a secure Ingress, i.e. access the microservice via https rather than http, but I am struggling to understand how to configure it in k8s.

**Attempt to solve the issue**

* A Google Cloud Platform project has been created
* A kubernetes cluster has been created
* ...
* ...
* ...

# Answer
@euank indicated that it is not required to create a DNS record on GCP in order to make it work.

https://github.com/jetstack/cert-manager/issues/399
> 1 votes

---
Tags: kubernetes, google-cloud-platform, certbot, dns
---


# Question
Title: Deploy two independently hosted services in production at once without any downtime
I have two different services. Let's say Service A and Service B. Service B is dependent on Service A. Both code bases are hosted in separate repo in Bitbucket and code bases are maintained by different teams. Service A has three backend servers and Service B has seven backend servers. I use Jenkins as a CI for code deployment. So, whenever we merge the code in master branch, Jenkins start the deployment. The process is same for Service A and Service B but they have two different Jenkins server.

Now, problem is that if Service A team deploy something in production independently with major changes in code base. Then whole application will break. Because Service B is not ready yet to compatible with the code changes of Service A. Let's say, parameters of a function has been changed where Service B is expecting a different parameter.

I am looking for a solution, where Service A code merge in master gonna happen only it is time for deployment of Service B. So, every deployment of Service B, it first deploys the code of Service A and then follows with Service B. This whole deployment will take a lot of time. Main application should not be down by this time (because once Service A is deployed then user has to wait for Service B deployment, otherwise they will face errors).

So, I am thinking if it's possible where new deployment will happen in a separate instance When deployment is completed in that instance then it will replace the previous instance. So, code will be updated in main production without any down time or error.

What is the recommended way to achieve this ? Can I do something with `Ansible` in this case ?

# Answer
> 2 votes
I think you're taking the problem on the wrong side, you may start a whole new infrastructure aside and then switch loadbalancers from blue to green, loosing sessions, breaking ongoing transactions etc.

The proper way to achieve what you're doing is coding differently, when A relasease a breaking API ni version n, a should be able to continue to answer call to api in version n-1. the usual method is to have the version in the uri or an header, for exemple:

`<a-host>/api/1/getCustomers` and `<a-host>/api/2/getCustomerList` (for exemple if n is 2). With that, you can do a rolling update of B, with backends still using api v1 and new ones using api v2.

# Answer
> 0 votes
I will make the following assumptions:

* Service B communicates with Service A using any Application layer protocol (such as HTTP)
* Service B is the only consumer of Service A
* Users only interacts with Service B

**Solution with down time**

When upgrading Service A with breaking changes, move these changes to another "version".

The goal is for you to proactively create a new interface version (i.e. API version) that will implement the breaking changes. In some situation, this might require heavy refactoring. In other (depending if your code base was properly layered), not. Then you can start implementing "contract" tests between Service A and B to make sure A doesn't break the contract with B.

The problem here is that when deploying this solution, for a short while you will not have access to neither API.

**Solution without down time**

Make the same change as the solution above, but also consider a deployment strategy where Service A.1 is always up while we introduce Service A.2. When no requests to A.1 is incoming, it may go down to leave only A.2 alive to serve all the requests. This is the blue-green deployment strategy.

This is more a cherry on the top if your overall solution requires minimal downtime deployment.

---
Tags: jenkins, ansible, continuous-integration, linux, bitbucket
---


# Question
Title: Best way to isolate programming projects?
Not sure if this is the best place for the question but I have multiple development projects with different dependencies.. I want to know what the best practice is to isolate each project?

I have many unrelated projects that a designed and updated on my laptop but I typically push them to a rPi or VMs (esxi). I would like to isolate the various projects so its easier to track which dependencies belong to which project and to stop the OS from having libs and other deps that aren't needed after a project is 'completed'. I do code some for work but its largely a hobby helping neighbors and the local SPCA.

What I use: I typically program on a Debian box or Windows 10 depending on what I need to accomplish. I program in Java, Python, and the typical web stuff (jscript, html, css).. Bash and Powershell if you consider either programming too. Version control is done via Gitlab.

# Answer
A more generic approach would be use a separate virtualized environment for each project, see . That could be either:

* a container image - more lightweight, faster and definitely more popular these days, see
* a virtual machine (VM) image, slower and a bit more heavier to maintain than containers

You'd have all the project dependencies (IDEs, libraries, tools, etc) installed inside the virtualized environment, not on your system itself.

You'd need to create/bring the project's corresponding image on your machine when you start working on it and you can delete or save it for later re-use on some external storage when done. A shared artifact respository would be recommended as storage if more than one developer can work on these projects.

A huge advantage of this approach is that such virtualized environments play really well with automated CI/CD pipelines, in true DevOps spirit, helping to minimize/eliminate the differences between the development and staging/production environments.

Depending on the programming language and/or the supporting development tools being used on a particular project you may have some more specific approaches/alternatives. For example `Pipenv` or Virtualenv/ can be used to obtain such isolation for Python projects. But these solutions are typically not usable in other contextes.
> 1 votes

# Answer
http://microservices.io/

This website explains that microservices are not the holy grail but gives an impression how it could be implemented and how it could be useful.

Regarding the question:

> Best way to isolate programming projects?

there is no one-size-fits-all. It depends on the goal. If your company is commercial then the most important thing is that new features while be added. So a monolith would be good first. If issues occur one could apply the cheese-slicer method and extract the problematic functions into microservices. The most important thing is to get it to the customer as soon as possible.
> 0 votes

---
Tags: python, java
---


# Question
Title: Deploying a SaaS solution which is custom for every customer
So we have 5 repositories (Individual web services ex. CMS/E-commerce and internal APIs) which are currently deployed as Docker containers.

What we plan to do is to use these services and make a standalone SaaS for customers. This is straightforward however, each customer over here will have some kind of Business customizations (Business Logic etc) so this means based on our current deployment technique each customer will need their own 5 repositories.

Just to explain things more clearly:

Each customer will have their own aws instance which will be having the docker containers of our products.

However, with this approach, we will have to create N * 5 repositories and custom deployment scripts (Ansible+Terraform) for each client.

One solution that I had was to create individual branches for each project within the same 5 repositories however, this is not a good scalable method.

Open to any idea, to be honest since this is something new for me.

# Answer
> 2 votes
I'd investigate what would it take to bundle all these customizations into something like a per-customer configurable *theme* that can be served by a single, more generic application code.

It probably won't be easy and it may take a long time to reach such capability, but it can be done gradually: start with the N\*5 repos and, as you inch forward toward this ideal solution you'll see the deltas in the application code from the repos starting to shrink - the differences will gradually migrate towards these custom themes/artifacts.

When there are no more app code differences left then you can replace the N\*5 repos with just 5 app code repos for now common app code and N\*5 theme configurations/artifacts repos. Much easier to maintain.

---
Tags: docker, amazon-web-services, ansible, git, terraform
---


title: What are fetching DevOps analogies?

text:
# Question
Title: What are fetching DevOps analogies?
Some presenters use analogies to clarify a certain technology, e.g. Pizza as a Service 2.0 that explains the differences between different as-a-Service (aaS) stacks.

Advantages of this Pizza analogy is that it consists of multiple analogies, i.e. runtime aka pizza and homemade aka legacy.

When one Googles "DevOps analogy", various images are shown, but not of them is very catchy.

**Definition of "fetching"**

1. Show the image in a presentation
2. Talk 30 seconds about it
3. During the elevator pitch more and more people understand DevOps and it is completely clear by them.

# Answer
> 3 votes
DevOps is the Industrialization of IT

# Answer
> 4 votes
Mostly for devs but knowledgeable to others with the "disaster girl" meme: "Works on my machine.. Ops problem now!" This illustrates that lack of responsibility can endanger the whole company, and the value of software working only in a specific environment is not absolute.

Also, the matrix of hell. Adding Docker might seem as one column more, but containers will become to-go technology and the standard architecture on long term. So, you can run Docker containers also with Kubernetes or Apache Mesos.

# Answer
> 4 votes
The most prominent DevOps analogy I can think of is the **Pet vs. Cattle** analogy on disposable infrastructure. Although, I would argue that is less about the *fetching* that is associated with the image, and more about how easy it is to understand and relate to.

# Answer
> 2 votes
Another one that I like is this one from this website https://devrant.com/search?term=devops

as I have heard it multiple times and it frustrates me because it is silo behavior and anti devops. I basically want to enforce it, when you change it you have to release or when you break it you fix it. In practice it is not that simple as a mindset has to be changed.

# Answer
> 1 votes
Another analogy was found here https://devrant.com/search?term=devops

I think that this on is also applicable as there are still developers that continue throwing things over the wall.

I have to admit that I feel like this and that this encourages me to learn programming. I am now learning Java and want to get certificates. I am now studying for Java Oracle associate.

# Answer
> 0 votes
Based on a suggestion in one of the comments to one of the answers by @PeterMuryshkin I read more about Industry4.0 and I think it could be an DevOps analogy.

Another DevOps analogy could be industry 4.0:

> Industry 4.0 is a name for the current trend of automation and data exchange in manufacturing technologies. It includes cyber-physical systems, the Internet of things, cloud computing and cognitive computing. Industry 4.0 is commonly referred to as the fourth industrial revolution.

In order to introduce industry 1.0 the functional process, i.e. how to produce coton manually has to be clear in order to automate this, 2.0 automated more and 3.0 as well. Nowadays DevOps is also about automating more and more, but in order to do that the process should be clear as well. As 4.0 is about moving to the cloud, e.g. AWS, GCP, AWS, CI/CD and self-healing systems this could be an analogy too.

# Answer
> 0 votes
DevOps could also be compared with a commando squad, that consists of a small number of specialists. I always have to think about the first level of Commandos 1 behind enemy lines. There were three characters:

* marine
* driver
* green beret

Each of them possesses unique qualities, i.e. diving, swimming, rowing (marine), diver (machine gun, driving), green beret (climbing, carrying barrels).

All of them were able to take out enemies aka automation in DevOps. Whether Operations could be compared with the marine, driver of green beret does not matter. Operation, Development and Quality Assurance all have their specialties. Combining these elements is essential to release software more often.

If for example one of the commandos died in the game, the game was over. All of them had to work together in order to accomplish a mission. I can remember that each of the commandos were isolated in the beginning of level 1 and had to take out enemies themselves, but they were dependent on each other as well.

The marine was required to bring both the driver and the green beret to the other island as he was the only one that could row the boot. Once on the island the Green beret was needed as he was the only one that could move explosive barrels that were required to blow up the radio station.

When they were working together there was a higher chance that they could survive as three shots were needed to take out an enemy. If they shoot together the enemy was taken out immediately.

---
Tags: culture
---


# Question
Title: How to pull a docker image from a private docker registry using Helm?
When the default values.yaml is inspected it is not clear how to pull a private docker image.

```
# Default values for sonatype-nexus-apt.
# This is a YAML-formatted file.
# Declare variables to be passed into your templates.

replicaCount: 1

image:
  repository: nginx
  tag: stable
  pullPolicy: IfNotPresent

service:
  type: ClusterIP
  port: 80

ingress:
  enabled: false
  annotations: {}
    # kubernetes.io/ingress.class: nginx
    # kubernetes.io/tls-acme: "true"
  path: /
  hosts:
    - chart-example.local
  tls: []
  #  - secretName: chart-example-tls
  #    hosts:
  #      - chart-example.local

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  # limits:
  #  cpu: 100m
  #  memory: 128Mi
  # requests:
  #  cpu: 100m
  #  memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

```

# Answer
Based on this Github documentation it is possible to pull a docker image from a private docker registry:

values.yaml

```
imageCredentials:
  name: credentials-name
  registry: private-docker-registry
  username: user
  password: pass

```

templates/imagePullSecret.yaml

```
{{- define "imagePullSecret" }}
{{- printf "{\"auths\": {\"%s\": {\"auth\": \"%s\"}}}" .Values.imageCredentials.registry (printf "%s:%s" .Values.imageCredentials.username .Values.imageCredentials.password | b64enc) | b64enc }}
{{- end }}

```

templates/secret.yaml

```
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.imageCredentials.name }}
type: kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: {{ template "imagePullSecret" . }}

```

at the end of the templates/deployment.yaml add the following:

```
imagePullSecrets:
  - name: {{ .Values.imageCredentials.name }}

```
> 9 votes

---
Tags: docker, helm
---


title: Does Docker on MacOS support tmpfs?

text:
# Question
Title: Does Docker on MacOS support tmpfs?
I am new to Docker and I want to run Docker with ram only storage. In other words, I do not want Docker to save anything on persistent storage. I found out that on Linux Docker supports tmpfs. Is there any such support for MacOS? I have tried setting --tmpfs flag and docker run commands run fine on mac with that flag.

# Answer
According to the docs tmpfs is linux only.

**RAM disk**

If you want "RAM only storage", you can literally use a RAM disk. ie:

```
diskutil erasevolume HFS+ 'DockerDisk' `hdiutil attach -nomount ram://2097152 `

```

Then you can point volumes at `/Volumes/DockerDisk`. Drop and recreate at will.

Docker will still store its VM and images on disk. See: Docker \> Preferences \> Disk.
> 5 votes

---
Tags: docker, containers, docker-machine
---


# Question
Title: Globally configuring Gitlab with Jenkins in docker container environment
I have Jenkins users authenticated by OpenLDAP and also gitlab CE instance authenticated by the same OpenLDAP.

And all the three are running in three different docker containers in the same network.

I am trying to pre configure Jenkins and Gitlab so that they can be resolved globally in the container world.

I have Jenkins Gitlab plugin pre configured in Jenkins. To add Gitlab in the global configuration, a Gitlab API token is required to be added in Gitlab section along with the server URL for accessing all the resources of the Gitlab instance.

I am trying to do a curl operation from a Jenkins container to Gitlab using a user name and password to Gitlab requesting for Authorization code and a Private token for each a user as to be created in Gitlab. All the user authentication is happening using the OpenLDAP. For Gitlab to generate an Authorization code the user has to be created in Gitlab, which would not be happening if the users are authenticated using OpenLDAP.

My questions are:

1. Is it necessary to configure Jenkins with Gitlab globally to have Jenkins complete access at Gitlab resources, since we would require to authenticate once again from each job on a project level?
2. How to preconfigure a private token or personal access token in Gitlab from a docker container perspective to be persistent and to be shared with Jenkins when the container spins up for the first time?
3. When Gitlab is authenticated using OpenLDAP, without the user previously signing in into Gitlab, is a user not created in Gitlab? How to generate an authorization code for a user in this case?

# Answer
1. yes, it will need access to Gitlab API to report back Jenkins build status.
2. You can create a service account in LDAP such as Jenkins which can be used to generate Gitlab API tokens and use it to preconfigure Jenkins.
3. Same as point #2

For more info - https://docs.gitlab.com/ee/integration/jenkins.html
> 1 votes

---
Tags: docker, jenkins, gitlab
---


# Question
Title: GitLab Runner is using its Docker host's hostname as Git server FQDN/hostname
When using a GitLab Runner with a "docker" executor, the following build error occurs:

```
Running with gitlab-runner 10.7.0 (7c273476)
  on main 80500676
Using Docker executor with image centos:latest ...
Pulling docker image centos:latest ...
Using docker image sha256:e934aafc22064b7322c0250f1e32e5ce93b2d19b356f4537f5864bd102e8531f for centos:latest ...
Running on runner-80500676-project-1-concurrent-0 via hostname-of-gitlab-runner-and-gitlab-server...
Cloning repository...
Cloning into '/builds/some-group-name/some-repository-name'...
fatal: unable to access 'http://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@hostname-of-gitlab-runner-and-gitlab-server/some-repository-name/some-repository-name.git/':
Could not resolve host: hostname-of-gitlab-runner-and-gitlab-server
ERROR: Job failed: exit code 1

```

It looks like the GitLab Runner is using the hostname of its Docker host as the FQDN or hostname of the Git server to pull the code from. This doesn't make sense, as it's not in any of the configuration files:

This is the GitLab configuration:

```
$ sudo cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0

[[runners]]
  name = "main"
  url = "http://127.0.0.1:1337/"
  token = "80500676b5518df3f3f3b1f772e20b"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "ubuntu:latest"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    #extra_hosts = ["hostname-of-gitlab-runner-and-gitlab-server:actual-fqdn-of-gitlab-server-where-git-repo-is"]
  [runners.cache]

```

This is the relevant configuration for `external_url` in the GitLab server configuration:

```
$ sudo egrep '\bexternal_url\b' /etc/gitlab/gitlab.rb
##! For more details on configuring external_url see:
external_url = 'https://server:actual-fqdn-of-gitlab-server-where-git-repo-is'
# gitlab_pages['artifacts_server_url'] = nil # Defaults to external_url + '/api/v4'

```

This is the relevant configuration for `host:` in the GitLab server's generated configuration:

```
$ sudo egrep -C 2 -n '^    host: ' /var/opt/gitlab/gitlab-rails/etc/gitlab.yml | head -n 5
11-  gitlab:
12-    ## Web server settings (note: host is the FQDN, do not include http://)
13:    host: hostname-of-gitlab-runner-and-gitlab-server
14-    port: 80
15-    https: false

```

When I change `hostname-of-gitlab-runner-and-gitlab-server` to `actual-fqdn-of-gitlab-server-where-git-repo-is`, and run `sudo gitlab-ctl reconfigure`, it gets reset.

If I edit `gitlab.host` in `/var/opt/gitlab/gitlab-rails/etc/gitlab.yml` directory, and run `sudo gitlab-ctl restart` without running `sudo gitlab-ctl reconfigure`, it will stay intact.

**Some unanswered questions are:**

1. How can I permanently set the `gitlab.host` configuration value that gets generated in `/var/opt/gitlab/gitlab-rails/etc/gitlab.yml` when `sudo gitlab-ctl reconfigure` is run?
2. Why does `gitlab.host` in `gitlab.yml` keep getting set to the hostname of the GitLab server?

# Answer
> 3 votes
Neil, it's because the `=` sign in your `/etc/gitlab/gitlab.rb` file.

When you'll look on the full comment before the `external_url` line, you'll see a link to the documentation:

```
## GitLab URL
##! URL on which GitLab will be reachable.
##! For more details on configuring external_url see:
##! https://docs.gitlab.com/omnibus/settings/configuration.html#configuring-the-external-url-for-gitlab
external_url 'http://gitlab.example.com'

```

In the default `gitlab.rb` file (just after installing the Omnibus GitLab package) and also in the linked documentation you can see, that the `external_url` value is assigned without the `=` sign. It's done like that because the `external_url` is a method, not a variable. So here you're not assigning the value, but calling the method with this value. Ruby allows to call a method without brackets around the arguments, and this is a commonly used pattern when designing a DSLs based on Ruby.

When you try to set the value with:

```
external_url = 'https://server:actual-fqdn-of-gitlab-server-where-git-repo-is'

```

then the `external_url` method is not receiving any argument and in that case Omnibus GitLab falls back to a default value which is based on host's `hostname`.

In most cases the hostname of the server where GitLab is installed will be the same as the hostname that the server is accessible with. In some cases - including yours - the `external_url` needs to be set explicitly.

So the solution seems to be:

1. Change `external_url = 'URL_HERE'` to `external_url 'URL_HERE'`
2. Execute `sudo gitlab-ctl reconfigure`

This should set the proper values in `gitlab.yml` file and resolve your problems.

---
Tags: docker, gitlab
---


title: AWS CodePipeline staged deployments

text:
# Question
Title: AWS CodePipeline staged deployments
I am implementing a CICD pipeline using CodePipeline, CodeCommit, CodeDeploy and CloudFormation (I skipped the build part of the pipeline and am going to commit built code for the moment).

I already have a CloudFormation template that creates the required infrastructure to host the application, a CodeDeploy Application (in a nested stack) and a CodePipeline that gets the source code from CodeCommit, does In-Place deployments via CodeDeploy and finally sends an approval email (in a nested stack).

I would like to implement the following:

1. In order to create the required infrastructure to host the application for the first time, I launch a CloudFormation template that creates (I already have this):

* The required infrastructure to host the application (AutoScalingGroup and Launch Config, TargetGroup and ListenerRules, IAM roles)
    * CodeDeploy Application and DeploymentGroup (this will be in a nested stack)
    * CodePipeline pipeline pre-configured to carry out the next steps (this will be in a nested stack)
2. When source code is committed to CodeCommit, CodePipeline launches a new CloudFormation stack containing the required infrastructure to host the application, then deploys the new version of the code to this stack, without affecting the existing stack for the old version of this application. The new stack will have a listener rule with a different host name, so I can test the application while users keep using the old version;
3. Once the code has been deployed, CodePipeline sends an approval email (through an SNS topic and I already have this);
4. If the email is approved, the old stack (created in step 1. ) is deleted
5. I will manually swap the listener rules for the two stacks to direct users to the new stack created in step 2, then delete the old stack.

I am struggling to implement step 2 and currently trying to use a Deploy action with a CloudFormation deployment provider to carry out step 2. above. I am struggling to pass the output artifacts (new source code) from the previous stage to the new stack.

Can you provide some guidelines on how to implement ***step 2*** ?

# Answer
> 2 votes
I managed to setup an environment that does what I described above. Here are the full details:

1. Launch a template that creates a CodeCommit repository for the application. This repository will be used to store both the source code and the CloudFormation (CF) templates to create the required infrastructure for the application;
2. Launch a template that creates a CodePipeline pipeline for the application. The pipeline will deploy a stack for the infrastructure and one for deploying the code through CodeDeploy (the CodeDeploy stack will will create a new CodeDeploy Application and deployment group, and will deploy the latest code to the infrastructure created previously)

The pipeline has a paramater "StackBlueeGreen" that is used to determine whether the stack is the Green or the Blue version.

When the pipeline runs successfully, it will create a separate infrastructure

> (NameOfTheApplication-B or NameOfTheApplication-G)

and deploy the code there. The second infrastructure will have its own load balancer listener rule setup with a staging URL

> (something like URLOfTheApplication-staging.somedomain.co.uk)

This allows developers to test the new version of the application withuot affecting the previous version. Once developers are happy with the new version, the listener rules for the old and new infrastructure need to be changed manually so that users are now directed to the new infrastructure. If there is an issue with the new version, rolling back is as simple as swapping the load balancer listener rules again.

Finally, the pipeline stack need to be updated: the StackBlueGreen parameter needs to be switched to G/B depending on how it is currently set. If this is not done, next time there is a change, the new version will be deployed on the same infrastructure.

\[The setup described above works very well and meets all the requirements outlined in the OP. An improvement that I would really like to implement is automating the switching of the load balancer listener rule, deletion of the old infrastructure and updating the pipeline stack. I will add to this question if I manage to implement this improvement. Meanwhile, any input would be much appreciated!\]

---
Tags: amazon-web-services, continuous-deployment, build-pipeline
---


# Question
Title: How to move disk image in the command line?
As answered in this questions: Does Docker on MacOS support `tmpfs`? In order to run in memory docker, I need to move docker volumes to ramdisk. In macOS I can do it through docker preferences. However, I would like to write a script that could do it for me everytime I run docker on my machine. Is it possible to specify this setting using cmdline in mac?

# Answer
> 2 votes
Docker is running in a VM on MacOS. One of your options is to put the entire VM on a tmpfs. You could use this shell script to create and mount the ramdisk: https://gist.github.com/koshigoe/822455

Here is a guide for resizing the machine. https://gist.github.com/stefanfoulis/5bd226b25fa0d4baedc4803fc002829e

The trick is to stop docker first, then you can make changes and move the

> ~/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/Docker.qcow2

wherever you like and symlink there from this location.

# Answer
> 0 votes
I might be misunderstanding the question.

You tell docker where to put your volume data when you run the container. See the docs. It's not something you need to define in a global config. For example:

```
docker run -v /Volumes/DockerData/nginx_conf:/etc/nginx/conf.d nginx:latest

```

The above would run nginx but pull whatever config files you have in nginx\_conf.

---
Tags: docker, docker-compose, macos
---


# Question
Title: How do I replicate a Jenkins setup via automation?
I have a Jenkins setup running in production. I want to automate the Jenkins setup (installation) along with all the jobs that are set up in Jenkins.

One crude way I can think of is to copy the whole jobs directory to the new Jenkins setup.

How do I deal with this problem?

# Answer
> 7 votes
Jenkins itself can be configured in many ways. The same applies for Jenkins jobs.

Probably the most easy way is to configure Jenkins manually and replicate its setup by backing up and restoring. Unfortunately, this means dealing with copying XML files.

But fortunately there are other ways.

# `init.groovy.d` directory

When Jenkins starts, it checks `init.groovy.d` directory and executes Groovy scripts there. To enforce locale I'm using a snippet.

```
// Copied from /var/lib/jenkins/init.groovy.d/set-locale.groovy
def pluginWrapper = jenkins.model.Jenkins.instance.getPluginManager().getPlugin('locale')
def plugin = pluginWrapper.getPlugin()

plugin.setSystemLocale('en_US')

```

Another example of setting up Jenkins by Groovy scripts is located on https://gist.github.com/hayderimran7/50cb1244cc1e856873a4 \- I'm using that snippet to setup local Jenkins users on test machines.

# Generating jobs

## Jenkins Job builder

One approach of generating jobs is Jenkins Job Builder, which creates XML definitions jobs from YAML configurations. I don't like this approach personally as I consider it to be a bit hackish - looks like "generation of XML files without knowing what it means". But some big players are using this approach - JJB originates from Openstack.

## Job DSL plugin

Another approach is Job DSL plugin (plugin page). There is special job type called *seed job* which creates other jobs from Groovy DSL. The seed job can pull the others job description in from Git, so you can easily version them, reuse them, etc.

I'm using personally this approach, as it was available when I started with Jenkins. I've even found some Jenkins tool to create DSL sources from existing jobs, but I'm not able to find it again.

To provide a seed job, you can still use the `init.groovy.d` approach.

## Pipelines

The currently recommended approach is probably related to Jenkins Pipelines. You can store your building pipelines in the source VCS repository next to your code.

I haven't explored this very much, but I'm planning to do.

# Answer
> 3 votes
The simple way to deal with this is as you have described. However, some drawbacks of it are:

* manual Jenkins setup.
* No guarantee that your new Jenkins instance is one-on-one identical to the previous in terms of version, plugins, etc.

*Here's one way I have achieved this in the past.* This creates disposable Jenkins, that can come up with the same plugins and tools every time.

Note: my setup assumes

* Jenkins job history is backed up. Artifacts are not stored within Jenkins.
* Your Jenkins infrastructure is on the cloud, e.g. AWS
* Ansible used for automated provisioning of infrastructure
* Jenkins 2.x in use with full use of Pipeline-as-Code
* A defined set of Jenkins pipelines.

### Machine/Instance Creation

* Ansible provisioner creates AWS Linux instance
* Adds it to Ansible dynamic inventory

### Jenkins Install

* Ansible installs Jenkins on the instance
* Along with any ancillary tools, e.g. `git`, `maven` etc. required for Jenkins jobs to execute.
* Start Jenkins service, wait for it to start.

### Jenkins Configuration

Ansible tasks for:

* Wait for Jenkins CLI to be available.
* Download Jenkins CLI JAR file
* Execute Jenkins CLI to create users, install plugins, restart Jenkins etc and create jobs.
* Restore job history from backup.

### Caveat

* Works best for greenfield and simpler Jenkins setups. Is harder (not impossible) to achieve with brownfield and more complex projects.
* A large amount of training in Infra-as-Code mindset is required with this approach, such as for example "Don't make plugin installations manually".

---
Tags: jenkins, automation
---


# Question
Title: What are off-the-shelf options for creating a microservice that consults a REST interface and shows the outcome in a graph?
I am planning to create a microservice that reads a database and shows a graph. Are there any people who have created such a service? What are things to consider before starting? I am thinking about django and Springboot. On the other hand, perhaps it is from a "do not reinvent the wheel" perspective better to choose something as graphite, but then still has to configure it.

I am basically seeking for a tool that is able to read a REST interface and shows the results in a graph or otherwise a framework preferably written in Kotlin, golang or java.

The aim is to show how fast something is increasing in a graph. At a certain moment I also would like to show the R2, i.e. consistency of the points, slope and so on.

While writing this question, at a certain moment I would also like to predict whether something will be increased in the near future, i.e. applying data science.

A cron job runs every night and inserts data to the Postgres database. A part of the table looks as follows:

```
ID = 61
PACKAGE_NAME = FIREFOX
DATE = 2018-04-29
VERSION = 59.0.2

ID = 62
PACKAGE_NAME = TOMCAT8
DATE = 2018-04-29
VERSION = 8.0.51

ID = 63
PACKAGE_NAME = TOMCAT85
DATE = 2018-04-29
VERSION = 8.5.30

ID = 64
PACKAGE_NAME = TOMCAT9
DATE = 2018-04-29
VERSION = 9.0.7

ID = 65
PACKAGE_NAME = FIREFOX
DATE = 2018-04-30
VERSION = 59.0.2

ID = 66
PACKAGE_NAME = TOMCAT8
DATE = 2018-04-30
VERSION = 8.0.51

ID = 67
PACKAGE_NAME = TOMCAT85
DATE = 2018-04-30
VERSION = 8.5.30

ID = 68
PACKAGE_NAME = TOMCAT9
DATE = 2018-04-30
VERSION = 9.0.7

```

This data has to be displayed in a graph like Tiobe:

# Answer
I am currently looking into http://www.chartjs.org/. It looks like that this is able to create graphs. I will add more to this answer if I have created a graph using this library.
> 1 votes

---
Tags: microservices
---


# Question
Title: How do I get dict key from ansible playbook variable?
Following is the desire output:

```
domain.com is in gh4.allserevers.com

```

The idea is to pass domain name ie **domain.com** in playbook and playbook will lookup its **A record** then check it against the predefined variable. If ip is found in variable then it will display the corresponding key.

Here is my ansible-playbook role:

**roles/test/tasks/main.yml**

```
    ---
    - name: Simple A record (IPV4 address) lookup for domain.com
      set_fact:
         IP: "{{ lookup('dig', 'domain.com.')}}"
      register: output
      tags: test

    - name: checking if ip belongs to gh4
      debug:
        msg: "domain.com is in gh4"
      when:
        - IP in hosts.gh4
      tags: test

```

**roles/test/vars/main.yml**

```
    ---
    hosts:
      gh4: [127.0.0.1,127.0.0.2,127.0.0.3,127.0.0.4,127.0.0.5]
      gh5: [127.0.0.7,127.0.0.8]
      gh6: [127.0.0.10,127.0.0.11,127.0.0.12]

```

# Answer
If I understand your question correctly you want to pass in a domain to find what could be a physical location or data center. I hope this is helps you!

**roles/test/vars/main.yml**

```
---
domains:
  gh4: [127.0.0.1,127.0.0.2,127.0.0.3,127.0.0.4,127.0.0.5]
  gh5: [127.0.0.7,127.0.0.8]
  gh6: [127.0.0.10,127.0.0.11,127.0.0.12]

```

**roles/test/tasks/main.yml**

```
---
# Make sure we have a domain passed in
- name: Ensure domain is defined
  assert: { that: domain is defined }

# Obtain the IP
- name: Simple A record (IPV4 address) lookup for {{domain}}
  set_fact:
      IP: "{{ lookup('dig', '+short', '{{domain}}')}}"
  register: output
  tags: test

# Check if the IP belongs to any location
- name: Checking location of IP
  set_fact:
    domain_location: "{{item}}"
  when:
     - IP in domains[item]
  with_items: "{{domains}}"

- debug:
    msg: "domain.com is in gh4.allserevers.com{{domain_location}}"
  when: domain_location is defined

```

**Example Run** With domain.com pointing to 127.0.0.12

```
root@DESKTOP-U4VN2CS:/mnt/c/ansible/roles# ansible-playbook ../playbooks/test.yml  -e domain=domain.com
PLAY [127.0.0.1] ****************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************
ok: [localhost]

TASK [test : Ensure domain is defined] ******************************************************************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}

TASK [test : Simple A record (IPV4 address) lookup for domain.com] **************************************************
ok: [localhost]

TASK [test : Checking location of IP] *******************************************************************************
skipping: [localhost] => (item=gh4)
skipping: [localhost] => (item=gh5)
ok: [localhost] => (item=gh6)

TASK [test : debug] *************************************************************************************************
ok: [localhost] => {
    "msg": "domain.com is in gh6.allserevers.com"
}

PLAY RECAP **********************************************************************************************************
localhost                  : ok=5    changed=0    unreachable=0    failed=0

```
> 2 votes

---
Tags: ansible
---


# Question
Title: How to use Jenkins Pipeline and choose triggers for email notifications?
Before Pipeline, I used Email-ext plugin to define triggers for sending emails (on first failure, and when back to success).

But in the Pipeline version, I can't control the triggers, I can only "send always", or send based on current build result.

How do I send emails in pipeline based on more general triggers? (first failure, first success, etc)

# Answer
> 1 votes
You can use snippet like

```
post {
    changed {
        emailext body: '$DEFAULT_CONTENT', recipientProviders: [brokenTestsSuspects(), brokenBuildSuspects(), developers()], subject: '$DEFAULT_SUBJECT'
    }
}

```

in declarative pipelines.

Changed post-condition is called when the build result is changed (i.e. of first failure and back to success). There are post-conditions like "regression" if you want more control.

# Answer
> 1 votes
This seems not to be possible at the moment with pipelines. See this Jira ticket.

---
Tags: jenkins, continuous-integration, jenkins-pipeline, jenkins-plugins
---


# Question
Title: Understanding DevOps opponents on stakeholder level beyond typical examples
What are typical stakeholders and their reasons not to support/enable DevOps principles?\*

Background: in small to medium organizations, certain level of DevOps is not a very big problem due to thinner silos and faster communications paths.

But how do you do DevOps at large organizational scales and which groups of stakeholders do you deal with?

I can think of these typical examples:

* Developers, either junior or more experienced who would like to "just do their job they are paid for - establishing software solutions and coding them"
* Management/Deciders concentrating on the service aspect in terms of man-months, where you find out a thing like a toolchain is non-existent to them: they recognize people doing their work on computers, not more.

Others?

* (And how can DevOps be a benefit to them? - as a side thought to be separate question, "how can DevOps be a benefit to stakeholder group X?")

# Answer
People resist change, particularly when it's perceived to be outside of their control. Organizations that have been around a while naturally devolve into silos, and organizational structures often reflect that (e.g., database administration teams often manage all of the databases for the org, even though they are often distanced from the business purpose of those databases). DevOps has the potential to disrupt silos, and may cause realignment of org structures. For example, you could have product teams composed of developers and operations people (like DBA's) that focus solely on a single product.

For some mid-level managers, that could threaten their sense of job security. The benefit of a DevOps mindset is that it encourages everyone to understand the value stream to the customer, and the relationship of all the roles in delivering that value. IT could mean that the manager of DBA's suddenly has to understand development processes because they're now managing a product team.
> 2 votes

# Answer
DevOps fundamentally is an arbitrage play: it exploits the idea that if the things you are developing and operating are known to be stateless and immutable and ephemeral you can move very quickly, because there are a bunch of things you just don't need to do. And this is true, you can, and you don't.

But "traditional" IT is *all about* managing statefulness and controlling the transitions between states (or "mutation" if you prefer). A database server, a file server. A database server for example: once you have made changes to a schema and data has gone in, you can no longer "roll back" because you probably have nowhere you can put that data that satisfies the constraints of all the stakeholders. If the database server goes down you can't just spin up a new one, it has to be *exactly the same* to the greatest extent possible, and you need to have a plan for resolving the deltas. DevOps doesn't really address this and even worse, the typical DevOps pitch includes some jabs at "traditional" IT for being slow and obsolete, without any acknowledgement that maybe there were reasons for things to be done the "old fashioned" way other than the practitioners being dinosaurs.

What that sounds like to a stakeholder is that DevOps people are willing to take risks with the company's most precious assets - customer records, accounts, inventory control, all the other things that typically reside in the stateful parts of the technology infrastructure. Therefore to work with and bring on board skeptics the very first thing you must do is convince them that you have mitigated all the risks of operating the stateful parts of the infrastructure in a DevOps fashion.

I will wager that some variation on this theme is the root cause of TSB's woes at the time of writing. They failed to exercise sufficient care with the stateful parts of their system.
> 2 votes

# Answer
At one point a higher level tools manager, asked to review/vote on a proposal for using a CI tool which would allow gradual elimination of long-lived branches and migration towards true CI methodology (i.e. Trunk-Based Development \- TBD) for a large scale product line, opposed it. The stated reason was belief that it wouldn't scale to the very large team (1000+ developers) working on the product.

I may be wrong, but I got the feeling another reason behind the vote was fear of having his ranks (which included an army of sync-masters for driving the many, quite difficult branch merges) shrink - in TBD there wouldn't be any such branch merges for them to work on.
> 0 votes

# Answer
The biggest challenge I've encountered with people resisting the change is the reason of not seeing the value of it, or build automation in general. His idea was, we were our getting product(s) out the door, and on time. So why do we need to "create work" for ourselves, and take a developer off of working on a product. After 4 years since then I've finally got the owners on board with CI by capturing metrics on how much time is wasted copying files manually from bin folders to SVN for qa to copy from SVN to environments. This along with the overhead of managing what build is in what environment and other pitfalls of you are testing the wrong build etc. He has finally came around, but I think he feels that the time could be put to other tasks.
> 0 votes

---
Tags: culture
---


# Question
Title: Why screen folder in /var/run disappears in Docker container after running Ansible playbook?
### Files

I've got the following **`Dockerfile`**:

```
FROM ansible/ansible:ubuntu1604py3
ENV DEBIAN_FRONTEND noninteractive
RUN pip3 install ansible
RUN printf '[local]\r\nlocalhost ansible_python_interpreter="env python3"\r\n' > /etc/ansible/hosts
ADD . /opt
RUN ansible-playbook /opt/install_screen.yml -c local -v
RUN useradd -d /home/ubuntu -ms /bin/bash -g root -G sudo -p ubuntu ubuntu
USER ubuntu
WORKDIR /home/ubuntu
CMD ["/usr/bin/env", "screen"]

```

and the following **`install_screen.yml`** playbook file:

```
- hosts: localhost
  tasks:
    - name: Install screen (a full-screen window manager)
      apt: name=screen state=present
      register: screen_installed
    - name: Configure screen
      shell: bash -x /etc/init.d/screen-cleanup start
      register: screen_configured
      when: screen_installed is success
    - shell: ls -la /var/run/

```

---

### Build

Now, I build the Docker container in a shell by:

```
$ docker build -t screen .

```

and the *Configure screen* task shows the following output:

> `changed: [localhost] => {"changed": true, "cmd": "bash -x /etc/init.d/screen-cleanup start", "delta": "0:00:00.009613", "end": "2018-05-01 23:09:49.159514", "rc": 0, "start": "2018-05-01 23:09:49.149901", "stderr": "+ set -e\n+ test -f /usr/bin/screen\n+ SCREENDIR=/var/run/screen\n+ case \"$1\" in\n+ test -L /var/run/screen\n+ test -d /var/run/screen\n+ find /var/run/screen -type p -delete\n++ stat -c%a /usr/bin/screen\n+ BINARYPERM=2755\n+ '[' 2755 -ge 4000 ']'\n+ '[' 2755 -ge 2000 ']'\n+ chmod 0775 /var/run/screen\n+ exit 0", "stderr_lines": ["+ set -e", "+ test -f /usr/bin/screen", "+ SCREENDIR=/var/run/screen", "+ case \"$1\" in", "+ test -L /var/run/screen", "+ test -d /var/run/screen", "+ find /var/run/screen -type p -delete", "++ stat -c%a /usr/bin/screen", "+ BINARYPERM=2755", "+ '[' 2755 -ge 4000 ']'", "+ '[' 2755 -ge 2000 ']'", "+ chmod 0775 /var/run/screen", "+ exit 0"], "stdout": "", "stdout_lines": []}`

then my `ls -la /var/run/` shell command shows:

> `changed: [localhost] => {"changed": true, "cmd": "ls -la /var/run/", "delta": "0:00:00.004997", "end": "2018-05-01 23:09:49.442670", "rc": 0, "start": "2018-05-01 23:09:49.437673", "stderr": "", "stderr_lines": [], "stdout": "total 24\ndrwxr-xr-x 6 root root 4096 May 1 23:09 .\ndrwxr-xr-x 1 root root 4096 May 1 23:09 ..\n-rw-r--r-- 1 root root 0 Mar 28 19:06 init.upgraded\ndrwxrwxrwt 2 root root 4096 Feb 28 19:13 lock\ndrwxr-xr-x 2 root root 4096 May 1 23:09 mount\ndrwxrwxr-x 2 root utmp 4096 May 1 23:09 screen\ndrwxr-xr-x 2 root root 4096 May 1 23:09 systemd\n-rw-rw-r-- 1 root utmp 0 Feb 28 19:13 utmp", "stdout_lines": ["total 24", "drwxr-xr-x 6 root root 4096 May 1 23:09 .", "drwxr-xr-x 1 root root 4096 May 1 23:09 ..", "-rw-r--r-- 1 root root 0 Mar 28 19:06 init.upgraded", "drwxrwxrwt 2 root root 4096 Feb 28 19:13 lock", "drwxr-xr-x 2 root root 4096 May 1 23:09 mount", "drwxrwxr-x 2 root utmp 4096 May 1 23:09 screen", "drwxr-xr-x 2 root root 4096 May 1 23:09 systemd", "-rw-rw-r-- 1 root utmp 0 Feb 28 19:13 utmp"]}`

### Problem

It seems after installing `screen` and running `/etc/init.d/screen-cleanup start` during provisioning, the container have the problem running `screen` command as below:

```
$ docker run -it screen
Cannot make directory '/var/run/screen': Permission denied
$ docker run -it screen ls -la /var/run/screen
ls: cannot access '/var/run/screen': No such file or directory

```

In above Ansible's output I can see that `/var/run/screen` folder exists as per the following line (during `ls -la /var/run/` run):

> `drwxrwxr-x 2 root utmp 4096 May 1 23:09 screen`

But it disappears after Ansible playbook command is finished.

### Workaround

The following one-time workaround works to recreate the folder:

```
$ docker run -it -u root screen bash
ubuntu# bash -x /etc/init.d/screen-cleanup start
...
+ mkdir /var/run/screen
...

```

# Answer
> 2 votes
Rather than solving the immediate problem you're having, let me suggest an alternative solution to your actual problem:

> I need screen, so I can use multiple interactive shells in one instance of Docker. If I run multiple Docker instances, each takes 1G of RAM, so 10+ instances will kill my RAM.

Run screen (or tmux, but screen if you insist) locally on your workstation. Start up your docker container, and in the windows where you want to do something on the container, use `docker exec -it <container-id> bash` to get a new interactive shell on the container. This will be much simpler than running screen inside the container, and allows you to also have windows running commands on your workstation.

You may also want to look into using Vagrant with the Docker provider to simplify the process of doing development on a Docker container.

---
Tags: ansible, containers, dockerfile, provisioning
---


# Question
Title: Is it possible to keep all the DSC configurations in Version Control and make Azure Automation fetch it from there automatically?
The Azure Automation DSC seems to know to integrate with GitHub, but it seems to expect only runbooks there. At least, this is what I understand from the following UI screen:

I have just started to learn DSC and have no runbooks. So far I only used DSC configurations, no runbooks.

So, is it only possible for runbooks? Does it mean that nobody works with just DSC configurations?

**P.S.** Knowing that DSC stands for Desired State **Configuration** I find the term DSC **configurations** a bit funny.

# Answer
> 2 votes
The article "Continuous Deployment to Azure Automation DSC", or series of articles, walks through setting up a CI/CD pipeline for DSC using VSTS (Visual Studio Team Services). Building a VSTS account in Azure is free and it provides some number of build minutes for free as well.

Basically, rather than setting up a runbook, you can configure a CI/CD pipeline using build services in VSTS to perform whatever orchestration activities you require for DSC and VSTS enables you to save your code in their source control management system.

---
Tags: automation, azure
---


# Question
Title: What do you need for a monorepo for a large codebase?
From certain size of the codebase, would you still have Git or are there more specialized solutions?

(Also to checkout just a part of the codebase)

# Answer
> 4 votes
The answer is: a bit of both. To satisfy the constraints of "use git" and "manage a vast codebase" Microsoft developed a new filesystem (previously they were using a variant of Perforce called SourceDepot). It's open source but I have no personal experience of using it.

Why would you want a monorepo? The most obvious reason is that you can modify an API and all the callers of that API in an atomic commit. Also there are advantages to being able to do a `git log` search across the entire codebase...

# Answer
> 4 votes
Git works for monorepos, but it has a few problems:

1. You have to check out the entire repo.
2. You have to fetch the entire history (generally - shallow clones are an option, but usually not useful in actual development work).
3. Natively, everyone has read+write access to every directory if they have it at all.

Google, probably the most famous monorepo user, developed Piper to handle their needs. But you aren't Google, and so their solutions are probably not yours.

One of the key advantages to a monorepo is that you can make globally-atomic changes (i.e. you don't need to version many things because you can change the caller and the callee in the same commit). To power this, you really want to have a unified build system that tracks dependencies across the entire repo. Bazel is an open-source extraction of Google's build system, Blaze, and it attempts to do that (although it's young and immature and missing a lot of features that are necessary for non-Google use). Pants is a similar system out of Twitter.

If you're building tons of code when you make such an atomic change, then you probably also want a build farm that allows you to do that not on your local machine. Similarly, you'll need a powerful CI system to handle running tests across everything as you update.

# Answer
> 1 votes
Opinions differ on what a large code base is. If you are speaking about a company with 100 engineers, I would argue that Git should still be able to handle it. It has been developed for the needs of the Linux kernel, which is not a small project on its own.

Independent of the way how you store the repository, you may run into problems. For instance, if you are working on a large Java code base and are using tools like Eclipse or IntelliJ, they will use more memory and generally become slower.

On the other hand, having the option to operate on all code at once (e.g., when applying refactoring, or source code transformations) is one of the main advantages of monolithic repositories.

When you ask about whether you need specialized tools, then up a certain code size, the answer is yes. According to Google, which arguably has the biggest C++ codebase in the world, all available tools (open source or commercial) did not satisfy their requirements. They ended up developing an inhouse system called Piper:

# Answer
> 0 votes
If I understand it correctly the "need" for a monorepo is simply the fundamental need of a single/coherent versioning scheme applied to a software project containing multiple loosely-related components/sub-projects which are/could be otherwise managed/versioned independently in separate repositories.

Similar, if you want, with the need of using a regular source repository to provide a single/coherent versioning scheme for a multitude of source files, each with their own, independent modification history.

Using an actual monorepo solution is definitely one, but IMHO not the only way of addressing this need.

Another possible approach is using an umbrella project repository containing one or more manifest files with the exact version of each of the individual project component repositories.

Even if the component repositories have their versions modified by independent, non-atomic commits, the project itself can be managed coherently simply by combining all related component repository version changes into a single commit to the manifest file(s) in the umbrella repository.

Such approach has several advantages over migrating to an actual monorepo solution:

* no need to change the existing component repositories
* can support mixes of components with different repository technologies
* each component repository can still be developed and managed independently
* adding/deleting project components is almost trivial
* integrating 3rd party (upstream) components is a whole lot easier
* the project history can be kept much cleaner, not polluted with all the details of each individual component repository change (which typically would be irrelevant for the other components)
* no need to worry about size/performance/scalability of a single repository, the solution itself is highly scalable.

---
Tags: git, monorepo
---


# Question
Title: Is Azure WebDeploy safe for production Web Apps?
We use Azure Web Apps in production. We deploy using WebDeploy either by right-clicking the project in Visual Studio and choosing "Publish..." or via CI in AppVeyor, which is WebDeploy ready.

My question is: **Is WebDeploy safe for production environments?** Will it interrupt running processes, or does it work some magic to make sure it doesn't? Can we publish to production using WebDeploy during business hours (when our customers are active) and not worry about it? A good answer will provide some documentation on this, as I have not been able to find this yet.

# Answer
> 1 votes
WebDeploy by itself cannot guarantee if your application pool will not be recycled during deployment. To be on the safe side, I would use deployment slots. Also I would recommend to tag question with `azure-web-sites` so right people will help you.

# Answer
> 0 votes
I have been using web deploy for ages and it’s been very stable except for couple of azure outage times.

Suggest using deployment slots. You can use a CD like visual studio team services and first web deploy to the slot and add a step to swap the ‘staging’ slot with ‘production’.

If you are using a basic deploy approach you can turn on auto swap which triggers upon successful deployment to the ‘staging’ slot. I personally don’t like this and have seen issues with this.

---
Tags: deployment, continuous-deployment, azure, appveyor
---


# Question
Title: How to integrate a platform as a service with configuration management tools?
Is a configuration management tool (Chef/Puppet/Ansible) necessary for managing an application on a PAAS system (Azure app service, Heroku, Elastic Beanstalk)? How would it be utilized? While a tool such as Chef does not inherently work on a PAAS system, there are other tools (such as Habitat) that can support this functionality. What kind of steps would it take to integrate these technologies?

# Answer
A configuration management tool is not required to manage a PAAS. For example, when one has created a VM on Azure then it is possible to export the configuration snippet as powershell or as yaml and use yaml to deploy and manage systems.

In my opinion one should restrict configuration management tools like puppet or ansible to manage configuration files and installing additional packages and yaml to manage the PAAS as I prefer KISS.

If one would like to manage different PASS then one could consider to use terraform.
> 2 votes

# Answer
Modern configuration management tools can do a wide variety of things, so you can certainly use them to manage a PaaS. But it makes more sense to use a tool like Terraform to handle provisioning, since that's its specialty.
> 2 votes

# Answer
ARM Templated Deployment for the PaaS and the underlying components managed by Microsoft in the case of Azure. Don’t need a seperate CM. This is the beauty of being pure PaaS.

terraform is a wrapper for ARM. Unless you are cross cloud no need for this.
> 0 votes

---
Tags: configuration-management, platform-as-a-service
---


# Question
Title: Is there a common name for describing "spans" of versions?
I'm looking for a way to pithily define a abstract term that declares all versions that a bug effects, or that a feature behaved a certain way. For example, feature X worked way Y in versions 1.3 to 1.5, but then they changed it. I thought of calling 1.3 to 1.5 a "version span", but wondered if there was some common terminology that I would not have to explain all the time to everyone?

This could get more complicated for bug fixes that are done on different minor versions of multiple major versions. For example, a bug that was first introduced in 1.4 and fixed in versions 1.6 and 2.2. 2.1 also exhibits the bug because 2.1 branched from 1.4. So now you have a "version span" of 1.4 - 1.6 & 2.1 and all their patch versions and build versions etc.

# Answer
> 4 votes
I have never heard of a common name for what you describe; and rarely, if ever, have I seen a notation where someone references to something which has a "begin" and "end" version. It just is not that applicable in a general fashion; this is probably why there is no such name. Also changelogs usually don't contain this information except for extremely bad breaks that need some kind of notice about which span(sic) of versions one should rather stay clear of.

The closest you could get to a common name (popularized by Jira, for example) is probably to call 1.3 in your example the "affected version", and 1.5 the "fixed version"; possibly allowing multiple values there. But they also have no name for the *span* as an entity.

Note that it is not really a problem if you have multiple old maintenance releases like you are describing. Those would just lead to multiple spans distributed over the branches.

Regarding your edit:

> Instead of having to look through release notes this would be a table of features mapped to versions. Release notes typically have "bugs not fixed" but release notes never show bugs not yet found. My customer has multiple variants/branches of systems deployed all over the world, and wants a quick and easy way to know which variants and deployments have which features and bugs.

Been there done that. You are doing well to approach this with care, it can really "explode" after a while. If you are using a tool like Jira, you should really find a field for each issue which allows you to mention releases related to the issue. If you decide do go manually, just list your version ranges in whatever format seems right to you.

I would not bother with finding a name for a range of versions, but just call them "affected versions".

---
Tags: configuration-management
---


# Question
Title: Cannot Change Docker Size on Ubuntu 16.04
`Docker version 18.03.1-ce, build 9ee9f40`

I am trying to increase the docker container sizing on Ubuntu 16.04.

I have created the following file at `/etc/docker/daemon.json`

```
{
    "storage-opts": ["dm.basesize=20G"]
}

```

However when I `sudo service docker start` I get: `Job for docker.service failed because the control process exited with error code. See "systemctl status docker.service" and "journalctl -xe" for details.`

If I remove `/etc/docker/daemon.json` then docker can start.

Inspecting the recommended logs yields

```
systemctl status docker.service
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: inactive (dead) (Result: exit-code) since Sat 2018-05-05 12:58:26 PDT; 4min 35s ago
     Docs: https://docs.docker.com
  Process: 3047 ExecStart=/usr/bin/dockerd -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 3047 (code=exited, status=1/FAILURE)

May 05 12:58:26 stage systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
May 05 12:58:26 stage systemd[1]: Failed to start Docker Application Container Engine.
May 05 12:58:26 stage systemd[1]: docker.service: Unit entered failed state.
May 05 12:58:26 stage systemd[1]: docker.service: Failed with result 'exit-code'.
May 05 12:58:26 stage systemd[1]: docker.service: Service hold-off time over, scheduling restart.
May 05 12:58:26 stage systemd[1]: Stopped Docker Application Container Engine.
May 05 12:58:26 stage systemd[1]: docker.service: Start request repeated too quickly.
May 05 12:58:26 stage systemd[1]: Failed to start Docker Application Container Engine.

journalctl -xe
--
-- Unit docker.socket has begun starting up.
May 05 12:58:26 stage systemd[1]: Listening on Docker Socket for the API.
-- Subject: Unit docker.socket has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.socket has finished starting up.
--
-- The start-up result is done.
May 05 12:58:26 stage systemd[1]: docker.service: Start request repeated too quickly.
May 05 12:58:26 stage systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit docker.service has failed.
--
-- The result is failed.
May 05 12:58:26 stage systemd[1]: docker.socket: Unit entered failed state.
May 05 13:01:16 stage sudo[3069]:      mjb : TTY=pts/0 ; PWD=/var/www/app ; USER=root ; COMMAND=/bin/cat /etc/docker/daemon.json
May 05 13:01:16 stage sudo[3069]: pam_unix(sudo:session): session opened for user root by mjb(uid=0)
May 05 13:01:16 stage sudo[3069]: pam_unix(sudo:session): session closed for user root

```

How can I increase the default `dm.basesize` of Docker on Ubuntu such that, when I start docker using the command `sudo service docker start` it starts and has an increased `dm.basesize`?

# Answer
You can try below /etc/docker/daemon.json

```
{
    "experimental" : true,
    "storage-driver" : "devicemapper",
    "storage-opts" : ["dm.basesize=20G"]
}

```

Then try :

```
systemctl daemon-reload
systemctl restart docker

```
> 1 votes

# Answer
The only log message there is "Start request repeated too quickly." so we're missing the beginning of the error. You can run as root `systemctl reset-failed` and then `systemctl start docker` followed by `journalctl -u docker --no-pager -n 50` to show the last 50 lines of logs.

Most likely, your problem is that your storage driver is not device mapper, and this option only applies to device mapper. On an 18.03 install, ubuntu will be overlay2 with a recent enough kernel, or possibly aufs on older kernels. Without using the device mapper storage driver, you should not be setting that option.
> 2 votes

---
Tags: docker
---


# Question
Title: What courses would be useful for a DevOps Engineer?
I am a 'DevOps Engineer' based in the UK. I have 2.5/3 years experience in the IT field; starting out as an apprentice now a mid-level engineer learning/practicing Linux/Windows System Administration, Continuous Delivery (Jenkins), Continuous Integration/version control (Git, Visual Studio), AGILE/SCRUM methodology, configuration management (SaltStack), virtual machine technologies (AWS EC2, VMware etc) as well as S3 and BASH/PowerShell scripting.

I haven't got any formal qualifications nor do I think it is *necessary* to excel in the field. However my manager has asked myself and my colleagues to find some courses and/or qualifications which will help us become fully-fledged DevOps Engineers and I think it'd be stupid not to do so!

Could anyone recommend any useful courses/qualifications?

# Answer
> 11 votes
Linux Academy has several webinars and tutorials (including sandbox VMs for hands-on experience) in several technologies favored by DevOps shops including AWS, Docker, Ansible, Jenkins, Puppet and many more.

# Answer
> 4 votes
In my opinion you'll benefit the most if you choose the roadmap offered by RH. Review their courses (incl. DevOps) here

I myself however would also like to hear opinions from other professionals.

# Answer
> 1 votes
I had originally made this a comment, but I think it's worthy of it's own answer. Having been through a few different certification programs, I definitely think the RHCE (RedHat Certified Engineer) program is worth it's weight in gold. The prerequisite is RHCSA (RedHat Certified Systems Administrator), and RedHat does offer a "both-at-once" class where you have 4 days to cover both topics, and two exams on Friday. (I would only recommend this for someone who is already extremely familiar with administrating a RedHat system.)

If your employer is willing to fit the bill, I think that it would be a great start. RedHat also offers quite a few "DevOpsy" type courses as well, like OpenStack/OpenShift, Ansible (a HUGELY popular DevOps product), and many others.

You can see their offerings here.

Edit: I also have seen quite a lot of demand for AWS as well, so that would also be a good place to start. The "Free Tier" Amazon offers is a great place to start playing around with it and get some practical experience.

# Answer
> 1 votes
Microsoft in collaboration with EdX is running a comprehensive DevOps course. It covers Chef/Puppet for configuration management, Selenium for testing, Docker, Nagios, Loggly etc. Obviously being Microsoft they use Azure, but all this tooling is Open Source and so the skills are very transferrable. All the courses are free, you only pay if you want the Certificate.

---
Tags: learning
---


# Question
Title: Mitigating Maven Central risks as seen from the DevSecOps perspective
AFAIK there are two official primary repositories for Maven packages (Java language):

* search.maven.org offered by Sonatype Inc.
* mvnrepository.com offered by a private person @frodriguez

Now obviously it makes sense to have a complete archive for backward compatibility purposes. However already as mentioned back 2013 in the interview with Sonatype's CEO, there are lots of downloads for components with known vulnerabilities. So many seem just to not care.

So is backward compatibility of so much value here? Why not take some action to reduce risks? Of course there is the OWASP dependency check plugin but I think on long term it is possible to implement a deeper risk prevention strategy.

Question: What could be a feasible way to motivate Maven and repository makers to establish some kind of early time warning, like issue special headers on download, or and additionally for example **reject download of vulnerable components *by default***? This would in my opinion save a lot of time and electrical power if not to expect every build on the planet to do the check if this can be done centrally effectively.

UPD. In year 2017, over 80% of projects still seem not to recognize the risks, despite court cases.

# Answer
> What could be a feasible way to motivate Maven and repository makers to establish some kind of early time warning, like issue special headers on download, or and additionally for example reject download of vulnerable components by default?

Your concern, i.e. the meat of your question is a very valid one: how to improve security when downloading "stuff" from the internet. Maven, yes, but also other places like Dockerhub, Debian, Ubuntu etc..

I have no idea what stance the Maven developers have on this; I assume they have thought about it and come to the conclusion that it is not a good idea, for whatever reason. Generally speaking, I see no particular reason to place this responsibility on the repository, though.

Having separate responsibilities between componentes (here, Maven Central as one component) is a very powerful concept in IT. You find it in Unix command line tools, in OOP class design, or server architecture. In this particular case, it seems perfectly valid to me to let Maven Central simply be a huge repository of "stuff". Its main function is to *be there* and supply the packages.

Placing a security filter, if you will, between `mvn` and Maven Central is then pretty easily achieved. For example, a company I work for runs their own hierarchy of Nexuses; while the developers can pretty much do whatever they want on their laptops, the automated build environments have no internet access and must go through curated Nexuses. Each project has their own Nexus (for seldom used packages), and they chain up to a common intra-company global Nexus.

Developers, then, have no way to insert packages into those Nexuses, but a dedicated team reviews new packages and decides whether they can be used.

Note that this extends not only to security, but also, for example, to licenses. Depending on the policy of the company, not every open source license may be acceptable. So this can also be screened for.

Another big reason would be non-security related bugs. Say you find that X of your projects use package Y, and Y is found out to contain a serious bug, maybe with the result of being able to trash all your data. Would Maven Central remove that version from their repository? Certainly not! The bug may only appear in a very small edge case, which, by chance could be very relevant to your company. So, by having your own repository structure inbetween, you can flag the affected versions as "invalid", and thus enforce bug fixes traveling downward to your projects.

Finally, a company might want to phase out old versions of packages as a matter of routine, simply to keep up with the times. Using such a "filter" approach helps with that as well.

Back to Maven Central: why do they not do it? The question is not just a technical one, but also organizational. Who decides what is a relevant breach of security? Are we only talking about actual bugs? Are we also excluding packages that are safe in the hands of a responsible and knowledgeable user, or do we also by default remove packages that lead to possible holes indirectly (i.e., in a Unix package management, one may designate the `telnetd` package as insecure although it is, per se, perfectly fine).

So there you have it. Such filtering is a complex issue, and complex stuff is best distributed to different entities instead of forcing it on one central body.
> 2 votes

# Answer
First of all there is **one** Maven central repository and that is Maven central. The website mvnrepository.com is just a search engine for Maven central.

Secondly there is a gazillion of paid services that already scan artifacts for you and allow you to know about known vulnerabilities beforehand. So that problem is already solved.

Finally several people who download code from Maven central are used for university projects, quick demos, QA environments so of course not all people care...
> 1 votes

---
Tags: culture, security, java, devsecops, dependency-management
---