title: Large number of host emulation

text:
# Question
Title: Large number of host emulation
What would be the best way to emulate (or really to instantiate) a large number of hosts / virtual machines / docker containers in the cloud?

Assume I want to run a scenario where 10,000 hosts are running the same application. Is there a cloud service which provides the ability to "clone" a single host to 10K instances? Is there a way to configure slight changes in those hosts (in terms of Geo location / IP / hostname / MAC address)?

# Answer
you create a Docker Swarm stack file:

```
---
version: '3.1'

services:
  ubuntu:
    image: ubuntu # or your custom Docker image
      deploy:
        replicas: 10000

```

Then, with `docker stack` you can run your 10000 Ubuntu's on one or - probably better in this case - a set of Swarm hosts. This could be bare metal or AWS. Enjoy!

For MAC address is very interesting question which I hope other colleagues here can answer. Geolocation should be no problem as this is AFAIK a system setting - virtual OS does not implement a physical GPS sensor.

Docs give an example how to set a fixed MAC address on `docker run` \- possibly this can be translated to the stack YML file as well.

```
docker run --mac-address=".." ...

```
> 6 votes

---
Tags: docker, amazon-ec2, cloud, vmware
---


# Question
Title: How to connect a Bamboo remote agent from within Intranet
We have a Bamboo server running in the cloud and we would like to connect the machine that is in our local network. The problem is that this PC only has access to proxy that has only port 80 available and company would not allow to open any ports.

We want to avoid setting up new Bamboo instance in our local network.

# Answer
So I am assuming that

1) you have a PC on your LAN. The PC can only talk to port 80 outgoing via a proxy

2) you have a bamboo server running a cloud service

3) you wish to initiate a connection from the bamboo server to the PC

Don't get your hopes up but this answer https://stackoverflow.com/questions/19161960/connect-with-ssh-through-a-proxy

explains how to use a http proxy to tunnel ssh. Unfortunately most http proxies disallow the CONNECT method

Once you have the ssh to the remote cloud service set up then try it again with a -R option to allow a reverse tunnel

Then you should be able to connect via the reverse tunnel back to the Lan PC
> 1 votes

---
Tags: continuous-integration, bamboo
---


# Question
Title: How to refresh user's content of the site after release?
I have an `Angular / Web Api 2` site that uses `ui-router` for SPA behaviour and `gulp` for file transformations.

There is an issue where when I release the site, the user has to refresh their browser to view new content (usually). I have cache busting implemented on my `js / css` files, but it seems that the `HTML` views themselves are a problem. At the same time I can't help feel like I'm missing something blatantly obvious here.

How do I get my tech stack to refresh the user's content (eg. when adding a new field to an existing form) so that the user receives the most up to date version of the site after release?

# Answer
Here is what works for us.

We have a "version" file deployed with the application itself. Then, application issues a "check version" request periodically that checks if the version that the application has initially loaded is different from the version specified in this version file. If it is different, we show a notification that there is a new version available and suggest to refresh the browser window to get it.

I think that's what Google Inbox, for example, does. They have this periodic `checkJsVersion` request and, if there is a newer version of the web app available, they show this information box on the bottom left with a "New version is available, hit Refresh to get it" button.

There is also a different approach when the information about a new version of the app is *pushed from the server to the client*.
> 4 votes

---
Tags: release-management
---


title: Developing inside a Docker image?

text:
# Question
Title: Developing inside a Docker image?
Every time I onboard a new person on my team I have to go through the same routine of installing everything, figuring out what changed since last time we onboarded someone etc. Not fun.

I've heard about people maintaining docker containers with the development environment, but I'm not sure how to accomplish this.

Let's say you work on a repository containing a React webapp. Our IDE of choice is IntelliJ. It'd be cool if a new dev would just get a container with all the necessary dev stuff installed.

How do I make this happen?

# Answer
The onboarding experience should be almost as simple as telling your new developer to just clone the repo and run `docker-compose up`. Personally I wouldn't bother worrying about IDE integration because people might prefer to use different IDEs.

Every project/application (if you have multiple) should be able to run separately and each project/application should have it's own codebase. That's a concept from The Twelve-Factor App methodology.

In the root of each project add a `Dockerfile` and a `docker-compose.yml` file.

> NOTE: I created Craft Kits which you can use to automatically generate these files for the framework/language you're using with one command (e.g. `mason craft nodejs`), you don't even need to know Docker.

The `Dockerfile` will build your application into an image, installing and dependencies your application needs to run. This image is the image that runs your app.

```
FROM node:8.8-alpine

# Set our workdir
WORKDIR /app

# Install and cache app dependencies
COPY package.json /app
RUN npm install

# Add project files
COPY . /app

# start app
CMD [ "npm", "start" ]

```

The `docker-compose.yml` file will spin up your environment. It will define all the services Docker should start for your application

```
version: '2'
services:
  web:
    build: .
    volumes:
      - ./:/app
    ports:
      - "3000:3000"

  mongodb:
    image: mongodb
    volumes:
      - ./storage/data/mongodb:/data/db
    ports:
      - "27017:27017"

```

When you onboard a new developer, you can just tell them to `git clone` your repo and to fire up their development environment by running:

```
docker-compose up -d

```

The advantage of this approach when you are developing locally, running `docker-compose up` will mount your code as a volume.

Then when you want to deploy, when you build your `Dockerfile` it will copy your code into your image so everything is packaged up nicely and easily moved around.

We use the same approach for apps on Codemason, my service for building, managing and scaling container-based applications. When you push new code, we automatically build an image from your Dockerfile. Since that image contains everything your app needs to run, it can be easily launched on any server and scaled up/down across multiple servers.
> 8 votes

# Answer
If you are open to try other things than IntelliJ, there is for example Eclipse Che which runs in container. Ideally there will be plugins for other components of your DevOps toolchains.
> 0 votes

# Answer
There are several ways to achieve it, depend on which one, you like.
The most simple and easy is, get the docker image and create environment in it and save it for later, so when the member joins, you just need to start that image and whole environment is created FYI :
Docker for dev
or
Dev Environment

NOTE: These links are for PHP you need to figure out for other languages that you prefer. Hope this helps
> 0 votes

---
Tags: docker, containers, toolchain
---


# Question
Title: Scheduling collisions possible in IaC - Infrastructure as Code - solutions vs. software defined data center?
To clarify the terms first,

* Infrastructure as Code (IaC) can be delivered with upcoming technologies like Docker Swarm, Kubernetes, OpenShift, OpenStack and possiby some others run on bare metal. - You can code the infrastructure.
* Software defined data center (SDDC) are typically enterprise and mostly more traditional solutions from IBM, VMWare, recently also Nutanix any many others. -You can define the infrastructure through some sort of UI.

Correct me if I am wrong.

Then, typically, my assumption/observation larger companies (much larger than a s startup, much smaller than Google) would provision you with some SDDC resources and there you can play with IaC.

In such a case which I consider yet not very rare, does this setup actually make sense if you want high performance?

An SDDC solution would have some sort of scheduler, and your IaC CPUs are virtual CPUs on real CPUs, so in most pessimistic scenarios your Spark shares resources with say corporate mail server or some backup solution. And there can be priorities set between them, so the sharing is not equal. And collision cases described below from random towards a notorious DOS (Denial of Service) bias.

Collision case 1. Then your Docker Swarm, or Kubernetes scheduler would "see" a machine and say, well, the load there was not so much, so I take it - and then it finds what if was not expecting because the SDDC system scheduler has decided to take it already for something else.

Collision case 2. Just in the moment the IaC scheduer decides to compute something, the SDDC scheduler decides to unschedule it, so what would happen is kind of paralyzing spikes setting IaC to a slow motion mode.

# Answer
> 3 votes
I disagree with your definition, kubernetes, openshift and other are scheduler and as such should go in the SDDC part. IaC things are Vagrant/Terraform/Cloudformation/Vmware Cloud management. They are about writing code to define the infrastructure.

With this in mind k8s may fit both, a `deployment.yml` is IaC to drive the k8s SDDC.

Let take exemple for a k8s cluster running on top of vSphere infrastructure:
You have a scheduler k8s on top of another scheduler vSphere. SDDC on top of another SDDC.

K8s is concerned by its node load (which are vSphere VM), vSphere is concerned by its nodes load (ESXi hosts).
With this in mind k8s will move pods (containers) around its nodes, where vSphere will move k8s nodes around its hosts.

This indeed could lead to a spike or freeze of k8s nodes when both wish to move their loads at the same time. The solution is just an architecture solution: disable vSphere DRS on your k8s nodes so there won't be a move of your k8s nodes around while they are already moving containers.

Each time you overlay systems with the same behavior, you have to plan their definition properly to avoid race conditions.

Worst case would be running a docker swarm cluster on top of kubernetes on top of mesos on top of vSphere infrastructure on top of Nutanix partitions. (Absolutely doable but I don't think anyone would do that out of demonstration purposes)

So yes, this setup makes sense, it mainly require a careful design of all layers to achieve the best performances.

---
Tags: performance, virtualization
---


title: Waiting for Solr to load

text:
# Question
Title: Waiting for Solr to load
By current state of the art, we cannot enforce an ordered start sequence of service containers directly, neither in Docker Swarm nor in Kubernetes. (Maybe there is a standard solution for that via a specialized orchestraton service)

Now, imagine we want to have a Solr instance with considerable amount of data, and want to get it ready before other services can start their work.

What is the best way to verify this status?

It seems for example that calling and validating the expected list of loaded collections is not enough:

```
/solr/admin/collections?action=LIST

```

Other ideas?

# Answer
> 1 votes
This has worked - the following query has to return content-type application/xml:

```
 /solr/COLLECTION/select?q=*:*&rows=0

```

---
Tags: docker, kubernetes
---


# Question
Title: Data sent from Logstash via elastic output plugin not showing in Kibana, but file output plugin works fine - what am I doing wrong?
I have an "ELK stack" configuration and, at first, was doing the standard 'filebeat' syslog feeding from logstash with the elasticsearch output plugin. It worked just fine.

Now I have added a TCP input port (with assigned "type" for this data so as to do if \[type\] == "thistype" to differentiate in filters), its own grok filter, and output to both elasticsearch with its own unique index name and document\_type name and file. When the data comes in over the TCP port it writes the properly formatted data to the output file as expected in the file output plugin but no data is showing up in Kibana when I choose the given index. Kibana recognizes the index from my output configuration and also lists all the fields/keys I assign in the grok filter; however, again, no data is searchable. But it sees the property names (and even adds the propertyName."keyword" property itself on them!) The data is definitely being grok'd properly as it is, as I mentioned, appearing in the file plugin output.

What am I doing wrong here? My configuration is as follows:

```
input {
  tcp {
    host => "10.1.1.10"
    port => 12345
    type => "odata"
    id => "odata"
    codec => line
  }
}

filter {
  if [type] == "odata" {
    grok {
      match => { "message" => "%{QUOTEDSTRING:oid},\"%{WORD:oword1}\",\"%{IPV4:oclientip}\",\"%{DATA:otimestamp}\",%{QUOTEDSTRING:opl},%{QUOTEDSTRING:oos},%{QUOTEDSTRING:oua}" }
      remove_field => "message"
    }

    date {
      match => ["otimestamp", "YYYY-MM-dd HH:mm:ss Z"]
    }

    mutate {
      remove_field => "otimestamp"
      remove_field => "host"
      remove_field => "@version"
    }
  }
}

output {
# the if .. is here because there are other types that are handled in this output since I centralize the input, filter, and output files to three distinct files.
  if [type] == "odata" {
    elasticsearch {
      hosts => ["10.1.1.1:9200", "10.1.1.2:9200"]
      sniffing => false
      index => "odataindex"
      document_type => "odatatype"
    }
    file {
      path => "/tmp/odata_output"
    }
  }
}

```

Again, the grok filter is fine; the data is ingested via tcp just fine; and the file output plugin outputs the interpreted/grokd fields fine. Kibana sees the "odataindex" index and the fields (such as oid, oclientip, oua, etc.) from the grok filter. It just doesn't return any data when I do a search.

Any ideas? I am new to elastic/kibana/logstash and would also appreciate any tips on debugging these things.

Thank you in advance!

# Answer
> 1 votes
Being unfamiliar with Kibana, I was not aware of the time constraint on data on the default search/display of just 15 minutes. The data coming in was timestamped (@timestamp key) via the 'date' plugin with the original open date, NOT the time of the, via TCP port to elastic, insertion event; thus, no data was showing and I had no idea that by default only the past 15 minutes of data based on @timestamp were displayed. If I had read the part about the time constraint I would have known. So I just adjusted the time to go back infinite years and saw my data.

So, if anyone else is having this problem, it's probably because you created a time-dependent index and have not clicked the 'time' button in the top right corner and changed the time frame.

This, my friends, is why you read the manual!

---
Tags: elastic, logstash
---


# Question
Title: Git seems to recognize http.sslcainfo property from .gitconfig but ignores it on execution? SEC_E_UNTRUSTED_ROOT
all, I have a working certificates chain (testable with OpenSSL) but somehow I cannot manage to tell Git to load these certificates.

I get **the same** "untrusted root authority" error (SEC\_E\_UNTRUSTED\_ROOT) **independently** of whether my git configuration points to an existing or a fake certificate chain file.

For details, please check the attached screenshot.

Setting I use in `.gitconfig` for fake file:

```
sslCAInfo = C:/tmp/foobar.crt

```

or, for real file which works with OpenSSL:

```
sslCAInfo = C:/tmp/ca-bundle.crt

```

Console transcript:

```
C:\tmp>openssl version
OpenSSL 0.9.8h 28 May 2008

C:\tmp>git --version
git version 2.12.2.windows.2

C:\tmp>git config --list
http.sslverify=true
http.sslcainfo=C:/tmp/ca-bundle.crt

C:\tmp>dir
24.04.2017 13:45 10.875 ca-bundle.crt

c:\tmp>openssl s_client -state -connect https://mygithost:443 -CAfile .\ca-bundle.crt

Verify return code: 0 (ok)

C:\tmp git clone https://mygithost/bitbucket/scm/my.git
Cloning into ...
fatal: unable to access ... : schannel: next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.

C:\tmp> git -c http.sslverify=false clone https://mygithost/bitbucket/scm/my.git
Cloning into ...
Resoliving deltas: 100%, done.

C:\tmp>git config --list
http.sslverify=true
http.sslcainfo=C:/tmp/foobar.crt

C:\tmp\xxx\git pull
fatal: unable to access ... : schannel: next InitializeSecurityContext failed: SEC_E_UNTRUSTED_ROOT (0x80090325) - The certificate chain was issued by an authority that is not trusted.

```

# Answer
> 2 votes
Been hit by this just today after a fresh install so here's how I got over it:

From your logs (emphasis is mine):

> fatal: unable to access ... : **schannel**: next InitializeSecurityContext failed:

git is configured to use schannel (windows native implementation), but schannel use windows certs bundle and not a the cainfo bundle.

To switch to openssl to use a custom bundle file use this:

`git config --system http.sslbackend openssl`

And now git will honor the CA bundle passed in http.sslcainfo.

Alternatively as my error was a problem of cypher suite you may be interested by the following links:

# Answer
> 1 votes
I had something similar a year ago, so I hope I remember -

You should have all the certificate chain in the crt file. Verify that you have all the intermediate certificates between your bitbucket certificate and the root one (inclusive).

You can have a look at the correct format for appending them in the pem format (which I believe is your case with the crt) over here.

My certificate was signed by Comodo when I had that issue, their certificate and the intermediate weren't in the default ca-bundle supplied by most systems.

Hope that helps!

**Edit:** Now I noticed it happens also with the fake one - try to check the systems default known certificates (each system has such). On windows you can go to the Windows Certificates mmc snap-in.

---
Tags: git, windows, ssl
---


# Question
Title: What's the term for moving/promoting code between deployment environments
I'm looking for a term that unambiguously describes the process of **promoting** a "deployment bundle" (for lack of a more accurate/technical term) from one deployment environment to another, from Development towards Production (or whatever environments there might be in between). I am talking specifically about the context of release management, where each environment has its own dedicated purpose (development, integration testing, user acceptance testing, staging, production, etc.), and which succeed each other in a defined, usually the same, order (of importance or quality).

I had previously thought this process was called **staging** (since I heard the term in this exact context, in multiple projects and teams), before research - especially this discussion: What is, or could be the formal and/or officially established definion of a "stage"? \- proved me wrong.

My guess is that I might end up using the term **deployment** itself, but I'd like to further differentiate between the processes of *deploying source code to a server* and *moving to a different environment in accordance with some specific process* (e.g. end-to-end test successful, reviewed by release manager, etc.), especially since there might be a different(ly implemented) process for each.

More broadly speaking, I'd like to clear some ambiguity in DevOps terminology, so please also correct any wrong assumptions I might have made in my question.

# Answer
I don't think there is a generic term for promoting to production. I can tell you that within the devops community, terms are frequently interchanged. As long as the point being addressed is well thought out and made clear, the point can be understood. Some of the terms that are used when referring to the movement between environments are **deploy**, **propagate**, **move**, **promote**, and **release**. I will say though, whenever I hear the term **stage**, I usually associate it with a pre-production environment. I think most others would assume the same.

Within your organization/groups, I think it is important to try to be consistent with your terminology. This eliminates any confusion and assumptions people can make. An example would be, in my organization, we use the term **deploy** whenever we are referencing moving between environments except when referring to the production environment which we use the term **release**. This is to emphasize that it is the final movement in the pipeline/process.
> 6 votes

---
Tags: terminology, deployment, release-management
---


# Question
Title: PRs not automatically getting trigger in Jenkins github organization
We are using Github for source control and Jenkins for our builds.

In Jenkins, we are using GitHub Organization Folder plugin along with Multibranch pipeline plugin. This causes the repository to show up in Jenkins, and each repository has a Jenkinsfile in it. And each PR(in github) has a status check, which checks if the build of the PR is good or not.

Everything works fine, except for the fact that once a PR is opened or updated, the PR is not built in Jenkins automatically. You have to manually go to Jenkins and trigger the PR build.

As far as master branch is concerned, it works like a charm. If the master branch is updated, Jenkins automatically triggers the build.

I have attached the screen shot of the branch configuration in Jenkins for the GitHub Organization. So my question is what should be done for PR opening or updating, that would cause Jenkins to start build automatically.

# Answer
> 3 votes
It sounds like maybe you need to configure a webhook for git if you are using the git plugin, but I think that may just build the master branch. I haven't used git much, so you might be able to use the web hook to have it build the pull request - I don't know.

Instead, you might want to check out the pull request builder plugin and see if that does what you are looking for.

# Answer
> 1 votes
According to the official documentation for the GitHub Branch Source plugin:

> While the build triggers are often enough, you can set up webhooks to automatically trigger builds when changes are pushed to your GitHub repositories. To do this you must have a GitHub login with a token.
>
> 1. Go to the main configuration settings page, Manage Jenkins \> Configure System
> 2. In the GitHub Plugin Configuration section, add a server with your credentials
> 3. If you need a token, generate one with the Additional Actions \> Convert login to password and token
>
> You can also configure this manually through GitHub itself by registering the URL provided in the help section of the server config.

---
Tags: jenkins, jenkins-pipeline
---


title: Parallelizing dependent CI subjobs

text:
# Question
Title: Parallelizing dependent CI subjobs
Imagine you have resources for a job which has part 1 and part 2.

Part 2 requires part 1, but both take considerable time.

And, part 2 requires part 1 only at the end.

Would you start them both in parallel and let part 2 somehow wait for part 1 getting ready? (chances are it 50/50 it will be ready by then)

# Answer
> 4 votes
The critical question is - can you teach your job #2 to patiently wait for job #1 to complete when it gets to that dependency point and appropriately proceed or bail depending on job #1 result?

If you can't, then the answer is no - you'll have more trouble dealing with #2 failures due to incomplete job #1, too much noise in the process.

If you can (and if you have enough resources) - yes, by all means - your process will be faster. You *might* not even need special support from the CI system for such dependencies, it could be completely oblivious, it would simply see the 2 jobs as parallel.

A middle-ground possibility, if your CI systems supports it, would be to stagger the jobs: start job #2 with a delay sufficient to bring the chances of job #1 completing before job #2 needs it in a much more comfortable range (say 99%). The delay would be plain eyeballed or empirically determined, either fixed or, in a more advanced approach, automatically derived from historic measurements/stats (if available).

# Answer
> 1 votes
Yes, you should definitively run both parts in parallel if you have the resources. This means that your job will run as fast as your slowest running part. This puts emphasis on optimizing your slowest running part. It might be something as simple as running some processes of part 1 ahead of time before part 2 even begins.

---
Tags: continuous-integration, automation
---


title: Dynamic service redirections in Docker

text:
# Question
Title: Dynamic service redirections in Docker
Imagine you have decided to abstract from exact service location.

Normally, you have a cody world with the service name=DNS.

Say, you have prototyped something with S3 compatible API like exoscale/pithos.

But then you think okay this was actually a mockup and now you go for real S3 in production.

Would you rewrite the stack, trick around with `${ENV_VARIABLES}` or is there some more elegant way to proceed?

# Answer
> 1 votes
As per https://12factor.net/config, configuration like this should be stored as environment variables.

> The twelve-factor app stores config in environment variables (often shortened to env vars or env). Env vars are easy to change between deploys without changing any code; unlike config files, there is little chance of them being checked into the code repo accidentally; and unlike custom config files, or other config mechanisms such as Java System Properties, they are a language- and OS-agnostic standard.

A service that you connect to is suitable to be defined as configuration, so the use of env vars is appropriate here.

---
Tags: docker, microservices, networking
---


# Question
Title: Why it is recommended to run only one process in a container?
In many blog posts, and general opinion, there is a saying that goes "one process per container".

Why does this rule exist? Why not run ntp, nginx, uwsgi and more processes in a single container that needs to have all processes to work?

blog posts mentioning this rule:

# Answer
> 123 votes
Lets forget the high-level architectural and philosophical arguments for a moment. While there may be some edge cases where multiple functions in a single container may make sense, there are very practical reasons why you may want to consider following "one function per container" as a rule of thumb:

* Scaling containers horizontally is much easier if the container is isolated to a single function. Need another apache container? Spin one up somewhere else. However if my apache container also has my DB, cron and other pieces shoehorned in, this complicates things.
* Having a single function per container allows the container to be easily re-used for other projects or purposes.
* It also makes it more portable and predictable for devs to pull down a component from production to troubleshoot locally rather than an entire application environment.
* Patching/upgrades (both the OS and the application) can be done in a more isolated and controlled manner. Juggling multiple bits-and-bobs in your container not only makes for larger images, but also ties these components together. Why have to shut down application X and Y just to upgrade Z?
    + Above also holds true for code deployments and rollbacks.
* Splitting functions out to multiple containers allows more flexibility from a security and isolation perspective. You may want (or require) services to be isolated on the network level -- either physically or within overlay networks -- to maintain a strong security posture or comply with things like PCI.
* Other more minor factors such as dealing with stdout/stderr and sending logs to the container log, keeping containers as ephemeral as possible etc.

Note that I'm saying function, not process. That language is outdated. The official docker documentation has moved away from saying "one process" to instead recommending "one concern" per container.

# Answer
> 67 votes
Having slain a "two processes" container a few days ago, there are some pain points for me which caused me to use two container instead of a python script that started two processes:

1. Docker is good at recognizing crashed containers. It can't do that when the main process looks fine, but some other process died a gruesome death. Sure, you can monitor your process manually, but why reimplement that?
2. docker logs gets a lot less useful when multiple processes are spewing their logs to the console. Again, you can write the process name to the logs, but docker can do that, too.
3. Testing and reasoning about a container gets a lot harder.

# Answer
> 21 votes
The recommendation comes from the goal and design of the Operating-system-level virtualization

Containers have been designed to isolate a process for others by giving it its own userspace and filesystem.

This is the logical evolution of `chroot` which was providing an isolated filesystem, the next step was isolating processes from the others to avoid memory overwrites and allowing to use the same resource (I.e TCP port 8080 for example) from multiple processes without conflicts.

The main interest in a container it to package the needed library for the process without worrying about version conflicts. If you run multiples processes needing two versions of the same library in the same userspace and filesystem, you'd had to tweak at least LDPATH for each process so the proper library is found first, and some libraries can't be tweaked this way, because their path is hard coded in the executable at compilation time, see this SO question for more details.
At the network level you'll have to configure each process to avoid using the same ports.

Running multiple processes in the same container require some heavy tweaking and at the end of the day defeat the purpose of isolation, if you are ok to run multiples processes within the same userspace, sharing the same filesytem and network resources, then why not run them on the host itself?

Here is the non exhaustive list of the heavy tweaking/pitfalls I can think of:

* Handling the logs

Either being with a mounted volume or interleaved on stdout this brings some management. If using a mounted volume your container should have its own "place" on host or two same containers will fight for the same resource. When interleaving on stdout to take advantage of `docker logs` it can become a nightmare for analysis if the sources can't be identified easily.
* Beware of zombie processes

If one of your process in a container crashes, supervisord may not be able to clean up the children in a zombie state, and the host init will never inherit them. Once you exhausted the number of available pids (2^22 so roughly 4 millions) a bunch of things will fail.
* Separation of concerns

If you run two separated things, like an apache server and logstash within the same container, that may ease the log handling, but you have to shutdown apache to update logstash. (In reality, you should use the logging driver of Docker)

Will it be a graceful stop waiting the current sessions to end or not ? If it's a graceful stop, it may take sometime and become long to roll the new version. If you do a kill, you'll impact users for a log shipper and that should be avoided IMHO.

Finally when you have multiple processes you're reproducing an OS, and in this case using a hardware virtualization sounds more in line with this need.

# Answer
> 11 votes
As in most cases, it's not all-or-nothing. The guidance of "one process per container" stems from the idea that containers should serve a distinct purpose. For example, a container should not be both a web application *and* a Redis server.

There are cases where it makes sense to run multiple processes in a single container, as long as both processes support a single, modular function.

# Answer
> 5 votes
The process I'll called as service here, **1 container ~ 1 service**, if any of my service is failed then I'll only spin up that respective container and with-in seconds everything is up again. So, there won't be any dependencies between services. It is best practice to keep your container size less than 200 MB and max 500 MB (exception to windows native containers are more than 2 GB's) otherwise, its going to be the similar as virtual machine, not exactly but, performance suffice. Also, take into consideration few parameters as scaling, how could I make my services resilience, auto-deploy, etc.

And, its purely your call how you need to make your architectural patterns like micro-service in polygot environment using the containers technology that best suite your environment and will automate the things for you.

---
Tags: docker, containers
---


title: How to store docker images in Nexus OSS

text:
# Question
Title: How to store docker images in Nexus OSS
The following documentation was found that indicates that:

> simply create a new docker (hosted) repository as documented in Repository Management

The problem is that the format docker seems not to be available when trying to configure a "Hosted repository"

# Answer
> 1 votes
Nexus 2 was deployed instead of Nexus 3. The docker feature seems to be introduced since Nexus 3.

This blog indicates how to configure a docker repository and how to push and pull docker images.

The blog contains multiple pictures. The following picture indicates how to enable the docker registry:

---
Tags: nexus, n3dr
---


# Question
Title: Strange orchestration behaviour of a Docker Swarm manager - pile of containers
For some unknown reason, the manager host which gets the command to launch a stack, tries to pile up all the containers on itself; other worker/manager hosts stay unemploed.

Unfortunately there are no log entries for that.

Has anybody observed this behaviour already with Docker CE as well?

# Answer
I would check that all nodes are active and ready with command

```
docker node ls

```

With

```
docker swarm leave

```

and

```
docker swarm join \
--token <SWMTKN token> \
<manager private IP>

```

you may rejoin malfunctioning node.
> 1 votes

---
Tags: docker, docker-swarm
---


# Question
Title: What is the optimal speed relation of CI/CD jobs to a (reference) local machine?
Today, you could more or less setup a CI/CD toolchain by youself, even locally. Indeed I have seen teams doing it in an ITSM environment because the communication with management did not work.

Nevertherless we know CI/CD is a very important, if not most important, company production line - it produces value, even if it is not seen so by ITSM to give it operations status and priority (in short, SLA for systems which should survive project time instead of setting them up in every project and department over and over).

Let's ignore for simplicity network downloads.

Question: what should be the execution speed of CI/CD relative to a setup on a local machine (if possible):

* Same is ok?
* Faster - how much faster?
* Slower?

# Answer
You want the execution speed of the CI pipeline as fast as (practically) possible, irrespective of being a local or shared setup or the time required to set it up - this is what effectively dictates the overall development speed in the respective project.

But it should be noted that by speed here I don't mean the *duration* of each CI execution. The duration of such execution may be directly related to the coverage of the CI QA verification, which translates into stabiliity of the development process and the quality of the product. For example including a smoketest in the CI verification will increase the duration of the CI execution, but will also raise the product quality - it's a good thing.

But I have to mention: such private CI setups (even those SLA-backed CI setups you mention) are often an indication of what the ThoughtWorks team called the CI Theatre. Simply using CI tools doesn't make the process a CI one. Using project/feature/integration branches which still require subsequent branch merges (other than cherry-picked fixes) before release means development is still done in silos (those branches are the silos) - waterfall style.

Side note: Frankly, if I'd be part of a team in which management doesn't understand the importance of CI (let alone not being able to effectively communicate on such critical subject) I'd look for another job - I doubt they'll remain in business for very long ;)
> 3 votes

# Answer
They should be roughly equal.

If the CI system is faster than running it locally, your local development is going to be uncomfortably slow.

If your CI system is much slower than running it locally, it will become something that is an obstacle rather than something provides value.
> 0 votes

---
Tags: methodology
---


# Question
Title: Adam Smith vs. fullstack developers - and productivity in DevOps?
By Adam Smith, labour division can make you *by 240 times* more effective (on example of a pin factory producing pins in 18 steps).

**Why then are multi-skilled roles are so in demand if this actually reduces productivity - or was Smith just wrong, why then?**

Searches for "fullstack developer" still trend on Google, however apparently slower than two years ago:

=====

To sum up, a *full stack developer* would be able do virtually all the value chain (correct me if I am wrong):

* Discuss with customers and refine workable agile requirements for his part of the job
* Decide which architecture, tooling and components pick up - just give him a notebook
* Write code for frontend, backend, ingration, which is cross device compatile and does not requrire much testing, or includes it
* Profile and scape data, use Cloud AI/ML APIs for advanced features
* Write up required IaC code and rollout
* Be on call in case of error or sales processes
* Be aware of security relevant design, overall patching, migration and modernization
* Account time table in a scroutinized way to ease invoicing of the employer
* ... did I forget anything?

UPD - "*we need the productivity of specialisation but we don’t want the insular worldview of “extreme division of labour*”. (DevOps Guys, "DevOps, Adam Smith and the legend of the Generalist", 2013-2016)

# Answer
> 12 votes
There are two types of work:

1. **Exploitation** \- Well defined work that can be easily divided into well-defined stages, where each stage can be learned and mastered on its own and handover between stages does not require communication.
2. **Exploration** \- Undefined work, which requires learning and experimentation to accomplish each stage and handover between stages requires massive amounts of communication of all learning and status of the project.

Adam Smith concerns himself wholly with exploitation and not at all with exploration. The work done in **Research & Development** departments of the industry is by its definition mostly exploration and so it is not covered in any way by Adam Smith.

But we have seen that in later continuous improvement stages, which are partly exploitative work, the application of CI/CD can bring similar gains in productivity, which could in some way be probably traced down to Adam Smith by someone very imaginative.

# Answer
> 6 votes
Adam Smith did not need to consider passing of information from one stage to another. This is a critical part of any significant IT project. So a fullstack developer has the significant advantages that:

* they don't have to talk to anybody in another department to get things done
* they don't have to wait for those other people to get around to it
* there's much less chance that something will get lost in translation between one layer and another

For more on the importance of information passing in IT projects see Fred Brook's *Mythical Man Month*.

# Answer
> 4 votes
IMHO the answer has a lot to do with scale and resource availability.

I believe Adam Smith's theory can only be applied at large scale - entire nations/economies in the original context, or, in the SW development context - large development teams.

In a large team it is, indeed, more efficient to hire more specialized human resources:

* they aren't rockstars - often considered a danger sign in such large organisations
* they're typically easier to find, cheaper that those with wider expertise spectrum
* they typically don't raise attrition risks - for example they won't be unhappy because they only use a subset of their skills.
* in a (maybe weird) devopsy comparison the "cattle vs pets" principle can be applied in such larger organisations, and for quite similar reasons. These specialized resources are, from the business perspective, literally just capita in human resource pools, which sizes that can be rapidly adjusted as needed, typically by hiring/firing.

Oh, and such teams can only be functional if they are complemented by high-quality architect resources which would be necessary to split the product in the smaller, specialized tasks that can be addressed with the specialized resources.

In smaller scale or even one-man teams (typically startups or even isolated smaller teams in larger organisations) it's ineffective or even impossible to use such resources and still get the job done:

* they simply don't have the budget/resources to hire the many different specialized resources necessary to cover the entire product development
* they actually seek/appreciate rockstars which can wear multiple hats and immediately switch roles with great flexibility, without incuring delays and additional HR costs

# Answer
> 3 votes
I consider myself as a fullstack developer on the basis of the following combination of responsibilities:

**Front end and back end programming**

I can do UI changes until some extent: write html, css (as a web developer) and in other hand in some extend provide data to the UI from database, provide it in a service etc.

I leave testing, architecture and those aside, meeting customers may be added to the working description.

**Opposite**

The opposite for my point of view would be strict roles of UI guys and back end guys.

**Conclusions**

I don't see full stack really that full as you mentioned, more like a fancy expression like agile or cloud that in certain conditions just have to be mentioned to attract people's attention and the real implementation may vary vastly. At least about scrum and agile I have seen so many variations that the terms have run out of meaning.

# Answer
> 3 votes
In short, I don't think Adam Smith was wrong, but I do think there are some serious differences between his model of labor division in manufacturing and silos in software development.

First, the pin factory example (as far as I know) was merely hypothetical; although most modern manufacturing factories can trace their roots to this division of labor, I am unaware of any studies that have actually tested scientifically this hypothesis.

Second, Smith was primarily concerned with manufacturing material goods; there are certain tangible outputs associated with material production that do not have similar analogs in software development. For example, in pin making, physical dimensions are important as a functional requirement; there's no obvious comparison to that in software. This is important because tangible objects can be replicated through exact repetition; software development is never the same problem twice. Developers develop common methods to produce predictable results, but you never code the same problem twice. Any component developed in the stack has intricacies unlike physical components, and those intricacies have interactions beyond tangible measurement (height, weight, length, etc). A pin pointer doesn't care how the wire cutter works, as long as the wire gets cut according to spec. In software development, the boundaries are never quite that clear.

Full stack developers are not expected to do all of the work themselves (they're not intended to be a single pin maker), but they are expected to be capable of understanding all of the elements of the stack and how those elements interact. A full stack team should be comprised of T-shaped individuals who specialize in one or more areas, but understand the spectrum (and may be able to step in at some level).

Where I think Smith's work holds true in software development is in the area of context switching (or multitasking); if a single developer is responsible for all areas of development, it takes time to shift from responsibility to responsibility. At scale, collaboration between team members with different experiences in the same product team can balance context switching and complex interactions.

# Answer
> 3 votes
An important point to understand is that labour division doesn't always mean a different person per step.

I'd take my own history in a car factory, I was on a seat assembly chain, to get a full seat with airbag, leather, head rest etc. the chain was split in 9 stages. We were 9 working on the chain. Each person was doing only one stage at a time, but every hour we were shifting on the next stage to avoid too much repetition. The workday was 8h long so we didn't get on each stage every day.

This is exactly a labour division where you do only one step of the assembly at a given time, that doesn't prevent you to be able to do the full assembly.

As already stated in other answers this is a little different than software development but I think this sheds light on why a Full Stack developer isn't mutually exclusive with labour division, someone with the abilities to handle the whole life cycle of an application is not required to do it all the time.

Generally speaking when I hear FullStack developer I think more of someone able to code efficient back-ends and nice UI at the same time, in opposition with Front and Back Dev.

---
Tags: culture, project-management
---


# Question
Title: A systematic way to test RESTful APIs with curl?
I have noticed during integration testing that I actually work on recurring use cases with restful APIs (or in general HTTP interfaces) I check here and there with bash+cURL.

It starts looking quite messy and gets harder to maintain. Why deliver mess?

Typical use cases are:

* Check that an URL returns http response code e.g. 200
* Check that content type matches some MIME you need in that case
* Check that returned content matches some pattern, or passes an abstract validation procedure

What I have found so far and consider a workable option without reinventing the wheel, is:

* have a go with PyCurl - in a hope it fully implements all cURL options esp. proxying but also other switches I might need
* use Python's built in unit testng

Then I could have, for example, one unit test per service I'd like to check:

```
import unittest, pycurl

class TestService (unittest.TestCase):

    def test_1(self):
        self.assertEqual(pycurl.returncode("some_url"), 200)

    def test_2(self):
        self.assertTrue(pycurl.response("some_url").matches ("xxx") )

    def test_3(self):
        self.assertTrue (pycurl.ContentType("some_url").equal("xxx"))

if __name__ == '__main__':
    unittest.main()

```

Does this make sense or is there a more high-level (but not too complex to pick up and integrate) tooling?

# Answer
You could look at tools such as Postman which focuses on testing REST APIs with JavaScript - it has some nice features but you lose the use of Python.

Instead, I'd suggest looking at REST-related plugins for pytest, a Python test framework that simplifies your test code, while still running tests written using `unittest`.

Pytest has a huge set of plugins that simplify various tasks, including:

* Tavern, which specialises in testing REST APIs and seems highly relevant here - sort of "Postman for Python unit tests".
* pytest-curl-report \- when testing with the `requests` library, will print a `curl` command you can use to reproduce the error from shell.
* Testinfra \- focuses on server testing (e.g. state of OS packages, files, processes, etc, usually tested on remote servers) - strongly recommended if you also need this type e.g. to test Ansible code.

+ For those who use Puppet or Chef, Testinfra is similar to Beaker (with RSpec), ServerSpec or InSpec.

If you don't like Tavern, you can of course use `pycurl` with `pytest`, which makes it easier to diagnose exactly what failed. This example from the pytest-curl-report site uses only generic pytest features:

```
$ py.test test.py
============================= test session starts ==============================
platform darwin -- Python 2.7.9 -- py-1.4.27 -- pytest-2.6.4
plugins: curl-report, httpbin, cache, capturelog, cov, flakes, pep8
collected 1 items

test.py F

=================================== FAILURES ===================================
______________________________ test_requests_get _______________________________

    def test_requests_get():
        r = requests.get('http://httpbin.org/get')
>       assert False
E       assert False

test.py:7: AssertionError

```

Pytest lets you write all tests with a plain `assert`, and optionally include a helpful message as part of the output. For example, one of your tests could be written:

```
def test_2():
    assert pycurl.response("some_url").matches ("xxx"), "xxx not found in response"

```
> 9 votes

---
Tags: testing, continuous-deployment, python, curl
---


# Question
Title: Teaching an old dog new tricks: Finally adopting the micro-service architecture
Is it too late to teach an old dog a new trick?

I've always been the guy with a monolithic application/codebase.

For more than 10 years now, I've gotten good at caching, writing good SQL queries, using Redis to store frequently called read data, using background jobs to perform updates, edits, long-running tasks, e.t.c

Then again, I haven't had to deal with more than 100 million rows in a db before.

Not until now, I've used up all my tricks and alas no significant improvement.

Now I'm coming for advice from you smart devops people.

For my current app, just recently I had a rails app, my DB, and some script in the /lib, Redis, Sidekiq, all on one, you guessed it, server.

I read up on Docker and now I have Sidekiq, Redis, my app, and my db all on different containers.

Now, I'd like to split some functions on my app itself. This is where I need your help.

For example:

Script A: I have this ruby script(a ruby class) that crawls the web for backlink data. Script A only gathers links and removes links that have been parsed already.

Script B: Another script that gets the links script A collected and retrieves the information I want from that link.

Let's stop at that for now, so it doesn't get confusing.

I'd like to run these scripts on their own servers, and they would connect to my app via api's.

Both scripts are less than 1000 lines of code and are written in Ruby.

Requirements

1. Both scripts require Selenium, Watir, and headless chrome to run.
2. The server must be able to autoscale as needed.
3. If the server is rate-limited or stuck in a link blackhole, I should be able to spin up a new server(host), with a different wan IP, with all the content of the script in as little time as possible.
4. The server should be affordable, as it's these are long running tasks. The script does not require a database as it only sends back important data back to my app via API.
5. In short, both scripts should be on their own server (host) and it should be easy to spin up a new one in no time.

How do you suggest I go about it?

# Answer
In your approach I see a few scalability problems - I'm judging from a Google App Engine (GAE) context, where scalability is achieved via breakdown of the work in small tasks/work items and strict limitation of the response time for a task execution:

* you're duplicating some piece of functionality: both scripts need to parse the same page, leading to your requirement #1
* the task execution time for script A is fundamentally unbound, it can vary widely depending on the number of links discovered which need to be checked if already crawled

I'd approach this a bit differently to avoid these issues. I don't even see a need for separate services for this particular job:

* the service pulls individual page URLs as separate tasks/work items
* it checks if the URL was already crawled and, if so, does nothing, otherwise continues
* gets the page content and parses it to extract 2 kinds of info:

+ the list of other URLs to crawl, creating and enqueueing new work items for each one of them (it doesn't check here if they were already crawled, which may take a long time depending on the number of URLs)
    + the other info needed from the page

In this approach the duration of each task execution is reduced to the minimum necessary: one page parsing and only one check if a URL has already been crawled. And each page is only parsed once.

The number of pending work items would be the driver for scaling - if it goes above a level new instances can be started, if it drops below a level and multiple instances are running some may be stopped. If implemented as a GAE app the GAE infra can take care of that for you automatically.

This solution can be implemented using a noSQL database (like Google Datastore, in a GAE context), which can scale better than a SQL one.
> 4 votes

---
Tags: amazon-web-services, microservices, serverless, servers
---


title: Automated Testing Jobs

text:
# Question
Title: Automated Testing Jobs
We have multiple environments close to 7 for an application that has UI and also integration server that accepts SOAP xmls.

Its hard to maintain the environment integrity when many people working, developing, testing on these servers. Almost every 2nd day one or the other environment has issue and lot of effort and time goes in fixing that.

What we are looking for is an automated testing tool that has a capability to run/schedule test cases both UI based (like macros) and XML based. Lets say this tool runs the test cases every morning on each of the environment and sends out an email report of the output/results.

Based on your experience which tool fits best to our requirement. Please suggest.

Regards, Viv

# Answer
> 3 votes
I see a mismatch between the problem you describe and the solution you're asking for. Your problem seems to be managing your environments. You seem to think automated testing is the answer.

But why not automate managing your environments to prevent the manual effort that goes into fixing them?

It's very important to reset test environments to a known good state before running automated tests. Without a reset, it takes effort to figure out if a test failure is due to code changes or environment changes. It's about the same effort as manually fixing the environment in my experience.

There are many approaches for creating or resetting test environments. Probably it's a too broad question and a matter of opinion.

Once you have a repeatable way to provide a known good test environment, there are many tools for automating regular test runs and emailing results. Probably it's a too broad question and a matter of opinion.

---
Tags: automation, testing
---


# Question
Title: Sarbanes-Oxley (SOX) Versioning Strategy
When dealing with publishing SOX application artifacts what is the best strategy for versioning SNAPSHOTS?

I'm currently following the semver pattern `<major>.<minor>.<patch>` and wondering if it is necessary to get granular with the version strategy and append something like a git hash to the end? e.g. `<major>.<minor>.<patch>-SNAPSHOT<githash>`

Or can I treat SNAPSHOTS as a SNAPSHOT should be and drop the hash all together?

My argument for the latter is that SNAPSHOTS are not cut as release candidates (rc) until they actually enter the release pipeline and therefore can be clobbered, especially when following a mature CI pattern where changes are continuously merged to master following early testing validation.

In addition to this, typically in an organization, a change request is only opened and tracked at the time you are performing a release.

(additional convo happening here)

# Answer
> 1 votes
I suggest to follow the principles that are defined in the Semantic Versioning 2.0.0 documentation.

---
Tags: versioning
---


# Question
Title: From 6 months to 16 seconds? grokking DevOps speedup data from DevOps report by DORA and Puppet Labs
Puppet Labs' annual DevOps reports are a very nice and representative source of information, AFAIK. Indeed, with Kim Gene on board they are THE industrial gurus I'd say.

Now I have a question to the following data in this year's report (page 21).

Can we multiply the data for 2016 and 2017?

For example, take the lead time for changes.

```
2555x440=1124000 # more than a milion times faster?

```

Does this mean then, if in 2015 it took 6 months (say 180 days real time i.e. 4320 hours) to deliver a bugfix, then

* in 2016 it took just under 2 hours,
* and in 2017, *16 seconds*?

Is the warp speed delivery speed real now? In 2014, management was very sceptical about even 30x speedup. Now I'm sceptical.

**Question:** this must be absolute minimum values for very lightweight microservices. That is, as the bugfix commit arrives..

* you really count from this moment
* you have no compilation step
* the Docker image is very small
* you have really fast (or, fastest you can get) infrastructure.

Right?

# Answer
> 5 votes
I wouldn't exactly calculate it like that. It doesn't necessarily mean that the *actual* verification time for a changeset is decreasing that much.

The verification process itself can be (and often is these days) a processing pipeline in which the *average/effective* verification time per changeset can, indeed, decrease to insanely small amounts.

For example the processing pipeline can process multiple changesets simultaneouly. If you have, let's say, 60 proposed changes, each one of them touching a different file in a repository, you can combine them into an equivalent single uber-changeset touching 60 files. You could then process this uber-changeset through a verification step that takes, let's say, 1h. If the verification passes you obtain a speedup factor of x60 and an effective verification time per changeset of 1 min, even if the actual verification time was 1h. Granted, you'll lose some of the speedup when the verification fails and you need to perform bisections to identify the changeset at fault, but that's a different story.

Or you can have each of the changesets verified independently, through 60 1h verifications performed in parallel. Same effective verification time, but with a whole lot higher verification resource usage.

The point, I guess, is that there are ways of building processing pipelines which can greatly reduce the effective per-changeset verification times and still keep very large scale projects agile.

Another point to consider is the branching strategy. Those 6 months weren't the actual verification time. Most of that time was simply spent navigating through the branch spagetti and fighting the integration hell, just to propagate the fix into the proper shipping branch - effects of the waterfall development model which was considered the norm at the time.

Today CI/CD (the proper one, not the CI Theatre) and Trunk Based Development eliminate all that wasted time/effort and side-stepping, with the fix being virtually ready for delivery right from the moment it is integrated in the trunk - no unpredictable massive branch merging obstacles ever stand ahead of shipping.

---
Tags: speed
---


# Question
Title: How can I automatically post Codefresh build results to pull requests in Bitbucket?
I have a BitBucket Cloud repository with a WebHook to trigger Codefresh builds on Pull Requests. I would like the build results to appear as comments on the PRs, similar to the way Travis integration works for GitHub.

I couldn't find any settings for this in my BitBucket repository or in Codefresh, but Codefresh's documentation seems to suggest that this happens automatically when you setup integration with GitHub.

I can't find any examples of BitBucket repository with automatic comments. Is it supported?

# Answer
> 3 votes
One could update the build status in bitbucket as follows:

> Adding a build result to a commit
>
> To associate a build result with a particular commit, you need to POST a JSON object to the build status REST resource at:
>
> `https://<bitbucket-base-url>/rest/build-status/1.0/commits/<commit-hash>`
>
> The format of the JSON object that should be used as the request body is:
>
> ```
> {
>     "state": "<INPROGRESS|SUCCESSFUL|FAILED>",
>     "key": "<build-key>",
>     "name": "<build-name>",
>     "url": "<build-url>",
>     "description": "<build-description>"
> }
>
> ```

One could run such a snippet by codefresh at the end of the build using a script to notify bitbucket regarding the build status of a commit.

> One green build
>
> The following curl command will add a build to the commit 9e72f04322c4a1f240e0b3158c67c3c19cdd16e7:
>
> ```
> curl -u username:password -H "Content-Type: application/json" -X POST
> http://localhost:7990/bitbucket/rest/build-status/1.0/commits/9e72f04322c4a1f240e0b3158c67c3c19cdd16e7
> -d @build0.json
>
> ```
>
> Where build0.json contains:
>
> ```
> {
>     "state": "SUCCESSFUL",
>     "key": "REPO-MASTER",
>     "name": "REPO-MASTER-42",
>     "url": "https://bamboo.example.com/browse/REPO-MASTER-42",
>     "description": "Changes by John Doe"
> }
>
> ```

---
Tags: continuous-integration, bitbucket, bcbsn, codefresh
---


# Question
Title: Gitlab CI Runner command line doesn't care about --parameters
I'm trying to write a windows batch file to automate the gitlab runner registration :

```
SET /p token=Token ?:
gitlab-runner register --locked false --run-untagged true --tag-list java --name foo --token %token% --url https://myurl --executor

```

I found the parameters in help documentation (`gitlab-runner register --help`) but it doesn't care about them. It still continues to ask question whose answers are in the parameters.

Thanks for helping

# Answer
You're missing the `--non-interactive` tag to your registration command.

Full command should be:

```
gitlab-runner register --non-interactive --locked false --run-untagged true --tag-list java --name foo --registration-token %token% --url https://myurl --executor shell

```

Relevant documentation for non-interactive registration.
> 3 votes

---
Tags: gitlab, gitlab-ci-runner
---


title: CasC with Bamboo

text:
# Question
Title: CasC with Bamboo
While trying to get a clear picture how to setup a seamless CasC (Configuration as Code) build and deployment plan management, I have to admit that I get quite puzzled.

Is the following (subjective) *ideal scenario* possible?

* Users can manage their plans as code or via UI as YAML
* Plan changes in UI result in a commit to the CasC code base (YAML as well)

In the meantime, my impressions are that just pieces of that are possible but no integraton setup yet:

1. Admins can manually export (and import?) plan configurations as YAML via UI
2. Bamboo Specs allow Java language coding of plans.

Looks like #1 is too limited (possibly usage of REST API could change that? But still you'd need admin rights) and #2 will be overdesigned for most cases.

Anybody has a solution for the (subjective) *ideal scenario*?

# Answer
This slide share gives an idea how to handle CasC in Bamboo.

The presenter advocates to use yaml, but according to him there are also downsides.
> 2 votes

---
Tags: bamboo, casc, yaml
---


# Question
Title: Founded argument to have no sandbox CI/CD?
CI/CD environment is the production carrier for DevOps teams.

An error here is a show stopper for possibly many teams.

Additionally, it is a product offered to teams which needs maintenance (quartely new features sometimes and security patches anyway) and room for experiment (new metrics plugins for example).

How to allow this? The only argument against a sandbox I can imagine is its cost. But then, isn't a transition to IaC, aka Pipelines as Code providing a "serverless" CI/CD like in Gitlab or Travis, not the next step? Otherwise, CI/CD is the new "pet" system?

# Answer
A CI/CD system is, after all, just another software product/system. Regular upgrades and at least some changes in the plugin installations could be treated as changes of the software product itself.

I see no founded reason for which such software product can't also use as part of its own development process a CI/CD solution. *Maybe* even itself - the most recent production version, for example - if it's flexible enough (I'm still investigating such possibility for the CI system I'm working on).

Now, a CI/CD system deployed/actively used in a certain software development project is effectively a software product deployed in a specific environment.

Changes to the configurations of the CI/CD system aren't actually software changes, they're changes of the environment in which the software system is deployed. These are trickier to address in a CI/CD pipeline. Some even impossible. You'd have to replicate the environment as close as possible. But it's unlikely you'll be able to do that at 100% - just like no dev/QA/etc environment is *exactly* as the production one.

Some environment changes could be tested in a CI/CD pipeline. For example changes to a test script: both the old and the new versions can be executed as parallel pipeline jobs with the same set of artifacts for a period of time, long enough to determine the test quality and stability before making the new version the official one and retiring the old one.

Finally, don't forget that *most* CI systems are reactive: they *detect* breakages, but human action is required to identify the culprit and/or make repairs before the development process can be resumed. Applying such CI system to an actively used CI/CD system one would allow the show-stopper problems you mention. Only a non-blocking CI system can completely *prevent* them (**disclaimer**: I'm the founder of the company owning the referenced page and offering such system).
> 1 votes

---
Tags: continuous-integration, continuous-delivery
---


# Question
Title: How to run database migrations in an auto-scaled group
We are just about to move from a monolithic architecture to an auto-scaled group and I'm not sure how I should be running my database migrations (Laravel).

I am thinking that a script will run when a new box comes online, this will git pull my latest code down. Should this script also execute my database migration? I am not sure how I can get it run on only one box?

# Answer
How are you doing your deploys? Nuke + rebuild the ASG (whether one node at a time, or by replacing the whole group at once), or do you have a script that redeploys all of your active nodes?

And also, how are you triggering your deploys?

Ideally you want to be running/triggering deploys from some form of CI server like Bamboo or Jenkins. If you're doing this, you can run your DB migrations from the build server (or its component build agent instances).

This would allow you greater control over your environment, as you could, for example, restore DB from snapshot if the deployment fails.
> 1 votes

# Answer
Many orm's store migration state in the DB itself, but if you do it 'manually' its not hard to build yourself either. Just have a "migrationstate" table where all executed migrations are stored. That way when a node comes online it can simply check that table against its local "migrations" folder, and execute what's missing - this should cause only your first node after a version bump to execute the migrations.

As a side note: CloudFoundry solves this by allowing each instance (node) in your scale set to access its identifier. The first one gets 0, the next 1 etc. That way a node knows if its the first one, and can execute migrations. I don't think ASGs have the same.
> 1 votes

---
Tags: amazon-web-services, autoscaling
---


# Question
Title: Ansible sporadic errors with Windows machines
I'm running into some on and off issues when using windows hosts in my Ansible playbooks. I'm running Ansible 2.3 with pywinrm 0.2.2 installed. I'm using basic authentication with the local Administrator user.

Sometimes I receive this issue when I run a task:

```
 [WARNING]: FATAL ERROR DURING FILE TRANSFER: Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/connection/winrm.py", line 267, in _winrm_exec
  self._winrm_send_input(self.protocol, self.shell_id, command_id, data, eof=is_last)
File "/usr/local/lib/python2.7/dist-packages/ansible/plugins/connection/winrm.py", line 248, in _winrm_send_input
  protocol.send_message(xmltodict.unparse(rq))
File "/usr/local/lib/python2.7/dist-packages/winrm/protocol.py", line 207, in send_message
   return self.transport.send_message(message)
File "/usr/local/lib/python2.7/dist-packages/winrm/transport.py", line 191, in send_message
   raise WinRMTransportError('http', error_message) WinRMTransportError: (u'http', u'Bad HTTP response returned from server. Code 500')

```

Other times, when I try to run a `win_shell/win_command/raw module` and `with_items` on a group of windows hosts it seems to fail on temporary files created by Ansible.

The task I'm trying to run is:

```
- name: Check services up
  win_command: 'sc queryex {{ item }} | Findstr RUNNING'
  with_items: '{{ component_services }}'
  register: command_result
  ignore_errors: yes

```

And the error I may get is:

```
changed: [172.16.104.169] => (item=Dnscache)
failed: [172.16.104.176] (item=Dnscache) => {"failed": true, "item": "Dnscache",
  "module_stderr": "Exception calling \"Run\" with \"1\" argument(s): \"Exception calling \"Invoke\" with \r\n\"0\"
     argument(s): \"The running command stopped because
           the preference variable \r\n\"ErrorActionPreference\"
           or common parameter is set to
   Stop: (0) : cannot open \r\nC:\\Users\\ADMINI~1\\AppData\\Local\\Temp\\RESB3FF.tmp
  for writing\r\n(1) :
     using System;\r\n\"\"\r\nAt line:45 char:1\r\n+
     $output = $entrypoint.Run($payload)\r\n+
      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~\r\n+
  CategoryInfo          : NotSpecified: (:) [], ParentContainsErrorRecordE \r\nxception\r\n+
  FullyQualifiedErrorId : ScriptMethodRuntimeException\r\n",
  "module_stdout": "", "msg": "MODULE FAILURE", "rc": 1}
     changed: [172.16.104.141] => (item=Dnscache)
     changed: [172.16.104.168] => (item=Dnscache)
     changed: [172.16.104.145] => (item=Dnscache)

```

Both issues are absolutely random and may even not appear at all over a sequence of different runs.

Any assistance?

# Answer
You should probably create an Ansible issue for this as it's most likely a bug in Ansible.

The first error makes me think about WinRM pipelining:

* Ansible 2.3.0 introduced an always-on WinRM pipelining feature (similar to SSH pipelining), and that may be behind this.
* SSH pipelining can cause issues in Ansible for Linux, and it can be useful to turn it off, but this isn't yet possible for WinRM pipelining.

This related issue includes some Git commits that will re-enable 'non-pipelined' mode in a future release (should now be released in 2.4, possibly with a backport as part of 2.3.2 - see this comment)

Try upgrading to Ansible 2.4.1+ (which generally works well) to get the fix. Or try downgrading to Ansible 2.2.3 to see if this helps - this will disable WinRM pipelining and may avoid other regression bugs in this area.

* If you installed Ansible using `pip`, you can do `pip install ansible==2.4.1` to upgrade (or `ansible==2.2.3` to downgrade), then if that doesn't help, do the same with `2.3.1` to upgrade again.
* you should also upgrade to latest `pywinrm` as mentioned in the issues above
> 2 votes

# Answer
I've found Ansible 2.3.2 to be the most stable, tho I haven't spent much time with 2.4.1 yet. 2.4.0 definetely has some stability issues when it comes to winrm.
> 1 votes

---
Tags: ansible, windows, powershell
---


title: Delivery in a SalesForce environment

text:
# Question
Title: Delivery in a SalesForce environment
In an organization with a hundred developers, and several thousand salespeople using SalesForce to create their application. Given the restrictions of SalesForce environment and deployment options.

What would be the process and which tools to use to reach a delivery pipeline where changes can be delivered to production many times per day?

The basic flow is:

1. salespeople think of ideas for improving their flow (bugs, new dev).
2. product managers distill the requirements and pass onto developers.
3. developers write code and create configuration changes in git.
4. magic happens (currently takes about 3-4 months).
5. new changes are in production and salespeople can use it.

# Answer
I would recommend to create an issue about CD in the ticket system of salesforce. I have googled a little bit and this seems to be their issue tracker.

I would prevent to implement CD in salesforce as that will be very cumbersome and will take a lot of time. In my opinion this issue should be solved by SalesForce itself.
> 1 votes

---
Tags: continuous-delivery
---


# Question
Title: How to make custom SLES (SuSe Linux Enterprise) Dockerfiles?
To use a dockerized SLES, you need to use an official base image and run it on a SLES host.

But this does not seem to work out of the box - what needs to be done to build a custom image?

Dockerfile:

```
FROM suse/sles12:latest
RUN zypper ref -s
RUN zypper --gpg-auto-import-keys ref -s

```

Docker build output - image will be created but obviously it is not possible to install any packages because the central repo seems to be not preconfigured.

Vendor documentation: https://www.suse.com/documentation/sles-12/book\_sles\_docker/data/customizing\_pre-build\_images.html

```
Sending build context to Docker daemon  3.584kB
Step 1/3 : FROM suse/sles12:latest
 ---> d0f7e807658b
Step 2/3 : RUN zypper ref -s
 ---> Running in 235316f5bbe1
Refreshing service 'container-suseconnect'.
Problem retrieving the repository index file for service 'container-suseconnect':
[container-suseconnect|file:/usr/lib/zypp/plugins/services/container-suseconnect]
Warning: Skipping service 'container-suseconnect' because of the above error.
All services have been refreshed.
Warning: There are no enabled repositories defined.
Use 'zypper addrepo' or 'zypper modifyrepo' commands to add or enable repositories.
 ---> 21e8beb32227
Removing intermediate container 235316f5bbe1
Step 3/3 : RUN zypper --gpg-auto-import-keys ref -s
 ---> Running in 72ba1ad22da8
Refreshing service 'container-suseconnect'.
Warning: Skipping service 'container-suseconnect' because of the above error.
All services have been refreshed.
Warning: There are no enabled repositories defined.
Use 'zypper addrepo' or 'zypper modifyrepo' commands to add or enable repositories.
Problem retrieving the repository index file for service 'container-suseconnect':
[container-suseconnect|file:/usr/lib/zypp/plugins/services/container-suseconnect]
 ---> 6426e55c2d87
Removing intermediate container 72ba1ad22da8
Successfully built 6426e55c2d87

```

UPD. While this one is working:

```
 docker run -it suse/sles12 bash -c "zypper -n ref -s"
 Adding repository 'SLES12-Debuginfo-Pool for sle-12-x86_64' ...[done]
 .....

```

UPD. Finally, the following worked for me: run a container, install everything there and finally commit the container to a new image (`docker commit container-id image`).

# Answer
As @Tensibai already indicated in one of the comments, some packages are missing:

```
Use 'zypper addrepo' or 'zypper modifyrepo' commands to add or enable repositories.

```

Either use `zypper addrepo` or `zypper modifyrepo` in your dockerfile to solve the issue.
> 1 votes

---
Tags: containers, dockerfile
---


title: Azure and DNS services

text:
# Question
Title: Azure and DNS services
I have a `Virtual network` (MyVPN) resource in Azure, being a class C network, MyVPN has `Azure provided DNS service` as DNS servers, so all virtual machines with a NIC connected to this network are able to resolve their hostname successfully, for example virtual machine `myvm01` can ssh to `myvm02` using only the hostname instead of IP.

However, they are unable to make a reverse DNS lookup, which makes sense since by default Azure does not provide a `reverse DNS lookup zone`. By following Microsoft's documentation I managed to create it, sadly I don't really know if I should delegate to `Azure provided DNS service` and if so, how to do it (I tried following this)

What should I do now to allow `myvm01` be able to resolve the IP of `myvm02` to its hostname?

# Answer
> 1 votes
There is another manual for enabling reverse lookup DNS in azure.

> **Will my reverse DNS records resolve from the internet?**
>
> Yes. Once you set the reverse DNS property for your Azure service, Azure manages all the DNS delegations and DNS zones required to ensure that reverse DNS record resolves for all Internet users.
>
> **Are default reverse DNS records created for my Azure services?**
>
> No. Reverse DNS is an opt-in feature. No default reverse DNS records are created if you choose not to configure them.

It seems that reverse dns has to be enabled if it is used internally.

---
Tags: azure, dns
---


# Question
Title: What is OpenWhisk and how to setup a cluster using this technology?
On IBM bluemix platform it is possible to write cloud functions using OpenWhisk, but what is OpenWhisk and how to setup a cluster using this technology?

# Answer
According to this GitHub readme

> OpenWhisk is a serverless event-based programming service.

In order to get an idea how to setup this tool on a cluster one could run the following code that is defined in the readme:

> ```
> # Clone openwhisk
> git clone --depth=1 https://github.com/apache/incubator-openwhisk.git openwhisk
>
> # Change directory to tools/vagrant
> cd openwhisk/tools/vagrant
>
> # Run script to create vm and run hello action
> ./hello
>
> ```
> 1 votes

---
Tags: serverless, ibm-cloud, openwhisk
---


# Question
Title: Is making kubernetes masters schedulable a really bad idea?
It's a cash strapped start-up. What are the worst consequences of running only three nodes, three schedulable masters?

Are there some ways can one mitigate against the risks?

# Answer
> 3 votes
https://kubernetes.io/docs/tasks/administer-cluster/highly-available-master/

> **Best practices for replicating masters for HA clusters**
>
> Try to place master replicas in different zones. During a zone failure, all masters placed inside the zone will fail. To survive zone failure, also place nodes in multiple zones (see multiple-zones for details).
>
> Do not use a cluster with two master replicas. Consensus on a two-replica cluster requires both replicas running when changing persistent state. As a result, both replicas are needed and a failure of any replica turns cluster into majority failure state. A two-replica cluster is thus inferior, in terms of HA, to a single replica cluster.
>
> When you add a master replica, cluster state (etcd) is copied to a new instance. If the cluster is large, it may take a long time to duplicate its state. This operation may be sped up by migrating etcd data directory, as described here (we are considering adding support for etcd data dir migration in future).

I would like to emphasize the following quote

> A two-replica cluster is thus inferior, in terms of HA, to a single replica cluster.

If less then 3 nodes will be used then it is basically more safe to run 1 master, but if one goes down there will be an issue as well. So it is better to run at least three masters.

---
Tags: kubernetes, high-availability
---


# Question
Title: Is it possible to create a robot account in bitbucket without creating a new email account?
**Introduction**

According to this article a user account could be given to a robot for Continuous Integration purposes.

**Method**

The IT department created a new email account, this was used to login to the git repository, a ssh key was created, the public key added to bitbucket and finally the jenkins user was able to create commits in bitbucket.

**Problem**

As the email account is only used by the robot and the robot itself is not reading emails it feels that the email account is misused.

**Question**

The question is whether it is possible to create a robot account in bitbucket without creating a new email account.

**Discussion**

If there is no separate email account how could bitbucket deviate between the robot account and an existing real user? A unique email account should be used right?

# Answer
> 4 votes
After doing some research and comparing various documents it seems that a bot account is basically a real user account and needs a separate email account in order to create such an account as using an email account to register a new account results in, e.g.: `This email account is already in use`.

Combined with quote retrieved from the link that is defined in the question:

> User accounts are intended for humans, but you can give one to a robot, such as a continuous integration bot, if necessary.

It seems to be necessary to create a new email account if a bot account has to be created as:

1. git repositories, like github, bitbucket and gitlab check whether an email account already exists if a new user is created
2. to clarify that the user is a bot and not a human

---
Tags: bitbucket
---


# Question
Title: In kubernetes, using an nginx ingress controller what is recommended way of adding back end services?
I'm running on a simple KVM host and have poor man's load balancing with round robin DNS to and nginx ingress based on this chart which is documented here

So to direct traffic to backend pods it seems from those docs that all you would need to do is apply appropriate labels and magic will ensue.

It's a bit too magical for me and I'd rather just edit the nginx config and put reverse proxy entries in for the service urls for my microservices.

What's the typical way of doing this?

# Answer
What you have installed, the `ingress controller`, is only one part of the story. The other part is an `ingress`: in an `ingress` you define what pods/services should receive traffic of what domain and request path. The `ingress controller` picks this information up, and dynamically reconfigures the underlying proxy, in your case nginx. You can find examples of ingresses in the official documentation: https://kubernetes.io/docs/concepts/services-networking/ingress/#types-of-ingress
> 2 votes

# Answer
Ingress controllers allow you to not manually wire stuff up, that's the whole point.

If you'd rather do it manually You could have an nginx array (either inside Kubernetes our outside of it) and just point to the ip/port of the services that expose your microservices. If you use an nginx array external to Kubernetes you'd have to use the externalIP service type. If you run the Nginx array on Kubernetes you could store your "manual proxy config" in a configmap so that all your nginx proxies get the same config.

I would definetely advise to not do this tho. Ingress controllers are built to simplify traffic routing, and as your complexity grows you will definetely want to take advantage of them. A good labeling scheme is absolutely paramount in Kubernetes, and that goes for ingresses aswell.
> 2 votes

---
Tags: kubernetes, containers, microservices, load-balance
---


# Question
Title: How to set minikube to auto start on windows log on?
How I can set minikube to auto start on windows 10 login? I checked but I can't find anything on google.

I saw that there is a default parameter `autostart-enabled=off` but I dont know where I can set it.

At the moment after I restart my pc I can see in Hyper-V Manager that the minikube machine is running but when I try minikube dashboard in terminal I get:

> minikube is not currently running so the service cannot be accessed

# Answer
> 5 votes
Should the normal startup functionality of w10 work?

> To change which apps run at startup, press and hold (or right-click) the Start button, select Task Manager, and then select the Startup tab. Choose an app, then select Enable or Disable.
>
> To add or remove an app from the Startup tab, press the Windows Logo Key + R, type shell:startup, and then select OK. This opens the Startup folder. Select Start . Find the app you're looking to add or remove, and press and hold (or right-click) it. Select More \> Open File Location. In the file location folder, press and hold (or right-click) the app and select Copy. Paste the shortcut to your app into the Startup folder to have it run at startup.

Info from:

https://support.microsoft.com/en-us/help/4026268/windows-change-startup-apps-in-windows-10

# Answer
> 2 votes
You could create a bat script `minikube start` and add it as a scheduled job that start on start up of windows

---
Tags: windows, minikube
---


# Question
Title: Can be DevOps culture mapped to a specific type of organization as described by Laloux?
In his book, "Reinventing organizations", Frederic Laloux describes a historical evolution of organizations and their systems of internal structure, management and values.

Where will an organization place on this chart when it would adopt a set of DevOps best practices and culture recommendations? Which stage of development is a target for DevOps practices and what development of the organization is being achieved? Does it correspond to this model?

Here is a picture of the Laloux Culture Model:

# Answer
> 5 votes
This question perfectly illustrates the different approaches in adopting DevOps by organizations. While both approaches target the 3rd circle on the culture model - the traditional corporations - there is a difference in aim and results.

1. Adopt DevOps **best practices** and **tools** for increased effectiveness.
2. Adopt DevOps **culture** and **values** for organizational transformation.

Established corporations often aim at (1) and either omit or fail to adopt (2). They will still gain increased effectiveness and improved organizational key performance indicators, but they will fundamentally stay at the orange level of organizational development, failing to undergo the transformation. Such transformation always requires an existential threat to the organization and those are rarely recognized and acted upon until there is too late for the organization to transform.

New organizations and organizations with new leadership have an easy way to focus on (2) and so they either are raised up at the new developmental stage or manage to successfully seed business units operating with new culture and values that either subsume the entire organization eventually or the old business units slowly decay and whither while the new ones take over a large bulk of the business. It is almost like a parent passing his family business to a child who has been raised with different values and more progressive ideas.

Either way, you can say that DevOps as it is understood today, operates between the orange and green layer and has the potential to enable the transformation from **Orange** to **Green** under certain conditions.

The **core ideas** behind DevOps are a different matter. They are in a way a method for an organization to pass from any developmental stage to the next, but as DevOps is being codified through *specific* cultural recommendations, *concrete* values, and *actual* practices and tools, the *instantiation* of those core ideas in form of DevOps is squarely aimed at the border between Orange and Green.

---
Tags: culture, methodology
---


title: Sonar:6.7 docker container restart loop

text:
# Question
Title: Sonar:6.7 docker container restart loop
I am running an init.sh script in order to create mapped directories for Sonar and Postgres.

Both of the containers build fine and are able to be started, but for some reason, our Sonar container is in a constant restart loop after about 8-16 seconds.

In the logs we noticed that there was a missing plugin error for `"findbugs"`, although the compability matrix states that `findbugs` works with Sonar 6.7.

```
        2017.11.23 17:16:21 WARN  web[o.a.c.l.WebappClassLoaderBase] The web application [ROOT] appears to have started a thread named [Timer-0] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
     java.lang.Object.wait(Native Method)
     java.util.TimerThread.mainLoop(Timer.java:552)
     java.util.TimerThread.run(Timer.java:505)

```

This seems to be a popular error when I do a `sudo docker logs sonar_container`

Restarting:

```
VirtualBox:~/Sonar/docker-sonar-scpvm014$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
fa4b1c86433d        sonarqube:lts       "./bin/run.sh"           16 seconds ago      Up 5 seconds        0.0.0.0:9092->9092/tcp, 0.0.0.0:8060->9000/tcp   sonar_container
87db675ef2dd        postgres:9.5        "docker-entrypoint..."   18 hours ago        Up 11 minutes       0.0.0.0:5432->5432/tcp                           pgsql_container
:~/Sonar/docker-sonar-scpvm014$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
fa4b1c86433d        sonarqube:lts       "./bin/run.sh"           19 seconds ago      Up 9 seconds        0.0.0.0:9092->9092/tcp, 0.0.0.0:8060->9000/tcp   sonar_container
87db675ef2dd        postgres:9.5        "docker-entrypoint..."   18 hours ago        Up 11 minutes       0.0.0.0:5432->5432/tcp                           pgsql_container
~/Sonar/docker-sonar-scpvm014$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                                            NAMES
fa4b1c86433d        sonarqube:lts       "./bin/run.sh"           23 seconds ago      Up 12 seconds       0.0.0.0:9092->9092/tcp, 0.0.0.0:8060->9000/tcp   sonar_container
87db675ef2dd        postgres:9.5        "docker-entrypoint..."   18 hours ago        Up 11 minutes       0.0.0.0:5432->5432/tcp                           pgsql_container
~/Sonar/docker-sonar-scpvm014$ sudo docker ps
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                        PORTS                    NAMES
fa4b1c86433d        sonarqube:lts       "./bin/run.sh"           25 seconds ago      Restarting (0) 1 second ago

```

Does anyone have any experience with fixing this issue?

# Answer
> 1 votes
I ran into the same issue and found this docker-compose file that is able to start postgres and sonarqube.

https://github.com/SonarSource/docker-sonarqube/blob/master/recipes.md

> ```
> version: "2"
>
> services:
>   sonarqube:
>     image: sonarqube
>     ports:
>       - "9000:9000"
>     networks:
>       - sonarnet
>     environment:
>       - SONARQUBE_JDBC_URL=jdbc:postgresql://db:5432/sonar
>     volumes:
>       - sonarqube_conf:/opt/sonarqube/conf
>       - sonarqube_data:/opt/sonarqube/data
>       - sonarqube_extensions:/opt/sonarqube/extensions
>       - sonarqube_bundled-plugins:/opt/sonarqube/lib/bundled-plugins
>
>   db:
>     image: postgres
>     networks:
>       - sonarnet
>     environment:
>       - POSTGRES_USER=sonar
>       - POSTGRES_PASSWORD=sonar
>     volumes:
>       - postgresql:/var/lib/postgresql
>       # This needs explicit mapping due to https://github.com/docker-library/postgres/blob/4e48e3228a30763913ece952c611e5e9b95c8759/Dockerfile.template#L52
>       - postgresql_data:/var/lib/postgresql/data
>
> networks:
>   sonarnet:
>     driver: bridge
>
> volumes:
>   sonarqube_conf:
>   sonarqube_data:
>   sonarqube_extensions:
>   sonarqube_bundled-plugins:
>   postgresql:
>   postgresql_data:
>
> ```

---
Tags: docker
---


# Question
Title: "Software Development" which environment is it: Manufacturing or Projects?
Is it Manufacturing or Projects?

Definitions I am using for these:

> **Project**
>
> Planned set of interrelated tasks to be executed over a fixed period and within certain cost and other limitations.

and

> **Manufacturing**
>
> The process of converting raw materials, components, or parts into finished goods that meet a customer's expectations or specifications. Manufacturing commonly employs a man-machine setup with division of labor in a large scale production.

I can suggest that in software development *manufacturing* translates to:

```
"raw materials" = requirements and specifications
"parts" = software libraries
"finished goods" = service/software deployed and serving customers
"man-machine setup" = configuration of environments
"division of labor" = dev and ops and qa, etc...
"large scale production" = lots and lots and lots of code

```

When I write "Software Development" I actually mean the whole cycle that includes lead-time from the point where some customer need is discovered, to the point of customer value created and delivered.

# Answer
> 2 votes
According to Wikipedia

> Manufacturing is the production of merchandise for use or sale using labour and machines, tools, chemical and biological processing, or formulation.

and

> a project is an individual or collaborative enterprise, possibly involving research or design, that is carefully planned, usually by a project team, to achieve a particular aim.

Based on the definitions Software Development could both be seen as **Manufacturing** and **a Project**.

It could be seen as **manufacturing** from a deliverables perspective as Development activities mean that that the software is created, but does not guarantee quality. Therefore QA is required and to get it online Operations tasks are needed as well. In summary, Software Development could be seen as **manufacturing** as multiple specialisms are required to make the feature available for a customer.

It could also be compared with **a project** as the Dev team itself does R&D and creates a (software) product.

From a DevOps perspective I would advocate to compare Software Development with **manufacturing** rather than **project** as DevOps is about let work multiple specialisms together and not about creating silos like **project**.

---
Tags: software-engineering
---


title: Nginx /foo and /foo/ redirects

text:
# Question
Title: Nginx /foo and /foo/ redirects
I want both http://domain/foo and http://domain/foo/ requests to serve a given resource. Here's the relevant part:

```
location / {
       rewrite /(.*) /ipns/QmdpoFuwY/$1 break;
       proxy_pass http://127.0.0.1:8080;
}

```

with this config only:

```
curl -X GET http://domain.io/foo/

```

returns a resource while:

```
curl -X GET http://domain.io/foo

```

gives a `Path Resolve` error.

# Answer
One could replace

```
rewrite /(.*) /ipns/QmdpoFuwY/$1 break;

```

with

```
rewrite ^(.*[^/]) /ipns/QmdpoFuwY/$1 break;

```

and try again.

The issue was solved by inspecting the logs. If for example one navigated to `/foo/` instead of `/foo/` the log indicated:

```
2017/10/15 14:51:28 [error] 7#7: *1 "/etc/nginx/html/index.html/foo.html" is not found (20: Not a directory)

```

It turned out that the regex did not match `/foo/`. One could also enable the rewrite logging to facilitate debugging:

```
rewrite_log on;

```
> 1 votes

# Answer
Your nginx code is incorrect, for example, because you're effectively losing the extra functionality afforded by http://nginx.org/r/proxy\_redirect default value of `default`.

As per the documentation at http://nginx.org/r/proxy\_pass, the correct way would be to map location on the front-end and back-end directly within the `proxy_pass`, especially if you don't need to be using any variables:

```
location / {
    proxy_pass http://127.0.0.1:8080/ipns/QmdpoFuwY/;
}

```

---

Otherwise, back to your question, there is absolutely no reason to have `/foo` and `/foo/` point to the same resource.

In fact, it would be perfectly legal for one to be internally expanded to `/foo.html`, whereas for the other to `/foo/index.html`, so, the fact that different results are returned is entirely normal, and doesn't need to be fixed.
> 0 votes

---
Tags: amazon-ec2, servers, http
---


# Question
Title: Using Anaconda instead of pip+virtualenv
We are currently using `pip`+`virtualenv` and install all our Python dependencies during the application build step on the CI. This requires compiling some of the packages, which contain C modules, which takes time and requires the build servers to have required system-level "development libraries" installed.

From what I understand, Anaconda can install *pre-built packages* allowing to skip the complication step altogether making the builds faster and not needing to have certain development packages.

Has anyone gone through `pip`+`virtualenv` to `conda` transition? What could be the downsides and potential problems of switching to `conda` for building and installing Python dependencies?

# Answer
In my field (science) Anaconda is probably the most common Python distribution in use. conda is the package manager (just to be clear of the distinction).

It is difficult to find fault in conda when compared with pip+virtualenv.

The one point that I would make is that the Anaconda distribution is meant to be installed per user. If you have a Python environment that is shared across users there is no way of doing the equivalent of `pip install --user` with conda (but read on for a surprising solution...). Nor users can create conda environments.

So, if you are considering a single user installation go ahead and try Anaconda. It is worth a try

Note that you can use pip with Anaconda... they are not incompatible. In fact that can be quite useful in a shared installation.

I have setup a shared-user conda installation where users can install user-space packages with... `pip install --user`.

In a shared installation I do not see how users can create their own conda environments (conda has environments). And I bet conda+virtualenv would be a mess.

Note that conda is not Python tied: First you can have a conda installation with both Python 2 and 3 - and conda manages environments very well. But you can also have conda install Perl, R. In fact that is quite common. Our Perl interpreter is provided by Anaconda ;)

Also I suspect conda is probably stronger in Linux and Mac OSes. Windows is less well supported. For other operating systems, there are probably no packages (or they will be limited).
> 3 votes

---
Tags: deployment, python, builds, virtualenv
---