# Question
Title: How to implement Continuous Deployment in Angular, PHP API and Ansible projects?
I am working on an Angular2 project with Yii2 (PHP/MySQL) as its API point. We will need to deploy the Angular app in different languages for different customers (using AOT compilation from angular-cli).

What is the best way to achieve this? I am looking at Docker or Ansible(-container) to make this work. But the examples are mainly for a 1:1 setup. But I would need a 1:n, something like:

`deploy app-de new_costomer prod_server`

Additional to that, when a new version of the app is available i want to update the entire network with the new app (that requires uploading of the src/ folder AOT compiled, and db migration scripts on the Yii2 side).

Ideally it would do something like:

`deploy update network-all prod_server`

Any ideas?

To clarify: I have a number of customers, each customer should get its own container (docker, through ansible-container). There is the AOT compiled Angular stuff (just JS) and the PHP backend with MySQL.

Each customer gets this setup on a server (so 3 customers = 3 docker containers with the Angular + PHP + MySQL).

We push updates to both Angular and PHP quite frequently (i.e. the entire DIST folder needs to be uploaded or at least a diff, migrations scripts for the PHP/MySQL backend needs to run, etc.).

And this from one command (because I obviously do not want to have SSH into each container to do all that semi-manually).

Since I have never done this kind of setup before, I would like to get some ideas on how this can achieved with things like Ansible (or the like).

# Answer
> 7 votes
Let's assume you have the following scenario: You have many customers, but most use the exact same app, except for some configuration changes. You would like each of your customer to get the latest of the app ASAP.

If this is your scenario then try make everything that needs to be differentiated between the apps a configuration change. You build one (or one set of) docker containers that can handle any of your customer based on the configuration it was given. Then on deployment time you simply supply the proper configurations for this container for each of your customer.

The benefit of this approach is that you don't really incur massive overheads when a new customer comes. You generate their configuration, create the new servers, add some entries into an inventory file, and the next time you deploy, it gets there as well. The more well your CD pipeline is set up the more you can automate any of this process.

So how would the flow generally look like:

1. You have your application codebase, that can cater for **all** of your customers
2. You build the docker container for your application (one for the frontend, one for the backend, possibly a few more supporting ones)
3. You have a deployment script that uses a configuration file to determine where to deploy and what.
4. You run this deployment script. It will go over all of your customers, and simply deploy the latest version of your app with the appropriate config.

If you have a new customer, the only place you have to change is the deployment configuration file. If you are using ansible most likely you have to change your inventory and your vars file. Let's say you have get a new customer, who would like the app to run under Spanish locale, and has access to the beta features. This customer's configuration would maybe look like this:

```
config.js:

SETTINGS = { locale: 'es', beta: true };

config.php:

define('CUSTOMER_LOCALE','es');
define('CUSTOMER_BETA','true');

```

You then make sure that during deployment your container gets these files mounted on startup.

---
Tags: deployment, ansible, docker
---


title: What is a container?

text:
# Question
Title: What is a container?
There are some questions about ***containers***, such as:

**My questions**:

* What is actually a "container" (in the context of DevOps)?
* Why are they used?

# Answer
> 20 votes
The very first thing to know about a container is:

It is, first and foremost, a ***process.***

Once that is understood, one can start to understand how containers compare and contrast with virtual machines. Containers and VMs both share *isolation* from their hosts. The method of isolation is the critical difference.

Container processes use extensions to the OS kernel host on which they run to isolate themselves from other processes. Other extensions also provide disk and resource isolation. Containers *share* their kernel and memory with the host OS.

Virtual Machines use a *hypervisor* to isolate VMs from their hosts. This is a layer of software that forwards requests for resources from "guests" (VMs) to the hardware. Disk isolation is provided by disk virtualization. VMs *do not* share a kernel with the host- they load their own kernels into memory space dedicated to the VM.

One important impact of this difference is that a container must be kernel-compatible with its host. For example, it is not possible to run a Windows Nano Server-based container on a Linux host, or an Ubuntu container directly on a Windows host. Virtual Machines, by contrast, can run any kernel regardless of host OS. When running a Linux container on a Windows host, Docker runs the container in a Linux VM.

The operational differences are in agility: containers start and stop about as fast as a normal process. VMs are "heavier," requiring dedicated resources set aside for them and take longer to start and shut down.

Containers offer a lot of flexibility to a DevOps operating model:

* Containers isolate software component dependencies. Developers can use containers to guarantee that an application module will function as well on their development machines as in QA/UA/Production
* Containers use software-defined networking to communicate
* Container definitions are declarative and can be source-controlled
* Container management systems (Kubernetes, DC/OS, Swarm) can manage hardware resource (compute/RAM/storage) pools and dynamically scale containers

# Answer
> 8 votes
The word *container* refers to a lightweight virtualisation technology available on modern Linux kernels, this technology is very similar to FreeBSD jails.

An older, non-container-able, Linux kernel is able to run processes concurrently. Some attributes of the system are private to process, like the process environment or the process memory: only the process owning these attributes and the operating system itself are able to access to this data. (There is a lot of loopholes, like some *ps* implementations, but that's essentially true!) Some other attributes are shared among the processes, like the filesystem and network interfaces for instance.

A modern, container-able, Linux kernel is able to handle more attributes of the system as private data associated to a process or a group of processes. The resulting context is a *container* and instead of running a program in the “initial containers” using the filesystem and the network interfaces initialised by the operating system, it is possible to run processes in other containers, so that they see a different filesystem and a different list of network interfaces. Therefore, two processes running in distinct containers only really share the kernel. You are maybe familiar with the *chroot* command which can run a process in a distinct file-hierarchy, containers take the idea a few step further.

Of course, this is just a very coarse explanation, but I hope it helps to clarify the idea of what containers are. Now, what are they good for?

A popular interface to the container capabilities of Linux kernels is implemented by *docker,* a command-line utility that can be used to produce artefacts representing file-systems (*docker images*) and run processes in containers where these file-systems are accessible. This software suite is also able to build ad-hoc virtual networking systems to let several containers communicate on a private network.

Container-based technologies are convenient to:

* Describe *scalable* complex deployments.
* Provide application developers an environment very similar to the production environment.
* Implement immutable server pattern, as software artefacts typically describe a full operating-system, not just an application package.

(As you seem familiar with other virtualisation technologies like Virtual Box, you might remark that these technologies can also conveniently address the three points above. Nowadays, there is quite a small spectrum of virtualisation technologies, and we can compare the question of their popularity in certain contexts with the popularity of computer languages: it depends probably of the technical merits of each individual solution, but also to a lot of factors that I will just label “chance”.)

# Answer
> 7 votes
Usually containers refers to something like docker containers which have popularized the name

I quote there from docker definition:

> Using containers, everything required to make a piece of software run is packaged into isolated containers. Unlike VMs, containers do not bundle a full operating system - only libraries and settings required to make the software work are needed.

The root naming comes linux containers (lxc) whose goal was to isolate a process from its host system, the first goal was to avoid compromission of the process to take over the host system.

Now they are used in a wider scope. In a modern 'container' definition, you'll more or less release a package for runtime which already include your application, it's underlying middle-ware if needed and all necessary libraries and be sure it will run on any compatible system.

The second advantage is that it allows to use multiple applications with the same dependency at different version without having to heavily tweak its environment variables so it load the correct one.

Unlikely to a VM system like virtual box virtual machine or an EC2 instance on AWS, containers are virtual only on the file system level and isolated only on the memory stack. They still share the same host and the operating system under them will arbitrate cpu ticks.

A virtual machine is virtual at the hardware level, and you run an operating system within, a container is virtual at the OS level, and your run a process within.

# Answer
> 3 votes
From AWS's definition:

> Containers are a method of operating system virtualization that allow you to run an application and its dependencies in resource-isolated processes. Containers allow you to easily package an application's code, configurations, and dependencies into easy to use building blocks that deliver environmental consistency, operational efficiency, developer productivity, and version control. Containers can help ensure that applications deploy quickly, reliably, and consistently regardless of deployment environment. Containers also give you more granular control over resources giving your infrastructure improved efficiency.

Containerizing is packing all the essential necessities needed for doing a particular task/environment so that it is self-sufficient and can be run on any platform, which takes away a lot of pain, when it comes to setting up and installing stuff.

---
Tags: terminology, containers
---


# Question
Title: What is the difference between feature flags and feature toggles (if any)?
Part of an answer to my (meta) question "Rename feature-flags to feature-toggles?" is like so:

> ... There's could be valid questions about feature flags (formalism, documentation about them, process of definition), which may not be related to feature **toggles** but to feature **ramp up** which carry a less binary meaning. ...

**My question**: In the context of DevOps, what's the difference between feature flags and feature toggles (if any)?

PS: Looking at https://martinfowler.com/articles/feature-toggles.html , I don't get the difference between such flags and toggles.

# Answer
If you take it on an objective only point of view, there's no difference.

But feature *toggles* carry a heavy binary "on/off" for the whole application idea, whereas feature *flags* are a more relaxed term which can encompass ramp up testing more easily.

When you launch a ramp up testing and turn the new feature on for 10% of your traffic or user base and move up to 20% the next week and so on, the toggle term become counter productive. Some people (managers/commercials/some client support people) may not really understand only a part of the clients get the new feature.

Ramp up can be defined as (quote from wikipedia on software testing technology):

> The usual sequence is to ramp up the load: to start with a few virtual users and increase the number over time to a predetermined maximum. The test result shows how the performance varies with the load, given as number of users vs. response time.

I like etsy's feature API (even is not maintained) as source for explanation, it's an easy to undertand framework which has a good readme around feature flags and their uses.

So the difference can be done in the usage of the terms to just help avoiding confusion on what happens on the application.
> 8 votes

# Answer
I would add that a toggle is on/off (show ads on the site, don't show ads) and perhaps augmented by a flag like (West Coast gets ads from provider A, East Coast gets ads from provider B). Toggling turns off all ads. A feature flag might be able to switch from provider A to provider C.

I also think a key part is deciding how your switches and settings work - do you have to roll out a new build? or can your control tower determine east coast/west coast and answer "which provider" all on the fly with an easy, almost foolproof interface.
> 3 votes

---
Tags: terminology, feature-flags, feature-toggles
---


# Question
Title: Would choosing the integration branch strategy be part of DevOps?
In a **mature** DevOps organisation would choosing a product/product line integration branch strategy be part of DevOps?

What do I mean by integration branch strategy?

For example a product team has to integrate 20 features in a certain interval of time to meet some time to market release requirements.

One potential approach would be to use feature branches in a waterfall manner as they've deen doing for years - each feature is developed and tested in their own branch and only merged into the main branch (from which the release branch will be pulled) after metting the respective feature QA goals.

But the total integration time estimate - 20 feature branch merge windows multiplied by the average merge duration (from the team's branch merge historical data) is maybe too high.

Another approach is to pull say 4 intermediate integration branches, each of which being used by 5 of the features branches. This way all 20 features will be integrated into the 4 intermediate branches in a 5 branch merge windows interval. Which would be followed by a 4 other branch merge windows interval for integrating these 4 branches into the main branch, bringing the total integration estimate to 9 windows, much better than 20. Yes, some of the gain will probably be cancelled by higher degree of difficulty of last 4 merges.

Another approach would be to go straight for trunk-based development (true CI integration into the main branch) with practically no branch merges.

The process of selecting one such strategy is what I'm referring to as branch strategy.

Note: the actual SCM repository is irrelevant, it can be any combination of repos managed together as a single one, the term `branch` denotes just the common logical representation of whatever physical/actual repo branches exist in that combination.

# Answer
Branching strategy is a very clearly a business decision in which the DevOps engineer should be the primary expert, explaining all the trade offs. There are some strong positions on this issue in the community, very clearly Jez Humble is on one side with his book of Continuous Delivery. You could see him talking about it here. Here would be a video by panel of experts on the topic. And here is my highly up-voted answer on StackOverflow specifically on topic of feature branches. So clearly this is a big topic for DevOps.

One thing that people often say (Jez Humble mostly) is that feature branches are evil. You will need a (n-1)(n-2)/2 merges *\[combination number n-1 over 2\]* for n independent feature branches. Which comes to some very high numbers at 10+ feature branches. And because merging is hard and no publicly available version control systems do it will, then the entire Software Industry is moving away from software releases and software on premises to Software as a Service, where the only way is forward and software is both developed and maintained for you by the vendor.

As far as I know, there is only one company that can do high quality, business critical software on premises, with maintaining 10+ years of past major releases, back-porting patches through multiple waves of code refactoring and capable of working regularly on dozens of major feature branches per release and thousands small features developed on their individual feature branches and being continually merged as full features and not individual commits. The code has also been released on 14 different architectures. That company is Oracle, the product is RDBMS and they have been practicing what is now called DevOps since at least 1990. Hundreds of code deployments a day back in 2001 would be pretty standard. The reason why they are able to do it is through rigorous coding standards, invariant test suite and a proprietary version control system I've spent 10+ years of my life developing. A system that can do 99% of all merges through automation.

Everyone else pretty much only patches the most recent major release version past a certain stage/size of a project or it takes extreme amounts of human resources to merge all the patches to all the possible release versions and extreme amounts of human or compute resources to test them all. For example in Linux kernel you would have a maintainer for X.odd development version and later stability branch maintainer for X.even version, but you would see not patches for X-1.Y versions. Sometimes Linux distributions would maintain kernel version in their LTS release, but only at high cost and with varying code quality among distributions. And that is a high visibility project with enormous resources poured into it.

The primary reasons for this discrepancy are differences in:

1. Adherence to Coding Standards and Code Styles
2. Complete coverage by invariant test suite, which can catch bugs for which no tests were written specifically.
3. Version Control System capable of high levels of Automation.
> 3 votes

# Answer
DevOps should take part in this decision, but so are developers, product managers, testers, support and possibly even sales and marketing to some degree.

Branch strategy might affect the final product release, so it shouldn't be a one team decision.
> -7 votes

---
Tags: culture, branch
---


# Question
Title: What are pros and cons of Hackathons on implementing a DevOps culture?
I am curious about the efficacy of using a "hackathon" to focus development early in a new project before moving a more formal DevOps process.

Does letting things start out fast and loose to get things going hamper later efforts to add process?
If so, how does one weigh the benefits of building that early momentum vs. establishing a good DevOps process?

# Answer
> 9 votes
If the project is for a startup's minimum viable product or some other sort of sanity-check/proof of concept effort - hackathon before DevOps is often recommended - basically in the "fail-fast/fail cheap" lean startup strategy.

Some might recommend hackathon before DevOps in other situations as well, for time-to-market reasons. But if DevOps doesn't followup fast enough that initial market bridgehead can easily collapse. Possible reasons for that include:

* that early momentum can be ephemeral - easily lost as soon as the DevOps process starts to be established ("it's too hard, it's slowing us down")
* typically hackathons cause accumulation of technical debt. The cost of managing and eliminating it can outweighs the benefits of that early momentum.

# Answer
> 5 votes
DevOps is about the merging of information and processes between those who develop the software and those who deploy the software.

If the goal of your hackathon is to produce code which can not, or should not be deployed anywhere, then sure it makes sense to get your proof of concept (PoC) coded before developing the deployment process.

However, if you wish for a client (i.e. anybody who isn't one of the developers) to be able to view the PoC from a remote computer, then part of your hackathon already includes DevOps. To not use DevOps for a PoC or Minimally Viable Product (MVP) would be a waste of resources and time, as you work on coordinating the deploy process with operations.

DevOps will aide you in deploying more iterations of your PoC or MVP to be reviewed by more people in less time, and will thus enhance your Hackathon efforts.

# Answer
> 4 votes
Hackathons are a great way to promote people to share knowledge, collaborate, and share risk and responsibility. The traditional (non-devops) organization is often power-centric and promotes people to avoid talking directly to each other. Allowing people to only communicate to each other via their managers. This type of culture in organizations is sub-optimal for good performance, which is what DevOps all about.

The term 'power-centric' and 'generative' culture come from the R. Westrum model on how you can measure culture. The article presents a way to determine which culture your organization has, and what are the pros & cons of each type of culture (there are just three types).

From the article:

> *There is wide belief that organisational culture shapes many aspects of performance, including safety. Yet proof of this relationship in a medical context is hard to find. In contrast to human factors, whose contributions are many and notable, culture’s impact remains a commonsense, rather than a scientific, concept. The objectives of this paper are to show that organisational culture bears a predictive relationship with safety and that particular kinds of organisational culture improve safety, and to develop a typology predictive of safety performance. Because information flow is both influential and also indicative of other aspects of culture, it can be used to predict how organisations or parts of them will behave when signs of trouble arise. From case studies and some systematic research it appears that information culture is indeed associated with error reporting and with performance, including safety. Yet this relationship between culture and safety requires more exploration before the connection can be considered definitive.*
>
> **A typology of organisational cultures by R. Westrum**

The *big* question that DevOps researches asked themselves, is how can you take an IT organisation and transform the culture from power-centric towards generative. **Hackathons are one of the definite tools** that leadership can promote, which have an actual effect breaking down old ways of doing things and really changing culture.

This is why the Westrum model has been heavily quoted both in the 2016 State of DevOps Report, and in DevOps Enterprise forum guidance materials.

The PuppetLabs State of DevOps Report is a survey of more than 25k technical professionals. They were asked questions about various aspects related to DevOps. The section on "Lean Product Management", and the section on "Organizational Culture and Identity" specifically quote the Westrum model. Using it to explain how better culture is a key factor in better organizational performance.

The paper on Metrics for DevOps Initiatives at the DevOps Enterprise Forum Guidance explains how to *measure culture* using the Westrum model, and what organizational performance can be expected from it.

---

Key messages from the Westrum model:

> * To be able to work with and understand organisational culture, we need a typology of organisational environments.
> * Information processing style is a useful focus for such a typology, because information is important directly and is correlated with other features of the organisation’s culture.
> * Three typical styles of information processing are pathological, bureaucratic, and generative.
> * These styles are shaped by leaders’ preoccupations, including focus on personal needs, bureaucratic objectives, and the organisation’s mission.
> * These styles are associated with different responses to signs of trouble and opportunities for innovation.
> * Culture is mutable. With new leadership, an environment with one kind of culture can change into another.

Implications of Westrum model to clinical practice, most of which apply directly to IT organizations:

> * Leadership in the medical unit shapes the culture, which shapes the information flow.
> * Good information flow and processing has important effects on patient safety.
> * In particular, an open and generative culture will mean better uptake of innovations and better response to danger signals.
> * A generative culture requires that alignment, awareness, and empowerment replace suspicion, isolation, and passivity.
> * A culture of conscious inquiry will assist in getting fundamental improvements to the system, rather than just quick fixes.

---
Tags: culture, process
---


# Question
Title: Advantages of Linux Containers over Docker
I'd like to understand the differences between Linux Containers and Docker, especially what kind of advantages Containers have in comparison to Docker.

# Answer
> 6 votes
Docker is a specific implementation of Linux containers, or if you want to be more precise Docker is a distribution of tools that includes `runc` which is an implementation of Linux containers. Other implementations include rkt, LXC, LXD, and (I think) Snappy from Ubuntu.

---
Tags: docker, containers, linux
---


# Question
Title: What is a good logging practice for distributed tasks?
I have the following setting:

> Create multiple workers, do a computation and terminate them after the computation is done.

So, every-time it’ll be a different instance running the task, so each host will have its own a log file, this will result in a huge list of files.

Is it a good practice? If not, what would be a better way for logging the task processing in this particular use-case?

PS: My infrastructure is serverless. So, for now, I am logging to (AWS)CloudWatch. But, please answer the question independently of AWS, and suiting a serverless setup as much as possible.

# Answer
> 12 votes
"Serverless" mostly just means you've got relatively simple microservices, generally just a little webapp or a single function that is automatically connected to a REST frontend. The same concepts apply as you would use for a more traditional web services: usually some mix of remote syslog and ElasticSearch writers.

Networked or remote syslog has been around for a long time and has a fairly robust set of tools around it. You would have to run the central syslog server(s) but the protocol is very simple and there are pure client libraries in every language that you can use for sending logs. One common problem with remote syslog is that it has traditionally been based around UDP. This means that under heavy load, some log messages may be lost. This could be a good thing, helping avoid a cascade overload, but it is something to be aware of. Some newer syslog daemons also support a TCP-based protocol, but client support is less unified so just do your research.

More recent but very popular is logging to ElasticSearch. This is mostly useful because of the Kibana dashboard and Logstash tooklit (often called ELK, ElasticSearch+Logstash+Kibana). Amazon even offers a hosted ElasticSearch option, making it somewhat easier to get started. ES uses a relatively simple REST API, so any language with an HTTP client (read: all of them) should be okay with logging to ES but make sure you are careful with blocking network operations in cases of partial system outages (i.e. make sure your app won't get stuck in a logging call that will never succeed and stop servicing user requests).

More complex logging topologies are bounded only by your imagination, though these days you'll see a lot of use of the Kafka database/queue/whatever-you-want-to-call-it as a nexus point in very complex log distribution systems.

On the "serverless" side, you'll generally want to integrate with these systems directly at the network level, so sending log data directly to syslog or ES from your service/function, rather than writing to local files (though maybe echo to those too for local debugging and development).

# Answer
> 6 votes
This answer is more about scalability considerations - if the number of workers can be high and/or multiple of them can produce logs at high rate at the same time.

Yes, using multiple logfiles simultaneously is a good practice.

Attempting to combine into a single logfile logs from multiple workers **in real time** will raise problems:

* using blocking mechanims to prevent message loss will slow down the workers
* log messages can appear out-of-order in the combined logfile
* a centralized logging facility which combines the logs can be overloaded due to limited write speed, messages would be lost

Sharding logfiles (using multiple logfiles active in the same time) is itself a technique used by some hosting providers offering high performance, scalable centralized logging services. For example, when exporting logs to files Google's StackDriver Logging produces multiple sharded logfiles. From Log entries in Google Cloud Storage:

> When you export logs to a Cloud Storage bucket, Stackdriver Logging writes a set of files to the bucket. The files are organized in directory hierarchies by log type and date. The log type can be a simple name like `syslog` or a compound name like `appengine.googleapis.com/request_log`. If these logs were stored in a bucket named `my-gcs-bucket`, then the directories would be named as in the following example:
>
> ```
> my-gcs-bucket/syslog/YYYY/MM/DD/
> my-gcs-bucket/appengine.googleapis.com/request_log/YYYY/MM/DD/
>
> ```
>
> A single bucket can contain logs from multiple log types.
>
> The leaf directories (`DD/`) contain multiple files, each of which holds the exported log entries for a time period specified in the file name. The files are sharded and their names end in a shard number, `Sn` or `An` (n=0, 1, 2, ...). For example, here are two files that might be stored within the `directory
>  my-gcs-bucket/syslog/2015/01/13/`:
>
> ```
> 08:00:00_08:59:59_S0.json
> 08:00:00_08:59:59_S1.json
>
> ```
>
> These two files together contain the `syslog` log entries for all instances during the hour beginning 0800 UTC. To get all the log entries, you must read all the shards for each time period—in this case, file shards 0 and 1. The number of file shards written can change for every time period depending on the volume of log entries.

Such high-performance logging services can also offer alternatives to logging to files, management of logfiles can thus be avoided altogether if that is of interest:

Finally - if real-time logfile merging is not a requirement having multiple logfiles can help with offline log management:

* easy to devise progressive log backup, compression, archiving and eventual disposal schemes
* parallel processing of multiple sets of logs (logfiles) is possible, reducing/avoiding bottleneck effects
* no file splitting and re-writing necessary

---
Tags: logging, architecture, serverless
---


title: Getting a single revision from Git

text:
# Question
Title: Getting a single revision from Git
Having the full Git revision history has a lot of benefits as part of the development process.

But our product is the source code, we are using scripted languages that don't need compilation or processing, and then the Git history becomes a burden on deployment- in our example we deploy a clean virtual environment following every change, having several deployments on a single machine.

There are some ways to reduce the amount of history, for example shallow clones whose efficiency depends on how deep the revision is in the branch, doing a fetch instead of clone but then you still get history from the revision and back, or getting the full repo once then pull when needed but this is wasteful in terms of disk space and tends to be less reliable.

Is there a way to get a single revision from Git without it's history ?

# Answer
### Shallow clone

You can indeed get a shallow clone out of Git using:

```
git clone --depth=1 <url>

```

This will still clone the repo and create a `.git` folder with the objects, only smaller in size (difference depending on your total file size vs. history size).

### Git archive

You can also use git-archive to extract an archive of the repo:

> Creates an archive of the specified format containing the tree structure for the named tree, and writes it out to the standard output. If is specified it is prepended to the filenames in the archive.

In the examples it shows for instance:

> `git archive --format=tar --prefix=git-1.4.0/ v1.4.0 | gzip >git-1.4.0.tar.gz`
>
> > Create a compressed tarball for release.

### Hosted Git, archive API

If you are hosting your repo on GitHub, then you can use their archive API:

`https://api.github.com/repos/<username>/<repository>/zipball/<commit_hash>`

Bitbucket.org has a same functionality for this:

`https://bitbucket.org/<username>/<repository>/get/<branch_name|commit_hash|tag>.zip`
> 16 votes

# Answer
Don't deploy your git repo. Develop a real deployment methodology. Even if it's as simple as tarring up an archive (= building an artifact with just the necessary files in it to be deployed) of deployed scripts.

Even if you shallow-clone the tip of your source repository, you likely don't need unit tests, documentation, linting profiles, and other supporting ephemera in your deployed environment.

**Note**: for repositories of scripted languages that don't really have a "build" step, a trivial way to release an artifact would be to package them in an archive, such as tar or rpm. Then, to "deploy", you untar the archive or install the rpm. this removes the need for git tooling in your deploy chain (not all prod servers will have those dev tools).
> 11 votes

# Answer
> Question is there a way to get a single revision from Git without it's history ?

To get a repository, no there's no way, mainly because there's no 'revision'. Git store commits, which are changes from previous state.
If you want your repository at a specific point in time you have to pull the commit at this time and all it's ancestors or you'll get only the changes made in the commit.

To avoid confusion: Shallow cloning is getting the needed history and then truncating it to free space, the tree is still made from history.

For the solutions, @7ochem answer does cover them.
> 6 votes

---
Tags: git
---


title: Is there no Oracle JDK for docker?

text:
# Question
Title: Is there no Oracle JDK for docker?
```
REPOSITORY          TAG       IMAGE ID       CREATED             SIZE
an-image            1         X              26 seconds ago      279 MB

```

when the docker image will be run, the following message is shown:

```
No java installations was detected.
Please go to http://www.java.com/getjava/ and download

```

When Oracle JDK is deployed the docker image size is more than doubled!

```
REPOSITORY          TAG       IMAGE ID       CREATED             SIZE
an-image            2         X              26 seconds ago      666 MB

```

**Discussion**

When "Is there no Oracle JDK for docker" is googled only some links to docker image that contain docker are returned. When Oracle JDK is deployed in the docker image the size is more than doubled. I want to keep the docker images as small as possible, but the Oracle JDK seems to be larger than the image itself!

**Question**

Is there no Oracle JDK for docker?

# Answer
No. Because you cannot distribute Oracle JDK or JRE, the license in effect doesn't allow distribution. When distributed by a third party (embedded with your app) all the liability for it not working is on that party. This is why you will not find Oracle JDK/JRE on any of the public Docker registries, or in any Linux package repositories for that matter.

You can create your own image, and install Oracle JDK or JRE using the webupd8team/java package for Ubuntu/Debian.

And if you are using Alpine Linux this blog post explains which dependencies are required, and links to StackOverflow for installation instructions.

**Update**: Oracle has an official Docker image on the *Docker Store* now. https://blogs.oracle.com/developers/official-docker-image-for-oracle-java-and-the-openjdk-roadmap-for-containers

Link to *Docker Store* Oracle Java Image - https://store.docker.com/images/oracle-serverjre-8
> 23 votes

# Answer
As docker is not an operating system, no there's no JDK for docker.

As I understand your question, it's that a full JDK is too large for your wishes, in this case you may try switching to just the JRE and include only necessary libraries.

JDK is the acronym of Java Development Kit, JRE is the one for Java Runtime Engine. By "lazyness" due to the complexity to list every needed dependency no one strip down to the necessary libraries only, but that is the way to go if you wish to reduce the installed size to strict minimum.

A more usable way could be strip off some of the installed things which are taking a lot of space:

```
/usr/lib/jvm/jdk1.8.0_101$ du -hs *
776K    bin
4.0K    COPYRIGHT
5.7M    db
208K    include
4.9M    javafx-src.zip
187M    jre
133M    lib
4.0K    LICENSE
2.0M    man
4.0K    README.html
4.0K    release
21M     src.zip
108K    THIRDPARTYLICENSEREADME-JAVAFX.txt
176K    THIRDPARTYLICENSEREADME.txt

```

There's two \*src.zip which can be removed, they won't be of use, and as you can see the lib directory is half the size and within it you have a bunch of things you don't really need:

```
/usr/lib/jvm/jdk1.8.0_101$ du -hs lib/*
120K    lib/amd64
1.2M    lib/ant-javafx.jar
18M     lib/ct.sym
160K    lib/dt.jar
20K     lib/ir.idl
36K     lib/javafx-mx.jar
400K    lib/jconsole.jar
12K     lib/jexec
60M     lib/missioncontrol
4.0K    lib/orb.idl
8.0K    lib/packager.jar
2.4M    lib/sa-jdi.jar
18M     lib/tools.jar
34M     lib/visualvm

```

You can without problem remove missoncontrol on a server, I assume you can get rid of visualvm also, you'll have to check if you need something in this lib directory, but I can't tell without knowing your app..

One thing you should do in your dockerfile while installing oracle jdk is to remove the downloaded `.tar.gz` file which is roughly the same size as the resulting directory as `.jar` files are already compressed, the tar.gz is just there to "package" them.

You have to do all this in the same `RUN` command, chaining with `&&` to avoid adding layers where you won't remove size by removing files from previous layer.
> 8 votes

# Answer
There is no official Oracle JDK implementation provided by Docker. Docker used to support their own Java library on hub.docker.com, but they deprecated it in favor of the actual OpenJDK implementation since it was "OpenJDK-specific since it was first introduced". Their reasoning:

> As all of the major upstream Linux distributions are unwilling to redistribute Oracle Java in their own distribution channels, we have chosen to follow them. See references below on how each distribution does not distribute Oracle Java.

So if you want to keep your Docker image as small as possible, consider using OpenJDK instead. There's an official OpenJDK Dockerfile repository or you can just use `docker pull openjdk`. The basic "easy to run" Dockerfile for OpenJDK 7 is as follows (taken from the website listed in the previous sentence):

```
FROM openjdk:7
COPY . /usr/src/myapp
WORKDIR /usr/src/myapp
RUN javac Main.java
CMD ["java", "Main"]

```
> 3 votes

---
Tags: docker, oracle-jdk
---


# Question
Title: Are git pre-receive/update hooks serialized?
I'm investigating the possibility of implementing a "pre-commit" verification flow, enforced on the central SCM server side. By "pre-commit" in this paragraph I don't mean the `git` `pre-commit` hook, I really mean *before* the change becomes part of the integration branch, visible to all developers working on the branch.

For `git` in particular, which is a distributed SCM, the `commit` phase happens on the local server, which is not what I need. So I can't use the `git` `pre-commit` hook.

From what I gather the `pre-receive` hook would be what I'm after:

> This hook is invoked by 'git-receive-pack' on the remote repository, which happens when a 'git push' is done on a local repository. Just before starting to update refs on the remote repository, the pre-receive hook is invoked. Its exit status determines the success or failure of the update.
>
> This hook executes once for the receive operation. \[snip\]
>
> If the hook exits with non-zero status, none of the refs will be updated. If the hook exits with zero, updating of individual refs can still be prevented by the `<<update,'update'>>` hook.

Or maybe the related `update` hook:

> This hook is invoked by 'git-receive-pack' on the remote repository, which happens when a 'git push' is done on a local repository. Just before updating the ref on the remote repository, the update hook is invoked. Its exit status determines the success or failure of the ref update.
>
> The hook executes once for each ref to be updated, \[snip\]

The complication is that the verification itself can take a while.

My question: is the execution of these hooks serialized (by `git`)? Meaning will another hook invocation (due to a subsequent `push` operation performed **before** the previous hook execution completes):

* fail,
* block until the previous, still running execution finishes,

or

* simply run in parallel with the previous hook execution?

# Answer
In general git allows concurrent operations because one of them will eventually fail. Some specific database updates will have file locks (like updating the index files) but multiple receives can be in flight at once I think. In this case whichever verification completed first would be allowed in, and the second would fail due to having an invalid history (i.e. not being a fast-forward push).

In general this kind of operation is not well suited to git's server-side hooks since if something does fail, the person will then have to do some surgery on their branches to get things back in a valid state. These kinds of checks are generally run on the workstation side, though then you run into the "fun" that is distributing git hooks as they can't actually be checked in directly.

Another common solution is to have a CI service like Jenkins running verifications (tests) and updating a second branch or a utility tag showing which commits are acceptable.
> 3 votes

# Answer
Each Git hook is executed when a particular Git command is run. See the Git documentation on hooks for specifics. If you have two pushes updating the same ref, whichever one completes the "update" hook first gets to update the ref, and the other will get an error. It doesn't matter which client started to push first. It's all about which got to update the ref first.

When a push occurs, this runs a number of hooks. These three are run before a ref can be updated.

* pre-push - Client Side, called by git push and can be used to prevent a push from taking place.
* pre-receive - Server side, invoked by git-receive-pack on the remote repository. Just before starting to update refs on the remote repository, the pre-receive hook is invoked. Its exit status determines the success or failure of the update. If the hook exits with non-zero status, none of the refs will be updated. If the hook exits with zero, updating of individual refs can still be prevented by the update hook.
* update - Server side, invoked by git-receive-pack on the remote repository. Just before updating the ref on the remote repository, the update hook is invoked. Its exit status determines the success or failure of the ref update. The hook executes once for each ref to be updated.

My 2 Cents: **I think you're asking the wrong question.** Any use of a Git hook should be quick and simple. If your verification requires more time than a second or two, it's too long and would be misplaced in a Git hook. A long running hook will frustrate users because they'll think Git is unresponsive. My suggestion is to apply the FAIL FAST, FAIL OFTEN principle.

In your case of a long running verification, use of a tool to perform the verification outside of Git is ideal, such as a unit test for example, runnable from CI or locally. Proper use of pull requests (i.e. merging short lived branches) should check that your verification passed and then allow incorporation of such a change.
> 3 votes

---
Tags: git, git-hooks, workflow
---


# Question
Title: How to implement the four-eyes principle for emergency fixes?
Consider this scenario (any comparison with real world situations is purely by accident):

* **3:07 am**: incoming support call "*Something in production went down, I need your help!*".
* **3:12 am**: connected to the system (logon accepted) ... and no time for coffee.
* **3:15 am**: lucky you, right away you could spot the issue via some error message somewhere.
* **3:17 am**: use your SCM toolbox to grab the code, fix the issue, test it, great ... my fix works!
* **3:20 am**: get in touch with the DevOps-team to ship the fix and to get production running again.
* **3:21 am**: red flag ... "*To respect , we need 2 more eyes to get approval for this fix*".
* **3:22 am**: ggggrrrreat, now what, who else can we call (= wake up some manager)?

If you implemented some approval procedure similar to my answer to "What are possible implementations (or examples) of the four-eyes principle?", then you're out of luck ... here are your choices:

* Your fix will be stuck (read: production will be down) until 2 more eyes got involved.
* You figure out a way to get around the missing eyes.

***So how to implement the four-eyes principle for emergency fixes?*** ... So that you get production up and running asap, i.e. around **3:25 am** ... And so that you can also close the call (and go back to where you came from)?

# Answer
> 8 votes
In the SCM-world where I'm mostly familiar with, the above scenario is typically addressed by what's called the "***abbreviated***-approval list procedure.

Here is a blueprint of it:

* Define your business hours, say from 8 am to 6 pm.
* Define a ***complete*** approval list of (say) 3 levels of approval (for roles X, Y and Z).
* Define an ***abbreviated*** approval list of (say) only 1 level of approval (only for roles X).
* ***Planned changes*** always require all approvals from the complete approval list.
* For ***Unplanned changes***, the complete approval list is used *also* to gather the required approvals, provided the approvals are to be issued *during* the defined business hours.
* For any approvals of ***unplanned changes*** that are to be issued *outside* the defined business hours:
    + Only the approvals from the abbreviated approval list (such as role X above) are required to authorize the change. And after the authorization by the abbreviated approval list is given, the deployment of the change (in the target environment) will actually be performed.
    + But additional ***post***-approvals will be needed afterwards (within a reasonable amount of hours/days), i.e from all roles contained in the complete approval list (such as role Y and Z above), which are not also contained in the abbreviated approval list (such as role X above). And if within the (upfront) agreed amount of hours/days not all post-approvals have been issued (e.g because the fix worked "this" time, but was only like a temporary fix), then the change might be subject to a rollback. While there is at least 1 outstanding post-approval, the change is marked as "waiting post approvals".

With such solution in place, the call can be closed around **3:23 am** ... since there will be no more red flag at **3:21 am** ... ggggrrreat, time for a beer to celebrate my fix to get production going again (instead of coffee) ... and fingers crossed the outstanding post approvals will come in soon ...

# Answer
> 3 votes
In the case of off-hours emergency fixes, it's more practical to require less sign-off for changes than your normal procedure. Generally, you can deploy a fix and then do post-approvals the next business day. If the fix is not approved, it can be reverted and replaced with a permanent solution.

During an outage situation, the number one priority should be to restore service. If your organization does not recognize this relaxed process during an outage, then yes, your only option is to start waking more people up for sign off.

---
Tags: configuration, four-eyes
---


# Question
Title: How to investigate a main process that has died in a docker container?
Sometimes you have to investigate a container, which is stopped, or a container which after starting up dies very quickly and stops.

`docker exec -ti <id> bash` only works on running containers, once it finishes, the bash prompt terminates as well.

With `docker start` you cannot supply a different command, and if the container dies abruptly again you won't have enough time to get into the container and do your investigations.

We can do `docker commit`, then `docker run` on the new image with a different command, but I'm wondering if there are any other alternatives.

**Note**: `docker logs` just returns whatever the apps printed to stdout/stderr. That might not be enough to figure out what the problem was.

# Answer
> 9 votes
General ways to track why a process in Linux failed are good. One such way is to run a process using `strace` which will tell you the system calls process did and usually point to the reason for a failure.

You can create a `Dockerfile` that looks something like this:

```
FROM original_image

RUN apt-get -y update && apt-get install -y strace

# build with `docker build -t debug_version`

```

Then run your new image using `docker run debug_version strace original_cmd`.

For processes that fork children (and then die) you want to run `strace` with the `-ff` option. You can also map some file using Docker data volumes and use the `-o` option from `strace` to write to it. But in general `strace` will leave output on stdout, which is readable using `docker log`.

Related Q: Linux process terminates mysteriously

# Answer
> 3 votes
As far as I am aware, `commit` and `run` are the best options here to give you full access to the container as it was when it died.

Ideally, your container would spit out some more useful information when it fails, but that is another topic altogether.

Edit: to expand my answer, if the container is dying right on start, you can also use `docker run` to specify an alternative `--entrypoint` and `CMD`. Generally I'll set this to a loop or something that won't exit on its own. Once you're in the container, you can manually run the steps that are failing, and then inspect the outcome without having to worry about the container exiting.

---
Tags: docker, containers, post-mortem
---


# Question
Title: What is the cleanest branching strategy to use when creating reusable artifacts?
If I have 3 environments : integration, staging, and production, what is the cleanest branching strategy to use, assuming I want to reuse my deployed artifact?

For example, should feature branches be merged into master and then built to be deployed to integration? Or is there another strategy that allows you to have 1-to-1 mapping between your releases and SCM?

Some more details ... Say I merge into master, and that kicks off a build, which produces an artifact. That same artifact should be deployed to all environments to ensure consistency. Is it best to start that process with integration? In that case, there is a rather large potential window where untested code is in master before verification can occur. Is it better to use an integration branch, deploy that to integration, then merge to master and use that artifact for staging and production? Or does that just increase the need for duplicate testing?

# Answer
> 6 votes
The simplest/cleanest branch strategy is IMHO the one used in **continuous deployment**: a single/main integration branch which is also your release branch. From What is Your Branching Model?:

> Commits can go all the way to production from one trunk/master, if the automated build says the commit was good. It’s the turbo-switch for TBD, where no humans can hold up a release by taking their sweet time testing it and giving a belated seal of approval. Github, Etsy, Netflix (and many more startups) are all here.

You'll effectively use the CI build artifacts from that main branch across all environments. Well, as long they build and pass the respective QA verifications, obviously.

**Note:** there are no feature branches in this strategy, at least not long-lived ones - in order to eliminate/minimize the integration delays and costs. Practically you can eliminate branch merges from your development.

---
Tags: branch, scm, workflow
---


title: How to use feature flag toggles?

text:
# Question
Title: How to use feature flag toggles?
What are the different ways to use feature flag toggles in applications?

If you were to explain to a developer the exact things that should be done in order to get from nothing to a full feature-flag-toggled application, what would those steps be?

# Answer
> 16 votes
Feature flags are an engineering device that can be used to avoid long-lived branch and conflicts in product development. Here is how it can be used the context of an object-oriented language to help developers collaborate on a specific product feature while one handle a new version. This solution can also be used in non object-oriented contexts, provided a notion of “interface” exists. (*cf.* OCaml module system.)

For the purpose of illustration, we assume a tool presenting reports about data stored in a database. The code implements a *DatabaseClient* class used to perform requests. As the dataset grows, it becomes clear that some alternative data layout would improve the application performance. Therefore *Alice* will develop a new version of the *DatabaseClient* able to retrieve data from the structures with improved layout, while *Bob* will maintain the historical *DatabaseClient*.

With the following steps, Alice and Bob can collaborate on short-lived branches while minimising their conflicts.

1. *Alice* rename *DatabaseClient* to *DatabaseClient\_v1* and create a delegate class called *DatabaseClient* that uses an object *DatabaseClient\_v1* and implements an interface called *DatabaseClientInterface.* (If possible, this *DatabaseClientInterface* should be a code artefact but duck-typed languages not always support this.)
2. Bob reviews changes made by Alice in 1 and is aware that his maintenance job should happen on *DatabaseClient\_v1*.
3. Alice introduces a new configuration flag in the application that governs the behaviour of the *DatabaseClient* delegate and implements a *DatabaseClient\_v2* placeholder, a class implementing the *DatabaseClientInterface* whose methods all throw a “Not implemented” exception.

After this, Alice and Bob can collaborate without explicit synchronisation, because code written in their respective iterations is subject to the *DatabaseClientInterface*. This minimises the risk of a conflict resulting from their concurrent work.

Iterations from Alice can be very short, like implementing a test, implementing a method, or even partially doing so, because in production, the code is not selected for use and does not need to be fully functional. The automated testsuite should be configured so that the *DatabaseClientInterface* always uses *DatabaseClient\_v1* while Alice can easily toggle to *DatabaseClient\_v2* when running the testsuite locally – or in a custom CI setup. Once everything is ready, a single commit can perform the change, by updating the configuration value governing the *DatabaseClient* delegate.

# Answer
> 7 votes
The steps are quite "easy", to move to a feature flag app you need basically two things:

1. A flag repository (file/data base/env variable)
2. Conditional statements to change the behavior according to the flag.

The basic of feature flag is to turn them on/off, but quickly you'll wish to release a new feature in a ramp-up manner, for example: 1 server on 5 hosting the app have the feature "on" to start, you then turn on the feature on another server, until all servers have it "on".

This means you have to be careful about your feature being compatible with the app without it (extra column in DB for example).

Frameworks exists in various languages to avoid reinventing the wheel, the now un-maintained one from Etsy's have an interesting readme to explain how it works.

# Answer
> 2 votes
The embedded software world often uses build-time flags, in the app code itself (`#define`/`#ifdef` statements, for example) and/or in build tools configuration files (`makefile`'s, for example).

Build flags can be used, in similar manner, not only for features, but also for all kinds of code refactoring, migrations, debug support, etc). They allow committing in the integration branch partial or unverified changes without breaking the build or causing regressions in the features/projects already working in the branch. Excellent for handling point fixes alongside large/risky/slow progress changes (which would otherwise require a long-lived branch) in a continuous integration manner.

But in addition to verifying the already existing branch code for regressions, it's also possible to perform progress/stability verifications of the new code. For this the build-time flags need to be toggled.

One way of toggling the flags would be to use, in a separate verification pipeline of the CI system of the same branch (if it has support for such functionality), a patchfile toggling the flag - to be applied to a separate workspace prior to the build. A different set of artifacts would be built in this workspace and then verified.

Alternatively a long-lived feature branch can be pulled from the main integration branch, but the only change in this feature branch would be the toggled flag. Due to this tiny change the feature branch can be automatically sync'd extremely fast - practically shadowing very closely the main integration branch. A separate CI execution on this branch would not need a preliminary patchfile anymore. It'd be trivial to carry around such feature branch even for an extended period of time.

It may also be possible to create, in the main integration branch, new build artifacts which would really be just clones of the existing build artifacts but with the flags toggled. This way neither the preliminary patchfile nor the feature branch would be necessary to verify the new code, right in the main branch.

---
Tags: terminology, continuous-delivery, feature-flags
---


# Question
Title: What functional differences are there between kubernetes, rancher and mesos?
What are the main feature differences the popular container orchestration tools kubernetes, rancher, and mesos?

What would lead you to choose one over the other?

# Answer
First of all, Rancher actually contains implementations of both Kubernetes *and* Mesos within itself. However, they did make their own system called Cattle which is heavily based on Docker's Swarm. I'll touch upon this in the section for Rancher.

Secondly, since each one offers similar base features (load balancing, resource isolation, etc) I'll try to go more into what makes them different rather than focusing on differences among those common features unless they're significant.

# Kubernetes

Highly performance focused, also featuring cloud storage orchestration (a feature missing from Mesos, although there's probably a plugin for it). Has API options to allow for automated scaling of resource usage as needed by individual containers (and for the reverse if a container is not being hit hard while others are).

Something important about Kubernetes is that unlike other container orchestration software it doesn't provide a comprehensive configuration or any kind of comprehensive self-healing. Instead, it focuses on continuous deployment of multiple apps with an easy rollback system at the app level (as a result you might want to look into micro-services when using it).

Each app is a small piece and can be deployed/configured individually. They make a point in their docs to say that Kubernetes is not a traditional PaaS (platform as a service) system since it lacks middleware for virtual hardware or databases and it doesn't build your app itself. It's designed to (as they say themselves) remove the need for manual container orchestration, instead automating the process by continuously pressing towards a target app state.

# Mesos

Monolithic in comparison to Kubernetes. Focuses on the big picture moreso than the individual services, although it still allows for management of individual services. Provides built-in middleware for things like Spark, Hadoop, etc. Best usage of Mesos will involve many plugins as it's designed to be easily extended.

If you need fine-grained control over managing your application (insofar as there's a plugin available for what you want to do or you have a team member willing to build one if there isn't) you'll want to use Mesos.

# Rancher (and Cattle)

Potentially the best option in that it is itself a superset of the previous two, having an implementation of both. This might also be seen as a downside as more complication in your management software is rarely a good thing and might lead to unforeseen issues.

Rancher features an application catalog that allows for one-click deployment, something Kubernetes doesn't have due to its design philosophy. However, seeing as Rancher has an implementation of Kubernetes, you can use Rancher if you feel these features are missing from Kubernetes.

Cattle is based off of a stack system, where you group related services together. It also houses Rancher Compose, a feature similar to the Docker service of the same name. This is probably the most interesting part of Cattle, the rest of it being fairly standard (although the Secret management is still in beta). If you have experience with Docker Compose you should read up on it here (I don't, so I'm probably not the best person to write about it).

Resources: "What is Kubernetes?", "Overview of Rancher", "mesos.apache.org: 'What is Mesos?'"
> 18 votes

---
Tags: kubernetes, rancher
---


# Question
Title: How to avoid a regular user can only test in production?
When configuring test environments, the issue often comes up where I ask the question to the customer like in my question title here:

> **How to avoid a regular user can only test in production?**

There are typically the environments like unit test, system test, integration test, volume test, user acceptance test, etc (I bet there are yet another dozen of names like those). But they all relate to environments in which something differs from how things currently work in the real production environments. So none of these is what my question is about.

To make this a less abstract question, consider this sample below:

> Can non-SE employees only test SE sites in production?

I run into this scenario over and over (earlier today it happened again ...):

* I discover some facility on an SE site that gets my attention (that I want to start using).
* I start looking around in help pages, etc, and either don't find what I'm looking for, or what I find is not 100 accurate or clear to me.
* I just try to start to use it, see what happens, hoping I will find the answer to what I couldn't find in the help pages (by just using that facility in some SE site ... read: by ***testing in production***).
* I then just wait and see what happens, which may be things like:
    + I learn about something new/interesting and it works great.
    + I get accused of trying to "fuzzle the system".
    + I get corrected by a mod because of something I did (which I shouldn't have done).
    + I discover a bug, which I either report (on meta.SE), or which I decide to consider as "a bug becomes a feature" (read: I don't want to disclose this handy feature).

However, doing some of those "experiments in production" may damage your reputation (not really your "points", but your real reputation).

So as a good SE citizens, I would be happy to do those experiments in an environment that is designed *just for that*: a none-production environment designed for users like me who want to do such experiments.

Voilà, if you were asked to configure test environments for SE sites, what would you do (or recommend SE management) about this DevOps issue?

PS: to my knowledge, at this very moment, there is no such SE environment that I can use right now to go check if a user that I "invite right now" for DevOps (while it is in private beta), and which I know did NOT commit to DevOps, will actually be able to login to the DevOps site. So instead of discussing it with mods on some other site, I already ... ***tested it in production***. And despite what some moderator try to make me believe ... it did work (and the invited user now also has an account on DevOps.SE).

# Answer
### A Sandbox could be part of the solution

To bring material to the subject you can check the questions with tag on MetaSE, there are **sanboxes** for:

* Q/A formatting.
* Comments formatting.
* Chat.

There's a feature request about "Can we get a Stack Exchange sandbox?" which has no status (neither declined, pending, review nor in-progress meta-tag).

So I assume the answer (to the specific sample in the question) is that there's no way to test out of real SE sites for now and my recommendation to SE would be to accept this feature request.

### When is a sandbox appropriate?

I don't think it's always needed to give users a sandbox, and it can be overkill sometimes. Defects should be spotted on Q/A environments, and when they're not, well, either you can know there's a glitch by a fine monitoring of your logs, or you will just never know until someone tells you.

### Sanboxes come at an additional cost

Even if you deploy a sandbox for your users to try and confirm a bug, nothing tells you they will inform you of a bug/problem and you had to go through adding a public environment with **dummy or anonymised data** with no idea of it's ROI.

So It boils down to: "You can create a sandbox for your users but it adds complexity to your infrastructure for a hardly measured return"

### Recommendation

My personal opinion would be to put the efforts in log analysis to spot unwanted behaviors instead.
> 6 votes

---
Tags: testing, scm
---


# Question
Title: What can Vagrant, Puppet, Docker, Chef do for a Full stack Web developer?
First of all let me say that I'm aware of related questions here in SO:

Or articles out there:

But I'm having trouble understanding real benefit they provide for a web developer (e.g. Full stack developer with (PHP/react)). You know? None of them is that clear for me in:

* what facilities they could provide & how they make my job easier?
* When do they become semi vital for project success & non-self-repeating activities?

Would you mind clarifying these concepts/techs from my POV? If you mind, please avoid comparing with routine "Vs." literature & Try to introduce real-life good-parts. A brief explanation from anybody would sufficient.

# Answer
First of all, I strongly advise against "feature comparisons" blog post for similar software, they're quickly outdated and I'll try to keep this answer generic for a system configuration manager (SCM) and let docker out of the talk for the first part.

What it can brings you:

* Reproduction of configuration from environment to environment
* Versioning of the configuration
* Enforcing the configuration

To get a real life example for a PHP application you'll need:

* an operating system
* a webserver
* php libs
* configuration of your app for external services (smtp, database IP, etc.)

What a SCM can provide is starting from a minimal base OS:

* taking care of installing the webserver (apache/nginx)
* installing PHP (either by package or source)
* installing the necessary libs
* Setting up your site config (vhost in apache)
* Setting up your php.ini with any tweaks needed
* deploying your app from a repository (from git to nexus, wide range of choice)
* Writing your application configuration (here can be the smtp or database server for this environment)
* Starting the webserver
* Eventually running some smoke tests on your app.

Now if someone comes to connect to the machine and do a manual change to any of the configuration, the goal of a SCM is to keep the machine in its desired state, at next run, the change will be detected and the file changed will be put back to its desired state.

This has the advantage of enforcing any change to be put in the desired state description and not be forget between environments. Usually chef or puppet are set up to run on a periodic schedule, default for Chef is 30 minutes.

Now the big difference with docker is that you will do the same thing, installing you middleware, libs, forcing a templatized configuration, etc.
Once in the runtime phase, if there's a breach in your app and an attacker use it to tweak your config or modifying your webserver configuration it can stay unnoticed for a while if it doesn't break the app.

A SCM will revert the changes if you use it properly it will even remove unwanted sites in the `sites-enabled` directory of apache and thus will raise the complexity for an attacker.

Likewise if your webserer dies, the container will stop and unless you have a system to monitor it an relaunch it you're down. A SCM will notice the service is stopped and restart it at its next run, less immediate than runit but it will correct a rotate script which didn't properly restart the service (like the default apache logrotate conf on Ubuntu which may time out and let your apache stopped)

A SCM may be interesting even to create your docker containers, you know you'll have the same behavior on a centos or ubuntu machine (most community code handle using the proper package names if you go this way) and so it will remove problem of edit to package names in a dockerfile.

---

The main idea behind a SCM is to manage your infrastructure as you manage your application, code it, version it, test it and be sure it will always behave the same.

If you rely on human procedure for configuration of the middleware/libs/app config you can be sure there will be a step forget during the deploy process once in a while, and spotting this problem can be really hard. With a SCM you prevent this omission.
> 6 votes

---
Tags: docker, chef, vagrant, puppet
---


title: How to identify yourself on a CV

text:
# Question
Title: How to identify yourself on a CV
So here is your job role:

* You help in designing common modules that allow other teams to easily deploy their code onto production. This includes Java code, dockerfiles, ansible modules, build pipeline configurations, etc.
* You help promote DevOps practices for both other developer teams and the operation teams
* You are partially responsible for the staging environments
* You are responsible for investigating technologies that help the applications scale better and be more reliable

How would you identify your team? How would you identify yourself? You know that you could cheat and include DevOps in your name, but that would be wrong. On the other hand you want to be more specific, so a simple Software Engineer won't cut it, especially since you want the annoying recruiters to contact you for jobs related to this practice, not just because you like it, but because you know the average salary is also higher.

# Answer
Technically mentioning DevOps *might* be wrong (the question you referenced is also subject to opinions).

But it can also be seen as clarifying/disambiguating - the term is widely used presently on the job market, not only according to your salary reference but also from the job ads themselves - see What is the difference between Sysadmin and DevOps Engineer?.

You'd only speak the recruiters' language - which is IMHO more important from the communication perspective than potentially being technically correct.

Language and culture are fluid - something wrong may become de-facto correct and vice-versa.
> 3 votes

---
Tags: culture, job-role
---


# Question
Title: How can Jenkins help to implement CI/CD for Drupal?
Assume you want to implement (= CI) and (= CD) for building and maintaining websites using Drupal (let's assume for Drupal Version 7). Often times Jenkins seems to be recommended as one of the most appropriate solutions for doing so.

But what kind of CI/CD related functionality can I actually use it for? Here are some of things I'm thinking of, but I'm not sure if I'm correct on that, and/or if these are the only things:

* CI related: Automated testing.
* CD related: Migrating custom module development between dev / staging / QA environments.

# Answer
> 4 votes
What you've listed is actually two different things, continuous delivery and continuous integration. When people talk about CI they typically are only referring to the testing part; automated unit, function, and integration testing with some sort of version bump or code promotion at the end. Looks like the Jenkins wiki has some specific pointers about Drupal.

As far as environment promotion, you're getting into continuous delivery which Jenkins can also be used for. Consider looking at the more modern Jenkins Pipelines for CD pipelines.

---
Tags: jenkins, continuous-integration, continuous-deployment, drupal
---


# Question
Title: How to implement the manual step at the end of continuous delivery?
The accepted answer to my question about "How does continuous integration relate to continuous delivery / deployment?" also explains the ***small difference*** between *continuous delivery* and *continuous deployment*. It appears to be related to the answer to a question like "How do you want to deploy to production, whereas these are the (exclusive) options to pick from:

* Auto(matic).
* Manual.

I cannot imagine that there will be a poor "operator" at the other side of the DevOps wall, who is going to have to do something that corresponds to whatever that "manual" means ... My questions:

* Is my reference (in my question) to "distribute" versus "install" close to a possible implementation of such "manual"-thing? Here is a relevant quote of my related question:

> * ***distribute*** to some target environment, using something like FTP (if standard copies cannot bridge the gap), but do not yet activate it in the target. That's what should be similar/close to *continuous delivery*, or not?
> * ***install*** (or *activate*) in some target environment, combined with things like binds, stop/start operations, etc. That's what should be similar/close to *continuous deployment*, or not?

* What are other possible implementations of it?

# Answer
> 5 votes
Personally, I consider the *distribution* of the software to a target just an intermediary step of a deployment - installation/activation of that software is still needed to complete that deployment.

To me the *delivery* (as in *continuous delivery*) stops when the software to be deployed is created and made *available* for deployment (i.e. for distribution, installation and activation).

So, to answer your first question: No, I wouldn't consider distribution and installation as reflecting the manual step differentiating continuous delivery from continuous deployment.

Yes, in some (hopefully rare) cases that manual step is just the final human decision for deploying into production, reflecting the cultural mistrust in process automation and the mental comfort of having a human double-checking and signing off the deployment decision (thus assuming responsibility for it) even if that decision is purely made based on an algorithm that can be automated (like counting pass/fail testing results).

But in general it simply reflects the fact that the decision for performing the deployment in production is not simply the result of an automated algorithm. Here would be some example such cases:

* the automated decision is overwritten
    + deployment can be signed off even if not all quality criteria are met (we all know that this is not just a theoretical case)
    + the deployment is held for whatever reason, even if all criteria is met (due to market timing implications for example)
* the automated algorithm is not (yet) implemented/deployed
* the algorithm includes checks for some criteria depending on human decisions (say results from manual testing)
* deployment is actually done in a 3rd party customer environment, following additional customer checks

So, I wouldn't look at the manual step simply as an implementation issue.

# Answer
> 2 votes
One additional consideration if you are publishing something that you are expecting other projects to consume, deploy also takes on the meaning of "publishing for others to use"

Consider a workflow where you deploy a library to a common artifact repository. This part of the process might be part of you deploying another component which requires that artifact at build time or it might simply be an update to a common library. But regardless, for that artifact, it's life cycle doesn't necessarily end by being made available for consumption by others, but the deployment of that artifact to the artifact repository may be the final stage in the developers work after they have decided to cut a new release version and before others can safely consume the new version.

---
Tags: continuous-deployment
---


# Question
Title: What exactly are "Truly reproducible builds"?
What exactly are they? Why are they important, in the domain of Continuous Delivery?

Context: I have seen in one of (I guess reddit)'s comments that Truly Reproducible builds are still an under-research technology, and is very difficult to create.

So, I wanted to know why are they so difficult to create?

# Answer
> 8 votes
### What exactly are they?

Here is a quote from reproducible-builds.org:

> Reproducible builds are a set of software development practices that create a verifiable path from human readable source code to the binary code used by computers.

### Why are they important?

IMO the easiest way to explain their importance is to consider them as a variation of a backup procedure.

As an example:

* Assume a business which uses (depends) on some software package licensed from some software vendor. Whereas the business only gets the executables, not the sources, etc that were used to create those executables.
* Everything goes well, but at some point something goes wrong with the software vendor, e.g. they go out of business (eg bankrupcy).
* This may expose a risk to the business (in the long run). I.e. if there is no procedure/agreement in place for the business to get (legal) access to all required sources, documentation, build procedures, etc related to anything from the software vendor used (back in the days) when the executables (used by the business) were created (and shipped to the business).
* That's were "*Software Escrow*" comes to the rescue: if there is such agreement in place, one would think that via a 3rd party, it would still be possible for the business to get access to "*whatever was used*" to be able to reproduce the executables, so that from there on, the business might have a chance to continue to use that software, and where apropriate start maintaining it themselves (for only running their own business).
* However, the "*whatever was used*" in the previous bullet is the toughest part to make this work. It requires that the 3rd party performs appropriate validations upfront. And trust me, it takes a while before you can recreate an executable for which you can prove that, apart from (eg) the link date, it is a perfect match with what the software vendor delivers to the software agent.

### And why are they so difficult to create?

If the above sample is still not clear enough, imagine you're my software escrow agent, tell me what you need as input to recreate a copy of the software licensed by my customer. Get it? You didn't forget to check about which version of my compiler, maybe my OS, compile/link options, versions of reusable components (includes), libraries, etc?

# Answer
> 4 votes
To provide a practical example of an attempt at creating a truly repeatable build consider the following -

A build pipeline which starts with a git repository for which no user can ever rewrite history or delete unmerged branches.

The first "build" step after checking out the source code is to spin up a container which contains all the build time dependencies.

The output of the build time container running is a container which contains the compiled binary.

More important to the repeatability of the build the following tags are added to the final container:

* The exact hash of the source code in the original repository and the url of both the git repo and a tar ball snapshot of the code that is uploaded to an artifact repository.
* The exact version of the build container that was used to run the build.
* The exact version of the original base image that the binary was loaded into.
* The values of all build-time variables used to create the binary.
* The version of docker that all three containers where built with as well as the version they where running under when they built.

By adding all of this meta data we can ensure that at any point in the future we can pull out the exact set of build dependencies (via the build container), compile the binary with an exact known set of steps (enshrined on the build container) and package this into another known base image with all run-time dependencies (using the base image tag) and this can all be based on the exact correct version of the source code based on the tag on the container.

Theoretically this should give us he ability to exactly reproduce a build version.

The importance of this is that it allows us to look at what's running in production and, even if everything has progressed versions significantly, go back and pull out the version of code, base image and build container originally used so that we can, for example, apply a hot fix to that version before rebuilding exactly as before so that we can redeploy knowing that it is exactly the same artifact with the only delta being the hot fix.

---
Tags: terminology, continuous-delivery, builds
---


# Question
Title: What are the Devops-related conferences happening around the world?
This is a similar question like the Statistics Conferences question at CrossValidated

What are the Devops or Devops-related conferences happening around the world?

Rules:

* Include a link to the conference
* Please include links for the talks, e.g. youtube, the conference site or some other video streaming site

# Answer
Velocity Conference is one of the most popular Devops conferences, and also is an O'Reilly conference.

From it's website:

> At Velocity, systems, site reliability, and systems engineers, systems architects, and application developers learn to build and manage fast, resilient, and highly available websites and apps.

The videos of the conference are present on O'Reilly's youtube channel.
> 2 votes

# Answer
This question reminded me about this quote about the Etymology of DevOps, From Wikipedia:

> At the Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure". The term DevOps was popularized through a series of "devopsdays" starting in 2009 in Belgium. Since then, there have been devopsdays conferences, held in many countries, worldwide.

So at the risk it may not 100% fit the rules in your question, I thought I'd post what, IMO, is "the reference" to get an up-to-date answer if you ever wonder something like

> **What's up in DevOps-land?**

In such case, you probably can find the answer to your question at http://www.devopsdays.org/ .

At this very moment, it shows links to these locations:

> Beijing - Vancouver - Denver - Atlanta - Tokyo - Seattle - Zürich - Austin - Stockholm - Salt Lake City - Toronto - Amsterdam - Washington, DC - Minneapolis - Portland - Dallas - Raleigh - Chicago - Detroit - Istanbul - Paris - Hartford - Philadelphia - Oslo - Cape Town

PS: From the above quote, it should also be clear what the relationship is with DevOps and Chocolats, Beer, Fries and ... Drupal.
> 1 votes

---
Tags: conferences
---


# Question
Title: How to promote a build when certain value in output is above threshold?
I'm using the Promoted Builds Plugin and I'd like to promote my freestyle job to have a gold star when a certain value in my output is above a given threshold (e.g. above 1.00).

Here is a sample output from backtesting jobs:

```
Pass  Profit Total trades Profit factor Expected Payoff Drawdown $ Drawdown %
   1  622.71           76          1.38            8.19     526.77       4.94
  10  301.09           36          1.38            8.36     526.77       4.93
  14  240.96           62          1.18            3.89     589.60       5.60

```

My aim is to promote the build to gold when the first line right after the header line has a `Profit factor` column value above `1.00` and only to silver star when it is below.

How can this be achieved?

# Answer
What you are looking for is at the bottom of the GitHub README describing Job DSL for the Promoted Builds Plugin. Link to GitHub is on every plugin's page. There are also links to plugins it depends on and you should read the documentation for those like the Job DSL plugin wiki. You can also ask questions in the Job DSL forum or simply read the documentation. Here is a plugin that extends the promotion syntax of the DSL.
> 4 votes

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: Git clean/smudge filters for ansible vault secrets
I am trying to setup clean/smudge filter in git to have automatic encrypting and decrypting of files containing secrets thru ansible-vault command.

Peculiarity of the ansible-vault command is that it is not idempotent (it creates a different binary each time it is invoked on the same data).

I started with the implementation suggested in this blog page. Unfortunately it did not work correctly, as whenever smudge is called (be it a git checkout , or just git status), the secret files looks as modified for git, even if it is not.

So I wondered if git would be comparing the binary he has in the index with the clean filtered current file, and I tried to build on those script like follows:

```
#!/bin/sh -x
# clean filter, it is invoked with %f

if [ ! -r "$HOME/.vault_password" ]; then
  exit 1
fi

tmp=`mktemp`
cat > $tmp

# get the plain text from the binary in the index
tmphead=`mktemp`
git show HEAD:$1 > $tmphead
contenthead=`echo "embedded" | ansible-vault view $tmphead --vault-password-file=$HOME/.vault_password`
export PAGER=cat
echo -n "$contenthead" | tee $tmphead

# if current and index plain text version differ
if [ "`md5sum $tmp | cut -d' ' -f1`" != "`md5sum $tmphead | cut -d' ' -f1`" ]; then
  tmpcrypt=`mktemp`
  cp $tmp $tmpcrypt
  # generate a new crypted blob
  echo "embedded" | ansible-vault encrypt $tmpcrypt --vault-password-file=$HOME/.vault_password > /dev/null 2>&1
  cat "$tmpcrypt"
else
  # just return the HEAD version
  cat "$tmphead"
fi

rm $tmp $tmphead $tmpcrypt

```

The difference here is that it tries to compare the current and HEAD versions of the plain text (unencrypted) secret files, and only in case they differ output a new binary blob encrypted with ansible-vault.

Unfortunately, after this change git continues to think the secret file is always modified. Even after `git add`ing the file again, so that the git blob is computed, git thinks the file is different and let the change go into the commit. Note that `git diff` return empty changes, as it should.

For reference, this is smudge:

```
#!/bin/sh

if [ ! -r "$HOME/.vault_password" ]; then
  exit 1
fi

tmp=`mktemp`
cat > $tmp

export PAGER='cat'
CONTENT="`echo "embedded" | ansible-vault view "$tmp" --vault-password-file=$HOME/.vault_password 2> /dev/null`"

if echo "$CONTENT" | grep 'ERROR: data is not encrypted' > /dev/null; then
  echo "Looks like one file was commited clear text"
  echo "Please fix this before continuing !"
  exit 1
else
  echo -n "$CONTENT"
fi

rm $tmp

```

and this is diff:

```
#!/bin/sh

if [ ! -r "$HOME/.vault_password" ]; then
  exit 1
fi

export PAGER='cat'
CONTENT=`echo "embedded" | ansible-vault view "$1" --vault-password-file=$HOME/.vault_password 2> /dev/null`

if echo "$CONTENT" | grep 'ERROR: data is not encrypted' > /dev/null; then
  cat "$1"
else
  echo "$CONTENT"
fi

```

# Answer
The problem here is caused by the random salt in ansible-vault encryption. You can hack VaultEditor class to pass the salt to it from an argument in ansible-vault. The random salt is generated in `lib/ansible/parsing/vault/__init__.py` on this line. It is called from lib/ansible/cli/vault.py where you can easily add the argument for fixed salt. If you do change it, please submit an upstream patch to Ansible, I'd love to use it.

This problem is further discussed here on hacker news. And there are other implementations with tools that take fixed salt, namely gitcrypt, transcrypt. Here is also a link to one more implementation using ansible-vault called ansible-vault-tools, but this one has the same salt issue as far as I know.
> 9 votes

---
Tags: ansible, git, automation, ansible-vault
---


# Question
Title: How to avoid continuous integration-caused instabilities in test environments?
Assume you're using continuous integration processes which frequently update some target environments, so that every time there are some changes "you" can test your changes right away. That's part of the goals of CI, no?

But, also assume that you have other people involved in your test cycle, e.g. managers or customers. Makes sense to get other people involved in trying to review (break?) your upcoming changes, no?

But if you **continuously** keep delivering changes in the environment in which those other people are, seriously, trying to test them, then multiple issues may arise, such as:

* **`they`** might waste their time in reporting issues which, by the time they save the (in depth) report, they cannot even reproduce the issue themselves anymore (e.g because accidently you also ran into the same issue, and already fixed it in their environment).
* **`you`** might not be able to reproduce issues they reported, since the environments in which they ran into some issue, is no longer identical (you (!!!) might have overlayed their environment).

So what can you do (how to configure things?) to avoid such (frustrating) situations?

# Answer
I'll give my experience on this one, mostly because it showcases why some answers are not always applicable.

Some context to start:

* We have 7 environments to host roughly 80 applications, most of them rely on each others through webservices or shared tables on db2-iSeries.
* For good or bad, the iSeries are our DB system of reference.
* This last point invalidates any idea of bringing the app with its dependencies in an isolated environment as bringing up an AS400 for each would cost too much and we wouldn't have the hardware to run it anyway.

What we are doing is not a complete automated Continuous Delivery, we have a schedule of releases to bring up a coherent lot of applications for the general operations. Aside from this each test teams can trigger a release in one of the Q/A environment for the application they are testing and can put a lock on some application version to avoid another team request breaking their tests.

Dependencies of applications are checked before release, so the system won't release something if other applications can't be updated or doesn't match its dependencies needed. The main idea is to allow updates when it won't impact someone, if there are no tests planned, it should flow from previous environment (and we're aiming at removing the scheduled releases in the 5 firsts environments on mid term now we have validated this 'on demand' method system).

The short version is to have a 'semaphore' system around the applications in the environment, a team should be able to lock its target application with its dependencies (and transitive dependencies) for the time of manual tests.
The implementation of this semaphore is highly dependent on your automation system so I won't extend on that.

Of course the easy way is, as others mentioned, to create a fresh environment for an application with all its dependencies to avoid the semaphore described above.
> 10 votes

# Answer
Sounds like you're talking about a test environment which is constantly re-used without being **reliably** re-initialized for every test execution. This makes such test an unreliable one. Similar, from the reliability perspective, with manual testing, if you want.

IMHO you shouldn't be using such testing *inside* your CI/CD qualification purposes as that will effectively invalidate your qualification process (at least in that area). Saying that the software passes test X without actually executing test X for every software version delivered or without having the certainty that the `pass` result obtained is not accidental (due to false positives) will erode your testing's confidence level. False negatives are not credibility damaging, but they are also undesired because of the unnecessary "noise" they create.

It's fine to execute such testing *outside* your CI/CD qualification process. But you'd be treating a failed result in such testing just like a customer-found bug: you'd need to reliably reproduce the issue to be able to develop a fix for it and confirm that the fix is working. And you can't really do that if the testing is not reliable.

If you plan to address the issue then ideally you'd first develop an automated, reliable test case for reproducing the issue. Which you'd use to develop a fix and confirm its effectiveness (test result should transition from FAIL to PASS). You can (should?) also place this testcase inside your CI/CD qualification process to prevent future re-occurence, if desired - to increase your overall software release quality level.
> 8 votes

# Answer
The usual approach is to create different environments:

DEV - this is the place where dev team mess the things. Here are create all changes tunings, deploy new version and so on. Here is the place where CI is integrated fully.

PREPROD/QA - this is the place "play" QA/test/validation team do tests. This environment usually freeze during the tests. Integration of CI with this environment is only to provide new version of the product, configurations, etc.

PRODUCTION - is it need to explain :)?
> 7 votes

# Answer
To add to Romeo Ninov's answer, internally within an environment you need to try and separate out the applications as much as possible. This is partially why docker has been so successful for dev/test. It let's you almost pretend that you aren't sharing an environment at all.

The other option is to have very clearly defined servers on which the applications run which are separate from the rest of the infrastructure that makes up your environment. Ie. All the environment management or enablement machinery goes on separate, long lived servers. Then you hook in new short lived servers based on a known image to test an application and, if any changes are made to the base image, you need to apply those changes **everywhere** for every new component. Which means testing the changes against everything.

If an appdev team ask for a change that breaks someone else's application, then tough luck, they need to internally create a mitigant in their code and keep their specific requirements separate from the environments offering.
> 3 votes

# Answer
If you're doing CI/CD, that implies that there are some automated tests happening (CI) prior to deployment (CD). If you're finding a lot of issues in your test environment, that means they aren't being caught by the tests being run prior to deployment; this indicates insufficient automated testing. If the developers are having issues where defects are cropping up in the test environment(s), they need to improve their automated test suites in order to prevent this. This will also improve quality and reliability overall, all the way through into production.
> 3 votes

---
Tags: continuous-integration, testing
---