title: How to measure human utilisation?

text:
# Question
Title: How to measure human utilisation?
In The Phoenix Project the single graph in the entire book shows that as a person's workload increases into the high 90%s the wait time on them increases exponentially. In fact in the book it claims that:

Wait Time = (Percentage Busy/Percentage Free)

So if a resource is busy for 35 out of 40 hours per week then:

```
Wait Time = 0.875/0.125  = 8

```

However if a resource is busy for 39 out of 40 hours per week then:

```
Wait Time = 0.975/0.025  = 39

```

This is multiplied up by the number of waits in the workflow. It's obviously something we want to keep an eye on!

So, given all of this it's obviously vital that we keep people's utilisation at a sensible level. Given the book's insistence of the importance of this formula it offers little practical advice on how to measure these values. My question is, how can you practically measure a person's percentage busy?

# Answer
**TL;DR**: It is the **wrong thing** to measure. By measuring and increasing utilization in employees across the board, you **create problems** in the system and **reduce** overall **throughput**.

## Throughput Accounting

What you actually want to measure is throughput, inventory and operating expense all together and try to reduce inventory and reduce operating expenses while at the same time maximizing throughput. This method is known as throughput accounting.

In software development, inventory is the work in progress that is not bringing benefits to the customer yet. Anything that has been done, but not released. Throughput is the amount of work useful to the customer that is being released. Any work done that is not directly useful to the customer is accounted as operating expense.

## Simple system

In a **simple system** with *single human* or multiple humans working *independently* with independent equipment as much as each one does will directly increase the **throughput** of the whole **system**. This leads to the common **misconception** that is basis for this question that increasing human utilization will lead to increased throughput in all systems. But you still measure the **throughput** of the system, **inventory** and operating **expenses**.

## Complex System

In a complex system, even with as little as two dependencies, increased utilization of one part of the system can directly lead to decrease of utilization in the bottleneck, which decreases throughput of the whole system. Any increase in productivity outside of the bottleneck is a **mirage**.

*Example:* Team of software engineers has all code reviewed by the software architect, who also makes plans for new features. This person is a bottleneck, code not reviewed by the architect will simply increase inventory, if the architect does not have time, no new features will be properly planed. If you start measuring utilization of software engineers, they will each try to make more changes, rather than better changes. The time the architect will need to spend on each change will increase and the total time spent reviewing will increase further by the increased amount of changes to a point where no time would be left to plan new changes. Eventually the whole system grinds to a halt. If on the other hand they decrease utilization, even spend time idling, they spend longer on each change or peer reviews, this could lead to decrease in time required for reviews and eventually increase in throughput. This is just single team with 2 dependencies. Engineers depend on architect for new changes to be planed and for changes to be reviewed.

Clearly the benefits are to be gained in properly managing the bottleneck and trying to gain **productivity at the bottleneck**, where **hour gained**, is hour of **throughput of the entire system**.

This is the real message of The Phoenix Project and comes directly from the Theory of Constraints by Eliyahu M. Goldratt. You might also read an article on utilization thinking vs throughput thinking. I would also suggest to read more on Critical Chain Process Management.

*Remember:* What you measure is what you get. And you definitely **DO NOT WANT** to get increased individual utilization. A road to Hell is paved with good intentions.
> 7 votes

---
Tags: culture, systems, the-phoenix-project
---


# Question
Title: How to model Man, Machine, Measures, and Process in a DevOps world?
In The Phoenix Project when on one of the tours of the plant we're told that each workstation is a combination of Person, Machine, Measurement, and Process. This makes a lot of sense, after all we have people, servers, KPIs, and instructions.

However, whenever I model a process (the lifecycle of a Support Ticket for example) I struggle to take this into account.

My workflow states typically include:

* First Line Assistance
* Tech/Dev/More Technical Team Assistance
* Code Review
* Testing
* UAT
* Deployment

I can quite easily measure the cycle types, throughputs, and queue times of each of these states but I don't feel this does justice to the Man, Machine, Method concept. It's an idea which is frustratingly hinted at in the book but not expanded on...

We know that wait time is a function of utilisation so monitoring how busy people and servers (the finite resources) are is critical. Is there any defined process for expanding my measurements from a simple finite state machine to the Man, Machine, Method, Process idea in the book?

# Answer
What they are talking about is Kaizen 5M (Man, Machine, Material, Method, Measurement). It is approach to continuous improvement at every station in the process and the Ms are possible points of improvement and to which there is a corresponding question (5Qs). Sometimes Environment is added for 6th, like in this process that explains how to construct the questions using the Ishikawa diagram. These are pretty much essentials of the TPS / Lean Manufacturing. But the improvements are not in utilization, they are improvements in quality. You never strive for utilization as that is counterproductive to the throughput of the system.

It is important to understand that Man, Machine, Material, Method and Measurement are not easily separated. Sometimes Machine, Material and Measurement come on one side together and Man and Method on the other side. As you can replace a Man and a Method on that work station.

In terms of software development, you have Software (Machine), Issues (Material), Code Quality/Acceptance (Measurement), Man (Programmer) and Method (Development Process). The man trains on the machine and becomes familiar with it, with the material being worked on, understands the measurement required, learns the process. All those are living in the Man's brain and so are not easily separated once learned. Changing a Method is only possible if the Man has not fully internalized it yet, otherwise it gets easier to change Man and Method. Also Machine, Material and Measurement are often tied together through automation and configuration. That is why those are being drawn on opposite sides of the diagram.

If you read the book carefully, it does not really talk about utilization other than on the bottleneck of a value chain. In order to elevate and exploit the bottleneck. Several methods are employed for that in the book, including Kanban.

You don't want to optimize the individual stations of your process (Customer-\>Support-\>Development-\>Review-\>Testing-\>User Acceptance-\>Deployment-\>Customer), but you need to model the transitions between those work stations, their dependencies and to monitor Work In Process (WIP) moving through the system. Usually through an Issue Tracking Software (or Kanban system), which is equivalent to materials tracking in manufacturing. Where the WIP piles up in front of the work station in your critical chain process, you will find your bottleneck and that is the place where you want to optimize using Kaizan (5Ms,5Qs)

**Notice:** *I've added Customer both at the start and the end of your process, as each value chain has to start and end with a Customer otherwise it does not represent value to the company.*
> 7 votes

---
Tags: culture, metrics, systems, the-phoenix-project
---


title: What is an artifact (or artefact)?

text:
# Question
Title: What is an artifact (or artefact)?
The question about "What is an artifact repository?" contains an answer with an interesting explanation about the repository part of it. And from reading the entire answer, I am not sure what exactly an "*artifact*" means in the context of DevOps.

Any suggestions?

Ps: From one of the answers I seem to understand that maybe **artefact** is what I'm wondering (confused?) about ...

# Answer
Wikipedia has a very good answer to this question. Artifact, sometimes also called Derived Object, is a product of some *process applied* to the **Code Repository**. Originally they were called **Build Artifacts**, but as more processes were applied other than build to create them, the first word was simply dropped.

The major distinction is that artifacts can be recreated from the code repository using the same process, providing you have preserved the environment in which the process was applied. As this process can be time consuming and the environment can be preserved imperfectly to be able to recreate the artifacts in the exact same way, we started to store them in **Artifact Repositories**.

Storing them apart from **Code Repository** in an **Artifact Repository** is a design decision a DevOps engineer would make. Some companies, namely Perforce, suggest to use their Code Repository as Artifact Repository as well. There are different requirements in terms of *access*, *auditing*, *object sizes*, *object tagging* and *scalability* on each repository and so depending on situation it is often better to use two distinct products. For example Git repositories are copied in their entirety to every development machine and so storing artifacts in the code repository would increase its size beyond all reason, although lately there are ways to mitigate this. Another decision to make is which artifacts to store. Some companies store even intermediate artifacts as individual object files, to speed re-builds, others store simply just the final binaries. Not all artifacts have the same value. Artifacts resulting from a release build could have different requirements than artifacts resulting from a developer build.

Most common artifacts are results of the following processes: **Configuration**, **Preprocessing**, **Compilation**, **Linking**, **Automated Testing**, **Archiving**, **Packaging**, **Media files creation and processing**, **Data File Generation**, **Documentation Parsing**, **Code analyzing**, **QA**, etc.
> 29 votes

# Answer
There are two usages of the word “artefact” and one makes source code an artefact while the second makes it not being an artefact: this can indeed be quite confusing!

**“artefact” as a concrete thing, vs. an ideal thing** – This meaning is the common meaning of the word “an object made by a human being, typically one of cultural or historical interest” and is not technical jargon. Here is an example in technical context: *When you debug a software, you learn something about the software. It is often a valuable investment to turn this learning into a software artefact, like a regression test. Otherwise this learning will be forgotten and the the effort made to acquire it will be wasted.* In this meaning, source code is considered an artefact.

**“artefact” as something produced by a recipe** – This meaning uses the popular image of the alchemist using some esoteric recipe to produce a magical device, often called an artefact. It is technical jargon used to distinguish between the source code, which corresponds to the recipe in the alchemist's metaphor, and anything derived from that source code, which corresponds to an artefact in the alchemist's metaphor. For instance *I just automatised the artefact production for my plop-fizz program, now source tarballs, signature files, DEB and RPM packages can all be instantiated in just one command!* This meaning does not recognise source code as an artefact, as the term is used to denote what is produced from this source code.
> 8 votes

# Answer
I suppose the answer may vary from place to place. Where I work at the moment an artifact is anything consumed by some other entity, except for the source code used for development - this goes into source control.

This includes binaries of the product or other needed products, libraries, object files, test artifacts like media files or test data.

Source code is not considered an artifact. Unless it matches the definition of "consumed by" - in our case including third party libraries, script code used for testing or other purposes (but not the development version itself).
> 3 votes

# Answer
Side note on the culture side. While in DevOps we consider the concept of "artifact repository" as a given situation, there seems to be not so much linkage to the organizational process.

Culture problem: if an organization uses ITIL, the certified people would say "we need to have a definitve media library, such a repository to place software configuration items we have produced". So people who care about well structured IT processes do not know which (not-management) tools support that and are in use. Vice versa, if you need a justification for a Nexus or Artifactory language, you might have hard times to explain it depending on the organization.

Further reading: https://en.wikipedia.org/wiki/Definitive\_Media\_Library
> 1 votes

---
Tags: terminology, artifacts
---


title: What is Serverless?

text:
# Question
Title: What is Serverless?
All the cloud providers are marketing their "serverless" solutions. The promise is that serverless is going to replace the way developers are currently develop their software, and operations manage it in production.

What is "serverless"? Where can one learn more about it, and how it can be used today?

# Answer
> 46 votes
Wikipedia's article on serverless computing provides a decent introduction to the topic:

> Serverless computing, also known as function as a service (FaaS), is a cloud computing code execution model in which the cloud provider fully manages starting and stopping of a function's container platform as a service (PaaS) as necessary to serve requests, and requests are billed by an abstract measure of the resources required to satisfy the request, rather than per virtual machine, per hour.

The idea is that a developer **shouldn't need to care about the server infrastructure at all**. The cloud provider manages the physical servers, the operating system used and all the traditional difficulties involved with running a server.

Serverless computing changes your architecture from thinking about what **machines** are doing to what **functions** are doing. AWS Lambda is the example that comes to mind - you pay for and run **functions**, without any mention about what type of physical infrastructure is running below. There are also competing serverless hosts such as Azure Functions (or you can simply search if you're not interested in either of those).

There are quite a few advantages to serverless (although you do need to write in a slightly different way than you're used to in some cases, because it's a totally different architecture):

* Scalability essentially comes for free - because you're just paying to run a function, the cloud provider can easily dedicate more hardware as needed to run your code. You can also potentially scale as demand rises, rather than paying a fixed rate whether your application is used once or a million times.
* The server software and hardware no longer needs to be managed by a developer - the cloud provider handles that. If you've ever used something like Arch on a server, you'll know how easy it is to wipe out a critical package and break everything!
* It frees up developers to focus on what they're good at - **code**. Most developers probably won't be great at both server infrastructure **and** programming - serverless just takes one problem out of the equation.

# Answer
> 16 votes
Martin Fowler has a good overview:
https://martinfowler.com/articles/serverless.html

**TL;DR**
"Serverless" describes application development and architecture that designs applications in which the infrastructure is *ephemeral*, meaning that they tend to be container-based and can "come and go" based on a dynamic scaling mechanism. This prescribes the use of stateless architectures that depend on distributed technologies (such as key/value stores.)

# Answer
> 14 votes
'Serverless', like many things in our space, is becoming an overloaded term.. but generally what it means is "Functionally, Our architecture does not depend on the provisioning or ongoing maintenance of a server"

The first instance that comes to mind is a single page javascript app, that uses local storage, and is stored on something like Amazon S# or Github Pages (or any static site - those are just common examples). Imagine something like a 'todo' or 'getting things done'-style application that runs entirely in your browser. Your browser hits a service like S3 to download the code, and the items you store are all stored in local storage in your browser. There is no server you maintain for this.

The second instance, and is a bit more complicated (and also the one that popularized the term 'serverless'), uses a service like AWS Lambda. Let me explain this by presenting the problem it solves:

Many times in my career I've solved a business problem for a client with little more than some ruby code that performed a periodic extract, transform, and load (typically written as a rake task). Once solved, I'd typically automate it with cron. Then the problem becomes 'where do I host this thing that runs once every hour?' For some clients, we'd set up a server in their existing infrastructure. For others, we'd set up an EC2 instance, even though it was idle 99% of the time. In either of those circumstances, there is a server that requires provisioning, patching, monitoring, updating, etc.

With Amazon Lambda, I can take that rake task and run it on their service as a pure 'function'. I can even schedule it. No longer would that client need a piece of infrastructure for such a simple once-an-hour thing.

With 'serverless' there is still a server, just like with 'cloud' there is still a computer. There is just a level of abstraction on top of it that takes some of the environmental responsibilities for you.

# Answer
> 12 votes
All great answers already. I was going through the blog post Thinking Serverless — How New Approaches Address Modern Data Processing Needs in HighScalability, where I've come across this excellent explanation of what serverless means:

> The phrase “serverless” doesn’t mean servers are no longer involved. It simply means that developers no longer have to think that much about them. Computing resources get used as services without having to manage around physical capacities or limits. Service providers increasingly take on the responsibility of managing servers, data stores and other infrastructure resources…Going serverless lets developers shift their focus from the server level to the task level. Serverless solutions let developers focus on what their application or system needs to do by taking away the complexity of the backend infrastructure.

And, one more learning from my personal experience in building serverless infrastructure is:

* Serverless data pipelines don't really need to be real-time only. One can build efficient batch-processing serverless pipelines via AWS's CloudWatch alert mechanisms, which allows one to monitor the tools and raise alerts for (AWS)Lambda to compute.

Like for example: Store the files in an AWS SQS queue. Once the number of messages in the queue reaches 10, fire an event to Lambda.

# Answer
> 7 votes
Beside simply explaining the definition of the term Serverless, the origins of the term and it’s history give some insight as well into its meaning. The concept originated with the JAWS framework by Austen Collins, which got later renamed as Serverless. I've learned of it first in AWS re:Invent 2015 session, which can be helped to date the concept. This is what actually coined the term Serverless, which then spread quickly to other cloud systems:

---
Tags: terminology, serverless
---


# Question
Title: How to restrict filesystem access in Atlassian Bamboo builds?
We have Atlassian Bamboo running on Ubuntu. When a developer is setting up a build, then he or she has the possibility to run shell script tasks. This is useful to run (custom) commands on the codebase you are building.

However, the scripts that run can also access the filesystem outside their job directory in the Bamboo working directory (`<Bamboo-home-dir>/xml-data/build-dir/JOB_KEY`). So JOB\_A can also access files of JOB\_B: `cd ../JOB_B`.

Is there a possibility to restrict this access?

P.S. I am aware of the fact that builds are ran by (local or remote) agents in Bamboo and you can build different projects by different agents. However, if two projects are built by the same agent, the projects could access each other's files.

# Answer
> 9 votes
There is currently no ability to restrict jobs which are able to run on the same agent from potentially interacting with each other. There are a bunch of feature requests asking for this sort of granularity, but if I understand your question correctly the most fitting request would be this one BAM-2504 Jira Ticket

It's a huge gap in the product line, the only solution I've found is similar to what is proposed by the request linked above, essentially your bamboo process would need to run with sufficient privileges to impersonate a set of users who represent the projects which you would like to keep separate.

Once you have this sort of mechanism in place you just need to try to enforce that all plans run as one of the impersonatable accounts, depending for example on the project or the creator etc.

Problematically the way that access controls currently exist this would mean that few core admins would need to set up all the plans so that they can he sure of the required permissions split rather than letting non-administrative users edit and create their own plans.

If this sort of approach isnt feasible, which it isn't once you get into the "many hundreds of users" type of range, then the only thing you can really do to try and stop build jobs from interacting with each other is implement the very weak controls that bamboo gives you.

I've tried two approaches to doing this:

1. Delete or cripple (remove all capabilities from) your local agents and for every different project/team/whatever that on boards to your bamboo instance you need to force them to BYO build server. In most instances I've been involved in the cost of an agent is utterly trivial compared to the cost of potential data leakage or malicious plan interactions.
2. Make sure that projects that do have, or that think they do have, sensitive data involved in their plans always sanitise their environments after they build. This shifts the burden off the team administering the tools and onto the projects writing their plans and forces them to defensively clean up any information that they don't want potentially available to others.

Neither solution is even close to perfect but to the best of my knowledge that's about as much separation as you can enforce if you have a shared bamboo instance.

---
Tags: builds, bamboo, permissions
---


# Question
Title: How to implement a frozen test environment?
Here is a partial quote from an answer to the question about "How to avoid continuous integration-caused instabilities in test environments?":

> This environment usually freezes during the tests.

My question: what are sample implementations of a frozen environment? I.e. what can you do to technically enforce that nobody (except if allowed by an authorized user like a release manager) will be able to change anything in such frozen environment.

**Clarifications**:

* I'm not talking about what (I think) is called "frozen periods" during (eg) year-end processing in banks. That is about not being allowed to apply any (repeat any) changes to production environments, to reduce the risk of new changes/fixes being introduced that may impact the year-end processing.
* Assume that users who are allowed to approve/apply changes anyway (such as the release manager in my example), will only do so in exceptional cases. Such as where during testing a high severity issue is encountered, for which deferring a fix to a next release is not an option (since it would production at risk if the release would be activated without such fix).
* This could just be about suspending any automated update during the time of the test. The point is: avoid someone else upgrading an Application A to version Y while another team is still testing application B in version X which rely on application A. This could mean having a guard to avoid a test team to require an update on a dependency under test.

# Answer
> 3 votes
My answer in the context of the question is a very expensive test environment (like mainframes or very large telecom equipment, for example) which are expected to be shared by multiple users for multiple tests, even simultaneously.

In many cases such equipment has a software management system responsible for, among other things, controlling software (un)installation/upgrades/downgrades/etc. Which should have some mechanism for blocking such operations (which would be, if I understood the question correctly, equivalent to freezing the environment) based on some more or less programmable policies.

In such case a specific policy could be developed exactly to support the desired freezing schedule necessary for the test environment. Ideally automated, accepting freezing/thawing triggers from the external sources which could be the test execution wrappers or CI systems. And maybe with human-triggered overrides, if deemed necessary.

Of course, such feature of the software management system would be helpful when testing other components of the equipment, but maybe not for tests of the software management system itself.

# Answer
> 4 votes
TeamCity has a Shared Resources build feature which allows you to define a resource which multiple Build Definitions depend upon it. Build Definitions can either require a Read Lock or a Write Lock, you can also define if these locks are exclusive or allow a degree of parallelism.

If we make the following assumptions about a shared environment named *PreProd*:

* A Shared Resource exists named "PreProd".
* All build definitions, such as deployments, that make any change to that environment take an exclusive **Write Lock** on "PreProd".
* All build definitions, such as non-restrictive tests, that perform read-only operations on the environment take a non-exclusive **Read Lock** on "PreProd".
* TeamCity is the only process that can do anything on PreProd, albeit perhaps via another tool.

Therefore the following are true:

* When a deployment is happening then nothing else can use PreProd, they will be queued.
* When a test is running, any deployment would be queued until the tests complete.

You can use a similar mechanism with Jenkins using the Exclusion plugin. In fact, you could build this functionality into any process using locking or semaphore - for example Apache ZooKeeper or HashiCorp Consul.

# Answer
> 0 votes
This sounds like an anti pattern to me. I believe everybody or nobody should have access to all the environments.

If users are subverting the process then I would take a serious look at the process to try and ensure it's not getting in people's way.

Implementing an automated mechanism that enforces a specific state is also useful for encouraging people to do things the right way. This can be via Config Management or destroying any immutable instance if someone SSHs to it

---
Tags: automation, testing, configuration
---


title: What is "Infrastructure as Code"?

text:
# Question
Title: What is "Infrastructure as Code"?
The phrase "Infrastructure as Code" has been mentioned several times in the last two weeks in different contexts. What does it actually mean in a practical sense to have Infrastructure as Code?

# Answer
> 12 votes
**TL;DR**: Infrastructure as Code is a way to automate and backup your environment. In ideal case, after a disaster, you *could restore* your Infrastructure *fully and automatically* by Provisioning new resources, Restoring Configuration from Code Repository and Recovering Data from Backup.

## Overview

Infrastructure as Code relies on three main concepts:

### Automation

Configuration Management is in its 3rd generation of tools. Building on CFEngine a new set of tools for Automated Configuration Management is being widely deployed now. The most popular in alphabetical order are *Ansible, CFEngine, Chef, Puppet, PowerShell DSC and SaltStack*. Each will have a **Language** to describe state of your infrastructure, **Code Modules** to apply those changes and provide ability to extend the tools, some **Agent** to execute those on the servers and a **Central Repository** of information.

They will generally operate in push or pull mode, either connecting to servers from a central location(s) and executing changes remotely or running on each server and pulling information about state from central location and that either in client/server model or in a distributed way.

The **important concept** is for the system administrator or site reliability engineer to **not make changes directly** to the infrastructure, but let the automation do changes. Anything **done manually by a human** should be considered either perishable, being soon corrected back by automation or in stricter form **violating the integrity** of the infrastructure and triggering destruction and rebuild of the affected components.

### Code Repository

Code Repository, ideally separate from Repository holding Software, would be used to manage all changes to the Infrastructure and related Automation. It should hold Configuration files and templates, Playbooks (Cookbooks) describing process of changes to be reviewed, Code extending the CM automation tools, Provisioning configurations, Infrastructure Tests and Alerts, Staging/Deployment Tests, Documentation, Manual (not yet automated) Process Descriptions.

The important concept is to institute peer reviews for changes, to have record of all changes and ability to automatically revert to previous state in case of unpredictable and/or untested issues, ability to deploy to staging environment and test configuration changes and ability to automatically deploy changes without variation caused by human error.

### Managed Infrastructure

Managing Physical Infrastructure is a real world task that goes beyond software and requires very different set of skills. By being able to abstract this layer through Cloud Computing or a Managed Datacenter, you have your team focus on the part of managing Infrastructure that adds business value.

While Cloud Computing offers a way to start and scale quickly at a later stage companies often realize some benefits and even significant savings in moving parts of the Infrastructure in their own data centers for a hybrid model. Owning or renting the hardware does not mean that you also have to employ the people who handle it. At this scale you need data centers geographically distributed around the world and having people with all the required skills in all places would be very expensive. Flying them around the world adds high latency to any changes and additional level of operational inefficiencies, which is another reason for outsourcing the datacenter management.

The important takeaway is that **Managed Physical Infrastructure** is often forgotten or **overlooked concept**, but just as important. Even if you've got everything automated, all configuration stored in a backed up code repository, unless you have a way to **quickly provision**, you have a **huge bottleneck**, which could easily **erase all the benefits** you've gained by the other two steps.

# Answer
> 6 votes
Before explaining what it exactly is, let me quote a really nice definition, straight from Wikipedia:

> Infrastructure as Code (IaC) is the process of managing and provisioning computing infrastructure (processes, bare-metal servers, virtual servers, etc.) and their configuration through machine-processable definition files, rather than physical hardware configuration or the use of interactive configuration tools.

Okay, now let's look at one such IaC tool, Terraform to understand the concept better:

Also, this is what Terraform say about itself:

> Terraform enables you to safely and predictably create, change, and improve production infrastructure. It is an open source tool that codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

This means, one can code the entire infra. which includes creation of cloud(/infra) resources like the server instances, load balancers, etc, along with the complete configurations (which includes the basic settings tweaks, security settings, regions, etc) as code, which can be editable, versionable and of course, reviewable.

This is a sample example of Terraform code for provisioning AWS resources:

```
resource "aws_elb" "frontend" {
  name = "frontend-load-balancer"
  listener {
    instance_port     = 8000
    instance_protocol = "http"
    lb_port           = 80
    lb_protocol       = "http"
  }

  instances = ["${aws_instance.app.*.id}"]
}

resource "aws_instance" "app" {
  count = 5

  ami           = "ami-408c7f28"
  instance_type = "t1.micro"
}

```

**Bonus PS**: Also, one needs to understand the differences between provisioning and orchestration tools. Devs confuse one for the other very often and tend to make the mistake of trying to tweak and use a tool for what it is not intended to be used for.

---
Tags: terminology, culture, infrastructure-as-code
---


# Question
Title: Is it possible to autodeploy on every commit in a specific branch using Travis CI and GitHub?
I'd like to deploy files using Travis CI and it only works for a tagged commit. When committing to the branch, there is a warning:

> Skipping a deployment with the releases provider because this is not a tagged commit.

Is there any way to deploy on branched commit using Travis CI?

To clarify, it works when I tag the commit, but I'd like to deploy files on each commit for the given branch.

# Answer
> 4 votes
Travis CI supports deployment on branch release by using the following syntax:

```
deploy:
  on:
    branch: release

```

The problem is that GitHub does not support it.

As per GitHub Releases Uploading page at Travis CI:

> Please note that deploying GitHub Releases works only for tags, not for branches.

For GitHub the only workaround is to push tags (e.g. `git push origin master --tags`).

Or solution is to use different supported provider.

# Answer
> 6 votes
Instead of using Travis' deployment features, build the deployment logic in to your normal "test" script(s). For a PaaS this might be as simple as pushing to another git remote if all tests pass, but the sky is the limit.

---
Tags: git, release, deployment, travis-ci, github
---


# Question
Title: Which university has a good postgraduate DevOps program?
I am looking for a university that has a good PhD program in theoretical DevOps topics. Especially any program that is trying to advance the beyond Deming's Theory of Profound Knowledge, Goldratt's Theory of Constraints and other related topics. Do you know a professor specifically interested in running a program like that and currently doing advanced research in this field?

I have found some companies, which do consulting in this area and try to push it forward, but I am having problems to find some program in academic setting with sufficient credibility.

# Answer
> 4 votes
I'm not really familiar with the topics you mentioned. However I went ahead to go checkout kuleuven.be, i.e its Computer Science departement (with sufficient credibility, trust me).

Here are some interesting articles in which DevOps is mentioned:

* Automated workflow regression testing for multi-tenant SaaS: Integrated support in self-service configuration dashboard.

Here is part of the abstract of it:

> ... This effectively makes these tenant administrators part of the **DevOps** team, and in turn creates the need for the SaaS provider to provide them with Quality Assurance tool support. One such tool is a regression testing framework that allows them to make sure that a new version of a workflow can behave similarly as to a successful execution of a previous version. ...
* Variability middleware for multi-tenant SaaS applications.

Here is part of the abstract of it:

> ... The focus has been mainly on (1) the pre-deployment phases of the development life cycle and (2) fine-grained (tenant-level), run-time activation of specific variants. However, with upcoming trends such as **DevOps** and continuous delivery and deployment, operational aspects become increasingly important. ...

Both articles have links with contact info to their authors.

---
Tags: learning
---


# Question
Title: How to set up AWS Trusted Advisor alert for Service Limit warnings?
All I want in life is for AWS to shoot me an email when any of Trusted Advisor's Service Limits go to WARN or ERROR status. I've checked their guides, tried chatting with support and no one can get that to happen, so I'm reaching out here.

# Use Case

We spin up a lot of EC2 instances and CloudFormation stacks and everything uses lots of resources. Somehow we always hit a limit we didn't know about in the middle of a validation sprint and end up blocked for an hour while AWS takes their sweet time bumping our limits. I'd like to set up some kind of event to **email devops when one of the Service Limits is close to being reached**. I am alright with it checking as infrequently as once a week, but would prefer once a day if that kind of granularity is possible.

### Attempts

I tried setting up a CloudWatch Event, but the only option I could get was to email us when the Service Limits were *refreshed* \- which is useless, because then you're already in there manually looking at it and there's no point in an email.

### Options

Support told me there's no default functionality for this inside of AWS, so I'm looking for either custom event options, or I will also accept something leveraging Jenkins2 and the AWS CLI if that is a possible route. If Jenkins is the way to go it doesn't need to send an email - I'd probably have it slack our devops channel, if that changes the answer any.

# Answer
That's on my backlog but still on early stage, I just surfaced the idea, so here's are my findings so far:

Using Trusted Advisor as a Web Service

Mainly: Poll a Trusted Advisor Check for Status Changes

This documentation example can be implemented with aws cli with the `support` subset of commands and targetting the 'us-east-1' region, you'll then have to filter on the `region` field in the `TrustedAdvisorResourceDetail` structure part of the list.

I still didn't had time to write a script to poll the API (I would aim at ruby with the AWS SDK per our usage more than a shell script to group the requests).

Hope this help getting started even if that's not really a complete answer.

Courtesy of Romeo Ninov: For a recap of which options is available for which account type you can check this site
> 7 votes

---
Tags: amazon-web-services, amazon-cloudwatch, aws-cli
---


# Question
Title: What are further known DevOps operation models?
It seems that many people somehow try to define the software development process using a DevOps tools model but there is no alignment; still I assume that this is more a question of maturity/alignment, not of matter of taste.

**Question**: What are further known DevOps operation models?

Some examples I've seen at available popular sources:

* Plan, Develop, Test, Release, Operate
* Plan, Code, Build, Test, Release, Deploy, Operate, Monitor
* Requirements, Development, Build, Testing, Deployment, Execution

Here is a quote from the Wikipedia page for the DevOps toolchain:

> A **DevOps toolchain** is a set or combination of tools that aid in the delivery, development, and management of applications throughout the software development lifecycle, as coordinated by an organisation that uses DevOps practices.
>
> Generally, DevOps tools fit into one or more activities, which supports specific DevOps initiatives: Plan, Create, Verify, Preprod, Release, Configure, and Monitor.
>
> Another proposed toolchain model was referenced in 2014 as coming from "an unknown source" as Plan, Code, Build, Test, Release, Deploy, Operate and Monitor.

However, that same Wikipedia page also contains an image which looks like so:

This image, IMO, does not correlate with the text on that page.

*DevOps Toolchain stages on Wikipedia, By Kharnagy - Own work, CC BY-SA 4.0Wikipedia*

Sources: https://devops.com/devops-2017-cdos-care/

# Answer
> 2 votes
Your question doesn't seem to make any assumption about the platform/OS it is about. Which is why it may make sense to add an answer about how part of such toolchain looks like in a mainframe environment. My answer is based on using the SCM product where I'm most familiar with (not sure if it's needed to disclose the product name).

Your example about "*Plan, Code, Build, Test, Release, Deploy, Operate, Monitor*" seems to be missing these crucial phases:

1. Enforce predictable problems in production to be addressed before they actually have an impact in production (sounds like sales talk, no?). You might think this is just "impact analysis", but that is not it:

* **Impact analysis** is what you do in the plan-phase (we all know that, right?).
    * **Audit** (= name of the phase we use for it), is around the time that the developer thinks/claims to be ready. Something like "unless there is anything else that may have happened in production while I was working on my change, I'm done and recommend to move forward". Think of things like "version regression issues". Eg because last night a fix was applied to production and you didn't retrofit that fix, so if you move forward with your change, you'll wipe out the fix again ... bad luck if you do!
2. Give me a red button, so that if anything else fails (and production is impacted), I just have to push a magic button, be patient (for a few secs) and be 100% sure a rollback completed with 0 rebuilds of any kind (= **Backout** is the name of the phase we use for it).

This boils down to "component level backups" of any files/components you will be updating by your change in production, whereas these backups are made at the very moment the change is going to be applied. and this for any type of artifacts included in the change.

If later on (5 mins, 2 days or 1 week later) something goes wrong in production, and it turns out to be caused by your change, all you have to do is to have an automated procedure (the red button!) in place that you can launch, and which simply restores the backup you should have available.

Attention, if there were database updates involved, this backout may not be an option anymore. While things like "backups overlaid" (because of concurrent changes/fixes got applied to production) is another scenario where the red button is no longer available.

**Easy, no**?

PS: I leave it to everybody's own judgement if my answer is yes or no DevOps-compliant.

# Answer
> 2 votes
I don't think the image you mentioned is really in contradiction with the corresponding text. From the Preprod section further down on the same page (emphasis mine):

> **Preprod**
>
> Preproduction or "preprod" refers the activities involved once the release is ready for deployment, often also referred to as staging or **packaging**.

---
Tags: terminology, culture, toolchain
---


# Question
Title: How to publish Jenkins job icons externally?
I'm using Promoted Builds Plugin to assign certain icons to Jenkins jobs on certain conditions and I'd like to publish these promotion icons.

How can I generate external links to these icons per each job, so they're accessible externally?

—————

For example I'd like to create the table in `README.md` on GitHub like:

```
Job 1 | ![](link_to_icon_of_job_1)
Job 2 | ![](link_to_icon_of_job_2)
Job 3 | ![](link_to_icon_of_job_3)

```

# Answer
> 11 votes
The "Embeddable Build Status Plugin" will do exactly that.

Note: This requires your Jenkins server to be accessible from the internet if you're using GitHub because they cache all images in their CDN.

# Answer
> 4 votes
I was trying to do something similar (embed build status icons into email notifications) but since our build system is not publicly accessible, I ended up posting the status icons to imgur and referencing those images by storing the URL in variables according to state.

Here are some snippets from my email template Groovyscript:

**Defining the URLs:**

```
def images = [:]
images["SUCCESS"] = "http://i.imgur.com/uXlqCxW.gif"
images["PASSED"] = "http://i.imgur.com/uXlqCxW.gif"
images["UNSTABLE"] = "http://i.imgur.com/QkQbxR3.gif"
images["SKIPPED"] = "http://i.imgur.com/QkQbxR3.gif"
images["FAILURE"] = "http://i.imgur.com/LUveOg7.gif"
images["FAILED"] = "http://i.imgur.com/LUveOg7.gif"
images["ABORTED"] = "http://i.imgur.com/jSdrWWP.gif"
images["NOT_RUN"] = "http://i.imgur.com/jSdrWWP.gif"

```

**Using the URLs in the email:**

```
<img src="${images[build.result.toString()]}" />

```

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: How to automate submission to Amazon Marketplace?
Is anyone aware of a way to automate the submission of an AMI to Amazon's Marketplace? I want to take a public AMI and share it to the AWS Marketplace so it can be scanned for listing and then submit the product metadata sheet.

Our preferred environment is Python, but I didn't see any relevant hooks in the `boto3` library so I'm open to alternatives.

# Answer
# Snapshot Permissions

Boto3 has a function that allows you to create volume permissions, which is what AMI Sharing with AWS Marketplace requires you to do. `snapshot.modify_attribute` will allow you to share your AMI with the marketplace account like so (you can also use a JSON representation if you prefer, it's in the docs):

```
response = snapshot.modify_attribute(
    Attribute = 'createVolumePermission',
    OperationType = 'add',
    UserIds = [
        '679593333241', # Marketplace user ID
    ]
)

```

There is also an official API function that will help you achieve the permissions requirement listed in AMI Sharing with AWS Marketplace if you decide to migrate from boto3 (or if a reader isn't using boto3). If your AMI is private, you need only make this API call:

```
https://ec2.amazonaws.com/?Action=ModifySnapshotAttribute
&SnapshotId=ID_HERE
&CreateVolumePermission.Add.1.UserId=679593333241
&AUTHPARAMS

```

This should add the permission that the marketplace needs to access your AMI. As for replacing `ID_HERE` with the snapshot's ID, you can do that with boto3 by using your snapshot object's snapshot\_id property `snapshot.snapshot_id` before making the API call (that being said, there's little reason for you to use this API since you have to use boto3 anyways).

# Product Metadata

You should be able to use the relevant ServiceCatalog functions in boto3 such as `create_product` to upload product metadata. There are also equivalent official API functions (this one is just CreateProduct, they're not listed in a way that allows me to link to them nicely). It's worth noting that the official API only accepts JSON whereas boto3 will generate the JSON itself. Which one is more useful to you depends on how your data is structured in your product metadata file(s). Since I don't know that I can't directly write the code to do it.
> 6 votes

---
Tags: amazon-web-services, continuous-deployment, automation, amazon-ec2
---


# Question
Title: How does 'Environment Management' work in a DevOps world?
I've had some very interesting conversations today about the role of Environment Management within teams that follow DevOps practices. The traditional role of an *Environment Manager* was someone who:

> Took overall responsibility for coordinating multiple teams and components at all stages of the software development life cycle, SDLC, to ensure that a known sets of software versions are delivered and available in a timely manner.

It is my belief that in a cross-functional Agile team following DevOps practices these responsibilities belong to the team as a whole, ultimately being accountable to the Product Owner as the representative of the business.

**In short, my question is** who is... responsible for Environment Management in a team following DevOps practices, i.e., one that does not have an external Environment Manager function available to them.

# Answer
> 3 votes
I've never heard of an "Environment Manager." Release Management, on the other hand, has historically been consolidated to a single person or team.

In a DevOps model, Release Management is more of a process supported by elements from both Dev and Ops. Organizations at high levels of "DevOps maturity" don't really have a distinct Release Management *event*\- the act of releasing is baked into the Continuous Delivery pipeline (see "push on green" What is "Push on Green"?)

It's possible the OP is also referring to *Configuration Management*, where environment-specific configurations are managed manually, or by tools like Puppet or Chef. That work is traditionally an Ops-managed task, with higher-maturity orgs handling that through code. Responsibility may be shared between Dev and Ops, or there may be a dedicated SRE team that deals with Config Management.

# Answer
> 1 votes
I see three way to handle this need.

1. The software declare its dependencies and is released with each dependency at the correct version independently of others. This may create problems if two version of the same dependency are not compatible for database access for example and need a system above to ensure only compatibles version are deployed toward an environment, the way to achieve it may vary depending on the tooling and is quite hard to get it right.
2. The software declare its dependencies, and if current available version of one dependency on the target environment doesn't match, the deploy stop and keep actual version the software. The team has then to coordinate with the dependency team to know when they'll be able to deploy. Main drawback is the delay this can bring in the release train. Here again the way to achieve it may vary depending on the tooling.
3. Cross teams made from members of each team, to handle the coordination of versions toward release. This help getting things at the right time but doesn't enforce any control.

Best setup in my opinion is 2+3 above, to handle any problem of missed dependency while enforcing a communication between separate teams and allowing releases compatible with multiple version of the dependencies. this need to keep trace of this kind of behavior change to remove the unneeded old API handling once it is not on line anymore.

---
Tags: culture, agile, environments
---


# Question
Title: What are possible implementations (or examples) of the four-eyes principle?
Michael Grünewald recently posted this comment:

> A very important method you do not mention is the “four-eyes principle” which is used in the finance – either as a regulatory obligation or as a safe-guard. In the software industry it is implemented in various ways, as e.g. code reviews but also can be used to validate commands affecting live systems.

Correct me if I'm wrong, but I was taught that the **"four-eyes principle"** is about something that is "approved to happen", after at least 2 human beings (and/or automated processes) gave their prior blessing. Or to use the (slightly corrected) wording about the **"two-(wo)man rule"** from Wikipedia:

> The two-man rule is a control mechanism designed to achieve a high level of security for especially critical material or operations. Under this rule all access and actions requires the presence of two authorized people at all times.

Regulatory obligations are, pretty sure, off topic here, but in the context of "safe-guard", what are the possible conceptual implementations of this four-eyes principle, which probably could apply to any platform / OS / hardware being used?

# Answer
One of the implementation on code is the Pull Request model (PR) popularized by GitHub.

The main reasoning behind is that only a small set of maintainers of the product will be allowed to merge code into the release branch. Every new feature/bugfix will happen on a new branch and once done will be defined as a pull request.

This allows to test on the merge from actual release (master) code with the code in the PR in an automatic way (Travis being the most popular for public project actually) and give a first feed back on the code quality. Travis CI (for example) can run on the result of actual master with the code from the pull request merged into it, so it will fail if the merge is impossible, or if the commands defined in the travisci.yml return a non zero exit code

Once the automatic tests have passed, for the 4 eyes principle, it still requires a configurable number of persons to review and approve the change before it gets merged obviously at least 1 person (who is not the PR author) to enforce 4 eyes reviewing the changes.

There's a wide range of options to merge automatically once the quorum of reviewers is met or to still need a manual merge by a maintainer.

The permissions to review and merge can be separated which helps giving more people the right to "vote" on the merge status while keeping the possibility to restrict who can actually do the merge.
> 11 votes

# Answer
### Code Reviews

This is about having at least 1 other person look at the code written by somebody, eg to evaluate if it meets some predefined criteria like:

* Coding standards (indentations, etc).
* Inline documentation.
* Maintainability of the code.
* Error handling.
* Completeness (eg `if/then/else` or `case/when` constructs cover all possible cases).

### Approvals to update some target environments

This is about having at least 2 confirmations from some person and/or automated system before it is allowed to update some target environment (which may be live, or may be something like some master file / baseline library). Some examples are:

* Only a limited set of warnings are allowed when transforming (building) source components in executable components.
* Some set of automated tests must have completed without any probems.
* Some human beings must have indicated their prior approval (and without that, any attempts to update the target environments will automatically fail).
> 9 votes

# Answer
These are strategies/patterns I can think of:

### Separation of Duty

DevOps, to my view at least, does not means embodying of both dev and ops in a single person. So it is still possible to separate the duty such that the one writing the code (dev) is not the one executing it (ops).

For example, if an SQL statement is to be executed on the live environment, one writes the SQL and another executes it. What this presupposed is the the one executing need to also have an understanding of the SQL and not just execute.

### Deploy trigger

While there are merit to continuously deploy. Team in a more regulated industry can appoint another (separate) party to trigger the deploy instead of deploying automatically. Checklist, automated tests, checksums are possible checks before triggering deploy.

Once triggered, the automation can go ahead to execute the deployment.

### Pair Programming

Personally I have not cited this technique as a method to auditor to satisfy the check and balance principle. But potentially I think it can be a strategy.

### MFA

I might be stretching a little with this one, but it is possible that for some reason that you don't want unilateral entry into a system, someone can hold the password and another person hold the token or device for one time code. So that, 2 person must be present to assess the system.
> 6 votes

---
Tags: configuration, four-eyes
---


# Question
Title: How can successfully pre-verified changes cause regressions that should have been caught?
In a CI context one of the commonly-used measures of increasing the quality levels of the integration branch is a mandatory set of pre-commit quality verifications (typically including building some artifacts, performing unit tests and even some feature/integration tests).

Yet some regressions (build breakages, various test failures) are detected by the CI system verifications in exactly the areas which were supposed to be covered by these mandatory pre-commit verifications.

During analysis of these regressions an argument often heard is that the developer who committed the change identified as root-cause of the regression has successfully passed all such verifications. And often the claim is supported by hard evidence indicating that:

* after the final version of the change was reached it was ported to a fresh workspace based on the tip of the branch
* the required artifacts were built from scratch (so the build was totally fine, no cache-related issues, etc.)
* all mandatory tests passed, including those covering the area in question and should have detected the regression
* no intermittent false-positives affected the respective verifications
* no file merges were detected when committing the change to the branch
* none of the files being modified was touched by any other change committed in the branch since the fresh workspace was pulled

Is it really possible for a software change to cause such a regression despite correctly following all the prescribed processes and practices? How?

# Answer
There's one possibility I can think of, if when the dev are working on their own workstation, with sometimes images baked for virtual box to run on their workstation where your infrastructure doesn't use the exact same image.

The dev will need, while developing a feature, need to add a JVM parameter or whatever change to the middleware early in its work and forget it.

Before commiting, all unit/integration tests run on its workstation works great, as the baked image is shared, it works on every develloper system.

But when going through CI, it fails because the change to the middle-ware wasn't implemented, either because the dev forgot to ask for it, or just because the team in charge of updating the base images/provisioning system didn't had the time or did forget to update the system.

That's a good thing it break in CI, because it tells early before going into production that the system won't work as expected, but sometimes it becomes a hell to find the missing parameter.

This last point advocate to avoid rejecting commits, and just break on CI on a feature branch, thus it won't block anyone else, and let the dev fix the problem early, when the change is needed and prevent this change to be forget in the flow.

FWIW, we did exactly this here, developers had whole access to development machines and releases in Q/A were failing because a parameter change has been forget, we did move to chef to handle the configuration of the middleware (tomcat now) so each needed change to the infrastructure has to be coded somewhere and will be reproduced in all environment.
> 5 votes

# Answer
Sure it is. Production is always different. Real money. Real load. Real users. Real pain. This is why it is so important to put any significant change behind a feature flag. Your deployment should not change anything. Turning on a feature is the only thing that should make significant changes to your site.
> 2 votes

# Answer
The breakage is always theoretically possible because the pre-commit verification performed by the developer is done in isolation and thus can't take into account other in-flight changes being verified in parallel. Such changes can interfere with each-other and cause regressions without actually have collissions detectable at the SCM-level.

A simple example of such interfering changes:

Let's assume the code in the latest version of a project branch includes a certain function, defined in one file and invoked in a couple of other files. Two developers working in parallel on that project are preparing to make some changes to the code.

Developer A reworks that function removing or adding a mandatory function argument and, of course, updates all invocations of the function in all the associated files to match the updated definition.

Developer B decides to add an invocation of said function in a file which didn't contain any such invocation before and is thus not touched by developer A's changes. Of course developer B is filling the argument list to match the function's definition visible in the latest label. Which is the old definition, as developer A's changes aren't yet committed.

Both developers correctly perform the pre-commit verifications with a pass result and proceed to commit their code changes. Since the two changesets do not touch the same files no final merge happens, which typically would be an indication of potential problems, warranting a closer look and maybe a re-execution of the pre-commit verification. Nothing whatsoever can give even a subtle hint that something may go wrong.

Yet the end result is catastrophic - the build is broken as the function call added by developer B doesn't match the function definition updated by developer A.
> 2 votes

# Answer
When you find this type of issue, you should write some new extremely quick running acceptance tests that can eke out these issues, and add them to your build verification tests that run prior to your integration tests. You should constantly be shifting left and trying to shorten the feedback loop to the developers committing changes. If you cannot find a way to do this, perhaps your architecture isn't as agile as it needs to be.

@Dan Cornilescu - your scenario is valid for tightly coupled architectures which is why loosely coupled architectures (microservices of versioned RESTful APIs) have emerged as the current best practice in high performing organizations. These matrix of services organizations have other complexities to overcome though.

Sometimes you need to refactor your entire architecture to overcome issues such as these. I believe it was both Google and eBay that have **completely** rearchitected their platforms 5 times (in a span of something like 10 years) due to constraints their previous architecture imposed upon them.
> 1 votes

---
Tags: continuous-integration, regression
---


# Question
Title: How to storing encrypted secrets in code for a serverless.com project?
With serverless.com, the easiest way to expose a secret to an AWS Lambda function is to store it in the `serverless.yml` file (encrypted with KMS, for example).

But committing encrypted secrets to Git is not the best thing in the world. Requires code changes when the secret has to change, for one thing.

But in terms of security alone, what are better alternatives? e.g. the secret could be stored in S3 (encrypted), with the Lambda given access to that location and KMS key, but is that actually better in any meaningful way?

# Answer
There are a couple of elements to consider here:

1. It is valuable to have the capability to maintain configuration with a separate release cadence from the code that is being configured. Doing so encourages rotation of credentials in an automated manner, do it regularly make it painless.
2. It is entirely reasonable from an "Infrastructure as Code" point of view to store all configuration in Git, albeit probably in a different repository as the scope of configuration is likely to span more than one service.
3. Assuming both of the above statements are relevant to your situation, I would be inclined to encrypt the data with KMS and store it in S3 or DynamoDB. I would even go so far to say that in very simple situations it might be suitable to encrypt and store the data within KMS itself.

There are several open source libraries that will support this operation:
> 7 votes

# Answer
I typically treat secrets as configuration data in scope. As such, it does not live alongside the code as they are handled by different release schedules and processes. A separate git repo, KMS, dynamo, s3, or inside your config management system (chef-vault/encrypted database in the chef world) are good places. Basically, you don't want to have to build and deploy a new release of your software to get a secret updated.

If you secrets management needs are more complex, something like Hasicorp Vault (https://github.com/hashicorp/vault) would be a good option.
> 1 votes

---
Tags: security, aws-lambda, serverless, encryption
---


# Question
Title: What are best practices and/or tools for updating AMIs on a regular basis?
I have a Master-Workers style computation model. And, I launch the workers via an AMI through an Ansible playbook.

The workers' AMI has all the necessary configs needed for doing the computation (which is mostly a pipeline of ML tasks).

However, we keep updating and optimizing our algorithms and ML models almost on a regular basis, which means I have to keep my Worker AMI's up-to-date.

So, **is there a tool or a technique which would help me do that?**

Also, **is it a scalable practise, or is doing the containerized way better?** \[By which, I mean launching an EC2 instance and pulling a container and do the computations inside a container\].

# Answer
> 4 votes
We are planning to automate this process for our Jenkins EC2 slaves.

Currently we have to manually update the AMI ID every time we build a new AMI, but by taking advantage of the `config.xml` file where Jenkins store all its configuration, we should be able to automatically update the AMI value in this file and then restart Jenkins to take those changes into effect.

EDIT: see comments below for a better way to make those changes rather the config.xml file

# Answer
> 3 votes
I can answer on how we do it actually:

We're using a private Gitlab on AWS and its CI system. We have a repository for our build environment (a docker image with packer/scripts/json describing amis) and one with a .gitlab-ci.yml which describe the build and export tasks for each of our base AMI.
Each project build its own AMI from our base one on every push and we use a DynamoDb table to store the versions and ami ID for deploys.

This could be an approach, where your ML taks update would trigger the AMI build.

If you go on the docker way, you should have a look at AWS ECS which will cut the burden of starting an EC2 instance before pulling the docker image.

Another option could be to move totally to Amazon Machine Learning but in this case you will raise the burden of migrating you jobs outside of AWS if the need arise.

There's no "best" way in my opinion, all are valid and the one you're the most comfortable with should be your choice :)

# Answer
> 3 votes
You should take a look at packer, we've found it very helpful in automating builds of image types, including AMI's.

We make use of the chef client provisioner with cookbooks to manage the changes applied to an image, although many other provisioners (shell, puppet, ansible, etc) are supported. We can output docker images, AMIs and vagrant boxes from shared configuration, all version controlled with the rest of our config management code.

See here for a breakdown of where this fits in a broader process, but your approach sounds sensible.

Using tools like packer means that if you decide to switch to containers, that the image build side is pretty trivial to change over.

# Answer
> 3 votes
Using containers for such computation nodes makes a lot of sense, and it would offer other advantages, too. But it's not a silver bullet. You'd still have to work at it.

To really answer your question I need to know how you're deploying your system (is it with an ASG? manually? with some tool, like Ansible?).

If you're using an ASG, dockerizing it wouldn't solve the problem of having to update the AMI. It would still need to be updated, since the launch-configuration refers to it, and the ASG will spin up "old" versions. of course, you can make your ASG instances smarter, by using Chef or something equivalent and let them provision themselves and fetch the latest version of what they're supposed to be running. But that means investing in this extra infrastructure, if you don't already have it; and it also means increased scaling latency, since it takes time to do this self-update when a new instance is spun up.

If you're not using an ASG, you can just push the latest image onto your instances when you deploy. Here a dockerized solution would indeed be pretty handy. e.g. if you join your instances into a Docker swarm, you can deploy a new version with one instruction and it'll be done with a rolling upgrade, etc.

So, as you can see, there are a bunch of ways to do it. Depends on what you're already doing now and what your needs are.

# Answer
> 2 votes
Perhaps you might benefit from abstracting your platform (the AMI with base dependencies) from your software. Do you really need to re-roll the AMI every time you update your algorithms and models? Or, do you re-roll when you have a library change and add a deployment setup to your setup that sets up your ML stack on the AMI? If you can rework your execution model to support this you will likely have a higher iteration velocity.

If you can't, I would suggest having your CI process generate and publish new AMIs or images.

---
Tags: amazon-web-services, containers, amazon-ami
---


# Question
Title: How to automatically remove a dead node from RabbitMQ cluster
I am planning to create RabbitMQ cluster using Ansible on AWS VPC with Amazon internal load balancer as the frontend to point connections to it.

Any suggestion how to remove a dead node from RabbitMQ cluster based on autoscaling rule where nodes can go up and down, or if you are using spot instances?

When a node goes down, RabbitMQ does not remove it from the replication list automatically, I can see `Node not running` in the management UI.

I managed to join to the cluster a scaled instance automatically via Ansible and userdata.

# Answer
Consider using the rabbitmq/rabbitmq-autocluster plugin:

> A RabbitMQ plugin that clusters nodes automatically using a number of peer discovery mechanisms:
>
> * Consul,
> * etcd2
> * DNS A records
> * AWS EC2 tags
> * AWS Autoscaling Groups

There is a fair bit of configuration to plug in to get this setup including setting IAM policies and adding EC2 tags to the instances you want to be party to your cluster.

If you were to use AWS Autoscaling Groups then you would add the following to your `rabbitmq.config`:

```
[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_autoscaling, true},
    {aws_ec2_region, "us-west-2"}
  ]}
].

```

If you are not using AWS Autoscaling Groups you can still achieve the desired result using tags on your EC2 Instances:

```
[
  {rabbit, [ ... ]},
  {autocluster, [
    {backend, aws},
    {aws_ec2_tags, [{"region", "us-west-2"}, {"service", "rabbitmq"}]},
    {aws_ec2_region, "us-east-1"},
    {aws_access_key, "..."},
    {aws_secret_key, "..."}
  ]}
].

```

With all of that said I strongly recommend using Consul by HashiCorp as your service discovery mechanism, in the long run, you get significantly more flexibility in terms of decoupling your parts of your system from each other.
> 4 votes

---
Tags: amazon-web-services, ansible, aws-vpc, rabbitmq
---


# Question
Title: How can DevOps help to improve Software Escrow procedures?
Consider a software vendor and a licensed customer of some software of this vendor, wheres the software being licensed is either used on premise (at the customer's location), or in the format of SaaS solution (hosted by the vendor). However the customer only gets access to what's needed to use/run the software (executables and similar things), so not the source components and anything related to that to create the executables.

To protect the customer's business continuity in scenarios where something may go wrong with the vendor (eg: bankruptcy), both parties may agree to some sort of Software Escrow (SE ... 'also') agreement (also called Source code escrow). With such agreement both parties agree to get a 3rd party involved (= a "Software Escrow agent"), trusted by both parties. These are the highlights of such SE-agreement (= specs of the actual SE-process):

* ALL pieces of software components ( related to the licensed software) get deposited by the software vendor, at some agreed location related to the SE-agent. Such deposits include the executables, but also the source components and anything related to that to create the executables (even documentation, instructions, etc to create the executables).
* Since the software vendor may create multiple releases for the duration of the software license, and the customer has the right to receive such new releases (as per the license agreement), part of such SE-agreement is that "with every major new release" (whatever "major" may mean ...), the deposit delivered to the SE-agent will also be updated/refreshed.
* If specific conditions are satisfied (e.g: bankruptcy of the vendor), then the Software Escrow agent will deliver to the licensed customer, upon request of such customer, of copy of everything that was deposited, so that customer will be able to continue using the software, and where needed even adapt the source code to continue to use it for the customer's business.

A common practice for such SE-agent to get involved, is some sort of a legal person/entity, such as a lawyer. But to actually "process the SE-deposits" (by the SE-agents), all sorts of release management and/or software delivery tasks need to be performed by somebody or something (the poor SE-agent) who probably doesn't know at all what the licensed software is supposed to do ... fun guaranteed!

**My question**:

How can DevOps help to improve Software Escrow procedures as described above? Like what kind of -tools would you recommend to be used for the fulfillment of which part of the SE-agreement? And where appropriate using which (preferably open source) software solutions for it?

**Notes**:

1. To not further complicate things, just assume that it is agreed between all parties involved, that the SE-agent does NOT have to do any type of "*verifications*" about the deposits being done. That is: whatever is deposited is assumed to be complete, up-to-date, documented, etc.
2. About "major new release": assume there are between 1 to 3 every year, which means that the licensed customer only expects to be able to get access (via the SE-agent) to those releases. Even though if there have been intermediate deliveries (like fixes or beta versions) to the licensed customer, those types of deliveries are considered out-of-scope. Even if it was only because:

* the SE-agent charges "for each deposit to be processed by the SE-agent".
    * the licensed customer rarely changes releases, and is only interested in being able to use the SE-agreement if things go wrong, for the very release they are running at the moment things go wrong.

# Answer
> 5 votes
A very interesting question. On the assumption that the goal of a Software Escrow process is to allow for a 3rd Party to take over or nominate an additional party to fulfil the responsibility of the software vendor, I would suggest the following elements of a DevOps operating model that would support software escrow:

1. **Infrastructure-as-code** \- effectively documenting through an executable specification of dependant infrastructure, stored and versioned in source control provides the environments that the source code is developed in. Unlike static documentation in text files because it is executed on a regular basis by the software vendor to build their own environments it does not go out of date of suffer from "bit rot". This will always hold the most value when the entire development pipeline is built and maintained in source control applying Infrastructure-as-code principals.
2. **Continuous Integration** \- the goal of continuous integration is to execute a set of steps that integrate the solution on a regular basis, ideally upon each change. Typically this means upon check-in and push to a central repository a set of tests are executed that validates the process. From a software escrow perspective I would expect this to also push a working version to a secondary "backup" repository owned and operated by the 3rd parties. It is important to note that this does need to be both legally and financially "unlinked" from the software vendor.
3. **Continuous Deployment** \- the goal of continuous deployment is to deliver software in a working state on a frequent basis. In this sense the 3rd party is just another deployment target to deploy the outputs to, albeit probably not actively spinning up infrastructure on the 3rd parties fabric.

Some other considerations to bring into the equation are:

* the move from static documentation to infrastructure as code significantly reduces the toil involved in updating documentation that describes how to install, configure and recover software.
* don't forget secrets management, such as X.509 Certificates, symmetric keys, passwords and licence keys, these could be stored in source control however this does have it's own drawbacks.

From a tools perspective this will depend very much upon the development environment, I don't believe there are any tools specific to software escrow however:

At the end of the day if you are able to employ Infrastructure-as-code, Continuous Integration and Continuous Deployment principles to a software package it can be used to fulfil your obligations under a Software Escrow contract.

# Answer
> 1 votes
> But to actually "process the SE-deposits" (by the SE-agents), all sorts of release management and/or software delivery tasks need to be performed by somebody or something (the poor SE-agent) who probably doesn't know at all what the licensed software is supposed to do ... fun guaranteed!

I wouldn't do business with an SE provider which would allow such unfortunate situation to exist - they don't know what they're doing.

If the SE deposits are to be processed in any manner by the SE agent the exact and complete processing procedure needs to be *fully documented* in the agreements so that it is actually usable.

That should also include the exact environment specification (or procedure to reproduce it) and the toolchain used for such processing. In other words the choice is not really open unilaterally to the SE agent. Except maybe for the actual storage strategy (personally I'd include that choice in the agreement specification as well, even if the choice is done unilaterally by the SE agent).

A good SE agreement would also ensure that each and every SE deposit by the vendor goes through that processing and the customer always gets and *qualifies/signs off* the result of that particular processing, not an alternate result coming directly from the vendor - to validate that the procedure itself remains up2date and effective for each and every SE deposit.

Otherwise the ability to reproduce the "SE withdrawals" at a later time (if/when needed) is questionable, which practically voids the whole SE story.

---
Tags: release-management
---


# Question
Title: Can GitHub's Pull Request model be used for implementing post approvals?
The accepted answer to "What are possible implementations (or examples) of the four-eyes principle?" suggests that GitHub's Pull Request model is a possible implementation.

And my own answer to "How to implement the four-eyes principle for emergency fixes?" explains the concept of ***post-approvals***.

**My question**: Can GitHub's Pull Request model be used for implementing post approvals also? If so: how? If not, is there anything else in Git (GitHub?) that I could use for it?

# Answer
Yes, I do believe so. To explain that I need to lay some groundwork on how I have implemented something similar, I have simplified the model in an attempt to make it as clear as possible.

## Assumptions

I am making the assumption here that Jenkins, TeamCity or similar is being used as the CI/CD tool of choice. Additionally GitHub is being used and that there is a well-defined and appropriately controlled branching structure:

## Configuration

In this example, GitHub is configured as follows:

* The Black 'Master' branch can only be merged into using Pull Requests, direct commits are disallowed.
* The Blue 'Development' branch can accept direct commits or merges; in practice, there may be additional restrictions on this branch.
* The Red 'Hotfix' branch can accept direct commits and merges.
* Required Status Checks are enabled in strict mode to prevent pull requests merging when the branch is failing the build.
* If the Hotfix branch is ahead of Master then pull requests *into* Master will be blocked, either with the Status API or the Pre-Receive Hooks on GitHub Enterprise.

The CI/CD tools are configured as follows:

* Builds from the Development branch cannot be deployed to production.
* Builds from Master can be deployed to all environments.
* Builds from Hotfix can be deployed to all environments.
* Deployments from Hotfix will notify some non-development function, for example, the Change/Release team and ask them to perform the *post-approval*.

**Notes**

Master is protected as it represents the current state of production, to do this practically you may have another "Release" branch that deployments are made from and only when successful merge into the Master branch.

## Key Points

The Blue Development branch is basically a free-for-all. Hotfix is kind of a free-for-all but any deployments trigger a type of Break Glass by notifying a non-development function who will perform the *post-approval* and in the process will merge the change into Master.

It's essential merges into Master stop while Hotfix is ahead of Master to:

1. Prevent a deployment from Master overwriting the Hotfix, which could result in an regression.
2. Prevent a Hotfix sitting in the Hotfix branch languishing without being merged in.

In some organisations it may help to prevent all pushes to the central GitHub repository while a Hotfix is pending post-approval.
> 4 votes

---
Tags: git, configuration, four-eyes
---


# Question
Title: What strategies can be employed to secure sensitive data in log files?
Working in highly regulated environments data is classified in different ways depending on the sensitivity. In some cases, this is legally enforced and must be treated differently.

Examples of a data classification policy are:

* **Highly Restricted** Data, such as Passwords, Private Keys, SAML Tokens, and Credit Card Numbers.
* **Restricted** Data, such as Usernames and Customer IDs.
* **Unrestricted** Data, pretty much anything else.

This classification comes with certain obligations:

* Any data that is **Highly Restricted** must never be made available in log file under any circumstances.
* **Restricted** data could be made available in log files under specific conditions. For example, if an incident were to occur with a service then the on-call engineer would be able to enact a Break-Glass procedure to access this data to diagnose the problem. The Break-Glass procedure would, in turn, trigger a review, an audit, and possibly temporary revocation of privilege from that engineer.

What strategies can be employed to achieve this, particularly given that there are a broad range of logging, monitoring and instrumentation tools available on the market that don't provide a direct answer to this problem?

For example, both Splunk and AppDynamics have the ability to impose different access controls upon conditions of the telemetry being exposed; this means you can create a filter that masks out `<customerId>NNNNNNNNNNNN</customerId>`. However, neither of these tools give you the capability automatically identify credit card numbers and mask them automatically.

**Note**: I believe this is related to DevOps because in a traditional tiered support model a relatively small group of people could have access to the data and filter it manually, by devolving responsibility for operating platforms to development teams this data is potentially exposed to a far wider audience.

# Answer
I think the solution comes down to a broad spectrum of approaches that ensures data protection:

* **Data Classification**: The most efficient technical strategy is to categorise the data at the point of creation rigorously. At its core, the developers are responsible for ensuring that all logged information is assigned a category. Categorization can, for example, be achieved through Splunk Metadata which, in turn, can be used to direct log entries to different buckets based upon their data categorization.
* **Event Partitioning**: There is often a desire to log sensitive information along side non-sensitive information. For example, if a new user was to sign up you may log:

+ Customer ID (Restricted)
    + Customer Type (Restricted)
    + Aquisition Source (Unrestricted)
    + Correlation ID (Unrestricted)

It is possible to split this one "Event" into two parts, one containing the *Restricted* information and one containing th *Unrestricted* information. This aligns with the first point by allowing filtering rules to direct to different buckets.
* **Data Masking**: In specific circumstances it may not be possible to categorise the data at source, in my experience logging solutions do allow for Masking Rules to X out sensitive data. In the linked example a `sed` command is used to apply a regular expression to all data from a specific source. Once the restricted data has been masked out then the event can be considered to be Unrestricted. Care must be taken with the rule to ensure that information critical to an event such as a correlation ID doesn't match the Regular Expression used to match sensitive data.
* **Event Filtering**: As a last resort it may be necessary to filter all events of a particular type or source into a separate bucket if they contain sensitive data that cannot be categorised or masked. In this case the information in a restricted bucket could only be accessed through a Break-Glass mechanism to bypass access controls under the provisions of an incident.

With each of these solutions, testing is key to ensuring that nothing slips through the cracks. Logging and Event Management must be considered to be a first class Non-functional Requirement with the same level of development rigour applied to a solution - including peer review and tests to ensure that data is properly categorised and partitioned in the tool of choice.
> 4 votes

# Answer
It depends on what you mean by "log files" I supposed. If you mean the telemetry data you use to verify your system is operating correctly I would say "Don't log sensitive fields." You don't need that kind of information to alert you to high or low transaction rates, response times to your dependent services, etc.

If you mean data for billing or audit purposes I would suggest you establish a write-only pipeline where the data is written but can't be read by the writer. Then your billing and auditing pipeline kicks in and you have the controls there to audit who looked at individual records.

In the end security comes down to documented processes with their own secure log files, etc. At some point you have to trust someone or some process since all data is written so that it can later be read.
> 0 votes

---
Tags: security, logging, data, financial-services, data-protection
---


# Question
Title: Why can this ENV not be found when it is defined in one layer using a Dockerfile?
When the following Dockerfile:

```
ENV A=123 \
    B=$A
RUN echo $B

```

is run, it results in:

```
Step 2/2 : RUN echo $B
 ---> Running in x

 ---> y
Removing intermediate container z

```

When two `ENV` layers are defined:

```
ENV A=123
ENV B=$A

```

it works:

```
Step 2/2 : RUN echo $B
 ---> Running in x
123
 ---> y
Removing intermediate container z

```

# Answer
> 3 votes
The `ENV` keyword will be translated to `export` command (builin of the shell usually), if you remove the multi-line declaration you end up with this equivalent:

```
export A=123 B=$A

```

What happens here is that when the shell parse the line to give it to `export` input, `A` is not yet exported and available as an environment variable.

Step by step this will give:

* You type `export A=123 B=$A`
* The shell parse the line and pass `A=123 B=` to `export`'s stdin
* `export` parse each argument in the form `key=value` and as such export `A` to be `123` and `B` to be empty

On the second form, A is available to the shell when the line is parsed, and as such the resulting `export`'s stdin is `B=123`

# Answer
> 0 votes
```
ENV abc=hello
ENV abc=bye def=$abc
ENV ghi=$abc

```

will result in def having a value of hello, not bye. However, ghi will have a value of bye because it is not part of the same command that set abc to bye.

---
Tags: dockerfile
---


# Question
Title: What is the definition of "cattle not pets"?
The term "treat your servers like cattle not pets" has proliferated in recent years, particularly when applied to Docker containers and Virtual Machines

What does it actually mean?

# Answer
> 92 votes
Randy Bias chronicles the history of the term stating that it probably originated in 2011 or 2012 when Bill Baker used the analogy when describing "scale-up" vs. "scale-out" architectural strategies. Bias adopted this into his presentations about cloud architectural patterns:

> In the old way of doing things, we treat our servers like pets, for example Bob the mail server. If Bob goes down, it’s all hands on deck. The CEO can’t get his email and it’s the end of the world. In the new way, servers are numbered, like cattle in a herd. For example, www001 to www100. When one server goes down, it’s taken out back, shot, and replaced on the line.

Bias continues to define Pets as

> Servers or server pairs that are treated as indispensable or unique systems that can never be down. Typically they are manually built, managed, and “hand fed”. Examples include mainframes, solitary servers, HA loadbalancers/firewalls (active/active or active/passive), database systems designed as master/slave (active/passive), and so on.

and cattle as

> Arrays of more than two servers, that are built using automated tools, and are designed for failure, where no one, two, or even three servers are irreplaceable. Typically, during failure events no human intervention is required as the array exhibits attributes of “routing around failures” by restarting failed servers or replicating data through strategies like triple replication or erasure coding. Examples include web server arrays, multi-master datastores such as Cassandra clusters, multiple racks of gear put together in clusters, and just about anything that is load-balanced and multi-master.

Fundamentally, what Bias and Baker are trying to convey is there has to be a transition from how we treat servers from being "Unique Snowflakes" with names and emotional attachments, to a model whereby if we have a problem with the server we create a replacement and destroy the problematic server.

Finally, it is probably worth mentioning that in regulated environments taking a server out the back and shooting it may not be optimal. In these cases it is often advantageous to "freeze" the server, for example using `docker pause` to freeze a container. This can then be used to perform a Root Cause Analysis as part of the Incident or Problem Management Process.

# Answer
> 30 votes
To add to Richards answer, generally the analogy is helpful in terms of considering the impact of the loss of a server.

If you would feel some sort of distress over the loss of any individual piece of infrastructure, then consider it a pet (read antipattern).

If you would feel pretty comfortable knowing that if any of the fleet stopped functioning there would be no real impact to operations, then you are talking about cattle.

It's often tempting to use the analogy to simply classify your servers, ie "our workload nodes are cattle but our load balancers are pets" but falling into that trap is exactly the problem. There is no place for pets in a modern computing environment (ie. In the cloud, on commodity hardware etc.) If all of your servers are considered cattle, and are easily replaceable, then you can start looking at things like chaos monkey to help build assurance that your infrastructure truly is resilient.

---
Tags: terminology, snowflake-server, cattle-vs-pet
---


# Question
Title: What opensource solutions can be used to implement anomaly detection based on Riemann metrics?
When Riemann used as system and/or application metrics collector, what are opensource tools can be used in order to implement anomaly detection monitoring?

# Answer
I can think of 3 possibility (Not exactly Riemann metrics based, but around anomaly detection):

1. Outdated but the most efficient in anomaly detectionI know: Shyline + Oculus from Etsy Both tools available from etsy's github as Skyline and oculus, check the network graph for more up to date forks. Best one I found is earthgecko/skyline with its doc
2. Graphite and checking that the Standard deviation fits in the Holtz Winter's Confidence bands
3. ELK stack in version 5 with timelion which, as above, allow to compute a standard deviation and alert if it goes out of confidence bands with the proper license.
> 4 votes

---
Tags: monitoring, network-monitoring
---


title: What Is SecOps?

text:
# Question
Title: What Is SecOps?
First when I heard the word SecOps, I thought of it as a management approach that aims to connect security and operations teams, in much the same way that DevOps unifies developers and operations team.

But isn't security just a piece of the DevOps puzzle ?
DevOps already includes processes like component monitoring ,version management, benchmarking, code review, continuous monitoring ..

What other can SecOps add to a DevOps team , or is it another buzz word maybe ?

# Answer
I'd agree this is a buzzword as much as DevOps can be.

Main task of a SecOps added on top of an usual operational engineer tasks is to take the burden of following CVE publication feeds, handling the remediation, usually handling things historically handled by the security or network administration team (Firewall rules, Web Applications Firewall exceptions)

If you see a Sysadmin in a DevOps org as a sysadmin able to read and be part of the application code, a SecOps would be a sysadmin able to be part of the security rules of the infrastructure around the servers themselves.

In some structure keeping silos of responsibility (Sales, Business, Dev, Ops, Security, Monitoring), Security Engineer and Operation Engineers are as separated as Developers and Operations Engineers, while not embracing a full DevOps organization, moving to a SecOps model is a first step to reunite two separated team historically closer and less antagonist in the silo based organization. Some people are also more comfortable to add either the operation or security facet in their current job than growing code skills.

To summarize, I would define SecOps as a first step toward a DevOps org, aiming at getting a multi-skill team around security/network/operating systems engineers where they are separate teams in an existing department.
> 18 votes

# Answer
I do agree with Tensibai's Answer in that SecOps is as much as a buzz-word as DevOps itself and that SecOps is a stepping stone between a siloed organisation and cohesive organisation.

I have observed the flip side to also be true, that is, if you have an organisation who is operating using a DevOps Model, with DevOps ways of working and following DevOps Practice, they may appoint someone in a SecOps role to integrate IT Security practice into the model.

Quite commonly this is branded as DevSecOps or DevOpsSec so as to bridge all three disciplines.

Further Reading:

Of the above I am currently reading Jim Bird's book, have had the pleasure of working with the organizer of DevSecCon in a previous role and had the privalage of attending the Velocity Session in Amsterdam last year.
> 9 votes

---
Tags: terminology, culture, security
---