# Question
Title: How to explain devops to non-techy managers?
I'm totally excited about DevOps. I know that DevOps is the methodology that will move us into building an IT infrastructure that will streamline and move our company forward.
But how do I sell this to my bosses, especially non-technical bosses?

We are going to implement an automation project which will include, automated deployment , infrastructure cloudification , continuous integration process .. we definitely need to persuade our bosses to invest at higher levels in this.

**Note**: we started to improve our process by automating tests, releases and supervision , it's a step towards adopting devOps but the automation project itself is in stand by as we need more investment.

# Answer
Being a consultant I am contractually obliged to answer, "it depends". With that out of the way, I can actually answer your question.

What does it depend on? Well, that could come down to what your boss thinks about DevOps:

1. If your boss has heard of the term, via their obsession with CIO.com maybe, then ask them what they think it means. From there work out what the difference is and if their view is compatible. Identify a suitable project to trial DevOps on and pitch it to them. Remember that at the core DevOps is culture so consider how that could be applied to a project.
2. If your boss has never heard of the term then build a business case for DevOps. Use the Puppet Labs State of DevOps and material from books such as The Phoenix Project to write the business case. Find a problem that your boss has and DevOps could solve and use it as a conversation starter. As kenchew said you don't have to mention DevOps, you might for example, suggest that Operations be more involved in a project or more test automation to be planned as part of project delivery.
3. If your boss thinks DevOps is just another buzz word then do one of the above but absolutely don't mention DevOps, look at other similar models such as Site Reliability Engineering, Platform Engineering or Continuous Deployment and find out how they could solve the problem.

The key is to focus on understanding what your boss is motivated by, then carve out a small bit of time, money and people to take tangible steps to solving that problem.

I highly recommend the book To Sell Is Human by Daniel H. Pink, fundamentally Daniel Pink talks about how selling something is a very human thing to do, all we need to do is tap into the needs and align our "pitch" by proposing a solution that serves those needs.
> 15 votes

# Answer
You don't.

Despite your enthusiasm about DevOps, non-tech bosses don't really share your fascination with the techie lingo.

First, show your bosses the benefit of a small pilot projects you have done. Collect some useful data points to prove your case. (Found this question that could help: What are some methods to measure the ROI for DevOps?)

Next, tell your bosses you have a project that could bring more benefits but need a small investment. (Try to figure out a project that don't let your bosses fall off the chair. You should have a feel of what this figure is if you have been working with your bosses for a while.)

Once you got the investment, do an excellent job of achieving the objective. Better yet, wildly exceed it!

Now, when finally the bosses ask you "So what did you do that brought us so much benefits?"

This is the time that you proclaim:

"DevOps"

And ask for more investment for your next devops project.
> 10 votes

# Answer
Any business initiative would get a traction if you show its relevance to either topline or bottomline of the organization.

Internal initiatives such as devops can only affect the bottomline. You need to identify the costs of recurring work performed by the individiuals and how automation would reduce that expenditure.

Even though non-techy managers may not understand the difference between choosing chef over puppet, they have some understanding of Industry trends. You can make them aware of the costs of delays due to builds not being available, costs of regression issues and how your approach can reduce those costs. If you can show tangible plan for improvement in bottomline, and if it is better than the other action items on their plate, you would get a go-ahead.
> 4 votes

# Answer
My go-to line of reasoning for people who are unfamiliar (or just mistaken) about the term DevOps boils is down to "delivering business value more frequently". This, in my experience, is something very few managers are able to object to. They get it.

If they say something like "we just need someone to straighten out our devops, probably just a few weeks of work; so there's a limit on how much we're going to invest in devops right now" I just try to explain that's like saying "we don't want our company to deliver too much business value. we just need a bit more, but that's it".

It's just rhetoric, of course, but I find it's effective, much more so than than telling them to read a book about Toyota.
> 4 votes

# Answer
Everything in the previous answers is true, but I think there are some things missing to actually get the approval and commitment from your bosses (BTW: most people only have 1 boss at max ...).

Sooner or later Mr. Murphy will come along (= *Anything that can go wrong, will go wrong, and it shall go wrong when it shouldn't go wrong*). And at that point some boss(es) will want to get answers to questions like this one:

> What happened when and why, and which authorized user actually approved it ... upfront?

And at that point you'll get the real ROI from DevOps-practices you'll have in place ... And/or all of a sudden get all sorts of huge budget approvals to implement what you seem to be looking for.

Even if it would take too long for Murphy to come by, your company might also run into requirements such as what Richard described in the question "What processes or tools enable Segregation of Duties when engineers both deploy and run code?" (those kind of requirements scare CxOs ...).

But, if you ever have to introduce "DevOps" to somebody who is new to it, it may help them to "warn" them upfront like "***OK, so you want to start DevOps practices, great! But be aware that it is like switching to another religion ...***"
> 2 votes

---
Tags: culture, project-management
---


# Question
Title: What are ChatOps approach use cases of production infrastructure management?
Many "simple" use cases shown in examples of ChatOps approach, creation of cloud instances, deployments to test environment etc., are there any use cases for production infrastructure management?

# Answer
> 7 votes
ChatOps is an absorbing topic; practically it means something very different to each team using it. So much so that Atlassian have put together what could be considered to be a Chat maturity model.

1. **Sputnik**: Trying chat out to see how it works.
2. **Mercury**: Starting to move to chat.
3. **Gemini**: Chat as the primary source of communication, experimenting with bots.
4. **Apollo**: Chat as the nexus of communication and work, including bots.
5. **Elon Musk**: Chat is the primary centre of mission critical work.

**Important:** Chat and ChatOps is very specific to an organisation, think of Chat as the Fabric for your team's bots to existing within it's the bots that provide the functionality that enables business and technical processes.

With the brief lesson in ChatOps Theory over I can talk about the experiences that I have had with ChatOps:

## DevOps Support

Slack and PagerDuty has fantastic integration allowing any newly raised incidents in PagerDuty then posted in one or more Slack Channels with information about the incident and buttons for acting upon the knowledge:

Beyond connecting PagerDuty and Slack we also connected all of our monitoring and alerting tools into Slack so that it was the central point for all real and potential problems, including the ability to query New Relic directly from Chat.

## Chat as the Modern Command Line

With another client we built custom bots that executed scripts on remote machines, we did this using a combination of calling the management APIs in Azure and running Cluster Fork commands:

> `@checky scale serviceA 30`: Scale ServiceA to 30 machines.
>
> `@checky kill serviceA#4` : Kill instance number for on ServiceA.
>
> `@checky exec serviceA#1,5 ps -awx` : run the `ps` command on serviceA instances 1 and 5 and post the output to chat

## StackOverflow Close Vote Reviewers

We have some chatbots running in the SOCVR channels that help us to:

* Find questions that need close votes.
* Identify comments that contain "Heat" - i.e. might be offensive or lead to escalations.
* Identify questions that are of poor quality that might need review.

## Further Reading

# Answer
> 2 votes
A prime use-case for Enterprise IT would be to reinvent their process for **handling Major Incidents** (sometimes called "Incident War-room") by utilizing the ChatOps practice. I've captured this and additional use-cases relevant to Enterprise IT - in a blog series:

https://community.hpe.com/t5/IT-Operations-Management-ITOM/ChatOps-ChatBots-and-Enterprise-IT/ba-p/6924240

https://community.hpe.com/t5/IT-Operations-Management-ITOM/Chat-Bots-in-the-Enterprise-Part-2/ba-p/6925811

(and you can also continue to the 3rd part in that series from those links...)

I'd love to get your comments/feedback.

---
Tags: toolchain, infrastructure-as-code, chatops
---


# Question
Title: Who is this Chaos Monkey and why did he crash my server?
I had a perfect server, it was so pretty and rock solid and so I named it Petra. It was perfect in every way, everything was configured and tuned just right, it had perfect 100% service record and 753 days of uptime. I've spent a lot of time and effort making sure it run so well. No other server in the company had been this good. But last night this evil monster crashed my server for no reason.

Of course I was notified at 2am and it took me until morning to get it up and running and everything configured and tuned up, but I'm afraid it is not going to be as good as before. It might take weeks before it is back to it's former glory. Now my uptime is gone, I don't have even measly three 9s and who knows what this will do to my reputation. Who is this Chaos Monkey and why did he do that to my server and why is he trying to ruin me?

# Answer
> 39 votes
**TL;DR**: Chaos Monkey was developed in 2010 at Netflix and released into wild in 2012 is part of the Simian Army, wildly popular among devoted followers. Built on principles of chaos engineering, the army increases resiliency to failure by injecting constant failure to the system.

## Concept

Chaos Monkey was developed specifically for AWS where it will randomly kill instances within an Auto Scaling Group. It is meant to run during the business hours when engineers are alert and can quickly react to discovered failures.

## Simian Army

Members of the army would *sow chaos* through other means:

* Latency Monkey will introduce random delays to services.
* Chaos Gorilla (Kong) will simulate outage of entire availability zone.

Other Monkeys are *helpful* and remove the weak members of the herd:

* Conformity Monkey shuts down instances not following best practices.
* Security Monkey looks for known security vulnerabilities in configuration and services.
* Doctor Monkey shuts down unhealthy instances not conforming to certain metrics.
* Janitor Monkey looks for unused resources to reclaim.

## Failure is Inevitable

Failure in the System is inevitable, something will always go wrong. You might not be able to choose what, but you can try to choose when. By introducing small errors throughout the day, you ensure that your engineers are present. By killing non-conforming services quickly, you ensure that failures happen often before deployment. By making the environment more adversarial, you ensure that it will be the developers who run into issues long before any service makes its way into production. Failures will be quickly apparent in integration phase of new services with the old ones, but that is ok, because the old production services are already resilient.

## Cattle not Pets

Everyone will tell you lately: Do not treat your servers as pets. There is a power in numbers and any single point of failure will bring down the system. No matter how well you can tune and optimize your server, no matter how beefy hardware you can get, how much it can handle, it will never be a match for herd of small scalable instances. Chaos Monkey encourages you to think about removing all points of failure, because sooner or later, the Monkey will come! Everyone fails and even the Amazon S3 had an unpredictable outage.

## Anti-Fragile

So what is the theory and why does it work? Nassim Nicholas Taleb in his book Antifragile describes a concept where living self-aware systems, will benefit from a small levels of randomness and actually become better in face of adversity. This is similar to annealing.

He does also describe an evolutionary way, where *fragility of parts* in a system is *transferring into antifragility of the whole*. The transfer occurs on two levels:

1. By a small random variations - *developers making changes* \- the most fit for the environment will survive and propagate - *pass tests and get deployed*. **Standard Development Life Cycle**.
2. By failure of parts not capable to withstand a larger level of randomness in the environment, the remaining parts that were able to withstand it compose a system that is as a whole better able to deal with changing environment than before. This is essentially **Chaos Monkey**.

Larger levels of randomness can be withstood using the second approach.

# Answer
> 9 votes
Some additions to your own answer to this question ...

### Additional monkeys

The article about "How chaos boosts performance" describes a few more of these monkeys, i.e.:

> * **10-18 Monkey**: finds configuration and run time problems in instances that serve customers in multiple regions.
> * **Chaos Kong**: simulates an outage of an Amazon region.

Remark: The same article also mentions "Chaos Gorilla: simulates an outage of an Amazon availability zone", though it could well be that this has been renamed now to "Chaos Kong: simulates an outage of an Amazon region" ... Talking about *Chaos*! I couldn't find any confirmation/docu on that so far, at least there does not seem to be an issue for it in the issue queue. An undocumented change might have made it to production on github ... Gggggggrrrrrreat!

### Setup and use your own Monkeys.

Head over to github to get in touch with the Simian Army (same link as the very first link in your own answer). Here is a quote of what you'll find there:

> Simian Army consists of services (Monkeys) in the cloud for generating various kinds of failures, detecting abnormal conditions, and testing our ability to survive them. The goal is to keep our cloud safe, secure, and highly available. More details can be found at this blog.
>
> Currently the simians include Chaos Monkey, Janitor Monkey, and Conformity Monkey.
>
> Refer to the Quick start guide to get started setting up and using the Monkeys.

You can even configure the monkyes, so that they fit your business needs.

If you dig deep enough within those Github links (i.e within the Support link), you'll also find a link to join the SimianArmy Google Group.

# Answer
> 3 votes
> One Server to rule them all, One Server to find them,
> One Server to bring them all and in the outage bind them

You, Sauron, forged this One Server, in the Darkness of Mount Doom your Datacenter in a desire to rule All applications.
Hopefully the Fellowship of Devops did unite to tell you:

After a long fight, Frodo the Chaos Monkey has been able to melt your One Server and bring freedom to all applications, driving you to the way of reproducible Servers at the same time.

Credits:

---
Tags: amazon-web-services, process, automation, servers, sysadmin
---


# Question
Title: What are alternatives for Docker in Docker (dind) usage for Jenkins slave running in container?
When using Jenkins slave running in container, what are alternatives or "safe" (in terms of security) implementations for Docker in Docker.

Architecture example:

1. Jenkins master running as Docker container
2. Jenkins slave running as Docker container
3. Test task executed in Docker container

Some known alternatives:

Constraints clarification: the idea is to use Jenkins as an orchestrator and Docker for clean environment, the rest of architecture is a subject to change.

# Answer
To avoid relying on `dind` (out of building new images from a dockerfile which should be ok) I'd look around Jenkins's kubernetes plugin.

It obviously needs to bring to life a kubernetes cluster but allow to have a one time jenkins agent to run the test in a clean isolated environment, and then tear it off. The maintenance costs of the kubernetes cluster should not be greater than maintaining your Jenkins slaves.

The bonus point is that you can version you slaves with specific environment and don't have to tweak around for multiples version of ruby/java/python and let the task choose the proper "environment" by specifying the agent image to use and have a ready to test environment without any tweak pre-test.

I'd head toward kurbenetes as container scheduler because it is relatively lightweight to configure and maintain.
> 5 votes

---
Tags: jenkins, docker, docker-in-docker
---


# Question
Title: What is Ansible's config equivalent of `--vault-password-file`?
According the help of ansible-playbook one could use `--user=REMOTE_USER` to define the ssh user, but one could also define `ansible_ssh_user: REMOTE_USER` in either the host- or group\_vars.

**Question**

What variable need to be defined in either the group- or host\_vars directory to prevent that `--vault-password-file` has to be defined while running `ansible-playbook`?

**Attempts**

* When `ansible_vault_password_file: ~/.vault_pass.txt` is defined in the config the decryption fails:

```
ERROR! Decryption failed on /path/to/vault

```
* No associated vault variables was found in this documentation

# Answer
> 15 votes
Here is the definition:

```
DEFAULT_VAULT_PASSWORD_FILE = get_config(p, DEFAULTS, 'vault_password_file', \
'ANSIBLE_VAULT_PASSWORD_FILE', None, value_type='path')

```

---

This means that you either put in ansible.cfg or playbook:

```
vault_password_file: ~/.vault_pass.txt

```

Or in your shell defined this variable:

```
export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txt

```

# Answer
> 3 votes
You can set an environment variable `ANSIBLE_VAULT_PASSWORD_FILE` storing the path the the vault password file. This way you won't have the always use the `--vault-password-file` switch when running a playbook.

This is described in Ansible's Vault documentation, available here.

So, add `export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txt` to your `~/.bash_profile`, source from it and you're ready to go.

If you need different vault passwords for different groups of hosts, then you should do the following:

> Inside of this subdirectory, create two files named vars and vault. Inside of the vars file, define all of the variables needed, including any sensitive ones. Next, copy all of the sensitive variables over to the vault file and prefix these variables with vault\_. You should adjust the variables in the vars file to point to the matching vault\_ variables and ensure that the vault file is vault encrypted.

This is an example for best practices approach for managing sensitive information on per group basis. More information is available in Ansible's documentation here (The above text is copied from there).

---
Tags: ansible
---


title: What is a "Walking Skeleton"?

text:
# Question
Title: What is a "Walking Skeleton"?
One of my agile teams has taken an interesting approach in the early stages of their project. Instead of starting the project with a Sprint 0 where they setup code infrastructure and decide on the solution architecture, they have started building a "Walking Skeleton", which they describe as a DevOps practice.

What this seems to come down to is building something very small (in the case of an API a single endpoint that just returns `200-OK`), getting this working in continuous integration, and building out the continuous delivery pipeline to deploy this through each of the environments:

> Development ► Test ► UAT ► Pre-production ► Production

In the process they have managed to tick off many of the non-functional requirements that could have been missed if deployments were left to the last minute.

My question is this: what is a "Walking Skeleton" and what benefit does it provide to a Agile team following DevOps practices?

# Answer
> 51 votes
A "Walking Skeleton" is a form of "proof of concept" of your basic architectural concept. Where a proof of concept typically focuses more on a single functionality, a "Walking Skeleton" is a minimalistic end-to-end implementation. A "Walking Skeleton" is not an outline of your concept (only a "skeleton") but is really executable and shippable (it can "walk" :O) and should be accompanied with tests.

Alistair Cockburn has described it (and is being quoted often):

> A Walking Skeleton is a tiny implementation of the system that performs a small end-to-end function. It need not use the final architecture, but it should link together the main architectural components. The architecture and the functionality can then evolve in parallel.

The advantage here for DevOps is that a "Walking Skeleton" should be developed early on in the project and results in **working, shippable and testable code**. This way DevOps can set up a full continuous integration chain early in the project, instead of being onboarded in the final phase of projects. This means that any issues that would arise are also being tackled in an early stage instead of rush work at the end.

---
Tags: terminology, culture, practices, patterns
---


# Question
Title: Should I use multiple beats input in Logstash?
After some researches around the `beats` input plugin and specially this rewrite I wonder if I should use only one beat input or multiples to handle multiples entry types.

I'll have events coming from roughly 500 machines, with a 20/80 windows/linux distribution. I plan to use multiples beats shipper, filebeat, metricbeat and maybe packetbeat.

Is there an interest at using one input per type/os couple or would just one input and "if type=..." in the filter pipeline be enough ?

# Answer
> 6 votes
So still not benchmarked much, but after a quick code inspection and a bunch of reading about netty and logstash in version 5 the input is not the bottleneck to worry about.

Logstash team did put a bunch of work in the way the filters and outputs plugins are run in parallel, the beats input plugin wait for a batch of events, and the performances problem have indeed been solved in version 3.1.0 by the rewrite I quoted in the question.

---
Tags: architecture, performance, logstash
---


# Question
Title: Why is ansible not sending the correct sudo password?
What does it mean when this:

```
EXEC sshpass -d29 ssh -C -vvv
  -o ControlMaster=auto
  -o ControlPersist=60s
  -o ControlPath="/root/.ansible/cp/ansible-ssh-%h-%p-%r" -o StrictHostKeyChecking=no\
  -o Port=58030
  -o User=em7admin
  -o ConnectTimeout=30 127.0.0.1 /bin/sh
  -c 'sudo -H -p "[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: "
  -S -u root /bin/sh
  -c '"'"'echo BECOME-SUCCESS-mxraphooaxhpruunceorxkcelobceggf; LANG=C LC_MESSAGES=C LC_CTYPE=C /usr/bin/python'"'"''

```

returns this:

```
debug1: Sending command: /bin/sh -c 'sudo -H -p \"[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: \" -S -u root /bin/sh -c '\"'\"'echo BECOME-SUCCESS-mxraphooaxhpruunceorxkcelobceggf; LANG=C LC_MESSAGES=C LC_CTYPE=C /usr/bin/python'\"'\"''
debug2: channel 2: request exec confirm 1
debug3: mux_session_confirm: sending success reply
debug2: callback done
debug2: channel 2: open confirm rwindow 0 rmax 32768
debug1: mux_client_request_session: master session id: 2
debug2: channel 2: rcvd adjust 2097152
debug2: channel_input_status_confirm: type 99 id 2
debug2: exec request accepted on channel 2
debug2: channel 2: read<=0 rfd 6 len 0
debug2: channel 2: read failed
debug2: channel 2: close_read
debug2: channel 2: input open -> drain
debug2: channel 2: ibuf empty
debug2: channel 2: send eof
debug2: channel 2: input drain -> closed
debug2: channel 2: rcvd ext data 67
[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: debug2: channel 2: written 67 to efd 8
debug2: channel 2: rcvd ext data 18
debug2: channel 2: rcvd ext data 67
Sorry, try again.\n[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: debug2: channel 2: written 85 to efd 8
debug2: channel 2: rcvd ext data 18
debug2: channel 2: rcvd ext data 67
Sorry, try again.\n[sudo via ansible, key=mxraphooaxhpruunceorxkcelobceggf] password: debug2: channel 2: written 85 to efd 8
debug2: channel 2: rcvd ext data 18
debug2: channel 2: rcvd ext data 36
Sorry, try again.\nsudo: 3 incorrect password attempts\ndebug2: channel 2: written 54 to efd 8
debug1: client_input_channel_req: channel 2 rtype exit-status reply 0
debug3: mux_exit_message: channel 2: exit message, exitval 1
debug1: client_input_channel_req: channel 2 rtype eow@openssh.com reply 0

```

I've tried many combinations of ansible\_ssh\_pass and using the password from the commandline (I need to get ansible running so I can get the public keys on these devices), but still it's sudo that isn't working. Sudo does work for the username and password on the device.

The only wacky thing I'm doing here is setting the ansible\_port after the play starts.

```
- hosts: devices
  gather_facts: False
  pre_tasks:
  - name: Get ephemeral port
    local_action: ephemeral_port
    register: ephemeral_port
  - name: Run local ssh command to enable proxy
    local_action: shell /usr/bin/ssh -p {{proxy_port}} -o 'HostKeyAlias=[{{proxy_ip}}]:{{proxy_port}}' -o 'UserKnownHostsFile=/home/user/.ssh/known_hosts' -i /home/user/.ssh/id_dsa -o 'ConnectTimeout=5' -o 'ControlMaster=auto' -o 'ControlPath=%r@%h:%p' -o 'ControlPersist=10s' -f -N -l user -L {{ephemeral_port.port}}:{{silo_private_ip}}:22 {{proxy_ip}}
  - name: Change ansible port to ephemeral port
    set_fact:
      ansible_port: "{{ephemeral_port.port}}"
  tasks:
    - name: test
      sudo: yes
      shell: hostname

```

I even had it working once, but I deleted the old playbook and I can't remember what was in it - this playbook is a lot simpler than the one I started with and I don't really want to start.

If I take out the pre\_task part it works fine too (I just ran it once, copied the port number and hard coded it in as the `ansible_port` in the host\_vars file)

# Answer
> 3 votes
The cure (or bandaid) was to set Pipelining=False in the ansible.cfg file.

This kind of stinks because I like Pipelining and it works, but I guess ansible is gonna treat that first ssh statement as something 'it owns' even though it's a local action.

---
Tags: ansible, ports
---


title: Ansible expected paths error

text:
# Question
Title: Ansible expected paths error
I have a webserver playbook containing the following:

```
- hosts: webserver
  become: true
  roles:
    - apache2
    - { role: demo_app, db_user: demo, db_pass: demo, db_name: demo }

```

My folder structure is

```
.
├── ansible
│   ├── ansible
│   ├── ansible.cfg
│   ├── ansible.pub
│   ├── dev
│   ├── docker-compose.yml
│   ├── Docker.zip
│   ├── env
│   │   ├── ansible
│   │   ├── ansible.pub
│   │   ├── Dockerfile
│   │   ├── init-fake.conf
│   │   ├── ssh_config
│   │   └── ssh_host_config
│   ├── playbooks
│   │   ├── database.yml
│   │   ├── hostname.yml
│   │   └── loadbalancer.yml
│   ├── startup.sh
│   └── temp
│       ├── ansible
│       ├── ansible.cfg
│       ├── ansible.pub
│       ├── dev
│       ├── docker-compose.yml
│       └── Docker.zip
├── ansible.cfg
├── ansible.pub
├── control.yml
├── database.yml
├── demo
│   └── demo.conf
├── dev
├── docker-compose.yml
├── Docker.zip
├── env
│   ├── ansible
│   ├── ansible.pub
│   ├── Dockerfile
│   ├── init-fake.conf
│   ├── ssh_config
│   └── ssh_host_config
├── loadbalancer.yml
├── main.yml
├── playbooks
│   ├── hostname.yml
│   ├── stack_restart.yml
│   └── stack_status.yml
├── roles
│   ├── apache2
│   │   ├── defaults
│   │   │   └── main.yml
│   │   ├── files
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── meta
│   │   │   └── main.yml
│   │   ├── README.md
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   ├── tests
│   │   │   ├── inventory
│   │   │   └── test.yml
│   │   └── vars
│   │       └── main.yml
│   ├── control
│   │   ├── defaults
│   │   │   └── main.yml
│   │   ├── files
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── meta
│   │   │   └── main.yml
│   │   ├── README.md
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   ├── tests
│   │   │   ├── inventory
│   │   │   └── test.yml
│   │   └── vars
│   │       └── main.yml
│   ├── demo_app
│   │   ├── defaults
│   │   │   └── main.yml
│   │   ├── files
│   │   │   └── app
│   │   │       ├── demo.py
│   │   │       └── requirements.txt
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── meta
│   │   │   └── main.yml
│   │   ├── README.md
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   │   └── demo.wsgi.j2
│   │   ├── tests
│   │   │   ├── inventory
│   │   │   └── test.yml
│   │   └── vars
│   │       └── main.yml
│   ├── mysql
│   │   ├── defaults
│   │   ├── files
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── meta
│   │   │   └── main.yml
│   │   ├── README.md
│   │   ├── tasks
│   │   ├── templates
│   │   ├── tests
│   │   │   ├── inventory
│   │   │   └── test.yml
│   │   └── vars
│   │       └── main.yml
│   └── nginx
│       ├── defaults
│       ├── files
│       ├── handlers
│       │   └── main.yml
│       ├── meta
│       │   └── main.yml
│       ├── README.md
│       ├── tasks
│       ├── templates
│       │   └── nginx.conf.j2
│       ├── tests
│       │   ├── inventory
│       │   └── test.yml
│       └── vars
│           └── main.yml
├── site.yml
├── startup.sh
├── templates
├── webserver.retry
└── webserver.yml

```

**Ansible version is 2.2.1**. I'm getting the following error when trying to run ansible-playbook webserver.yml

```
TASK [demo_app : copy demo app source] *****************************************
fatal: [app01]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to find 'demo/app/' in expected paths."}
fatal: [app02]: FAILED! => {"changed": false, "failed": true, "msg": "Unable to find 'demo/app/' in expected paths."}
    to retry, use: --limit @/work/webserver.retry

PLAY RECAP *********************************************************************
app01                      : ok=6    changed=0    unreachable=0    failed=1
app02                      : ok=6    changed=0    unreachable=0    failed=1

```

The relevant task is:

```
- name: copy demo app source
  copy: src=demo/app/ dest=/var/www/demo mode=0755
  notify: restart apache2

```

I've tried altering the above copy src to `/demo_app` and also `src=demo_app/files/app` but this didnt help.

# Answer
The task which fails is:

```
- name: copy demo app source
  copy: src=demo/app/ dest=/var/www/demo mode=0755
  notify: restart apache2

```

If you have a look at your folder structure and more specifically the `demo_app` role, you have:

```
│   ├── demo_app
│   │   ├── defaults
│   │   │   └── main.yml
│   │   ├── files
│   │   │   └── app
│   │   │       ├── demo.py
│   │   │       └── requirements.txt
│   │   ├── handlers
│   │   │   └── main.yml
│   │   ├── meta
│   │   │   └── main.yml
│   │   ├── README.md
│   │   ├── tasks
│   │   │   └── main.yml
│   │   ├── templates
│   │   │   └── demo.wsgi.j2
│   │   ├── tests
│   │   │   ├── inventory
│   │   │   └── test.yml
│   │   └── vars
│   │       └── main.yml

```

The `copy` task will look for the `src` in `demo_app/files/`. So, either change your task to:

```
- name: copy demo app source
  copy: src=app dest=/var/www/demo mode=0755
  notify: restart apache2

```

Or create a `demo` directory in `demo_app/files/` and place the `app` directory in it (like `demo_app/files/demo/app/`).

Setting an absolute path to the directory in the task will also work, but don't think it's an elegant solution.
> 7 votes

---
Tags: ansible
---


# Question
Title: How to get rid of develop branch for simplified Git flow
In a continuously developed web project (not a product) we currently have the following branching strategy, roughly based on *git flow*:

* develop branch: latest working version
* master branch: version to be released / released version
* feature branches: features in development
* hotfix branches: urgent bugfixes in released version

**Master** is read only, updated via pull requests from **develop** or **hotfix branches**. Each update results in a release candidate being built and deployed to the staging system. Release candidates are deployed to production after manual approval.

**Feature branches** are created based off **develop**, or from the last commit that has been merged into master. A pull request from a feature branch to develop is built, deployed to a free test system where integration tests and acceptance tests (automatic & manual) are executed. When successfully tested and reviewed, the PR gets merged, so that it will become part of the next release (i.e. merge from develop to master).

## My goal

I would like to simplify this and get rid of the develop branch. The develop branch has mostly historical reasons and since it is always a successfully tested version, I think it is unnecessary to keep it separated from master. Removing it will also simplify the release process because there is no additional merge anymore.

### I have the following constraints:

* releases are scheduled and should not be fully automated
* while feature branches are typically short lived, some stay unmerged for several weeks (e.g. a redesign) but need to be tested as well (currently as open pull requests to develop)
* sometimes a single feature should be released outside the regular release, effectively turning it into a hotfix. With the current strategy I can rebase a feature branch and merge it directly into master
* it also happens that we need to hold back features after acceptance tests with external systems on staging failed

### Where I am not sure about the transition:

* currently I am building pull requests for testing and merge commits for releases. Can I unify this?
* how to deal with hotfixes when the master is ahead of the latest release. Should I build and deploy releases directly from hotfix branches?
* is there a sensible way to deal with features that should be excluded from a release after they have already been merged? Is a separate develop branch really an advantage for these cases? Most of the time I end up reverting and re-reverting commits manually anyways.

# Answer
> 8 votes
IMHO the problems you're facing are just a side effect of the poor branch strategy you started with: you're effectively plowing new development on `develop` (i.e. what converges towards the *future* production code) **through** the *current* production code on `master`. This leads to contradicting requirements and problems since typicaly the future code diverges from the current one:

* the new development destabilizes production - regressions seen after merging `develop` into `master`
* stabilizing the production slows down future development - you need to stabilize `develop` to make it good enough for merging into `master`

Dropping `develop` won't help (much) - you're not eliminating the problem, you're just transferring the `develop`-specific portion of the problem into `master`.

A better approach would be to move the production **behind** the current/future development, to prevent interference with development for future releases, as illustrated in this image from What is Your Branching Model?:

>

Please note that I'm only referring to the release branches in the above image, not on what's happening in the trunk.

How would this work for you:

* the `develop` branch dissapears, as you wished, absorbed into `master`
* the `master` branch is your trunk, this is where development happens with no speed restrictions (it's **never** merged into production).
* production is/are one or more `release` branches pulled from a `master` label/tag considered close enough to production quality (a short list of remaining todo items can be addressed in that branch if needed). These branches can only receive direct hot-fixes and/or cherry-picked fixes from `master`, they are **never** merged with `master` or other branches
* hotfixes are direct commits into `release` branches

If a hotfix applies only to a production version but not to the `master` it is directly committed to the `release` branch. If it applies to both it's typically committed to `master` first and cherry-picked/double-committed to the `release` branch as well.

Now looking at what goes into `master` (which is past the point where the current `release` branch is pulled off), you have 2 options:

* continue with feature branches just like you do today, except they'll be based off `master`, not `develop`. Converting them to hot fixes remains possible - you'd just have to rebase them to the corresponding `release` branch instead of `master`
* switch to continuous integration and reap its benefits (can be done at any time going forward), including in a progressive manner - gradually pull fewer and fewer feature branches.

If you like this approach here's how you get there from where you are today:

* establish a release naming strategy:
    + you can't have an ongoing release branch with the same name
    + you can't (shouldn't actually) rebase or sync a production release branch
* pull a `releaseX` branch from `master` immediately, following that naming strategy
* stop commits from going into `develop`, they'll be soon going straight to `master`.
* merge `develop` into `master`
* instruct developers to rebase their workspaces on `master` instead of `develop`.
* open `master` for commits
* delete `develop` if you desire (or leave it permanently locked/read-only for reference)

# Answer
> 11 votes
Let's say you take out the master branch (you can rename develop to master to confuse your team if you like later) and simply use tags for releases either on develop or hotfix branches. You took out a branch, but the difference is just a change in syntax. Change for change sake.

Now let's say you actually take out develop with keeping the locked master branch. What will happen is that integration of code will slow, it will live longer separated in feature branches, especially close to release dates. This will Increase the difficulty of merging each time and slow the process down.

Within the constraints you have laid down I don't see any positive effects of making such change. It would require relaxing the constraints, especially the first one.

# Answer
> 5 votes
You are already building and testing code on each of the pull-request and hot-fix branches. This means that in aggregate, the sum of all branches pending on pull-request are your virtual `develop` branch.

You can create a system when in a test environment, several pull-requests are cherry picked into a temporary branch that is not published to the main repository. This branch is used to integrate a test environment that includes `master` and several additional pull-requests, but once the testing is done, this branch is no longer available anywhere.

When you create a release from `master`, you would usually create a tag on that release. Later hotfixes can use that tag to create a new hotfix branch from which a deployment will be made, even though the edge of `master` is already ahead. On this hotfix branch you would probably also tag a minor release, and make sure that the changes were merged into `master`.

Removing merged features from a release is quite hard to do with git. The best mechanism for this would be to use `git revert` on the merge commit. But that makes it almost impossible to get these features/changes back, and history becomes all muddled.

A much better way to handle separation for deployment of code, and release of features, is *feature flags*. If your developers can hide their features behind some conditions in the code itself, you could deploy their code, but turn off the feature. This is a separate topic, but a lot of information about this exists (including a Q&A on devops.SE).

# Answer
> 3 votes
Well @dan-cornilescu says it well for your particular problem, but the more general case for Trunk-Based Development (mentioned in the Continuous Delivery, Lean Enterprise, and The DevOps Handbook) is made here: https://trunkbaseddevelopment.com/

---
Tags: git, release-management, builds, branch
---


# Question
Title: Tool for storing per environment configuration
I have a requirement to store configuration information on a per environment basis in a tool.

This is a tool with a GUI for adding/updating configuration values (e.g connection strings). This should have a default value and be able to change this based on different environments.

There should be an API to retrieve these configuration values during deployment to a particular environment to add to the application.

I have searched for a while and can't see any tools that would fit this bill. Are there any suggestions?

**Note**: Currently the settings are in TeamCity variables and deployment is via PowerShell scripts.

# Answer
There are many tools that can do something like this, including configuration management tools like Chef, Ansible, or Puppet; and KVS tools like Consul and etcd. You could also integrate it as a build step in your CI server, or sidestep the issue using live configuration at runtime against an external configuration store (again, something like Consul or etcd, or any database at all).
> 6 votes

# Answer
Possibly a different repo? One with branches for QA, UAT, Prod (of nor more). A repo that's different to your regular "Code as Code" and "Infrastructure as Code" repos.

It is highly nuanced. How much config per env. Is it toggled between releases? Should those toggle states maintain state despite binary deploys. Which of client, customer, guest or user to you maintain config for?

I wrote a bunch of blog entries (and prototypes/demos) on the topic over 5 years - including UIs for the toggling (if you need them).
> 1 votes

---
Tags: deployment, configuration-management, configuration, software-recommendation, casc
---


# Question
Title: Is Continuous Delivery the Control Framework for DevOps?
I've been reading Jim Bird's DevOpsSec book, one of the statements in Chapter 4 - Security as Code is as follows:

> Agile ideas and principles—working software over documentation, frequent delivery, face-to-face collaboration, and a focus on technical excellence and automation—form the foundation of DevOps. **And Continuous Delivery, which is the control framework for DevOps**, is also built on top of a fundamental Agile development practice: Continuous Integration. *(O'Reilly 2016, ISBN 9781491971413)*

My understanding of a control framework was that it declares certain Control Objectives such as:

* Payments are made only for authorised products and services received.

This may be satisfied by some process built into the system, and would likely have a number of checks and balances; however, you wouldn't say that that process *is* the Control Framework.

To me it feels that that sentence should read: "And Continuous Delivery *satisfies* the Control Framework for DevOps".

Am I totally wrong? Is Continuous Delivery the Control Framework for DevOps?

Or, am I right and Continuous Delivery is not the control Framework for DevOps?

# Answer
> 3 votes
In my opinion you are right, Continuous Delivery (CD) is not the control framework of Devops, at least it is not the only possible one.

But in the context of the book you're quoting it gets the most used possibility when you start to include security baseline and assessments as part of the product delivered.

In a security context, you'll add smoke tests in a post deploy phase to ensure your application is not subject to XSS or Sql injection for example. If any of this tests fail, block any deploy in other environment, mark the deploy as failed and eventually rollback to previous version.
This enforce the security rules to be satisfied, and in this way act as a control framework to ensure production deploy are 'safe' for known vulnerabilities.

# Answer
> 2 votes
I would say 'Yes' in the context of 'Security as Code', e.g.:

As the author stated, if I have continuous delivery that means I have continuous integration, which implies I can easily integrate and propagate change throughout the system, and likely this is due to prescriptive code-based configuration.

This alone could be a form of control as you can hopefully easily propagate compliance related changes across your infrastructure

However, I think configuration management exemplifies compliance controls in a devops lifecycle. Look at Chef\DSC and InSpec\ServerSpec. You can run configuration management during the bake phase to set initial configuration and then run it on a schedule after deployment as a form of compliance to eliminate drift. InSpec\ServerSpec tools sit outside that CM and 'audit' the state of the systems to ensure the CM is doing its job.

---
Tags: terminology, control-framework
---


# Question
Title: What do businesses that employ multiple development teams use to implement DevOps in a structured manner?
Adding DevOps practices to a single development team is a relatively simple proposition as it usually comes down to introducing either people to the team to build up a DevOps, Release Engineering or Platform Engineering capability within an organisation.

Some of the pitfalls organisations are trying to avoid are:

* Inability to report progress consistently across multiple development teams, for example, if one team reports their progress in terms of the number of equally sized stories completed in 2 weeks; whereas another team reports in storypoints completed in a three-week sprint.
* Inconsistent application of a practice means team members can not easily move between teams within the same organisation without learning new ways of working.
* Mismatches in development cadence across multiple teams makes coordinating dependent releases difficult.

What can be employed to achieve this in a consistent manner in an organisation that has a large number of development teams, potentially in multiple separate business units?

# Answer
> 6 votes
Large enterprises will commonly adopt a delivery framework, or operating model, that has been built to support large enterprises. From an Agile/DevOps perspective, which is my area of expertise there are three frameworks of interest:

* **Scaled Agile Framework (SAFe®)** defines DevOps as delivering business value through the successful operation of software in an environment. This is achieved by giving early attention to the act of "moving into production" and accomplished, in part, by integrating personnel from the operations function within the Agile Teams.
* **Disciplined Agile 2.X**, also known as DAD, defines DevOps as "DevOps is the streamlining of the activities surrounding IT solution development and IT operations." this is achieved by building feedback loops between the development and operations functions within the business. DAD goes on to describe Disciplined DevOps as the intersection between the following roles: Enterprise Architecture, IT Operations, Business Operations, Release Management, Data Management and Development.
* **Large Scaled Scrum** does not specifically call out DevOps, Release Engineering or Platform Engineering; however, it does describe a methodology for scaling out Scrum in an organisation that is growing fast, integrating a DevOps specialism within these teams is both logical and well understood.

As far as I am aware there are no statistical data about the enterprise adoption of each of these frameworks and whether they have adopted DevOps practices in addition to Agile practices, however from experience many organisations appear to gravitate towards Scaled Agile Framework.

Kristof Horvath discusses the application of DevOps in these enterprise-grade agile frameworks in his article: Scaling Agile in Large Enterprises: LeSS, DAD or SAFe®?.

It is worth noting that adopting an "enterprise-grade" framework is not a requirement in an enterprise, at the end of the day Agile, DevOps and Lean are about adopting the right practices for your organisation, not the most popular ones.

# Answer
> 2 votes
None of the issues you present appear to be DevOps-specific to me.

> Some of the pitfalls organisations are trying to avoid are:
>
> * Inability to report progress consistently across multiple development teams, for example, if one team reports their progress in terms of the number of equally sized stories completed in 2 weeks; whereas another team reports in storypoints completed in a three-week sprint.

This is a problem any company with multiple development teams, or multiple teams of any sort that share a common functionality, will face. The simplest answer is to move them all onto a shared cycle.

> * Inconsistent application of a practice means team members can not easily move between teams within the same organisation without learning new ways of working.

Again, not a problem specific to DevOps. As with developer teams, you need to increase visibility into other teams and introduce standards to keep people in sync.

> * Mismatches in development cadence across multiple teams makes coordinating dependent releases difficult.

Once again, not DevOps-specific, but a classic problem of business that comes from far before electronic computers. Reduce dependencies and make it clear when they happen that they are happening, and the effects of slippage, and keep all affected parties informed of schedule changes.

# Answer
> 2 votes
> Inability to report progress consistently across multiple development teams, for example, if one team reports their progress in terms of the number of equally sized stories completed in 2 weeks; whereas another team reports in storypoints completed in a three-week sprint.
>
> Inconsistent application of a practice means team members can not easily move between teams within the same organisation without learning new ways of working.

I view this as both a failure to standardize process across teams and a management problem if people aren't sticking to process. Individual teams should be flexible in terms of how they manage their people, but processes should be transparent and followed.

Imagine a fast-food chain restaurant. Each restaurant may re-organize things a bit according to the architecture of their specific location, to best suit their needs. However, the general components and the ingredients and the intended SLAs for delivery will be the same. An employee should be easily able to transfer between any location with some minor re-orientation.

Challenges faced by each 'location' means you may need to regularly re-visit the overall strategy, and you should be 'trying things out' in various teams and regularly rolling those changes back into the overall process. Process is just like software delivery, too.

> Mismatches in development cadence across multiple teams makes coordinating dependent releases difficult.

Again, need authoritative control and agreement between teams... a corporate brand with many locations, not a food court of different franchises.

Etsy has their famous example of 'schema change thursday', where they allow non-contract-breaking-changes to be deployed as often as needed, but anything that will require coordinated change between systems happens on a specific day every week.

---
Tags: operating-model, methodology
---


# Question
Title: Are there any best practices and/or tools for 'nohup' monitoring?
I use `nohup` for starting Ansible playbooks inside the Master which is spawned via boto3 from AWS Lambda.

This playbook creates multiple workers, make them do some tasks, and terminates them.

So, I want to design/build a reliable monitoring system for the same. For now, I am writing the logs of `nohup` to a `tasks.log` file inside master, and pushing it to a store(for now, S3) once the Ansible plays are completed.

**But, I wanted to know whether there is a much more elegant (and/or) better method for doing the same? And also, whether there are good tools which would help me in the process?**

# Answer
> 8 votes
I'm not sure it's a best practice but what I would do is using filebeat to read the log in near real time and push it to ElasticSearch.
AWS provides an ElasticSearch Service which can save you from this part.

The drawback of using filebeat directly to elasticsearch is that the time of events will be the time the prospector read them and if your logs have a timestamp they can be "ingested" in different order than produced (from multiple workers).

You can add Logstash inbetween, sending from filebeat to logstash and then to elasticsearch. Logstash is able to interpret your logs and split them into fields to search again, in the same manner you can parse your date in log so events will be stored with a correct timestamp.

You may use logstash directly to read the logs, but it's quite heavy on resources so I would keep it separated to no impact the workers.

If you buy a license from x-pack you can add alerts on messages directly from elasticsearch.

If you want to save you from the burden of maintaining logstash there's services like logmatic.io which provides you with the full stack, you just have to push your logs and a bunch of parsing are handled from stock.

---
Tags: ansible, aws-lambda, monitoring, practices
---


# Question
Title: How to synchronize deployments (especially of database object changes) on multiple environments
I am the DevOps engineer and a software engineer in a team where months back, the developers moved from having a central Oracle DB to having the DB on a CentOS VM on their individual laptops. The move from a central DB was to reduce dependency on the DBAs and also to eliminate issues that stemmed from inconsistent data.

The plan for sharing and ensuring synchronization of the Database with everyone on the team was that each person will share change scripts with everyone. The problem is that we use Skype for communication (we just setup slack but are yet to start using it fully), and although people sometimes post the text of DB change scripts, it could be missed by some. The other problem is that some developers miss posting the changes. Further, new releases are deployed in Production without being deployed on the Test and Demo environments.

This has posed a serious challenge for us, especially myself who of recent, became responsible for ensuring that our Demo deployments were in sync with the Production deployments. Most of the synchronization issues border on the lack of sync of the Database due to missing change scripts or missing DB objects. Oracle is our DB of preference.

A typical deployment in the Demo environment is a very painful process that involves testing an application and as issues occur due to missing DB table columns, functions, stored procs, we have to look for the missing DB objects, apply them to the DB and then continue until all issues are resolved.

How can I solve this problem to ensure smooth, painless and less time-consuming deployments? Can migrating our applications to Docker help with the DB synchronization issues and the associated lack of discipline of the developers? What process can we put into place to improve in this area?

# Answer
I would integrate the schema management into the application itself (or along with it).

Any change to the schema should be committed along the application code (and hence tagged also).

There's already a bunch of possibilities listed in this question: What practices or tools enable Continuous Deployment of Databases

With this kind of tools, using an in memory database like h2 or a central database and having the application check the schema version (stored in the DB) when started allow to get an updated database schema at application startup.

This will not solve your discipline problem and may create new ones, as this usually comes with a strict versioning of the schema, enforcing the version can be done with pre-commit hooks but that's a hard work.
> 9 votes

# Answer
In our company we manage our application code within a VCS (Git) and most of the applications we work with install their core database from within the setup scripts that ship with the application.

If we would have to extend or customize an application for one of our customers which involves having to extend the database or do database customizations, then we will accompany our code with a setup script that will perform the database operations through the application's database layer/wrapper/driver.

This way, once the code is merged into the VCS repository, every developer that will check out this code, will retrieve these database changes. Also after the code is merged, these database changes will end up on every environment (test, UAT and production nodes) and this way all database changes will roll out smoothly.

This is an easy to implement method. You could also dive into source control tools specific for databases if you need more advanced management possibilities.
> 4 votes

---
Tags: deployment, continuous-delivery, databases, environments, flyway
---


title: Simple CI/CD Containers in AWS

text:
# Question
Title: Simple CI/CD Containers in AWS
I am using AWS Code Pipeline, Code Build to create a new Docker container and push it to ECR.

My application is a simple straight-forward single-container based. What would be less friction approach to pull down current running Container and re-launch a new Container from the ECS registry (output of Code Build thru Code Pipeline).

I tried CloudFormation with EC2 user data, custom scripts on one side and CloudFormation with ECS with task definition on the other side (not successful yet). I strongly feel there must be more obvious and simpler approach.

# Answer
I would keep the ECS container instances (I'm talking about the *Docker hosts* \- I don't like AWS terminology here) and the deployment as two separate things.

Get your ECS stack up and running. You can manage it through CloudFormation and Auto-scaling groups, that's fine. Just think of your cluster as a platform where you will deploy *to*, not something you need to *redeploy*.

Then, for CD, the easiest method by far is to update the service definition to use a new task definition and let ECS rolling update the containers for you.

Every time it start a task, ECS will run docker pull image:tag even if it has the image locally to make sure it has the latest version of that image:tag. So the image tag you use really don't matter (there is no need to change the tag on every build).

That means that you can build myimage:latest over and over in order to deploy it easily.

What you need is a task definition where the image = myimage:latest. Create a service with that task definition and every time that ECS start a task (an instance of your service) it will be the most recent "myimage:latest" you have built.

From there, you are missing only one piece in the puzzle, from CodeDeploy, you can call something, perhaps a lambda function, to create a new revision of your task definition and update your service and ECS will automatically create new tasks for that revision and remove the old tasks.

An example:

Let's assume you have created a service called MyService. That you have configured that service to run 2 tasks for the task definition MyTaskDefinition:1 (revision 1). In that task definition, you have one container definition which image is set to "myimage:latest".

1. Yesterday you have build myimage:latest which had the ID (SHA) 365d8f7bf565.
2. Your container instance ABC is running a task named MyTaskDefinition-**1**-containerName-someLongId. when you inspect that container, it's running the image "sha256:365d8f7bf565.........."
3. Your other container instance DEF is running another task. It has a similar name (only the ID differ), but it's running the same image.
4. You push a change to your repo.
5. CodePipeline picks up that change, build and publish the image to ECR.
6. That new Docker image is also myimage:latest, but its ID (SHA) is f7ec5e54ac96
7. Now you need to add a step to your pipeline to use Lambda functions and the AWS NodeJS SDK to do some calls to your cluster:
    1. Create a new task definition (which will be exactly the same as before). That will be MyTaskDefinition:2
    2. Update your MyService to use MyTaskDefinition:2 (instead of 1)
8. ECS will create new tasks. The container names will MyTaskDefinition-**2**-containerName-someLongId. When you inspect those containers, you'll see that they will be running "sha256:f7ec5e54ac96.......". Perhaps you'll have 2 tasks on the container instance ABC, perhaps they will be sprayed out (that depends on your service's configuration)
9. After some time ECS will remove the old task MyTaskDefinition-1-containerName-someLongId from ABC and DEF.

Note: you don't actually need to create a new task definition. If you want you could instead retrieve the service's tasks list and manually stop them one by one. You should wait for ECS to restart a task before stoping a new one (that is: stop the first container, wait for ECS to replace it, stop the second container). When ECS restart the container, it will grab the most recent myimage:latest built, as explained before. I just think creating a new task definition is easier and less error prone (no logic required to wait and check, ECS will handle the rolling update for you if you have a new task definition).
> 18 votes

# Answer
For simple use case described I would suggest checking Elastic Beanstalk for Docker, it's not the minimal solution like bare ECS usage, but you can benefit from auto-managed and configured services like ELB, EC2 AutoScale, health monitoring and much more.

High-level summary:

1. Configure Elastic Beanstalk to use specific tag myimage:tested
2. Use Code Pipeline/Build to build, test and promote tag "tested"
3. Trigger Elastic Beanstalk deployment, which will pull promoted image myimage:tested to all instances, different deployment strategies available.

This approach based on re-using same tag, alternative approach would be generating tag with build id, for example myimage:tested-42, this will require updating Elastic Beanstalk each time with new tag, but gives more granular control on deployed revision.
> 3 votes

# Answer
I second elastic beanstalk for its simplicity; it is very easy to setup and deploy.

If you are familiar with docker-compose, another approach would be to define docker-compose.yml and directly deploy on ECS with ecs-cli.
> 0 votes

---
Tags: continuous-integration, containers, continuous-delivery, amazon-ecs
---


# Question
Title: Where can I find professional insights into the impact of DevOps upon SecOps?
I'm writing about the arrival of DevOps practices such as continuous integration (CI) and continuous delivery (CD) and their impact upon SecOps (or whatever the equivalent team was called). My audiences are SecOps and DevOps professionals who are familiar with industry tools and terminology.

I'm looking for a resource, such as interviews with Security professionals, that describe the impact this had upon security practices. Specifically, the context would be small to medium-sized organizations (approximately 25 to 5000 FTEs) with one or more dedicated software development teams that reorganized as DevOps and implemented CI/CD.

Also, if you would like to share your experiences, I would appreciate hearing about them here. Please describe the organization at a high level; which DevOps practices were adopted/discontinued; which SecOps practices were adopted/discontinued; and particularly any key challenges and how they were overcome.

I know some SE groups are sensitive about receiving open-ended questions like this. However, I think it would be valuable for other visitors to read about and generalize from your professional experiences. If you feel like this question should be placed in a different forum, please let me know and I'll be glad to move it there.

# Answer
> 4 votes
As alluded to by Tensibai ♦, I think the answers you are looking for start from my answer to "What is SecOps?", specifically from the "Further Reading" section:

Of the above I am currently reading Jim Bird's book, have had the pleasure of working with the organiser of DevSecCon in a previous role and had the privilege of attending the Velocity Session in Amsterdam last year.

I would advise you to reach out to Francois Raynaud and ask the question of him, he has worked in the space for a long time and is really helping the community evolve as part of DevOps.

Beyond my links, the O'Reilly Security Conference has published videos on Safari Books Online:

---
Tags: continuous-integration, continuous-delivery, continuous-deployment
---


# Question
Title: How can I autoscale instances based on Kafka topic lag?
I have autoscaling groups that I want to dynamically scale up/down based on the amount of lag showing in certain Kafka topics. There's a 1:1 relationship between topic and autoscaling group that needs to be scaled. I'm fairly sure I can't use CloudWatch metrics to approach this.

Is there a way I could accomplish this using Jenkins without creating race conditions and other issues?

Note: My application is not constrained by CPU, Disk I/O, or any other metric provided by CloudWatch; the constraint is the application's ability to process from incoming Kafka topics.

# Answer
> 7 votes
CloudWatch is absolutely the way to go on this front, you can surface Kafka Metrics in CloudWatch either by:

1. Having a separate process that pushes the metrics into CloudWatch.
2. Have your Producers, Consumers or Stream Processors push the metrics you need into CloudWatch.

The process you follow will broadly be broken down into:

1. Publish Custom Metrics.
2. Create an AutoScaling Launch Configuration.
3. Create an AutoScaling Group and Policies based upon your Custom Metric.

Depending on your cost model I would also consider how you can use spot instances to keep the costs down, i.e. if you have non-critical but high compute workloads use Spot Instances as part of your autoscaling configuration.

---
Tags: jenkins, amazon-web-services, autoscaling
---


title: What is an artifact repository?

text:
# Question
Title: What is an artifact repository?
What is an artifact repository? If it's just a place to store files, can't I just use a source control system?

# Answer
> 40 votes
During development you generate a fair amount of different artifacts. These might include:

* The source code
* The compiled application
* A deployable package
* Documentation

and potentially others as well

While you could use a source control system to store all of them, it's usually massively inefficient, as source control systems are usually designed to handle text based files, and not binary files. You might be able to use them as a simple storage mechanism, if most of your releases are text based, and you don't have to store a lot of binary data.

Artifact repositories however are designed to store all kinds of files, including binary ones. This includes anything from zipped up source codes, to build results, to things like docker images as well. Also, they usually not only store these artifacts but also help manage them using various additional functions, for example:

* Versioning support: properly store some metadata, like when each artifact was built, what their version number is, store their hashes, etc.
* Retention: make sure you only keep the important artifacts, and automatically delete ones that are only snapshots / not needed anymore, etc. based on various criteria you can set up
* Access control: set up who can publish and who can download the various artifacts
* Promotion: ability to promote artifacts. For example you can have snapshot artifacts with a short retention period on a server near your coders, and a separate repository near the live servers, where only artifacts that have been deemed deployable appear. This also includes support for various version channels, and moving artifacts between them (like promoting a specific version from beta to stable).
* Act as a native repository for the artifacts. Meaning you can use it as the main repository for maven, rubygems, docker, etc. This can also include caching of artifacts from the official repositories as well.

# Answer
> 7 votes
There's repository managers and Universal package repository managers (UPM).

UPM’s can store all your build artifact for Jenkins, teamcity etc. and can generally also act as repository mangers for many different types of binary artifacts Maven, npm, NuGet and more.

These would be tools like Jfrog Artifactory, Inedo ProGet, and Sonatype Nexus.

A pretty decent comparison is here: https://binary-repositories-comparison.github.io/

---
Tags: terminology, continuous-delivery, continuous-deployment, artifacts
---


# Question
Title: What does a "DevOps Transformation" contain?
Some consulting companies are promoting a service called "DevOps Transformation". Multiple big companies are talking about the subject at conferences and meetups around the world.

What does such "DevOps Transformation" entail? How does it look like in actionable terms, both for successful transformations, and failed ones.

# Answer
> 14 votes
I need to put my answer to this question in the context of what DevOps is, more specifically within the DevOps transformations I have been part of, DevOps is the ownership of the full Software Development Lifecycle. All of the practices in the chart are an important part of DevOps, and they enable and underpin both Systems Thinking and Amplification of Feedback Loops.

However, the key differentiator between CI/CD and DevOps is the actual operation of software in a production environment, where it can deliver value to its customers and the business it serves.

As a consultant participating in or leading a DevOps transformation I have the following aspects in the front of my mind:

* **Culture**: As Dave quite rightly pointed out a Culture of Continuous Experimentation and Learning is critical to the success of any transformation. From a DevOps perspective this comes down to how do we engender a culture that is supportive of the chosen DevOps model, this model could be "You Build It, You Run It" or it could be more along the lines of Google's Site Reliability Engineering practice.
* **Operating Model**: This is the part of the business proposition that articulates how the organisation will be delivering value, generally by articulating the People, Process and Tools used tied together at a high level. Without an operating model, you have no blueprint for the way the organisation adopt the practices that the culture defines, this, in turn, leads to a lack of clarity and divergent behaviours.
* **C-Level Aircover**: It is often our job as consultants working within transformation programmes to make radical changes to the way business works. You are going to upset people, and some people won't like the changes - it is important that you have "air cover" from above to change things and move forwards.

Once the high level is in place it is important to find something you can realistically deliver:

1. Start as small as possible, ideally, once you have some people who understand the culture, a sketch of an operating model and buy-in from the executives create the "Minimum Viable Project", don't attempt to boil the ocean by introducing DevOps to an audience of thousands. Set an achievable goal:
    * Automate the creation of the infrastructure from Product X.
    * Automate the delivery of Product X into Azure across all environments.
    * Hand-back support from outsourcer Y to a development team in London.
    * Create a set of tests around our riskiest feature and run them in continuous integration.
2. Great you have some success under your belt now it is time to start baking this into more teams, add another couple of teams into the mix and get them up and running. Don't be afraid to offer "White Glove Support" at first to assist them in the transition; they will need a lot of hand-holding over the coming weeks and months.
3. Now you have several early adopters following a new way of working; you have some critical mass, it is time to start evangelising the work you are doing with a wider audience:
    * Hold regular **show-and-tell** sessions ask the early adopters to demonstrate how successful they have been.
    * Offer **drop-in sessions** to allow other parts of the organisation to explore how they can come on board with your team.
    * Enable the creation of **Communities of Practice** focusing on specific disciplines: Continuous Deployment, Automated Testing, Business Communication, Risk Management, Monitoring and Alerting, etc.
4. Stay the course and close off the transformation by onboarding the rest of the organisation. Understand the relationship between the Gartner Hype Cycle and the Adoption Lifecycle. Prepare for the Transformation Programme to fall into the "Trough of Disillusionment", stay the course and keep the end goal in sight.

For a deeper exploration of the final point read Geoffrey A. Moore's Crossing the Chasm. I could quite literally write a book about how to deliver a DevOps transformation, however by the time I had finished it there would probably be no more DevOps transformation work for me to do.

# Answer
> 10 votes
DevOps tends to break down across three major dimensions:

**Culture**
DevOps culture emphasizes high levels of trust, collaboration and communication between *all* stakeholders, especially Dev, Ops, and Security. The natural tension and competition between these groups creates friction, and often dysfunction. DevOps is (arguably) first and foremost about aligning efforts between these teams.

**Process**
DevOps development processes align closely to Agile processes. Ops is encouraged to take up Agile-like practices to better align to Dev efforts. DevOps-aligned processes are designed to support high velocity and fast feedback loops throughout the development/delivery lifecycles. Continuous Integration, Continuous Delivery, and Continuous Improvement (kaizen) are focus areas of DevOps process.

**Technology**
DevOps isn't a tool, but it is supported by tools. There are whole families of tools supporting a range of areas including Continuous Integration, Source Control, and Application Lifecycle Management.

A "DevOps Transformation" must address elements of all three, but not necessarily all equally at the same time. There is a natural progression and "critical path" for transformation. The argument could be made, for example, DevOps is dependent on some form of Agile practice, at least within the Development team/teams. Issues with culture may need to be addressed before investment is made in tooling.

References:
Culture: https://www.andykelk.net/devops/using-the-westrum-typology-to-measure-culture
Technology: https://xebialabs.com/periodic-table-of-devops-tools/

---
Tags: terminology, culture
---


# Question
Title: Why does `docker-login` fail in BitBucket pipelines?
**Aim**

to login to a private docker registry using BitBucket pipelines.

**Attempt**

Based on this documentation the following bitbucket-pipelines.yml:

```
image: docker:latest

pipelines:
  default:
    - step:
        script:
          - docker login -u $D_REGISTRY_USER -p $D_REGISTRY_PASS $D_REGISTRY_URL

options:
  docker: true

```

was created. When BitBucket pipelines run this it results in:

```
+ docker login -u $DOCKER_REGISTRY_USERNAME -p $DOCKER_REGISTRY_PASSWORD
$DOCKER_REGISTRY_URL
Warning: failed to get default registry endpoint from daemon
(Error response from daemon: authorization denied by plugin pipelines: Command
not supported.).
Using system default: https://index.docker.io/v1/
Error response from daemon: authorization denied by plugin pipelines: Command
not supported.

```

# Answer
At the moment the `docker login` and `docker push` commands are only available to Pipelines Alpha members.

You can read about the features available to Alpha at "Pipelines Alpha".

In order to participate, the above link includes instructions on how you can opt-in to the added functionality.

Personally I received an e-mail from Atlassian a day after signing up saying my projects now include Alpha features. So this feature works in my pipelines without any issues.
> 4 votes

---
Tags: docker, bitbucket
---


# Question
Title: Why trigger something remotely (using Jenkins) and then just forget about it?
When using Jenkins to trigger some process (eg a remote build) at a remote location, why would you ever want to "***fire***" something, and then just "***forget***" about the outcome of what got triggered remotely? At least that is what "fire and forget" seems to be about.

Such "fire and forget" sounds like not respecting some of the basic rules in IT: in whatever process (or program, etc) you write, try to always be prepared for unexpected conditions (return codes, etc) that may arise. So that you'll also have appropriate error handling in place.

To the extend it does make sense, in the (mainframe) SCM tool that I'm most familiar with, you'd start some FTP process from a dev environment to a remote target (eg to distribute and activate executables). And sooner or later (asynchronously) somebody gets some kind of acknowledgement back (from the remote target to the dev site) that indicates "success or failure". And if within a reasonable amount of time no feedback at all is returned, you know that something is not working as it should. Using such scenarios, you'll never *forget* what got *fired*.

Any examples for which using "fire and forget" in Jenkins does make sense?

# Answer
It seems like ***forget*** in this context is not really what it sounds like. Instead it is rather something like "*`Don't keep me waiting for the process to complete, so that I can do something else while the process is running. And when the process is done (succeeded or failed), we'll continue processing with whatever is next`*".

An example '*for which using "fire and forget" in Jenkins does make sense*' (as in my question) is creating parallel chains to reduce the duration of pipelines, and combine it with using the Join plugin. Some more details about this plugin (from the linked page):

> This plugin allows a job to be run after all the immediate downstream jobs have completed. In this way, the execution can branch out and perform many steps in parallel, and then run a final aggregation step just once after all the parallel work is finished.
>
> The plugin is useful for creating a 'diamond' shape project dependency. This means there is a single parent job that starts several downstream jobs. Once those jobs are finished, a single aggregation job runs.
> 1 votes

# Answer
I wouldn't describe Jenkins jobs as "fire and forget." They are designed to run automatically without the need for human interaction, however, that doesn't mean the result of a job is not visible. Quite the opposite- detailed logs of all activity are available for jobs ranging from command-line tools to custom deployment jobs.

The testing results from Jenkins jobs are routinely displayed on large-screen TVs all over the Development floor of many companies I've seen. There are whole plugins devoted to making this type of information available to developers, Ops, and management in a dashboard format.
> 6 votes

# Answer
In this case, "Fire and Forget" doesn't mean what you think it means.

It isn't the case that you fire the build and then forget about the outcome. What actually happens is that you fire the event, and then forget about what the process is doing up until the point where the process returns feedback to you and reminds you about what was fired.

As an example, the old way of doing things might be to trigger a build and then let it run while you watch the output. You sit there watching the results of the build as they occur and don't work on anything else productive during that time. Or you do work on something productive, but you have one eye on the build process. When it is done, you need to either be paying attention, or remember to check on it to see the results and then continue based on that.

In Jenkins model of "Fire and Forget", you have some automated process do the build for you, and your mind is not focused on the build process until something goes wrong, or the build completes. At that point, you get a message from Jenkins, either as an email or in a program like slack, which now reminds you of the build process and tells you all the information you need to know to move on. In the meantime, you were working on some other task with your full focus, because you knew that you didn't have to keep an eye on it. The automated system would alert you to anything you needed to know.
> 6 votes

---
Tags: jenkins
---


# Question
Title: How to manage Free and Open Source Software dependencies?
We exist in an increasingly complex ecosystem of Free and Open Source Software, FOSS, and it's dependencies. Having done a bit of analysis on one medium size project there are over 1,500 dependent software packages, not counting different versions of the same package or any packages developed internally for reuse.

How do we manage this complexity in an enterprise that needs to be able to answer the following two questions categorically

* Have we met the licence terms of all of the software we are using?
* What is our exposure to vulnerabilities in free or open source software?

I can think of two architectures that would support the answering of these questions, however, the enormity of the problem could well be clouding my judgement:

## Approach #1: Walled Garden

Effectively firewall off the sources of these open source packages, i.e. npm, Docker Registry, nuget, etc., then create an internal repository of approved packages, implementing some process to whitelist packages.

## Approach #2: Audit

Allow packages to be downloaded from the internet freely, however, perform source code analysis as part of the build pipeline to report on the packages currently being utilised.

Practically an organisation could use *either* solution or if necessary *both* solution to provide a degree of checks and balances.

Are their other architectures or tools that support the management of Free and Open Source dependencies?

# Answer
> 3 votes
Any organization that needs the level of control you've specified is essentially forced into the preapproval process ("walled garden"). It's a pain for developers, but it's a necessary process to keep the company safe. I've even heard of companies with programs that scan all their workstations regularly to find and remove unapproved software.

It's worth noting that there are a few ways to implement it. As you note, you can use an internal mirror. Another popular option is to *vendorize* dependencies into your application's repo.

Security is one of the main issues with a frozen version approach, as you've taken on responsibility for maintenance. This generally means having someone whose responsibility it is to follow the release discussions of every package you've frozen. It also requires allotting continuous time for updating dependencies, because otherwise it's very easy to get caught with a security vulnerability that's only been fixed in a version that's incompatible with your app.

In general, I find that companies often underestimate how much additional work they're taking on with this change, and strict compliance is actually much more costly than occasionally ending up with licensing or "disappearing dependency" problems.

# Answer
> 2 votes
There are enterprise tools that manage these opensource dependencies namely Jfrog Artifactory with the Xray feature and Inedo ProGet with features for license filtering and vulnerability scanning.

Basically, allows you to restrict or permit download of package so companies can set policies to ensure development isn't breaking rights of use or building application with vulnerable components. These tools also have local caching features so that if a dependency gets removed or edited (think left-pad) then development isn't impacted.

\__

Disclaimer, my day job is at Inedo :)

---
Tags: dependency-management, open-source, build-pipeline
---


# Question
Title: Separate SQL Server or just separate SQL Database for test and production?
I'm new to SQL Server, so this may be more of a SQL Server management question.

I'm creating test and production databases for a service, and I imagine I'll be wiping the test database quite a lot. Additionally, I'm going to want different replication and audit logging strategies.

Does it make sense to have both SQL Databases on the same SQL Server, which seems to be what the Azure Portal makes easiest to manage, or does it make more sense to create a separate logical SQL Server for the test and production databases?

# Answer
> 8 votes
Go with a separate logical server:

* It is going to cost the same because you are charged on the **tier** of the database not the database server.
* By providing separate servers you provide **isolation** between the two databases protecting one from the other.
* Unless you use contained users your users will be **shared** between Test and Production, if you happen to use the same user for both databases a change of the password in production would also require the same change in Test.

You might want to use the same server for a database if:

* You are creating a very large number of Databases **across many Dev/Test** environments as there is a limit to the total number of SQL Servers.
* You have two databases that need to be deployed as part of the same ARM template.

# Answer
> 6 votes
Putting them both on the same server is certainly the easiest. However, I'd argue in a business situation you shouldn't be *able* to do it.

A production database is one of the most important resources of a web app. It likely has much confidential data in it. It's also probably critical to the site's stability. So the production database credentials should be tightly controlled, and firewall rules should prevent access to it except from a small number of vetted machines. In addition to security considerations, the firewall rules also help prevent accidental test operations on the production database - if it can happen, it will sooner or later.

Edit: Shortly after I originally wrote this answer, Digital Ocean (one of the largest hosting providers) had an outage that would've been prevented if they had segmented their network appropriately. Learn their lesson instead of repeating the mistake.

---
Tags: sql, azure
---


# Question
Title: How should I store my environment variables?
This is a very broad question about methods and advice regarding environment variables / structure. But ultimately I'm looking for answers for the very specific question of 'How should I store my environment variables?'

Firstly some clarifications:

* An environment to me can be from 3 to 10 servers and is a way of containing a specific customer's infrastructure.
* Inside each environment there are a few variables which are mostly automatically generated from a few key inputs (name, size etc).

As it stands right now we're storing all of our environment variables in a structure like so:

```
<playbook>.yml                   # Various playbooks for deployment
roles/windows                    # Ansible role for Ubuntu
roles/ubuntu                     # Ansible role for Ubuntu
config/hosts/<name>.yml          # Ansible inventory
config/hosts/vars/<name>.json    # Environment specific variables

```

Right now the config is initialised as a submodule in the git repository above. As the variables file changes rather frequently this has caused issues with the data changing, once, twice or even three times between commits making changes increasingly difficult to trace.

As I personally see it going forward we should be looking to store all our customer variables in a centralised / scalable way and then hook into it with a dynamic inventory with .

I understand there are a few technologies which seem to do a part of what could be required such as Consul but they seem to work best in an environment which serves one big application rather than lots of smaller slightly differing ones.

I essentially see us having to write an inventory script and then just shove all of our data into some not-for-purpose built database and then keep going on as if nothing changed. I see this conceivably as a way to potentially trim down on a lot of the data we currently store and perhaps look into different ways of storing the data rather than just scaling up what serves it again.

I'm hoping somebody has some kind of experience in implementing infrastructure as code when having to deal with lots of smaller environments as opposed to one, two or three huge ones.

Any suggestions?

# Answer
I've had two runs at doing environment variables in a scalable way and neither has ended up perfect because, as I've discovered, is a very tricky thing to get right. I'll give a summary of both of my experiences below:

**Common Factors**

* Environment variables are stored in a separate repository from the original source code (they are submoduled together but are still based on separate repos)
* There is a separate "build" process for the artifact and it's variables.
* There is **not** a separate release process for the environment variables. If you want to change environment variables you need to go through the same change review boards and usual

**Using Consul KV Pairs**

The environment variables are loaded from an artifact repository (never the original git repo) and loaded into a namespaced KV pair tree, for example

/env/dev1/my/application/v1.1.1

Where the preceding dev1 is the name of the environment, the my/application is the application namespace, and the v1.1.1 is the version of the environment variables to use.

To developers all these things are invisible. At runtime the platform checks that the environment exists in the current consul cluster (if it doesn't there's a problem and it's errors out), it then checks the subtree for the applications namespace (this way there can be no cross contamination where one app references another apps vars) then the version number of the configuration is taken from label connected to the deployable artifact. Updating this label is the key thing here because it means if we lost both production data centers we could stand up the environment again by simple reading the meta data from our deployable artifacts and loading *all the environment variables* into the KV store.

*Problems With This Approach* Developers always, and I mean every single time, found a way to slip configuration changes into the environment that had significant impacts to how the application would run. Because it always ended up being easier to get configuration changes approved than code changes.

**Storing a "Deployment" Artifact with variables embedded**

This tightly couples the exact version of the artifact to the version of configuration. If you changed configuration then you had to rebuild this deployment artifact.

The deployment artifact itself was essentially a yaml file which contained the URL to the releasable binary and all the configuration attached to it.

The platform contains componentry to read variables and then out them into the process tree of the application when it starts up.

This has so far been a lot more successful because there is an artifact we can trace the history of and which we can hold up to any review board and say "this is the only artifact we care about, we don't need t look at any other changes, only changes to this thing" (ie. Version of application to deploy, environment variables included etc.

This makes it just a little bit harder for developers to try and build logic into their application that will change its behavior based on variables so they can slip in changes without going through appropriate testing cycles.

**Bonus Points**

Consider application secrets. Our solution to this has so far been to provide a public RSA key which development teams use to encrypt an extended Java key store (almost every language has a library somewhere that can read Java keys stores) this is then treated like a third type of artifact and is pulled onto the server, decrypted with our platform private key, and provided to the application at run time.

Admittedly secrets management is its own can of worms. But it's probably worth considering.
> 13 votes

# Answer
If your environments are per customer, I would suggest *in your specific case* to have a **repository per customer**. (In general it is repository per environment.) This repository would have a standard directory structure for environment variables, ansible variables and inventories, strongly encrypted secrets (account access tokens, private keys, etc.). You would git submodule the code into those repositories. I would probably do it in multiple repositories. One for ansible roles and modules, one for maintenance and deployment scripts, one for each major application running in the environments.

Now you can optionally actually fork the code or otherwise pin the submodule at specific tag for **release**, making sure that the code managing the customer's environment would not change unless tested and released.

If you are using an **artifact repository**, make sure the artifacts are properly versioned and those versions are specified in the environment variables properly.

**Automation** is important because the environment variables should not be updated by humans where possible, but generated by scripts. Make sure that there are nearly no manual updates in the per customer inventory and developers only update the code repositories. If they want to make configuration change, it should be done to one of the generating scripts, which is then run to generate the variables and the diff is committed into the customer repository. It pays to setup continuous integration for this process. Without this at some point there will be too many repositories to **maintain**.
> 3 votes

---
Tags: ansible, configuration, databases, provisioning, environment-variables
---


# Question
Title: Continuous Integraton using Jenkins Pipeline for a .NET project
The project: A .NET standalone application consisting in:

* The main application that is a Windows Service containing a web server used for the web interface. Project is 90% .NET (Framework, Standard and Core) plus a few C++ libraries
* Windows Agents that will be deployed naturally on Windows machines
* ASP.NET Core Agents that will be deployed on Linux machines
* WIX used for packing and creating the installer

We don't have a dedicated and experienced Devops team member, but we currently have in place a sort of CI/CD pipeline in VSTS. Nothing fancy, just a bunch of build definitions that run the unit tests for every commit and nightly the integration and Selenium tests split into several builds across multiple VMs (due to MSTest parallelization at the assembly level in order to decrease the run time), and another one that is manually started to create the installer.

In the near future we may need to switch to Jenkins and after spending some time messing with builds in it I grew really fond of the Pipeline plugin (especially with the Blue Ocean one). On the other side, I discovered that if you're not developing in the Java world, you're kind of screwed with even the most basic tools required for .NET like MSBuild and MSTest or VSTest lacking Pipeline support.

Only to be able to run the tests and display the results we had to do a bunch of steps, mostly consisting in batch scripts and add extra tools to be able to build, perform dotnet restore, nuget restore, run the tests, transform the `.trx` file into JUnit supported ones to be able to display them.

Considering all the info above, I humbly ask the following:

1. What should an "upgrade" of the existing build system look like?
2. Are there any buried resources/guides/starting points in how we could make use easier of the Pipeline in the .NET world?
3. Anything else that I may be missing?

# Answer
Specifically in answer to your 3rd question, if you are willing to look outside of the Jenkins Ecosystem there are alternatives out there that might be of value to you.

For my clients who use the Microsoft Stack and have fewer than four teams, I have been recommending the use of AppVeyor it is highly tuned for the .NET Stack and integrated very naturally with `msbuild` and Wix.

There are two ways of configuring AppVeyor, either through the web-based user interface or via the `appveyor.yml` file checked into the root of the git repository. I would strongly recommend the latter, you are welcome to start from this template:

```
version: 3.0.{build}

build:
  parallel: true
  project: EVEMon.sln

cache:
  - packages -> **\packages.config

install:
  - nuget restore

```

The major drawback of AppVeyor and it's cousins TravisCI and CircleCI is they don't play particularly nicely with on-premise source code management solutions, you do really need to be using GitHub or BitBucket.
> 4 votes

---
Tags: jenkins, jenkins-plugins, build-pipeline, dotnet
---