# Question
Title: Is there an ansible module that check whether a port is not in LISTEN state?
The Ansible module index has been consulted, but no module was found that checks whether a port is available? Is there a module that supports this?

**What does `available` mean?**

Available means that a port, e.g. `80` is not in LISTEN state. The following indicates that port 111 is unavailable.

```
user@localhost ~ $ ss -nat
State      Recv-Q Send-Q Local Address:Port    Peer Address:Port
LISTEN     0      128       *:111                   *:*

```

**What is the aim of this question?**

The aim of this question is to find a way to stop an Ansible run when port 80 is already allocated by another process than certbot. Certbot requires port 80 to be available, otherwise the cert cannot be renewed. The issue can be found here.

# Answer
> 12 votes
I think you're looking for the `wait_for` module. It will allow you to check and make an action when a certain port is available and more.

# Answer
> 4 votes
Based on @13nilux' answer the following code has been created:

```
- name: Check whether port 80 is available
  wait_for:
    port: 80
    state: stopped
    timeout: 10

```

When port 80 is listening the run will fail if for example nginx is listening:

```
TASK [role_under_test : Check whether port 80 is available] ********************

fatal: [localhost]: FAILED! => {"changed": false, "elapsed": 10, "failed": true, "msg": "Timeout when waiting for 127.0.0.1:80 to stop."}

 [WARNING]: Could not create retry file

'/etc/ansible/roles/role_under_test/tests/test.retry'.         [Errno 30] Read-

only file system: u'/etc/ansible/roles/role_under_test/tests/test.retry'

```

# Answer
> -1 votes
Letsencrypt is long-time renamed to certbot. You can automate certificate renewal process by configuring a host which serves a domain(s) by installing certbot and configuring web server for handling a directory with auth tokens as `/.well-known/acme-challenge/` for domains under Let's Encrypt-signed certs, then using Ansible playbook for manual renewal process or run the same command by cron.

---
Tags: ansible, certbot
---


title: Baking Immutable Images

text:
# Question
Title: Baking Immutable Images
I am working on a project and we are trying to bake immutable images on containers as IaC. These images can have J2EE, .NET or Python apps on it. All OS patches should be applied to this image frequently and app updates should be also updated on this ready to run images. Having said that, an option we discuss includes Terraform for provisioning, Packer for building images. Many internet resources deploy additionally Ansible, Chef or Puppet into this mix. My question why Packer is not enough to cover baking images why would I need to consider Ansible/Chef/Puppet additionally? What can't Packer do and others can for these requirements? Thanks for your time and help...

# Answer
> 6 votes
An idea key to your question is that of *immutable infrastructure*, which is the idea that an image is built once, deployed many times, and never changed at runtime. If the contents of the image need to change, an entirely new image is built from scratch and new instances replace the ones running the old image.

This is the opposite of what you might now call *mutable* infrastructure, where often a machine (or container) is booted from a generic OS image and then its contents managed by a configuration management tool on an ongoing basis. In this model, instances generally live longer and are changed in-place to respond to new requirements.

Packer is intended as a building block for *immutable* infrastructure. It is intended to orchestrate the creation of customized images that boot immediately into the desired state. If immutable infrastructure is the right fit for your use-case (which is not necessarily the case!) then indeed tools such as Ansible, Chef and Puppet *may* be unnecessary.

However, running these tools in a "one-shot" mode (e.g. *Chef Solo*) with Packer can often be useful for non-trivial images: these tools generally provide more powerful abstractions than Packer alone provides, making it easier to do things such as configure the init system to run certain services, or create user accounts. In this case, the configuration management tool would run as a *provisioner* in Packer, creating the necessary configuration within the image, but is not involved in the ongoing upkeep of instances booted from the image.

Most practical systems include a mixture of mutable and immutable infrastructure, since e.g. immutable infrastructure is inappropriate for stateful, centralized systems. In that case, using a traditional configuration management system can be useful to create a common base of functionality between your immutable and mutable systems, applying configuration in a one-shot way for immutable images and in an ongoing, stateful way to long-running, mutable machines.

# Answer
> 3 votes
Image generation and distribution quickly rises to O(n^2) or higher, and as terraform is explicitly focused on instantiation it relies on external provisioners for internal state.

It will work at a smaller scale, but will greatly complicate a hybrid-cloud model and will have problems if the systems are long lived.

It also tends to cause difficulties with refactoring an iterating but these would be dependent on your use case.

For example, lets say you wanted to deploy a new version of a python module. If you had a wheel or a requirements.txt you could deploy this to cloud provider a,b,c on OS version c and d. But with fully baked images you would have to produce and test images a\_c, b\_c, c\_c, a\_d, b\_d,c\_d. If you have to do this per iteration the costs to developer productivity grow very quickly.

Note that you may be assuming that the images will be fully immutable, but without a configuration management tool you have no way to ensure that they are.

As containers in general right now work under a trusted model, and it is fairly trivial for any process or person who has access to launch a machine under frameworks like docker to make arbitrary changes to anything on the system (every API user is root, even on the host for docker) this is a contingency you want to account for.

If you keep the loose coupling between cloud provider, application and base OS it is much easier to iterate over time and vendor mitigation is far easier.

If you maintain a terraform configuration for GCE, AWS and Azure, a packer configuration per OS then an application deployment method separately it may increase the initial complexity but those costs will quickly amortize away over the total life cycle of the project.

I personally find a tool like ansible, which can be run without centralized infrastructure, and which can make orchestrated changes based on state to be more flexible in a pure deployment model but I have always had to retrofit in some form of configuration management at some point.

This may not directly map to your specific use case but it is a very good reason that it is a common practice.

# Answer
> 0 votes
I don't see the relation between packer and ansible/chef/puppet the way you do. Packer might give you a means to declare that you are building an image but if you aren't going to use something like ansible/chef/etc to load up the stuff you want in that image then how do you propose to get the stuff into the image?

Usually the default answer to this would be "well we would script it" but she'll scripts are just the glue that can stick *anything* together, they aren't designed specifically to handle infrastructure and configuration.

I've been working for a little while on a project where we have made the concious decision to try and write no shell scripts, at all, even where it's a little less convenient we write everything in a clearly defined, preferably declarative, language that someone else is maintaining standards for (ie. We don't need to make sure that ansible can do the things we need it to do, we let the developers do that, as opposed to scripting everything and needing to write all the detailed actions outselves).

If you consider your configuration management tools as just a nice way to declare what is actually in he base image then your off to a pretty good start if you want developers to trust the image their code will be running on.

---
Tags: docker, ansible, terraform, packer
---


# Question
Title: Application calling AWS internal load balancer in same subnet is timing out
## Some background:

I've created a moderately complex network using Amazon's vpc. It's a three-tiered network across two availability zones. Each layer has a subnet in zone-a and zone-b. The presentation layer is at the top, there’s an application layer in the middle, and a core layer at the bottom.

All of the security groups and ACL's for the subnets are currently allowing ALL inbound and outbound traffic to help me reduce surface area of the issue.

The presentation layer’s routing table is pointing all traffic to an internet gateway. The NAT gateway is in a segregated subnet also pointing all traffic to the internet gateway.

My application has two components, a UI (React.js) and an API (Node/Express). They are deployed as docker images. In front of each is a classic load balancer.

The UI-ELB is internet facing and resides in the presentation layer, routing traffic from 80/443 to port 8080 and is associated with my app-ec2 that is placed in the application-layer subnet.

My API has an internal load balancer in front of it. The API-ELB is in the application layer (in the same subnet as the app-ec2), and takes traffic on port 80/443 and routes it down to the api-ec2 in the core on port 3000.

Both load balancers are offloading the certificate before passing traffic to their instances.

I have both my load balancers associated as alias's in Route53, and referenced in the applications by their pretty url (https://app.website.com). Each load balancer passes the defined healthchecks and reports all ec2 instances in use.

Lastly, on the API I have enabled cors using the cors nodejs package.

Here's a quick and dirty diagram of my network.

## The problem:

The APP-ELB successfully routes me to the application. However, when the app tries to send a GET request to the API-ELB, it first sends an OPTIONS request that tims out with the error code 408.

**Where it gets weird**

Some of the weirdest things I've encountered while debugging are:

1. I can SSH into the app-ec2 instance and can run a successful curl against the API-ELB. I’ve tried many, and they all work. A few examples are: `curl -L https://api.website.com/system/healthcheck` and `curl -L -X OPTIONS https://api.website.com/system/healthcheck`. It always returns the desired information.
2. I've moved the entire application out of my network into a public default vpc and it works as it's supposed to.
3. I have the api-ec2 writing all network requests to the console. While it shows the healthcheck requests, it does not show any requests from the app-ec2. This leads me to believe traffic is not even reaching the api.

Really the biggest thing that has me at a complete loss is that curling the internal api elb works, but the axios request to the same exact url does not. This doesn't make sense to me at all.

## What I've tried

I originally spent a lot of time playing with ACL rules and security groups thinking I did something wrong. Eventually I just said, "screw it", and opened everything up to try and take that piece out of the equation.

I've spent way to much time playing with Cors on my api. Eventually landing on the configuration I have now, that is the default `app.use(cors())` callback provided by the cors node package. I've also included the `app.options('*', cors())` that is recommended in the documentation.

I've google everything under the sun, but specifically whether I need to define some special custom headers with the elbs? But can't seem to find anything. Plus, when I moved my app out of the network it worked just fine.

I'm sure I've tried many other things, but these seem to be the most pertinent. What am I missing? I realize this is potentially a very vague and broad issue, and an enormous post, but I appreciate any insight and your time in reading in it!

# Answer
So what you have actually is this:

As your API ELB is in a private zone it can't be accessed from the internet.
Your frontend in React.js just run in User's browser and not on the UI servers, those server just serve static files.

You have two options, configure your frontend servers to redirect API calls to the API ELB or just update the API ELB to be internet facing.

The usual pitfall of JavaScript apps is forgetting they run inside user's browser and not on the frontend servers as a JEE application would.
> 9 votes

# Answer
This sounds like an asymmetric or n-path routing issue. Here is what is probably happening:

Machine A at IP address 192.168.1.1 makes initiates a \[SYN\] request through the LB at 192.168.1.10. the LB then proxies the payload to Machine B at 192.168.1.2, so the payload now has source: 192.168.1.1 and has has destination: 192.168.1.2 (which used to be 192.168.1.10).

So what happens now when 192.168.1.2 responds with a \[SYN, ACK\]? What *should* happen is that Machine B should respond to Machine A *through the load balancer* \- typically because of a default route or gateway on the server that routes traffic through the LB. In this case however, the machine is on the same subnet, so the route/gateway is not used and the routing table ignored by the server. This means that when the server responds, the \[SYN,ACK\] appears to Machine A to come from an IP different than the IP that Machine A initiated the request with - it was expecting a source IP of 192.168.1.10 (the LB) but is seeing a \[SYN,ACK\] coming from 192.168.1.2 (machine B) and thus the LB is unable to establish a connection with the machine B in this scenario because the response went to the wrong device.

The reason this works for external traffic is because of your default route - the responses to everyone else are routed through the ELB. The ELB sees that it was initiating a connection and automagically intercepts the response and swaps the source of 192.168.1.2 back to 192.168.1.10.

So, for one solution to this issue, you could implement one-armed load balancing (also known as a load balancer on a stick). What this will do is use a Source NAT on the inside interface of the load balancer (so assume you had outside interface 192.168.1.10 on your load balancer and 192.168.1.11 on the inside interface). This will make all traffic appear to be coming from 192.168.1.11 from the perspective of Machine B which should solve your connection issue.

It appears however, that your AWS ELB doesn't support SNAT, so you will either need to put your hosts and ELB on different subnets or use something that supports SNATs like F5's Virtual Edition which comes in hourly or BYOL flavors. Beware connection limitations with SNATing though - if you need over about 30k simultaneous connections you will run into SNAT port exhaustion and need to start using a SNAT pool..

Hence, you best solution (for cost and to prevent future issues) would be to make sure the client and server are on different subnets.

The best way to confirm would be to use tcpdump on the connecting host and/or back-end server and look for responses coming directly to/from the back-end server instead of going through the load balancer. You can then load your dump file into WireShark to figure out exactly what is going on.
> 1 votes

---
Tags: amazon-web-services, amazon-ec2, infrastructure, software-engineering, cors
---


# Question
Title: How to detect what interface has private IP in Ansible
I plan to reconfigure `iptables` on our servers (mostly CentOS 7, some CentOS 6) with different network configurations, some on 1Gbit interface, some on 10Gbit, some with VLANs and some with configured LACP.

I want to use Ansible, but couldn't find how to detect what inteface has private address. Private address we could get from inventory as `ansible_ssh_host`.

Is there any Ansible module that could help? Or I should write shell/Python script?

I checked all solutions on StackOverflow and ServerFault but haven't found anything.

# Answer
> 4 votes
Peter's module not working in CentOS and I write my own module:

```
#!/usr/bin/python
# encoding: utf-8

import subprocess
from ansible.module_utils.basic import *
from netaddr import IPNetwork

def get_priv_interface_info_for_ip(ip):
    network_config = subprocess.Popen(["/usr/sbin/ip", "-o", "addr", "show"], stdout=subprocess.PIPE)
    result = network_config.stdout.read()
    interface = ""
    network = ''
    ip_addr = ip
    for str in result.split("\n"):
        if 'inet ' in str:
            if ip in str:
                interface = str[3:str.find(" ", 3)]
                network = IPNetwork(str[str.find("net", 3) + 4:str.find("/", 3) + 3]).cidr

    return interface, ip_addr, network

def get_pub_interface_info_for_ip(ip):
    network_config = subprocess.Popen(["/usr/sbin/ip", "-o", "addr", "show"], stdout=subprocess.PIPE)
    result = network_config.stdout.read()
    interface = ''
    network = ''
    ip_addr = ''
    for str in result.split("\n"):
        if 'inet ' in str:
            if ip not in str and '127.0.0.1' not in str:
                print str
                interface = str[3:str.find(" ", 3)]
                network = IPNetwork(str[str.find("net", 3) + 4:str.find("/", 3) + 3]).cidr
                ip_addr = str[str.find("net", 3) + 4:str.find("/", 3)]

    return interface, ip_addr, network

def main(argv=None):
    if argv is None:
        argv = sys.argv

    fields = {"ip": {"required": True, "type": "str"}}
    module = AnsibleModule(argument_spec=fields)
    my_ip = module.params['ip']
    ifname_priv, ip_priv, network_priv = get_priv_interface_info_for_ip(my_ip)
    ifname_pub, ip_pub, network_pub = get_pub_interface_info_for_ip(my_ip)

    module.exit_json(changed=True, ifname_priv=ifname_priv, ip_priv=my_ip, network_priv=str(network_priv), ifname_pub=ifname_pub, ip_pub=ip_pub, network_pub=str(network_pub))

if __name__ == '__main__':
    main()

```

It works for servers with two network interfaces and provide information about public and private network interfaces.

Actual version of this module could be found in my GitHub.

# Answer
> 2 votes
I couldn't find it in Ansible, so I wrote something to help a couple of weeks ago:

```
import fcntl
import json
import os
import platform
import subprocess
import socket
import struct
import re
from random import randint
import time
from ansible.module_utils.basic import *

def get_interface_info_for_ip(ip):

  ifconfig_process = subprocess.Popen(['/sbin/ifconfig'], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
  ifconfig_stdout_lines, ifconfig_stderr_lines = ifconfig_process.communicate()

  for ifconfig_stdout_line in ifconfig_stdout_lines.split('\n'):
    if 'mtu' in ifconfig_stdout_line.split():
      current_ifdesc = ''
      current_mtu = ''
      current_ip = ''
      current_netmask = ''
      current_mac = ''
      ifdesc_line_match = re.match(r'(?P<ifdesc>[a-zA-Z0-9]+):.+mtu\s+(?P<mtu>[0-9]+)', ifconfig_stdout_line )
      if ifdesc_line_match:
        current_ifdesc = ifdesc_line_match.groupdict()['ifdesc']
        current_mtu = ifdesc_line_match.groupdict()['mtu']

    if 'inet' in ifconfig_stdout_line.split():
      inet_line_match = re.match(r'\s+inet\s+(?P<ip>[0-9.]+)\s+netmask\s+(?P<netmask>[0-9.]+)', ifconfig_stdout_line)
      if inet_line_match:
         current_ip = inet_line_match.groupdict()['ip']
         current_netmask = inet_line_match.groupdict()['netmask']

    if 'ether' in ifconfig_stdout_line.split():
      ether_line_match = re.match(r'\s+ether\s+(?P<mac>[0-9a-f:]+)', ifconfig_stdout_line)
      if ether_line_match:
         current_mac = ether_line_match.groupdict()['mac']

    if current_ifdesc and current_mtu and current_ip and current_netmask and current_mac and current_ip == ip:
      return current_ifdesc, current_mtu, current_ip, current_netmask, current_mac

def main(argv=None):
    if argv is None:
      argv = sys.argv

    module = AnsibleModule(
        argument_spec = dict(
           silo_ip = dict(required=True),
        )
    )
    my_ip = module.params['ip']
    ifdesc, mtu, ip, netmask, mac = get_interface_info_for_ip(my_ip)

    module.exit_json(changed=True, ifdesc=ifdesc, ip=ip, mac=mac, netmask=netmask, mtu=mtu)

if __name__ == '__main__':
    main()

```

Call the file `get_ip_facts`, put it in your library path and call it with:

```
name: get facts for ip address
get_ip_facts: ip={{ whatever_your_ip_is }}

```

---
Tags: ansible, linux
---


# Question
Title: What is recommended books reading order offering thought leadership to read on DevOps to go from beginner to expert?
For sure there is partly an opinion question, but many colleagues around me have little knowledge about books offering thought leadership on DevOps. So this thread might be of great convinience.

# Answer
One fantastic read that offers great thought leadership on this issue and is specifically tailored to the non-technical is The Phoenix Project which is available in paperback, a Kindle E-book and as an Audiobook. This book offers an excellent starting place to educate on the who, what and why of DevOps, wich is often seen as nebulous buzzword. The official synopses states,

> Bill is an IT manager at Parts Unlimited. It's Tuesday morning and on his drive into the office, Bill gets a call from the CEO.
>
> The company's new IT initiative, code named Phoenix Project, is critical to the future of Parts Unlimited, but the project is massively over budget and very late. The CEO wants Bill to report directly to him and fix the mess in ninety days or else Bill's entire department will be outsourced.
>
> With the help of a prospective board member and his mysterious philosophy of The Three Ways, Bill starts to see that IT work has more in common with manufacturing plant work than he ever imagined. With the clock ticking, Bill must organize work flow streamline interdepartmental communications, and effectively serve the other business functions at Parts Unlimited.

By novelizing this story the authors are able to expertly illustrate and explain in a detailed and entertaining manner:

1. What DevOps means and entails
2. What teams and individuals are involved in DevOps
3. How DevOps can benefit a companies' workers and business
4. Challenges and weaknesses of the traditional model

While this will not explain or recommend how to do DevOps on a technical level, it will make and explain the business case and method for doing so, thereby taking you from beginner to intermediate.

The authors have also published a followup tome, the DevOps Handbook which bills itself as a how-to on "Creating World-Class Agility, Reliability, and Security in Technology Organizations," however one reviewer, FA Calkins cautioned,

> Dont buy this book if you expect it to teach you how to build a CI/CD DEVOPS environment. It is basically a review/summary of a bunch of research papers that the authors have read, but does not provide any specifics details of how to set up a CI/CD pipepline (only theory). You can get most of the content of this book by reading articles from the web. Also, you can get more specific details of how to implement CI/CD with specific tools, programs, and scripts from other sources. Dont waste your time trying to read this book if you want to deploy CI/CD.

Though I have not yet read it (it is on my reading list!), the description claims that the book includes

> * More than 40 DevOps case studies, including Amazon, Etsy, Capital One, Google, Facebook, Intuit, Nationwide Insurance and many more
> * More than 400 pages of DevOps applications, lessons and 'how-to’s'.
> * DevOps data gathered from more than 25,000 data points.

These resources should at least get you started on the business/philosophy piece.
> 1 votes

---
Tags: learning
---


# Question
Title: What are feasible quality criteria for Git repositories?
If you manage a lot of Git repositories (or other, but Git is a specific and common technology), it could happen that depending on users there will be quite different usage profiles. On the worst case end you could face issues with broken repositories (unplanned work, risks) and wasted storage (which is claimed to be cheap but is still accounted for).

Who is responsible for imposing minimum acceptance criteria for "quality of usage" (I cannot find better term for now) and what are realistic quantifiable measures you could install and link to actionables?

Examples of anti-patterns and related risks you might be dealing with:

* Folks upload files, also binaries, which are neither part of the software nor test documents
* Binary libraries instead of using dependency management
* Repos become too big (over 15G as stated here \[1\])
* Repos are below 15G but the history is many gigabytes
* Other?

Or, should anybody care about these things at all? (formal objections, not opinions, why not care)

\[1\] https://www.quora.com/What-is-the-practical-maximum-size-of-a-Git-repository-full-of-text-based-data

# Answer
> 5 votes
Who is responsible for checking that people don't check in commits that have large files? The same people who are responsible for checking that the commits aren't bad in other ways: everyone.

If git is new in your company, make sure that the git education includes this sort of thing. After that, it's up to code reviewers to notice mistakes and correct them.

If it becomes a repeated problem, then whoever is in charge of developer tooling can write some checks that run every night or whatever and let people know that something slipped through. But that seems like overkill in most situations.

# Answer
> 2 votes
Having spent lots of time on github and various private corporate repos on gitlab and such there are certain things that make a huge difference in how good a repo seems to me:

* Does the `README` clearly state what the software does? Hopefully it also provides some code snippets, usage examples, and links to the preferred support forum. Any build or runtime prerequisites should be clearly spelled out. Are the primary authors clearly identified?
* Are there releases? If nobody has tagged a release it is easy to assume that the repo is still in early development and not ready for secondary consumption.
* Is there a `LICENSE`? If I don't know that I can *not* legally use this software and it makes it much harder to decide to invest the time into it. On a private corporate server I would assume that everything is valid for internal use, but in some cases that might not work either.
* Is there a link to a CI environment? If the repository doesn't even show the minimal effort at providing tests and access to a CI environment that regularly runs those tests it is a strong vote against it.
* Are the build instructions for a new contributor clear? Some people may have this containerized, but the build instructions for doing it manually should be viable as well.
* In a corporate environment I would expect that you could impose other restrictions semi-automatically. Does the code fit the code formatting guidelines? Are binaries excluded (a good idea if you have a central artifact repository for such things)? I wouldn't set an arbitrary size limitation. Some projects may just grow that large. That's not great from an organizational perspective, but it is easy enough to implement technically.

As for "who" does this I'd say it ends up being a combination of people. To start you need buy-in from engineering management. Then it could be a DevOps person or some random developer who implements it in a particular place.

---
Tags: git, scm, risk
---


# Question
Title: How to export Cucumber Reports via Jenkins Pipeline
I'm using the Cucumber Reports Plugin for Jenkins and I want to be able to export the HTML reports so they can be posted on an external portal. I can see that there is a folder named `cucumber-html-reports` in the build directory in Jenkins, but how do I access this? Is there some way to access these files from within a pipeline?

# Answer
> 3 votes
Working off what @chupasaurus said, here's what I came up with:

```
node('master') {
    dir("../builds/${BUILD_NUMBER}/") {
        sh "cp -r cucumber-html-reports $WORKSPACE"
    }

    archive "cucumber-html-reports/*"
}

```

Obviously all this does is archive the report for that build, but you can easily extend this to copy it somewhere else where it can be hosted or manipulated. Just make sure you put this after the Cucumber Reports plugin step.

# Answer
> 1 votes
You should add export script (bash script which does exporting html files produced to the place they belong) after Cucumber Reports plugin step, as it should be used by job runned either solely or by pipeline.

---
Tags: jenkins, jenkins-plugins, testing, jenkins-pipeline
---


# Question
Title: Terraform: How to use a variable inside of another variable?
I'm trying to design it so that all configurable parameters exist within variables.tf...the issue I'm running into currently is for provisioning EC2 instances. I'd like to be able to define the security group to use, but the input variable only accepts strings, so I wasn't able to reference the security group id.

I moved on to trying this, which is looking up the resource id dynamically by inserting the name (look at security\_groups\_name and subnet\_name:

```
variable "ansible_ami" {
    type =  "map"
    default = {
            name = "ansible"
            ami = "ami-4836a428"
            size = "t2.micro"
            ebs_size = "8"
            ebs_type = "gp2" #general purpose ssd
            enable_public_ip = true
            security_groups_name = "JumpBox"
            subnet_name = "pubnet1"
    }
}

resource "aws_instance" "ansible" {
    ami = "${var.ansible_ami["ami"]}"
    instance_type = "${var.ansible_ami["size"]}"
    security_groups = "${aws_security_group.${var.ansible_ami["security_groups_name"]}.id}"
    subnet_id = "${aws_subnet.${var.ansible_ami["subnet_name"]}.id}"
    associate_public_ip_address = "${var.ansible_ami["enable_public_ip"]}"
    #user_data = ""

    ebs_block_device {
        device_name = "/dev/sdb"
        volume_size = "${var.ansible_ami["ebs_size"]}"
        volume_type = "${var.ansible_ami["ebs_type"]}"
    }

    tags {
        Name = "${var.ansible_ami["name"]}"
    }
}

```

However, that is not working either since i'm getting syntax errors from the lines where I'm trying to next variables. I'm looking for ideas or direction on how to accomplish this.

# Answer
Largely inspired by the documentation the idea is to create a terraform datasource from a filter of the available resources and then use this as entry point in your ami resource:

```
variable "security_group_id" {}
variable "subnet_id" {}

data "aws_security_group" "selected" {
  filter {
    name = "${var.ansible_ami["security_groups_name"]}}"
  }
}

data "aws_subnet" "selected" {
  filter {
    name = "${var.ansible_ami["subnet_name"]}"
  }
}

resource "aws_instance" "ansible" {
    ami = "${var.ansible_ami["ami"]}"
    instance_type = "${var.ansible_ami["size"]}"
    security_groups = "${data.aws_security_group.selected.id}"
    subnet_id = "${data.aws_subnet.selected.id}"
    associate_public_ip_address = "${var.ansible_ami["enable_public_ip"]}"
    #user_data = ""

```

I'm not using terraform myself, so this could be plain wrong.
> 1 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: What is the difference between packages and artifacts?
I would like to understand the difference between package and artifact.

Also In what context we should use these terms.

# Answer
> 7 votes
Mainly an artifact is the result of of a build phase, this mean a package is an artifact of a kind.

A package is usually a way to install a software or application, it includes the software itself and some intelligence to setup and configure the software.

Calling an artifact a package usually comes when the artifact (whatever it is, from a .deb to a simple zip file) contain more than just the application, it could be a simple script to help configuring the software and setting proper permissions or other softwares needed.

# Answer
> 0 votes
Also, the term "artifact package" refers to a logical package that can assemble artifacts created in a single build as well as multiple builds.

---
Tags: terminology, deployment, release-management, release, artifacts
---


# Question
Title: How to login using serverspec using ssh key?
The serverspec doc indicates that it is possible to login using ssh and serverspec:

> ```
> require 'highline/import'
>
> if ENV['ASK_LOGIN_PASSWORD']
>   options[:password] = ask("\nEnter login password: ") { |q| q.echo = false }
> else
>   options[:password] = ENV['LOGIN_PASSWORD']
> end
>
> set :ssh_options, options
>
> ```

How to login using an ssh key?

# Answer
serverspec is using this Ruby class to ssh to servers

One should set the path to the key into the ~/.ssh/config file:

```
Host X
    HostName X
    User Y
    PreferredAuthentications publickey
    IdentityFile ~/.ssh/Z

```

Now `rake spec` runs the test on the server.
> 0 votes

---
Tags: serverspec
---


# Question
Title: How can I calculate costs of deploying 1000 instances of Drupal?
What are the things that should I investigate for making an estimate of the cost of deploying 1000 instances of Drupal, for example?

I was thinking in:

* Database space
* Database usage (in ops/second? how can I test it?)
* Application uploads space
* Application processing time (again, how to measure?)
* Bandwidth usage (for medium/low traffic sites)
* DevOps / Technical Support time

This is all about bringing a SaaS platform. Is there anything else I should take in consideration?

How can I know how many "instances" would be able to run in a single host? Much like a Shared Hosting

Would Amazon EC2 / Google / Azure be cheaper than having dedicated servers?

# Answer
> 3 votes
In reply to 'Would Amazon EC2 / Google / Azure be cheaper than having dedicated servers?'

I've done a lot of investigating into this and in every case they are NOT.

80 Linux servers running 24h/365 - annual cost: Total cost per year Google £37,515.46 AWS £29,196.53

These costs do NOT include data transfer.

You can buy a dedicated cloud appliance for less than £50k, fully supported, that will run more servers and last you 3 years (min).

---
Tags: containers, cloud, drupal, capacity-planning
---


# Question
Title: How to performance test websites using Gatling that are encrypted using Certbot?
**Aim**

to performance test a UI

**Method**

* Run Gatling recorder
* Once the gatling recorder proxy has been started, configure this proxy in the webbrowser that will be used for the test
* Login to the UI
* Navigate to the website
* Save the recording

**Results**

When a http site is recorded, then it is recorded, but when an https site then gatling records nothing

**Discussion**

Gatling can record websites, but when these are encrypted then one could define the following:

HTTPS mode:

* self-signed certificate
* provided keystore
    + keystore file
    + keystore pasword
    + keystore type
* certificate authority
    + ca certificate
    + ca private key

How to record websites that are encrypted using Certbot?

This topic has been created in the Google user group.

# Answer
If it is possible to switch https to http temporarly then gatling can record the clicks. When the recording was done, https was activated again and the test could be run against https as well. In summary, recording works if the site is http and running it works against https. Note that the recorder file contains http and this need to be changed to https.
> 0 votes

---
Tags: gatling, certbot
---


# Question
Title: Deploying a set of cloud instances based on a list of parameter differences
Sorry for the general and vague nature of my question. Let me describe with some more detail what I would like to do (or what I think I would like to do).

I have a shell script for ubuntu 16.04 that sets up my instance how I want it. But it needs to use two different variable inputs for each different instance. For example, if across all the instances that I wanted to set up, I wanted to put in different timezone, I could start with a list of all the timezones, and iteratively deploy a machine for each timezone. Maybe the solution I look for would drop a textfile with the timezone I wanted on each instance. That would be enough for me to read the file and set it up. I just need a solution that will allow me to create any number of cloud instances of ubuntu 16.04 and deliver seperate textfiles with these timezones. They are all identical except the file with the timezone.

I'd like to be able to create all these instances simultaneously from the command line or through some R package for an API interface. Digital Ocean, AWS, or even some local option if I should be considering that.

Is this a job for docker or vagrant? What is the tool for this kind of deployment? Maybe a script to loop through creating instances with these?

# Answer
This is why configuration management software exists. Example for Ansible and AWS:

```
- name: Create a sandbox instance
  hosts: localhost
  gather_facts: False
  vars:
    timezones_path: /some/path/timezones.list
    key_name: my_keypair
    instance_type: m1.small
    security_group: my_securitygroup
    vpc_subnet: my_vpc_id
    image: my_ami_id
    region: us-east-1
  tasks:
    - name: Load timezones list
      shell: "echo {{ item }}"
      with_lines: "cat {{ timezones_path|quote }}"
      register: timezones

    - name: Launch instances
      ec2:
         key_name: "{{ keypair }}"
         group: "{{ security_group }}"
         instance_type: "{{ instance_type }}"
         image: "{{ image }}"
         wait: true
         region: "{{ region }}"
         vpc_subnet_id: "{{ vpc_subnet }}"
         assign_public_ip: yes
         count: "{{ timezones|length }}"
      register: ec2

    - name: Add new instance to host group
      add_host:
        hostname: "{{ item.1.public_ip }}"
        timezone: "{{ item.0 }}"
        groupname: launched
      with_together:
        "{{ timezones }}"
        "{{ ec2.instances }}"

    -name: Tag instances with timezones
      ec2_tag:
        region: "{{ region }}"
        resource: "{{ item.id }}"
        tags:
          Timezone: "{{ item.timezone }}"
      with_items: "{{ groups['launched'] }}"

    - name: Wait for SSH to come up
      wait_for:
        host: "{{ item.public_dns_name }}"
        port: 22
        delay: 60
        timeout: 320
        state: started
      with_items: "{{ ec2.instances }}"

- name: Configure instance(s)
  hosts: launched
  become: True
  gather_facts: True
  tasks:
    - name: Run deploy script
      shell: "/path/to/script.sh --tz {{timezone}}"

```

Path to script in last line should be for host which runs Ansible, not remote.
> 1 votes

---
Tags: docker, vagrant
---


# Question
Title: Jenkins Testing PR's - allowing broken builds
I have a very large codebase that currently takes ~30 minutes to build via Jenkins. It's connected to our Source control/review system (Let's just say GitHub for now) so that every PR automatically gets tested and must build before we allow it to be merged.

We're seeing a behavior where 2 PR's (We'll call them PR#a and PR#b) can exist and each one is tested successfully (code builds, unit tests run, etc). However, they're each tested against the tip of develop. That means once PR#a is merged, PR#b shows that's been tested and valid.. However merging it exposes an API break merged as part of PR#b, and now the build is broken pending a fix.

What's the typical way to prevent this? If not in GitHub, what about other systems? (GitLab, Gerrit, or Stash/Crucible ?)

# Answer
On GitHub - you can't. Basically, their PRs are temporary branches, and there's no check if base commit is HEAD of branch to merge in. This problem is solved with gating systems like Zuul, which puts all PRs into queue to run tests.

You are talking about systems for managing and storing code, while the issue is in CI scope (i.e. Gitlab flow has this issue described in "Testing before merging" paragraph), Zuul is made exactly to fix this problem, because the amount of merges to OpenStack codebase per day is large (year ago they had 6 zuul instances to run tests for merges).
> 2 votes

---
Tags: jenkins, gitlab, builds, bitbucket
---


# Question
Title: Does full automation still provide a better ROI at small scales?
Obviously, Automated configuration management and deployment makes sense at large scales when you need to manage hundreds of servers and you need those servers to be identically configured to a consistent standard.

However, in situations where systems are small and need only 2, 3 or 4 servers, it can often be faster in build time to simply stand up a first instance and clone subsequent instances - changing IP addresses and hostnames by hand.

The theoretical advantage to automated deployment and configuration management is that code can be deployed faster, tested faster and services are provided more reliably by automating out the problem of human error. Future upgrades of the underlying OS (eg, CentOS 6 -\> CentOS 7) are better/faster and in the long run the above benefits outweigh the initial larger investment of programmatically automating deployments using a configuration management system?

Are there any real-world non-anecdotal studies and data-points to back each of the above claimed benefits to small shops with very low numbers of servers to manage?

# Answer
> 8 votes
There is definite value in automating manual tasks and placing systems under configuration management that is done using code, not using paper and human intervention.

One huge benefit is the reduction in the amount of rework. You can consider any problem caused by a human mistake in repeating the same sequence a case of rework. As well as any bug that needs to be fixed more than once.

Rework is costly, and there has been some research done to quantify its cost. You can read about it in this whitepaper - https://devops-research.com/roi/

The same whitepaper also quantifies the cost of failure for services/systems, and the slowness to resolve failure. All of which are greatly reduced by having a robust automated process that prevents human mistakes from ending up in production systems.

The topic of automating manual tasks has been also investigated in the *State of DevOps Report* that has been issued every year since 2013. Most of these are available at https://devops-research.com/research.html as well (with the exception of 2013). In every report it was mentioned as the biggest driver for excellence in high-performing organizations of all sizes. There are plenty of reasons and explanations that can be used as proof for your case.

# Answer
> 4 votes
The main advantage to automation isn't just automation itself, but also that it provides you with an identical configuration across all of your resources.

Say, for example, you stand up a webserver hosting your application and all relevant configs. Then you register it as an Amazon AMI so you can deploy multiple copies (say, one for dev, one for test, and one for production). You roll it out, and with how AWS works, you don't even need to change any IPs by hand.

A few weeks later you realize you need to change SSL session cache size in nginx, or users of your application can't upload large files.

Since you only have a few boxes, you do the update by hand. You do these config changes a few more times and well-document them.

Then a week later you deploy a new box (a customer is asking for a UAT environment to test their integration). You're sick, so someone else is standing up the server instead of you. Or maybe your instance in AWS dies, so you have to rebuild production in the middle of the night after a good binge-drinking session. Everything works perfectly... except for one specific config-related feature. Cue wasting two hours of figuring out why it's happening as an account manager is riding your ass, only to realize the other admin forgot a single stupid config setting when deploying a new server.

Have this happen to you a few times, and you'll want to find a way to minimize these issues.

So why not take a few days of your time and write out a simple deployment pipeline with Ansible or Puppet and put it in git? From that point on, any config changes are only a push away, and you'll rarely have to worry about a similar incident occurring.

Now, it's not peer-reviewed, APA-cited research papers you're asking for, but point me a sysadmin that's never had this happen to them, and my company will probably throw them a job offer with a pretty nice signing bonus.

# Answer
> 4 votes
One of the things I have the hardest time getting across to people is that the man-hours saved doing repetitive tasks is often only a small part of the value of automation. The bigger part is often hard to measure, and nearly impossible to estimate when automating something for the first time: *how it changes the way you work*. Talking with developers, this is a lot easier, because they know test automation (unit tests) and it's an easy comparison to make. It's things like:

* How does your workflow change when you can do X as often as you want with negligible costs and in very little time?
    + For unit tests, it means bugs get caught much, much earlier - you can test *everything* multiple times per day, instead of shipping something to QA once a week. Turnaround on iterations gets orders of magnitude shorter.
    + For infrastructure, it means that doing a short-lived test environment or proof of concept is a non-event, as is scaling out, as is replacing an unhealthy instance. The LoE nears zero.
* How does your workflow change when you think about automation as part of your process?
    + For unit tests, this means you design your code differently, to facilitate automation, which (often) leads to better, cleaner code.
    + For infrastructure, this encourages you to develop a standard plan for how you're going to automate things - when a new service is being developed, you know what questions you need to ask about it.
    + Developers start to change the way they develop as well, thinking about keeping stateless, graceful shutdown, fast restarts, how configuration is handled, keeping log or data file paths configurable, thinking about what paths the application user needs access to and what access it needs, etc.

The "pets vs. cattle" shift was driven in large part by automation; when creating a working server is a non-event, then servers become fully fungible and you no longer need to care about individual instances.

---
Tags: automation, configuration-management
---


# Question
Title: System test frameworks for scale testing on physical and virtual systems
I'm searching for a framework for running system tests. The requirement are:

* Able to manage provisioning to multiple physical machine
* Able to manage provisioning to multiple virtual environments (AWS, Azure, etc)
* Able to run tests on a given machine be it physical or virtual.
* Not bound to a single language or platform. We wish to at least test against Linux, Windows at least.

Is there anything that meets those requirements?

Thank you!

# Answer
Test Kitchen? Your requirements are huge and vague but it's probably your best bet.
> 1 votes

# Answer
So you are looking for a software which is resource agnostic

* provision any resource type
* configure any resource type
* execute on any resource

I am biased but you can look at some job schedulers. We are currently using ProActive from Activeeon which should answer your needs but other may have similar behaviors.
> 0 votes

---
Tags: testing, systems, control-framework
---


# Question
Title: Understanding how Docker volumes work with compose
In the Docker Compose example for WordPress, the Ycode listing \[1\] says:

```
volumes:
   - db_data:/var/lib/mysql

```

What is exact meaning here? To compare, while using the Docker client, you can map volume to a host folder \[2\]:

```
-v /src/webapp:/webapp

```

> This command mounts the host directory, `/src/webapp`, into the container at `/webapp`.

\[1\] https://docs.docker.com/compose/wordpress/#define-the-project

\[2\] https://docs.docker.com/engine/tutorials/dockervolumes/#locate-a-volume

# Answer
> 6 votes
`db_data:/var/lib/mysql` simply means that the `db_data` volume that was previously created will be made available in the container at `/var/lib/mysql`.

In you example it's created at the very end of the compose file.
From the Docker documentation: "The docker volume db\_data persists any updates made by Wordpress to the database."

Compose volumes works the same way as the Docker engine.

In you case db\_data is just a named volume instead of a path like your second example.

---
Tags: docker, docker-compose
---


# Question
Title: Is DevSecOps indeed a new practice worth a professional term?
Or rather marketing slang? Currently, I am not always sure how to build a consense which words deliver informational value and which are thought to attract customers. So, DevOps would include security testing if required without calling it DevSecOps? Otherwise we will have DevPerfOps, DevDevOpsOps etc

# Answer
> 5 votes
In my experience, the purpose of the "DevSecOps" (or whichever order you prefer) has mostly to do with including Security as a first-order concern of software delivery. Security is commonly treated as a "bolt-on" (often meaning *optional*,) non-functional requirement. DevSecOps attempts to describe a process where security concerns are folded into the delivery process at every step of the value stream.

---
Tags: terminology
---


title: Docker memory storage

text:
# Question
Title: Docker memory storage
Docker daemon allows, according to documentatio, using memory storage driver for its images. Has anybody any daily business experiences with that?

# Answer
> 6 votes
According to the documentation, memory storage is intended for testing purposes only and should never be used in production. The only practical purpose I can think of for this might be state testing, such as with Serverspec or Inspec, to validate configurations. Using memory storage for this purpose *might* be a little faster and use less disk storage. I'm not sure those benefits are worth the effort to implement.

---
Tags: docker, scalability
---


# Question
Title: Unable to get jupyter/scipy-notebook running on docker
I pulled the docker image and is up to date. When I run it it says that it's going to be accessible at 8888 on my localhost but when I try to access it resource can't be found.

Here is the output:

```
sudo docker run jupyter/scipy-notebook #jupyter
[sudo] password for tech:
Execute the command
[I 06:16:54.964 NotebookApp] Writing notebook server cookie secret to /home/jovyan/.local/share/jupyter/runtime/notebook_cookie_secret
[W 06:16:55.137 NotebookApp] WARNING: The notebook server is listening on all IP addresses and not using encryption. This is not recommended.
[I 06:16:55.195 NotebookApp] JupyterLab alpha preview extension loaded from /opt/conda/lib/python3.5/site-packages/jupyterlab
[I 06:16:55.211 NotebookApp] Serving notebooks from local directory: /home/jovyan
[I 06:16:55.211 NotebookApp] 0 active kernels
[I 06:16:55.211 NotebookApp] The Jupyter Notebook is running at: http://[all ip addresses on your system]:8888/?token=8bc0bf76526f10c078aad705f48f80598fe17aade9897db8
[I 06:16:55.211 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[C 06:16:55.212 NotebookApp]

    Copy/paste this URL into your browser when you connect for the first time,
    to login with a token:
        http://localhost:8888/?token=8bc0bf76526f10c078aad705f48f80598fe17aade9897db8

```

what could I be doing wrong?

# Answer
> 2 votes
It turned out to be an issue with port mapping. After mapping local machine port 8888 to docker's 8888 I could launch the program in browser.

```
sudo docker run -p 8888:8888 jupyter/scipy-notebook

```

---
Tags: docker, python
---


# Question
Title: Is the use of ChaosMonkey restricted to AWS?
According to this documentation, ChoasMonkey seems to be restricted to AWS:

> Get Auto Scaling Tools
>
> You can use the new AWS CLI tools:
>
> ```
> $ sudo pip install awscli
>
> ```

Is it possible to use ChaosMonkey on other platforms as well?

# Answer
Quoting the standalone ChaosMonkey README:

> ## Requirements
>
> This version of Chaos Monkey is fully integrated with Spinnaker, the continuous delivery platform that we use at Netflix. You must be managing your apps with Spinnaker to use Chaos Monkey to terminate instances.
>
> Chaos Monkey should work with any backend that Spinnaker supports (AWS, Google Compute Engine, Azure, Kubernetes, Cloud Foundry). It has been tested with AWS, GCE, and Kubernetes.

As said there, you'll find supported Spinnaker platforms on https://www.spinnaker.io

Your quote comes from the quickstart documentation of Symian Army, as for any quickstart it is opinionated to keep things as simple as possible, if you don't match the quickstart environment, you will have to dig deeper in the documentations to adapt to your needs.
> 3 votes

# Answer
The `Chaos Monkey` tool, released by Netflix might be restricted to AWS at this time. But the approach is definitely not restricted and can be implemented anywhere where there are distributed systems comprised of a multitude of redundant parts, hopefully with automated healing ability.
> 1 votes

---
Tags: resilience-testing
---


# Question
Title: Terraform - resource repeated multiple times
I have a terraform plan which consists of multiple `aws_sqs_queue` resources which more of less share the same config. Before I remove any repeating configs and use variables I wanted to run `terraform plan` to see if it works. However I keep running into -

```
module root: 1 error(s) occurred:

* data.template_file.ep_match_result_queues: resource repeated multiple times

```

Ive tried googling but there is not much information available. Any help will be much appreciated. My plan looks like this: (ignore extra braces, typos as this is just a skeleton to give you a general idea how the plan is sturctured):

```
data "template_file" "ep_match_result_queues" {
  template = "${var.namespace}-sm-ep"
}

resource "aws_sns_topic" "sns_topic_name" {
  name            = "ep_sm_match_result_topic${var.environment}"
  display_name    = ""
  policy          = <<POLICY
{
 #policy
}

resource "aws_sqs_queue" "queue1" {
 #config
}

resource "aws_sqs_queue" "queue2" {
 #config

    redrive_policy             = <<POLICY
{
 #policy
}
POLICY
}

resource "aws_sqs_queue" "queue3" {
#config
}

resource "aws_sqs_queue" "queue4" {
 #config

    redrive_policy             = <<POLICY
{
 #policy
}
POLICY
}

resource "aws_sqs_queue" "queue5" {
 #config
}

resource "aws_sqs_queue" "queue6" {
  #config

    redrive_policy             = <<POLICY
{
 #policy
}
POLICY
}

resource "aws_sqs_queue" "queue7" {
 #config
}

resource "aws_sqs_queue" "queue8" {
  #config

    redrive_policy             = <<POLICY
{
 #policy
}
POLICY
}

resource "aws_sqs_queue_policy" "queue_policy" {
  queue_url = [ ... ]
  policy    = "${data.aws_iam_policy_document.match_result_queues_policy.json}"
}

data "aws_iam_policy_document" "match_result_queues_policy" {
 #policy
}

```

# Answer
Terraform scans the configuration directory for all files whose name end in `.tf`, and loads them all as configuration.

A common cause of this problem is if you accidentally duplicate one of the `.tf` files under a new name, and do not update its contents. In that case, Terraform will see both files define the same resource and produce the error shown here.
> 1 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: What are known useful and misleading memes in the DevOps culture?
After discovering the Cattle vs Pets and 12factor apps DevOps memes which now will help me a lot with communication, I ask myself what are other known memes in the DevOps culture?

Side note: best is to establish fact why a meme is a meme e.g. more than 10K Google query results?

# Answer
> 8 votes
Useful memes:

There are some horrible memes pushed by vendors worth mentioning.

*WARNING*: **Bad** memes to use for explaining DevOps (traps for young players) -

---
Tags: culture
---


# Question
Title: Why is the AWS EC2's spot price greater than the on-demand price?
I was trying to provision spot instances via Ansible yesterday, and almost all my requests failed, even when I put my spot price == the on-demand price of that instance.

So, when I had a look at the spot pricing graph, I found something very interesting:

>

The spot price of the instance in us-east-1a is more than the on-demand price, which confused me. \[in fact, ~5x times higher\]

**Aren't spot instances preferred for the low cost?** If yes, then why is the price higher than the on-demand price?

According to AWS's docs:

> Spot instances provide you with access to unused Amazon EC2 capacity at steep discounts relative to On-Demand prices.

Also, **does this mean that people are bidding over the on-demand price?** If yes, then why so? Aren't they better off with an on-demand instance?

Or did I understand the concept of spot instances wrong?

# Answer
> 31 votes
This is actually a great example of people slightly abusing spot. People are saying 'Our workload is really important but we don't want to pay full on demand price', so they set a bid price higher than on-demand on the assumption that it is very unlikely to be terminated, but still want to get the 'cheapest possible' spot price on offer.

There have been cases where people enter, for example, $1000 (I've been told of at least one time this happened) because they want the benefit of the spot market. Of course naturally at some point the demand comes in and the spot price SOARS to make people pay higher than on-demand.

The way the spot market works is that Amazon have X instances spare capacity, and they count from the top down until they fill the need for all X instances. The 'price', then, is the lowest price at which they can fulfill those X instances.

So imagine Amazon have 10,000 instances - well they will count down to (say) $0.43 until they've got those 10,000 instances fulfilled. But if that supply suddenly drops to 100 instances, then maybe a few people put bid prices of $10,000 for their 100 instances, suddenly they'll be paying that $10k per hour.

Tl;dr Understand how spot works, and set a cap you are prepared to pay.

# Answer
> 13 votes
There are 2 reasons for this:

* A lot of users are using the spot instance at some times (Think about batch processing, boot up 100 machines as spot instance and crunch away).
* For a spot instance you don't pay the bidding price, you pay the current spot price. The bidding price is the cutoff point. If the current spot price exceeds that of the bidding price AWS will terminate that instance.

This last on is also the reason that some user will massively overbid on the spot prices. They don't want their instance to shutdown every so often, so they bid a price so high that the spot price will never reach. Since they will only pay the current spot price the instance will be much cheaper 99% of the time.

# Answer
> 8 votes
A useful bit of information for understanding *why* someone would bid over the on-demand price can be found in the Introduction to Spot Instances:

> Spot Instances can be used to help you meet occasional needs for large amounts of compute capacity (note that the default limit for Spot Instances is 100 versus the default limit of 20 for On-Demand Instances.) If your needs are urgent, you can specify a high maximum price (possibly even higher than the On-Demand price), which will raise your request’s relative priority and allow you to gain access to as much immediate capacity as possible given other requests and the Spot Instance capacity available at the time.

If you have a huge spike in demand for your service (perhaps being linked to from another popular site—see the Slashdot effect), bidding above the on-demand price for a spot instance would help you to get access to far more instances, as noted by the document.

Of course, this isn't sustainable in the long run, and you'd get a much better deal for a long-term compute by just buying on-demand instances (plus there's less risk of being outbid!).

If you're in a situation where you need **a lot of computing power, *fast***—more than you could get from just buying on-demand instances—overbidding might make sense.

# Answer
> 5 votes
If you look at the charts closely, you will see that a spike is always of a very short duration - just enough time for the automated monitoring systems written by the owner to terminate those systems gracefully. In addition, you will occasionally find that the price drops down to 0 immediately following a spike. That is because all systems in that data center are in use as on demand systems, with no systems available for spot pricing the price is effectively zero.

When your spot instance flagged for termination, a message indicating this will be available on the system at the local meta-data uri of http://169.254.169.254/latest/meta-data/spot/termination-time. There will be **3 minutes** until it terminates. More then enough time in most cases to handle termination automatically. Bidding above the demand price is only necessary for deployments which require more then a few minutes to terminate gracefully.

If it is not possible to design your system to terminate gracefully, archive data, etc in 3 minutes you can place a bid higher then the demand price to gain time. The system can even be designed to proactively monitor the current spot price and swap over before the price goes over. But for those times it does, you will need to make a business decision on how much it is worth for the time to terminate gracefully.

It is foolish to pay $100/hour for 4-5 hours in order to keep your system. However, if it will take your system 30 minutes to terminate all the processes gracefully, you can make a business decision how much it is worth to potentially lose any data, or degrade your horizontally scaled service. An e-commerce site with a net profit of $10,000 an hour can certainly afford to pay $1000 to keep 2 spot instances running for 15 to 30 minutes while bringing up demand systems and archiving data.

Web based application can use Elastic Load Balancer to help in addressing termination automatically. A smart implementor would put in place a set of scripts to handle the alert. They could maintain 2 low cost on demand instances which are load balanced - and then use up to half a dozen medium cost systems via spot instances to maintain high performance and spend less then a single on demand system of the same capacity.

Leave 3 of them paying up to $100/hour and 3 of them paying only up to half the on demand price. As AWS terminates instances, ELB will adjust automatically. Giving the automated system up to an hour to adjust for a mere $200.

---
Tags: amazon-web-services, amazon-ec2
---


title: Scope for one click deploy

text:
# Question
Title: Scope for one click deploy
So one of the goals of a software company is to have one click deploy. I think this is a major responsibility of the dev ops team. Whether or not it is should be reserved for another conversation. My question is that our company has multiple individual artifacts that make up a release. Should the developers be able to one click deploy all these artifacts at once, or should each artifact be deployed individually?

# Answer
> 2 votes
The answer is: it depends.

Before I answer, I need to provide definitions for a couple of terms to prevent ambiguity.

1) *Deploy* is defined as the *act of changing executables in an environment*. This might mean that new source is copied over, and some process is restarted. Or a new version of a docker container is placed into an orchestrator - doesn't really matter.

2) *Release* is defined as the *act of making new functionality available to users of the system*.

Do note that while in most organizations *release* and *deploy* happen together, they definitely don't have to be.

> Every released change (usually) requires new code to be deployed. But not every deployment of code means new changes were *released* to be used.

Let us first look at the best case possible. Where you don't have any *button* that requires pushing, to get a deployment to happen. Every change sent to your version control system - automatically gets tested for correctness and is automatically deployed to your system.

Now your question is optionally several different questions:

* Do I have a way to *release* a new feature and is that way manual? (button)
* Do I *release* ALL available (multiple) features at the same time when that button is pushed?
* Do my feature *releases* require changes to happen in multiple decoupled parts of the system?

Since releasing new functionality requires some business approval. For example, the UX and correctness of the new functionality must actually bring value to your customers. It makes sense to have a "dark" release that only some friendly customers can see and provide feedback for. Then when these customers are happy with the value they get, your business can decide to make this new functionality generally available (GA).

Should this feature break the system when released, it is possible to turn off the switch that made it released in the first place. All of which does not require any new code (or old code) to be deployed to the system, because the code is already there.

When you consider that *deploy* and *release* are separate, it makes the decision to deploy everything together or separately simpler.

---

The manufacturing world and Toyota Production System specifically have taught us that having small batches is much more beneficial than batching work items together. TPS even goes as far as to strive for single-piece-flow, where each work item is delivered to the next work center alone and not in a batch.

Agile methodologies have taken a cue from TPS and also describe the benefits of having small-batches. These would be described as a "sprint" with a definite deployed deliverable at the end and striving to keep the sprints as short as possible. This allows to run short experiments and receive feedback regarding the value of the changes created as soon as it is possible to do something about them.

Continuous Delivery is taking this even further by saying that any finished code, should be deployed to your systems as soon as it was auto-tested. And a release can happen at any time when someone decided to do so. And experiments can be validated as part of the development cycle. Moving the definition of "done" for a developer away from "I finished writing my code" and towards "I received feedback from customers that they received value". The shorter the feedback cycle is - minutes, seconds, etc... the better. And batching is a deterrent to short feedback.

By having a button that by definition batches things together, would mean that risk of problems is higher. Learning from TPS, Agile and CD means that single-piece is far superior and has many benefits.

---

The answer "it depends" is correct because all of what I wrote above highly depends on the maturity of your organisation. The amount of automation you have. The amount of communication you have. And many other factors.

Separating release from deployment requires continuous integration and deployment. As well as a system that enable to select which features to release. For example https://github.com/intuit/wasabi or https://www.infoq.com/presentations/etsy-deploy

# Answer
> 0 votes
All these artifacts should be deployed at once. Otherwise, it is not truly "one click."

---
Tags: deployment, build-pipeline
---


title: How do I launch Lambda in my own VPC?

text:
# Question
Title: How do I launch Lambda in my own VPC?
The VPC configurations option in Lambda tells this:

> All AWS Lambda functions run securely inside a default system-managed VPC. However, you can optionally configure Lambda to access resources, such as databases, within your custom VPC.

**So, can't I launch the Lambda function in my own VPC?**

# Answer
The API for creating a function includes a section to choose which VPC to use for running that function.

source: https://docs.aws.amazon.com/lambda/latest/dg/API\_CreateFunction.html#SSS-CreateFunction-request-VpcConfig

It doesn't have to be the default VPC, it can be any VPC.

Consider that "EC2 server inside a VPC" only means that it is some virtual machine that has a NIC connected to a network you have control over in your AWS account (the VPC).

With Lambda, you can choose to run functions on instances that have this NIC connected to a VPC under your control, or you can forego this control and let the Lambdas run where ever they usually do (default system-managed whatever) and just not care about it.
> 3 votes

# Answer
I have also asked the AWS Support folks for help too, as Lambda's security was crucial for us \[HIPAA compliancy\].

This was their response:

> Lambda has two options, you can have your lambda functions without a VPC or you could place your lambda function within a VPC. If you want to put your lambda function within VPC, then the most important thing to note is lambda function must be placed in private subnet. If you want to access internet resources from Lambda or make any AWS SDK API Calls, then you need to have the internet access for which you need to have a NAT Gateway or NAT Instance setup for the VPC, where your lambda function resides and have the private subnet point to the NAT Gateway. When lambda functions are placed within public subnet ( with an IGW route), then the lambda function won't work.

So, a lambda function can be put into a private subnet of a VPC, but not inside a public subnet. And for being able to create instances and ssh'ing into them, from this Lambda, add the private subnet to the security group of the public subnet \[or another hacky way, is to add the VPC's IP range to the security group\].
> 4 votes

---
Tags: security, aws-lambda, aws-vpc
---


title: Release/file sharing service

text:
# Question
Title: Release/file sharing service
At my company we use FTP to share releases and e-mail to notify customers of the availability.

I'm looking for a service offering:

1. Easily upload of a file
2. Users need to sign in to download
    * and keep a record of which user have downloaded what
3. proper UI

Any ideas?

# Answer
What you're after is an Binary repository manager

Quoting from Wikipedia with added links:

> Notable Universal package managers include:\[6\]

I know for sure Nexus and Artifactory match your requirements (even if proper UI is a bit subjective and you mileage may vary).
> 8 votes

# Answer
In DevOps, it is not always about just piciking the right tool, but understanding what is happening also in terms of the workflow. Interesting aspects are here delivered value (like saved time) and how the process can be scaled if you get more customers.

Without knowing further details, I would suggest to investigate what your customers do after they have got the release.

* Do they use it to install at their workplaces? In this case, customer-friendly mechanism is some sort of autoupdate through update site or providing a link to a CDN, content delivery network service (you just distribute your releases after all as *binary content*).
* Or, are these customers technical users, like system engineers, who have to upload the releases which are actually *system components* to further FTP interfaces? Here you might want to find out whether it makes sense to investigate seamless integration with their content delivery pipeline.
* Or, are these releases *executable installers*? In this case, you might consider something either like a Bamboo Deploy Plan or integration with customers' configuration as code system (like Chef or Puppet).
> 3 votes

---
Tags: release-management, artifacts, release
---


# Question
Title: Is it possible to upgrade Jenkins from Jenkins it-self?
On *Manage Jenkins* I've a message that:

> New version of Jenkins (2.47) is available for download (changelog).

Is there any way of upgrading Jenkins from the GUI similar as it's done for the plugins?

I'm using Jenkins ver. 2.43 on Linux.

# Answer
Yes, it is possible. In Windows, this is what I see when there is a new version available:

When you click the Upgrade Automatically button, Jenkins downloads and install the new war file itself.

Please, check Jenkins wiki for further details. There are some problems if you installed Jenkins from a Debian package.

> If you run Jenkins with `java -jar jenkins.war`, Jenkins will be able to update itself from the Manage Jenkins page. This is the simplest way to do automatic upgrades.

### \*nix/Mac

> If you've installed via the OS X installer, the `jenkins` running the process does not own the `.war` file. **To get auto-upgrade working, just fix the permissions so the `jenkins` user can write the WAR** (in `/Applications/Jenkins`).

### Debian Package

> Using the Upgrade-Button from within Jenkins (available since 1.318) will not work if Jenkins was installed from a Debian package (results in permission denied errors when trying to download the new WAR file)!

### Windows Auto-Upgrade

> If you install Jenkins as a Windows service, Jenkins will be able to update itself from the *Manage Jenkins* page.
> 19 votes

# Answer
You can overwrite the existing jenkins.war file with the new one and then restart Jenkins.

This file is usually located in /usr/share/jenkins.

If this is not the case for your system, in Manage Jenkins -\> System Information, it will display the path to the .war file under executable-war.
> 3 votes

---
Tags: jenkins, upgrade
---


# Question
Title: How do I hire a good DevOps, fitting my company?
Good developers should fit rule 11 of the Joel Test:

> Do new candidates write code during their interview?

However, I know little to nothing about Docker containers, how to configure Jenkins or AWS, etc. What would be a good way to test if the person who applies can manage such tasks?

Sub-questions: I believe that a good devops employee can configure the build chain once and it should work from then on out. Can I then also expect him or her to be a developer in the company (DEVops)? As DevOps is a relatively "new" job, is it maybe better for small companies to share a devops employee or hire him or her as a freelancer whenever you need him or her?

My confusion about the term originates from job offers like this, even here on stackexchange (= link where above screenprint was taken from).

I found this helpful, but possible opinionated article you might want to read if you stumble on this question: https://hackernoon.com/devoops-some-common-anti-patterns-1850ac2f5074

# Answer
It is a common misbelief that “DevOps” is a role in a company. The word merely denotes an organisational shift in software companies, this shift can be shortly described by the “You build it, you run it!” coined by Werner Vogels:

> There is no need to distinguish between building and running, and according to Werner Vogels, it is much better than that:
>
> > Giving developers operational responsibilities has greatly enhanced the quality of the services, both from a customer and a technology point of view. The traditional model is that you take your software to the wall that separates development and operations, and throw it over and then forget about it. Not at Amazon. You build it, you run it. This brings developers into contact with the day-to-day operation of their software. It also brings them into day-to-day contact with the customer. This customer feedback loop is essential for improving the quality of the service.
>
> This lesson is interesting, but this particular change in an organization is not always easy to implement.

This shift is concomitant of an increasing importance of continuous integration and deliveries methods. Please note that these ideas are much older than the formalisation of “agile” methodologies, some of them are already present in the NATO Scientific Committee report about software engineering published in … 1968!

While the wording can be overlooked, the short description of the DevOps proposal (this site) mentions “software engineers”:

> Beta Q&A site for software Engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure.

---

Now if you are looking for a software engineer that can help your team getting started with these methodologies, engage them on this topic. Even if you are not familiar with this topic, following the “You build it! You run it!” slogan you can start from a programming problem and lead the interview towards question about deployment and maintenance of the application.
> 13 votes

# Answer
At the risk of getting vote down, off-topic or even heresy, I going to tell you, "Don't listen to the purists telling you that DevOps is not a role."

You are on the right track. **It should fit your company.** That's what matters when hiring.

If you need a DevOps, to work like a DevOps, produce results like DevOps, it is a DevOps.

* Don't let anyone dissuade you otherwise.
* Advertise anyway.
* Interview anyway.
* Hire anyway.

While at that, find out more about DevOps, how the concept can fit into your company, how does the person fits in.

Distill out more about the role you need to fill. Search for questions you can ask. There are plenty of example, even if you are not subject matter expert. Some good old-fashioned patience and hardwork.

Do you need an employee? Or freelancer? A jack of all trades? Only you know your company well.

Create your unique culture that is your interpretation of DevOps.
> 8 votes

# Answer
"Hiring a DevOps" will be tough because DevOps isn't a role. IF you can find an engineer with software development *and* engineering skills, you're on the right track..

**However**,

Your hiring process will be complicated by a few factors:

1. Site Reliability Engineers (Google's term for admin/engineers with coding skills) are rare
2. Rare skills are expensive
3. Rare skills are... rare.

The good news is you may not need a fully qualified SRE.

Your use case, as it were, is hiring engineers to establish your infrastructure (AWS) and automate deployments (Jenkins.) Those skills are not all that rare, but you may have a challenge finding them in a single person.

You also mention that you yourself don't have those skills. This is a further complicating factor. You don't have the skills necessary to evaluate the skills of your candidates. You must ultimately find a way to develop (or hire, as in an Engineering Manager) the skills needed to manage this type of resource.
> 6 votes

# Answer
There are two possibilities:

1. You need help figuring out what are the changes your company needs to go through on your journey to DevOps.
2. You know what you need and just look to hire people to implement this vision for you.

If you need someone to help you figure out which issues you need to concentrate on, there is a really good company called DORA (DevOps Research and Assessment), which is cofounded by Dr. Nicole Forsgren, Jez Humble and Gene Kim. Nicole has run the State of DevOps survey for years and has been keeping detailed statistics of the industry for years. Jez Humble is a guru of Continuous Delivery and literally wrote the book on it. Gene Kim is the author of The Phoenix Project and the last two are co-authors of The DevOps Handbook. DORA provides an assessment of your company and help you answer those questions.

If you have a pretty good idea where you need to go and need someone to help you get there, getting a recruitment agency is probably not a good idea at this point. I would suggest to go to DevOps Enterprise DOES Conference or local DevOpsDays Conference and talk to people in the hallways. Look for recommendations and maybe even open a booth, make a poster about hiring.

Second option would be to come here and start asking those questions in this forum and look for the experts that answer them and get up-voted. Many of those might be actually looking for a job or a challenge. Sometimes even asking questions about your current situation and use case here could point you to the right course of action.
> 2 votes

# Answer
I look for a number of things in the people I have hired into this space.

* a minimum of 'ha' level of agile (google shu, ha, ri for agile adoption if you need more)
* technical hands-on experience - coding, automation, etc, but I'm tool and language agnostic
* learning personality - they take the initiative to constantly learn
* inquisitive and not looking to place blame - they can accept failure as a learning lesson
* they are a team player who can trust and be trusted since psychological safety is the number one success factor for high performing teams
* they can coach and mentor - they share their knowledge without being elitist - prima donnas have no place on high performing teams
* they have "enough" social skills since they need to be team members and communicate well

I have yet to find a perfect candidate, so it's always some type of trade-off. In today's world, I looked through 65+ resumes that were "pre-qualified" as candidates by agencies and HR, discarded 50 or more as not meeting my minimum requirements on paper, interviewed 15+, and hired 3. This process took me the better part of 9 months.
> 2 votes

# Answer
Aside other questions I believe the following answer is yet missing - I want to summarize what I have learned so far in a mostly humble tone:

Introducing DevOps skills to your team is probably not enough (that is to consider that you cannot hire a DevOps)! (Like asking on a lower level what are best tools to make things work, here you seem to want your organization work).

The real question is: what is your oganizations digital transformation strategy and **how much good DevOps will help you to create more customer value and reduce your costs**, still having a good climate?

DevOps reveals the need for change, and change requires taking responsibility.

Imagine a situation were working DevOps might introduce pressure on other teams. DevOps is sometimes a result of evolution from Agile and Continous Integration - what are your assets here?
> 1 votes

---
Tags: culture, software-engineering
---


# Question
Title: Installing phantomjs karma launcher in Ubuntu-Docker context
Karma is a nodejs tool to automate browser testing. It seems that it depends on nodejs plugin fsevents.

But, fsevents is for processing Mac OS X events, therefore I get the error "not supported on your system" for the phantom karma launcher.

Questions I have found on GitHub seem to come from other Mac OS users dealing with upgrading npm.

My question is: how to install karma launcher in headless ubuntu/docker context?

# Answer
Because fsevents is an optional dependency, the solution has been here:

```
npm install --no-optional karma-phantomjs-launcher

```
> 1 votes

---
Tags: testing
---


# Question
Title: Which organization types are targeted by DevOps transformation?
Due to my lacking vocabulary in this context I am not sure how to designate the two types of organizations I would like to talk about before I can put my question. Anybody capable of better wording feel free to improve this. Probably there are mixed types but I suppose I have to polarize a little bit to make the point.

**OrgType1:** "digitalized product organizatons" either B2B or B2C, they have their products, as they become more and more digital, they have some own IT product teams and IT operations. They have found out that they need (more than before) product diversification quite recently.

**OrgType2:** "digital consulting service organizations" (not necesarily just IT consulting but they are typical example here) these guys are B2B only, if they have sustainable products and related operations then it is rather a side effect. Rather, their software product lifecycle is limited to just one or several projects, and that diversification has always been their core business. Actually, the real product is here much more every single member of this organization who depending on the role might, but by far not always does, develop software.

OK then, so OrgType1 is clearly what we already know that DevOps is of a great help, because there are established delivery processes, they get disrupted, need to be optimized, and so on and so forth. Well, bigger organizations consist of many parts, regional business units, business lines, departments, and every one has its combination between operations and their-own-breed-of Devops. Okay - anyway there can be some evolution.

OrgType2 can care about DevOps only so much as it is important for their customer (or they even sell DevOps consulting ;) but then if it comes to onsite software development, you somehow start from zero in almost every project, and this is magnified as with OrgType1 through all departments, business units and so on but it is worth because every "DevOps entity" starts just with 1-2 people who need to discover what they are up to. Seems like OrgType2 is "siloed by design" in a much more dense way which makes it also somehow scalable and persistent.

This all considered and taken into account, the question from my current perspective is: how do DevOps practices apply for the OrgType2, maybe there are resources where this has been explored?

# Answer
> 3 votes
DevOps applies to Organization Type 2, the Business to Business entity as much if not more than the first Organization type. There are two possible scenarios here:

**Scenario 1**

In the first they are selling a product or service to businesses. In terms of DevOps, this makes them virtually indistinguishable from Organization Type 1. Though their customer is a business/group of individuals, they are providing a product or service to a customer. This customer just happens to be a business instead of a private individual, this is is irrelevant to execution of DevOps. Contracted with specific software goals, clearly defined requirement or otherwise, the best value and customer service will be provided by a DevOps model for reasons that do not need to be restated here.

**Scenario 2**

This scenario happens to be a bit more exceptional as in this case, Organization Type 2 is providing consulting services to a business. Presumably and usually, a business hires a consultant for *business* assistance within the consultant's domain of expertise. Businesses have technical support for assistance with specific technologies and can contract en engineer for migrations, but when issues such as risk management, overall architectural design, process, flow and high-level questions arise, this is when a consultant is hired for their *business* advice.

This is why it is important to remember that DevOps is not a role. I would argue that DevOps represents a business or organizational management processes - or even a philosophy. It is more or less a specific way of implementing an agile methodology that has been extended well beyond just development to encompass many disciplines and departments. As such, DevOps is a package deal - in order to be successful at DevOps, you have to be committed to a specific business philosophy of organizational management.

In fact, in order to illustrate this point, In "The Phoenix Project", Board member Erik Reid repeatedly drags the Book's main character, Bill Palmer, to a parts manufacturing plant in order to illustrate and teach Bill things about technology and software development by drawing business lessons from the manufacturing sector.

This is authors Gene Kim, Kevin Behr and George Spafford's way of driving home the point that DevOps is more than just way of doing things, or specific technologies, but is a philosophy of business simply known by various nom de plumes.

So, in order to provide the best value to the company to which a consultant is consulting, it is incumbent upon the consultant to demonstrate and explain the value of DevOps and assist a company in successfully executing on this business philosophy and guide the company to specific technologies and ways of doing that. In short, it is the consulting companies job to act as an "Erik Reid" to whatever business has contracted their consulting services.

In short, DevOps should be very important to Organization Type 2 because it is their job as a consultant to explain to their customer why it is important - they should *always* be selling DevOps consulting. Consultants working with a company who refuses to consider DevOps should consider politely suggesting that perhaps they aren't the right fit. their customer can hire a generic code monkey from anywhere if they want to continue to (unsuccessfully) continue to do projects the way they have always done them.

# Answer
> 1 votes
I feel dev ops is involved more than just the product developed, but the process that is used to go from an idea to in the field. Org 2, I think, could use dev ops to set up a uniform way to gather requirements, incorporate/enforce branching strategy, setup/maintain CI. This is just the dev team.

Every organization isn't just dev. Org 2 could may very well have a sales team. Is there anything that can be done for the sales to gather requirements in a more effective manner, or organize them. Is there a common UI element that all the applications share that all customers complain about.

These are just some of my views on Dev Ops. I am very new to the role. I view the role as being responsible for all the transitions in the development process as well as the development process. Making it easy to go from idea to requirements, to user stories, to work tracking, to testing, to release, then gather feedback to use again as requirements gathering.

I think sometimes you have to create the data you want to mine. This year (2017) Microsoft's conference dedicated a whole day I believe to Dev ops. I was unable to go but the conferences can be found on you tube. One you may be interested in (and I have queued up to watch) is dev ops for every team. Another I just came accross in browsing Keynote with Donovan Brown: From 0 to DevOps (I haven't watched either of those but they seem applicable)

---
Tags: transformation
---


# Question
Title: Different execution contexts for same POM - use properties files?
Imagine that a Maven POM configures a set of integration tests for web apps (could be a functional test as well).

* in developer context, you would clearly use Chrome and configure that accordingly
* in the CI, you want a headless test with another browser driver

Same would be if you decide to configure all fixed dependencies' versions in on place, and for a specific stage/release.

For such cases, what is the best strategy to have environment-specific POM properties files for Maven, and is it a good approach?

# Answer
> 1 votes
I'm not an expert on this, but maintaining separate pom files seems like a recipe for things to get out of sync.

It would make more sense to define specific dependencies within the same pom.xml and execute them depending on specific goals or targets. I.e. -Pdev.

You could also then define a list of shared dependencies used by all targets (i.e. chrome driver or what not), and specific dependencies for each target.

---
Tags: continuous-integration, automation, maven
---


# Question
Title: What is, or could be the formal and/or officially established definion of a "stage"?
Many of us use this term in daily business quite intuitively: "build stage", "test stage" and used long before the rise of DevOps.

But is there a formal and/or well-established definition of this term?

Additional question: Who and where used it for first time?

# Answer
> 4 votes
Often in technology we co-opt English words and give them alternate meanings. However, in this case it's really just the standard definition:

> A level, degree, or period of time in the course of a process

# Answer
> 2 votes
I would argue that referring to these milestones as "phases" would be preferable to "stages." This is because "staging" is already a more prevalent term in use in the industry.

Staging refers to a specific environment used for walking through testing the deployment process. It is distinct from the QA environment used for bug testing and sanity checking and testing environments used for User Acceptance & integration Testing (UAT). All of these could be considered types of test environments.

The origin of this term comes from theater in which actors will walk through a scene - often stopping to adjust, clarify, and note physical positions on stage. This is often also referred to as "blocking" in theater.

Thus, if you call these milestones "stages" you could end up having a "staging stage" and for the sake of clarity, "staging phase" or "staging milestone" would be preferable and more prevalent. These are the more common and customary terms, driven by the terminology of project management.

I would typically dis-recommend using the term "testing" to refer to an environment, because it can be confusing as to *which* testing environment you are referring.

---
Tags: terminology, term-disambiguation
---


title: What is the future role of tester?

text:
# Question
Title: What is the future role of tester?
If the new paradigm is "You build it, you run it" (Werner Voegels, Amazon CTO) which obviously puts much more responsibility - and pressure - on software engineers, what does this change introduce to the task of the test team?

# Answer
IMHO that depends on what the role of the tester was before such transformation. BTW, I believe my answer applies to the DevOps transformation in general, not only to the `You build it, you run it` paradigm.

If the role was a drone job - mindlessly executing tests - it's bound to go away, automation will eat such jobs.

If the role included writing the test plans and specifications they could continue to do that or even morph into writing the corresponding automation scripts, side-by-side with the developers. Or standalone - in some environments it is mandatory to use a completely separate methodology/infrastructure/personnel for testing than the one(s) used for development. In many organisations testers are software engineers, just as developers.

If the role was executing intentionally unreliable/surprise tests, specifically requiring human traits (emotion, subjectivity, inspiration, reflexes, intuition, etc) to complement automated testing - that also won't change.
> 10 votes

# Answer
> "You build it, you run it"

This quote aim at giving emphase on the break between silo teams, one the principle of devops is to avoid silo achieving one task.
While the idea here focus on the build and run phases, the important idea is to bring a whole team together, from architecture to exploitation **roles**. A "Devops Team" would be made of all roles taking part in the software life-cycle, including tester role, with no-one in the team handling a single role.

The main change for a tester would be to learn to voice his/her opinions and feedback on the plan phase in an agile team and probably to take part in some operational tasks.

But all in all, there's no exact rule, what someone previously in a silo would do in a "Devops team" depends on what he/she is more interested in and comfortable to do. That's one of the challenge of a newly created team, spot the best abilities of everyone to share the load on the most efficient way.
> 4 votes

# Answer
I'm a senior engineer, in that capacity I make hires.

At this point in the devops progression, I'd find it difficult to hire a Test who could not easily transition to a Infrastructure Engineer role (what others might call Devops role)

I would not expect a tester for instance to know the intricacies of JVM multithreading or be very opinionated in python class design. I would expect them to be able to understand the source that they may write stub code or generate synthetic data.

I would expect a modern tester to know the basic arsenal of devops within my team. That would include: the provisioning of bare metal servers using our CM tools, the deployment of code via various artifacts or containers.

I would further require them to have some capacity as a Data Engineer in the sense of knowing how the data moves through a system. Thinking about where the single points of failures are and how to simulate failures, throttling etc.

I would never hire a tester to perform module level testing, which I would define as the interaction of a handful of classes (say 1-20) behind a network barrier.

tl;dr I would hire testers to setup environments, simulate (or replay) data through that environment and to cause chaos in the environment.
> 3 votes

---
Tags: culture, testing
---


# Question
Title: What are the pros and cons of running application testing during vs. after a docker build?
Our dockerfile creates the application environment (e.g. env variables, install libraries with apt-get etc.). It also pulls and copies the python application code from its git repo.

However, we are debating whether the application tests (some unit, some integration) should be put within a docker `RUN` command (e.g. `RUN /bin/bash -c "source activate cool_env; pytest"`) or after the build using the CI stack (e.g. Jenkins, Openshift) to execute the tests on the built container.

What are the pros and cons of each?

# Answer
I support J.Doe's suggestion for separate Docker files for each of the build and test stage. Such approach also allows you to:

* re-run the test stage for whatever reason (known intermittent failures, for example) without re-doing the build stage
* run multiple *different* test stages in parallel using the same build - for example instead of running a single long test stage made up of serialized tests, for a potentially significant overall pipeline speeding up.
> 3 votes

# Answer
If you run everything in the same container...

**PRO:** you have solved the CasC challenge for configuring the testing environment inline as well

**CON:** you miss the blackbox tesing part, imagine your container wouldn't accepts connections from outside. Ouch!

Possible solution: I'd go therefore for a multi-stage Docker-based declarative pipeline: one Docker environment for each stage where artefacts cascade along it.
> 2 votes

---
Tags: jenkins, docker, testing, python
---