# Question
Title: How to order physical backup of S3 data?
I'd like to order backup of data to be copied out from S3 to a physical drive (SATA HD) and to be shipped to the address. I know it's possible using Amazon Glacier, however I cannot find such option anywhere. What would be the easiest way to do that?

# Answer
> 12 votes
AWS Import/Export is a service that accelerates transferring data into and out of AWS using physical storage appliances, bypassing the Internet. AWS Import/Export Disk was originally the only service offered by AWS for data transfer by mail. Disk supports transfers data directly onto and off of storage devices you own using the Amazon high-speed internal network.

Import/Export was released on May 2009, and described in this blog post - https://aws.amazon.com/blogs/aws/send-us-that-data/

Released on October of 2016 there is also Amazon Snowball, a service for customers who want to transport terabytes or petabytes of data to and from AWS, or who want to access the storage and compute power of the AWS cloud locally and cost effectively in places where connecting to the internet may not be an option.

Blog post describing the snowball service - https://aws.amazon.com/blogs/aws/aws-importexport-snowball-transfer-1-petabyte-per-week-using-amazon-owned-storage-appliances/

The documentation explains how to prepare a device for export, and how to create the Amazon S3 Export process.

Here are some older docs:

# Answer
> 6 votes
I think you want Amazon Snowball, it's a HD shipping service by Amazon to/from AWS.

> Snowball is a petabyte-scale data transport solution that uses secure appliances to transfer large amounts of data into and **out** of the AWS cloud.
>
> With Snowball, you don’t need to write any code or purchase any hardware to transfer your data. Simply create a job in the AWS Management Console and a Snowball appliance will be automatically shipped to you

---
Tags: amazon-web-services, amazon-s3, storage
---


title: Why has Hudson been renamed to Jenkins?

text:
# Question
Title: Why has Hudson been renamed to Jenkins?
There are two similar CI: Hudson and Jenkins. The Jenkins project was forked from Hudson after a dispute with Oracle. Why was it renamed back to Jenkins and what was the dispute about?

# Answer
The Jenkins website has a blog post on this issue:

> But one issue, which we feel is the most significant issue of all, one for which we now believe no resolution is possible: the rights to the name Hudson.
>
> Oracle has told us that they have trademark applications filed in both the EU and US for Hudson, based on Hudson's creation by Kohsuke while working at Sun. The problem is that this trademark ownership gives Oracle the ability to revoke the Hudson project's right to call itself Hudson at any time, and while Oracle has made an attempt to offer some guarantees (most notably, that binary releases of Hudson, once they've been released with the name Hudson attached, will always retain the right to the name), they are not offering any binding guarantee that the Hudson project will be able to retain its use of the name in perpetuity.
>
> Therefore, to continue using the name Hudson means ceding some of the project's independence to Oracle - if the project and its governance board opted to go in a direction Oracle disapproved of, Oracle would be able to take away the naming rights. Or, in a less dramatic scenario, Oracle could insist on certain changes to the code, infrastructure decisions, process, etc, regardless of opposition from the Hudson development community, in order to retain the rights to the name.

The Jenkins team were concerned that using the name 'Hudson' would give Oracle far too much control over the project, because there was a constant threat that if the project didn't do what Oracle wanted, they'd simply yank the rights to the trademark.

In order to retain independence, the developers of Hudson voted to rename the project Jenkins. However, here's where it gets confusing. Oracle chose to **continue developing Hudson**, essentially making Jenkins a fork because the code started to diverge. There is some dispute about whether this is really a fork or a rename, although it's more an argument over semantics than constructive discussion.

As of now:
> 12 votes

---
Tags: jenkins, hudson
---


# Question
Title: How to store Jenkins job results and produce a summary report?
I'd like to store custom job result data into some format (e.g. in XML) that Jenkins can read and produce the summary report on the job page (ideally in the form of table).

I'd like to know how this can be achieved?

# Answer
> 12 votes
Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page):

> ... allows an easy and fully customized build report display.
>
> Features
>
> * Allow a rich summary report visible from both project and build page
> * Reports must be written in an XML format according to the syntax described in section Syntax Description
> * This plugin allow the parsing of several XML files
> * Reports displayed are ordered according to the XML file names
> * Element displayed are based on

In general, the answer to such questions is usually found by searching the endless Jenkins plugin index.

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: How to get mobile push notifications on Jenkins jobs?
I'd like to receive automatic push notification to my mobile about completed Jenkins jobs and their status (success or failure) in any form. Is it possible?

# Answer
For real push notifications you will need good amount of mobile contracts and infra.

Maybe you can consider using some kind of IM which will send you message to your IM
> 2 votes

# Answer
Jenkins has a bunch of plugins:
> 4 votes

---
Tags: jenkins, jenkins-plugins, notifications
---


# Question
Title: Is it possible to package an application in a .nupkg when creating an application revision in AWS CodeDeploy?
My build process packages my application in a `.nupkg` instead of a `.zip`.

Assuming my .nupkg contains a correctly-implemented `appspec.yml` and is otherwise appropriately bundled, can my deployment revision use it?

# Answer
> 2 votes
Yes. AWS CodeDeploy bundletype is specified separately from the revision location, so you could upload a .nupkg to Amazon S3, set the bundle type to 'zip', and since a nupkg is simply an archive CodeDeploy will treat it as such and upload it to your EC2 instance.

In your nuspec, make sure your appspec.yml gets placed in the root of your .nupkg. My project contains the appspec.yml and powershell scripts to manage windows services in a directory structure like this:

```
deploy/appspec.yml
deploy/lib/get.ps1
deploy/lib/start.ps1
deploy/lib/stop.ps1
...

```

You can include this structure at the root by simply recursively retrieving this directory structure (via double wildcards `**`) in the nuspec files element:

```
<files>
  <file src="deploy\**" target="" />
</files>

```

And then after packing and outputting the nupkg, you can create the application revision as below:

---
Tags: amazon-web-services, aws-codedeploy, nuget
---


title: How to use cloud-init with Terraform?

text:
# Question
Title: How to use cloud-init with Terraform?
I am working with Digital Ocean and Terraform and I already can automate the domain, subdomain, network preferences and the host but there is a section called `User data` that looks like this:

The description of that field says `Allows the use of Cloud-init to configure your droplet`. Looking around I found the documentation.

How to take advantage of this while using Terraform?

# Answer
Cloud-init files are essentially bootstrap codes, that run before each startup, and can - among others - modify files, set up services, create users, etc.

Not all types of droplets support all functionalities of cloud-init, for example CoreOS uses it's own implementation, with a very limited subset of valid values.

To use this in terraform, simply provide the cloud-init file during droplet creation:

`main.tf`:

```
resource "digitalocean_droplet" "web" {
  image              = "coreos-stable"
  name               = "web"
  region             = "lon1"
  size               = "2gb"
  private_networking = true
  ssh_keys           = ["${digitalocean_ssh_key.dodemo.id}"]
  user_data          = "${file("web.conf")}"
}

```

`web.conf`:

```
#cloud-config
coreos:
  units:
    - name: "etcd2.service"
      command: "start"
    - name: "fleet.service"
      command: "start"

```

This will for example create a droplet, where CoreOS will run etcd2 and fleet during startup

You can find some more examples in this repository, where I show how one can use these configuration options to set up some simple docker based services on CoreOS
> 25 votes

# Answer
When you create an Auto Scaling group with Terraform, you can specify the `user_data` to be used by instances created by this ASG. Documented here - https://www.terraform.io/docs/providers/aws/r/launch\_configuration.html#user\_data

You can also create a single EC2 instance, and provide `user_data` to be used - https://www.terraform.io/docs/providers/aws/r/instance.html#user\_data

The AWS EC2 documentation explains how `user_data` is passed to the `cloud-init` service which is running on most Linux distributions available as AMIs on AWS - http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/user-data.html#user-data-cloud-init
> 6 votes

---
Tags: terraform, cloud-init
---


# Question
Title: Relation of devops with release management
How does DevOps deal with release management?

Is release management is the part of devops lifecycle?

Do devops take active part in release management or the organization has hire to another professional altogether.

# Answer
> 5 votes
Your use of the term "devops" suggests that you see it as a role. I humbly suggest that developers that are part of a Scrum team aren't called "Agiles," so DevOps practitioners aren't "devops." :)

Release management is absolutely part of DevOps. It's one of several specialty engineering roles. Staffing for the skill of release management varies widely from one company to the next. The size of the company, the industry, and the size of the IT department all influence how (and if) that skill is isolated to a person, a team, or distributed across IT.

Organizations of medium to large size tend to have dedicated staff that understand the practice and tooling of Continuous Integration. This includes some systems admin skills, and increasingly includes development skills, at least scripting. The growth of Infrastructure as Code as a practice is blurring the lines between release managers and developers.

---
Tags: release-management, release
---


title: How to deploy software every second?

text:
# Question
Title: How to deploy software every second?
Companies such as Amazon, Etsy, Google, Facebook and others are publicly saying they deploy software thousands time per day.

What are they doing that is so different from other companies? How can other software companies deploy software thousands times per day, what is stopping them from doing so today?

# Answer
> 3 votes
The "how" of extremely high-frequency deployment is all about *very* mature engineering practices and tooling. The "unicorns" (Gene Kim's term for the type of company you list) have some of the best talent in the industry, and invest in the best technology.

One of several key practical differences between these unicorns and most other orgs is the level of automation they employ. It is very difficult to automate the quality checks needed to maintain a constant flow of work from DEV, through testing environments, to Production. Development *and* Operations must maintain high service levels internally. Ops must provide self-service resources so developers aren't waiting on tickets to get new infrastructure.

# Answer
> 2 votes
These big companies are using unconventional solutions, such as the BitTorrent deployment system and geographically distributed content delivery networks (CDNs). When Facebook updates its code or generates a new build, the binary files needs be pushed to all of the company's servers (~1.5GB), so they created its own custom BitTorrent tracker designed to obtain data slices from other servers that are on the same node/rack which reduces total latency.

Source: Exclusive: a behind-the-scenes look at Facebook release engineering

Facebook deployment takes the following steps:

> 1. Update goes to an "a2" tier which rolls update out to a small, random collection of users.
> 2. Check-in procedure on company IRC where all developers who submitted code for the update have confirm that they are ready to respond if there is a problem when the update goes out.
> 3. Head of release engineering issues command in terminal to begin deployment
> 4. Watch web-based monitoring dashboards as the update rolls out.

Frequency of deployments is one minor update on most business days, one major update on a weekly basis.

Source: How Facebook Does Deployment

# Answer
> 1 votes
Not exactly a full answer, but you need mostly three things:

* Automation of the delivery pipeline (Continuous Delivery)
* A very, very strong test system (including canary release)
* Good coding practices to ensure versions can run concurrently on the same environment.

This usually include tooling for code review, code linting, unit testing, orchestration, configuration management, integration testing and a strong culture for the change.

---
Tags: deployment, automation, dtap
---


# Question
Title: What is the origin of "Release early and often"?
Where does the origin of the Release Early, Release Often (RERO) principle come from?

# Answer
> 4 votes
One of the earliest references to this term originate in Jim McCarthy's book Dynamics of Software Development. This term seems to also be embedded in the Lean Startup movement, which built upon ideas in Lean Manufacturing, TRIZ, and other sources explaining how to evaluate an experimental product and check if it fits the market before investing too much effort in its completion.

# Answer
> 3 votes
Eric S. Raymond is credited with popularizing "Release Early, Release Often" in his essay "The Cathedral and the Bazaar" where he credits Linus Torvalds as using that approach in the development of Linux to it's success.

---
Tags: release-management, agile
---


# Question
Title: How to terminate EC2 instance in Jenkins via Vagrant when job is stopped manually?
I've freestyle jobs which aims to run EC2 instances via Vagrant (`vagrant up`), however when I cancel the job manually (by clicking X button), the instances on EC2 are still running. How can I make sure that the instances are terminated when the job is cancelled?

# Answer
> 3 votes
This can be achieved by using Post build task plugin and by adding the following *Script* command in *Post-build Action*, *Post build task* solved the problem:

```
vagrant destroy -f

```

Plus adding rule with *Log text* as "Build was aborted" make sure that above command is only invoked on aborted builds.

For example:

---
Tags: jenkins, amazon-ec2, jenkins-plugins, vagrant
---


title: How is DevOps related to Agile?

text:
# Question
Title: How is DevOps related to Agile?
Agile has been around for quite a while. And several offspring came out of it, such as Extreme Programming, Scrum, Lean Software Development and others.

Isn't DevOps just another offspring of Agile? Is there any substantial difference between DevOps and Agile?

# Answer
> 4 votes
DevOps and agile aren't same thing but they are complimentary to each other. Agile development is development that adheres to the principles stated in The Agile Manifesto. In brief, agile is the word for an environment in which the priorities, according to the authors of the manifesto, are-

* Individuals and interactions over processes and tools
* Working software over comprehensive documentation
* Customer collaboration over contract negotiation and
* Responding to change over following a plan.

The point of DevOps is to deliver technology to business units in a timely fashion and ensure the technology runs without interruption or disruption. To achieve it's goal DevOps mostly focus on-

* deep communication between software development and IT operational groups
* automated deployment processes

Below are list of relation/differences with agile and DevOps-

* Agile software development is a methodology for developing software, but DevOps, on the other hand, is all about taking software which is ready for release and deploying it in the safest, most reliable manner possible.
* Agile is all about development. Sometimes, it takes over the entire company. Even when it does, agile discipline doesn't inevitably lead to DevOps. The practice of DevOps involves a separate discipline and methodology from those of agile.
* Agile practices, like Continuous Delivery practices, can be part of a DevOps activities.

# Answer
> 3 votes
I would say that DevOps is a *peer* of Agile, not a child. Agile methods apply largely within software development. DevOps, by contrast, applies the same Lean Manufacturing/Lean IT concepts *across the software delivery lifecycle,* which begins with the business and ends with business value delivered (code running in Production.)

I believe, after some thought on the subject, that there is also a dependent relationship between Agile and DevOps. DevOps is intensely focused on automation. There is limited benefit in automating a process that executes infrequently. Waterfall development methods are a perfect example of a slow, infrequent process.

Agile methods are focused on maintaining a constant flow of work through Development. This is a perfect, natural fit for DevOps, which keeps that flow going all the way from the business to the end state, which is code running *in Production.*

# Answer
> 1 votes
DevOps is not an 'offspring of Agile', though it has been influenced by concepts from the *Manifesto for Agile Software Development*. DevOps consists of a set of characteristics of building, deploying, and monitoring software systems, as well as configuration management of source code, with a heavy emphasis on continuous integration and continuous delivery (CI/CD).

Adam Jacobs effectively defines *DevOps* in this presentation: AWS re:Invent 2016: What hundreds of companies show us about how to build a DevOps Culture (DEV320)

On a related note: about the use of the word *Agile*, Dave Thomas corrects its usage in the GOTO 2015 presentation Agile is Dead

# Answer
> 1 votes
Scope-wise, Agile covers **development** of software, i.e. mostly/only the `Dev` aspects of `DevOps`.

They best work together, however they are not exactly conditioned/dependent on each-other:

* non-devopsy orgs can still use agile in development
* devopsy orgs can still use non-agile development methodologies

---
Tags: culture, agile
---


# Question
Title: How to prevent failure from introduced changes?
The DevOps Report by Puppet Labs states that "devopsy" companies have 3x times lower failure rate. While other companies must have a fairly regular rate of failures from deployments to production.

What are the companies who have very low failure rates do differently? How can other companies mimic such decrease in failures of changes in their production (and other) environments?

# Answer
This is pretty deep in to "opinion based" but here goes:

While most definitions of "DevOps" are independent of specific tools and techniques, it is very often combined with things like test automation (e.g. Jenkins, Travis-CI) and deployment automation (e.g. Chef, Capistrano, Kubernetes). Those two together act as a forcing function to push up quality levels of software by making it both easier to use tests to ensure things don't break and making it easier to fix breakages when they do occur.

That said, I really doubt that number captures the full story. Like so many things, "software quality" is easy to talk about but difficult to define.
> 0 votes

---
Tags: deployment, automation, culture, dtap
---


# Question
Title: How to upgrade job-dsl files to Jenkins 2 pipelines?
Is there a common practice for migrating a large repository of Jenkins job-dsl to Jenkins2 pipelines? I am looking for ways to test the behaviour of builds, that go a little further than comparison of generated artifact from same code base.

Actually we have a quite large infrastructure, deployed via Ansible, and we got a verification environment which is topologically equivalent to production, so that would not be a problem. I am more interested in validation practices we could automate, as we build more from than 300 project repositories, there are a few thousand jobs spread across a few Jenkins masters.

# Answer
No, there still isn't any common practice for this since this is not very common.

I would recommend you to create a `Development Jenkins Environment` and run tests against it. Right now you will have to do that from scratch since there are not many open source repositories that could help you.

So, how does this `Jenkins Development Environment` look like?

For me it is a `docker-compose` file that you can run and get everything you need. If you can automate the creation of the jobs it will be a plus.
> 6 votes

---
Tags: jenkins, job-dsl, jenkins2
---


# Question
Title: Is it possible to build Jenkins jobs without using build numbers?
I have a job which is running very frequently, so I'd like to avoid high number of builds. Is there any way to drop job build numbers, so the job is constantly ran, but without high numbers of builds?

# Answer
It's not possible to avoid setting the `BUILD_NUMBER` environment variable, but you can avoid using it. Try using the Jenkins plugin Build Name Setter Plugin. It adds an option in your build configuration to set the build name to whatever you want, and you can ignore the `BUILD_NUMBER` environment variable this way.

Consider `1.250.5` versus `1.250.${BUILD_NUMBER}`. The former will be constant no matter how many times you build and the latter will increment by one every time you build. It's also possible to use other macro values in between ${} to make more interesting dynamic build numbers.
> 7 votes

# Answer
No, but you can just ignore it which you would be doing anyway if it wasn't there.
> -3 votes

---
Tags: jenkins, jenkins-plugins, builds
---


# Question
Title: To design a blue green deploy how to publish websocket traffic from live to hot swap server
A blue green deploy involves pumping live prod data flows (blue) into a hot swap non prod environment (green) in preparation to deploy to live the green environment, such that the green has full data sync with the previous prod blue environment.

I am wondering what people are using to live copy ongoing websocket traffic from blue to green do I write my own or is there a publish/subscribe websocket library or possibly a different approach to blue green?

My app has nodejs REST servers which also manage websocket traffic from mobile devices ... a mongodb server, etc ... each in containers on GCE/AWS

I realize I could just keep the mongodb in sync from blue to green however that would not exercise the green nodejs servers with live traffic which is a nice regression sanity check I am looking for

Possibly if I just live forward HTTP traffic the underlying websocket which runs atop HTTP will just take care of itself and not demand specific blue green setup

# Answer
> 5 votes
With a minimally engineered server code (a Google AppEngine application in the example) it's possible to duplicate and forward incoming HTTP traffic received by a production server to a test server (or vice-versa) for the purpose of comparing their responses. See https://stackoverflow.com/questions/38799566/can-i-asynchronously-duplicate-a-webapp2-requesthandler-request-to-a-different-u.

# Answer
> 6 votes
Websockets don't really have any handling for this per se. You could build your own system but generally the idea for HA is that if anything happens you'll set the client up to automatically reconnect. There will be some non-zero downtime during a blue-green swap but if your app is well built it should handle that already.

If you wanted something more smooth, you could build a message in to your protocol to connect to a new server and then drop the original connection.

---
Tags: deployment, bluegreen, http
---


# Question
Title: Ansible playbook for configuring runit to supervise an uberjar?
We are using Ansible 1.9 for deploying Spring Boot microservices. We are using CentOS 6 and the services are packaged as rpms. We intend to configure the executable jar files to be supervised by runit, instead of SysVinit or Upstart.

Is it reasonable to package the runit config inside the RPM, and let Ansible only deploy specific properties into `/etc/sysconfig`, or does a better way to use Ansible for the full runit config exist, so that the service would be decoupled from that host configuration?

# Answer
> 3 votes
So both have their place. On one hand, putting service definitions in the package helps to keep the number of "moving pieces" to a minimum. On the other, it also requires a more complex package build process to at least some degree (like you would have to build your own packages for things instead of using distro packages). If you have the capability to do it in packages, that seems better since moving pieces in prod will always eventually break.

---
Tags: ansible, microservices, runit, spring-boot
---


# Question
Title: How can I manage changes in desired capacity due to scaling policies in AutoScaling Group state?
How can I manage changes in desired capacity due to scaling policies in AutoScaling Group state in terraform?

Specifically, suppose I provision an aws\_autoscaling\_group resource with terraform with a desired\_capacity of 4 and a scaleup policy on high CPU utilization. Later, the autoscaling group has updated via scaling policies to a desired capacity of 6 but this state isn't captured in my terraform .tfstate.

If I want to make a subsequent change to my autoscaling group state via terraform, how can I make the change without resetting the desired\_capacity (since it has not changed in the .tf)? Can I automate updating the desired\_capacity to match current group size somehow, or should I just not set desired\_capacity at all?

# Answer
> 10 votes
The `desired_capacity` in Terraform is marked in the documentation as optional. So with a proper `min_size` value, Terraform can wait until the minimum capacity is reached before continuing.

The above, with the addition of scale policy can effectively manage capacity without being specific about `desired_capacity` in your Terraform code. This will prevent you from having Terraform override the capacity in place at the moment of execution.

---
Tags: amazon-web-services, terraform, autoscaling, capacity-planning
---


# Question
Title: How do I work around the fact that AWS SQS is not HIPAA compliant?
I have a use case where data from S3 is queued into AWS SQS, which is in turn connected to CloudWatch, whose metrics will be triggering AWS Lambda.

However, I want the architecture to be HIPAA compliant. So, I have come up with this idea:

1. Once my S3 bucket gets a file,
2. Fire up a Lambda function, which does hashing/name scrambling of the files, and copies to another S3 bucket (via `aws cp`)
3. Connect the bucket with the hashed/scrambled names to the SQS queue

**Is this a good and secure practice? Or is there a better workaround?** (Would be more than happy if I can send encrypted keys of S3 to SQS. But not sure if I can or if it is possible)

# Answer
According to Amazon AWS

> Customers may use any AWS service in an account designated as a HIPAA account, but they should only process, store and transmit PHI in the HIPAA-eligible services defined in the BAA. There are ten HIPAA-eligible services today, including AWS Snowball, Amazon DynamoDB, Amazon EBS, Amazon EC2, Amazon Elastic MapReduce (EMR), Amazon Elastic Load Balancing (ELB), Amazon Glacier, Amazon Relational Database Service (RDS) \[MySQL, Oracle, and PostgreSQL engines only\], Amazon Aurora \[MySQL-compatible edition only\], Amazon Redshift, and Amazon S3.

source: https://aws.amazon.com/compliance/hipaa-compliance/

This means that as long as you are not storing or transmitting PHI in SQS, just the information about where this PHI is being stored - you probably can pass an audit reg. HIPAA compliance.

In the architecture you describe, the SQS queue does not need to include any PHI content. This would make it comply with the above statement.

More information about HIPAA compliance on AWS is available in this whitepaper from January 2017 - https://d0.awsstatic.com/whitepapers/compliance/AWS\_HIPAA\_Compliance\_Whitepaper.pdf

Specifically SQS is mentioned and explained in the HIPAA FAQ - https://aws.amazon.com/blogs/security/frequently-asked-questions-about-hipaa-compliance-in-the-aws-cloud-part-two/.

**update**: As of May 1st 2017, SQS is now HIPAA compliant. https://aws.amazon.com/about-aws/whats-new/2017/05/amazon-simple-queue-service-sqs-is-now-a-hipaa-eligible-service/
> 19 votes

---
Tags: amazon-web-services, amazon-s3, amazon-sqs
---


# Question
Title: What is a good strategy to keep my site online when S3 goes offline?
What is a good strategy to keep my site online when S3 goes offline?

If S3 US East 1 goes offline, how should I have my app configured/structured to prevent that taking my entire site offline?

What are the best strategies to diversify in this sort of situation?

# Answer
> 29 votes
In March 2015, Amazon AWS announced they support S3 replication across regions. When a certain region in S3 goes offline, you can serve files from your mirror in another region.

source: https://aws.amazon.com/blogs/aws/new-cross-region-replication-for-amazon-s3/

The practice of keeping your infrastructure online by doing a switch over to another region is a complex one, but S3 is a relatively small and simple component. Netflix has a great article on their experience with Chaos Gorilla.

This also applies to service degradation, like increased latency. Not just when a service you depend upon is completely offline. Netflix has an article on this as well: Chaos Engineering Upgraded.

# Answer
> 11 votes
What you are asking for is, basically, high-availability. In order to make a system highly available, you need three things:

1. Eliminate single points of failure
2. A mechanism to switch from an endpoint to another
3. A way to detect failures

## Eliminate single points of failure

In the case of S3, the point #1 is addressed, as Evgeny pointed out, by S3 cross-region replication.

Replication, however, is not instantaneous and you'll want to check if you want to make your application replication aware or not. In the event of an outage, it is possible that something that was written to your source bucket has not yet made it (wasn't replicated) to the destination bucket. You have to think how the application would handle such a scenario. That really depend on the type of data, what is being done with it and (potentially) the end-users or management expectations.

## A mechanism to switch from an endpoint to another

For S3, that means that in the event of an outage, you want the application to stop reading and writing from/to bucket A and use bucket B instead.

How to achieve this is, as far as I know, up to you for now. Some other AWS services offer completely transparent fail-overs, but I'm not aware of such a thing for S3 at the moment.

There are various ways to achieve this. One example is using a proxy that will route the traffic to the appropriate bucket. During an outage, you would update/change the proxy to route the traffic to a bucket not affected by the outage. Another example would be to make your application configuration dynamic and store it in a key-value store. If the application reads the KV store for updated properties often enough, you can switch where you read from and write to (Spring Cloud has support for an "EnvironmentChange" listener, for example).

## A way to detect failures

Well, that one is easy, I think. Simply setup a write + read loop and alert as soon as something isn't right :)

## Closing notes

* If your application is writing to the bucket, you have to think about what would happen in the case of a fail-over. Have all the writes made it to the destination bucket (and can you tell)? Can you allow writes to the destination bucket (making it the new "primary")? Careful planning will avoid split-brain or lost updates scenarios.
* Depending on your SLA, you may want the points #2 and #3 to be automated or automatic. That requires additional planning, tooling and testing, but well-written scripts will always react faster and in more predictable ways than human can (failures also have the annoying habit of happening in the middle of the night when human intervention is something hazardous.
* It's worth mentioning that even cross-region replication doesn't completely eliminate single points of failure. Sure, if a region goes down, you are covered. But what if a US-wide AWS outage happens? Azure had a partial but global outage last year and one in 2014 too.

---
Tags: amazon-web-services, amazon-s3, high-availability
---


title: What is "black box" testing?

text:
# Question
Title: What is "black box" testing?
What is "black box" testing and how it is different from ordinary testing? For example black box tests prior to deploying to production.

**Note**: "black box" testing is a common requirement when applying for devops roles (not necessary a tester thing).

# Answer
> 6 votes
I will make my answer, based on the knowledge for security testing, but IMHO this can be generalized.

* Black box testing - when the tester know nothing about the system, components, liaisons, connections, etc. This can be helpful more like UI/UX testing, functional testing. Example: you do not work for Microsoft and also you do not have the source code and need to test MS Office
* Gray box testing - when the tester have some knowledge about how the system work, some information about internals.
* White box testing - when the tester have or can get all the information about internals, connections, principles of work of the tested product. Example: testing Linux kernel.

For reference:

Black box testing (Wikipedia)

White box testing (Wikipedia)

Gray box testing (Wikipedia)

# Answer
> 2 votes
As answered by Romeo Ninov in Black Box testing, tester is unaware about the application internal structure.

> This method is named so because the software program, in the eyes of the tester, is like a black box; inside which one cannot see. This method attempts to find errors in the following categories:
>
> * Incorrect or missing functions
> * Interface errors
> * Errors in data structures or external database access
> * Behavior or performance errors
> * Initialization and termination errors

Refrence:

---
Tags: terminology, testing, dtap
---


# Question
Title: Key attributes of Continuous Integration (CI) software
I work in the IT department of automation/dev company and we are just about to implement/add CI into our set of tools. And we are having difficulties to choose the one.

At the moment we've been thinking about these systems:

* Jenkins
* CircleCI
* TravisCI

**Question:** What are key attributes of CI's soft which the systems should have and based on which we should compare them?

---

**EDIT:** We are expecting from CI the standard things: From building the applications, running tests (Unit/Integration/Performance/..), saving statistics and providing information either via email/on the page (reports).

Issue there is that we don't know which if there are more functionality and if, what is it? This is what I would be looking for under words **"key attributes"**. The names above are just for reference, I'm looking for answer **"why use** this or that" over simplified **"use this or that"**.

We are developing in .NET (C#) environment with some libraries written in C++/VBA.

# Answer
> 13 votes
In general, there are five main differences between different CI software solutions.

1. **Cost**: Is the software open source and free or proprietary? Does your DevOps team already have a budget for software or are you expected to compare free options?
2. **Maintenance**: Is the software something you need to host on your own and maintain, or is it being offered as a service? If you need to host it on your own, do you have the team members available who can run the maintenance? If it is being provided as a service, where does your code reside?
3. **Security**: Are there any security or compliance concerns about where the build machines are hosted? Are the permissions required by the CI software into your SCM well understood?
4. **Community**: Does the CI software have a good community around it with the plugins and support for the technologies you use? Is it easy to get support or troubleshoot when things go wrong?
5. **UX**: How easy is the software to use? Does the CI software support scripts stored within your SCM for the tasks you need to do? Is the reporting easy to understand or will it require some training?

# Answer
> 6 votes
The **SCM system** you use can be essential in making your CI choice.

Using a private/intranet solution, for example, pretty much excludes `CircleCI` and `TravisCI` as these only support cloud-based `GitHub` and/or `Bitbucket`.

`Jenkins` has plugins supporting many SCM systems out there, see Which SCM tools does Jenkins support?. But using a less popular or a wrapped/customized SCM system may require development of a new plugin.

Also particularities of your **building/testing framework/infrastructure** may matter, for similar reasons (configuration, location/access method and compatibility/support).

---
Tags: jenkins, continuous-integration, travis-ci, circleci
---


# Question
Title: How to connect a build node to a Jenkins master?
I'd like to connect a build node to the Jenkins master. Ideally without using a web browser or being in front of the physical machine. What would be the easiest way to achieve that?

# Answer
> 3 votes
Connecting a node to Jenkins through a terminal only is a two step process.

First, you should ssh into into your node machines and set them up with the following steps:

1. Make a jenkins user shell: `sudo adduser jenkins --shell /bin/bash`
2. Create the directory after switching to the jenkins user: `mkdir /home/jenkins/jenkins_slave`
3. Create your SSH credentials to be used in the following bash script.

Then, once your nodes are set up, you will need to connect to your jenkins machine and run bash script such as this one to connect your nodes to your master:

```
#!/bin/bash

JENKINS_URL=$1
NODE_NAME=$2
NODE_SLAVE_HOME='/home/build/slave'
EXECUTORS=1
SSH_PORT=22
CRED_ID=$3
LABELS=build
USERID=${USER}

cat <<EOF | java -jar ~/bin/jenkins-cli.jar -s $1 create-node $2
<slave>
  <name>${NODE_NAME}</name>
  <description></description>
  <remoteFS>${NODE_SLAVE_HOME}</remoteFS>
  <numExecutors>${EXECUTORS}</numExecutors>
  <mode>NORMAL</mode>
  <retentionStrategy class="hudson.slaves.RetentionStrategy$Always"/>
  <launcher class="hudson.plugins.sshslaves.SSHLauncher" plugin="ssh-slaves@1.5">
    <host>${NODE_NAME}</host>
    <port>${SSH_PORT}</port>
    <credentialsId>${CRED_ID}</credentialsId>
  </launcher>
  <label>${LABELS}</label>
  <nodeProperties/>
  <userId>${USERID}</userId>
</slave>
EOF

```

---
Tags: jenkins, linux
---


# Question
Title: How to change existing policies in an organisation?
I assume that an organization wishing to do a DevOps transformation has some problems and policies it is interested in changing. This interest can come from top managers, middle managers, or even from bottom up. One of the biggest factors impeding this change is making other people buy into doing the change.

For example, in many cases pushing "new" ideas such as Agile, often fails. People resist change, and it seems like a wall that stops good things from happening. Yet there is a mandate for good things to happen.

What methods can be used to affect employees in an organization starting its DevOps transformation? Especially specific techniques and ways that are found to work. Specific as-in more Engineering, less hand waving.

# Answer
You have to understand that processes change people who follow them. As people learn, internalize and get better at a process, it changes the way they learn how to solve a particular problem. A set of similar processes reinforce each other into a mindset the person uses to solve a category of problems and eventually form a set of values that guides decisions and new solutions to new problems.

Even if you change the process, without the change to the mindset and even more crucial to the values, the person will simply adapt the new process to conform to the same values, same mindset or even same solution as in the original process. At a certain point, it is not possible to divorce this person in this position from the mindset acquired or change the underlying values.

To institute a change, you have the following two options:

1. Bring in a person who will already have the right values and mindset and in best case scenario, understand the process that needs to be followed without your help.
2. Bring in and empower new employee, either recently hired, a new hire or a transfer from a different team in the organization and train him in the new process, hoping to instill the new mindset, hoping the new set of values will emerge.

If the change is local, you might prefer an internal transfer as that person would already share the global company wide values which you want to retain. In case of a larger change, you need to bring in someone from outside to have a fresh perspective and to not share the company wide values which you might be trying to change.

The important part is to empower the person, team or business unit to follow the processes and insulate them from the old team, other teams or the rest of the company, respectively, which might still follow the old set of processes. As it is very hard to insulate such change agent from management above, if the change is to be larger, it often needs to follow all the way up the management chain or come all the way from the top of it.

**Note**: It is hard to bring change to more than just your team without the support of management. Even inside your team it is difficult if others are already set in their ways. For a new team in a new company a successful evangelist can often affect the forming policies even without the support of management simply by being a leader or creating the path of least resistance for others to follow. But in established company, see above.
> 7 votes

# Answer
## Hack your Team

Bringing about change in your organisation is hard. People have habits, they resist change, and they are often comfortable with the status quo. To bring about change, in no particular order, here are a few tools you can use.

1. **Cause others to experience the problem that DevOps solves.** Many times the benefits of DevOps is only understood on a theoretical level by your team. Most of the problems that happen during deployment are hopefully and rarely experienced by the rest of the development team or management. To fix this, make sure that you are vocal about issues when they arise and mention how this problem wouldn't have happened if the team was using a continuous integration solution. Another possibility is to be sure you ask developers to fix problems that their code caused during deployment instead of fixing it yourself.
2. **Find the leaders**. It is common for people to follow the leaders, be they management or just the most popular/commanding person in the group. Get those leaders on board with your desire to move to a DevOps culture, and devise public ways in which they can be seen using or advocating best practices.
3. **Build Trust**. We are more likely to agree to things from people after we have already agreed with them once or twice before. Ideally, you can find small improvements that can be made without a shift in culture and build upon that success. However, if that isn't an option, ask them simple questions and offer simple suggestions so they get into a habit of saying yes or agreeing with you.
4. **Don't be ashamed to repeat yourself.** Repetition works and eventually sinks in. Whenever possible mention how great things would be if the team was using DevOps. However, this only works if you have first built trust within your team.
5. **Make it pleasurable**. If you are allowed to build a proof of concept for your DevOps situation, use cute emoticons and cheerful colours in the reports and notifications. Post funny gifs when a build fails. Make sure you aren't annoying with your updates.
> 5 votes

---
Tags: culture, transformation
---


# Question
Title: Is cross-region replication 100% fool-proof for S3 region outages?
Amazon S3 has an option of cross-region replication which should be pretty fault-tolerant against region/zone outages.

Does that mean those who are ranting about the outage did not make use of this aspect?

Or is that cross-region replication is not completely fool-proof and would not have helped?

# Answer
> 11 votes
The drawback when there's replication come from the note below:

> Amazon S3 routes any virtual hosted–style requests to the US East (N. Virginia) region by default if you use the US East (N. Virginia) endpoint (s3.amazonaws.com), instead of the region-specific endpoint (for example, s3-eu-west-1.amazonaws.com).

When you use replication you usualy let AWS takes care of routing the alias to one region by targetting `s3.amazonaws.com` in your REST request from your servers and let the redirect do it's job.

Whenever N.Virginia is down, the magic cease to work and you're out of luck to access your data and have to update your configuration to choose a specific region endpoint.

The problem does *not* come from the DNS (a request to the bucket itself will work) but from S3 clients, which will connect to the S3 API endpoint before accessing the bucket, in this case the dns resolution is done on `s3.amazonaws.com` and this is us-east-1 endpoint.

When you use regions alias, you loose the ease of load balancing over regions with the health check from AWS included.

If you use DNS cname targeting the regions to switch quickly, you're responsible of your DNS TTL but nothing guarantee cache servers of client ISP will honor your value (one of many cache your client may encounter).

And lastly, if you try to load balance yourself you'll probably create the same SPOF than AWS already have with the added burden of maintaining it.

AWS is working on it but that's all the information I have at time of writing.

# Answer
> 10 votes
Many big companies would be at fault for not using this feature. It does add additional cost, and historically any kind of *real* disaster recovery solution is untested even if implemented.

Other than the cost issue, companies who are actively using cross-region replication can offer a valid concern regarding the latency it takes for an object to replicate. S3 does not allow (as far as I know) read-after-write consistency on replicated objects, while it does allow it for a bucket in a single region.

This SE question raises a concern where objects are not being replicated properly, or take too long to replicate. Provided that cross-region replication is done in a mode of eventual consistency, there are a lot of concerns to address.

---
Tags: amazon-s3, high-availability
---


# Question
Title: How can I keep track of my Cloud Resource use?
I am trying to automate my AWS application deployment with Jenkins.

Right now if we want to update the application in any of the environments, say UAT, we build our docker images, find the current ECS Task and update it with the new image, find the running ECS Cluster, and update the tasks.

Broadly, what is best practice for keeping track of cloud resource Id's (ECS Cluster Id's, ECS Task Id's, EC2 Id's, etc) in your Continuous Integration environment?

# Answer
A judicious usage of one of the cloud "orchestration" tools, such as Terraform or possibly Fugue seems to be the best way.

You can start small, pick a less important and not very extensive environment, carefully decipher it into automatable code and proceed from there.

Broadly this is referred to as infrastructure-as-code, for googling and other buzzword-oriented media applications.

More specifically, say, if you had your environment described in Terraform's HCL files, you would have required ID's exported as "outputs" and could operate on those from scripts/Jenkins jobs/etc
> 9 votes

# Answer
Netflix have presented their method of dealing with the problem of resource sprawl. They created the Janitor Monkey which is responsible to clean-up resources when it is apparent that those are not being used.

To track the created resources in Amazon AWS, generate an audit log, and allow to search through time (and history) they also created Edda. Edda allows to store and search through your AWS resources.

There is some mention that Janitor Monkey might be integrated with Edda, but they have not published much information about these since.

Today with AWS CloudTrail and AWS Config Rules it is possible to achieve similar results without third-party software. When Edda and Janitor Monkey were announced (2012-2013) both of these services were not yet ready to solve these kinds of problems.

With CloudTrail and Config, the tracking of resources is already in place. All to do is write a script that will decide what to do with these by going over the list from time to time.
> 1 votes

---
Tags: continuous-integration, terraform, automation
---


# Question
Title: How to automate TeamCity installation using docker?
JetBrains TeamCity docker image describe build steps for already installed TeamCity.

Is there any way to automate installation of TeamCity it-self?

For example creating the admin account, enabling standard authentication and setting up the project?

# Answer
Automation of installations like this can often be done with config management tools. GitHub and the public repos for these tools is a good place to start.

There is a Puppet installer for TeamCity here: https://github.com/haf/puppet-teamcity

Here's a Chef cookbook: https://supermarket.chef.io/cookbooks/chef-teamcity

You can also use tools like Packer to generate your image(s). Packer can substitute for a Dockerfile in generating a Docker image. Packer will also run Provisioners (including Chef/Puppet/Ansible) to configure your image.
> 9 votes

---
Tags: docker, automation, teamcity
---


# Question
Title: How to automate installation of Windows installer on Appveyor CI?
I've the installer in EXE format (`mt5setup.exe`) and I'd like to download it and automate the installation process on Appveyor CI. On Linux I would normally use xdotool.

Since this installer doesn't have any silent flags to use, how can I automate the installation process headlessly?

# Answer
The installation of Windows application can be automated using AutoHotkey (AHK), similar as Winetricks does this for all common apps which support.

Here is example of AHK script which aims to install mentioned application:

```
Run, mt5setup.exe
WinWait, MetaTrader 5 Setup
ControlClick, Button1
Sleep 100
ControlClick, Button3
WinWait, MetaTrader 5 Setup, Installation successfully completed
ControlClick, Button4
Process, Wait, terminal.exe
Process, Close, terminal.exe

```

To integrate it with CI, AHK needs to be installed by downloading ZIP file, uncompressing it and passing the above AHK script. Suggested `appveyor.yml` file:

```
install:
- curl -sLo https://www.metatrader5.com/en/download
- curl -sLo http://www.autohotkey.com/download/AutoHotkey104805.zip
- unzip *.zip
before_test:
- dir
test_script:
- AutoHotkey.exe install.ahk
build: off
platform: x86

```

where `install.ahk` is the file to contain above AHK script which can be dynamically generated or downloaded from some repository.
> 4 votes

---
Tags: automation, appveyor, installation
---


# Question
Title: How do you fetch a Jenkins job's artifacts from a remote system?
I'd like to download an artifact file from Jenkins CI (e.g. file from the latest successful job) from a remote system. How can I do it? Do I need a plugin to do this?

# Answer
The public URLs of artifacts can be found in Workspace of the job. It is advised to use ***view*** version of URL, so it is more consistent.

For archived files, Archived Artifact Url Viewer can provide URL to a file inside a zip or jar archive within the artifact folder of a build
> 2 votes

---
Tags: jenkins, jenkins-plugins, artifacts
---


# Question
Title: What are the different job permissions possible in Jenkins, and what are the reasons you would use each?
I'd like to understand better job permissions in Jenkins. What type of user permissions can I setup in Jenkins? And when would you use each?

# Answer
> 3 votes
There are 4 type of user permissions in Jenkins:

* Open

Jenkins is accessible for everyone with no permissions. Basically anyone can do anything. This should be used for internal use only.
* User authentication (all admins)

All users are admins. The authentication method can use Jenkins’ own user database, LDAP, Unix user/group database, servelet container (e.g. Tomcat or Glassfish user DB) or other authentication plugins. This can be used when there is no need to restrict users from doing certain actions.
* Matrix based security

This is a global permission per role/group security strategy where each user where permissions are associated with groups to which users belong to. This is the most common authorization to use for multiple users.
* Project based matrix

It is authorization strategy where user permissions can be configured per individual jobs. Once activated, permission matrix can be activated on each job page. Useful when specific jobs shouldn't be accessed by certain users.

Related page: Securing Jenkins - Access Control.

---
Tags: jenkins, permissions
---


# Question
Title: Can you use `aws-sdk` to store objects in Google Storage?
Google Storage claims they support *interoperability* in their solution with Amazon AWS S3 and Eucalyptus Systems APIs.

source: https://cloud.google.com/storage/docs/interoperability

Can the `aws-sdk` for Java, or Node.js, or Python, be used to store and retrieve objects with the Google Storage APIs?

# Answer
> 8 votes
Yes, you can use the AWS SDK to store objects within google storage.

From Simple migration:

> In a simple migration from Amazon S3 to Google Cloud Storage, you can use your existing tools and libraries for generating authenticated REST requests to Amazon S3, to also send authenticated requests to Google Cloud Storage. The changes you need to make to your existing tools and libraries are described in this section.
>
> To get set up for a simple migration do the following:
>
> * Set a default Google project.
> * Get a developer key.
> * In your existing tools or libraries, make the following changes:
>     + Change the request endpoint to use the Cloud Storage request endpoint.
>     + Replace the Amazon Web Services (AWS) access and secret key with the corresponding Cloud Storage access key and secret key (collectively called your Google developer key).
>
> That's it! At this point you can start using your existing tools and libraries to send keyed-hash message authentication code (HMAC) requests to Google Cloud Storage.
>
> When you use the Cloud Storage XML API in a simple migration scenario, specifying the `AWS` signature identifier in the `Authorization` header lets Cloud Storage know to expect `x-amz-*` headers and Amazon S3 ACL XML syntax in your request.
>
> **Note:** A simple migration from Amazon S3 is an easy and quick way to get started using Google Cloud Storage, with a minimal investment in time and changes to your existing code. After you have some experience with a simple migration, you can fully transition to Google Cloud Storage to take advantage of all its features. For more information, see Full Migration.

# Answer
> 7 votes
Most of the AWS SDKs mentioned support constructing a custom endpoint, for example as described here for Node.JS.

Setting the endpoint to `storage.googleapis.com` and using your GCP access credentials should enable you to operate the Google storage backend service "like S3".

---
Tags: amazon-s3, storage, google-cloud-storage
---


# Question
Title: How to manage Consul and its quorum in auto-scaling environments?
We have auto-scaling Docker environments in which we use Consul for service discovery. These environments can add or remove one instance every few minutes.

Our early Consul testing showed that it was very easy for Consul to loose its quorum. Perhaps naively, our very first experimentation was a setup in which we would start a Consul server on all instances and have that Consul server join the cluster. That part was working fine.

However, Consul does not reap unreachable nodes quickly (it take about 72 hours?) in a very scalable environments that means that the list of Consul servers keeps growing and over time, most of them are "unreachable" and at that point, the cluster loses its quorum.

We've seen armon's response from almost two years ago on this issue on GitHub: https://github.com/hashicorp/consul/issues/454#issuecomment-125767550

> Most of these problems are caused by our default behavior of attempting a graceful leave. Our mental model is that servers are long lived and don't shutdown for any reason other than unexpected power loss, or a graceful maintenance in which case you need to leave the cluster. In retrospect that was a bad default. Almost all of this can be avoided by just kill -9 the Consul server, in affect simulating power loss.

We were trying to avoid running dedicated, long-lived nodes. Keep in mind that at no point, we remove N/2+1 instances from an auto-scaling group. The EC2 cluster is able at any point in time to reach most of the nodes and should be able to vote whether a node should be removed from the Consul (or other tool) cluster.

# Answer
> 6 votes
I would set the `leave_on_terminate` option to true. As per the documentation

> `leave_on_terminate` If enabled, when the agent receives a TERM signal, it will send a Leave message to the rest of the cluster and gracefully leave. The default behavior for this feature varies based on whether or not the agent is running as a client or a server (prior to Consul 0.7 the default value was unconditionally set to false). On agents in client-mode, this defaults to true and for agents in server-mode, this defaults to false.

What happens when a node is shutdown gracefully is to send SIGTERM to all processes before shutdown, with this setting on the consul agent will leave the cluster so it won't be considered as a node which can restart and be back in the cluster in a few hours (which is what your quote says it does by default).

---
Tags: docker, autoscaling, consul
---


# Question
Title: What are HiveQL limitations compared to SQL?
I'm aware that Apache Hive provides SQL-like interface to query stored data. So I would like to ask, what are the main limitation of Hive-based SQL-like compared to other relational SQL query languages (such as MySQL)?

# Answer
The Wikipedia page for HiveQL provides a brief summary of the differences:

> While based on SQL, HiveQL does not strictly follow the full SQL-92 standard. HiveQL offers extensions not in SQL, including multitable inserts and create table as select, but only offers basic support for indexes. Also, HiveQL lacks support for transactions and materialized views, and only limited subquery support. Support for insert, update, and delete with full ACID functionality was made available with release 0.14.
>
> Internally, a compiler translates HiveQL statements into a directed acyclic graph of MapReduce, Tez, or Spark jobs, which are submitted to Hadoop for execution.

One of the biggest limitations is with transactions, as documented in their wiki \- it appears that ACID semantics have been added recently, so the support is not quite as mature as it would be in a typical RDBMS like MySQL.

Hortonworks have also published a cheat sheet for converting SQL to HiveQL \- it looks pretty useful for determining the differences between Hive and a normal RDBMS, although it does appear to be slightly outdated (it talks about v0.10 and v0.11, when the latest release as of March 2017 is 2.1.1).
> 4 votes

---
Tags: sql, hive
---


# Question
Title: How efficiently does GlusterFS deal with failures?
What happens when one of the storage bricks goes down (HDD failure), how does GlusterFS deal with data recover? Is the stored data still safe?

# Answer
> 4 votes
Gluster has built in data "translators" that automatically replicate data across all of your bricks. The particular type of translator you're interested in is called AFR for automatic file replication. The AFR translator also uses the DHT (distributed hash table) translator. It's important that you have at least two master bricks since if you only have one, you have a single point of failure. As long as you have at least two bricks, an auto-healing process is triggered by GlusterFS's daemon (it's automatically installed when you invoke `gluster` the first time) using the other master server's replicated/translated data if anything goes wrong. Actual disaster recovery that the auto healing can't fix requires going through a careful process documented here.

Other wisdom on the topic (for example, Redhat's documentation) recommends having at least six bricks in two sets, actually. This way, "even if we lose two bricks from each set, there is no data loss". Basically, the more bricks you have, the more redundancy you have, and the less chance of data loss you have (however, it'll become cost-prohibitive and time-prohibitive after a while).

# Answer
> 2 votes
Usually you have at least two replicas of data between the nodes of GlusteFS cluster.

If you need details about handover between the nodes (in case of one node fail) you can check this answer in Unix and Linux SE site

---
Tags: storage, glusterfs
---


# Question
Title: How to deploy a custom build of a Jenkins core plugin?
How to deploy a custom build of a core plugin bundled in Jenkins?

# Answer
> 8 votes
This section of the official Jenkins Plugin Tutorial will probably be of use.

According to the writers of that documentation, you need to copy your .hpi file to `$JENKINS_HOME/plugins`, remove the plugin's development directory, and finally create an empty file called .hpi.pinned for maven to use. Then you should be ready to restart Jenkins and use your custom build of a core plugin.

They also suggest running `mvn clean` when you're ready to update Jenkins to remove old files (after changing core plugins, old .jar files might still be around after updating).

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: What steps could be taken to inherit and override some parts of a Chef cookbook?
With 3200+ cookbooks available in Chef supermarket, often there is already a cookbook that covers the need I need to solve.

Almost every time there is some change required to a template, or a variable, or a resource in an existing cookbook.

What are the steps to override and/or improve existing cookbooks? Without forking their code, and retaining the ability to get version updates in the future.

# Answer
There are a lot of specifics but the overall pattern we use is "wrap and extend". The general idea is to make a cookbook that depends on the community cookbook, usually named `mycompany_originalthing`, and then make recipes in that which call `include_recipe 'originalthing::whatever'` but with either more stuff added before/after or with calls to things like `edit_resource` to change resources. Avoid `edit_resource` when possible since it leads to brittle code, but it is there if you need it. You can also use wrapper cookbooks to set attributes, subclass or wrap custom resources, and so on.

For the specific case of "I need to tweak a template in a community recipe" it would look like this:

```
include_recipe 'original::whatever'

edit_resource!(:template, '/path/to/something') do
  source 'mytemplate.erb'
  cookbook 'mycompany_original'
end

```

You can find more details about `edit_resource` and friends at https://coderanger.net/rewind/

With cookbooks based around custom resources instead of recipes, things can get more complex but the specifics depend a lot on the how exactly the cookbook you are extending is written.
> 13 votes

---
Tags: chef
---


# Question
Title: How to add a new column with custom value extracted from the job?
The current list of Jenkins jobs has 4 columns:

* Name
* Last Success
* Last Failure
* Last Duration

Is there any way to add a new column which would have some custom value extracted from the job?

# Answer
> 4 votes
From what I can tell you can definitely customize the columns for a custom view (I can't find where to change it for the "All" view, but it's probably possible). Unfortunately it doesn't look trivial to add a custom column without writing a plugin, for example the jacoco plugin:

https://github.com/jenkinsci/jacoco-plugin/blob/master/src/main/java/hudson/plugins/jacococoveragecolumn/JaCoCoColumn.java

Jenkins Java Doc:

http://javadoc.jenkins.io/hudson/views/ListViewColumn.html

Edit: This plugin might get you closer to what you're looking for:

https://wiki.jenkins-ci.org/display/JENKINS/Extra+Columns+Plugin

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: How to find the constraint in a software development shop?
Let us define a constraint as anything that prevents the system from achieving its goal and then look at the goal of DevOps in an organization.

A good definition is "DevOps enables a fast flow of features from development to IT operations to the customers." Improving DevOps can be achieved by removing impediments to flow.

First, we must find these constraints in the flow. And maybe there are multiple obstacles, how can we decide on which one to remove first?

A famous quote from Dr. Eliyahu M. Goldratt is

> Any improvement not made at the constraint is an illusion

This constraint, once found, is an important focusing mechanism for management of the system. So, how can you identify the constraint in an IT organization? And what to change about it to get the benefit of better flow?

# Answer
To identify a constraint in any process is a relatively simple task. Work moves from a person to person throughout the organization and it will simply pile up in front of a constraint. You can look for people or teams with the highest number of blocking unresolved issues for example. It depends on how you track the flow of work, but if you do track it, then identifying constraint is usually not too difficult. It helps to know the value chains for your organization and simply follow along them.

Constraints can be of three types:

* People: A particular skill or knowledge unique to an individual.
* Policy: A written or unwritten policy or process, which limits the work either by increasing latency or reducing throughput.
* Equipment: Particular computer system can limit the speed at which you can pass work through it or the number of people that can use it at the same time.

In regards to software, the Software Delivery Process is often a constraint and thus Continuous Integration / Continuous Delivery is such a powerful way to elevate this constraint and subordinating the organization to this constraint is often needed to increase the throughput of work. By automation in Software Delivery you remove slow people and often costly policy obstacles and approvals. Equipment in form of Computer System upgrades is a relatively small cost comparatively to the other two.

QA is another example of possible constraint in the process. Manual quality controls are work intensive and even automated testing can be constraint by equipment, in case the software is to run on expensive hardware. But the most common constraint in QA is policy, which tends to snowball with ever problem. The latency of the QA process and the small throughput can lead to larger units of work being put through, which can push back the cost into code integration.
> 6 votes

# Answer
Unfortunately there is no one answer here. If you ask each member of your team, they will probably all have different answers. Instead of trying to a root cause analysis on something as generalized as "culture", try looking at return-on-investment instead. A fix to something near a bottleneck will usually have a better RoI than one further from the bottleneck, but if the further one will take 5 minutes and get you 90% of the benefit then who cares, do the needful.
> -1 votes

---
Tags: terminology, culture, workflow
---


# Question
Title: How to deterministic and reproducibly assign increasing port numbers in Ansible?
I am new to Ansible, but I have to maintain a set of playbooks, which correspond to services to be setup in a given environment. They need to be assigned a port, certificates, etc. This results in many files with lists of essentially always the same names and an assignment to them.

In many cases I think I can easily reuse the service\_name as variable, but when mapping to IPs, ports or other numeric identifiers I have not yet figured out a way to deterministically assign them different numbers in a way that is reproducible, and preferably remains the same even when new services are added. I have considered using a SQLite database to store the services from and to generate the values from their ids, but I have no idea how to integrate that with Ansible.

I assume that assigning increasing port numbers is not something entirely new; it is something a lot of sysadmins to on a day-to-day basis so there has to be some way to do that.

**Edit**: We directly add the port numbers etc. in `group_vars/all.yml` like this:

```
ports:
    service1:1024
    service2:1025
    service3:1026

```

The inventory is generated automatically, since we create additional jails (BSD) and depends on the roles that will be executed.

# Answer
Disclaimer: I'm not using Ansible.

What I would do is use a random "predictable" number. According to Ansible doc you can seed the random number generator:

> As of Ansible version 2.3, it’s also possible to initialize the random number generator from a seed. This way, you can create random-but-idempotent numbers:
>
> "{{ 59 |random(seed=inventory\_hostname) }} * * * * root /script/from/cron"

So in your case for a port number (I assume unpriviledged) I would go for a variable with something like:

```
port="{{ 32767 |random(start=1024,seed=service_name) }}"

```

Max at 32767 to avoid clashing with any client initiated port (See Ephemeral port for the reason).
> 5 votes

---
Tags: ansible, ports
---


# Question
Title: How do you manage your inventory of spot instances on AWS with Ansible
I am new to Ansible, with some experience in Chef. I am considering learn and use Ansible to manage environments.

I was wondering, what would be the best practice to manage inventory of AWS spot instances?

For example, when a spot instance is shut down the old IP is not relevant anymore in the inventory of hosts.

Are there any other alternative approaches for the use case of elastic environments?

# Answer
> 15 votes
You are looking for *Dynamic Inventory*

Please look into `ec2.py` and `ec2.ini` of Ansible, and also the `Dynamic Inventory` docs of Ansible are very good for understanding how to manage your AWS inventory.

---
Tags: ansible, amazon-web-services, amazon-ec2
---


# Question
Title: What is the difference between SRE and DevOps?
What is the difference between SRE and DevOps?

Site Reliability Engineering and Development Operations seem to overlap a lot in detail. How do I know which group is responsible for what, and how do I know what jobs would be appropriate for my skillset?

It seems like SRE is about maintaining servers and network, and DevOps is about maintaining code, is that correct? Isn't there still a fair amount of overlap between those two?

# Answer
> 56 votes
> DevOps is about maintaining code, is that correct?

**DevOps** is not "just" about code, or systems, or any one thing. DevOps is a very general term that covers *all* things related to *software delivery.*

**Site Reliability Engineering** is a term popularized by Google. From this article https://landing.google.com/sre/interview/ben-treynor.html we can distill their TL;DR:

> Fundamentally, it’s what happens when you ask a software engineer to design an operations function.

Operations, Engineering, and Software Development are blurring together. The degree of automation required to create and maintain a mature infrastructure requires skills from all three. SREs are admins, *and* engineers, *and* developers.

See also: http://shop.oreilly.com/product/0636920041528.do

# Answer
> 24 votes
Dave Swersky already posted an excellent response above with Ben Treynor's definition of SRE, which is as poignant today as it was in 2003.

> Fundamentally, it’s what happens when you ask a software engineer to design an operations function.

So, in an attempt to further define "DevOps", here's an excerpt from the book *Effective DevOps* by Jennifer Davis & Katherine Daniels:

> Devops is a way of thinking and a way of working. It is a framework for sharing stories and developing empathy... \[it's\] is not just another software development methodology.
>
> \[P\]ractices can include software development methods, or features like infrastructure automation and continuous delivery, \[although\] it is much more than just the sum of these parts.
>
> While these concepts are related and may be frequently seen in devops environments, focusing solely on them misses the bigger picture--the cultural and interpersonal aspects that give devops its power.

In summary: an effective SRE will leverage DevOps practices.

--

Also:

> How do I know which group is responsible for what?

Ownership should not be implicit; communicate!

# Answer
> 18 votes
**Site Reliability Engineering** falls under more or less traditional operations, but heavily automated and version controlled, what is also called Infrastructure as Code. It is a well defined **vertical role**. In modern DevOps this is the vertical slice that relates to Operations. You can have a team of SRE.

**DevOps** as such is a cultural change for an organization. In addition to the vertical, top down, management structure, it creates a horizontal connection between the teams along the delivery paths of work throughout the value chains. For an engineer it is a loosely defined **horizontal role** binding several teams together, ensuring that work passes smoothly and quickly throughout the organization. You cannot have a team of DevOps engineers, that is an *oxymoron*, as spanning the team boundaries is a crucial part of the role.

# Answer
> 2 votes
Another way to describe differences between *Site Reliability Engineering* and *DevOps* is to look at Wikipedia's explanation of a **`Site Reliability Engineer`**, which starts like so:

> Site reliability engineer (SRE) is a job description given to software engineers focused on reliability, scalability, and the development of cloud computing infrastructure, known as Site Reliability Engineering (SRE).

So you may consider SREs as people involved in ***`Building walls`*** ...

However, Wikipedia's explanation of **`DevOps`** starts like so:

> DevOps ... is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes. It aims at establishing a culture and environment where building, testing, and releasing software can happen rapidly, frequently, and more reliably.

What it doesn't say is that all these DevOps practices are actually triggered by a **Demand** from the business side. So combined with building and testing (the **Develop** part of it) and the **Release** of software, DevOps is about a **DDR** (= Demand-Develop-Release) culture and environment, which some people may remember from these 9 seconds of a historical speach to **`Tear down this wall`**.

Refer to the question about Brief outage planned for Wed, May 3, 2017 at 8pm US/Eastern (like a fire drill for computers) for an example about all SE sites ... posted (signed) by a user with job title ***SRE Manager, Stack Overflow, Inc.***

---
Tags: terminology, sre, job-role, team-role
---


# Question
Title: How to implement the immutable server pattern without loosing the ability to do post-mortems?
The immutable server pattern is a deployment discipline favouring the reproducibility of deployments. It is characterised by the fact that “a server that once deployed, is never modified, merely replaced with a new updated instance” and implementing this discipline requires an automation of server deployment. This automation has numerous operational advantages, one of the most important is allowing the quick and reliable replacement of failing instances in an infrastructure. This automation also implies that server deployment is described by versioned software artefacts and is subject to iterative improvements.

A popular aspect of implementations of this discipline is the removal of remote access methods to the server once it has been launched (esp. removing SSH access). Removing remote access is an easy way to ensure that the configuration of the server matches the configuration prepared by the deployment automation.

However, when investigating the causes of a software failure in a , relying on structured monitoring is not always enough and remote access to the machine could be necessary. It is a common practical situation that server monitoring does not cover all of the failure sources, or that monitoring can be impaired by the server failure itself, which would likely be the case if the server runs out of memory or reaches its process limit.

How to implement the immutable server pattern without loosing the ability to do post-mortems?

# Answer
First of all, removing ssh on an immutable server doesn't guarantee there'll be no change, it's more that as there should be no need to change something you reduce the attack surface by removing a remote access channel.

One way to keep a sort of post-mortem is log centralisation. There's a myriad of methods to achieve it, ELK stack, Splunk, syslog...

Another more crude way to keep a post mortem for an immutable server is to have a script on the shutdown process (an immutable server failing would be shutdown and a new one spin up to replace it) to gather a core dump of the program, a memory dump and send them to a remote system for analysis along with most of the logs.

Main advantage of this solution is that you get back only failing system information at time of problem, allowing to gather larger informations than getting them periodically.

It's hard to be more specific on how to achieve this, each distribution has some way to get things and I've no generic example.
> 9 votes

# Answer
The fact that you don't have SSH access doesn't mean there is no way to access the machine. Most likely you'll be running it on some cloud operator, where you can also do the following:

* take a snapshot of the machine. You could simply take a snapshot of the box before destroying it, for later analysis.
* access the machine through the console. You'll probably need to have the root password for this, but some cloud providers can inject a random root password for console access at any time.

These are essentially "physical" access to your machine, and will be available even if you remove other types of access. You could limit these interfaces as well though.

Apart from this as @Tensibai said the better thing to do is have proper logging and monitoring set up, so any time you'll have to do a post mortem, there are enough data available to do it.
> 7 votes

---
Tags: immutable-servers, post-mortem
---


# Question
Title: Vagrantfile change with Ansible playbook to control file permission on Vagrant synced folder for Trellis
I use Trellis for my WordPress development. It works great, except I'm finding it hard to change file (or directory) permission within the synced folder.

Generally I'm OK with the default permission setup. However, sometimes I need to give write permission to some of the sub-directories of the main (NFS) synced directory.

This is the Vagrantfile that uses Ansible playbook for the main configuration of the VM (I use VirtualBox). I'm not a Ruby programmer nor have I worked with Ansible, however, by the looks of it, the following is the part of the `Vagrantfile` where file permissions are being set:

```
if Vagrant::Util::Platform.windows? and !Vagrant.has_plugin? 'vagrant-winnfsd'
    wordpress_sites.each_pair do |name, site|
        config.vm.synced_folder local_site_path(site), remote_site_path(name, site), owner: 'vagrant', group: 'www-data', mount_options: ['dmode=776', 'fmode=775']
    end
    config.vm.synced_folder ANSIBLE_PATH, ANSIBLE_PATH_ON_VM, mount_options: ['dmode=755', 'fmode=644']
    config.vm.synced_folder File.join(ANSIBLE_PATH, 'bin'), bin_path, mount_options: ['dmode=755', 'fmode=755']
else
    if !Vagrant.has_plugin? 'vagrant-bindfs'
        fail_with_message "vagrant-bindfs missing, please install the plugin with this command:\nvagrant plugin install vagrant-bindfs"
    else
        wordpress_sites.each_pair do |name, site|
            config.vm.synced_folder local_site_path(site), nfs_path(name), type: 'nfs'
            config.bindfs.bind_folder nfs_path(name), remote_site_path(name, site), u: 'vagrant', g: 'www-data', o: 'nonempty'
        end
        config.vm.synced_folder ANSIBLE_PATH, '/ansible-nfs', type: 'nfs'
        config.bindfs.bind_folder '/ansible-nfs', ANSIBLE_PATH_ON_VM, o: 'nonempty', p: '0644,a+D'
        config.bindfs.bind_folder bin_path, bin_path, perms: '0755'
    end
end

```

How can I make changes to this file in a way so that I can control permission to specific files and folders when I need. Since changing the permission within the vagrant synced folder on the host machine doesn't reflect on the Virtual Machine, I must at least be able to do it by changing the Vagrantfile and then re-provisioning.

How can I achieve this easily that will persist upon future re-provisioning?

# Answer
There are two parts of this Vagrantfile, one does the mounting for Windows hosts (the first half), the other one for UNIX hosts (the second half), but essentially the do the same - just with different plugins.

You can see from the `config.vm.synced_folder` and `config.bindfs.bind_folder` commands where they set up the appropriate directories with the proper permissions.

To add a new directory with different permissions simply add them to the list. Note that `File.join('a','b')` simply converts the path to `a/b`, but it's a nicer way to do it, as it's platform independent (on Windows it can also handle `\` style paths).

So for example if you want to give the `/tmp/needswrite` folder more permission you can simply add the following two lines to their appropriate places:

Windows config:

```
config.vm.synced_folder File.join(ANSIBLE_PATH, 'tmp', 'needswrite'), File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), mount_options: ['dmode=777', 'fmode=777']

```

Unix config:

```
config.bindfs.bind_folder File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), perms: '0777'

```

This will for example set their permission to `777` instead of the default `755`

Note that in the Windows config you have to map from `ANSIBLE_PATH` to `ANSIBLE_PATH_ON_VM`, while on the Unix config bindfs will do a remount, so you need to match them with the same directory name.

You can also play with users and groups:

Windows config:

```
config.vm.synced_folder File.join(ANSIBLE_PATH, 'tmp', 'needswrite'), File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), owner: 'new-owner', group: 'new-group', mount_options: ['dmode=755', 'fmode=755']

```

Unix config:

```
config.bindfs.bind_folder File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), File.join(ANSIBLE_PATH_ON_VM, 'tmp', 'needswrite'), u: 'new-owner', g: 'new-group', perms: '0777'

```

Here we set them to use user `new-owner` and group `new-group`
> 4 votes

---
Tags: ansible, vagrant
---


# Question
Title: Is there any way I can rotate AWS KMS keys every time interval < 1 year?
According AWS KMS, keys can be rotated once every year. However, there is no way I can define a custom time period for that.

However, I can keep changing the same in the bucket's policy **manually**.

**So, is there anyway I can automate that?**

# Answer
> 8 votes
The AWS documentation is clear that automated key rotation is only supported by Amazon on a yearly basis. However, you can use the CLI to manually create new keys, and then use an automation technique such as cron to automate the manual process.

In this case you can likely have cron run a bash script to read from a list of keys and credentials and then use the AWS kms cli to "manually" change your keys.

---
Tags: automation, amazon-s3, amazon-kms
---


title: Is DevOps compatible with ITIL?

text:
# Question
Title: Is DevOps compatible with ITIL?
In my career, I have been both a software developer and ITIL practitioner in an operations role. Thus DevOps was a natural progression for me.
However, I have always struggled with the highly specialised language that ITIL introduces and making that "Developer Friendly" enough to not be a complete turn off to developers.

ITIL is an internationally recognised IT Service Management framework that has been developed over 30 years as a set of practices that have a proven benefit to the operational stability and maturity of an organization.

Is DevOps truly compatible with ITIL, or in essence do we need to take the spirit of ITIL and "translate" it to language that is better understood by development teams:

* Incident & Problem Management → Production Defects, Bugs or Issues
* Change & Release Management → Continuous Delivery
* Event Management → Logging, Telemetry, Instrumentation and Alerting

# Answer
In my opinion, the DevOps culture come along with a methodology change toward Agile process management.
ITIL is heavily aimed at a clear formalism of the process and the results and thus more adapted to a Waterfall model.

This doesn't mean ITIL is incompatible with Devops, but usually this will be two separate process with different timelines. I mean that the inclusion of a new product within ITIL referential will usually be delayed until the product/application has been released in production for a while, where early pitfalls and some documentation needed to integrate ITIL has been done and adapted after the product is "live".

One of the thing in ITIL is the Service Design, which is assumed to be defined before any development task, an agile process will/may review the design in each iteration, breaking the formalism needed in an ITIL process.

The main goal of ITIL is, as you said, to provide a framework to ensure nothing is omitted between the design/conception and maintenance phase (Build/Run). In a devops culture, the whole team is responsible of all phases on long term, hence why the formalism is reducted.

That doesn't mean we have to forget ITIL, the core principles are absolutely good and, in my opinion, should be used as a checklist to build the initial backlog of a product. It's just that following the ITIL principle with all its formalism goes against the time to market reduction goal of a quick iterative software development and sometimes is not even applicable as there's less transmission of information needed between teams, as the tasks are done by the same team.
> 26 votes

# Answer
I'm ITIL certified (though it has been a while.) I agree with Tensibai: ITIL and DevOps aren't *incompatbile*, but that doesn't necessarily make them great friends.

The argument can be made that the processes in ITIL must happen in some way, especially for larger organizations. Successful integration of DevOps practices, where ITIL is already practiced, requires careful planning, communication, and execution. Then again, that's true of any DevOps Transformation.

For a "greenfield" transformation where neither ITIL or DevOps are in place, I'd craft a combination of both using "mapped" terminology as you have described. As long as everyone in the organization is on the same page, using the same language, ITIL and DevOps can add value when combined.
> 9 votes

# Answer
I liked the answers provided by the IT Skeptic on an episode of DevOpsCafe.org If I remember it correctly, his line of thinking is that if you actually really understand ITIL, there is very little conflict. That most of ITIL guidelines are very general and that the conflicts are largely between some implementations of ITIL, not behind the actual specification.
> 0 votes

---
Tags: terminology, itil, operations
---


title: What is Value Stream Mapping?

text:
# Question
Title: What is Value Stream Mapping?
I have heard a lot about Value Stream Mapping and how it can be used to analyse the value stream of manufacturing processes including the process of delivering software. I have never seen it successfully delivered in the context of software development and operations teams.

What is Value Stream Mapping and how can it be applied to software delivery?

# Answer
> 9 votes
A *value stream* is a process that adds value to a collection of things that have less inherent value. The canonical example there is an assembly line. I have little interest in a disconnected collection of parts (steering wheel, shifter, gas pedal.) On the other hand, I'll pay good money for a car. Car manufacturers *add value* to the parts in the form of facilities, skilled work, supply chains, etc. Parts go in one end of the stream, cars come out the other.

Enterprises use complex processes to deliver their products and/or services. *Value stream mapping* is a process used by companies to figure out *how and where they deliver value.* The purpose of value stream mapping is to develop a clear understanding of the steps, time, and resources required to support a business capability that delivers value.

Value stream mapping is useful in *kaizen*, or Continuous Improvement. The "systems thinking" employed in DevOps and Lean encourages small, incremental improvements made throughout a complex process. A value stream map can expose areas of waste, such as unnecessary or inefficient steps in a value stream. A complete map tends to expose those areas of waste. This helps prioritize improvements.

Value Stream Mapping can be applied at large scope to a business process, or to a smaller scope within software delivery. The software delivery process itself mirrors an assembly line. Code is written, reviewed, tested, integrated, and tested again. It is delivered from one environment to another (in manufacturing this is called a *work center*) until it is finally delivered to Production.

# Answer
> 7 votes
**What is Value Stream Mapping**

A Value Stream Map is a representation of the flow of work, inventory and information from supplier to the customer through your organization. The VSM enables you to see at a glance where the delays are in your process, any constraints and excessive work or inventory.

For IT organizations, one common notation is that inventory is a number of tasks in the backlog. Joel Spolsky wrote an amazing article that explains and expands on this this concept.

Usually a VSM is a graphical chart that includes:

* your customer
* your supplier (if and when you use external contractors to do work)
* the process inside of your organisation (what happens to items of work in your org.)
* the feedback loops (customer asks for features, you communicate requirements to the supplier, etc...)

**Why do Value Stream Mapping**

In bigger organizations, when work is split into specialized cells - often the concept of the whole system is not in focus. This leads to problems like optimizing to local optima. For example, if you have a team that manages a Jenkins server and another team of developers who need to use it - the Jenkins admins can do improvements and work that looks good in their JIRA. But actually, hurts the developers using the system. In a Systems Thinking organization, people would work towards improving the system as a whole, not adding improvements that are local and either have no effect on the whole or hurt the process.

The purpose of a VSM is to provide optimal value to customers of the organization, with minimum waste in the value creation process (Lean). "Value" is what the customer of the organization is buying.

**How to create a Value Stream Map**

Creating a VSM has a goal of its own, improving the organization. Thus the first step is mapping the current flow of work and information. Next step is designing and implementing some improvements, and in the future continue iterating and improving as well. The VSM is a visual dashboard showing the work and measurements around it, so the bottleneck is clearly visible and can be controlled (using Theory of Constraints).

Steps to create a VSM based on the description in most Lean books:

1. Specify the value from the standpoint of the customer.
2. Identify the VSM for each product family (there can be more than one)
3. Improve the flow of work.
4. Establish pull. The customer can now pull work out of the VSM, not get pushed work from it.
5. Iterate until perfection.

In an organization that is using Kanban to manage work, it is used to manage the pull of value from the customer standpoint. Unfortunately, often people don't really know how Kanban works and why it is defined as it is, so they use Kanban to push work onto customers (or the market) - this often leads to a lot of wasted work and frustrated people who don't see their work having an impact.

Great reference for further reading about the subject:

---
Tags: terminology, process, project-management, analysis
---


# Question
Title: How to store credentials that are required by an application?
Everyone is saying that storing credentials in version control (git) is a bad thing. So there must be other ways to store credentials which are much better.

An application must receive credentials from somewhere to use services it depends on. These credentials are usually stored in configuration files. Manually entering each server to create that file is out of the question, since servers come and go without human intervention.

How to manage an application's credentials?

# Answer
> 8 votes
Proper management of an application's secrets has always been a challenge. New challenges came with the adoption of the cloud. There's a great OWASP presentation about the reality and challenges of storing secrets in the cloud.

You might be surprised to hear that storing secrets into the source code is one of the solution (or "architecture") presented. That's because, right now, there is no perfect architecture or way of doing this. In the end, your secrets might be encrypted... but what is guarding the encryption key? "Turtles all the way down", they said.

Every type of secret management has its strengths and weaknesses and the presentation already covers that. Instead, I'll try to go over some of the features you might be looking for in an secret (credential) management solution:

* Access control: can you give write access to admins and read access to applications? Can you limit what applications are able to read (application A has access only to those secrets)?
* Audit logs: required for many compliancy reports and a good way to figure if something is awry
* Secure storage of the secrets: how is the solution storing the secrets? Encrypted DB? Encrypted FS? Who/what holds the encryption key, if any? How is this key used - once at startup and then securely discarded?
* Keys/passwords rotation or renewal: if a secret is compromised, can you revoke it and send an updated secret to the applications? Can/should the applications pool the secret management service?
* Compatibility: Some of these solutions offer tight integration with certain languages or framework. Some offer REST API. Are you interested in this?

By looking at these items, how they are important to you and how they are implemented by the solution, you will be able to choose one of the secret management service out there.

# Answer
> 4 votes
For a purely EC2-based environment, the easiest solution is to use AWS IAM roles and S3 bucket policies. Some formulations of this pattern also include KMS for encryption and DynamoDB instead of S3 for storage but I've not found a hugely compelling reason to use either. Check out https://github.com/poise/citadel, it is specifically aimed at Chef users but the underlying pattern works for anything, you just might have to make your own helper for doing the actual download from S3.

---
Tags: deployment, security, credentials
---


# Question
Title: How do I automate deployment to Docker with CircleCI?
My circle.yml file and my CircleCI account are set up to use Docker for testing already, but I'd like to move to continuous deployment upon a successful build.

Here's what I have so far in circle.yml:

```
machine:
  services:
    - docker

dependencies:
  override:
    - docker info
    - docker build -t pgeiss/appname .

test:
  override:
    - docker run -d -p 8080:8080 pgeiss/appname; sleep 10
    - curl --retry 10 --retry-delay 5 -v http://localhost:8080/res.json

```

# Answer
A few things need to happen for this to work properly. First, add a `deployment` section to circle.yml:

```
deployment:
  main: # or whatever your deployment is called
    branch: master # or whatever branch you want to deploy
    commands:
      - docker login -e (your email here) -u (your username here) -p (your password here)
      - docker push pgeiss/appname
      - ./start.sh

```

*Thanks to this blog post for the following script.* Then, make a file called start.sh in the top level of your repository (if you use a different name, change the circle.yml's last line) that contains the following:

```
#!/usr/bin/env bash

echo "stopping running application"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker stop dodsv'
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker rm dodsv'

echo "pulling latest version of the code"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker pull pgeiss/appname-webapp:latest'

echo "starting the new version"
ssh $DEPLOY_USER@$DEPLOY_HOST 'docker run -d --restart=always --name dodsv -p 80:5432 pgeiss/appname:latest'

echo "success!"

exit 0

```

Finally, to allow the script to work you'll need to set the script to be executable with `chmod` and add the environment variables and your `DEPLOY_USER`'s ssh key (preferably) or credentials (if no ssh key) to CircleCI. After doing that CircleCI should deploy your app after a successful build.
> 7 votes

---
Tags: docker, automation, circleci
---