# Question
Title: Problem in integrating Travis-CI with core PHP projects
I am trying to integrate my project coded in core **PHP**, with **Travis-CI** but it always fails.

Travis reports a failure even when I have only one file in my project.

## PHP file Code :

```
<?php

phpinfo();

?>

```

---

## `.travis.yml` File Code

```
language: php
php:
  - '5.4'
  - '5.5'
  - '5.6'
  - '7.0'
  - '7.1'
  - hhvm
  - nightly

```

# Answer
You need to write some unit tests and set the `script` variable in your `.travis.yml` to actually run a file.

By default, Travis CI runs the command `phpunit` without any arguments. When this happens, `phpunit` doesn't know what you're asking it to do, and shows a help message, then exits with error code 2 (i.e. **non-zero**, which indicates an error occurred).

The Beginner's Guide shows you a possible solution of setting the `script` variable to run a `Test.php` file with your tests in (insert this into your `.travis.yml`):

```
script: phpunit Test.php

```

There is an example of a valid `Test.php` in the example repo that Travis provides:

```
<?php
    class Test extends PHPUnit_Framework_TestCase
    {
        public function testOnePlusOne() {
            $this->assertEquals(1+1,1);
        }
    }
?>

```
> 10 votes

---
Tags: continuous-integration, travis-ci, php
---


# Question
Title: Are there any drawbacks of using a deb package as if it was a container to deploy an application?
My team is currently trying to decide if we should deploy our Nodejs app as a deb package instead of trying to run it in a container such as Docker.

I got this idea from reading this blog here which makes some good arguments for using a deb package for a pre-existing python application. The main point from this blog which is appealing to us is the issue of maintaining the Docker ecosystem (port sharing, permissions, hosting of Docker Images, etc.)

It seems like "dep-packages as the original containers" makes a lot of sense for small services where there is no concern of port conflicts and where all the dependencies are maintained within a virtual environment.

My gut, however, is telling me that if deb packages were a good fit, it would be more common and docker would be advertised as a more language specific solution. Are there any drawbacks of using something like deb packages to deploy our services, instead of using a full system such as docker?

# Answer
> 17 votes
First, while **Docker** is sometimes seen and used as a *ad hoc* packaging system, it actually solves a totally different problem: **Docker** is about *running* programs. The **Docker** system allows to describe *services,* that can be *scaled* at will and to control *swarms* of containers. **Debian packages** are for installing programs and they are able to handle *dependencies* between software versions. **Docker** certainly don't qualify as a descent packaging system: each “package” can only have one dependency, the system has no “recursive build” option and does not support complex version constraints!

A possible answer would be that, if you are willing to write a **Debian package** for your application, you can also use **Docker** to deploy your application. This can be achieved with a configuration script `apt_setup.sh` which would look like

```
apt-key add - <<EOF
-----BEGIN PGP PUBLIC KEY BLOCK-----
<YOUR RELEASE OFFICER PGP KEY GOES HERE>
EOF

cat >> /etc/apt/sources.list <<EOF
deb https://my.organisation.org/repo debian-jessie main
apt-get update -y
apt-get upgrade -y
EOF

```

and a `Dockerfile` along the lines of

```
ADD apt_setup.sh /root
RUN sh -ex /root/apt_setup.sh && rm /root/apt_setup.sh
RUN apt-get install -y my-node-js-package

```

(In your specific situation, the `apt_setup.sh` would be more complicated, adding the *nodesource* repositories and some helper packages such as *apt-transport-https*.)

It is therefore really possible to use **Debian packages** and **Docker** simultaneously, however …

> My gut \[…\] is telling me that if deb packages were a good fit, it would be more common

This is a correct hitch which leads us to ask ourselves why **Docker** proves to be popular as a *ad hoc* packaging system, while it is not intended to be one. (See above.)

The “official” packaging system from a given distribution is just a possibility among many others to install software in some computing environment. There are many other sources available, like community-specific package managers such as **npm** or **opam,** port trees like **pkgsrc** and plain source code distribution. From this perspective, it is easy to understand the success of **Docker** as an *ad hoc* packaging system:

* **Docker** specifications are very close from a shell script and whatever source it comes from, we install software using the shell.
* **Docker** has a “built-in” (paying) service for hosting artefacts it produces, the **Docker Hub**.

Now what are the strength of **Debian packages** over **Docker images** as a package system? The tight control over dependencies at installation. (The possibility to upgrade and downgrade also exists but has no practical importance if we are implementing the pattern.) This leads to the

### Conclusion

If you only have a single product deployed in a single version (which is typical for SaaS), your version management needs are very simple and using **Docker** as a *ad hoc* package manager should not have any hard drawbacks. As soon as you work with several versions of a single product or several products, the complexity of the version constraints problem you need to solve increases and you need an appropriate tool for this, which might be **Debian packages** or some configuration management system if you are mixing software from different origins.

# Answer
> 7 votes
A Debian (or RedHat) package to install applications has been a good practice when done correctly. Packages are used for the purpose of deploying applications which are infrequently changed. Debian packages involve some overhead, like version management, dependency management, pre&post-install scripts, etc...

In many cases upgrading from some older version to a new version requires the careful writing of scripts, attention to details in version, etc. Because mutating existing state is difficult. It would be much easier to replace the current state altogether with a new state, without mutating anything.

Once you decide to completely replace your configuration or dependencies or application on each deployment because it is easier and less error prone. Most organizations (used to) switch to a whole new VM or cloud instance. Which means that installing the package would be done on a "clean" server, and mutating the files and configuration on the server is not a problem anymore.

Those developers who created packages and did not understand the fallacy and complexity in mutations suffered quite a lot of difficulties as a result.

Replacing VMs is sub-optimal when all you need is to replace an application, which is why lightweight containers were introduced as an answer. Using Docker (or other LWC) you can replace the userbase, including all the dependencies, without replacing the server itself. You can also host multiple versions of the same application, with different dependencies, on the same server and only switch the incoming network traffic on upgrade. As well as switch network traffic back on rollback (blue-green), something that was remarkably hard in the case of managing deployments via packages.

Containers introduce a way to bundle all the application code, and dependencies, and configuration, into an image. This image has multiple properties that make it much better than traditional operating system packages. For example, it has tags that enable versioning, but it also has layers, which enable to save on space. It allows an easy way to ship these images to servers, and development environments, by using a registry. And these images can be executed as containers in any environment and any server, almost identically. This includes the developer's laptop as well as the production environment. Again, something that was much more difficult to do with VMs and/or with package-based versions of the software. Having the same image tested on the developer's laptop, and staying the same bits and bytes in production removes a lot of "works on my machine" problems.

# Answer
> 7 votes
Yes, there's drawbacks.

With a .deb package you won't be able to have two version of the same application on the same host. You will have to rely on the distribution available packages, if your app rely on nodejs for example, either you'll be stuck with the distribution version or you'll have to install your own.

Now when you want to host multiples applications on the same host, you'll be hitting a wall very quickly when they depend on the same thing (let's keep nodejs here) in two different versions.

The main goal of docker is to isolate each application from the hosting system and others applications on the same host. there's two reason to do this isolation: 1. to avoid compromission of the app to be able to take over the host or impact another application 2. to give the application its exact dependencies and prevent it to be impacted by a system update or another application dependency.

# Answer
> 1 votes
Talking specifically about the image packaging piece of Docker, not the container runtime there are a few minor bits. The biggest is that a Docker image is more like a chroot, which means you are prevented from accidentally depending on shared system state since every file in use must be explicitly included in the image while a system package might pick up dynamic links you didn't expect or otherwise get more intertwined with other packages. This can come up with complex C dependencies getting loaded without your knowledge, for example OpenSSL. Additionally using deb packages don't de-duplicate shared bits in the say Docker's storage system does. For some this might be a good thing, better I/O performance and fewer moving pieces, but for others it might be a problem.

---
Tags: docker, deployment, artifacts
---


# Question
Title: How can I validate GoCD configuration before applying it
We have a GoCD installation where we like to keep the pipeline configuration XML files (`cruise-config.xml`) in a separate git repository, and install it (preferably via a pipeline and automatically) to the GoCD server whenever it is changed.

However we couldn't find an easy way to validate the XML config file before sending it off to GoCD, and obviously we do't want to install invalid configs, as that would break our pipelines. Are there any tools, or utilities that help us with validating the config file?

# Answer
The best solution we found so far, is using the `MagicalGoConfigXmlLoader` interface from the go source code, as it calls an internal `validateCruiseConfig` command, that does all validations, or die if the file is invalid:

```
MagicalGoConfigXmlLoader loader = new MagicalGoConfigXmlLoader(new ConfigCache());
loader.loadConfigHolder(configAsString);

```

The main problem with this solution that it essentially requires the whole GoCD server source code checked out, in a working state to work properly. Also as it's an internal API it is subject to changes between GoCD versions, making GoCD upgrades harder.
> 4 votes

---
Tags: continuous-integration, configuration, gocd
---


title: Which SCM tools does Jenkins support?

text:
# Question
Title: Which SCM tools does Jenkins support?
I'm aware that Jenkins supports Git and SVN. What other SCM tools does Jenkins support?

# Answer
> 18 votes
It supports **AccuRev, CVS, Subversion, Git, Mercurial, Perforce, Clearcase** and **RTC**.

For all of them exists a plugin and as you probably already know, Jenkins is not limited to only that list, anyone can create a SCM plugin for other options if they want to.

Here is the link to the plugins:

# Answer
> 9 votes
In addition to the list in the previous answer, there is also the plugin that integrates Serena Dimensions CM with Jenkins. Some more details about this plugin (from the linked page):

> The plugin allows a Jenkins job to be associated with a Serena Dimensions CM stream or project, automatically updating the Jenkins workspace with file content from the Dimensions CM repository.
>
> The plugin currently supports
>
> * Polling
> * Checkout
> * Build change log reporting
> * Tagging
> * Artifact Upload
> * Tag Deployment
> * Launching Tagged Builds via Dimensions Builder

**Video tutorial**: Dimensions CM: Jenkins Plugin Tutorial

# Answer
> 4 votes
Your question doesn't specify which platform (OS) it is about. But if it is about good old mainframes (running zOS, whereas the z stands for "zero downtime ..."), then these would be some possible options to pick from (quotes are from the linked pages):

* Compuware Source Code Download for Endevor, PDS, and ISPW Plugin.

> ... allows Jenkins users to download Endevor, PDS, or ISPW members from the mainframe to the PC. Source can then be accessed on the PC, for example, for SonarQube analysis and reporting.
* IBM zOS Connector.

> ... provides its functions via FTP connection to IBM z/OS LPAR. You can configure your SCLM project on z/OS and then check for the changes via Jenkins.
>
> Features include:
>
> + Submission of user JCL job (with optional log collected upon finish).
>     + Introduction of SCLM as SCM for your projects allowing to checkout SCLM changes.
>     + The ability to build SCLM projects currently can be performed only via 'Submit zOS Job' build action.

PS: if you're not familiar with the typical SCM solutions on this platform (such as Endevor or ISPW), then have a look at the typical SCM solutions exist for managing mainframe software.

---
Tags: jenkins, git, jenkins-plugins, scm, subversion
---


# Question
Title: How can continuous integration scale for very large projects/teams?
Traditionally CI systems only monitor the quality of the code in an integration branch, signalling when regressions occur. Human intervention is required for repairs.

As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears—progress on the branch becoming effectively negligible.

Splitting into multiple teams each working on a separate integration branch which would be merged at a later moment *might* help, but greatly extends the project duration, since it simply delays the necessary integration for a later moment while adding churn/noise/technical dept related to the partial integration from individual branch merges. Costs also increase due to CI setups for each of the branches, each with their own build/QA resources, etc. And overall quality is not necessarily better.

Scalability is, at best, dubious.

Is there a method for making such large projects actually scale?

# Answer
> 5 votes
Yes, but it's by preventing the things you mention in your question as much as possible. You are right that there are only so much developers that can work on the same code before things become unmanageable. You need someone or something that has an overview of all changes and makes sure integration works fine and regression doesn't occur. This is hard to scale if everything happens in one repository.

Before starting any work you need to separate functionality in such a way that things can be integrated later with little or no effort. This means that you want to split functionality into parts that are independent, or at least have as little dependencies as possible. How you do this depends very much on the product you are developing. In the past separation of functionality was done by creating different libraries. The downside to this is that you need a good method of managing those libraries. Old versions of Windows for example had a bad library management system which caused the infamous DLL Hell. Nowadays micro-services is (becoming) a popular way to do this, although this also has it's drawbacks (e.g. communication overhead, more services to monitor)

If separation based on functionality is difficult, another option may be to separate development in time. Instead of multiple teams working simultaneously on many different features, you have 1 team that works on only 1 or a few features at a time.

When separation of work is not possible, you'll need tools and tests that make sure things align. Automated unit and integration testing can be a big help here, but ultimately there will be limits to how big a project or team is to be workable.

# Answer
> 4 votes
> As the number of developers working on the same branch increases the risk of breakages/blockages increases. Eventually a point is reached where on average by the time a breakage is fixed a new one appears

While the first part of the sentence is probably statistically correct I disagree with the second.

CI and integration branches cannot defeat GIGO (garbage in garbage out) If the developers has no discipline then you will end up in a halted branch, but integration branch tests should only act as the lat barrier, not the first.

Developers should use proper coding standards, code reviews, local unit tests and possibly pre-integration branch tests to make collisions less likely.

I am working daily on such a system with hundreds of contributors and surprisingly it's uncommon to see it hangs.

I don't have enough data but it might be true that cost is high, on the other hand it's difficult to judge the alternative cost of late integration and bugs.

# Answer
> 0 votes
Another approach is to switch to a different, non-traditional CI system, capable of **preventing regressions** by identifying and rejecting the offending candidate changes prior to their commit rather than detecting regressions and waiting for recovery while development on the branch is blocked. I like to call such system a non-blocking CI system.

Simply bumping up the developer-executed pre-commit verification requirements in an attempt to reduce branch regressions is not sufficient for such guarantee. It may actually cause a regression rate *increase*: longer verification times =\> higher number of changes simultaneously verified in isolation (thus not accounting for each-other) =\> higher risk of functional conflicts and interference. I'm explaining this in a bit more details in the Extending the pre-commit verification coverage should increase our branch stability myth.

The CI tool also needs to be capable of operating at scale, of course. And such tools exist, see Is there a CI tool that guarantees no regressions in the branch quality level?

Disclosure: I'm affiliated to the company mentioned in the above-links.

---
Tags: continuous-integration, scalability, capacity-planning
---


# Question
Title: What are ways to mitigate the effects of Mythical Man Month?
**Brooks's law:** *Adding manpower to a late software project makes it later.*

In his book *No Silver Bullet — Essence and Accidents of Software Engineering* Frederick Brooks defines the concept of Mythical Man Month:

Brooks's assumption is that complex programming projects *cannot be perfectly partitioned into discrete tasks* that can be worked on without *communication between the workers* and without establishing a set of *complex interrelationships between tasks and the workers performing them*.

Since 1982 we have certainly moved forward and gathered some more experience in mitigating this problem. What are some of the solutions that you have successfully applied at your work to add resources to a project without creating more problems.

# Answer
**What is the MMM**

First I want to explain the context for Brook's Law. What was the assumption that made him create it back in 1975?

> A Man-month is a hypothetical unit of work representing the work done by one person in one month; Brooks' law says that it is impossible to measure useful work in man-months.

source: https://en.wikipedia.org/wiki/The\_Mythical\_Man-Month

Back in the day, complex programming projects would mean big monolith systems. And Brooks claims that these cannot be perfectly partitioned into discrete tasks that can be worked on without communication between developers and without establishing a set of complex interrelationships between tasks and the people performing them.

This is very much true in highly cohesive software monoliths. No matter how much decoupling is done, still the big monolith mandates time required for the new programmers to learn about the monolith. And increased communication overhead which will consume an ever increasing quantity of the time available.

But does it really have to be this way? Do we have to write monoliths and keep communication channels to `n(n − 1) / 2` where `n` is the number of developers?

We know there are companies where thousands of developers are working on big projects ... and it does work. So there must be something that changed since 1975.

---

**Possibility to mitigate the MMM**

In 2015, PuppetLabs and IT Revolution published the results of the 2015 State of DevOps Report. In that report, they focussed on the distinction between high performing organizations vs. non-high performing.

The high-performing organizations show some unexpected properties. For example, they have the best project due-date performance in development. Best operational stability and reliability in operations. As well as the best security and compliance track record.

One of the surprising things highlighted in the report is the deployments per day metric. But not just deployments per day, they also measured deploy/day/developer and what is the effect of adding more developers in high-performing organizations vs. non-high performing.

This is the graph from that report -

While low-performing organizations align with Mythical Man Month assumptions. The high-performing organizations can scale the number of deploy/day/dev linearly with the number of developers.

An excellent presentation at DevOpsDays London 2016 by Gene Kim talks about these findings.

---

**How to do it**

First, how to become a high-performing organization? There are a couple of books that talk about this, not enough space in this answer so I'll just link to them.

For software and IT organizations, one of the critical factors for becoming a high-performing organization is: **focus on quality and speed**.

For example Ward Cunningham explains Technical Debt as all the things we allowed to be left unfixed. This is accepted by management because it always comes with a promise that it is going to be fixed when there is time.

There is never enough time, and technical debt just becomes worse and worse.

What are these things that cause technical debt to grow?

* legacy code
* manual configuration of environments
* manual testing
* manual deployments

Legacy code As defined in Working Effectively with Legacy Code by Michael Feathers is any code that doesn't have automated testing.

Any time shortcuts are used to get the code to production; the operations is burdened with the maintenance of this debt forever. Then the process of deployment becomes longer and longer.

Gene tells a story in his presentation about a company who has six-week long deployments. Involving tens of thousands of extremely error prone tedious steps, tying up 400 people, and they would do this four times every year.

One of the tenets of DevOps is that reliability comes from doing smaller deployments more frequently.

---

**Example**

These two presentations show all the things that Amazon did to decrease the time it takes them to deploy code to production.

According to Gene, the only thing that is changed across time in these high-performing organizations is the number of developers. So from the Amazon example, you could say that in four years they increased their deployments ten times just by adding more people.

---

This means that under certain conditions, with the right architecture, the right technical practices, the right cultural norms, developer productivity **can** scale as the number of developers is increased. And DevOps is definitely in the middle of all this.
> 19 votes

# Answer
What I have done (and this is only subjective) is as follows:

When a manager thinking about a due date wishes to add people into my team to cut the time needed and seems under MMM, I first discuss with him or her on why this could be bad. My favorite analogy for this is to remind them that if one woman can have a baby in nine months, nine women won't have one baby in one month, but they'll have nine babies in nine months. The time isn't cut, just the parallel processing is better.

When the decision is forced upon our team, we usually try to either further divide some tasks, and when this is not possible we usually rely on pair programming, where one programmer is responsible for typing, and the other dictates the code (and switching periodically).

The code writing itself is slower, but there's fewer typo/linting and bugs when testing because of the inevitable review done while writing. I feel the overall code quality is also a little better, but I've no metrics to support that claim.
> 5 votes

# Answer
Speaking exclusively from a CI perspective, increasing the number of developers working on a project typically translates into more people working in the same branch.

Traditional CI systems have a scalability issue in this respect: the probability of breakages/regressions/blockages increases slowing down the integration speed and inviting smaller teams to break off and move on to child branches (i.e. further slowdowns). See How can continuous integration scale for very large projects/teams?. This plays right along the Mythical Man Month concept.

The solution I suggest in my answer to that question, a *highly scalable* CI system would enable migration towards true CI - single branch/trunk based integration for entire greater teams (even with huge sizes).

With everyone of the same page, using the same automated tools/processes and the vast majority of the QA verifications automated inside the CI system itself it becomes much easier to switch roles and focus between team members. The whole development proces becomes smoother, more predictable, more relaxed.

Bringing new people aboard in such an environment, getting them productive simply by offloading less difficult tasks from the more experienced team members which can then take on more difficult ones is thus easier.

All of these can be seen, I believe, as soothing the Mythical Man Month concept effects.
> 4 votes

---
Tags: project-management, process, software-engineering
---


# Question
Title: How can Agile methodologies, CI/CD be applied to massive, monolithic software projects with 100s/1000s of developers?
Using continuous integration represents an (almost) mandatory requirement for a software project to be agile. And without CI there can be no continuous delivery/deployment.

But CI can face scalability challenges, see How can continuous integration scale for very large projects/teams?.

Splitting work on multiple smaller pre-integration branches would allow work on those branches to be agile, but then merging of those pre-integration branches needs to be handled. And those would be massive, unpredictable merges, rather difficult to plan and incorporate into an Agile framework. Until the merge is *actually* performed one can't really tell if it'll take 2 hours, 2 days or 2 entire sprints to complete.

So how can Agile methodologies still be applied to massive, monolithic software projects with 100s/1000s of developers?

# Answer
Well, you won't be able to apply an Agile methodology to a large team. One of usual principle is to work on Pizza Team (Less than 10 persons which can share a large pizza for diner together) because the lack of formalism advocated by Agile process makes it hardly applicable to a large team, the information would be scattered and some would be lost in the crowd.

So before thinking about the CI system, you'll have to review the organization around the software you build.
This mainly need to refactor a large project in smaller independent pieces, this could be separate softwares (and you'll take the path of microservices) or it could just be an architecture design within the software, where a class/object/library is given to a small team. This team has to document its service input and output (service contract/service orientation).

Once this is done you can split the responsibility of each independent piece to a smaller team which will be able to apply an Agile process.
This may solve the CI/CD problem, splitting the code base would allow each team to follow it's own pace, as long as it's interface is properly versioned and that the input/output are fixed for a type of call you can make parts of the product evolve while some other part keep a previous behavior.

The main drawback is that the service contracts from each part *must* be compatible between versions in use, which need a dependency system to know which part use which version of what before dropping an unused version of an API. This brings problems to make breaking changes and usually need a careful transverse planning between producer and consumers.
> 4 votes

# Answer
We went from a large, monolithic backend application worked on by 100s of devs into docker based microservices, deployments using ansible and similar DevOps fluff. Here are some notes on the process:

1. Determine the domain boundaries inside your monolithic app. Make sure calls between boundaries happen as less as possible (by call I mean either in-process calls, or database access).
2. Split the devs into teams, and assign each of them one slice of the domain. (It is possible that one domain will be worked on multiple teams as well)
3. Specify cross boundary calls still happen, and define an API around it. Make sure only this intermediate library is used when doing cross domain calls.
4. Have proper tests around these intermediate libraries, that check that they are working properly.
5. Split up the code based on these domain.

Now, each of your teams should be both smaller, and work on their piece of code without worrying too much about the others. The tests you put into these intermediate libraries should cover regressions from outside callers, and the internal code is your responsibilty. Obviously if the intermediate libraries needs changes, or if your tests tell you that you are not adhering to the API specifications anymore (which can, and will happen occasionally), then you need to start talking to the other team(s) using these APIs.

Once you are happy you can try to replace these intermediate libraries to be HTTP API calls instead of in-process calls. Once this is done, you can actually split up your large app into real, smaller pieces (they will probably still not be "micro" services, but much-much smaller and more manageable). Also they will at least belong to one specific domain, and also have a clear, well defined API that others (including new microservices mostly free of your legacy big app) can use to obtain their data.
> 3 votes

# Answer
Yes, splitting the bigger team into smaller, agile-size sub-teams is obviously necessary. But it's far from sufficient to also be performant overall, at the bigger team level.

With a highly scalable CI system in place it is possible even for very large teams to use trunk-based integration, in a single master branch, thus completely avoiding the intermediate integration branches and the unknowns/risks/delays/costs associated with their merges which would otherwise represent serious obstacles impacting the sub-teams. See How can continuous integration scale for very large projects/teams?

With these obstacles out of the way the sub-teams can really make the best out of the agile techniques as they're automatically all on the same page and benefit from **all** advantages of the CI methodology.

Such approach works equally well:

* for monolithic codebases/producs which can't/won't be split into micro-services for whatever reason
* for codebases/producs split into micro-services, even if for reasons other than independent deployment capabilities - which don't want to deal with inter-service dependency management and prefer to stick to monolithic deployments (they still share the same development branch)

Note: the actual SCM repository is irrelevant, it can be any combination of repos managed together as a single one, the term branch denotes just the common logical representation of whatever physical/actual repo branches exist in that combination.
> 2 votes

# Answer
I've been working on couple of Agile projects with over 100 developers without any problems, so from my personal experience, to deal with such big workflow, here are some suggestions:

* Team definitely should to be split (max. ~12 people each).

Make sure you've the right people with the right skills in each team, so they're independent (cross-functional teams). DevOps doesn't have to be part of development teams.
* Each team should work on their own components/modules, so their work won't conflict with other teams.
* Once feature/component or bug is completed and tested in separate branch, developer should be responsible for sending and maintaining a valid mergeable *Pull Request* to be approved by other team members.
* Often commits and pulling from master branch will avoid any surprises about code conflicts.
* Each Pull Request should be automatically tested in CI and once approved, should be merged (either development or master branch).
* Deployment to development branch should be done on each Pull Request being merged and tested on development environment by automated tests and manually, so only stable code can progress upstream.

To summarize, the main point in order to avoid spending too much time on merging complex branches, introduce *Pull Request*, so each requester is responsible for mergeability of his own code and other developers are responsible for reviewing the change, so no any unpredictable branches or conflicts are possible in this case.

So yes, Agile methodologies can scale for massive scale projects.
> 1 votes

---
Tags: continuous-integration, culture, agile, scalability
---


# Question
Title: Is Elastic Beanstalk suitable for enterprise-grade CD?
I'm working with a project that's using Jenkins to build and deploy microservices to Elastic Beanstalk. We deploy an integration branch to a test environment, release branches to a staging environment, and then a final master build to production. I have a couple of concerns with doing it this way: first, it means we end up with a matrix of one build per project per environment, duplicating efforts; and two, it means we aren't deploying the same build artifacts to production that were validated in staging.

I'm inclined to abandon Beanstalk and move to plain ASGs using something like Chef for deployments. That would leave us with one build per project, producing a build artifact, and we could deploy the same artifact to production that was approved in staging. Transitioning has a not-insignificant up-front cost, however. Is there some way to use Beanstalk better that would allow for more reliable, easier-to-manage CI/CD?

**Note**: Promoting the same build artifact is exactly what I want to do, but from the docs I don't see any clear way to do that; it explains how to deploy to EB from your app source, but not how to promote an existing version to another environment, unless I managed to scroll right past it. If it is available in EB itself, there may be a limitation in the Jenkins EB deployment plugin that prevents it from being done in Jenkins specifically, but I haven't seen a way to do it at all.

# Answer
> 4 votes
IMO opinion your issue isn't with Elastic Beanstalk in that scenario, it's with Jenkins, or at least the way you're using it. You should really concentrate on building "a thing" only once, regardless of what that thing is.

Full Disclosure: I work for ThoughtWorks and am incredibly biased to GoCD. I'll try to spell out what I mean in as neutral was as I can. I will use our tool's docs are examples, but hopefully people can extrapolate to their systems.

Somewhere early in your pipeline you're building "artifacts". This could be binaries representing all or part of your application, or they could be output from any number of tools such as testing tools. These artifacts should be stored by the system and never built again. The system should then fetch the artifact from the proper revision when needed.

For example...

1. I build a .jar file and run some unit tests on it, basic C/I stuff. If it passes, that .jar file and the output of the tests gets uploaded to that specific pipeline job.
2. The next pipeline could be your deployment to a more complex testing environment. It should fetch the *exact* jar from the *exact* job that built it. It then executes Elastic Beanstalk to deploy that jar to the correct environment.
3. The next pipeline is your staging deployment. It goes all the way back to the first pipeline and fetches the *exact* jar from the *exact* job that built it. In then executus Elastic Beanstalk to deploy that jar to the correct environment.

These are each separate pipelines because that allows you to run more in parallel or on demand without blocking.

You could use Elastic Beanstalk, Chef, Puppet, Ansible, uDeploy or any number of other tools to do the actual deployments. That's not where your issue is coming from. Continuous Integration servers weren't built originally to do this. Of course there are lots of plugins you can use to get to the same place if that's your preference.

Continuous Delivery servers like GoCD, Chef Automate and ConcourseCI were built specifically to solve things like this.

---
Tags: jenkins, amazon-web-services, continuous-integration, continuous-delivery
---


# Question
Title: Which Release Management aspects help explain the difference between Waterfall and Agile?
When explaining DevOps to somebody, it happens that a question comes up like:

> **How does Release Management using the Agile methodology differ from Waterfall?**

So what kind of criteria can you use to explain these differences to such audience?

# Answer
IMO DevOps is culture, much like Agile (without choosing an agile methodology.) Therefore you don't "do" DevOps.

You "do" a release methodology called Continuous Delivery as part of a DevOps Culture. (full disclosure, I don't think I've ever referred to CD as a release methodology before, but in my jetlagged state I think it works)

If you'll buy that, then here's the definition of Continuous Delivery from one of the people who wrote the book by the same title, Jez Humble.

> Continuous Delivery is the ability to get changes of all types—including new features, configuration changes, bug fixes and experiments—into production, or into the hands of users, safely and quickly in a sustainable way.
>
> Our goal is to make deployments—whether of a large-scale distributed system, a complex production environment, an embedded system, or an app—predictable, routine affairs that can be performed on demand.
>
> We achieve all this by ensuring our code is always in a deployable state, even in the face of teams of thousands of developers making changes on a daily basis. We thus completely eliminate the integration, testing and hardening phases that traditionally followed “dev complete”, as well as code freezes.

So then, you can work in an Agile methodology, having software you can demonstrate to the business, making sure you're doing proper automated testing, reacting well the change and all of the things that make it better than waterfall. All too often that **doesn't** mean you could actually deploy it to production.

You end up with something like this:

So the software will (probably) be better when you're done then if you didn't have some sort of iterative approach, but you don't really know because real users haven't ever seen it.

What you really want is something that looks more like this:

Every iteration, something gets deployed to production. So, the software is **deployed**. If you decide to create downloads, open up the web server, or however you get software into the hands of users it's **released**.

## What the heck!? I asked about DevOps! Nobody asked about Continuous Delivery!!

So what does DevOps have to do with this?

It's very, very hard (approaching impossible) to truly have your software in a state where you could deploy it whenever you want unless that team is working in a DevOps culture. A culture where System Admins, DBA, SREs, Security People, Devs, QAs, etc., etc., are all part of a single team and not siloed part of an organization with handoffs.

**Note**:

About part of a comment posted to this answer, which was like so:

> About your "... software in a state where you could deploy it whenever you want ...": that reminds me about "automatic pilot" software (in a plane) ... My favorite question about that: "**Imagine an update is applied to such software ... How would you feel about doing so inflight ... While you're in onboard?**".

I love that question (in bold, in the above quote)! The idea of "is it really ready?" is something I rant about all the time - blog. IMO it's vital that you're confident in security, performance and other too often "secondary" tests in order to practice CD. The features are done when they're done, but hackers are always there.
> 11 votes

# Answer
Not sure if there aren't any others, but these are the criteria I use:

```
+-------------------+-----------+-----------+
! Criteria          ! Agile     ! Waterfall !
+-------------------+-----------+-----------+
! Release Events    ! Frequent  ! Rare      !
! Risk              ! Less      ! High      !
! Required Effort   ! Smoother  ! Peaks     !
! Volume of changes ! Small     ! Huge      !
+-------------------+-----------+-----------+

```

And if you really want to experience the difference yourself as a user of some software, then think of using some software (like a Linux distribution), where you have a choice between using either of those releases:

* a "**`Rolling`**" release (==\> Agile).
* a "**`Long Term Support`**" release (==\> Waterfall).
> 2 votes

---
Tags: terminology, culture, release-management, agile, waterfall
---


title: Why does development oppose operations?

text:
# Question
Title: Why does development oppose operations?
I'm still a student, but I'm not knowledgeable about operations, and my English is still bad.

My question is: **Why does development oppose operations**? When does developing oppose operations?

# Answer
> 24 votes
The point of DevOps, is that development **shouldn't** oppose operations, instead they should support each other.

Traditionally, due to waterfall deployments and large scale updates, development would cause operations a variety of problems when deploying due to inadequate testing, changing server environments, the list goes on and on. Essentially, the updates were too large for the operations team to be able to effectively deploy them without some problems arising in the process. These problems might be why you believe that **development opposes operations**.

On the other hand, DevOps works to reduce update size, decrease rigid environments, and generally improve the handoff of the application between development and operations by increasing the amount of times the handoff occurs each year. With the increased number of deployments comes less headaches for operations, because they have either automated a large amount of work required to update the products, or they better anticipate and prepare for the updates.

Tldr: DevOps aims to nullify the theory that **development opposes operations** by creating a mindset where operations and development work together to frequently deploy products in a timely and easily reproducible way.

# Answer
> 12 votes
I think you already got some comprehensive responses, but you said your english isn't great, so I'll try to provide a very brief and understandable answer:

* The primary goal of development is to make changes.
* The primary goal of operations is to keep the environment stable.

These two things conflict. That being said, development and operations should not oppose one another. They should work together to make sure both goals can be achieved. This is the purpose of DevOps.

# Answer
> 11 votes
The tension between development and operations is often caused by misalignment of incentives and attempts to optimize within the team.

**Developers** are often judged by the speed and quantity of issues they can get through and merged into code repository and their reward is often not tied to that code actually working or working correctly. Much less scaling, performance and other factors.

**Operations** are often judged on the stability of the environment and how well the code works in production, but rarely on the quality of the process for bringing changes in quickly.

This creates the issue where developers are incentivized to create a lot of code and throw it over the wall to the operations team and the operations team has incentive to accept as little change as possible to ensure the stability of the environment.

DevOps is in a way the set of solutions to this problem:

* Some of them can be organizational, where the processes and incentives of the teams can change. For example if developers work is only marked as done when it has been running in production for some time, there were no issues and the operations team agrees to take ownership of the code. Similarly operations can be judged more on the speed with which the code is being accepted, while the environment is still within some bounds of stability.
* Another part of the solution can be in communication and cross-polination, where you embed operation people in development team and vice versa. You break the wall between those teams and DevOps engineers are often the result of this type of bridging.
* Tools supporting processes like Continuous Integration and Continuous Deployment are another part of the solution, which can increase the speed of changes, while retaining high quality and fast recovery of the production environment in case of any issues through rollback of code or fast progress of a fix into production.

# Answer
> 7 votes
Most organizations deal with complexity by breaking down their organization into functional parts and demanding that each part figures out how to improve itself. This is often called the "silo" approach.

It is important to understand why this silo approach blocks the success of the business and often fails to improve the organization as a whole. And it doesn't affect just development and operations - it affects all the other functional silos within a large organization such as quality assurance team, finance, product and project management.

As managers of each functional silo are commanded to improve by cutting cost or increasing speed, their reaction is often:

* I am totally overwhelmed fixing problems from the last improvement effort. Leave me alone.
* What do you want me to do, fulfill my responsibilities or work on improvement projects? I do not have time to do both.
* Not again! Here comes another program of the month.

With these typical reactions, any executive launching a minor improvement effort is rarely greeted with enthusiasm. Managers find themselves in fierce competition with other functional areas over resources needed to execute their improvements. So, no wonder the command to improve intensifies cross-functional battles!

Even when a manager completes his part of an improvement project, he then meets other functional areas and other organizations (suppliers, contractors, etc...) that did not do their job. This diminishes or totally negates the results.

This cross-functional tension is not limited to improvement efforts. It is in the very heart of day-to-day decision making and the judgment of management effectiveness across an organization. Here is one real-life example:

> A finance manager was told, "improve." He decided to block hiring contractors that cost more than the nominal price accepted in the market. He implemented the new policy and claimed to have saved $1 million dollars this year. With developers and IT unable to hire contractors to help them use container and container orchestration since these contractors are expensive. The IT operations manager in the same company calculated that not improving in their infrastructure was costing the company well over $100,000 in extra expenses each month. At that rate, the annual savings in hiring contractors were eaten away before the year ends.

You can imagine that the relationship between these two functional areas was not exactly lovey-dovey.

These cross-functional conflicts are driven by the silo approach, where the organization measures each silo in improvement independently. If you are a cost center, improvement naturally means greater efficiency or cost reduction within your silo.

In this frame of reference, costs are seen as obeying the "additive" rule. The costs of each silo added together, equal the total cost of the organization. Therefore, managers see any cost reduction in their area as "good," since they see a direct translation to cost savings for the company as a whole. In this frame of reference, improvement efforts are spread everywhere to attack cost and waste throughout the organization.

A great example when the development team/s start working Agile and pushing code to QA and Operations every two weeks instead of every quarter like they used to. The QA and Operations are not ready for such a change and are blamed for slacking off. Again, this does not contribute much to the love between the people in development and operations.

---
Tags: operations
---


# Question
Title: How to decide between Rolling Releases versus LTS Releases?
There are software packages that offer a choice between either of those types of releases:

* a ***Rolling*** release.
* a ***Long Term Support*** (= LTS) release.

Consider the available Ubuntu releases as an example of this. And imagine that:

* some software requires a Linux server,
* whereas the decision is made (for whatever reason) to go for Ubuntu,
* the decision makers do know what these release types actually mean,
* it turns out that any release starting from 16.04 and above will do,
* but whatever release is selected, all Dev / QA / Operational servers must run the very same release.

**Question**:

What are the ***criteria*** one should use to decide about which type of release to select?

# Answer
> 6 votes
I don't think there's a good answer that applies to all software packages or situations.

Rolling releases are more risky as there is no predictability around when things will change. You may have installed the thing last night and already there's an update to deal with. But what does that update do? Is there enough information to know if you want it? Sometimes, but it depends entirely on the developer/company releasing the package.

Some things to consider before going with a rolling release package:

* Who are the primary users of the package, and in what environment?
* What is your tolerance for risk?
* What is your process for performing upgrades?

LTS releases are intended to be used for longer periods of time and often have published lifetimes that enable you to plan and test migrations to the next LTS release well before updates to the package/system end.

My recommendation is that servers (with the exception of sandbox/R&D machines) should use LTS releases. On desktop machines it depends on the end user and what that individual wants.

In fact, I would go so far as to say unless there are specific and very good reasons to go with a rolling release, always use LTS releases.

**Note**:

When dealing with higher-ups, appealing to their desire not to spend money is always a good tactic. Rolling releases are more expensive in the long run when you consider the amount of time spent by employees testing, installing, dealing with bugs from new versions, etc. While you'll have to do some of this with LTS releases, their longevity means it's significantly less.

# Answer
> 1 votes
A rolling release is newer, with cutting-edge functionality, but potentially more defects. LTS is more stable. If stability is your priority (as it generally is for DevOps), LTS would be the release to use.

**Note**: Deploying your own code often is unrelated to choosing between a bleeding edge or LTS release of a third party product. You want your external dependencies to be as stable as possible to mitigate risk so that you are more free to do more frequent releases of your own product.

---
Tags: release
---


# Question
Title: How to check Jenkins build executor free memory?
In Jenkins when clicking on *Build Executor Status* I can only see free disk space related statistics (URI: `/computer`). How I can monitor free system memory (RAM) in Jenkins?

I'm asking, because sometimes when I had too many executors (despite having swap space configured, but not the one below), Jenkins was freezing or crashing a lot.

# Answer
Have a look at the Monitoring plugin. Here are some more details about it (from the linked page):

> * Charts of memory, cpu, system load average, http response times by day, week, month, year or custom period
> * Statistics of http requests with mean response times, mean cpu times, mean response size by request and by day, week, month, year or custom period
> * Errors and logs
> * Current http requests
> * Threads
> * Heap histogram (instances and sizes by class)
> * Http sessions
> * Process list of OS
> * MBeans
> * Actions for GC, heap dump and invalidate session(s)
> * Report in html or pdf
> * In English, German, French, Portuguese or Chinese
> * Jenkins security
> * For Jenkins nodes (slaves in general):
>     + The report for the nodes is available at `http://yourhost/monitoring/nodes`
>     + Charts aggregated for all nodes of memory, cpu, system load average, number of running builds, build queue length, build times by period
>     + Detailed statistics of the build times and of the build steps by period
>     + Threads, process list and MBeans for each nodes
>     + Heap histogram aggregated for all nodes
> * For each individual node (each node in `http://yourhost/computer`), reports and actions are available from the "Monitoring" page in the contextual menu or in the detail of the node:
>     + Threads, process list, MBeans of that node only
>     + Heap histogram of that node
>     + Actions for GC, heap dump
> * And more...

Credits: Assaf Lavie (who first mentioned it in a comment, but for some reason didn't post that as an actual answer ...).
> 10 votes

# Answer
As for workaround, the actual memory can be checked by invoking Groovy commands directly in *Script Console* (at `/computer/(master)/script`). Example command:

```
println "free -m".execute().text

```
> 5 votes

---
Tags: jenkins, jenkins-plugins, monitoring
---


# Question
Title: What is a 'Feature Flag Toggle' and when to use them (or not)?
There are some questions about **`feature flag toggles`**, such as:

**Questions**:

* What is actually a "feature flag toggle" (in the context of DevOps)?
* Why are they used?

# Answer
Without repeating the content of https://martinfowler.com/articles/feature-toggles.html, since it is an amazing in-depth explanation on what feature flag toggles are. I will just focus on the DevOps aspects.

According to the 2014 State of DevOps Report prepared by PuppetLabs, there are four major metrics to measure IT performance:

* Lead time for changes
* Release frequency
* Time to restore service
* Change fail rate

These also contribute to organisational performance overall. So it means that if your IT is doing great on these metrics, your bottom line gets more $$$.

Continuous Delivery is enabled by these metrics, and has been described in depth in the book Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble.

In context of Continuous Delivery, there is an important distinction that differentiates it from Continuous Deployment. And that is the decision when to do a **release** of features (to customers).

Keeping changes smaller in size, and deploying (copying code) half-baked features to production systems with a feature flag toggled *off* allows to shorten **lead time for changes**.

When features are finally finished, doing a *release* is a decision left to the business. Maybe a release of a new feature needs to be aligned with some marketing, or a release in another part of the business like a feature in the mobile app.

Features can be released using A/B experiements to only a part of the customer base, or to specific people, or even direct to general availability (GA). Although releasing to GA is often done only after there is enough certainty that the feature works as expected. One might argue that this in effect affects **release frequency** to be higher.

*This decoupling of **release** and **deploy** is almost impossible to achieve without feature flag toggles.*

Naturally when no deployment is required to toggle a feature *off*, then the **time to restore service** is lowered substantially.

And by using feature flags that release features to a small slice of the customer base, the **change fail rate** metric can be improved significantly as well.

---

So a simple mechanism called *feature flag toggles* enables much better IT performance, and in turn improves organisational performance overall.

Great example of how this is done in real companies can be found at Flickr (on of the earliest public posts on the subject), and at Etsy. But many others have adopted the practice and talked about it in length, for example the famous engineering culture at Spotify videos.

Etsy are showing off their internal tool to manage feature flags, called Catapult, in multiple presentations found around the web. And Intuit release an open-source tool called Wasabi that helps manage feature flags.
> 14 votes

# Answer
Ken Mugrage posted an interesting comment below my question, with a link to an illuminating explanation of "Feature toggles", with a summary of it like so:

> Feature toggles are a powerful technique, allowing teams to modify system behavior without changing code. They fall into various usage categories, and it's important to take that categorization into account when implementing and managing toggles. Toggles introduce complexity. We can keep that complexity in check by using smart toggle implementation practices and appropriate tools to manage our toggle configuration, but we should also aim to constrain the number of toggles in our system.

Not only does the above summary help to understand what this is about, but it also contains some examples that explain *why they are use*. And after digesting it a bit further, it seems that "Feature Toggles" and "Feature Flag Toggles" are pretty much a synonym of each other.

But, solution (answer) to the problem (question), changes the problem ... one might ask related questions such as:

* What are the pros/cons about using them? It's a powerful concept, but also a scary one if not used wisely (and secured appropriately) ...
* What could be some examples (good ones and bads ones) where they are used? I can think of quite a few of them, some of them I've used myself quite some time ago (from before DevOps was even a thing).
> 1 votes

---
Tags: terminology, feature-toggles
---


# Question
Title: What are the limitations of Puppet in comparison to Ansible?
I would like to understand the differences between Puppet and Ansible, especially what kind of Puppet limitations has in comparison to Ansible.

Are there any things you cannot do it in Puppet, but you can in Ansible? In other words, why some people moving away from Puppet to Ansible?

# Answer
There are of course several pro and cons for each of Puppet, Ansible, Chef and *add your favorite tool here* as well. So I'll try to stay away from opinion, and share what is great in Ansible as a matter of fact.

The main capability that puts Ansible above the others is not having to rely on some custom/additional agent running on the target nodes, instead being founded on ssh connections only. Yes, it still requires an ssh server, Python and a bunch of Python libraries on the nodes, and if your distro of choice (or, good luck, there are some windows nodes) does not ship with them, it will be a little bit painful to bootstrap. But that's unlikely, and may well even make you think again about your distro.

That will simplify monitoring, not eat additional resources, not force the system to run a daemon as root all the time, and in general feels better inside the UNIX philosophy. Chef has `chef-solo`, Puppet can be run master-less, but they both work "the other direction," by cloning and via hooks respectively. While with Ansible, a merge in the source repository can trigger the deployment in a fashion we all are comfortable with, be it in Jenkins, in the git master, or in some other tool like Rundeck for instance.
> 21 votes

# Answer
No, people moving away from Puppet to Ansible (or vice versa) has nothing to do with what can or cannot be accomplished with either tool. Puppet/Chef/Ansible - it's mostly a matter of taste.

For example, Ansible is based on Python, and Python developers typically feel more at home with it (no need to learn a DSL), or Ruby (for Chef)). Easier for Python developers to extend Ansible as well.

But in essence they're all very similar in terms of what you can achieve. Some have relative strengths in some areas, and weaknesses in others, but typically the choice between them is made by style/culture/preference of the team.
> 11 votes

# Answer
Until Puppet 4.0 there was no easy way to orchestrate application spread over multiple servers or services, as it was hard to specifically order actions in Puppet, which was a design choice. Ansible was better at orchestrating and ordering the steps, especially across multiple servers. This was especially significant in applications where the wrong order of steps could lead to errors unrecoverable through repetition of those steps until an eventual consistency could be reached.

That is no longer an issue and so the distinctions are largely preference based.
> 8 votes

---
Tags: ansible, comparison, puppet
---


# Question
Title: How to achieve a smooth transition from “the one big VCS repository for all products” organisation model to the “many small VCS repositories” model?
It is a common scenario that the codebase of a product held by a in some VCS system evolves to a point where that codebase can arguably be seen as containing several products. Splitting the codebase across several VCS repositories, each dedicated to a single product, can leverage several benefits (see *Benefits of having a product per VCS repository over the bloat repository model* below). On the technical side, splitting the codebase is a rather easy step as most VCS support this operation. The split however might rise engineering issues related to automated testing, continuous delivery, service integration or monitoring (see *Issues raised by the split*.) Organisations planning to perform such a split therefore need to know how to perform this transition as smoothly as possible, that is, without interrupting their delivery and monitoring pipeline. The first step of this is probably to better understand the notion of *project* and how to delineate the split in a monolithic codebase.

In the answers to this questions, I would like to see:

1. An attempt to give a working definition of what a product is, which gives practical criterions to actually delineate products in an existing codebase.
2. According to this working definition, elaborate a plan that actually perform the split. We can make the simplifying assumption that the codebase is processed by a fully automated implementing and . That is, each branch is validated by an automated testsuite implemented in the current codebase and each merge to some “magic” branch generate that are tested and deployed. (*Product artefacts* are *e.g.* source tarballs, documentation, binary software packages, **Docker** images, AMIs, unikernels.)
3. Such a plan is satisfying if it explains how to circumvent the

## Issues raised by the split

1. How automated testing procedures relate to the pre-existing monolithic repository and the split repositories?
2. How automated deployment procedures relate to the pre-existing monolithic repository and the split repositories?
3. Where is stored the code for automated deployment procedures themselves?
4. Where are stored , and strategies?
5. How to ensure that a developer needs only one codebase at a time (but possible uses artefacts from other codebases).
6. How can a tool like **git-bisect**

---

## Marginal note: Benefits of having a product per VCS repository over the bloat repository model

Having several small repositories holding the codebase for a specific product has the following advantages over the “bloat repository” approach:

1. With a bloat repository it is hard to roll back a release when a product is unstable, because history is mixed with other product history.
2. With a bloat repository, it is hard to review project history or pulls, with small repositories, we are more likely to read this information. (This might be specific to VCS like git, where unlike svn, we cannot checkout subtrees!)
3. With a bloat repository, we have to do much more branch-dance when we develop. If we have N repositories we can work in parallel on N branches, if we have only 1 repository we only can work on one branch, or have a load of working copies which also are a hassle to handle.
4. With several small repositories, the logs give a heat map of the project. They can even be used as a proxy of knowledge diffusion in the dev team: if I did not commit in repo X since 3 months, it could be good to assign me in a team working on repo X so that I stay aware of the developments in that component.
5. With small repositories, it is easier to gain a clear overview of a component. If everything goes in a single big repository, there is no tangible artefact delineating each component, and the codebase can easily drift towards the big ball of mud.
6. Small repositories force us to work on interfaces between components. But since we want to have a good capsulation, this is a work we should do anyway, so I would count this as an advantage for small repositories.
7. With several small repositories, it is easier to have several product owners.
8. With several small repositories, it is easier to have simple code standards that are pertinent to a full repository and that can be automatically verified.

# Answer
> 9 votes
It is a fascinating question for which real answers may not actually exist; I appreciate that while you tried to keep the question contextualized on the VCS, it naturally scaled by itself up to infrastructure design and implementation planning.

Though, it seems many of us are working of this kind of transitioning, which can be exciting, but at the same time so frustrating and painful; and I would like to share my experiences and views, trying not to be pedantic, and just because I may not be such a good engineer, also not to be boring.

*Design*

Infrastructure and architecture should go together to write a modern software. Tightly coupled, if you want. It might sounds weird to use those words, but we are not certainly talking about code itself here: I mean they must be part of the same blueprint. When the clouds arrived, and people started to write software for them, how many people then realized that by putting the mudballs there, they just would be the same mudballs in a different place(?) Maybe a few forward thinking people could foresee that, and they are likely working in devops today. As devops is just a buzzword with so many differnt meanings for differnt people, I have seen places in which the devops team would sit in every architecture meeting; other places in which is automation only. To achieve this kind of transformation, we have to fight our way to sit there.

*Confidence*

The transition must be kept isolated, in the sense that a consistent cut of history must exist, that provides the transition itself and itself only, without any other change (after several months of preparation). With what confidence one would approve it and push the red button?

I mean the codebase must change to accommodate the new VCS structure, and it will be very difficult to keep it merged during development. (for this issue there may be facilitating strategies, I'll talk about a common one later, that can help parallelize development a bit).

Well I bet the only way is with behavioural testing, and the same suite of behaviour tests should be launched to verify the old with new codebase. We are not verifying the application behaves as expected here, but that the transition does not alter the behaviour. Having failing tests may be a good thing! If they continue to fail!

In fact it is very uncommon for mudballs to be well tested; usually the code is very tightly coupled, and likely, for most legacy code, it was not developed with a test driven approach, not even unit tests.

If such test code is missing whatsoever, it shall be written first.

*Strategy*

Yes, the transition must be kept isolated; but at the same time integrated. I know I may sound crazy here, but I wouldn't find other words to describe how confidence can keep up to business. Very few, if none at all, companies would like to stop development for a big monolithic codebase, to make space for this transition, and we are not making it just happen within a toss of a coin. Maybe hundreds of developers might be continuously contributing to the codebase (I would use the *tampering* word here, from our POV). While our work must address a specific snapshot to provide confidence, we have to keep ourselves rebased (not in a git meaning here), to avoid to fall behind forever.

The implementation strategy in here can give different experiences. A common line of development is to wrap/adapt (exposing endpoints with optionally rearranged schemes) newer implementation branches (well, living in other repositories in this case), when they need to interact with the core. Transitioning with a strategy like this, along with refactoring, can at the same time offer a POC scenario for the VCS transition, and later on a step by step approach. See it like sculpting the ball of mud. Yeah life offers so many funnier things.

*Technical Debt*

Business management spheres started to understand technical debt and keep it into consideration. Nope, scratch that, not true. While is it increasingly common to gather measurements and quality data, in terms of static code analysis, code reviewing, behavioural test results, and performance, and generate nice reports and everything... it still remains incredibly difficult to make the business accept a continuous refactoring approach. The *business value* of it. *"We are following an agile process, and this will not bring in any enhancement to the features, wouldn't it?"*. Basically, by doing so they are negating the existence of technical debt. I see this as the common missing necessary step to be able to start any transition from monolith to microservices architectures.

*Reaggregation*

After all this, you might still want to provide a single repository-like view in which you can build more than one single product. For any reason, ie curr/next release, multibrand, customer builds. Tools like google repo may help in this case. Never used one myself, but I know I'll need one day.

*Integration testing*

With microservices, integration testing assumes a different context, of "testing own API". Higher layers of testing, functional or performance, can and will exists, but do they mean much without proper integration testing? It would be like having integration testing without unit testing. Of course not. That's why, if you need to git bisect, you will execute it in a microservice repository branch, forget about running that in the mudball repository.

PS. this is a draft, my English is bad and I'll fix it one day

---
Tags: deployment, culture, repository
---


# Question
Title: Which ways are there to decouple deployment from release?
One method of continuous deployment is to decouple deployment from release, i.e. deploy updates without activating the changes immediately.

I know that can be used for this, but I'm wondering if there are other techniques for "non-features".

For example, would you build a feature toggle for a bugfix? Probably not and one could argue that bugfixes should be deployed as soon as possible, because it can only get better. And after a bugfix has been released, I sure don't want to toggle it off anymore. But is this the case? It might be a risky change that you want to release in a controlled way. And if there *are* unexpected side effects, it's good to be able to roll it back. So, feature flags for every change?

And what about visual changes? For example, can you implement something like a feature flag in CSS? Does it even make sense?

# Answer
For software in the web app category, depending on you infra/hosting provider such decoupling *might* be possible to switch incoming traffic across (or split it between) different deployed versions of the sw, practically covering any of the changes you mention: bugfixes, visuals, etc.

Such support would typically not require feature toggles. And it might be applicable regardless of the app being monolithic or split in microservices.

For example Google's App Engine Paas offering has support for traffic splitting and migration.

From Splitting Traffic:

> You can use traffic splitting to specify a percentage distribution of traffic across two or more of the versions within a service. Splitting traffic allows you to conduct A/B testing between your versions and provides control over the pace when rolling out features.

From Migrating Traffic:

> Traffic migration switches the request routing between the versions within a service of your application, moving traffic from one or more versions to a single new version.
> 7 votes

# Answer
While with monoliths you might be restricted to switches, with microservice architectures, you can split every deployment pool of nodes providing a service (ie. pods). You then activate the deployment of the newly changed product in a subset of pool, and carefully monitor it; you can even choose which amount of the pool to deploy the change to, say for instance, activate the change for 15% of the traffic. You might find the feature called "rolling update" in literature.
> 5 votes

---
Tags: deployment, release-management, feature-toggles
---


# Question
Title: How to communicate queue-based processing delays to non-technical team members?
I am responsible for a set of SQS queue-processing jobs with a scaling policy on the `ApproximateNumberOfMessagesVisible` CloudWatch metric. These jobs can fail to keep up with the amount of messages sent for any number of reasons:

* Service degradation reduces capacity of messages able to be processed.
* `AutoScaling` max limit reached while queue depth continues to rise.
* S3 Outage impacts other dependent AWS services (`AutoScaling` service) that the queue processing job uses to keep up with demand.

When discussing outages with non-technical team members, I would like to communicate specific delays of queue processing that can translate into customer-visible degradations. How can I do this with SQS queues?

# Answer
As with any outage communications, a non-technical reader will be primarily looking to understand:

* How long was it?
* How bad was it?

Amazon CloudWatch metrics provides the following metrics for SQS queues that can help answer these questions:

* **NumberOfMessagesSent:** The number of messages added to a queue.
* **NumberOfMessagesReceived:** The number of messages returned by calls to the ReceiveMessage API action.
* **ApproximateNumberOfMessagesVisible:** The number of messages available for retrieval from the queue.

When graphed correctly, these metrics can be powerful visual aides in describing queue processing delays. Here are a couple of examples from an outage I experienced where the job's capacity to process queue messages was severely degraded:

## NumberOfMessagesSent & NumberOfMessagesReceived

* **Graph Type:** Line Graph
* **Statistic:** Sum
* **Period:** 5-minutes

This graphs the contrast between messages sent and received, which helps isolate which processing component is responsible for the delay. In this graph, the received metric drops dramatically while the sent metric continues on its normal trend, so we can infer that the issue lies in the queue reading component instead of the queue writing component.

**Does this answer how long/how bad event was?** Yes; describes processes impacted over time.

## NumberOfMessagesReceived & ApproximateNumberOfMessagesVisible

* **Graph Type:** Stacked Area Graph
* **Statistic:** Sum
* **Period:** 5-minutes

This graphs the queue depth on top of messages received, which helps show how far the queue backed up and how it recovered. In this graph, we can see that the queue depth backed up dramatically while the queue reading component was having issues, and began recovering when the queue reading component began reading messages again.

**Does this answer how long/how bad event was?** Yes; describes messages impacted over time.

---

## Graphing Discussion

In both graphs, queue processing can generally be considered healthy when the lines overlap, and unhealthy when the lines diverge. This is an easy pattern to teach to a non-technical team member, and can help them to quickly dissemenate where and how there are problems when presented with these graphs.

To further communicate specific points in graphs, you can simply annotate them:

**Graphing Tips:**

* Label units and axes.
* Use consistent colors for matching metrics across graphs. Note that NumberOfMessagesReceived is orange in both graphs; this will help visualize the same metric across different graphs.
* Vertically align graphs that describe similar metrics so that they are easier to compare across time.

**Note:** I've formatted these graphs for presentation on StackExchange, so these aren't necessarily how I would present them in an outage post-mortem. I have explicitly removed values from the left axis here to obscure them from StackExchange; you'll want to keep them in your post-mortems.

---

## Additional tips

* **Empower Your Team:** After training your team members to read these graphs, there is no reason keep them hidden away. Consider setting up a CloudWatch Dashboard and giving your non-technical team members read-only IAM access to CloudWatch, so that they can view these graphs any time.
* **Set Up Notifications:** Consider setting up Cloudwatch Alarms based on the ApproximateNumberOfMessagesVisible metric if it exceeds some agreed-upon high value, and subscribe team members to notify them of potential issues. Cloudwatch Alarms have description fields that are sent along with the notification emails -- make sure to include a human-readable description to help your non-technical members disseminate the alarm.
* **Explore Other Data:** Per Evgeny's comment, explore other data beyond what CloudWatch provides and think about how you can convey that data to your team. His example of using message lifetime in the queue to create a histogram is a great example of this creative thinking, and can be accomplished by logging both the message send and message receive times in your application. You can get the message Sent Timestamp via the SentTimeStamp Attribute on each queue message of the ReceiveMessage API response. More details here.
> 16 votes

---
Tags: amazon-web-services, amazon-sqs, amazon-cloudwatch, post-mortem
---


# Question
Title: How to transition from a complex branching reality to a single-branch model?
In large organisations, using the waterfall methodology typically results in very complex branching structures (aka *branch spagetti*).

What branching strategies can be used to transition from a complex branching reality to a single-branch model like trunk-based development?

Update:

To clarify, the question is about the **migration/transition itself**, not about the before and the after methodologies, which are pretty clear.

It can't really be "at EOB today we're still waterfall with gazillion of branches but tomorrow 1st thing we'll switchover to trunk-based, single-branch CI".

# Answer
Because you mention waterfall, I understand that the numerous branches you are alluding to are feature-branches rather than maintenance-branches.

In this setup, I also assume that these branches are created according to a waterfall plan that tries to minimise conflicts. This implies that the goal of the development is to produce several distinct products. When using a single-branch development model, it is important to also work on a single product. If several products are simultaneously developed in a single-branch development model, it effectively “glues” together versions of theses products, so that we might have in version *a* of the repository a healthy product *X* and a buggy product *Y*, while in version *b* the product *X* has a regression and *Y* a bug fix, but we have no version where both *X* and *Y* are healthy. Such a situation would force us to consider *X* and *Y* as being developed in distinct repositories, which is a hint that they should be.

Therefore, the first steps should perform a repository split:

1. Arrange the repository so that it is easy to split it into several small repositories. For instance, rearrange the current repository so that each top-level directory corresponds to a repository you want to create in the future. Doing so, you can continue to use the *branch-spaghetti* discipline everybody is familiar with.
2. When step 1 is completed, refine the *branch-spaghetti* discipline by requiring that any single branch can only touch files in one top-level directory.
3. When each branch complies with step 2, perform the split. Developers can easily convert their pending changes to patch a single repository, just by removing the first-level of the path.

Now that the split has been performed, you can start working on the branch discipline itself.

4. Introduce programming techniques helping the development of short-lived branches. Branches being short-lived is a crucial aspect of all single-branch methodologies. One of their goals is to reduce the time spent on merging and debugging long-lived branches. A popular technique is the introduction of “feature-flags” where a “factory” uses a configuration flag to either produce the historic version of an object or the new, initially partially developed, version of that object.
5. By now, you have zillions of repositories with only a few branches in each, and you can turn the “we globally adopt the trunk-based development discipline” button, without seeing the original *branch-spaghetti* mountain collapsing on the trunk.

The actual split of repositories might be optional, but you would then have to adopt policies that cleanly delineate the allowed scope of each submitted patch (to limit the risk of conflicts when merging changes in the main branch). Reducing the overhead bound to conflicts is one of the goals of single-branch model methodologies, so I assume this is relevant in your context.
> 11 votes

# Answer
When migrating from something to something else, there are only two things you need to define:

1. What is your target
2. How to get there (the migration plan)

The first part is, sadly, often overlooked or **way** too vague. You cannot simply say that what you have is a mess and you want to organize it. What would that mean? Everybody would have a different interpretation (aka: every dev thinks that *his* or *her* way of doing things is the best).

Chances are, all the branches you have are serving or have served a purpose. Without a clearly defined target process, people will keep doing what works for them the way it suits them best (and rightly so).

For example, your target should be defined as clearly as Vincent Driessen defined his "successful Git branching model". If you look at this model, it is very precise: It says where stable code should be, and where unstable features should be developed. It also says how - and when - to branch, update and merge back. You know what each branch is for, and what to do with them. We use a variation of what was put forward by Vincent and our variation is defined in our wiki.

The important point is to have all the team understand and agree on a target. It might be worth it to remind people that you are not looking for their personal favorite branching model, but a model that all the team members can agree on and use easily.

Once you have your target, you'll be able to elaborate you migration plan. That plan can be as long or as short as you'd like. I've seen such branching model imposed overnight; at other places, it was done over 2 or 3 sprints. It doesn't matter much to me, as long as we are improving.

You can start with the "biggest" or more important branches. E.g.: "from now on, master must always be in a state to be deployed in prod and the dev branch must always compile" (or whatever your rules). Then, enforce version (release) branches. Afterwards, enforce feature branches. After that, impose a code freeze on version branch, if it make sense.

DevOps is all about communication, openness and efficiency. These concepts must be kept in mind and communicated throughout the process.

I would suggest to invite some people outside of the development team to the process meeting as observers. Ops or middle management might have a thing or two to say about your model. The developers' needs should be prioritized, but if the branching model is impossible to align with the way things are managed, you'd be better knowing now and not in a month or two.

If you have really big teams, try to include everyone nonetheless. With very big teams, you'll end up with two or three meetings anyway. So invite team leaders in the room, but have a webcast available and let everyone know about it. If anyone has a suggestion or concern, they'll be able to voice it to their team leader and if it is valid, it will be addressed on the second or third meeting.
> 6 votes

# Answer
It is actually very simple to convert a multi-branched hydra repository into a single branched model.

First, you want to start with the branches which have the least difference between itself and master or trunk. Examine their age and relevance. If they are still relevant, start merging them together and resolving conflicts. If they are no longer relevant, then delete them.

Continue this process until you have managed to merge all your branches, resolved all conflicts, and you have only a single branch remaining.

You can follow this simple outline to get started:

1. Create a copy of your master/trunk branch and call it `temp_master`
2. Find the branch with the greatest divergence from master/trunk.
3. Determine if the branch needs to be kept, archived, or deleted.
    1. If it needs to be kept, continue to step 4.
    2. If it needs to be deleted or archived, delete and archive it, and then return to step 2.
4. Repeat step 2 to find the next branch with the least divergence.
5. Merge the two branches found in step 2 and step 3, resolving all conflicts.
6. Merge your these two branches into your `temp_master` branch.
7. Test the code in temp\_master code to see if it compiles, and builds, and run any other automated tests you have for sanity.
    1. If any tests fail, go back, find out why, and fix them, and repeat the process.
    2. If tests still fail, pick two different branches to work with.
8. Repeat steps 2 - 7 until you have only two branches, your master/trunk and `temp_master`.
9. Finally, merge `temp_master` into master/trunk and live with your new single-branch model.
> 3 votes

# Answer
For large organisations with typical 4 weeks sprint cycle Git-Flow is preferred approach because You get benefit of Feature branch Master production ready branch is always deployable Also, master branch is kept clean from unwanted commits by following two commit cycle (from feature to `Develop` and `Develop` branch to Master).

Moreover, branching is also determined by frequency of production releases. For frequent deployment to production its better to have a Feature branch or Centralized model. In this case overhead of managing branches is shifted to robust testing in lower environments to maintain production stability.
> -4 votes

---
Tags: continuous-integration, branch, waterfall
---


# Question
Title: How to safeguard Ansible deployment to mitigate accidents?
Recently the Amazon S3 had a major outage in the us-east-1 region. It looks like it was likely caused by a spelling error when running a maintenance playbook in Ansible or a similar tool. You can put a shell script wrapper around ansible-playbook to look like:

```
#!/bin/bash
/usr/bin/ansible-playbook "$@" --list-hosts --list-tasks
read -p "Are you sure? (y/n) " answer
test "$answer" = "y" || exit 0
exec /usr/bin/ansible-playbook "$@"

```

But what are some other ways you use to improve the safety and reduce a chance of error causing a major outage for your company.

# Answer
> 6 votes
We're using jobs in jenkins to trigger deployments. It ensures that no matter who does the deployment, the ansible command that is run will be the same. A nice bonus is the build logs record when deployments were triggered, who triggered them and what exactly happened during the deployment.

It's certainly not foolproof, but it's been a nice improvement over running ansible playbooks by hand.

For larger/riskier changes this should ideally be combined with some form of change management so changes are made only after another person/team reviews the change and the approach to the change to help identify and resolve potential issues early.

Additionally it never hurts to have a teammate who understands the change you're making be present and watching while you make big changes so they can watch for and help prevent mistakes in the execution of the change.

# Answer
> 4 votes
**Categories of errors**

There are two ways of looking at human factors that lead to problems and accidents:

1. You can see the human error as the cause of a mishap. In this case "human error," under whatever label—loss of situation awareness, procedural violation, regulatory shortfalls, managerial deficiencies is the conclusion of your investigation.
2. You can see the human error as the symptom of deeper trouble. In this case, human error is the starting point for your investigation. You will probe how human error is systematically connected to features of people's tools, tasks, and operational / organizational environment.

The first is called the *human* approach, and the second as the *system* approach.

To explain failure using the human approach, you would seek failure and find people's inaccurate assessments, wrong decisions or bad judgments.

To explain failure using the system approach, you are not trying to find where people went wrong. Instead, find how people's assessments and actions made sense at the time, given the circumstances that surrounded them.

For example, Donald Berwick of the Institute for Healthcare Improvement (IHI) argues that improving patient safety requires *changes in the design of systems*:

> ...We are human, and humans err. Despite outrage, despite grief, despite experience, despite our best efforts, despite our deepest wishes, we are born fallible and will remain so. Being careful helps, but it brings us nowhere near perfection... The remedy is in changing systems of work. The remedy is in design. The goal should be extreme safety. I believe we should be as safe in our hospitals as we are in our homes. But we cannot reach that goal through exhortation, censure, outrage, and shame. We can reach it only by a commitment to change, so that normal, human errors can be made irrelevant to the outcome, continually found, and skillfully mitigated.
>
> Donald M Berwick. Not again! BMJ 2001

---

**Removing mistakes from the system**

A great way to find (and correct) the various ways failure happens after the fact, is searching for the root cause without blaming the people. This is often called "blameless post-mortems", and Etsy Code as Craft blog post expands on the concept. People at Etsy presented and wrote more about it on other forums and blogs.

To prevent mistakes in the first place, some culture traits are a must. Procedures and various artifacts created in the system must test that using them by humans is very clear and self-explanatory. Often those who create are not those who consume, leading to a disconnect and lack of clarity. The system is then not safe to operate since the only person who knows all the assumptions is the one who created it (and no one else).

*Effective control measures*

Put in place effective control measures to stop the process when an error occurs. This is mistake-proofing. Effective control measures are design changes that prevent or stop processes from continuing when an error has occurred by introducing a process failure

Example:

In 1896, Sakichi Toyoda invented Japan’s first power loom called “the Toyoda Steam power loom.” This development increased productivity by twenty times, and the quality of the textiles improved and caused a revolution in the textile industry in Japan. But here’s the subtle but very important discovery and principle:

> when the needle broke, the machine stopped

Sakichi Toyoda created an innovation to the Loom that would later become one of the pillars in the Toyota Production System (Lean). That pillar we now call Jidoka, sometimes called “smart automation with a human touch” or “autonomation.”

In large part, Andon (stop at first defect) and Poka-Yoke (mistake proofing) are later developments that find their influence from the Loom.

*Removing Single-Point Weaknesses*

The term single-point weakness refers to the creation of redundancies in the system as an approach to improving system reliability. Redundancy is created by increasing the number of systems or individuals involved in the process. Having more backup systems or more checks (double, triple, or more) increases the probability that the process will proceed correctly.

One great example for this is the "four-eyes principle," which means that "all business decisions and transactions need approval from the CEO and CFO. Since the CFO is not reporting to the CEO, there is an independent controlling mechanism in place".

source: https://en.wikipedia.org/wiki/Two-man\_rule

*Make Hazards Obvious*

If hazards are made obvious, or impossible to reach, humans cannot create mistakes. For example, color-coding is a common approach to making mistakes more obvious. Or if you think of various computer sockets that can only be inserted one way and not the other, etc.

---

Some great books are talking about the subject, and it would not be a good answer without mentioning them:

# Answer
> 1 votes
As @bradim said using your CI/CD tool to initiate the deployment instead of hand based commands is usually good step forward, as is adding tests in your pipeline that actually test your deployment scripts on your staging (or a freshly created) environment, where you can pick up bugs earlier.

I would also add that instead of calling your ansible scripts directly, you can also add tools like Ansible Tower into your flow, which will let you track the changes that have been run more easily, and can give you an additional step of security into your flow.

---
Tags: ansible, process
---