# Question
Title: Is DevOps restricted to companies with SaaS products?
The practices describing DevOps, such as continuous delivery, automation, etc. are relevant to products that provide continuous service, such as SaaS products.

For example, a software development company that mostly does projects for other clients might never be maintained these after the project is over. And client projects are not shared with other clients, because irrelevant.

Does DevOps even apply to companies who develop multiple projects that are one-offs? What DevOps practices apply in this case, if at all?

# Answer
> 32 votes
Absolutely not!

DevOps is all about breaking down the traditional silos (departments) in order to be more efficient.

Better communication between teams, improved visibility and reliable and automated process are ways to achieve a better product.

I used to work for a big media company where we would support an internal tool and develop public-facing websites.

The benefits of DevOps in our case were the following:

* Through continuous building, we let know the development team earlier rather than later if there are integration or build problems with their code. They can fix issues while their mind is still on the piece of code they just committed.
* Through continuous testing and delivery (into QA), we enabled the QA team to find problems earlier and report them earlier. This reduced the time it took to find and correct bugs as well as reduce the complexity of these investigation.
* With out log collection & aggregation tools, we gave to the developers access to something they wouldn't usually look at (they were very keen on the debuggers :) - understanding how logs are seen and used by other teams improved the overall quality of logs
* We often shared information and created documentation to share knowledge between teams, trying to break down walls. By understanding the Ops' needs, we create a few guides for what should always be kept in mind when bootstrapping an application (where/how to manage properties, etc.). By understanding the Dev's reality (code more features, faster, gogogogo!) we were able to have the ops create servers and clusters that were better suited to the dev's needs.
* The overall quality of deployments was greatly improved too. Deployments were handled by our team, so we had perfect visibility on both Ops and Dev. This eliminated many issues related to the "code hand-off" where the dev would hand over a package and one-page document to the ops saying "Install this!".

Overall, I would say that regardless if you are updating your production environment once per day or once per month and regardless of how many customers you have or your business model, every enterprise can use better communication, better tools, better visibility, faster feedback, etc.

# Answer
> 13 votes
My team and I are responsible for developing "one-offs", products that once finished are given to the client for upkeep or in some cases managed by us for a fee.

We still need to maintain a solid development pipeline to handle the constant feedback from our clients in order to ensure that we ship them something reliable and proven to run.

While the client doesn't care about DevOps (in most cases), it is still helpful for us. With DevOps, we can rapidly push new builds, so clients can see feedback in minutes not hours, and we are also able to catch any errors/bugs with our testing via Jenkins/Travis.

To ensure our deployment strategies are the same across projects, we focus on containerizing our applications. Using Docker, we are able to easily hand off the application to our clients.

The cost saved by DevOps is hard to determine. We do have extra costs in the form of software we choose to use for the pipeline (Travis, Jenkins, Puppet, what have you), but we also save time and money by fixing bugs/ giving the clients feedback quickly. Our quick response time keeps our customers happy, in turn, keeping our wallets happy.

# Answer
> 3 votes
I've worked for companies producing software as shrink-wrap products, as fully installed and supported deployments and as embedded code in devices. In all of those companies, DevOps provides essential support for development:

* Automated, reproducible builds of software, with known, controlled configurations of compilers, libraries and other build tools.
* Automated, repeatable system tests for regression testing and for new functionality tests.
* Other automated and regular actions (for example, continuously updated sample screen shots available in all supported languages, for translators to verify and for technical authors to incorporate into user manuals).

In all cases, these are things that individual developers *could* do as one-offs, but that would not be good use of developer time, nor have the same assurance of configuration control that the automated builds have.

# Answer
> 3 votes
The activities on the development and implementation of software previously did not require deep integration between the departments. But for today it is necessary to closely cooperate all departments (Development, IT Operations, Quality Assurance, etc.).

For developers, change is what they get paid for. Business always needs changes to match the modern world. This understanding pushes developers to produce the maximum possible number of changes. IT professionals have a different understanding, in which change is harm. Each of them thinks that it works correctly, benefiting the business. Indeed, if we consider them separately, they are both right.

All employees must understand that they are part of a single process. DevOps cultivates thinking, which makes it possible to realize that the personal decisions and actions of everyone should be directed towards the realization of a single goal. And success should be measured relative to the entire development-to-delivery cycle, and not from the success of individual roles. As a result of close cooperation between developers and maintenance specialists, a new generation of engineers is formed, which take the best achievements of both disciplines and combine them for the benefit of the user. This is manifested in the appearance of cross-functional teams with experience in development, configuration management, database management, testing and infrastructure management.

So the methodology is useful not only in SaaS.

# Answer
> 0 votes
Not at all. While there are already great examples on this thread, I'd like to share an anecdote of my own. In 2001 I inherited a software project with a release that involved the creation of a CD. The documentation for the release process included 40(!) steps documented by a previous engineer, that included some ridiculous hand-written instructions like "open this config file and change the name of the .jar file on line 41 to include the version number of the new release".

We aggressively automated every step of that build process, replacing hand written instructions like that with things like 'patch' scripted with bash. We even had to open tickets with our third party install-tool vendor to make their project files patchable.

Once that process was automated, we bought a 'CD Jukebox'. Every night after the tests would pass, the build machine would automatically create a new install CD. Our testers could come in the next morning, grab a disk, and make sure everything was installable.

We certainly have tighter feedback loops when we can deploy software as a service, but the core principals of automation, feedback, cycle time, small releases, etc. all apply.

---
Tags: terminology
---


# Question
Title: What branching strategies/workflows are available for Git repositories?
I'm looking for a high-level, summary description of the recommended branching strategies/workflows primarily applicable with Git, which would serve as a guide for navigating this environment.

Ideally a summary of pros/cons and maybe intended usage of such strategies/workflows should be mentioned as well.

# Answer
> 11 votes
The Atlassian website includes a comparison of different workflow strategies available when concerned with Git. More strategies exist, like the one used by Linux Kernel team, but are not relevant to most organisations.

The workflow types most commonly used are -

1. Centralized \- every change is added using a single branch.
2. Feature Branch \- every change is added on its own branch and considered for merging into a central branch for integration.
3. Gitflow \- based on feature branch workflow, but also adds another branch for stable releases, and another bunch of branches for hotfixes on that stable release branch.
4. Forking \- in OSS projects often the whole project is copied over to an additional repository and changes are merged between multiple public repositories, or not merged, but still available to the public.

Depending on the organization, one of these might be more suitable.

1. Centralized - non-restrictive, allows everyone to shoot anyone to "break the build" by pushing changes, least overhead and process involved. The advantage is a continuous integration of changes, even when those don't work.
2. Feature branch - allows anyone to add changes by adding some overhead process, but integration into main collaborative work is a controlled process. Has the disadvantage of integrating changes late, when requests to merge are forgotten or not attended.
3. Gitflow - huge overhead in managing the various branches, causes a lot of confusion for regular people who just want to get their changes into the branch. Can work well when most developers only see the "feature branch" part of this. Merging between the development and stable branches often happens late, and integration is not trivial. Adds a *lot* of extra overhead and terminology that everyone must understand well.
4. Forking - the wild west, requires a server that supports managing multiple repositories and repositories per-user (GitHub, GitLab, BitBucket Server, etc...). Allows most freedom for each contributor to do whatever he wants, but does not force integration. Demands that integration is done piecemeal in small bits and very controlled, usually by using pull requests across repositories. Works well in unorganized teams such as OSS project contributors.

# Answer
> 2 votes
You can read book about best practices for git: https://git-scm.com/book/en/v2

Example for some git strategy in project:

1. Creates branch named like a task, feature/XXX-1
2. Sending task for review
3. If not review, fixes and again review
4. If reviewed go to test
5. If test not passed, fixed and again review/tests

First time review:

```
git pull dev
git checkout -b feature/XXX-1
git add .
git commit -m 'XXX-1 My task'
git push origin feature/XXX-1

```

If need fixed after review:

```
git push origin :feature/XXX-1
git reset --soft HEAD~1
git add .
git commit -m 'XXX-1 My task fix'
git push origin feature/XXX-1

```

---
Tags: git, branch, workflow
---


# Question
Title: What role do Configuration Managment tools play in immutable infrastructure?
If I'm using **immutable** servers/containers, do I need tools like Chef, Puppet, Ansible, or Salt? Those config management tools are designed to establish a configuration and then maintain it.

If I'm deploying immutable servers, should I use configuration management tools only for initial provisioning?

# Answer
> 13 votes
There are a few answers to that:

1. Something needs to build those immutable images. It is certainly easier to use old-school-style procedural scripting to build something when starting from a known starting state but this can still get very unwieldy over time (e.g. Dockerfiles), especially when you end up wanting a big matrix of different images for things like different versions of software, different environments, etc. Packer and other image build tools integrate nicely with Chef, Ansible, Puppet, Salt, and more.
2. Immutability is a spectrum, not a binary. Even in a "very immutable" deploy it is not uncommon to have some configuration files that need runtime management. This is also a place where you can use CAPS tools, though lighter-weight options like Consul Templates or etcd might make more sense depending on the overall infra. If you're doing immutable app servers but your database servers are more traditionally managed using Chef, then it might make sense to use Chef even on the immutable side for minor management tasks.
3. Zero-day management. Immutability is nice, but what are you going to do when the next OpenSSL 0day drops? If you have your build pipeline in a position to create immediate hotfix images and deploy them, then that's great. But many people will probably not have that kind of quick-turn ability.
4. Things that can't be immutable. The un-answer, but a whole infra is rarely 100% immutable. Things like database servers and developer workstations (yes, those are part of your infra) are somewhere between difficult and impossible to make immutable.

# Answer
> 0 votes
Absolutely immutable environment is a bad idea, i think.

CM tools like Ansible, Chef, Puppet can be useful in many cases including yours.

I'm using ansible for initial provisioning of VM instances in GCP.

---
Tags: containers, immutable-servers
---


title: Separate Jenkins config from build logs

text:
# Question
Title: Separate Jenkins config from build logs
My `JENKINS_HOME` is (something like) */var/lib/jenkins*

In there, there is *jobs* directory, in it a subdirectory for each job. In each job subdirectory, you have `config.xml` plus the *builds* subdirectory.

I would like to somehow separate my config.xml from the same file structure that contains builds so that I can check them in my version control but to not clutter its view by having all the build logs that each execution writes. If I change a job config, the corresponding config.xml is changed and I can (actually in a Jenkins job) check it in my version control automatically or if I change it manually in a text editor, it gets checked out and deployed. But I don't want the builds in the version management.

Is there a way to separate just the job configs from the build logs in the filesystem?

# Answer
> 4 votes
If you're interested in only backing up configs, try installing the SCM Sync Configuration Plugin. This plugin syncs with a repo each time you change either a job or the Jenkins configs, but does not keep builds.

> SCM Sync Configuration Jenkins/Hudson plugin is aimed at 2 main features :
>
> Keep sync'ed your config.xml (and other ressources) jenkins/hudson files with a SCM repository
>
> Track changes (and author) made on every file with commit messages

As a bonus you can set it to take a commit message each time you change a job, for easier restoration if necessary.

# Answer
> 2 votes
It *might* be possible, depending on your version control system.

In Git, for example, for a dir structure like this:

```
dir_in_question/
  .git/
  .gitignore
  builds/
  config.xml

```

I can prevent git from seeing any of the changes under the `builds` directory simply by adding inside `.gitignore` (which I prefer to add to the git as well) this line:

```
builds/

```

Another thing to try would be to make `builds` a symlink to a location outside the version control system, typically another filesystem. As a side effect you'd gain protection against builds filling up the jenkins' filesystem :)

---
Tags: jenkins
---


# Question
Title: How to run experimental Docker features on CircleCI
When a build is started on CircleCI one of the first things that is done is:

> Setup a remote Docker engine
>
> ```
> Specified reusable docker engine, but build has not been whitelisted.
> Contact CircleCI to be whitelisted
> Allocating a remote Docker Engine
> ...
> Remote Docker engine created. Using VM 'prealloc-wrjtu1qd-1491949826270'
> Created container accessible with:
>   DOCKER_TLS_VERIFY=1
>   DOCKER_HOST=tcp://<IP>:2376
>   DOCKER_CERT_PATH=/tmp/docker-certs615987123
>   DOCKER_MACHINE_NAME=51123
>
> ```

Subsequently when `docker build --squash -t imagename .` is run on CircleCI it results in:

```
Error response from daemon: squash is only supported with experimental mode
Exited with code 1

```

**Discussion**

It works to use experimental features locally after changing the `/etc/docker/daemon.json` as follows:

```
{
    "experimental": true
}

```

and restarting the docker systemctl service, but how to do that on CircleCI? It seems impossible as the build output indicates that a connection is made to a remote docker that reside on the CircleCI systems.

# Answer
> 5 votes
You are running into the limitations of managed services. Many managed services are great to start experimenting with ideas and can show a great value for the casual user. But they trade ease of use for flexibility and functionality, therefore as soon as we start to have *serious* needs and are ready to spent a significant amount of effort building up on these ideas, the first move we should do is to opt for a flexible a fully functional solution. In my judgement, you are hitting the line where you need to look for something else – and you will notice that you need to learn a different tool, because that's not free software and you cannot run it yourself.

Here we are speaking about Circle CI, but the following limitations are common to many managed services:

* Managed services put a hard contraint on software versions they allow us to use. These constraints may match what we need today, but does not need to at a later time. They may be already out of sync today. In any case this is often a show stopper, because being unable to use the same software versions in development environment and in managed environment is a useless source of bugs and difficulties of all sort.
* Managed services provide restricted access to their functionalities. This is the problem you are facing here: you want to run docker, but you do not have access to the daemon, only to a subset of its functions. These limitations are sometimes not well documented nor brought in full light by advertisement material. This just means work spent on understanding and integrating these features is worthless until we leave the managed service for another better solution.
* Managed services provide restricted access to the data we store there. So, to take the example of managed services for logging and monitoring, data access is slow, inconvenient and expansive. If we run our infrastructure, we are free to run any fancy custom analysis on our monitoring artefacts, because data is easier to reach.

Note that this is not a case against managed services. A hand of them are able to combine ease of use together with flexibility and functionality. But many of these service trade ease of use for flexibility and functionality: we can use them as an expedient to quickly build a simple functionality but they will prevent us to walk beyond that point. We need to be aware of this trade-off and consider.

# Answer
> 8 votes
CircleCI now allows you to use a machine executor which provisions a separate VM for you, with Ubuntu 14.04 and Docker version 17.06.0-ce installed. This allows you to turn on experimental features for the Docker daemon.

You need to use the `machine` key instead of the `docker` key, to run your job in a separate virtual machine instead of just a Docker container.

You can only choose 2 images for the machine:

* `circleci/classic:latest`: Ubuntu 14.04 with Docker 17.03.0-ce, or
* `circleci/classic:edge`: Ubuntu 14.04 with Docker 17.06.0-ce - the one with experimental features.

You'll also need to install dependencies on the machine yourself, as it is quite bare. For example, if you need PHP for your tests, you'll need to run `sudo apt-get install -y php5`.

Here is a sample *.circleci/config.yml* that builds a Docker image using experimental feature `docker build --squash`:

> .circleci/config.yml
>
> ```
> version: 2
> jobs:
>   build:
>     # Run in a separate virtual machine instead of a Docker container.
>     machine:
>       enabled: true
>       # Use Ubuntu 14.04 with bleeding edge Docker daemon 17.06.0-ce.
>       image: circleci/classic:edge
>     steps:
>       - checkout
>       - run:
>           command: |
>             # Restart Docker with experimental features on.
>             sudo sh -c 'echo '\''DOCKER_OPTS="--experimental=true"'\'' >> /etc/default/docker'
>             sudo service docker restart
>
>             # Install dependencies for tests etc.
>             sudo apt-get update
>             sudo apt-get install -y php5
>
>             # Build image with experimental feature --squash.
>             docker build --squash -t myuser/myimage .
>
>             # Login and push Docker image to registry.
>             docker login -u $DOCKER_USER -p $DOCKER_PASS
>             docker push myuser/myimage
>
> ```

---
Tags: docker, circleci-2.0
---


# Question
Title: Is Gradle used to automate deployment of EAR and WAR files to application servers?
I'm getting a bit deeper into Gradle and want to further automate the process. Generally speaking what are the approaches towards interacting with application servers?

I'd want to automate deploying an `EAR`. Is there even a need, or case to be made, to step out of Gradle for that purpose?

I imagine it's much the same regardless whether it's `EAR` or `WAR` files and regardless of the specific application server -- but I'm using Glassfish.

# Answer
> 2 votes
If you want just to deploy faster to server(propably not only Glassfish) you can just write code in gradle to do that. Example taken from here

```
/**
 *  ~/.gradle/gradle.properties:
 *  glassfishHome=/path/to/glassfish_home
 *
 *  or in Netbeans, right click project, Properties, Manage Build in Tasks, Run
 *  Add line to Arguments: -Dorg.gradle.project.glassfishHome=/path/to/glassfish_home
 *
 *  For more information about Exec tasks see
 *  http://www.gradle.org/docs/current/dsl/org.gradle.api.tasks.Exec.html
 */
task run(dependsOn: 'war', type:Exec) {
    workingDir "${glassfishHome}${File.separator}bin"

    if (System.properties['os.name'].toLowerCase().contains('windows')) {
        commandLine 'cmd', '/c', 'asadmin.bat'
    } else {
        commandLine "./asadmin"
    }

    args "deploy", "--force=true", "${war.archivePath}"
//from: https://www.schuermann.eu/2013/08/03/gradle-glassfish.html
}

```

In other words just use your asadmin

---
Tags: terminology, deployment, automation, toolchain, gradle
---


# Question
Title: Why do some components vanish from the catalog list if one component has been installed in Rancher?
If one adds a component from the catalog in Rancher and one would like to add another component than the list seems to be reduced. Why do some components vanish from the catalog list if one component has been installed in Rancher?

# Answer
You haven't specified anything concrete, so I can't give you a specific reason... But catalog items can:

* Be mutually exclusive (e.g. you can't use ipsec and vxlan at the same time)
* Be deployed at most once in an environment
* Require the environment be a specific orchestration type
* Require a certain range of Rancher versions
* etc
> 1 votes

---
Tags: rancher
---


# Question
Title: Where should I store my build configuration files?
At my work we use git for our source control, MSBuild for our builds, and Jenkins for our CI. Our git repo will have different branches for the cycle of the product (develop, QA, release).

My question is where should I store the msbuild xml? If I check it in alongside the product, say in develop and there's a feature branch, the file gets merged when a team does a merge from the parent. So that's not an option.

I also want the build config to stay with the version of the product. So the build.xml will go from develop-\>QA-\>release.

How is everyone else handling this?

# Answer
> 0 votes
I think you are trying to solve a different issue (environment) with how Git versioning / branching works

As suggested by @Dan you need to keep the build.xml alongside product code but at the same time the build.xml should not change for each environment (database=beta, prod ,qa) etc.

There are two ways to solve it,

a. Make your application dynamic to read system environment variables (like ENV=qa , ENV=beta or ENV=PROD and have different property files for each environment and keep the build.xml static (with just referring the variable)

But drawback in this approach is that each time you add a new environment a new property file gets added to your code

b. Have all the dependent resources (like database end point / credentials , host\_name , email server and credentials, . etc) everything into system environmental variables or service startup variable for e.g in JVM it is -D and maintain a separate repo for all the configurations and execute the same using a config management tool like Ansible/Salt , Chef. This way you decouple platform code form configuration / dependent resources and keep the code base rolling with new features without depending upon the environment.

# Answer
> 1 votes
Normally the build configuration needs to stay consistent with the product code, so it should be stored into the same git repo as the product code.

It's true, the file would get merged when a branch merge from the parent is performed.

But often it's a trivial (empty/fast forward) merge - if there were no changes to the msbuild xml file in the parent branch.

And if the merge is not a trivial one it means there were changes in the build process in the parent, which **need** to be taken into account in the child branch as well since it also picks up the code updates that go with those build changes. In other words the msbuild xml merge is **required** in such case to avoid build breakages.

---
Tags: jenkins, git, release-management, builds, release
---


title: Docker image registries

text:
# Question
Title: Docker image registries
We are using AWS **EC2 container services** and we are using Amazon container registry but the problem with container registry is that it's only available in one region although it's have URI but is **docker hub repository** is good to go with EC2 container services or using Amazon ECR. Thanks

# Answer
> 3 votes
It depends on several factors.

If your images can be public or must be private. If you can make a public image, docker hub could work.

With ECR you will have a fine grain control of your permissions, integrated with IAM

On the other hand, ECR it's only available on one region so if you pull an image from a different region you will be charged for that traffic and the latency could be a problem so if you need a multi-region registry yo should implement that mechanism by yourself.

In brief, I wouldn't use public registry for production applications. You'll have more control with a private registry like ECR or other 3rd party solutions.

---
Tags: docker, amazon-web-services, amazon-ecs
---


# Question
Title: What is recommended to document an IT technology stack, including their relationship to one another, in a graph database?
Working for a large company with over 500 IT staff and over a 1,000 servers, with each server running its own business applications, we have a tremendous information and co-ordination challenge in knowing which IT staff member to contact for which server. The problem of co-ordination is compounded with different IT staff being responsible for different layers of the IT stack. E.g., there are different teams who are responsible for hardware, virtualization, operating systems, application servers and the applications themselves.

Considering this challenge, within DevOps there is a requirement to define and document all the components that constitute the various technology stacks within an IT environment. Traditionally this might have been accomplished with a propriety CMDB solution.

I've investigated typical CMDB solutions such as BMC Atrium and others for this purpose, the problem however is that they stop at the level of documenting IT assets themselves, at a high level, as per the ITIL framework, but to do not address documentation of the IT Technology Stack in detail. I have also investigated tools such as Puppet, Ansible and Salt, but these tools focus more on software deployment and configuration, and not on the people co-ordination around software.

A workable solution for example, would define the various concepts, along with the key attributes important to these concepts:

* Hardware
* Virtualisation
* Operating Systems
* Application servers
* Applications

These concepts would then be associated with one another in terms of their relationships in order to form solutions. E.g., an application would depend on an application server, which would depend on an operating system, etc., Together this solution would be defined at the "Finance System". Having defined a system, all the metadata associated with these systems would be captured in order to facilitate co-ordination for each layer in the stack. I.e. the co-ordination of the technical support staff for each layer.

The purpose of such a solution would to do various queries against technology stacks, such as:

* To determine who supports which components
* Which components are out of date
* Which components need to be patched

With this in mind, what open source tools exist to define all the components of an IT technology stack, including their relationship to one another, in a graph database such as Neo4J?

# Answer
> 5 votes
Considering your first paragraph, the organization you're describing is a highly siloed org, which is exactly what a DevOps organization tend to avoid.

> Considering this challenge, within DevOps there is a requirement to define and document all the components that constitute the various technology stacks within an IT environment. Traditionally this might have been accomplished with a propriety CMDB solution.

What you're describing here could be ITIL, which is a management system requiring documentation and you mix it with the fact a DevOps team will usually define the underlying layers as code, as such it gets back to any development documentation with the caveats of Code is Documentation often seen in Scrum methodology for an agile methodology of development (quick and short iterations aiming at minimal working solution at end of the iteration)

Disclaimer for the rest of this answer: I know more of chef and inspec and that's why I take them as exemple here; but they are not the only tools existing on the market, I won't open a debate on them as the better one is the one you're more comfortable with.

As such the rest of the question is a little biased and I, personally, didn't encounter an organization documenting the layer relation you describe more than the infrastructure as code and configuration management system code documentation. (Again, this doesn't mean no-one does it, I just never heard of it).
To illustrate from my company in a chef environment, an application cookbook will declare its dependencies (tomcat, jboss, nginx/php and on which OS, needed mount points for some shared datas and its DB schema name mostly) and expose its services URIs to be consumed by chef for other applications configuration, this sound like defining your 'Finance System' and the documentation for it is on this application cookbook README, with some more files if really needed.

Configuration management systems usually have a central reporting place, "chef-server" for datas an "manage UI" for presentation in chef world "ansible tower" for ansible world to name two of them, but they usually aim more at giving an oversight of the overall managed system than graphing the dependencies.

That said, for chef, the chef-server also act as a CMDB you can query with various tools (it return JSON datas from HTTP requests), the inter applications dependencies can be expressed in various way and there's no 'out of the box' method, each company will have its own way to declare them in the system for configuration purposes and as such you may leverage this to build your graph, but that's on your side.

In an infrastructure as code point of view, the infrastructure needs would be kept with the application, it is still the application who know what it needs under as middle-ware, which OS, with which locale, what are other services dependencies and what services this application offer).

Last thing I can think of if you want to manage those dependencies for documentation only are tools like glpi which is mainly a CMDB and a ticketing system, it takes advantage of documenting assets and their relation to be able to tell what is impacted when you open a ticket saying an application is down. coupled with ng-inventory it allows to query system states and as such can fulfill your query for patch needs, but in my opinion this is an audit system task, like could do inspect integrated within a chef run for exemple, as the next phase would be to fix the outdated systems by updating/patching them.

---
Tags: automation, toolchain, dependency-management, open-source
---


# Question
Title: Managing Data bags in different environments
Hi I got stuck with one of the scenario, I have two environments Dev and Prod. I need to pass different values in dev and different vales in prod. So how can I manage it using one Data bag. this is my general Data bag in dev environment

I am using this data bag in Dev environment, so I have to pass different username, passwd and jdbc password for prod environment. How can I do this? I am planning to do in below manner

```
{
 "id": "some_data_bag_item",
 "production" : {
 "id":"Databag_item",
 "username": "username",
 "passwd": "****",
 "jdbc_password": "****"
}
 },
  "testing" : {
    "id":"Databag_item",
    "username": "username",
    "passwd": "****",
    "jdbc_password": "****
  }
 }

```

Is this is a right way? Do we need to add anything else in recipe? Do chef will find out which environment it is automatically?

# Answer
> 3 votes
https://www.digitalocean.com/community/tutorials/how-to-use-roles-and-environments-in-chef-to-control-server-configurations

> For instance, if a node is in the "production" environment, you could want to run a special recipe in your "nginx" cookbook to bring that server up to production policy requirements. You could also have a recipe in the nginx cookbook meant to configure special changes for testing servers.

Example

> ```
> name "web_server"
> description "A role to configure our front-line web servers"
> run_list "recipe[apt]", "recipe[nginx]"
> env_run_lists "production" => ["recipe[nginx::config_prod]"], "testing" => ["recipe[nginx::config_test]"]
>
> ```

---
Tags: chef
---


# Question
Title: What processes enable software vendor/supplier management in a DevOps way of working?
Within a traditional separate Development and Operations organisations, procedures and processes are put in place to ensure that relationships between software vendors and suppliers are maintained. These relationships exist so that when for example a vulnerability is discovered operations teams are able to identify the business impact and mitigate appropriately.

If that organisation was built around ITIL service management principals then they would likely be doing one or all of the following:

With the assumption that the Operations role doesn't exist as a separate entity within a DevOps enabled organisation, how is an organisation collectively or severally meant to maintain supplier/vendor relationships so that when a vulnerability is publicised the teams can respond by reaching out to the vendor to understand remediation and mitigation actions.

Conceivably we could:

* Ask all of the Scrum teams to own their vendor relationships and act accordingly, this seems scalable but at the cost of a high overhead across an organization.
* Maintain a central function responsible for these relationships, this central function may not have the full context the the Cross-functional teams have.
* Use a 3rd Party automated service to scan infrastructure and applications to maintain a dynamic inventory which can be assessed for known vulnerabilities, this however doesn't actually maintain a relationship with vendors and is only as good as it's database of known vulnerabilities.

What strategies do you use?

# Answer
Since you reference the handling of vulnerabilities specifically in your question, I'll speak to this regarding how we handle this within our organization and group of companies.

We have an Information Security department who are responsible to monitoring and detecting vulnerabilities. You could think of this team as being responsible to technical information security management. When vulnerabilities are detected using regular scan with tools such as Nessus or published US-CERT by the information security community, this team passes these vulnerabilities to each Information Security Officer within the group.

The Information Security Officer, who is responsible and accountable for risk management, is tasked at a high level with:

* Establishing Information Security Policy, Standards, Procedures and Guidelines
* Promoting Information Security Awareness
* Completing Information Security Assessments and Audits

Having received the information regarding vulnerabilities, the Information Security Officer evaluates the vulnerability, determines the criticality of the vulnerability and it's impact on the business applications and environment. This information is then given to the appropriate Development Manager.

Development Managers are responsible and accountable for supplier and vendor management. Business Applications are allocated by business domain to various Development Managers. The Development Managers work with either a vendor and/or their own Scrum team to resolve the vulnerabilities withing their domain. Should specialist advice be required, this is provided by the Information Security Officer.

Regarding your possible options.

* Scrum teams might well lack the organizational clout to negotiate and compel vendors to address vulnerabilities quickly. It's also unlikely that the scrum teams themselves would have been in a position to ensure that all the correct legal clauses have been put into the vendor and supplier contracts to ensure that there is an obligation to handle vulnerabilities within a given SLA.
* Maintaining a central function ought to work to a degree, however as you point out they would lack the full context. Three tiers of management, covering three different aspects will address this short coming.
* A 3rd Party automated service will be able to alert you to vulnerabilities, but it will not necessarily be able to correlate those results to a specific business application, and as a consequence a given vendor or supplier. You will still need individual people to manage this aspect.
> 2 votes

---
Tags: terminology, operating-model
---


# Question
Title: Ideal methods of handling server logs on AWS
We currently have some ruby applications hosted on EC2 servers on AWS, and every few weeks we see the disk space get completely consumed by the app log as well as the worker logs.

What are the best practices regarding log storage for a cloud hosted environment such as ours? We've considered log rotation, pushing the logs to CloudWatch or simply truncating log files.

# Answer
I like to use ELK, Elasticsearch, Logstash and Kibana with Beats.

Filebeat, that belongs to Beats series will forward the logs that read from a file to logstash.

In logstash you could tag, filter, parse and modify the log entries that are stored in Elasticsearch.

For visualizing you can use Kibana. You can create a great dashboards based on the logs with all the information that really gives value to your business

If you don't want to install agents, you can forward logs using rsyslog directly but with al beats series you could get more in deep information rather than simply logs.

This ELK suite could give you much control over your application but you must maintain it
> 5 votes

# Answer
Completely agree with RuBiCK's answer.

I handle logs of 3 servers. We keep only really important logs in the system. Rest are pushed instantly to elasticsearch database.

My view is that Using elasticsearch would be beneficial than log rotation as you can keep logs for as much time as you want. And also, you can easily make visualizations and dashboards which can help in faster analysis. It also provides full text search so you can search for any kind of error OR status in your logs instantly.

I use rsyslog to push logs to elasticsearch. It is really a fast and reliable way to send logs. Moreover, you don't need to add any agents in your system.

Here is link which can help you. https://sematext.com/blog/2013/07/01/recipe-rsyslog-elasticsearch-kibana/

Also, do check what is being logged. Some apps like Rabbitmq store a lot of info in logs.
> 3 votes

# Answer
Also check the content of the log files and ask "is this relevant logging"? Sometimes, complete stack traces are logged. If this is the case, what is causing this and how to suppress it? Some apps define several log levels. If the app is running without issues, just set the log level to normal. If there are issues, change it to debug for a short moment.

If finetuning the logging does not help, then introduce log rotation or keep the logs for only one month, but the first thing I would do is inspecting the log and see whether there is superfluous information that could be removed by modifying the code.
> 2 votes

---
Tags: amazon-web-services, amazon-ec2, logging
---


title: Is Docker right for my use case?

text:
# Question
Title: Is Docker right for my use case?
My company has a system we sell that consists of basically a mini-computer "Smartbox" that is running Ubuntu 12.04. This box runs a Django application plus a number of different upstart processes related to it. Not much else. We have thousands of these boxes out on the field. We manage the package dependencies, process registration, etc. through a deb package with varying degrees of success.

We need a way to efficiently and robustly push updates out to our users out in the field. We also need something that as we upgrade the OS (we are way overdue for a Ubuntu upgrade as you can tell) we can feel relatively secure about our packages "just working".

I don't know much about Docker, but when I first heard about our problem (I'm a new hire), Docker was my first thought. But the more I thought about it I felt like maybe it wasn't, as these boxes are ours we control the OS on it which is a big part of the value proposition of Docker, or so I understand. So if we KNOW our boxes will always be Ubuntu and we basically just have a Django app plus some processes to run, is Docker any better than a deb package?

TL;DR: Docker vs deb packages for a distributed appliance that will always run Ubuntu so platform independence is not that important.

# Answer
I'm not 100% sure I understand from the question, but it sounds like the Docker solution would be to go from having an (physical?) appliance with an OS and your app installed on it, to having an appliance with an OS and Docker on it, running a single container with your app in it. That doesn't obviate the need to update the OS in the host, and it adds a layer of complexity (and more updates to contend with, as you'll now have to keep Docker *and* the OS patched) with no readily-apparent benefit as far as the specific areas mentioned in the question are concerned.

However, if you're talking about going from a *virtual* appliance to a Docker container, that could potentially smooth things out for you, but it also adds Docker as a dependency for your product; you're shutting out anyone who isn't using Docker and doesn't want to add it to their stack just to use your product. You could continue to support those that don't/won't use Docker by continuing to ship the (now "legacy") virtual appliance as before, but now you've just doubled your workload because you have two distributions to support instead of one.
> 8 votes

# Answer
I've worked with Docker a long time. The platform independence is nice, but it's not what I consider most useful about Docker.

First and foremost, you get repeatability. You can create a Dockerfile, debug in a container on your developer machine, run tests on a continuous integration server, and then in your final product, and you know it will behave the same in all those environments. No forgetting a dependency that a developer had installed on their machine. Also, your developers don't have to use Ubuntu at their desk. Important to keep us Arch Linux users happy :-)

Second, for your upgrade scenario, you can have multiple versions pulled to a machine at once. If you do a `docker pull myapp:2.0` while `1.0` is running, you can swap to `2.0` extremely quickly. Much faster than doing a full OS upgrade would normally take. If you use an orchestrator with multiple instances of microservices, you can even do rolling upgrades that don't interrupt service at all.

If you use a microservices model, Docker also provides sandboxes that can limit the amount of damage attackers can do in the event of an exploit. Instead of gaining control over an entire machine, they are only gaining control over one container.

The main downside is you need the host OS and some sort of orchestration. There are lots of choices for that, but don't underestimate the amount of work it takes to evaluate one, put it in place, and maintain it.
> 6 votes

# Answer
By and large docker provides many advantages for both developers and operations personnel. I use docker for some of my applications and find it to be a very reliable and robust approach.

My issue with you adopting docker is that I do not hear you asking the right questions and you could potentially make your life more complicated without addressing your most important requirements.

The first question you should ask (you said that you were new) is how are updates to the OS & application handled now? Does the current methodology work for you (your company)? What works well? What could be improved? Can you do a physical configuration audit on your target machines in the field to verify that they have the correct OS patches, application & were there any unauthorized changes.

I love docker, but I would not jump on the docker bandwagon without first assessing where you are right now including what works well and what needs to be improved.
> 2 votes

# Answer
While there is some tiny overlapping region Docker and the Debian packaging systems essentially solve two very different problems:

* The Debian packaging system is built to install software on a host and upgrade it as easily as possible. It is able to handle complex dependency and constraint patterns between software components, like “software X version A requires software Y with version B or newer installed” or “software X should never be installed with software Z version C”.
* The Docker system is conceived to easily describe and deploy services, especially micro-services, possibly on several hosts – e.g. a Docker swarm or a Kubernetes cluster.

These two problems are essentially orthogonal, which means that given the deployment problem to solve, one can use one of them, both of them, or maybe even none of them, as part of the solution. When using both of them, the Debian package is used in the production of the Docker image, and your Dockerfile (the recipes used to prepare the Docker image describing the “virtualised system” ran in a container) would essentially register your Debian repository in the sources of the Debian packaging system and install your package.

With this in mind, it seems to me that what you are really looking for is to implement the immutable server pattern. The recent development in cloud technologies made possible to upgrade software not by using the classical software upgrade system from a software package system (such as the Debian packaging system) but rather by simply replacing the full server at once. (Some persons did this before this development by having three OS-s on a server, two used in alternance to run the appliance and a mini-OS dedicated to performing the appliance replacement. While not overly complex, this seems to have always remained a niche.) This technique can be of interest for you because if you are used to upgrade software on your server using the package manager, the final state of the server depends of the “upgrade history” of the server – especially if errors occur in the upgrade process. This heterogeneity is bad, because it makes production problems hard to reproduce and diagnose, and your mixed experience

> We have thousands of these boxes out on the field. We manage the package dependencies, process registration, etc. through a deb package with varying degrees of success.

could relate to this. The immutable server pattern wipes this source of errors by essentially destroying the notion of “upgrade history” from the problem.

Now there are various options to implement the immutable server pattern, two popular choices are to use Docker images, images or to use “master instance images” from your cloud provider (these are called AMIs in AWS and just Custom Images in Google Compute Engine). Your use-case forbids the use of cloud based techniques, I will therefore assume Docker images as the only eligible choice. (For the sake of completion, it is certainly possible to use other approaches, for instance using Virtual Box or similar virtualisation solution, as an alternative to Docker.)

When using the immutable server pattern technique, you introduce a new artefact (the Docker image) representing your server and this artefact can be tested as well, and it is very easy to obtain a setup replicating truthfully your production settings – aside from service load.

Now to consider the concrete problem you described, let's assume implementing the immutable server pattern with Docker is actually what you want. Since the Docker system and the Debian packaging system are complementary rather than mutually exclusive (cf. intro) we still have to address the question if you should you prepare a Debian package for your software.

The pertinence of using a Debian package to install your software (in the Docker image or on a host) lies in the complexity of the versioning problem you have to solve. If you run at the same time several versions of your software, occasionally need to downgrade, and have complex version requirements that you need to carefully document, having a Debian package is a must-be. Otherwise, this step can be skipped – but since you already put an effort to produce and deploy these packages, there is no real value into ditching your work. I would therefore suggest continue to produce your Debian packages.
> 1 votes

# Answer
I think it could be a good option (further tests are necessary)

You could provide a URL with all tags/version of the container you have made and the clients will read that URL to see if there is a new version of the container.

You can store personal files/settings on local and you will never lost that information in upgrades and you will ensure that what you have made and tested will work for everyone in the same way.

Even you could give the users the possibility to choose which version from the available they want to use (if you want to give that possibility).

It will be like ""only upgrade one package"", talking about only retrieve a new version of the container, much better that dealing with debian packages ;)
> 0 votes

# Answer
Docker sounds reasonable to me, as you could make and test changes to the container in house, and then depending on your release process, restart the containers always pulling :latest or something similar which would provide a tested upgrade.

Considerations you would need to deal with include data storage as containers don't retain changes on restart, so you would want a data volume. There are probably a lot more considerations you would have as well once you dig into it. The system that I am currently working with (all docker-based) has been in development for over a year now and we are still finding areas in which we need to make changes to the container, configurations, etc.
> -1 votes

---
Tags: docker
---


title: Testing Jenkins jobs and pipelines

text:
# Question
Title: Testing Jenkins jobs and pipelines
Currently, we have a fair number of Jenkins jobs and pipelines for builds, testing, deployments and other automated activities.

Every time we change or add a new job, we only test it manually - e. g. going over the "happy path" (when job is done with no errors), testing a couple of negative test cases when a job or pipeline fails - checking the error code and notifications.

This approach clearly is not reliable and does not scale well. How can we improve this process? Is there a place for test automation when it comes to checking how Jenkins jobs and pipelines work?

# Answer
I'm posting this here not because I endorse these solutions (in fact, I've never tried them), but just because they are a potential answer to your question:

You can start with JenkinsPipelineUnit, a unit testing framework for Pipeline scripts.

There is also a project called jenkinsfile-runner which executes your Jenkinsfile in a transient, headless Jenkins instance. Supposedly, this can be used for integration testing Jenkinsfiles and Pipeline shared libraries. However, as of summer 2018 there is no documentation on how to use this tool for integration testing, and I have been unable to find any examples of anyone using this tool "in the wild".

See also the related bug report on the Jenkins JIRA: "Test framework for Jenkinsfile".
> 10 votes

---
Tags: jenkins, testing, automation
---


# Question
Title: Accessing the committer's name as a Jenkinsfile environment variable
Using the `env.BRANCH_NAME` variable accessible in the Jenkinsfile outside of the `pipeline{}` block, I have set up a variable used to conditionally set the node used to build the git branch. I would like to also access the name of the developer who committed the change, but I do not know how to access it.

I need a variable assigned before the pipeline block to assign to `pipeline { agent { label selected_agent }}`. If I attempt to use a method call instead of a variable there, the Java parser complains:

> java.lang.ClassCastException: org.jenkinsci.plugins.pipeline.modeldefinition.agent.impl.Label.label expects class java.lang.String but received class org.jenkinsci.plugins.pipeline.modeldefinition.ClosureModelTranslator

I cannot run code inside the pipeline block before the agent block to define the variable, and the agent block has to be the first part of the pipeline declaration.

If I could run bash commands, I could access the author's name, and in fact I do use it at the end of the Jenkinsfile to pull the git committer's name for the status email, but if I attempt to use sh() before the pipeline block I get this error:

```
Required context class hudson.FilePath is missing
Perhaps you forgot to surround the code with a step that provides this, such as: node

```

I am not sure what else to try. Does anyone have a suggestion?

# Answer
I'm not entirely sure what you're trying to ask in your question, so I'm answering the question I believe you're trying to ask.

A build does not necessarily have a single commit author. It has a list of committers, which can be empty or can contain many committers. To create this list, you can use this snippet in a scripted Pipeline (note that you may need to disable the sandbox in order to use `.collect()`):

```
changeAuthors = currentBuild.changeSets.collect { set ->
  set.collect { entry -> entry.author.fullName }
}.flatten()

```

I use this with Git as my SCM; I'm not sure if it works with other SCMs.

See also the following questions on StackOverflow (some answers may be outdated):
> 4 votes

---
Tags: jenkins, jenkinsfile, environment-variables
---


# Question
Title: How to structure DevOps related code and configs in a code repository?
We've been growing as a company, our products are expanding and our DevOps related activities and efforts are growing as well - we've switched from Bamboo to a more flexible and configurable Jenkins, using deployment pipelines and other plugins; switched to Ansible and starting to use Docker here and there internally.

All of these things require some level of coding or configuration - Ansible scripts and configs, Jenkins groovy scripts, Dockerfiles and YAML configs.

For now, we've created a separate "ops" repository with high-level directories for `jenkins`, `ansible`, `docker` and `other` (which is a terrible name, but for now all "other" DevOps automation things are there).

Our approach does not feel right and might not scale, but what are the best practices and recommendations to keep DevOps-related code in a code repository or repositories?

# Answer
> 4 votes
The current organisation of the code and configuration you describe is structured by the technical solutions involved. This is a bad design that will add a lot of overhead in our maintenance activities and will add a lot of traps in our way as well. Instead, that organisation should be structured around the artefacts we are deploying.

The reason for this is that we want to consider artefacts (*e.g.* a docker image or a software package) as the objects of the following verbs:

* build
* test
* deploy

to consider a minimal set of automated tasks we want to perform. If we want to change something about how the test verb is implemented, it is easy to visit the folder corresponding to that artefact in the appropriate repository and then discover the jenkins-specific automation items that needs to be updated. Instead, if the automation recipes are structured around technical solutions, then we need to figure out of the blue that jenkins is involved in the test procedures and find there the artefact related automation items. In complex situations, the organisation around technical solutions makes updates very hard, because we have to know a priori all the technical solutions involved in some service to update them accordingly.

For instance a repository containing the code for a website and a micro-service “a” could have the following sub-directories dedicated to operations:

```
./ops/website
./ops/micro-service-a

```

each having three scripts called `build`, `test` and `deploy`. Now that the organisation of automation items has somehow been clarified, let's turn our attention to configuration.

The main conditions and requirements about the configuration organisation are set by the `deploy` verb when applied on a service-like artefact. The `deploy` verb should have the following parameters:

* the version of the artefact to deploy,
* the deployment target of the artefact, which describes the concrete environment where the deployed artefact will run (*e.g.* a cluster and endpoints it should talk to)
* the credentials it should use to connect to other endpoints (*e.g.* databases)
* the runtime configuration of (like how long cache entries should live, etc.)

From the operational perspective, this breakdown of the parametrisation matches the natural degrees of freedom of the deployment problem – aside from the credentials that could be bundled with the runtime configuration, but it is better to separate them to avoid spreading them carelessly.

# Answer
> 6 votes
I can answer bout docker, one of the best practice to use docker is to keep the docker file and the compose files in the same repository of the project, so wherever you clone the project you can build the docker image, and it is good to keep multiple versions of docker compose files for example (prod, staging, dev) so you can build the image and run the container with specific option for each env for example for dev machine you can use specific network and run more dependencies container or whatever.

# Answer
> 4 votes
Each tool's code goes into its own repo. for e.g

1. Jenkins Groovy template into a Jenkins repo
2. Ansible YAML playbooks in its own repo (with the roles , tasks , inventory sub directories
3. Cloudformation / Terrform templates in its own repo
4. Docker files in its own 5.. And So on

This would help you scale better in terms of process orchestration and maintaining various branches for each environment

This would give you more granular control and offload all your versioning overhead to the version control systems. Also create separate branches for each environment and tag the code for every production release (as we do for application code base). Think Infra and process in terms of code. (Any change in process has to be codified and sent to QA, SIT , UAT and then to PROD) similar to application.

For e.g you may have V2.1 of Ansible running in Production ( master branch) but V2.0 of docker containers running in Prod (master branch)

Similarly keep your DB scripts / bash scripts in their own repositories and maybe you can have a healthcheck file (JSON/YAML) configured to show the versions of all the tools / parts in each deployed URL for tracking and automating purposes. (So that your webhooks reads the URL and automates deployments)

# Answer
> 3 votes
Making a distinction between Ops, Dev and DevOps promotes isolation and enforces a "throw it over the wall" mindset. To increase cooperation between teams one should put everything in a repository that is required to build and deploy the project.

Having said that, the answer to the question:

> How to structure DevOps related code and configs in a code repository?

is that if config is required to run the project then one should put it in the same directory.

---
Tags: jenkins, docker, ansible, git, repository
---


# Question
Title: monitoring/alerting system which can support very arbitrary data source and logic
I am looking for a monitoring/alerting tool/system which can support very arbitrary data source and logic.

(following is a made up example to more easily demonstrate my needs)

I have a database where I can find out the query execution times for my clients(which are individual companies). The query times are expected to vary greatly between clients, because of the data of each company. I want to have an alert when the query time for a client is ?% over some median value from the past, and where I am comparing Monday's data against previous Mondays' data(I am being difficult for demonstration purpose).

I want to avoid defining a separate alert for each company client.

I want to avoid defining a separate alert for each kind of query I want to monitor.

I want to avoid having each source pre-push data to one central place.

I would like to monitor some other things that has nothing to do with databases, and want to apply some entirely different logic for my alerts - e.g. some REST api call to some other system, or some ELK or time series data, or some hardware monitoring data.

I am contemplating a custom solution, but want to see if there is already a solution out there that can meet some or more of my needs. I have seen/used some monitoring solutions out there such as Nagios, though not an expert on those. Will appreciate any pointers or recommendation.

# Answer
Consider using something like nagios or Zabbix. While it might be nice to avoid custom queries and alerts, this is impossible. The odds of someone using your specific suite of tools in your configuration are slim-to-none - and the odds of having pre-canned monitoring for that suite in that specific configuration are even slimmer. Simply put, there is no way to avoid this custom monitoring. While monitoring, alerting and reporting management and configuration can be tedious, there is no easy shortcut for doing it properly and it's probably going to take more time, dedication and resources to do it properly than you think.

Out of the 4 monitoring systems I have used extensively, I must recommend nagios - it seems to be the best suited for configuration management systems and most compatable with DevOps philosophy.

It may sometimes be useful to use a system such as elastalert or similar systems as middleware for alerting/monitoring/reporting, but you will have to judge each situation/monitor on a case-by-case basis, but you will find these tools (and most monitoring systems) have some SDK, API or tool capable of helping you with your monitoring customization. Just choose the easiest to configure system with the lowest ROI and support cost and be sure to stick with it and follow your monitoring migration project to completion. Be sure to consider dedicated resources/personnel for this task if possible.
> 6 votes

# Answer
Adding my brick to the list here, I'm at a point where updating continuously the monitoring system to add new instances and cleanup old ones is a pain.

As such I've turn toward prometeus which works with a simple contract, an exporter does the collection job (could be the cloudwatch\_exporter on AWS or when using netdata querying it (either a central one or each separately, but the latter lost the advantage of auto-discovery) (doc for the netdata part here) and there's also a node exporter from prometeus team.

The main advantage behind prometheus is the use of mathematics functions and grouping in the alert definition by label from the metric name. This allow to define alerts on various way, for my infrastructure monitoring I work with standard deviation or standard variance to avoid fixed level of alert on cpu usage, it alerts me if the stddev goes up by 20%.

So for your case you could export your database or push the metrics to prometeus and assuming a metric name `client_queries` with the client name as the label `client` you could do something along the line of this (untested of course and just to illustrate):

```
ALERT ClientQueryDriftUp
  IF avg(client_queries[5m]) > avg(client_queries[5m] offset 1w) * 1.2
  ANNOTATIONS {
    summary = "{{ $labels.client }} query has gone up over 20% of last week",
    description = "{{ $labels.client}} query average is high ! (current average value: {{ $value }}s)",
  }

```

Or if you just want an alert if the difference between averages is over 1 second (assuming values are in milliseconds) you can use this IF condition:

```
abs(avg(client_queries[5m]) - avg(client_queries[5m] offset 1w)) > 1000

```

Of course this answer is just an overview on how it can tackle your problem and is far from exhaustive about it.
> 5 votes

# Answer
I would suggest elastalert as an alert system for elasticsearch, it has amazing capabilities, you can execute any query on elasticsearch and define an alarm depending on the response and it support multi ways for alert like sending email or slack or telegram, even execute specific commands.

For monitor data in elasticsearch as you are already using ELK, you can use Kibana for monitoring not only for making elasticsearch query and create very valuable Dashboards

For hardware and services I think Nagios do very good job there, but for more easy solution you can use Monit it's very easy to use and configure and can do a lot of jobs and service management, But the same as you are already using ELK, you can configure metricbeat to monitor the server (CPU, RAM, Disk usage, etc...)

For me I think the best thing is to use ELK with all the integration that you need, that will allow you to more focus on, and centralize your monitoring and alerting systems
> 3 votes

# Answer
We use Datadog's anomaly detection for this purpose:

https://www.datadoghq.com/blog/introducing-anomaly-detection-datadog/

We push custom metrics which in your case would be something like query execution time tagged by client name, and you can create a single Multi monitor that would alert based on anomalous behavior on this metric.
> 2 votes

---
Tags: monitoring
---


title: OMD vs Ubuntu install issues

text:
# Question
Title: OMD vs Ubuntu install issues
When installing OMD 1.3 on Ubuntu, I get the following output:

```
dpkg: dependency problems prevent configuration of omd-1.31.20160708:
omd-1.31.20160708 depends on libapache2-mod-proxy-html; however: Package libapache2-mod-proxy-html is not installed.
omd-1.31.20160708 depends on apache2-mpm-prefork; however: Package apache2-mpm-prefork is not installed.
omd-1.31.20160708 depends on libboost-program-options1.54.0; however: Package libboost-program-options1.54.0 is not installed.
omd-1.31.20160708 depends on libboost-system1.54.0; however: Package libboost-system1.54.0 is not installed.
omd-1.31.20160708 depends on libperl5.18; however: Package libperl5.18 is not installed.
omd-1.31.20160708 depends on php5-cli; however: Package php5-cli is not installed.
omd-1.31.20160708 depends on php5-cgi; however: Package php5-cgi is not installed.
omd-1.31.20160708 depends on php5-gd; however: Package php5-gd is not installed.
omd-1.31.20160708 depends on php5-mcrypt; however: Package php5-mcrypt is not installed.
omd-1.31.20160708 depends on php5-sqlite; howeve
dpkg: error processing package omd-1.31.20160708 (--install): dependency problems - leaving unconfigured

```

I have updated and upgraded my system quite a few times and still issue persists and has confirmed that the dependencies exist.

Does anybody know if the above packages and their versions are required? I have the dependencies installed, but on the latest version and I am assuming that OMD and the packages that comes with it requires the older version.

Any advise or solutions to this issue?

System:Ubuntu 16.04.3 LTS OMD Version: 1.31.20160708

# Answer
According to How to let `dpkg -i` install dependencies for me? it sounds like `dpkg` doesn't have support for automatically installing a package's dependencies.

Personally I'd switch to a package manager which has such support, like `apt-get`. You'll like it ;)

Alternatively you could:

* use a "helper" application around `dpkg` as suggested in the above-mentioned answer
* manually collect the entire list of missing dependencies and install them as well. But it can be very tedious.

> I have the dependencies installed, but on the latest version

The version is part of the dependency, so technically the above means you don't actually have the dependencies installed. At least not for that particular version of OMG.

> and assuming that OMD and the packages that comes with it requires the older version?

Bingo. You have 2 options:

* re-create an older version of your system which is compatible with your particular version of OMD. It could be based on Docker or some VM technology (or some plain old hw you may have lying around). I won't consider downgrading your system a solution :)
* look for a newer version of OMD, hopefully one that's happy with what you have installed and on which its dependencies can be installed (some packages suport multiple versions installed on the same system). Also note from omdistro.org, you may be looking for another package:

> With the growing popularity of OMD, we came to a point were it makes sense to develop two specialized versions of OMD.
>
> * Check\_MK Raw: Check\_MK Raw Edition focuses on Check\_MK. Use this if you mainly use Check\_MK. This is a smaller package including all components to run Check\_MK.
> * OMD-Labs: OMD-Labs contains all components of the original omd and some more addons like Grafana and Influxdb as well as Naemon and Icinga 2.
> 2 votes

---
Tags: monitoring, nagios
---


# Question
Title: Application configuration of a complex enterprise system
For a complex, distributed Enterprise system running on premise, I'll be driving an initiative to harness application configuration management. Today we have software delivered by almost 20 various scrum teams with only some coordination, having central and embedded software in Java, .NET, C++ and SQL worlds (MSSQL+Oracle). Creating a consistent system configuration addressing project specific functional requirements every time becomes a painful project.

What I would like to introduce is a type of configuration meta model starting from the business layer (domain of customers, product managers, etc) going down to individual parameters (domain of developers an system engineers). I imagine something like a web car configurator which in the end generates a consistent configuration. The whole complexity of relations between functionalities (hundreds) and parameters (tens of thousands) would become explicitly maintained in such a tool.

Now, are there any off the shelve tools/frameworks supporting something like this? Gut feeling tells me that adapting any existing solution to a very specific needs of a large enterprise system might actually be more complex than building it by ourselves.

# Answer
I highly recommend Consul for configuration management as it is a high reliability, distributed system that provides quite a few and easy to use features with a simple Rest API. Also the community edition is free :), although for a large enterprises there is an enterprise offering.

I am a particular fan of the Key Value store, because it allows me to have separate configurations for my micro-services and back-end services (written in C#/.Net Core and Java respectively) in all three environments (Dev, Beta, Prod).

Whenever I add a new service or application, the configuration for each environment is added to the Key Value store and a simple config client downloads the configuration during the application's bootstrap/loading routines.

You can read a bit more here but in a nutshell:

> Consul has multiple components, but as a whole, it is a tool for discovering and configuring services in your infrastructure. It provides several key features:
>
> Service Discovery: Clients of Consul can provide a service, such as api or mysql, and other clients can use Consul to discover providers of a given service. Using either DNS or HTTP, applications can easily find the services they depend upon.
>
> Health Checking: Consul clients can provide any number of health checks, either associated with a given service ("is the webserver returning 200 OK"), or with the local node ("is memory utilization below 90%"). This information can be used by an operator to monitor cluster health, and it is used by the service discovery components to route traffic away from unhealthy hosts.
>
> KV Store: Applications can make use of Consul's hierarchical key/value store for any number of purposes, including dynamic configuration, feature flagging, coordination, leader election, and more. The simple HTTP API makes it easy to use.
>
> Multi Datacenter: Consul supports multiple datacenters out of the box. This means users of Consul do not have to worry about building additional layers of abstraction to grow to multiple regions.
>
> Consul is designed to be friendly to both the DevOps community and application developers, making it perfect for modern, elastic infrastructures.
> 2 votes

# Answer
Consider using something like Puppet, Salt Stack or Chef - all of which can be installed on Linux or Windows. You should then make it the responsibility of the respective development teams for managing a module for their application - just like they should be managing their installers.

The configuration code for the management system can then be managed in git or some other version control system just like your existing code. Upgrades are then as simple as checking out the latest module version, or a specific commit ID or tag into the module directory.

For certain things (eg, time source) you can use an NTP module that applies to all hosts instead of just a specific host. This will allow your project managers, customers, ops team, whatever to manage some things at the organizational level/domain and some developers or teams to manage some things at the individual level/domain.

This can even work for your networking team who can use these configuration management system to manage firewalls and switches using SDN (Software Defined Networking) technology.
> 1 votes

---
Tags: configuration-management
---


title: Storing Meta Data for Apps/tools

text:
# Question
Title: Storing Meta Data for Apps/tools
Have created a number of images via packer and soon coming to the conclusion that binaries/images soon rack up in qty and need to find away to store config data.

We are going to use ansible to hold config data for apps and was wondering about how people manage versions of binaries, versions of images, ingress egress data etc etc. I wondered if anyone knew of either a regularly used/industry approach to managing app data, versioning, config data etc. My default thought was a scheema to use in ansible, or perhaps a tool that can manage this data.

A combo of packer images, binaries and other images I am beginning to think that these files could increase in numbers v v quickly. How do people manage this info??

# Answer
Problems with golden images are that you have to store a full system image with overhead of data which is not build-specific, and using conventions for configuration to make deploy process the same for all versions.

TL;DR: Infrastructure as a Code, version-specific storage for data outside the app package.

The problem is easily solved by storing config files and templates, data and installation scripts (including ansible playbooks to manage) inside both code and packages/images, so install and initial configuration would always be version-specific (switching to the same commit you are deploying to have configs you need). This could be done with VCS and current package management software (deb, msi, PKG etc. and scripts for unattended deploy and installation) with ease. To avoid using different images of OS and other environment, storing and using a single *golden image* for all builds is a standard practice: it provides consistency and you can automate CI/CD pipeline for image update process.

If some data is stored outside (*i.e. a lot of current Windows apps outside of Store are installed by running a downloader, then installing app*) and you have to keep large number of them (no specific branches only like stable-beta-alpha), then storage should be versioned too (i.e. using version-specific hashes as names) and links in install scripts or artifacts would point to the one version they need. But you also have to keep the register of pairs `build:data version`, so when there's no need to use specific version, deletion of related data could be performed within the same remove process. Also, you have to decide whether you want to store complete images or use some differential systems (incremental or differential snapshots of files/filesystems), latter is much more complex and hard to manage so it's not used widely AFAIK (with an exception for some DBs, i.e. Facebook stores MySQLs snapshots and binary transaction logs so they could deploy or restore exact state up to certain transaction id).
> 3 votes

---
Tags: ansible, build-pipeline, packer
---


# Question
Title: Jenkins deployment key is not associated with the repository
Using jenkins to pull from bitbucket repository, I got this error:

> git@bitbucket.org:source/source.git +refs/heads/*:refs/remotes/origin/*" returned status code 128: stdout: stderr: repository access denied. deployment key is not associated with the requested repository. fatal: Could not read from remote repository.

I have added ssh keys in the bitbucket repo, I have run ssh -v git@bitbucket.org and this is the output:

> authenticated via a deploy key.
>
> You can use git or hg to connect to Bitbucket. Shell access is disabled.
>
> This deploy key has read access to the following repositories:
>
> source/source: root.staging\_quc -- root@QUICKRESPONSE debug1: channel 0: free: client-session, nchannels 1 Connection to bitbucket.org closed. Transferred: sent 2968, received 1872 bytes, in 0.0 seconds Bytes per second: sent 82471.7, received 52017.2 debug1: Exit status 0

I don't really know what to do, hope you guys have any clue on this one I will update this question

# Answer
Hello guys I have found the solution, So I create ssh keys from my **Jenkins Server**, and then called the private key from my code, this is how it looks like on my project structure:

```
root@staging:~/project# tree
.
├── deployment.yaml
├── server
│   └── host
└── the_private_key

1 directory, 3 files

```

and my deployment.yaml:

```
---
- hosts: staging
  become_user: sudo
  tasks:
  - name: populate my project
    git: repo=git@bitbucket.org:source/source.git
         version=master
         dest=/var/www/test/
         key_file=the_private_key
         accept_hostkey=yes

```

And then I copied **Jenkins Server** public key to **Target Server**, then everything worked well
> 1 votes

# Answer
First make sure that you added the public key to the right repo with the right permission, usually I add it to 'Access keys' so the user has read only permission, then to make sure that is working add the private key to you system so you can use it with using git:

```
ssh-add private-key
git clone <your git repo>

```

If it's working, add those credential to you jenkins and use them later
> 1 votes

---
Tags: jenkins, bitbucket
---


# Question
Title: What is the difference between DevOps engineer Vs DevOps consultant Vs DevOps architect?
What is the difference between DevOps engineer Vs DevOps consultant Vs DevOps architect?

Considering Role wise & Salary wise & Seniority wise & job demand wise, etc.

# Answer
An ***architect*** is somebody who thinks and creates designs.

A ***consultant*** is somebody who gets hired for things like "Teach us how to do so and so", or to answer questions like "What's your opinion about this design?".

An ***engineer*** is the one who is objective oriented and pragmatic, something along the lines like "*How can I get that design (created by the architect, and validated by the consultant) implemented and make it work"?*.

Their *salaries* may vary (there are cheap and expensive ones), same for *seniority* (there are juniors and seniors, while everybody starts as junior, not all of them ever make it to senior). And *job demand* is pretty variable (by region, over time, economical factors, depending on hypes, etc).

BTW, it doesn't really matter what the actual "subject" is, it could be "IT", it could be "Road Construction", or it could even be "DevOps" ...

PS: What's in a name (or jobtitle), how about a ***DevOps BA***, as shown in this post ...
> 16 votes

# Answer
You might want to check the accepted answer to this question: Why shouldn't I try to hire a 'DevOps Engineer'?

Or maybe the article "7 DevOps roles you need to succeed", which includes these (critical?) roles:

1. DevOps evangelist
2. Release manager
3. Automation architect
4. Software developer/tester
5. Experience assurance (XA) professional
6. Security engineer
7. Utility technology player
> 4 votes

---
Tags: terminology
---


# Question
Title: Does a company need DevOps if it already delivers software at acceptable speed?
I mean, DevOps is, from the business view, about speed and quality, right?

What does it mean if a company is growing and fast enough for their customers without devops? Does one of the following apply?

1. They did, or do software development at sufficient level, no change is required.
2. They do DevOps but don't know it/gave it another name.
3. They do something totally different and better than DevOps as we know it and we really should find out what it is (before it is too late).
4. They are awfully wrong: their speed and quality are below their knowledge and they will know it as of next financial report (in case they manage to survive long enough to see it).

# Answer
> 11 votes
DevOps isn't just a *different* way of doing things, it is a *better* way of doing things. Perhaps it will help me to make my point if I rephrase the question in a different context - the context of the industrial age, the railroad and the steam hammer:

> Does a railroad company need Steam Hammers if it already lays track at an acceptable speed?
>
> ---

> I mean, steam hammers from the business view are about speed and quality, right?
>
> What does it mean for a construction company contracting for the railroad that is laying track "fast enough" for their customers without steam hammers? Which of these might apply?
>
> 1. They lay track at a sufficient level, no change is required
> 2. They use steam hammers but don't know it or call the steam hammer something else
> 3. They use something totally different and better than a steam hammer and we really should find out what it is (before it is too late)
> 4. They are awfully wrong: their speed and quality are below their knowledge and they will know it when they begin to lose bids to other contractors.

The answer is obviously probably #4. This hypothetical company may not see the problem with the next financial report - perhaps it will take a few years - but eventually it will become clear how and why competitors outperform businesses still manually "laying track" as it were. Sure, it's possible they have come up with a diesel powered hammer that is more efficient and less prone to malfunction or breakdowns, but not likely.

Thinking that they can outperform the steam hammer is going to get this company left behind. Sure, an individual group may have their steam hammer blow up and fall apart and perhaps that one group has such trouble with their steam hammer that it would have been more efficient in *that one case* to do things the old fashioned way, but they are the exception to the rule and the problem likely was that their steam hammer was poorly built and/or poorly maintained.

The industrial age is coming and the railroad company can either check out the competition and then try to get their own steam hammers, or get left behind and eventually succumb to their irrelevance and antiquity.

DevOps is no different. It's just another iteration of automation that companies can either embrace or ignore at their own peril much like the steam hammer.

---
Tags: culture
---


# Question
Title: error: exportArchive: The operation couldn’t be completed. (IDEDistributionErrorDomain error 3.)
I am using Jenkins automation build for iOS.

Previously I have been using xcode 8.1 with no error showing. But i have updated the xcode to 8.3 and I'm facing errors now. I have the scripts as below:

OLD SCRIPT

```
xcodebuild -exportArchive -archivePath Sample.xcarchive -exportPath Sample -exportFormat ipa -exportProvisioningProfile 'Sampledemo'

xcodebuild: error: invalid option '-exportFormat'

```

NEW SCRIPT

```
xcodebuild -exportArchive -exportOptionsPlist exportOptions.plist -archivePath Sample.xcarchive -exportPath Sample  PROVISIONING_PROFILE_SPECIFIER="Sampledemo"

error: exportArchive: The operation couldn’t be completed. (IDEDistributionErrorDomain error 3.)

Error Domain=IDEDistributionErrorDomain Code=3 "(null)" UserInfo={IDEDistributionErrorSigningIdentityToItemToUnderlyingErrorKey={

```

but I'm facing this issue now. How can I solve the issue ?

# Answer
> 1 votes
https://forums.developer.apple.com/thread/75038

> Hi!
>
> I think I've got it figured out...
>
> instead of:
>
> ```
> -exportProvisioningProfile "MyProvisioningProfile"
>
> ```
>
> use:
>
> ```
> PROVISIONING_PROFILE_SPECIFIER="MyProvisioningProfile"
>
> ```
>
> We've got unity building the project every time and it doesn't fill in the PROVISIONING\_PROFILE or PROVISIONING\_PROFILE\_SPECIFIER. And it looks like it will figure out the alphabet soup that belongs to PROVISIONING\_PROFILE by looking it up itself provided you have downloaded it already.

---
Tags: jenkins, continuous-integration, continuous-delivery
---


# Question
Title: How to reconcile Infrastructure as Code and not storing env config in code?
When developing a Web App, and trying to adhere to what is generally considered best practice, e.g. The 12 factor app methodology etc. One key concept is to keep configuration and sensitive data out of your source code - access keys, connection strings etc. Using something like `.env` files to set different connection strings for a dev, test and production environment.

However, another key point (not sure is is mentioned in the 12 factor app), seems to be Infrastructure as Code. I completely see the benefit of this, however, I don't see how it can be reconciled with not storing/hard coding sensative data. Surely to entirely automate the provisioning of a new environment, a series of env variables has to be specified - in code...

How can these two seemingly contradictory pieces of standard practices be reconciled?

My current approach is something like this:

1. Hosted Git Repo with however many dev branches and a master branch
2. When dev branches are merged with master branch, the CI build is triggered, this builds the project and runs unit and integration tests.
3. The git repo also has a production branch, when master is merged into production, the same build runs, but if it is successful, a CD release is created. (In my case this is a deployment to an Azure App Service)

So my three environments are the local dev machines (before a PR into the master branch in the hosted repo), the CI build server and the production app service. At the moment, locally we have a `.env` file which isn't checked into the git repo. On the CI server, the `NODE_ENV` value is set to test, so the code uses hard-coded unit test values (nothing sensitive). Then in production we manually set the connection strings etc. in the app service settings. This, I feel, is potentially an issue.

So possibly I could add an infrastructure folder to the repo, with Terraform code in it, then as part of the release process, terraform plan could see if there is anything that needs changing between the code's infrastrucutre and the real world infrastructure (and if so, execute it). But where do I store the production env variables?

# Answer
> 5 votes
There are, as always, a few ways to solve this.

You can use a central source to keep secrets that each server reads from ala Hashicorp Vault. While popular this is not my preferred approach as its rather complex. There are quite a few key value stores that can provide similar functionality such as AWS Parameter Store.

You can manually put data in these stores and/or store it encrypted in git. Using tools such as kops (AWS only) or ejson (provider agnostic) you can have the secrets in the same repo as the app. Then have the app either fetch all the secrets at deploy time or just the decryption key so it can load the encrypted file.

# Answer
> 2 votes
This is a common problem in configuration management, and while I can't speak for Chef as I haven't worked with it, I can tell you how this is solved by both Puppet and SaltStack.

Firstly, this distinction becomes a little less important with interpreted languages since there isn't much difference between including this sensitive data in a configuration file and including it in a text file that is interpreted (and I'm not aware of any compiled configuration management systems)

But secondly, both Puppet and SaltStack, so you can simply make the equivalent of a .env file and include it just as you do with your applications.

Typically this is not the recommended approach however. Instead, SaltStack recommends you use Pillars for managing this data, which are stored separately from the code (called "States"). This pillar data is uniquely calculated for each minion and sent directly from the server only to the minion only using standard SSL encryption - so one minion cannot see another's pillar data. In fact, minions can't edit pillar data and they *definitely* can't rewrite pillar data *on the server* \- this data is unicast only.

Similarly, Puppet utilizes Heira for achieving something very similar. This allows you to store your code and metadata separately. You will then reference the pillar in your salt state or heira value in your manifest using a variable/function call thereby allowing your code to populate these values at runtime.

---
Tags: continuous-integration, continuous-delivery, terraform, environments, infrastructure-as-code
---