# Question
Title: Automation and the .NET framework which tools to use?
I am aware of the de facto and popular tool choices for various programming languages such as Go, NodeJS, Java, Python etc. However, I do not know what toolchain is reasonable or even (hot) in the .NET world. I have heard that people use Octapus deploy, is this still a valid choice? Is NuGet still the defacto package manager? What about code inspection, automatic QA etc.?

I would like to get a feeling of the complete toolchain for .NET and what is popular right now when considering development, automatic test, and delivery.

# Answer
Somewhat stealing from Ian Margett's answer as the architecture is common amongst most Microsoft/.NET development organizations, the high-level target operating model for looks something like this:

The goal is to create a Continuous Deployment pipeline, using existing off-the-shelf software, namely TeamCity, ProGet, SonarQube and Octopus Deploy:

* **GitHub** is the source code management tool, however, it could be BitBucket or Visual Studio Team Services. The branching model and code review process are out of scope at this high level.
* **TeamCity** is chosen as the Build System due to its tight integration with Octopus Deploy and good all-round support for .NET, msbuild and PowerShell. TeamCity is also used as the orchestrator of deployments in Octopus Deploy.
* **ProGet** is the package management solution which both stores Octopus Packages and proxies public package/image repositories. The rationale for not using the built-in TeamCity NuGet store is purely for scalability reasons.
* **SonarQube** provides continuous code quality management and reports are published as part of the TeamCity build outputs.
* **Octopus Deploy** is used as the deployment tool for both infrastructure and code into the target platforms.

I've seen this broad approach implemented across two companies and successfully implemented it in two additional companies - in the most recent case we swapped TeamCity out for AppVeyor which worked, albeit a little painful when setting up firewall rules.
> 11 votes

# Answer
You mention a few different categories in your toolchain for .NET. Yes, NuGet is still the default package style – and a lot of people use a Universal Package Manager to manage their NuGet feeds.

For Deployment, Octopus is indeed an option for pushing out artifacts, but it doesn’t enable some of the other aspects you were talking about.

An ARA tool would probably be a better fit it does more than just Deployment automation and ARA tools are more "Hot" in the DevOps world right now - especially with evolving things like WinOps.

As far as other tools, take a look at the DevOps toolchain wiki page and the WinOps focused tools section.

\*Full disclosure I work at Inedo, and we make a solution for both of these options (with .NET in mind)
> 6 votes

# Answer
Experience at my place has been using Octopus Deploy, and Proget - had great success building a good pipeline and its scaling well. Also plays nicely with unit testing and automated functional test tools. We are predominantly in Azure on .Net but also deploying to private cloud and on premise
> 4 votes

---
Tags: automation, toolchain, dotnet
---


# Question
Title: How to capture tools release to trigger upgrade?
We capture our applications release by using commits from source code management, new package into the artifactory system etc.,

How can we capture my tools release from vendor so that I will trigger my upgrade script to upgrade my tools itself, for ex: upgrading Jenkins whenever new release available from Jenkins itself?

List of tools am expecting to upgrade:

```
1. Jenkins
2. GitHub Enterprise
3. Atlassian Jira
4. Atlassian Bamboo
5. Atlassian BitBucket Server

```

Basically, continuous deployment(with proper pipeline including testing) to production without human interaction for these tools.

Note: I am looking for some example, my requirement is for more than 10 tools. So for Jenkins, it will be Linux and Ubuntu.

# Answer
> 3 votes
For tools which don't offer access to a source code management repository that you can automatically query for updates you could start with a manual trigger, then take a closer look for how is that trigger activated by humans and maybe automate the task:

* there could be upgrade email notifications sent to a subscriber list. An automatic email processing entity could be subscribed to get these messages and distil them into an automatic trigger
* the tool itself may performs such checks, maybe displaying messages in its log(s). In such case an automated tool could be parsing the log and derive an automatic trigger from it. If the upgrade availability is displayed in a GUI it's may be more difficult, but not impossible, to automate that as well
* the tool provider may (or could be persuaded to) offer a programmatic way to query for or be notified about upgrades - API, webhook, service, etc

I'd suggest keeping the human monitoring and ability to activate those triggers in parallel with the automated system - just in case there are unexpected changes rendering the automatic triggers ineffective. Such monitoring could be more relaxed (say once a week/month), just as a precautionary safety net.

# Answer
> 3 votes
From your list of tools:

Jenkins and Github have apt/yum repositories you can use to install and upgrade.

For Atlassian products, best up to date information about repositories I can found is This ShipIt experiment and it seems unlikely to change from this forum post
The workaround may be parsing the mailing list to update the deploy.

Now if you really want something automated to handle the installation and configuration of your tools, you can try a Configuration Management System like Chef/Puppet/Ansible/Salt.

For Chef I know better, there's cookbooks for Jira and Confluence maintained.
For the rest you'll be on your own writing the bits to deploy and configure.

This means having 4 CI/CD systems (at least), two 'groups' validating each other, as a pipeline won't be able to validate and upgrade itself properly (inception). That sounds overkill in my opinion.

At the end of the day, there's no silver bullet for all and every software you may use, each will have its own particularity and way to be installed and you'll have to workaround those bits.

---
Tags: jenkins, continuous-integration, continuous-delivery, continuous-deployment, upgrade
---


# Question
Title: How to include a custom Slack message only on completion?
We're running Jenkins with the Slack Notification Plugin. Is there a way to send the custom message only on completion, not on start? To be clear, I **don't** want to disable the build started message, I'd just like the custom text to only show on completion (success).

Use case is jobs like delete stack. We use the custom message to make it clear which stack the job deleted, but you end up with slack messages like (paraphrased)

```
Delete job started
  Deleted Alex stack

Delete job finished
  Deleted Alex stack

```

Obviously you can see where this causes confusion. I'd rather get

```
Delete job started

Delete job finished
  Deleted Alex stack

```

"It can't be done" is an acceptable answer if that's the case. I know I can do a pipelines workaround but I'd love a quick fix for the jobs we haven't converted yet.

# Answer
> 7 votes
As far as I was able to tell it's not possible.

We solved this by using Pipeline putting the custom messages in the Jenkinsfile.

The flexibility gained by having the Jenkinsfile in the git repo really helped our developers have a better understanding of the build process, and allowed them to make necessary changes to the build commands without the communications headache.

It's a nice excuse to move over to Pipeline.

It also happens to be very easy to do:

`slackSend (color: colorCode, message: customMessage)`

---
Tags: jenkins, slack
---


# Question
Title: What are the main differences between Jenkins and TeamCity if one is used to working with Jenkins?
These tools seem to share very similar characteristics.

How complicated would it be to start using TeamCity after getting used to working on Jenkins? Are there specific concepts one needs to be aware of?

# Answer
**TeamCity :**

It does look nicer, if this is important for your team then it should definitively weight in. That said, if it is VERY important than you probably will end up creating tools or some sort of dashboard overlay to support your team at which point what you really want is the one with the best API. Have not tried Jenkins API so I cannot compare, the TC API though should get you what you need.

Their support is pretty good, the guys respond relatively fast and are courteous. However that does not mean you will get what you want. If you use the system in an unconventional way you can very well get your bug put on the shelves... happened to us. At which point it becomes quite frustrating to use, you are faced with a black box leaving little alternative than to work around it. At this point things can get hackish and ugly.

It is generally pretty quick to get done what you want and the log interaction quasi API is a very nice feature if you do a lot of custom scripting in your pipeline.

**Jenkins:**

Battle tested and Extensive.

But it is somewhat less pretty, would not go as far as saying it is ugly though, one could say that functionality comes before appearance.

I'm pretty sure you can find private paid support plan by third party companies if you look around. If this is important for your shop don't just block at the "OpenSource" part of the deal, the community is quite expansive.

Lots, I mean LOTS of plugins. Again don't just just limit yourself to the official channels, lot more plugins can be found in github and other places.

I found both to be about equally fast to get started with, though with Jenkins you might need to be more dynamic with plugins than with TeamCity. So if you have a tight IT department and wont get admin access to the server this could pose a problem. Compounded by their much faster release cycle than TeamCity (weekly).

I found that Jenkins does support more release cycle paradigms than TeamCity. Might be easier to find a process template out of the box that is a closer match to what you have in mind. I say this with reserves as I have not dealt with TeamCity for 2 years now.

I personally prefer Jenkins mostly because I am biased towards Open Source for such tools and in a lesser degree because I found it's structure and config mechanism agree more with me.

YMMV
> 8 votes

# Answer
On the whole, the user experience is pretty similar. TeamCity has a prettier UI, but isn't particularly easier to use. In terms of functionality, the two are effectively equivalent. Most of the terminology is the same as well.

The plugin ecosystems are fairly different, however; you'll definitely want to look at what plugins are available for TeamCity to achieve what you're trying to do, as this will likely be the biggest pain point in terms of a transition. If you're used to running certain Jenkins plugins, you'll need to learn a) what functionality is provided by TeamCity without any plugins needed, and b) what plugins are available to add any remaining functionality, and how they differ from plugins you're used to in Jenkins.
> 6 votes

# Answer
I agree with Adrian on most points. TeamCity's UI is definitely prettier and you get a lot more built-in functionality out of the box with TeamCity than with Jenkins. But Jenkins is open source and while the quality (and doc) varies a lot from plugin to plugin, the ecosystem is extensive.

I've been using Jenkins for years and just started using TeamCity recently. For example, setting up dependent jobs is much simpler and more intuitive in Jenkins than it is in TeamCity.
> 4 votes

---
Tags: jenkins, continuous-integration, teamcity
---


# Question
Title: Where is the apache2 filebeat module configured?
I am setting up the Elastic Filebeat beat for the first time. I have followed the guide here, and have got the Apache2 filebeat module up and running, it's connected to my Elastic and the dashboards have arrived in Kibana.

Currently it's using the default path to read the Apache log files, but I want to point it to a different directory.

There is a setting `var.paths` documented here for this purpose, but I can't see where this setting is applied in the configuration for Filebeat.

I've looked through the Yaml files in the installation and can see the Apache2 module default config, but it doesn't look like I should modify that.

I'm guessing I should add something to the filebeat.yml, but there doesn't seem to be a placeholder for "modules.apache" or similar.

I'm trialling on MacOSX, will be deploying on Ubuntu, but I don't believe this to be a platform-specific question.

# Answer
> 3 votes
After looking a bit further, I found it in the sample called `filebeat.full.yml`

* So I added this section to the filebeat.yml
* Restarted filebeat

After running at 100% CPU for minute, Kibana is now immediately showing the results. That was easier than I expected.

```
#==========================  Modules configuration ============================
filebeat.modules:

#------------------------------- Apache2 Module ------------------------------
- module: apache2
  # Access logs
  access:
    #enabled: true

    # Set custom paths for the log files. If left empty,
    # Filebeat will choose the paths depending on your OS.
    var.paths: ["MY-CUSTOM-PATH"]

```

---
Tags: configuration, logstash
---


# Question
Title: How to implement blue-green deployment with one master branch
I was told that they easy way out is to have two production branches, master/blue and master/green, git merge action triggers CI to deploy to the matching stacks. Is that possible to do it with single master branch?

Some more details about our stacks.

* CloudFront, origin set to S3 bucket which serves all the front-end SPA (single page app) static assets.
* SPA app talks to AWS API Gateway, which connects AWS Lambda.
* AWS Lambda is responsible for db CRUD operations with DynamoDB.
* Front-end app and Lambda function each has their own Git repo, Git master branch merge action triggers 2 Circle CI jobs, one copy front-end assets to S3 bucket, the other deploys Lambda with serverless framework.

For blue-green deploy, I'm thinking to have 2 instances of S3, API gateway and Lambda function. To make the blue-green switch, we'll need to change origin from one S3 bucket to another, then create invalidations to clear the cache.

To make the single production git branch blue-green deployment to work, Circle CI needs to know is the current active stack is blue or green, copy & paste / deploy to the other stack, if Circle CI is not able to do that, the alternative that I could think of is to somehow put the checkout\_current\_active\_stack action in the git precommit hook? To be specific:

1. (in the git precommit hook) send out a HTTP request to the current active stack API (AWS lambda function, which keep track of the current stack information)
2. deploy base on the return value ('blue' or 'green')

# Answer
> 5 votes
It is worth separating your Continuous Integration "Build" from your Continuous Deployment "Pipeline". For the build portion, CircleCI seems to be your tool of choice, for the latter I recommend using something like AWS CodeDeploy which natively supports Blue-Green Deployments, for completeness you can use CircleCI to orchestrate AWS CodeDeploy:

Separating your pipeline phases has several benefits:

1. You only need to compile/test once from the `master` branch; then promote the packages through the Artefact Repository.
2. Your packages are stored forever in an Artefact Repository, for example Sonartype Nexus or JFrog Artifactory, this means you can reproduce an old build as you have the specific packages used for deployment.
3. CodeDeploy will handle the release patterns for you without having to think too much about how to achieve Blue-Green Deployments.

It is also worth stating that you can model the whole pipeline in Application Release Automation tools such as BuildMaster or Nolio. Personally, I prefer to build my pipelines from multiple 3rd party SaaS solutions.

---
Tags: amazon-web-services, circleci, git-hooks, bluegreen
---


# Question
Title: What are best practices for using Ansible Vault on public CI's and Source Control Repositories like BitBucket?
**Introduction**

On a private CI's and Source Control Repositories like Gitlab-ce it is possible to copy an ~/.vault\_pass.txt to the server and let it use by the CI to decrypt files using Ansible.

**Problem**

On public CI's and Source Control Repositories like Bitbucket it is not possible to copy a ~/.vault\_pass.txt to the CI server it self.

**Discussion**

In Bitbucket it is possible to define encrypted variables, but when this file is checked the only VAULT related variables are:

* `ANSIBLE_ASK_VAULT_PASS`
* `ANSIBLE_VAULT_PASSWORD_FILE`

These variables are not an option to solve the issue as when the `ANSIBLE_ASK_VAULT_PASS` is set `ansible-vault` still prompts:

```
user@host $
Vault password:

```

When the same password is entered it can open the encrypted file, but the aim is to open the file without needing a file or entering a password in a prompt.

Another attempt to solve the issue was running `export ANSIBLE_ASK_VAULT_PASS=<ansible-vault-password>`, but the interactive mode persists.

Another option is `export ANSIBLE_VAULT_PASSWORD_FILE=~/.vault_pass.txt`, but then this file needs to be pushed to the repository, but source Control repositories should not contain passwords.

# Answer
`--vault-password-file` can instead point to an executable script that writes to stdout. This little-known feature should solve your problem.

First, write a simple, executable script that prints a system environment variable, and check that into your source control. Then, use Bitbucket's encrypted variable feature to set that environment variable to your ansible-vault secret. Finally, execute like so:

`ansible-playbook site.yml --vault-password-file ./mypass.sh`.

References:

1. http://docs.ansible.com/ansible/playbooks\_vault.html#running-a-playbook-with-vault
2. https://groups.google.com/forum/#!topic/ansible-devel/1vFc3y6Ogto
> 13 votes

# Answer
Using

```
ansible-playbook site.yml --vault-password-file ./mypass.sh

```

resulted in:

```
ERROR! Problem running vault password script / p a t h / t o
/ e c h o _ v a u l t _ p a s s . s h ([Errno 8] Exec format error). If this is
not a script, remove the executable bit from the file.

```

Based on this post the following was defined in bitbucket-pipelines:

```
image: docker:latest

pipelines:
  default:
    - step:
        script:
          - echo $ANSIBLE_VAULT_PASSWORD > .vault_password.txt
          - ansible-playbook -i ansible/inventory ansible/site.yml --vault-password-file .vault_password.txt

```
> 3 votes

# Answer
You could have a test set up that does not run production, and load different files for those.

Create a host\_vars/localhost/vault that only works for the local test installation.

That way you could use a open vault password that works only for that local host vault.
> -1 votes

---
Tags: ansible-vault, bitbucket
---


# Question
Title: Git & Jenkins: get latest green commit on branch
We're just starting to push for CI-CD and as a baby step we're going to try updating a stack with the latest green develop once every couple of hours. I am fairly new to Git/Bitbucket, and can't figure out how to ensure the checkout that Jenkins makes gets the last commit to have been marked green by Jenkins, rather than just "the last commit" as a blanket statement.

We have the Bitbucket Build Status Notifier plugin installed, so Bitbucket does track which commits are green after our unit tests run. Is there a way to leverage this info to make sure the right commit is picked?

# Answer
> 7 votes
You don't make any mention of the scripting language you want to use, so I will talk specifically about the HTTP requests to the BitBucket API:

## Assumptions

If you have a BitBucket Repository that has three commits in in it the first and the last are failing the build, the middle is passing:

* 4768815 ❌
* 49d7110 ✅
* 42d357f ❌

## Get the list of commits

You can get the list of commits by calling the following API method:

> `https://api.bitbucket.org/2.0/repositories/{{owner}}/{{repo_slug}}/commits`
>
> * `owner`: RichardSlater
> * `repo_slug`: greencommitproofofconcept

The response looks like this:

```
{
  "pagelen": 30,
  "values": [
    {
      "hash": "4768815fdc27abf4be17096e7c460f7f68f5d39b",
      "repository": { ... },
      "links": {
        ...
        "statuses": {
          "href": "https://api.bitbucket.org/2.0/repositories/RichardSlater/greencommitproofofconcept/commit/4768815fdc27abf4be17096e7c460f7f68f5d39b/statuses"
        }
      },
      "author": { ... },
      "parents": [ ... ],
      "date": "2017-04-10T11:38:18+00:00",
      "message": "README.md edited online with Bitbucket",
      "type": "commit"
    },
    {
      "hash": "49d7110b98616358d16055960a4abdf2926b890d",
      ...
    },
    {
      "hash": "42d357f1df7a7d7bcf1f10a9f3a5a40d85d5b11c",
      ...
    }
  ]
}

```

If you parse the JSON and loop over the responses you can extract out the statuses from:

```
values[n].links.statuses.href

```

Where `n` is the index, i.e. `0`, `1` or `2` in the above example. If you were to construct this from scratch it would be in the following format.

## Get the list of statuses from the commit

> `https://api.bitbucket.org/2.0/repositories/{{owner}}/{{repo_slug}}/commit/{{sha}}/statuses"`
>
> * `owner`: RichardSlater
> * `repo_slug`: greencommitproofofconcept
> * `sha`: 4768815fdc27abf4be17096e7c460f7f68f5d39b

**Note:** this is a Hypermedia API which means the urls *could* change so I would recommend using the links from the previous response rather than trying to generate them from scratch.

The response from the above HTTP request will be something like:

```
{
  "pagelen": 10,
  "values": [
    {
      "key": "POC-01",
      "name": "Build #1",
      "repository": { ... },
      "url": "http://devops.stackexchange.com/q/809/397",
      "links": { ... },
      "refname": null,
      "state": "FAILED",
      "created_on": "2017-04-10T13:04:28.261734+00:00",
      "updated_on": "2017-04-10T13:04:28.261759+00:00",
      "type": "build",
      "description": "Changes by Richard Slater"
    }
  ],
  "page": 1,
  "size": 1
}

```

From this response you can extract the `state` using:

```
values[n].state

```

Again where `n` is the `status` \- there could be many of them if one commit resulted in many builds.

If the state for the build you care about is `SUCCESSFUL` then you have your answer and you can immediately return the `sha` for the commit.

Loop over all of the commits from the first phase, if you run out of commits follow the `next` page `link` that is included in the call to `/commits`.

## Complete Flow Diagram

At a high level the flow will look like this:

Don't forget this is a Hypermedia API so wherever possible have your code follow the links in the API rather than trying to "guess them".

# Answer
> 3 votes
In a typical Continuous Delivery/Deployment pipeline you would have the following happen:

1. Developer pushes one or more commits, or, a pull request is merged.
2. Jenkins automatically builds and executes tests.
3. If successful Jenkins publishes a deployment package to an Artefact Repository; if failure publishes nothing and notify developers.
4. Deployment automation uses packages from the Artefact Repository and deploys them.

The goal is to avoid **building** the solution from source twice, you build it once and deploy it many times. You can implement approvals in Sonartype Nexus to define the environment approval process, i.e. Dev → Test → UAT → Stage → Production.

**That said...** if you have read all of the previous and still want to get the latest green build from source control then you can use one of two techniques:

1. Have Jenkins tag the branch with a suitably named tag, i.e. `master-green` then use that instead of `master` when you want the latest green build.
2. Use the BitBucket commits to get a list of commits and commit/{sha}/statuses over each one to find the commit with a Green status. I have expanded on this solution in another answer.

Feel free to post a follow-up question if you would like the specific detail on how to use the above approaches.

---
Tags: jenkins, git, bitbucket, bcbsn
---


# Question
Title: Is it possible to host an ECS cluster with p2 or other GPU instance types?
It seems from the GUI that it is not possible (the GPU instance types are simply not listed when you go to create an ECS cluster and look at the list of instance types). However, there is an official AWS blog article here indicating that Amazon itself uses ECS in this way.

I see that there is a an ECS CLI command register-container-instance that could theoretically be used like so:

1. Manually create an EC2 instance with a GPU type.
2. Use `register-container-instance` to register your GPU EC2 instance to an ECS cluster.

However, the documentation notes that this command is supposed to be internal use only, and also it is a bit of a hack and a lot of extra work.

# Answer
You can use custom instances of any type with ECS -- the key is running the ECS container agent on the instance of your choice. So, in general, I followed this documentation on that process with success -- http://docs.aws.amazon.com/AmazonECS/latest/developerguide/ecs-agent-install.html -- but do note that you probably want to also specify the ECS cluster into which the instance goes (otherwise it will go to "default" cluster).

To specify the cluster you specify env var ECS\_CLUSTER, typically in `/etc/ecs/ecs.config` prior to running the container agent. See here.
> 3 votes

---
Tags: amazon-web-services
---


# Question
Title: Where does CircleCI-2.0 store the ssh private keys?
**Aim**

To deploy docker container using ansible using CircleCI-2.0.

**Method**

A private key has been created and added using the CircleCI-2.0 UI.

*.circle/config.yml*

```
version: 2
jobs:
  build:
    docker:
      - image: ubuntu:xenial
    working_directory: ~/project

```

**Results**

Running ansible results in:

```
Failed to connect to the host via ssh: Warning: Permanently added
Permission denied (publickey,password)

```

**Discussion**

The assumption was made that the keys would have been stored in ~/.ssh, but this seems to be empty.

> Is your target host accepting this key (when tried manually)?

Yes. Log in using `ssh root@host -i key` succeeds.

> Are you using the same user in the container?

Yes. The `root` user is used both manually as from the CircleCI-2.0.

**Conclusion**

In summary, the question is where does CircleCI-2.0 store the private keys that have been submitted using the UI? The `~/.ssh` directory is empty.

# Answer
> 5 votes
I still don't get how your `config.yml` could launch ansible without a step part. Here is my findings, but I'm unsure it is what you're after, but that was too long for a comment.

According to the documentation here you have to add a `add_ssh_keys` with the fingerprint of your key as seen in the UI.

```
- add_ssh_keys:
    fingerprints:
      - "48:a0:87:54:ca:75:32:12:c6:9e:a2:77:a4:7a:08:a4"

```

---
Tags: circleci-2.0
---


title: Upgrading jenkins using chef

text:
# Question
Title: Upgrading jenkins using chef
Can you please help me out how to upgrade an existing application which is jenkins using chef. I am trying to upgrade Jenkins from 1.x to 2.x and I have a cookbook for 1.x. what steps do I need to take to upgrade 1.x version to 2.x along with plugins. Environment: Rhel and using rpm package to install jenkins

# Answer
> 3 votes
So first of all, Chef is just a manner of upgrading/configuring, it will help you keep a reproducible state and keeping configuration in line but it won't do black magic and translate your jobs for you.

Now for Jenkins 1 to Jenkins 2, Jenkins-ci site says Jenkins 2 is Backward Compatible.
While this is true, when you update the pipeline plugin, not all plugins are compatible yet. There's a compatibility list on the pipeline plugin github page.

I would recommend installing a fresh Jenkins 2 and porting your jobs from Jenkins 1 to Jenkins 2. This will spot breaking plugins in Jenkins 2, allow you to strip up not needed anymore plugins and refactor your jobs.
This way you can also take advantage of defining the pipelines with a Jenkinsfile along with the code of your application.

---
Tags: jenkins, continuous-integration, chef
---


# Question
Title: How to define failure situation on jenkins
I created a project in Jenkins and set it to run tests on every pushes. but even when one of the pushes does not pass the tests, Jenkins build returns success. How can define failure situation if tests do not pass in Jenkins?

# Answer
> 6 votes
By default Jenkins assumes whether the build is `SUCCESS` or not from build process exit code.

Which means that build exit with `code 0` considered as SUCCESS, rest all are considered as FAILURE.

If your build step/process exit with `code 0`, even during the failure case then Jenkins will report the build is `SUCCESS`.

So I my suggestion is to run the all build steps manually and try to get the exit code on each step (for both PASS and FAIL case).

---
Tags: jenkins, continuous-integration, continuous-delivery
---


title: Terraform apply exit code on error

text:
# Question
Title: Terraform apply exit code on error
I'm using Terraform as part of a Jenkins Pipeline. When `terraform apply` encounters an error, it does not return a non-zero exit code. When running into a problem on Jenkins, it does not stop the pipeline or mark the build as FAILURE.

If terraform encounters an error, I need the job to stop and not progress.

Looking at the docs for `apply`, it doesn't look like I can provide a flag to return helpful exit codes. The docs for `plan` has a `-detailed-exitcode` flag, but doesn't look like that'll help me.

# Answer
> 3 votes
Turns out this was not a real issue. My apologies. At the end of that shell script, it did an echo which is what was returning the non-zero status code. After I took that out, it worked as expected.

---
Tags: jenkins, terraform
---


# Question
Title: How to let build fail when docker image size exceeds certain threshold?
`docker images` returns:

```
REPOSITORY            TAG            IMAGE ID       CREATED             SIZE
<none>                <none>         fcc125757e66   About an hour ago   270 MB

```

**Attempts**

`docker images | awk '{print $6}'` returns:

```
SIZE
ago
hour
ago
ago
ago

```

**Expected outcome**

`270` so this could be used in an if statement:

```
if [ VAR > 250 ] then
   echo "Image too large"
   exit 1
fi

```

# Answer
> 7 votes
The output of `docker images` is really intended for humans, and it's not very parse-friendly. Instead, most Docker commands support a `--format` flag using Go templates. As you can see in the `images` documentation, you only really care about the `.Size` value, so you really want this:

```
docker images --format "{{.Size}}"

```

This just gives an output like this:

```
90.6 MB

```

If you have multiple images, they will each be on a separate line. I'm going to assume you will only ever have one line, because it vastly simplifies the next steps.

To get rid of the `MB`, we can simply use `sed`:

```
docker images --format "{{.Size}}" | sed "s/MB//g"
# => 90.6

```

But, you probably **don't** want this for your bash script. Most shells don't support floating point comparisons, so inputting 90.6 would just fail. Instead, let's use `bc` to truncate the number first:

```
docker images --format "{{.Size}}" | sed "s/MB/\/1/g" | bc
# => 90

```

Note that I've replaced the `MB` with a division by 1, which truncates in `bc`.

You can then pipe it into a shell script. I tested with this script:

```
VAR=$(cat)

if [ $VAR -gt "250" ]
    then
    echo "Image too large"
    exit 1
fi

```

Then run with:

```
docker images --format "{{.Size}}" | sed "s/MB/\/1/g" | bc | ./check_size.sh
# Exits with 0 (for my example)

```

---
Tags: docker, continuous-integration
---


title: How does Cacti compare to Nagios?

text:
# Question
Title: How does Cacti compare to Nagios?
So far I have been using the Nagios monitoring system. However, I'm not familiar with Cacti. How does Cacti compare to Nagios?

# Answer
While Romeo has laid out the basic facts I'd like to add a bit of experience to the answer:

* **Basics** Cacti is focused on graphs. It doesn't do any of the up/down monitoring that is provided by something like Nagios. Classically Nagios does not provide any useful graphs, but more on that later.
* **Config Scaling** Cacti tends to be configured through the GUI manually. Eventually you will hopefully scale enough to find this frustrating. I've switched to Munin in places where there's lots of configuration management and it is easy to turn a server inventory into a config file with templates. Nagios tends to be configured with config files so hopefully you script the translation of your inventory in that format too. Then you can scale to thousands of nodes without redesigning your monitoring system.
* **Distinctions blur.** The classic distinction between Cacti/Nagios of graphing vs up-down status has been blurred in a number of Nagios derivatives. Icinga and OMD, for instance, integrate graphing into a Nagios-style monitoring architecture.
* **Plugin Availability** Doing something in the Nagios world gives you access to tons of well tested plugins to monitor almost anything under the sun. Cacti also has plugins, but not nearly as many.

So obviously my Cacti experience hasn't been that great and I'd recommend you look elsewhere for your long-term graphing needs.
> 6 votes

# Answer
Nagios is generic monitoring solution which can be extended by using snmp agents, custom plugins and so on.

> Cacti is a complete frontend to RRDTool, it stores all of the necessary information to create graphs and populate them with data in a MySQL database

(from here)

Cacti can be used as graph solution in Nagios to represent in graphic manner historical data (memory usage, processor usage, network bandwidth and so on). Cacti of course can be used also as standalone system and generate graphical presentation of data, collected from logs, via scripts and so on.
> 5 votes

---
Tags: nagios, cacti, network-monitoring, comparison
---


# Question
Title: How to create a centralised dashboard to monitor multiple cloud (Amazon and Google)?
I'm looking for a single product or service which can be used as a centralized dashboard to monitor cloud infrastructure provide by Amazon Web Services and Google Cloud Platform in addition to the application.

I'm looking at few options like sensu which support all of the services in Amazon Web Services, but I didn't find any plugins to monitor Google Cloud Platform.

Are there any other services or products that allow me to create this, my preference is Open Source?

Note: Everyone has access to GCP and AWS and if anyone launches something it will be easy to have one dashboard for monitoring.

# Answer
I think what you are looking for is an Open Source project that can take inputs from both Amazon CloudWatch and Google StackDriver; there isn't a huge amount out there at the moment, but I will detail what I know.

I have made the assumption that you know how to import your application telemetry into the solutions below.

## Open Source

Open source solutions to this problem have been lagging behind as late due to favourable prices of SaaS and built-in solutions. The products below are constantly evolving so if they don't do what you need now they may be in the future.

* **Elastic Stack**, formerly known as the ELK stack is made up of three individual products from Elastic.co: ElasticSearch, LogStash and Kibana. It is possible to ingest data from CloudWatch with a plugin and to export StackDriver data to Google Pub/Sub and then ingest into LogStash.
* **Grafana**, is slightly better than Kibana at displaying time-series data. However, dosn't have as robust an ecosystem around it. Grafana has built in support for Amazon CloudWatch, Google StackDriver is not supported as yet. However, someone is working on it in the form of a plugin.

## Software as a Service (SaaS)

Software as a Service monitoring solutions is a multi-billion dollar market, as large enterprises will readily pay for a SaaS or off-the-shelf solution to their dashboarding challenge.

## Single Pane of Glass

In the last three years, various products have gone to market providing a "Single Pane of Glass". The advantages of these products are:

* **Heterogeneous Platforms** \- you can use many different monitoring tools and products and then unify them with a Single Pane of Glass.
* **Data Normalization** \- most monitoring tools have idiosyncrasies in the way they present data a Single Pane of Glass helps to normalise these so that metrics from Google, Amazon and your App make sense when displayed in the same graph.
* **Business View** \- a Single Pane of Glass usually has the capability to integrate with non-technical data sources. As an example displaying data from financial tools to show a measure of business success. This view makes for some compelling C-Level dashboards that demonstrate how technology is providing value to the firm.
> 14 votes

---
Tags: amazon-web-services, monitoring, google-cloud-platform
---


# Question
Title: How to use tags in Amazon Web Services and Azure
I am currently building a solution on Azure (though the question should relate to AWS equally) and I am using a `UniqueId` tag to locate and use objects.

For instance, we have a storage account that is used for caching applications' blobs. The storage account is created with a random name but is found and presented to the application through its `UniqueId=AppCache` tag.

At the moment I am taking the approach that as more tags become useful I'll start to add them.

Ultimately we're looking for a hands off 'ChatOps' style solution. So everything created should be automated.

Should I be looking at a structured approach to tagging (and what does that look like) or is the organic approach more sensible?

# Answer
I would set up a base taxonomy to start with, you can use automation across your estate to ensure that all resources follow the taxonomy. You have a choice between simply deleting resources that don't comply or reporting on it to a central team who are responsible for chasing down and educating the owners.

I have followed the Best Practice set out by Microsoft for both Azure and Amazon:

* **Owner** \- normally this is a team distribution list although sometimes it is an individual if it is a "10% time" project. To make sure that the e-mail address entered is valid we run a hourly scan of our estate to detect if there are new e-mail addresses in this field then send an e-mail with an activation link, failing to activate results in deletion after three days.
* **CostCenter** \- the numeric cost centre, we use this to feed into the end of month reconciliation.
* **SDLCPhase** \- one of `Development`, `Test`, `SIT`, `UAT`, `PerfTest`, `PreProduction`, `Production`. We use this to define different automated behavour:
    + **Development** and **Test**: deleted overnight and at weekends to save on costs. Teams are responsible for reprovisioning using the automation in the morning.
    + **SIT**, **UAT** and **PerfTest**: deleted when the cost center in our finance system is closed down.
    + **PreProduction**: scaled down overnight and when the cost center is closed down.
    + **Production**: we validate that all resources deployed to the production subscription have this tag, if they don't we send out e-mails extensively.
* **DropDeadDate** \- for us, this is a required field for everything except Production. We run a nightly scan and send e-mails to the owner 7 days and 1 day before the **DropDeadDate** then delete the resource after it has passed.

In terms of the automation around this I have a tendency to use Janitor Monkey and some custom bots written is Azure Functions that sit in Slack Channels and do my bidding.
> 9 votes

---
Tags: amazon-web-services, azure
---


# Question
Title: How do I detach an existing volume and attach a new volume with Terraform?
Is there a way to detach EBS volume from existing EC2 instance and attach a new EBS volume using `Terraform v0.9.2` ?

I prefer to replace the volume rather than trying to destroy the instance and create a new one with new volume.

File system is unmounted from the device within the operating system before detaching the volume and also there is no issue to detach the volume from AWS console.

```
resource "aws_ebs_volume" "create_volume" {
  availability_zone = "eu-central-1a"
  snapshot_id = "${data.aws_ebs_snapshot.mysql.id}"
  type = "gp2"
  tags {
    Name = "${var.instance_name}"
  }
}

resource "aws_volume_attachment" "mysql_data" {
  depends_on = ["null_resource.stop_mysql_service2"]
  device_name = "/dev/xvdf"
  volume_id = "${aws_ebs_volume.create_volume.0.id}"
  instance_id = "i-0d48be4266da"
  skip_destroy = true
  force_detach = true
}

$ ./terraform apply
data.aws_ebs_snapshot.mysql: Refreshing state...
aws_ebs_volume.create_volume: Creating...
  availability_zone: "" => "eu-central-1a"
  encrypted:         "" => "<computed>"
  iops:              "" => "<computed>"
  kms_key_id:        "" => "<computed>"
  size:              "" => "<computed>"
  snapshot_id:       "" => "snap-0afb2303c60f"
  tags.%:            "" => "1"
  tags.Name:         "" => "mysql"
  type:              "" => "gp2"
aws_ebs_volume.create_volume: Still creating... (10s elapsed)
aws_ebs_volume.create_volume: Creation complete (ID: vol-04c7f4)
null_resource.stop_mysql_service2: Creating...
null_resource.stop_mysql_service2: Provisioning with 'remote-exec'...
null_resource.stop_mysql_service2 (remote-exec): Connecting to remote host via SSH...
null_resource.stop_mysql_service2 (remote-exec):   Host: mysql
null_resource.stop_mysql_service2 (remote-exec):   User: ubuntu
null_resource.stop_mysql_service2 (remote-exec):   Password: false
null_resource.stop_mysql_service2 (remote-exec):   Private key: true
null_resource.stop_mysql_service2 (remote-exec):   SSH Agent: true
null_resource.stop_mysql_service2 (remote-exec): Connected!
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2: Still creating... (10s elapsed)
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec): .
null_resource.stop_mysql_service2 (remote-exec):  * Attempt to shutdown MySQL Community Server 5.6.35 timed out
null_resource.stop_mysql_service2: Still creating... (20s elapsed)
null_resource.stop_mysql_service2: Still creating... (30s elapsed)
null_resource.stop_mysql_service2: Creation complete (ID: 1850529311883)
aws_volume_attachment.mysql_data: Creating...
  device_name:  "" => "/dev/xvdf"
  force_detach: "" => "true"
  instance_id:  "" => "i-0d48be4266da"
  skip_destroy: "" => "true"
  volume_id:    "" => "vol-04c7f4"
aws_volume_attachment.mysql_data: Still creating... (10s elapsed)
Error applying plan:

1 error(s) occurred:

* aws_volume_attachment.mysql_data: 1 error(s) occurred:

* aws_volume_attachment.mysql_data: [WARN] Error attaching volume (vol-04c7f4) to instance (i-0d48be4266da),
message: "Invalid value '/dev/xvdf' for unixDevice. Attachment point /dev/xvdf is already in use", code: "InvalidParameterValue"

Terraform does not automatically rollback in the face of errors.
Instead, your Terraform state file has been partially updated with
any resources that successfully completed. Please address the error
above and apply again to incrementally change your infrastructure.

```

# Answer
The old volume is never detached because `skip_destroy = true`. From the terraform docs:

> skip\_destroy - (Optional, Boolean) Set this to true if you do not wish to detach the volume from the instance to which it is attached at destroy time, and instead just remove the attachment from Terraform state. This is useful when destroying an instance which has volumes created by some other means attached.

If you remove that, the old attachment will be destroyed and the attachment point should be available for the new attachment.

You may need to manually delete the old attachment this one time, I believe Terraform may have already deleted it from the state.
> 10 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: How to check AWS spot pricing automatically?
I've been thinking about a Jenkins job to check the pricing for the agent boxes we spin up; every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc.

My first instinct was that the best route is to have an hourly Jenkins job that runs a `aws ec2 describe-spot-instance-requests` and checks for failed requests (and then slacks us on a failure). But I'm wondering if there's a cleaner method that involves actually comparing hard prices (and could therefore tell us exactly what's off, and by how much), rather than looking at successful/failed requests.

Anyone set up something similar? How'd you do it?

# Answer
> 15 votes
Spotted an open source tool called autospotting that just might help with this:

> Once enabled on an existing on-demand AutoScaling group, it launches an EC2 spot instance that is cheaper, at least as large and configured identically to your current on-demand instances. As soon as the new instance is ready, it is added to the group and an on-demand instance is detached from the group and terminated.

We have this in our to-do pipeline, will be able to add more context once we finish that.

Update:

Another tool that was recently demo'ed in a conference was mapbox/spotswap

This works slightly differently. It monitors a normal ASG with either On-Demand or Reserved instances and then, if scale arises, bids and provisions spot instances of similar compute level on a separate ASG.

# Answer
> 6 votes
I would personally consider a model like this:

```
Timed Lambdas -> Checks spot price -> Push to ElastiCache

```

Then when you need instances:

```
Timed lambdas -> Pulls spot price from ElastiCache, sets it as environment variable on your Machine where you spin up IaC from -> This is parsed as argument to IaC code and pushes out the spot price

```

You could set some tolerances, too, within the lambdas (i.e. 10, 25, 50% increases based on importance) and a hard cap of on-demand, for example. It's also a great place to build the logic to handle, for example, finding the cheapest AZ, finding the relatively cheapest spot price (`2xt2.medium` vs `t2.large`), etc.

# Answer
> 4 votes
Let me give you a tool-agnostic way of trying to do this.

> every so often the price spikes and it'll be an hour or two before anyone notices that no new agents are coming up, and then we have to go in manually and check the spot price and adjust accordingly, or switch zones, etc

We faced the same problem in the infrastructure which we're building. So, we had `if-else` style blocks to set the bid price, depending on the on-demand price of the instance.

AWS has an API for getting the on-demand price of an instance. We used this Python wrapper for the purpose.

So, once we got the on-demand price (let's say `X`), we plugged in `if-else` style blocks, which are `0.4*X`, `0.6*X`, `0.8*X`, `X`, which means we are trying for a bid price in the range 40%, 60%, 80% of the on-demand price. If all fails, then we are falling back to creating on-demand instances.

Also, as this is independent of AWS's current spot pricing, we never pay a price above the on-demand price.

But, if you're looking for a way to do it on-the-fly, then Hashfyre's solution should be the way to go.

---
Tags: jenkins, amazon-web-services, amazon-ec2
---


# Question
Title: What 'ChatOps' books and resources are available?
We have defined what ChatOps is in the form of the following question:

* What are ChatOps approach use cases of production infrastructure management?:

> Practically it means something very different to each team using it. So much so that Atlassian have put together what could be considered to be a Chat maturity model.
>
> 1. **Sputnik**: Trying chat out to see how it works.
>     2. **Mercury**: Starting to move to chat.
>     3. **Gemini**: Chat as the primary source of communication, experimenting with bots.
>     4. **Apollo**: Chat as the nexus of communication and work, including bots.
>     5. **Elon Musk**: Chat is the primary centre of mission critical work.

What other books and resources are available that describe the people, process and tools used to implement ChatOps within an organisation.

# Answer
> 1 votes
From this 'awesome' chat ops resource collection.

> ## Articles
>
> ## Videos
>
> ## Books

# Answer
> 3 votes
One of the people active in ChatOps community is Jason Hand from VictorOps, who wrote a book published by O'Reilly called Managing Operations in Group Chat. This book is available for free as part of their content marketing campaign. The previous ChatOps for Dummies is no longer available and is replaced by the above book.

You should definitely take a look at Slack Apps, especially the Developer Tools. There is article by SlackStorm on how to integrate Ansible and Slack.

You might also monitor the subreddit on chatops. There is a long sidebar of resources including the video from GitHub ChatOps presentation and many more.

I would also recommend to look at Cog from operable.io and the Cog Book.

---
Tags: toolchain, operating-model, operations, chatops
---


title: Zip/tar a folder using chef resource

text:
# Question
Title: Zip/tar a folder using chef resource
Can someone help me how to take backup for a specific folder and also zip/tar those folders. I am using this for taking a backup actually, but I would like a compressed output:

```
bash "backup #{dir_name}" do
    user 'user'
    cwd "#{path}"
    code <<-EOH
      cp -rp #{dir_name} #{dest}/#{dir_name}_bkp_#{timestmp}
    EOH
end

```

Is there any chef resource to zip/tar a specific folder ?

# Answer
Well that's not exactly about chef, you're basically asking how to use `tar`.

The command to create an archive of a folder would be
`tar -zcvf <backup_thing>.tgz <path_to_backup>`

In the tar options:

* `z` pass the resulting tar archive to gzip for compression
* `c` instruct tar to compress (create the archive)
* `v` tells tar to be verbose, so you know at which point it is working, it has a performance impact
* `f <path>` tells tar to output in a file or device

A correct chef resource for this would be as follow:

```
execute 'Backup #{dir_name}' do
  user 'user'
  command "tar -zcvf #{dest}/#{dir_name}_bkp_#{timestmp}.tgz #{dir_name}"
  live_stream :true # to see the progress during chef run, remove it and v in tar option if you don't care of it.
end

```

Using an execute resource avoid spawning a bash process to run tar, it is not needed, the `shell_out` class used by `execute` under the hood handles it already.

Be warned this is not idempotent, each run of chef will redo this backup command, that's maybe not what you're after.
For an update of jenkins (or something else) out of usual maintenance I would do a manual backup, check it is ok, and then run the upgrade (via chef eventually).

I'd recomend taking the tutorial at https://learn.chef.io and reading through the documentation at https://docs.chef.io
> 7 votes

---
Tags: chef, backup
---


# Question
Title: Constructing Dependencies from a Pillar in Saltstack
I'm trying to create a dummy state in Salt to pull dependencies from a list derived from a pillar.

In my `top.sls` file, I have:

```
base:
  '*':
    - components

```

Then, in `components.sls`, I have:

```
{% if 'components' in pillar.items() %}
include:
{% for component in pillar.get('components',[]) %}
  - {{ component }}
{% endfor %}
{% endif %}

{% if 'components' in pillar.items() %}
components:
  require:
{% for component in pillar.get('components',[]) %}
    - {{ component }}
{% endfor %}
{% endif %}

```

When I check the pillar contents, I see:

```
$ salt my-minion-id pillar.items

my-minion-id:
    ----------
    components:
        - a-dependency-name

```

But when I try to see the set of states, I see:

```
$ salt my-minion-id state.show_sls components

my-minion-id:
    ----------

```

...and that's it.

My ultimate goal, here, is to treat the minion as hostile, so I'm trying to pull role details out of pillars instead of assigning roles in my `file_root`'s `top.sls`. It's my understanding that every minion has full access to the contents of the file\_root, and I don't want a hostile minion to know the specific firewall or services of any unrelated node.

**Edit:**

I discovered that my pillar structure didn't support pillar merging, as it used lists, so I switched to a pillar-structure that used no-value dicts:

```
my-minion-id:
    ----------
    components:
        ----------
        a-dependency-name:
            None

```

# Answer
> 4 votes
This templated SLS file works splendidly:

```
{% if 'components' in salt.pillar.items() %}

include:
{% for component in salt.pillar.get('components').keys() %}
  - {{ component }}
{% endfor %}

{% endif %}

```

However, it requires changing my pillar key structure. From the original question, you see the structure as:

```
my-minion-id:
    ----------
    components:
        - a-dependency-name

```

Instead, it needs to be:

```
my-minion-id:
    ----------
    components:
        ----------
        a-dependency-name:
            None

```

Note that `a-dependency-name` is now a dict, and has a single key/value, `None`. Thus the pillar SLS file needs to change from

```
components:
  - a-dependency-name

```

to

```
components:
  a-dependency-name: ~

```

While you can get away with not using no-value dictionaries (and thus get rid of the `.keys()` in the template) and use lists instead, if you do that, you can't merge the `components` from multiple different pillars; each pillar Salt applies will override the `components` key from the previous, and the last pillar read will win. If you want to merge pillars (we *are* talking about role-based assignment, here), this would appear to be the necessary construct.

With that, the output of `salt my-minion-id state.show_sls components` is then correct:

```
$ salt my-minion-id state.show_sls components
my-minion-id:
    ----------
    a-dependency-name:
      ----------
      ...

```

`pillar.items()` in Jinja templates turns out not to be *quite* equivalent to `salt.pillar.items()`; if you try using `pillar.ls()`, for example, you may see the following error:

```
Rendering SLS 'base:components' failed: Jinja variable 'salt.pillar object' has no attribute 'ls'

```

Whether that means one should avoid the implicit `salt.` prefix shortcut Jinja provides, or instead use a construct like `{% if pillar['components'] is defined %}` (thanks, @brousch, for the advice), I can't say.

# Answer
> 3 votes
You can also do it like:

```
{% if pillar['components'] is defined %}
include:
{% for component in pillar['components'] %}
 - {{ component }}
{% endfor %}
{% endif %}

{% if pillar['components'] is defined %}
components:
  require:
{% for component in pillar['components'] %}
 - {{ component }}
{% endfor %}
{% endif %}

```

---
Tags: dependency-management, salt, jinja, templating
---


# Question
Title: How to cache third party resources in a build pipeline?
A often requires to access third party resources, aside from the source code of the artefacts it is meant to build. Each of these third party resources introduce a risk factor in builds, for instance:

* Distribution repositories are temporarily unavailable, therefore installing distribution packages temporarily fails.
* Distribution repositories are permanently unavailable, same and worse as before.
* Some NPM free-software package stops to exist, because its author reclaimed that withdrawal.
* Some source file or binary ad-hoc package for a third-party tool is not available anymore.

This list could be extended *ad lib* and while studying the various reasons leading to some third-party resource to disappear is much more entertaining as it sounds, it first of all has for us a disastrous consequence: the *build-pipeline* is broken.

What the is the best way to cover the two following cases when building *docker* images:

1. Caching files downloaded with a *curl* or *wget*.
2. Caching Debian packages.

There are several plausible strategies that can be considered here. For instance using a proxy and configuring the build system to route all its requests through this proxy . A second, cheap, possibility would be to wrap *curl* calls to store results in a local cache.

(In this question I would consider sharing the cache among several units of the build pipeline as a question of secondary importance.)

# Answer
I'm not overly familiar with Docker particularly but it sounds a lot like issues one would get when building a normal .NET application or java app with the need to store their external dependencies (DLL or jar).

To solve this I'm quite fond of artifactories. Two of the most popular are sonatype's Nexus and JFrog's Artifactory. Both offer a free Open Source version and both will have some way to store customised package or use it as a raw depot. There are other options out there, I just listed these two because I've worked with them already and know they will do what you need. As a bonus you might also be able to use them to manage the docker image directly as well.

There are others too worth mentioning, you can find a comparison table for them, you'll get a better view of their capabilities. (thanks to Karl Harnagy for the link)

Note that some of these options might require one or another of the paid versions so YMMV.

Some continuous integration systems will also support the concept of artifacts (from experience Jenkins and team-city do this, others might as well though) which should allow you do achieve similar results directly.

If none of these do it for you you can look into other technologies or roll your own. I personally like to leverage whatever I already have this reduces the maintenance required in long term and makes passing the buck easier when comes time to look for new challenges elsewhere.

Hope this helps
> 6 votes

# Answer
It looks like what you're looking for is less about caching and more about mirroring. Because your requirements have to deal with permanently unavailable packages, you need to persist the packages yourself in something more permanent than a cache. In the past, we've stored packages in persistent, cloud-based stores like S3. However, there is nothing to preclude you from setting up a custom filestore on the build server.

More concretely, it's probably easiest to set up a proxy service that looks for a resource in your filestore (S3) and returns the package if it's found. Otherwise, it would just request the resource from the upstream, populate your filestore, and return the package.
> 1 votes

---
Tags: docker, build-pipeline
---


# Question
Title: Is there a way to disable a Jenkins plugin for a single job?
I use the Build timeout plugin to alert when jobs are stuck. I have one job that triggers this alert because it contains a step with a long execution time. I would like to disable the build timeout plugin but only for that job, is there a way to do so? I'm using Jenkins v2.22.

We don't use the built-in option for the job not to terminate on timeout, we use sensu plugins to monitor the state of the likelyStuck variable.

# Answer
Sadly I just switched jobs and I don't have access to a Jenkins instance (least not yet) so I cannot try it out for you.

That said some ideas quickly jump in my mind that could get you there.

I've had some success with the conditional build step plugin though I don't know if it would work for build timeouts.

If not directly perhaps as a means to change some values through the rest API ? Maybe using job parameters ? You could also split the job in two and call the appropriate one (with or without timeout) using a conditional build step (mentioned above)...

If you are int a pipeline you could use the timeout in a jenkins file. https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#code-timeout-code-enforce-time-limit.

let me know if any of this is helping you.
> 7 votes

---
Tags: jenkins, jenkins-plugins, sensu
---


# Question
Title: How to avoid "retry storms" in distributed services?
A "retry storm" is caused when clients are configured to retry a set number of times before giving up, a retry policy is necessary because of packets loss will occur in normal operation of a service.

Take this example:

If for example the services as a whole were scaled to support 80,000 requests per seconds and run at about 80% of capacity, a spike in traffic that caused the service to receive 101,000 requests per second would cause 1,000 of those requests to fail.

When the retry policies kick in, you end up with additional 1,000+ requests, depending on where the failure was detected which would push the service as a whole up to 102,000 requests per second - from there your service goes into a death spiral doubling the number of failed requests every second.

Other than massive over-provisioning of services beyond the projected peak transaction, which would be inefficient. What strategies can you employ to avoid "retry storms"?

# Answer
> 12 votes
It depends on what you're trying to avoid.

If you are trying to avoid any service interuption of something which is a genuinely critical service (I'm thinking in terms of "people will die if my API call is not appropriately served") the you need to just budget for the huge inefficiencies that come from vastly over provisioning dedicated resources. And yes they have to be dedicated, none of this allowing for traffic spikes stuff, multiple services spiking would thus cause an outage.

In the *far* more likely scenario that you're service going down would be inconvenient you can tackle the problem both from the client and server sides. Although it's worth noting that it's logically impossible to actually solve the problem of to much traffic because without processing the traffic (which consumes resources) you can't know if it's a retry, if it's a retry for a request that was successful but incorrectly handled by the client, if it's a DDOS, etc. But you *can* mitigate impact.

In the **client code** write sensible retry logic which has an upper limit and a mechanism for gracefully failing. That way you don't stick your users in an infinite loop of failing requests and you just give them an error telling them to try whatever they just did in little while.

For your **server side** infrastructure the simplest solution is to throttle. Hard limits on requests, especially if you can try and spread them logically based on your specific use case (ie. If you have a centralised service make some hard decisions, do you want to start blocking geographically distant requests which might be resulting in threads hanging server side? Or do you want to distribute your inevitable yet minor outage evenly? etc) It basically boils down to the fact that returning a 503 intentionally from a gateway is a hell of a lot cheaper than letting the request go through and sending a 504 anyway. Basically force clients to behave based on what you can currently provide and provide the correct responses so that clients can react appropriately.

# Answer
> 8 votes
One way of preventing these retry storms is by using backoff mechanisms.

From the Implement backoff on retry section of Google App Engine Designing for Scale guide:

> Your code can retry on failure, whether calling a service such as Cloud Datastore or an external service using URL Fetch or the Socket API. In these cases, you should always implement a randomized exponential backoff policy in order to avoid the thundering herd problem. You should also limit the total number of retries and handle failures after the maximum retry limit is reached.

Most of the GAE APIs already have such backoff mechanisms/policies enabled by default.

---
Tags: serverless, cloud, distributed-systems
---


# Question
Title: Where can I find older versions of GitLab Runner?
I'm stepping through the documentation here for installing GitLab Runner. When I register the runner, I get this compatibility error:

```
ERROR: Checking GitLab compatibility... not-compatible  reason:=Gitlab
Runner >= 9.0 can be used ONLY with GitLab CE/EE >= 9.0 results=405
runner=FsqqGSrw statusText=405 Method Not Allowed

```

We're operating on an isolated network with GitLab 8.13.0, and no plan to update at the moment. All I can find in the documentation is this link which provides the latest 9.0 version of the runner.

Does anyone know where I might find older versions of GitLab Runner?

# Answer
As described in the documentation you are following to install on the first paragraph:

> You can download a binary for every available version as described in Bleeding Edge - download any other tagged release.

Following the link in this sentence allow your to find the tag list page

Last version in 1.X (Hence compatible with 8.X versions of Gitlab is 1.11.2 at time of writing, you can download it from The tag details page

If you prefer another archive format, you can choose in the dropdown of download at the right of the tag page description:
> 7 votes

# Answer
According to the runner readme, the old version compatible with 8.x can be downloaded from here.
> 1 votes

---
Tags: continuous-integration, gitlab
---


title: Laptop Internet connection monitoring

text:
# Question
Title: Laptop Internet connection monitoring
I have a need to monitor laptops both Linux and Windows that are deployed to remote locations which we don't have physical access to all the time.

The laptops are used for different services but need Internet access, what we would like to have is a solution to alert us when a Internet has lost Internet connection

For web servers we use nagios and pingdom but wondering if someone has come across a solution that is lightweight and for laptops

Ideally having a dashboard to view what laptops are connected would be the best

# Answer
> 5 votes
I've started abusing SyncThing for this purpose (among others), configuring it as a system service on laptops, then locking down it's UI to prevent it from being able to be used by a local connection originator to manipulate or access files. I get connectivity monitoring and robust `rsync`-like backup of field data.

It's fantastic at punching through difficult network conditions, so once you peer a node with a central node (say, a backup landing pad itself backed up via more conventional means), you can use Syncthing's API on your landing pad node to track connectivity.

If you're looking for end-to-end IP connectivity, and not just "can the remote host reach the Internet at all" knowledge, you can disable NAT traversal and relay use, at which point there will need to be a clean, unfirewalled path between the two nodes.

The downside is that SyncThing is *very* much a rapidly-developed project, with the issues that carries with it.

---
Tags: monitoring, network-monitoring
---


# Question
Title: Using requisite injection to order states
Given three states, `/root/a`, `/root/b` and `/root/c`, I want `/root/c` to execute before `/root/b`, and `/root/b` to execute before `/root/a`.

Given a Salt SLS file `salt://ordertest/init.sls`:

```
/root/a:
  file.managed:
    - source: salt://ordertest/a
    - user: root
    - group: root
    - mode: 600

/root/b:
  file.managed:
    - source: salt://ordertest/b
    - user: root
    - group: root
    - mode: 600

/root/c:
  file.managed:
    - source: salt://ordertest/c
    - user: root
    - group: root
    - mode: 600

```

I can test to see the order in which the listed states would be applied. You'll see `/root/a` precede `/root/b`, and `/root/b` preceed `/root/c`.

```
$ salt my-minion-id state.apply ordertest test=True
my-minion-id:
----------
          ID: /root/a
    Function: file.managed
      Result: None
     Comment: The file /root/a is set to be changed
     Started: 13:54:23.538144
    Duration: 31.765 ms
     Changes:
----------
          ID: /root/b
    Function: file.managed
      Result: None
     Comment: The file /root/b is set to be changed
     Started: 13:54:23.570065
    Duration: 16.632 ms
     Changes:
----------
          ID: /root/c
    Function: file.managed
      Result: None
     Comment: The file /root/c is set to be changed
     Started: 13:54:23.586831
    Duration: 17.124 ms
     Changes:

Summary for my-minion-id
------------
Succeeded: 3 (unchanged=3)
Failed:    0
------------
Total states run:     3
Total run time:  65.521 ms

```

If we use `require` to directly tell `/root/b` to require `/root/`, and `/root/c` to require `/root/b`:

```
/root/a:
  file.managed:
    - source: salt://ordertest/a
    - user: root
    - group: root
    - mode: 600
    - require:
      - /root/b

/root/b:
  file.managed:
    - source: salt://ordertest/b
    - user: root
    - group: root
    - mode: 600
    - require:
      - /root/c

/root/c:
  file.managed:
    - source: salt://ordertest/c
    - user: root
    - group: root
    - mode: 600

```

We get the ordering we want:

```
$ salt my-minion-id state.apply ordertest test=True
my-minion-id:
----------
          ID: /root/c
    Function: file.managed
      Result: None
     Comment: The file /root/c is set to be changed
     Started: 13:32:33.624165
    Duration: 31.807 ms
     Changes:
----------
          ID: /root/b
    Function: file.managed
      Result: None
     Comment: The file /root/b is set to be changed
     Started: 13:32:33.656166
    Duration: 17.247 ms
     Changes:
----------
          ID: /root/a
    Function: file.managed
      Result: None
     Comment: The file /root/a is set to be changed
     Started: 13:32:33.673589
    Duration: 17.126 ms
     Changes:

Summary for my-minion-id
------------
Succeeded: 3 (unchanged=3)
Failed:    0
 ------------
Total states run:     3
Total run time:  66.180 ms

```

But if we try to use `require_in` to have `/root/c` place itself before `/root/b`:

```
/root/a:
  file.managed:
    - source: salt://ordertest/a
    - user: root
    - group: root
    - mode: 600
    - require:
      - /root/b

/root/b:
  file.managed:
    - source: salt://ordertest/b
    - user: root
    - group: root
    - mode: 600

/root/c:
  file.managed:
    - source: salt://ordertest/c
    - user: root
    - group: root
    - mode: 600
    - require_in:
      - /root/b

```

It doesn't work; `/root/c` stays at the end of the sequence:

```
my-minion-id:
----------
          ID: /root/b
    Function: file.managed
      Result: None
     Comment: The file /root/b is set to be changed
     Started: 13:36:54.037214
    Duration: 32.401 ms
     Changes:
----------
          ID: /root/a
    Function: file.managed
      Result: None
     Comment: The file /root/a is set to be changed
     Started: 13:36:54.069797
    Duration: 16.968 ms
     Changes:
----------
          ID: /root/c
    Function: file.managed
      Result: None
     Comment: The file /root/c is set to be changed
     Started: 13:36:54.086908
    Duration: 17.349 ms
     Changes:

Summary for my-minion-id
------------
Succeeded: 3 (unchanged=3)
Failed:    0
------------
Total states run:     3
Total run time:  66.718 ms

```

Here, you can see that `/root/b` was ordered before `/root/a` as appropriate, but `/root/c` did not get ordered before `/root/b`.

So how can I get this to work? I could use `order` instead of `require` and `require_in`, but that would mean I couldn't use `require` targeting any state that uses `order`, which could get quite messy and poorly-defined.

# Answer
This turns out to be a known issue:

> The require\_in requisite does not support everything that require does, mainly id does not support sls or state\_id without specifying a state module.

So by modifying our SLS file to include the state module (specifying `file: /root/b` as the `require_in` target instead of simply `/root/b`), we get the correct result.

I don't know if the other requisite injectors (`watch_in`, etc.) have the same limitations, so I will likely standardize my coding style on explicitly including module names in such identifiers.
> 5 votes

---
Tags: configuration-management, dependency-management, salt
---


# Question
Title: How can I make use of bash functions in a makefile?
If I need to use something like NVM in my makefile, I run into a problem, because NVM exports some bash functions that are not available in Make, even if my shell is defined as bash in the makefile.

How do I cause make to "inherit" all the parent shell's functions? Is there a way to make NVM accessible to a makefile, without writing `export -f` for a zillion little NVM bash functions?

# Answer
> 6 votes
There are several options that can be used here. Maybe one of the easiest is to install each function as a real script in some directory that is added to the path. Here is how to do this:

First, we choose a path name where to store all these functions, it can be a a directory in our project, where other utility scripts used in our Makefile live. We call it *nvm\_install\_dir*

Then we write a `_nvm_trampoline` script which use the name it is called as to trigger the right function, and create as many file aliases to `_nvm_trampoline` as there are functions in the script. Note the leading underscore, hinting at the “private” character of the script.

This script can go along the lines of

```
#!/bin/sh
. "${NVM_DIR}/nvm.sh"

if [ "${0##*/}" = '_nvm_driver' ]; then
    : NOP
else
    eval "${0##*/}" "$@"
fi

```

Here we use `${0##*/}` to remove any path element from the name under which our script is called. We install that script under `${nvm_install_dir}` and run once the following utility script:

```
nvm_install_dir='SET-TO-ACTUAL-PARAMETER-VALUE!'
nvm_file="${NVM_DIR}/nvm.sh"
nvm_trampoline='_nvm_trampoline'

nvm_list_functions()
{
    awk -F'[(][)]' '$1 ~ /^nvm_[^ ]*$/{print($1)}' "${nvm_file}"
}

nvm_install()
{
    nvm_list_functions | {
        while read nvm_function; do
            ln\
                "${nvm_install_dir}/${nvm_driver}"\
                "${nvm_install_dir}/${nvm_function}"
            chmod 755 "${nvm_install_dir}/${nvm_function}"
        done
    }
}

nvm_install

```

It is a good practice to pack the *nvm\_install* procedure in a real function instead of just inlining its body in the script, as it gives better testing options. (It is easier to comment out the call to *nvm\_install* than the body of the function, if we want to experiment with the script.)

Ater this, the directory pointed to by *nvm\_install\_dir* is populated with aliases to `_nvm_trampoline` that are executable and delegate their work to the corresponding function. We only need to add this directory to our *PATH* when running *make*.

A second approach would be to generate pseudo commands for each *nvm* functions, with the following script:

```
nvm_file="${NVM_DIR}/nvm.sh"

nvm_list_functions()
{
    awk -F'[(][)]' '$1 ~ /^nvm_[^ ]*$/{print($1)}' "${nvm_file}"
}

nvm_generate()
{
    nvm_list_functions |  {
        while read nvm_function; do
            nvm_FUNCTION=$(printf '%s' "${nvm_function}" | tr '[a-z]' '[A-Z]')
            printf '%s=sh -c \047source "${NVM_DIR}/nvm.sh"; %s "$$@"\047 %s\n'\
                   "${nvm_FUNCTION}"\
                   "${nvm_function}"\
                   "${nvm_function}"
        done
    }
}

nvm_generate

```

The output of this program consists of *make* variable assignments

```
NVM_ECHO=sh -c 'source "${NVM_DIR}/nvm.sh"; nvm_echo "$$@"' nvm_echo
NVM_CD=sh -c 'source "${NVM_DIR}/nvm.sh"; nvm_cd "$$@"' nvm_cd
…
NVM_COMMAND_INFO=sh -c 'source "${NVM_DIR}/nvm.sh"; nvm_command_info "$$@"' nvm_command_info
…

```

It can be saved to a file `Makefile.nvm` or `nvmtools.mk` that can be included our Makefiles. Calling the function `nvm_command_info` is done with

```
${NVM_COMMAND_INFO} arg1 arg2 …

```

---

Using a customised bash profile is possible, using the assignment `SHELL=/bin/bash --rcfile PATH-TO-CUSTOM-PROFILE -i` but I would consider this esoteric enough to puzzle the maintenance programmer.

# Answer
> 3 votes
One possible issue is that the NVM install script only writes the commands to set itself up to either `.bash_profile` OR `.bashrc`. Depending on your systems configuration, it may only be setup for interactive shells, not ones spawned by other programs (more info). Sourcing the nvm setup script can be a good option, as a previous comment noted. If you have a number of commands to run for 1 make target, look into .ONESHELL. Sourcing `$NVM_DIR/nvm.sh` should take care of it.

Another option that invokes bash directly with the interactive flag might work as well.

---
Tags: make
---


# Question
Title: Autoscaling containers using local request counters
Somewhere around I got to know about different approaches which were used to scale our webapp which included **Scaling using local request counters**. Below that they had written the drawbacks of this approach adding that

> Each instance would reach the threshold almost at the same time and hence each would demand a new instance, leading to a large number of instances even though the number should have increased by one only

I was curious to know if there's a solution to this problem or any workaround ?

# Answer
The problem can easily be solved using the following components:

* On the instances serving your *webapp* continue to monitor the number of incoming requests – and anything else you see fit.
* Publish the number of incoming requests to a monitoring system. If it is not yet implemented, this step will improve your monitoring capacities, and will help you to monitor the load on each hosts as well as host balancing.
* From the incoming requests, deduce an estimated required number of *webaspp* servers needed to serve that work load, as well as the difference between the actual number and the estimated needed number. In the case you describe, it seems that the estimated number is just a scalar function of the total number of incoming requests on a recent period of time. On other systems or after some times, more subtle strategies can be implemented. Monitoring these quantities and the difference could ease the traceability of the auto-scaling strategy, and will monitor its responsivity.
* Last implement the auto-scaling itself, at this point, this is really just reading some number from your monitoring system and writing it to your scaling system.
> 3 votes

# Answer
One possible approach is to allow such instances to make demands for new instances based on local request counters, but instead of directly reacting to those demands you would funnel them to a central instance creation logic.

That logic would immediately react to the first demand, but also start a "cool off" countdown timer. Any subsequent demand received while the timer is still active would be considered to be caused by the same traffic spike that triggered the first demand and would thus be ignored.

A similar logic could be used to gradually shutdown idle instances, if the local counters remain below a minimum level.

**Note:** the local counters would need to operate in a leaky bucket manner or be periodically reset for such approach to be possible - so that the demands are repeated if the traffic remains high.

Another possible approach is to just publish the local counters. A central piece of logic would periodically collect and aggregate these local counter values in order to decide on lauching new instances or shutting down existing ones.

The advantage of such method is the lack of a *single* global counter which would require write access locking (to prevent corruption) which would be a scalability limitation.

This method is described in mode details in Sharding counters.
> 2 votes

---
Tags: containers, autoscaling
---


# Question
Title: How do I configure Travis CI build testing for a Java Eclipse project?
I have an Eclipse Java project on GitHub. I would like to set up auto-build testing with Travis CI. However, when I enable build testing for my project, compilation always fails with the following error.

```
Buildfile: build.xml does not exist!
Build failed

The command "ant test" exited with 1.

```

How do I configure Travis CI to run build tests for my Java Eclipse project?

# Answer
> 10 votes
The easiest method is to have Eclipse generate an Ant build script for you.

Right click on your project in the Package Explorer and select **Export** in the context menu. Choose the export type **General -\> Ant Buildfiles** and click **Next**.

On the next screen, make sure your project is selected. You can keep leave the options on their default settings. Click **Finish** to generate your `build.xml` file.

Lastly, to ensure Travis CI runs the project correctly, create a .travis.yml in your project's root folder. For Java projects, it should contain at least the following.

```
language: java
jdk:
  - oraclejdk8

script: ant build

```

Source: Kofun devblog - Using Travis CI with Java Eclipse Project

---
Tags: travis-ci, github, eclipse
---


# Question
Title: Referencing current variable in Ansible conditional statement?
I'm using the willshersystems.users role from ansible galaxy for creating users on ubuntu servers. Now that I've got it working consistently, I'm trying to set the state attribute based on whether the user should have access to a specific server given a list.

Is something like the below possible? My google fu seems to be failing me at the moment, any advice would be very appreciated!

```
vars:
  user_servers:
    user1:
      - testing1

  users:

    - name: user1
      groups: '{{ user_groups }}'
      state: "{{ ( inventory_hostname_short in user_servers[self.name] ) | ternary('present','absent') }}"

roles:
  - role: willshersystems.users

```

# Answer
> 4 votes
How about something like this:

```
hosts: all

vars:
  user_servers:
    user1:
      - testing1

  users:
    -name: user1
    -groups: '{{ user_groups }}'
    -state: present

roles:
  - name: Add filtered users
    tasks:
      - name: Filter users
        debug: User {{ item }}
        with_items: "{{ users | default([]) }}"
        when: {{ inventory_hostname_short }} in user_servers[item]
        register: filtered_users

      - name: Add filtered users
        include_role: wilshersystems.users
        with_items:
        - {{ filtered_users }}
        loop_control:
        - loop_var: users

```

---
Tags: ansible
---


# Question
Title: Best practice when automating Drupal (or other soft) installation regarding database settings
Suppose I have Drupal and CiviCRM installations to automate.

These, however, need particular configs for working together (some file edits and database configs)

Should I make a fixed version install (eg. Drupal 4.7.5 \+ CiviCRM 4.5.6), install them and configure, and then save the edited files plus database?

Or should I put more effort in my build script and make some queries and seds to edit those configs in the database and files?

# Answer
Your question sounds like you should have a look at what I call **F**eatures **C**onfiguration **M**anagement, aka **FCM**. FCM is not only about using Drupal's Features module, and not about ***Configuration Management*** (as introduced in Drupal version 8). Instead, it is a special case of **S**oftware **C**onfiguration **M**anagement, aka **SCM**, applied to Drupal.

Mostly because Features can be considered as a code generator, whereas that generated code may be considered as the *build script* (as in your question). Which can then also be used to be migrated through multiple environments.

Have a look at my (popular) answer to "How to use the Features module in a 3 dev environments?" for (way) more details about all this.
> 7 votes

---
Tags: automation, drupal
---


# Question
Title: What are best practices of structuring encrypting variables since Ansible 2.3.0?
This document indicates that since Ansible 2.3 it should be possible to encrypt certain vars using a `!vault |` prefix instead of putting a variable and keys in a vault file and encrypt it completely.

> ```
> notsecret: myvalue
> mysecret: !vault |
>           $ANSIBLE_VAULT;1.1;AES256
>           66386439653236336462626566653063336164663966303231363934653561363964363833313662
>           6431626536303530376336343832656537303632313433360a626438346336353331386135323734
>           62656361653630373231613662633962316233633936396165386439616533353965373339616234
>           3430613539666330390a313736323265656432366236633330313963326365653937323833366536
>           34623731376664623134383463316265643436343438623266623965636363326136
> other_plain_text: othervalue
>
> ```

At the moment the structure is as follows:

```
ansible/group_vars/testing/vars
ansible/group_vars/testing/vault

```

When an encrypted variable is moved from the vault to the vars directory and `ansible-vault decrypt ansible/group_vars/testing/vars` is run it returns:

```
ERROR! input is not vault encrypted data for ansible/group_vars/testing/vars

```

> This vaulted variable be decrypted with the supplied vault secret and used as a normal variable. The ansible-vault command line supports stdin and stdout for encrypting data on the fly, which can be used from your favorite editor to create these vaulted variables; you just have to be sure to add the !vault tag so both Ansible and YAML are aware of the need to decrypt. The | is also required, as vault encryption results in a multi-line string.

**Questions**

1. Should variables that need to be encrypted, be encrypted one by one using the command line?
2. What are best practices to restructure the old Ansible structure? E.g. remove the vault files and put all encrypted vars in the vars file?

# Answer
> 2 votes
Having been working with ansible vault quite a bit recently (specifically regarding what the encrypt and how to encrypt those things without making the code unreadable) I've found very little incentive to change my habits in 2.3.

If I'm going to have a series of encrypted variables, or files, keep them separate (I prefix them all with `vault_`) and just leave it at that.

A few times I have used the !vault feature but am left distinctly unamazed as it seems easier to just be very explicit about what is and is not coming from a secrets file. That way no one editing my plays makes assumptions about what is and isn't sensitive data.

# Answer
> 1 votes
I have decided to remove the `ansible/group_vars/testing` directory and replace it with a file, i.e. `ansible/group_vars/testing` that contains the encrypted vars:

```
mysecret: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          66386439653236336462626566653063336164663966303231363934653561363964363833313662
          6431626536303530376336343832656537303632313433360a626438346336353331386135323734
          62656361653630373231613662633962316233633936396165386439616533353965373339616234
          3430613539666330390a313736323265656432366236633330313963326365653937323833366536
          34623731376664623134383463316265643436343438623266623965636363326136

```

Instead of running `ansible-vault edit ansible/group_vars/testing/vault` one could pipe the values that need to be encrypted to ansible-vault, i.e. `printf mysecret | ansible-vault encrypt`. The latter will encrypt `mysecret` and the output could be added to the `ansible/group_vars/testing` file. When `ansible-playbook` will be run the encrypted variables will be decrypted if the ansible vault file is specified of course.

If the encrypted value needs to be debugged then the following code could be used:

```
- debug: msg="Decrypted value: {{ encrypted_var }}"

```

---
Tags: ansible-vault
---


title: What is a Platform Engineer?

text:
# Question
Title: What is a Platform Engineer?
I've been called many things over my career, most of them polite - more recently the work to build CI/CD pipelines has been termed "Platform Engineer" or "Platform Engineering". The risk of such a title is that you get typecast into the role of an engineer who develops platforms for use by Developers and doesn't necessarily include any of the Ops bit of DevOps.

In my experience, however, the roles have involved a great deal of Operations Discipline work such as designing Major Incident Management Processes, Alerting, and Monitoring solutions.

In the role it "feels" like it has a lot in common with Site Reliability Engineering, in that both roles are engineering roles somehow set apart from software engineering they seem to have a similar Google Trends Pattern though:

Is there a canonical definition of a Platform Engineer or is it like SRE simply an implementation of a DevOps capability within an organisation?

# Answer
I think you would be hard pressed to get a straight definite definition of a Platform Engineer.

Where I'm currently positioned everyone on the team starts out with their own special role but very quickly the end-game seems to be to feel comfortable enough to be called, or to call ones self, a platform engineer.

When I was first told that officially a platform engineer the brief explanation of what that meant, for me at least, was that I was going to need to do a whole lot more defending of my platform.

I wasn't quite so focused on helping our developers directly (although that's still the goal obviously) but as the platform owners our jobs was mostly to make sure that our developers where being good citizens by making sure we designed the platform so that we could show them what they did wrong if something happened and we could give them all the information they needed to fix their code.

The obvious implication of this is that our platform is perfect, which it obviously isn't. But at least we can support the developers in a way that gets them working on a solution on their end while we're fixing up our side of the agreement.
> 16 votes

# Answer
Source: I'm a Platform Engineer, (though not in a strict DevOps sense).

I manage Office 365 for my employer. Particularly the Identity, Authentication, Messaging portions. I don't build Exchange servers (under normal circumstances) but I do manage all changes in the service and "own" the platform.

As hvindin's answer also calls out, I am equally responsible for evangelizing the platform I own as well as maintaining it.
> 3 votes

---
Tags: terminology
---


title: Assertions and constraints

text:
# Question
Title: Assertions and constraints
I'm constructing a template to build a configuration file, and the service that consumes this file places constraints on identifier lengths.

If an identifier is longer than, say, 6 characters, the service will get part-way through applying the configuration, fail, and leave the node in an inconsistent state.

How can I perform an assertion to trigger a deployment transaction failure, preventing the target nodes' service from being misconfigured?

My particular circumstance is Salt, but I would be curious to see how other systems solve the problem as well.

# Answer
In Ansible: you can use `assert` or `fail` module.

```
- name: "Make sure web_sites is dictionary"
  fail: msg="web_sites should be dictionary"
  when: web_sites is not dict

- name: "cluster_name should be shorter than 6 chars"
  assert:
       that: cluster_name|len <= 6

```

In Puppet: there is fail function evaluated during parsing phase which cause parsing failure on server (see question on StackOverflow)

```
 if length($cluster_name) > 6 {
      fail("Cluster name is too long. Should be less than 6 chars.")
 }

```
> 7 votes

---
Tags: configuration, configuration-management, linting
---


# Question
Title: How can I develop and support mainframe software without going broke?
Mainframe (zOS) software is still very much in use right now. Many of the most critical applications of banks, insurance companies, airlines, global companies, etc are running on (good old) mainframes.

However, for companies that are in the business of developing and supporting mainframe softwares, it may be (terribly) expensive to have the required infrastructure for developing, testing and supporting such software.

What are possible solutions for developing and supporting mainframe software without paying fortunes for the required infrastructure (and spending all software revenue to it)?

# Answer
One possibility would be to find a 3rd party business owning such infra and selling on-demand usage/access time/connectivity to it as a service for more convenient expenses. For example Mathru Technologies (found through plain google search).

Another possible approach is to obtain such services from strategic partners or maybe even your very customers. Maybe even for free, depending on the quality of your relationship.

You could also ask the mainframe community for advice and/or leads in this direction.
> 5 votes

---
Tags: mainframe
---


# Question
Title: How are you enforcing git behavior, including locally (particularly on Windows)?
I'm taking point on moving this .NET shop from svn to git, and have identified some ancillary issues I'd like to have a solution for before we flip the switch.

The one I'm asking about in particular in this question is line-ending enforcement. By default git for windows installs with the 'checkout crlf, commit lf', which won't work for a bunch of source that is (as far as I'm aware) exclusively comprised of crlf endings.

I don't know that I'd blindly trust any given dev to configure this correctly even given instruction, so I'm considering one (or both) of the following but was curious if anyone here had gone another route.

* A pre-commit hook that checks for any lf line endings (or maybe all lf line endings), and rejects in that event.
* An install script distributed to devs that populates the global config with the 'as-is, as-is'.

P.S. While writing this it occurred to me that the initial conversion from svn to git could commit the default way and as long as people stuck to the default that would be pretty seamless as well. Having been a dev using git in a .NET shop that installed git with the non-default 'as-is, as-is', I've created my own issues there too (they'd all rolled default prior to my arrival). So I'm still leaning towards some sort of enforcement mechanism.

# Answer
To answer the question of how to enforce something locally, you can't without doing some very heavy lifting around managing and enforcing the state of every developers workstation, and I'm usually of the opinion that developers should probably be local admins on their development machine because if they aren't then they will just spend their time figuring our how to get those privileges anyway.

And that's probably because you shouldn't need to care about the state of local configuration when using distributed version control. You should only care about the state of **your** configuration. Assuming you are using git as a centralised version control system, because that's what basically everyone does anyway because its easiest, then let's just assume that "your" configuration is the copy of code saved on the central server.

If this is the case then you shouldn't accept merges that, as you mentioned, break crlf/lf line endings. So you enforce that when some other client tries to push changes you with server side logic that rejects the request to pollute *your* repo with *their* potentially breaking stylistic choices.
> 6 votes

# Answer
We require a review process using Pull requests in github onto our main dev or master branches. During that review process, we will mark pull requests as requiring changes if many files have white space or line ending differences, and insist that they follow the formatting of the dev or master branch they are making the pull request for.

There are also some nice tools depending on which languages you use and which CI tools you use, which can, during the build process or pull request step auto format the code based on rules that you set up. That also helps with keeping code looking consistent and minimizing formatting issues when committing code.
> 4 votes

# Answer
You could use per repository config to override the user's config on a per-repository basis. When done on the repo considered to be the central source it should propagate with clones and pulls to other repos including local ones thus overriding centrally the local config.

You can further enforce this through hooks in your central repo to check that file endings are what they supposed to be and reject the merge/push should it not be kosher. However these hooks will not clone with the repo, least not directly but this is not necessary since the rules only really need to be enforced on the central repo.

Using templates in your git init you can ensure that your repos are created equals with all the gitignore, gitattributes etc files just the way you want them to be.

Last thing, not directly related to your question, I found the biggest friction point when taking svn saavy team to a git based workflow what to get them used to the extra repo in the middle. I found that this visual cue sheet helped a bit explain which command had what effect to which part.

Hope this helped a bit.
> 3 votes

---
Tags: git, scm, windows
---