# Question
Title: Best way of binary repository for .NET projects
I am very to new to .NET projects and I am wondering which is the best tool to store binaries of .NET project. How good to store binaries of .NET projects in artifactory and why? can some one answer generally Any comments can be helpful.

# Answer
I'd recommend ProGet. If you're familiar with Artifactory you might find this comparison helpful: https://inedo.com/support/kb/1115/a-comparison-artifactory-vs-proget
> 2 votes

# Answer
It is wise to think about package management early on.

There are multiple options in terms of package management. The 2 largest are jFrog Artifactory and Sonatype Nexus. Both offer support for multiple formats such as maven, NuGet, Docker, npm, ...

If you only have standard requirements, both of these will suffice. If you have specific requirements, it's best to look at a comparison such as this one to find out which one has got what you need.

Both of these offer free versions with all the functionality you will need included.

When it comes to choosing one: If you have experience with one, i would suggest taking that one. Otherwise, you can't really go wrong with either one, since both are widely used by small and large teams around the world.
> 2 votes

---
Tags: continuous-integration, dotnet, repository
---


# Question
Title: Windows network share application deployment automation
We have a historically grown setup of interface applications that needs to be inventorized and cleaned up. Afterwards deployment should be automated and application versions somehow documented.

My problem is that this setup is on a Windows network share only accessible from my work computer (Windows 7). There is currently no way to change that setup.

Tools that I already looked at are Puppet, Capistrano, Octopus Deploy, RapidDeploy and Ansible. The last one seems to be exactly what I need, but is only available for Linux.

Which tool can be recommended to be used for this task?

# Answer
> 1 votes
Check out Otter it's basically the only Configuration management/Infrastructure as Code tool with native Windows support or Possibly BuildMaster as well which is more on the release coordination deployment automation side of things.

If you're a Windows heavy shop definitely take a look at the tools mentioned on the Winops wiki page \- will probably help point you in the right direction.

---
Tags: ansible, deployment, windows
---


# Question
Title: Downloading multple files from remote using chef resource
I am wondering if there is a resource to download multiple files from remote using chef resource. I want to use:

```
remote_file 'Download remote file' do
  path /opt/
  source http:///xxx
  mode '0644'
  action :create
end

```

...for one file. What if I want to download multiple files in a same place with different urls? Or how can I modify this resource to make multiple downloads which have different URLs?

This is how I added attributes in recipe:

```
node['file']['name'].each do |pkg|
    remote_file "path/plugins/#{pkg}" do
        source "node['file']['url']/#{pkg}"
        action :create
    end
end

```

in attributes

```
default['file']['name'] = %w(
    xx-2.0.jar
    xx-2.jar
    xx.jar
    )

```

# Answer
what about something like that:

```
%w{
        mysql-community-common-5.7.16-1.el7.x86_64.rpm
        mysql-community-libs-5.7.16-1.el7.x86_64.rpm
        mysql-community-client-5.7.16-1.el7.x86_64.rpm
        mysql-community-server-5.7.16-1.el7.x86_64.rpm
}.each do |pkg|
        remote_file "/tmp/#{pkg}" do
          source "https://s3.amazonaws.com/tmp/mysql/#{pkg}"
        end

       rpm_package pkg do
        source "/tmp/#{pkg}"
        action :install
      end
end

```

another way:

```
urllist = {
  { 'url': 'http://some.url1/', 'path': '/some/path1/', 'filename': 'some.file' },
  { 'url': 'https://some.url2/', 'path': '/some/path2/', 'filename': 'another.file'}
}

urllist.each do |urlinfo|
  remote_file "#{urlinfo['path']}/#{urlinfo['filename']}" do
    source "#{urlinfo['url']}/#{urlinfo['filename']}"
    owner 'someowner'
    group 'somegroup'
    mode 0755
  end
end

```
> 5 votes

# Answer
What I do for this case is using node attributes hash for a key value mapping as follow :

In `attributes/default.rb`

```
default['namespace']['files']['file1']='http://server1/path/source_fileX'
default['namespace']['files']['file2']='http://server2/path/source_fileY'

```

Or with hash notation :

```
default['namespace']['files']={'file1' => 'http://server1/path/source_fileX',
'file2'=>'http://server2/path/source_fileY'}

```

Then in recipe:

```
node['namespace']['files'].each do |filename,src|
  remote_file "/path/destinaton/#{filename}" do
    source src
    mode '0644'
    action :create
  end
end

```

This way you can map a file with its source taking advantage that node attributes are a hash structure.

If course you can set the full path instead of just the file name and avoid the string interpolation in the recipe remote\_file.

(Typed on phone, please forgive typos)
> 2 votes

---
Tags: chef
---


# Question
Title: Jenkins cannot reach nodes after upgrade
I just upgraded Jenkins from 2.39 to 2.57. After the upgrade, Jenkins can no longer communicate with my SSH build nodes. I get the following error:

```
<===[JENKINS REMOTING CAPACITY]===>channel started
Slave JVM has not reported exit code. Is it still running?
[04/28/17 01:29:32] Launch failed - cleaning up connection
[04/28/17 01:29:32] [SSH] Connection closed.
ERROR: Connection terminated
java.io.EOFException
    at java.io.ObjectInputStream$PeekInputStream.readFully(ObjectInputStream.java:2335)
    at java.io.ObjectInputStream$BlockDataInputStream.readShort(ObjectInputStream.java:2804)
    at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
    at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
    at hudson.remoting.ObjectInputStreamEx.<init>(ObjectInputStreamEx.java:48)
    at hudson.remoting.AbstractSynchronousByteArrayCommandTransport.read(AbstractSynchronousByteArrayCommandTransport.java:34)
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:59)
Caused: java.io.IOException: Unexpected termination of the channel
    at hudson.remoting.SynchronousCommandTransport$ReaderThread.run(SynchronousCommandTransport.java:73)

```

Based on Googling 'Slave JVM has not reported exit code.' it seems like a Java version mismatch (JENKINS-30561, JENKINS-29120), but I can confirm that my build nodes are using Java 1.7:

```
$ java -version
java version "1.7.0_80"
Java(TM) SE Runtime Environment (build 1.7.0_80-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

```

Does anyone have ideas what could be causing this? More info can be provided on request.

# Answer
I found out via this StackOverflow question which pointed me to JENKINS-27624 which indicates that Java 8 is now required.

Update: I upgraded Java on all my nodes this morning and they are up and working.
> 5 votes

---
Tags: jenkins, jenkins-plugins, upgrade
---


title: Maven command for Sonarqube analysis

text:
# Question
Title: Maven command for Sonarqube analysis
I am planning to build a Maven project in Jenkins. I tried to do some research how to give goals for Maven for doing few steps in one command but cant get understood how to do.

What I mean is how can I give a goal where Maven can build Maven project like creating artifact, skip tests and analyse Sonarqube analysis with particular URL like (`sonar:sonar -Dsonar.host.url=http://url`)?

I tried but I am unable to give all goals at same time. I am planning to give in these three steps in three different goals. Any way that I can give in single step?

# Answer
> 4 votes
Well, you can pass multiple goals to maven in command line simply by passing them as arguments. Skipping test require a java system property that you can also pass as an argument to the command line.

For packaging a maven artifact, executing a sonar analysis with the maven goal but skipping executing the tests you should use :

`mvn package sonar:sonar -DskipTests`

Before that, you need to decide if you want to put sonar settings in the maven configuration (i.e. `pom.xml` or `.m2/settings.xml`) or in the Jenkins server configuration.

* With maven only, the `sonar:sonar` goal needs some requirements and maven configuration that you can find here in the documentation. Jenkins use will be reduced to execute maven, you won't need to install a Jenkins plugin.
* Using Jenkins with sonar also needs some requirements and Jenkins configuration that you can find here in the documentation. Be sure to install the Sonarqube scanner Jenkins plugin. This plugin enable some Jenkins integration features.

---
Tags: jenkins, continuous-integration, builds, build-pipeline, hudson
---


# Question
Title: What are the pro and cons of SnowFlakes Servers, Phoenix Servers and Immutable Servers?
I'm curious about a matrix like comparison on security/ease of management/forensic ability for each type of server. I may forget some more key features of each type also.

I've a general idea about the types but a reference matrix would be of help when choosing between them on some cases (when the automation become complex for the application for example).

To avoid concern about it being too broad, I feel that splitting it in multiples question would scatter the information and that a question about security comparison would need to compare each type also.

# Answer
> 17 votes
The term Phoenix Server was coined by a fellow of Martin Fowler, and all three terms described in short articles on Martin's bliki.

The pros and cons of each such server are described in the articles. The main difference being in the way the server is managed.

Servers exist to fulfill the role of a container for some application(s). Since applications change often, it is often required to change some attributes of the container - such as packages, configuration, etc. It is also sometimes required to change attributes of the container itself because of external reasons, such as security vulnerabilities that require patches to be installed.

There are several ways to change an existing server:

1. Manually create the server initially, then keep changing its content (mutating) each time a change is required.
2. "Bake" an image for a server based on a recipe, usually in an automated way (not manually). Then create servers from that image. And repeat this process on every change.

The former is called Snowflake, while the latter is a practice that allows Phoenix and Immutable server types. Where Immutable states that no changes are made to an existing server once it was created, and Phoenix means that a server is destroyed completely and a new one is used to replace it during the change process.

# Answer
> 10 votes
As I was more thinking of a listing of advantages and drawbacks of each type, here's is my view (not exhaustive, it's the important operational ones in my opinion):

1. **Snowflakes Servers**

* **What they are**: Systems with their specific configuration, no other servers in the data centre have the exact same parameters. They are usually manually administered.
    * **Advantages**:

    + Fitted to the needs of what's running on them.
        + Long-lived, updates are usually shorts.
        + Adapted to special cases where the tweaks are well documented by the product hosted.
    * **Drawbacks**:

    + Sometimes updates leave unused files, the cleanup could be complex.
        + When the changes have to be made to multiples machines it takes a while.
        + Nothing prevents undocumented change.
        + In the case of corruption, you have to rebuild a base OS and restore, some OS tweaks can't be restored and should be reapplied, it's easy to slip over a line and forget an important tweak.
        + Usually long to provision due to the manual configuration.
2. **Phoenix Servers**

* **What they are**: Automatically configured by some code.
    * **Advantages**:

    + Defined by code, version-able.
        + Easily replicated to a point in time.
        + Long-lived, short updates also.
        + Changes to controlled files are documented and can't be forgotten.
    * **Drawbacks**:
    * Sometimes updates leave unused files, the cleanup could be complex.
    * Not everything is under code management, some tweaks by a human can be missed if not included into the automation.
3. **Immutable Servers**

* **What they are**:
        + Automated one-time provisioning from a master image with generally no access.
    * **Advantages**:

    + Defined by code, version-able.
        + Easily replicated to a point in time.
        + Reduced attack surface due to the usual removal of remote access.
        + Fixed configuration, no change can break something
        + Easily scalable 'on demand' from the master image.
    * **Drawbacks**:

    + They are immutable, you have to ensure you can roll an update quickly in case of a 0day flaw impacting you.
        + Not all applications fit well inside this model (Databases, for example, a complete replace on same data is not always possible, there's migration to handle).
        + Brings some new challenges for forensic analysis of crash and log management.

None of those patterns is exclusive, you have to chose the best one according to your actual need. Snowflakes bring a lot of concerns in case of recovery after a disaster so the choice is usually more between Phoenix and Immutable.

# Answer
> 2 votes
All three are patterns of sorts, it isn't is a case of picking and choosing which to use in any specific circumstance but a case of knowing when to recognise the patterns that can help or hurt you.

# Snowflake Server

A Snowflake Server is very much an anti-pattern representing the case when a server evolves in an uncontrolled manner to the point when it cannot be easily reproduced.

I have had numerous run-ins with this kind of server in production, they are fairly easy to spot as there is usually a large number of failed changes and comments such as "it \[the change\] worked in Development/Test/UAT/Staging".

# Phoenix Servier

A Phoenix Server is more of a principal than a pattern as Martin Fowler puts it:

> A server should be like a phoenix, regularly rising from the ashes.<sup>\[a\]</sup>

If you were to apply IT Service Management (ITSM) or ITIL language to the same situation you would likely call it an IT Service Continuity Plan or Recovery Plan:

> A separate plan for each service should provide detailed procedures and step-by-step guidelines for each stage of an incident so that the Recovery Teams are able to restore the services and thereby to meet the agreed process and component RTOs.

# Immutable Server

An Immutable Server or Immutable Infrastructure is the process by which we treat all deployed infrastructure, configuration and code as utterly immutable, i.e. unchanging. When we deploy anything new we spin up new infrastructure and deploy the code to this. Interestingly this mostly satisfies the needs traditionally fulfilled by Evergreening.

---

## Notes

* **a:** Martin's colleague Kornelis Sietsma came up with the term "Phoenix Server" on an internal discussion list.

---
Tags: servers, architecture
---


title: How to improve DRP-testing of SE sites?

text:
# Question
Title: How to improve DRP-testing of SE sites?
Have a look at what's described in the question about "Brief outage planned for Wed, May 3, 2017 at 0:00 UTC, 8pm US/Eastern (like a fire drill for computers)", which is about testing of the existing "**Disaster Recovery Planning**" (= DRP) of the entire family of SE sites.

If you were in charge of this, what would be your recommendations to improve these kinds of DRP-testing in production?

# Answer
> 7 votes
> **NOTE:** It's probably not worth reading too much into the outward comprehension of how good-or-not StackExchange is at managing their disaster recovery scenarios. I suspect they are following much of the best practice below and simply testing scenarios to validate their configuration.

Depending on the environment you operate within:

* A disaster recovery plan might form part of a larger Business Continuity Plan, a Business Continuity Plan would also likely consider operational risks to you people, organisation, locations, information, partners and management systems.
* A disaster recovery plan might be broken down into many IT Service Continuity Plans for individual services. The disaster recovery plan would likely incorporate people and process along with the technical aspects of the service.

Given these definitions, you might consider ways of improving the capability of the organisation as a whole to be resilient against failure:

* **Service Recovery**:

+ Enable individual services to be Active-Active across two geographically dispersed data centres. This does assume that applications are capable of replicating state between data centres, for example using BASE Semantics for data.
    + Create Self-Healing services, this means expecting failure and building with Resilience Engineering in mind. An example is by using a tool such as Chaos Monkey to simulate a failure.
* **Disaster Recovery Plan**:

+ Again enable Active-Active mechanisms across data centres, the difference from SRPs is that capacity needs to be carefully considered, i.e. if you have to DCs in an Active-Active pattern and one fails then a single DC must be sufficiently scaled to support 100% of the traffic.
    + War Games and Rehearsals are really important for disaster recovery plans as this tests the people and the process, in the most mature DevOps environments much of this can be automated as evidenced by Chaos Gorilla.
* **Business Continuity Plan**:

+ On the basis that this is a DevOps site I won't go into the long process of building Business Continuity plans. However, the rules of not keeping all of your eggs in one basket apply - have a plan for when your office is flooded:
        - Allow your staff to work remotely from home one day a week, this tests your BCP strategies.
        - If possible have to geographically and politically separate locations for your workforce.
        - Define and test a clear process for communicating business continuity events and practice them through fire drills.

---
Tags: testing, disaster-recovery-plans
---


# Question
Title: How to get environment variables in ECS container?
What AWS CLI command should I be using to access the environment variables in an AWS ECS cluster's task's container?

I've seen `describe-cluster`, `list-tasks`, `describe-tasks`, `describe-container-instances`, etc... I can't seem to find one that lets me look at environment variables.

Is this doable via the AWS CLI?

# Answer
Environment variables are defined inside the container and some are passed in the task definition. So you can use describe-task-definition to see the extra variables.

To get the default environment of the container image, you can register-task-definition for task with command 'env' and the specific container image for which you want to find out and then run-task this task.
> 10 votes

---
Tags: amazon-ecs, aws-cli
---


# Question
Title: How to ensure equitable distribution of SQS messages in a distributed systems setting?
I have multiple servers, each having a script polling an SQS queue \[all polling the same queue\].

So, is there any way I can ensure an equitable distribution of messages to all those clients \[i.e. my worker servers here\]. Like for example, if there are 100 messages in the queue, then `20-20-20-20-20` if there are 5 workers, and so on.

Can AWS ELB (Elastic Load Balancer) help me do it? If yes, then how? If not, then is there an alternate service in the AWS ecosystem which can help me do it?

**Or am I overthinking this?** I mean, can this be solved straightforwardly in the polling script? \[Please keep in mind the race conditions involved due to multiple clients polling a single queue\]

# Answer
> 6 votes
If there are 100 messages in the queue and 5 consumers, the initial distribution will be no more than 10-10-10-10-10.

A single response can never return more than 10 messages.

This seems like a non-issue.

Race conditions related to multiple consumers should be a non-issue as well. SQS is *designed* for multiple simultaneous consumers.

Use long polls and a 20 second max wait timer and be amazed. (No, a 20 second wait does not delay messages by 20 seconds. It doesn't delay them at all. You sort of need to see it in action to really understand how it works.)

You're definitely overthinking some things, I suspect.

# Answer
> 3 votes
Good architecture in how you use SQS queues will solve your problems. If we assume there are, say, 3 minutes of processing per message, then you can almost guarantee equal distribution of the messages as this is very large compared to the time needed to poll the queue, if you delete the message from the queue only after it has been processed.

Do be aware there is a 12 hour Visibility Timeout limit on any SQS message, so if you dont delete it by then, it will re-appear on the queue. I suspect this is probably not a limitation for you, but do keep it in mind.

# Answer
> 3 votes
Long polling is always beneficial since it results in higher performance at reduced cost for the majority of use cases. Unfortunately, you cannot control the number of messages each worker receives from the queue due to the distributed nature of the queue. But there are some client-side workarounds that can help you in balancing the load for workers.

**So, this is what we did as a workaround for this:**

As one of the workaround, the poller script can control the number of messages each worker receives. A threshold can be set for maximum number of messages each worker can handle. This threshold can be a dynamic value and would probably be `ApproximateNumberOfMessagesVisible` divided by number of pollers/poller scripts. You can then keep the visibility timeout to be any lower value so if all the poller scripts are long polling at the same time, one of the pollers grabs the message, decides it is overloaded based on threshold, does not delete the message, the message goes back to the queue and it could be grabbed by other pollers who still has the capacity to grab the message. The threshold parameter can be fine tuned to meet the application's needs.

---

Also, having failover mechanism would help too, like how the answers in this post describe. However, I can't afford to have failover queues in a distributed architecture, as it would increase the complexity. So, the above workaround was a better idea for my team.

---
Tags: amazon-web-services, amazon-sqs, distributed-systems, aws-elb
---


# Question
Title: What are best practices of deploying docker-compose using Ansible?
Ansible is used to deploy docker-compose on a system and subsequently start the container.

When ansible runs from the local system, the docker-compose file is deployed using Ansible.

When the code is pushed to bitbucket the file content is:

```
../../../../docker-compose.yml

```

and the CI deploys this on the system as well.

**Structure**

```
project/ansible

project/docker-compose.yml

project/ansible/roles/role/files/docker-compose.yml

```

**Discussion**

In order to prevent that the docker-compose.yml need to be changed on two places, a symlink has been created, but that seems to be ignored by bitbucket as the link instead of the content reside in this file.

This Q&A indicates that:

> The data referenced by the symlink is not stored in the repository.

and creating a symlink to avoid code duplication in docker-compose.yml seems to cause the problem.

# Answer
> 2 votes
Instead of using a symlink that will be stored as is in BitBucket one could use:

```
- name: Copy docker-compose.yml
  copy:
    src: ../../../../docker-compose.yml
    dest: /path/to/docker-compose.yml

```

---
Tags: ansible, docker, bitbucket
---


# Question
Title: Jenkins Pipeline downstream job; exit unless it hasn't run in the last x hours
Is there a way in Groovy directly or by integrating Jenkins plugins to have a downstream job complete only if it hasn't been run in the last X hours already?

Let's say I have unit tests, when green, kick off a job to create an image. I want to put an intermediary step between the two that will check to see when the last iteration of the image creation job ran, and if it was less than say 3 hours ago, NOT run the image creation job.

There isn't a way to make this work with cron that I can see, since the upstream job runs at random whenever commits are made, obfuscating any timing control. I can't just schedule the image creation because sometimes we get 10+ commits in an hour, and sometimes there's barely one a day. When that single one of the day does turn green I'd rather immediately create an image for use downstream, rather than waiting as much as 3 hours if I schedule it.

I'd prefer not to make it a 1:1 ratio commit to image only because storing them gets tricky in large numbers, and downstream from the image you'd never need them as frequently as the aforementioned 10 per hour.

# Answer
Took a while and it's a little on the hacky side since I'm no Groovy expert, but I got it working with this;

```
long now = System.currentTimeMillis()

node('jenkins2_dedicated_slave'){

    stage ('check time') {

        sh '''lastBuild=$(mktemp)

        curl -u [jenkins username]:[API key for user] "http://[jenkins URL]/[job name]/lastSuccessfulBuild/api/json" > $lastBuild

        jq -j '.timestamp' < $lastBuild > time.txt'''

    }

    archiveArtifacts artifacts: 'time.txt'
    long lastBuild = readFile('time.txt') as Long
    int finalNum = (now - lastBuild) / (1000*60*60)

    if( finalNum < 8 ) {
        currentBuild.result = 'SUCCESS'
        echo finalNum + " hours since the last build. Downstream job not run."
        return
    }

    stage ('downstream') {

        //run downstream job

    }
}

```

Basically this will

1. Run a curl command on the Jenkins REST API to get the info on the last successful build and store it in a temp file
2. Use jq to parse said info and extract the timestamp field, which is in Epoch time. This is stored to a file.
3. The file is archived to the job and then read into a variable. This timestamp is compared to the current time, (gotten at the top of the job), and if the break between them is less than eight hours the job is "failed" out with a success and error message.
4. If the time is over eight hours, the job continues on to build the downstream job.
> 2 votes

# Answer
To synthesize Alex and Steve Johnson's answers, you could put this at the top of your downstream job:

```
lastBuild = currentBuild.getPreviousBuild()
delay = 2

if Date(lastBuild.timestamp) >= new Date().minusHours(delay) {
    currentBuild.result = 'SUCCESS'
    echo "Less than ${delay} hours since the last build. Job will not run."
    return
}

```
> 4 votes

# Answer
This might work. I used an example X of two hours prior.

```
lastBuild = currentBuild.getPreviousBuild()

if Date(lastBuild.timestamp) < new Date().minusHours(2) {
  <do your thing here>
}

```

You may have to whitelist `Date()`.

Cobbled together from this Cloudbees answer.
> 1 votes

---
Tags: jenkins, jenkins-pipeline, groovy
---


title: Pass string to Jenkins job from Slack

text:
# Question
Title: Pass string to Jenkins job from Slack
In an ideal world our less tech-knowledgeable employees could just use a Slack command to run a job and get the results slacked back to them, negating the use of the Jenkins GUI. I found Start a build in Jenkins using a Slack Command on GitHub, but it's got a few limitations including that it seems to require a 1:1 ratio Heroku instance to Jenkins job, which isn't great for our purposes.

My thought is to have people use a singular command like `/jenkins`, and use parameters after it that will be parsed by one job, and used to kick off others downstream. That keeps us to one slack command / one instance, but still gives us all the flexibility we need.

The issue falls in getting the parameters to Jenkins. I'm picturing something like the following workflow;

1. User types `/Jenkins job2 param1 param2` in Slack. This is calling an umbrella job that will parse the parameters to find out what to run.
2. Command is somehow sent to the umbrella jenkins job. Heroku instance, direct integration, w/e.
3. Umbrella job receives `job2 param1 param2` as a string. (Perhaps as the value of a single parameter, perhaps pre-split into multiple fields)
4. Umbrella job splits the string if it hasn't already been done, and figures out that the user wants `job2` run, using parameters `param1` and `param2`.
5. Umbrella job does a buildjob to run the downstream job as called, which will Slack back to the user their results.

It's steps 2-3 that are tripping me up. **How can I pass a string (with spaces!) from Slack to my Jenkins job?**

If my entire premise is wrong I'm also open to having my frame challenged.

# Answer
> 5 votes
The plugin you found doesn't have a 1:1 ratio between Heroku instance with build jobs since it takes the job name as parameter, so you should be able to run any number of jobs from a single Heroku instance.

As for passing a space in a job parameter, the word split is made on spaces so quoting won't change anything. You could possibly try to use the HTML spacing character `%20` and see if it works?

Personally I would try to use the plugin to run the correct job with the associated parameters directly instead of going through filtering jobs.

---
Tags: jenkins, slack
---


# Question
Title: Jenkins Artifactory plugin not downloading in multi-thread
In my company we're having a bad time trying to download with the Artifactory plugin in Jenkins because we have a lot of tiny files in our repository and apparently the plugin works in a single thread, so it takes a REALLY long time. We tested the JFrog CLI where we can set the number of threads for the download and that actually works great. Still, we really need to use the original plugin.

Is there any workaround for this? Maybe it has something to do with the Artifactory server configuration?

**Note**: We have so many small files because that's how the development is being done in the company I'm afraid. We thought about archiving all the files before we store them but that would probably be non-economical in terms of storage, since every uploaded zip file would probably have a different checksum.

# Answer
> 6 votes
You could try to run the artifactory file upload in parallel if you are using the Jenkinsfile syntax: https://github.com/jenkinsci/pipeline-examples/blob/master/pipeline-examples/parallel-from-list/parallelFromList.groovy

Here is a simpler example to run things in parallel if you only have a fixed number of things you want to do in parallel:

```
parallel (
    "task1" : {
        //runTask1
    },
    "task2" : {
        //runTask2
    },
    "task3" : {
        //runTask3
    },
)

```

This can be enclosed inside a `node`, in which case they would all share the same workspace or each command can define `node` inside the task code to run into another node in which case it would have a different workspace for each task.

---
Tags: jenkins, jenkins-plugins, jenkins-pipeline, artifacts
---


title: Execute the whole Playbook serially

text:
# Question
Title: Execute the whole Playbook serially
Adding `serial: 1` to a Play will execute that one Play one host at a time. Is there something similar for the whole playbook?

That is, all plays will execute only in one host, and when they are finished then all plays will execute on the next host, an so on.

# Answer
> 1 votes
I think Ansible serial works only for the entire play, have you tried to use roles ?

For example:

```
- hosts:
    - web
    - db
  serial: 1
  roles:
    - { role: nginx, when: ansible_os_family == 'Debian' }
    - role2
    - role3

```

---
Tags: ansible, configuration-management
---


# Question
Title: How do I dynamically take variables from previous plays in Ansible?
Example, I have 3 tasks, which are registered `ec21`, `ec22` and `ec23`.

`ec22` runs when `ec21` fails, and `ec23` runs when `ec22` fails.

Now, I have another task below it which is :

```
- name: Add new instance to host group
  add_host:
    hostname: "{{ item.private_ip }}"
    groupname: launched
  with_items: "{{ ec2x.instances }}"

```

where `ec2x`is any of the previous tasks.

So, how do I set `ec2x` here? cause, **Any of my 3 tasks can run**. So, it can be `ec21` or `ec22` or `ec23`. So, how do I dynamically write that in the adding task after them?

# Answer
> 5 votes
It feels dirty, but I guess something like that would work:

```
- name: ec21
  [...]
  register: ec21_result

- set_fact: end_result= "{{ ec21_result }}"
  when: ec21_result|succeeded

- name: ec22
  [...]
  register: ec22_result
  when: ec21_result|failed

- set_fact: end_result= "{{ ec22_result }}"
  when: ec22_result|succeeded

- name: ec23
  [...]
  register: ec23_result
  when: ec22_result|failed

- set_fact: end_result= "{{ ec23_result }}"
  when: ec23_result|succeeded

- name: Add new instance to host group
  add_host:
    hostname: "{{ item.private_ip }}"
    groupname: launched
  with_items: "{{ end_result.instances }}"

```

Basically setting the `end_result` as soon as you succeed a play.

---
Tags: ansible
---


# Question
Title: Reconciling Continuous Deployment with strict change management policies
I'm curious how others architect their DevOps practices in a strict change management environment, such as with a Change advisory board (CAB) approval process.

I understand that automation can improve your audit process by guaranteeing a more rigorous, provable, and repeatable process. But it feels like Continuous Deployment is more or less impossible in such a situation. Since it can take a week or more to get changes approved, you lose the ability to deploy fast and often. What steps do you take to work within these processes short of just submitting change requests and waiting for approval?

# Answer
If you must adhere to the change process, you'll be limited according to the limitations of the change process, full stop. If changes must be approved prior to deployment, you cannot do continuous deployment. If approval takes a long time, you cannot deploy quickly. There's no workaround whereby you can both follow the process and not be impacted by it. That's the cost of following the change process, and it's a cost worth bringing to the attention of the stakeholders in that process.

All is not lost... you can maximize the automation *around* the process, in order to minimize errors; all of the CD steps *except for* the linkage between generating a stable artifact and deploying that artifact to production. That linkage would be replaced by either some kind of user intervention (button, CLI command, etc.), or linked to the record of approval (e.g. when a change request ticket is moved to the "approved" state, fire off the associated deployment). You just have to squeeze as much benefit as you can out of it, while following whatever mandatory process you've been saddled with. It won't make approvals go any faster, of course.
> 7 votes

---
Tags: continuous-deployment
---


# Question
Title: Jenkins Pipeline job; proper quoting for slackSend step
This is probably 100% user error, but I can't get a pipeline job to use a variable as the channel name in a slackSend step.

This uses both the Slack plugin and the Build User Vars Plugin. I'm trying to use `BUILD_USER_FIRST_NAME` to identify which channel to Slack back to (ie, the one starting the build should be the one to get the message).

Got halfway there with this article on how to use the build user plugin in a pipeline job, but substituting the variable is tripping me up.

The relevant part of the job looks like this;

```
stage ('message') {

    wrap([$class: 'BuildUser']) {

        sh "printf '%s' ${BUILD_USER_FIRST_NAME} > name.txt"

    }

    archiveArtifacts artifacts: 'name.txt'
    name = readFile('name.txt')
    echo "name is $name" //works fine, I get Alex

    if ( "$output" != null ) {
        slackSend (channel: '@$name', color: '#36A64F', message: "Job succeeded")
    } else {
        slackSend (channel: '@$name', color: '#36A64F', message: "Job failed")
    }

}

```

I've tried the following in place of the channel bit, all have failed with indeterminate `Slack post may have failed. Response: Invalid channel specified` errors.

```
channel: '@${name}',
channel: '@" + ${name} + "',
channel: '@"${name}"',

```

Expected output is that it would resolve to `channel: '@Alex',`

What's the proper syntax?

# Answer
In Groovy you have to use double quotes to get string interpolation:

```
    if ( "$output" != null ) {
        slackSend (channel: "@${name}", color: '#36A64F', message: "Job succeeded")
    } else {
        slackSend (channel: "@${name}", color: '#36A64F', message: "Job failed")
    }

```

You can also probably do something like this to avoid it entirely and have less duplication:

```
name = '@' + readFile('name.txt')

slackSend (channel: name //etc.

```
> 6 votes

---
Tags: jenkins, jenkins-plugins, jenkins-pipeline, slack
---


# Question
Title: How to remove a non-existing GitHub Ansible role from Galaxy?
According to this documentation it is possible to remove a role from Galaxy, but the following fails:

* `ansible-galaxy login` succeeded
* `ansible-galaxy delete 030 https://github.com/030/ansible-firewall` resulted in:

> ERROR! Galaxy user 030 does not have access to repo 030/https://github.com/030/ansible-firewall

# Answer
> 7 votes
You have to pass only the repository not the full HTTP URL of the repository when you want to delete it from what I've read in the documentation. So the command should be:

```
ansible-galaxy delete 030 ansible-firewall

```

---
Tags: github, ansible-galaxy
---


title: Sensu TCP Connection Issue

text:
# Question
Title: Sensu TCP Connection Issue
I have a fresh sensu/uchiwa stack running on a virtual machine. It is a "standalone" installation - redis, rabbitmq, sensu-server, sensu-api, sensu-client, and uchiwa are all installed on the same machine. I have a single check which the client.json subscribes to.

In short, something appears to be wrong. I logged into the uchiwa dashboard, and there is a warning message that "Datacenter sensu-81 returned: 500 INTERNAL SERVER ERROR".

The `sensu-client`, `sensu-api` and `sensu-server` logs are filled with these messages and only these messages:

```
==> /var/log/sensu/sensu-api.log <==
{"timestamp":"2017-05-11T21:00:34.758243+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:34.758784+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

==> /var/log/sensu/sensu-client.log <==
{"timestamp":"2017-05-11T21:00:35.973060+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:35.974858+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

==> /var/log/sensu/sensu-server.log <==
{"timestamp":"2017-05-11T21:00:37.489540+0000","level":"warn","message":"transport connection error","reason":"tcp connection lost"}
{"timestamp":"2017-05-11T21:00:37.489704+0000","level":"warn","message":"transport connection error","reason":"possible authentication failure. wrong credentials?","user":"sensu"}

```

It suggests that there is an error connecting to the transport layer, so I checked that my transport.json was configured as expected, and it was (the sensu default):

```
{
  "transport": {
    "name": "rabbitmq",
    "reconnect_on_error": true
  }
}

```

So I checked sensu's rabbitmq.json to check that it was configured as expected, which again it was. (Note: I have a line in my hosts file which translates monitor to 127.0.0.1. I am aware of the issue with using "localhost" causing problems with IPv6, so I double checked that the hosts file maps "monitor" to 127.0.0.1). To further confirm that these credentials are working, I turned on the rabbitmq web-based management and logged in successfully using those credentials.

```
{
  "rabbitmq": {
        "ssl": {
      "cert_chain_file": "/etc/sensu/ssl/cert.pem",
      "private_key_file": "/etc/sensu/ssl/key.pem"
    },
        "host": "monitor",
    "port": 5671,
    "vhost": "/sensu",
    "user": "sensu",
    "password": "sensu"
  }
}

```

I also checked that the configuration of rabbitmq itself was as expected (ssl turned on and such):

```
[
    {rabbit, [
        {ssl_listeners, [5671]},
    {ssl_options, [{cacertfile,"/etc/rabbitmq/ssl/cacert.pem"},
                   {certfile,"/etc/rabbitmq/ssl/cert.pem"},
                   {keyfile,"/etc/rabbitmq/ssl/key.pem"},
                   {verify,verify_peer},
                   {fail_if_no_peer_cert,true}]}
      ]}
].

```

I'm at a loss where to continue debugging.

I also checked the api.json and uchiwa.json config files, just to make sure they had credentials which matched, which they do:

api.json:

```
{
  "api": {
    "host": "monitor",
    "port": 4567,
    "user": "admin",
    "password": "secret"
  }
}

```

uchiwa.json:

```
{
 "sensu": [
   {
       "name": "",
       "host": "monitor",
       "ssl": false,
       "port": 4567,
       "user": "admin",
       "pass": "secret",
       "path": "",
       "timeout": 5000
   }
 ],
 "uchiwa": {
   "users": [
    {
        "password": "admin",
        "username": "admin"
    }
],
   "port": 3000,
   "refresh": 5
  }
}

```

Attempting to talk to the sensu api directly from inside the box gives a 500 error also:

```
vagrant@vagrant-ubuntu-trusty-64:/etc/default$ curl -I http://localhost:4567/stashes -u admin
Enter host password for user 'admin':
HTTP/1.1 500 Internal Server Error
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Origin, X-Requested-With, Content-Type, Accept, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS
Access-Control-Allow-Origin: *
Connection: close
Content-length: 0

```

# Answer
Ends up the rabbitmq logs held the answer. I saw that something was wrong, but the messages were so cryptic I couldn't tell what was actually breaking. Turns out it was an SSL issue caused by an old version of Erlang, which is what apt was installing by default on ubuntu 14.04 (erlang version R16B03).

This amqp issue was what pointed me to the solution:

https://github.com/squaremo/amqp.node/issues/224

I needed to upgrade to Erlang \>= 17.5, then it worked as expected.
> 5 votes

---
Tags: monitoring, sensu
---


# Question
Title: Looking for ways to host application in AWS and have it monitored using opensource
I am a newbie in the world of DevOps and have been assigned a task to implement web applications for the customers in AWS and that application monitored (security, metrics and logging) using any of the open source tools. I dabbled with Nagios a little and have gone through the AWS course on security and networking before.

Has anyone done a similar kind of project before? Could anyone share their approach so that it will help me prepare a design document?

# Answer
You can try built-in tools that provide by AWS.

AWS calls them **Management Tools**.

**Amazon Inspector**

> ### Automated security assessment service to help improve the security and compliance of applications deployed on AWS.
>
> Amazon Inspector is an automated security assessment service that helps improve the security and compliance of applications deployed on AWS. Amazon Inspector automatically assesses applications for vulnerabilities or deviations from best practices. After performing an assessment, Amazon Inspector produces a detailed list of security findings prioritized by level of severity. These findings can be reviewed directly or as part of detailed assessment reports which are available via the Amazon Inspector console or API.
>
> — AWS Inspector

**Amazon Web Services (AWS) Monitoring With Nagios**

> Implementing effective Amazon Web Services monitoring with Nagios offers the following benefits:
>
> Increased server, services, and application availability Fast detection of network outages Fast detection of cloud computing environment problems
>
> — Nagios

Some other tools:
> 1 votes

---
Tags: amazon-web-services, nagios
---


# Question
Title: What is the difference between using brackets (`[` `]`) and just specifying the command in the CMD option in Dockerfiles?
Why does `CMD ["mysqld"]` result in:

```
db_1   | 2017-05-14T16:34:11.829163Z 0 [Note] mysqld (mysqld 5.7.18) starting as process 1 ...
db_1   | 2017-05-14T16:34:11.833159Z 0 [Note] InnoDB: PUNCH HOLE support available
db_1   | 2017-05-14T16:34:11.833190Z 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
db_1   | 2017-05-14T16:34:11.833199Z 0 [Note] InnoDB: Uses event mutexes
db_1   | 2017-05-14T16:34:11.833211Z 0 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
db_1   | 2017-05-14T16:34:11.833222Z 0 [Note] InnoDB: Compressed tables use zlib 1.2.3
db_1   | 2017-05-14T16:34:11.833232Z 0 [Note] InnoDB: Using Linux native AIO
db_1   | 2017-05-14T16:34:11.833583Z 0 [Note] InnoDB: Number of pools: 1
db_1   | 2017-05-14T16:34:11.833729Z 0 [Note] InnoDB: Using CPU crc32 instructions
db_1   | 2017-05-14T16:34:11.835500Z 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
db_1   | 2017-05-14T16:34:11.844395Z 0 [Note] InnoDB: Completed initialization of buffer pool
db_1   | 2017-05-14T16:34:11.846532Z 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().

```

and `CMD mysqld` in:

```
db_1   | 2017-05-14T16:35:43.121085Z 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).
db_1   | 2017-05-14T16:35:43.123152Z 0 [Note] mysqld (mysqld 5.7.18) starting as process 8 ...
db_1   | 2017-05-14T16:35:43.126844Z 0 [ERROR] Fatal error: Please read "Security" section of the manual to find out how to run mysqld as root!
db_1   |
db_1   | 2017-05-14T16:35:43.126882Z 0 [ERROR] Aborting
db_1   |
db_1   | 2017-05-14T16:35:43.126905Z 0 [Note] Binlog end
db_1   | 2017-05-14T16:35:43.127142Z 0 [Note] mysqld: Shutdown complete
db_1   |
strength_db_1 exited with code 1

```

# Answer
CMD with brackets is contained within an array. Without brackets, the command is executed as `/bin/sh -c mysqld`. Best practice afaik is to always use brackets.
> 10 votes

---
Tags: dockerfile
---


# Question
Title: Memory being abruptly freed up every day at about the same time
I have a few VMs on Windows Azure that run our ecommerce website, and lately we started using Telegraf, InfluxDb and Grafana to keep an eye on these machines. After a couple weeks of gathering data, I have noticed a weird pattern related to the Memory *Available* metric:

Everyday almost always at the same period of the day, I have noticed that there is an abrupt amount of memory being freed up which, due to my very very very limited DevOp skills, I cannot figure out what is causing this.

Here's a chart that shows this pattern:

My question is: What could lead to something like this? I feel tempted to suspect that a Memory Leak is to blame but...Free memory never drops below 70% and only happens in two of the VMs with the most traffic!

Should I be concerned when I see something like this?

P.S.: I have stared gathering metrics for Private e Virtual bytes for each of the windows services we have running and for the w3wp process...although I have read that this metrics are not very reliable to find out if you have a memory leak, but at least I'll try to get some sort of trend and see if it correlates with the pattern shown above.

# Answer
I think have found out why this graph looks like this.

I am also gathering metrics for the ASP.NET Applications / Errors Total performance counter, and I have noticed that exactly at the same time that a memory available surge happens, the Total Errors metric resets to 0.

According to msdn this counter resets to 0 everytime an application restart/shutdown happens.

This leads me to believe that the cause for this Memory Available sawtooth pattern is due to Application restarts happening.

Here's how my charts look like:

**UPDATE**

This happens because the Private Bytes of the W3WP process hit the limit for a recycle (We have a private bytes limit configured in the app pool). And looking closely to the Private Bytes chart, we can see something abnormal is happening because Memory usage leaps from 650MB to 3.2GB and a few hours later leaps from 3.6GB to 16.6GB! This is when the recycle happens.
> 7 votes

# Answer
I've seen this same "sawtooth" pattern in other systems, in particular a Java-based data tool. Based on your description, I think you're looking at .NET garbage collection (assuming this is a .NET app.) Java and .NET are both memory-managed languages and frameworks that use garbage collection.

A *memory leak* is typically found in frameworks that lack memory management, or in a program on a memory-managed framework that is overriding or confusing the garbage collector.

The fact that these are your highest-traffic servers makes sense. You're seeing the .NET framework allocate memory as needed, then the garbage collector kicks in on a regular cycle and reclaims unused memory using the garbage collection algorithms. Unless you're tracking specific performance issues I don't think this memory usage pattern is a problem.
> 7 votes

---
Tags: azure, windows
---


title: Unable to create databags in chef

text:
# Question
Title: Unable to create databags in chef
I am very new to and trying to secure passwords using databags. Unfortunately, I tried in different ways but unable to succeed. I followed different tutorials first and second. I understood only two steps but later on I was unable to move forward.

1. I created secret key using openssl
2. In link1 step 2 says `knife data bag create passwords mysql --secret-file /tmp/my_data_bag_key` but I don't know where it is creating the databag.

Can some one guide me from step two? I am trying to secure my database password. I got stuck after creating secret file.

# Answer
I'm using the following ruby script to encrypt/decrypt chef data bags locally: https://gist.github.com/anonymous/4923ebeee53fe4d43a1cfb70a2abaadd

to encrypt:

`ruby encrypt.rb PATH_TO_SECRET_FILE < UNECRYPTED_DATA_BAG.json > ENCRYPTED_DATA_BAG.json`

and to decrypt:

`ruby decrypt.rb PATH_TO_SECRET_FILE < ENCRYPTED_DATA_BAG.json > DECRYPTED_DATA_BAG.json`
> 2 votes

# Answer
You can create Chef data bags in two ways - by loading it from a json file on your disk, or by creating a new data bag which you then write content into (the two commands in the doc).

Encrypted data bags are stored on the server, and anyone who wants to read them (a workstation or node) needs to download and decrypt it with the secret key that you pass directly or from a file (see the command flags here).

Once you have stored your content (your secret key from openssl) in data bags, you can access them in recipes with `data_bag_item(bag_name, item, secret)` (again, see the docs for sample code).

Hope this helps!
> 2 votes

# Answer
A data bag is stored on the chef server itself. My chef days are behind me, but you should be able to enter the data bag content either at the create step or the edit one.

For the step 2, just execute the command that you copied:

```
knife data bag create passwords mysql --secret-file /tmp/my_data_bag_key

```
> 1 votes

---
Tags: chef
---


# Question
Title: What is the simplest way to list which attached disks are encrypted (and which not) on AWS?
An auditor is asking for proof that we've encrypted all the disks on our AWS EC2 VPC. I'd like a way to list all the disks and whether they're encrypted or not.

I know I can build a report with the AWS API (or the CLI) - but I was looking for a simpler approach than that, hopefully with the console.

My question is: **What is the simplest way to list which attached disks are encrypted (and which not) on AWS?**

# Answer
> 2 votes
To get the number of non encrypted volumes you can run this command:

```
aws ec2 describe-volumes --region <your_region> --filter "Name=encrypted,Values=false" --query "length(Volumes[])"

```

`length` will return the length of the array `Volumes` flattened by the selection operator `[]` (more details on JMESPath documentation).
As we filter the slection for non encrypted volumes (`--filter "Name=encrypted,Values=false"`) this should allow to demonstrate to the auditor the number is 0 not encrypted volumes.

Same filter can be applied in the console, in the ec2 page, under 'Elastic Block Store' =\> 'Volumes', type `Encrypted : Not Encrypted` to filter the view to non encrypted volumes only. you may add `Attachment Status : Attached` to list only attached volumes.

# Answer
> 0 votes
According to this document you can run command like:

```
aws ec2 describe-volumes --region us-east-1

```

(feel free to set region you use) and search for field in json output, named "encrypted"

---
Tags: amazon-ec2, encryption
---


# Question
Title: Deriving an item value from a structured file in Zabbix
Zabbix's builtin items for monitoring system parameters are reasonably rich, but there are occasionally things it doesn't seem to make trivial to obtain, such as how many dirty cache pages there are.

For example, consider `/proc/meminfo` on your typical Linux system:

```
Inactive(file):  1348288 kB
Unevictable:       18876 kB
Mlocked:           18876 kB
SwapTotal:       8388604 kB
SwapFree:        8388604 kB
Dirty:             39620 kB
Writeback:           948 kB
AnonPages:       6184596 kB
Mapped:          5319216 kB
Shmem:            427020 kB
Slab:             741476 kB
SReclaimable:     633088 kB
SUnreclaim:       108388 kB
KernelStack:       14528 kB
PageTables:       118816 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:    16575460 kB
Committed_AS:   19184632 kB
VmallocTotal:   34359738367 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
AnonHugePages:         0 kB
ShmemHugePages:        0 kB
ShmemPmdMapped:        0 kB
CmaTotal:              0 kB
CmaFree:               0 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:     9692684 kB
DirectMap2M:     7049216 kB

```

What I want is 39620, from the `Dirty` line. With a combination of `sed` and `grep`, this would be as simple as `export field=Dirty; grep -e "$field:" /proc/meminfo | sed -e "s/^$field: \+\([0-9]\)\+.*$/\1/"`; fairly straightforward.

I *could* do this with a UserParameter:

```
UserParameter=proc.meminfo[*], grep -e "$1:" /proc/meminfo | sed -e "s/^$1: \+\([0-9]\)\+.*$/\1/"

```

and then access the value with the item key `proc.meminfo[Dirty]`.

I'd *prefer* not to use a UserParameter for this, though, as UserParameter require me to deploy additional configuration to each node, and also the process spawns consume system entropy. Both are things I generally prefer to avoid.

# Answer
If your Zabbix agent is 2.2 or later, you could use Zabbix agent item `vfs.file.regexp` like so:

```
vfs.file.regexp[/proc/meminfo,^Dirty.*([0-9]+),,,,\1]

```

If your Zabbix is 3.4 or later, you can use a master item to collect the data blob (perhaps with `vfs.file.contents`) and then extract and parse the desired values with item value preprocessing.
> 3 votes

---
Tags: monitoring
---


title: Where to Store Container Configuration

text:
# Question
Title: Where to Store Container Configuration
(I reviewed this question but the one answer doesn't offer practical solutions.)

The general wisdom when it comes to containers appears to be:

***"Make them immutable and identical."***

I take this to mean that *every* container for a given application function (a web server, for example) should be bit-for-bit identical. The code, the *internal* config, and the file system should be built the same way every time for a given version.

This begs the question: where to store the config that makes a given app container a DEV, TEST, QA, or PROD container? If the environment config is stored inside the container, that makes them different. This seems to run contrary to the identical/immutable goal.

A better model would be to somehow keep environment-specific configuration outside the container. There are two methods I can think of to do this:

1. Store environment-specific configurations on a mounted volume, and make the volume mapping a deployment variable
2. Use configuration service such as in the External Configuration Store pattern

My particular use case is for .NET web applications, where the web.config is a file stored with the application.

Both options appear to have potential pros and cons. Is one of these options better, or am I missing a potential best option?

# Answer
we do this with .conf files and loading them via compose

env\_file: /path/to/our.conf

they can also be used with

docker run --env-file /path/to/our.conf ourcontainer:version

In some cases we use the variables defined in our conf files directly, meaning the code we are running reads them direct, in others we need to modify other files on a per environment basis so we use sed or other tools to make changes to a file we have turned into a template
> 1 votes

---
Tags: deployment, configuration, environments, containers
---


# Question
Title: What Puppet modules do include best practices regarding CI and rspec-puppet like Geerlingguy's Ansible roles?
It is possible to test Puppet code using rspec-puppet, but modules that are supported by the community like NTP do not seem to run test regularly like is done for all Geerlingguy's Ansible roles, e.g. Apache.

The aim is to find a Puppet module on Github that shows the best practices regarding testing using rspec-puppet on build using a CI like Travis. When such a module is found one could read and understand the .travis.yml and apply it to custom modules.

# Answer
Go to https://forge.puppet.com and look for modules with a high "Quality Score"

For example https://forge.puppet.com/puppetlabs/stdlib/scores.

Then you can take a look at the Project URL and view it's .travis.yml and the tests in spec/
> 2 votes

---
Tags: ansible, puppet, rspec-puppet
---


# Question
Title: Site monitoring browser-only tool running in frontend only?
Infrastructure monitoring tools like Nagios or much more "low-level" curl are well known in the DevOps community.

But, given I would like just to track some APIs in similar manner just in frontend, what are known more or less mature popular tools or components to support this functionality? i.e. like Nagios but running completely inside the browser (loading static configuration file from server together with the JS libs would be ok).

# Answer
You have a few options for this:

If you can pay for this, use commercial API monitoring solutions like Runscope or APImetrics. They're easy to setup, have no maintenance overheads, and show you detailed data and alerting etc. The problem with this approach is that they cost money, and if you have a miniscule ops budget or this is a hobby project, it can be hard to justify spending on this.

...which brings me to the alternative: roll your own. If you have your own system monitoring and metrics setup (Nagios/Icinga/Sensu, and Grafana etc), you can write check scripts for whatever data you need (availability, response time or latency etc). Some of these tools integrate with time series databases like graphite/influxDB, so you can push the check data to that and use a visualization tool like Grafana to show graphs and charts based on this data (here's a blog post describing Icinga-Grafana integration)
> 2 votes

# Answer
I don't think there are any popular tools for doing this, because it's not a popular way of monitoring. Anything running completely inside the browser will only execute checks as long as it's open in a browser window, so it's not going to be a popular method of monitoring, full stop. It's just not reliable.

That said, it would be pretty easy to hand-roll something like this in a couple hours with simple HTML/CSS/JS. It might not be pretty, but it would meet the stated requirements.
> 0 votes

---
Tags: monitoring, nagios
---


title: Amazon ECS Basic Definitions and Terms

text:
# Question
Title: Amazon ECS Basic Definitions and Terms
I'm a beginner starting to learn about Amazon EC2 and I want to get clear on the following terms, if anyone can supply a basic definition and perhaps an example to tie them together that would be phenomenal:

Task, Service, Container, Instance, Cluster (of containers), State (of a cluster, most odds), ECS.

A partial answer will also be valuable.

# Answer
> 4 votes
AWS provide different services and categories them on base of computation, storage, databases, management, Analytics, Messaging, Developer Tools etc.

**ECS** comes under the computing services.

## Amazon EC2 Container Service (ECS) – Container Management for the AWS Cloud

> Run your applications packaged as Docker containers.EC2 Container Service (or ECS for short). This service will make it easy for you to run any number of Docker containers across a managed cluster of Amazon Elastic Compute Cloud (EC2) instances using powerful APIs and other tools. You do not have to install cluster management software, purchase and maintain the cluster hardware, or match your hardware inventory to your software needs when you use ECS.

Here I will explain the steps to set up the docker based application in ECS and terminology.

1. **Cluster:** A **cluster** is a pool of EC2 instances in a particular AWS Region, all managed by ECS. One cluster can contain multiple instance types and sizes, and can reside within one or more Availability Zones.

This is the first step to creating ECS.before anything you need cluster it can be default or you can create new one.

2. **EC2 Instance type:** In cluster you need to choose the Amazon EC2 instance type for your container instances. The instance type that you select determines the resources available for your tasks to run on.

Amazon Elastic Compute Cloud (Amazon EC2) is a web service that provides secure, resizable compute capacity in the cloud.it's like your PC you can install mongo, MySQL (for MySQL RDS is best), PHP, node or whatever you want in it.While Creating EC2 you have to define which OS you want to use in ec2.You can access EC2 using **pem file** that will be provided by AWS during creating ec2.https://aws.amazon.com/ec2/

3. **Container:** – A container is a packaged (or “Dockerized,” as the cool kids like to say) application component. Each EC2 instance in a cluster can serve as a host to one or more containers. For every task definition, you need to specify container which tells about CPU, memory and other resources.
4. **ECR Registries** Amazon EC2 Container Registry (Amazon ECR) is a managed AWS Docker registry service that is secure, scalable, and reliable. Amazon ECR supports private Docker repositories with resource-based permissions using AWS IAM so that specific users or Amazon EC2 instances can access repositories and images. Developers can use the Docker CLI to push, pull, and manage images.

Once the above environment is ready you can place task or start services.

5. **Task definition:** A task definition is like a blueprint for your application. In this step, you will specify a task definition so Amazon ECS knows which Docker image to use for containers, how many containers to use in the task, and the resource allocation for each container.
6. **service** services manage the task that you have created.once you have created a task definition, you will configure the Amazon ECS service. A service launches and maintains copies of the task definition in your cluster. For example, by running an application as a service, Amazon ECS will **auto-recover** any stopped tasks and maintain the number of copies you specify.

**NOTE**: Elastic load Balancing can be setup before any thing done. when you want to create services you can configure with load balancer.load balancer conttain target group and container which need to be balanced.https://aws.amazon.com/elasticloadbalancing/

7. **Elastic Load Balancing** Elastic Load Balancing automatically distributes incoming application traffic across multiple Amazon EC2 instances. It enables you to achieve fault tolerance in your applications, seamlessly providing the required amount of load balancing capacity needed to route application traffic

# Answer
> 1 votes
* **EC2** (Elastic Compute Cloud) Instance - A Virtual Machine server
* **ELB** (Elastic Load Balancer) - A service in EC2 which manages incoming traffic using DNS and managed instances (not seen to the user). Redirects HTTP/TCP requests to instances registered in the ELB. Includes health-check mechanism to include/exclude EC2 instances.
* **ALB Target Group** \- A group of EC2 instances, usually containing duplicates of a service listening on a specific TCP port. Includes health-check mechanism.
* **ALB** (Application Load Balancer) - A new generation of ELB. Allows to map multiple URL paths, and `Host:` headers to Target Groups of EC2 instances.
* **EBS** (Elastic Block Storage) - An attachable disk for EC2 instances
* **ASG** (Auto Scaling Group) - A mechanism to create/increase/decrease the number of EC2 instances in a group.
* **Launch Configuration** \- Template describing how to create EC2 in ASG.
* **Image** (Usually Docker Image) - A special format of file you can "pull" and "push" to a Docker Registry. Contains "user-space" executables, libraries and files.
* **Container** (Usually Docker Container) - An execution of an Image, running under a Docker Daemon service.
* **ECS** (Elastic Container Service) - Orchestrator for Docker containers in EC2.
* **ECS Cluster** \- Collection of EC2 (usually in ASG) that ECS uses for running containers.
* **ECS Host** \- One EC2 instances out of all the instances registered in ECS Cluster.
* **ECS Task Definition** \- A template on how to run a specific container. Includes most of the arguments/parameters you can find with `docker run` command.
* **ECS Task** \- A running container somewhere in the ECS cluster. Usually used to run containers on a schedule, or one-offs.
* **ECS Service** \- A service that manages running of ECS Tasks. Can also register listening ports of containers in a specific ALB Target Group.
* **ECS Agent** \- Opensource service, running as a container on each ECS Host. Responsible to receive commands from ECS service regarding containers to run/stop/etc...
* **ECR** (Elastic Container Registry) - A managed service in ECS used to store/push/pull Docker images. Similar to Docker Hub, Quay, and other artifact storage services used to manage images as a service.

---
Tags: amazon-ecs
---


# Question
Title: Pros/cons of discontinuing a DevOps workflow?
I'm trying to evaluate whether or not it is a good idea to move away from a devops-style workflow to the traditional dev-then-ops (not sure what you call that).

We are a small 5 person department tucked away within a 4000 employee traditional media (eg non-software) company. Two years ago we began building software to allow our department to significantly scale up our production. We've been pretty successful and the greater company is beginning to take notice. To date, we have been solely responsible for the design, development and deployment of what has become a ~10 service AWS microservice platform. Our team doesn't identify as DevOps, but without question we are living the DevOps life, with each developer intimately familiar with both the code and the system it runs on.

One of the questions we will face shortly is what "efficiencies" are shared between us and the IT department for our parent company. Our project owner usually prefers outsourcing over in-house learning, so in our case these efficiencies probably means getting as much IT work "off our plate" as possible. Currently, I would say our team has a 70/30% split between experience in coding vs infrastructure. The IT department is solidly in the IT realm, with no visible crossover into software development.

Our project owner (a non-technical individual) is hoping that by handing off as much work as possible to the IT team we will see a ~1:1 boost in productivity for each hour of operations work that we shed. I'm skeptical about this though. Our product is still pre-beta (despite already being a significant business asset) and in our limited experience with the IT department there are usually significant delays for things as simple as filesystem permission changes.

Right now, my ideal solution would be for the IT department to "adopt" us and allow us to continue deploying our own work, while ensuring that we meet the standards & requirements of the IT office. I'm not sure how realistic that is though. Plus it is nearly the opposite approach our project owner is advocating since it would add additional operations work in the short term.

In our situation, what are the likely pros/cons of staying with DevOps approach vs handing off IT?

# Answer
> 10 votes
***It's not a good idea.***

In my experience you'll gain the disadvantages of both while any projected advantages will somehow fail to materialize.

Itemized:

1. You will lose speed.
IT will comply with their own standard. The new task (for them) will follow the same 'sluggish' template all their work now has. Be prepared they will find it challenging - so even less speed than standard simple actions.
2. You will fail to offload.
IT will lean on you guys for every single anomaly. You will put in effort to get one guy up to speed - and the next thing you now you'll be repeating yourself because the following task/problem/day there will be a new guy, again.
3. Documentation will be required, but it will not help.
Again template behaviour will be that short manuals will fail to catch every anomaly, and thorough texts will not be read for being too long. So any investment here will be a loss, likewise the enormous effort needed to implement improvements to get your tools to 'shrink-wrapped'quality.

Last but not least any problems will reflect on you guys. Tar, tarbrush principle.

If the above sounds cynical, well, I'm afraid I've been there. Repeatedly.

***What to do instead?***

Go shopping in the IT department, find yourself a useful candidate, and keep this guy 'on loan' to relieve you workload.

# Answer
> 6 votes
You can find many of the answers in the result of DevOps survey which you should ask the product owner to read. This is a document written specifically for business people with little technical knowledge talking in the terms he should understand.

At average you will need 1 extra developer for every 4 people to keep same level of feature development (38% vs 49% time spent on new work). Your mean time to recover from failure will drop up to 25 times. Your work will be 20% less enjoyable and you will be 40% as likely to recommend your work to a friend. Just those three facts should be enough.

# Answer
> 4 votes
What you will lose by fitting into the IT organization is the "Dev" part of your little DevOps team. When teams become segmented into artificial roles of NetOps, SysOps and Dev, you introduce the following problems:

1. **Un-needed red tape and isolation** \- To do anything the developers will have to submit a ticket to IT and wait for them to implement it. They will no longer be able to implement and interact with it themselves - up to and including their Dev and QA instances which will limit what infrastructure they can code out. They are stuck at the VM barrier instead of being able to code on the full stack. If what they submit to IT looks like DevOps code, they will be ill-equipped to handle it and might revert back to manual deployments.
2. **Neglect** \- Alternatively, they may just deploy it as is and then neglect the beast because they don't know how to interact with it - and they aren't developers, so code is not their problem.
3. **Outages** \- One of the oft overlooked benefits of DevOps is it's programmatic nature. Sure, it might take longer to deploy that server by treating it as code, but this automates out human mistakes. The way it went into dev is they way it will go into QA/Test is the way it will go into prod, thereby reducing outages. When developers lose access to the network equipment they need to deploy their service or the compute infrastructure not only does it take longer, but it introduces more fallible humans which will cause more outages.
4. **Documentation** \- In some senses, DevOps code is self docuementing. You know how the server was built and deployed because the code tells you. In 5 years, when it is time for an upgrade to CentOS 8 or whatever, no one will know how to deploy your application anymore - including at the network, storage, monitoring and backup layers.

In short, you should suggest that your project owner take the time to read The Mythical Man-Month to disabuse him or her of the notion that you will see a 1:1 relationship in productivity and The Phoenix Project which is a good novelized (and entertaining) illustration of what is gained and lost by using DevOps in non-technical language for non-technical people. If the project owner has any sort of commute an Audible audiobook of The Phoenix Project is available.

# Answer
> 3 votes
I would suggest that you adopt some of the IT team, and give them thorough training in the new system.

Once they understand the system fully then it makes sense to offload it to them.

Otherwise, you will become a Support Center for IT - and spend a lot of time firefighting as they learn the intricacies of the new system.

---
Tags: process, operating-model
---


title: User management using chef

text:
# Question
Title: User management using chef
I need to install a jenkins on a server using a user jenkinsuser and group jenkinsgroup. so in my recipe I defined as

```
#group
group jenkinsgroup do
  system true
end
# creates user
user jenkinsuser do
  gid jenkinsgroup
  system true
end

```

but what will happen if the user and group exist already in node (where I am trying to install jenkins) Does my resource do nothing as a user and group exists and go for next steps or will it throw an error as user exists?

# Answer
It does nothing. A huge part of the purpose of Chef is that converges are *idempotent*, meaning you can run them repeatedly with no cumulative effects. I highly recommend consulting the Chef documentation as it will provide an excellent foundation for understanding how and why Chef works.
> 6 votes

---
Tags: chef
---


# Question
Title: Is there evidence to suggest that the Docker/Hashicorp ecosystem requires a higher calibre of software development engineer?
Our build system at work is a series of Go pipelines with go agents running in docker containers on a docker cluster. It compiles java and runs tests. This is in a financial services institution.

We're using consul and registrator for service discovery and vault for secret storage.

I've experienced some pushback in my workplace about the perceived complexities of using docker for running compiles and tests. To counter this I've written lots of documentation, and run training courses.

There are some comments online that docker is complex.

In the Adrian Cockcroft videos, he mentions that in some cases they had to hire Google-level engineers, to wrap their heads around the concepts of distributed computing. In another video he mentions that it took them seven years to achieve their vision of containerisation.

My question is: **Is there evidence to suggest that the Docker/Hashicorp ecosystem requires a higher calibre of software development engineer?**

# Answer
I apologize for the novel length response, but I want the industry to start having these discussions as the potential advantages are huge.

The problem is not that it is necessarily more complicated but that it requires experience that is difficult to find today and which goes against traditional models.

This is not an uncommon occurrence in the computing industry, and I personally remember a time when TCP/IP was considered too complicated. The talent base will grow over time though.

Here are the main barriers I see today:

Tools:

On the platform side most Engineers are just really starting to leverage automation. Unfortunately they are leveraging configuration management tools which were designed to solve a different problem. As an example Puppet and Chef assume idempotency and long lived systems. A large portion of the industry is just starting to get good at those tools but they are inappropriate for a model where dynamic istanciation is the rule and long lived systems are rare.

Most of the monitoring tools or management tools in use also assume long lived systems do not work well in a dynamic environment.

Then there is the whole *we must have a single source of truth* problem that I will ignore, however it is a significant barrier.

Virtualization:

This is mostly an issue with vendor lock-in and market forces. But many SE's have been separated from the simple mechanics of virtualization that their is a knowledge gap. These individuals are capable of the work but have spent most of their time managing vendors so understanding concepts like linux namespaces or c-groups is almost foreign.

Networking:

Docker and Kubernetes really need network segmentation, and tenant isolation. While this happens at a google scale most companies are using fragile and expensive *enterprise* solutions. These tend to either have no automation or massive centralized management. Really each swarm or cluster needs it's own segment and the technologies like vxlan or sdn that enable this are just popping up.

Security:

Related to the lack of overlay networks, or shared VPCs between swarms and clusters across security domains this is a very hard nut for companies to crack right now. Most security teams have no way to figure out how to audit these systems. Most teams I see are trying to fix it with solutions that don't directly address the problem. Segmentation and limited super user access will improve security in the long run, but like most humans they can't really parse json policy files with ease.

Related to the knowledge gap in virtualization, most team do not understand the attack surface involved here. Mix this in with a tenancy to give into the need for super user access this is an unsolved problem for organizations. As an example the default AWS VPC limit per region pretty much means anyone running kubernetes in AWS today most likely has serious security issues and any breach will be massive.

This is less of a problem on GCE but it is a very real concern today. This is far from an unsolvable, but it is a real problem.

In closing:

With all of those negatives I still think it is one of the most promising directions looking forward. But our industry today is full of companies that struggle to even implement OpenStack, which is really just a a bunch of python wrappers, a message bus and a database.

It will be a hard sell for a while, but the advantages of true loose coupling will probably win out in the end.
> 4 votes

# Answer
Your probably looking more at the calibre of your platform engineers. You don't want to make developers jobs hardware nor do you want your software engineers to need to handle more complexity, it's simply contrary to your goal.

Instead it sounds like you are looking for a platform engineer who is able to wrangle the relevant technologies together so that they become transparent to the user's (devs) as the devs becomes more comfortable using them they will be able to stretch out and use some of the more unique features, but until then, providing some abstraction from the underlying complexity is probably a good idea.
> 1 votes

---
Tags: containers, learning, team-role
---


# Question
Title: How do you monitor Node Micro services running inside of Docker Containers?
This article about "How healthy is your Dockerized application?" explains the trouble with monitoring, but it doesn't provide any good examples of how to actually monitor a microservice inside of the docker container.

We are currently using PM2 monit to monitor our microservices, but as we put them into docker containers we lose the ability to access this data within one screen for all the various microservices which each run in their own docker container.

Dockerswarm monitoring will tell us the state of the containers, but not the microservice running inside of them.

What's a solid proven way of solving this problem?

# Answer
> 4 votes
This is an area where Kubernetes has the correct model, there should be a load balancer between all systems which should have functional health checks.

Once you start to add Nagios, Zabbix or other types of monitoring to the system you start to build a large state machine. This will break the loose coupling model and introduce inter-dependencies that inhibit the ease refactoring. While not set in stone the key differentiation between microservices and other variants of SOA is this loose coupling.

If the services are fine-grained and perform a single function, implement a health check at an upstream load balancer, then monitor the active pool members.

As an example in HAproxy

```
backend myapp
[...]
option tcp-check
tcp-check send GET\ /health HTTP/1.0\r\n
tcp-check send Host:\ foo\r\n
tcp-check send \r\n
tcp-check expect rstring ^HTTP/1.1\ 200\ Ok
tcp-check expect string container\ Good
server srv1 10.0.0.1:8080 check
server srv2 10.0.0.2:8080 check

```

In theory you don't care about the performance of an actual container, just that your overall performance is good.

This method makes it easy to have the system self repair and to scale with a minimal amount of complexity.

Basically you only have to check if the number of systems you expect are alive, and if not you spin up some more. If you need to add capacity you simply change the number of expected nodes.

This also simplifies refactoring as you only need to replicate or modify this test with no external dependencies or state machine.

It should also reduce down time and middle of the night Pagerduty alerts as the system self repairs.

As for the overall systems metrics, which are needed to trace down issues like latency I would want them in a central location using a tool like elasticsearch. If you use syslog, logstash or log4??? to collect metrics that will be far more useful in the long run. When systems are small and simple traditional polling based monitoring may provide enough metrics but it is preferable to have them in a format that is searchable and more importantly relational to other systems.

Solutions like monit still have their place, but it is to monitor the long lived components like the VMs or bare metal hosting your swarm, but the containers themselves should be decoupled from that system to get the most benefits from a micro-services model.

# Answer
> 4 votes
One commonly used solution(not container-specific) is to build a health check API within your service which tests all the functionality you want monitored (say availability of DBs and other dependencies) and the app itself, and returns some expected output (say \< service \>: \< status \>). You can then trigger alerts from a monitoring service like Nagios if this API does not return an ok for all the services. This will also fail if the microservice itself is unhealthy.

This approach also has the benefit of running a functional test of your service (by hitting an API end point).

This approach does not cover some edge cases though - eg. the microservice runs (but particular APIs fail).

---
Tags: docker, monitoring, microservices
---


# Question
Title: What is the best way to do REST with puppet?
I have tried making REST calls with puppet and it was... well... awful. I didn't really want to use pure ruby to interact with my load balancer and my concern was that for REST calls, every time the puppet run occurs I would be idempotently slamming my web server with a ton of requests - first checking if a particular item is out of spec and then possibly resetting it. Using curl via an exec and an unless statement was arduous to say the least.

Furthermore, the idempotent philosophy of puppet doesn't always play well with REST APIs that are not idempotent.

Are there any simple ways, ready-made methods or plugins for dealing with REST APIs via puppet?

**Note**: one feature request to watch: https://tickets.puppetlabs.com/browse/PUP-6658?filter=-2

# Answer
> 1 votes
While it may not be the answer you are looking for, puppet's DSL lacks verbs on purpose. Puppet's idempotent philosophy is a huge reason it is so useful for some use cases but for this type of need you will want to use a tool which is more targeted to that role.

When you hit limitations due to the idempotent philosophy of Puppet it is a good indication that another tool is required for that need.

I would look for publicly maintained tools for the task maybe in Ansible or Terraform.

We leveraged the "puppet kick" functionality of puppet and then tried to extend it after it was deprecated and the pain of trying to ignore their design philosophy is an expensive one.

---
Tags: puppet
---


# Question
Title: Assign DNS name to Docker Swarm containers with public IPv6
I'm trying to understand how I would set up a Docker container (running in Swarm mode) with publicly available IPv6.

I know Docker Swarm uses mesh routing, but I don't quite understand how I can dynamically assign DNS names to addresses.

As an example, let's say I want to run a Git server (with HTTP 80 443 and SSH 22) on `git.example.com` and a regular web server at `example.com` (with HTTP 80 443). Both have public IPv6 addresses, but I don't want to hard-code the IPv6 address (for example at CloudFlare). After all, this is DevOps!

How would I go about doing this? I assume I need to run my own DNS server for this.

**Note**: After doing some more research, it seems this is possible with the Docker HTTP Routing Mesh (HRM) which is unfortunately only part of the Enterprise Edition. Does anyone know a similar, open-source approach to this?

# Answer
Assuming that "publicly available" is on the public internet it will be difficult.

DNS isn't really that reactive to changes and is challenging to run in a secure fashion on the public internet.

It also depends on what protocol you are running, because of an oddity around SRV records where pre-existing protocols were excluded from supporting it in the RFC.

But in general the public and the private solution I would implement are similar with an additional layer for a public internet solution. (assuming no use of a 3rd party or geo specific answers)

On the internet edge use Unbound DNS server on the edge, and have it relay queries to consul which would handle service registration.

If this is for private usage this would allow for some data-center awareness and fail-over.

If it is just for internal use you can just use consul and either configure your internal DNS servers to delegate subdomains or forward to the default domain.
> 3 votes

---
Tags: docker, docker-swarm
---