# Question
Title: What are the measures of size and complexity of IT infrastructure
How can I compare different infrastructure in regards to size and complexity. What can I measure and compare e.g. number of nodes, number of servers, architectures etc.

How do those measures and variables differ in what they compare? Which of them make sense to compare and what is the cardinality at which there is actual difference in the kind of work done, rather than just a bit of extra work.

What will let me decide that "*This infrastructure is very large*" or "*This infrastructure is very complex*" and what is the difference?

# Answer
> 8 votes
> **TL;DR:** Measures of **size** can be split roughly in three different categories that I would define as **depth** in what you manage vs. what you outsource or consume as a service, **breadth** of services supported and **height** in numbers of instances,servers and customers. Measures of **complexity** largely depend on selected **systems architecture**, **organizational structure** of people supporting it and **skill sets** required. Where depth and skill sets go hand in hand adding both to size and complexity.

# Size mesures

Note: *Most of the following size requirements can also add to complexity through systems architecture, skill sets and organization structure needs.*

## Depth of Infrastructure

How much of the infrastructure you are outsourcing to others in increased levels of depth:

1. Do you simple use Software as a Service for all that you do?
2. Do you operate fully in public cloud, private or hybrid cloud or using some PaaS?
3. Do you use Infrastructure as a Service?
4. Do you use Hosted and Managed Infrastructure in rented DC space?
    * Do you own or rent the hardware?
    * Does the provider manage infrastructure monitoring?
    * Does the provider manage basic system administration?
    * Does the provider manage hardware failures and maintenance?
    * Does the provider manage racking and server installation?
    * Does the provider manage internal networking?
    * Does the provider manage internet connectivity and routing?
5. Do you have just data centers with contracted remote hands?
6. Do you host everything on premises or in your own datacenters?

## Breadth of Infrastructure

What are the different types of services you support?

* Compute Resources
    + Bare metal servers
    + Virtualization layer (VMWare)
    + Container layer (docker, k8, mesos)
    + Serverless layer (lambda, functions)
* Storage Resources
    + Standalone storage appliances
    + RAID in servers
    + Large standalone relational database clusters
    + Time series databases
    + Object store clusters
* Network Resources
* Observability Resources
    + System Log servers
    + Metrics and Graph systems
    + Search clusters
* Automation Resources
* Backup/Recovery Resources
* Complex compounded services
    + ELK, Hadoop, etc.

## Height of Infrastructure

* What is the scale for each resources that you need? Do you operate the services on single server/instance or do you need use clusters of machines?
* What is the level of redundancy you need?
* What are your availability requirements?
* What are your requirements for latency and throughput of services?
* Do you need geographically distributed infrastructure? (international business, latency requirements or regulation compliance like GDPR, data localization laws, etc)
* Do you need multiple datacenters in each geography?

# Complexity Measures

Just very briefly...

## System Architecture

When it comes to Infrastructure complexity, it quite closely follows the complexity of the distributed systems supported by the infrastructure. You have to take into account two types of systems:

1. Distributed systems supporting individual services.
2. Distributed system created by interdependencies of the services.

### Distributed Systems Complexity

Every service your infrastructure supports can have in and of itself a different level of complexity with varying levels of requirements on the infrastructure. Systems supporting services can range through:

* Single threaded.
* Multi-threaded (shared memory, shared disk)
* Parallel systems with data sharding
* HA Failover (Primary / Standby) (Cold, Warm, Hot)
* HA Cluster (N+M)
* Real-time Clusters

### Interdependence of Services

Let me start with an example. Let's say your infrastructure reports test results into ElasticSearch cluster. Your pager depends on monitoring and test data provided by ElasticSearch. Geographical distribution of ElasticSearch cluster makes it depend on your datacenter networking connectivity. Now one of your internet providers decided to make an unannounced maintenance on Saturday night, throughput drops, your traffic is rerouted to backup provider, monitoring traffic is de-prioritized to customer data traffic, ingest of monitoring events slows and your pager goes crazy.

Every time two services, two parts of the infrastructure depend on each other, they create a new single distributed system, whose complexity should be judged independently. Such dependency can be either removed or reduced. Remember that system is only as redundant and as available as the intersection of all services that it depends on.

Further examples of complexity increasing factors:

* Dependencies on external services.
* Attempts to mitigate failure from service dependency.
    + Multiple providers
    + Caching of data

## Organizational structure

This is a chapter in and of itself ... people are often overlooked part of the overall system of IT infrastructure. We rarely think about redundancy, availability and latency factors when it comes to people, but just as with computers, these same issues affect organizations maintaining infrastructure and the complexity of it can sometimes easily outweigh the complexity of the computer systems. People involved in maintaining your infrastructure can span multiple time zones, languages, geographic locations, companies, pay scales and legal codes. Any of those factors are signs of increase in complexity.

---
Tags: terminology, infrastructure, complexity
---


# Question
Title: Is there a CI tool that guarantees no regressions in the branch quality level?
Traditionally CI systems only perform **monitoring** of the quality levels in an integration branch, by performing QA verifications on the codebase where the changes are already committed, watching for regressions and sending notifications for human intervention.

But when these regressions are detected the branch has already been in trouble at least since the respective QA verification started and will remain in such state (or even get worse!) until all the culprits are identified, repairs for them committed and a new QA verification confirms the branch quality level was restored. The branch can be considered blocked for normal development during all this time.

Is there a CI tool capable of actually **preventing** such regressions from happening, which would perform *pre-commit* QA verifications and allowing commits only when the codebase updated with the respective commits would be passing those pre-commit QA verifications as well, thus guaranteeing a minimum branch quality level?

**Update:** the assumption is that suitable automated QA verifications with appropriate coverage to be able to detect the respective regressions are available for invocation by the CI tool(s).

# Answer
ApartCI is a CI system designed exactly to prevent regressions, thus guaranteeing flat or increasing branch quality level. Still in beta.

It orchestrates centralized pre-commit verifications in such manner as to ensure that a change is committed only after it is verified, together with all other changes committed before it, to meet or exceed the latest branch quality level.

This is the key difference compared to traditional developer-driven pre-commit verifications, often done in parallel , which leaves room for regressions caused by interfering changes which were never tested together.

The tool is also designed to easily scale \- capable of sustaining very high rates of incoming candidate changes and supporting 100s/1000s of developers working in the same integration branch.

**Disclaimer:** I'm the author of the tool and founder of the company offering it. Apologies for the ad.
> 0 votes

# Answer
As far as I can tell, you're looking for a tool that will **reject** commits that break the build—a CI tool probably won't be able to prevent regressions by actually fixing your code, but it can stop you from adding bad code to the repository.

Atlassian has a few interesting applications of Git hooks:

> Server-side pre-receive hooks are an especially powerful compliment to continuous integration because they can prevent developers from pushing code to master, unless the code meets certain conditions – like elite ninja guardians, protecting it from bad code.

If you're using Git, the hooks are **very** powerful (and there are similar hooks for SVN, Mercurial and other version control systems), and you might find it useful to use them to run pre-commit checks.

The Git documentation has a page on creating a hook to reject pushes if they don't meet a certain criteria which could easily be adapted to this use-case.

However, a lot of people would argue that rejecting commits is a bad idea on a `feature` branch—you'll just be wasting time fighting against your CI system when the build breaks for some reason, instead of actually fixing the bug.

On the `master` branch, it could make sense to reject broken merges, because you might want to ensure it always builds. For a `feature` branch, you **will** inevitably break things, and since the code isn't going into production **now**, it makes more sense just to warn you than actually reject your commit altogether.
> 6 votes

# Answer
No tool could possibly *guarantee* no regressions - that depends much more on your tests than the tool executing them. However, you can help to prevent regressions that will be caught from entering into the integration branch. You can do this with pre-commit hooks, but it's often easier with pull requests (which hopefully you're already using for peer code review).

If a branch is up-to-date with its upstream (where the PR is merging to), and its tests pass, then they will still pass after the merge; the state of the target branch after the merge will match the state of the source branch before the merge.

It's generally not particularly difficult (depending on the tools used) to indicate both whether the source branch in a PR is up-to-date with the target, and if it has a passing CI build. You can use this as a requirement (by policy, and/or enforced in software) for merging the pull request.
> 2 votes

# Answer
True continuous integration tools (as opposed to just continuous testing) like Reitveld and Zuul can help, though they are only as good as the tests you write and code reviews you do.
> 1 votes

# Answer
Use GitLAB, you can set in project-settings to only allow a merge when the pipeline succeed, so can have a truly Continuous Integration, combine that with adding your QA to the list of merge approvals and with Dynamic Environments, you can have quality assurance before you merge to the master.
> 1 votes

---
Tags: continuous-integration, software-recommendation
---


# Question
Title: Blue/green deployment with micro services and DNS
Currently, we have on our test environment an *ECS Cluster* with micro services deployed. All micro services are talking to each other through *Route53* DNS CNAMEs like service1.company.com, service2.company.com, etc.

Here is a simplified schema of the infra:

The *ALB* listens on port 80 and everything works fine.

Now comes the troubles: I'm trying to build a blue/green deployment infra.

I basically know how to do it (*HAProxies*, 2x *ALBs*, 2x *ECS Clusters*). What I want to acheive can be seen https://www.youtube.com/watch?v=KJOORneBTHI

I'm stuck with the fact that we are using CNAMEs for our micro services communications.

* So on the **blue** part:

CNAME service1.company.com -\> ALBBlue

* On the **green** part:

CNAME service1.company.com -\> ALBGreen

But that's something I cannot do of course.

How can I do?

# Answer
> 5 votes
You should have 3 names when you're willing to do a blue/green deployment.

Two set of names, one for blue, one for green, which will works as is, and a production entrypoint which will point to either the blue name or green name.

The point of a blue/green deployment is to be able to test the full deploy before switching the clients entry point, so your services calls themselves don't have to use the entry point name itself.

---
Tags: amazon-web-services, microservices, amazon-ecs, dns, bluegreen
---


title: Chef syntax with "do" statement

text:
# Question
Title: Chef syntax with "do" statement
While reviewing a chef.io whitepaper, I encountered the following syntax:

```
# Load your AWS credentials databag
include_recipe ‘aws’
aws = data_bag_item(‘aws’, ‘main’)
aws_elastic_lb ‘elb_qa’ do
  aws_access_key aws[‘aws_access_key_id’]
  aws_secret_access_key aws[‘aws_secret_access_key’]
  name ‘elb_qa’
  action :register
end

```

Can you explain or refer to documentation about the "do" statement?

The common example for the Ruby "do" statement is using "each", which cannot be found in the example above. What does the syntax mean? what here is a parameter, variable, assignment, function, etc.. ?

# Answer
what you're looking at is a resource declaration, mostly a class taking a block as parameter.

In ruby the `do code end` syntax is the prefered way over the curly braces syntax `{ code }` which is prefered for inline code.

So mostly, what you have is the same as:

```
aws_elastic_lb('elb_qa'){ aws_access_key aws[‘aws_access_key_id’]; aws_secret_access_key aws[‘aws_secret_access_key’]; name ‘elb_qa’; action :register }

```

In recent version of chef (14.X) the name attribute should be the name of the block (what is just after the resource type (aws\_elastic\_lb)

More details in this SO quastion about ruby syntax.

Concerning Chef itself I highly suggest you follow the tutorials at https://learn.chef.io for a better overview of what is what in cookbooks.
> 0 votes

# Answer
Following Tensiblai's guidance, I found this documentation explaining the parts comprising a resource:

see: https://docs.chef.io/resource.html

> A resource is a Ruby block with four components: a type, a name, one (or more) properties (with values), and one (or more) actions. The syntax for a resource is like this:

```
type 'name' do
   attribute 'value'
   action :type_of_action
end

```
> 0 votes

---
Tags: chef
---


# Question
Title: Multiple Ingress controllers in Kubernetes
We currently have at our company a Kubernetes cluster with an ingress controller that manages the traffic of the services and their endpoints.

We want to change our URL gradually, first add the new URL to point to the same services that are currently being used by the old URL, and once we see that everything works fine, we'll just make the old URL redirect to the new one.

That's the plan in a gist, however, I have no idea where to start. Do I add another ingress controller that is just copied from the one we currently have and update the SSL certificate to match the new URL?

Thanks!

# Answer
> 1 votes
I assume you've created your nginx ingress controller in it's own namespace? By default it's called `ingress-nginx`.

You can create your second Nginx Ingress controller in a seperate namespace so that it's logically isolated from your first. You'll need to give it a different name, but I see no reason why you can't do that.

Doing so will spin up a seperate AWS ELB, which you can then map to your new DNS record and then you can slowly migrate your services over.

---
Tags: kubernetes, ingress
---


# Question
Title: Alternating VC branches to build deployment artifacts for different environments
We use `Jenkins` for CI. The app is written in `Java`, built with `Maven` (invoked manually in local dev environments and by Jenkins), deployed on various `Tomcat` servers. Nothing unusual about the application stack. Version control is `SVN` for the time being (we're looking at `Git` and I hope that comes soon).

We have a multi-phase Jenkins job called something like **build.all** which does `svn update` using the Maven plugin and then runs `mvn clean install` from the root dir where the code is checked out by svn. Then, another set of jobs deploys the artifacts made by **build.all** using deploy plugin by sending them over to the target servers using a Tomcat manager web service. The deploy jobs specify the latest successful build of **build.all** as the source of artifacts to deploy.

That works great when we are in a single branch mode. The hard setting of **build.all** as the source of artifacts to deploy is what is giving me headaches while trying to support transition periods during which some environments receive deployments from branch X and some from branch Y.

Without belaboring you with the minutiae of random low level tactics I'm speculating might make this possible -- can you recommend a high level strategy to support parallel artifact builds from different versioning branches? One goal is to not have to change the job source of artifacts to deploy because there is a rather large number of jobs that do deployments and changing them all by hand would be prohibitively labor intensive. For example, is there a way to maintain a key lookup mapping environments with source branches (or references thereof) so if a deployment job starts with *STAGE* or *UAT*, Jenkins looks up which build job (specific to a branch) is mapped to that environment?

# Answer
> 1 votes
You can create a release process to manage deployments in your environments. A plugin that can help you is the maven-release-plugin, where you 'freeze' the package and can promote them between environments (DEV -\> UAT -\> PROD). This ensures that the artifact (package) is exactly the same as the Production environment.

---
Tags: jenkins, branch, subversion
---


# Question
Title: Jenkins pipelines vs matrix vs multiple jobs for multiple branches across multiple platforms
I've never done any CI work before, so I'm struggling to decide how to set up Jenkins as my first project at my new job.

From what I (sort of) understand:

* Jobs are the generic work unit
* Pipelines are like a job but are defined from a series of steps from scripts stored in a VCS repo
* Matrix is like a job, but you can define axes to execute commands across

The basic setup for my project is 2 git repositories serving 3 platforms (one Windows, one Mac/Unix), each of which has 3 branches for different phases (so 9 builds total). The team wants something that will automate builds from each branch as things are committed to them, and then run some tests (which I'm also going to write). There are 2 machines dedicated to each branch. This will not include any kind of general deployment. It'll only be to the test machines.

I've set up a Jenkins master server that each branch stack connects to. I have 3 jobs set up at the moment targets a label I've assigned to the test machines, since I'm just testing with one branch for each platform. Each job uses the "run script" option, and contains something like:

```
# sets up the required authentication
$SLAVE_HOME/pre-build.sh

# then run the build script
./build.sh

# then a post-build/install script that automates installation of the packages,
# runs the product, and verifies it's started properly.
$SLAVE_HOME/install.sh

```

The scripts are unique to platform, and may eventually be unique to branch depending on if build steps change

I'll also need to integrate testing at some point - I'm planning to try using SikuliX for mac/unix and either that or Selenium+Winium for Windows. This will include aggregating results and build status to post on a not-yet-written monitoring page. That will add another couple platform-specific "global" steps.

Setting up 9 jobs that are very similar feels like I'm doing something wrong. I've considered a matrix per platform with branch axis, but I'll also need to report on build status/outputs, and having 6 outputs tied into one job might be problematic. Tests may also differ between branches as features are changed.

# Answer
You can use the Jenkins pipeline (Jenkinsfile) to run the multiple branches in multiple stages, with the conditional 'IF'. Below is an example from Jonico. I always use this example to create continuous delivery of monolithic applications. https://gist.github.com/jonico/e205b16cf07451b2f475543cf1541e70
> 2 votes

# Answer
To be honest, I'm not sure exactly what your question is here (StackExchange sites usually work best when you formulate your post as a single question rather than just generally asking for help). So I'm going to attempt to address what appears to be your main issue:

> Setting up 9 jobs that are very similar feels like I'm doing something wrong. I've considered a matrix per platform with branch axis, but I'll also need to report on build status/outputs, and having 6 outputs tied into one job might be problematic. Tests may also differ between branches as features are changed.

This is where Shared Libraries come in. Jenkins Shared Libraries allow you to refactor common code into their own individual repositories outside of your project repos. This allows you to treat your Pipeline code, or snippets of your Pipeline code, as independently-versioned libraries.

For instance, my team has 150+ repos which require near-identical Pipelines. Instead of synchronizing 150 Jenkinsfiles, I have one Shared Library which is used by all of these repos to run tests and deploys.
> 1 votes

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: Built codebase doesnt appear after deploying
I'm packaging and shipping a custom code base of Wordpress with a Dockerfile in to my private container registry.

Dockerfile:

```
FROM wordpress:latest
ADD . /var/www/html

```

Then on the deployment server I just use `docker-compose.yml` and grab my custom image. and also I'm mounting this needed directory to host.

docker-compose.yml

```
volumes:
  - .:/var/www/html

```

Container starts and runs without any hustle. However the problem is the code (files, directories) I ported earlier doesn't show on the deployment. Only the official `wordpress:latest` data are there.

What I'm missing here?

---

**Another try**

Then I tried something like this. Instead of directly `ADD . /var/www/html/`, I copied code base to another directory `ADD . /app` on build and on the deployment docker-compose.yml, I tried to `cp -a /app/. /var/www/html` with a `command` parameter. Command and `cp` works. However it happens before `wordpress:latest` script runs and it avoid coping official wordpress files, which I don't want. I expect some official files from `wordpress:latest` to be copied to the `/var/www/html/` then my customer data to be copied from `/app`

Whats the missing ingredient here?

However in that case

# Answer
> 1 votes
Well after some research, it seems the initial issue was caused by the volume mount configuration in the host's docker-compose.yml

Even I put below line on build `Dockerfile` which copies the files to the given path

```
ADD . /var/www/html

```

Since the host's docker-compose.yml got line like below

```
volumes:
    - .:/var/www/html

```

The container mounting this directory as a new directory to it's Unix like filesystem. So it masks the underlying files. So on the file system its a new directory and doesn't represent the underlying files. Basically its empty. This Github issue confirms this.

Then it runs the `wordpress:latest`'s `docker-entrypoint.sh` Which set `/var/www/html` as the working directory and if it doesn't contain `index.php` and `wp-includes/version.php` (obviously it doesn't due to above matter, directory is new) inside, it downloads and extract WordPress source to this directory. That's why after the mount, my old bundled data wasn't there.

So I was basically tried to do something which was not intended.

As @Jeremy commented,

> as /var/www/html? It sounds like your mount is failing - and that the ADD Is superfluous

---

**Another try** Was totally wrong. Because I was adding a custom `ENTRYPOINT` script to my build `Dockerfile`, so it keeps ignoring the base image `wordpress`'s `ENTRYPOINT` and `CMD` that's why it wasn't copy new source to the directory.

---
Tags: docker, docker-compose, dockerfile
---


# Question
Title: Rancher & gitlab-runner - is there a way to get the gitlab-runner to tell Rancher to start the CI test environment instead?
I’ve got an AWS gitlab instance successfully doing CI with gitlab-runners running under vagrant on my local dev machine. I’d like to move the runners off my machine and onto a standalone server...

I’ve put Rancher on a box and can get the gitlab runner installed, but I can’t get my head around how to not do docker-in-docker, or whether this actually matters.

My understanding is that unless I do `_some_configuration_I_don’t_know_` that Rancher runs the gitlab-runner as a Docker container, and the gitlab runner will in turn start a docker container inside itself to run my CI test environment as specified in the `.gitlab-ci.yml`

Is there a way to get the gitlab-runner to tell Rancher to start the CI test environment instead?

# Answer
The short answer is "it depends". Rancher itself runs using system-docker and the docker instances it starts for you, such as gitlab-runner are run separately in regular docker. Gitlab-runner then will start runners depending on *its* configuration:

> There are three methods to enable the use of... docker run during jobs; each with their own tradeoffs.

https://docs.gitlab.com/ee/ci/docker/using\_docker\_build.html#runner-configuration

**My eventual solution**

I'd initially installed Rancher 1.6 and just couldn't get the dependencies working (Rancher OS running Rancher 1.6 and Rancher agent running on a couple of Alpine nodes).

The upgrade to Rancher v2 was a steep learning curve as you need to learn the Kubernetes ecosystem tooling (helm, kubectl, etc). I still couldn't get a gitlab-runner working under Rancher v2, but **upgrading to v2.0.8 fixed it**. I could add the custom helm chart for gitlab-runner and after cribbing the settings from https://gitlab.com/charts/gitlab-runner/blob/master/values.yaml I could get gitlab-runner working under Rancher v2.
> 1 votes

---
Tags: docker, gitlab, rancher
---


# Question
Title: How to revert the Jenkins setup if plugin/Jenkins updates went wrong.
I would like to upgrade my Jenkins version(From 2.60.2 to 2.121.1) and some of my plugins. Currently, there are some jobs configured in it. I know there are some configuration changes needed for jobs after upgrading the plugins. Is there any way to restore all the settings(data + conf + plugins) to the previous state if my updates went wrong.

# Answer
> 2 votes
As Tensibai mentioned in the comment - better way is backup you Jenkins instance before an update

There are different ways to do this:

# Plugins

Jenkins has a lot of backup plugins, most popular is:

PeriodicBackup Plugin \- be aware, has status `This plugin is up for adoption` right now, that means there is can be lack of support

thinBackup Plugin \- the same about current status of plugin

SCM Sync configuration plugin \- see attention on the plugin page, this is deprecated plugin, the suggestion about **migrate their jobs to Pipeline Plugin** and **use the Configuration as Code Plugin** also make sense

All of them easy to use, also you can discovery any Jenkins plugin here

# Your own solution

Jenkins has good structure for backup, it is not hard to understand what exactly you need to backup and how.

You can write script\job with simple steps `copy`, `archive`, `move` to achieve backup of your Jenkins.

More information and discussion about how to backup jenkins can be found here: How do you back up Jenkins jobs & master configs?

## As for me

I'm using `Periodic Backup` plugin + my own scripts for transfering backups into different places. But this plugin is not really stable and has opened issues, so i'm not recommend this for highload production use. That's fine for personal not-critical systems. Now I decided to migrate to my own solution with simple job to archive and copy all configs of my Jenkins instances.

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: Do AWS backups of large empty volumes take longer than small ones?
AWS provides backup functionality for RDS databases and snapshots functionality for EBS volumes.

Given a small amount of data, would the backup and restore time change depending on whether it's stored on a large disk or a small one?

In case you are asking yourself why one would put a small amount of data on a large disk: EBS gp2 volumes offer more IOPS (IO operations) with more storage (3 IOPS per 1 GB). So if you need a disk with plenty of IO operations but little persistent data, you might end up with a few megs of data sitting on a 1000 GB disk.

If my assumption that backups might take longer for larger disks is true, then it might be worth deciding for the more expensive io1 disk type instead, which can scale with up to 50 IOPS per GB.

# Answer
> 2 votes
Firstly I would argue that backup time does not matter, as it usually doesn't correlate with the restore time.

In regards to restore time, snapshots restore on volumes almost instantly, as data is only copied onto the volume from S3 when requested.

So as long as you only request actually used data from the volume, it would only matter how much data is used on the volume to determine the time to restore.

This link explains the process well https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-restoring-volume.html

---
Tags: amazon-web-services, amazon-ebs
---


# Question
Title: Is there a way to run with_items loops in parallel in Ansible?
I'm running Ansible 2.2, but can upgrade if it helps.

I saw this and was pretty excited, but it doesn't seem to be in this (or any) version of Ansible documentation.

The problem I'm trying to solve is I've got 1000 users that I need to manage on a Centos box.

It takes quite a while to run this task serially. And even more annoying, everything shows up as changed because the "expires" command on the user module always marks the thing as changed.

this also looked promising, but it took the same amount of time to run each command in the with\_items loop and didn't go any faster (I never bothered to wait long enough to get to the end).

Skipping tasks is fast now (a lot faster than it was in Ansible 2.0), if I can't figure out how to make this work in parallel, I think I'll go back and figure out how to skip pointless tasks and if all else fails, I'll write my own module. But it seems like I should be able to do all this faster in Ansible.

---

This is what I want to run in parallel, `host_authorizations` is a list of usernames and other data.

```
  - name: Create/modify OS user accounts
    user: name={{ item.username }} group=sshusers shell=/bin/bash home="/home/selinux-modules/{{ item.username }}" state=present expires={{item.expiredate|default(omit)}}
    with_items: "{{ host_authorizations }}"
    tags: full_maintenance

```

# Answer
As @webKnja mentioned this is possible with `async` mode. I have recently discovered it myself and learned that you can use it in 3 different ways depending on your needs.

1. **Execute and poll** the results, notice the `poll:5`, This will poll the results every 5 seconds. You may save some time with this method.

```
 - name: My long runing task
   some_module_name:
     ip: "{{item.fabric}}"
     username: "{{user}}"
     password: "{{password}}"
     secret: "{{secret}}"
   loop: "{{zoning_list}}"
   register: _alias_vc_0
   async: 60
   poll: 5

```
2. **Fire and forget** `poll: 0`, This is very quick option since Ansible it's just shooting out those tasks. The down side is that we don't know what was the outcome of the task i.e. `changed: True/False`. Of course it's a downside if you care about the feedback ;).

```
 name: My long runing task
 some_module_name:
   ip: "{{item.fabric}}"
   username: "{{user}}"
   password: "{{password}}"
   secret: "{{secret}}"
 loop: "{{zoning_list}}"
 register: _alias_vc_0
 async: 60
 poll: 0

```
3. **Fire and forget with `async_status`**, the syntax for the task is the same as example **2** whowever it will require additional task `async_status`. This is my favorite since it's relatively fast (faster then normal looping or the `execute and poll`) and allows you to capture the feedback although will need to deal with new `register` for your `async_task`.

`retries: 20` -- how many attempts before failing.

`delay: 2` -- how many second to wait between polls.

```
    - name: My long runing task
      some_module_name:
        ip: "{{item.fabric}}"
        username: "{{user}}"
        password: "{{password}}"
        secret: "{{secret}}"
      loop: "{{zoning_list}}"
      register: _alias_vc_0
      async: 60
      poll: 0

    - name: Wait for My long running task to finish
      async_status:
        id: "{{ item.ansible_job_id }}"
        #jid: "{{ item.ansible_job_id }}" # ansible version > 2.8
      register: _jobs_alias_vc_0
      retries: 20
      delay: 2
      until: _jobs_alias_vc_0.finished
      loop: "{{_alias_vc_0.results}}"

```

**A word of caution**, depending on the task yo may not be able to use the `async` option. I had examples where I was interacting with system which was not able to handle multiple requests for the same resource. I found `async` option best working if I have to perform the same task across multiple hosts. That's where I was able to "save" the most time.

Since you posted the link to Ansible documentation in the question I'm not going to do that.
> 32 votes

# Answer
It is possible to achieve this using `async` mode. Please find some references for how to do this below.

Refs:

```
---

- name: Run tasks in parallel
  hosts: localhost
  connection: local
  gather_facts: no
  tasks:
    - name: Pretend to create instances
      command: "sleep {{ item }}"  # Instead of calling a long running operation at a cloud provider, we just sleep.
      with_items:
        - 6
        - 8
        - 7
      register: _create_instances
      async: 600  # Maximum runtime in seconds. Adjust as needed.
      poll: 0  # Fire and continue (never poll)

    - name: Wait for creation to finish
      async_status:
        jid: "{{ item.ansible_job_id }}"
      register: _jobs
      until: _jobs.finished
      delay: 5  # Check every 5 seconds. Adjust as you like.
      retries: 10  # Retry up to 10 times. Adjust as needed.
      with_items: "{{ _create_instances.results }}"

```
> 6 votes

# Answer
To answer your question: No, as of now Ansible can't run loops in parallel.

I'd use `newusers` instead, which is made for bulk user creation. Create a file with all users in it, copy it over to the host, and run `newusers /path/to/user/list` in a `command` or `shell` task.

For example:

```
- name: Create multiple users
  shell: newusers /path/to/user/list && touch /path/to/user/ansible-users-added
  args:
    creates: /path/to/user/ansible-users-added

```
> 3 votes

---
Tags: ansible
---


# Question
Title: Docker - which cache is used to build image
I have got my docker image called `samplecontainer`, during build I am not specifying `--cache-from` option.

Is it possible to know which image is used during build as cache?

# Answer
## Answer

The default location of the image cache is under `/var/lib/docker`.

## More answer

The "cache" features in docker are more like a communication vehicle for the documentation, than a proper, separate data structure. There is no special storage which makes up this cache.

For every filesystem-modifying line (RUN/ADD/...) in your Dockerfile, `docker run` creates a new image, with the image of the previous line (or the FROM line at the beginning) as the base. Docker uses the hashes of the previous image together with the hash of the RUN line to calculate the hash of the new image. If that new hash already exists, then it is reused.

Those intermediate images are exactly the same kind of object in storage as your final image (i.e., a subdirectory of `/var/lib/docker`, by default), they just don't have a tag pointing to them, so they can be cleaned away somewhat more aggressively by `docker image prune -a`, for example. But technically they are just the same as tagged images.

All options which talk about caches in the docker documentation (i.e., `build --rm=false`, `build --squash`etc.) just modify the behaviour so that said image handling a bit different, but under the hood there's just images, nothing special related to caching.
> 2 votes

# Answer
You can run the history command to see the different layers of your container

```
docker history samplecontainer

```

On the result you can see the image id of each layer
> 0 votes

---
Tags: docker, cache
---


# Question
Title: Pass parameters to jenkins job with the help of userContent
I am trying to create a so-called proxy page which will accept user parameters in Jenkins and then pass them to url for running pipeline with specified parameters.

I am trying to achieve it with the help of userContent Jenkins feature:

```
<font face="verdana" size="2px">
   <form action="http://user:token@jenkinshost:port/job/dev/job/dev-pipeline/buildWithParameters?n1" method="post">
       Number1:<input type="text" id="n1" name="n1"><br>
       <input type="submit" value="Submit">
  </form>
</font>

```

But nothing works.

I was even trying to set up the next policy through Jenkins console:

```
System.setProperty("hudson.model.DirectoryBrowserSupport.CSP", "sandbox allow-same-origin allow-scripts; default-src 'self'; script-src * 'unsafe-eval'; img-src *; style-src * 'unsafe-inline'; font-src *");

```

It also didn't work.

Am I doing something wrong by missing something?

# Answer
> 1 votes
Looks like I read documentation not attentively.

**userContent** supports only static content but not dynamic.

Thus, PHP/JS or something similar can not be used there: https://wiki.jenkins.io/display/JENKINS/User+Content

---
Tags: jenkins
---


# Question
Title: Encrypting docker images (end to end); transporting over offline channels
We are developing and building docker images in our intranet, and need to deploy them to several hosts belonging to us (developers, internal testing, external testing and so on). Some of these are in our intranet, some are accessible for 3rd parties on the internet.

The final deployment is inside the customer's intranet on several nodes (production, different test stages). These are behind firewalls that don't regularly let them access things outside their intranet, i.e., while some of them can be allowed to access an external registry for deployment, others can't and images have to be delivered manually through some arcane software upload tool.

I am looking for a way to have a registry on the internet (possibly ran by ourselves on some VM out there, preferably not) which allows the images to be stored encrypted (preferably GPG or similar, not a simple password). But then also being able to rather simply delivering "half" of the stuff via some upload mechanism manually. The customer is very paranoid, so keeping the end-to-end-encryption up is quite important.

Is there a tool that springs to mind, which is perfectly able to handle it? One solution would be to mirror the full registry to the customer premises on a dedicated host, keeping the encryption part intact.

A "standard" solution would be great, I'd be loath to hack something together if there is something lean/lightweight/established/stable around, already.

EDIT (+edit to the title): An extensive permission scheme like the one offered by Portus is a good start, but I am looking ideally for end-to-end encryption of the actual images. The customer is ultra paranoid and just getting started with Docker, cloud-based services etc.

# Answer
Google Container Registry is a potential option.

1. Registry is private
2. Image layers are stored on a Google Cloud Storage bucket, which is encrypted by Google (RSA) and can be additionally encrypted with a client key (RSA)
3. Access control is granular down to each individual object stored on GCS bucket
4. Client-side authentication is enforced via oauth2. Clients configure docker to use the google cloud sdk as a credentials manager
5. Images are pulled via ssl connections

The fact that the registry is part of Google's Cloud Platform also means you can build up to some other interesting things. For example, you could set up a VM to serve as reverse-proxy and set up the GCP firewall to only allow 443 from a certain set of IPs (like your clients' offices).
> 2 votes

# Answer
Have a look at https://github.com/Senetas/crypto-cli

It allows you to encrypt/decrypt containers when pushing and pulling from a public or private repo.

(Full disclosure I know the guy who wrote it)
> 0 votes

---
Tags: docker, deployment, encryption
---


# Question
Title: What's the difference between DevOps and Automation?
I see that whenever someone does DevOps, it's mostly about automating things like deployment etc.

But where does automation end and DevOps begin?

# Answer
> 47 votes
A big part of DevOps is making it possible to release very often. That comes with automated build, automated testing, etc. You can say that to achieve its goals, DevOps need to use automation to be efficient.

Here's how DevOps and automation are related. DevOps is not just automation, there's more to it. Conversely, automation is not exclusively used by "DevOps people". A lot of automation was taking place in IT before DevOps came around.

Please don't consider to above diagram to represent all that is DevOps, or all this is automation. It is to help the reader picture how the two concepts are related.

# Answer
> 15 votes
Automation is a key attribute of DevOps, but it's not the full story. The question is kind-of like "What's the difference between time-boxing and Scrum?".

You'll hear DevOps called a 'culture', a 'movement', a 'methodology', and all kinds of things that won't box it in well enough for you to understand it, even though those descriptions are accurate. In a nutshell, DevOps is about the confluence of Agile methodologies, automation, and virtualization that enables a new feedback loop in the management/control/steering of a software project.

With aggressive automation, things that use to take a long time and be subject to human error now happen quickly and without incident. As a result, we tend to do them more often. A primary example of this is a 'deploy into production'. We used to save large batches of work and deploy them off-hours just in case 'something went wrong'. But now we can deploy changes several times a day, in a way that the chances of 'something going wrong' is dramatically reduced, and in a way that the impact of something going wrong is much smaller when it does occur.

Once we have this repeatable process in place, we begin to see it as a 'pipeline'. Requirements go in, code deployed into production comes out. We automate everything along this pipeline - tests, documentation, merges, deploys, and more tests, and so on... Because people focus on the automation, they don't see the 'pipeline mentality' that drove it. **This** is the management methodology - the attention paid on the pipeline - that makes DevOps more than Automation.

Once we have that automation in place, the feedback loops kick in. We start to measure things like **cycle time** so we can figure out things we had previous tried to guess with estimates. Things about the architecture that make automation/continuous delivery hard tend to be replaced with alternative architectural patterns that make automation/continuous delivery easier (several great examples of this are documented in the book 'Evolutionary Databases'. 'Green/Blue Deployments' are another).

Notice I was able to provide this description without talking about Jenkins, Check, Puppet, Ansible, Vagrant, AWS, or any of the other tooling that supports it. This is what we mean by the larger level buzzwords like 'methodology'. In the end, any set of tools can be replaced... What we're left with is the core management principles enabled by automation and the focus on the pipeline.

# Answer
> 14 votes
DevOps is really a cultural shift - it's intended to be about breaking down the traditional barriers between operations and development (and really also with QA and the rest of the business!). The idea is that rather than having departmental 'silos' you can work directly with other teams to get things done quicker and more efficiently.

It's all about removing constraints and streamlining processes. Automation comes into this heavily because having repeatable processes helps remove constraints. For example: if someone from ops has to do a manual release process to get code out to an environment, there's a couple of things that can get in the way - one is that there has to be someone in ops free to do the deploy, and two, there is less confidence in the release process because manual work is error-prone.

# Answer
> 5 votes
DevOps includes automation but that's only part of it. DevOps is a cultural change to break down the silos between the different parts of the organization to provide a complete value stream. Providing a culture where business, development, quality assurance, infrastructure, security, ops, etc all work together to provide value to who ever is the end user. DevOps is not a tool, you can't buy it, you have to change your culture.

Automation is a key part of DevOps in that it allows the speed of delivery with quality. Automation for the deployment process is one of the areas many people focus on first as it is one of the best ways to gain value quickly and provides a high return on investment through not only reducing the time for the deployment but also standardizing the process and removing errors.

# Answer
> 2 votes
DevOps movement consists of four major areas abbreviated as CAMS:

1. Culture
2. Automation
3. Measurement
4. Sharing

Here is the original defining post from 2010.

In each area there are some tools, processes and practices that are generally accepted, although the subject is not well defined for Best Practices, there are in most cases some Good Practices to follow.

Automation by itself is a bit wider subject, but in context of DevOps it is just a subset of what is being covered. Take a note that we lead with culture, although many DevOps practitioners new to the field often overlook it at their own peril and skip directly to automation.

# Answer
> 1 votes
**I would like to add my 2 cents:**
**1) *Automation*:**
Something that we are nowadays having to move towards. It has become more of a necessity wherein the preferred way would be to automate pieces, if not the entire process. This approach gives the users (developers) flexibility to use a fixed step along with being able to customize as needed.
The advantage in this approach is that we can automate the parts that we want to while the individual process can be tied together by the developer. The more granularized the automation steps, the better control they have.
Also, there are many tools for automation in spaces like robotic automation, SOP automation (for servicing industry), report automation (like Splunk), etc.
**2) *DevOps*:**
Given the state of delivery quality and timeliness that is being expected out of the current world, there is a necessity to extend automation of the software delivery process. To enable this and provide value to the customer in the fastest way possible, DevOps does extensively rely on automation tools.
The advantage in this approach is that, the individual steps can be automated to bring about consistency across enterprise while the overall orchestration can be modified to suit the process needed by that project.
The individual automation tools (in a way) like Chef for deployment, Docker via Dockerfile, Maven for build, etc. can be tied together probably via Jenkins to provide the required solution while at the same time bringing down the time needed for implementation or usage.
Hope this helps to add any value to the thought process that you may already have.
**Edit:** Forgot to add that I had talked about the Process and Tools - 2 of the 3 aspects in DevOps. As mentioned by the others, the 3rd and equally important aspect is People. One of the major differences I would assume between this and automation would be that people are more prone to absorb automation more frequently than they would resist DevOps. I feel that this is due to the nature of DevOps itself, in that automation is usually associated with making things easier for them, while they feel that DevOps is changing the way that they are comfortable with.

# Answer
> -2 votes
Automation and DevOps are unrelated. DevOps is more like combined engineering where the Developers of a site or service are all the Operators of that site or service. Why is this novel? In my experience the first thing the Ops team did when anything more exciting than a network blip happened was to call the Dev team. Why did they do that? Because all the Ops team did was monitor and keep a list of dev phone numbers to call.

Notice I've said nothing about automation.

Automation is about repeating success. If I do steps a,b and c and process X always works, then steps a, b and c are good candidates for automation. Then I can use the time for what used to be a manual process to do things that make me more money. Automation is successful when it's simple. Deploying new releases, running integration tests, torquing a nut on a bolt, backing up data, balancing credits vs. debits, etc. are all great candidates for automation because the steps are repeated whether by a person or a robot.

**Note**: What is new is that Developers are also the operators. There is not another group. Cooperation in my case was rare. If there was nothing in the Troubleshooting Guide (aka TSG) then you were guaranteed a phone call. In my experience Ops was the first call in case of backhoe. Problems between services were out of their wheelhouse.

---
Tags: continuous-integration, terminology, automation, culture
---


# Question
Title: WinNMP (Nginx) returning "Directory access is forbidden."
I installed WinNMP with the typical install. and added two files that I am using to check the installation(among others)

```
C:\Proj\MyRG\public_html\test.htm
C:\Proj\MyRG\public_html\application\index.htm
c:\winnmp\conf\nginx.conf file contains:
    root "C:/Proj/MyRG/public_html/application"
C:\Windows\System32\drivers\etc\host file contains both these lines after the install:
    127.0.0.1 localhost
    127.0.0.1 ProjWinNMPTest.test #WinNMP ProjWinNMPTest.test #WinNMP public_html.test

```

Regardless of how I set WinNMP Manager--\>Options(button)--\>Projects Dir when I read the test files from a browser

```
http://127.0.0.1/public_html/test.htm
    correctly reads the simple htlm file.
http://127.0.0.1/public_html/application/index.html
    returns "directory access is forbidden
'http://127.0.0.1/public_html/index.html
    returns "404 file not found"

```

1) How can I set the root file for 127.0.0.1 to point to `C:/Proj/MyRG/public_html/application` instead of `C:/Proj/MyRG/public_html` ?

2) How can I get permission to access the directory ?

# Answer
I found the problem it is unrelated. The index.html file itself was the 'directory access is forbidden
> 1 votes

---
Tags: nginx
---


# Question
Title: Can't install MongoDB docker on raspberry
If I want to execute any docker file from https://hub.docker.com/r/library/mongo/tags/ on my raspberry Pi 3B with raspbian stretch installed I always get the same error namely something like :

```
Pulling mongodb (mongo:)...
latest: Pulling from library/mongo
ERROR: no matching manifest for unknown in the manifest list entries

```

How can I resolve this error and what causes it? If you need any more details please let me know I'm pretty new to docker so I don't really know what's relevant.

# Answer
This means that the container that you are trying to pull is not available for the architecture, in this case it would be **linux/arm64**. Maybe there is an image available now. I would search for 'docker mongodb arm64' to see. I see support for mongodb under arm64v8 architecture, which would be in Raspberry Pi 4. Maybe you want to upgrade, as your chip is arm64v7.

This issue explains error message in a bit more detail.

Answers under this issue are going to help you run MongoDB under ARM64.
> 4 votes

---
Tags: docker
---


# Question
Title: Error during setup of clairctl for container scanning with clair
I am trying to setup clairctl to scan our application images and so wanted to install the full stack of clairctl, clair scanner & postgress database. I did the following steps:

1. Acquired a VM running CentOS 7, 2 vCPU and 8 GB RAM.
2. Installed docker-compose as described here: https:// www.linuxhelp.com/how-to-install-docker-compose-on-centos-7.
3. Installed GO as described here: https:// linuxize.com/post/how-to-install-go-on-centos-7.
4. Downloaded and installed clairctl by referring to this page: https:// github.com/jgsqware/clairctl. In this step I faced some problems and tried to work around them as described below.

i) Downloaded the binary release of clairctl by running “curl -L https:// raw.githubusercontent.com/jgsqware/clairctl/master/install.sh | sh”

ii) Executed the command “git clone git@github.com:jgsqware/clairctl.git $GOPATH/src/github.com/jgsqware/clairctl” which failed with error message

```
[root@clair-server ~]# git clone git@github.com:jgsqware/clairctl.git $GOPATH/src/github.com/jgsqware/clairctl
Cloning into '/src/github.com/jgsqware/clairctl'...
The authenticity of host 'github.com (192.30.253.113)' can't be established.
RSA key fingerprint is SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8.
RSA key fingerprint is MD5:16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'github.com,192.30.253.113' (RSA) to the list of known hosts.
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights and the repository exists.
[root@clair-server ~]#

```

So did the following to workaround the problem. If this is a generic issue which could affect many, you may want to update the project’s README.md with the workaround so that it helps other users of clairctl.

```
git clone git clone https:// github.com/jgsqware/clairctl.git
export GOPATH=$PATH:~/go
mkdir -p $GOPATH/src/github.com/jgsqware/clairctl
mv clairctl $GOPATH/src/github.com/jgsqware/

```

iii) Then executed the commands below which passed:

```
cd $GOPATH/src/github.com/jgsqware/clairctl
docker-compose up -d postgres

```

iv) Then executed the command “docker-compose exec clairctl clairctl health” which failed with error. I Googled about this error, examined the file docker-compose.yml. Based on it, I executed “docker-compose up -d clairctl” which is not mentioned. This is when images for clair and clairctl were downloaded and started. Then I ran “docker-compose exec clairctl clairctl health” which was successful. You may want to add the missing command: “docker-compose up -d postgres” to the documentation so that it helps other users of clairctl.

5. The current situation is given below. Is there some way for the second command (without docker-compose) to work? Could you please help me here?

```
[root@clair-server clairctl]# docker-compose exec clairctl clairctl health
Clair: ✔
[root@clair-server clairctl]# clairctl health --log-level debug
2018-09-19 09:45:27.038127 D | config: No config file used
2018-09-19 09:45:27.038608 D | clair: requesting health on: http:// localhost:6061/health
2018-09-19 09:45:27.040779 E | clair: requesting Clair health: Get http:// localhost:6061/health: dial tcp [::1]:6061: getsockopt: connection refused

Clair: ✘
[root@clair-server clairctl]#

```
6. I want to scan our application container and so did “docker login” and then analyze, but face problem seen below:

```
[root@clair-server clairctl]# docker login -u username -p the-password
[root@clair-server clairctl]# docker-compose exec clairctl clairctl analyze my-org/my-container:186 --log-level debug
2018-09-19 13:49:10.858615 D | config: Using config file: /home/clairctl/clairctl.yml
2018-09-19 13:49:10.859175 D | dockerdist: Downloading manifest for my-org/my-container:186
2018-09-19 13:49:10.859395 D | dockerdist: Retrieving repository client
2018-09-19 13:49:11.240796 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
client quit unexpectedly
2018-09-19 13:49:15.415043 C | cmd: retrieving manifest for "my-org/my-container:186": errors:
denied: requested access to the resource is denied
unauthorized: authentication required
[root@clair-server clairctl]#

[root@clair-server clairctl]#clairctl analyze my-org/my-container:186 --log-level debug
2018-09-19 09:51:33.513208 D | config: No config file used
2018-09-19 09:51:33.513675 D | dockerdist: Downloading manifest for my-org/my-container:186
2018-09-19 09:51:33.514031 D | dockerdist: Retrieving repository client
2018-09-19 09:51:37.131535 D | dockerdist: endpoint.TLSConfig.InsecureSkipVerify: true
2018-09-19 09:51:41.656177 D | dockerdist: manifest type: *schema2.DeserializedManifest
2018-09-19 09:51:41.656235 D | dockerdist: retrieved schema2 manifest, no verification
2018-09-19 09:51:41.656286 I | config: retrieving interface for local IP
2018-09-19 09:51:41.656301 D | config: no interface provided, looking for docker0
2018-09-19 09:51:41.656830 I | clair: Pushing Layer 1/7 [sha256:605ce]
2018-09-19 09:51:41.656982 D | clair: Saving sha256:605ce1bd3f3164f2949a30501cc596f52a72de05da1306ab360055f0d7130c32[https:// registry-1.docker.io/v2]
2018-09-19 09:51:41.657113 D | clair: auth.insecureSkipVerify: true
2018-09-19 09:51:41.657150 D | clair: request.URL.String(): https:// registry-1.docker.io/v2/my-org/my-container/blobs/sha256:605ce1bd3f3164f2949a30501cc596f52a72de05da1306ab360055f0d7130c32
2018-09-19 09:51:42.809395 I | clair: pull from clair is unauthorized
2018-09-19 09:51:43.941475 I | clair: adding layer 1/7 [sha256:605ce]: pushing layer to clair: Post http:// localhost:6060/v1/layers: dial tcp [::1]:6060: getsockopt: connection refused
client quit unexpectedly
2018-09-19 09:51:43.941545 C | cmd: pushing image "my-org/my-container:186": pushing layer to clair: Post http:// localhost:6060/v1/layers: dial tcp [::1]:6060: getsockopt: connection refused
[root@clair-server clairctl]#

```

I want to eventually use the CLI to integrate clairctl with Jenkins to implement a quality/security gate. Is there some way for the scans to proceed (ideally without docker-compose) to work? Could you please help me here? I tried to Google around and read threads about how to overcome this barrier, but could not figure out a way. I thought of posting the question in clairctl/Lobby on Gitter, but thought I couldn't express myself in detail.

Any guidance, help would be highly appreciated. Could you please help me?

# Answer
I eventually figured out that Clair server was running inside the container and its service ports - 6060 (API) and 6061 (Health) were available only inside the container context. I confirmed this by running these commands:

Command to see the open ports inside the clair container

`docker-compose exec clair netstat -anp`

Command to see the open ports inside the clairctl container

`docker-compose exec clairctl netstat -anp`

The clairctl analyze operation was failing inside the container as docker login context was not available inside. I figured out that I will have to expose the ports 6060, 6061 from inside Clair server outside the container context. Only then could it be visible using clairctl binary copied outside the container where the docker login context is available. I did the following three changes post which I could run the clairctl commands successfully.

1. Copied the file https://github.com/jgsqware/clairctl/blob/master/clairctl.yml.default as clairctl.yml into $HOME directory from where I had done docker login and was running clairctl commands.
2. Updated the value of `uri` parameter in above file from `uri: http://clair` to `uri: http://localhost`
3. Added the `ports` clause in section for `clair` into file /root/go/src/github.com/jgsqware/clairctl/docker-compose.yml as seen below

```
  clair:
    image: quay.io/coreos/clair:v2.0.0
    restart: unless-stopped
    ports:
      - "6060:6060"
      - "6061:6061"

```

This exposed the ports 6060, 6061 inside the clair container to the localhost, after which commands like `clairctl health` and `clairctl analyze <container-image-name>` worked successfully.
> 0 votes

---
Tags: containers, docker-compose, security, clair
---


title: How to calculate server capacity?

text:
# Question
Title: How to calculate server capacity?
I am trying to answer a question:

### Do we need more worker machines to handle current load?

We have a variable number of jobs coming in and each job takes different time. Here is a snapshot of our system for a period 12 hours.

But the number of jobs in system changes based on day and time.

How can I calculate throughput of our system and build monitoring around max load that our system can handle?

# Answer
> 1 votes
*(N.B. this answer is, in the spirit of the question, high-level and not very concrete...)*

> How can I calculate throughput of our system

If by "calculate" you mean to take some data points, dry run some formulas and get a pretty close approximation to your throughput - that is pretty much impossible, unless you have a *lot* more information - which may also include pretty random parts, hence quite hard to do theoretically.

If you mean "measure", then it could be as simple as checking your logfiles and counting lines per time unit.

> and build monitoring around max load that our system can handle?

Unless you're NASA, that's probably a case for trial and error. Run your system(s) and see how much troughput you get. Increase worker nodes. See what happens.

If you already know the behaviour of your overal system - i.e., whether it scales linearly, whether there are bottlenecks like databases, lock contention and such, then you can take shortcuts or good guesses.

That's one of the reasons for doing the container-based scaling we do these days - you can throw a few more workers at it relatively easily.

How you actually (technically) do that depends on what platform you are using. AWS, Kubernetes/OpenShift etc. come with techniques or settings do do it automatically for you. I assume those work mainly with the metric of "free workers" - trying to get the "free:busy worker" ratio into some target corridor so that every new request has a very high likelihood to hit a free worker, at any time, thus getting (theoretical) constant time for each request.

# Answer
> 0 votes
KPI (key performance indicator) or software metrics could help. For starters after a bit googling, Kubernetes has self-healing mechanism in which whether thresholds could be set up to heal after exceeding some measures is something I never tried. I'd love to know if it is doable though.

Read: Monitoring Kubernetes performance metrics.

---
Tags: capacity-planning
---


# Question
Title: Staging - Test multiple microservice versions together
I been trying to search for answer for this question but so far I didn't found anything useful.

Our problem is that we have 2 microservices with many staging branches, each of these microservices are on different servers and environments. We want to test each microservices branch with each other in our staging environment so microservice1-branch20 with microservice2-branch10. They need to communicate two-way so they both need to know that they only can contact with that version of the microservice.

Is there any good way to do these tests without putting them together in one docker container?

# Answer
Yes, this can be done by Service Discovery. The product family you're looking for is of products such as Consul. You should have the container services report their identities and versions when they load. Report their version and ip (for example), and have the tests run against these values.

This will require some effort with installations and scripting, and has a learning curve - but this is the way to go.
> 1 votes

# Answer
You need apply Continuos integration, so exist many combinations but general you need know your

1.-Programming language.

2.-Operative System and version.

3.-Data type( sql, mongodb,file, etc).

4.-Protocol communication and port.

5.-A script that check o verify outputs.

6.-A file with result/or flag

travis is very well option, because they run test with his machines, and too you can add service git(github option) , with file script called `.travis` you can run your test, exist other ci services and integrations but general working like it(without docker).

```
language: python
cache: pip
python:
  - '3.5'
  - '3.6'

install: pip install -r requirements-dev.txt
script: pytest
deploy:
  provider: pypi
  user: acme-bot
  password:
    secure: "<something long>"
  distributions: sdist bdist_wheel
  on:
    tags: true

matrix:
  include:

```

Well if you network communication require additional service networs you too need a service VPN
> 0 votes

---
Tags: microservices
---


# Question
Title: What is a Notes Day and can it be used in DevOps transformation?
Lately I am hearing about more and more companies adopting the Pixar concept of a Notes Day and I was wondering what exactly is a Notes Day, what would be a good format for such event and mainly if it could be successfully used in a DevOps inspired transformation of a company?

# Answer
> 3 votes
In his book Creativity, Inc, Ed Catmull mentions a practice originating in movie business, when directors would receive feedback on their movie in progress, so called "**Notes**". At Pixar he was in charge of the company and that was his "movie" and in essence he envisioned a similar way to receive feedback on running the business. It was an event carefully planed in advance where all employees would take a day off from their work and use their creativity to improve how things are done. It was called "**Notes Day**".

Creativity, Inc and Pixar inspired other creative companies to adopt some of the same management practices. Here is a brief article on some of the basics. One such creative company adopting Notes Day as practice is Deeson as you can read in this blog. Even tech companies follow suit, this is an article by Slack on being inspired by Ed Catmull's ideas.

A place where I've experienced it personally is Cisco Meraki, former startup and now a business unit of Cisco, which is still operating independently, much like Pixar after being acquired by Disney. Notes day is a big event every year, the entire engineering organization plans a 4 day off-site event with engineers from all around the world coming to HQ for the week and then joining us at the offsite. The event helps build teams, foster new connections, propagate culture and a whole day is dedicated to planing, discussing and presenting projects that yearly improve many of the aspects of our day to day lives. Preparations start much earlier in the year and every year the format is tweaked a bit, the design team always takes a huge part and Sean Rhea is traditionally the driving force behind the event. You can read some more in a blog article by Adam Berman.

# Answer
> 4 votes
I see some interesting parallels in this story and *The Phoenix Project.* (Spoilers ahead for the book, if you haven't read it read this anyway :))

I take it the title for "Notes Day" comes from the term of art in Hollywood, where "notes" are constructive criticism about a work. In this story, Catmull plays the part of Parts Unlimited CEO Steve Masters. Catmull accepts that he can learn from his employees how to do a better job, as Masters does in *Phoenix Project* when he does his public mea culpa and apologizes to Bill Palmer.

In a sense, Catmull is *also* Brent- he's a major constraint. If his time was so precious, it meant that Pixar was struggling to delegate authority. It seems Catmull was critical to every decision, rather than able to set large-scale goals and leave it to employees to figure out how to achieve them.

An event like this might work well at some companies, less well in others. I think it's the key idea behind Notes Day that is most valuable: accountability and transparency from the highest levels of management to the last employee. That value can be spread throughout a company in a variety of big and small ways.

---
Tags: terminology, culture, conferences, transformation
---


# Question
Title: How do I get the find module to work in Ansible?
My playbook does not create a /tmp/results.txt file, but I expect it to do so with the content of the results of the `find` module.

Here is my playbook:

```
- name: Do stuff
  become: yes
  hosts: fun
  tasks:
  - name: Delete directory
    find:
      paths: /home/jdoe/subdir
      patterns: "jj*jj"
      use_regex: yes
    register: find_results
  - shell: "echo {{ item.path }} > /tmp/results.txt"
    with_items: "{{ find_results.files }}"

```

A managed node with the "fun" label has this subdirectory: /home/jdoe/subdir/jj3jj/

What is wrong with my playbook?

# Answer
Remove *use\_regex: yes*. Quoting from the doc

> If false the patterns are file globs (shell) if true they are python regexes.

If you want to use Python regex then the pattern would be probably something like this

```
patterns: '[j]{2}.[j]{2}'

```

You probably want to append to file **\>\>**

```
echo {{ item.path }} >> /tmp/results.txt

```
> 1 votes

---
Tags: ansible
---


title: Tips for separating DevOps Git repos?

text:
# Question
Title: Tips for separating DevOps Git repos?
We currently have a Git repo with automation code (with tests), iaas-code and lamba-code.

* The automation code is mostly Fabric, Python and unittests.
* The Iaas-code is mostly CloudFormation, Docker-compose and some Terraform.
* The AWS lamdas are Serverless code and some Fabric.
* There is also some code with Dockerfiles that builds a Docker image from this Git repo.

We're considering separating git repos and using git submodules as it seems much cleaner. Do you have any tips for this process?

My concern is that most tips regarding mono-repos vs. separate repos are mostly for programmers that build micro-services and not for DevOps / SysAdmins.

# Answer
When considering mono-repo vs many-repos you might look at the following questions, which should also affect the composition of the code directory tree structure. *There is a slight general trend toward mono-repos.*

### Discoverability

How easy it is for developers and ops people to discover the code for some feature. Are there clear guidelines on what goes where? Are they enforced somehow through automation? Are new repos automatically linked as submodules into correct location? Does the build or deploy process depend on the tree structure making sure people need to put it there in order for it to be even noticed? What is the size? How long it will take to grep code in certain place? Are there generic `lib/` directories where devs and ops end up sharing libraries that create new unexpected inexplicit dependencies?

### Deployability

Does any of the code need to be present on machines that you deploy to? Are you expecting to do such deployment through syncing repository or do you have other established deployment mechanisms. In the first case, will other code creep in during the deployments that does not need to be there?

### Durability

Will the concept you are using stand the test of time? Are you a one product shop, but maybe in your future there are new products? How would adding a new product affect the code? Will there be new mono-repo for the new product? What about shared code between products? Looking at your company roadmap, how will whatever you do fare 6m, 2y, 5y down the road? Choices like this tend to be hard to change in practice (even though they shouldn't - in theory).

### DevOps

You really want to take down barriers between Devs and Ops. So even just having separate mono-repo for ops could create such barrier. The operation and deployment of code, standing up environments, setting up services, it should all be in reach of developers and with explicit expectations to develop such code in coordination with the ops team, not being dependent on someone doing it for them. If things go well your developers follow through with code for months after it has already been in production and have hand in all aspects of the deployment. How easy it is to make commits that contain both product and deployment changes? How can those commits be grouped together? What is the visibility of ops into product runtime code and devs into product management code?
> 2 votes

# Answer
My advice would be to not separate these repositories as it introduces unnecessary risks in terms of repositories not being available due to access control, availabilty or billing issues at some point when you need 2 or more of them working together.

It also reads contrary to the term "DevOps" where Software developers and System admins are supposed to work together and get a grasp of each others domain instead of traditional "over the wall" responsibilities. If Dev and Ops are supposed to work together, why would Ops and Ops be separated?
> 1 votes

# Answer
I lean towards more repositories or submodules.

Sure there might be some more configuration to do (hooks, permissions etc.), but understanding a repository with 5 files in it is a lot easier than understanding one with 5 folders.

As you mentioned, programmers use different repositories when implementing micro services. They split them up logically based on interfaces or APIs. As a sys admin you could split up repositories by defining logically what each of your components are responsible for.

For example, if your automation code is for testing/monitoring your Iaas I would put those together in the same repository. The tests verify the validity of your infrastructure. As for your lambda functions, could they potentially be reused for other types of Iaas? Can they be extended for other functions?

Also ask yourself if someone new started today, could they understand your repo and submit a pull request? Or would they be overwhelmed?

Less complex repositories enable other users to be able to make meaningful contributions in less time.
> 0 votes

---
Tags: git
---


# Question
Title: Can the Angular CLI be used to build and deploy on a remote server via ssh?
I'd like to use the Angular CLI to build the app and deploy it on my remote server via SSH. Is this feature available?

# Answer
> 1 votes
You need to use continuous deployment tools. Not available yet.

---
Tags: deployment, build-pipeline
---


# Question
Title: How the use Docker exec to update code into a running container
I am using docker-compose to manage Docker. Now suppose I want to update a source code in one of the Docker containers.

Can I do it using the exec but then how to copy the code from host to running container ?

Or is there any better solution to do the above task ?

# Answer
> 1 votes
It's a bad practice. You need to use Continuous Integration rebuild docker image (see Dockerfile) every time when source code changed.

# Answer
> 0 votes
In terms of doing it once to verify how this is done you can use the `docker cp` command from the host machine -\> https://docs.docker.com/engine/reference/commandline/cp/

But I agree with Meiram, this is bad practice. You want to be able to have the ability to revert back to a previous version of not only the code but also the container.

There is a very good tutorial here integrating Docker/DockerHub with Github, Jenkins, and even Sonarqube.

https://medium.com/@hakdogan/an-end-to-end-tutorial-to-continuous-integration-and-continuous-delivery-by-dockerize-jenkins-f5b9b45b610d

---
Tags: docker
---


# Question
Title: Docker security risks passing secrets over environment variables
I'm new to Docker and want to put my Django web application into containers.

For the last couple days I'm struggling with passing credentials safe to containers.

Online there is always the hint that secrets should not be passed as environment variable because they can be retrieved via inspect or from the images. My hope is that it can be neglected for my setup.

The setup

* I provide the credentials via .env file of docker compose. This file is not on version control.
* The containers are running as non-elevanted users (a different special user).
* Docker command on the host can be used as root only.
* I'm the only user of the host server (Ubuntu 18.04).
* I assume breaking into the host is much harder (I'm using SSH public key authentication with non-root user only) than breaking into container of the Django app.
* I don't want to use kubernetes or swarm.

I'm newbie to Docker and not a security expert. Do I have overlooked something? Why should I care about the docker inspect or images in this setup?

Thank you in advance

# Answer
> 2 votes
As long as no one else is logging into the server or able to run docker commands against the host, then passing secrets as an environment variable into the container which is only stored on the host is relatively secure. About the only risk I can think of is that it is a clear text value on the disk, so you have the risk of the secret escaping with a lost harddrive or backup.

Docker's swarm mode secrets have the following advantages over environment variables and other ways of managing secrets:

* Values are encrypted on the manager hard drives, though typically, the decryption key is also there to make restarting managers easier.
* Values are not written to disk on workers, they are injected as a file on a tmpfs filesystem mounted into the container.
* Values are not able to be displayed with an inspect command on the service, container, or the secret itself. You must deploy a service with the secret to get access to that secret.
* Secrets are automatically sent to workers on a need to know basis from the managers, no need to build an external process to synchronize files across hosts.
* The secret file may be limited with unix filesystem permissions, e.g. only enabling root read access, and having your container drop access to another user after loading the secret.

That said, no matter what, all secret management solutions are a way to give your application a clear text password or other credential. And with a process, every way of doing that has some level of risk. One of the best ways to reduce that risk is to make sure every service gets its own individual secret, with only the access needed by that specific service, and that it is frequently rotated. It's also useful to look at additional layers of security, like network policies you can configure with service mesh solutions.

# Answer
> 1 votes
ENV variables are a good way of passing secrets to a container. What you don't want to do is bake the secrets directly in the image. As you have rightly done, do not store secrets in your source control repo as well.

Besides ENV variables, you could also consider using an external secrets repository, such as Hashicorp Vault, or possibly even AWS secrets. See also vault's github page. Your container would still need a token/credentials required to Authenticate/Authorize with the key store, but if required, these can be made to expire/rotate.

So you are on the right path. Some additional reading is worthwhile at this point:

---
Tags: docker, linux, security
---


title: AWS ECS/EKS: EC2 vs Fargate mode

text:
# Question
Title: AWS ECS/EKS: EC2 vs Fargate mode
Amazon offers the option of launching their container management services `ECS` and `EKS` in two modes: `EC2` and `Fargate`.

In the former, normal VMs (`EC2` instances) are used and `ECS` or `EKS` (depending on the selection) clusters are created.

In the later, there is no concept of vm/instance present; you just have to deal with the containers and plan their capacity so that the system knows what memory/cpu needs you have.

Given that `Fargate` apparently accounts for a (far?) more cost-efficient solution given that you are charged only for the resources' consumption of your containers, I would like to know what are the benefits (or example use cases) where anyone would like to go with the `EC2` \- based solution.

# Answer
> 6 votes
**Update 12/05/2020:** Fargate prices have been **greatly reduced** over the last months and are now **comparable with EC2 prices** for the same CPU / RAM configurations. That makes Fargate a valid option for 24x7 workloads.

---

*Originally written in 2018 - no longer correct!*

Fargate is more expensive than EC2 for the same vCPU/RAM amount.

For example:

* m5.large (2 vCPU, 8 GB RAM) costs $0.096/hr which is ca $69.12/month
* Fargate container with 2vCPU and 8 GB RAM costs $0.2028/hr or $146.02/month

If you want to run your container 24x7 you'll be much better off running it on an EC2 instance.

However if your containers only run briefly to complete a task and then exit, or if they scale up and down based on demand it will be much easier for you to run them in Fargate - you won't need to scale up and down the underlying EC2 cluster to support the load.

With Fargate you pay premium for the flexibility.

In many cases it works out better to run on Fargate even if it's more expensive - we spin up batches of hundreds of containers at a time a couple times per day for some processing and each container runs for only about 10 minutes. If we had to scale up the EC2/ECS cluster before each run, wait for it to settle, deal with the failures, then run our batch job and then scale down again the overhead would be quite high and our batch processing would take much longer.

Here Fargate works great for us. I wouldn't use it for an always-on service though.

Hope that helps :)

---
Tags: amazon-web-services, amazon-ec2, amazon-ecs
---


# Question
Title: How to "inject" AWS Lambda code into a CloudFormation?
I'm exercising an idea of not using S3 bucket to refer to the code of a lambda function.

The usual example of a lambda CloudFormation code might look like:

```
    MyLambda:
        Type: AWS::Lambda::Function
        Properties:
        FunctionName: hello
        Handler: index.handler
        Runtime: nodejs8.10
        Role: !GetAtt LambdaExecutionRole.Arn
        Code:
            ZipFile: |
            <can we "inject" the code and/or binary here?>

```

But what I would like to do instead is:

* inject that code from the file system, after potentially running my (unit) tests against that code;
* maybe even just inject a compiled binary instead of actual code;

It is possible to do similar using CLI (create a zip manually, refer to it). I understand that CloudFormation changes would be running on the AWS infrastructure and not locally or a build server, so I'm open to other non-CloudFormation options.

So my question is - what is the best way to deploy the AWS functions sticking to "infrastructure as code", preferably sticking with CloudFormation. Or is S3 + CloudFormation really the best option?

# Answer
> 3 votes
You've got a couple of options:

### 1. Embed the source code to the CloudFormation template

I do just that in my ec2-start-stop demo. You'll see file `ec2-start-stop.template.yml` with these lines:

```
    StartStopLambda:
      Type: AWS::Lambda::Function
      Properties:
        [...]
        Runtime: python3.6
        Code:
          ZipFile:
            Fn::Join:
            [...]
            -
              - "%%{ec2-start-stop.lambda.py}%%"

```

Then I've got the actual Python script referred in the `%%{ec2-start-stop.lambda.py}%%` and a simple `import-files.py` script that *somewhat intelligently* embeds the python file into the yaml file and creates a valid standalone CloudFormation template with the lambda code in it.

Note that you're limited to 4096 bytes including new lines for your lambda code.

---

### 2. Use `aws cloudformation package / deploy`

If you've got a more complex Lambda that needs external libraries or doesn't fit in the 4kB limit you can use `aws cloudformation package` and `aws cloudformation deploy` to facilitate the deployment to CI/CD.

In this case your *Lambda resource* in the template points to a local directory, e.g. `lambda/`. The `package` command then zips up the contents of the directory, uploads to the provided *S3 Bucket* with a unique name and generates an updated CFN template with the link to the uploaded file. The `aws cloudformation deploy` command then takes the template, creates a *change set* and applies the changes, thus updating the Lambda.

---

I've used both methods in CI/CD pipelines and both work well because the source files are kept in separate files until deployment.

Hope that helps :)

---
Tags: amazon-web-services, amazon-s3, infrastructure-as-code, aws-lambda
---


# Question
Title: Where do you start with hosting the infrastructure for your infrastructure?
So I'm trying to get started with the HashiCorp stack, but the question I keep asking myself, when looking at any of this DevOps infrastructure stuff, is what infrastructure should you be using to host the tools that will be managing - well - your infrastructure?

Like, for example, should I just be spinning up individual VMs to host each part of the HashiCorp stack? Should I be doing that instead using containers? Should I be managing those containers? With the same tools I'm going to be using to manage the containers managed by those containers?

# Answer
> 2 votes
Not a stupid question, but it's lacking details. If by stack you mean Terraform, Vault, Consul etc, then you would go about provisioning machines/VMs/containers for those services likely using some form of ansible or equivalent.

It would be prudent to store those scripts in git or any other source control system.

> Like, for example, should I just be spinning up individual VMs to host each part of the HashiCorp stack? Should I be doing that instead using containers?

Personally, I find containers more flexible especially when upgrading/ testing. The only downside is you have to remember to mount a data volume into the container to persist any configuration data which would normally live beyond the lifecycle of the container.

---
Tags: terraform, azure, project-management
---


# Question
Title: Is there an alternative to the Jenkins dir() to change to a directory on a Docker container?
Here's my current set-up:

1. Jenkins Master running in Docker on an Ubuntu 18.04 host.
2. A Node running Ubuntu 18.04 with Docker installed.
3. Desired job spins up docker containers on the node which build products in parallel and output to a shared volume.

The issue I'm running into is I'm unable to switch to a directory on the container in which I need to run build commands. The directories exist and can be access using `sh()` to run various commands for verification. When I try:

```
dir("/path/on/container") {
    sh "pwd"
}

```

The console output appears that the `dir()` function is executed, but there's a failure when tryining to run anything between the curly braces. The failures are (full error text at the end of this post): `hudson.remoting.Channel$CallSiteStackTrace: Remote call to DockerHostNode` and `java.nio.file.AccessDeniedException: /path/on/container`.

```
[DockerTest] Running shell script
+ pwd
/home/jenkins/workspace/DockerTest
[Pipeline] dir
Running in /path/on/container
[Pipeline] {
[Pipeline] echo
Are we in the project dir?
[Pipeline] sh
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to DockerHostNode
        at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1741)
        at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:357)
        at hudson.remoting.Channel.call(Channel.java:955)
        at hudson.FilePath.act(FilePath.java:1070)
        at hudson.FilePath.act(FilePath.java:1059)
        at hudson.FilePath.mkdirs(FilePath.java:1244)
        at org.jenkinsci.plugins.durabletask.FileMonitoringTask$FileMonitoringController.<init>(FileMonitoringTask.java:171)
        at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:197)
        at org.jenkinsci.plugins.durabletask.BourneShellScript$ShellController.<init>(BourneShellScript.java:189)
        at org.jenkinsci.plugins.durabletask.BourneShellScript.launchWithCookie(BourneShellScript.java:110)
        at org.jenkinsci.plugins.durabletask.FileMonitoringTask.launch(FileMonitoringTask.java:98)
        at org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.start(DurableTaskStep.java:264)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeStep(DSL.java:270)
        at org.jenkinsci.plugins.workflow.cps.DSL.invokeMethod(DSL.java:180)
        at org.jenkinsci.plugins.workflow.cps.CpsScript.invokeMethod(CpsScript.java:122)
        at sun.reflect.GeneratedMethodAccessor263.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.codehaus.groovy.reflection.CachedMethod.invoke(CachedMethod.java:93)
        at groovy.lang.MetaMethod.doMethodInvoke(MetaMethod.java:325)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1213)
        at groovy.lang.MetaClassImpl.invokeMethod(MetaClassImpl.java:1022)
        at org.codehaus.groovy.runtime.callsite.PogoMetaClassSite.call(PogoMetaClassSite.java:42)
        at org.codehaus.groovy.runtime.callsite.CallSiteArray.defaultCall(CallSiteArray.java:48)
        at org.codehaus.groovy.runtime.callsite.AbstractCallSite.call(AbstractCallSite.java:113)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:157)
        at org.kohsuke.groovy.sandbox.GroovyInterceptor.onMethodCall(GroovyInterceptor.java:23)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.SandboxInterceptor.onMethodCall(SandboxInterceptor.java:155)
        at org.kohsuke.groovy.sandbox.impl.Checker$1.call(Checker.java:155)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:159)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
        at org.kohsuke.groovy.sandbox.impl.Checker.checkedCall(Checker.java:129)
        at com.cloudbees.groovy.cps.sandbox.SandboxInvoker.methodCall(SandboxInvoker.java:17)
        at com.cloudbees.groovy.cps.impl.ContinuationGroup.methodCall(ContinuationGroup.java:57)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.dispatchOrArg(FunctionCallBlock.java:109)
        at com.cloudbees.groovy.cps.impl.FunctionCallBlock$ContinuationImpl.fixArg(FunctionCallBlock.java:82)
        at sun.reflect.GeneratedMethodAccessor157.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.cloudbees.groovy.cps.impl.ContinuationPtr$ContinuationImpl.receive(ContinuationPtr.java:72)
        at com.cloudbees.groovy.cps.impl.ConstantBlock.eval(ConstantBlock.java:21)
        at com.cloudbees.groovy.cps.Next.step(Next.java:83)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:174)
        at com.cloudbees.groovy.cps.Continuable$1.call(Continuable.java:163)
        at org.codehaus.groovy.runtime.GroovyCategorySupport$ThreadCategoryInfo.use(GroovyCategorySupport.java:129)
        at org.codehaus.groovy.runtime.GroovyCategorySupport.use(GroovyCategorySupport.java:268)
        at com.cloudbees.groovy.cps.Continuable.run0(Continuable.java:163)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.access$101(SandboxContinuable.java:34)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.lambda$run0$0(SandboxContinuable.java:59)
        at org.jenkinsci.plugins.scriptsecurity.sandbox.groovy.GroovySandbox.runInSandbox(GroovySandbox.java:108)
        at org.jenkinsci.plugins.workflow.cps.SandboxContinuable.run0(SandboxContinuable.java:58)
        at org.jenkinsci.plugins.workflow.cps.CpsThread.runNextChunk(CpsThread.java:182)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.run(CpsThreadGroup.java:332)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup.access$200(CpsThreadGroup.java:83)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:244)
        at org.jenkinsci.plugins.workflow.cps.CpsThreadGroup$2.call(CpsThreadGroup.java:232)
        at org.jenkinsci.plugins.workflow.cps.CpsVmExecutorService$2.call(CpsVmExecutorService.java:64)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at hudson.remoting.SingleLaneExecutorService$1.run(SingleLaneExecutorService.java:131)
        at jenkins.util.ContextResettingExecutorService$1.run(ContextResettingExecutorService.java:28)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:59)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
java.nio.file.AccessDeniedException: /path/on/container
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:84)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
    at java.nio.file.Files.createDirectory(Files.java:674)
    at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
    at java.nio.file.Files.createDirectories(Files.java:767)
    at hudson.FilePath.mkdirs(FilePath.java:3271)
    at hudson.FilePath.access$1300(FilePath.java:211)
    at hudson.FilePath$Mkdirs.invoke(FilePath.java:1252)
    at hudson.FilePath$Mkdirs.invoke(FilePath.java:1248)
    at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3084)
    at hudson.remoting.UserRequest.perform(UserRequest.java:212)
    at hudson.remoting.UserRequest.perform(UserRequest.java:54)
    at hudson.remoting.Request$2.run(Request.java:369)
    at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

```

The path on the container is a docker volume and the containers are all running with -u root, so I'm not entirely sure why I'm getting an AccessDenied. The only thing I can think is that, because of the previous "Remote call to DockerHostNode", it's trying to access `/path/on/container` which doesn't exist of the node itself. Any ideads? Here's the full error text:

# Answer
> 2 votes
I could not achieve changing the working directory using the suggestion by Vishwas, although I did try creating a Jenkins user inside the container with the same uid as the host. It just didn't work for me. I did however, realize that I didn't actually need to change the directory on the container because the workspace was already in the container.

I'm using the Docker Plugin from CloudBees. After reading their documentation, I realized the workstation folder I thought was on the host machine, was actually in the container. Part of the documentation stated:

> The above is a complete Pipeline script. inside will:
>
> 1. Automatically grab an agent and a workspace (no extra node block is required).
> 2. Pull the requested image to the Docker server (if not already cached).
> 3. Start a container running that image.
> 4. **Mount the Jenkins workspace as a "volume" inside the container, using the same file path.**

After seeing item number four, I ran `docker container inspect <containerID>` and I saw the workspace `/home/jenkins/workspace` was actually a volume mount that was mapped to the `/home/jenkins/workspace` on the Docker host machine as well. Using this as my workspace, I was able to build the various applications within the container and then moved the results to a second volume used for application collection later in the pipeline. It's not an exact answer to my original question, but it works!

# Answer
> 1 votes
If your master is running as Jenkins user, you should make a user in the docker image which you use as to create your node and do the key exchange. For example `jenkins`user with same UID needs to exist on both master and node docker container.

---
Tags: docker, jenkins
---


# Question
Title: How can I use a local file for a nested Cloudformation stack?
How and is it possible to use a local file for a CloudFormation stack besides an S3 bucket?

# Answer
I'm afraid you can't. The nested stack files must be in S3 as far as I know. You have a couple of options:

1. Have some **automation** in place (e.g. *aws-cli* \+ *shell script*, or better *ansible*) that when executed will upload all the templates to a designated S3 bucket and then execute the the CloudFormation deployment.
2. Use some **templating engine** like *Ansible + Jinja* or similar that can build a big CFN template from little snippets saved in separate files. With Jinja you can also use loops, lookup variables, etc, to build your final template file. Then again you can use the same *Ansible* playbook to deploy it. Or do it manually as it will only be a single file.

Hope that helps :)
> 5 votes

---
Tags: amazon-web-services, aws-cloudformation
---