# Question
Title: How to "docker-compose build" a service from code located in local git repository?
I have a git repository placed in local file system at `/home/nou/code/lib.git` Also I have `/home/nou/docker-compose.yaml` that is used to deploy a service based on the code from the repo:

```
version: '2'
services:
  lib:
    container_name: lib
    build: git://./code/lib.git/
    #build: git://./code/lib.git
    #build: git:///home/nou/code/lib.git/
    #build: git://file:///home/nou/code/lib.git/

```

As you can see, am trying to build lib service from the local git repo, but the only result I observe is such error (or similar):

```
nou@ubuntu:~/$ docker-compose build
Building lib
ERROR: error fetching: fatal: unable to connect to .:
.: No address associated with hostname

: exit status 128

```

How can I use local git repo to build services using docker-compose?

# Answer
> 6 votes
What you are specifically after (building from a bare repository on a local filesystem) isn't functionality offered by Docker, and by extension, docker-compose. Docker supports building from a few different URLs, but not bare repositories on local filesystems. You can try a few workarounds:

* Build straight from Github: `docker build https://github.com/docker/rootfs.git`
* Run a Git daemon to use the `git://` protocol: `git daemon --base-path=. --export-all` (in `/home/nou/code`)
* Running a Git daemon from within Docker, mounting your bare repository as a volume and building inside it via a mounted Docker socket
* Converting your bare repository to a normal one

https://docs.docker.com/engine/reference/commandline/build/#build-with-url

# Answer
> 4 votes
You can try this, it works for me:

1. Create a user named "git"

```
$ sudo adduser git
$ su git
$ cd ~
$ mkdir .ssh && chmod 700 .ssh
$ touch .ssh/authorized_keys && chmod 600 .ssh/authorized_keys

```
2. If you don't have ssh keys for "nou" user then, from "nou" user run

```
$ ssh-keygen

```

and accept all defaults.
3. Add "nou" user's ssh public key to git user ~/.ssh/authorized\_keys

```
$ sudo cat nou_user_home/.ssh/id_rsa.pub >> git_user_home/.ssh/authorized_keys

```

After this try below command from "nou" user:

```
docker build git@localhost:/home/nou/code/lib.git

```

If `Dockerfile` is not at the root of cloned repository then:

```
docker build -f folder_containing_dockerfile/Dockerfile
git@localhost:/home/nou/code/lib.git

```

If this all works fine then you can try with your `docker-compose` file.

---
Tags: docker, git, docker-compose
---


# Question
Title: What tool is able to analyze images by connecting to clair?
This clair documentation indicates that a number of tools should be able to scan images:

* clair-scanner
* reg
* klar
* clairctl

analyze-local-images

All of them have been tried, but none of them seems to work. All kind of issues occurred, including this:

```
2017-09-19 11:11:23.022625 I | Analyzing faild: Could not analyze layer:
Got response 400 with message {"Error":{"Message":"could not find layer"}}

```

Another issue that occurred using clairctl:

```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x518490]

```

> does this happen with all images including official like ubuntu, alpine, tomcat?

Using clairctl it was possible to scan postgres an other images that are available on dockerhub, but scanning local images failed

# Answer
> 0 votes
Clair-scanner is able to scan the images locally.

https://github.com/arminc/clair-scanner/issues/13

> ```
> docker run --net=host -d --name db arminc/clair-db:2017-09-18
> docker run --net=host --add-host postgres:127.0.0.1 -d --name clair --net=host arminc/clair-local-scan:v2.0.1
> ./clair-scanner nginx:1.11.6-alpine example-nginx.yaml http://127.0.0.1:6060 127.0.0.1
>
> ```

---
Tags: security, clair
---


title: Dynamic YAMLs for Docker Compose

text:
# Question
Title: Dynamic YAMLs for Docker Compose
Currently I find myself managing a set of embarassingly similar YAMLs for environment profiles. I see there a scaling trouble if there will be *lots* of dynamic environments (i.e. fluid acceptance test configurations, with disposable environments you can do that!)

Has anybody done a working implementation to build YAMLs dynamically on scale based on some generic represenation ( kind of CasC level 2 beyond some naive scripting), or if this makes no sense, why and which way is to prefer then.

# Answer
> 3 votes
One could use the same docker-compose file for dev, test, acceptance and production (DTAP) systems. Things that deviate per environment, e.g. passwords could be defined in an .env file.

If .env files are not a solution for some reason then one could create different docker-compose files, e.g. docker-compose.override (dev), docker-compose.test, docker-compose.acc and docker-compose.prd.

I personally prefer the first option, e.g. using .env files and a docker-compose.override for dev.

# Answer
> 1 votes
SaltStack is great at generating these kinds of files based on metadata you feed to it via "pillars" as Salt itself uses YAML files heavily. Salt uses a Jinja templating engine (though others are also available).

Similarly, Puppet also uses YAML heavily and is able to do the same using an ERB template.

If you find that some of this metadata are things like a listening IP address or a hostname, these systems automatically generate variables such as this using facter facts or salt grains respectively which can further reduce the amount of metadata you need to suply.

Chef is probably also capable of this, though I am not familiar with that configuration management system.

---
Tags: docker-compose
---


title: Scaling issues with Nagios as CaSC

text:
# Question
Title: Scaling issues with Nagios as CaSC
If Nagios server would use a CasC source of configuration in a larger organization is there a design flaw in this approach which will limit scaling up?

To avoid broken configuration, test instances of Nagios could be validated in a CI environment.

Usage scenario: imagine a central Nagios service with more users than you want to be able to break it, who would like to add their services for monitoring.

# Answer
> 2 votes
Personally I don't think using CasC itself will have any negative scalability impact.

Fundamentally CasC means the actual config files are not hand-maintaned, instead they're auto-generated following version-controlled rules - which can be a lot more reliable.

But once the config files are generated - the service using them functions just as it dit with manually-crafted config files.

If anything - using CasC should make such large nagios system more reliable - if done correctly CasC should reduce/eliminate the risk of human error in modifying the config files.

# Answer
> 1 votes
From my experience in managing a Nagios deployment with version controlled configuration files and CI/CD, it works really well. You can collaborate with other teams more easily since you can grant access to a git repository, and you gain all the benefits of CI/CD e.g. rollback and automatic testing. One thing that may be a bottleneck is how frequently you're reloading Nagios, but this would only be an issue when you have many deployments per hour.

Nagios itself also has a lot of limitations and is missing a few key features for a modern environment (scaling, configuration inheritance, APIs), but strictly speaking this doesn't detract from using CasC with Nagios.

# Answer
> 1 votes
Having a CasC setup means that you are in a better position to deal with any scalability issues that should arise later. It should take you a while to max out a nagios box. As long as you haven't starved it of memory or something you should be able to get thousands of nodes on a single nagios box. How many depends on how many checks you're doing and how expensive those checks are. But if you get to the point that a single nagios box is not enough being in a CasC situation gives you the power to shard things in a convenient way and go on with life. There would be much less manual intervention than if you had to split your configs by hand.

You might not reach the maximum for a single instance, but want the reliability of having local nagios boxes in each data center. Having CasC lets you do this more easily and transparently as well.

---
Tags: nagios, scalability, casc
---


# Question
Title: Set a Jenkins job to not to clone the repo in SCM
I have integrated Jenkins with Bitbucket using the Bitbucket Plugin. As per the plugin's Wiki, a given job will be triggered if the repository is set in the SCM of the job. As you know, if one set SCM in a Jenkins job, this is cloned in pre-build stage.

So far so good. However, the main purpose of the job I'm setting has nothing to do with the repository's content; instead, I just want the job to process the payload sent by Bitbucket. One could say it's not a big deal in terms of storage to clone a repository despite you really don't need it. I don't think so, adding unnecessary steps, consuming time and resources is not a good practice.

So, the question is: **Does anyone know how to set an SCM in a Jenkins job but prevent it to clone the repository?**

# Answer
Yes, definitely. I do this all the time. You can specify configuration options for your pipeline and one of them is `skipDefaultCheckout`, which causes pipeline to skip the default "Declarative: Checkout SCM" stage.

The `skipDefaultCheckout` option is documented in Pipeline Syntax and here's an example Jenkinsfile showing how to use it:

```
pipeline {
  agent { label 'docker' }
  options {
    skipDefaultCheckout true
  }
  stages {
    stage('commit_stage') {
      steps {
        echo 'sweet stuff here'
      }
    }
  }
}

```
> 27 votes

# Answer
In case you're not using the Declarative Pipeline, you can avoid checking out from SCM by:

```
node {
        skipDefaultCheckout()
        //...
}

```
> 8 votes

# Answer
I think what you want to achieve is processing a webhook payload in a Jenkins job. Using bitbucket plugin won't be necessary and it is probably strongly designed to clone the repo.

I belive this stackoverflow answer could help you.
> 2 votes

---
Tags: jenkins
---


# Question
Title: The best way to share volume with config data between multiple Docker containers
I have Docker image which contains config files of VirtualBox and I want to use this image for running multiple containers in parallel.

Config files of VM are being stored on host OS, however, I am afraid of issues with config data of VMs which may occur because of parallel accessing by multiple containers. All containers will be started by Jenkins automatically.

Therefore, could you please tell what is the best way to share data between containers to avoid data corruption because of parallel usage?

Please note: I can not run FreeBSD container under Linux and I can not setup FreeBSD as host OS to use FreeBSD containers as long as Docker is an experimental thing in FreeBSD and Kubernetes is not supported under FreeBSD.

# Answer
> 1 votes
One could use read-only volumes.

> ```
> docker run -d \
>   -it \
>   --name=nginxtest \
>   -v nginx-vol:/usr/share/nginx/html:ro \
>   nginx:latest
>
> ```

If one would like to use the same volume in multiple containers then one could specify the following when running each container:

```
-v vol:/usr/share/nginx/html:ro

```

Note: one could use `ro` to ensure that the volume is read-only to prevent that multiple containers will write at the same time to the same volume and prevent issues.

---
Tags: docker
---


# Question
Title: AWS Redshift UNLOAD not firing S3 PUT event
I am trying to use Redshift to UNLOAD data to S3 and then pick that data up with a Lambda function and process it. All of the constituent parts (the UNLOAD, the Lambda function) now work, however the Lambda event trigger (`S3:ObjectPUT`) is never firing. If I upload manually via the console, it all works. If I run the UNLOAD the data appears in the bucket, but the trigger event doesn't run.

I have tried creating / recreating the function (as per this forum thread - https://forums.aws.amazon.com/thread.jspa?messageID=768407&tstart=0), but to no avail.

Has anyone else had this - is there something I'm missing?

# Answer
> 1 votes
You did not tell how big data amounts you tested with, but if you test ok with little data block successfully and larger real situation fails, there may occur same as occurred here:

> The larger files are almost certainly being uploaded via S3 Multipart Upload instead of a regular Put operation.
>
> ..
>
> It sounds like the function is only subscribed to s3:ObjectCreated:Put events currently, and you need to add s3:ObjectCreated:CompleteMultipartUpload to the configuration.

---
Tags: amazon-s3, aws-lambda
---


# Question
Title: What's the impact of the General Data Protection Regulation (GDPR) on DevOps?
The General Data Protection Regulation (GDPR) is a set of rules to improve the protection of data about European citizens. Quote from this link:

> The EU General Data Protection Regulation (GDPR) is the most important change in data privacy regulation in 20 years.

Refer to GDPR Key Changes for a summary of what it is all about, especially the *Data Subject Rights*, such as:

* Breach Notification.
* Right to Access.
* Right to be Forgotten.
* Data Portability.
* Privacy by Design.
* ...

Some GDPR facts:

* GDPR will be ***enforced*** as of 25 May 2018 (that's pretty close).
* Quote from GDPR Key Changes (about *Territorial Scope*):

> ... will apply to the processing of personal data by controllers and processors in the EU, regardless of whether the processing takes place in the EU or not. The GDPR will also apply to the processing of personal data of data subjects in the EU by a controller or processor not established in the EU, where the activities relate to: offering goods or services to EU citizens (irrespective of whether payment is required) and the monitoring of behaviour that takes place within the EU. Non-Eu businesses processing the data of EU citizens will also have to appoint a representative in the EU.

For anybody who is old enough to remember the *Y2K*-hype back in the 90s (i.e. the impact of it on IT systems): IMHO addressing all those Y2K issues was a piece of cake as compared to the challenge ahead with this GDPR thing.

**Question(s)**: What is the impact of GDPR on DevOps, more specifically I wonder about:

1. What kind of changes will be needed in the DevOps to make them GDPR-compliant?
2. How can DevOps-practises help (facilitate, simplify, etc) a company to become GDPR-compliant, similar to how SCM tools helped in the 90s to become Y2K compliant?

# Answer
"Enforcing" means lawyers going after possible culprits. Frankly, in my opinion (and those of some folks around me which are more a mix between IT-guy and lawyer), nobody knows how that will work out exactly.

Those iLawyers expect a lot of very interesting court cases in the first months to tests out the water, and see what precedence cases are going to come up. There will probably be plenty of law firms fishing for nice fat fish to fry.

For me personally, I don't really see that much impact specifically regarding DevOps. The applications themselves need to be adapted as necessary, obviously. Also, the actual data storage, be it RDBMS, elastic search indexes or whatever, will have to be looked at.

Aside from that, one benefit of DevOps, specifically if you work with some kind of container system and infrastructure-as-code is that you should generally know *exactly* where your data, log files etc. is stored, and what kind of them you have; much more so than in classical servers, where your data and especially logs might be strewn all over the place. It should also be very easy to regularly roll your data as applicable, i.e., get lost of old stuff.

So it can help to, for example, take your catalogue of i-a-c files and walk through them diligently - you can then be pretty sure to have found everything you need to be aware of.
> 3 votes

# Answer
With DevOps being a marriage between Dev and Ops (and even SecOps), I see different relationships with each one of them.

From the Dev aspect DevOps can help with GDPR compliance just like it can help with any other type of compliance required from software products: via (good) compliance QA testing.

As long as the compliance requirements are well defined it is possible to devise corresponding QA testing to verify those requirements. Simply by incorporating those QA verifications inside the products' DevOps chain - in the CI/CD pipelines, for example, the products' compliance can be measured and controlled.

From the Ops (and SecOps) prospective things are a bit different (I'm less familiar with these), but I presume GDPR will impose additional requirements which I suspect would translate into additional operating policies.
> 0 votes

---
Tags: toolchain, data-protection, gdpr
---


# Question
Title: What are best practices for implementing and using Elastic in an organization?
At the moment there are multiple microservices that log to individual files. If there is an issue then the team is debugging like headless chickens. In order to prevent this the aim is to introduce aggregated logging, e.g. elastic. What are best practices to migrate to such an aggregated logging system?

**Current idea**

What endpoint should be configured?

# Answer
> 5 votes
One best practice would be to log everything to the same location. Your "Current idea" suggests that you'd be running the Elastic stack locally, which is great for a local development environment, but for anything outside that you want to be shipping the logs from all your microservices to the same location. This enables you to do correlation between logs, and have a "single pane of glass" approach to your logging.

To achieve that, you're going to have to run it on a server or use a managed service such as Elastic Cloud or AWS Elasticsearch. You can install the components through package managers e.g. yum or apt, https://www.elastic.co/start, or it's also possible to run it in Docker. The `stack-docker` repository is great for running locally, but it's not suitable for any sizeable deployment due to a lack of scalability.

The endpoint you ship to will depend on what method/protocol you're using to ship logs, for example Nginx running on an AWS EC2 instance might use Filebeat and ship to a load balancer that sits in front of Elasticsearch, or if you're using Docker you might use the gelf Docker logdriver to ship your logs to Logstash.

Some more best practices:

* Log all your messages in JSON, with a focus on objects/fields rather than long string messages.
* Exceptions should be actionable, anything else can be a warning.
* Avoid multiline logs.
* Use correlation IDs. If your app is passed a correlation ID it should include it in every message triggered by that call, otherwise it should generate one. The correlation ID should be included in calls to other microservices.
* Err on the side of logging more rather than less, you never know when you may need that bit of data, or what trends you might find by analysing an otherwise inconspicuous piece of data.
* Separate your environments logically so you can have different retention periods and archiving on your logs.

---
Tags: elastic
---


# Question
Title: Sharing Gems between Chef and System Ruby
I know about the `chef_gem` resource obv, but in the interests of good system hygiene I would like to avoid as much as possible duplication, so I would like to share Gems between the embedded Ruby within Chef, and the system-wide one. Additionally I have become aware of a troubling situation: the Gems I have in the system Ruby come from the known-good local Yum repo and are installed via RPM over the LAN whereas `chef_gem` goes off to the public Internet, and recent events in the Python community have made me very concerned about this. I have tried setting `ENV['GEM_PATH']` in my recipe both before and inside a `ruby_block` but this appears to have no effect, I get a failure on `require`. What am I missing?

# Answer
> 2 votes
To take your points one by one:

> but in the interests of good system hygiene I would like to avoid as much as possible duplication

Can you ensure your system ruby will evolve at the same pace as the Chef's ruby ? Main point of chef coming as a bundle with its own ruby comes back from Chef 10 which was installed via a simple gem command and was usually a problem switching between ruby needed for applications and ruby needed by chef, the same goes for gem dependencies. I would advise against mixing system and chef ruby.

> the Gems I have in the system Ruby come from the known-good local Yum repo and are installed via RPM over the LAN whereas chef\_gem goes off to the public Internet

While I understand and agree with the worrying about recent pipy events, there's two things here:

* Your system ruby install gems from yum repository because your system ruby's `gem` command has been tweaked for that, that's specific to your distribution. -
* Chef's ruby use usual ruby method going to rubygems.org, as any ruby installed by rbenv or any other way than yum install would do.

If you really want to use a local repo you'll have to set up a gem server (alternatively, some artifact repositories can handle that also) and set-up the sources for gem using chef's `gem sources` in `/opt/chef/embedded/bin`.

If you're using a chef client above 13.0 your can use the `rubygems_url` in client.rb to tweak that:

> ```
> rubygems_url
>
> ```
>
> The location to source rubygems. It can be set to a string or array of strings for URIs to set as rubygems sources. This allows individuals to setup an internal mirror of rubygems for “airgapped” environments. Default value: https://www.rubygems.org.
> Changed in Chef Client 13.0.

And finally:

> I have tried setting ENV\['GEM\_PATH'\] in my recipe both before and inside a ruby\_block but this appears to have no effect, I get a failure on require

Chef omnibus package hardcode some values specially to avoid interfering with the system ruby, I think the GEM\_PATH environment variable is ignored when searching for gem you `require` within the recipe code. I don't think there's a workaround for this, and again you're heading to a compatibility problem on mid term as you'll have a drift between chef's ruby and system ruby.

---
Tags: chef, security
---


# Question
Title: How can I see HEAD temp pointer reference in Git?
Not the current working HEAD REFERENCE which points in our current branch.

# Answer
> 2 votes
You can directly see `HEAD` if you look into the file `.git/HEAD`. What you find in there is all there is to it, there is nothing more.

`HEAD` does not, strictly, point to the current branch but is the commit that is currently checked out. It is, by definition, the first entry that `git log` lists.

If you do a `git checkout branchname` then, yes, indeed, `.git/HEAD` will contain the `ref` of the branch you checked out, probably something like `refs/head/branchname`. This in turn is literally the file that contains the actual commit hash; you can see it in `.git/refs/head/branchname`. But if you do `git checkout commithash` then `.git/HEAD` will contain said hash, and you will be in a "headless" state (git will give you a longish message explaining what that is). In this case, there is no "current branch".

---
Tags: git
---


# Question
Title: How to route to specific, of multiple datacenters, with a single domain
The data of each customer only exist in one of several global datacenters, that all answer to the same hostname (ex. api.company.com)

**Only a single domain for all customers globally**
I want to make sure that all global customers only have to use a single domain: api.company.com to access their data, and not something like api-region3.company.com, but it may not be possible.

**GEO Location DNS may go to wrong datacenter**
I cannot use geo-location DNS directly, because that would mean that someone might end up accessing a datacenter that does not contain the data for that customer.

**Possible solution**
I could use a proxy at each of the datacenters to relay the request to the correct datacenter, based on some information in the request, but I'm not sure if there's some better established industry standard way of doing this, without duplicating the data across all datacenters. In case proxying/relaying is the best option, what is some tried-and-tested options for actually setting this up?

# Answer
> 4 votes
Proxying is gonna cause you some issues when someone takes a trip to China and tries to access data in the USA, depending on what your app does they could cause you some $$$ issues when your Chinese POP is sending/receiving all of their traffic each way.

Instead I suggest:

If your API is authenticated and you're not going to replicate any of the data and you're using AWS then I'd use lambda@edge to do what you need.

You can use GEO-DNS to route to a lambda function which handles authentication. This lambda function can also hold a record of where users need to go and issue a redirect to the correct endpoint.

This does mean that you'll still need domains like `api-region3.company.com` but it will allow `api.company.com` to perform the task of being universal. It's down to the client application then to hold on to that URL? At least using redirects instead of proxies should help with network traffic.

---
Tags: amazon-web-services, distributed-systems
---


# Question
Title: Ways to the DevOps hiring process optimization through CALMS paradigma?
Much of DevOps recruitement happens to follow along the lines of keyword matching, which leads in my opinion to solely technology focus.

Now, DevOps is about so much more than just technology, and DevOps Engineer is not just a better system administrator with some coding skills.

**Senior DevOps role/profile means to me also offering seniority in many other foundations and practices beyond infrastructure and software engineering skills** like Lean, Measurement and being open and communicative (who asks DevOps hires for their communication skills, honestly?!)

So, can a job ad / interview be more efficient in some way - for example, by applying questioning CALMS categories as well? - Leading to questions like "now, how do you apply lean principles? How were cultural aspects addressed in your recent DevOps projects?"

Further elaboration:

* **C**ulture (e.g. strategies for conflict management and attitude to failures, own and others')
* **A**utomation (here you ask about Puppet/Docker etc skills)
* **L**ean (foundations of Lean? Waste types?)
* **M**easurement (ask for tools like JMeter but go also to things like sampling, data modeling..)
* **S**haring (obviously knowledge management and according tools)

UPDATE - so why wouldn't employers/recruiters structure the interview by CALMS as shown below (additionally, the "automation" section could be formulated along the DevOps model (document link, readonly)?

Side note - so for example is actually not just a soft skill anymore, for DevOps it is one of the core skills - like all the others in this domain.

# Answer
> 5 votes
This is a brilliant idea, also because of Daniel Kahneman who showed that if you split a single score into 5 weighed scores and add numerical criteria and bounds to those, you will significantly reduce bias. You could design not just the resume scoring, but the entire hiring process, with phone screens, onsite interviews, everything in this way. It would significantly reduce the inherent bias of the interviewers. We have actually started to do something similar for all hiring.

Obviously, inside each area, your should add weight to what is important to the company for the position, but you are hiring a well-rounded engineer and you want someone who will propose major changes to how your organization operates, you are not simply hiring someone for specific skills to work in a limited area. Many people simply see this role as a higher paid Release and Build Engineer and if that is the case, that is what you should hire and advertise for.

For a DevOps hire, I would suggest replacing the Lean with Learning. It is originally CAMS and even though some extend it to CALMS to include Lean, that is somewhat restricting as DevOps is based on so much more than just Lean. It is also Deming's ideas about Special and Common Cause Variation and System Thinking, Nash's Equilibrium (if each optimizes for themselves, the result could be suboptimal, compared to when everyone includes the interest of the group), Shewhart's Statistical Process Control, Goldratt's Theory of Constraints, Taleb's Anti-Fragility and so many more.

This would also allow you to include participation in conferences in Learning and presentations at conferences or meetups as Sharing. In a position where you are not always a part of a team or your company might not be big enough to have your peers as your co-workers, it is even so much more important to establish and maintain out of workplace relationships and learning opportunities. We generally grouped those two under Culture.

I would personally put under Culture the soft skills required to be effective in improving processes at your organization. CMMI, Kanban, Work in Progress limits, Agile practices, etc.

JIRA seems more like Sharing tool and Git is more closely related to Automation.

# Answer
> 2 votes
**EDIT**

I believe that this depends from organisation to organisation and what a DevOps / Senior DevOps is expected to do, therefore, your first sentence is 100% accurate. Because, a DevOps should be able to use a set of tools that the company uses and also improve or bring new set of tools which enables the company and its developers to work faster and waste less.

In my opinion a DevOps should have strong SysAdmin skills and obviously coding skills as Puppet, Chef, Python, Bash will be used extensively as well as some knowledge of the code that goes on the servers at least to be able to do minor debugging on why the application doesn't behave as expected from one environment to the other.

Now, as a Senior DevOps, CALM may be applied however, the Lean and Measurement principles might / might not apply. For example we are developing applications using Chef / Puppet / Ansible to automate the mundane things and keep everything in sync which obviously saves time and produces less *waste*.

Regarding measurement, I am not sure if that's applicable in most of the cases. However, the other CALM principles are part of a DevOps position.

Having good communication skills is also important as a DevOps, but more important as a Senior DevOps because you will have to not only deal with your team and share knowledge and with the developers as you are there to support them, but also you might have to create reports and keep presentations in front of the management.

I like the spreadsheet you added, and is good to have a point system, however, some of the companies are also adding more skills / technologies in a job advert than required.

Also, after a telephone interview (if there's one) I'd find it useful that in an interview you will be given some problems to resolve or at least to show your debugging process and how you will behave in given situations. Personally, I dislike written tests as I believe there are 'n' ways of solving a problem, and also, sometimes *Google* is your friend, as you are not expected to know everything by heart.

As a DevOps / Senior DevOps I believe there's a line between the applications that are used and knowledge. You could be amazing at using these new / old tools or write code, but when it comes to debugging or just understanding what the problem with a server, Jenkins job might be you are not able to do so.

Finally, the spreadsheet presented I think is a way of assessing a DevOps knowledge also for a Senior position I might add there some Interpersonal and Management skills to make it complete.

And when it comes to the selection process you could have a look at the spreadsheet and pick the person with a score that you believe is the right one for your organisation as well as keeping in mind his(er) behaviour during the interview and the way (s)he presented / answered those questions.

---
Tags: culture
---


# Question
Title: Does Docker allow containers to be OS agnostic?
We currently use a Windows based server to host 2 VMs. One VM uses Windows with a Windows app and the other VM uses Linux with a Linux app.

Is the below scenario possible?

Bare server with Windows 2016 (no VMs). Install Docker engine. Install Windows app container. Install Linux app container.

Does the Docker engine abstract away the OS such that this configuration will run both apps? Meaning that the Docker engine works as an "interpreter" akin to how Java code can run on various OS's?

# Answer
> 11 votes
> Does the Docker engine abstract away the OS such that this configuration will run both apps?

No, it does not. Docker uses containerisation as a core technology, which relies on the concept of *sharing a kernel* between containers. If one Docker image relies on a Windows kernel and another relies on a Linux kernel, you cannot run those two images on the same OS.

When you install Linux Docker on Windows, you're actually setting up a Linux VM in Hyper-V, the LinuxKit details are here. You can open the Hyper-V console and see a VM called "MobyLinuxVM". There's a lot of work done to abstract this away from you e.g. the `docker` binary can connect to the Linux VM and run the same commands you're used to, networking between the host and VM tries to be seamless, volume mounting, etc. Ultimately it's not Docker that lets you run a Linux Docker image on Windows, it's just plain old virtualisation.

# Answer
> 2 votes
Yeah go nuts, you can run whatever you want in the container so long as the host supports virtualisation and can run the docker binary!

---
Tags: docker, containers, linux, windows, servers
---


# Question
Title: Deploying to VSphere with Packer and/or Terraform?
**Scenario**:

Creating multiple VMs and deploying them to VSphere. Current development uses Packer and Ansible to provision a Fusion VM, with the aim of using Terraform to deploy to VSphere.

**Issue**:

I've been having loads of issues with uploading Fusion VMs to VSphere and having to use work arounds such as this, and been wondering if there is a better of doing this.

**Question**:

I'm questioning the use of Packer in this scenario (which deploys to Fusion), and doesn't seem to currently support a Packer-\>ESXi (VSphere) builder unless it's an ISO.

Would it be possible to remove Packer entirely from this scenario, so instead of building a Fusion image, provisioning the image, and uploading it to VSphere, use Terraform to deploy an ESXi image (template), and provision that image in-place using Ansible as a provisioner? (such that everything is done in VSphere rather than Fusion then VSphere).

What are the advantages and disadvantages of doing this?

# Answer
> 1 votes
I'm using vagrant with vsphere plugin, starting from a 'base' image already present on vsphere.

The new machine is then fully configured (inlcluding middleware) by chef after vsphere customization.

That's one in many possibilities, unsure this question lead to actionnable answers out of everyone 'workaround'.

---
Tags: ansible, deployment, terraform, packer, provisioning
---


# Question
Title: Docker Container always stopped and started
I am using docker compose to build two containers, one is nodejs with pm2 and another is nginx.

The nginx one runs well but the nodejs with pm2 one keeps restarting: it runs for one second, then stops for ~10 seconds, then starts again and runs for one second, as you can see this is the output of `docker ps -a`:

I have used `docker attach 'ID'` but it still restarted.

Using `docker logs` shows nothing, but jenkins says that pm2 was started successfully:

This is the content of my Dockerfile:

```
FROM node:8
MAINTAINER test
RUN npm install -g pm2

WORKDIR /var/www/marketing

EXPOSE 80

RUN npm install
RUN pm2 start npm -- stage

```

If any of you could help me, I will update this question if you guys have any request. Thank you

# Answer
> 6 votes
You should keep at least one service running in foreground to keep the container running, you can do that using `ENTRYPOINT` or `CMD` or both. For more info check Dockerfile: ENTRYPOINT vs CMD.

You can also use a docker image for pm2 from docker hub, that will save your time and you can check the Dockerfile for it to see how it is written. Check the keymetrics/pm2-docker-alpine project, for example.

# Answer
> 4 votes
To give visability to the error messages mount a volume to your container so after it crashes the logs are available ... problem otherwise is once crashed the logs go away ... for example this docker-compose.yaml excerpt is how to mount a volume

```
  nodejs-enduser:
    image: your-image-name
    restart: always
    volumes:
      - /pathA:/pathB

```

pathA is full path to some logging dir on host where container is launched from

pathB is path processes write to from inside container

If you wrap your docker launch using something like supervisord then its config defines where standard out / stderr are written to ... that is /pathB ... then the app inside the container just writes to stdout / stderr

Alternative approach is to spin up an empty container then once running you log into it using `docker exec -ti container-id bash` Once inside you then manually issue your Dockerfile commands to install and execute the app ( nodejs ) where you can then see the errors as they happen ... here is the Dockerfile for this do nothing container

```
FROM ubuntu:16.04

ENV TERM linux
ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update
RUN apt-get install -y wget  curl   net-tools

# COPY .bashrc /root/
#  list your normal Dockerfile code COPY steps here

CMD ["/bin/bash"]

```

now build above Dockerfile

```
docker build --tag stens_ubuntu .

```

and run

```
docker run -d  stens_ubuntu  sleep infinity

```

now log into it

```
docker exec -it $( docker ps | grep stens_ubuntu | cut -d' ' -f1 )  bash

```

here you are at the command prompt inside the running container where you now issue whichever commands you need to run your normal app as listed in your original Dockerfile ... beauty is now you are interactively seeing the error happen ... until it crashes however the messages will be shown

# Answer
> 2 votes
Docker monitors the process started by ENTRYPOINT or CMD and since you don’t have such entry in your Dockerfile your container will terminate early in its lifecycle.

Also be aware that the processes that are started via ENTRYPOINT or CMD should run in **foreground**, not started as a daemon.

---
Tags: docker, node.js
---


# Question
Title: Why should a developer care about Docker?
Generally a developer cares about satisfying business requirements. He / she might have the expertise in a particular stack or a framework. But should he / she make an effort to learn docker and it's various deployment methods (swarm, kube, mesos, etc) ?

Simply put why should a developer care about docker ?

PS : The parent question to this post is Implication of introducing docker to the development team

# Answer
> 8 votes
Probably not the answer that you're looking for, but an answer nonetheless :)

Learning about docker and its deployment methods could actually be included in the business requirements by making it part of the project or team development environment, just like the code language(s), version control system, compilers, test infrastructure, etc - to work in that team or on that project one needs to know about and use all of these, can't "bring your own" (in most cases).

Things get a bit more complicated if by "a developer" you actually mean the majority of or even entire development team. Pushing a tool in the development environment without actually having any of the developers supporting it will be really tough. Spend the time to create one such supporter first from the team's technical leadership.

Side note: it *might* also not be necessary for each and every developer in the team to become a docker expert. Pre-established usage recipes wrapped in simple, cheatsheet-ready commands often allow developers to use docker-based solutions without actually knowing too much about their inner workings, which could be fairly acceptable, especially in large teams. Just like being able to contribute code without knowing all the details about how the end-product is being built.

# Answer
> 7 votes
I'll give you my perspective. Developers should care about docker as there are other developers who are willing to use docker and have already built an expertise in it. They are willing to take up the roles of a DevOps engineer along with being a developer. So the Ops part of DevOps is what they are now building expertise on.

These days, you'll find more and more guys who can develop, orchestrate, automate tests, automate jobs and build tools to monitor and take this complete package into production single-handedly. These are the guys who are pushing docker and other tools among the developer community.

Also, the tide of market is towards virtualization, auto-scaling, automation, machine learning and docker fits in all of these. It has become very imperative to use docker. The businesses are willing to pay 2x for a single guy who takes all these responsibilities and when there is demand for such guys, the supply will also begin. This is from point of view of an employee-employer.

Technically, in the organisations I have worked, there are separate development and DevOps teams, although they work very closely for deliveries. The DevOps engineers and developers share a vast majority of skill-sets here and hence there is a negotiation of duties sometimes.

The bare minimum a developer can do is to share his binaries, but he needs to understand that the binaries will be used to run inside a docker container and for that he needs to understand how docker works. For kubes, swarms, mesos etc, the developer may not even care what's being used, but basics of docker should be very well understood by the developer and a mindset should be there from the beginning to build the application loosely coupled for re-use as micro-services. If the application is built from that mindset (which requires basics of docker), then the DevOps engineers can take it up from there to auto-scale, orchestrate, test, deploy and monitor.

Also, most of the times there is no one size fits all kind of thing. A developer does not know clearly how to build a docker friendly app and a DevOps engineer quite rightly does not know the internals of the app building process. Hence, most of the times, organisations prefer to give both these tasks to the same guy to speed things up. If there are separate things, then a continuous feedback mechanism is required from the DevOps team to the dev team for making the apps more futuristic and docker/cloud/scaling ready.

# Answer
> 7 votes
It is not about Docker or any other containerisation technologies out there.

Containers like Docker, rkt, etc. are just way of delivering your application in similar fashion to static binary. You are building your deployment that it contains everything it need inside and end user doesn't need anything more than runtime.

These solutions are similar to fat JARs in Java, where everything you (in theory) need is just runtime (JRE) preinstalled and everything Just Works™.

---

The reason why developers need to understand (they do not need to learn how to operate such tool, only why this is needed) orchestration tools is that this allows you to have some advantages over "traditional" deployment.

## Cattle, not pets

EngineYard have written good article about that. Whole point of that is that when your server dies, then you shrug and wait as new will appear. You treat them as cattle, you have tens, hundreds, thousands of them running and when one goes down neither you or your clients should be ever aware of that.

Orchestration tools achieve that by monitoring status of all applications (pods/jobs, whatever) in cluster, and when it sees that one of the servers stops responding (goes down) then it automatically moves all applications that were running on that server somewhere else.

## Better resource utilisation

Thanks to orchestration you can run multiple applications on one server and orchestrator will track resources for you. It will rearrange applications when required.

## Immutable infrastructure

Thanks to the automatic failover handling in orchestrators you can ran your custom images in the cloud as is. When you will be needing update, you just build new image, set your Launch Configuration to use that one now and just roll. Everything will be handled for you:

1. Create new server with new configuration.
2. Kill one running server.
3. Your orchestrator will move everything to other machines (including new one).
4. If there are any old servers left then go to 1.

## Simpler operations

* Not enough resources? Add new machine to cluster.
* Need more application instances? Increase number and go on.
* Monitoring? Done.
* Log management? Done.
* Secrets? Guess what.

---

**TL;DR** Whole point isn't about Docker but about orchestration. Docker is just a extended version of tarball/fat JARs that is required for proper orchestration.

# Answer
> 4 votes
Here are for example some arguments from a blog post published back 2014 and titled in way quite matching your answer:

> * Much more flexible injection of new technologies into the environment
> * there is still a massive pain point between committing the final tested code and then getting it running on the final production servers. Docker vastly simplifies this final step
> * Docker makes it trivial to keep legacy OS, no matter what flavor of Linux you are running

From: https://thenewstack.io/why-you-should-care-about-docker/

# Answer
> 4 votes
If you are running your production in docker container it's crucial that those container are being made by the same developers that have build the app running on them. Who else is better place to know what external dependency are needed and so on ... ?

Also pipeline can fail at any step during a CD, particularly when it's the docker image build step, sometimes it's a file who is missing or a lib that's needed.

At work we've introduce all devs to docker explaining them the basics to build the dockerfile in order to serve their app, also we made the pipeline easy so one can only add a name and a dockerfile and its app will automatically be build on the next push regardless of the tech running it.

Docker quickstart is really a great introduction to do so, after what the devOps team guide the dev in their choice of distro (a lot of them don't know things like `alpine`).

Our job is to provide them an easy access to the tools, they do the rest so they can fix it when something is wrong. Docker is really a part of the development process and the devOps team provide them docker images that match our needs and that are easy enough so it take only a couple minute to create a new app and deploying it without assistance.

# Answer
> 2 votes
Docker gets lots of press and blog mentions which leads to developers getting interested in using it. For some people it is the interest in playing with a new technology or understanding how things work. For others it is a desire to add keywords to their resume. Either way, the more developers know about how things work and how they get deployed the less surprised they will be later. From what I've seen there's a decent amount of pre-existing interest in this so it shouldn't be that hard to encourage it further.

# Answer
> 0 votes
Well, if you ever used VMs for testing you may want try to use containers and docker is actually a great stuff for testing and it's much more simpler to use instead of LXC :)

---
Tags: docker, project-management, orchestration
---


# Question
Title: Trigger PagerDuty alert after a number of incidents
Is it possible to only have an alert triggered after a certain number of incidents from an integration?

For example, if my application reports a single non-critical failure of a kind, that's worth-while to troubleshoot, but probably not so bad as to wake up somebody in the middle of the night.

However, if the application is reporting the same failure over and over again, it's a symptom of a bigger problem that somebody should look at ASAP.

Any ideas?

# Answer
> 2 votes
The easiest way to build this would be to have some sort of metrics-based alerting system like Prometheus, datadog, etc. Those systems allow you to have a counter that gets incremented and you can see how many incidents happened in a given period of time on a pretty graph. Most metrics systems will tie nicely to PagerDuty for the alerting piece. Having any of these metrics systems functioning will also give you a historic baseline for your systems that is often incredibly handy in figuring our what went wrong and when.

The threshold should probably be number of incidents over a given period of time. If you just pick an arbitrary number, say 100, for this you will always eventually get to that limit. If you see 10 incidents in an hour that could be bad for you.

The hardest part of this would be that I don't think PagerDuty makes it easy to access metrics about the number of incidents. You can look at pretty graphs in their UI, but there's no easy way to feed that into your metrics system. You would need to add a bit of code to the other places you have creating incidents and have them increment this counter.

# Answer
> 0 votes
The way we handle this is to run our alerts through Sensu first. You can configure a Sensu check to require multiple failures to alert, and then also configure the check's notify to get sent to PagerDuty. This way, by the time PagerDuty hears about it (and pages you), it has already passed the "N failures in a row" criteria.

If you need to check not for number of failures in a row, but rather N failures across a time range, then that's a good use case for adding elasticsearch or graphite or similar into the mix. Whenever there's a failure, log an error or increment a metric, and then your check can simply look over the time range and see if the aggregate is above the limit.

---
Tags: monitoring, pagerduty
---


# Question
Title: How to Configure jenkins email notification for some commercial email services?
For configuring jenkins email notification with some corporate email server or commercial email services.

How we can get mail server setting details like mail server name, port number, using SSL, etc. for such email services?

# Answer
You may consider these two Jenkins plugins for it:

https://wiki.jenkins.io/display/JENKINS/Mailer

https://wiki.jenkins.io/display/JENKINS/Email-ext+plugin

The first one is mostly basic and the second one is extended for additional user customizations. All necessary instructions regarding configuration, etc are described there.

> > > How we can get mail server setting details like mail server name, port number, using SSL, etc. for such email services?

I am afraid that the best approach here is either refer to your system administrator who is resposible for it or contact with the provider who provides mail services.
> 1 votes

---
Tags: jenkins, configuration
---


title: Creativity use case scenario?

text:
# Question
Title: Creativity use case scenario?
i am a devops learner, just finished graduation actively looking for jobs in the field of DevOps.

I am learning many tools like GIT, Jenkins, Chef, Docker , Nagious, ELK, AWS, etc.

Can anyone link all the DevOps tools together and give me a scenario where and how a company uses these tools at different times?

# Answer
> 3 votes
Your question is quite vague/generic.

People store their source code (or generally, files) in git; let Jenkins run jobs (like CI/CD pipelines), use Docker to put their software into well-defined containers, Nagios to monitor their systems, ELK to collect, store and visualize (mostly logging, event) data, AWS to host their stuff, and Chef to manage their configurations.

If you are studying that and learning about it, then you should have gotten that kind of information, really. The scenario in real world companies is simply to use those tools for what they are made, in different combinations as needed.

If you really have just gotten some deficient information from your college or wherever you studied those things, then get a book like The DevOps Handbook and give it a read. It should clear up how all of this would work together in the real world.

---
Tags: docker, jenkins, amazon-web-services, chef, kubernetes
---


# Question
Title: How to prevent hug of death on EC2 Instance?
I've a iOS app on the app store and recently I've received a huge surge of traffic to my landing page hosted on EC2 and resulted in the page not responding, luckily I managed to recover it by restarting and upgrading the instance to a t2.medium.

Now I'm looking to hire someone to implement a technology to prevent the same death again. My experience only stretch that far allowing me to understand basic devops stuff but not enough for load balancer on AWS, I want to know what is a affordable implementation for my instance.

My landing page and iOS app backend are hosted on the same instance.

# Answer
If you want something quick to get this sorted without much more knowledge I'd recommend elastic beanstalk. It's another AWS application which will handle the load balancer configuration and instance scaling for you.

There's no extra cost on top of the load balancer and instances so you can keep using t2 type instances but let elastic beanstalk scale as much as you want to help.

Auto scaling isn't instant and in times of spiking traffic will take a short amount of time, minutes normally, to be able to handle a spike but it will be much better than manually scaling the instance size and is really easy to get to grips with.
> 8 votes

# Answer
I would recommend auto-scaling as mentioned above with the addition of some CloudWatch alarms to start the auto-scaling process when specific thresholds start to increase, not when it's already to far gone.

For example; configure CloudWatch to monitor your server, when the CPU is at 50% or higher for a period of 30 seconds or more start the auto-scaling process.

This may not be completely faultless but it's easy to do via some online guides and all configurable via the GUI.

Also, if your landing page is static why not host that on a free-tier t2.micro and use another t2.micro free-tier for your app?
> 1 votes

# Answer
I’d love to help with this if you’re looking for some help. Depending on your page you may not need ec2 at all. For instance if your serving something static or JavaScript it can be served from s3 with a cloudfront distribution. Or we could possibly use an auto scaling group if absolutely necessary.
> 1 votes

# Answer
There are two general strategies for dealing with traffic surges: increasing capacity and reducing cost.

Increasing capacity means auto-scaling, which everyone was very excited about when public clouds first became available. In its most basic sense, this will boot up more webservers for you based on load and add them to a load balancer, but since can be a pain to manage, there are more automagic solutions as well, like Elastic Beanstalk.

The trouble with automated capacity expansion is that its also automated bill expansion - 10x normal traffic means 10x servers means 10x money you have to pay. That's why, while it's a useful strategy to keep in mind, I think you should always start by seeing how much you can cheat.

By cheat, I mean cache, which rests on the idea that most of the time you can give users slightly out of date data and they won't notice, and that can save you tremendous amounts of time. Imagine that you have a page that you decide it's ok if it's five seconds out of date, and it gets 20 req/s. Without caching, you're running that calculation 1200 times a minute, whereas with caching it's only 12. You can see how this can make a tremendous difference.

There are of course many types of caching, and a successful website will use several of them. But for your use case, there are two pretty good and easy options.

The first is to make the site completely static. This assumes that you can do so, but if you can, then you just have Nginx serve up the html directly, and it can serve *tons* of requests with no sweat.

If you need some level of dynamicity, then doing some full-page caching is a good option. Nginx has some capability to do this, but I really like Varnish because of its flexibility.

Whatever option or options you go with, make sure you do load testing to validate that you've set it up properly; sometimes fixing one spot exposes a new bottleneck.
> 1 votes

# Answer
I would like to share our experience with AWS. We deployed our application on EC2 and face the same problem and also high cost. We deployed our application **Amazon EC2 Container Service** although our application was monolithic, but we achieved

* Availability
* Cost Effective
* Scalability

Application load balancer will handle the traffic and will route traffic to the healthy instance and you can run multiple tasks of same services without worrying of scaling and balancing the load.

> This architecture makes it easier to implement failure isolation. Techniques such as health-checking, caching, bulkheads, or circuit breakers allow you to reduce the blast radius of a failing component and to improve the Overall availability of a given application.
> 0 votes

# Answer
This depends a lot on the specific architecture, but for example:

* Front load your website with CloudFront to reduce load on the host
* Use client side hosting service in something like S3 for scale
* Elastic Load Balancer with an Autoscaling group
> 0 votes

---
Tags: amazon-web-services, amazon-ec2, load-balance
---


# Question
Title: Jenkins node (agent) on shared web server - what are the right permissions
I have a shared web server with multiple sites and each site has a dedicated user used as the owner and with the group apache.

I want to install Jenkins node on the server, but how will it be able to change files and run commands like `git pull`, I think that using `runuser` will miss the point.

The script on the node will basically run `git pull`, `drush` commands (Drupal sites), rsync from remote server and more. I will also need to run chmod and chown at the end, but maybe for those commands I will just use the sudoers file (?).

Thanks.

# Answer
A not very devopsy, but quite simple and effective solution with no security implications (no credential exchange or authentication required) would be to establish a file-based "message" exchange scheme, in a "mailbox" - a well-known filesystem location (setup once and owned by `root`) with 2 directories:

* one owned by the jenkins user which creates **request** files containing deployment request information, one for each site
* one owned by the apache group where each site dedicated users create their own **response** files containing request handling information for deployment requests for their site

When jenkins processing reaches a deployment stage for a particular site it creates a corresponding **request** file with the necessary information in it.

Each site user periodically (cron-driven, for example) checks for **request** files pertaining to their site, handle the request as appropriate, following their own site policies and provide status updates in the respective **response** files, which the jenkins user checks periodically.

When a request handling is completed the jenkins user removes the **request** file, signalling that it received the "message" and then the site user periodical job can remove the corresponding **response** file.

The names of the **request** and **response** files can be used to encode the particular site and request identification, so that the periodical checks don't have to fumble through multiple files.

The scheme can easily work across machines (if, for example, some of the sites are migrated to other servers) simply by placing the "mailbox" on a shared filesystem accessible from all those machines.

OK, an example, as requested. Just a basic skeleton, in python, hopefully self-documenting.

Prerequisites:

```
sudo mkdir -p /var/message_box/requests
sudo chown jenkins /var/message_box/requests
sudo chmod go-w /var/message_box/requests
sudo mkdir /var/message_box/responses
sudo chgrp apache /var/message_box/responses
sudo chmod g+w /var/message_box/responses

```

The `mailbox.py` file:

```
#!/usr/bin/python2.7 -u

import logging, os, re, getpass, sys, time, yaml

class Mailbox(object):

    base_dir = '/var/message_box'
    request_filename_format = '%s.%s.yaml'  # username.id.yaml

    def __init__(self):
        pass

    @property
    def request_dir(self):
        return os.path.join(self.base_dir, 'requests')

    @property
    def response_dir(self):
        return os.path.join(self.base_dir, 'responses')

    def msg_filename(self, user, request_id):
        return self.request_filename_format % (user, request_id)

    def request_file(self, user, request_id):
        return os.path.join(self.request_dir, self.msg_filename(user, request_id))

    def response_file(self, user, request_id):
        return os.path.join(self.response_dir, self.msg_filename(user, request_id))

    def create_msg_file(self, user, request_id, data, is_response=False):
        assert user and request_id and data and isinstance(data, dict)
        msg_file = self.response_file(user, request_id) if is_response else \
                   self.request_file(user, request_id)
        with open(msg_file, 'w') as fd:
            fd.write(yaml.dump(data))

    def msg_file_data(self, user, request_id, is_response=False):
        msg_file = self.response_file(user, request_id) if is_response else \
                   self.request_file(user, request_id)
        if os.path.exists(msg_file):
            with open(msg_file) as fd:
                data = yaml.load(fd.read())
            if data and isinstance(data, dict):  # expected data format
                return data
        return None

    def create_request(self, user, request_id, data):
        self.create_msg_file(user, request_id, data)
        logging.info('created request %s for %s' % (request_id, user))

    def create_response(self, request_id, status, response_data=None):
        assert status
        user = getpass.getuser()
        self.create_msg_file(user, request_id, {'status': status, 'data': response_data}, is_response=True)
        logging.info('created response %s with status %s for %s' % (request_id, status, user))

    def handle_requests(self):
        user = getpass.getuser()
        while True:  # keep handling requests indefinitely
            time.sleep(1)  # new request polling rate, in seconds
            for filename in os.listdir(self.request_dir):
                m = re.match('(.*)\.(.*)\.yaml', filename)
                if not m:  # not a valid request filename
                    continue
                [username, request_id] = m.groups()
                if username != user:  # not a request for this user
                    continue
                if os.path.exists(self.response_file(user, request_id)):
                    # request handling already started
                    # you may add here recovery code for request handling interrupted for whatever reason
                    continue
                msg_data = self.msg_file_data(user, request_id)
                if not msg_data:  # unexpected data format
                    continue

                logging.info('received request %s: %s' % (request_id, msg_data))

                # mark the request handling start
                self.create_response(request_id, 'in_progress')

                time.sleep(5)  # mock-up, replace with whatever request handling means

                # mark the request handling done
                self.create_response(request_id, 'done')  # you can add response data to the dict if needed

                logging.info('handled request %s, waiting for confirmation' % request_id)

                while True:  # wait for confirmation receipt before cleaning up
                    time.sleep(1)  # confirmation receipt polling rate, in seconds
                    if not os.path.exists(self.request_file(user, request_id)):
                        # the deletion of the request file is the confirmation receipt
                        logging.info('confirmation for request %s received, cleaning up' % request_id)
                        os.unlink(self.response_file(user, request_id))  # cleanup response file
                        break

    def execute_deployment(self, deployment_id, deployment_user, deployment_data):

        self.create_request(deployment_user, deployment_id, deployment_data)
        started = False
        while True:  # wait until it's done
            time.sleep(1)  # polling rate, in seconds
            msg_data = self.msg_file_data(deployment_user, deployment_id, is_response=True)
            if msg_data:
                status = msg_data.get('status')
                if status:
                    if not started:
                        logging.info('request %s handling started' % deployment_id)
                        started = True
                    if status == 'done':  # job completed
                        logging.info('request %s handling completed, cleaning up' % deployment_id)
                        # cleanup request file, which is the confirmation receipt
                        os.unlink(self.request_file(deployment_user, deployment_id))
                        break

def usage(err_msg, option_parser):
    if err_msg:
        logging.error('%s\n\n%s\n' % (err_msg, option_parser.format_help()))
    sys.exit(-1)

if __name__ == "__main__":
    import optparse

    logging.basicConfig(level=logging.DEBUG, format="%(levelname)5s  %(asctime)s %(filename)s:%(lineno)d] %(message)s")
    os.umask(022)

    p = optparse.OptionParser()
    p.add_option('-c', '--command', action='store', dest='command', choices=['deploy', 'handler'],
                 help='command/mode: <deploy|handler>, mandatory', default=None)
    p.add_option('-i', '--ID', action='store', dest='id',
                 help='deployment ID, mandatory for deploy command', default=None)
    p.add_option('-u', '--user', action='store', dest='user',
                 help='deployment user, mandatory for deploy command', default=None)
    p.add_option('-a', '--artifact', action='store', dest='artifact',
                 help='deployment artifact, mandatory for deploy command', default=None)

    opts, _ = p.parse_args()

    if not opts.command:
        usage('command is mandatory', p)

    mailbox = Mailbox()

    if opts.command == 'deploy':
        if not opts.id or not opts.user or not opts.artifact:
            usage('ID and user must be specified for deploy command', p)
        data = {'artifact': opts.artifact}
        mailbox.execute_deployment(opts.id, opts.user, data)

    elif opts.command == 'handler':
        mailbox.handle_requests()

```

The jenkins user driving the deployments, deployment info hacked to a string for this example - 'artifact' :

```
$ ./mailbox.py -c deploy -i 20 -u dancorn -a artifact
 INFO  2017-10-17 13:32:34,663 mailbox.py:49] created request 20 for dancorn
 INFO  2017-10-17 13:32:35,666 mailbox.py:109] request 20 handling started
 INFO  2017-10-17 13:32:40,678 mailbox.py:112] request 20 handling completed, cleaning up
$ ./mailbox.py -c deploy -i 123 -u dancorn -a artifact
 INFO  2017-10-17 13:33:32,359 mailbox.py:49] created request 123 for dancorn
 INFO  2017-10-17 13:33:33,362 mailbox.py:109] request 123 handling started
 INFO  2017-10-17 13:33:38,375 mailbox.py:112] request 123 handling completed, cleaning up
$

```

The apache users would launch the handler which in this example remains running (could be converted to a daemon, or a cron-driven approach):

```
$ ./mailbox.py -c handler
 INFO  2017-10-17 13:32:34,819 mailbox.py:77] received request 20: {'artifact': 'artifact'}
 INFO  2017-10-17 13:32:34,821 mailbox.py:55] created response 20 with status in_progress for dancorn
 INFO  2017-10-17 13:32:39,827 mailbox.py:55] created response 20 with status done for dancorn
 INFO  2017-10-17 13:32:39,827 mailbox.py:87] handled request 20, waiting for confirmation
 INFO  2017-10-17 13:32:40,828 mailbox.py:93] confirmation for request 20 received, cleaning up
 INFO  2017-10-17 13:33:32,888 mailbox.py:77] received request 123: {'artifact': 'artifact'}
 INFO  2017-10-17 13:33:32,889 mailbox.py:55] created response 123 with status in_progress for dancorn
 INFO  2017-10-17 13:33:37,891 mailbox.py:55] created response 123 with status done for dancorn
 INFO  2017-10-17 13:33:37,891 mailbox.py:87] handled request 123, waiting for confirmation
 INFO  2017-10-17 13:33:38,893 mailbox.py:93] confirmation for request 123 received, cleaning up

```
> 2 votes

---
Tags: jenkins, continuous-integration
---


# Question
Title: DMAIC - Define, Measure, Analyse, Improve, Control in the DevOps world?
Six Sigma's DMAIC (see also Wikipedia) has been there before continuous procedures, so I think more or less these concepts have been nourishing , then CI/CD and now DevOps as well.

Still I am not sure whether we have an established data model for example to allow DMAIC across many iterations.

Question: do you know of examples of practices/ aligning DMAIC with DevOps (at the measure side, first of all)? Does it make sense?

# Answer
Yes, DMAIC makes sense. If processes like CD need to be improved it is important to have a baseline. Therefore the **M**easure step is useful. The team investigates the status of process X, e.g. CD by documenting the current status.
> 5 votes

---
Tags: methodology, itil
---


# Question
Title: How to handle updating binary files on a dev server with git
I have a dev site on a server which is synced with my git repo. The git repo does not track the media/ folder.

I want to update the content on the dev server to mirror the live server, so will copy over the database and the media/ folder from live to dev.

My worry is when I do this, git will pick up the changes in the filesystem with the new media/ files and won't allow me to do a git pull.

What's the best practice to handle this?

# Answer
Git doesn't consider the untracked files for making decisions about accepting or rejecting a `git pull` operation, so you should have no worry about it.

If still in doubt you can easily verify it: just pull another copy of the same git repo in some other location - for test only - mess with untracked files in it and pull again.
> 4 votes

# Answer
You can add `/media` to .gitignore
> 2 votes

---
Tags: git
---


# Question
Title: How can I download an artifact from Jenkins using the Google Login Plugin?
I recently spun up a new Jenkins instance and installed the Google Login Plugin for authentication via Google. I would like to download build artifacts to servers using a shell script, basically curl or wget, but cannot authenticate properly from command line.

I tried the following `wget` command without success.

```
wget --auth-no-challenge --http-user=XXXX --http-password=XXXX https://____/job/_____/lastSuccessfulBuild/artifact/target/ROOT.war

```

I also tried the same without `http-`:

```
wget --auth-no-challenge --user=XXXX --password=XXXX https://____/job/_____/lastSuccessfulBuild/artifact/target/ROOT.war

```

The response is always

```
401 Invalid password/token for user: XXXX

```

I attempted to follow this post but ran into an issue.

I have seen this workaround, but it seems like I'd be adding a security hole.

Does anyone have experience with this plugin or in general just downloading from an OAuth-protected endpoint?

# Answer
Thank you to the comment from user54 for the answer.

Both of my pasted `wget` URLs work as long as I'm using the correct value for password. In this case, I needed to use my user's API Token as described in the answer to another question: https://stackoverflow.com/a/34634247/317951
> 2 votes

---
Tags: jenkins, continuous-integration, google-cloud-platform
---


# Question
Title: Proper mapping for service architectures, Docker and BitBucket mapping - how do you do it?
With Atlassian BitBucket, you can have:

* Projects, which incorporate..
* Source code repositories.

Now, you could have a microservices with more than just one codebase and need therefore more than just repository. What is the best practice to reflect this if you have many different projects?

My idea now is to encode microservice prefixes to repository names, but is this really the way to go?

Like:

* Project A
    + service1.repo1
    + service1.repo2
    + service2.repo1
* Project B
    + serviceN.repoX

# Answer
One could also create libraries and call them using a dependency manager like a package.json if node is used or ivy if the code is written in java.
> 1 votes

---
Tags: bitbucket, methodology
---


# Question
Title: Creating a Separate Technical and Business Versioning System
A common problem at my company is versioning. We follow a versioning system *formatted* very similar to semantic versioning (but not by *definition*). Our current versioning is not only used by development, but also by marketing/sales in their release notes, advertising, etc. Often times, there are conflicts as to what the versioning should be. What the development team defines as a major change (non-backwards compatible api change) is often not considered a "major" change by marketing. Other times, marketing will push to increment the major because their is a significant customer workflow change, but from a development perspective, the change is very insignificant (often times just an application code patch).

I am pushing heavily to separate our versioning into two distinct systems, a business versioning and a technical versioning system. This would allow both teams to define their own rules for versioning that best suites their needs. While the feedback in general is positive, some of the concerns are the confusion that this will cause, in our cross team communication and ticketing systems.

What are some of the complications of having two versioning systems? What are some of the benefits of a separation? What kind of impact will this have on the teams and the company as a whole?

# Answer
It seems to me that your customers should be dictating the versioning. Both teams seem to be heading towards the same thing but only partially understanding the customer impact. For development, when they significantly re-architect components, there is risk because this could corrupt data or leave core features in a malfunctioning state - this should be reflected by a major version number change.

This doesn't mean much to marketing - everything looks the same, so why would the customer notice this change? No need to reflect this as a major change... unless you are a developer, in which case you risk a huge amount support interactions over those bugs.

Similarly marketing is looking at the user interface with the assumption that with a significant overhaul to the GUI, customers might get lost or hate the new user interface. This could lead to significant support interactions helping customer navigate the new UX, so should be reflected as a major version number change to the customer. There is a lot of risks there... Unless you are developer. Then this is purely cosmetic. Everything is still functioning the same on the back end.

So it seems you need to decide what it is that you feel version numbering is for.

* Are version numbers used to signal risk to customers?
* What source control system are you using
* Does Development even need version numbers *at all* or are commit IDs sufficient?
* Does Development plan features against version numbering for the purposes of setting deadlines?

Depending on the answers, it might be ideal to use a single version numbering system led by marketing - but marketing will need to do a better job of understanding the risks of under-the-hood changes. If this is needed as a designator for setting deadlines for change integrations, then two version numbering systems might be approperite:

Side note: git tags would be great for any version numbering scheme you have mentioned.
> 3 votes

# Answer
In my opinion the software version should not be updated if the documentation changes, while the software itself does not change.

I do not prefer to use different versioning for technical documentation and software as the docs are related to a certain software version. Different versioning will result in confusion.

In conclusion, one could use the same versioning, but if only the doc is updated one could add metadata, like 1.2.3-20171017. By using metadata one could still see that the doc belongs to version x of the software and one could also see that the doc has been updated.
> 1 votes

---
Tags: versioning
---


title: AWS Storage Cost for a custom AMI

text:
# Question
Title: AWS Storage Cost for a custom AMI
How does AWS calculate the cost of a custom AMI?

If I would create an AMI which is based on Ubuntu 16.04, do I need to pay for the complete size of AMI (e.g. 4GB) or just the differences of my custom AMI to the base AMI (e.g. 0.4GB)

# Answer
AWS stores **whole** AMI as a single non-differential snapshot thus they charge for a full size of files in an image.
> 3 votes

---
Tags: amazon-ec2, amazon-s3
---


# Question
Title: How can I ensure consistency between new microservices?
My organisation is experiencing an explosion of microservices. We currently have no formalised way of bootstrapping new projects. I'm finding that a team will come to me with a bug in their deployment or build process, and I will spend time on it only to realise I've already resolved it in another project. There's also a lot of inconsistency between projects that I'd like to see standardised.

The changes often involve a single file (e.g. serverless.yml or a Makefile) so a solution involving shared libraries e.g. git submodules doesn't seem viable. Each project will have its own set of configuration that needs to be maintained, e.g. Dockerfiles or serverless.yml, so centralised configuration management solutions for VMs aren't really applicable.

How can I ensure that new microservices conform to organisation standards and include bugfixes/features from existing projects in a way that's easy and intuitive for developers who want to start new projects? What are some best practises around resolving these issues?

The current workflow we have is to ask the person next to you "what project should I clone from to use as a template?" and then delete all the stuff that's not necessary for that project.

# Answer
> 5 votes
I'll add an answer of what my solution is so far, but I'm still really interested in hearing how other organisations are solving this problem and the best practices they have.

To resolve the issue of not having a consistent base to create projects for, my idea is to create a repository (repositories?) of boilerplates/templates and use cookiecutter as a tool to scaffold out new microservices. This way, each project is created from a standard base with all the lessons we've learned as an organisation integrated in to it. Any changes we make can be integrated upstream in to the boilerplate repository. I imagine we'll have templates for Nodejs Docker images, Serverless SPAs, Python lambdas, etc.

To resolve the problem of changes made to the templates being propogated downstream to every project, my solution is to implement a process where owners of microservices are made aware of changes to the template and are then responsible for propagating those changes to their microservice.

# Answer
> 4 votes
Use a configuration management/automated deployment system. This is what these were designed for. Things like Kubernetes, Puppet, Chef, Ansible and Salt Stack are designed for just this purpose and can be utilized with Golden Master templates, kickstart scripts, Amazon AMIs or just a Docker container. This allows you to use default base templates and then layer on additional roles. This will ensure builds are documented completely (as code) and it will be fast and easy to deploy to production, and will be deployed exactly identically to what was designed or deploy additional instances when the need for scalability or redundancy arises or something breaks. Changes/updates can also be integrated this way. Just as you release software updates, you can release updates to your configuration and the configuration code can be managed just as your software code is managed - in the same repos and with the same workflows. And when upgrade time comes, there is no mystery how the thing is built, just look at the script.

One way that configuration management systems do this is through heavy use of templating for your configuration files. For example, there are generally many lines that will be the same or similar in your environment. SaltStack utilizes jinja templates and puppet uses Embedded Ruby templates. Using AWS as an example, you may need to set a, api key, IAM role, region (or randomly select a region from a list of regions), a VPC, etc that is all the same across all instances. But then you need to have your functions and outputs unique. Or better yet you could write a puppet module or salt formula which defines "contracts" and use those contracts (api definitions) for both inputs and outputs saving you from having to configure your microservices two or three places.

SaltStack for example recently had a meetup to discuss this particular scenario. Furthermore, SaltStack is also able to manage and deploy docker containers natively. (Puppet also has a module for Docker) Likewise Ansible has playbooks and docs for working with serverless deployments and Docker containers.

Also, if you wish to continue with your serverless theme/paradigm, Puppet, Ansible and saltstack all are masterless or support a masterless mode, if you wish to continue this theme.

# Answer
> 3 votes
This question is broad so if my answer is a little off-base feel free to add context and specific examples so that I have a better understanding.

Using a machine image such as AWS' AMI would allow you to create a base or golden image, which you could then maintain and distribute or implement in your continuous delivery. Using this architecture you are ensuring that every microservice is deployed on consistent hardware with identical configuration so any problems you face are related to the microservice/application configuration.

You can further this immutability with the addition of configuration tools such as Ansible and Packer. Using configuration management you could provision the machine image with whatever you want on it (including the application). Packer would then allow you to take a snapshot of that machine image and each deploy would be identical.

Using this example you could 'bake' a base AMI with the correct packages, updates, and configuration installed with the help of Ansible and Packer. Additionally, you could look at 'Ansible-Pull' to complete the deploy by pulling the application code, making any changes, and deploying the microservice on top of that base image.

However the most important advice I can give is to just come up with a solution that the entire organization can support and maintain. It's worth trying to establish an SDLC that solves your particular set of problems, matches the culture and attitude of leadership, and embraces modern architecture practices.

I've been with three organizations and we've taken three very different approaches.

Good luck!

---
Tags: project-management, microservices
---


# Question
Title: Automatic Docker Hub builds from GitLAB, not github
As per title. I would like to configure automated builds on Docker Hub, as specified here.

My build context (the build context is a repository containing a Dockerfile) is hosted in GitLAB.

# Answer
> 2 votes
This feature is not supported as the issue in the link says.

They promised it for late 2016 but now at 2017 the issue is still open.

One comment was to look for gitlab CI, which might be my best guess of workaround for now, waiting the Docker Hub version to come.

---
Tags: docker, git, gitlab
---


# Question
Title: Using docker with users leaves files belonging to the subuid/subgid
# Background

I wish to make sure all people and automated processes working with my project use the exact same version and environment of some tool which has to do work on the *host* file system.

Instead of running `/usr/bin/xyz` or whatever, people will then use a wrapper/shim script similar to:

```
docker run -ti --rm \
  -v $PWD:/workdir \
  -v $HOME/.ssh:/root/.ssh \
  dockerizedxyz "$@"

```

This is working splendidly but...

# Problem

Obviously, if I stop right here, then files created by the `xyz` running inside the container will be ending up belonging to `root:root` on the host. It is clear why this happens, and as expected.

Hence, I have enabled user namespaces for the docker demon. This works fine as well. My host user has UID 1000 and the subuid 493216.

If I run the docker-xyz-shim in a directory (on the host) that belongs to 1000, and if that container starts to create files, my expectation at first had been that they would have ended up as belonging to 1000 on the host (i.e., forward- and backward-mapping).

But, lo and behold, they belong to 493216, i.e., they belong to the subuid instead of the host uid. There is only a forward-mapping happening (from host-uid to sub-uid, no backward-mapping from sub-uid to host-uid).

I understand why that is, and that this is working as intended. First, there is a 1:n mapping involved, and it would be non-trivial to map the potentialla 65536 sub-uid back to the one host-uid. It allows us to do all kinds of things that would be practical in other use-cases. But in my particular case, I actually do wish the files to end up with 1000 on the host.

*Is there anything I can do about it?*

# What I tried

I added the host user 493216 to the host group 1000 and set the `g+rwx` permissions. If I do a `su - 493216` on the host, I can now edit the files belonging to 1000. But from inside the container, the files are shown to be owned by `nobody:nobody` and I can read them, but not write them.

I can create new files inside the container if I do a `chmod 777` on a directory on the host (it then belongs to `nobody:nobdy` with 777 permissions in the container), and they end up belonging to `root:root` in the container (fine) and to 493216:493216 on the host (to be expected after all this).

EDIT: replaced "git" with "xyz", this is not about git at all, but about how to modify host files from inside a docker container without having `root` on the host.

# Answer
```
docker run -ti --rm \
  -v $PWD:/workdir \
  -v $HOME/.ssh:/root/.ssh \
  --user $(id -u):$(id -g) \
  dockerizedxyz "$@"

```

As long as you're not doing anything funny with the entrypoint in the container, this will leave files owned as the user that invoked Docker.
> 2 votes

---
Tags: docker, permissions, user-namespaces
---


title: Is it bad to use sudo from Jenkins?

text:
# Question
Title: Is it bad to use sudo from Jenkins?
I use Publish Over SSH plugin to deploy my apps from `Jenkins` to different environments. Some deployment jobs do environment preps and things like stop and restart the app server system service. Some of those commands require `sudo`.

I am just curious if it can be a bad security practice to require sudo within remote publish and execution Jenkins jobs. Should we change the security policy on the target host to allow the required functions to be performed without sudo?

# Answer
No, it's actually *really good* security practice to sudo when you need higher level privileges. This is the reason that most distributions prohibit default login and actually force you to `sudo su` instead of just typing `su` \- they want to encourage you to use sudo for it's security benefits. There are few reasons why:

* **Auditing.** When you sudo, the system keeps a record of *who* sudoed and the command they ran. This adds a great deal forensically when you need to go back and figure out what happened - to assign blame, catch nefarious activities or just for troubleshooting purposes (what in the hell happened last night at 7PM before this server went AWOL?)
* **Sudo rules.** Just because a user needs to run *something* with escalated privleges doesn't mean that they need to run *everything* with escalated privleges. Using su or adding a user to wheel allows a user to do *everything*. With sudo however, it is possible to specify what users can and cannot do using Command Aliases which allow for the use of wildcards, thereby allowing for flexible rules that grant users privleges to use some commands, but not others.
* **Hardening.** It is possible to disable root. As in, for all intents and purposes, it doesn't exist and is only useful/usable in single user mode - requiring a reboot of a server (and the only reason this is allowed is for password recovery). `sudo su` no longer works. This is done by setting root's shell to /bin/nologin. which is a great way to prevent many root escalation vulnerabilities. This, of course assumes that you are properly using the bullet above (and you can still grant a specific, admin user all permissions, though this will undermine your security stance).

These things are, of course, only part of your security model. For example, unless you have no users with full administrative privileges (and sometimes even then) your user may be able (probably is able) to delete logs and history - unless you use a log server and syslog to ship logs off-box. Then there's no way to put that cat back in the bag. Of course, if you use a central authentication directory structure, then this comes back into play: If I compromise one account, it is compromised on *all* the servers in the organization - including probably your log server.

In short, security is complicated, but sudo is a great tool in architecting a secure environment and more people should be using it to it's fullest extent.
> 6 votes

# Answer
Whether you allow remote `sudo` or remote access to something that does `SUID root` you have a pretty similar attack surface. I would keep `sudo` in the chain because it lets you limit the commands easily and has logging that will be vital if you need to audit things later. `sudo` also has a much longer history in production. Doing something else will have less history and higher changes of unpleasant surprises.

There are other things you can do to make this more secure though:

* tighten up `ssh`
* only allow the restart commands for a few specific users including one for jenkins
* only allow logins for those specific users from all internal IP's or just jenkins and jump box IP's
* store logs on remote boxes so they can't be mucked with
> 7 votes

---
Tags: jenkins, security
---


title: Evolution of DevOps?

text:
# Question
Title: Evolution of DevOps?
I head that the present DevOps jobs were previous know as build and release engineer. I want to go back to past 10 years. What would be the different job position roles a personal might have played from past 10 yes?

# Answer
I don't view DevOps as a job position. Yes, I also have job offers out there for titles like "DevOps Engineer" \[sic\], but that just means I am looking for people who mainly can take a team with little DevOps knowhow and guide them along (be it through just doing the bulk of the work, or evangelism or whatever). But at the end of the day, everybody on the team needs to have a working understanding at least of what's going on.

DevOps is more of a feeling, a culture, and a set of common tools and techniques. A guideline if you wish. With memes like having the same environment across dev, qs, prod, immutable containers being shipped across all those, "everything as code", applying Agile and Lean to ops, having devs and ops in the same team and such.
> 6 votes

---
Tags: continuous-delivery
---


# Question
Title: Why does the jenkins-cli.jar linting functionality not work?
The jenkins-cli.jar has been downloaded.

It is able to run.

When one tries to check the syntax of a jenkinsfile by running `java -jar ~/Desktop/jenkins-cli.jar -s https://<jenkins_url> -noCertificateCheck declarative-linter < Jenkinsfile` it results in:

```
Sep 27, 2017 10:46:18 AM hudson.cli.CLI _main
INFO: Skipping HTTPS certificate checks altogether. Note that this is not secure at all.
Sep 27, 2017 10:46:34 AM hudson.cli.CLI$5 run
WARNING: null
java.io.IOException: Stream is closed
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3512)
        at sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3486)
        at java.io.DataOutputStream.writeInt(DataOutputStream.java:197)
        at hudson.cli.PlainCLIProtocol$EitherSide.send(PlainCLIProtocol.java:188)
        at hudson.cli.PlainCLIProtocol$EitherSide.send(PlainCLIProtocol.java:195)
        at hudson.cli.PlainCLIProtocol$EitherSide$1.write(PlainCLIProtocol.java:215)
        at hudson.cli.CLI$5.run(CLI.java:699)

```

@Tensibai asked whether there is an authentication issue. Authentication was done, but it is unclear whether it was succesfull or not:

```
$ java -jar ~/Desktop/jenkins-cli.jar -noCertificateCheck -s https://<jenkins url> login --username <username> --password <password>
Sep 27, 2017 10:57:17 AM hudson.cli.CLI _main
INFO: Skipping HTTPS certificate checks altogether. Note that this is not secure at all.

```

# Answer
After upgrading Jenkins the issue was solved. It turned out that the issue was version related.
> 1 votes

---
Tags: jenkins, linting
---


# Question
Title: Declarative Jenkins Pipeline to Poll different branches?
We have a gitlab enterpise that host multiple project in one repo. Something like:

```
Repo/
  projectA/
  projectB/
  projectC/

```

We have three pipelines that will run base on changes(whitelist region) when polling. They will start polling when received a `curl` via the `notifyCommit?` in another job.

However, the pipeline can only work on `branch` that is run previously.

I have also try to use parameters $branch on `branch to build` and update the `curl` job with parameters like:

`http://ourjenkins/git/notifyCommit?url=git_url.git&branch=$GIT_BRANCH`

but, it still will only poll the last run branch.

Does anyone know how we can have the declarative pipeline to poll multiple branches and run only base on whitelist directory changes?

# Answer
> 1 votes
One could use the PollSCM plugin.

https://jenkins.io/doc/book/pipeline/syntax/

```
triggers { pollSCM('H 4/* 0 0 1-5') }

```

> The pollSCM trigger is only available in Jenkins 2.22 or later.

---
Tags: jenkins, continuous-integration, jenkinsfile
---


# Question
Title: Docker compose host variable in extra_hosts not working
I have a docker compose file with `.env` file, for example the `.env` file has this `HOSTNAME=a1.test.com`, and a `docker-compose` file has this:

```
...
extra_hosts:
  - "${HOSTNAME}:1.1.1.1"
  - "test.com:1.1.1.1"

```

When I run the container the `/etc/hosts` file content is:

```
1.1.1.1
1.1.1.1    test.com

```

So the docker compose not putting the variable data during the run process.

I tried this `hostname: "${HOSTNAME}"` and it is working fine the host name of the container is correct `a1.test.com`.

Any idea how can achieve what I want?

# Answer
I've tried your case, and it works fine on my side:

```
version: '2'
services:
  test:
    image: ubuntu:16.04
    command: sleep 9999
    extra_hosts:
      - "${HOSTNAME}:1.1.1.1"
      - "test.com:1.1.1.1"

```

and here is what I get in the container

```
cat /etc/hosts
...
1.1.1.1 a1.test.com
1.1.1.1 test.com

```

*(edit: indentation)*
> 4 votes

---
Tags: docker, docker-compose
---