# Question
Title: Gitlab Runner not running scripts with Windows shell configuration
I have a Gitlab Runner running locally on my windows box. It works fine for the before script section.

my `.gitlab-ci.yml` follows:

```
before_script:
  - pushd . && uru 233 && popd && set HOME=c:\ && ruby -v && bundle install

rspec:
  script:
    - bundle exec rspec

rubocop:
  script:
    - bundle exec rubocop

flay:
  script:
    - bundle exec flay *

```

It does not do anything after the before script, though. Does not run any of the jobs.

How can I find out what's going on?

I was able to fix it with a super-non-ideal/hacky solution that I don't like:

```
rspec:
  script:
    - pushd . && uru 233 && popd && set HOME=c:\ && ruby -v && bundle install && bundle exec rspec

rubocop:
  script:
    - pushd . && uru 233 && popd && set HOME=c:\ && ruby -v && bundle install && bundle exec rubocop

flay:
  script:
    - pushd . && uru 233 && popd && set HOME=c:\ && ruby -v && bundle install && bundle exec flay *

```

People sending comments have also prompted me to add that in Windows it seems to be stopping at the first array element in every script session.

With a Docker configuration for the runner I can do things like:

```
my_job_name:
  script:
    - a command
    - another command

```

With the windows shell runner it just stops after `a command` if I do it this way. Stringing the commands together seemed to work, but I would rather not have to put everything on one line.

# Answer
stages are required

```
stages:
  - stage1

stage1:
  stage: stage1
  script:
    - echo hello

```

Could you try the following:

```
before_script:
  - pushd . && uru 233 && popd && set HOME=c:\ && ruby -v && bundle install

stages:
  - rspec
  - rubocop
  - flay

rspec:
  stage: rspec
  script:
    - bundle exec rspec

rubocop:
  stage: rubocop
  script:
    - bundle exec rubocop

flay:
  stage: flay
  script:
    - bundle exec flay *

```
> 4 votes

---
Tags: gitlab
---


# Question
Title: What approach should I follow for adopting continuous integration?
I have a WAMP server which hosts a Django application by using a Windows server.

Currently I am deploying everything manually. Now I want apply Continuous Integration concepts to an existing application.

What approach should I follow and what technology should I use for this?

Which is better server to host Django application by CD in mind

```
 is it Puppet,

       Chef,

       Jenkins etc

```

Currently I am using Git as source control.

# Answer
I am in a similar pair of shoes as you - I am currently, slowly, getting into the DevOps topics after having many, many years of conventional development (and close connections to conventional ops) behind me.

You seem to have read a few articles, or heard a few things. That is good. Let me suggest a practical approach how to go on:

1. Take your favourite application (which you or your team wrote yourself); not too large, preferably with a limited amount of services (like Apache, DB etc.) working together with your app.
2. Install `docker`, read a bit on the Dockerfile and the whole build process, and just go ahead to create one or more Dockerfiles for that application. You know already how to deploy it manually; now make it run inside a container.
    * Make it easy on yourself. Use a `FROM`parent container that is a full-blown distribution of your choice. I like `debian:jessie`, but you can pick whichever you are familiar with.
    * At the start, do not worry about image sizes. This is just a pet project. Start your Dockerfile with the `FROM`, then as many `apt-get update; apt-get install ...` as you need to get your application running. Add stuff with `ADD` and `COPY`liberally.
3. Learn how to actually run those containers; how `EXPOSE`works, how to `--link`the app together with the DB and the Apache (in case you made a separate container for that - you can skip this obviously if your app is able to serve all stuff including static pages itself). Figure out how to mount/bind `volumes`, how to debug a container by opening a shell inside it and so on. `docker-compose` is nice if you have more than one, and pretty simple to get going with.
4. Now you probably have a working setup. That's great! You have learned the basics, and you immediately feel the great power that comes from containerizing your app. You can deploy it instantly on any server, you can give it to your developer colleagues and go up and running in zero time, and so on. Enjoy that feeling for a while.
5. Make the image smaller. There are a few obvious things you can do (google...), and a few not so obvious ones (thinking is like googling, just better... :) ). If you need help on that, post more questions here.
6. Get Jenkins (feel free to do so with docker, again, if you're adventurous) and make a job to build the image from the Dockerfile and run your test suite directly in the container you built. Jenkins can check your repository for updates automatically, you don't need to bother with `git` hooks at this moment.
7. You're done! This was the CI part. For CD, after the tests are green in Jenkins, simply tag the build and send it on to your deployment server.

This should take you, if you are an experienced developer and have done these things "dry" already, not much longer than a few days. Docker itself is a pretty simple affair, there is not much more to it. One further part that makes sense to add to the puzzle is Ansible, which makes it pretty easy to manage another machine remotely (i.e. for installing packages, deploying stuff, starting services and whatnot).

If you have Docker and Ansible down, just get your CI/CD pipeline to work. Google advanced, but necessary stuff like security and such. After that, you should be experienced enough to decide in which direction to go next (i.e., larger stuff like Openshift/Kubernetes, or just whatever AWS/Azure or whatever have to offer, etc.). There are a myriad of different options out there, and much of the choice seems to be pretty opinion-based. If you get into High Availability, swarms and such, it will also pretty soon get more non-trivial. But starting out with Docker should be fun and relatively straightforward.
> 4 votes

# Answer
As you are already using Bitbucket, Bitbucket pipelines could be useful.

> What languages are supported?
>
> Node, Ruby, Python, PHP, and more. Anything you can run in a Docker container is supported.

---

> why not jenkins?

Jenkins could be an option as well. If Bitbucket could trigger Jenkins or Jenkins could check whether there are code changes.
> 3 votes

# Answer
You've chosen a fairly difficult stack to support. WAMP is largely used in dev environments, and the windows builds of mysql, php, and apache typically lag their Linux counterparts by a significant amount. You can likely reuse your code, barring any hard coded or absolute paths, on Linux. Once you move to Linux, there are lots of great deployment options, the most obvious of which is probably docker. Since your stack includes a full relational database solution, and a web server, you're up against a lot of potential complexity. This makes it near impossible to give you a good answer beyond scripting something via powershell and rolling your own if you want to keep this stack on windows. You can keep windows and swap your stack and switch languages (probably a terrible idea, depending on how far you are into this) or you can switch operating systems and keep your code, but potentially fight with a whole world of new OS frustrations. I recommend this path, but there's no solver bullet here.

So you have two pretty significant hurdles: 1. Get this app running on LAMP. 2. learn a deployment tool for LAMP.

Alternatively you might be able to find a specialized WAMP host who will let you do something like FTP your code to them, which you could script. That's not really a production-worthy plan if you ever intend to scale this application, but sometimes it's smart to act small.
> 2 votes

---
Tags: continuous-integration, continuous-delivery, windows, bitbucket, python
---


title: Kubernetes user mode in production

text:
# Question
Title: Kubernetes user mode in production
Docker daemon has to run in root mode which brings up security discussions.

How is this with Kubernetes daemon - can it run not as root?

Note: both of these products allow you to reduce the scope of kernel capabilities allowed for containers but this is not the question.

# Answer
> 1 votes
https://kubernetes.io/docs/tasks/configure-pod-container/security-context/

https://linux-audit.com/linux-capabilities-hardening-linux-binaries-by-removing-setuid/

> With Linux capabilities, you can grant certain privileges to a process without granting all the privileges of the root user. To add or remove Linux capabilities for a Container, include the capabilities field in the securityContext section of the Container manifest.

In summary, in k8s it is possible to assign certain privileges to a pod. Assign the privileges that are required by a certain container.

There is an open issue:

https://github.com/kubernetes/kubernetes/issues/7961

---
Tags: docker, kubernetes, security
---


# Question
Title: What OS should a master node CI virtual machine be?
Say you are developing a new CI/CD system, deploy builds across Linux, Windows and Mac, and have now reached the point of wanting to deploy a master node which hosts your chosen CI tool. How do you choose what OS and version your master node VM should be?

For example with Gitlab, I can have the VM with Ubuntu, Debian or CentOS, or with Jenkins, the VM could be Mac, Windows, Ubuntu, Debian etc...

What's the major deciding factor when choosing an OS and version for the master node of the CI system?

# Answer
> 5 votes
I think the major deciding factor is the expertise of you and the rest of your company in the chosen OS. If you are a Windows shop, and your company is willing to leverage the cost of the node, it's probably best choice to host it on a Windows VM. For choosing a Linux distro, I would see what is most common across your company. In my opinion, it is not worth the benefit of creating a vm with a one off distro just because of a few particular features, unless those features of the distro are absolutely necessary.

---
Tags: jenkins, continuous-integration, continuous-delivery, gitlab
---


# Question
Title: Make use of CIs as Maven Repository in order to use it for building Docker images without including dependencies
I am trying to make images from our company Maven based applications. These apps are using some libraries which are included in a Maven repository, but fills a lot of disk space. They also have some dependencies which should be downloaded. I have seen this question which is talking about downloading dependencies (and unfortunately has no approved answer), but for me it is more important to use my local repository without adding it to the image (which increases image size from 300M to 4G) and caching downloading dependencies is in the second place.

If I wanted to `ADD` my local repository to the image, I would have make this `Dockerfile`:

```
FROM java:8.0_31
MAINTAINER Zeinab Abbasimazar (zeinab.abbasi@peykasa.ir)
ADD .m2/ $HOME
WORKDIR .
RUN apt-get update;apt-get install -y maven;cd Development/;mvn clean install -P test
CMD ["./Component.sh &"]

```

Which makes this image:

```
sudo docker images
REPOSITORY                                   TAG                 IMAGE ID            CREATED             SIZE
maven                                        3.3.9               44ab79a0bd9e        2 hours ago         4.678 GB

```

This size for an image is not affordable for me; could anyone please help me on this? Is there any workaround using `gitlab` or `Jenkins` to play `Maven Repository` role?

# Answer
> 2 votes
Docker images are after all VM templates, i.e. they have to be more or less self-contained: you get the image, you run the environment with all the dependencies.

To approach the challenge **operationally**, Docker supports reuse of environments through the `FROM` statement i.e. you could maintain a base image and on top a much smaller image with the app itself.

But, from the **strategic point of view**, I would say your architects might want consider splitting the whole thing into smaller microservices - that would allow for 5 smaller images. Of course I cannot judge whether it's really possible.

# Answer
> 0 votes
I'm thinking about different approach to your problem. Instead of adding maven repository into the image, I will mount a volume to $HOME/.m2/repository, and another volume to $PROJECT\_DIR

```
ENV PROJECT_DIR=/project
VOLUME /repository
VOLUME ${PROJECT_DIR}
RUN mkdir -p $HOME/.m2 /repository && ln -s /repository $HOME/.m2/repository

```

After that, add a condition check in component.sh whether this is the first run and then build the required component.

```
...
if [[ ! -e $PROJECT_DIR/app_is_ready ]]; then
  cd $PROJECT_DIR
  mvn clean install -P test
  touch $PROJECT_DIR/app_is_ready
fi
<<actual run commands>>

```

Please be note that the example code is not handling the race condition of two (or more) instances using the same volume for PROJECT\_DIR started at the same time.

Hope this help!

---
Tags: docker, dockerfile, maven, repository
---


# Question
Title: Docker Images via Container vs Dockerfile
Pardon the dumb questions, but I've been learning about Docker and container technologies and I was learning about using Dockerfiles to specify build instructions for an image, and how each instruction creates a layer that will impact the overall size of the image generated.

I also learned that you can also attach to an image, as a base, and use it as any other server/vm, and commit the container as an image. This is the method I've been using in my few weeks of experimenting with Docker.

Is there any benefit to using one method vs the other? Is reusability enhanced either way? I was also able to push the image I created via container to the registry, is this best practice, or should I be keeping a repository of Dockerfiles?

# Answer
If one uses a Dockerfile then a colleague could also understand what happened (documentation as code). If one runs a container, enters it, runs commits then it would be hard to understand what packages were installed. Especially after a couple of months.
> 7 votes

---
Tags: docker, dockerfile, containers
---


# Question
Title: Is it ok to have a Jenkins Server without slave node?
In **my team** we have one Jenkins server with **one master node & one slave node** in the same server. We use Jenkins Pipelines and surround everything with `node{}`.

In a **colleague's team** I have seen that, for some reason, they have **one master node and no slave node**. All their builds run in master node.

Questions:

* Is this a bad practice?
* Are there any bad consequences for having only one master node?
* Is my setup (one master & one slave) a bad practice?

# Answer
https://wiki.jenkins.io/display/JENKINS/Jenkins+Best+Practices

> In larger systems, don't build on the master.
>
> If you have a more complex security setup that allows some users to only configure jobs, but not administer Jenkins, you need to prevent them from running builds on the master node, otherwise they have unrestricted access into the JENKINS\_HOME directory. You can do this by setting the executor count to zero. Instead, make sure all jobs run on slaves. This ensures that the jenkins master can scale to support many more jobs, and it also protects builds from modifying potentially sensitive data on $JENKINS\_HOME accidentally/maliciously. If you need some jobs to run on the master (e.g. backups of Jenkins itself), use the Job Restrictions Plugin to limit which jobs can be executed there.

---

> One master node and no slave node. All their builds run in master node. Is this a bad practice?

Running the jobs on the master nodes means that the jobs have unrestricted access into the JENKINS\_HOME directory

> Are there any bad consequences for having only one master node?

As the slaves have unrestricted access into the JENKINS\_HOME directory this could be unsafe

> Is my setup (one master & one slave) a bad practice?

It is better than only one master as long as all the jobs run on the slaves

---

In summary, from a security perspective it is a bad practice to run the jobs on the master.
> 14 votes

# Answer
There are some situations where running jobs on the master node is fine, but you do have to be careful. Like the other answer mentioned, generally speaking running jobs on the master is bad practice since the jobs aren't sandboxed on the master node.

That being said, sometimes you do want to run jobs on the master node. For instance, two of the jobs on my Jenkins installation are:

* find jobs whose dependencies have been updated and schedule builds for them against the new versions of the dependencies
* find open pull requests that haven't been built yet and schedule builds for them

I (as the Jenkins admin) am in total control of these jobs, neither of these particular jobs touch any files on disk (they're essentially "meta" jobs), I want them to be able to run even when all the executor slots on my slaves are busy, and I would prefer these jobs don't block more important jobs in the queue. For these reasons, I allow these two jobs - and **only** these two jobs - to run on the master node.
> 5 votes

---
Tags: jenkins, architecture
---


# Question
Title: Dockerize CI installation on master node VM
Carrying on from this question, would you want to dockerise your CI (Gitlab (especially if Gitlab also contains all your repositories) or Jenkins) installation on your master node virtual machine?

What are the advantages and disadvantages to this, than just installing normally onto the VM without docker?

# Answer
> 3 votes
**Pros**

* possible to run the CI on a container orchestration platform

**Cons**

* docker in docker in docker

# Answer
> 3 votes
If it's just the master, I would say it's okay. The main concern would be any recursive dependencies where your CI tool manages the Docker platform that it runs on.

Build agents are another case. I've run CI with Docker in both configurations, and I prefer running outside of Docker and invoking it during your build, as it allows consistency between your dev and CI platform. Eg: https://github.com/amaysim-au/docker-serverless

---
Tags: docker, continuous-integration
---


# Question
Title: Salt State is running against all servers except one, why?
I recently asked a question here: Is it possible to use multiple if statements in a salt state?

Thanks to the answer I have managed to update my state to only check specific files exist on specific servers defined in the if statement. However, there's an issue with one server, the state runs fine against it and doesn't error, however it is only checking 32 files on TEST-SERVER-04, whereas it's 'DEV' counterpart (defined in the same if statement) has all 50 files checked.

For the life of me I can't work out why! If I try running the state directly against the server (i.e. salt 'test-server-04' state.sls filecheck') it still does the same thing! I have checked my syntax repeatedly, checked for white lines etc but to no avail! I'm sure somebody will be able to point out what I have done wrong here!

Below is a snippet of the affected code:

```
######################################################################################################################## // THIS STATE IS USED TO CHECK THAT ALL MINIONS (JL/TL LINUX SERVERS) IDENTIFIED AS 'WEBSERVER' HAVE THE CORRECT CONFIGURATION FILES AND SERVICES IN PLACE.
#######################################################################################################################

checkservices.sh:
  file.exists:
    - name: /opt/checkservices.sh

####################################################################################################################
# // THIS SECTION CHECKS THE CONFIG FILES IN /OPT/TOMCAT/CONF EXIST
####################################################################################################################

catalina.properties:
  file.exists:
    - name: /opt/tomcat/conf/catalina.properties

logging.properties:
  file.exists:
    - name: /opt/tomcat/conf/logging.properties

web.xml:
  file.exists:
    - name: /opt/tomcat/conf/web.xml

tomcat-users.xml:
  file.exists:
   - name: /opt/tomcat/conf/tomcat-users.xml

server.xml:
  file.exists:
    - name: /opt/tomcat/conf/server.xml

context.xml:
  file.exists:
    - name: /opt/tomcat/conf/context.xml

catalina.policy:
  file.exists:
    - name: /opt/tomcat/conf/catalina.policy

####################################################################################################################
# // THIS SECTION CHECKS THE CORRECT JHC SCRIPTS ARE IN /OPT/TOMCAT/BIN (JHC JETS ADMINISTRATION SCRIPTS)
####################################################################################################################

jhcshutdown.sh:
  file.exists:
    - name: /opt/tomcat/bin/jhcshutdown.sh

jhcstartup.sh:
  file.exists:
    - name: /opt/tomcat/bin/jhcstartup.sh

jhc-tomcat-util.sh:
  file.exists:
    - name: /opt/tomcat/bin/jhc-tomcat-util.sh

jhc-tomcat-before-startup.sh:
  file.exists:
    - name: /opt/tomcat/bin/jhc-tomcat-before-startup.sh

jhc-tomcat-after-startup.sh:
  file.exists:
    - name: /opt/tomcat/bin/jhc-tomcat-after-startup.sh

jhc-tomcat-after-shutdown.sh:
  file.exists:
    - name: /opt/tomcat/bin/jhc-tomcat-after-shutdown.sh

shutdown.sh:
  file.exists:
    - name: /opt/tomcat/bin/shutdown.sh

startup.sh:
  file.exists:
    - name: /opt/tomcat/bin/startup.sh

version.sh:
  file.exists:
    - name: /opt/tomcat/bin/version.sh

tool-wrapper.sh:
  file.exists:
    - name: /opt/tomcat/bin/tool-wrapper.sh

setclasspath.sh:
  file.exists:
    - name: /opt/tomcat/bin/setclasspath.sh

digest.sh:
  file.exists:
    - name: /opt/tomcat/bin/digest.sh

daemon.sh:
  file.exists:
    - name: /opt/tomcat/bin/daemon.sh

configtest.sh:
  file.exists:
    - name: /opt/tomcat/bin/configtest.sh

catalina.sh:
  file.exists:
    - name: /opt/tomcat/bin/catalina.sh

bootstrap.jar:
  file.exists:
    - name: /opt/tomcat/bin/bootstrap.jar

#####################################################################################################################                      THIS SECTION CHECKS THAT THE TOMCAT LOGS ARE PRESENT IN /OPT/TOMCAT/LOGS/
####################################################################################################################

jhc.jets.log:
  file.exists:
    - name: /opt/tomcat/logs/jhc.jets.log

catalina.out:
  file.exists:
    - name: /opt/tomcat/logs/catalina.out

{% if grains['host'] in ['BCA-AJT-TC-04', 'BCA-AJU-TC-04', 'BCA-AJT-TC-05', 'BCA-AJU-TC-05', 'BCA-AJT-TC-07', 'BCA-AJU-TC-07'] %}

demo-admin.log:
  file.exists:
    - name: /opt/tomcat/logs/demo-admin.log

{% endif %}

minion.log:
  file.exists:
    - name: /var/log/salt/minion

lynis.log:
  file.exists:
    - name: /var/log/lynis.log

#####################################################################################################################                       TEES FINANCIAL SPECIFIC CONFIG
####################################################################################################################

{% if grains['host'] in ['BCA-AJU-TC-04', 'BCA-AJT-TC-04'] %}

tees-pom.properties:
  file.exists:
    - name: /opt/tomcat/webapps/website1/META-INF/maven/jhc.client.F63.web.figaroweb/ajb-2/pom.properties

tees-pom.xml:
  file.exists:
    - name: /opt/tomcat/webapps/website1/META-INF/maven/jhc.client.F63.web.figaroweb/ajb-2/pom.xml

tees-web.xml:
  file.exists:
    - name: /opt/tomcat/webapps/website1/WEB-INF/web.xml

tees-jets-jca-client.xml:
  file.exists:
    - name: /opt/tomcat/webapps/website1/WEB-INF/classes/conf/jets-jca-client.xml

tees-index.html:
  file.exists:
    - name: /opt/tomcat/webapps/website1/index.html

tees-logo.png:
  file.exists:
    - name: /opt/tomcat/webapps/website1/statics/images/logo.png

tees-logo_website1.png:
  file.exists:
    - name: /opt/tomcat/webapps/website1/statics/images/logo_website1.png

tees-notfound.gif:
  file.exists:
    - name: /opt/tomcat/webapps/website1/statics/images/notfound.gif

tees-sessionKeepAliveImage.png:
  file.exists:
    - name: /opt/tomcat/webapps/website1/images/sessionKeepAliveImage.png

tees-site.properties:
  file.exists:
    - name: /opt/jhc/ajb-2/config/site.properties

tees.properties:
  file.exists:
    - name: /opt/jhc/ajb-2/config/ajb-2.properties

tees-common-site.properties:
  file.exists:
    - name: /opt/jhc/ajb-2/config/common-site.properties

tees-environment.properties:
  file.exists:
    - name: /opt/jhc/ajb-2/config/environment.properties

tees-Admin.properties:
  file.exists:
    - name: /opt/jhc/ajb-2/config/Admin.properties

tees-log4j-Client.properties:
  file.exists:
    - name: /opt/jhc/ajb-2/config/log4j-Client.properties

tees-log4j-Admin.properties:
  file.exists:
    - name: /opt/jhc/ajb-2/config/log4j-Admin.properties

tees-website1.properties:
  file.exists:
    - name: /opt/jhc/ajb-2/config/website1.properties

tees-ehcache.xml:
  file.exists:
    - name: /opt/jhc/ajb-2/config/ehcache.xml

{% endif %}

#####################################################################################################################                                           DART CAPITAL SPECIFIC CONFIG
####################################################################################################################

{% if grains['host'] in ['BCA-AJT-TC-03', 'BCA-AJU-TC-03'] %}

dart-pom.properties:
  file.exists:
    - name: /opt/tomcat/webapps/website2/META-INF/maven/jhc.client.F63.web.figaroweb/website2/pom.properties

dart-pom.xml:
  file.exists:
    - name: /opt/tomcat/webapps/website2/META-INF/maven/jhc.client.F63.web.figaroweb/website2/pom.xml

dart-index.html:
  file.exists:
    - name: /opt/tomcat/webapps/website2/index.html

dart-logo.png:
  file.exists:
    - name: /opt/tomcat/webapps/website2/statics/images/logo.png

dart-notfound.gif:
  file.exists:
    - name: /opt/tomcat/webapps/website2/statics/images/notfound.gif

dart-sessionKeepAlive.png:
  file.exists:
    - name: /opt/tomcat/webapps/website2/images/sessionKeepAliveImage.png

dart.log:
  file.exists:
    - name: /opt/tomcat/logs/dart.log

dart-site.properties:
  file.exists:
    - name: /opt/jhc/website2/config/site.properties

dart-common-site.properties:
  file.exists:
    - name: /opt/jhc/website2/config/common-site.properties

dart-environment.properties:
  file.exists:
    - name: /opt/jhc/website2/config/environment.properties

dart-Admin.properties:
  file.exists:
   - name: /opt/jhc/website2/config/Admin.properties

dart-Client.properties:
  file.exists:
    - name: /opt/jhc/website2/config/Client.properties

dart-log4j-Client.properties:
  file.exists:
    - name: /opt/jhc/website2/config/log4j-Client.properties

dart-log4j-Admin.properties:
  file.exists:
    - name: /opt/jhc/website2/config/log4j-Admin.properties

dart-ehcache.xml:
  file.exists:
    - name: /opt/jhc/website2/config/ehcache.xml

dart-website2-client.log:
  file.exists:
    - name: /opt/tomcat/logs/website2-client.log

dart-website2-admin.log:
  file.exists:
    - name: /opt/tomcat/logs/website2-admin.log

{% endif %}

#####################################################################################################################                                           website3 SPECIFIC CONFIG
####################################################################################################################

{% if grains['host'] in ['BCA-AJT-TC-07', 'BCA-AJU-TC-07'] %}

website3-pom.xml:
  file.exists:
    - name: /opt/tomcat/webapps/website3/META-INF/maven/jhc.client.F63.web.figaroweb/website3/pom.xml

website3-pom.properties:
  file.exists:
    - name: /opt/tomcat/webapps/website3/META-INF/maven/jhc.client.F63.web.figaroweb/website3/pom.properties

website3-index.html:
  file.exists:
    - name: /opt/tomcat/webapps/website3/index.html

website3-logo.png:
  file.exists:
    - name: /opt/tomcat/webapps/website3/statics/images/logo.png

website3-notfound.gif:
  file.exists:
    - name: /opt/tomcat/webapps/website3/statics/images/notfound.gif

website3-sessionKeepAlive.png:
  file.exists:
    - name: /opt/tomcat/webapps/website3/images/sessionKeepAliveImage.png

website3-site.properties:
  file.exists:
    - name: /opt/jhc/website3/config/site.properties

website3.properties:
  file.exists:
    - name: /opt/jhc/website3/config/website3.properties

website3-common-site.properties:
  file.exists:
    - name: /opt/jhc/website3/config/common-site.properties

website3-environment.properties:
  file.exists:
    - name: /opt/jhc/website3/config/environment.properties

website3-Admin.properties:
  file.exists:
   - name: /opt/jhc/website3/config/Admin.properties

website3-Client.properties:
  file.exists:
    - name: /opt/jhc/website3/config/Client.properties

website3-log4j-Client.properties:
  file.exists:
    - name: /opt/jhc/website3/config/log4j-Client.properties

website3-log4j-Admin.properties:
  file.exists:
    - name: /opt/jhc/website3/config/log4j-Admin.properties

website3-ehcache.xml:
  file.exists:
    - name: /opt/jhc/website3/config/ehcache.xml

website3-website3.properties:
  file.exists:
    - name: /opt/jhc/website3/config/website3.properties

{% endif %}

#####################################################################################################################                       MONEY MATTERS SPECIFIC CONFIG
####################################################################################################################

{% if grains['host'] in ['BCA-AJT-TC-05', 'BCA-AJU-TC-05'] %}

website4-pom.xml:
  file.exists:
    - name: /opt/tomcat/webapps/website4/META-INF/maven/jhc.client.F63.web.figaroweb/ajb-1/pom.xml

website4-pom.properties:
  file.exists:
    - name: /opt/tomcat/webapps/website4/META-INF/maven/jhc.client.F63.web.figaroweb/ajb-1/pom.properties

website4-index.html:
  file.exists:
    - name: /opt/tomcat/webapps/website4/index.html

website4-logo.png:
  file.exists:
    - name: /opt/tomcat/webapps/website4/statics/images/logo.png

website4-notfound.gif:
  file.exists:
    - name: /opt/tomcat/webapps/website4/statics/images/notfound.gif

website4-sessionKeepAlive.png:
  file.exists:
    - name: /opt/tomcat/webapps/website4/images/sessionKeepAliveImage.png

website4-site.properties:
  file.exists:
    - name: /opt/jhc/ajb-1/config/site.properties

website4.properties:
  file.exists:
    - name: /opt/jhc/ajb-1/config/ajb-1.properties

website4-common-site.properties:
  file.exists:
    - name: /opt/jhc/ajb-1/config/common-site.properties

website4-environment.properties:
  file.exists:
    - name: /opt/jhc/ajb-1/config/environment.properties

website4-Admin.properties:
  file.exists:
   - name: /opt/jhc/ajb-1/config/Admin.properties

website4-log4j-Client.properties:
  file.exists:
    - name: /opt/jhc/ajb-1/config/log4j-Client.properties

website4-log4j-Admin.properties:
  file.exists:
    - name: /opt/jhc/ajb-1/config/log4j-Admin.properties

website4-ehcache.xml:
  file.exists:
    - name: /opt/jhc/ajb-1/config/ehcache.xml

website4-website3.properties:
  file.exists:
    - name: /opt/jhc/ajb-1/config/ajb-1.properties

{% endif %}

##################################################################################################################### THIS SECTION CHECKS THAT SERVICES ARE RUNNING ON EACH TOMCAT SERVER
####################################################################################################################

{% if grains['host'] in ['BCA-AJT-TC-04', 'BCA-AJU-TC-04', 'BCA-AJT-TC-07', 'BCA-AJU-TC-07'] %}

httpd:
  service.running:
    - name: httpd
    - enable: True

httpd.conf:
  file.exists:
    - name: /etc/httpd/conf/httpd.conf

php.conf:
  file.exists:
    - name: /etc/httpd/conf.d/php.conf

{% endif %}

rsyslog:
  service.running:
    - name: rsyslog
    - enable: True

salt-minion:
  service.running:
    - name: salt-minion
    - enable: True

sshd:
  service.running:
    - name: sshd
    - enable: True

ntpd:
  service.running:
    - name: ntpd
    - enable: True

check_jets:
  cmd.run:
    - name: ps aux | grep 'org.apache.catalina.startup.Bootstrap start' | grep -v grep

```

It may also be worth mentioning that I have used the above method in multiple areas across my state for the same purpose, yet targeting dev/test-server-01, 02, 03 etc without problems. It's specifically a problem with this section of code as other sections which target TEST-SERVER-04 in the state work fine (as I mentioned it runs 32/50)

# Answer
I have found the answer to my own question after some more investigation.

The reason the state was skipping the contents of my if statement was because the hostname of the server was in lower-case (god knows why). Therefore the in-built salt grains.item host which is called upon didn't recognise the server I was specifying.
> 2 votes

---
Tags: salt, jinja, yaml
---


# Question
Title: Error Vagrant Provisioning with Virtual Box on Fedora
I'm Provisioning Vagrant Box with Virtual-box on Fedora 26.

```
$ vagrant -v
Vagrant 1.9.1
$ virtualbox -v
virtualbox 5.1

```

Below is the error;

```
muhammadtaqi@localhost etc]$ vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
==> default: Importing base box 'ubuntu/trusty64'...
There was an error while executing `VBoxManage`, a CLI used by Vagrant
for controlling VirtualBox. The command and stderr is shown below.

Command: ["import", "/home/muhammadtaqi/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-trusty64/20170918.0.1/virtualbox/box.ovf", "--vsys", "0", "--vmname", "ubuntu-cloudimg-trusty-vagrant-amd64_1506105327930_50139", "--vsys", "0", "--unit", "6", "--disk", "/home/muhammadtaqi/VirtualBox VMs/ubuntu-cloudimg-trusty-vagrant-amd64_1506105327930_50139/box-disk1.vmdk"]

Stderr: 0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Interpreting /home/muhammadtaqi/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-trusty64/20170918.0.1/virtualbox/box.ovf...
OK.
0%...
Progress state: VBOX_E_FILE_ERROR
VBoxManage: error: Appliance import failed
VBoxManage: error: Could not create the imported medium '/home/muhammadtaqi/VirtualBox VMs/ubuntu-cloudimg-trusty-vagrant-amd64_1506105327930_50139/box-disk1.vmdk'.
VBoxManage: error: VMDK: Compressed image is corrupted '/home/muhammadtaqi/.vagrant.d/boxes/ubuntu-VAGRANTSLASH-trusty64/20170918.0.1/virtualbox/box-disk1.vmdk' (VERR_ZIP_CORRUPTED)
VBoxManage: error: Details: code VBOX_E_FILE_ERROR (0x80bb0004), component ApplianceWrap, interface IAppliance
VBoxManage: error: Context: "RTEXITCODE handleImportAppliance(HandlerArg*)" at line 886 of file VBoxManageAppliance.cpp

```

# Answer
Try to start with a clean sheet by removing the .vagrant.d folder or start to remove the corrupted zip. When the file or folder has been removed, one could run `vagrant up` again and check whether it works.
> 1 votes

---
Tags: ansible, vagrant, virtualization
---


# Question
Title: Where to start? Scaling PHP applications on AWS
Currently all of our PHP (Laravel) applications are running on one EC2 instance. To deploy code, we SSH onto the live box and pull from CodeCommit.

We now need to scale and deploy properly. I'm not sure where to start.

Most examples seem to use Elastic Beanstalk but I have been advised that this could be too restrictive.

I guess we want to end up with a load balancer and application servers which are created on demand.

Can anyone point me in the right direction? (I am a bit overwhelmed by all the AWS documentation.)

**Ok, this is the plan:**

I have signed up for A Cloud Guru and I'm going to work through some of the videos on there.

I'm going to sign up for a AWS Free Tier account and start tinkering.

I'm considering the *AWS Certified Solutions Architect - Associate* course, this looks like a great way to develop myself.

# Answer
Here's a rough approach to scaling any stateless app on AWS:

1. Run the app in an Auto Scaling Group (ASG). An ASG makes it easy to manage multiple servers, will automatically replace failed servers, and allows you to automatically scale the number of servers up or down in response to load.
2. To run an ASG, you need to create a Launch Configuration. This is a "template" that specifies the configuration for each server in the ASG: i.e., what type of instance to run (e.g. `t2.micro`), what AMI to run on the instance, the security group settings, etc.
3. The AMI you run on each instance in the ASG should have your app already installed (i.e., your PHP app). You can define your AMI as code using a tool like Packer.
4. The app in your AMI should be configured to run on boot using a process supervisor (e.g., using systemd or supervisord; this will also ensure the app is restarted automatically if it crashes.
5. Every time you change something in your app, run a build process to create a new AMI, update the Launch Configuration to use this new AMI, and roll that AMI out across your ASG. For example, you could do a zero-downtime deployment (somewhat similar to a blue-green deployment) by doubling the size of the ASG, waiting until all the new servers (from the updated Launch Configuration) launch and pass health checks, and then reducing the ASG back to its previous size (the ASG will terminate oldest instances by default).
6. With an ASG, you have more than one server, but you want to give your users a single endpoint to use. The typical solution to this is to deploy a load balancer in front of the ASG. If you're building a typical HTTP/HTTPs app (e.g., just about all PHP apps), you'll want to use the Application Load Balancer (ALB); if you're building an app that listens directly for TCP connections, you'll want the Load Balancer Classic or Network Load Balancer. Whichever load balancer you pick, AWS will create a domain name for it that you can use, and any requests you send to that domain name will be routed in round-robin fashion to different servers in your ASG.
7. You should a) configure the load balancer to perform health checks on your app so that it only routes to healthy servers and b) configure your ASG to register all of its servers in the load balancer automatically and to use the load balancer for health checks.
8. You can create Auto Scaling Policies for the ALB that change the number of servers in your ASG depending on various CloudWatch metrics. For example, you could configure the ASG to add servers when average CPU usage is over 80% and to remove servers when CPU usage drops below 60%.

For more info, check out A Comprehensive Guide to Building a Scalable Web App on Amazon Web Services.
> 3 votes

# Answer
> Currently all of our PHP (Laravel) applications are running on one EC2 instance. To deploy code, we SSH onto the live box and pull from CodeCommit.

The first thing you need to decide on is a Continuous Deploy strategy. There's a lot of servers out there with different use cases, you don't have to pick the most popular, just pick the one that fits your use case.

> I guess we want to end up with a load balancer and application servers which are created on demand.

AWS has some great reference architectures but overall I would say start simple. At a base level you'll be using Route 53, Certificate Manager, ALB (ELBv2), VPC, and EC2. I would get used to the options you have with each of these and play with them. AWS CSA will not teach you about reference architecture, rather it will go over what each product is used for.

Before you go building anything for production I would also recommend using an infrastructure deployment tool. AWS provides CloudFormation out of the box but that can be difficult to interface with. Some others are Sceptre and Terraform. These allow you to store your infrastructure as code in a repository and deploy updates via a CD server.

Also keep in mind if you don't have Continuous Integration then you'll need to work on that, followed by Continuous Delivery, then by Continuous Deployment. Each of these is a critical step in the right direction.

**Note on "Created on Demand"**

Please keep in mind that many people come to the cloud thinking that "scaling" is instantaneous. It's not. Depending on your provisioning it can take some time. You will not scale yourself out of burst traffic. Your traffic will need to be profiled, you will need to determine your instance capacity (for web servers I measure instances in requests per second with a load testing tool), and plan accordingly.

Scaling is primarily designed for a gradual rise in traffic over time, such as if your website traffic grows gradually larger around noon (peak) and then settles late in the day.
> 1 votes

---
Tags: amazon-web-services, autoscaling, load-balance, php
---


# Question
Title: Puppet master certificate generation not working
***Puppet Master:***

* Docker container with cent-OS-6
* configured in /etc/hosts
* configured /etc/puppet/puppet.conf with **server**,**dns\_alt\_names** and **certname**

**Puppet Agent:**

* Docker container with ubuntu OS
* configured serverip and agent ip in /etc/hosts
* configure /etc/puppet/puppet.conf with **server** value

**NOW trying to Generate Certificate in SERVER Machine with**

```
sudo -u puppet puppet master --no-daemonize --verbose

```

Displays:*`Notice: Starting Puppet master version 3.8.7`*

*Nothing happens related to Certificate Generation.*

I believe there aint any mistake in hosts file and other configuration missing. here might have things gone wrong.

**In the client machine** If I run this command **puppet agent -t** . I get the following error. As the certificate is not generated in server. I think I have to fix that first than this .

```
Warning: Unable to fetch my node definition, but the agent run will continue:
Warning: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Info: Retrieving pluginfacts
Error: /File[/var/lib/puppet/facts.d]: Failed to generate additional resources using 'eval_generate': Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Error: /File[/var/lib/puppet/facts.d]: Could not evaluate: Could not retrieve file metadata for puppet://www.puppetmaster.com/pluginfacts: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Info: Retrieving plugin
Error: /File[/var/lib/puppet/lib]: Failed to generate additional resources using 'eval_generate': Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Error: /File[/var/lib/puppet/lib]: Could not evaluate: Could not retrieve file metadata for puppet://www.puppetmaster.com/plugins: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Error: Could not retrieve catalog from remote server: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run
Error: Could not send report: Server hostname 'www.puppetmaster.com' did not match server certificate; expected one of puppetcertificate, DNS:puppet www.puppetmaster.com, DNS:puppetcertificate

```

# Answer
> 2 votes
First, try setting dns\_alt\_names in /etc/puppet/puppet.conf:

```
[main]
    dns_alt_names = www.puppetmaster.com
[master]
    autosign = true

```

Then see if your puppet-agent -t run works properly. be sure that you have properly set your server on the puppet client in /etc/puppet/puppet.conf too:

```
[agent]
    server = www.puppetmaster.com

```

---
Tags: puppet
---


# Question
Title: Multiple paths with different expiry time in gitlab-ci runners
I'm having issues with parsing a `build` directory between stages using Gitlab-CI.

Gitlab-CI wipes the created build artifacts between stages which seems weird. I'm trying to store the build artifacts between the build and the test stage, however the build stage also has the build artifact which I want to keep and also the build artifacts which are required to run the next stage.

Is it possible to have multiple expiry times with different paths using the `artifacts` option?

I have tried the following, which only keeps the second definition of `paths` (the build/test\* paths), and not the first `paths` (.dmg) declared.

```
  artifacts:
    paths:
    - build/*.dmg
    expire_in: 1 week
    paths:
    - build/test1
    - build/test2
    - build/test3
    expire_in: 15 mins

```

I have tried using the `caches` however can't seem to get that working... Any suggestions would be great appreciated!

# Answer
> 5 votes
This has been answered over on SO using a work around seeing as it doesn't seem possible according to the documents.

Basically, this can be done in 3 stages.

**Stage 1:** Build and store all artifacts.

```
build_stage:
  script:
    - build
  artifacts:
    paths:
    - build/*.dmg
    - build/test1
    - build/test2
    - build/test3
    expire_in: 15 mins

```

**Stage 2.1:** Do the next official stage of the job (ie run tests in my scenario) using the artifacts `build/test1, build/test2` and `build/test3`.

```
test_stage:
  script:
    - test
  dependencies:
  - build

```

**Stage 2.2:** Running concurrently with Stage 2.1 you can just have an empty job but sets a new artifact expiry date.

```
overwrite_artifact_stage:
  script:
    - echo 'saving artifact'
  artifacts:
    paths:
    - build/*.dmg
    expire_in: 1 week

```

# Answer
> 3 votes
It seems you're trying to define the artifacts globally, that won't work with multiple paths since you're overwriting the definition of the key. Instead you should define the paths per job.

artifacts: expire\_in: 1 week

```
build:buildjob
    paths:
    - build/test1
    - build/test2
    - build/test3
    expire_in: 15 mins

test:testjob
      paths:
        - build/*.dmg
      artifacts:
        expire_in: 1 week

```

---
Tags: gitlab
---


# Question
Title: Building a Docker container using Gitlab CI Docker runner
I am trying to build a docker image using Gitlab Docker runner.

For this I have configured the runner with a shared volume in */etc/gitlab-runner/config.toml* called */builds*:

```
concurrent = 1
check_interval = 0

[[runners]]
  name = "My Docker Runner"
  url = "http://host"
  token = "xxxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache", "/builds"]
  [runners.cache]

```

I created a *.gitlab-ci.yml* in my repo which loosely looks like this:

```
image: alpine:latest

variables:
  GIT_STRATEGY: none

test_ci:
  script:
    - apk update
    - apk add docker
    - apk add git
    - docker info
    - git clone http://gitlab-ci-token:${CI_JOB_TOKEN}@host/user/repo.git /builds/user/repo
    - docker build -t database .
    - docker run database

```

I have also created a *Dockerfile* in the repo which looks like this:

```
FROM postgres:9.6

ADD /builds/user/repo/script.sql /docker-entrypoint-initdb.d/

```

Everything works alright except for the Docker *ADD* instruction which fails because the */builds/user/repo/script.sql* cannot be found:

```
ADD failed: stat /var/lib/docker/tmp/docker-builder886348850/builds/user/repo/script.sql: no such file or directory

```

So I imagine that the */builds* volume I am trying to share between containers is not working as I am expecting it to ... Could someone help me with this? I would like to be able to share the code which is cloned on the runner container to the *docker build* command and then to the newly created container.

# Answer
Weird Docker. I changed the *WORKDIR* in the *Dockerfile* and it worked:

```
FROM postgres:9.6

WORKDIR /builds/user/
ADD repo/script.sql /docker-entrypoint-initdb.d/

```

**EDIT** The `/builds` folder does not exist on the host system. I thought that adding it to the runner's config would create a volume container that would be shared between containers. But no. So I removed it from the runner config, */etc/gitlab-runner/config.toml*:

```
[[runners]]
  name = "My Docker Runner"
  url = "http://host"
  token = "xxxxxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = false
    disable_cache = false
    volumes = ["/var/run/docker.sock:/var/run/docker.sock", "/cache"]
  [runners.cache]

```
> 3 votes

---
Tags: docker, continuous-integration, gitlab
---


# Question
Title: Kubernetes flannel pod getting the wrong network
Flannel running in a pod is getting the wrong subnet and networking is just not happy, the symptom is flannel is being assigned /24's from the 10.105.0.0/16. it should be assigning /26's from 10.105.5.128/21. Thanks for any help.

here are the details:

```
/usr/bin/kubeadm init \
--kubernetes-version v1.7.5 \
--pod-network-cidr 10.105.5.128/21 \
--service-cidr 10.105.5.136/21 \
--token XXXXXXXXXXX

```

kube-flannel-rbac.yml is loaded after kube-flannel.yml only modified bit(SubenetLen and Network) from kube-flannel.yml:

```
{
  "Network": "10.105.5.128/21",
  "SubnetLen": 26,
  "Backend": {
    "Type": "vxlan"
  }
}

```

DNS is set in the systemd file to:

```
--cluster-dns=10.105.5.136.10

```

Using Ubuntu 16.04 LTS and stock kernel

here is the docker daemon.json file:

```
{
"hosts":[
    "fd://",
    "0.0.0.0"
],
"ip-masq":false,
    "experimental": true,
"registry-mirrors": [
    "http://hub.xyz.com"
],
"insecure-registries": [
    "http://hub.xyz.com"
],
"tls": true,
"tlsverify": true,
"tlscacert":"/etc/docker/ca.pem",
"tlscert":"/etc/docker/cert.pem",
"tlskey":"/etc/docker/key.pem"
}

```

all kuberentes components are 1.7.5 installed from ubuntu k8s repos

here is the log of the kube-flannel container:

```
I0926 03:29:10.214198      89 main.go:446] Determining IP address of default interface
I0926 03:29:10.216166      89 main.go:459] Using interface with name eth0 and address 10.105.5.12
I0926 03:29:10.216261      89 main.go:476] Defaulting external address to interface address (10.105.5.12)
I0926 03:29:10.242216      89 kube.go:283] Starting kube subnet manager
I0926 03:29:10.242055      89 kube.go:130] Waiting 10m0s for node controller to sync
I0926 03:29:11.242864      89 kube.go:137] Node controller sync successful
I0926 03:29:11.242957      89 main.go:226] Created subnet manager: Kubernetes Subnet Manager - kube-m2.XXXXX.com
I0926 03:29:11.242969      89 main.go:229] Installing signal handlers
I0926 03:29:11.243984      89 main.go:330] Found network config - Backend type: vxlan
I0926 03:29:11.288902      89 ipmasq.go:51] Adding iptables rule: -s 10.105.0.0/21 -d 10.105.0.0/21 -j RETURN
I0926 03:29:11.306692      89 ipmasq.go:51] Adding iptables rule: -s 10.105.0.0/21 ! -d 224.0.0.0/4 -j MASQUERADE
I0926 03:29:11.314413      89 ipmasq.go:51] Adding iptables rule: ! -s 10.105.0.0/21 -d 10.105.0.0/24 -j RETURN
I0926 03:29:11.323229      89 ipmasq.go:51] Adding iptables rule: ! -s 10.105.0.0/21 -d 10.105.0.0/21 -j MASQUERADE
I0926 03:29:11.329676      89 main.go:279] Wrote subnet file to /run/flannel/subnet.env
I0926 03:29:11.329746      89 main.go:284] Finished starting backend.
I0926 03:29:11.329829      89 vxlan_network.go:56] Watching for L3 misses
I0926 03:29:11.329903      89 vxlan_network.go:64] Watching for new subnet leases

```

# Answer
You have either a bad understanding or misconception about how a cidr works.

10.105.5.128/21 is just one address in the range 10.105.0.0 - 10.105.7.255.
So either your cidr bits are not the one you're expecting, 10.105.5.128/26 would be a valid network definition but that's a /26 subnet within the 10.105.0.0/21, or you're not using the proper network definition.

You can check those division on this very helping site (the link already use your networks above.
> 2 votes

---
Tags: docker, kubernetes
---


title: How to Chef things that don't exist yet

text:
# Question
Title: How to Chef things that don't exist yet
Let us say I have some Chef code like:

```
require 'mixlib/shellout'
yum_package 'somepackage'
myvar =  Mixlib::ShellOut.new('/bin/somecommand').run_command.stdout.strip

```

Where `/bin/somecommand` does not exist yet because it is installed by `somepackage`. This will fail at recipe compile time for that reason, but will obviously work at convergence time providing the package installs successfully (and if it doesn't then obviously the recipe has failed anyway). This also fails if the package is installed be a previous recipe in the runlist since they are all compiled together upfront. How do I include things in a Chef recipe that that recipe or runlist installs itself?

# Answer
I'd go with `node.run_state` to store a transient variable in a run and define it in a `ruby_block` so it happens at converge time, something like this:

```
yum_package 'somepackage'

ruby_block 'set myvar' do
  block do
    node.run_state['my_var'] = Mixlib::ShellOut.new('/bin/somecommand').run_command.stdout.strip
  end
end

```

As far as I know requiring 'mixlib/shellout' is not necessary.
> 4 votes

# Answer
The specifics depend on the exact use case. If the value is only used as a property on another resource, you would use the `lazy` helper method. If you don't need the output, you would use the `execute` resource. In some more complex cases you might use a `ruby_block` resource or write your own custom resource.
> 2 votes

---
Tags: chef, configuration-management, infrastructure-as-code
---


# Question
Title: Is there a tool that makes it possible to validate Jenkinsfiles online?
At the moment I am editing a Jenkinsfile and then let it run unless Jenkins reports a issue. This approach costs a lot of time. I prefer to validate the syntax before committing the Jenkinsfile. Is there a tool that solves this issue?

In gitlab there is a URI, i.e. `/ci/lint` that makes it possible to submit a gitlab file, click on the check button and then the UI will indicate whether the syntax is correct or not.

# Answer
> 13 votes
Here is some documentation on the Jenkins pipeline linter and its commands. Do you need to validate **before** a commit? If not, it would be really trivial to run the linting command prior to your pipeline running, and simply fail if it does not pass.

From Command-line Pipeline Linter:

> Jenkins can validate, or "lint", a Declarative Pipeline from the command line before actually running it. This can be done using a Jenkins CLI command or by making an HTTP POST request with appropriate parameters. We recommended using the SSH interface to run the linter. See the Jenkins CLI documentation for details on how to properly configure Jenkins for secure command-line access.
>
> *Linting via the CLI with SSH*
>
> ```
> # ssh (Jenkins CLI)
> # JENKINS_SSHD_PORT=[sshd port on master]
> # JENKINS_HOSTNAME=[Jenkins master hostname]
> ssh -p $JENKINS_SSHD_PORT $JENKINS_HOSTNAME declarative-linter < Jenkinsfile
>
> ```
>
> *Linting via HTTP POST using `curl`*
>
> ```
> # curl (REST API)
> # Assuming "anonymous read access" has been enabled on your Jenkins instance.
> # JENKINS_URL=[root URL of Jenkins master]
> # JENKINS_CRUMB is needed if your Jenkins master has CRSF protection enabled as it should
> JENKINS_CRUMB=`curl "$JENKINS_URL/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,\":\",//crumb)"`
> curl -X POST -H $JENKINS_CRUMB -F "jenkinsfile=<Jenkinsfile" $JENKINS_URL/pipeline-model-converter/validate
>
> ```
>
> **Examples**
>
> Below are two examples of the Pipeline Linter in action. This first example shows the output of the linter when it is passed an invalid `Jenkinsfile`, one that is missing part of the `agent` declaration.
>
> *Jenkinsfile*
>
> ```
> pipeline {
>   agent
>   stages {
>     stage ('Initialize') {
>       steps {
>         echo 'Placeholder.'
>       }
>     }
>   }
> }
>
> ```
>
> *Linter output for invalid Jenkinsfile*
>
> ```
> # pass a Jenkinsfile that does not contain an "agent" section
> ssh -p 8675 localhost declarative-linter < ./Jenkinsfile
> Errors encountered validating Jenkinsfile:
> WorkflowScript: 2: Not a valid section definition: "agent". Some extra configuration is required. @ line 2, column 3.
>      agent
>      ^
>
> WorkflowScript: 1: Missing required section "agent" @ line 1, column 1.
>    pipeline }
>    ^
>
> ```
>
> In this second example, the `Jenkinsfile` has been updated to include the missing `any` on `agent`. The linter now reports that the Pipeline is valid.
>
> *Jenkinsfile*
>
> ```
> pipeline {
>   agent any
>   stages {
>     stage ('Initialize') {
>       steps {
>         echo 'Placeholder.'
>       }
>     }
>   }
> }
>
> ```
>
> *Linter output for valid Jenkinsfile*
>
> ```
> ssh -p 8675 localhost declarative-linter < ./Jenkinsfile
> Jenkinsfile successfully validated.
>
> ```

---
Tags: jenkins-blueocean
---


# Question
Title: org.jenkinsci.lib.envinject.EnvInjectException: java.io.IOException: remote file operation failed
I've the following error in Jenkins console when running job which uses Windows slave:

```
Started by user
  [EnvInject] - Loading node environment variables.
 ERROR: SEVERE ERROR occurs
 org.jenkinsci.lib.envinject.EnvInjectException: java.io.IOException: remote file operation failed: C:\Program Files\Java\jre1.8.0_91 at hudson.remoting.Channel@2f4af34f:Management VPC Slave: Windows (sir-rvhi7z8q): java.io.IOException: Unable to delete 'C:\Program Files\Java\jre1.8.0_91\bin\awt.dll'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
 at org.jenkinsci.plugins.envinject.util.RunHelper.getBuildVariables(RunHelper.java:137)
 at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironmentWithoutJobPropertyObject(EnvInjectListener.java:236)
 at org.jenkinsci.plugins.envinject.EnvInjectListener.setUpEnvironment(EnvInjectListener.java:50)
 at hudson.model.AbstractBuild$AbstractBuildExecution.createLauncher(AbstractBuild.java:534)
 at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:454)
 at hudson.model.Run.execute(Run.java:1737)
...
 Caused by: java.io.IOException: remote file operation failed: C:\Program Files\Java\jre1.8.0_91 at hudson.remoting.Channel@2f4af34f:Management VPC Slave: Windows (sir-rvhi7z8q): java.io.IOException: Unable to delete 'C:\Program Files\Java\jre1.8.0_91\bin\awt.dll'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
 at hudson.FilePath.act(FilePath.java:994)
 at hudson.FilePath.act(FilePath.java:976)
...
 Caused by: java.io.IOException: Unable to delete 'C:\Program Files\Java\jre1.8.0_91\bin\awt.dll'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts.
 at hudson.Util.deleteFile(Util.java:250)
 at hudson.FilePath.deleteRecursive(FilePath.java:1211)
...
Caused by: java.nio.file.AccessDeniedException: C:\Program Files\Java\jre1.8.0_91\bin\awt.dll
 at sun.nio.fs.WindowsException.translateToIOException(Unknown Source)

```

which stops the job at the very start.

I've no idea why *EnvInject* plugin would want to remove some Java files (such as `awt.dll`).

Any idea how to solve this problem?

# Answer
> 5 votes
The *EnvInject* plugin is basically attempting to load the build environment which uses configuration of JDK tools in Jenkins which fails during reinstallation of Java JDK library on the Windows slave (as per given configuration).

This JDK configuration can be found in *Global Tool Configuration* (at `/configureTools`) for the given JDK profile which was selected in that job configuration. The installation process failed to reinstall *Java JDK* due to running Java process in the background (such as `java -jar slave.jar`) which was using these *DLL* files.
So it sounds like it's not possible to reinstall the same Java JDK files which are used to run the Jenkins slave process.

The solution was to disable *Install automatically* option in *Global Tool Configuration*, as Java JDK was already installed on that slave.

Related: JENKINS-46989 - EnvInjectException: remote file operation failed.

---
Tags: jenkins, java
---


# Question
Title: Running multiple websites on Docker in isolated or shared database containers?
So I have been reading up on Docker, and I cannot quite get my head around this concept. There are literally dozens of tutorials on the web about spawning a Wordpress instance on Docker within 5 minutes. All these sites spawn their processes separately; a singe SQL container, a single webserver container and a PHP + filesystem container.

Apparently the Docker philosophy is to keep everything nice and small and separated. Proper isolation, distribution and scalability; all fine. But what if I need two Wordpress sites, or ten?

The advantage of separate containers is that every site can be instantiated with its own passwords and other secrets via a docker-compose file, but having 10 database containers running next to each other feels like wasting resources. Once single SQL container should be capable of running 10 databases.

I was wondering if running multiple instances is indeed resource heavier, and/or if the added value of strict isolation outweighs the resource usage?

(I have chosen SQL, but the question basically applies to all resources that can be shared across sites. Think of mailservers, virus scanners, etc.)

# Answer
> 3 votes
Docker and all the other fancy names are just tools. The examples are just examples.

Resource-wise, Docker changes nothing much beyond usual Linx/Unix multiprocessing. It does not have the overhead of VMs, and little overhead over just starting the process directly ( https://stackoverflow.com/questions/21889053/what-is-the-runtime-performance-cost-of-a-docker-container ).

So it is really up to you to decide how to use it. If you have closely related applications under your own command, maybe in your intranet, why not use a shared DB. If they are customer DBs, or you might need to run different versions of the software, or different incompatible configurations, then separate services might be fine.

For RDBMSses especially, they are usually very good at managing resources (RAM, HDD) and so for this case specifically, running only one may just be the best thing. It's really more about what your needs are than general guidelines.

---
Tags: docker, containers, scalability
---


# Question
Title: Does my organization need adopt Agile Soft. Dev. before adopting DevOps?
Agile software development is the methodology of choice for software shops today. But there are still organisations which are not practicing Agile in their software development, who might be interested in adopting DevOps.

When I say Agile Software Development, I mean any offspring that came out of the Manifesto for Agile Software Development. Such as Extreme Programming, Scrum, Lean Software Development and others.

Is Agile software development a mandatory prerequisite of adopting DevOps on an organisation level?

# Answer
Agile Software Development is not *required* to do DevOps, but I believe the argument can be made that the value proposition for DevOps is often a lot lower without Agile.

DevOps is a lot of things, but *automation* is a central theme. The value of automation increases in direct proportion to the frequency with which Development creates new releases. Frequent deployment has a positive impact on certain types of products, specifically consumer applications. High velocity all the way through the software delivery lifecycle returns value for each iteration (anyone seen the CA ad where the zombies want new features in their apps?)

Without Agile, high-frequency releases are extremely difficult, if not impossible. If the Development team is releasing software once a quarter, or twice a year, DevOps can still automate the process, but then what is the point? The investment in time, training, and resources to adopt DevOps may be partly returned in quality, however, the best value is in maintaining high velocity throughout the delivery lifecycle.

One could also argue that if you're going to adopt DevOps, why *wouldn't* you also adopt Agile? The principles that make them both work, work well together. Practicing DevOps by itself, without Agile, could create an imbalance between Ops and Dev, in which Ops is outperforming Dev for service delivery.
> 8 votes

# Answer
# NO

I would argue that *Mature* DevOps operation, does require a *Mature* Agile process. You are unlikely to be able to get the full confidence to continuously deploy or allow your developers to initiate the deployment process without a *mature* Agile process in place.

However, I believe it is very important to make it clear that an organisation does NOT need to adopt their agile process before building up their DevOps culture and infrastructure. In fact, I would argue that it is actually easier to adopt Agile once you have some basic DevOps working in your company. Rather than Agile being a prerequisite for DevOps, I would suggest that DevOps be used to help advance your agile implementation.
> 4 votes

# Answer
At the Agile 2008 conference, Andrew Clay Shafer and Patrick Debois discussed "Agile Infrastructure". This was the start of the DevOps movement.

But while DevOps has it's roots in Agile Manifesto, the fact is that it is not strictly required for the development teams to be Agile. Most of the Agile Principles are indeed important to follow, but no specific Agile Methodology is prescribed and if the development team can come up with their own way of working, while keeping more or less to the Agile Principles, you can realize the benefits.
> 2 votes

# Answer
Not necessarily; as the other answers have stated, Agile methodologies and DevOps mentality can build off of each other, but to me, DevOps is broader than either agile or lean development efforts. Companies that adopt DevOps look at the entire value stream, including operations, development, product design, marketing, security etc... the movement can originate within any silo with the intent of dismantling barriers.
> 2 votes

# Answer
IMHO it doesn't *need* to.

From DevOps (emphasis mine):

> DevOps (a clipped compound of "software DEVelopment" and "information technology OPerationS") is a term used to refer to a set of practices that emphasize the collaboration and communication of both software developers and information technology (IT) professionals while automating the process of software delivery and infrastructure changes.\[2\] It aims at establishing a culture and environment where building, testing, and releasing software **can happen** rapidly, frequently, and more reliably.

I highlighted `can happen` as it is not `must happen`.

Nothing in the above description actually **requires** an agile development methodology.

But I suspect often transitions towards DevOps often incorporate insertion of agile methodologies in the development process as they really fit well together.
> 1 votes

# Answer
The concept of the devops has arisen precisely because of the proliferation of agile, so it is appropriate to use the devops approaches in a agile. For example, with agile, continuous integration is used (which is part of the devops approach), orchestration of services and containers for testing and rolling out new releases for test and prod environments.
> 1 votes

---
Tags: culture, agile
---


# Question
Title: What is the difference between the roles Director of DevOps vs Head of DevOps and related areas of responsibility?
Is there really a difference? Would "Director" lead several "Head of"?

# Answer
> 4 votes
It is not clear whether there is a clear difference between a Director of DevOps and a Head of DevOps, e.g. according to this vacancy both roles are the same, while according to this the Head of Devops is the owner and responsible for DevOps, while a Director of DevOps is responsible for implementing a DevOps culture.

---
Tags: culture
---


title: Reusable module in Terraform

text:
# Question
Title: Reusable module in Terraform
I want to get an `autoscaling group name` variable in `module_apps` and use it in other modules - for example `module_init`. However when I try to run `terraform plan` it does not work.

# content of module\_apps

```
resource "aws_autoscaling_group" "example"
{
  name = "asg-ad"
}

output "asg_name"
{
  value = "${aws_autoscaling_group.example.name}"
}

```

# content of module\_init

```
module "module_init"
{
  source = "../module_apps"
  asg_name = "${module.module_init.asg_name}"
}

```

Trying to run `terraform plan` seems that it can't get the source module:

```
o:Get: file:///home/user/terraform/application/module_apps
o:Get: file:///home/user//terraform/application/module_init
o:Get: file:///home/user//terraform/application/module_elb
o:Get: file:///home/user//terraform/application/module_deloy
o:Get: file:///home/user/terraform/application/module_apps
o:Get: file:///home/user//terraform/application/module_init
o:Get: file:///home/user//terraform/application/module_elb
o:Get: file:///home/user//terraform/application/module_deloy
o:Get: file:///home/user//terraform/application/module_init
o:Get: file:///home/user//terraform/application/module_elb
o:Get: file:///home/user//terraform/application/module_deloy
o:Get: file:///home/user/terraform/application/module_apps
o:Get: file:///home/user//terraform/application/module_init
o:Get: file:///home/user//terraform/application/module_elb
o:Get: file:///home/user//terraform/application/module_deloy
o:Get: file:///home/user//terraform/application/module_init
o:Get: file:///home/user//terraform/application/module_elb
o:Get: file:///home/user//terraform/application/module_deloy
o:Get: file:///home/user/terraform/application/module_apps
o:Get: file:///home/user//terraform/application/module_init
o:Get: file:///home/user//terraform/application/module_elb
o:Get: file:///home/user//terraform/application/module_deloy

```

# Answer
If it doesn't outright fail you can expect to see a cycle warning where you've created a loop in Terraform and it's trying to fulfil your request for `asg-name` before the asg has been created.

To get what you need you'll need to have something like the following:

# **module\_apps/module\_apps.tf**

```
resource "aws_autoscaling_group" "example" {
  name = "${var.asg_name}"
}

```

# **module\_init.tf**

```
variable "asg_name" { default = "asg-ad" }

module "module_init" {
  source = "../module_apps"

  asg_name = "${var.asg_name}"
}

output "asg_name" {
  value = "${var.asg_name}"
}

```

This way, if you call `module_init` somewhere else you'll have `asg_name` available to you. Always be careful of trying to use an output of a module to create the module, Terraform doesn't handle loops like that well and the errors can be confusing to diagnose.
> 1 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: Build FreeBSD Docker image using Ubuntu host OS
I know that in generally it is impossible to build FreeBSD Docker image on Linux host OS due to different kernels and sys calls. Relevant discussion.

However, I would like to know maybe there is a simple workaround to overcome this limitation?

P.S. I don't want to consider Vagrant usage inside Docker container and I don't want to consider usage of FreeBSD as host OS.

# Answer
> 5 votes
As you can't build a docker image for an arm kernel from a x86 one neither.

Nor can't you build an image with a glibc lib targetting a 4.x kernel on a host running a 3.X kernel.

Docker isolate processes and give a virtual filesystem but that's still the host kernel who own the process, so obviously without a layer of hardware virtualization like virtualbox, vmware, xen, kvm, there's no chance to build an image with binaries made for a different kernel type.

---
Tags: docker, linux
---


title: DevOps practice on AWS instances

text:
# Question
Title: DevOps practice on AWS instances
I am new to DevOps, started learning the different tools. I would like to practice all of the tools like Jenkins, Chef, GIT, Puppet, Ansible, Docker, Kubernetes, etc .

My question is If I would like to practice all of the tools on AWS Free tier Instances How many of them are needed? and which tools I can combine them and install on which instances ?

Please help me in this regard.

# Answer
> 5 votes
You might not be able to get this done on the free tier. Puppet for example isn't going to want to start because of the RAM limitations. The AWS free tier uses thje t2.micro instance which only has 1GB of RAM. Your operating system alone probably needs 512 to run at idle. This leaves you a mere 512 MB or ram for all of the things that you listed.

While the issue with puppet not starting can be overcome by editing `/etc/sysconfig/puppetserver` and setting something like:

```
JAVA_ARGS="-Xms512m -Xmx512m"

```

(this issue arises because puppet expects 2 GB or RAM out of the box)

You are bound to find that running all of these on a single server is debilitating and painful with only 1 GB of RAM. You should probably consider using something like virtualbox on your PC or laptop so that you can get some more ram than that offered by the AWS free tier. I'd say you want at least 4 or maybe 8 GB of RAM. While this probably *can* be done, I don't recommend it. You may have to stop one service to run another which then makes it difficult to have one talk to the other (eg, ansible talk to Jenkins while puppet is running, for example.)

# Answer
> 3 votes
The Azure free trial gives you $200 in credit for the first month which is plenty of capacity spin up many VMs of an adequate size to run all the tools you mention. I would advise not starting the trial period until you have a good plan of what you want to do, to get maximum benefit from it.

> Provision up to 14 virtual machines, 40 SQL databases or 8 TB of storage for a month

# Answer
> 1 votes
The good news for your question is that all of these tools can be played with on one free tier instance in AWS.

Using Docker/K8's, which it seems you want to learn as well, you'll be able to run all of the other tools that you want to learn as containers on that one host which should have enough grunt to allow you to play around.

# Answer
> 0 votes
You can use 2 instance in AWS free tier if you want to practice as client and Server configuration. When instance is not required shut it down.

---
Tags: jenkins, docker, amazon-ec2, chef
---


title: Workplace viewport scripting

text:
# Question
Title: Workplace viewport scripting
Working with DevOps toolchains requires switching contexts between code, dashboards and command line terminals (local and remote). That is just for one project - and many of us work in more than one.

To reduce switching and navigating, folks and me too use more and more monitors - 4K (UHD) is a welcomed technology.

Say altogether, a set of windows and terminals for a context is a DevOps viewport (or maybe there is a better word).

Now, what are - if any - convenient means to "script" windows/terminals configurations as a CasC (Configuration as Code) artefact so you could easily load and switch your CI/CD environment viewports?

* Windows 10 environment?
* Ubuntu Linux environent?

# Answer
> 1 votes
There are tools around the web for saving desktops. Still best solution I found out to be running a set of virtual machines on your computer, having one setup in one vm.

That takes some space, but you can save the state of vm upon vm shut down and come back to exactly same state when you come back. Maybe sessions have died but the windows certainly are as you left them.

Solution is stated on http://www.makeuseof.com/tag/save-restore-windows-session/ and it is for windows but I suppose the same will apply for Linux vm too.

---
Tags: automation, casc
---


# Question
Title: Is it possible to design headless tests with JMeter/Selenium? (and Docker)
There are of course Dockerizations of Selenium and JMeter - but can they work together? All the tutorials I can find discuss desktop solutions so not headless.

# Answer
> 3 votes
JMeter supports headless operation, in fact it's even the recommended mode for best load testing results. From Non-GUI Mode (Command Line mode):

> For load testing, you must run JMeter in this mode (Without the GUI) to get the optimal results from it. To do so, use the following command options:
>
> **-n** This specifies JMeter is to run in non-gui mode

As for Selenium, things are a bit debatable, the Does Selenium support headless browser testing? SO post is probably interesting, personally I find Ross Patterson's and General Redneck's answers most useful - basically recommending the use of a virtual X server (Xvfb, for example) instead of a headless browser.

A nice related post: How To Run Your Tests Headlessly with Xvfb.

---
Tags: docker, selenium
---


# Question
Title: Jenkins Select unique parameter with multiple values
Hello I want the following behaviour.

A want my job to be parametrized. The thing is that I want to select a unique parameter with multiple values.

I have a unique parameter named number with 2 options (1, 2) and then set of values linked to each option:

> Parameter name: var
>
> Options:
>
> 1
>
> 2

Let´s say:

* One set of values:

> * value1: "1.0.0"
> * value2: "myname"
> * value3:"build/libs"

* Another set of values:

> * value1: "2.0.0"
> * value2: "myname2"
> * value3: "var/lib"

In the "Build section" I select "Execute shell" with the following lines:

```
echo ${var.value1}
echo ${var.value2}
echo ${var.value3}

```

So when building the job I would like to select from a dropdown list with 2 values as said above: 1 and 2.

If I select 1, I would like to take the first set of values so that it would dislplay:

```
1.0.0
myname
build/libs

```

On the other hand if I select 2, I would like to display the second set of values:

```
2.0.0
myname2
var/lib

```

I have tried "Extended Choice Parameter plugin" but I don´t know how to manage this situation.

Thank you.

# Answer
> 1 votes
My suggestion is to set job parameterized with two options, and just set up variables in the Shell Build step, like this:

```
if [ "$VAR" == "1" ]; then
    value1="1.0.0"
    value2="myname"
    value3="build/libs"
elif [ "$VAR" == "2" ]; then
    value1="2.0.0"
    value2="myname2"
    value3="var/lib"
fi

echo ${var.value1}
echo ${var.value2}
echo ${var.value3}

```

---
Tags: jenkins
---


title: How to make Tiller highly available?

text:
# Question
Title: How to make Tiller highly available?
I'm using helm to deploy my services to a Kubernetes cluster.
I want to improve the service availability of my tiller pod. So I scaled the tiller deployment to 2 replicas.
Helm is working as before with no issues that might be related to this.

Does anyone have an idea why this might NOT be a good idea to increase reliability of helm and tiller?

**EDIT:**

* Helm v3 is expected to drop the usage of tiller. Making this issue not relevant
* If you want a tillerless experience in helm v2, you can use a helm-tiller plugin

# Answer
Running two tillers does not make it HA. There is an open issue at the moment.
> 5 votes

---
Tags: kubernetes, helm
---


# Question
Title: Why is Jenkins unable to read config.xml?
```
java.io.EOFException: no more data available - expected end tag </hudson> to close start tag <hudson> from line 2, parser stopped on END_TAG seen ...<globalNodeProperties/>\n\n... @42:1
    at org.xmlpull.mxp1.MXParser.fillBuf(MXParser.java:3035)
    at org.xmlpull.mxp1.MXParser.more(MXParser.java:3046)
    at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1384)
    at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
    at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109)
Caused: com.thoughtworks.xstream.io.StreamException:  : no more data available - expected end tag </hudson> to close start tag <hudson> from line 2, parser stopped on END_TAG seen ...<globalNodeProperties/>\n\n... @42:1
    at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:126)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:135)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.hasMoreChildren(AbstractPullReader.java:87)
    at com.thoughtworks.xstream.io.ReaderWrapper.hasMoreChildren(ReaderWrapper.java:32)
    at hudson.util.RobustReflectionConverter.doUnmarshal(RobustReflectionConverter.java:309)
    at hudson.util.RobustReflectionConverter.unmarshal(RobustReflectionConverter.java:270)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
Caused: com.thoughtworks.xstream.converters.ConversionException:  : no more data available - expected end tag </hudson> to close start tag <hudson> from line 2, parser stopped on END_TAG seen ...<globalNodeProperties/>\n\n... @42:1 :  : no more data available - expected end tag </hudson> to close start tag <hudson> from line 2, parser stopped on END_TAG seen ...<globalNodeProperties/>\n\n... @42:1
---- Debugging information ----
message             :  : no more data available - expected end tag </hudson> to close start tag <hudson> from line 2, parser stopped on END_TAG seen ...<globalNodeProperties/>\n\n... @42:1
cause-exception     : com.thoughtworks.xstream.io.StreamException
cause-message       :  : no more data available - expected end tag </hudson> to close start tag <hudson> from line 2, parser stopped on END_TAG seen ...<globalNodeProperties/>\n\n... @42:1
class               : hudson.model.Hudson
required-type       : hudson.model.Hudson
converter-type      : hudson.util.RobustReflectionConverter
path                : /hudson
line number         : 42
version             : not available
-------------------------------
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:79)
    at com.thoughtworks.xstream.core.AbstractReferenceUnmarshaller.convert(AbstractReferenceUnmarshaller.java:65)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:66)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.convertAnother(TreeUnmarshaller.java:50)
    at com.thoughtworks.xstream.core.TreeUnmarshaller.start(TreeUnmarshaller.java:134)
    at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.unmarshal(AbstractTreeMarshallingStrategy.java:32)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1189)
    at hudson.util.XStream2.unmarshal(XStream2.java:114)
    at com.thoughtworks.xstream.XStream.unmarshal(XStream.java:1173)
    at hudson.XmlFile.unmarshal(XmlFile.java:160)
Caused: java.io.IOException: Unable to read /var/lib/jenkins/config.xml
    at hudson.XmlFile.unmarshal(XmlFile.java:162)
    at jenkins.model.Jenkins.loadConfig(Jenkins.java:3078)
    at jenkins.model.Jenkins.access$1200(Jenkins.java:307)
    at jenkins.model.Jenkins$16.run(Jenkins.java:3096)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
    at jenkins.model.Jenkins$7.runTask(Jenkins.java:1090)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:47)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1124)
    at jenkins.model.Jenkins.<init>(Jenkins.java:929)
    at hudson.model.Hudson.<init>(Hudson.java:86)
    at hudson.model.Hudson.<init>(Hudson.java:82)
    at hudson.WebAppMain$3.run(WebAppMain.java:235)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:252)

```

# Answer
There would be many reason for it. If you are writing/updating `JENKINS_HOME/config.xml`, then the possibility of a syntax error may be in `config.xml`. So you can recover it with the following steps:

Step 1: Stop jenkins service

```
cmd> service jenkins stop

```

step 2: `mv JENKINS_HOME/config.xml` to another file for backup

```
cmd> mv JENKINS_HOME/config.xml JENKINS_HOME/config.xml_bak

```

step 3: Start jenkins service

```
cmd> service jenkins start

```
> 8 votes

# Answer
It looks like your XML tags aren't balanced. You should run your config file through an XML validator to see what's wrong with it if you can't spot it by eye.
> 2 votes

---
Tags: jenkins
---


title: SSH to VM inside Docker container

text:
# Question
Title: SSH to VM inside Docker container
I have a Docker image based on Ubuntu Xenial. Inside that container I try to start FreeBSD VM by using Vagrant and VirtualBox (Note: host OS for Docker is VM based on Vagrant and VirtualBox). VM is being provisioned fine, however, Vagrant can not ssh there. I also tried ssh with debug within container:

```
# ssh -vvv -p 2222 vagrant@127.0.0.1
OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "127.0.0.1" port 2222
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to 127.0.0.1 [127.0.0.1] port 2222.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /root/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.2
ssh_exchange_identification: read: Connection reset by peer

```

The same from host OS:

```
# ssh -p 1111 vagrant@127.0.0.1
ssh_exchange_identification: read: Connection reset by peer

```

I run container with the next parameters:

```
docker run --privileged -v /dev:/dev -p 1111:2222 vm:latest

```

I tried to run lightweight image like it was described here and everything started successfully.

Any suggestions/ideas regarding it?

Note: I start a VM in a docker container because I don't want to make garbage etc within my desktop OS and I don't have a hardware where I could test it. Therefore my approach is to use VM for this.

# Answer
It looks like issue with ssh timeout is related to nested visualization restrictions. Thus, despite of enabling visualization in BIOS VM in container will still not be provisioned. I found similar answer here.

Once I started Docker container on host OS, issue has gone away and VM successfully provisioned.
> 0 votes

---
Tags: docker, vagrant
---


# Question
Title: Autorun ElasticSearch with Docker Compose
All of the tutorials online say to manually start elasticsearch as a daemon with a docker command from the command line. Is there a way to auto run it like redis, postgres and other services do from the docker compose yaml file?

# Answer
https://github.com/elastic/stack-docker/blob/master/docker-compose.yml

This is an official docker-compose for running Elasticsearch, and will make a good starting point for whatever you're trying to achieve. This would be the key part:

```
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:${TAG}
    environment: ['http.host=0.0.0.0', 'transport.host=127.0.0.1', 'ELASTIC_PASSWORD=${ELASTIC_PASSWORD}']
    ports: ['127.0.0.1:9200:9200']
    networks: ['stack']

```

But I'd recommend cloning the entire repository locally and running `docker-compose up` to experiment.
> 1 votes

---
Tags: docker, docker-compose, elastic
---


# Question
Title: How to connect to external etcd server inside of docker container (python application)?
I have a `Cement` application that runs inside of a docker container. In this image I have to connect to etcd. Inside of my docker image I have config that for etcd is set to `localhost` that I know is **not** correct!

On my local machine etcd is up and running:

```
tcp4       0      0  localhost.2379         localhost.51949        ESTABLISHED

```

The question is how should I connect to `etcd` server which is outside of docker container? `etcd` is installed normally without using docker.

Output of my python application when I run it with `docker run -p 8084:8084 my_python_image`:

```
Traceback (most recent call last):
  File "my_app.py", line 64, in <module>
    app.run()
  File "/usr/local/lib/python2.7/site-packages/cement/core/foundation.py", line 882, in run
    return_val = self.controller._dispatch()
  File "/usr/local/lib/python2.7/site-packages/cement/core/controller.py", line 477, in _dispatch
    return func()
  File "my_app.py", line 45, in default
    module_rpc = rpc.Rpc('my_module', params=app.config.get_section_dict('etcd'))
  File "/usr/local/lib/python2.7/site-packages/rpc_server/rpc.py", line 31, in __init__
    self.etcd_client.write(self.etcd_nodes, None, dir=True)
  File "/usr/local/lib/python2.7/site-packages/etcd/client.py", line 500, in write
    response = self.api_execute(path, method, params=params)
  File "/usr/local/lib/python2.7/site-packages/etcd/client.py", line 893, in wrapper
    cause=e
etcd.EtcdConnectionFailed: Connection to etcd failed due to MaxRetryError(u"HTTPConnectionPool(host=u'localhost', port=2379): Max retries exceeded with url: /v2/keys/my_module/nodes (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fcc33eaa690>: Failed to establish a new connection: [Errno 111] Connection refused',))",)

```

# Answer
> 1 votes
One option is to use the `host` network driver. If you run the container with the host network driver, `localhost` will resolve and route to your local machine where etcd is running.

```
docker run --network=host --userns=host -p 8084:8084 my_python_image

```

When using the host network driver, the IP of the container will be the same as your local machine's IP address, giving it access to all of the services that are listening on it.

The `--userns` flag is required for kernels that have namespacing enabled and provides additional permissions to the container in order for it to use host networking.

# Answer
> 3 votes
The error you see in your container is:

> etcd.EtcdConnectionFailed: Connection to etcd failed due to MaxRetryError(u"HTTPConnectionPool(host=u'localhost', port=2379): Max retries exceeded with url: /v2/keys/my\_module/nodes (Caused by NewConnectionError(': Failed to establish a new connection: \[Errno 111\] Connection refused',))",)

If you take a close look there, you can see that it's trying to connect to `localhost`. Inside your container, `localhost` is means "the current container" (much like `localhost` on your host means "the current host"). If you want to connect to an `etcd` instance *outside* of the container, you'll need to use the ip address (or hostname) of another server.

If you're running (a) running Docker locally, (b) etcd on your host is listening on all addresses, and (c) there are no firewall rules that would otherwise prohibit the connection, then you can use the address of the Docker bridge associated with your container as the address of your host. This will be the default gateway as seen from inside the container. You can extract the address with something like:

```
ip route show | awk '$1 == "default" {print $3}'

```

---
Tags: docker, containers, python
---


title: Multiple gradle builds with docker

text:
# Question
Title: Multiple gradle builds with docker
I have a multi-branch setup on Jenkins for my Gradle project. I would like to be efficient in terms of performance and cost as well (money!). Meaning, if would like to build my project many times, for different branches, commits, in parallel. For that I'm thinking about having everything inside of containers, but I'm not sure if my containers would share the libraries resolved by gradle. Moreover, I don't want to resolve dependencies for every time I bring them up, it would be time consuming and network usage as well.

Or I'm thinking about deploying a simple vm with Ansible and run my builds from there, making this whole process simpler.

So I wonder which approach would is better? Anyone faced a similar issue? Does someone has any benchmark to share?

# Answer
> 3 votes
There is currently an issue open to share Gradle cache between containers:

https://github.com/gradle/gradle/issues/851

I think the best solution at the moment is to bake as many dependencies in to the Docker image as possible, and then use a volume to share between sequential builds, limiting it to one concurrent build per host. You can mount the cache on to the host with a docker-compose.yml like so:

```
version: '3'
services:
  gradle:
    image: gradle:4.0.1-jdk8-alpine  # I'd suggest using this as a base to build your own Gradle image with dependencies baked in
    working_dir: /usr/src/app
    volumes:
      - ./:/usr/src/app:Z
      - ./.gradle:/home/gradle/.gradle:Z

```

Alternatively, another solution would be to look at network-level caching e.g. Squid.

---
Tags: docker, ansible, build-pipeline, gradle
---


# Question
Title: What are best practices to override complete sections using docker-compose?
The docker-compose.override contains:

```
volumes:
  - ./a:/path/to/a

```

The docker-compose:

```
volumes:
  - a:/path/to/a

```

When docker-compose up is run the override overwrite the `./a` overwrites the `a`.

When a new volume is added to docker-compose, the override will try to mount is as well.

docker-compose.override

```
volumes:
  - ./a:/path/to/a

```

The docker-compose:

```
volumes:
  - a:/path/to/a
  - b:/path/to/b

```

How to prevent that the override will use volumes that are defined in the docker-compose file?

When docker-compose.override contains

```
volumes:
  - a:/path/to/a
  - .b:/path/to/b

```

the b is still mounted...

# Answer
> 5 votes
According to this documentation it does not seem to be possible to override complete sections in docker-compose by using different docker-compose files.

> If both files are present on the same directory level, Compose combines the two files into a single configuration.
>
> The configuration in the docker-compose.override.yml file is applied over and in addition to the values in the docker-compose.yml file.

---
Tags: docker-compose
---


# Question
Title: How to access local Docker storage on Windows 10?
Where does Docker daemon store image and container data on the local disk ?

# Answer
Most of the persisted data is written to: `C:\ProgramData\docker`

you should have directories here such as:

```
.\windowsfilter
.\volumes
.\config
.\image

```

etc

The layer and image data is stored under these.
> 1 votes

---
Tags: docker, windows
---


# Question
Title: Can we consider usage of git lfs for artifacts equally mature strategy to an artifactory?
* Git lfs is just support for large files in Git -\> good for example for a video file sample if you need it, right?
* But, if you have large artifacts like a trained model or an uber-jar or some other artifact, what is the benefit to version this binary in git?

# Answer
**What is Git-LFS?**

Git Large File Storage (LFS) prevents that large files need to be stored in git itself.

https://git-lfs.github.com/

> Git Large File Storage (LFS) replaces large files such as audio samples, videos, datasets, and graphics with text pointers inside Git, while storing the file contents on a remote server like GitHub.com or GitHub Enterprise.

**What is artifactory?**

Artifactory stores artifacts, e.g. docker images, jars, wars

**Artifactory vs. Git-lfs**

Artifactory is used to store artifacts while git-lfs is used to prevent that large files are stored in git. If one will store large files in git and one would like to clone this repository it will take a decade before it is cloned.
> 1 votes

---
Tags: artifacts, methodology, artifactory, git-lfs
---