# Question
Title: Join a Linux Docker swarm with a local Windows Docker for test purposes
I'm currenlty trying to connect/join a Linux Docker Swarm cluster with my Windows machine in order to test a stack without sending the stack over and over to my test cluster under Linux. My problem is that I can't join the cluster with my Windows Docker.

My cluster had initially 1 manager and 4 workers. I recently created and added a new manager to it in order to add my Windows Docker node as manager. With this, I shouldn't have any issue with the manager quorum (since my two managers will be always up, and my Windows Docker could be interrupted at any time).

I configured my local firewall - and also on each nodes of my Linux cluster - to accept connections on ports (the following can be found at the end of this documentation) :

* TCP port 2377 for cluster management communications
* TCP and UDP port 7946 for communication among nodes
* UDP port 4789 for overlay network traffic

The advertising network in my cluster is `192.168.0.0/24`, one of my manager's IP address is `192.168.0.100` and my PC's IP address is `192.168.0.143` (same network). They can ping each others and I tested the ports defined before.

So, when I perform the `docker swarm join-token manager` on a manager I have something like this:

```
docker swarm join --token SWMTKN-1-27hzpcnek5... 192.168.0.200:2377

```

If I copy/paste it to my PC:

```
Error response from daemon: manager stopped: can't initialize raft node: rpc error: code = Unknown desc = could not connect to prospective new cluster member using its advertised address: rpc error: code = DeadlineExceeded desc = context deadline exceeded

```

But, even with this error, on my cluster, if I run `docker node ls`, my node appears in the list:

```
8380zb5jeke0elwohr7aieke8     linuxkit-00155d00d946   Ready               Active

```

But on my PC Docker thinks that it's not member of a cluster:

```
Error response from daemon: This node is not a swarm manager. Use "docker swarm init" or "docker swarm join" to connect this node to swarm and try again.

```

The Docker version is different in the cluster and on my PC:

* Cluster:
    + Client: `17.09.1-ce`
    + Server: `17.09.1-ce`
* PC:
    + Client: `18.03.1-ce`
    + Server: `18.03.1-ce`

Could the issue came from this?

N.B.: I work with Bash on Ubuntu on Windows, but the result is the same on PowerShell.

N.B. 2: I don't know if it's useful, but since Docker for Windows uses Hyper-V, I have a NAT interface: `10.0.75.0/24` (and the VM's IP address is `10.0.75.1`). I tried to use this interface as advertising address and listening address but the issue returned is the same

Does anybody already encounter this issue? Any ideas?

Thanks! :)

**EDIT 1:**

Disaggreeing with @BMitch's answer, after tests, I don't think there is any issue with the difference between managers' Docker versions.

As I said in my previous comment on BMitch's answer, I had to add the option `--advertise-addr 192.168.0.200` \- which is my leader manager address - to the docker join command. Until this, I used `--advertise-addr 192.168.0.143`, my PC address.

I hope it will help others.

**EDIT 2:**

If I run `docker info` on my PC I got this:

```
Swarm: active
 NodeID: o9wehjcyhe3n6xxzy1epnhxdp
 Error: rpc error: code = Unknown desc = The swarm does not have a leader. It's possible that too few managers are online. Make sure more than half of the managers are online.
 Is Manager: true
 Node Address: 192.168.0.200
 Manager Addresses:
  192.168.0.200:2377
  192.168.0.200:2377
  192.168.0.201:2377

```

There are two managers with address `192.168.0.200:2377`.

**EDIT 3:**

I finally gave up and installed a Linux server on another machine in order to join the cluster. That's not what I wanted but I can work now.

If any have information about this issue, I invite you to share it. Thanks :)

# Answer
> 2 votes
Why do you want to create a new Manager node, risking damaging your raft consensus and cie?

I find it way easier to simply expose the **docker socket** locally, as if you were working inside the node, but with your Windows environment.

To do so, simply open a **ssh tunnel** that exposes `/var/run/docker.sock`:

```
 ssh -M -S ~/.docker.sock \
     -fnNT -4 -L localhost:1337:/var/run/docker.sock \
     USER@MANAGER_IP

```

Refer to `man ssh` to see what all these options mean.

It will open an ssh tunnel; you will still be in your local environment shell, only thing left to do is set the proper `DOCKER_HOST` environment var, so your docker cli is bound to your swarm manager.

`export DOCKER_HOST=localhost:1337`

And that's it.

Note: Docker also includes an option to do this; I've quickly searched for it but can't manage to find it. It's in the most recent version of docker.

# Answer
> 0 votes
You cannot mix docker engine versions on the swarm managers. They need to each be running the same version, particularly when adding, removing, promoting, and demoting nodes. When you need to upgrade the engine on the managers, you can upgrade the engines in place, without promote/demote, but I would strive to minimize the time in this state and avoid using any newer features in the swarm.

Adding a second manager to a cluster is also not recommended. Should any issue occur, like the problem you're currently experiencing, you will lose quorum with only a single manager running. In effect, you've doubled the potential points of failure rather than creating an HA infrastructure. With quorum, you do need at least 3 nodes to be managers to have the ability to lose a single node. It's best to do this to an existing cluster by adding the additional managers first as workers, and then after you verify connectivity, promoting those nodes to managers.

---
Tags: docker, windows, linux, docker-swarm, networking
---


# Question
Title: passing args to shell module in Ansible adhoc command
Is it possible to pass `args`, to the shell module when running it as an adhoc command?

I can pass the `free_form` parameter using `-a`, but I am trying to figure out how to specify additional arguments, such as `chdir` or `executable`.

For example, what is the adhoc equivalent to this task?

```
- name: Change working dir to somedir/ and run somescript.sh using bash
  shell: somescript.sh >> somelog.txt
  args:
    chdir: somedir/
    executable: /bin/bash

```

# Answer
Append the parameters after the command. See the example below.

```
# ansible -m shell --args "pwd; touch test; ls -1 chdir=/scratch/tmp creates=/scratch/tmp/test" localhost
localhost | CHANGED | rc=0 >>
/scratch/tmp
test

# ansible -m shell --args "pwd; touch test; ls -1 chdir=/scratch/tmp creates=/scratch/tmp/test" localhost
localhost | SUCCESS | rc=0 >>
skipped, since /scratch/tmp/test exists

```
> 8 votes

---
Tags: ansible
---


# Question
Title: Overview of Gitlab CI/CD process for multi-project SOA application. Releasing and Versioning
I have a SOA-application. Each service is a separate project in Gitlab with its own Dockerfile. So, in terms of structure it looks as follows

```
\UserService
    Dockerfile
    \src

\OrderService
    Dockerfile
    \src

```

There is one developer (or group of developers) responsible for just one Gitlab project (or service as I call it). So, in fact each group technically can tag its project with its own version, so that `UserService` can have version `1.2.3`, while `OrderService` can have version `5.7.8`. It does not look good, because both services are part of one SOA-application.

So, in terms of DevOps I need know what is the best practice for versioning SOA-application containing physically separate services with their own code base?

My another question is about Docker. We are using docker for packing services. I know, that we can tag Docker images also with version number. In terms of best practices should this image version have something to do with versions of the source code of each single service?

Besides, I do not know what should we treat as a release and its version. I guess there will be a build pipeline in Gitlab, which will build multiple Docker images and tag each one with a version. Should we treat the collection of these images as a release or should there be some kind of a bundle, physical release with again its own version.

I know there may exist different practices, I want to know some real world approaches.

# Answer
> 1 votes
Managing versions of different services can be tricky. For now I've never found a good HOW TO out there. Like most things you do what suit you the best. Here is a couple of suggestion you could use:

-\> **Tag docker image with their commit sha**: Pretty easy to do when you are using gitlab this is an already set environment variable that you can uses when building you image in the CI/CD pipeline. It's easy to track what code this container. Also when rolling back one of your service you can see directly the merge request you are rolling back on.

-\> **Continuous deploy on your staging environment**: When one is merging a new feature on the master of one of your services deploy it as soon as possible so you can test the impact on all other services.

-\> **When staging looks good trigger a release job**: That job release shall be present on all your services. The purpose of this job is to:

* Check if the service current version (commit sha) match the currently deployed in production.
* If not trigger the deploy job to production for this commit sha pipeline.
* If the deploy to production job work then we shall store the current sha deployed in a new folder somewhere in a K/V store that way we keep a single source of truth of what's is being deployed and when

More on that K/V store and release job:

This shall be trigger form a different pipeline on a project you use only to remotely trigger the updated versions on your whole SOA-application. That way we have a repo that contain a script that will trigger the job release of all the services. It will create a new folder on your K/V store and all the triggered release job of your application will write their current version in it (updated or not) Once finished only service that had change will be deployed but you will have a major version for your wall application like this on the K/V store:

```
release_0.1.5:
 - user_service = COMMIT_SHA
 - payment_service = COMMIT_SHA
 - organisation_service = COMMIT_SHA
 - ect..

```

When this job is replay it will detect that the folder already exist and so will trigger the deploy of all the version present in it so you can rollback your App to an older state.

Deploy to production job on each of your service can be protected to only be triggered from the main release project. But for quick fix reason you could also let it open just make sure it update the current K/V folder.

---
Tags: docker, gitlab, cicd
---


# Question
Title: Unable to run playbooks using Ansible best practices layout
I am very new to ansible and I think I am lacking something basic that I am not finding in the examples or on github as to the directory structure and its use.

I have set up my directory structure using ansible best practices document located here:

https://docs.ansible.com/ansible/latest/user\_guide/playbooks\_best\_practices.html#directory-layout

But the documentation is not clear on how the structure should be used in practice. I have multiple clients in which the inventories must be kept seperate. My directory structure is set up as follows:

```
inventory/
   client1/
      group_vars/
        all
        credentials.yml <encrypted>
        router
        switch
        firewall
      host_vars/
        router1
        switch1
        firewall1
      hosts
   client2/
      group_vars/
        all
        credentials.yml <encrypted>
        router
        switch
        firewall
      host_vars/
        router1
        switch1
        firewall1
      hosts
roles/
  role1/
     tasks/
       main.yml
  role2/
     tasks/
       main.yml

```

The hosts files contain routers, switches and firewalls (grouped as such) like this:

```
[router]
router1 ansible_host=1.2.3.4

[switch]
switch1 ansible_host=5.6.7.8

[firewall]
firewall1 ansible_host=9.10.11.12

```

The group\_vars have router, switch and firewall files with the appropriate variables. There is a credentials.yml in the ./inventory/client name/group\_vars directory that is encrypted and should be loaded for every playbook. I understand precedence and I am not having any trouble there.

If I run ansible-playbook -i /path to hostfile/ how do I pick the specific host I want to run against?
`ansible-playbook -i inventory/client1/hosts myplaybook.yml` \<--where can I put the specific host that is in the hosts file? I have tried several different methods and none seem to work. My guess is that I am running the ansible-playbook command from the wrong directory.

If I run ansible-playbook in the correct inventory folder (./inventory/client1/) I can pick the host, but how do I point to the playbooks that are in the root folder without backing up directories each time like this:

```
ansible-playbook -i hosttarget ../../myplaybook.yml

```

It seems poor form to have to back out of the directory each time just to run a playbook. If I just put the playbook name, it does not find it so I am thinking that it was not the intention to run ansible-playbook in this way either.

How can I load additional credentials files or group variables? It would be nice if there was a way to pull this from the directory that contains the inventory but I can't figure that out. I was thinking of possibly loading extra\_vars in the playbook and setting a value for a variable called "client" in the "all" file, but it seems kludgy and prone to error. I was hoping that there was some type of automatic pull that can be done like the loading of the group\_vars and host\_vars but maybe I am missing something.

```
cat ./inventory/client1/all
client=client1

extra_vars=~/inventory/{{ client }}/credentials.yml

```

My thought is that since the inventory layout is shown as Ansible best practices, then maybe I am not fully understanding of how I should be running playbooks.

It would seem to me that I should be able to run ansible-playbook in the root directory that contains all my plays and point it to an inventory file easily. The playbooks should be reusable across all clients with no changes required and there should be an easy way to load variable files without having to specify them each time at the command line. Ansible already does this very well with group\_vars and host\_vars so I am thinking that I am just missing something simple.

I have a workaround for this now using the following command line: ansible-playbook -i inventory/atcog ecats\_l2l.yml --extra-vars @inventory/atcog/group\_vars/credentials.yml

This run a configuration playbook that runs a task that builds a VPN. This VPN is built for a client called atcog. I have to add this very long --extra-vars @inventory/atcog/group\_vars/credentials.yml to access the credentials file for the client "atcog". I also have to manually modify the playbook to have it only run the play on a single host by adding - hosts: atcogfirewall-1. It seems like there should be a better way.

So let's say I now want to push this VPN to another client. I now have to edit the playbook and change - hosts: krifirewall-6, then I need to send this command to make sure the right credential file is used: ansible-playbook -i inventory/kri ecats\_l2l.yml --extra-vars @inventory/kri/group\_vars/credentials.yml

Seems like a lot of places where a typo or forgetting to update a playbook host entry can cause this to run on the wrong device or run with the wrong credentials.

So if I add something like extra\_vars=~/inventory/{{ client }}/credentials.yml in the playbook I can likely mitigate the need to put the --extra-vars in the command line but I can't figure out a way to resolve the whole choosing the individual host issue without manually updating the playbook each time which is just asking for human error.

Here is a screenshot of the layout. The playbook I am running is in the root directory but is not shown as I could not capture it all on 1 screen.

# Answer
> 3 votes
Generally, Ansible is very flexible and often there are more "correct" solutions.

Start for example with the question "Where do the configuration data come from?", put the default data to the roles and decide which variables should be configured in "*group\_vars/host\_vars*", in the roles, and which in the playbooks. Review Variable precedence: Where should I put a variable?. To set paths to the inventory, roles, and others see Ansible Configuration Settings.

To answer your questions:

> Q1: If I run ansible-playbook -i /path to hostfile/ how do I pick the specific host I want to run against? ...where can I put the specific host that is in the hosts file?

To pick for example "*router1*" start a play with:

```
- hosts: router1

```

> Q2: How do I point to the playbooks that are in the root folder without backing up directories each time like this: "ansible-playbook -i hosttarget ../../myplaybook.yml"

Configure DEFAULT\_HOST\_LIST. For example:

```
# ANSIBLE_HOSTS=$PWD/inventory/client1/hosts ansible-playbook myplaybook.yml

```

> Q3: How can I load additional credentials files or group variables?

There are a lot of options described in Variable precedence: Where should I put a variable?. For example "*include\_vars*" might be useful to load additional credentials. For "group variables" pick from the options.

---
Tags: ansible, yaml, ansible-inventory
---


title: How Do I Log Task Output To A File?

text:
# Question
Title: How Do I Log Task Output To A File?
One of my ansible tasks imports an Oracle database using `impdp`.

This generates a *lot* of output to the console so I have set `no_log: True`.

However, when this fails I want to see the log!

How can I make this particular task log to a file and not to the console?

# Answer
> 4 votes
I think that all you need to do is to register the output of every command you need (store it in a variable) and then simply dump the variable in a file. That way you can review it later.

```
tasks:
  - name: Dump all vars
    action: template src=templates/dumpall.j2 dest=/tmp/ansible.all

```

Then in dumpall.j2:

```
Module Variables ("vars"):
--------------------------------
{{ vars | to_nice_json }}

Environment Variables ("environment"):
--------------------------------
{{ environment | to_nice_json }}

GROUP NAMES Variables ("group_names"):
--------------------------------
{{ group_names | to_nice_json }}

GROUPS Variables ("groups"):
--------------------------------
{{ groups | to_nice_json }}

HOST Variables ("hostvars"):
--------------------------------
{{ hostvars | to_nice_json }}

```

The example I'm using is from here

# Answer
> 4 votes
\[Converting my comment to an answer\]

One way to do it would be to write the logs to some external file, and then having a task after it which makes use of failed\_when condition, and remove the log file, if the previous task was successful.

Something like this should help you.

```
 - name: Run Py script
      command: <>.py  > <>.log
      become: yes
      register: PyScript
      ignore_errors: True

    - name: PyScript on success
      command: rm <>.log
      when: PyScript|succeeded

```

Note: This might not be the best way to handle your problem. But, this was a hack which helped me do my logging and monitoring.

# Answer
> 3 votes
I solved this by adding

```
ignore_errors: true
register: results

```

to the no\_log-task. This makes ansible continue to the next task, even when the task fails. Then for the next task define a debug task, which always fails and outputs the registered variable, but only runs when the previous task failed:

```
- name: Error output
  debug:
     var: results
  failed_when: true
  when:
     results is failed

```

So even with no\_log: true, this will make ansible display the output of the failed task. This solution is not logging it to a file as requested but fullfils your need to 'see the log when failed' , and of course, you can redirect or use tee to output the full ansible output to a file, which will, with this solution also contain the the log of the failed task.

# Answer
> 2 votes
What I do when I've a command to execute and wish to get the log only in case of failure is as follow (prefixed by a shell comamnd like `/bin/sh -c '...'` in case the initiator doesn't use a `system` call or execute the command directly without shell):

```
command 2>&1 > command-log.txt || cat command-log.txt

```

This will redirect error and standard output to a file and display the content of the file in case of failure only. If the command is very verbose and you don't wish to keep the log when it's ok you can go with:

```
command 2>&1 > command-log.txt && rm command-log.txt || cat command-log.txt

```

Quote for `&&` and `||` usage from sh manpage:

> The symbol && (||) causes the list following to be executed only if the preceding pipeline returns a zero (non zero) value.

That's probably not the most idiomatic way to do it with ansible but has the advantage of being very portable with any configuration management system giving the ability to display command stdout.

# Answer
> 0 votes
Assuming ansible properly throws errors to stderr you can capture the output of errors in any program to a file using output redirection:

`some command 2> error.log`

However I don't think this is the case.

Instead you will likely want to refer to this guide to decide when errors will occur http://docs.ansible.com/ansible/playbooks\_error\_handling.html and then grep your output for strings that indicate an error before outputting to a file

ie.

`ansible-playbook my-playbook | grep 'error' > error.log`

# Answer
> -2 votes
Tee will be a very simple tool for logging, you can refer the following command.

```
eric@eric-MacBookPro:~$ ansible -m ping all | tee > /tmp/ansible.log
eric@eric-MacBookPro:~$ cat /tmp/ansible.log
localhost | SUCCESS => {
    "changed": false,
    "ping": "pong"
}

```

# Answer
> -2 votes
I think what you're looking for might be too simply redirect stdout and street to file.

Typically , some-command &\> logfile.log

or some variant....

---
Tags: ansible, logging
---


# Question
Title: Configuration Management Tools and Rollbacks / Approaching Immutable Infrastructure
I am interested in whether there are any configuration management tools out there that make it easy to roll back.

So far it seems that with Ansible, Chef, and Puppet you have to write the roll back tasks/recipes/whatever to get back to your previous state.

Coming from a bit of personal interest and experience with NixOS I have seen that with pure Immutable Infrastructure rollback is easy.

I am in Enterprise environments where NixOS is not happening. How can Enterprises begin to approach Rollbacks better with tools that work with Windows and Enterprise Linux Distros?

# Answer
You hint in the title about immutable infrastructure, so it sounds like you already know the solution: don't change existing servers, but bring up new ones with your changes, switch over to them, and switch back to the old ones if necessary.

Theoretically any of the standard config management tools can roll back by simply checking out a previous commit and rerunning the config. However, practically speaking this often isn't the case, as you may have introduced a new config rule (and so whatever it is doing is simply ignored in older versions) or you've written a rule that isn't easily reversible (for instance, does some destructive data operation). If you were **very** careful in how you always write your playbooks, you could avoid this, but a few less error-prone solution is just to use an immutable infrastructure approach.
> 3 votes

---
Tags: ansible, infrastructure-as-code, immutable-servers, nixos
---


# Question
Title: How do I troubleshoot missing data in my Prometheus database?
I have been gradually integrating Prometheus into my monitoring workflows, in order to gather detailed metrics about running infrastructure.

During this, I have noticed that I often run into a peculiar issue: sometimes an exporter that Prometheus is supposed to pull data from becomes unresponsive. Maybe because of a network misconfiguration - it is no longer accessible - or just because the exporter crashed.

Whatever the reason it may be, I find that some of the data I expect to see in Prometheus is missing and there is nothing in the series for a certain time period. Sometimes, one exporter failing (timing out?) also seems to cause others to fail (first timeout pushed entire job above top-level timeout? just speculating).

All I see is a gap in the series, like shown in the above visualization. There is nothing in the log when this happens. Prometheus self-metrics also seem fairly barren. I have just had to resort to manually trying to replicate what Prometheus is doing and seeing where it breaks. This is irksome. There must be a better way! While I do not need realtime alerts, I at least want to be able to see that an exporter failed to deliver data. Even a boolean "hey check your data" flag would be a start.

How do I obtain meaningful information about Prometheus failing to obtain data from exporters? How do I understand why gaps exist without having to perform a manual simulation of Prometheus data gathering? What are the sensible practices in this regard, perhaps even when extended to monitoring data collections in general, beyond Prometheus?

# Answer
> 5 votes
I think your can do some kind of alerting on a metric `rate` with something like this:

```
ALERT DropInMetricsFromExporter
  IF rate(<metric_name>[1m]) == 0
  FOR 3m
  ANNOTATIONS {
    summary = "Rate of metrics is 0 {{ $labels.<your_label> }}",
    description = "Rate of metric dropped, actually: {{ $value }}%",
}

```

The main idea is to alert whenever the metric rate is at 0 for 3 minutes, with the proper metric name and a label somewhere telling from which exporter it comes it should give you the correct information.

Choosing the right metric to monitor by exporter could be complex, without more insight is hard to give a better advice out of vacuum.
This blog post could be an inspiration also for a more generic detection.

# Answer
> 1 votes
There are a few reasons which might have caused the gap. Most likely the exporter isn't reachable in which case the `up` timeseries will be 0. You can alert on this like this (taken from https://prometheus.io/docs/alerting/rules/#templating):

```
# Alert for any instance that is unreachable for >5 minutes.
ALERT InstanceDown
  IF up == 0
  FOR 5m
  LABELS { severity = "page" }
  ANNOTATIONS {
    summary = "Instance {{ $labels.instance }} down",
    description = "{{ $labels.instance }} of job {{ $labels.job }} has been down for more than 5 minutes.",
  }

```

On the status page you should also see that it's down including an error message. Unfortunately there is no way to see past error but there is an issue to track this: https://github.com/prometheus/prometheus/issues/2820

Your Prometheus server can be also overloaded causing scraping to stop which too would explain the gaps. In that case you should see `Storage needs throttling. Scrapes and rule evaluations will be skipped.` errors in the log and increases in the `prometheus_target_skipped_scrapes_total` metrics. You should alert on that too, e.g:

```
ALERT PrometheusSkipsScrapes
  IF rate(prometheus_target_skipped_scrapes_total[2m]) > 0
  FOR 5m

```

# Answer
> 1 votes
I'm not sure if this is the same problem you are seeing. But we saw these random gaps for Java applications that did not specifically set the GC settings. Meaning we were seeing gaps when the data collection stopped because the activity for the instance stopped while the JVM was doing a Full GC. If you are using Java this can happen. Our fix was to explicitly use the G1 garbage collector (Java 8+) with a specified limit on the length of GC activity to prevent these time gaps in the data collection.

---
Tags: monitoring, prometheus
---


# Question
Title: Jenkins input pipeline step - Give submitter as a group
I could find that “submitter” parameter accepts individual user IDs and external LDAP groups. But “Jenkins-local” groups defined by any authorization strategy are not working. I have a limitation not to use external LDAP groups.

The workaround that I have used is define the comma separated list of individual users as a Jenkins global variable and manage the list from there. This gives a better maintainability. However, it mandates that the stage is wrapped within node. As the pipeline waits for the approver to approve this step, it holds a node instance and this causes the rest of the jobs to wait for available instances. Is there a way to use a Jenkins global variable outside of node for the input step?

```
node{
stage ('Approve?')
{
    timeout(time:2, unit:'DAYS')
    {
        input message: 'Can you approve this ??', submitter: MY_APPROVERS
    }
}

```

}

# Answer
> 1 votes
We have moved onto using Jenkins templates and I have a workaround to keep the individual approver names list in one place. Sharing the sample code for your reference.

```
#!/usr/bin/env groovy
import com.mm.Constants
def call(String teamName = 'LOGIN') {
   // team name of null means UI
   teamName = teamName ?: 'UI'

   timeout(time:2, unit:'DAYS')
       {
           input message: 'Can you approve this ??', submitter: Constants."${teamName}_APPROVERS"
      }

```

}

```
#!/usr/bin/env groovy
class Constants {
   static final LOGIN_APPROVERS = 'approver1,approver2';
   static final UI_APPROVERS = 'approver2,approver3';
}

```

Below is how you call from the pipeline

```
stage ('Ship to QA?'){
    echo "Waiting for QA approval"
    shipToQA('LOGIN')
}

```

---
Tags: jenkins
---


title: Persisting data in docker context

text:
# Question
Title: Persisting data in docker context
I am new to docker and flask web framework, my docker context has a flask file, a mod\_wsgi file, and the python script. As I am new to flask, there are a lot of small changes which I keep doing. For any change to the docker context to take into effect I have to:

1. Kill the container and prune it out of the system.
2. Make changes to the Dockerfile and then rebuild the container.
3. In some cases I have to even remove the image and rebuild it as the changes are not taken into effect if cache is retained.

Is there any way in which I can have the changes into the docker context taken into effect during run time. I tried to have the docker context shared as a volume as that is the preferred way to persist data in Docker containers and services, but that did not work, any changes still were not reflected.

Could you please let me know of any way I can achieve this. Any suggestion would be greatly appreciated.

Thanks and regards, Shailesh

# Answer
> 1 votes
Mounted volumes can be used to preserve state between runs, apart from that Docker is stateless by design. To make a local directory available inside the Docker container, you can use the `-v` option, for example:

`docker run -v /path/to/local/dir/:/mountpath/inside/docker ...`

But if I understand your question correctly, you are asking for a faster feedback cycle while working on the Dockerfile. For that, I would recommend to start the container, and attach an interactive shell to try out your changes. Once you made some progress, update the Dockerfile.

```
$ docker ps
CONTAINER ID  IMAGE        COMMAND  CREATED           STATUS        PORTS  NAMES
f1217635695d  ubuntu:18.04 "bash"   23 seconds ago   Up 21 seconds         stoic_austin

$ docker exec -it f1217635695d bash
root@f1217635695d:/# ...

```

Also note that proper use of caching can speed up rebuilds of the Docker container after changes significantly.

> (from your question) 1. Kill the container and prune it out of the system.

Pruning should not be necessary unless you run out of disk space. It should be sufficient to update the tag, for example:

```
docker build -t test .
docker run --rm -it test

```

`test:latest` will point to the last image id that was created. `docker images` will show that the previous images are not deleted, but as said, if you have enough disk space available, it is not a big concern.

---
Tags: docker
---


# Question
Title: How do I get the hosts from a specific section of an Ansible `hosts` file in a Jinja2 Template
I have a `hosts` file including the following contents

```
[elasticsearch_nodes]
host1 os=linux
host2 os=linux
host3 os=linux

```

In a `.j2` template I want to get the the list of `[elasticsearch_nodes]`

```
{{ some_ansible_magic_here }}

# should output "host1","host2","host3"

```

How do I do this?

I do have this as well to help with the quotes and the commas when I have the list.

```
{% for host in some_sort_of_ansible_magic_to_get_me_that_list_i_need %}
 "{{ host }}"{% if not loop.last %},{% endif %}
{% endfor %}

```

# Answer
> 2 votes
The magic you need is magic variables, in particular the `groups` variable, which is

> a dictionary/map with all the groups in inventory and each group has the list of hosts that belong to it.

Then you can use template filters to format the list: use `map` with `regex_replace` to add the quotes to all list elements and then `join` them into the final result.

```
{{ groups['elasticsearch_nodes']|map('regex_replace', '(.*)', '"\\1"')|join(',') }}

```

---
Tags: ansible, jinja
---


# Question
Title: Installing azcopy in Gitlab Runner (Alpine Linux)
How can I install azcopy and use it in my gitlab runner CI/CD script?

First I have two scripts that download the tar file and extract it.

However, if I try running executable `./azcopy_linux_amd64_10.1.0/azcopy` I get an error saying `/bin/sh: eval: line 102: ./azcopy_linux_amd64_10.1.0/azcopy: not found` I know this directory/file exists because I've ran `ls` commands.

I've also tried copying the binary to `/usr/bin` and/or `/usr/local/bin` but I still get the error: `/bin/sh: eval: line 102: azcopy: not found`

What am I doing wrong? How can I extract and run the executable, or otherwise install azcopy in my gitlab CI/CD script

# Answer
> 11 votes
This is a glibc dependency issue on Alpine.

This `azcopy` binary is compatible with Linux distributions that use glibc, GNU's C standard library. Alpine Linux uses a different libc implementation, musl-libc, which is generally not compatible to glibc. When trying to run non-Alpine-built binaries on Alpine, they'll usually fail to link since the glibc shared object, `libc.so.6`, is missing. `azcopy: not found` is actually the result of the dynamic linker failing to resolve the requires libraries.

You could verify by running `ldd ./azcopy`:

```
    /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)
    libpthread.so.0 => /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)
    libc.so.6 => /lib64/ld-linux-x86-64.so.2 (0x7fa0bc9c3000)

```

Note the `libc.so.6` dependency. On vanilla Alpine, there's only the musl libc binary, `libc.musl-x86_64.so.1`.

Fortunately, Alpine provides a light compatibility layer for glibc, through the libc6-compat package. It installs a `libc.so.6` link which is redirected to the musl shared object, and adds some missing glibc interface functions. This seems to be enough to running azcopy.

Therefore, for running azcopy on Alpine, just install the `libc6-compat` package, using:

```
apk add libc6-compat

```

# Answer
> -1 votes
After doing the extract you should set it executable with `chmod +x azcopy`

You can then run it with a `azcopy --help`

---
Tags: docker, gitlab-ci-runner, alpine
---


# Question
Title: Which system user should I use to run Postgres?
I am using a postgres docker image configured with the following environment:

```
    image: healthcheck/postgres:alpine
    environment:
      POSTGRES_USER: postgres

```

I have the following questions regarding this setup:

* What is the default user of postgresql?
* How does hardcoding the postgres user improve security?
* What special rights does the postgres user have?
* What is the upside and downside of using an unprivileged user ?

# Answer
> What is the default user of postgresql?

postgres

> How does hardcoding the postgres user improve security?

It's not using the root user.

You c*an* use any account you like, but consistency is a Wonderful Thing, especially when you have lots \[and lots\] of Postgres clusters to manage.

> What special rights does the postgres user have?

Not many.

It installs itself a hole through the local firewall but, otherwise, it's just another user.

> What is the upside and downside of using an unprivileged user ?

You should **never** run applications (or databases) as root (unless they *actually* need to be).

*Always* keep the biggest and best tools in the toolbox for *yourself*, ready to sort out the mess that Applications (or Users) make. If you run stuff as root and it decides to do something really stupid to your machine, *you* may not be able to fix it.
> 1 votes

---
Tags: postgresql
---


title: Using password protected ssh keys

text:
# Question
Title: Using password protected ssh keys
If you use password-protected SSH keys, is there a way to pass the password at the command line automatically instead of manually typing it?

E.g. something like:

`ssh-add <key> -p password`

# Answer
> 1 votes
First you need to install sshpass.

* Ubuntu/Debian: `apt-get install sshpass`
* Fedora/CentOS: `yum install sshpass`

---

**Example:**

```
sshpass -p "YOUR_PASSWORD" ssh -o StrictHostKeyChecking=no YOUR_USERNAME@SOME_SITE.COM

```

**Custom port example:**

```
sshpass -p "YOUR_PASSWORD" ssh -o StrictHostKeyChecking=no YOUR_USERNAME@SOME_SITE.COM:2400

```

**Notes:**

* `sshpass` can also read a password from a file when the `-f` flag is passed.
    + Using `-f` prevents the password from being visible if the `ps` command is executed.
    + The file that the password is stored in should have secure permissions.

# Answer
> 0 votes
But storeing your password in a command is not the best way - it should be better to remove the password for this case - so you do not reveal your key password

```
$ ssh-keygen -p [-P old_passphrase] [-N new_passphrase] [-f keyfile]

```

---
Tags: jenkins
---


# Question
Title: How do you monitor status and view logs for multiple distributed java applications at a centralized place?
We have multiple java applications running on distributed hosts in our system. As theses applications are processing huge amount of data, it is designed this way. However, it had been a challenge in getting the status of these applications, view logs etc., across the different hosts. All these applications uses log4j and is logging to a file in the hosted machine. Is there a open source, proven solution that can be used to get status and view logs for distributed java applications at a centralized place?

# Answer
> 1 votes
You can use Glowroot

Glowroot is Open source Java Application Monitor. Very easy to use and Very low overhead.

Steps to run:

1. Download and unzip glowroot-0.13.4-dist.zip
2. add `-javaagent:path/to/glowroot.jar` to your application's JVM args
3. Go to http://localhost:4000
4. and monitor all this

# Answer
> 0 votes
You could use graylog with the GELF logformat from your apps as central logserver, we do this with highly distributed systems. So you would be able to get errors and troughput metrics as graphs to. Or if you need more insights pinpoint

---
Tags: java, distributed-systems
---


# Question
Title: Testing R package on Travis fails because of TeXLive incompatible versions
I am writing an R package and using Travis for Continuous Integration. The tests on macOS fail because the system has a newer version of TeXLive and is unable to install the `inconsolata` font to build vignettes.

The Travis logs show:

```
sudo tlmgr install inconsolata upquote courier courier-scaled helvetic
tlmgr: The TeX Live versions supported by the repository
http://mirrors.concertpass.com/tex-archive/systems/texlive/tlnet
  (2016--2018)
do not include the version of the local installation
  (2019).

```

and further on, R is unable to build the vignettes:

```
* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
LaTeX errors found:
! LaTeX Error: File `inconsolata.sty' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)
! Emergency stop.
<read *>

l.276 ^^M

!  ==> Fatal error occurred, no output PDF file produced!
* checking PDF version of manual without hyperrefs or index ... ERROR
* DONE
Status: 1 ERROR, 1 WARNING

```

This failure throws a warning for R, which is upgraded to an error because I have `warnings_are_errors: true` in `.travis.yml`.

I tried this StackOverflow answer of reinstalling `inconsolata` in `.travis.yml`:

```
before_install:
  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
      brew cask install basictex;
      sudo tlmgr update --self;
      sudo tlmgr update --all;
      sudo tlmgr install titling framed inconsolata;
      sudo tlmgr install collection-fontsrecommended;
    fi

```

and this TexStackExchange thread of changing the TeX repository and running the same commands as Travis does when installing the system:

```
before_install:
  - if [[ "$TRAVIS_OS_NAME" == "osx" ]]; then
      sudo tlmgr option repository http://mirror.ctan.org/systems/texlive/tlnet;
      sudo tlmgr update --self
      sudo tlmgr install inconsolata upquote courier courier-scaled helvetic;
    fi

```

and neither worked.

How can I install the `inconsolata` font?

# Answer
> 1 votes
This issue is due to the CTAN mirrors having yet to update. The issue appears in the Travis R community forum:

> might be related to the recent switch to TexLive-2019 as the LaTeX package manager seems to fail finding inconsolata in the repository.
>
> ...
>
> It looks to me like the mirror chosen was has not updated to 2019, so I think this will resolve itself naturally when the mirrors catch up. If not I can look into it.

In the meantime, I solved it by forcing a previous version of TeXLive with a downgrade to OSX 10.12 in `.travis.yml`:

```
# macOS 10.12 version
osx_image: xcode9.2

```

See this Travis page for the correspondence between XCode and OSX versions.

---
Tags: travis-ci, macos
---


# Question
Title: Travis tests for R package fail with little explanation
I am writing an R package and using Travis for Continuous Integration. Tests and `R CMD CHECK` pass locally and for 3 versions of R under Linux (oldrel, release, and devel). All three versions fail under macOS with no information in the Travis logs:

```
...
R CMD check fail logs
0.01s$ for name in $(find "${RCHECK_DIR}" -type f -name "*fail");do echo ">>> Filename: ${name} <<<";cat ${name};done
log
R CMD check log logs
...

```

I checked in the Travis logs that all the tests I wrote in `testthat` fail. The Travis tests once passed for both Linux and macOS, so I could dissect the code changes to detect the issue, but ideally I could use something more direct and faster than groping in the dark waiting for an answer from Travis.

Any ideas?

# Answer
If you are using the recommended setting for R packages of upgrading warnings to errors, i.e. with this line in `.travis.yml`:

```
warnings_are_errors: true

```

then the place to look is the previous heading in the Travis log, called `Checking package`.

### Example of problem with missing font

In my case, the Travis log under the heading `Checking package` showed that the `inconsolata` font was missing, so the vignettes failed to build:

```
* checking PDF version of manual ... WARNING
LaTeX errors when creating PDF version.
This typically indicates Rd problems.
LaTeX errors found:
! LaTeX Error: File `inconsolata.sty' not found.
Type X to quit or <RETURN> to proceed,
or enter new name. (Default extension: sty)
! Emergency stop.
<read *>

l.276 ^^M

!  ==> Fatal error occurred, no output PDF file produced!
* checking PDF version of manual without hyperrefs or index ... ERROR
* DONE
Status: 1 ERROR, 1 WARNING

```

This caused an error for LaTeX, a warning for R, and an error for Travis.

### Solution to problem

In my case, the problem was due to Travis using TeXLive 2019 and the CTAN mirrors having yet to update to 2019.

The issue appears in the Travis R community forum:

> might be related to the recent switch to TexLive-2019 as the LaTeX package manager seems to fail finding inconsolata in the repository.
>
> ...
>
> It looks to me like the mirror chosen was has not updated to 2019, so I think this will resolve itself naturally when the mirrors catch up. If not I can look into it.

In the meantime, I solved it by forcing a previous version of TeXLive with a downgrade to OSX 10.12 in `.travis.yml`:

```
# macOS 10.12 version
osx_image: xcode9.2

```

See this Travis page for the correspondence between XCode and OSX versions and the original thread in DevOps StackExchange.
> 1 votes

---
Tags: travis-ci
---


# Question
Title: How to manage Docker images and their child images in a pipeline?
I have various Jenkins agents images and I am trying to add automation to it.

I have a base java image called `agent-base` and those who inherit from that one e.g. `agent-ansible`, `agent-terraform` and so on. In some case, I even have a 3rd level.

My questions:

* What would be the best way to update the child images when a parent is updated? Downstream pipelines perhaps?
* Or should I keep all the Dockerfiles in a single repository and have a script that know the images hierarchy?
* More importantly, how should I tag (version) my child images? Should it contain my parents version too? In the tag itself or as labels.

# Answer
There is a good practice called multistage build, in which you have one Dockerfile containing various image definitions that depend from one another, this helps you keep all your images updated with the latest changes and copy artifacts from one to another; this also helps reducing the image size if done right. A simple example is like the following:

```
FROM golang:1.9.2 as builder
ENV CGO_ENABLED=0 GOOS=linux
RUN go build -a \
             -installsuffix cgo \
             -o app .

FROM alpine:3.6
WORKDIR /root/
COPY --from=builder app .
CMD ["./app"]

```

In this example, in the first image you compile a go example application using the golang official image and then, using alpine to keep the size of the image small, you copy the executable to a new image and execute it. This way you can keep your containers in one file and build all at the same time or you can build just a target stage:

```
docker build --target builder -t somehub/sometag:latest .

```

You can even use remote images as stages:

```
COPY --from=nginx:latest /etc/nginx/nginx.conf /nginx.conf

```

If you need anymore information regarding Docker multi-stage builds you can find it in the following link.
> 1 votes

# Answer
It's not good idea to store multiple Dockerfiles inside one repository. The best way is 1 repo 1 Dockerfile.
> 0 votes

---
Tags: docker, jenkins, jenkins-pipeline
---


# Question
Title: How to handle AWS Offsite or cross-account RDS (mysql) backups?
AWS Provides a great solution to provide point-in-time recovery backups within RDS. However, while we take every measure to protect the root account, I'm paranoid that if someone were to gain access they can wipe the entire AWS account including all backups and therefore take the whole company down. What are the best ways to handle this? Do you have 100% trust in your root account and policies to prevent someone malicious from wiping the database and all backups?

What are the best ways to handle an offsite backup to a different RDS account? Should we simply execute `mysqldump` periodically from a separate system, or is there a better way?

# Answer
> 2 votes
On top of protecting your root account as best as you can, you should be able to reduce the risk of such resource deletion if you have more than 1 resource to protect.

For example if all AWS resources you had to protect were 2 RDS database servers, you could create 2 other accounts that have access to only copy the snapshots of one of your database servers each - and no other access to any other resource. This setup would then require an intruder to hack both root accounts of your main and one backup account each, which should be significantly more difficult.

To copy snapshots from one account A (your current RDS account) to account B (your new account) you should need the following parts (untested)

* Create a new IAM role in Account A, for access from account B:

* Create a new policy for reading RDS snapshots from account A

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
        "Effect": "Allow",
        "Action":
            [
                "rds:CopyDbSnapshot",
                "rds:DescribeDbSnapshots"

            ],
        "Resource": "arn:aws:rds:::"
        }
   ]
}

```

and assign it to the IAM role

# Answer
> 0 votes
In terms of your root account, you should be very paranoid about the power it wields and protect it as thoroughly as possible.

I would suggest that you get yourself a Hardware Key Fob as listed in MFA devices on this page: https://aws.amazon.com/iam/details/mfa/

Having a physical factor means that you can be sure only people with access to the physical device could log in and do that kind of damage. You can even secure it in a physical safe.

If your root account is considerably secure, I would be of the opinion that is Good Enough for DB back-ups. If you're worried about those then s3 or other amazon services will have the same problems. While you could look at other storage options, realistically if your aws root account is compromised it might just be the end of days - I would focus on the security aspect personally.

---
Tags: amazon-web-services, backup, mysql, amazon-rds
---


# Question
Title: How do -dev apk packages differ on Alpine for the purposes of creating docker images?
I recently started a new position at a company that is transitioning over to docker for local and prod, and inherited a few 'base image' repositories for php, node and a few others.

Unfortunately the creator is no longer at the company, so I'm going through the system and trying to piece it together. None of the images are used in production yet, and it's fair to say they are probably not road ready yet.

The images are based off alpine and I noticed particularly in the php image that most of the apk dependencies are using `-dev` distros:

```
apk add --no-cache $PHPIZE_DEPS bzip2-dev freetype-dev gmp-dev icu-dev ...

```

I couldn't easily find out the difference between, for example, `bzip2` and `bzip2-dev` but my assumption would be that these packages are aimed at people who develop those packages. Next best guess would be they have some advantage for general developers (php in this example) but if so I'm not sure what those benefits would be.

What is the purpose of the `-dev` packages, and perhaps some guidance on where they would be appropriate (e.g. local only or fine for prod?)

# Answer
`dev` packages are typically intended for development and build purposes. They usually contain header files and other resources for providing programmer APIs. For example, `python-dev` contains C headers for C/C++ applications that integrate python functionalities (which of course isn't necessarily python itself).

The "non-dev" packages provide the actual application or library: for example, `python` provides the python interpreter binary, the `libpython3.so` shared library, and compiled python modules.

For inspecting the contents of specific packages, you could use Alpine's excellent package explorer: locate the package of interest (for example, `python3-dev`), and click the "Contents of package " link at the bottom, to see the complete list of files installed by the package.
> 1 votes

---
Tags: docker, php, alpine
---


# Question
Title: Error with declaring volumes in Docker-compose, 'no such file or directory, open '/app/package.json'
I am a very beginner in the Docker world, while I am trying to experiment with Docker file and Docker-Compose I am facing the below error, I now am able to understand it is related to declaring volumes in Docker compose, if I remove volumes, everything works fine, may be it is trying to refer back to the package.json file in my local drive and somehow container is not finding the package.json inside the container, yet to figure out how to solve this.

```
enoent ENOENT: no such file or directory, open '/app/package.json'

```

Here is the output from docker-compose:

```
Starting frontend_web_1  ... done
Starting frontend_test_1 ... done
Attaching to frontend_test_1, frontend_web_1
web_1   | npm ERR! path /app/package.json
web_1   | npm ERR! code ENOENT
web_1   | npm ERR! errno -2
web_1   | npm ERR! syscall open
web_1   | npm ERR! enoent ENOENT: no such file or directory, open '/app/package.json'
web_1   | npm ERR! enoent This is related to npm not being able to find a file.
web_1   | npm ERR! enoent
test_1  | npm ERR! path /app/package.json
test_1  | npm ERR! code ENOENT
test_1  | npm ERR! errno -2
web_1   |
test_1  | npm ERR! syscall open
web_1   | npm ERR! A complete log of this run can be found in:
web_1   | npm ERR!     /root/.npm/_logs/2019-04-22T04_10_39_226Z-debug.log
test_1  | npm ERR! enoent ENOENT: no such file or directory, open '/app/package.json'
test_1  | npm ERR! enoent This is related to npm not being able to find a file.
test_1  | npm ERR! enoent
test_1  |
test_1  | npm ERR! A complete log of this run can be found in:
test_1  | npm ERR!     /root/.npm/_logs/2019-04-22T04_10_39_236Z-debug.log
frontend_web_1 exited with code 254
frontend_test_1 exited with code 254

```

Dockerfile.dev

```
FROM node:alpine

WORKDIR /app

COPY package.json .

RUN npm install

COPY . .

CMD ["npm", "run", "start"]

```

Docker file contents are below

```
version: '3'
services:
  web:
    build:
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "3000:3000"
    volumes:
      - /app/node_modules
      - .:/app
  test:
    build:
      context: .
      dockerfile: Dockerfile.dev
    volumes:
      - /app/node_modules
      - .:/app
    command: ["npm", "run", "test"]

```

My directory structure as as shown below,

Executed the below statement and the corresponding response,

```
$ docker run -it --rm -v "$(pwd):/app" busybox ls -l /app

the input device is not a TTY.  If you are using mintty, try prefixing the command with 'winpty'

```

Update: I now am able to understand it is related to declaring volumes in Docker compose, if I remove volumes, everything works fine, may be it is trying to refer back to the package.json file in my local drive and somehow container is not finding the package.json inside the container, yet to figure out how to solve this.

# Answer
> 1 votes
As mentioned in the post/question, it was an issue with volumes, my PC's firewall was blocking the Docker from accessing the drives in my local machine, I reset the credentials in 'Docker for Desktop' app in the shared drives section and everything was back working once again :). Thank you everyone, in fact BMitch was suggesting the same solution, but I was a completely beginner and I did not comprehend it well. Thanks everyone for your responses.

---
Tags: docker
---


title: Docker not exposing port on mojave mac

text:
# Question
Title: Docker not exposing port on mojave mac
I used brew to install docker, docker-machine and docker-compose. Everything is working, but when I use expose to map port on my host machine, it is not working.

When I ssh in docker machine service is running perfectly inside docker machine, but outside nothing coming up.

My software versions are

```
Docker version 18.09.5, build e8ff056

docker-machine version 0.16.1, build cce350d

docker-compose version 1.24.0

```

and I am using `xhyve` for virtualization

Here is my Dockerfile

```
FROM node
  LABEL maintainer xyz@gmail.com
  RUN git clone -q https://github.com/docker-in-practice/todo.git
  WORKDIR todo
  RUN npm install > /dev/null
  EXPOSE 8000
  CMD ["npm", "start"]

```

I use following command to run docker

```
docker run -i -t -p 8000:8000 --name example1 todoapp

```

I have read many article and noticed there is issues in docker and mac related, but not sure how can I solve it. Please any suggestion. I am not using mac docker app due to system resources limit.

I also read somewhere that I should use sudo than it should work, but I tried and get following error

```
docker: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

```

I think sudo will require when port is below 1000 but my port is 8000 so I don't think I need sudo here.

# Answer
> 2 votes
Most likely the port is published on the VM and you are trying to connect to your laptop's IP and not seeing it. Newer versions of docker that use xhyve include some networking tricks to publish the port all the way to your laptop, but docker-machine with VirtualBox does not do this.

You can lookup the IP of your VM with `docker-machine ls` or see it in the variable `echo $DOCKER_HOST` and connect to port 8000 on that IP to reach your application.

# Answer
> 0 votes
That shows that your docker service is not yet running . Try start the service

```
open -a Docker

```

---
Tags: docker, dockerfile, ports
---


# Question
Title: Is OpenStack the same type of software as Azure, AWS, etc.?
I'm not sure how to ask it. Is something like OpenStack the same thing as AWS and Azure, except it is Open Source? If I wanted to create an onsite cloud for a Production environment, can OpenStack (or other) be used?

# Answer
> 1 votes
Generally yes - all of them are large ecosystems of related software products working together to implement "clouds".

There won't be an exact 1:1 mapping for each and every one of their individual products, features and APIs, so don't expect that. But there is a quite significant overlap in capabilities and functionality in general. Just like with the public cloud providers: they're similar to each-other, but not exactly 1:1 compatible.

You (and your customers) will have to learn the specifics of using the desired cloud products in the ecosystem, just like with the other cloud infra providers.

But you'll also need to learn how to install, maintain and operate the infrastructure in general as well as the individual products you're gonna offer, which is something users of the public cloud providers don't need to worry about.

# Answer
> 0 votes
Yes

In the cloud theory there 's 3 types of cloud : Public Cloud, Private Cloud and Hybird Cloud

Openstack one kind of the private cloud, the alternative is opennebula and cloudstack

And Azure, AWS and GCP is kind of Public Cloud Service

And the hybird cloud is the cloud that combine 2 of that type of cloud

---
Tags: cloud, openstack
---


# Question
Title: etcd cluster is unavailable or misconfigured (kubernetes multi-master)
I am new to kubernetes and I am working on creating multi-master-kubernetes with 2 masters nodes.
Could anyone explain why when I execute the command
`ETCDCTL_API=2 etcdctl member list`, I get:

> etcd cluster is unavailable or misconfigured;

And when I execute `ETCDCTL_API=2 etcdctl --ca-file=/etc/etcd/ca.pem --cert-file=/etc/etcd/kubernetes.pem --key-file=/etc/etcd/kubernetes-key.pem member list`, I get this result:

> cc59f69db7afb8fc: name=x peerURLs=https://x.x.x.x:2380 clientURLs=https://x.x.x.x:2379 isLeader=false
> e91f43559989b2a6: name=y peerURLs=https://x.x.x.x:2380 clientURLs=https://x.x.x.x:2379 isLeader=true

Am I missing something?

# Answer
> 0 votes
It was a stupidity of me for not paying attention to etcdctl version. In my case, I had to use
`ETCDCTL_API=3 etcdctl member list`
And do not forget to export the env.

---
Tags: kubernetes
---


# Question
Title: use different kubectl configs in different terminal sessions
If I run `kubectl get svc`, it seems to be the same across terminal sessions - how can use a local configuration that's only loaded for my current terminal session?

If the answer is a local config file and using KUBECONFIG env var to point to it, then how can I generate a local config file? If I use:

```
export KUBECONFIG=`dirname "$BASH_SOURCE"/conf`
aws eks update-kubeconfig --profile=alex --name ignitecluster

```

I get this error:

> Content not a dictionary.

# Answer
> 1 votes
Set the `KUBECONFIG` environment variable with a value that includes the relevant Kubernetes configuration file, the environment variable only applies to your current session/window.

---
Tags: kubernetes, kubectl
---


# Question
Title: Download Google Cloud Bucket as a file (zip)
I use Firebase. If you want to backup a Firebase bucket, they recommend doing this: https://firebase.google.com/docs/firestore/manage-data/export-import

What is the problem? You are backing up a bucket inside other bucket. I want to backup my bucket in an external device. I just want to download all my bucket content (or a folder inside that bucket) as a zip file, and save it in my awesome pendrive.

Is there a way to do it? I could not find a way to: 1. Download a bucket as a file in my computer 2. Or, download a folder that is inside a bucket.

How I'm currently doing the backup? Running this in the Google Cloud console

```
gcloud beta firestore export gs://my-product-name-backup/12-31-2018 --collection-ids=users,problems,seeds,feeback

```

So, my bucket is called `my-product-name-backup`. I'm OK with downloading the whole bucket, or just a folder (for example: `my-product-name-backup/03-03-2019`)

# Answer
You seem to be confused a bit about how Firestore stores data and what Google is suggesting you do.

> What is the problem? You are backing up a bucket inside other bucket.

This is not correct. Firestore stores its data inside some sort of mostly opaque database format. If you are copying this out to a GCS bucket, that really *is* an export into a different storage medium and a different format.

That being said, it's still reasonable that you might want to keep your data backups outside of Google (although GCS has a good SLA). To do that, after you've exported your Firestore data to a GCS bucket, use the gsutil utility to copy that bucket down:

```
gsutil cp -r gs://your-bucket-name .

```
> 2 votes

---
Tags: google-cloud-platform, backup
---


# Question
Title: how to run an Ansible command for specific number of hosts
I am new to Ansible and i have a question about limitting hosts in a host group at running commands.I have these host groups in my inventory:

```
[primary]
host1
host2
[standby]
host3
host4

```

i have thses configuration in my `.yml`:

```
  ---
  - hosts: primary
  roles:
     - automation

```

for example i want to run this command `ansible -playbook -i /path test.yml --tags automation` on `host2` only. what should i do?

# Answer
Use the `--limit` option to select a subset of hosts:

```
ansible-playbook -i /path  test.yml --tags automation --limit host2

```

Its description from ansible-playbook documentation:

> further limit selected hosts to an additional pattern
> 2 votes

---
Tags: ansible, linux, automation, centos
---


# Question
Title: Set environment variables then run script in Jenkins Scripted Pipeline
I am new to Jenkins, Groovy and pipelines. I have created a simple pipeline stage like so:

```
//working build but not setting env variables
node('build-01') {
    stage('Building') {
        echo "[*] Starting build (id: ${env.BUILD_ID}) on ${env.JENKINS_URL}"
        try {
            sh 'ls -l'
            //ls shows the damn file
            sh '. setup-target'
        } catch(all) {
            sh "echo 'Failed to run setup-target script with error: ' ${all}"
        }
    }
}

```

This works. But I want to modify/add environment variables to the session running this script (this script is a bash file with the correct shebang line on top). So I did:

```
node('build-01') {
    withEnv(["CMAKE_INSTALL_DIR=${WORKSPACE}", "SDK_INSTALL_DIR=${WORKSPACE}"]){
        stage('Building') {
            echo "[*] Starting build (id: ${env.BUILD_ID}) on ${env.JENKINS_URL}"
            try {
                sh 'ls -l'
                //ls shows the damn file
                sh '. setup-target'
            } catch(all) {
                sh "echo 'Failed to run setup-target script with error: ' ${all}"
            }
        }
    }
}

```

This errors out with:

> > /home/jenkins-sw/ci/workspace/myWorkSpace@tmp/durable-6d30b48d/script.sh: line 1: .: setup-target: file not found

and

> > Failed to run setup-target script with error: hudson.AbortException: script returned exit code 1

But the environment variables are set, I check this by doing a `sh 'printenv'` right below the `ls -l` line. Interestingly `ls -l` does show the script.

What am I missing?

# Answer
I figured it out. I was cloning directly into the workspace and then setting my environment variables to point to the workspace as well. I modified both those things. I now create a dir in my workspace and clone into it and I also set my environment variables to directories inside my workspace. Like so:

```
node('build-01') {
    withEnv(["CMAKE_INSTALL_DIR=${WORKSPACE}/cmake_install", "SDK_INSTALL_DIR=${WORKSPACE}/sdk"]){
        stage('Building') {
            echo "[*] Starting build (id: ${env.BUILD_ID}) on ${env.JENKINS_URL}"
            try {
                sh 'ls -l'
                //ls shows the damn file
                dir('path/to/checkout') {
                    sh '. ./setup-target'
                }
            } catch(all) {
                sh "echo 'Failed to run setup-target script with error: ' ${all}"
            }
        }
    }
}

```

This works.

Note: Since this site is in beta phase and I barely got any views I posted the same question to stackoverflow, here it is for refernce: https://stackoverflow.com/questions/56015463/set-environment-variables-then-run-script-in-jenkins-scripted-pipeline
> 0 votes

---
Tags: jenkins, jenkins-pipeline, shell-script, groovy, environment-variables
---


# Question
Title: Cheapest and easiest way to run a periodic Docker process
I have a dockerized Python process that a client would like to run once or twice a month. It calls a number of Google APIs and stores the results in a Google Sheet. It does not accept network requests. I want to provide one "magic button" that will spin up the Docker container and then shut everything down.

Some background: I had previously deployed this to an AWS EC2 micro instance via Docker Cloud. My client got a Docker Cloud account and was able to run the container on demand, relatively painlessly. There are two problems with this workflow:

1. The provisioned EC2 instance (Docker host) was running 24/7, whether the Docker container was run up or not. This became expensive.
2. Docker cloud shut down a month ago.

Alternatives seem to be Google and AWS Kubernetes services. My reservation is that they will be too complicated for my client to use. Advice?

# Answer
AWS offers fargate now, which is managed by them rather than being a service like ECS which you ran the servers for and they manage the control plane. Fargate is basically "serverless" ECS. You can use a cloudwatch scheduled even to run something occasionally on a schedule on a fargate ECS cluster. Therefore you're only paying for what you're using. Something to consider is fargate is quite a bit more expensive than ECS running on your own EC2 instance, but if you're running in frequently you'll likely see a pretty marginal savings.

Another option would be to automate something similar. So create an ECS cluster with an ASG backing it, and adjust scale to match ECS demand, and then use cloudwatch to launch the container. ECS can then scale up and down, but this can be tricky because this method is usually used to autoscale ECS clusters based on cloutwatch triggers, so you might have to hack around in cloudwatch and possibly hack together a lambda script, since you're technically not using it the way it's intended. In the end this would probably be the most cost effective, but in your case the cost savings could be negligible.

In my opinion Kubernetes is a ten ton hammer that you'd be using to hang a picture on the wall. It's way overkill.

You can explore those 2 options and see if they are cost effective, and easy enough to manage. Other container orchestration is usually going to be overkill, ECS has the advantage(usually a disadvantage) of being stupid simple and you pay very little and have basically no operational overhead for the control plane.
> 3 votes

# Answer
The cheapest way is to use Heroku's Container Registry.

It's entirely free, easy to use and deploy (You don't even need the heroku CLI , just docker push your image to registry.heroku.com with your auth token)

It may not be the best though but based on your comments I think it should be fine for you, see cons here.

Another cheap alternative is hyper.sh.
> 2 votes

# Answer
You can mix the powerful of Zappa and Hug to convert your code in a serverless AWS Lambda function adding only a decorator to your python process main function

```
import hug
[...]

@hug.get('/your_endpoint_name')
def your_function_name():
    """Here goes your code"""
    [...]
    return "Function finished sucesfully"

```

After this you can deploy to AWS Lambda with `zappa deploy prod` and you will only need to call the returned URL twice a month.
> 2 votes

# Answer
While I'd normally agree with the other answers that Kubernetes is overkill, KubeSail simplifies it as much as possible in order to make it easy to run tasks like yours. There is a free tier that should let you run your job indefinitely. You just need to log in with GitHub, get your Kube config, and then you can use the following:

```
apiVersion: batch/v1
kind: Job
metadata:
  name: my-python-script
spec:
  template:
    spec:
      containers:
      - name: my-python-script
        image: asciimo/my-python-image

```

Just save out the above to `my-job.yaml`, replacing `image: asciimo/my-python-image` with the name of your image on dockerhub (or another registry), then run

```
kubectl apply -f my-job.yaml

```

If you wanted to learn more about Jobs, the Kubernetes docs contain a lot more info and options: https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/

*Full disclosure, I am one of the KubeSail Founders*
> 1 votes

# Answer
Amazon's ECS service is really simple and allows you to schedule container tasks

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduling\_tasks.html
> 0 votes

# Answer
Ansible ec2 module have required functionality (launch instances, runs some tasks and then terminate them) out-of-the-box. And even have sample playbook.
> 0 votes

---
Tags: docker, cloud
---


# Question
Title: Pre-install Jenkins plugins using Docker image
If I have a jenkins docker image:

```
FROM jenkins/jenkins:lts

```

is there a way to pre-install certain plugins using the Docker image, for example all the bitbucket plugins:

I assume we'd probably have to unpack some tars to the filesystem somewhere like so:

```
 FROM jenkins/jenkins:lts

 USER root

 WORKDIR /jenkins/main/dir

 RUN tar -xvf some-jenkins-plugins.tar

```

# Answer
> 2 votes
This seems to be the right idea: https://stackoverflow.com/questions/29328278/installing-jenkins-plugins-to-docker-jenkins/29328489#29328489

so in the Dockerfile you'd just use:

```
COPY plugins.txt /usr/share/jenkins/plugins.txt
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/plugins.txt

```

I tried it and it worked for me.

---
Tags: docker, jenkins
---


# Question
Title: how to give a user privilege to run Ansible commands or just an specific tag?
for example, I have a tag named 'installagent' and i want to run this command `ansible -playbook -i /path agent.yml --tags installagent` from another server via ssh on my Ansible server with user 'testusr'. what should i do?

can i limit 'testusr' to only use specific tags?
( I read somewhere that i should give `sudo` permission to that user. if its true what should i do if i don't want to give `sudo` permission?)

# Answer
There is no tool in Ansible to authorize a user to execute selected commands or tags only.

It seems a good idea to use sudo to authorize user to execute selective commands.
> 0 votes

---
Tags: ansible, linux, permissions
---


# Question
Title: Connect two PCs to Azure vnet then RDP from one to the other
I have an Azure vnet setup, to which two PCs are connected via a point-to-site VPN. I was hoping to be able to then RDP from one pc to the other using the IP assigned by the gateway, but this doesn't work, just can't find the other pc. I've dropped firewalls etc... to test this, but still nothing.

Can anyone advise if this is actually possible, or if there is something else I can do?

# Answer
> 1 votes
I have fixed this partially by updating the routes.txt file in the installation folder for the client app by adding an entry for the subnet used by the gateway to assign IP address to new logins.

While this now routes to the correct PC when RDPing, unfortunately a new IP is assigned to the PC every time it connects to the VPN, making it difficult to know the IP address when connecting.

---
Tags: azure, vpn
---


# Question
Title: Docker NGINX issue -- unable to edit nginx.conf
I'm having trouble figuring out how to edit `nginx.conf` inside NGINX open source Docker container.

I'm trying to follow the directions here:

https://docs.nginx.com/nginx/admin-guide/installing-nginx/installing-nginx-docker/#maintaining-content-and-configuration-files-in-the-container

I've gotten through all 4 steps succcessfully. I have SSH access using the helper `mynginx4_files` container.

This part confuses me, though:

> * the /bin/bash argument means that the bash shell runs in the helper container, presenting a shell prompt that you can use to modify files as needed

This sounds good! Unfortunately, I don't see any way I can modify `nginx.conf`. There's no text editor that I can see. I don't see `vi` or `emacs` or anything else. (I searched through all 4 'bin' directories, and `whereis` and `find` didn't turn up anything for `vi`.) I tried to install `vi` using `apt-get` but was unsuccessful. I even tried ubuntu instead of debian, but even with ubuntu there is no text editor. I thought vi is standard on all Linux distributions. I don't know what's up, but I feel like I'm missing something big here. Any help greatly appreciated.

# Answer
> 1 votes
I suggest you mount your conf file as a volume:

```
docker run -v nginx.conf:/etc/nginx/nginx.conf ....

```

This way you can easily change the file outside the container and then just restart the container. If you change your config file inside the container and then you should have to restart nginx to pick-up the changes. At that point your container will stop because you stopped the main process.

Your local nginx.conf should already have the initial configuration which you can copy from inside the container (cat /etc/nginx/nginx.conf).

---
Tags: docker, nginx
---


# Question
Title: Docker Node.js Error: Cannot find module
When I start a dockerized Node.js testapp with

```
sudo docker-compose up

```

I get the following error:

```
Starting testapp_web_1 ... done
Attaching to testapp_web_1
web_1  |
web_1  | > testapp@0.0.1 start /usr/app
web_1  | > node index.js
web_1  |
web_1  | internal/modules/cjs/loader.js:613
web_1  |     throw err;
web_1  |     ^
web_1  |
web_1  | Error: Cannot find module 'mongodb'
web_1  | Require stack:
web_1  | - /usr/app/index.js
web_1  |     at Function.Module._resolveFilename (internal/modules/cjs/loader.js:610:15)
web_1  |     at Function.Module._load (internal/modules/cjs/loader.js:526:27)
web_1  |     at Module.require (internal/modules/cjs/loader.js:666:19)
web_1  |     at require (internal/modules/cjs/helpers.js:16:16)
web_1  |     at Object.<anonymous> (/usr/app/index.js:4:21)
web_1  |     at Module._compile (internal/modules/cjs/loader.js:759:30)
web_1  |     at Object.Module._extensions..js (internal/modules/cjs/loader.js:770:10)
web_1  |     at Module.load (internal/modules/cjs/loader.js:628:32)
web_1  |     at Function.Module._load (internal/modules/cjs/loader.js:555:12)
web_1  |     at Function.Module.runMain (internal/modules/cjs/loader.js:826:10)
web_1  | npm ERR! code ELIFECYCLE
web_1  | npm ERR! errno 1
web_1  | npm ERR! testapp@0.0.1 start: `node index.js`
web_1  | npm ERR! Exit status 1
web_1  | npm ERR!
web_1  | npm ERR! Failed at the testapp@0.0.1 start script.
web_1  | npm ERR! This is probably not a problem with npm. There is likely additional logging output above.
web_1  |
web_1  | npm ERR! A complete log of this run can be found in:
web_1  | npm ERR!     /root/.npm/_logs/2019-05-04T15_34_04_615Z-debug.log
testapp_web_1 exited with code 1

```

The project structur is as follows

```
- testapp
--- docker-compose.yml
--- dockerfile
--- src
----- index.js
----- package.json

```

**index.js**

```
'use strict';

const MongoClient = require('mongodb').MongoClient;
const express = require('express');

// Constants
const PORT = 8080;

// App
const app = express();
app.get('/', (req, res) => {
  res.send('Hello world\n');
});

app.listen(PORT);
console.log(`Running on Port:${PORT}`);

```

**package.json**

```
{
  "name": "testapp",
  "version": "0.0.1",
  "description": "Testapp",
  "author": "hi there",
  "main": "index.js",
  "scripts": {
    "start": "node index.js"
  },
  "dependencies": {
    "express": "^4.16.4",
    "mongodb": "^3.2.3"
  }
}

```

**dockerfile**

```
FROM node:12-alpine

WORKDIR /usr/app

COPY package*.json ./

RUN npm install

COPY . .

EXPOSE 8080

```

**docker-compose.yml**

```
version: '3'
services:
  web:
    build: .
    command: npm start
    volumes:
      - ./src:/usr/app/
      - /usr/app/node_modules
    ports:
      - 80:8080

```

I have alread read this stackoverflow thread but I couldn't solve this issue. I'm getting this error no matter what module I'm trying to use: mongodb, spdy, etc. Locally without docker the testapp is working. But I don't know what I'm doing wrong with docker. Can anyone help me?

# Answer
You need to remove the volumes all together. Files are copied by your Dockerfile build:

```
version: '3'
services:
  web:
    build: .
    command: npm start
    ports:
      - 80:8080

```
> 0 votes

# Answer
```
    volumes:
      - /usr/app/node_modules

```

This volume could be overwriting what you have installed during the `RUN npm install` stage of the Dockerfile. As far as I can see you're not doing any installation when starting the container, so the contents of the mount are empty. If you remove that mount, the `node_modules` path will be populated with what was installed during the `docker build` (as defined by the Dockerfile) phase.
> 0 votes

---
Tags: docker, node.js
---


# Question
Title: How to approach dynamic environments per Git feature branch workflow?
Our system consists of

* 2 frontend applications A and B (both in JavaScript/Angular),
* 3 backend applications C, D and E (main API and two microservices all in Go)
* MongoDB as F

Our team grows bigger, it's 10 developers at the moment.

We are working on 2, 3 new major features at the same time.

We only have two pre-production environments which makes it hard to test/present different features that are being built at the moment (and on top of that, to test fixed bugs and hot fixes etc.).

We use Git and try to follow git branching model (which might not be relevant here).

To make it easier, we can assume, that each repository has its own Dockerfile to create a fully working container with that application.

What are the options to make it super easy (one command) to publish/switch to a feature that a developer is working on so it's available on one of the environments (that could maybe be created dynamically?)

Should we create 6 new servers/containers if we only make changes in application A?

I would like to have easily accessible versions via a URL like `https://a166.preview.example.com` where `a` is the application name and `166` is the application version (or branch name).

# Answer
> 3 votes
What you're after is referred to as "Review Apps" by Gitlab and Heroku. It relies on a CI/CD system that runs pipelines on every branch, and exposes the branch name to the build (usually as an environment variable). You can then use the branch name to namespace the resources created by your Infrastructure as Code.

You have mentioned Docker so I will use Kubernetes as an example, but this approach can be applied to just about any deployment method.

Normally when deploying a Docker image, you would name your deployment after the app, e.g.:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
# ...

```

Using either some kind of preprocessor (Jinja, envsubst, etc.) or a tool like Helm we can pass in some parameters to make the resource include the branch name during the pipeline:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app-${BRANCH_NAME}
# ...

```

This way, branches do not deploy on top of each other. The resources they deploy are isolated to the branch.

For an API or web app, you'll want some kind of DNS name and load balancer. Kubernetes handles this quite easily with Ingress and DNS controllers, so I'll use CloudFormation as an example this time.

```
AWSTemplateFormatVersion: '2010-09-09'
Parameters:
  Branch:
    Type: String
  Environment:
    Type: String
  Zone:
    Type: String

Resources:
# ALB resources omitted for brevity...

  Route53:
    Type: AWS::Route53::RecordSet
    Properties:
      Name: !Sub
        - "my-app-${Branch}.${Environment}.${Zone}"
      HostedZoneName: !Ref Zone
      Type: A
      AliasTarget:
        HostedZoneId: !GetAtt ALB.CanonicalHostedZoneID
        DNSName: !GetAtt ALB.DNSName

```

Here we create a DNS record that is very specific to the branch. The DNS record resolves to a load balancer which should point to the containers deployed by the pipeline for this branch. The stack name is also specific to the branch, so pipelines on other branches will not affect it. The end result is that each feature branch will have a dedicated URL (`https://my-app-my-feature.dev.example.com`) that your developers can use to test and showcase to stakeholders.

> Should we create 6 new servers/containers if we only make changes in application A?

No. This isn't specifically related to Review Apps, but microservices in general should be self-contained: decoupled from each other so that they can be deployed in isolation. Removing dependencies and making small changes will increase your development/deployment velocity.

## Considerations

### Cleanup

What happens when you merge or delete your branch? Without setting up some kind of cleanup, those containers and DNS records will sit around forever. You need some kind of cleanup. Something that just runs periodically and deletes resources associated to old branches is pretty simple to write, or you could take webhooks from your git server and delete those resources on an event.

### Databases

Some resources may not warrant getting deployed per-branch, although this should be the exception rather than the norm. You probably don't want to deploy a new database every time someone makes a branch (perhaps unless you have a good schema management tool and DB population scripts). You can have all deployments connect back to the same DB. Ensure that any changes you make to the schema are backwards compatible at least one version behind.

---
Tags: docker, deployment, environments
---


# Question
Title: How to trigger Jenkins Pipeline by push in specific branch in Bitbucket?
I've created Jenkins Pipeline Job, activated "Build when a change is pushed to BitBucket" trigger. I've also specified branches to build. But pipeline is triggered by all branch changes. Also I've created Freestyle job with similar configuration and trigger works correctly. General Webhook didn't help, it triggered all jobs in Jenkins.

# Answer
While I don't know Bitbucket-specific solution, when it comes to triggering Jenkins builds, I prefer using a simple post-receive Git hook.

In Jenkins select `Trigger this build remotely` (or something similar, it's usually the very first option in `Build triggers` section), then set up the post-recevie hook to something like:

```
#!/bin/bash

username=user
token=1234567890abcdefgh

echo "Executing hook"

while read oldrev newrev refname
do
   branch_received=$(git rev-parse --symbolic --abbrev-ref $refname)
   echo "   /==============================="
   echo "   | PUSH RECEIVED"
   echo "   | Old revision: $oldrev"
   echo "   | New revision: $newrev"
   echo "   | Reference   : $refname"
   echo "   | Branch name : $branch_received"
   echo "   \=============================="
   curl -X POST -d "BRANCH=$branch_received" http://$username:$token@jenkins/job/job_name/buildWithParameters?token=$token
   res=$?
   if test "$res" != "0"; then
      echo "The curl command failed with: $res"
   fi
done

exit 0

```

Of course, the above runs for every branch and passes the branch name as a parameter to the job. In your case, since you have different job for every branch, you would call different URL depending on what branch was pushed and probably use `build` rather than `buildWithParameters` endpoint.
> 2 votes

# Answer
I recommend using the Bitbucket Branch Source Plugin which gives much better control over push build triggers than the default Pipeline job type.

The problem with your current setup is that the "branch to build" setting only configures the branch of the repository that Jenkins checks out, not necessarily which branches will trigger a build.
> 1 votes

---
Tags: jenkins, jenkins-pipeline, git, bitbucket
---


# Question
Title: Jenkins run build in docker on server (withDockerServer not working)
I am trying to run my build inside a docker; but I don't want the docker to run on the jenkins master but on another server that has docker running.

What does work:

* inside jenkins configure a docker Cloud to connect to docker server (using tcp://my-server:4243)
* configure a docker agent template for certain docker images and assign labels to those docker containers (these need to be containers with sshd running; java installed etc such that they can be connected as jenkins slaves)
* in my build I specify that I want the build to run on that label

What does not work:

```
pipeline {
        agent any
        stages {
                stage('test') {
                        steps { script {
                                withDockerServer([uri: 'tcp://my-server:4243']) {
                                        withDockerContainer([image: 'alpine']) {
                                                sh "whoami; pwd; ls /usr/local/bin"
                                        }
                                }
                        } }
                }
        }
}

```

in this case I get errors:

```
Caused: java.io.IOException: Cannot run program "docker": error=2, No such file or directory

```

since docker is not installed on my jenkins master; this means he is trying to execute the docker image on my jenkins master and not on my-server... any tips?

I am using:

* jenkins 2.150.3
* docker plugin 1.1.6
* docker pipeline 1.17

**EDIT** even with scripted pipeline this does not work:

```
node('master') {
        withDockerServer([uri: 'tcp://my-server:4243']) {
                withDockerContainer([image: 'alpine']) {
                        sh "whoami; pwd; ls /usr/local/bin"
                }
        }
}

```

he is still ignoring `withDockerServer` (it is shown in the pipeline.log, but the docker command is still run on jenkins master)

# Answer
apparently `docker` needs to be installed on jenkins master in order for jenkins to be able to launch the docker on the server.

After installation of `docker` (even without the `docker` daemon running on jenkins master); jenkins was able to launch the docker on **my-server**.
> 1 votes

---
Tags: docker, jenkins, jenkins-pipeline
---


# Question
Title: Preventing accidental production deployments in ansible
Last night I ran a playbook from a wrong branch that referenced an old inventory file with old IPs for each server, and deployed a dev build to a production machine. What would be a a good practice to avoid such mistake from happening in the future? How can I isolate environments better?

# Answer
The best practice is to separate production and staging(testing) inventory. See Staging vs Production.

> This way you pick with -i what you are targeting.

To make it more robust you might want to clone a separate structure for production and restrict access to it. Pull from staging before deploying to production.

In detail, this would mean to 1) put the directories into git, 2) make sure staging is committed, pushed and merged, and 3) then pull to the production and deploy. The workflow is up to you.
> 1 votes

---
Tags: ansible, environments
---


# Question
Title: Is there a way to map ports two docker container
Let's consider two docker conatiner named `app` and `mysql`. I am looking for a way, where I can map `app` and `mysql` ports, so that if I try to connect to `mysql` from `web`, using `localhost:3306` it can connect.

I know I can connect using `dns` and `docker network`. But I am curious if there is a way to do this.

# Answer
> 1 votes
On Linux, containers live in separate network namespaces, so localhost or 127.0.0.1 on the container is not localhost on the host. I'm not sure about other OSes. You could expose a port to the host like this:

```
docker run -p 3306:3306 mysql

```

Then you can access it from any container at the host IP address, which for the default network is 172.17.0.1:3306. You can add -p many times, read the docs here: https://docs.docker.com/

All that the -p does on Linux, is essentially:

```
sudo iptables -t nat -A DOCKER \! -i docker0 -p tcp --dport 3306 -j DNAT --to 172.17.0.2:3306

```

(Or whatever your mysql container IP address is.) Docker compose and similar utilities make containers able to see each other with **--link** eg.

```
docker run --link web --link mysql:localmysql app

```

The only thing this does is to put the correct IP address for the container in a file called /etc/hosts inside the container (but this might change in future.)

If you want to use --link, all containers need to be on the same network. You can connect them while running with:

```
docker network connect mynet web

```

The main reason for doing this is because the containers don't have fixed IP addresses. You can do:

```
docker run --network mynet --ip 172.18.0.10 -d app

```

To give your container a static IP - and then just use that instead of the name. The docker team keeps changing how things work, so you might need to run additional commands to allow different networks to connect to each other.

You can of course also do all of this manually, on Linux, using the firewall, on the host machine. (Start learning about it here: https://www.google.com/search?q=iptables (Also look at the Images tab!)

```
iptables -nvxL # shows the filter table
iptables -nvxL -t nat # shows the NAT table where port forwarding is done

```

To dynamically forward a port to a container, get its IP address, eg 172.18.0.10, then:

```
iptables -t nat -A POSTROUTING -p tcp --dport 1234 -j DNAT --to 172.18.0.10:4567

```

Be keenly aware that if you do this, and your container restarts, or gets another IP address, that this will not automatically update, and unless you use iptables-save and iptables-restore in your startup and cron scripts, these manually created rules will disappear. Also that the docker command and compose scripts won't know about any configuration that you manually add like this.

If you really want localhost in your container to point somewhere else, you can edit /etc/hosts inside the container, and change the IP address of localhost - be very keenly aware that other things might break, because localhost is supposed to point to the same host, not another one, so there will be unintended consequences at some point.

---
Tags: docker, networking, ports
---


# Question
Title: running a phpUnit test within a docker container
i want to run a specific version of phpUnit WITHIN a docker container. This container will use a specific version of php. i.e

php:5.6-apache

Its a lareval application. i have install phpunit via composer on the hostfiles and then used the volume command to transfer this to the container. my composer.json file has following entry:

> "require-dev": { "phpunit/phpunit": "^5.0" }

This is my docker run command to run the test on my testdev container:

```
docker run --rm -it -v ~/Users/mow/Documents/devFolder/testdev:/app testdev_php "php ./vendor/bin/phpunit"

```

This returns the error:

```
exec: fatal: unable to exec php ./vendor/bin/phpunit: No such file or directory

```

i am unclear why it says this because te vendor directory is at the root of my site directory.

this is my dockerfile

```
FROM php:5.6-apache

ENV S6_OVERLAY_VERSION 1.11.0.1

RUN apt-get update && apt-get install -y \
    libldap2-dev \
    git \
    --no-install-recommends \
    && rm -r /var/lib/apt/lists/* \
    && docker-php-ext-configure ldap --with-libdir=lib/x86_64-linux-gnu/ \
    && docker-php-ext-install ldap \
    && docker-php-ext-install mysqli pdo pdo_mysql

#install xdebug
RUN git clone https://github.com/xdebug/xdebug.git \
    && cd xdebug \
    && git checkout tags/XDEBUG_2_5_5 \
    && phpize \
    && ./configure --enable-xdebug \
    && make \
    && make install

RUN a2enmod rewrite

COPY ./docker/rootfs /
COPY . /app

WORKDIR /app

ENTRYPOINT ["/init"]

```

I guess the real question is this: what is the correct way to run a phpUnit test within a docker container so that its subject to the php version within that container.

# Answer
> 1 votes
Don't quote the command you're trying to run, just do:

```
docker run --rm -it -v ~/Users/mow/Documents/devFolder/testdev:/app testdev_php php ./vendor/bin/phpunit

```

Your image is looking for a file called "php ./vendor/bin/phpunit" instead of looking for "php" and passing everything else on the line to it.

This is the real reason docker requires all your parameters before the image, because the parameter after the image is the command and everything after it is parameters to that command - don't quote everything.

The Dockerfile equivalent is:

```
ENTRYPOINT ["/init","parameter1","parameter2","parameter3"]

```

If it still doesn't work, check the parent Dockerfiles for the location of "php"

```
docker run --rm -it -v ~/Users/mow/Documents/devFolder/testdev:/app testdev_php /usr/local/bin/php ./vendor/bin/phpunit

```

---
Tags: docker, docker-compose, dockerfile
---


# Question
Title: Install MDB2 in docker container while PEAR is down
While pear is down, the following instruction in my dockerfile is failing:

```
RUN pear install -o --alldeps MDB2-beta && \

```

I've searched on the internet how to install MDB2 without pear, successfully accomplished it by downloading the .deb file from https://packages.ubuntu.com/search?keywords=php-mdb2 and then doing `dpkg -i php-mdb2_2.5.0b5-2_all.deb`

My issue is that the same instructions are failing when placed in the dockerfile. Here's the instructions:

```
RUN wget http://mirrors.edge.kernel.org/ubuntu/pool/universe/p/php-mdb2/php-mdb2_2.5.0b5-2_all.deb -O php-mdb2_2.5.0b5-2_all.deb
RUN dpkg -i php-mdb2_2.5.0b5-2_all.deb

```

Here's the error message:

> dpkg: dependency problems prevent configuration of php-mdb2: php-mdb2 depends on php-common; however: Package php-common is not installed. php-mdb2 depends on php-pear (\>= 1.3.6); however: Package php-pear is not installed.

I've tried to manually search and install all dependencies, and dependencies' dependencies but failed at the end because one of them is being told to me that it'll break locales thus cannot be installed.

Also tried another solution I found in stackOverflow to install dependencies for dpkg but it also fails. Instructions:

```
RUN wget http://mirrors.edge.kernel.org/ubuntu/pool/universe/p/php-mdb2/php-mdb2_2.5.0b5-2_all.deb -O php-mdb2_2.5.0b5-2_all.deb
RUN dpkg -i php-mdb2_2.5.0b5-2_all.deb || true && \
apt-get -f install

```

Error received:

> dpkg: error processing package php-mdb2 (--install): dependency problems - leaving unconfigured Errors were encountered while processing: php-mdb2 Reading package lists... Building dependency tree... Reading state information... Correcting dependencies... Done The following packages will be REMOVED: php-mdb2 0 upgraded, 0 newly installed, 1 to remove and 9 not upgraded. 1 not fully installed or removed. After this operation, 528 kB disk space will be freed. Do you want to continue? \[Y/n\] Abort. ERROR: Service 'app' failed to build: The command '/bin/sh -c dpkg -i php-mdb2\_2.5.0b5-2\_all.deb || true && apt-get -f install' returned a non-zero code: 1 Makefile:8: recipe for target 'build' failed make: \*** \[build\] Error 1

Can you point me in the right path maybe?

# Answer
> 1 votes
You need to learn a bit more about Debian or Ubuntu. If you look at every single Debian or Ubuntu container that installs things, you will notice, amongst other things:

```
apt-get update && apt -f install [whatever]

```

The reason for this is that aptitude doesn't know where to find anything without the "apt update".

See for example https://github.com/dockerfile/ubuntu/blob/master/Dockerfile

```
# Install.
RUN \
  sed -i 's/# \(.*multiverse$\)/\1/g' /etc/apt/sources.list && \
  apt-get update && \
  apt-get -y upgrade && \
  apt-get install -y build-essential && \
  apt-get install -y software-properties-common && \
  apt-get install -y byobu curl git htop man unzip vim wget && \
  rm -rf /var/lib/apt/lists/*

```

This updates all packages, then installs build-essential, then software-properties-common and then the packages in the second to last line. Of course you don't need all of this.

It is important however to have the apt-get update && apt-get install on the same line, if you are ever going to deploy your Dockerfile to any CI system. If apt-get update is on its own line, then that layer will never get invalidated as the succeeding layers will have old sources that they build consequent layers from. Keep in mind that every single command generates its own layer, that is why commands are chained in this way - so as to create one layer instead of many that will never serve any purpose.

---
Tags: docker, php
---