# Question
Title: Where should we keep application properties file while doing build promotion using Jenkins?
We are using spring boot to develop Java backend services. `application.properties` file has the database configurations. We are deploying in 4 different environments (dev, test, UAT, and production). The properties files will be different for different environments. Where should we keep these files and how should do build promotion using Jenkins? We are using Apache Tomcat to deploy our WAR file.

Now we are keeping the properties file in `C:\Program Files\Apache Software Foundation\Tomcat 7.0\conf` directory and accessing using the following code.

```
@PropertySource(value = {"file:${catalina.base}/conf/application_dev.properties"})

```

Each time we deploy in different environments, we need to replace the respective file name `application_test/uat/production.properties`, commit the code and let Jenkins build and deploy. Now manually we have to keep the files in different environments too.

Please suggest how should we do it.

# Answer
> 2 votes
I disagree with @levi, you should stick to the 12 factor app method. Leaving configs in the repo scusk. Simple changes to a password or a small setting requires a commit and a pipeline rebuild. That's too much work for something so simple.

The last 3 companies I've worked for are using 'single branch' repos. There is only the master branch, and developers create 'feature' branches off of that. All the different environments are built from master.

In that case, we have a static config file, but all the values are injected by the build pipeline. You can pull them in from different sources: build server env variables, an external tool (Hashicorp Vault), or even another repo that has all the configs (this last one is least desirable).

# Answer
> 2 votes
The typical approach is to use a folder per environment that contains identically named files:

```
conf/dev/application.properties
conf/test/application.properties
conf/uat/application.properties
conf/prod/application.properties

```

Then parameterize the @PropertySource to use an environment variable such as "env":

```
@PropertySource("file:${catalina.base}/conf/${env}/application.properties")

```

Now you simply need to ensure that in each environment there is an operating system environment variable defined called "env" that is set correctly (i.e, dev, test, uat or prod). You haven't stated how your launch your application (j2ee server launch script on linux? springboot docker container?) but it shouldn't be a problem to set the environment variable in each environment.

Rather than load files from "${catalina.base}" you can load files from the classpath with something like:

```
// see https://stackoverflow.com/a/26387933/329496
@PropertySource("classpath:${env}/application.properties")

```

If you are building with maven you can create the folders under `src/main/resources` and they will be copied into the classpath:

```
src/main/resources/dev/application.properties
src/main/resources/test/application.properties
src/main/resources/uat/application.properties
src/main/resources/prod/application.properties

```

That simplified things quite a lot. Jenkins doesn't need to do anything at all as the same build/release artefact can be run in any and every environment. It is usually an anti-pattern to have Jenkins have to do extra work to reconfigure an application to work in a particular environment. Rather you should aim to have a single Jenkins release job build an artefact that can be simply copied between environments.

There is one downside to such an approach. If you want to create a new environment you have to add a new file into source control and create a new release artefact. With legacy technologies setting up new environments is a lot of work (e.g., commissioning VMs and databases) such that this "extra step" of adding a file to the code doesn't seem like a problem.

With more modern cloud-native technologies like Kubernetes, you can set up environments in seconds, and "on-demand". In which case you don't want to be pre-specifying your environments within your codebase. Rather you should follow a 12factor.net approach and define every single property as an environment variable and have spring just use them directly. If you look at the current springboot documentation it uses environment variables by default. This means if you haven't defined a `@PropertySource` but you are using things like:

```
@Value("${name}")
private String name;

```

they will bbe set from the environment variable of the same name but using all caps. This then moves the problem to "how do we make sure that the environment variables are correctly set up in each environment". Technologies that make it easy to spin up environments in seconds (kubernetes, cloudfoundry, swarm) also typically make it easy to manage environment variables. With kubernetes you create a "ConfigMap" (e.g. "my-app-properties") and have it mounted as the environment variables where every key+value defines a unique environment variable for the application. Each logical environment can then be a separate Kubernetes namespace (possibly on a shared cluster for dev/test but a dedicated cluster for prod) that has its own "my-app-properties" configuration object that defines the environment variables for that logical environment .

You can put these settings under source control but not in the application source repo. Rather you can take an "infrastructure as code" approach where everything that runs the application (e.g., all scripts and yaml to set up the environments) is in it's own separate git repo. You can then set up a configuration deploy job that is triggered by a git webhook on the configuration repo. This can push out the new environment variables. That allows you to have continuous deployment of configuration changes that are independent of the continuous deployment of application code.

If you are using Kubernetes then Helmfile is an excellent choice for that is it won't update anything in kubernetes that hasn't changed within git. This means it is safe to reapply all the configuration in the git repo on every push to a protected master branch; helmfile will double check what is already deployed and only update Kubernetes configuration objects that need to be updated.

# Answer
> 0 votes
You could have a branch per environment. So a "dev" branch would build and deploy the "dev" app to the "dev" environment.

Though according to 12factor you should have your environment settings outside of your app repo. Though sometimes it's more work to do that than to just break that rule.

---
Tags: jenkins, jenkins-pipeline, spring-boot
---


# Question
Title: call container B inside container A using docker networks
* I have two docker containers A and B that runs on the same user-defined network.
* Container A has a port published, container B hasnt.
* I do my requests on container's A published port, but container A needs to do requests on container B.
* The requests to B inside A are mapped to `0.0.0.0:(container B needed port)/route`.
* These fails.

How could I call B inside A without getting and changing IP based on the network every time?

# Answer
If you have a user-defined network set to bridge, you can access other containers by their container's name which is automatically as hostname for containers inside same network.

So from `containerA` you could do `ping containerB` without problems. In your case it would be changing the setting to `containerB_name:(container B needed port)/route`
> 2 votes

---
Tags: docker, networking
---


# Question
Title: How to add LinkedIn's Burrow to a Docker Compose?
It appears `LinkedIn` doesn't have an official Burrow docker image on Docker Hub, but there are others who have forked it.

However, I can't find any examples of how to add any of them to a docker compose file that spins up ZK and Kafka something like this.

What am I missing?

# Answer
> 2 votes
This is the example I was looking for: https://github.com/linkedin/Burrow/blob/master/docker-compose.yml

---
Tags: docker-compose, dockerhub, kafka
---


# Question
Title: How do configuration management tools deal with someone messing up configuration files manually?
I'm researching a bunch of configuration management tools, like, Salt, Chef, Puppet, Ansible, etc. and am trying to understand how they deal with someone manually changing a config file on the server directly:

1. A config file that is being directly managed by the CM tool, eg. the nginx/apache/Postgres config file
2. A config file that the system *implicitly* depends on, but is not being directly managed by the CM tool (i.e. it uses whatever defaults the underlying OS provides), eg. the `rsyslog` config file.

Do these tools snapshot / checksum all files in `/etc` (and similar) folders and alert the sysadmin whenever something has been changed behind their backs?

PS: My mental model right now is that of immutable config files & infra, like `nix`

# Answer
> 4 votes
If the file is marked as being managed by a configuration management tool, it deals with someone messing up a configuration file manually being messed up by a hapless admin by clobbering the configuration file using something like Jinja or ERB templates

While this can often feel frustrating to an old-school UNIX admin when their change is clobbered in minutes or hours by a configuration management tool, (and I have seen some use things like extended file attributes to hack around this) it is far superior in that it 1) enforces a cattle-not-pets mentality and 2) self-documents a deployment - a factor which is vastly undervalued considering that 5-10 years later when an upgrade of a system is needed, hundreds of config files may need to be scoured such as a host file, limits files, logrotate files, syslog files, systemctl defaults files, firewalld/iptables config files, selinux files, cron files, and much, much more may need to be considered using the "pets" deployment philosophy.

9 times out of 10 this comes from a lazyness/unwillingness to learn a newfangled configuration management technology. That being said, I highly recommend that if you are going to manage a config file using a configuration management tool, you use Salt, Chef, Puppet, Ansible, etc to flag the fact that you have taken over management of said config file so that the admin is not completely blindsided whenever possible and as often as possible.

You can also support additive configuration - EG, I have used salt to manage rsyslog.conf and *some* files in /etc/rsyslog.d/ but not all files. This allows for a baseline set of defaults, while still supporting some customization without (barf) using a configuration management tool - thus allowing you to "have your cake and eat it too".

# Answer
> 2 votes
For Ansible, it does not manage this for you and you have the following options.

Config files that you depend on, but where default are fine you will have to add to your configuration.

Files whose presence interferes you will have to ad a rule: state=absent For example I use Ansible to configure a Lamp server and need to manage a list of deleted sites.

For where you are changing lines in files, you may also need to add lines to remove

---
Tags: configuration-management
---


# Question
Title: What are tools that lend themselves for backup and recovery in a CI/CD environment?
I have used tar, dump, Amanda, Legato (EMC) Networker and other backup software for making backups of systems. However, I don't know the best tools for making backups in a CI/CD (Continuous Integration/Continuous Deployment) environment and doing "rolling recoveries" of systems "as you go" which is the case in a DevOps oriented system.

Many of these backup utilities are not necessarily best suited for a CI/CD environment because of the continuous changes taking place on both the development and production environment.

I'm looking at programs like Borg and Git based backups like Bup These programs would allow for many incremental backups and finer granularity of backups and work well with CD tools like Ansible. Even Ansible has a couple of way to make backups. However, there is no clean way to recover the most recent backup in an automated way.

Relying on some form of caching or mirroring of your content on the cluster and hope it's not corrupted is not a way to go :(

The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary. The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles. Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.

I would like to know what are the best backup and recovery tools used in the CI/CD environment? and an example of how you would configure it?

I don't expect a "magic bullet" just some possible solutions.

# Answer
> The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary. The containers, VMs, even the webapps would NOT be backed up as these should all be uniform and constantly updated under the CI/CD principles. Currently I use a combination of scp and tar into an archive directory sorted by date and machine. I like to find something better if possible.

Config files, containers and VMs *should* be backed up and versioned by a tool like git or SVN if you are doing configuration management properly.. Backing up config files and versioning them is the whole point of *configuration* management tools like, salt, puppet, ansible, and chef.

As you duly note however, dynamic content such as database data and similarly dynamic data is a more challenging issue. These however, are outside of the scope of *configuration* management, CI, and CD principles. These principles purposefully remain silent on dealing with these issues: because they have no new answers for these problems. Instead, you should use the many battle-tested techniques and strategies for dealing with this kind of data instead of expecting CI/CD/CM tools to solve problems that they were never intended to address.
> 3 votes

# Answer
> The things that would be backed up would be the config files of the servers, the database and uploaded content as these will almost always be unique and necessary.

But would they? Config files can go into `git`. Databases (for the "CI" part) should be 100% creatable from source as well (migrations, fixtures etc.), and may be so on each single test run anyways. I'm not sure what you mean with "uploaded content" in the context of a CI pipeline.

Your goal would be to reduce *everything* to some kind of middle layer that is acting as a barrier between the hardcore low-level stuff, and the application (including CI/CD). This could be straight Docker - i.e., packaging up *everything* (including your CI/CD driver, Jenkins or whatever) in Dockerfiles, with everything needed included as configuration - everything in `git`.

Or a little more advanced, a Kubernetes/OpenShift cluster which does the same; or the respective solution of the cloud providers (AWS, Azure etc.).

Below that border, you are running with classical solutions as you are used to; there should be very little "special" you need to take care of, and it should be easy (as nothing from the application should influence this at all) to make it so you can add nodes without backup/recovery at all. I.e., if some piece of hardware or non-containerized software breaks, you simply throw it away and replace it with a new one.

Dynamic data that needs to persist (e.g., the history of job runs stored by your actual CI driver) will be stored on dedicated volumes (which contain said data, and only it - not interspersed with OS or other files), which, in turn, you can backup with classical tools (offline backups etc.) - the biggest issue, there, is to decide whether to run the backup in "container land", or on the underlying abstraction layer; but aside from that, it would be just copying files around as usual (with whatever tool you like most), not special just because it's a "CI environment".

TL;DR: make as much as possible run in containers, with dedicated volumes; don't backup containers; backup volumes as usual.
> 0 votes

# Answer
I think from a DevOps POV that you are approaching this from slightly the wrong angle. DevOps thinking for example is that there is no need to backup a configuration file; the configuration file is written by a configuration management tool such as Chef and can be recreated at any time.

The database does need to be backed up obviously as that is the truly stateful component of the whole system; you don't mention which you are using but head on over to DBA.SE and someone (perhaps me) can give you some specific pointers on that side of things.

For the uploaded content, if this is outside the database, I'd recommend some filesystem-snapshot based approach, it's easy to do and low overhead.
> -1 votes

---
Tags: ansible, cicd, backup
---


title: OOM when running Bitbucket pipelines

text:
# Question
Title: OOM when running Bitbucket pipelines
as one (sometimes failing) step of our pipeline we

* setup environment via installation of docker which runs 3 containers through own `docker-compose`. Unfortunately we could not figure out how to impose any memory limits on individual containers in docker-compose v3 config.
* run tests in `sbt` with `SBT_OPTS="-Xmx1500m"` and every second to third time step is failing with OOM

definitions: services: docker: memory: 1024

Explanation on the memory usage is not clear on this case.

Could somebody please explain what's going on with the memory allocation to containers, docker host, and step env?

Interesting is that running the same tests, same containers for multiple times usually gives different results. Understandable that real load on the physical server changes over time and running times can differ - but memory?

Puzzled...

# Answer
> 2 votes
I suspect that Bitbucket pipelines is enforcing this memory limit, not docker-compose.

Similarly to docker-compose, the Bitbucket pipelines code is likely to use Linux CGroups to enforce the memory limit, through the Linux kernel's OOM killer.

The page you linked is about memory limits in additional containers (e.g. for a database) used by the pipeline, not the pipeline itself.

Some suggestions:

* Try editing `bitbucket-pipelines.yml` to include a `size` option for memory size - I would expect this is more useful than editing docker-compose options which may not be respected. See this doc page.
* Try running the build locally under Docker Compose to get an idea of memory usage.

It would help to include the details of your `bitbucket-pipelines.yml`, and the Bitbucket error message about being out of memory.

# Answer
> 0 votes
When you limit resources in docker-compose, the docker engine prevent containers to use more than the limits, docker do this by Linux's Cgroups. Its usage in docker-compose v3 is also straightforward: https://docs.docker.com/compose/compose-file/#resources

When your OS is in OOM state, this kills processes automatically and maybe your containers are good candidates in the state, as the problem I have faced on my Gitlab CI's build machine to build node projects!

---
Tags: docker, docker-compose, bitbucket-pipelines
---


title: Configuring Zookeeper from Ansible

text:
# Question
Title: Configuring Zookeeper from Ansible
I'm trying to configure a Zookeeper Cluster with Ansible. As you might know Zookeeper has two important config files.

**zookeeper/conf/zoo.cfg**

I create this file on each cluster member using this jinja2 template:

```
{% for host in groups['zookeeper'] %}
server.{{ loop.index }}={{ host }}:2888:3888
{% endfor %}

```

The result will look like this:

```
server.1=processing1.srv.mycompany
server.2=processing2.srv.mycompany
server.3=indexing1.srv.mycompany
server.4=indexing2.srv.mycompany
server.5=quorumandmonitoring.srv.mycompany

```

**data/myid**

Accordind to the docs this file needs to contain an unique id for each member of the cluster. To keep everything in sync I would like to have the value after `server.` in the `myid` of each host. This means:

* `myid=1` on `processing1`
* `myid=2` on `processing2`
* `myid=3` on `indexing1`
* `myid=4` on `indexing2`
* `myid=5` on `quorumandmonitoring`

And this is where my problem starts: how can I do this in a clever way? An ideal solution would be to use something like the `for host in groups` loop. I would rather not add `host_vars` to my inventory.

# Answer
> 1 votes
In lack of a better idea I chose this rather long approach:

**myid.j2**

```
{% for host in groups['zookeeper'] %}
{% if host == inventory_hostname %}
{{ loop.index }}
{% endif %}
{% endfor %}

```

The idea is to iterate over all hosts and check if the current host being processed matches the host in the list of hosts. If it matches I write the iteration counter to the file.

When I ssh into the boxes I can see their myid files were created according to my need:

```
[root@quorumandmonitoring data]# cat myid
5

```

---
Tags: ansible, jinja, templating, zookeeper
---


# Question
Title: How to create a preemptable docker container?
Is there any way by which I can set a lifetime for a container after that much of time it should be stopped automatically and only awake when someone send a request. I wanted something like **Preemptable Google Compute engine** instance or **Heroku free Dyno** which sleeps after 1 hr of inactivity and awake again when a request is send to that instance.

**Note**: I have docker compose project so I want to set the life time for one service which depends on other services also. I have a traefik container runs separately beside the docker-compose project which handles the request and sends to containers. So is there something like request\_post\_hook which can start that stopped docker-compose container when receive any request ?

# Answer
I'm not aware of any straightforward way to do that, without also running a different container to act as a middleware of sorts. You might want to look at one of the "Serverless" platforms. If your use case will fit that model, many of the docker/swarm/kubernetes serverless tools do what you're asking, but again, you're running other container(s) to manage it. I'd suggest starting with a glance at the OpenFAAS docs to see if it'll work for you.
> 1 votes

---
Tags: docker, docker-compose, docker-swarm, google-compute-engine
---


# Question
Title: How to add multiple jvm options in a saltstack jinja?
I have a config.jinja file with below jvm options:

```
`-Dappdynamics.agent.appName=AP_{{ pillar['region'] }}`

```

However the AP\_{{ pillar\['region'\] }} varies for different environments. For Dev/Prod it can be AP\_{{ pillar\['region'\] }} and for QA it must be AP without the pillar value. How can I define the appName option with multiple values to achieve above? Any suggestions?

```
if (pillar['region'] == QA)
   -Dappdynamics.agent.applicationName=ALIP
else
   -Dappdynamics.agent.applicationName=ALIP_{{ pillar['region'] }}
end-if

```

# Answer
Perhaps a macro might be what you are looking for.

Understanding Jinja:Macros

In the lib.sls example in the link (pasted below), it sets a value contingent on the value of the if statement. I think a similar approach could work for you.

```
{% macro pythonpkg(pkg) -%}
  {%- if grains['os'] == 'FreeBSD' -%}
    py27-{{ pkg }}
  {%- elif grains['os'] == 'Debian' -%}
    python-{{ pkg }}
  {%- endif -%}
{%- endmacro %}

```
> 1 votes

---
Tags: salt, jinja
---


# Question
Title: JFrog Artifactory or JFrog Bintray , when to use one over the other?
We are actually looking for a package management system for our project. The target is simple, have a centralized system to keep our packages ( apps and middlewares ) and use them with our CI/CD tools (Jenkins, Ansible, Docker ...)

Looking for options I discovered JFrog Artifactory and JFrog Bintray. they both seems to do the same job but I don't believe JFrog is maintaining two concurrent products with the same options.

What is the difference between Artifactory and Bintray?
How do we choose one over the other and why?

# Answer
> 8 votes
JFrog Artifactory and JFrog Bintray both manage binaries (and any other file type you can think of). I'd like to see them as two different parts of your CI/CD pipeline.

**Artifactory** is mostly meant to be used inside the organization. For managing all binaries coming in as dependencies (like maven central jars) and being produced by your build process (like your `pack.rpm` or `org/pack-docker` docker image). These can later be used by others in the org or released to **Bintray**.

**Bintray** is meant to provide a universal distribution platform for your end deliverables. For example, the `pack.rpm` you want to be available for your customers to download. And also that `org/pack-docker` docker image.

Look into the features lists of each and see how they differ in order to best match their intended role.

I hope this helps explaining the differences.

**Disclosure**: I work at JFrog

# Answer
> 0 votes
**TL;DR:**

* **Bintray**
    1. Software for other people's use.
    2. Publishing platform
    3. Released software
    4. Selective, distributed, download optimized, network heavy
* **Artifactory**
    1. Software for your own use.
    2. Caching platform
    3. All derived objects and binary dependencies for CI/CD
    4. Inclusive, centralized, upload optimized, storage heavy

**Disclosure**: JFrog did not offer me a job. :)

---
Tags: continuous-integration, artifacts, artifactory
---


# Question
Title: What are vertical scalability limits of JFrog Artifactory Pro?
The essential feature of JFrog Enterprise over JFrog Pro (X) is that it introduces high availability and site replication support.

Obviously, operations in geographically distributed organizations with thousands developers and build processes require some planning also for this aspect.

Nevertheless, is there any data/study to show order of magnitude of load where you would say okay this is where we need not just some vertical scaling but parallel (say using some reference hardware setup)?

# Answer
> 2 votes
As @XiongChiamiov mentioned, it's a very hard question to answer, but here's a clue:

Artifactory is a network-bond server. It means given enough storage, CPU and memory, it will max out your network interface first. Network interface is considered maxed out at half its rated bandwidth limitation. That can help you do the math.

---
Tags: artifactory, high-availability, operations
---


# Question
Title: How to use remote Ivy artefacts in Maven jobs with different binary repository managers?
Consider the following situation:

```
Alice -----              { Firewall }  Bob
|                                       |
Artifactory / Ivy repos  -  ---   --- Nexus / Maven Repos

```

That is, Alice uses Ant and Ivy to publish her artefacts in an Artifactory in Ivy repos, while Bob sits behind a firewall and builds Maven artefacts storing them in Maven repos in Nexus.

How can Alice provide her artefacts to Bob? (while both do not have to make large changes to their setup)

So far, I have considered the following options and decided to ask you before trying them out:

(1) Bob manages credentials to access Alice' artefacts in his build system and uses a Maven plugin for Ivy (which means configuration work for all relevant projects)

(2) Alice uses an Ant/Maven *task* to generate POMs for *publishing/deployment* step and offers an additional Maven repo, because Artifactory can support it different repo formats. This for Bob's most convenience though Alice needs to do configuration work of all relevant projects and there might be also other Bobs in future using other *binary repository managers*.

(3) Bob configures Nexus to be a proxy for Alice' Ivy repo (which is AFAIK was not possible as of 2014).

What would you do?

# Answer
> 1 votes
I'd go with (2). Since Bob has a more rigid setup (requires Maven metadata), and Alice can provide that (using more flexible Ant), that sounds like the easiest way to go.

---
Tags: maven, artifactory, ant, interoperability
---


# Question
Title: All volumes are deleted when EC2 intance is deleted via terraform
I am using a terraform template to create AWS resources. My count is 2, everything gets created as per plan, and, I am attaching a 2nd EBS volume of 1 GB to both machines, which is also happening well.

However, the only issue is when I try to delete one EC2 instance using the command below, both of the 2nd EBS volumes of 1 GB are getting destroyed. I checked they are connected on separate instances.

```
$ terraform destroy -target=aws_instance.jumpserver[1]
aws_vpc.main_vpc: Refreshing state... (ID: vpc-06b59734024ad6adc)
aws_key_pair.ProdKeypair: Refreshing state... (ID: ProdKeypair)
aws_security_group.sg_internet_facing: Refreshing state... (ID: sg-05a2739733f4f8a32)
aws_subnet.public_subnet: Refreshing state... (ID: subnet-0a8c6ea2718a44224)
aws_instance.jumpserver[1]: Refreshing state... (ID: i-05646d53baa34a988)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - aws_ebs_volume.vol_generic_data[0]

  - aws_ebs_volume.vol_generic_data[1]

  - aws_instance.jumpserver[1]

  - aws_volume_attachment.generic_data_vol_att[0]

  - aws_volume_attachment.generic_data_vol_att[1]

```

This is the main.tf:

```
    # Define webserver inside the public subnets
    resource "aws_instance" "jumpserver" {
      count                       = "${var.num_of_instances}"
      ami                         = "${var.ami}"
      instance_type               = "t2.micro"
      key_name                    = "${aws_key_pair.ProdKeypair.id}"
      subnet_id                   = "${aws_subnet.public_subnet.id}"
      vpc_security_group_ids      = ["${aws_security_group.sg_internet_facing.id}"]
      associate_public_ip_address = true
      source_dest_check           = false
      #   user_data = "${file("install.sh")}"

      root_block_device = {
        volume_type           = "gp2"
        volume_size           =  "8"
        delete_on_termination = "${var.delete_on_termincation}"
      }

      tags {
        Name = "${format("jump-%01d",count.index+1)}"
      }

      provisioner "remote-exec" {
        inline = ["sudo apt-get  -y install python"]

        connection {
          type        = "ssh"
          user        = "ubuntu"
          private_key = "${file(var.private_key_path)}"
        }
      }
    }

    resource "aws_ebs_volume" "vol_generic_data" {
      size              = "1"
      count             = "${var.num_of_instances}"
      type              = "gp2"
      availability_zone = "${element(aws_instance.jumpserver.*.availability_zone, count.index)}"

     tags = {
        Name = "${format("jump-%01d",count.index+1)}"
      }
    }

    resource "aws_volume_attachment" "generic_data_vol_att" {
      device_name = "/dev/xvdf"
      volume_id   = "${element(aws_ebs_volume.vol_generic_data.*.id, count.index)}"
      instance_id = "${element(aws_instance.jumpserver.*.id, count.index)}"
      count       = "${var.num_of_instances}"
    }

    # Define webserver inside the private subnet
    resource "aws_instance" "backendserver" {
      ami                         = "${var.ami}"
      instance_type               = "t2.micro"
      key_name                    = "${aws_key_pair.ProdKeypair.id}"
      subnet_id                   = "${aws_subnet.private_subnet.id}"
      vpc_security_group_ids      = ["${aws_security_group.sg_backend.id}"]
      associate_public_ip_address = false
      source_dest_check           = false
      user_data                   = "${file("install.sh")}"

      tags {
        Name = "backendserver"
      }
    }

```

# Answer
I could see some spelling mistakes in the below mentioned line, rest all are fine i think.

```
delete_on_termination = "${var.delete_on_termincation}"

```

Better you could avoid this line, This might be deleting the EBS when u terminating the ec2 instance. Skip this option and check.
> 3 votes

# Answer
There is an bug reported on TF AWS provider that is certainly related (terraform-provider-aws/issues/83). Unfortunately, there is no solution for it except waiting to see if the bug still exists with the config language improvements expected in TF 0.12.

What is your use case to delete this instance and ebs with `terraform destroy -target=aws_instance.xxx[1]` command? Do you want to recreate this instance in a second step or stay with only one instance?

What I would try as a workaround is tainting the instance and ebs volume you want to delete with `terraform taint aws_instance.xxx.1 && terraform taint aws_ebs_volume.xxx.1`. Then if you want to recreate them, just run `terraform apply` or if you want to stay with one instance run `terraform apply -var 'count=1'`
> 1 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: Jenkins Pipeline project can't when on branch
I want to use `when` on `branch` condition. The Jenkinsfile is like this

```
 stages{
        stage('Checkout project branch master'){
            steps{
                withCredentials([sshUserPrivateKey(credentialsId: 'credential', keyFileVariable: 'CREDENTIAL')]) {}
                git branch: 'develop', credentialsId: 'credential', url: 'git@bitbucket.org:user/demopipelinejenkinsfile.git'
            }
        }
        stage('develop'){
             steps {
                   echo 'on branch develop`
             }
        }
        stage('any_branch'){
             environment {
                   GIT_BRANCH = sh(returnStdout: true, script: 'git rev-parse --abbrev-ref HEAD').trim()
                   }
             steps{
                   echo "${env.GIT_BRANCH}"
                  }
        }
 }

```

The console output shows that the stage `develop` is skipped due to condition. However, the last stage still echoes the current branch is "develop"

**Question** How to use `when` with `branch` in **Pipeline** project (not **Multibranch** project)

# Answer
This is the way to go:

```
stage('Deploy'){
  when {
    expression {env.GIT_BRANCH == 'origin/master'}
  }
  steps {
    ....
   }
}

```

Take care, this is only working with the declarative syntax.

The environment step is used to "set up the environment" meaning this is the place to declare environmental variables.

Also, in my case I did not declare the GIT\_BRANCH var myself. But jenkins provides it as an environmental (therefore "env.GIT\_BRANCH") variable if you did "checkout scm" to checkout your repository. If you want to check all the available env vars, do this as a step: `sh 'printenv'`

check out this article for more info: https://jenkins.io/blog/2017/01/19/converting-conditional-to-pipeline/
> 4 votes

---
Tags: jenkins, jenkins-pipeline, jenkinsfile
---


# Question
Title: Hosted CI/CD for iOS that's not in the US?
**TL;DR: Do you know of any hosted CI/CD providers for iOS, who have their *datacenter/build-boxes* in Asia, or at least Europe?** (Bonus points if they provide both build and deploy, but build is the MVP.)

---

Back-story:

We're running CI/CD at-scale for iOS and Android: we run 10+ simultaneous builds both to test/verify our Merge-Request branches and to deploy trunk builds to our testers and stakeholders. We're using a SaaS/cloud provider we're pretty happy with... except for their location.

We're in Asia, as is our source. Client is in a highly-regulated industry and fighting hard to keep source on-prem, as they think their regulator can't handle source in the cloud yet. Please accept this premise. I understand why they need to let go of it. But for now... assume they can't.

What that means is: source is in Asia, but the CI/CD providers who build it are all seemingly in the US (Circle, Buddybuild, etc). Trans-pacific bandwidth sucks, particularly during the Asia business day. **The time taken by every clone before every build spikes to \>60 minutes for much of the business day.**

Going on-prem for the CI/CD would be pretty easy for Android builds, on docker containers. But iOS is the problem. You either need to put yourself in the business of teaching people to admin OSX and keep a fleet of build-boxes operational, or you need to let specialists solve that problem for you.

---

NOTES:

* I am ***not*** asking for recommendations, SE community! This is a factual technical question: are certain services, that meet certain technical requirements, available in a certain geography?
* We know that MacStadium can provide bare MacOS VM's in Ireland. But that means you have to admin your own whole CI process, plus a lot of low-level system admin tasks we'd rather avoid. It also means separating build from deployment, for sure. But the latency appears to be acceptable.
* We know of other people who have cloud CI/CD platforms closer to us... but without iOS/MacOS support.
* We know that shallow clones require less bandwidth, which would mitigate the problem, but they also have other complications that mean our current provider doesn't support it yet. And they don't entirely solve the problem in any case.
* We've experimented with using off-premise GitHub mirror, which solves some of the problem, but doesn't address the regulatory issues. It also doesn't work with many webhooks, and specifically the Merge-Request webhooks that are critical to our CI pipeline for new code. We could create an agent that would monitor the webhooks, then imperatively relay API commands to the other service-providers... but that's really jumping through hoops, plus we'd have created substantial new code to maintain.

# Answer
One could run Jenkins on MacOS.

https://medium.com/@dixitakansha15/continuous-integration-in-ios-using-jenkins-8e081753cc72

https://savvyapps.com/blog/continuous-integration-ios-jenkins
> 2 votes

# Answer
Have you seen https://buildkite.com/ and especially this post for IOS https://medium.com/@Pinterest\_Engineering/continuous-integration-for-ios-with-nix-and-buildkite-ef5b36c5292d? The company is based in Australia I believe.

My company uses Buildkite and I actually like it a lot (we don't build anything IOS/Mac); you do need to setup build agent which can be hosted on cloud provider or even bare metal. The CI/CD non-build server itself is hosted. It supports I think all of Github webhook.
> 2 votes

# Answer
Issue here is that price for bandwidth in APAC 10 times higher than US, so probably you should multiply Buddybuild prices by 10 and ask them enterprise support keeping result numbers in you mind. I just did a calculations intended to provide you service and I got around 8k/month for 10 sim builds. Can pm me if you are interested.
> 1 votes

---
Tags: continuous-integration, continuous-deployment, ios, software-as-a-service
---


# Question
Title: Is openstack public cloud like Google drive, aws where we can store data?? Or something else?
I am using cloudberry to backup data. There are lot of options where we can store data like Google drive , aws and also openstack. I got confused because I was thinking openstack as operating system which we can install in our premise..please someone explain..what's openstack actually??

# Answer
> 1 votes
Openstack is something you might install to setup an self-service IaaS platform that manages VMs, storage and network. Whereas the top three cloud providers have written their own proprietary IaaS platforms a small cloud provider, or a large company, cannot feasibly code there own cloud-like management software, so they might setup their “private cloud” using openstack (or, say, a big vendors distribution of it) so that many internal projects can use it as self-service IaaS. To quote Wikipedia with some expansion by myself:

> OpenStack is a free and open-source software platform for cloud computing, mostly deployed as infrastructure-as-a-service (IaaS), whereby virtual servers and other resources are made available to \[internal or external\] customers. The software platform consists of interrelated components that control diverse, multi-vendor hardware pools of processing, storage, and networking resources throughout a data center. Users either manage it through a web-based dashboard, through command-line tools, or through RESTful web services.

People might talk about it being “your operating system for private cloud” which is a bad analogy. In general the term “operating system” is often deliberate misused as it has positive connotations and might indicate to none technical people the broad benefits of investing in a given technology. For example people might also talk about container orchestrators such as Kubernetes or Docker Swarm being “the new operating systems of the cloud” which is a similar poor analogy.

If a backup project said it provides connectors for AWS S3, Azure Blob, Google Cloud Storage, and Openstack Swift then it would be more clear exactly what it does: write large binary backup files into a cloud store. Such a product could then accurately say it supports AWS, Azure, GCP, and Openstack. Just to be clear I am not saying the the technology you are looking at actually uses those APIs I am making a general point about openstack having feature parity with many public cloud technologies.

---
Tags: azure-devops, cloud, openstack
---


# Question
Title: Terraform | AWS | vpc_id causing apply to fail
I am working on setting up a Cloud9 environment using Terraform, I ran into an issue when it comes to my Terraform apply and specifically:

```
Error: Error applying plan:

1 error(s) occurred:

* aws_instance.cloud9Test: 1 error(s) occurred:

* aws_instance.cloud9Test: Error launching instance, possible mismatch of Security Group IDs and Names. See AWS Instance docs here: https://terraform.io/docs/providers/aws/r/instance.html.
        AWS Error: Value () for parameter groupId is invalid. The value cannot be empty

```

Here is my Terraform code that seems to be related and causing the issue.

```
resource "aws_vpc" "cloud9Test" {
  cidr_block       = "10.0.0.0/16"
  instance_tenancy = "dedicated"

  tags = {
    Name = "cloud9Test"
  }
}

resource "aws_security_group" "cloud9Test" {
    name = "cloud9Test-sg"
    description = "Cloud9 Group"
    vpc_id      = "${aws_vpc.cloud9Test.id}"
}

```

When I comment out the vpc\_id in the security group I don't have the issue anymore but I do want the security group inside my VPC.

How do I fix this?

Note I do not have anything that should overlap causing an issue, this is a fairly new account with only 3 ec2 instances and 1 cloud9 instance all created manually

# Answer
> 1 votes
I'll go out on a limb here and assume that the issue is actually in your "aws\_instance" block and is likely the result of how you are assigning the security groups in the "aws\_instance".

Copied from the first linked article, in the resolution:

```
change

security_groups = ["cloud9Test"]

to

vpc_security_group_ids = ["${aws_security_group.cloud9Test.id}"]

```

Two related articles:

https://github.com/KainosSoftwareLtd/aws-api-gateway-demo/issues/3

https://stackoverflow.com/questions/31569910/terraform-throws-groupname-cannot-be-used-with-the-parameter-subnet-or-vpc-se/34586893#34586893

If this is not on track, please post the "aws\_instance" declaration.

For your own sanity I'd also recommend giving the resources names that are more specific to their role, and not identical, for example "cloud9Test\_vpc".

---
Tags: amazon-web-services, terraform
---


# Question
Title: I trying on Windows 10, Trying to connect / login to work interactively on container but I'm getting this kind of error
I trying on Windows 10, Trying to connect / login to work interactively on container but I'm getting this kind of error

```
docker run --rm -p 8080:80 laravel-docker
 Error response from daemon: driver failed programming external connectivity on endpoint wizardly_diffie (0c95bf862e583179e0a988eae0b07606e41184548c494ddd523df95c1430422f): Error starting userland proxy: mkdir /port/tcp:0.0.0.0:8080:tcp:172.17.0.8:80: input/output error.

```

How to resolve this.

my docker version

```
$ docker --version
Docker version 18.09.0, build 4d60db4

```

# Answer
> 1 votes
Restart docker, it falls into this state after the OS reboots.

The below thread mentions running it a swarm as preventing the issue from reoccurring and several ways to resolve the underlying OS issue with the software that may, or may not be workable for you.

https://github.com/docker/for-win/issues/1038

---
Tags: docker, php, docker-machine
---


# Question
Title: How to monitor a micro-service oriented solution?
I run a micro-service oriented system, running under a private network, composed mainly of:

* databases (clusters)
* internal API's
* external/public API's
* workers (services)
* monitoring servers

All are running Debian (7/8) and mainly using nginx and supervisor.

I'm currently monitoring logs using ELK stack. But I do want to have a monitoring/alerting system to include the following:

* internal/public API's (calls)
* databases status (only basic, to know when the cluster does not have issues)
* ram/cpu for the servers

What do you recommend to use? I'm more open to open-source solution, but also can pay for services that are mature (not interested in startup solutions)

# Answer
> 2 votes
Prometheus is a popular tool for monitoring microservices deployed to the cloud in combination with Grafana. For a better understanding of your inter-service issues, a distributed tracing tool is needed as well.

---
Tags: monitoring, microservices, software-as-a-service
---


# Question
Title: Recommended JFrog Artifactory Layout setup for Maven Projects
We are very new in using Artifactory services, we would like to store all our artifacts in JFrog Artifactory server for dependency as well as for release management. I have currently created one Local Repository in the name 'myartifactory' and I am planning to use the following code in my projects pom.xml file for distribution management.

```
<distributionManagement>
    <repository>
        <id>central</id>
        <name>jd-releases</name>
        <url>http://jd-msg.staging.com/artifactory/myartifactory</url>
    </repository>
    <snapshotRepository>
        <id>snapshots</id>
        <name>jd-snapshots</name>
        <url>http://jd-msg.staging.com/artifactory/myartifactory</url>
    </snapshotRepository>
</distributionManagement>

```

Does the above method looks like a recommended way? How about having "Remote Repositories" and "Virtual Repositories"? Is it mandatory to have? What is the real life example for those?

In addition artifactory by default recommends some best practices for Local Repositories like the following,

Pre-defined Local Repositories: \- Artifactory comes with a set of pre-defined local repositories, which reflect best practices in binary repository management as follows:

```
libs-release-local - Your code releases
libs-snapshot-local - Your code snapshots
ext-release-local - Manually deployed 3rd party libs (releases)
ext-snapshot-local - Manually deployed 3rd party libs (shapshots)
plugins-release-local - Your and 3rd party plugins (releases)
plugins-snapshot-local - Your and 3rd party plugins (snapshots)

```

So I am bit confused about the right approach. Suggestions are welcome.

# Answer
> 0 votes
Within Artifactory there are three types of repositories:

1. **Local**: Local repositories are physical, locally-managed repositories into which you can deploy artifacts
2. **Remote**: A remote repository serves as a caching proxy for a repository managed at a remote URL
3. **Virtual**: A virtual repository (or "repository group") aggregates several repositories with the same package type under a common URL.

Local and remote repositories are true physical repositories (they exist either on your server on on a remote server somewhere else), while a virtual repository is actually an aggregation of them used to create controlled domains for search and resolution of artifacts.

If you need all of those types depends a bit on your use case. A real world use case for a remote repository would be Maven Central or jCenter where developers host their Maven packages. JFrog Artifactory can serve as a cache for those services so that not every call to resolve Maven dependencies has to go through the Internet. A real world use case for a virtual repository is that I have a remote repo (like the ones I already mentioned) and a few local repos too (like the ones you mentioned in your question). I want to make sure that my developers only need to configure one Maven repository as URL and my virtual repository allows me to do that. With a virtual repo, I can group together all sorts of other repositories and give one URL to developers so I can manage which dependencies are allowed as opposed to resolving them from the Internet every time.

Based on lessons learned we've included some default repositories into Artifactory that follow the flow of most developers that use Maven. Having said that, there is nothing stopping you from adopting your own workflow and not using the default repos.

---
Tags: artifactory
---


# Question
Title: Ansible how to know what return value is provided by a specific module
From Ansible's official doc

> Ansible modules normally return a data structure that can be registered into a variable, or seen directly when output by the ansible program

and

> Ansible modules normally return a data structure that can be registered into a variable, or seen directly when output by the ansible program

However, the document on aws\_ec2 module

1. Doesn't document return values
2. Has this example, which seems like it actually has some specific return values

```
# Stripped version

# Launch instances, runs some tasks
# and then terminate them

- name: Create a sandbox instance
  ... # hosts, vars...
  tasks:
    - name: Launch instance
      ec2:
      ... # ec2 module specs
      register: ec2 #This seems to be one of the specific return structure by this module

    - name: Wait for SSH to come up
      ... # some module
      with_items: "{{ ec2.instances }}" # This is where the registered return value is used

```

Question: How to see all possible return values of a module, even without documentation

# Answer
if you do something like:

```
- name: Launch instance
  ec2:
  register: data_struct

```

You can then output all the values returned by doing this below the above code:

```
- name: print all returned values
  debug:
    msg: "{{ data_struct }}"

```
> 10 votes

---
Tags: ansible
---


# Question
Title: Multiple Application in single SSL 443 server NGINX
I have two application served in a single server `app1.domain.com` and `app2.domain.com`.

I want to proxy pass the request from `app1` to serve static file and `app2` to `proxy_pass` `localhost:8000`

I already have both SSL certificate using letsencrypt, how can I serve both apps using a single server?

I thought I can use different `server_name`. I read from here: the article use multiple NGINX `server` with the same port 80. But when I try for the port 443 the test fails "duplicate listen options for..". Here is my current config

```
server {
    server_name app1.domain.com;

    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/app1.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app1.domain.com/privkey.pem;

    index index.html;
    root /usr/share/nginx/html/;

    location ~* \.(css|js|png|jpg|jpeg)$ {
        try_files $uri $uri/ =404;
        gzip_static on;
    }

    location / {
        try_files $uri /index.html =404;
        gzip_static on;
    }
}

server {
    server_name app2.domain.com;

    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/app2.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/app2.domain.com/privkey.pem;

    index index.html;
    root /usr/share/nginx/html/;

    location / {
        proxy_set_header  Host $host;
        proxy_set_header  X-Real-IP $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header  X-Forwarded-Proto $scheme;

        # Fix the “It appears that your reverse proxy set up is broken" error.
        proxy_pass    http://localhost:8000/;
        proxy_read_timeout  90;
    }
}

server {
    if ($host ~ "app1.domain.com|app2.domain.com") {
        return 301 https://$host$request_uri;
    }

    listen 80 default_server;
    listen [::]:80 default_server;

    server_name domain.id;
    return 404;
}

```

And I thought maybe I could use `if ($host = domain)` inside the `location` but it fails too

# Answer
> 1 votes
Oops, turns out I just have to remove the line `listen [::]:443 ssl ipv6only=on;` and it works.

it's leftover from running NGINX certbot, I thought that was important

---
Tags: nginx
---


# Question
Title: Connecting jaeger with elasticsearch backend storage on kubernetes cluster
I have a kubernetes cluster on google cloud platform, and on it, I have a jaeger deployment via development setup of jaeger-kubernetes templates because my purpose is setup `elasticsearch` like backend storage, due to this, I follow the jaeger-kubernetes github documentation with the following actions

Here are configured the URLs to access to `elasticsearch` server and username and password and ports

```
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production-elasticsearch/configmap.yml

```

And here, there are configured the download of docker images of the elasticsearch service and their volume mounts.

```
kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/production-elasticsearch/elasticsearch.yml

```

And then, at this moment we have a elasticsearch service running over 9200 and 9300 ports

```
 kubectl get service elasticsearch                                                                                [a89fbe2]
NAME            TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
elasticsearch   ClusterIP   None         <none>        9200/TCP,9300/TCP   1h

```
* I've follow with the creation of jaeger components using the `kubernetes-jaeger` production templates of this way:

---

```
λ bgarcial [~] → kubectl create -f https://raw.githubusercontent.com/jaegertracing/jaeger-kubernetes/master/jaeger-production-template.yml
deployment.extensions/jaeger-collector created
service/jaeger-collector created
service/zipkin created
deployment.extensions/jaeger-query created
service/jaeger-query created
daemonset.extensions/jaeger-agent created

λ bgarcial [~/workspace/jaeger-elastic] at  master ?

```

According to the Jaeger architecture, the `jaeger-collector` and `jaeger-query` services require access to backend storage.

And so, these are my services running on my kubernetes cluster:

```
λ bgarcial [~/workspace/jaeger-elastic] at  master ?
→ kubectl get services                                                                                [baefdf9]
NAME               TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                        AGE
elasticsearch      ClusterIP      None            <none>           9200/TCP,9300/TCP              3h
jaeger-collector   ClusterIP      10.55.253.240   <none>           14267/TCP,14268/TCP,9411/TCP   3h
jaeger-query       LoadBalancer   10.55.248.243   35.228.179.167   80:30398/TCP                   3h
kubernetes         ClusterIP      10.55.240.1     <none>           443/TCP                        3h
zipkin             ClusterIP      10.55.240.60    <none>           9411/TCP                       3h

λ bgarcial [~/workspace/jaeger-elastic] at  master ?

```
* I going to `configmap.yml` elastic search file `kubectl edit configmap jaeger-configuration` command in order to try to edit it in relation to the elasticsearch URLs endpoints (may be? ... At this moment I am supossing that this is the next step ...)

I execute it:

```
λ bgarcial [~] → kubectl edit configmap jaeger-configuration

```

And I get the following edit entry:

```
 apiVersion: v1
data:
  agent: |
    collector:
      host-port: "jaeger-collector:14267"
  collector: |
    es:
      server-urls: http://elasticsearch:9200
      username: elastic
      password: changeme
    collector:
      zipkin:
        http-port: 9411
  query: |
    es:
      server-urls: http://elasticsearch:9200
      username: elastic
      password: changeme
  span-storage-type: elasticsearch
kind: ConfigMap
metadata:
  creationTimestamp: "2018-12-27T13:24:11Z"
  labels:
    app: jaeger
    jaeger-infra: configuration
  name: jaeger-configuration
  namespace: default
  resourceVersion: "1387"
  selfLink: /api/v1/namespaces/default/configmaps/jaeger-configuration
  uid: b28eb5f4-09da-11e9-9f1e-42010aa60002

```

Here ... do I need setup our own URLs to collector and query services, which will be connect wiht elasticsearch backend service?

How to can I setup the elasticsearch IP address or URLs here?

In the jaeger components, the query and collector need access to storage, but I don't know what is the elastic endpoint ...

Is this `server-urls: http://elasticsearch:9200` a correct endpoint?

I am starting in the kubernetes and DevOps world, and I appreciate if someone can help me in the concepts and point me in the right address in order to setup jaeger and elasticsearch as a backend storage.

# Answer
> 1 votes
When resolving a service in Kubernetes,

> Assume a Service named foo in the Kubernetes namespace bar. A Pod running in namespace bar can look up this service by simply doing a DNS query for foo. A Pod running in namespace quux can look up this service by doing a DNS query for foo.bar.

Since Jaeger is in the same namespace, then yes, it can resolve the Elasticsearch endpoint with just `elasticsearch`. Your configuration is correct in that respect and requires no changes.

If Jaeger were located in another namespace, you could specify the full DNS record `elasticsearch.default.svc.cluster.local`.

---
Tags: kubernetes, elastic, storage
---


title: container on same network with host

text:
# Question
Title: container on same network with host
Our application is a combination of 4 different applications. The graphics part is running on a host and other 3 applications are running on 3 different **VMs**. Which uses bridge networking.

```
eg. Graphics app1 - 192.168.2.201 (Base Machine/HOST)
    Application 2 - 192.168.2.202 (VM1)
    Application 3 - 192.168.2.203 (VM2)
    Application 4 - 192.168.2.204 (VM3)

```

Which is running fine.

But, to get advantage of containers, we want to shift our VM applications into docker container. But in docker, when we are using Bridge Networking, it doesn't take IPs in HOST network, obviously because, Bridge network ID is not same as host. And application fails to run.

How can we achieve the Bridge Networking with same network ID (say 192.168.2.0)? We cannot use Host networking.

Note: our application specifically wants IP in 192.168.2.0 network, otherwise it fails.

# Answer
I could have mentioned this in comment first but I need 50 reputation to comment and I just joined this yesterday,

So, may I know what you are trying to achieve? If I interpret correctly, you want to move your VMs to Docker and then make those docker interact with each other as well as with graphics which is running on host? So consider it this way, you have 4 IP addresses, how would you make the communicate if it is simple LAN? Similarly you do it here. Allow hosts from that IP to work here. You can customize docker0 bridge to have gateway and CIDRs of your choice by modifying daemon.json

Customize the Docker0 bridge

You can create your user defined networks too.

May be this will help you too? Docker Networking User Guide

You can even write your own network plugin.
> 1 votes

---
Tags: docker, containers, networking
---


# Question
Title: How do Django deployments happen on Servers?
I am new to Django six months ago and I learned how to deploy the Django projects to EC2 instances.

Basically 1st time when the Django project is deployed to EC2, the instance will be configured with httpd.conf, wsgi, change permissions on files and folders and other stuff and the project will be cloned to the EC2 instance and server is restarted.

My question is: how do they do future deployments? They in this context is anyone who deploys Django on EC2 instances.

Do they login to EC2 instances and manually clone the repository from VCS site and restart the server?

Do they have any other automated mechanism to pull the code, ensure permissions and restarting the apache server etc.

How is it done basically every time they go for a release?

# Answer
Probably, you want need to go through something like Ansible first. Nothing is done manually these days, not even launching of EC2 instances or creating a VPC is manual and if it is, it needs to be changed immediately. You may also go through Terraform. You also need to understand CI/CD flow. So when any changes are pushed to git, a CI mechanism will execute which can include build, compile, unit test execution, integration test execution, code coverage reports etc which are all automated. Deployment can either be automated if CI is successful or on click of a button. Ansible can take care of the configuration changes which needs to be deployed including the application code changes. In fact, if you are in AWS, why don't you go for docker and deploy app in Docker using benefits of Amazon ECS. A Terraform code can do zero downtime deployments for you. This is not specific to Django. It is used for any tech stack.
> 1 votes

---
Tags: amazon-web-services, deployment, amazon-ec2, continuous-deployment, python
---


# Question
Title: Best way to launch 100 identical Docker machines on AWS?
I need to do some large scale testing on a system. The system consists of 2 main parts: Manager and Nodes. There will be only one manager server, but 100 nodes. Both the manager and nodes are Docker images.

I want to be able to spin up a setup on AWS, where an instance of a manager, and 100 instances (one instance per server) of the node is started.

What would I use for this? The manager and nodes should be able to communicate together, and the nodes should not be load balanced in any way. So they need a dedicated internal IP that the manager can communicate with.

I've looked at ECS and CloudFormation+EC2 with a custom AMI, but I'm not sure what is the way to go.

Ideally I want a script that I once in a while can run to launch the servers, and close them down easily.

Any suggestions on what I can do on AWS to launch this setup with ease?

# Answer
Look at **AWS Fargate** \- it lets you run your Docker containers without having to spin up the EC2 instances for an ECS cluster. You simply schedule the Tasks (or Services), each gets is own IP address and they can talk to each other.

You can also use **AWS Service Discovery** to facilitate the registration and lookup of the tasks' IP addresses.

BTW Note that the default Fargate concurrency limit is 20 Tasks/Services - if you need 100 of them you'll have to raise a Service Limit increase support ticket.

We've got it raised to 200 concurrent tasks because our processing runs 160+ containers in parallel, all on Fargate without a single EC2 to manage.

### Update - how to spin up 100 containers

I can tell you how *we* do it with Fargate - in our CI/CD pipeline we build the container image and upload to ECR. Part of the source is also a *CloudFormation (CFN) template* and in the *Deployment* stage of the pipeline we create/update CFN stack which creates **ECS TaskDefinition** with all the container parameters.

Then we've got a *scheduler* job written in Python and triggered by **CloudWatch Event** (cron-like scheduler) that looks up the *TaskDefinition ARN* and runs 160+ *Tasks* from that single TaskDef. You can give each Task an extra parameter, or override some config, etc. Or not and just spin it up.

We used to use EC2/ECS and had to do an extra step first - *scale out* the ECS AutoScaling group to tens of nodes to support our workload and then at the end scale it back in to 0. Other than that it's the same whether you run it on Fargate or on EC2/ECS.

Hope that helps :)
> 5 votes

# Answer
Why don't you use Terraform for the purpose? It can launch as many instances you want, create network for you, create security groups, ECS, IAMs or whatever you want to create and terminate easily when you want. This is widely used in industry now.
> 2 votes

---
Tags: docker, amazon-web-services, amazon-ec2, amazon-ecs, aws-cloudformation
---


# Question
Title: No Checkbox to" Restart Jenkins when installation is complete".Unable to restart Jenkins post plugin download
I have set up Jenkins version (ver. 2.155).

post adding plugin, i need to restart Jenkins but "Restart Jenkins" tab or link not working. I don't find my \[\] checkbox to check/tic click on arrow is not working (Refer image). Any help will be appreciated

# Answer
To restart Jenkins manually, you can use either of the following commands (by entering their URL in a browser):

`(jenkins_url)/safeRestart` \- Allows all running jobs to complete. New jobs will remain in the queue to run after the restart is complete.

`(jenkins_url)/restart` \- Forces a restart without waiting for builds to complete.
> 3 votes

# Answer
You can restart jenkins by visiting `/restart`. So if your url is `https://my-jenkins.local` you would visit `https://my-jenkins.local/restart`.

It's been answered many times before, so maybe search around next time ;). Hope that helps ya.
> 0 votes

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: Transfer files and execute commands from jenkins scripted pipeline to AWS instance
I want to transfer files and execute a few commands on remote AWS instance from Jenkins. I have the following methods in mind to do the same. Note that my Jenkins server is running in the local server, and I don't have SSH access to Jenkins

1) Using hardcoded pem file I have following scripted pipeline to transfer files from Jenkins pipeline to AWS EC2 instance.

```
stage('transfer docker image to ec2') {
  sh 'scp -oIdentityFile=/var/lib/jenkins/.key.pem docker_image.tar ubuntu@ec2-3-83-239-32.compute-1.amazonaws.com:'

```

However it is not working, and waiting for a long time without showing logs.

2) Using Jenkins "credential"

I have added the pem file in Jenkins credential, and then invoked the credential by the following method. (Referred from - https://stackoverflow.com/a/47627460/9403545)

```
withCredentials([sshUserPrivateKey(credentialsId: "bindu-test", keyFileVariable: 'keyfile')]) {
 stage('transfer docker image to ec2') {
  sh 'scp -i ${keyfile} test ubuntu@ec2-x-83-xxx-32.compute-1.amazonaws.com:'

```

Unfortunately, the above method also didn't work and got the following error.

> /tmp/../Hello directory doesn't exist.

I have tried to use Publish over SSH, but it doesn't seem to be supported in scripted pipeline.

Is there any way I can do this? P.S: Please keep in mind that I'm AWS newbie, it would be nice if the answer is in detail :)

# Answer
Below is sample groovy code for sending file or execute command over SSH in jenkins:

```
node(master)
{
  stage(Deploy)
  {
     sshPublisher(publishers: [sshPublisherDesc(configName: 'SERVER_NAME', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: 'apt-get update', execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '*.war')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
  }
}

```
> 2 votes

# Answer
You can use '**Publish Over SSH plugin**'. Using this plugin you can send files and execute the command on the remote server.

Click Here For install plugin.

# Configure

1. Click “Manage Jenkins”
2. Click “Configure System”
3. Go to “Publish over SSH” section
4. Enter “/Users/Shared/Jenkins/.ssh/id\_rsa” to “Path to Key”
5. Click “Add” at “SSH Servers”
6. Enter any logical name to “Name”
7. Enter IP Address or Hostname of the server to “Hostname”
8. Enter the user name to log in to “Username”
9. Enter any directory to “Remote Directory”
10. Click “Test Configuration”
11. Click “Save” at bottom of the page

# Create a job

1. Create or copy a job that can build the WAR correctly.
2. As I wrote in How to specify a Git tag to be processed, make a job to can specify a tag to be processed.
3. Click “Add post-build action”
4. Click “Send build artifacts over SSH”
5. Enter “Source files”
6. Enter “Remove prefix”
7. Enter “Exec command”

For More reference:

1. https://nozaki.me/roller/kyle/entry/articles-jenkins-sshdeploy
2. https://medium.com/@weblab\_tech/how-to-publish-artifacts-in-jenkins-f021b17fde71
3. https://wiki.jenkins.io/display/JENKINS/Publish+Over+SSH+Plugin

you can also do the same thing in pipeline script, see below images:

```
node(master)
{
  stage(Deploy)
  {
     sshPublisher(publishers: [sshPublisherDesc(configName: 'LAB-35', transfers: [sshTransfer(cleanRemote: false, excludes: '', execCommand: 'apt-get update', execTimeout: 120000, flatten: false, makeEmptyDirs: false, noDefaultExcludes: false, patternSeparator: '[, ]+', remoteDirectory: '', remoteDirectorySDF: false, removePrefix: '', sourceFiles: '*.war')], usePromotionTimestamp: false, useWorkspaceInPromotion: false, verbose: false)])
  }
}

```

Here configName: 'LAB-35' is your remote ssh server.
> 1 votes

---
Tags: jenkins, amazon-web-services, jenkins-pipeline, amazon-ec2
---


# Question
Title: Automatic Kubernetes deployments per Github pull-request
I'm looking for a tool/service that will achieve a similar result as Heroku Review Apps using Google services.

When a new pull-request is created in Github:

* Create new Cloud Build trigger for it with a given template configuration.
* When done with build, create new deployment for it in Kubernetes Engine given template configuration.
* Provide the public URL in the Github PR

When PR is closed:

* Remove trigger, container and deployment
* Remove public URL from Github pull-request.

# Answer
I don’t know of a specific tool but I have setup a lot of webhook handlers on OKD kubernetes to do git driven updates to kubernetes. All the scripts are on GitHub as OCD. The tools I used are:

1. adnanh/webook a go binary that you configure to catch git webhooks and run scripts. Here is my hooks.json config to match GitHub release events and run a build script on openshift kubernetes. Looking at the docs there are PR events you can match on. So you should be able to setup Webhooks to fire on PRs and trigger s2i builds.
2. hub is GitHubs official command-line tool. I put it into my tools image that runs my webhooks. It has some pull request commands. I haven’t checked them against your stated requirements. Hub is just a wrapper to their API so worst case you would need to use curl to update PRs.
3. Helm is the installer/template engine I use for creating apps. Helmfile is an excellent extension. So here is a generic “deployer” helm chart that can spin up a deployment. So OCD responds to GitHub events of new Helmfile declarative configuration being pushed and deploys or updated any helm releases modified in git.

For builds I had to do quite a bit to have an OKD container build tag the final container image with the GitHub release tag which is a generic “ocd-release-build” helm chart. I think that is necessary as each PR will need by tagged with the PR number in the container registry to deploy from that specific tag. I haven’t ever tried to get OKD to build a `refs/pull` but I would be shocked it didn’t work as expected.

So while OCD doesn’t yet spin up builds and deploys based on PR events it appears is likely that it could be extended to do that. We have migrated our apps from our in-house scripts over to OCD. A “PR previews” feature could be a very cool feature to add. Feel free to raise a feature request.
> 1 votes

# Answer
I have not try it myself yet, but k8s-deploy looks promising.

Other potential products that I have not tried yet:

My aim is to evaluate these tools and I hope that you could try the same and reply to this post. As soon as I find out what tool works nice I will let you know.
> 0 votes

---
Tags: kubernetes, google-cloud-platform
---


# Question
Title: How to create high availability databases?
With modern techniques, it's relatively trivial to create a clustered, high-availability solution for webservers by using docker (or comparable) to handle the front-end serving of pages to users.

e.g. for a LAMP stack, docker would handle the "A" and "P" parts.

Is there any way to make the database (e.g. mysql) work in a similar high-availability system with docker or comparable.

# Answer
There are docker images for running opensource databases in high availability mode such as https://info.crunchydata.com/blog/an-easy-recipe-for-creating-a-postgresql-cluster-with-docker-swarm

You can also find examples of Microsoft SQLServer container for Linux being run with high availability on Kubernetes https://docs.microsoft.com/en-us/sql/linux/tutorial-sql-server-containers-kubernetes

Your question is missing an important concept from the devops perspective. If we run a particular version of a database (MySQL, PostresQL, SQLServer) on either a physical server, or a VMware ESX virtual machine server, or an openstack Xen virtual server, or in Docker in a laptop or in an http://cri-o.io container runtime in Kubernetes it’s “the same software”. If the high availability is based on running two or three instances of the database, and them opening tcp sockets to exchange data, and perform leader failover, then as “it’s the same software” it doesn’t matter whether we are running the database cluster as physical, or on a virtual machine hypervisor, or in a container orchestrator. That means you can learn how to configure the software for HA as containers on your laptop to then know how to configure it for HA on physical servers.

Obviously you might get different performance on different hardware or virtual machines or container runtimes. Typically it’s disk bandwidth and memory that makes the biggest difference for a database not how the process is being run (within reason). Yet from the perspective of making a high availability setup if the software you are using to only needs normal disk and normal tcp connections you can expect it to work anywhere that lets you access disk and network. It is then a matter of making sure that you provide enough cpu, memory, disk and network of sufficient reliability to meet your NFRs and SLAs.

A lot of companies might setup virtual machine farms or container orchestrators while not thinking about the demands of relational databases running in high availability modes. For example if you configure a HA setup of two Postgres instances but they get placed in two different virtual machine on the same physical server then you have a very poor setup with respect to the likelihood of one machine fault killing both database servers. If you configure all disk to be NAS network drives then that might be fine to run application servers but terrible to run databases under load. Often people are only thinking about scaling up application servers and having many of them crash and restart on different machines as the aim without the same disk being attached at startup. Anyone trying to run a database cluster in such an environment is going to have many sleepless nights. This means it’s important to ensure that wherever you run your databases is sympathetic to the needs of databases to have relative stability. If the environment is optimised and operating as though everything hosted is a stateless application server workloads that is not fussy about being moved around and restarted at random you might regret setting up a database cluster there.
> 3 votes

# Answer
I do not run databases in containers as these are hard to scale and the data should be persistent. I use database pools in GCP, AWS and Azure.
> 0 votes

---
Tags: docker, high-availability
---


# Question
Title: What are the tools you recommend for deploying / automating / organizing vm in the private cloud
I manage a set of virtual servers in a private cloud on vSphere.

In this private cloud we have several sets of servers that host several web applications such as:

-Webpress web sites

-Phone applications in PHP

-Java web applications

-Alfresco

-Redmine

-Nagios

-Jenkins

-Odoo

And these applications are hosted in separate servers with an architecture that mixes several technologies:

-firewall

-reverse proxy / loadbalancer / SSL termination (Haproxy)

-caching servers

-web servers

-LDAP directory server

-Server to centralize the backup and send it to remote NAS

We have a problem of complexity to manage these applications as each application has a VM server that hosts it and another that contains its database and another that saves the data ..etc

What do you recommend to me as tools to make my life easier, on **deployment**, **automation**, **maintenance**, **organization of applications**, **security**

And on everything how to **organize and save my configurations for each server** ... etc

***Do you recommend another architecture or solution?***

# Answer
> 0 votes
**TL;DR** Roll out a container orchestrator for new builds (e.g. OKD or the commercial version OCP). Work on packaging existing apps to be managed by an automation system (e.g., Puppet Enterprise) so rollout, patching and upgrades are all automated. Take root access away from everyone else and have them only do installs via the automation tools.

**Long Answer** including one fully automated WMWare private cloud:

Moving to cloud native tech such as Kubernetes, CloudFoundry or Docker Swarm isn't an easy option if you have a huge set it existing apps. Yet you could potentially reap huge benefits from making such technology available for new development. What you want is for teams to "self-service" via, say, the Kubernetes webui and command line tools. OKD is a distro of Kubernetes that focuses on “self-service devops” that I have seen work very well for three large system builds at three large organisations. It can be installed on Linux VMs that you manage any way you want (e.g., Puppet Enterprise example below) and there is commercial support from a large Linux vendor.

With such an approach you setup a cluster you can expand to up to 2000 identical VMs. You grant teams a logical “namespace” (aka “project”) and assign a quota for memory and cpu. Teams then use the webui or command line to spin up dozens of containers that are automatically spread over the cluster of VMs. You can apply tags to VMs and projects to place different teams on different pools of VMs if you still want bill departments based on how many VMs they reserve. OKD does software defined networks and automated DNS for each project so it takes away the headaches of assigning IPs or managing DNS as it is fully automated. Also load balancing, fail-overs and scale out are all automated. It also does rolling deployments and will health check containers and restart failed ones.

New apps will need to be built to modern standards to work within such a container orchestrator. In four separate experiences of rolling this out I have seen skeptical developer teams love the immediacy of "just help ourselves" up to their quota. The forget they ever started out demanding root access. I worked at a big organisation where we automated setting up each new scrum team with with multiple environments on commercial OKD (called OCP) with databases, networks, git repos, and running sample code all created by a small webapp we wrote in less than 15 minutes.

With your existing stuff then the road out of cost and overhead is standardisation, standardisation and more standardisation. At the moment I am working with a private cloud platform which is VMWare vRA based. The vRA portal has blueprint that internal customers launch standardised host profiles for databases, message brokers, webservers and "vanilla” (plain) Linux hosts. The launched images have Puppet Agent installed and a Puppet Enterprise Server enforces, you guessed it, standards. If you use root to tamper with the config then puppet agent runs every 20 minutes and puts back the configuration to match the puppet profile.

The puppet profiles are all in git and a separate Puppet Hiera config repo defines all configuration setting for all hosts. One git repo has the Hiera config for the base OS with different overrides for database profiles or application profiles (eg applying kernel settings). A second git repo holds the Hiera config for the application profiles (e.g., install and start oracle database, or a message brokers, or a web server). A third repo holds the Hiera config for any host level overrides needs for tricky applications or production fixes yet to make it into the main configuration for that application profile.

Puppet chooses the most specific configuration settings for the actual hosts. It looks for host specific, not normally needed, then looks for app profile specific settings, not always defined, then uses the default for the OS. Everything is in git and managed by code review of pull requests. It’s proper infrastructure-as-code managed by GitHub Enterprise.

All of the above works for the “standard” software packages used at the organisation. For in-house apps they don’t yet have Kubernetes. Teams launch a “vanilla” Linux blueprint on the VMware vRA portal rather than a specific application blue print. So they might launch an Oracle database blueprint and three vanilla Linux ones to run their java code. With vanilla it has all the access management packages (Centify) and teams just run use scp to upload their java. Teams are using Ansible Tower to upload their Java apps via none root ssh access. That is driven by Jenkins deploy jobs that call out to Ansible Tower.

# Answer
> 0 votes
The answer will depend on how much you want to change. I will describe how most larger companies solves this nowadays.

Containerize the apps. Run them in Docker containers. And use a container orchestration tool, like Kubernetes to manage deployment. But to do this your application must probably be redesigned to follow The Twelve factor methodology.

Both Amazon, Azure and Google have Kubernetes cloud services. If you want to continue with private cloud, OpenShift is the most popular Kubernetes-product.

# Answer
> 0 votes
I suggest terraform. A lot of companies are using it and this tool is basically one of the industry standard. This tool is basically an acronym for infrastructure-as-code.

---
Tags: deployment, continuous-deployment, configuration-management, cloud
---


# Question
Title: Is there any latest technology which will replace Docker, Kubernetes in future?
What's the future of engineers starting career in tools like Docker, Kubernetes?

# Answer
> 0 votes
Perhaps serverless could substitute it, but it will be hard to predict. I would suggest to follow industry standards and run things using these tools in production to stay up to date. The most changes in all branches occur in IT. Having said this, every engineer has to learn new things everyday to stay up to date. Whether k8s and docker will be replaced? Probably yes, but on the other hand there are still VMs and on-premise hardware. Perhaps k8s and docker will be used less in the future, but when I do not know. I am not a fortune teller with a crystal ball.

---
Tags: docker, kubernetes, azure-devops, cloud, google-cloud-platform
---


# Question
Title: jenkins pipeline code to build docker images
I am trying to integrate docker build process in Jenkins. I know the things with docker plugin and configure docker agent and template to build it. I want these things to be done with Jenkins pipeline code.
Basically, I want below process 1. Pull the sources code from git and create maven build

2. Push artifactory .war to nexus
3. Pull another git repo having dockerfile and other supporting data
4. take artifactory (war) from nexus and add it to docker images build.
5. Push the newly updated image to docker repo
6. execute deployment on kubernetes.

I have steps to build maven code push it to nexus and deployemnt on k8s. But I am facing issue to build docker image using. I have jenkin setup using docker image. So need to use docker agent and related plugin in jenkin code.

# Answer
> 1 votes
On building the docker image, you need to have the docker daemon enabled. You can also use the jib, a Google project, which creates images for your Java applications without the Docker daemon

# Answer
> 0 votes
https://github.com/GoogleContainerTools/kaniko could be used. I have seen it in action, but now implement it myself yet.

---
Tags: jenkins, build-pipeline
---


# Question
Title: Automate creation of branch + pull-request
I have a repository (call it repo-X) of shell.nix files that define development environments for a number of separate projects (call them project-A, project-B).

When one of the projects releases a new version, I run a suite of automated tests in repo-X for the relevant shell.nix file, and if all goes well, I increment the version number in that file to match the new version of the project it supports.

I'd like to automate this process. I'm using Jenkins and GitHub to *build* PRs, but I'm not sure how I'd use CI to *author* a feature branch and create a corresponding PR.

I'm having a hard time even finding references/ideas, but maybe I'm searching for the wrong terms. What tooling should I look into for this?

# Answer
You want a RCS/VCS that handles the version incrementally for you. I'm not sure how this relates to CI as the idea is to Continuously Integrate instead of spawning more branches. I'm not sure what you're using for your PR's, so I'll use Atlassian Bitbucket as an example. In the response to pushing up a new branch you get back the url for creating a PR which has an API which you can call to create your PR. Git commands will do the branching for you:

```
# Get the latest description, usually based on tags : https://git-scm.com/book/en/v2/Git-Basics-Tagging
$branchName = git describe
# Checkout the new branch
git checkout -b $branchName
# Push up the branch to origin
git push --set-upstream origin feature/$branchName

```

It's best to have a firm grasp of git when dealing with VCS. * https://try.github.io/ * https://git-scm.com/book/en/v2/Getting-Started-Git-Basics
> 1 votes

# Answer
I am now starting to use the multi-branch pipeline and it looks really promising as it builds every branch that contains changes and a Jenkinsfile.

https://wiki.jenkins.io/display/JENKINS/Pipeline+Multibranch+Plugin
> 0 votes

---
Tags: jenkins, continuous-integration, git, github
---


# Question
Title: Deploy helm chart from Jenkins to Kubernetes
I want to deploy helm chart from Jenkins to Kubernetes but I don't find any Plugins in Jenkins for Helm.

How can I tell Jenkins to run Helm chart in Kubernetes ?

# Answer
Use a docker image within your pipeline that has `helm` and other tools installed.

To quote the docs:

> Pipeline is designed to easily use Docker images as the execution environment for a single Stage or the entire Pipeline. Meaning that a user can define the tools required for their Pipeline, without having to manually configure agents. Practically any tool which can be packaged in a Docker container. can be used with ease by making only minor edits to a Jenkinsfile.

It then gives an example that we can modify to show running helm:

```
pipeline {
    agent {
        docker { image 'your_image:1.0' }
    }
    stages {
        stage('Deploy') {
            steps {
                sh 'helm list'
            }
        }
    }
}

```

This is a total game changer. Docker is the plugin to end all plugins. Before Docker you had to find a plug-in written in Java that to exposes a DSL to be scriptable in a Groovy sandbox. That was because it was not secure or scalable to let you simply run your own tools and scripts on a shared Jenkins. You were not even allowed use the full power of Groovy or Java. With Docker it is safe run any script or tools. So you no longer have the indirection of Java plugins and you can use the full power of any Linux tool on any Linux distro.

This approach is exactly how new cloud based build engines work. With bitbucket.com pipelines and cricleci.com you don’t need any plugins. You simply say which containers to use and what commands to run within them just like the example above. The build engine simply checks out the code where the docker container can see it.

Once you have used Docker based builds for a while doing it the old way feels very restricted and limiting. For example consider having to move files around in the build. In a pipeline with Groovy if you are new to Jenkins you have to look at a huge amount of (very ugly) documentation to just to basic things. If you use docker you can simply use standard bash and Linux commands.
> 2 votes

---
Tags: jenkins, kubernetes, jenkins-plugins, helm
---


# Question
Title: What are some clear benefits of running master/agent configuration management?
Ansible seems to have a clear advantage over competition like chef and puppet because it is agent-less and saves a certain degree of overhead.

I have read several comparisons of the various configuration tools and while each tool has its own strengths and weaknesses, I realize that much of it is down to personal preference.

The benefit of being agent-less is a no-brainer but are there any advantages of the master/agent architecture with respect to configuration management tools?

# Answer
> 1 votes
I've found an interesting discussion on the topic configuration management pull vs push topology

One contributor writes that pull systems (master/agent) offer more flexibility and are easier to scale.

# Answer
> 0 votes
IMHO the only advantage of a master/agent pull architecture is that it can be used in a firewalled network (when the actual configuration is located outside that network) without opening up holes in the firewall, while the push architecture cannot.

In terms of flexibility, I think the most important issue is the minimal setup required on a node to become remotely manageable. Once this requirement is met both architectures can handle updates. A master/agent architecture needs the agent installed, configured and running. On one hand Puppet and Chef aren't (yet) by default installed on most systems, while sshd (needed by Ansible) is. On the other hand even Ansible needs sshd running and credentials configured on the node to be accessible, so I'd say there's not much of a difference from this perspective - especially if the minimal setup is achieved via customized system images.

In terms of scalability, I'd consider the master/agent architecture less scalable (but just a bit):

* if the master attempts to actively keep a live system state (i.e. the map of all the agents and their states) that would chew up some time/resources. Without a master, Ansible isn't susceptible to such scalability issue. Granted, even master/agent architectures can offer a scalable approach to providing an overall system state similar to Ansible's (and probably they all do nowadays).
* in the old days, when the master would actually drive the agent configuration tasks (as mentioned in the post you referenced in your answer), the scalability of such architecture could suffer a lot. But this is not a big deal anymore since for all the tools mentioned in your question the configuration is just transferred to the local node which then performs the tasks.
* when it comes to transferring the configs to the local nodes, a pull architecture is limited only by the capacity of serving those configurations (common to both architectures), but a master/agent one needs to also run the logic driving transfers and actually making the transfers, if applicable (it's possible to just instruct the agents to pull their own configs), which limits how fast the master can propagate configs to a large number of agents.

# Answer
> 0 votes
IMHO, the idea of having an agent or master or both boils down to this:

> A mechanism to **continuously** manage states of the nodes.

If such a tool allows this, then "self-healing" capability is the obvious benefits.

In the case of Ansible, as far as I'm concerned, there are no built-in a mechanisms to provide the self-healing capabilities (because no daemons are running continuously to fetch the states and to react to them.)

That said, self-healing doesn't sound like the problem a configuration management tool should solve so maybe it's not something one might consider "advantage" of other tools against Ansible.

---
Tags: ansible, configuration-management, puppet
---


# Question
Title: Trying to execute an appcmd command via Ansible
I trying to execute the below command and running into errors ... I tried with quoting around the arguments, that didn't help either

```
   - name: Add authorization rules to FTP site
     win_command: appcmd.exe
   - args:
      chdir: C:\Windows\System32\inetsrv\
      stdin: set config "FTP_FW"
   - section: system.ftpServer/security/authorization /+"[accessType='Allow',roles='domain\usera',permissions='Read, Write']" /commit:apphost

```

Tried with quoting, that didn't help ... Can you give me some pointers pls.

# Answer
> -1 votes
Thanks for the comments. I was able to figure out. I ran the complete command from win\_command instead of having the arguments in stdin.

---
Tags: ansible
---


title: How to use a plugin parameter?

text:
# Question
Title: How to use a plugin parameter?
In Ansible, I want to use a plugin parameter. For example, the `profile_tasks` plugin supports a parameter to "Adjust the sorting output of summary tasks".

I enable this plugin by whitelisting it in the `ansible.cfg` file by:

```
[defaults]
callback_whitelist = profile_tasks

```

But how do I use the parameter supported by the plugin?

Note: I just want to use it but I don't know how to. The document says "parameter" and normally I think the way to use a parameter is to "pass a parameter to a function/command line" so that's why I asked "how to pass a parameter". What I want to know is "How to use a plugin parameter".

# Answer
I've not tested it, but as the doc says, you can configure the plugin via two ways : ini entries or env.

Add below your snippet the following and it should work :

```
[defaults]
callback_whitelist = profile_tasks

[callback_profile_tasks]
sort_order = descending

```
> 2 votes

---
Tags: ansible
---