# Question
Title: Where do I use Ansible and where is Terraform used? ( Strengths & Weaknesses)
What is Ansible mainly used for?

Where do I use Terraform?

This question focusses on where the tools have relative strengths or weaknesses.

For example:

* Ansible for VM build and Software installation.
* Terraform for Configuration (Apply and Enforce) and Compliance Enforcements.
* Terraform is used to Setup Cloud Environments mainly etc.

Hope this helps the question be more precisely answered

# Answer
I think you have to consider the history of the two tools to answer this question properly:

* **Ansible** was created to manage the installation and maintenance of virtual machines, thus it originally only had the capability to install software into an operating system, perform systems administration tasks *within* that machine. Subsequently, it has been extended to be able to create cloud resources in AWS, Azure, GCP and others.
* **Terraform** was created to manage cloud resources, it's effectively a facade layer over the various APIs in AWS, Azure, GCP, etc. It has subsequently been extended to be able to cover many of the functions that Ansible also covers such as installing software.

My personal workflow is to use both applications for what they are **really good at**, thus I create infrastructure (Virtual Machines, Clusters, Storage, Networks, etc) in Terraform and then use Ansible to install and configure software as required.

I can, and have in some situations, just used Terraform or Ansible, it's just a little harder as you have to learn how to use one tool in a different way.

Bruce Becker pointed out in a comment that one of the main differences is the fact that Terraform is "eventually consistent", in that it does a good job of understanding the current state and then making changes that converge on a "desired state". This workflow is better for the majority of tasks than Ansible. In Ansible you can do a test run with `--check`, but this doesn't always take into account cascading changes.
> 6 votes

---
Tags: ansible, terraform
---


# Question
Title: Are there examples of other production grade container runtimes?
While we say "Docker is just one possible container runtime", are there other implementations used in production environments as of late 2019, and do they have different approaches? (while Docker being a console wrapper for LXC more or less, right).

# Answer
> 3 votes
## Are there other implementations

**Yes**. The Container Runtime group of the Cloud-Native Computing Foundation has a list of runtimes:

They include:

## Do they have different approaches

While they cannot all be compared with each other directly, they all offer a way to provide a container runtime. In some cases, this runtime leverages the kernel, in others, they are in userspace. While some are explicitly proprietary, others are explicitly Open. Some are supported by large corporations (Google, IBM, Amazon, Alibaba), others by foundations or open source communities.

All of these are used in production, but perhaps vary in their popularity.

---
Tags: docker, containers
---


# Question
Title: How I can use multiple hosts in AWX playbook?
I have workflow that create 2 virtual machines on GCP. When virtual instance created I store output with external ip to variable. In next playbook i want to use all of my ip addresses in host section. My code is:

```
---
- name: Download and install jenkins
  hosts:
   - "{{ node_1_ip }}"
   - "{{ node_2_ip }}"
  tasks:
   - name: Install java
     yum:
       name: java-1.8.0-openjdk-devel
     become: true

```

But when I run my playbook i have warning:

```
[WARNING]: Could not match supplied host pattern, ignoring: 35.228.80.9

```

35.228.80.9 is my node\_2\_ip. How can I apply my playbook to this 2 machines?

# Answer
I suspect you haven't set up the host in your inventory you can figure out where your hosts are being pulled from with the following command:

```
ansible-config dump | grep HOST

```

In practice, you probably just need to create an `inventory` file with the following content:

```
[nodes]
node01 ansible_host=IP OF YOUR FIRST NODE HERE!!!
node02 ansible_host=35.228.80.9

```

Then specify the `-i` parameter to `ansible-playbook`:

```
ansible-playbook -i inventory -k playbook.yml

```

Depending on your exact use cases it might be more scalable and maintainable to use GCE Dynamic Inventory, although this does require a little bit of extra setup.
> 2 votes

---
Tags: ansible
---


# Question
Title: EKS node cannot pull docker image from ECR: "no basic auth credentials"
Using the `eksctl` tool, I created an EKS cluster with 5 nodes. My application's docker images are stored in ECR registries in the same region. I deployed my kubernetes cluster and everything has been happy for the past 6 weeks or so.

This morning, I came in and found 3 pods were in an `ErrImagePull` state. Using `kubectl describe pod <pod name>`, I found the error:

`Failed to pull image "<ECR REGISTRY>/<IMAGE>": rpc error: code = Unknown desc = Error response from daemon: Get <ECR REGISTRY>/<IMAGE>: no basic auth credentials`

My understanding of EKS and ECR is that I don't need a pull secret (and I haven't used one for any of the other running pods) so my guess is that some process or docker image on that node died but I can't find any docs on this.

Update: I forgot all about this question. When I created the original node group, I failed to include the `--ssh-access` flag which prevented me from getting onto the node and see if a kubernetes process had failed. I never found the actual solution; I simply added a taint to the problem node, created a new node, and went about my business. I'm still trying to find time to spin up a new node group with ssh access.

# Answer
> 1 votes
Do your IAM roles that are attached to EC2 instances that are in EKS cluster have ECR iam policies? If not please update IAM roles Ref Link: https://docs.aws.amazon.com/AmazonECR/latest/userguide/ECR\_on\_EKS.html#:~:targetText=The%20Amazon%20EKS%20worker%20node,policy%20permissions%20for%20Amazon%20ECR.&targetText=When%20referencing%20an%20image%20from,tag%20naming%20for%20the%20image.

---
Tags: docker, kubernetes
---


# Question
Title: VSTS Azure for DevOps: Unit test checking mechanism on pull request
Is there a way in VSTS Azure for DevOps to implement unit test checking mechanism on pull request.

The condition is to only allow commits that change something in the unittest folder for example.

Notes:

* we are using GoCD as a CI/CD system so it's external building.
* Git is internal hosted within VSTS.

# Answer
> 1 votes
If you have the option, Github offers Actions which allow you to run workflows on different events - one of them being push.

If Github is not an option, I would recommend creating a build where deployment of the artifacts only happens if the build is done on master.

On your feature branches, on checkin, a build will happen which will pull the latest code, compile, and then run unit tests, security scanning, and code coverage (if applicable). Since this is a feature branch, the build ends there.

If it is on master, it then performs the steps to package/deploy the artifacts.

In Branches \> Master \> Branch Policies (in the ellipses menu), add a Build Validation.

I recommend marking it required.

If you aren't worried about merge conflicts causing a rebuild, you can make it expire immediately upon master changing (requiring a rebuild, which will fail on merge conflicts).

If you are a large team where you might frequently step on one another, you can make it never expire and leave it up to the developer to fix manually before they are able to merge.

---
Tags: git, azure-devops, cicd
---


# Question
Title: TeamCity - CI w/ Multi-project .NET solution using SOA build and delivery issues
I have a multi-project .NET solution that I'm trying to set-up with TeamCity (TC) for CI and Octopus Deploy (OD) for CD.

I originally set-up TC to pull the latest from SVC, build the solution and deploy it as a .zip on the Dev server, but I've been having issues trying to deploy the solution to IIS with OP. I was informed that I should instead have TC take the solution and create multiple packages from it for OP to pick-up. I currently don't know how to do this.

I have the following build steps:

And I have this in my General Settings Publish Artifact settings:

Visual Studio solution set-up (Blue shows the projects that make up the entire application we want to deploy. Green boxed will be the separate packages, BLL, Services and Repository because of project dependency get rolled up into the API project.):

I have manually deployed the solution in IIS, below is how it needs to deploy in IIS:

The root site is a .NET project using MVC that communicates, hubs is a Signal-R project and API has the repository project and services project under it. So essentially The root website is a client, it communicates can only see API and hubs and any communication to the repository project would go through the API project first, it's supposed to be using a service-orientated architecture.

The goal is that TC should spit out 3 packages, one for the root website, one for the Signal-R project and one for the API which holds several projects but all projects are entirely in a single solution and this isn't something that I can get around for the time being. Once TC can do that I will need to set-up OD to deploy each of those packages to their required locations (which should hopefully be pretty simple).

Any information/assistance with this is greatly appreciated as I do not have a Dev-Ops background and this is the first time I've had to set-up this kind of a project with TeamCity.

Kind regards.

# Answer
I managed to resolve this by installing the Nuget package Octopack on the projects that I wanted to be deployed as either separate websites or website applications.

In this context, I installed Octopack to the 3 projects in green squares and checked-in the changes. Then on the Octopus Deploy side, I created 3 separate IIS deploy steps and just followed the typical steps. Octopus deploy now receives a NuGet package from TeamCity using the Octopus deploy plugin, Octopack build step, from there Octopus deploy has 3 IIS deploy steps, one for the main website and 2 for the website applications, each was given a custom directory path but after I had installed the Octopack NuGet packages into the appropriate projects in the solution through Visual Studio it was smooth sailing just using the general user-guide and tutorial videos.

@richard slater - Thank-you for letting me know about Octopack, this was one of the key missing pieces that I needed.

TLDR - If you want to deploy projects in a solution as separate websites, install the Octopus Deploy plugin to TeamCity and follow the basic tutorials, add the Octopack NuGet package to the projects you want as separate websites or web applications, set-up the Octopack step to push to Octopus Deploy over the API and use IIS steps in Octopus Deploy and you should be golden.
> 0 votes

# Answer
So you need to create packages for each of the three sites so that you can deploy each one individually.

The first step is to create a `.nuspec` file for each of the IIS websites you want to package:

1. Install OctoPack and read the entire article
2. Create three empty text files next to the `.csproj`, give it the same base filename as the `.csproj`, thus if it is `MyApp.API.csproj` you want to create an empty text file called `MyApp.API.nuspec`.
3. In each of these `.nuspec` files paste in the following template:
```
<?xml version="1.0"?>
<package xmlns="http://schemas.microsoft.com/packaging/2010/07/nuspec.xsd">
<metadata>
    <id>Sample.Web</id>
    <title>Your Web Application</title>
    <version>1.0.0</version>
    <authors>Your name</authors>
    <owners>Your name</owners>
    <licenseUrl>http://yourcompany.com</licenseUrl>
    <projectUrl>http://yourcompany.com</projectUrl>
    <requireLicenseAcceptance>false</requireLicenseAcceptance>
    <description>A sample project</description>
    <releaseNotes>This release contains the following changes...</releaseNotes>
</metadata>
</package>

```
4. Customize the template for each of your projects.
5. Update TeamCity to add `/p:RunOctoPack=true` to the `msbuild` command line
6. Publish your packages with the following msbuild arguments:
    * `/p:OctoPackPublishPackageToHttp=http://your.octopusserver.com/nuget/packages`
    * `/p:OctoPackPublishApiKey=API-ABCDEFGMYAPIKEY`

You may have to play around a bit, but essentially the above is taken from this article so treat that as your bible.
> 1 votes

---
Tags: continuous-integration, continuous-deployment, cicd, teamcity, octopus-deploy
---


# Question
Title: On-Site Octopus Deploy - How to deploy projects in a multi-project solution to different IIS locations
I'm tasked with setting up CI/CD for my company. I'm using TeamCity which I have configured using the Octopus Deploy plugin to create a Nuget package to the appropriate dev server (alternatively, I can have TeamCity spit out a .zip).

The package itself is a solution that contains several projects, and each project needs to be deployed to different folders on IIS. Is this something that I can accomplish with Octopus Deploy, and if so how?

I've looked through the documentation and spent a lot of time Googling but I haven't found an example of someone doing this but I want to use Octopush Deploy because of the reporting and flexibility it provides (since I will need to set up pipelines to other environments in the future).

Information about the solution and environment: Multi-project solution .NET Framework 4.6.1 IIS version 6.2 Windows Server 2012 R2 Octopus Deploy v6.0 on-site TeamCity Professional 2019.1.1 (build 66192)

The solution is currently deployed on ISS like this:

Any assistance provided or links to resources is deeply appreciated.

Kind regards.

# Answer
> 0 votes
I managed to resolve this by installing the Nuget package Octopack on the projects that I wanted to be deployed as either separate websites or website applications.

In this context, I installed Octopack to the 3 projects in green squares and checked-in the changes. Then on the Octopus Deploy side, I created 3 separate IIS deploy steps and just followed the typical steps. Octopus deploy now receives a NuGet package from TeamCity using the Octopus deploy plugin, Octopack build step, from there Octopus deploy has 3 IIS deploy steps, one for the main website and 2 for the website applications, each was given a custom directory path but after I had installed the Octopack NuGet packages into the appropriate projects in the solution through Visual Studio it was smooth sailing just using the general user-guide and tutorial videos.

@richard slater - Thank-you for letting me know about Octopack and nuspec, these were some of the key missing pieces that I needed.

TLDR - If you want to deploy projects in a solution as separate websites, install the Octopus Deploy plugin to TeamCity and follow the basic tutorials, add the Octopack NuGet package to the projects you want as separate websites or web applications, set-up the Octopack step to push to Octopus Deploy over the API and use IIS steps in Octopus Deploy and you should be golden.

# Answer
> 1 votes
It *can* be achieved, however, you will make your life easier if you modify your TeamCity build process to create a package for each IIS Website then deploy each one individually using a Deploy to IIS step.

If it's impossible to change your build process then you will need to do some pre-processing or just deploy the websites through a custom script.

---
Tags: continuous-deployment, teamcity, octopus-deploy
---


# Question
Title: how long does it take to deploy a serverless function?
using any method - cli, serverless.com, terraform, roughly:

**how much time can i expect to pass between these two steps:**

1. submitting a new function to AWS (or azure or GCF), and
2. having that function hosted live in the cloud?

note that i'm ***not*** asking about cold start times.

i'm wondering how long it takes new function code to be entered into and hosted in the AWS lambda system (or azure, ibm, google etc) for a given region. that is, after taking steps to deploy some new lambda/serverless function code, approximately how soon will i be able to use it?

# Answer
> 1 votes
My general experience of Serverless across the three major clouds is that:

* GCP takes up to two minutes.
* AWS takes about a minute.
* Azure is a bit random and has bad days when it can take five minutes, but mostly around two minutes.

Some of it rather depends on how you measure it, and what you use to deploy it. Something like Serverless Framework will do some additional work to creating a CloudFormation template which adds to the overall time.

A former colleague of mine, James Randall, has published quite a few articles on various aspects of Serverless performance - I haven't seen anything specific around how long it takes for a cloud provider to distribute code to the point of execution, but you may be able to glean some additional answers from his work.

# Answer
> 1 votes
This will vary based on your cloud provider and what services you are using. AWS Lambda functions are designed to be highly available for the region. Per the AWS Lambda FAQ:

> When you update a Lambda function, there will be a brief window of time, typically less than a minute, when requests could be served by either the old or the new version of your function.

So the new function will not be instantaneously available, and there is a possibility that the old function may be called.

---
Tags: terraform, azure-devops, google-cloud-platform, aws-lambda, openwhisk
---


# Question
Title: stop pulling a docker image from the public docker-hub registry
Is there any way to stop pulling a docker image from the public docker-hub registry while building image locally. Note: We are maintaining a private registry that contains all the images. Only and only docker-engine should search for an image from the private registry and not from the public registry.

# Answer
> 2 votes
You cannot change the default registry (see this). You have to specify which registry to use like this:

```
docker pull myprivatereg:port/library/image

```

Or, in a Dockerfile:

```
FROM myprivatereg:port/library/image

```

(This question was also asked in StackOverlow).

---
Tags: docker-registry
---


# Question
Title: Jenkins Cancel (jobs) from Queue by Label
I'm freshman in Jenkins. I would like to cancel jobs **by it's own label** (hundreds) from my Queue of tasks given to slaves: `jenkins.model.Jenkins.getInstance().getQueue()` so i then could do: `items[i].doCancelQueue()`

Currently I only know how to do it, by its **name**: `items[i].task.getFullName()`.

I'm looking for something like this: (when I iterate over Slaves that's just give me list of available labels for a single slave)

```
for (aSlave in hudson.model.Hudson.instance.slaves) {
    println('Name: ' + aSlave.name);
    println('getLabelString: ' + aSlave.getLabelString());
}

```

My whole code for deleting jobs by name

```
import hudson.model.*

def q = jenkins.model.Jenkins.getInstance().getQueue()
def items = q.getItems()

if (job_name_fragment == '' && revision == ''){
    println("At least one param is requied")
    return 1
}

def count = 0
for (i=0;i<items.length;i++){
    if (items[i].task.getName().contains("trunk") || items[i].task.getName().contains("dev")|| items[i].task.getName().contains("Windows")){
        println("Task: " + items[i].task.getName() + " has been skipped")
        continue
    }
    if(job_name_fragment == '' || items[i].task.getFullName().contains(job_name_fragment)){
        if(revision == '' || items[i].params.contains(revision)){
            println("Task: " + items[i].task.getFullName() + " has been deleted from queue")
            items[i].doCancelQueue()
            count++;
        }
    }
}
println("")
println("Removed total: " + count + " tasks from queue")

```

Thanks

# Answer
> 2 votes
Found it by myself, if anyone will be interested check:

getAssignedLabel()

in. https://javadoc.jenkins-ci.org/hudson/model/Queue.Item.html

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: ansible aws windows ec2 - Internal Error: this connection module does not support running commands via sudo
I am getting below error when I am trying to run one playbook which connects to Windows machine hosted on AWS EC2. I enabled winrm and win\_ping worked fine but when I am invoking ansible-playbook I am not sure why it is using sudo as if the target machine is Linux host.

I tried disabling become=True in ansible.cfg but same issue occurred.

Is it compulsory to use ansible\_winrm\_transport=ntlm for the win\_ping to work. I feel that is causing this issue.

Any help would be appreciated.

```
fatal: [10.0.150.18]: FAILED! => {"failed": true, "msg": "Internal Error: this connection module does not support running commands via sudo"}

[root@localhost ]# cat ansible.cfg
[defaults]
log_path=/var/log/ansible.log
ansible_winrm_operation_timeout_sec = 200
ansible_winrm_read_timeout_sec = 500

[privilege_escalation]
become=True

[root@localhost ]# ansible --version
ansible 2.2.0.0

[root@localhost ]# cat hosts
[localhost]
localhost ansible_connection=local

[marcel]
10.0.150.18  ansible_user=Administrator ansible_password=xxxxxxxxxx ansible_port=5985 ansible_connection=winrm ansible_winrm_server_cert_validation=ignore ansible_winrm_transport=ntlm

```

# Answer
> 1 votes
According to the documentation:

> Prior to Ansible 2.4, become would only work when ansible\_winrm\_transport was set to either basic or credssp, but since Ansible 2.4 become now works on all transport types.

You're using Ansible 2.2. Might that be it? The error message might be a bit misleading.

# Answer
> 1 votes
You should try adding an entry to your priviledge escalation section.

Set `become_method` to `runas`. So in my opinion just change the `become_method` to `runas` and it should work just fine.

Do make sure the ports 5985/5986 are opened in the security group from the AWS console and try again. Do post the logs if you can.

---
Tags: ansible, amazon-ec2
---


# Question
Title: Azure Devops, multiple on-prem environments, variables per environment
I've configured first deployment pipeline to an on-prem DEV environment. The project consists of 3 identical environments, DEV, UAT and PROD While their setup is exactly alike, their config (database credentials etc.) is obviously not.

I'm using the DevOps `variables` to alter `web.config` and `appsettings.json` configuration files while deploying. Therefore I'd like to apply the same deployment pipeline, targeting 3 different environments with 3 different variables sets.

This is the way I've seen in Octopus. Is it possible in the DevOps?

# Answer
You can create a **Release Pipeline** with a stage for every environment; each stage has a different variable group, defining that environment's settings.
On each stage you can copy files, etc. and run tests against the environment.
If the test/stage succeeds, the pipeline goes ahead to the next stage. Between the stages, you can configure that an authorized person has to approve the next step.
The screenshot below is a release pipeline of one of my little side projects.
> 2 votes

# Answer
If I understand your question, you want a single variable to have different values per stage. This same concept Octopus took a while to get right with being able to manage the variables. Currently that isn't a thing that is available in Azure devops. You can create a pipeline that Ned Flanders suggested but you would need to define the variable multiple times for each environment and scope each variable to the stage it is deployed in. I think this is a need of azure devops and is one thing that octopus deploy does better than azure devops. I would recommend opening a feature request for this. Microsoft moved away from user voice and onto developer community. There is a request for this back in April that got closed in September due to low activity closed request. If you open up a new one, I'm sure we can get people here to vote on it
> 1 votes

---
Tags: azure-devops, azure, azure-pipelines
---


# Question
Title: What's the relationship between ContainerD and Docker?
I'm aware that Kubernetes uses `containerd` internally and that `containerd` is part of Docker. Does that mean that Kubernetes only uses certain parts of Docker to provide orchestration of containers? Additionally, how does `runc` fit into this ecosystem, I'm aware Kubernetes uses it because CVEs affecting `runc` impact Kubernetes.

I've not been able to find out what the technical components of Kubernetes are; there are plenty of diagrams out there showing the logical elements (i.e. masters, nodes, kubelets, deployments, services, pods, etc.). However, I'm trying to get into the details a bit more and understand what parts make up Kubernetes.

Is it possible to describe how Kubernetes is constructed from its components such as `runc`, `containerd` and Docker?

# Answer
> 9 votes
Containerd is a daemon for running containers. The Docker blog has a nice write up explaining how this fits into Docker. While containerd is used in Docker, it can also be used as a daemon for other container solutions. Containerd requires runc in order to function.

runc as defined by the project is a "CLI tool for spawning and running containers according to the OCI specification".

These tools are designed to follow the Open Container Initiative (OCI) specifications. This has become a standard and defines how Docker and other tooling interacts with containerd and runc.

When you look at most Kubernetes diagrams they will not show `containerd` or `runc` as they are abstracted away into the container engine you use (e.g., Docker, rkt). Because Kubernetes uses `containerd`, it will also use `runc`. This is what allows multiple container solutions that follow OCI specifications to integrate with Kubernetes.

So yes, you could break down how part of Kubernetes is built by looking at `runc`, `containerd`, and the container engine.

---
Tags: docker, kubernetes, containers, open-source
---


# Question
Title: How to parse command output (docker log) through goaccess?
How to parse `docker logs` output / tail output through goaccess? When I do:

```
docker container logs mycontainer > logs

```

and then `goaccess -f logs` it works perfectly, but if I do `docker logs --tail all mycontainer | goaccess -` it doesn't work :( but in the manual you can do this via tailing command: https://goaccess.io/man

```
tail -f access.log | goaccess -

```

Pls help :( I think the problem is that the native tailing differs from docker's?

# Answer
What do you mean by "it doesn't work"? I got an error suggesting that the log format wasn't correct:

```
Fatal error has occurred
Error occured at: src/goaccess.c - set_curses - 1394
No time format was found on your conf file.

```

When I edited `/etc/goaccess.conf` (i.e. uncommented/selected `time-format`, `date-format` and `log-format`) the `docker logs mycontainer | goaccess -` command works as expected.

As an aside, for my `nginx` proxy inside my docker container, I had to set my `log_format` as follows:

```
log-format %v %h %^ %^ [%d:%t %^] "%r" %s %b "%R" "%u"

```
> 4 votes

---
Tags: docker, logging
---


title: What is ansible-runner about?

text:
# Question
Title: What is ansible-runner about?
I've read the README file from the github repository as well as the documentation site of ansible-runner (from the official ansible project) but I don't understand what is ansible-runner about and in which case can it be useful ? Is it a project bring additional features or it is created to run ansible in a different way ?

# Answer
> 12 votes
This project is to support automation of running ansible playbooks. There is a GUI based project called **Ansible Tower** or AWX to do that, but **ansible-runner** seems to be the underlying layer of the automation in essence replacing **ansible-playbook** command.

There are some really serious issues for automation when trying to use **ansible-playbook** as part of some scripts. The output of **ansible-playbook** is the most awful thing under the sun when it comes to parsing. So you have to start with writing your own output plugin and then write a parser for that, then you need to figure out how to manage the environment variables and passwords and many other things. So **ansible-runner** is the answer to this. It would preserve the artifacts from running a task or playbook and make encapsulation of ansible in further automation possible.

You can run it from shell, you can run it as python module and there is even a reference container for it and obviously it is used by AWX. Further the project's event emitter will hopefully be much better pluggable to monitoring systems like ELK and to Message Bus type systems.

# Answer
> 1 votes
To complete Jiri's answer, I found out that `ansible-runner` is used as ansible runtime in the openshift-operator framework, so you run a ansible playbook or an ansible role in a pod to reconcile cluster status when changes appears on resources.

So `ansible-runner` is a really the runtime execution for ansible as `runc` or `cri-o` can be for containers execution.

---
Tags: ansible, automation
---


# Question
Title: problems running Ansible with become on domain-connected Windows Server
I am running several tasks on a Windows 2016 server with Ansible including:

* Joining it to a domain `SOMEDOMAIN`
* `SOMEDOMAIN\SOMEUSER` has been added to the local administrators group on the machine with Ansible.
* Running several tasks on the domain-connected machine as a domain-user `SOMEDOMAIN\SOMEUSER`

The last bit requires me to run Ansible tasks using `become` and `runas` (as per the Ansible Documentation on the topic).

## Problem

Unfortunately, even a trivial example like the below fails:

```
---
- name: add to local admin
  hosts: all
  vars:
    ansible_become_password: `some_password`
  tasks:
  - win_whoami:
    become: yes
    become_user: ''SOMEDOMAIN\SOMEUSER''
    become_method: runas

```

The error message reads

```
The full traceback is:
Exception calling "RunAsUser" with "7" argument(s): "LogonUser failed (Logon failure: the user has not been granted
the requested logon type at this computer, Win32ErrorCode 1385)"
At line:1087 char:9
+         $result = [Ansible.BecomeUtil]::RunAsUser($username, $passwor ...
+         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : Win32Exception

fatal: [SERVERNAME]: FAILED! => {
    "changed": false,
    "msg": "Failed to become user SOMEDOMAIN\\SOMEUSER: Exception calling \"RunAsUser\" with \"7\" argument(s): \"LogonUser failed (Logon failure: the user has not been granted the requested logon type at this computer, Win32ErrorCode 1385)\""
}

```

## Attempts

* I do understand this is related to a privilege missing on the machine.
* I also know that group policies get applied on the machine when it joins the domain, so its possibly something related to the group policy. (I don't have access to the domain controller to validate this)

I checked the Group Policy "Access this computer from the Network". It is set to `Everyone, Administrators, Backup Operators, Users` which is the default setting. My account is part of the `Administrators` group, so this doesn't look like the cause.

## The Question

What other possible causes are there? Or in other words, what other privileges (or modifications to the ansible task) are needed to make Ansible `become` with the `runas` method work on domain-connected Windows machines?

# Answer
> 2 votes
My problem resolves down to an issue with group policy, and is likely to be solved with the same answer as this question:

https://serverfault.com/questions/695839/error-1385-when-executing-runas-on-windows-enterprise-7

i.e. the required permissions for `become` to work on Windows, seem to be:

* Allow Logon as a Service
* Allow Logon Locally

### Alternate Solution

reference

> batch: Runs the process under a batch context that is similar to a scheduled task with a password set. This should bypass most WinRM restrictions and is useful if the become\_user is not allowed to log on interactively.

There is another permission called 'Allow Logon as a Batch Job". If the domain-user has this permission, the user can run scripts as though they were scheduled tasks.

You can run your scripts using Ansible `become` and change the logon type to `batch` as shown below:

```
win_whoami:
    become: yes
    become_user: ''SOMEDOMAIN\SOMEUSER''
    become_method: runas
    become_flags: logon_type=batch

```

# Answer
> 0 votes
I found the issue be that the "Secondary Logon" service had been disabled. Enabling it and running the Ansible playbook again worked.

I found the suggestion on this blog post: http://david-homer.blogspot.com/2018/01/pinvoke-createprocesswithlogonw-causes.html

---
Tags: ansible, windows, active-directory
---


# Question
Title: Can Jenkins master connect to a slave node in a private subnet?
We've got a Linux Jenkins master node set up in AWS behind a load balancer and I'm attempting to connect to Windows slave nodes in private subnets. Is this possible? The Jenkins URL is accessible when I RDP into each server, but when I attempt to configure the node using "Launch agent from browser" an error occurs.

```
hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to <JenkinsURL>:39057 (retrying:4)
java.io.IOException: Failed to connect to gpa-jenkins-
prod.aws.baxter.com:39057
    at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:242)
    at hudson.remoting.Engine.connect(Engine.java:686)
    at hudson.remoting.Engine.innerRun(Engine.java:547)
    at hudson.remoting.Engine.run(Engine.java:469)
Caused by: java.net.ConnectException: Connection refused: connect
    at sun.nio.ch.Net.connect0(Native Method)
    at sun.nio.ch.Net.connect(Unknown Source)
    at sun.nio.ch.Net.connect(Unknown Source)
    at sun.nio.ch.SocketChannelImpl.connect(Unknown Source)
    at java.nio.channels.SocketChannel.open(Unknown Source)
    at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:203)
    ... 3 more

```

What could be the source for this refusal?

# Answer
> 1 votes
As described in Jenkins documentation https://jenkins.io/doc/book/managing/security/#jnlp-tcp-port, you need to configure:

1. Jenkins to use fixed TCP port for JNLP agent, for instance 39057
2. AWS loadbalancer to listen on port 39057 and forward it to Jenkins on same port

---
Tags: jenkins, amazon-web-services
---


# Question
Title: Jenkins Declarative pipeline: Execute stage conditional on execution of previous stage
I have a Jenkins declarative pipeline with several stages. Some of these stages need to be executed *based on whether a previous stage has been executed*. In pseudocode:

```
pipeline{
  stages{
    stage('1: Always do this thing') {
      steps {
        sh 'do_thing.sh'
      }
    }
    stage('2: Maybe do this thing') {
      when{ condition }
      steps {
        sh 'sometimes_do_this_thing.sh'
      }
    }
    stage('3: Do this thing if 2 was executed'){
      when { pipeline.stages.2.was_executed }
      steps {
        sh 'conditional_path.sh'
      }
    }
  }
}

```

I accept that the best way to do this might be in the scripted pipeline, but I have declarative pipelines. Including some scripted bits in them is also acceptable, if kept to a minimum.

So: **What is the most elegant way to execute a stage in a pipeline conditional on the execution status of the previous one**?

# Answer
A pretty simple solution is to declare a global variable on top of your pipeline, and check it's value in your last conditional stage:

```
def flag = false;

pipeline {
    stages {
        stage('1 - Always') {
            steps {
                sh './always.sh'
            }
        }

        stage('2 - Maybe') {
            when { condition }

            steps {
                sh './maybe.sh'
                script { flag = true }
            }
        }

        stage('3 - If Maybe was executed') {
            when { expression { flag == true } }

            steps {
                sh './conditional.sh'
            }
        }
    }
}

```
> 7 votes

---
Tags: jenkins, jenkins-pipeline, groovy
---


title: Docker swarm container monitoring

text:
# Question
Title: Docker swarm container monitoring
I have been on a typical journey of:

1. deploying software on a machine
2. deploying software in a container
3. deploying software using docker swarm

The need for this journey is that swarm removes a large portion of micro-managing resources and machine allocation etc.

There is, however, a hurdle I am trying to overcome.

For point one and two, we have heavy monitoring of our software using a tool (Zabbix) which has given us great insights into any problems or metrics.

As we migrate to point three, how does one monitor an individual container?

**To give a real-world illustration:**

I have software that is "buggy", every now and then it dies. The health check would typically fail and a new container would start with no service interruption. My question is,

* How am I aware that this has occurred in order to investigate a potential problem ?
* One potential solution is to monitor the docker swarm event stream; is this the only solution?

# Answer
> 1 votes
The easiest way is not to handle this information)) You will get tired of such useless information like containers restarts.

Anyway, there is couple of tools to monitor docker with prometheus:

But it requires installation of prometheus, alert-manager, find metrics for triggering and set up alert rules.

Or maybe try to find zabbix tool for monitoring docker containers.

---
Tags: docker, docker-swarm
---


# Question
Title: How to get the next friendly name that Docker will assign to the container it will make?
**Background**

When you start a Docker container as in the following example:

```
$ docker run -ti ubuntu:latest

```

Docker docker automatically assigns the container an ID and a friendly name.

Docker friendly names are of the format `adjective_famousPerson`.

**Problem**

I would like to get the friendly name that Docker will generate for the next container, so that I can assign that name to the hostname parameter of my `docker run` command.

For example:

```
$ docker run -ti --hostname docker.next_friendly_name ubuntu:latest

```

**Question**

Is there a built in way to do this with docker?

I need a friendly name that is randomly generated, but I don't want to script it up myself if there is a way to do this with **Docker** or even **Docker Swarm** or **Kubernetes**.

Additionally, the `name` of the container and the `hostname` of the container need to match.

# Answer
As the name of the container is randomly generated you cannot get it before it's generated (by definition). You can use this service to generate random names :

```
NAME=$(curl -s https://frightanic.com/goodies_content/docker-names.php); \
docker run -ti --name $NAME --hostname $NAME ubuntu:latest

```
> 1 votes

# Answer
Yes, there is a built in way. You can define your own "friendly name" by using the --name parameter when creating the container:

```
docker run -ti --hostname OTHER_CONTAINER_FRIENDLY_NAME --name MY_NAME ubuntu:latest
```

If you are trying to automate this process then your build tool can supply the name from a variable or via the run script. If you are using a docker-compose file you can specify the name directly in the yaml:

```
services:
  myservice:
    container_name: example_friendly_name
```
> 0 votes

# Answer
You should be using `--name` to create your own friendly names. The randomly generated ones you are thinking of are done at run time so you have no possible way of knowing what the next one will be till it's generated.
> 0 votes

---
Tags: docker, kubernetes, containers, docker-swarm
---


title: Use Terraform to manage ssh-keys in GCE

text:
# Question
Title: Use Terraform to manage ssh-keys in GCE
I use Terrafrom and manage access to VMs in GCE by changing VM metadata.

Before I put it to string, but it not good, so I try to put them to variable and use it. But I faced problem. This is my code:

```
variable "ssh_keys" {
  type = "list"
  default = [
    {
      user = "atolkachev",
      key = "ssh-rsa my_key atolkachev"
    },
  ]
}

resource "google_compute_instance" "instance" {
  zone = "${var.region}-${var.zone}"
  project = "${var.project_name}"
  metadata = {
    ssh-keys = join("\n", [for user, key in var.ssh_keys : "${user}:${key}"])
  }

```

When I execute it, I get error:

```
Error: Invalid template interpolation value

  on main.tf line 24, in resource "google_compute_instance" "instance":
  24:     ssh-keys = join("\n", [for user, key in var.ssh_keys : "${user}:${key}"])

```

I tried to find solution, but failed.

# Answer
There was used wrong type for `variable`, should be:

```
variable "ssh_keys" {
  type = "map"
  default = {
    "atolkachev" = "ssh-rsa my_key atolkachev"
  }
}

```
> 1 votes

---
Tags: terraform, google-compute-engine
---


# Question
Title: How to dynamically create digital ocean droplets without password change?
I am working on a web app where that I divided into two blocks: 1. One for front-end. 2. One for backend doingheavy computation which won't be used every time Both parts of the web service are hosted on digital ocean. In order to cut costs, I created manually a snapshot of the back. The idea is to create a droplet from this image using a rest API if required by the front. At the start of the backend droplet, two docker containers are created immediately.

My problem is that since the droplet is a fresh ubuntu instance, my cron jobs are not executed after the creation of the droplet. I am suspecting the reason is because Ubuntu is asking for password change and blocking every thing.

For that, I need to rechange the password of the instance manually then reboot it. This is clearly not possible. Is there a way to automate this part ?

# Answer
> 0 votes
After more tests, I found out that in order to do this, I need to use an ssh key with the API. The password change phase is bypassed after this.

---
Tags: cloud
---


title: Jenkins and docker login issue

text:
# Question
Title: Jenkins and docker login issue
I am trying to create a Jenkins pipeline that will pull a public image from docker hub and build a Vue.js application inside that container. In my Jenkins server I already have other pipelines that use AWS ECR to push images too. It appears that my build that is trying to access a public image is trying to log into AWS ECR as well. I'd be okay with putting that public image on ECR if that would solve my issue.

Here is the output of the failed Jenkins build

```
[Pipeline] withDockerRegistry
$ docker login -u AWS -p ******** https://index.docker.io/v1/
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://registry-1.docker.io/v2/: unauthorized: incorrect username or password
[Pipeline] // withDockerRegistry

```

# Answer
> 1 votes
The error shows that you are using `AWS ECR` credentials to login to dockerhub which will not work. `Dockerhub` is a different registry(both public and private), however `ECR` is private registry only.

For pulling public images from `dockerhub` there is no need to login to dockerhub. You can simply use `docker pull` command and it will pull an image from dockerhub registry.

Example: `docker pull mongo`

However, when you want to pull an image from `ECR`, you need to first login to the `AWS ECR` and then only you can pull an image from `ECR`.

# Answer
> 0 votes
I think your jenkins need to create an rapo with aws first for that what we normally do is, let's first do a manual pull from jenkins machine from ecr. Use ecr login command $ecr login (with ur region codes etc) is there, try that first ( to get full command open ECR in aws and click on ecr push command, there u will get the full command). Then after successful login try again to pull the image this should pass. After getting this done try using jenkins pipeline. All the best.

---
Tags: docker, jenkins, continuous-integration, jenkins-pipeline
---


title: Issue pushing to ECR using Jenkins

text:
# Question
Title: Issue pushing to ECR using Jenkins
I'm trying to push an image to ECR via a Jenkins pipeline. Currently I have the ECR plugin installed, and a role assigned to my EC2 instance that allows ECR access.

I am using correct credentials that are valid for the region I am trying to push to.

However I am getting this error message

```
Pipeline] withDockerRegistry
Using the existing docker config file.Removing blacklisted property: credsStore$ docker login -u AWS -p ******** https:/123456790.dkr.ecr.us-east-1.amazonaws.com
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Error response from daemon: Get https://https/v2/: dial tcp: lookup https: Temporary failure in name resolution

```

Jenkins pipeline script

```
docker.withRegistry('https:/1234567890.dkr.ecr.us-east-1.amazonaws.com', 'ecr:us-east-1:aws-type-creds') {
    app.push(env.GIT_COMMIT)
 }

```

EDIT \** Now getting this error with small change to pipeline

```
+ docker push 1234567890.dkr.ecr.us-east-1.amazonaws.com/sample:66e4bf8547da1b2fd8f852e093fd152a6b1ee598
The push refers to repository [1234567890.dkr.ecr.us-east-1.amazonaws.com/sample]
1f15c22274e7: Preparing
70da67205593: Preparing
f0726b634cb9: Preparing
ab91817addbe: Preparing
1be049a99262: Preparing
cc82fb177433: Preparing
97a48aebb8e5: Preparing
1f7e5026a4f7: Preparing
9e864b659f2b: Preparing
f1b5933fe4b5: Preparing
cc82fb177433: Waiting
97a48aebb8e5: Waiting
1f7e5026a4f7: Waiting
9e864b659f2b: Waiting
f1b5933fe4b5: Waiting
no basic auth credentials

```

Line changed was line 2 of the pipeline code I showed before. It is now `docker.image('alerts').push(env.GIT_COMMIT)`

# Answer
The docker login failed.

Error response from daemon: Get https://https/v2/: dial tcp: lookup https: Temporary failure in name resolution

The url is invalid.

change from

`docker login -u AWS -p ******** https:/123456790.dkr.ecr.us-east-1.amazonaws.com`

to

`docker login -u AWS -p ******** 123456790.dkr.ecr.us-east-1.amazonaws.com`
> 1 votes

# Answer
Please see the below lines from the AWS ECR docs. (https://docs.aws.amazon.com/AmazonECR/latest/userguide/docker-push-ecr-image.html)

`Authenticate your Docker client to the Amazon ECR registry to which you intend to push your image. Authentication tokens must be obtained for each registry used, and the tokens are valid for 12 hours.`

Since you have already an IAM role to EC2 instance which will allow ECR access, you need to first get the authentication details (username and password) and login to ECR.

You can see various methods here to find out how you can get the authentication details and use them to login to ECR first. Once you are logged in, the you can seamlessly push images to ECR repo.
> 0 votes

---
Tags: docker, jenkins, jenkins-pipeline, docker-registry
---


# Question
Title: Ansible: What's the difference between task, role, play and playbook?
Ansible has tasks, roles, plays and playbooks.

Are those just names for different levels in a playbook organization?

If not, how, exactly, do they differ from and relate to each other?

# Answer
No, each has a different role and way of working.

Starting with the easy ones:

A **playbook** is simply a list of plays. The highest level of a playbook YAML is a list, and on that list only two things are accepted: a play definition or the keyword `import_playbook`, which imports a list of plays from another playbook file, as if they were defined in that place of the calling file.

The playbook is also the only thing that can be called directly from `ansible-playbook`.

A **task** is, paraphrasing the manual, '*nothing more than a call to an ansible module*'. That call definition, however, knows nothing of on which hosts it is supposed to run on.

The **play** is the element that ties tasks to the servers where they'll run. The key element here is the mandatory keyword `hosts`. This is the part of Ansible that tells which hosts are being affected and how.

So, the structure and behavior is as follows:

* Playbook: the highest level, just a list of plays
    + Play: ties tasks to host lists
        - Tasks: definition of a call to a module
        - Besides tasks, a play may have pre-tasks, post-tasks and handlers, which are all task-like, and roles.

A **role** is a different thing, as it is not defined within a playbook. Instead, they each have their own subdirectory under the directory `roles`. The objective of roles is to organize things. So, under the role's subdir, you can find tasks, handlers, files and templates, variables and defaults. All of these related to that specific role.

So, roles are an easy way to encapsulate and share Ansible information. You could have a subdir in `templates` per task, for example. But if you wanted to share that task, you'd have to pick the template as well. With roles, you can just zip the role dir and (almost) everything should be there.

Like a play, the role defines tasks and handlers. However, roles do not define on which hosts the role will be run. So roles must be referenced to from a play.

Roles can also declare dependencies (that is, references to other roles that need to be run before the declaring one).

---

*PS1. Most of this information is available piecemeal on SO (except for playbooks, I couldn't find a proper definition of that on the documentation or elsewhere in the Internet), but I haven't found a comparison between them all, and it took me some time to figure this all out, so I'm sharing that by answering my own question.*

*PS2. I really don't like the fact that roles are named 'roles'. It sounds a bit counter intuitive to me. The word 'role' alone alludes to user roles or server roles, to me. So something more related to Tower (users) or inventory (servers). I think the rationale for that naming is that they define 'one of the roles that a server functions'. Same could be said of plays, but roles are more abstract as they're not tied to specific hosts. Still, I find that naming confusing.*

**Edit**

A **handler** functions for the most part just like a task. However, it is not executed by default, with the other tasks, during the playbook run. Instead, each individual handler will be executed only if notified by a task.

On a task definition, the keyword `notify` identifies the handlers the task wants to notify. If the task is executed and reports a `changed` state (not `ok` nor `failed`), then the handler will be marked for execution.

After all roles and tasks have run, each notified handler will be executed, in the order they appear in the file. Check the docs for details.

Tasks will always run (unless skipped with `when`), and are responsible for the actual changes you want to implement. Handlers will be run only in response to tasks making changes, and are generally responsible for finalization steps (restart or reload a service that has had its configuration changed to make it effective, for example).

You could probably replicate handlers behavior with tasks, but it is such a recurring pattern that Ansible provides handlers as a helper.
> 34 votes

---
Tags: ansible
---


# Question
Title: For storing avatars in s3, which approach has the lowest cost?
On my site I have the ability for users to upload avatars (which they upload to s3). They can only have one avatar at the moment, so I currently just have the filename set to their user id.

However, they may replace their avatars, which causes issues. Because the s3 is propagating to CloudFront, I need to invalidate the cache for the filename when they upload their new avatar (because it replaces the old avatar file). But with CloudFront invalidations are expensive.

It almost seems like Amazon would prefer I just not delete the old files, and instead just give the new avatar a new name and leave the old one in there, even though it's no longer being used by anyone. That way I can avoid the invalidation.

The other option is not to use CloudFront at all, and just upload to s3, and make the s3 bucket public. But the problem with this is that CloudFront is much more generous with their GET request costs and amounts (2,000,000 per month with CloudFront and only 20,000 per month with S3 buckets).

Basically, I have these 3 options, and I'm not sure which to pick, given the above:

1) Replace the files in s3 and then invalidate the file in CloudFront

* **Problem:** invalidations are expensive

2) Just dump more files in the s3 bucket and use CloudFront

* **Problem:** the old, unused avatars just sit there wasting s3 bucket space

3) Don't use CloudFront, and just replace the file in s3 and then serve it from the bucket

* **Problem:** CloudFront has cheaper GET requests and better user experience with latency

**Edit**: In this case, I'm asking about AWS with s3 and CloudFront. I'm looking to most effectively automate the process by which a user can upload an avatar to my backend service. Specifically, the overwriting of old avatars has me confused. The general process flow of "PreSigned URL -\> Upload to s3 -\> Propagate via CloudFront" seems an adequate one, but I'm not sure how to accommodate the overwriting of old, no longer used files in this plan, considering the choice seems to be between an expensive invalidation, or the keeping of stale files.

# Answer
For the best performance naming many small files in an s3 bucket may not be the best idea unless they user\_id is a random string. See how the S3 hashing algorithm works https://aws.amazon.com/blogs/aws/amazon-s3-performance-tips-tricks-seattle-hiring-event/

So replacing it by a random hashed value may not be that bad. Of course you need to store the hash in the user profile or you need to calculate it from something. If you add something like profile id or last changed timestamp you can do sha1/md5 of (userid+lastchanged) and if you update timestamp you has the hash of the current avatar.

The older one you can simply delete. CloudFront will remove it from its cache after expiration https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/Expiration.html

This will more complex algorithm, but if the costs are the concern, a little bit of extra work seems to be a reasonable approach. Not sure how do you handle the uploads, but it may be a good opportunity to play a bit with step functions which would make sure that the new avatar is uploaded correctly and then the old one is removed as well.

It's kind of a programming question, but I get your point - it is very specific to AWS environment and requires a particular knowledge about how the infrastructure works.
> 3 votes

---
Tags: amazon-web-services, amazon-s3, amazon-cloudfront, cache
---


# Question
Title: What are the implications for Docker Community Edition of the Docker acquisition by Mirantis?
November 13, 2019, Mirantis Acquires Docker Enterprise Platform Business. The same day, Docker Inc. claims they will now focus on docker Desktop and Docker Hub.

Given those changes, will Docker CE continue to share the same codebase as Docker Enterprise, or should we consider it now as a fork like MySQL vs MariaDB?

# Answer
I'd ask the question the other way around, rather than will Docker CE continue to share the Docker Enterprise codebase, it's whether Docker Enterprise will continue to share the upstream Docker CE / Moby codebase. They've always maintained an internal closed source fork for the Enterprise code, which has additional features. However, the development process of CE has always been open source, and features added to CE were mirrored from CE to the Enterprise codebase. The acquisition of Enterprise didn't change that it was already a fork of CE. And the acquisition did not include the CE code or any of the other open source projects, those continue to be managed by Docker, which they will maintain as a core component to their business (CE is embedded in the various Desktop products, and Hub has little value without it).

The biggest downside I see to the acquisition is the loss of staff from Docker to Mirantis. I'm sure many people had a dual role working on Enterprise and CE, and a significant chunk of the staff went to Mirantis. That said, the CE repos are open source, Mirantis staff can submit PR's for changes they'd like usptream, and they can pull down new releases of CE into the Enterprise product. The end result for users may end up looking identical to what we have today, just two close companies coordinating, rather than two sides of the same company. From an open source perspective, having more outside contributions is actually a good thing, since it forces governance and planning into the open, that in the past may have happened with internal meetings.

Disclaimer: I have been exposed to private discussions from the Captains program on this topic (not much since they are still working this out) and have done my best to keep my answer to only the public knowledge.
> 3 votes

---
Tags: docker
---


title: AWS Batch use existing ECS cluster

text:
# Question
Title: AWS Batch use existing ECS cluster
I would like AWS Batch to use one of my existing ECS clusters.

The problem is that when I create an unmanaged "compute environment" from the AWS Batch service, it doesn't allow me to specify an existing ECS cluster and it will automatically deploy a new ECS cluster (It doesn't let you choose). The reason why I want to associate my existing ECS cluster with the unmanaged "compute enviromnent" is because I don't want to pay for two ECS cluster when I already have one...

Any suggestions/alternatives?

# Answer
I don't think what you want to do is possible.

I suggest you to use a managed compute environment so you don't care about cluster resources. Set the min and max CPU for the compute environment (min 0 is recommended so you don't waste many when no jobs are running). Then only set the resources needed by your jobs and batch will add compute capacity to satisfy the demand.

Your costs will be the same because batch will create an ASG(autoscaling group) and ECS cluster but it only charges you for EC2 instances.
> 1 votes

---
Tags: amazon-web-services
---


# Question
Title: How to verify hashes of Ansible Galaxy dependencies?
From a security perspective, I don't like the idea of relying only on Git tags, as they could be moved later. Unfortunately, I'm not aware of an official solution to enforce checks.

This is the best that I could come up with:

```
# requirements.yaml:
- src: git+https://github.com/geerlingguy/ansible-role-nodejs.git
  version: 405a113ccbde0c99614f6f815fb4285cb742ad25 # == release 5.0.0
  name: geerlingguy.nodejs.trusted_hash

```

And in the playbook, use it as:

```
- roles:
  - role: geerlingguy.nodejs.trusted_hash
    nodejs_version: "12.x"

```

It works, but feels a bit overly complicated. Is there an easier way?

# Answer
The feature is currently (Nov 2019) not directly supported in Ansible Galaxy.

There have been related discussions in the past (#14604 and #1358), but the latest state can be followed in the new issue #2108.

In the meantime, the workaround described in the question can be used, at the cost of some verbosity.
> 1 votes

---
Tags: ansible, security, ansible-galaxy
---


# Question
Title: Will data contained withing Persistent Volume survive cluster deletion?
I have a Kubernetes cluster based on 3 hosts managed via Rancher

I've created Persistent Volume on the first host and use it via a claim by a PostgreSQL pod. My app is generating and storing there some data.

Kubernetes docs state that "Persistent volume lasts through "Cluster lifetime".

My question is - if I will delete a cluster and then create it again, and then create Persistent Volume on the same host (first one) with the same configuration will data be still there?

My confusion stems from the fact that I don't quite get what is considered as "persistence" here. "Persistent Volume" entity within-cluster will obviously get deleted with a cluster. But whats about the data?

# Answer
> 3 votes
It depends on what your `persistentVolumeReclaimPolicy` is. If you manually created/defined the PersistentVolume, then the default action is to keep the data. (This sounds like what you're doing.)

If you are using something like EKS on Amazon, your PV's are dynamically generated EBS volumes. In that case, they will be deleted by default.

A little bit more about reclamation: https://kubernetes.io/docs/concepts/storage/persistent-volumes/#reclaiming

---
Tags: kubernetes, rancher, persistent-volumes
---


title: Jenkins triggered by multiple git repos

text:
# Question
Title: Jenkins triggered by multiple git repos
I've been looking around to see how to have ~10 different repos all on a git push trigger a single pipeline that does pretty much the same exact thing - build a docker image -except that the different repo names will be used for the docker image repo and the specific git commit will be the tag.

Is this possible or do I have to break it down into multiple pipelines? Ideally I would be able to have one that queues jobs when multiple pushes happen in a short period of time.

# Answer
> 1 votes
I would be inclined to use Pipeline Templating as this gives you the ability to define standard templates centrally but leave the SCM field to be defined by the individual job.

> Leveraging templates in this way lets you separate the business logic (what should happen when) of your pipeline from the technical implementation (what’s actually going to happen). The result of this is a CI/CD pipeline that’s proven to be significantly easier to manage when supporting multiple teams simultaneously.

I recommend that you **do not** have a single pipeline for all repositories as that is going to make it very hard to tell which repository is failing or slow via dashboards and alerts.

# Answer
> 0 votes
I use Pipeline Shared Libraries to share a single build pipeline across roughly 180 different repositories.

The pipeline itself is defined in a function in the shared library, and the Jenkinsfile for all of the repos is just a one-liner that calls that function.

There's also an example of how to share a build pipeline using shared libraries in an official Jenkins blog post.

---
Tags: docker, jenkins, continuous-integration, jenkins-pipeline
---


# Question
Title: How to pass groovy output to pipeline variable
I have the below code in the jenkisn pipeline:

```
   stage ("amd_distribution_input_transformation"){
        steps{
            script{
                    amd_distribution_input_transformation url: params.DOMAIN_DESCRIPTOR_URL, secret: params.CERDENITAL_ID
                }
            }
        }

```

groovy code:

```
def call(Map parameters)
{
    def CREDENITAL_ID = parameters.secret
    def DOMAIN_DESCRIPTOR_URL = parameters.url
    sh '''
        python amd_distribution_input_transformation.py
      '''
    }
}

```

in the amd\_distribution\_input\_transformation.py some code is running, and at the end, it returns JSON object named 'artifacts\_list'

my question is, how can I access this object in my next stage in the pipeline:

```
stage ("create bundle"){
            steps{
                script{
                    amd_distribution_create_bundle credential_id: artifacts_list
                    }
                }
            }

```

# Answer
> 1 votes
You can send the output of your python script to `stdout` and capture it in a environment variable and that variable can be passed to other stages in the pipeline. Also, the environment variables can be used inside the scripts.

You can do something like this:

```
def call(Map parameters)
{
    def CREDENITAL_ID = parameters.secret
    def DOMAIN_DESCRIPTOR_URL = parameters.url
    env.artifacts = sh(script: "python amd_distribution_input_transformation.py", returnStdout: true )

    }
}

```

Here, the output of the script `amd_distribution_input_transformation.py` will be stored in the variable `artifacts` and will be available further.

---
Tags: jenkins-pipeline, groovy
---


# Question
Title: What is a virtual repository in JFrog Artifactory / Artifact Repository Manager?
I want to understand what is a virtual repository in the context of Artifacts Repository Manager.

Before that, I referred the documentation. But, I didn't get the explanation as I was new to DevOps that time. Thanks.

> A virtual repository is a collection of local, remote and other virtual repositories accessed through a single logical URL.
>
> A virtual repository hides the access details of the underlying repositories letting users work with a single, well-known URL. The underlying participating repositories and their access rules may be changed without requiring any client-side changes.

Above is the documentation definition, my doubt is how virtual repositories are connected to either Local/Remote Repository.

# Answer
Local repositories serve artifacts from your local storage. These are found in paths like this one:

`http://<host>:<port>/artifactory/<local-repository-name>/<artifact-path>`

Remote repositories act as proxies to remote locations which may e.g. be other Artifactory servers. They also provide caching. These are found in paths like this one:

`http://<host>:<port>/artifactory/<remote-repository-name>/<artifact-path>`

Virtual repositories provide a way to logically group many repositories under a common name. Example path is following:

`http://<host>:<port>/artifactory/<virtual-repository-name>/<artifact-path>`

There is a default virtual repository called `repo` which groups all local and remote repositories.

Please try this article from JFrog for more details. It covers local, remote and virtual repositories in a friendly way:

https://www.jfrog.com/confluence/display/RTF20/Understanding+Repositories
> 4 votes

# Answer
> A virtual repository is a collection of local, remote and other virtual repositories accessed through a single logical URL.
>
> A virtual repository hides the access details of the underlying repositories letting users work with a single, well-known URL. The underlying participating repositories and their access rules may be changed without requiring any client-side changes.

Quote from the JFrog Artifactory User Guide.
> 3 votes

# Answer
Artifactory allows you to define a virtual repository which is a collection of local, remote and other virtual repositories accessed through a single logical URL.

A virtual repository hides the access details of the underlying repositories letting users work with a single, well-known URL. The underlying participating repositories and their access rules may be changed without requiring any client-side changes.
> 2 votes

---
Tags: artifactory, nexus
---


# Question
Title: Kubernetes - Deploying older version of app in new API
I'm working with yaml files built for an older version of the API

```
apiVersion: extensions/v1beta1

```

I tried changing to

```
apiVersion: apps/v1

```

But couldn't figure out the correct "selector" formatting. So I changed back to v1beta1

Then I ran across this command:

```
kubectl convert -f . | kubectl -n robot-shop create -f -

```

All seemed to deploy correctly. But now when I try to reach my app, I get connection refused. After some investigation, I took a look at iptables and see this:

```
iptables -L | grep 30080
REJECT     tcp  --  0.0.0.0/0            0.0.0.0/0
/* robot-shop/web:8080 has no endpoints */ ADDRTYPE match
dst-type LOCAL tcp dpt:30080 reject-with icmp-port-
unreachable

```

After some more research, this seems to have something to do with kube-proxy and "selector".

Question is, what's the right course of action?
1. Change the fw rule to ACCEPT
2. make the correct changes to yaml file (need help with that)
a. If so, do I delete the deployment and then make correct changes, then redeploy?

Here's my deployment yaml file:

```
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
  annotations:
    kompose.cmd: kompose -f ../docker-compose.yaml convert
    kompose.version: 1.10.0 (8bb0907)
  creationTimestamp: null
  labels:
    io.kompose.service: web
  name: web
spec:
  replicas: 1
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        io.kompose.service: web
    spec:
      containers:
      - image: linuxacademycontent/rs-web:latest
        name: web
        ports:
        - containerPort: 8080
        resources:
          limits:
            cpu: 200m
            memory: 100Mi
          requests:
            cpu: 100m
            memory: 50Mi
      restartPolicy: Always
status: {}

```

And my service yaml file:

```
apiVersion: v1
kind: Service
metadata:
  annotations:
    kompose.cmd: kompose -f ../docker-compose.yaml convert
    kompose.version: 1.10.0 (8bb0907)
  creationTimestamp: null
  labels:
    io.kompose.service: web
  name: web
spec:
  type: NodePort
  ports:
  - name: "8080"
    port: 8080
    targetPort: 8080
    nodePort: 30080
  selector:
    io.kompose.service: web
status:
  loadBalancer: {}

```

# Answer
> 1 votes
Prior to apps/v1 the selector field was implicitly picked up from the pod template's labels. So in your case it would be:

```
spec:
  selector:
    matchLabels:
      io.kompose.service: web

```

Starting with apps/v1 an explicit selector field is required by the Kubernetes API.

I would suggest not to change the iptables on your own as it it the responsibility of your Kubernetes network plugin and is most likely done well.

As far as I am concerned you may simple update the Deployment manifest and no uninstall is required.

---
Tags: kubernetes
---


# Question
Title: Trunk Based Development - release branch testing
We are currently using Git flow branching strategy and looking to go to a trunk based strategy - although we are quite a ways away from getting there I am trying to get all my ducks in a row, prioritize and eventually get there. I feel that its not practical to go from having multiple branches with merges etc directly to just having one branch with automated CD from that branch. I want to do this in steps while we work up our automated tests, code coverage etc. So, until then we will have to batch up our features for releases.

I was looking at something similar to this:

I could have a release branch where I would cherry pick the commits and batch them up. Obviously, we would have automated tests etc. running on the master branch. But, what about the release branches? What sort of tests should run on the release branch and at what frequency? Running all tests on both branches seems like an overkill. Since the commits are already tested in master branch, should we run only regression tests on release branch to identify any potential issues? Maybe also a CI build on the release branch that at least makes sure that the release branch is in a buildable state after a cherry-pick commit is completed?

I would appreciate your thoughts about this. Thanks for your time...VK

# Answer
> 3 votes
There are different strategies and, depending on your specific problem domain, you need to decide what makes sense for you.

For server side applications, especially micro services, you typically don't need release branches. You can work with pull requests, but once they hit master, it is common to deploy (from master) and never look back. Managing release branches comes with a cost and for server applications, there is not much benefit from my experience.

For other type of applications, release branches do make a lot of sense. In my current job, I'm working partly on Browser extensions where releases need to be planned. In contrast to the server side, pushing a broken extension release is something that you want to avoid at all costs, especially as rolling back is not possible. That is why we have code freezes (every two weeks or so).

At code freeze, we create a release branch from the current master. Trunk based development continues as before, but all new changes will not make it in the upcoming release. Instead there is a testing and stabilization window, where only bug fixes will be accept on the release branch. When a bug is found, it will be fixed on master first, and only then cherry-picked to the release branch. Once bugs are fixed, the release goes out and the whole process repeats.

Stable cycles of around 2-3 weeks tend to work well. Shorter is also good, but longer cycles can be problematic, as it encourages people to squeeze as much as possible in the upcoming release. Otherwise, if you miss it, you might have to wait for months. I would strong recommend to favor multiple small and predicable release cycles over big and unpredictable ones.

Our CI system runs for all development relevant branches (i.e., on master and all pull requests). To be best of my knowledge, we do not run tests for the release branches. Normally, a well-tested bug fix from master does not create problems on the release branch. However, I see no fundamental reason not to run tests on the release branch, too.

If you set it up from scratch, I would include release branches. However, from my personal experience, master (and pull requests) alone should give you most of the benefits already. Running tests on master helps to enforce that development does not break existing tests, or introduces warnings or other coding style violations.

In our setup, the CI system (Jenkins) runs after each commit. If there are no changes, it will not be triggered. However, in my previous job, we had costly integration tests, which were executed once a day but only on master, while unit tests ran after each commit. Again, that depends on your specific project.

---
Tags: azure-devops, testing, cicd, branch
---


# Question
Title: jenkins parameter behaves differently from internally defined variable
I set up my jenkins job to take in the parameter `payload` and set its value to `test\ntest`.

My jenkins script is the following:

```
def hardset = "test\ntest"

echo "param:"
def cleanParam = jenkinsParam.replaceAll("[\\n\\t ]","Yes");
echo cleanParam

echo "----------"

echo "hardset:"
def cleanHardset = hardset.replaceAll("[\\n\\t ]","Yes");
echo cleanHardset

```

The output is (cleaned up to remove jenkins pipeline echos):

```
test\ntest

param:
test\ntest

----------

hardset:
testYestest

```

Why? What's different between reading in a jenkins parameter and setting one in the script body?

In practice, I'm trying to replace troublesome `\n` coming in from a lambda payload, but this is throwing a wrench in my plans.

doing something like `def hardsetParam = jenkinsParam` to set it in the script body doesn't seem to work either, sadly.

In response to a comment - doing `echo jenkinsParam` gives me `test\ntest`. To my eyes, it's identical to hardset.

# Answer
> 2 votes
It seems that the `jenkinsParam` is holding the output after escaping the special characters. You are assuming that `jenkinsParam = "test\ntest"` and however it looks like it stores it as `jenkinsParam = "test\\ntest"`

See how it behaves if I use echo it these two different cases:

(I have tested this in groovy shell)

**1st Case:**

```
jenkinsParam = "test\ntest"
print jenkinsParam
test
test

```

\** 2nd Case:\**

```
jenkinsParam = "test\\ntest"
print jenkinsParam
test\ntest

```

In order to replace the `payload` which is stored in the `jenkinsParam` (2nd case behaviour), you need to replace the complete `\\n` from the string. Check the example below:

```
output = jenkinsParam.replaceAll('\\\\n', 'yes')
print output
testyestest

```

---
Tags: jenkins, groovy
---


# Question
Title: Dev environment for legacy Visual Studio code?
I work at a small company that has several legacy projects that all require different versions of visual studio to run. The company has servers on site and I have a small cloud budget. I want to set up the projects such that it doesn't matter which visual studio the project is opened it, it should still run the same. The projects are all asp.net, run on IIS, and use SQL Server. I'd like to know where to get started, such as should I try to containerize the applications? Should I set up some sort of VM Image with the projects on it? Any help would be appreciated!

# Answer
> 0 votes
> ... legacy projects that all require different versions of visual studio ...

Don't you just *love* Visual Studio?

> ... I want to set up the projects such that it doesn't matter which visual studio the project is opened ...

If your "legacy" goes back beyond .Net 2, then you're out of luck.

New, shiny versions of Visual Studio won't open .Net 1.\* projects and will shrink away *in horror* from anything that needs anything created in 'Studio 6 (except, possibly, C++, the one language that's survived .Net-ification largely unscathed).

Most of the newer stuff will happily co-exist, but you'll probably need to think about containerising the earlier stuff.

---
Tags: docker, windows, virtualization
---


# Question
Title: Can I create an Azure VM from Ansible with NICs in different resource groups?
I am working on creating an Ansible playbook to

* Ensure required network resources for an Azure VM are present, and
* Create an Azure VM attached to those resources.

Due to the way the organization is structured, all *network resources* (such as NIC, NSG, Subnets and Public IPs) are placed in a `Networks` Resource Group in Azure. And these are managed by a separate team.

Also, the mandate is to create the Azure VMs and all *non network* resources in a separate resource group (let's call it `AZRVMRG`.

The chunk of the playbook that creates the VM looks like this:

```
---
- name: "Create VM {{vm_type}} - {{name}}"
  azure_rm_virtualmachine:
    resource_group: "{{rg_name}}"
    name: "{{name}}"
    vm_size: "{{size}}"
    admin_username: "{{user}}"
    admin_password: "{{pass}}"
    os_type: "{{os_type}}"
    network_interfaces: "{{nic_name}}"
    image: "{{image}}"
    tags: "{{tags}}"

```

Where `{{rg_name}}` is the `AZRVMRG` RG, and `{{nic_name}}` is an existing NIC present in the `Networks` RG.

## Question

How do I reference the NIC present in the `Networks` resource group while creating the Azure VM within `AZRVMRG` resource group?

## What I've tried

I have tried "just" referencing the NIC name, but that gives me an error similar to:

```
 "msg": "Error fetching network interface AZRVMNIC - Azure Error: ResourceNotFound\nMessage: The Resource 'Microsoft.Network/networkInterfaces/AZRVMNIC' under resource group 'AZRVMRG' was not found."

```

So clearly, its looking under the same resource group.

## Notes

(I have seen VMs created in the Azure subscription this way, so this is possible in Azure. But Ansible does not document this approach in its official docs. )

# Answer
From close reading of the Ansible doc page on creating Azure VMs, it's clear that you can refer to a resource such as a virtual network or NIC using its resource ID. This lets you specify any resource, just like RM templates.

To specify a NIC created in another resource group by the networking team, with the `networking_rg_name` var set to that group name, you would use a resource ID like this:

```
- set_fact:
    nic_id: "/subscriptions/{{ subscription_id }}/resourceGroups/{{ networking_rg_name }}/providers/Microsoft.Network/networkInterfaces/{{ nic_name }}"

- name: "Create VM {{vm_type}} - {{name}}"
  azure_rm_virtualmachine:
    resource_group: "{{rg_name}}"
    name: "{{name}}"
    vm_size: "{{size}}"
    admin_username: "{{user}}"
    admin_password: "{{pass}}"
    os_type: "{{os_type}}"
    network_interfaces: "{{nic_id}}"     # Use the nic_id instead of name
    image: "{{image}}"
    tags: "{{tags}}"

```

This works for any resources that are in a different resource group, including virtual networks.
> 1 votes

---
Tags: ansible, azure
---


# Question
Title: How to check if pipeline parameter is empty
I have a pipeline job which includes some parameters:

In the pipeline file I have the below code:

```
stage ("create bundle"){
            steps{
                script{
                        amd_distribution_create_bundle credential_id: params.DISTRIBUTION_CREDENTIAL_ID, distribution_url: params.DISTRIBUTION_URL, gps_credential_id: params.GPG_PASSPHRASE, bundle_name: params.BUNDLE_NAME, bundle_version: BUNDLE_VERSION
                }
            }
        }

```

how can I ask before calling the groovy method 'amd\_distribution\_create\_bundle', if some field is empty?

# Answer
While creating the credentials parameter in jenkins job, you can specify `required: true`, then jenkins should validate the credentials paramter.

```
parameters {
    credentials(name: 'GPG_PASSPHRASE', defaultValue: '', credentialType: "Username with password", required: true )
}

```

You can also check the details mentioned in the link https://docwhat.org/jenkins-user-credentials

Adding Parameter

```
/* EXAMPLE */
parameters {
    credentials(
        credentialType: 'com.cloudbees.plugins.credentials.impl.UsernamePasswordCredentialsImpl',
        defaultValue: '',
        description: 'The credentials needed to deploy.',
        name: 'deployCredentialsId',
        required: true
    )
}

```

Using in the stage

```
/* EXAMPLE */
steps {
    withCredentials([usernamePassword(
        credentialsId: '${deployCredentialsId}',
        usernameVariable: 'DEPLOY_USERNAME',
        passwordVariable: 'DEPLOY_PASSWORD',
    )]) {
        sh './my-command.bash --username="${DEPLOY_USERNAME}" --password="${DEPLOY_PASSWORD}"'
    }
}

```

If you just want to check if the variable is empty or not, you can check `isEmpty()` in groovy script. Your code can be as follows:

```
stage ("create bundle"){
        steps{
            script{
                    if ( GPG_PASSPHRASE.isEmpty() ) {
                             GPG_PASSPHRASE = 'custom_string'
                     }
                    amd_distribution_create_bundle credential_id: params.DISTRIBUTION_CREDENTIAL_ID, distribution_url: params.DISTRIBUTION_URL, gps_credential_id: params.GPG_PASSPHRASE, bundle_name: params.BUNDLE_NAME, bundle_version: BUNDLE_VERSION
            }
        }
    }

```
> 2 votes

---
Tags: jenkins-pipeline
---


# Question
Title: Consul: how to remove dead client nodes quickly
Background: When running Consul client-mode in AWS, EC2s are constantly getting replaced, which results in a lot of dead client nodes when looking at the Consul UI or via `consul members`.

Already tried to enabled `reconnect_timeout`, `leave_on_terminate` (default to true on Client but tried to turn on anyway), and Autopilot `cleanup_dead_servers` to no avail.

The delay seems to be inconsistent as well, some got reaped after 4~7 days, some are never reaped, `consul force-leave` had to be used.

# Answer
> 4 votes
As a best practice, you should gracefully deregister the node gracefully. In this case, consul will know the the node has left and will be removed from the cluster. Otherwise, the consul cannot distinguish between a temporary failure, agent crash, network partition, etc.

There is a github issue as well related to this.

Quoting few important points from the issue.

> The nodes should automatically reap out after 72hours (not yet configurable, but soon). Otherwise, the best route is to issue a graceful leave before destroying the nodes (consul leave), so that they can be reaped immediately. They are kept around for that long since without a graceful leave, Consul cannot distinguish between a temporary failure, agent crash, network partition, etc.

.

> force-leave should push them into the "left" state. Nodes are not reaped until they are in failed for 72h or in the left state. "force-leave" just moves a node from the "failed" -\> "left" state. They are not removed from the members list for 24 or 72h.

---
Tags: linux, consul
---


# Question
Title: How to have Kitchen CI run a script on success?
I've got a Chef repo that uses test-kitchen. I have a pretty good battery of tests, and when I run the tests and they're successful, I have a `vendor.sh` script that does some cleanup and a Berks vendor. What I'd like to be able to do is automatically run that script. I suppose I could just put `kitchen test {box}` into that script, but then I'd have to find a way to detect success (maybe this is easy) and deal with output and things like that; I'd rather just have Kitchen do it.

I googled and searched and found nothing. Does anyone know a way?

# Answer
> 3 votes
I think what you are looking for is lifecycle hooks.

https://kitchen.ci/docs/reference/lifecycle-hooks/

For example, you could have this in you kitchen.yml file.

```

suites:
- name: default
  lifecycle:
    post_verify:
    - local: <script>

```

---
Tags: chef, inspec
---


# Question
Title: Ansible configure and compile in directory
So I have have an Ansible Task that extracts a programs sourcecode zip into a certain directory and is then supposed to compile and install it. The extraction of the zip works fine, but then I need to execute the commands:

```
./configure
make
sudo make install

```

And these commands need to be execute from within the extracted directory. I tried several things to execute it, I couldnt get it to work.

```
  - name: doit
    command: chdir=/tmp/proftpd/ ./configure && make && make install
    become: yes

```

I tried this but it fails:

```
  TASK [doit] ********************************************************************
fatal: [centos]: FAILED! => {"changed": true, "cmd": ["./configure", "&&", "make", "&&", "make", "install"], "delta": "0:00:00.152824", "end": "2019-11-21 14:20:48.656070", "msg": "non-zero return code", "rc": 1, "start": "2019-11-21 14:20:48.503246", "stderr": "configure: WARNING: you should use --build, --host, --target\nconfigure: WARNING: invalid host type: &&\nconfigure: WARNING: you should use --build, --host, --target\nconfigure: WARNING: you should use --build, --host, --target\nconfigure: WARNING: invalid host type: &&\nconfigure: WARNING: you should use --build, --host, --target\nconfigure: WARNING: you should use --build, --host, --target\nInvalid configuration `&&': machine `&&' not recognized\nconfigure: error: /bin/sh .//config.sub && failed", "stderr_lines": ["configure: WARNING: you should use --build, --host, --target", "configure: WARNING: invalid host type: &&", "configure: WARNING: you should use --build, --host, --target", "configure: WARNING: you should use --build, --host, --target", "configure: WARNING: invalid host type: &&", "configure: WARNING: you should use --build, --host, --target", "configure: WARNING: you should use --build, --host, --target", "Invalid configuration `&&': machine `&&' not recognized", "configure: error: /bin/sh .//config.sub && failed"], "stdout": "checking build system type... ", "stdout_lines": ["checking build system type... "]}

```

How can I achieve this properly with ansible?

# Answer
Ansible `command` module requires only one command and cannot be changed. You should use the `shell` module at the very least. Note however that this will break the idempotency of the playbook.

You should probably split the task up into three:

* configure task using the `shell` module. Use `creates:` to tell the task to be skipped if the configure task has already created the expected output file
* make task, using the Ansible Make module
* similarly for the make install task.

This should yield the same actual result, but would be more likely to be idempotent.
> 4 votes

# Answer
I just figured it out. Splitting it up into more commands worked perfectly

```
  - name: doit
    command: chdir=/tmp/proftpd/ ./configure

  - name: a shit here we go again
    command: chdir=/tmp/proftpd/ make

  - name: another one
    command: chdir=/tmp/proftpd/ sudo make install

```
> 0 votes

---
Tags: ansible, configuration-management, vagrant, installation
---


# Question
Title: Connect Docker container to both host and internal bridge network
I am trying to run a Docker container as a router between a private (`--internal`) Docker network and the predefined `host` network. This means the container needs to have two network interfaces: One "outside" interface, that can access all host IP addresses, and one "inside" interface, that acts as a gateway for the containers in the internal Docker network

The router container itself will then NAT network traffic from/to containers.

I have not found a way to configure Docker to run the container with those two interfaces. The closest I could get is having two `bridge` interfaces assigned, which is not exactly what I need.

Trying to connect manually results in an error:

```
# docker network connect host root_router_1
Error response from daemon: Container cannot be disconnected from host network or connected to host network

```

Can anybody show me how to achieve this, preferably even whith Docker Compose?

# Answer
> 4 votes
Docker does not allow to connect a container to the host network and any other Docker bridge network at the same time. I will try to illustrate the reason with an example:

* Let us think of a container C1. Hypothetically, C1 would be connected to the host network (--net=host) and a Docker bridge network Br1 (--net=Br1).
* A second container, let us say C2, is connected to Br1.

With the above setup, my guess is that the host network is visible from C2, and I suppose this is the reason why Docker automatically prevents us from unintentionally exposing the host network to non-host-specified containers.

That being said, if we have a set of containers, and we want all of them to be interconnected, with just a single container having access to the host network, my approach would be:

* \[C2,...,CN\] are connected to a user-defined Docker bridge Br1 (--net=Br1)
* C1 is connected to the host network (--net=host)
* C1 exposes a port in order to be accessible from the rest of containers

EDIT: We would still have to adapt iptables policies in such a way that C1 can be reached from the rest of containers (see https://docs.docker.com/network/iptables/)

# Answer
> 3 votes
According to this Q&A on github:

> The host network is special. You have to use network\_mode: host on the service

---
Tags: docker, docker-compose
---


title: AWS SQS - use case for "short polling"

text:
# Question
Title: AWS SQS - use case for "short polling"
When using event driven processing of work with AWS SQS and worker clients. the use of long polling is often recommended. AWS documents this here

Essentially when there are no message in the queue, the worker does not need to keep making requests to the queue (this reduces call costs), but new messages are still delivered immediately to a worker that is long polling.

Having build small systems a couple of times with SQS now, I question what the benefit of short polling is. I imagine it may be slightly faster, as only a subset of SQS servers are polled when short polling, but if that is the only downside, what sort of additional delay am I looking at? Does it depend on the number of messages I am processing? (Does SQS servers scale based on the number of messages?)

What other downsides are there, if any?

# Answer
> 3 votes
While you can benchmark this yourself, benchmark testing has shown that there isn't much of a difference in time between long and short polling when the queue is full. Additionally, with long polling you always have data coming back unlike short polling.

As for when to use short polling the AWS FAQ covers this:

> For example, if your application uses a single thread to poll multiple queues, switching from short polling to long polling will probably not work, because the single thread will wait for the long-poll timeout on any empty queues, delaying the processing of any queues that might contain messages.

Additionally:

> ..if your application expects an immediate response from a ReceiveMessage call, you might not be able to take advantage of long polling without some modifications to your application.

If you don't want to deal with asynchronous queues and multiple threads handling the queues in your application then short polling may provide an easier development path (although you'll have to deal with empty responses).

---
Tags: amazon-web-services, amazon-sqs
---


# Question
Title: How to find the PHP version that is used by the latest `phpdoc/phpdoc` docker image?
How do I get php version used by docker image phpdoc/phpdoc?

# Answer
The Dockerfile provided in the Q&A indicates that the image uses the latest `php:7` as a base image. One could inspect the version as follows:

```
docker run -it php:7 php -v

```

**Result**

```
PHP 7.3.11 (cli) (built: Oct 25 2019 02:24:51) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.11, Copyright (c) 1998-2018 Zend Technologies

```

---

**How to use an older version of php in conjunction with this container?**

It is possible to use a specific version by checking the tags of php on dockerhub. According to this list, `7.2.25` is currently the latest `7.2`. If one changes the from of the Dockerfile to `FROM php:7.2.25` and build the image again, then `7.2.25` instead `7.3.11` will be used inside the container.
> 1 votes

---
Tags: docker, php
---


# Question
Title: Build Docker Images with Ansible + Vagrant
So I am working on a research project and I am trying to automatically generate specific environments on different virtualization techniques.

So, first I made a Vagrantfile that pulls an Ubuntu image and then runs an Ansible playbook. I have a playbook that defines everything for my environment. Which means it copies files, installs and configures everything. For VirtualBox this is no problem, - it spins up my environment in VirtualBox.

Now I want to do this exact same thing, but on Docker. Normally one would write a docker file to define the environment. What I want to do is re-use the Ansible script to define the environment (despite being a bit counter-intuitive). While there seems to be some support for Docker in Vagrant - what is the best way to achieve what I want. I want to create a Docker Container - exactly how specified in the Ansible playbook. (Basically re-using the playbook I also used to setup VirtualBox). The output should then be a docker image that I can just start with docker run.

Whats the best way to do this? What do I have to look out for?

My current idea is the following:

```
1) Setup a VM with Vagrant.
2) Install Ansible and Docker in that VM.
3) Create a new Docker Image with Ansible. It just defines the Container OS and maybe opens a port so Ansible can SSH into the container?
4) Run the playbook in the container to configure everything.
5) Package the entire thing as a new image
6) Transfer packaged docker image to the host machine

```

# Answer
> 3 votes
If it's within scope, I would use Packer to build the artifacts.

Note that you need to make sure that the Ansible playbook will work in both types of virtualised environments.

This has the benefit of producing arbitrary types of artifacts, based off of a single input. You can use a single workflow for both the Vagrant and the Docker images, and they will be built off of the same state of the codebase (*i.e.* git hash).

---
Tags: docker, ansible, vagrant, environments
---


# Question
Title: Condition CircleCI build on what part of monorepo changed?
I'm working on a monorepo project that contains a few different subprojects in different languages. Right now I run three different CircleCI "jobs" on each commit. However, each of these packages compile and test themselves independently. Hence I don't actually have to spend time on running all three jobs every time.

Is there some way to condition a CircleCI job on which directory you committed code to?

# Answer
> 3 votes
This discussion gives a pretty good suggestion on how to accomplish what I want.

https://discuss.circleci.com/t/does-circleci-2-0-work-with-monorepos/10378/11

# Answer
> 2 votes
We're using CircleCI with **Go** monorepo.

Here’s how it’s done:

1. Define a job for each service in circle config yaml.
2. A git push triggers CircleCI job that finds which services are part of the change.
3. For each service run a CircleCI job with:

```
{
  "build_parameters": {
      "CIRCLE_JOB": "my-service"
  }
}

```

Also, published as an open source: https://github.com/Tufin/circleci-monorepo

# Answer
> 0 votes
I came here in addition of previous answer of @effi-bar-shean and @langkilde which give a good overview of the problem. My answer will be focus for JavaScript

> How to build on part of monorepo that have changed ?

You can use the 2.0 API of CircleCI which allow you to trigger multiples workflow separately. If you have a JavaScript monorepo you can use a tool like lerna to trigger the workflow of your choice

Example: The output of `lerna changed -a` gives you all repository that have changed then you have to trigger the workflow of theses packages. You can see a full example here

Here is an example of repository for Javascript monorepo

I hope my answer help you

---
Tags: continuous-integration, continuous-delivery, build-pipeline, circleci-2.0, monorepo
---