# Question
Title: ansible-playbook: how to dynamically, conditionally set a large deployment environment (variable space) in a playbook?
I am using an ansible playbook to deploy a php web application in aws. The deployment is repeated many times, with a different environment, for each customer. The different scenarios range from single server deployments to full deployments using elastic load balancers, and use varying code repositories based on product type.

To allow a human to deploy this app repetitively and adjust the deployment type and environment, I am running `ansible-playbook` with `--extra-vars "@deployment_vars_file.yml"` and copying and filling this vars file on a per-deployment basis, instead of putting the variables in `group_vars/all.yml`, so that I can keep a copy of the variable space I set for each deployment, and avoid overwriting (or .template suffixing) `group_vars/all.yml` and multiple roles' `vars/main.yml` files, for each deployment. I rarely rely on variable precedence, since the role and play variables inherit (are constructed out of) my deployment vars file variables.

My roles are already run conditionally at the play level, and I have a handle on how to run tasks/blocks/roles/plays conditionally, depending on the environment type that is defined in the `--extra-vars` file.

I need to define many **other** variables, dynamically (during the playbook run), conditionally (based on the values of variables in my `--extra-vars` file), with as little work as possible (i.e., in one place), at the beginning of the playbook. My roles' `vars/main.yml` files, as well as `vars:` play keywords, will inherit these other variables, (in addition to directly inheriting some of the `--extra-vars` variables). *The values of these other variables need to persist at the playbook level, not just within each play.* I don't want to have to do this in multiple locations with the same lines of code.

How can I do this, and what is the most simple way?

# Answer
Yes, an input host file for ansible looks like https://pastebin.com/Vtfw5FtV The upper part node stuff is generated using terraform, we can dynamically up/down scale the number of nodes with terraform and terraform outputs the yaml used for the hosts part of the file. The tenant-specifiek yml vars (about 150 additional lines of tenant specific config yml) are kept in a separate git location and a jenkins step/python concatenates them together and pushes them to the host directory location for each tenant and runs individual roles (hotfix) or all of 'em (full deploy) or perhaps just stop/start. The Jenkins job is pretty much a control screen with a lot of dropdowns, radio buttons and selectors. We can select how many nodes we want for a single service, which build to grab from nexus, do a full deploy, a single service hotfix or just a configuration hotfix. All this is selectable/executable per environment, per tenant and per individual module/service. The Jenkins screen is dynamically build from the host (targets) & roles (actions) directory.

Again, we're most likely abusing Ansible "best practices" a fair bit but for us this approach has been working remarkably well for last \>3 years in terms of overall manageability and gives us maximum flexibility of the whole CI/CD pipeline.
> 1 votes

---
Tags: amazon-web-services, ansible, amazon-ec2, deployment, php
---


# Question
Title: How CircleCI, Terraform and AWS should interact in CI/CD Pipeline
I'm new to devops and I'm have a hard time finding resources to help me with this question. My Goal is to have an automated CI/CD pipeline to launch my project.

Basically I have a GitHub repo with a Python Web application. I've also configured a pipeline in CircleCI that runs some tests in the project. That's what I've gotten to work so far.

Now, from what I've understood, this is what I would do next (This is what I'm unsure of so please correct me):

* In circleCI, generate a docker image with instructions on what kind of operating system etc and commands to pull project from GitHub repo and run database migrations etc.
* CircleCI runs terraform to launch this docker file in an EC2 Instance.

So my question is basically this:

1. Should the docker image be generated dynamically in CircleCI or static in the repo?
2. How does my source code get to the EC2 instance. Configure a `git pull` in dockerfile?
3. How do I tell CircleCI/Terraform how many instances I want to launch?

I know this is a question that would take too long to explain in full here. So I'm only looking to understand the basic workflow of how GitHub-\>CircleCI-\>Terraform-\>AWS interactive with eachother. Or any resources that explain this well.

# Answer
> 1 votes
There can be many correct answers, depending on the strategy you are doing. My take would be:

1. I wouldn't create a dedicated Docker image (e.g. index.docker.com/python:3.9-slim) with the application on it, but rather use the official Python Docker image, to build the app and the same to run it, so I'd package your app separately from the image that runs it. You can still compile your code as a Docker image if that's convenient and mount it as a volume. There are many benefits of this:

* your code can be mounted as read-only, lowering the attack-surface
    * faster build (no need to add a full Python every time), faster deployment (yes, Docker layers would still work, but if your code changes, Python has updates, you might have multiple hosts eventually don't count on layering).
    * separation of concerns: just because you update your Python interpreter's patch version (e.g. for security fixes), your app don't have to change. This can be an important factor if you need to have a decision log for your releases.
2. You can use, e.g. CoreOS and use a Systemd unit file to pull down and run it for you, or you can instruct your Docker daemon via the Terraform provider to run it and set the "restart = always" so it will be started on reboots too. I'd go with the latter as it requires no knowledge on the OS and there is a single way (Terraform) to manage the whole deployment.
3. Well, if you have multiple instances, high-availability, then you either

* need to do this in your app with a service-discovery tool (check e.g. Hashicorp Consul) or you can have a load-balancer, but that costs you money. Using load-balancers is the lazy way; using service discovery and doing the load-balancing on the client-side (app) offers you the best outcome, lesser architectural complexity, but much more work.
    * Use an orchestrator, e.g. Kubernetes, but then your whole manual deployment with Docker and Terraform is not relevant anymore because you are dealing with the deployment at a higher level. Although Kubernetes has a steep learning curve initially, it is very straightforward when it is up and running.
    * A runner-up solution could be a DNS-based solution. In this case, you manually set, e.g. 3 instances and then in your DNS zone, you set a round-robin.
    * You can't use, e.g. AWS Autoscaling Groups, because you want to manage your containers via Terraform, so if the ASG scales up an instance, somehow your Terraform needs to run. So unless you start up your containers via "user-data", ASGs doesn't make sense, but in that case, you can't manage them via Terraform because Terraform is likely aware of the container's state. Even if you manage it, it will be a fragile architecture that will bite you back when you need that high availability.

Note: You probably need at least 2 Terraform run, because the Docker Terraform provider requires access to the target Docker service via TLS/SSH or Unix socket, so your instances have to be up and ready *before* you run the Docker one. But that's a good idea anyway because if you add all these in your user-data/cloud-init/Ignition script, Terraform will destroy the instances, so your container updates won't be that quick.

Note 2: Do yourself a favour and if you need more than 1 instances, use an orchestrator like Kubernetes on Hashicorp Nomad.

---
Tags: amazon-web-services, terraform, amazon-ec2, github, circleci
---


# Question
Title: Minikube for Ubuntu 20.04 WSL stuck on 'Restarting existing docker container for "minikube" ...' after minikube start and 'StartHost failed' error
I've come across the following problem where my minikube cluster doesn't start and the command line becomes stagnant in the following way:

```
$ minikube start
�  minikube v1.17.1 on Ubuntu 20.04
�  Kubernetes 1.20.2 is now available. If you would like to upgrade, specify: --kubernetes-version=v1.20.2
✨  Using the docker driver based on existing profile
�  Starting control plane node minikube in cluster minikube
�  Restarting existing docker container for "minikube" ...

```

This all started after running `minikube stop` and then `minikube start` again after troubleshooting a kube-prometheus-stack helm release's pods not running. I've tried restarting my Docker daemon, rebooting my computer, searching online of course, and do see the minikube docker container created just after running `minikube start` despite the "StartHost failed" error and being stuck on updating the running docker container. I am not connected to a VPN, though I use one often.

Any help would be appreciated. Thank you!

# Answer
Not sure if complete reinstall is a valid option for you, but for me this helped:

1. `sudo minikube delete`
2. After delete is finished, `minikube start` again
> 1 votes

---
Tags: docker, kubernetes, minikube, ssh
---


# Question
Title: AWS ELB Application Load balancer, SSL not working
I've setup an "Application Load balancer" in AWS. It has two Listeners, one on HTTP (port 80) and one on HTTPS (port 443). It also has one target group that with only one FARGATE instance that I've registered in ECS.

In the Image below you can see the two listeners and the SSL Cert. I've attached to the HTTPS listener. You can also see that they have the same rule defined. Both of them target the same TargetGroup, over HTTP (port 80). From what I've understood that's an accepted practice to have a Load balancer listen to HTTPS but forward the request over HTTP since it's in the same VPC.

But for some reason only the HTTP listener is working. I've visited the site over HTTPS both directly through the LB's DNS Name and the domain I've attached to the SSL cert. both I both of the give the same result, it loads a while until it gets a timeout.

Any guess what could be wrong?

# Answer
> 1 votes
I solved it, I had forgotten that I had to add port 443 to the security group that my load balancer group belongs to.

---
Tags: amazon-web-services, amazon-ecs, load-balance, loadbalancing
---


# Question
Title: How do you calculate the compound Service Level Agreement (SLA) for cloud services?
Cloud services hosted by Amazon Web Services, Azure, Google and most others publish the **S**ervice **L**evel **A**greement, or SLA, for the individual services they provide. Architects, Platform Engineers and Developers are then responsible for putting these together to create an architecture that provides the hosting for an application.

Taken in isolation, these services usually provide something in the range of three to four nine's of availability:

* Azure Traffic Manager: 99.99% or 'four nines'.
* SQL Azure: 99.99% or 'four nines'.
* Azure App Service: 99.95% or 'three nine five'.

However when combined together in architectures there is the possibility that any one component could suffer an outage resulting in an overall availability that is not equal to the the component services.

## Serial Compound Availability

In this example there are three possible failure modes:

* SQL Azure is down
* App Service is down
* Both are down

Therefore the overall availability of this "system" must lower than 99.95%. My rationale for thinking this is *if* the SLA for both services was:

> The service will be available 23 hours out of 24

Then:

* The App Service could be out between 0100 and 0200
* The Database out between 0500 and 0600

Both component parts are within their SLA but the total system was unavailable for 2 hours out of 24.

## Serial and Parallel Availability

In this architecture there are a large number of failure modes however principally:

* SQL Server in RegionA is down
* SQL Server in RegionB is down
* App Service in RegionA is down
* App Service in RegionB is down
* Traffic Manager is down
* Combinations of Above

Because Traffic Manager is a circuit breaker it is capable of detecting an outage in either region and routing traffic to the working region, however there is still a single point of failure in the form of Traffic Manager so the total availability of the "system" cannot be higher than 99.99%.

How can the compound availability of the two systems above be calculated and documented for the business, potentially requiring rearchitecting if the business desires a higher service level than the architecture is capable of providing?

If you want to annotate the diagrams, I have built them in Lucid Chart and created a multi-use link, bear in mind that anyone can edit this so you might want to create a copy of the pages to annotate.

# Answer
> 37 votes
After reading Tensibai's excellent answer, I realised I used to be able to calculate this for network analysis purposes. I dug out my copy of High Availability Network Fundamentals by Chris Oggerino and had a crack at working this out from, not quite first principals.

Taking my serial example directly out of Tensibai's answer is simply a case of multiplying the probability of each component being available by the other:

So

> 99.95% * 99.95% = **99.9%**

Calculating it in parallel is a little more complicated as we do need to consider what the percentage *un*availability will be:

The calculation is done as follows:

1. Multiply the *un*availability of the two regions together.

> 0.1% * 0.1% = **0.0001%**
2. Convert that back to availability

> 100% - 0.0001% = **99.9999%**
3. Multiply the Traffic Manager availability by the availability of the two regions.

> 99.99% * 99.9999% = **99.9899%**
4. The result is the whole system availability.

> 99.9899% is close to **99.99%**

I ended up using Excel to perform the calculations, here is the values:

... and the formulas ...

# Answer
> 24 votes
I'd take that as a math problem with the SLA being the probability of being OK.

In this case we can rely on probability rules to get an overall.

For your first case the probability that App Service (A) and Sql Service (B) are down at the same time is the product of their probability:

```
P(A)*P(B) = 0.0005 * 0.0005 = 0,00000025

```

The probability that one of them is down is the sum of their probability:

```
P(A)+P(B) = 0.001

```

When two events are independents the resulting formula to take in account the probability of both being down is:

```
P(A,B) = P(A) + P(B) - P(A)*P(B) = 0.001 - 0,00000025 = 0,00099975

```

So the overall SLA would be `1 - 0,00099975 = 0,99900025` wich in percent is `99.900025 %`

A simplification is the product of the first probability: `0.9995 * 0.9995 = 0,99900025`.

Applied to your 1h/24h outage (4,166666% of a day) this gives (decimals are abbreviated):

```
0.0416 + 0.0416 - (0.0416 * 0.0416) = 0,081597222

```

So the probability of being OK is `1 - 0.0816 = 0.9184` in percent: `91,84%`

```
24 * 0.0816 = 1.95 h

```

This is less than the worst case of 2 hours because there's a chance both are down at the same time.

Keeping that in mind, you may notice the availability for each is `95,84%` and `0,958333333 * 0,958333333 = 0,918402778` which is our `91.84%` from above (sorry for the full decimals here, but they are needed for the demonstration)

Now for your second case, we'll start gain from our compound probability for each region (Sorry I dismissed the change for SQL to keep it reasonable), assuming there's no independent probability for the region itself and that each region is isolated and as such a DB failure take only its region down.

We have the traffic manager OK probability `P(T) = 0.9999` and each app+DB couple with a OK probability `P(G) = 0,99900025` from

How much region we have play a role as we have to apply the product of failure probability only to get the probability both region are down as the same time:
`0,00099975 * 0,00099975 = 0,0000009995000625` which means an overall availability of at least one region of `99,049375 %`

Now we have the overall regions availability, the product with the traffic manager one give us the overall availability of the system:

```
0.9999 * 0,9999990004999375 = 0,99989900059988750625

```

The overall availability is `99.989900 %`

Another source as explanation is available on Azure's docs (link courtesy of Raj Rao)

# Answer
> 0 votes
In the second diagram you multiply the two regions to calculate a full system availability. In all cases you are calculating a probability that the system shall be down (or up).

There is however, no reference to the worst case scenario where each component fails in a serial manner, and the duplicated components fail together. (i.e. traffic manager fails, comes back as both app services fail (or app-A and SQL-B), they are in turn restored as both SQL database fail)

So a dual site solution can't provide any more guarantee than a single site solution in the worst case. But can only reduce the probability of an issue occurring.

---
Tags: architecture, cloud, sla, infrastructure
---


# Question
Title: How to get into an App Container Manually with Garden-RunC Backend?
So I have been trying this-tutorial to access the runC container from the diego-cell VM. However, when I try to run

```
/var/vcap/packages/runc/bin/runc exec -t <instance_id> /bin/bash

```

I get the error

```
exec failed: container "<id>" does not exist

```

when I run cfdot for the given application it does list this instance-id, also when navigating into the location

`/var/vcap/data/garden/depot/<container_id>` it exists.

But runc in unable to open the container, also when I run

```
runc list

```

the list of running containers in the vm is empty

```
diego-cell/49dbeca6-36eb-4910-94b9-48960891a0b1:~# /var/vcap/packages/runc/bin/runc list
ID          PID         STATUS      BUNDLE      CREATED     OWNER
diego-cell/49dbeca6-36eb-4910-94b9-48960891a0b1:~#

```

I've run all as root as mentioned in the link.

Why am I not able to access the runC containers, What am I missing?

P.S: Although the intent is same, I'm not looking for cf ssh

# Answer
I suspect this may be something coming from containerd changes - running PCF 2.5 the runc list works fine, 2.6 runc list shows nothing in the inventory. I have *not* yet found the equivalent command that will allow me to find the container id and how to execute a shell entry to that container instance yet, but I'm hot on the trail.
> 1 votes

# Answer
What @Jon Ravenscraft posted as a comment on his own accepted (non) answer did in fact work for me as a solution. I am just formatting it here as an answer for anybody who stumbles here in the future:

* Follow these steps to get instance GUID (also included below for convenience)

```
# Establish the process-guid for the app:
cf curl /v2/apps/$(cf app <app_name> --guid)| jq -r '.metadata.guid + "-" + .entity.version'
b06b2901-2093-4be4-9f37-f7788d04ce8c-ff93f0e2-285f-4f18-a6fc-806802f3dde9

# Find the hosts where the app instances are running:
cf curl v2/apps/$(cf app <app-name> --guid)/stats | grep host

# Identify the Diego cell you want to ssh to by running bosh vms | grep <cell_ip>

# bosh ssh to the Diego cell (grab the cell_id from step 3).

# Once in the Diego cell, as root, run the following command to grab the container_guid/instance_guid:
cfdot actual-lrp-groups-for-guid <process_guid_from_step_1> | jq '.instance | select (.address == "<diego_cell_ip>")' | jq .instance_guid

```
* Then, from your terminal on the appropriate diego cell:

```
sudo /var/vcap/packages/containerd/bin/ctr -a /var/vcap/sys/run/containerd/containerd.sock -n garden tasks exec --exec-id my-shell --tty <instance-guid> /bin/sh

```

And voilà , you are dropped at a shell...

P.S. I believe the speculated reasoning given in the other answer is in fact, also correct, per https://docs.pivotal.io/pivotalcf/2-6/pcf-release-notes/runtime-rn.html:

> PAS v2.6 enables containerd mode by default to bring PAS in line with other container providers. When you upgrade to PAS v2.6, Garden switches the container runtime from runc to containerd. Since the restart of Garden and the re-creation of containers are a normal part of a PAS upgrade, there is no operational impact to running applications.
> 0 votes

---
Tags: containers
---


title: yarn server with docker-compose

text:
# Question
Title: yarn server with docker-compose
So I am trying to start up an server that has:

* a mongodb container
* a backend container
* a client container

Both server containers require me to run `yarn start` at the end. However, when I run this in the dockerfile the image never stops being built as the `yarn start` command technically never stops running.

The client requires the backend to be up already, and the backend requires the mongodb db instance to be up.

How do I get this server running in docker compose?

Here is my backend dockerfile

```
 FROM node:12-alpine

 WORKDIR /app
 COPY ./server /app

 WORKDIR /app/server
 RUN yarn
 RUN yarn start

```

It really does work until I get to yarn start (Which I should probably run in the docker-compose as it needs to mongodb container to be running. I just don't know how to run this in the `docker-compose.yml`, or force the client to wait for this command to be run before running its own `yarn start`).

# Answer
> 1 votes
`yarn start` should be the `CMD` of your docker image (I.e what is run when you start a container) and not a `RUN` command (which is run when creating the image used to satrt a container later)

From your exemple, something like this should be better:

```
 FROM node:12-alpine

 WORKDIR /app
 COPY ./server /app

 WORKDIR /app/server
 RUN yarn
 CMD yarn start

```

---
Tags: docker, docker-compose
---


title: How can I install Jenkins on Azure?

text:
# Question
Title: How can I install Jenkins on Azure?
Currently I have Jenkins installed on Windows Server as a Service but I am looking at installing Jenkins on Azure but the MSDN docs talk about creating a Jenkins server on an Azure Linux VM from the Azure portal, outlined in the following link: https://docs.microsoft.com/en-us/azure/jenkins/install-jenkins-solution-template.

Is this the only way or can I install it on a Windows VM and run it as if it was installed on a local Windows machine ? Thanks

# Answer
If I understand your question correctly then yes, you can create a windows VM in azure and install jenkins on it the same way as on a local machine.

They're might be some differences in the configuration of the environment for example about security, user access, routing etc
> 0 votes

# Answer
Do you really care about the OS behind jenkins that much? Jenkins is fine running in a Docker container in your choice of orchestration environments (Kubernetes, Dokku, etc.).

Running jobs on the controller is not a great idea, so the OS for build jobs is going to depend on the agents. Running agents in containers gives you a lot of flexibility, including having different pools of agents for different OS versions.
> 0 votes

---
Tags: jenkins, azure
---


# Question
Title: Docker: Let containers communicate to each other but don't expose to internet
I was recently shocked to find out that docker writes its own iptables rules and ignores ufw bindings completely. I had blocked all ports using the `ufw deny incoming command` and then selectively allowed only the ports that I wanted to expose.

I have a very simple setup that utilizes a postgresql container and a simple nodejs application. My primary goal is to keep the postgresql container port off the internet but let the node backend accept requests through port 3333. How do I do that?

My current `docker-compose.yml` file looks like this:

```
version: "3"

services:
  redis:
    image: redis:latest
    ports:
      - 6379:6379
    command: ["redis-server", "/redis.conf"]
    network_mode: host
  database:
    image: "postgres"
    env_file:
      - ./database/database.env
    volumes:
      - database-data:/var/lib/postgresql/data
      - ./database/init-db.sql:/docker-entrypoint-initdb.d/init.sql
    ports:
      - 5432:5432
    restart: always
    network_mode: host

  backend:
    depends_on: [database]
    build: ./server
    volumes:
      - ./files:/var/lib/files
    ports:
      - 3333:3333
    restart: always
    network_mode: host

volumes:
  database-data:

```

# Answer
> 1 votes
When you use network\_mode host, the published ports are ignored, which means the ports that are open in the containers will be the same port in the host. Does not matter what you had declared in `ports`

The default behavior is every service in the compose can talk to each other.

> Links are not required to enable services to communicate - by default, any service can reach any other service at that service’s name

let's use this simple compose as an example. There is no network defined, docker will create one.

```
version: '3.7'
services:
  nginx1:
    image: nginx
    ports:
      - '8080:80'
  nginx2:
    image: nginx

```

Nginx1 will be reachable through port 8080 on localhost and any other machines in the network.

```
$ curl -I http://localhost:8080
HTTP/1.1 200 OK
Server: nginx/1.19.10

```

While nginx2 will even have a port on the host.

```
$ docker-compose ps
      Name                     Command               State                  Ports
-------------------------------------------------------------------------------------------------
nginx1_1   /docker-entrypoint.sh ngin ...   Up      0.0.0.0:8080->80/tcp,:::8080->80/tcp
nginx2_1   /docker-entrypoint.sh ngin ...   Up      80/tcp

```

And they still reach each other by their internal port using the service name

Accessing nginx1 from nginx2

```
$ docker exec -it nginx2_1 curl -I http://nginx1:80
HTTP/1.1 200 OK
Server: nginx/1.19.10

```

Accessing nginx2 from nginx1

```
$ docker exec -it nginx1_1 curl -I http://nginx2:80
HTTP/1.1 200 OK
Server: nginx/1.19.10

```

# Answer
> 0 votes
Instead of using `network_mode` to `host`, you can simply make use of `links` (new versions uses `networks`) which will make the communication to happen between two containers.

I have updated the `docker-compose.yaml` file and added `links` to connect `database` service (database container) to be accessed from your `backend` service.

In your backend database configuration, you need to use `database` as the db host. (Here `database` is the service name which is defined in the compose.yaml file)

```
version: "3"

services:
  redis:
    image: redis:latest
    ports:
      - 6379:6379
    command: ["redis-server", "/redis.conf"]
  database:
    image: "postgres"
    env_file:
      - ./database/database.env
    volumes:
      - database-data:/var/lib/postgresql/data
      - ./database/init-db.sql:/docker-entrypoint-initdb.d/init.sql
    restart: always
  backend:
    depends_on: [database]
    build: ./server
    volumes:
      - ./files:/var/lib/files
    ports:
      - 3333:3333
    restart: always
    links:
      - "database"

volumes:
  database-data:

```

---
Tags: docker, docker-compose, security, ports
---


# Question
Title: AWS ECS vs EKS for hybrid cloud strategy
What can EKS deployment do that ECS deployment can't do? is there any specific circumstances that you have to deploy your application on EKS but not ECS.

# Answer
> 1 votes
In terms of general positioning this may be a good starting point.

Given in the title you mention "hybrid" deployments you may be interested in investigating and investigating ECS Anywhere and EKS Anywhere. They are both coming in 2021.

---
Tags: kubernetes, amazon-ecs, eks
---


# Question
Title: Cannot connect to chocolatey.org or any other internet site from within container
xpost from https://forums.docker.com/t/cannot-connect-to-chocolatey-org/38745

### Expected behavior

I'm running Windows Server 2016 Datacenter (MSDN download) 64 bit - Version 1607 - Build 14393.1715

I'm using Docker For Windows and running Windows Containers.

I figure this is the easiest thing in the world, to do this:

```
reboot my host machine
docker run -it microsoft/windowsservercore
powershell
iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'));

```

The expectation is for this to install chocolatey in the container.

*I'm actually going to be doing this in a dockerfile, but I'm going for minimum working example to get help right now.*

Here's an actual docker file that fails (it's not so creative, sadly)

```
FROM microsoft/windowsservercore

SHELL ["Powershell"]

ENV ChocolateyUseWindowsCompression false;
RUN iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'));

```

### Actual behavior

First things first: I'm running this in a Virtualbox VM. Everything else that I want to work works, including but not limited to: internal network connectivity and tests, sql server running, WCF services running against my docker-compose environment, data coming back correctly. All of the things I expect the system to do are working. This is the last-step, fingers crossed :D. The fact that I'm running this in VirtualBox shouldn't matter, since the Hyper-V is working and all my machines are working as expected. I just figure it's worth mentioning.

```
docker run -it microsoft/windowsservercore
powershell
PS C:\> iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
Exception calling "DownloadString" with "1" argument(s): "The remote name could not be resolved: 'chocolatey.org'"
At line:1 char:1
+ iex ((new-object net.webclient).DownloadString('https://chocolatey.or ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : WebException

PS C:\> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : 339189e8dd56
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : attlocal.net

Ethernet adapter vEthernet (Container NIC 6412b757):

   Connection-specific DNS Suffix  . : attlocal.net
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #4
   Physical Address. . . . . . . . . : 00-15-5D-80-E9-2C
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::a51b:820e:e45d:fab6%29(Preferred)
   IPv4 Address. . . . . . . . . . . : 172.22.202.147(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . : 172.22.192.1
   DNS Servers . . . . . . . . . . . : 172.22.192.1
                                       10.0.2.2
   NetBIOS over Tcpip. . . . . . . . : Disabled

```

From my host OS (Server 2016) I have this information:

```
PS C:\docker> ipconfig /all

Windows IP Configuration

   Host Name . . . . . . . . . . . . : WIN-CDPNLUMSTOE
   Primary Dns Suffix  . . . . . . . :
   Node Type . . . . . . . . . . . . : Hybrid
   IP Routing Enabled. . . . . . . . : No
   WINS Proxy Enabled. . . . . . . . : No
   DNS Suffix Search List. . . . . . : attlocal.net

Ethernet adapter vEthernet (HNS Internal NIC) 2:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #2
   Physical Address. . . . . . . . . : 00-15-5D-80-E5-17
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::5939:52e7:aaf3:5b70%9(Preferred)
   IPv4 Address. . . . . . . . . . . : 172.22.192.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :
   DHCPv6 IAID . . . . . . . . . . . : 301995357
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-4E-0C-48-08-00-27-51-5F-6A
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter vEthernet (HNS Internal NIC) 3:

   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Hyper-V Virtual Ethernet Adapter #3
   Physical Address. . . . . . . . . : 00-15-5D-B6-45-8D
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::1de4:bff7:36fb:ee92%13(Preferred)
   IPv4 Address. . . . . . . . . . . : 172.23.128.1(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.240.0
   Default Gateway . . . . . . . . . :
   DHCPv6 IAID . . . . . . . . . . . : 570430813
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-4E-0C-48-08-00-27-51-5F-6A
   DNS Servers . . . . . . . . . . . : fec0:0:0:ffff::1%1
                                       fec0:0:0:ffff::2%1
                                       fec0:0:0:ffff::3%1
   NetBIOS over Tcpip. . . . . . . . : Enabled

Ethernet adapter Ethernet:

   Connection-specific DNS Suffix  . : attlocal.net
   Description . . . . . . . . . . . : Intel(R) PRO/1000 MT Desktop Adapter
   Physical Address. . . . . . . . . : 08-00-27-51-5F-6A
   DHCP Enabled. . . . . . . . . . . : Yes
   Autoconfiguration Enabled . . . . : Yes
   Link-local IPv6 Address . . . . . : fe80::10de:923b:b866:221d%4(Preferred)
   IPv4 Address. . . . . . . . . . . : 10.0.2.15(Preferred)
   Subnet Mask . . . . . . . . . . . : 255.255.255.0
   Lease Obtained. . . . . . . . . . : Friday, August 19, 1881 6:47:42 PM
   Lease Expires . . . . . . . . . . : Tuesday, September 26, 2017 11:15:59 PM
   Default Gateway . . . . . . . . . : 10.0.2.2
   DHCP Server . . . . . . . . . . . : 10.0.2.2
   DHCPv6 IAID . . . . . . . . . . . : 50855975
   DHCPv6 Client DUID. . . . . . . . : 00-01-00-01-21-4E-0C-48-08-00-27-51-5F-6A
   DNS Servers . . . . . . . . . . . : 10.0.2.2
   NetBIOS over Tcpip. . . . . . . . : Enabled

Tunnel adapter isatap.{C420BD56-5715-49BF-9382-5EA99AA48563}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #5
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.{70EA4F39-FE0F-4040-B718-ACAC00765808}:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . :
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #6
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes

Tunnel adapter isatap.attlocal.net:

   Media State . . . . . . . . . . . : Media disconnected
   Connection-specific DNS Suffix  . : attlocal.net
   Description . . . . . . . . . . . : Microsoft ISATAP Adapter #7
   Physical Address. . . . . . . . . : 00-00-00-00-00-00-00-E0
   DHCP Enabled. . . . . . . . . . . : No
   Autoconfiguration Enabled . . . . : Yes
PS C:\docker>

```

### Information

DockerDebugInfo.ps1 output: https://gist.github.com/jcolebrand/170673e45bf8a659552f22f2071c30d6

### System information

Running Windows Server Datacenter (MSDN download) 64 bit - Version 1607 - Build 14393.1715

```
PS C:\docker> docker version
Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:57:19 2017
 OS/Arch:      windows/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.24)
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:59:47 2017
 OS/Arch:      windows/amd64
 Experimental: true

```

*If, for any forsaken reason this is important, the host OS does need to be activated. I'm doing this in a one-shot dev environment so wasn't particularly worried about activating the VM just yet. I can't fathom how this is important. Especially since everything else "just works".*

# Answer
> 6 votes
The problem appears to be that the Windows images configure DNS to first be the default gateway of the NAT interface, and then the DNS of the host machine. DNS lookups fail against that first resolver and external access.

Adding `--dns 8.8.8.8` (or your local resolver) to the CLI seems to work around the problem.

# Answer
> 4 votes
At a command prompt run `ipconfig /all` and look for a `DNS Servers` line in the output with something that looks like an IP address. That's your host machine's DNS server(s).

Edit `C:\ProgramData\Docker\config\daemon.json` and add a `dns` entry with that IP address. For example, if your DNS Server is 192.10.0.2 and 8.8.8.8 is Google's DNS as a backup:

```
{
    "dns": ["192.10.0.2", "8.8.8.8"]
}

```

Now restart Docker Desktop and the remote URLs can be resolved by Docker, either at build or runtime.

This solution came from Faithful Anere at https://medium.com/@faithfulanere/solved-docker-build-could-not-resolve-archive-ubuntu-com-apt-get-fails-to-install-anything-9ea4dfdcdcf2

# Answer
> 2 votes
As comments are transient, here's some info for the next person trying to solve this problem. Please feel free to edit this answer to provide more details, unless you're a Docker for Windows pro, in which case please give a better answer :D

I went to demo the problem for my IT group after having changed my NIC on my VM to bridged and adding my thunderbolt adapter. As there are security systems in place, I realized I needed to add the security software to my VM, but as it isn't joined to the domain, that was a problem. Switching it back to NAT on the wifi adapter, and then going to demo the problem in my Windows 2016 VM somehow magically made it start working.

Same network. Same VM configuration. Same software and host, no changes to anything I can track.

Having checked Get-ContainerNetwork or Get-WinNat showing nothing, I really had been at my end of diagnostics. Changing that NIC adapter and then changing it back apparently caused something to reset. Good times.

So, I don't have a solid answer, but I have a good workaround for the next person stuck in my shoes, which, given my track record of StackExchange questions and answers, will be me. But I hope it helps you out too, random internet asker.

---

And an XKCD tax, for those impossible to solve problems.

# Answer
> 1 votes
I recently had this frustrating problem but I managed to fix it today.

What I did, is in the desktop app on Windows / Mac you can edit the Daemon file. Under Settings in the Docker App under Docker Engine, I added the line at the bottom of the file just above the last curly brace. `"dns": [ "192.168.4.100", "8.8.8.8" ]`

This then allows the Docker Containers all that you now build to use your host's DNS server. Technically if you can access: https://chocolatey.org/install.ps1 then you should be able to access the choco repository.

I have also built the image in https://github.com/jasric89/vsts-agent-docker/tree/master/windows/servercore/10.0.14393 and labeled it in the repo:

microsoft/windowsservercore:10.0.14393.1358

I then set: `RUN choco feature enable --name allowGlobalConfirmation` before my first Choco Install command, this enables choco to install all the files and not error.

With all that set my Docker File Ran and built the image. Well in my Test Env now testing in my prod env. :)

Links that helped me:

https://github.com/moby/moby/issues/24928 https://github.com/jasric89/vsts-agent-docker/blob/master/windows/servercore/10.0.14393/standard/VS2017/Dockerfile https://docs.chocolatey.org/en-us/troubleshooting https://github.com/moby/moby/issues/25537

# Answer
> -1 votes
Try installation instructions from https://chocolatey.org/install#completely-offline-install (section "Installing behind a proxy"):

```
    @powershell -NoProfile -ExecutionPolicy Bypass -Command "[System.Net.WebRequest]::DefaultWebProxy.Credentials = [System.Net.CredentialCache]::DefaultCredentials; iex ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))" && SET PATH=%PATH%;%ALLUSERSPROFILE%\chocolatey\bin

```

---
Tags: docker, windows
---


# Question
Title: How to cache node_modules folder when using Jenkins Multibranch Pipeline
I'm using Jenkins Multibranch Pipeline to handle my CI/CD and I'd like to know if is there a way to cache my `node_modules` folder?

Every time I push a branch, I execute `yarn` to install the dependencies. when is `master` branch, it took 1s to execute this step because it's already cached. Otherwise, when is a new branch, `yarn` took 2 minutes because is a new environment. Is there a way to cache `node_modules` globally to be used by all branches/environments?

# Answer
> 4 votes
It is possible to do this (not with any built-in steps; you essentially have to either use a global cache or write your own caching tool), but I do not recommend it. When I've tried this in the past, I often ran into race conditions. For example, one build would write to the cache while another build simultaneously reads from the cache, or two builds would write to the cache simultaneously, etc. I got a lot of builds with corrupted modules as a result.

Caches do not have these kinds of race conditions when limited to a single job (or in this case, a single branch for a multibranch job) because Jenkins can be configured not to run multiple builds of one job simultaneously. If you take extra steps to ensure that all of the jobs which might access the cache can't run simultaneously, then you could probably get away with a shared cache. But at that point, you're likely to lose the added speed benefit of sharing the cache anyway, since you've massively reduced Jenkins' capacity to run jobs in parallel.

The other scenario where you could get away with a shared cache is if you write to the cache extremely infrequently - and I really do mean *extremely*. If the cache is rarely written to, then there simply aren't many opportunities for these kinds of race conditions.

# Answer
> 2 votes
Adjust Jenkins job "Build Environment"  see https://stackoverflow.com/questions/39829229/how-to-use-exclude-folder-option-in-delete-workspace-before-build-starts-secti/41728663

also add build step with shell/windows script with

```
npm install -D

```

# Answer
> 1 votes
The best way I have found so far is to use Docker containers with a docker file that does the installation of node\_modules before running other steps.

I make sure all jobs of this multi branch pipeline run on the same node and keep the latest docker image for EACH branch on this node locally. That way unless the dependencies in packages.json change, then docker will use the cache to build the new image (or just generate the artifact you want to create and transfer it to the local workspace). If all branches have the same dependencies, they will all use the same cached image, if not they will just create a new one.

Example:

```
## CHOSE YOUR LOCAL BASE IMAGE
FROM debian:10.0

### ADD YOUR DEPENDENCIES
ADD packages.json build-deps.sh

### INSTALL DEPENDENCIES (we use a custom script)
RUN build-deps.sh

### ADD THE REST OF YOUR CODE
### This will break the cache assuming your code changes each time,
### but previous docker build commands will be cached
### (if your dependencies didn't change)
ADD my-code.tar.gz .

### RUN YOUR COMMANDS
RUN run-your-scripts.sh

```

After building you image with docker build, you can then do soemthing like this to extract the artifact

```
docker run --name SOME-NAME YOUR-REPO:YOUR-TAG
sudo docker cp SOME-NAME:/path/to/artifact/artifact.tar.gz ${WORKSPACE}

```

I hope this helps

---
Tags: jenkins, node.js, cache
---


# Question
Title: Docker compose to ECS, two services in one task
I'm using this method to deploy my docker-compose project to AWS ECS.

A simplified version of my docker-compose.yml looks like this:

```
version: '3'

services:
  web:
    image: .dkr.ecr.eu-west-3.amazonaws.com/django:${IMAGE_TAG}
    build:
      context: .
      dockerfile: ./Dockerfile

  nginx:
    image: .dkr.ecr.eu-west-3.amazonaws.com/nginx:${IMAGE_TAG}
    build:
      context: .
      dockerfile: ./nginx.Dockerfile
    ports:
      - 80:80
    depends_on:
      - web

```

So, run `docker compose up` with my AWS ECS context and it works beautifully, cloud-formation creates all my resources and so on.

But, I can't figure out how to get these two services (web and nginx) to run in one task. Since they are in separate tasks at the moment it will create two instances of FARGATE or EC2 which is a bit overkill. I'd like two run the two containers on one instance but be able to scale up to more duplicate instances.

As stated in the link attached, I can modify the cloud-formation template to my liking, but something are limited to what I can modify. I don't think I can move that container service to another task since the image tag etc are created dynamically by `docker compose`.

My ECS cluster looks like this atm.

# Answer
What you are looking for is a `sidecar` pattern where you run a container of your main application and a number of additional containers (one or more) in support of it. This is a well established pattern with orchestrators such as ECS but it's not possible to declare it with a docker compose syntax today. There is an open request talking about this. Can I ask you to +1 it and add your commentary/request there? It would help Docker to sense demand for this feature. Thanks!
> 1 votes

---
Tags: docker, amazon-ecs
---


# Question
Title: How to Integrate Amazon API Gateway with EKS using Application Load Balancer (ALB)?
According to the AWS documentation, it is possible to integrate Amazon API Gateway with EKS using ALB. However, by far most of this documentation explains how to do this with NLB instead of ALB. So hopefully, someone can explain how to do this with ALB.

For example, do I need to create ingress to enable me calling my services? Or is there another way? How to configure my ALB and my API gateway?

# Answer
> 1 votes
From the same page:

```
The steps to integrate ALB and NLB with API Gateway are identical.

```

Which is followed by a sentence with a link to a documentation for an ALB ingress, including prerequisites for VPC subnets and EKS resources.

Yes, you have to create one unless you'd want to configure ALB manually instead of using `Ingress` manifests inside k8s.

---
Tags: kubernetes, amazon-web-services, load-balance, eks
---


# Question
Title: Are package facts getting invalidated automatically when I install remove packages?
In Ansible I can use a task such as:

```
- name: Gather the package facts
  package_facts:
    manager: auto

```

to gather the package facts and a condition such as:

```
when: "'foobar' in ansible_facts.packages"

```

to only execute a task when package `foobar` is installed. Gathering the package facts is a rather lengthy process compared to many of the other tasks in my playbooks.

For the sake of this question we can also assume that instead of `manager: auto` I'm using `manager: apt`.

Now I was unable to find any information regarding if or when the fact `ansible_facts.packages` gets invalidated after a subsequent package manager operation and a look at the fact via `debug` module doesn't give me any indication as to when it was gathered. Similarly I found no obvious way of checking against the cache time of the package manager. So in essence I was unable to come up with a means of only gathering (or re-gathering) the package facts when necessary.

So, questions:

1. does the `ansible_facts.packages` fact get invalidated after subsequent package manager operations?
2. if not: how can I get close to the desired "only gather package facts if something changed"?

NB: according to `ansible --version` I am running version 2.10.7.

# Answer
> 2 votes
Package facts (as any other facts like devices, disks and their partitions, IP addresses...) don't refresh automagically.

In your case, you can use a handler and notify it from any task that can possibly invalidate the cache. Here is a quickNdirty example:

```
- name: Pseudo code demo for package facts refresh
  hosts: localhost

  tasks:

    - name: Gather the package facts unconditionally at play start
      package_facts:

    - name: Make sure package X is installed
      package:
        name: package_X
      notify: refresh_package_facts

    # By default, handlers run at the end of the play.
    # If you want to force them to run at a certain point,
    # you need the following meta task.
    # Note => this will run !!ALL!! notified handlers
    - name: Run handlers if notified
      meta: flush_handlers

    - name: This task will run with an up-to-date cache
      debug:
        msg: "{{ ansible_facts.packages | length }} packages are installed"

  handlers:

    - name: Refresh the package facts
      package_facts:
      listen: refresh_package_facts

```

---
Tags: ansible
---


# Question
Title: Is it possible to deploy RDS in multiple region in AWS with read and write replica
I have an application that I need to make it accessible across 4 regions, I saw blogs and other posts in the internet that speaks about the read replica of AWS RDS MySQL Server, is it possible to enable write operations in the regions so that the data can be in sync and be writable across all the regions.

# Answer
You can look at Aurora Global Database for this. Based on this news announcement, you can also configure the setup, so you can write to your local db cluster and this is forwarded to your primary region.

Of course there would be some lag in write requests, essentially of the length of time it takes to write to the primary region and replicate the data changes back to the replica in your local region. Nonetheless this is likely easier to handle (and with the same methods in a web application), as if you wrote to the primary region yourself and checked if the data had arrived back in local replica.
> 1 votes

---
Tags: amazon-web-services, amazon-ec2, amazon-rds
---


# Question
Title: How to diagnose intermittent "error dialing backend: dial timeout" error in GKE
We have an issue in our GKE cluster that started suddenly yesterday where intermittently we get errors interacting with the control plane. This affects our Jenkins builds which use the Kubernetes plugin, but also even simply running and attaching to pods. For example, if we do a `kubectl run -i --tty --rm --image=busybox some-name`, about 20% of the time we'll get the following error:

```
Error attaching, falling back to logs: error dialing backend: dial timeout

```

Hoping someone can help suggest how we can diagnose further.

Verbose logs from the `kubectl run`:

```
I0421 09:38:47.720435    2483 round_trippers.go:420] POST https://35.189.86.41/api/v1/namespaces/default/pods
I0421 09:38:47.720448    2483 round_trippers.go:427] Request Headers:
I0421 09:38:47.720452    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:38:47.720456    2483 round_trippers.go:431]     Content-Type: application/json
I0421 09:38:47.720460    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:38:47.852510    2483 round_trippers.go:446] Response Status: 201 Created in 132 milliseconds
I0421 09:38:47.860462    2483 reflector.go:175] Starting reflector *v1.Pod (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
I0421 09:38:47.860489    2483 reflector.go:211] Listing and watching *v1.Pod from k8s.io/client-go/tools/watch/informerwatcher.go:146
I0421 09:38:47.860610    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods?fieldSelector=metadata.name%3Dsome-name&limit=500&resourceVersion=0
I0421 09:38:47.860619    2483 round_trippers.go:427] Request Headers:
I0421 09:38:47.860628    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:38:47.860633    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:38:47.912301    2483 round_trippers.go:446] Response Status: 200 OK in 51 milliseconds
I0421 09:38:47.919196    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods?allowWatchBookmarks=true&fieldSelector=metadata.name%3Dsome-name&resourceVersion=298429547&timeout=6m8s&timeoutSeconds=368&watch=true
I0421 09:38:47.919212    2483 round_trippers.go:427] Request Headers:
I0421 09:38:47.919219    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:38:47.919224    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:38:47.964479    2483 shared_informer.go:253] caches populated
I0421 09:38:48.002537    2483 round_trippers.go:446] Response Status: 200 OK in 83 milliseconds
I0421 09:38:49.416988    2483 reflector.go:181] Stopping reflector *v1.Pod (0s) from k8s.io/client-go/tools/watch/informerwatcher.go:146
I0421 09:38:49.417115    2483 attach.go:338] defaulting container name to some-name
If you don't see a command prompt, try pressing enter.
I0421 09:38:49.417419    2483 round_trippers.go:420] POST https://35.189.86.41/api/v1/namespaces/default/pods/some-name/attach?container=some-name&stdin=true&stdout=true&tty=true
I0421 09:38:49.417431    2483 round_trippers.go:427] Request Headers:
I0421 09:38:49.417436    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v4.channel.k8s.io
I0421 09:38:49.417440    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v3.channel.k8s.io
I0421 09:38:49.417444    2483 round_trippers.go:431]     X-Stream-Protocol-Version: v2.channel.k8s.io
I0421 09:38:49.417447    2483 round_trippers.go:431]     X-Stream-Protocol-Version: channel.k8s.io
I0421 09:38:49.417451    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:39:19.524483    2483 round_trippers.go:446] Response Status: 500 Internal Server Error in 30106 milliseconds
Error attaching, falling back to logs: error dialing backend: dial timeout
I0421 09:39:19.525463    2483 attach.go:338] defaulting container name to some-name
I0421 09:39:19.525813    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods/some-name/log?container=some-name
I0421 09:39:19.525829    2483 round_trippers.go:427] Request Headers:
I0421 09:39:19.525837    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:39:19.525843    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:39:49.585194    2483 round_trippers.go:446] Response Status: 500 Internal Server Error in 30059 milliseconds
I0421 09:39:49.591557    2483 round_trippers.go:420] GET https://35.189.86.41/api/v1/namespaces/default/pods/some-name
I0421 09:39:49.591578    2483 round_trippers.go:427] Request Headers:
I0421 09:39:49.591585    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:39:49.591611    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:39:49.636553    2483 round_trippers.go:446] Response Status: 200 OK in 44 milliseconds
I0421 09:39:49.639346    2483 round_trippers.go:420] DELETE https://35.189.86.41/api/v1/namespaces/default/pods/some-name
I0421 09:39:49.639366    2483 round_trippers.go:427] Request Headers:
I0421 09:39:49.639373    2483 round_trippers.go:431]     Accept: application/json, */*
I0421 09:39:49.639379    2483 round_trippers.go:431]     Content-Type: application/json
I0421 09:39:49.639386    2483 round_trippers.go:431]     User-Agent: kubectl/v1.18.3 (darwin/amd64) kubernetes/2e7996e
I0421 09:39:49.692164    2483 round_trippers.go:446] Response Status: 200 OK in 52 milliseconds
pod "some-name" deleted
I0421 09:39:49.692460    2483 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "Get \"https://10.31.60.49:10250/containerLogs/default/some-name/some-name\": dial timeout",
  "code": 500
}]
F0421 09:39:49.692479    2483 helpers.go:115] Error from server: Get "https://10.31.60.49:10250/containerLogs/default/some-name/some-name": dial timeout

```

# Answer
Google Support tracked down the issue down for us:

> this issue appears to have been caused by a known Issue affecting Konnectivity, under some circumstances a memory leak could occur, in the case of your cluster, this appears to end up causing connection timeouts after some time, this issue has already been addressed on the Konnectivity network proxy \[1\], and the fix is already available within GKE on version 1.19.9 or higher.

\[1\] https://github.com/kubernetes-sigs/apiserver-network-proxy/pull/168
> 1 votes

---
Tags: kubernetes, google-cloud-platform
---


# Question
Title: What do you tell someone new to going oncall or PagerDutyl?
\[\[ Based on tags and the wording this sounds like a PagerDuty-specific question, but it could apply to any tech or stack in my mind.... PagerDuty is just so dominant in SF-based jobs I've seen that is like tech Kleenex.... \]\]

I've been searching in my spare time for a few months for an answer to: **what do you tell a new recruit to pager duty** (or PagerDuty or )??? And I don't mainly mere "install the app and prey" , but those are key parts of it. I'm more looking for generic, general, globalizable advice like:

* ask for help as early as possible
* don't go sleep deprived "too" many nights in a row `~3`
* know who is likely to answer with low latency and who will notice work again at Monday 9am `DOH!`
* it is easier to find answer on StackExchange than random forums at 3am `free plug`
* triage: how big of a fire is it? how many fire trucks do you call in? Etc.
* when to merge incidents
* when to snooze incidents... but that is so scary!! "*You don't get fired for snoozing here?*" `hahaha.`
* good notification settings
* good phone/volume settings for getting woken up, even if you sleep like a log.
* scheduling and calendar layers
* rules of thumb and recommended tools
* know who to call for what subsystem (varies based on locality and org) but using a wiki or Confluence is often part of that

Maybe I'm searching wrong. I'm open to meta-suggestions for better ways to find this needle in the haystack. Searching for "starting pagerduty" makes sense to me, but it gets results of how to setup these things programmatically or technically, not how to instruct the humans in dealing with sorts of things. PagerDuty does try to address this, but it doesn't include the breadth of responsibilities that the job entails to me.

I also admit that I'm asking for everything and the kitchen sink and the city water supply for a bonus round. Getting little pieces of this as links and pointing folks at it would be a huge improvement over what I've found so far. I doubt anybody has done all of this in one doc or would ever want to do so.

I've actually already turned this question into an internal class. I'm happy to get permission from management to share my fun version of this after I pull out the proprietary bits. I'm still holding out for it having been done at some point before and I'm just too lazy to go that far into the search results. :)

# Answer
> 2 votes
There are a number of things that I think it's important to know before going on-call:

1. **How to triage an issue.** There should be a defined list of priorities with examples so that you can identify how serious a problem is.
2. **What the priority levels mean.** Can you ack this and then go back to sleep?
3. **Who to contact post-triage.** The on-caller's primary responsibility is *triage*, not fixing. Fixing the problem is good, too, but once you've triaged the issue, then it's up to you to rope in other folks from other teams and manage communications between them.

People jumping straight into fixing problems before triaging is the most common mistake I've seen (because, after all, we're used to fixing things and think of that as our job). But communications are often vital, and doing things in the wrong order can mean customers are left in the dark because you got caught up in trying to fix a problem before letting your support people know there *is* a problem.

# Answer
> 1 votes
Here's my two pieces of advice:

1. **Work out schedules so that you're not on-call 24 hours a day**. There should be a hand-off between daytime operations and after-hours support. I encourage people to split half their regular time between support weeks and project weeks; after-hours support should come during a project week.
2. **Get in the habit of taking notes during an incident.** That's the most important thing; even more important than "fixing" the issue. If bandaging the service requires you to stop taking notes, then it's time to call in additional support. **There should always be someone taking notes.**

---
Tags: monitoring, pagerduty, training
---


# Question
Title: See Azure DevOps WorkItems/Tasks in Planner?
We are using Microsoft Teams and Planner since a while. Now we are migrating from Bitbucket/Bamboo to Azure DevOps.

If I create a Team in Microsoft Teams with a Planner-Board, I can see all my tasks in Planner. But when I have tasks in Azure DevOps, I can't see this tasks in Planner. Teams and Azure DevOps is connected to Azure AD. Is there a way to show DevOps-Tasks in Planner?

# Answer
> 1 votes
I don't think that this feature is currently available out-of-the-box. However, you can build a Poower Automation Flow or Logic App that integrates the two of them.

In the long run, I would suggest that you remove Planner from the equation, and just use the kanban board in Azure DevOps for task management. You can display the kanban board inside Teams as well.

---
Tags: azure-devops, azure
---


# Question
Title: How to get code from Bitbucket via Octopus?
Is there a way that I can tell Octopus to take its code from Bitbucket and not GitHub (without using Bitbucket's pipelines)? I couldn't anything under community steps that would let me do that (somewhat surprisingly).

# Answer
It appears that Octopus supports a push from BitBucket, but not a pull (https://octopus.com/blog/bitbucket-pipes-and-octopus-deploy)

Github supports either option: https://octopus.com/docs/packaging-applications/package-repositories/github-feeds
> 1 votes

---
Tags: automation, bitbucket, octopus-deploy
---


# Question
Title: How to make Octopus Deploy trigger a new release when a new release made on GitHub?
I have a GitHub project that I'm using to trigger new releases on Octopus. I can see triggers for machine changes, or timed things, but not specifically when a new release is created on GitHub. This seems odd, certainly Azure offers it.

Does GitHub need to be configured to provide a feed, or is it possible for Octopus to check for new releases on GitHub and then use that to trigger a new release?

# Answer
> 1 votes
Octopus can be configured as either the target of a push from GitHub or you can set up a feed in Github, and pull from that feed.

In your case, it sounds like you want a push mechanism, such as GitHub Actions to do the heavy lifting.

---
Tags: automation, octopus-deploy
---


# Question
Title: When do I run applications on a server in a pod and when to run them directly on the base operating system?
I am trying to get started in the DevOps Field by setting up a small server currently with Virtualbox, later on for a small team. For my basic setup I plan to use gitea as self-hosted git and Jenkins as CICD tool which also provides a lot of extensions.

The server OS I chose is RHEL 8 since the systems on which I will setup up my DevOps system in a distant future will use this OS.

Now I find a lot of sources in the internet that point me to install both tools using containers combined with an orchestration tool like kubernetes. I have a certain degree of theoretical understanding of docker and kubernetes and also experimented a little bit with both tools.

My research also provided me information on the advantages of running Jenkins in a pod:

* only required in certain situations
* sometimes several instances of Jenkins for different projects can be required to run in parallel

At least were that the points that I found on different blogs websites that discussed this topic for Jenkins.

For Gitea I was not able to determine reasons why it should be run in container inside a pod in kubernetes but there is a helm-chart managed by the community of it. Also I found several sources on the internet telling me helm-charts are bad, don't use them.

To build, test and deploy applications (which will be Rest applications with react/node-js frontend and different backends) I will use container and kubernetes to orchestrate the container.

Could you help me understand why I should decide for or against containers/pods for the different applications and maybe even provide reliable sources, since most information I found are just opinions spread through the internet.

# Answer
> 1 votes
We're using a self-hosted gitlab, nexus and Jenkins (controller) and both are hard installs on CentOS7. Jenkins agent is a remote CentOS7. That approach made the most sense to us as opposed to breaking out the whole automated pipeline shebang to deploy gitlab/jenkins/nexus. Gitlab/Jenkins/Nexus are part of the foundation of our CICD infrastructure so it made no sense to make them ephemeral by using container, scaling or orchestration approaches. The actual software components we build & deploy are ephemeral (ie: a new build / config & deploy many times a day) and that infrastructure building, deploy & config is pretty much automated from scratch.

---
Tags: jenkins, kubernetes, containers, git
---


# Question
Title: Validating certificates with get_url or yum on CentOS 7 using Ansible
I'm pretty new to Ansible, but it seemed a better tool than a bash script for installing programs from source on remote servers...

Starting from my working-but-fragile bash script, I made an Ansible script to git clone, configure and compile what I need. However some of these sources are large, so I wanted to include the `--depth 1` parameter. This is a problem because the CentOS 7 version of git is 1.8.3 and Ansible ignores the depth command if it can't use at least git 1.9.1. Obviously for non-system use, I wouldn't mind having the latest git 2.15 anyway so...

To make a newer version of git available than is on yum's base/extras/updates repos, I found the IUS repository which has a "safe replacement" package for a few things including git. Helpfully there's an Ansible task set to install their repo. Here's the relevant task:

```
- name: install IUS release package
   yum:
       name: "https://{{ ius_distribution_abbrev }}{{ ansible_distribution_major_version }}.iuscommunity.org/ius-release.rpm"
       state: present
   when: ansible_os_family == 'RedHat'

```

Ansible however fails this task:

```
fatal: [MY_REMOTE_IP_ADDRESS]: FAILED! => {"changed": false, "failed": true,
"msg": "Failed to validate the SSL certificate for centos7.iuscommunity.org:443.
Make sure your managed systems have a valid CA certificate installed.
You can use validate_certs=False if you do not need to confirm the servers
identity but this is unsafe and not recommended. Paths checked for this
platform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem,
/etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible.
The exception msg was: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:579)."}

```

Both yum's ca-certificates and python-urllib3 packages are installed and on the latest version from the yum repos.

Per this comment on github, it seems that Ansible can't handle SNI to get the right SSL certificate without python 2.7.9+. But CentOS 7 ships with python 2.7.5. You can't upgrade it because get\_url has the same issue with SSL and so you would have to download the python source with `validate_certs=no`.

Worse, even when I did that, used `make altinstall` to put python 2.7.14 in `/usr/local/bin/` without touching the OS python and then set `ansible_python_interpreter: /usr/local/bin/python2.7`, (at least allowing get\_url to work) I'm then locked out of yum for the duration of the playbook because yum can't handle the newer version of python Ansible is trying to run it with.

* Is there a way to make this all work together properly without removing a security feature?
* Is Ansible the wrong tool for CentOS 7 because of the python version and yum?

# Answer
> 5 votes
While it's true that the version of python reported there doesn't have SNI support, I thought that RedHat backported it. But anyways, if not, you can do that yourself:

```
yum -y install gcc python-devel libffi-devel openssl-devel
pip install pyopenssl ndg-httpsclient pyasn1

```

(This works for `requests` at least, I'm not sure about Ansible.)

On a broader note:

> But CentOS 7 ships with python 2.7.5. You can't upgrade it because get\_url has the same issue with SSL and so you would have to download the python source with validate\_certs=no.

That's only the case if the server you're downloading Python from relies on SNI. I imagine most of their mirrors do not.

But really, you shouldn't be compiling Python on every individual server. You should do it once, either on a machine that you freeze into an image that you launch further machines off of, or you package it as an rpm and the machines just download that and install it. This allows you to control the download process.

Even simpler: download the package on your desktop (via verified https). Copy it to your file storage. Download from there onto your servers. There's no need to download directly from python.org every time, and it's more reliable to have a local copy anyways.

# Answer
> 1 votes
I suggest that you do not reinvent the wheel and use a ansible galaxy role that has already solved the issues you are reporting.

One could use Geerlingguys git role to install the latest git on CentOS7. It is possible to install this by issuing `ansible-galaxy install geerlingguy.git` and subsequently include `geerlingguy.git` in the roles in order to apply the role on the nodes.

# Answer
> 0 votes
Failure to validate SSL certificate seems to be happening widely across Linux distributions. Therefore I will post a solution (general idea) which worked for me. The surface problem seems to be that Python is not able to recognize the certificate bundle file. Meanwhile, commandline `curl` can recognize SSL certificate and download file. The path to the certificate bundle differs depending on your Linux distribution.

In my case,

Ansible version: 2.10, Debian distribution

after certificate was added by

```
- name: Add the intermediate certificate to certificate bundle
  command: update-ca-certificates

```

Then, specify the SSL\_CERT\_FILE variable to the certificate bundle.

```
- name: download a file
  get_url:
    url: "url"
    dest: "destination"
  environment:                                           # Explicitly specify SSL_CERT_FILE here.
    SSL_CERT_FILE: /etc/ssl/certs/ca-certificates.crt

```

---
Tags: ansible, git, ssl, centos
---


# Question
Title: Gitlab: Always having the possibility to launch a manual job (even if something failed before)
I'm using gitlab-ci. I have a pipeline that has the following steps :

1. code-verification
2. some tests
3. deploy (manual)
4. some more tests
5. rollback (manual)

Both the deploy and the rollback job have to be launch manually which means that I'm using `when: manual` in my gitlab-ci.yml file.

However, I also want the rollback to be possible *even* if a step/job has fail before (for example, a faulty deploy or a failing in the "some more tests" stage).

I was thinking of using `when: always` but since I'm already using the `when: manual`, it doesn't work (I tried).

For now, I just put the "some more tests" and the "rollback" in the same stage so that the failure of one doesn't impact the other. But is there another way to do this?

I really wish to have everything in the same pipeline (so no trigger) and to always have the possibility to launch the rollback job. No matter what happened before.

# Answer
You can use `when: on_failure`, in your case you can create a new job for rollback when your `some more tests` have failed.

### Example:

```
cleanup_build_job:
  stage: cleanup_build
  script:
    - cleanup build when failed
  when: on_failure

```

For more example: https://docs.gitlab.com/ee/ci/yaml/#when
> 2 votes

---
Tags: continuous-integration, gitlab, continuous-deployment, yaml
---


# Question
Title: Continuous deployment strategies vs DevSecOps
How do you implement security requirements in CI/CD while taking care of security requirements? What are best practices here?

Security requirements may differ, let's assume this simple level of security:

* Direct administrative access to target system shouldn't be given to the CI/CD system
* CI/CD system needs to be able to initiate deployment and show it's success/fail status (could be determined by a white box test routine i.e. without admin access rights placed on the CI/CD side)

# Answer
> 1 votes
As you stated, requirements will differ based on company and industry policy; for us, the adjustment from "separation of duties" to automated deployments has been rough. As interim steps, we've started:

1. Requiring multiple "sign-offs" before deployment; this is accomplished by both reviews for the PR in Azure DevOps and an audit log in Octopus.
2. Monthly review of deployments by higher-ups. This is mostly a rubber-stamp activity, but it satisfies auditors.

---
Tags: continuous-deployment, security, devsecops
---


# Question
Title: How to communicate in a multi-organization spanning project
In DevOps S-Pillar of "CALMS" says that it is also about sharing knowledge.

At a present project I am working on, I am sitting at the interface between two organizations together with another "DevOps"-guy. A lot of our time is wasted because we have partly redundant meetings (with the organization on the left side) and we wait partially longer times until somebody from the right side can assist us in solving some issue.

Sometimes I also spend hours to find out who is reponsible for getting some information for me that I need.

What would you try to address to make more sharing possible although it is different organizations that are working on that project?

# Answer
1. Establish a wiki (such as Confluence, DokuWiki, etc).
2. Establish a chat/communications channel (such as through Slack or Mattermost).
3. Share some code repositories (such as through GitLab, Bitbucket, GitHub).
4. Share some binary repositories (such as through Artifactory, Nexus, Pulp).
5. Occasionally send out a "newsletter" email that showcases what your team are trying to accomplish.
> 8 votes

# Answer
As Peter noted in his comment to the previous answer, getting stakeholders involved is key. It sounds like your parent organization is viewing DevOps as an interface between the teams, rather than an umbrella of reform.

IMO, the ultimate goal of DevOps is delivering value to clients faster, and communication is a key part of that. Here's some thoughts:

1. In Team of Teams McChrystal talked about the value of getting key players from multiple organizations to be within physical proximity of each other. If you're still going into the office, suggest that members of the organizations be moved to be near each other. IF that's not possible, implement a "buddy system"; have people from one org pair up with another remotely, and commit to checking in with each other daily (even if it's just 5 minutes).
2. Are the two organizations using the same work tracking system (or at least one that integrates with the other one)? Flow can often be improved if everyone's using the same tool set. It becomes easier to hand off items, as well as building that centralized repo.
> 0 votes

---
Tags: culture, process
---


# Question
Title: Log monitoring/metrics dashboard with images/video
I have a system that generates both textual log/metrics data, as well as images. I'm looking for some tool that would allow me to track the log data (a la Kibana or Grafana), but also let me embed the images, ideally in some way that was easily correlated with the logs based on time. That is, I'd have the normal charts derived from the log data, and if I click a point on a plot, it would show me the image from that time.

Does any system already exist that would let me do this? Free obviously preferred since it makes it easier, but this is ultimately a corporate project, so a paid tool also works.

# Answer
In terms of embedding in an external page, you have some options out of the box with both Kibana (docs here) and Grafana (docs here).

If I understand correctly, you are trying to generate realistic clickable thumbnails - in that case, you should try Grafana first since it better supports server-side image rendering
> 1 votes

---
Tags: grafana, kibana
---


# Question
Title: Pagerduty API integration and auto-resolution
It is possible to implement auto-resolution with Pagerduty API integration. When a monitoring system detects an issue based on some metric and a threshold, it would create a Pagerduty incident through the API integration. Later, if the metric falls below the threshold, it could automatically resolve the incident, in case it is still open.

This feature could be very useful for several reasons:

### Cases where a large number of incidents get triggered due to a common root cause

In this case, the on call person is spared from having to ack/resolve so many incidents and instead focus on troubleshooting and fixing the problem.

### Auto-resolution would capture the incident duration more accurately

Humans are not good in doing things on a timely basis. If we are computing MTTR based on manual resolution of incidents, it may lead to inaccurate measurements.

Has anyone implemented auto-resolution with Pagerduty? If so, would you be able to talk about the pros and cons?

# Answer
> 2 votes
PD has a lot of native integrations that can do auto-resolve but for those that don't you can use the API and/or rulesets to do that.

Here is a link to the ruleset documentation: https://support.pagerduty.com/docs/rulesets

And here is documentation on the incident resolution API: https://developer.pagerduty.com/api-reference/reference/REST/openapiv3.json/paths/~1incidents~1%7Bid%7D/put

In the API call you would pass the following json body to resolve it: { "incident": { "type": "incident\_reference", "status": "resolved" } }

---
Tags: monitoring, pagerduty
---


# Question
Title: Terraform: how do you inspect sensitive data?
In a resource of type `aws_ecs_task_definition` my plan has an line that looks like this:

```
~ container_definitions    = (sensitive) # forces replacement

```

Now in principle this could be doing anything - deleting all my containers, spinning up bitcoin mining containers, etc.

Is there any way I can look at the data, both the old and the new?

# Answer
When your configuration or a provider marks an attribute as sensitive, Terraform will always hide that value in any output that's intended for human consumption.

The real values are available in machine-readable output though. This is primarily with the aim of integrating with external software, but if you need to then you can also inspect the machine-readable output directly yourself.

You can get a machine-readable (JSON) rendering of a plan like this:

```
terraform plan -out=tfplan
terraform show -json tfplan

```

If you intend to read it directly in the terminal then it can help to pipe it into `jq`, if you have that utility installed:

```
terraform show -json tfplan | jq

```
> 4 votes

---
Tags: terraform, security
---


# Question
Title: How does one build a build pipeline with versioned build artifacts?
I'm not sure quite how to phrase this question. I'd like to learn either by answer here or sharing a reference to read from: ***How do I build a pipeline with versioned build artifact?***

I've inherited an established Jenkins build infrastructure that I recognize needs to change. This is especially evident because we want to move into a modern build approach with branches (dev, main, etc.).

This is basically how our builds work

```
lib-A    is created first
lib-B    is created next
lib-C    is the last library created
prod-DA  uses lib-A
prod-EC  uses lib-C
prod-FAB uses both lib-A and lib-B

```

Our current build infrastructure is what I'd describe as a *dev* infrastructure. In that it always uses latest.

When lib's A, B and C are created they are dropped into a shared folder to be used by everyone else.

When prod's DA, EC and FAB need to link with the library they just pull from the shared folder.

This is what I inherited. It's *adequate* for a single branch, *riding-latest* build system.

Is there a resource, pattern, book, website or toolset to help me learn how to change this to a versioned artifact scenario? Or how to convert to a mechanism that will scale and support branching?

I don't really even know what to call this, I'm really wanting to learn what you have to share.

Thank you for your patience.

# Answer
I know this question was asked a long time ago but I still want to answer it.

How products are build, versioned and delivered depend highly on the combination of the used tools, the infrastructure and the processes that have been established in your company. This also means there is no general way to it. However there are of course best practices like continuous integration/continuous delivery, nightly builds, common branching patterns etc.

In your case I would start with common branching patterns for the used version control system (svn, git, etc.) and think about how to support versioned artifacts . Next step is to get the build management tool (Maven, Gradle, Make, Cmake, etc.) build the specified version.

If these two parts are working you can think about how to automate it with jenkins.

One last thing: Your question was very general. I guess if you ask more specific questions like "How do I combine maven and git for building versioned artifacts" you get more answers.
> 1 votes

---
Tags: jenkins, build-pipeline, builds, artifacts, versioning
---


# Question
Title: How can I list all environment variables configured in Terraform?
I configured AWS provider in Terraform by providing an access key and secret as environment variables according to the documentation:

`$ export AWS_ACCESS_KEY_ID="anaccesskey"`
`$ export AWS_SECRET_ACCESS_KEY="asecretkey"`

I now want to view those variables on the console as well as other environment variables configured on my system but I can't find the right command.

**Update:** I found out that I didn't configure the AWS provider like I thought. The AWS CLI was already configured with my credentials stored in AWS credentials file: `~/.aws/credentials`.

# Answer
On your command line, run `printenv`

Reference: https://man7.org/linux/man-pages/man1/printenv.1.html
> 1 votes

# Answer
Your current shell environment variables, which are also including terraform-related ones can be listed by `env` command or you can do it in tf file:

```
terraform {
  required_version = ">= 0.12.0"
}

resource "null_resource" "get_tf_env" {
  provisioner "local-exec" {
    command = "env"
    interpreter = ["bash", "-c"]
  }
}

```

According to official docs you can only feed terraform with the envs starting with `TF_*`.
> 1 votes

# Answer
`printenv | grep TF_VAR` will list all variables that you defined.
> 1 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: when replicaset is 1 then what are 1/2 pods for prometheus server?
Using helm chart :

```
helm install monitor prometheus-community/prometheus -n monitoring -f ./values.yaml

```

I cannot get prometheus server to running.

```
# kubectl get pods -owide -n monitoring
NAME                                              READY   STATUS             RESTARTS   AGE   IP               NODE              NOMINATED NODE   READINESS GATES
monitor-kube-state-metrics-578cdbb5b7-ffnbc       0/1     Running            0          19s   192.168.23.10    kube-worker-vm3   <none>           <none>
monitor-prometheus-alertmanager-c94f7b6b7-znfp5   1/2     Running            0          19s   192.168.222.53   kube-worker-vm2   <none>           <none>
monitor-prometheus-node-exporter-5vksg            1/1     Running            0          20s   192.168.1.36     kube-worker-vm2   <none>           <none>
monitor-prometheus-pushgateway-8694884546-qgqd2   0/1     Running            0          19s   192.168.222.54   kube-worker-vm2   <none>           <none>
monitor-prometheus-server-5587c7f464-z9tcj        1/2     CrashLoopBackOff   1          19s   192.168.222.55   kube-worker-vm2   <none>           <none>

```

Even though I had 1 replicaSet out of 2 running or crashloopbakcoff.

```
# kubectl describe pod monitor-prometheus-server-5587c7f464-z9tcj -n monitoring
Name:         monitor-prometheus-server-5587c7f464-z9tcj
Namespace:    monitoring
Priority:     0
Node:         kube-worker-vm2/192.168.1.36
Start Time:   Sun, 09 May 2021 22:14:12 +0100
Labels:       app=prometheus
              chart=prometheus-13.8.0
              component=server
              heritage=Helm
              pod-template-hash=5587c7f464
              release=monitor
Annotations:  cni.projectcalico.org/podIP: 192.168.222.55/32
              cni.projectcalico.org/podIPs: 192.168.222.55/32
Status:       Running
IP:           192.168.222.55
IPs:
  IP:           192.168.222.55
Controlled By:  ReplicaSet/monitor-prometheus-server-5587c7f464
Containers:
  prometheus-server-configmap-reload:
    Container ID:  docker://d4fdd420ec67f42857a4f20dd56de146307c65c49cd8ffccf6153f0b54082e91
    Image:         jimmidyson/configmap-reload:v0.4.0
    Image ID:      docker-pullable://jimmidyson/configmap-reload@sha256:17d34fd73f9e8a78ba7da269d96822ce8972391c2838e08d92a990136adb8e4a
    Port:          <none>
    Host Port:     <none>
    Args:
      --volume-dir=/etc/config
      --webhook-url=http://127.0.0.1:9090/-/reload
    State:          Running
      Started:      Sun, 09 May 2021 22:14:19 +0100
    Ready:          True
    Restart Count:  0
    Environment:    <none>
    Mounts:
      /etc/config from config-volume (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from monitor-prometheus-server-token-njbp9 (ro)
  prometheus-server:
    Container ID:  docker://b53d3a56e2985594046a3b0018e9828597678268119c2dd7b07b1c7b315246e3
    Image:         prom/prometheus:v2.20.1
    Image ID:      docker-pullable://prom/prometheus@sha256:788260ebd13613456c168d2eed8290f119f2b6301af2507ff65908d979c66c17
    Port:          9090/TCP
    Host Port:     0/TCP
    Args:
      --storage.tsdb.retention.time=15d
      --config.file=/etc/config/prometheus.yml
      --storage.tsdb.path=/data
      --web.console.libraries=/etc/prometheus/console_libraries
      --web.console.templates=/etc/prometheus/consoles
      --web.enable-lifecycle
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    2
      Started:      Sun, 09 May 2021 22:20:24 +0100
      Finished:     Sun, 09 May 2021 22:20:24 +0100
    Ready:          False
    Restart Count:  6
    Liveness:       http-get http://:9090/-/healthy delay=30s timeout=30s period=15s #success=1 #failure=3
    Readiness:      http-get http://:9090/-/ready delay=30s timeout=30s period=5s #success=1 #failure=3
    Environment:    <none>
    Mounts:
      /data from storage-volume (rw)
      /etc/config from config-volume (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from monitor-prometheus-server-token-njbp9 (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config-volume:
    Type:      ConfigMap (a volume populated by a ConfigMap)
    Name:      monitor-prometheus-server
    Optional:  false
  storage-volume:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  prometheus-pv-claim
    ReadOnly:   false
  monitor-prometheus-server-token-njbp9:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  monitor-prometheus-server-token-njbp9
    Optional:    false
QoS Class:       BestEffort
Node-Selectors:  boardType=x86vm
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason       Age                     From               Message
  ----     ------       ----                    ----               -------
  Normal   Scheduled    9m19s                   default-scheduler  Successfully assigned monitoring/monitor-prometheus-server-5587c7f464-z9tcj to kube-worker-vm2
  Warning  FailedMount  9m18s                   kubelet            MountVolume.SetUp failed for volume "config-volume" : failed to sync configmap cache: timed out waiting for the condition
  Normal   Pulled       9m13s                   kubelet            Container image "jimmidyson/configmap-reload:v0.4.0" already present on machine
  Normal   Created      9m13s                   kubelet            Created container prometheus-server-configmap-reload
  Normal   Started      9m12s                   kubelet            Started container prometheus-server-configmap-reload
  Normal   Pulled       8m24s (x4 over 9m12s)   kubelet            Container image "prom/prometheus:v2.20.1" already present on machine
  Normal   Created      8m23s (x4 over 9m12s)   kubelet            Created container prometheus-server
  Normal   Started      8m22s (x4 over 9m12s)   kubelet            Started container prometheus-server
  Warning  BackOff      4m16s (x32 over 9m10s)  kubelet            Back-off restarting failed container

```

# Answer
> `monitor-prometheus-server-5587c7f464-z9tcj 1/2 CrashLoopBackOff`

`1/2` here means that 1 of 2 **containers** in the Pod is "Ready"

When you describe the Pod, you see the status of the containers:

```
Containers:
  prometheus-server-configmap-reload:
    Ready:          True
...
  prometheus-server:
    Ready:          False

```

You probably need to check the logs for the crashing container to see the reason.
> 1 votes

---
Tags: kubernetes, kubernetes-helm
---


# Question
Title: Reaching a Docker environment running in VMware from LAN
Here is my setup:

I have a Ubuntu 20.04.2 LTS machine running in VMware (vSphere client).

On this machine, I have a local Drupal website running in Docker using Wodby Docker4Drupal https://github.com/wodby/docker4drupal

The website can be accessed on the Ubuntu machine on http://drupal.localhost:8000.

The Ubuntu machine has an IP of **192.168.10.100** which I can ping from my local Windows machine which is on the same LAN (Windows computer IP address is 192.168.10.97).

Is there any way that I could access the Drupal website directly from the browser on a computer that is on the same LAN (ie. my Windows machine)?

I have tried to access http://192.168.10.100:8000 but I get a `404 page not found` message. On port `80` I get `ERR_CONNECTION_REFUSED`. Any idea?

I shall also mention that on the Ubuntu VM, I also get *the same* `404 page not found` if I try to reach http://0.0.0.0:8000 or http://127.0.0.1:8000 or http://localhost:8000. Only the hostname configured within the project resolves to the Drupal website (http://drupal.localhost:8000).

---

Here is the `docker-compose.yml` I am using.

```
version: "3.7"

services:
  mariadb:
    image: wodby/mariadb:$MARIADB_TAG
    container_name: "${PROJECT_NAME}_mariadb"
    stop_grace_period: 30s
    environment:
      MYSQL_ROOT_PASSWORD: $DB_ROOT_PASSWORD
      MYSQL_DATABASE: $DB_NAME
      MYSQL_USER: $DB_USER
      MYSQL_PASSWORD: $DB_PASSWORD
    volumes:
      - d4d-db-data:/var/lib/mysql # Use bind mount

  php:
    image: wodby/drupal-php:$PHP_TAG
    container_name: "${PROJECT_NAME}_php"
    environment:
      PHP_SENDMAIL_PATH: /usr/sbin/sendmail -t -i -S mailhog:1025
    volumes:
    - ./web:/var/www/html:cached

  crond:
    image: wodby/drupal-php:$PHP_TAG
    container_name: "${PROJECT_NAME}_crond"
    environment:
      CRONTAB: "0 * * * * drush -r /var/www/html/web cron"
    command: sudo -E LD_PRELOAD=/usr/lib/preloadable_libiconv.so crond -f -d 0
    volumes:
    - ./web:/var/www/html:cached

  nginx:
    image: wodby/nginx:$NGINX_TAG
    container_name: "${PROJECT_NAME}_nginx"
    depends_on:
    - php
    environment:
      NGINX_STATIC_OPEN_FILE_CACHE: "off"
      NGINX_ERROR_LOG_LEVEL: debug
      NGINX_BACKEND_HOST: php
      NGINX_SERVER_ROOT: /var/www/html/web
      NGINX_VHOST_PRESET: $NGINX_VHOST_PRESET
    #      NGINX_DRUPAL_FILE_PROXY_URL: http://example.com
    volumes:
    - ./web:/var/www/html:cached
    labels:
    - "traefik.http.routers.${PROJECT_NAME}_nginx.rule=Host(`${PROJECT_BASE_URL}`)"

  mailhog:
    image: mailhog/mailhog
    container_name: "${PROJECT_NAME}_mailhog"
    labels:
    - "traefik.http.services.${PROJECT_NAME}_mailhog.loadbalancer.server.port=8025"
    - "traefik.http.routers.${PROJECT_NAME}_mailhog.rule=Host(`mailhog.${PROJECT_BASE_URL}`)"

  traefik:
    image: traefik:v2.0
    container_name: "${PROJECT_NAME}_traefik"
    command: --api.insecure=true --providers.docker
    ports:
    - '8000:80'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

volumes:
  d4d-db-data:

```

# Answer
> 0 votes
It turned out it was only a setting in Traefik. I had to add a second router http rule for the nginx container in my `docker-compose.yml`:

```
- "traefik.http.routers.${PROJECT_NAME}_nginx_lan.rule=Host(`192.168.10.100`)"

```

So the full nginx section now looks like that:

```
  nginx:
    image: wodby/nginx:$NGINX_TAG
    container_name: "${PROJECT_NAME}_nginx"
    depends_on:
    - php
    environment:
      NGINX_STATIC_OPEN_FILE_CACHE: "off"
      NGINX_ERROR_LOG_LEVEL: debug
      NGINX_BACKEND_HOST: php
      NGINX_SERVER_ROOT: /var/www/html/web
      NGINX_VHOST_PRESET: $NGINX_VHOST_PRESET
    #      NGINX_DRUPAL_FILE_PROXY_URL: http://example.com
    volumes:
    - ./web:/var/www/html:cached
    labels:
    - "traefik.http.routers.${PROJECT_NAME}_nginx.rule=Host(`${PROJECT_BASE_URL}`)"
    - "traefik.http.routers.${PROJECT_NAME}_nginx_lan.rule=Host(`192.168.10.100`)"

```

It works fine. I can now access my Drupal website at http://192.168.10.100:8000 from any other device on the same LAN.

In my case, `PROJECT_BASE_URL` and `PROJECT_NAME` are configured in an `.env` file and correspond to `drupal.localhost` and `drupal` respectively.

# Answer
> 1 votes
If you are using the docker-compose file from that github repo it's using traefik and should correctly expose port 8000, that should default to 0.0.0.0:8000. Maybe Ubuntu firewall is blocking incoming TCP port 8000. That could explain why it works on drupal.localhost:8000 which most likely resolves to 127.0.0.1 but not from 192.\* LAN.

Check Ubuntu firewall:

`$ sudo ufw status`

Make sure incoming TCP 8000 is allowed from anywhere, or at least for your LAN ip range. If not, you can open it with:

```
$ sudo ufw allow from any to any port 8000 proto tcp

```

---
Tags: docker, networking, ubuntu, vmware
---


# Question
Title: How to use kubernetes_service data provider in Terraform?
I have simple workflow. Create a cluster, install ingress with helm and once it's ready, get load balancer's IP address to update CF dns records. It's all in one script.

After creating nginx-ingress controller, I was trying to get IP address of load balancer.

```
resource "helm_release" "nginx_ingress" {
  name       = "ingress-nginx"
  ...
}

data "kubernetes_service" "this" {
  metadata {
    name = "ingress-nginx-controller"
    namespace = "ingress-nginx"
  }
  depends_on = [ helm_release.nginx_ingress ]
}

```

This never worked for me and the reason was that it was trying to connect to localhost. Then I found this limitation.

> WARNING When using interpolation to pass credentials to the Kubernetes provider from other resources, these resources SHOULD NOT be created in the same Terraform module where Kubernetes provider resources are also used. This will lead to intermittent and unpredictable errors which are hard to debug and diagnose. The root issue lies with the order in which Terraform itself evaluates the provider blocks vs. actual resources. Please refer to this section of Terraform docs for further explanation.

and moved `kubernetes` provider to another module. However, I have now a new problem.

```
module "cluster" {
  source = "./modules/cluster"

  cluster_node_count = var.cluster_node_count
  age_path = var.age_path
}

module "network" {
  source = "./modules/network"

  host = module.cluster.host
  token = module.cluster.token
  cluster_ca_certificate = module.cluster.cert
  cloudflare_email = var.cloudflare_email
  cloudflare_api_key = var.cloudflare_api_key
  main_domain = var.main_domain
  cluster_subdomains = var.cluster_subdomains

  depends_on = [module.cluster]
}

```

This also doesn't work and giving

> Providers cannot be configured within modules using count, for\_each or depends\_on.

Without `depends_on` the `network` module is called at the same time as cluster, which of course leads to fail, since for the network module cluster should be ready.

So it's kind of dead-loop which I am not sure how to break.

# Answer
> 1 votes
The provider documentation you quoted here seems to have a mistake. It talks about using the two providers together in the same *module* but it seems like it really means in the same *configuration*.

The problem arises because Terraform needs to configure a provider in order to plan with it, and so everything inside a `provider "kubernetes"` block must be known at the time of planning. This means you can't use anything in there which isn't created yet.

That documentation is trying to tell you that you'll need to split the configuration that declares your Kubernetes cluster from all of your configurations that configure or make use of the Kubernetes cluster, and then you can `terraform apply` the creation of the cluster *first*, and then separately `terraform apply` your other configurations that interact with it using the kubernetes provider.

---
Tags: terraform, terraform-module
---