title: Is Helm not for me?

text:
# Question
Title: Is Helm not for me?
I've been studying Kubernetes and Helm lately (I'm new to both) and figuring out how I can fit them into my companies CI/CD process. On top of this, I want to adapt to the GitOps process, where you check out and release an environment. I firmly believe in fitting to tools to the process, instead of the other way around.

Because of this I'm beginning to think that Helm may not be for us? Let me explain.

In GitOps, you check out a whole environment, correct? You say, I want to checkout `rc-2.1.0` (release candidate) for example, and it will define ALL the microservices needed for that version. So if we had 20 microservices, only 3/20 may have changed between `rc-2.0.0` and `rc-2.1.0` but you still deploy all 20 of them. With Kubernetes, 17/20 will be a no-op since nothing's changed, and only the changed 3/20 will actually be updated. This way we can later go back to `rc-2.0.0` if need be for any reason, and get the 20 microservices defined in that version.

But I want to decouple the activity of deploying new versions, which takes time, to the actual action of "switching" to the new versions. The deployment is done with Deployments, while the action can be done by updating a Service's label correct? So I want to do the following

1. Build and push Docker images (we currently do this, and won't change)
2. Deploy new containers (pods) via Deployment using `kubectl apply -f deployment.yaml`
3. Wait for the deployments to finish
4. Update the `Service` to point to the new Deployment, which is instantaneous

Since a Helm chart deploys a whole package, it contains both Service and Deployment templates, I can't really "decouple" steps 2-3 with step 4 above?

So is Helm indeed not for me? If I'm completely off-base in my understanding of Helm, how can I accomplish this process using the tool?

**NOTE:** I know I can accomplish rollouts and rollbacks via Deployments, but again, if I understand correctly, this takes time.

# Answer
Yeah, this is what I had in mind all along. Blue/Green Deployments on Kubernetes. So in short, Helm as it stands, is **not** for me since it doesn't have the concept of a blue/green deployment. With Helm, when you install a new package, the Deployment gets updated with the new Pod versions, but the Service is still pointing to the Deployment. Helm doesn't have the concept of having 2 Deployments, a blue and a green, and doesn't have the concept of switching between the two. Kubernetes itself supports this though.

Now maybe Helm can work if you create your Helm charts **without** the Service, and run a script to update the Service to switch to the new Deployment later in your installation process.
> 1 votes

---
Tags: helm, kubernetes-helm
---


# Question
Title: Is there a better model for my CI/CD cycle?
I am having some troubles finding a better model for my CI/CD cycle.

This is mainly gitflow issue.

My current model : We have 2 bitbucket branches : master and develop.

Both are linked via webhooks to jenkins to be deployed to prod. and dev. servers.

QA team for testing features and releases on dev. server.

Team of developers for both frontend and for backend. With lots of troubles.

The problem : Team manager wants to be able to have better control over accepted/rejected features. Say : developer1 committed a feature to develop branch, developer2 added a hotfix to develop, we want QA team to easily test them both separately, and then testing both together with the least hassle possible. And afterwards easily select what commit stays and moved to master branch.

As I said, this is mainly gitflow issue.

Any suggestions?

# Answer
Key issue with this approach is that applying commits selectively is extremely problematic. Basically, how do you know whether commits are dependent or independent?

As you correctly mentioned, this is one of the issues with Gitflow itself, so the solution is shifting methodologies. Preferred way today is Trunk-Based Development (TBD) - https://trunkbaseddevelopment.com/ \- refer to "Accelerate" book for evidence.

Under TBD you can choose a style of work where you have short-lived feature branches. Once a feature is complete, pull request is created and QA can test this feature selectively under such short-lived branch and then approve the merge to main.

This way you know that your branches are independent (unlike commits in Gitflow) and approve features selectively. Note, that you can achieve something similar in Gitflow with multiple different branches, but this gets extremely messy there.
> 1 votes

---
Tags: git, deployment, continuous-deployment, continuous-delivery, bitbucket-pipelines
---


# Question
Title: Docker Build Ending Up `returned a non-zero code: 1`
```
Running mktexlsr /var/lib/texmf ... done.
Building format(s) --all.
        This may take some time... done.
build.sh: line 81: -: No such file or directory
The command '/bin/sh -c bash build.sh' returned a non-zero code: 1

```

You can view the code here.

https://github.com/austinsonger/GreenOpenVAS

# Answer
> 1 votes
Line 81 of build.sh is trying to run python3. But the problem is more fundamental because you aren't getting any of the earlier `echo`s or `apt-get` command's output. I would try:

```
chmod +x build.sh
./build.sh

```

to let the sh-bang line do its job.

---
Tags: docker, containers
---


# Question
Title: Increase concurrent users by limiting container resources with docker-swarm
Consider the scenario where I have:

* 4 replicas of an app
* 4 cores
* On load, the app consumes 100% for a single core

Can I tell each replica to only use 50% of the core and if so, would this allow at most 8 simultaneous connections to the app? I have just dipped my toe into this stuff so an feedback would be appreciated.

# Answer
> 1 votes
In general: No you can't. At least not on the Docker level.

You can maybe code your app in a way that it monitors such metrics and adjusts itself but Docker isn't able to.

What Docker CAN do is limit the app to 50% and kill it if it uses more than that by utilizing resource limits:

docs.docker.com - containers/resource\_constraints

docs.docker.com - docker-compose resource limits

---
Tags: containers, docker-swarm
---


# Question
Title: Two identical Azure deployments; one always throws 404 - how to find out what the difference is?
I am deploying a C# ASP.NET Core web service to Azure using Pulumi. I can deploy it in 3 ways:

1. Run it locally from Visual Studio, i.e., not using Azure at all.
2. Deploy it to Azure from my local developer computer.
3. Deploy it to Azure from Jenkins (whicn runs on a different computer).

I have this problem:

1. When I run it locally, I can call the service fine, e.g. from Postman or from a C# application. The web service returns what I expect.
2. When I deploy it to Azure from my local machine, I can also call it fine. The web service returns what I expect.
3. When I deploy it to Azure from Jenkins and then try to call the webservice, it returns "NotFound" to all calls no matter what I do. (This presumably means HTTP 404.)

The deployments in 2 and 3 should be exactly the same. My question is: **How can I find out what the difference is between these two deployments in Azure?**

The Jenkins-deployed webservice exhibits the following curious behaviour:

* It does not log any exceptions (even when I wait several minutes for them to show up).
* If I go to my resource group -\> Application Insights -\> Logs and search for "requests", it does list requests. Curiously, it says that it returned HTTP 200 to all the requests, even though what I get when calling them is 404.
* The above is true even for web service calls that should never return 200 (they should return 201).
* The above is true even for web service calls to methods that shouldn't even exist (i.e., when I deliberately corrupt the method URI before calling the service).

During deployment I authenticate with Azure using a service principal. My Jenkinsfile looks like this:

```
withVaultSecrets([
    "path/to/secret/in/vault": [
        "sp_name", "application_id", "object_id", "sp_secret"
    ]
]){
    script {
        env.PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
        env.ARM_CLIENT_ID = "${application_id}"
        env.ARM_CLIENT_SECRET = "${sp_secret}"
        env.ARM_TENANT_ID = "${azure_dev_tenant_id}"
        env.ARM_SUBSCRIPTION_ID = "${azure_dev_subscription_id}"
        env.AZURE_CLIENT_ID = "${application_id}"
        env.AZURE_CLIENT_SECRET = "${sp_secret}"
        env.AZURE_TENANT_ID = "${azure_dev_tenant_id}"
    }//script
    dir("./src/deploy/KmsStack"){
        powershell "pulumi login --local";
        powershell "pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase"
        powershell "pulumi up --yes"
    }//dir
}//withVaultSecrets

```

The script which I use to deploy locally looks like this, with the same service principal credentials:

```
cd $PSScriptRoot
cd webapi
dotnet publish /p:DisableGitVersionTask=true
cd ../deploy/KmsStack
$env:PULUMI_CONFIG_PASSPHRASE = 'jenkinspassphrase'
$env:ARM_CLIENT_ID = ...
$env:ARM_CLIENT_SECRET = ...
$env:ARM_TENANT_ID = ...
$env:ARM_SUBSCRIPTION_ID = ...
$env:AZURE_CLIENT_ID = ...
$env:AZURE_CLIENT_SECRET = ...
$env:AZURE_TENANT_ID = ...
pulumi logout
pulumi login --local
pulumi stack rm jenkinsfunctionaltest -y
pulumi stack init jenkinsfunctionaltest --secrets-provider=passphrase
pulumi stack select jenkinsfunctionaltest
pulumi up --yes

```

How can I find out why these two deployed services behave differently? The Azure portal GUI is rich and has lots of sections. Can you recommend me where to look? Might there be some security settings that differ? How can I find them?

Thanks in advance!

# Answer
We found out what was wrong. It was not an Azure issue. The problem was that we were deploying a bad ZIP file. The ZIP file was missing *web.config*, which meant that the web application could not start up.

We were zipping our published web application by having this in the CSPROJ file:

```
<Target Name="ZipOutputPath" AfterTargets="Publish">
  <ZipDirectory SourceDirectory="$(OutputPath)\publish" DestinationFile="$(MSBuildProjectDirectory)\kmswebapp.zip" Overwrite="true" />
</Target>

```

This turned out not to work because the compiler does things in a different order than we expected. At the time when it generated the ZIP file, *web.config* was not generated yet, so *web.config* never got packed into the ZIP file. Hence Azure could not start the application.

When we deployed from our local machines, it worked because we didn't clean the publish directory before each run, so there would be a *web.config* left over from the previous run, and this old (but unchanged) *web.config* would get packed into the ZIP file and deployed to Azure, so Azure would know how to start the application.

We solved it by removing the above from our CSPROJ file and doing (roughly) this in our Jenkinsfile:

```
powershell "dotnet publish ./src/webapi/WebAPI.csproj"
powershell "if (!(Test-Path('${publishDirectoryPath}/web.config'))){throw 'We need web.config to exist in the publish directory'}"
powershell "Compress-Archive -Path '${publishDirectoryPath}/*' -DestinationPath './src/webapi/kmswebapp.zip' -Force"

```

This generates a proper ZIP file including *web.config*, and Azure can now start our application so it can respond properly to requests.
> 1 votes

---
Tags: jenkins, azure
---


title: How does Docker Swarm balance tasks?

text:
# Question
Title: How does Docker Swarm balance tasks?
From docs.docker.com - force-the-swarm-to-rebalance I can see that Docker Swarm "tasks are given to less busy nodes" to reach eventual balance.

I'm trying to find more info on what "busy" means exactly. For us, running a lot of Java microservices, our primary resource is memory. We are constantly running into OOM killers on the hosts as Swarm keeps scheduling containers to nodes that are already very low on RAM.

I have a 7 nodes swarm in front of me right now with one node's `top` stating:

```
top - 22:55:04 up 234 days,  8:48,  5 users,  load average: 0.53, 0.68, 1.09
...
KiB Mem : 18330008 total,   425268 free, 13718032 used,  4186708 buff/cache
KiB Swap:   511996 total,   389836 free,   122160 used.  4225292 avail Mem

```

This is the lowest in free/available memory in the entire swarm. Yet this node ALWAYS gets the task I'm scheduling. Other nodes are much much more relaxed:

```
top - 22:56:28 up  5:49,  2 users,  load average: 0.38, 0.61, 0.79
...
KiB Mem : 18329940 total,  4899940 free,  7877052 used,  5552948 buff/cache
KiB Swap:   511996 total,   511996 free,        0 used. 10098352 avail Mem

```

Yet they will never get the task. None of the other nodes will. Only the above mentioned node will, which has by far the lowest available memory.

So what criteria does Swarm use to determine the "business" of a node to determine if a task should be scheduled on it? Available memory clearly does not seem to be a metric in the decision.

Sidenote: I am aware that I can assign resource limits to my containers to prevent the node from being over-scheduled. This is what I am doing right now and I assume it will fix my problem. But I want to know about Swarm's inner workings to determine where to schedule and where not to.

# Answer
> 2 votes
AFAIK, Swarm does zero real-time resource monitoring. It's only as smart as reservations and limits that you set on the Service beforehand.

First, it takes into account if the tasks of that Service are spread out. It will try not to schedule multiple tasks on the same node, and may ignore total task count per node to do so.

Then, it counts the number of tasks per node and finds the one with the least task count.

All these is restricted by resource reservations. If that node doesn't have the room (based only on the reservations you set), it'll look to the next lower task count.

This is also why Swarm alone can't do built-in auto-scaling because it has no way to track metrics over time.

It does know total CPU and memory, so it knows how to track "full" once you've set resource reservations.

More info: https://github.com/docker/swarmkit/blob/master/design/scheduler.md

---
Tags: docker-swarm
---


# Question
Title: Update volume size dynamically from inside a POD
I was wondering how I could dynamically **create and then mount new volumes from inside a pod ?** **At runtime !**

I have to download a lot of files with very different sizes and would like the storage to adapt to my needs...

---

**Use case example :**
*There is a decompression step. But I don't know how much will be decompressed before its completed.*
If I could allocate some more memory at runtime It would not end with a DiskFull error...

I can mount an initial volume at pod startup, then either I find a way to update this volume size or at least mount a new one in the same pod !

Any ideas are welcome
Thx

# Answer
In my opinion, you could create bucket (aws s3 for example) and store your data there (create simple app to make upload), thus, you don’t need to care with disk full error
> 1 votes

---
Tags: docker, kubernetes, volume, kubernetes-volume, persistent-volumes
---


# Question
Title: Does RAM Come Into The Container vs VM Decision?
I am currently running a Java trading system on a VM that needs ~32 GB RAM and runs (almost) 24/7. There are a couple of engines linked to it that require 4GB + 8BG respectively.

I am considering moving from VMs to containers.

Most of the articles I read talk about running web apps and the benefit of a container starting up fast but they don't talk about heavyweight applications.

For this question I would like to know whether the RAM requirements of the processes running within them are a consideration when comparing VMs to Containers, or is there no difference between the two?

# Answer
> 1 votes
> the benefit of a container starting up fast but they don't talk about heavyweight applications.

yes and no. The container only contains the app and runtime (e.g. JVM), but the app and runtime starts up with the same speed as when running on a VM. But a VM usually takes longer time to create and start since you also have to boot the operating system. So it depends on how you compare.

Running an app in a container does not do your app startup faster.

> For this question I would like to know whether the RAM requirements of the processes running within them are a consideration when comparing VMs to Containers, or is there no difference between the two?

The app will use the same amount of resources when run as a container as on VM. But with containers it is easier to manage multiple instances - so you could consider that - this is usually done to achieve better availability and zero downtime deployments - however your app need to be stateless if you consider scaling out to multiple instances.

---
Tags: docker, containers, virtualization
---


# Question
Title: TravisCI: Pandas version advanced (Jan 2021) numpy is now incompatible
I am very much a TravisCI beginner. That said, I maintain python package mplfinance and we run a TravisCI test suite for every Pull Request. We run the same tests for py36, py37 and py38. Beginning in January 2021, the Py37 tests began failing consistently with the error:

```
ERROR: pandas 1.2.0 has requirement numpy>=1.16.5, but you'll have numpy 1.16.4 which is incompatible.

```

Here's the thing: Some of the tests run using pytest under tox, and those tests are fine. But we also run a test from a shell script. In that test, for each PR, the script checks out and installs the package (using **`pip3 install .`**) first from the PR itself, and then from the version currently in the repository, and does some comparison of the two versions. This works fine for py36 and py38, and before January worked fine also for Py37. Here is an example of it working fine for Py37 *prior* to January:

https://travis-ci.org/github/matplotlib/mplfinance/jobs/751128518

In the above link you can see the following

```
Collecting pandas
  Using cached pandas-1.1.5-cp37-cp37m-manylinux1_x86_64.whl (9.5 MB)
Collecting python-dateutil>=2.1
  Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB)
Collecting cycler>=0.10
  Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB)
Requirement already satisfied: numpy>=1.15 in /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages (from matplotlib->mplfinance==0.12.7a4) (1.16.4)

```

Notice that in installs pandas 1.1.5 and numpy 1.16.4

Beginning in January this is what I see:

Example: https://travis-ci.org/github/matplotlib/mplfinance/jobs/753110408

```
Collecting pandas
  Using cached pandas-1.2.0-cp37-cp37m-manylinux1_x86_64.whl (9.9 MB)
Collecting kiwisolver>=1.0.1
  Using cached kiwisolver-1.3.1-cp37-cp37m-manylinux1_x86_64.whl (1.1 MB)
Collecting pillow>=6.2.0
  Using cached Pillow-8.1.0-cp37-cp37m-manylinux1_x86_64.whl (2.2 MB)
Requirement already satisfied: numpy>=1.15 in /home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages (from matplotlib->mplfinance==0.12.7a5) (1.16.4)

```

Now it is installing pandas **1.2.0**, and initially it appears satisfied with numpy 1.16.4, but then later in the file I see

```
ERROR: pandas 1.2.0 has requirement numpy>=1.16.5, but you'll have numpy 1.16.4 which is incompatible.

```

(Oddly, further up in the file, it appears that tox is installing numpy==1.19.4 and pandas==1.1.5 ??)

### Does anyone have any ideas what I can do about this?

Much appreciated. Thank you.

# Answer
This turned out to be a problem with **`pip`** determining the correct dependency: for some reason it thought that the pandas requirement was for numpy\>=1.15

The error message

```
ERROR: pandas 1.2.0 has requirement numpy>=1.16.5 ...

```

was coming from pandas itself (upon `import`).

The fix is to upgrade to a **newer version of `pip`** (inside the script running on travisCI). The newer version of pip is able to corrently detect that pandas needs a newer version of numpy.

Just put **`pip install --upgrade pip`** near the top of the script.

Not sure why TravisCI doesn't just use the latest `pip` by default.

Reference: https://travis-ci.community/t/11214
> 1 votes

---
Tags: travis-ci
---


# Question
Title: How do I set a base domain for gitlab's auto devops?
Docs seem to point to https://docs.gitlab.com/ee/user/project/clusters/index.html#base-domain

However it doesn't seem to specify how to set the base domain strangely enough.

Can't seem to find info no this via a web search, I've also looked at https://docs.gitlab.com/omnibus/settings/configuration.html with no luck.

# Answer
> 1 votes
https://docs.gitlab.com/ee/topics/autodevops/quick\_start\_guide.html#install-ingress

> Go back to the cluster page on GitLab, and go to the Details tab. Add your Base domain. For this guide, use the domain .nip.io. Click Save changes.

I think this option only showed up AFTER I installed the ingress.

---
Tags: gitlab
---


# Question
Title: Where/what is the `auto-devops` script in gitlab's "auto devops" staging stage?
https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/ci/templates/Jobs/Deploy.gitlab-ci.yml

```
staging:
  extends: .auto-deploy
  stage: staging
  script:
    - auto-deploy check_kube_domain
    - auto-deploy download_chart
    - auto-deploy ensure_namespace
    - auto-deploy initialize_tiller
    - auto-deploy create_secret
    - auto-deploy deploy
  environment:
    name: staging
    url: http://$CI_PROJECT_PATH_SLUG-staging.$KUBE_INGRESS_BASE_DOMAIN
  rules:
    - if: '$CI_KUBERNETES_ACTIVE == null || $CI_KUBERNETES_ACTIVE == ""'
      when: never
    - if: '$CI_COMMIT_BRANCH != $CI_DEFAULT_BRANCH'
      when: never
    - if: '$STAGING_ENABLED'

```

What does `auto-deploy` refer to? Is it a script? Where is it located? Where can I find out more info about it (because the staging stage is currently failing, and I'd like to debug what it's doing)?

The failing job logs:

```
To get the password for "user" run:
    export POSTGRES_PASSWORD=$(kubectl get secret --namespace initial-24739737-staging staging-postgresql -o jsonpath="{.data.postgresql-password}" | base64 --decode)
To connect to your database run the following command:
    kubectl run staging-postgresql-client --rm --tty -i --restart='Never' --namespace initial-24739737-staging --image docker.io/bitnami/postgresql:9.6.16 --env="PGPASSWORD=$POSTGRES_PASSWORD" --command -- psql --host staging-postgresql -U user -d staging -p 5432
To connect to your database from outside the cluster execute the following commands:
    kubectl port-forward --namespace initial-24739737-staging svc/staging-postgresql 5432:5432 &
    PGPASSWORD="$POSTGRES_PASSWORD" psql --host 127.0.0.1 -U user -d staging -p 5432
WARNING: Rolling tag detected (bitnami/postgresql:9.6.16), please note that it is strongly recommended to avoid using rolling tags in a production environment.
+info https://docs.bitnami.com/containers/how-to/understand-rolling-tags-containers/
Validating chart version...
Fetching the previously deployed chart version...
Fetching the deploying chart version... v1.0.7
secret "staging-secret" deleted
secret/staging-secret replaced
No helm values file found at '.gitlab/auto-deploy-values.yaml'
Deploying new stable release...
Release "staging" does not exist. Installing it now.
INSTALL FAILED
PURGING CHART
Error: release staging failed: timed out waiting for the condition
Successfully purged a chart!
Error: release staging failed: timed out waiting for the condition
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1

```

# Answer
Probably this https://gitlab.com/gitlab-org/cluster-integration/auto-deploy-image/-/blob/master/src/bin/auto-deploy

.....................................................................
> 1 votes

---
Tags: gitlab
---


# Question
Title: How to automate the key exchange in WireGuard when you deploy a cluster of machines?
Let's assume you want to deploy a cluster of machines on Hetzer Cloud. For simplicity let's call them `worker1`, `worker2`, `worker3`. They need to communicate with a server called `master`, which will be running on different account then the workers. Ideally, the whole setup should not be open to the internet. Unfortunately, Hetzner supports only private networks within the same account.

To make it work, you can setup your own VPN using WireGuard. Conceptually, it is not hard. You need to setup three connections (between the `master` and each `worker`). The tricky part is how to automate the key exchange. Ideally, it should not be more work if you deploy additional workers (e.g. 100 instead 3 workers).

Setting up such a VPN cluster sounds like a common problem, but I cannot find any recommendations on how to setup `1-to-n` or `n-to-m` connections, only tutorials on how to peer two machines. I'm thinking of automating the key exchange with Ansible (generate keys, gather them, install them on the `master`), but wanted to check first whether there is an easier solution to the problem that I missed.

In SSH, workers could share their key, which would simplify the problem. In WireGuard, keys cannot be shared, as far as I understood. How would you automate the setup of a VPN with WireGuard, so each worker can reach the master? Or is WireGuard the wrong choice for the problem?

Clarification:

* In my scenario, it is not possible to move the workers and master to the same account; otherwise, Hetzner networks would be the straightforward solution for setting up a private network.
* If you are not familiar with Hetzner Cloud, it is not a problem. You can assume that you get normal Linux machines, but then you are on your own (it does not support VPC peering across accounts as AWS does). Yet you can use all Linux tools available for creating the VPN setup. WireGuard would be my first choice, but I'm open to other techniques.

# Answer
The githubixx wireguard ansible role might be what you are looking for. The author of the role says:

> I use WireGuard to setup a fully meshed VPN (every host can directly connect to every other host) and run my Kubernetes (K8s) cluster at Hetzner Cloud (but you should be able to use any hoster you want). So the important components like the K8s controller and worker nodes (which includes the pods) only communicate via encrypted WireGuard VPN. Also (as already mentioned) I've two clients. Both have kubectl installed and are able to talk to the internal Kubernetes API server by using WireGuard VPN.

I reviewed the tasks and the template, but I haven't tried it out yet myself. It supports different versions of linux. It installs wireguard. It handles key generation, distribution, and wireguard configuration for a full mesh VPN. For each host it configures an interface with the host's private key as well as peer configuration for all the other hosts using their own public keys. You can set variables in the ansible inventory for each host (e.g. addresses) to tailor the configurations to your network.
> 1 votes

# Answer
(Disclaimer: As the question has not received an answer yet, I came to the conclusion that there is currently no out-of-the-box solution. Thus, I'm sharing what I ended up doing, and hope it will help others. Still, I would be interested to learn of a better solution if there is one.)

---

WireGuard can be used, but each of the n-to-m connections will need its own keys. Managing that by hand becomes tedious if the number of machines increases. Therefore, it is advisable to automated it:

1. Generate all keys in advance
2. Distribute the keys to the machine using Ansible (e.g. by generating systemd services `wg0.network` and `wg0.netdev`)

Note that step 1 assumes that the maximum number of machines is known in advance. If you cannot provide an upper bound, this approach will not work. Instead you will need to implement a dynamic key generation and exchange protocol.

If you end up using fewer machines then the configure upper bound, it will still work. Distributing the keys means that WireGuard will allow connections from different machines, but does not require that all peers are always available.

---

Scaling: In my setup, the upper bound was in the order of 100. That works without problems, but I did not test how it scales if you need to go higher (e.g. to the order of 1000 or 10,000).

What if there is no upper bound? -- That is a more difficult problem, and I don't know how to solve it best. As said, you would need to generate key dynamically and exchange them. I was considering opening SSH and using it to exchange on-the-fly generated WireGuard keys.

Note that with SSH, you can reuse the same key on different machines. That eliminates the need to exchange dynamically generated keys. But be aware that reusing the same key for all machines is a trade-off between convenience and security. If one machine gets compromised, the attacker will gain access to all other machines, too.

Maybe there is some solution that is more elegant and more secure. Perhaps directly using Wireguard is not a good idea if you need to support an arbitrary number of dynamic hosts. It is easy to introduce subtle security issues when implementing your own key exchange protocol.
> 0 votes

# Answer
I've got an Ansible role that may help you. It was designed to act as a gateway, but I think it may still work for you (hopefully without having to tweak it).

The most helpful aspect is that it can run in either client or server mode. So what you would do in your playbook:

1. Run it on `master` in server mode,
2. Save the generated public key to a variable.
3. Loop though the workers running it in client mode providing the server's public key.
4. Collect all of the public keys into a list variable.
5. Rerun it on `master` in server mode, this time with all of the worker keys.

Then, you should be good to go. I normally don't put all of this together, just do a single run for each device, but if you want to automate the entire thing (especially if you have many workers), then you should be able to get all of that into a single playbook.

For more information, see the role documentation.

I don't recommend sharing keys across devices for the reason you mentioned.
> 0 votes

---
Tags: ansible, deployment, automation, vpn
---


# Question
Title: Ansible fetch all vars files from a directory to generate a combined vars file
What I'm looking for is a task (or tasks) to read whatever .yml files exist in a specified folder and combine them into a file containing all the variables. For example:

pets/steves\_pets.yml:

```
cats:
  - spot
  - snowy
hamsters:
  - mephistopheles

```

And

pets/jacks-pets.yml:

```
dogs:
  - rex
  - woof
hamsters:
  - prometheus

```

Becomes

wherever/all-pets.yml:

```
cats:
  - spot
  - snowy
dogs:
  - rex
  - woof
hamsters:
  - mephistopheles
  - prometheus

```

In fact it's not even necessary to generate an all-pets.yml - as long as the full combined list is available to Ansible for use in roles / tasks. But a combined list file approach would be fine if easier.

Any pointers would be greatly appreciated!

# Answer
Sounds like you want the combine filter.

You could read each file into a specific variable, then use the combine filter to create `all-pets`. At that point, it would be in memory of course, so you might want to use the answer to this question to write it to a file:

```
- name: Read vars
  include_vars:
    file: "pets/{{ item }}_pets.yml"
    var: "{{ item }}"
  loop:
    - steves
    - jacks

- name: Combine vars
  set_fact:
    all_pets: {{ steves | combine(jacks) }}

- name: Write the file
  copy:
    content: "{{ all_pets | to_yaml }}"
    dest: pets/all_pets.yml

```
> 0 votes

# Answer
I think you can just include\_vars at directory level with `dir`

```
tasks:
  - name: "Read my pets from pets folder"
    include_vars:
      dir: "pets"
      name: all_pets

```
> 0 votes

---
Tags: ansible, jinja
---


# Question
Title: How to consolidate multiple servers for CPU and Memory with Kubernetes
**Setup**: I have 10 Photovoltaics based setup where IOT Gateway get data from sensors, devices and send to Kubernetes Cluster. K8S cluster exposed its ActiveMQ broker with NodePort.

**Observation**: AMQ has been scheduled on Node2 and i observed that CPU usage of this node is around 75%, As we are collecting lot of messages, But other nodes CPU usage is like 8% to 20%. If we put more load this CPU might reach to 100%.

Although we have CPU resource available but we cant use it, How can I do optimal usage of resources?

I read Apache Mesos and DC/OS but i think still its not possible to run a container workload which requests more cpu /memory than available on its single largest physical/virtul host.

Any thoughts in this regard?

# Answer
> 1 votes
Since Pod is running on single node it can use resources from this node only. And CPU/MEM can't be shared here. At least without special software for distributed computation.

In your case you should consider using ActiveMQ Clustering (https://activemq.apache.org/clustering). In that way you will be able to spread load across multiple k8s nodes with ActiveMQ pods running on them.

So use k8s Deployment + pod affinity to be sure pods will be scheduled to different nodes.

---
Tags: kubernetes
---


# Question
Title: Jenkins Docker - Make changes to gitconfig
I have a self-hosted Gitea organization, and the way to access the repos requires some "secret" `extraheader` in .gitconfig. (2FA is also enabled)

When I look at Jenkins "Configure System" settings, under the "Git plugin" section, it seems that the only .gitconfig options I can change through the UI are "user.name" and "user.email".

Is there any way I can change extra .gitconfig settings, such as adding `http.[url].extraheader`, or a way to run `git config --global` commands, before a `git clone` or `git pull` is performed?

# Answer
As far as I can tell, there is no way to tweak arbitrary git config settings from within Jenkins. You would have to tweak your git config on your build agents. If you're using permanent agents, this would involve logging in as the jenkins user and running `git config --global ...`. If you're using docker build agents, then you would need to bake the git config settings into your image.
> 1 votes

---
Tags: docker, jenkins, jenkins-pipeline, git, jenkins-plugins
---


# Question
Title: How to use Hashicorp Vault with NodeJS application?
I have Node JS app inside pods, which needs to read vault secrets. So far I found 2 methods for doing that

1. Using init container to mount secrets as .txt files and read/parse them in my app
2. Using node-vault connect to vault server directly and read secrets, which requires initial token

For (1) I found this article, where the author is considering it as not secure and complex.

> Secrets mounted as volumes are unwieldy—secrets can be stored as environment variables or mounted as a volume. The former technique is widely agreed to be less secure. If you opt for volumes, things quickly get complex when you have a large number of keys. Kubernetes creates one file per key, and you need to read all these files from within the application. There are workarounds, but they can be equally complex.

For the (2) I am not sure how to automatically unseal and obtain initial tokens without using Amazon secrets or other services. Of course I can't hard code initial token or `roll_id/secret_id`. What is the typical way of obtaining and passing initial token to the pods?

# Answer
As the Linux philosophy says:

> Make each program do one thing well

So it's worth it to use the vault agent, cause it's doing well what it would be done.

You can find an awesome example here
> 1 votes

---
Tags: kubernetes, hashicorp-vault
---


# Question
Title: Looping an Array with Missing Keys in Ansible Tasks
I'm attempting to create an Ansible Role to create users... I want it to be able to accept a list of dicts and if a key is missing from the dict, it will fall back to Ansible's default.

For example, given the following data:

```
---
users:
 - { name: foo, group: bar, create_home: no }
   { name: goo, group: bar }

```

I'd normally loop over the list with something like:

```
- name: Add users
  ansible.builtin.user:
    name: "{{ item.name }}"
    groups: "{{ item.group }}"
    create_home: "{{ item.create_home }}"
    state: present
  loop: "{{ users }}"

```

However, because `create_home` doesn't exist for the second user, it'd throw an error. How could I get it to fall back to Ansible's default for keys that don't exist?

# Answer
> 1 votes
Q: *"**How could I get it to fall back to Ansible's default for keys that don't exist?**"*

A: Omit such parameters, e.g.

```
    create_home: "{{ item.create_home|default(omit) }}"

```

---
Tags: ansible
---


# Question
Title: kubernetes quota vs limit. What are they? How are they different? When should I use one or the other?
I have been reading through the docs but it still is not clear as to why I should use one or the other. They have great examples of how to configure them but don't answer the why for quotes or limits very well.

I want to understand

```
kind: ResourceQuota

```

and

```
kind: LimitRange

```

I am not so interested in the yaml this is very clearly given in the examples. I want to understand the use-cases better.

# Answer
`ResourceQuota` is

> the total amount of memory and CPU that can be used by all Containers running in a namespace

`LimitRange`is

> if ... the Container does not specify its own CPU limit, then the Container is assigned the default CPU limit

* Example of ResourceQuota:

Imagine you have 2 namespaces in the same cluster, the namespace "backup" and namespace "test". "backup" run critical pods and you want to guarantee that will always have enough resources (CPU and memory) to run it. So you can configure a ResourceQuota in the namespace "test" to use only a part of the total amount of CPU and memory in your cluster.

Is like you say to k8s: "The sum of all memory and CPU of all pods in the namespace "test" can't be higher than what is specified in the ResourceQuota, 20G for memory and 12 vCPU's"

* Example of LimitRange:

You can specify in the yaml the minimum and maximum amount of CPU and memory a container can consume but instead of configuring in every yaml file you can set a default value with LimitRange.

Is like you say to k8s: "If there is no resource configured in the yaml file apply this LimitRange.
> 3 votes

---
Tags: kubernetes, containers, deployment
---


title: Monitor Jenkins node/slave health

text:
# Question
Title: Monitor Jenkins node/slave health
For a project I'm working on, we need to create a Jenkins node in an AWS environment. For this task, we will work with the docker image, which is available on https://github.com/jenkinsci/docker-ssh-slave . The Jenkins node will be responsible for executing AWS commands which can only be done from inside the AWS environment (not outside).

This Jenkins Master is running in our local environment. We will add the node as a permanent node in the Jenkins master. The Jenkins master is not responsible for setting up the Jenkins Node, since it has no permissions to do this ( this is managed in AWS ).

However, we would like to monitor the Jenkins Slave inside AWS to see if it it still running successfully. With other docker containers, we also expose a health check. If the health check fails, the docker container is treated as unhealthy, and is restarted accordingly. Just like we do with the other docker containers, we would like to expose such a health port as well on the Jenkins Slave. If it is not responding anymore, AWS will restart the docker container, and we have a stable system.

We already explored the Nagios monitoring, but it looks like this is exposing the health in the master, and not directly in the slave.

Is it available, or how can we add such functionality? Could you lead us in the right direction, or recommend plugins which need to be installed, ...

* Can we open the default HTTP API of Jenkins in such a Jenkins slave (https://wiki.jenkins.io/display/JENKINS/Remote+access+API)
* Another approach could be to add and start NGINX in docker and point the healthcheck to NGINX. But with this approach, we are not really checking the health of the the Jenkins slave.

# Answer
To me it looks like you are trying to reinvent the wheel. As I understood you would like to restart a container if a health check fails. As you are using docker you should use a orchestration platform in my opinion like Fargate (ECS) or Kubernetes (k8s). I know from the latter that it is possible to define healthchecks, like readiness and health probes. If the health check fails, kubernetes will restart the POD (auto healing).

It could be possible that you would argue that k8s is not an option. If that is the case, you could also have a look to AWS OpsWorks. Although I did not try it myself yet, it is able to perform auto healing as well.
> 1 votes

# Answer
Take a look at this Jenkins Plugin: https://plugins.jenkins.io/slave-status/

It'll expose a healthcheck endpoint + other information like memory on port 3141 (you can configure a custom port). You can then ping that healthcheck endpoint and send the data to CloudWatch.
> -1 votes

---
Tags: jenkins, amazon-web-services, jenkins-plugins
---


title: Nginx healthcheck does not stop

text:
# Question
Title: Nginx healthcheck does not stop
I have the following healthcheck condition for a nginx container inside my docker-compose.yml file:

```
healthcheck:
  test: ["CMD", "curl", "-f", "http://front-service:80"]
  interval: 15s
  timeout: 5s
  retries: 5

```

The problem is that I see in the logs continuous calls to `http://front-service:80` even after nginx is running fine.

```
front-service    | 172.18.0.4 - - [02/Mar/2021:09:58:23 -0300] "GET / HTTP/1.1" 200 2611 "-" "curl/7.64.0" "-"
front-service    | 172.18.0.4 - - [02/Mar/2021:09:58:38 -0300] "GET / HTTP/1.1" 200 2611 "-" "curl/7.64.0" "-"

```

Wouldn't the correct behaviour be that if the check succeeds, it stops pinging my frontend endpoint? I don't want it to keep pinging after it finds out that the server is alive.

# Answer
> 1 votes
If you see the logs, there's exactly 15s between the calls. Which is what you have configured in you docker compose file.

If you don't want it to keep pinging after it finds out that the server is alive, you have two options (which is not recommended to do):

* Don't use healthcheck, just add your logic in the `command` part
* Setup a higher interval for your healthcheck

Also, I have questions for you:

* Why do you want the healthcheck to be done only once?
* What if you server goes down, don't you want to restart it?

My recommendation is to keep this config as is because it does the work of checking for your service availability and will restart as needed and quickly.

---
Tags: docker-compose, nginx
---


title: Docker image, containers and layers

text:
# Question
Title: Docker image, containers and layers
I want to use docker to compile a custom PHP executable.

I plan to use alpine as a base image, installing the required tools (build-essentials, etc).

The goal is to have the most minimalist resulting image for later use in child images.

Would it make sense to tar the resulting php binary and publish to a host, and later have the child layers just pull in the php binary?

Or does it make sense to keep the original image? What happens to all the build tools and libraries? Do they become part of the image?

# Answer
This is what I would to better handle the base docker images lifecycle:

* Create a git repo to save your Dockerfile + configs
* Create a Dockerfile with alpine + all the configs/libraries you want
* Add your php.ini + php-cli.ini configs
* Have a Jenkins job to start on each push to that repo:
    + Build your docker image
    + Tag it incrementally
    + Push to your registry
* Use that base image to create your final docker image
> 1 votes

---
Tags: docker
---


# Question
Title: Docker desktop Ports binding not available
Why are the port options not available in docker desktop (windows 10)?

current settings (I have tried both engine options):

# Answer
> 1 votes
The Docker Desktop is giving the option to publish ports to the container ports that have been "exposed" in the image. Exposing a port in an image is documentation from the image creator to those running the image, to know what port the application inside the container should be listening. And in this case, it's Docker Desktop that's running the image and looking at that documentation.

More details on "expose" can be found in Docker's documentation: https://docs.docker.com/engine/reference/builder/#expose

---
Tags: docker
---


# Question
Title: Why is there insufficient memory on kubernetes node
I have an elasticsearch cluster in a kubernetes cluster. I have the data pods going to memory optimized nodes which are tainted so that only the elasticsearch data pods get scheduled to the. Right now I have 3 memory optimized ec2 instances for these data pods. They are r5.2Xlarge's which have 64G of memory. Here is the output of one of these r5 nodes. (they all look the same)

```
Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           32461564Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      65049812Ki
  pods:                        110
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         8
  ephemeral-storage:           29916577333
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      64947412Ki
  pods:                        110
System Info:
  Machine ID:                 ec223b5ea23ea6bd5b06e8ed0a733d2d
  System UUID:                ec223b5e-a23e-a6bd-5b06-e8ed0a733d2d
  Boot ID:                    798aca5f-d9e1-4c9f-b75d-e16f7ba2d514
  Kernel Version:             5.4.0-1024-aws
  OS Image:                   Ubuntu 20.04.1 LTS
  Operating System:           linux
  Architecture:               amd64
  Container Runtime Version:  docker://19.3.11
  Kubelet Version:            v1.18.10
  Kube-Proxy Version:         v1.18.10
Non-terminated Pods:          (5 in total)
  Namespace                   Name                                                        CPU Requests  CPU Limits  Memory Requests  Memory Limits  AGE
  ---------                   ----                                                        ------------  ----------  ---------------  -------------  ---
  amazon-cloudwatch           fluentd-cloudwatch-tzsv4                                    100m (1%)     0 (0%)      200Mi (0%)       400Mi (0%)     21d
  default                     prometheus-prometheus-node-exporter-tvmd4                   100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  es                          elasticsearch-data-0                                        500m (6%)     1 (12%)     8Gi (12%)        8Gi (12%)      14m
  kube-system                 calico-node-dhxg5                                           100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
  kube-system                 kube-proxy-ip-10-1-12-115.us-gov-west-1.compute.internal    100m (1%)     0 (0%)      0 (0%)           0 (0%)         21d
Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource                    Requests      Limits
  --------                    --------      ------
  cpu                         900m (11%)    1 (12%)
  memory                      8392Mi (13%)  8592Mi (13%)
  ephemeral-storage           0 (0%)        0 (0%)
  hugepages-1Gi               0 (0%)        0 (0%)
  hugepages-2Mi               0 (0%)        0 (0%)
  attachable-volumes-aws-ebs  0             0

```

Here is what my cluster looks like

```
kubectl get pods -n es
NAME                                                 READY   STATUS    RESTARTS   AGE
elasticsearch-client-0                               1/1     Running   0          77m
elasticsearch-client-1                               1/1     Running   0          77m
elasticsearch-data-0                                 1/1     Running   0          77m
elasticsearch-data-1                                 1/1     Running   0          77m
elasticsearch-data-2                                 1/1     Running   0          77m
elasticsearch-data-3                                 0/1     Pending   0          77m
elasticsearch-data-4                                 0/1     Pending   0          77m
elasticsearch-data-5                                 0/1     Pending   0          77m
elasticsearch-data-6                                 0/1     Pending   0          77m
elasticsearch-data-7                                 0/1     Pending   0          77m
elasticsearch-master-0                               2/2     Running   0          77m
elasticsearch-master-1                               2/2     Running   0          77m
prometheus-elasticsearch-exporter-6d6c5d49cf-4w7gc   1/1     Running   0          22h

```

Here is the events when I describe pod

```
Events:
  Type     Reason            Age                  From               Message
  ----     ------            ----                 ----               -------
  Warning  FailedScheduling  56s (x5 over 3m35s)  default-scheduler  0/11 nodes are available: 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules, 5 node(s) didn't match node selector.

```

Here is my resource limits and requests for the data pods

```
    Limits:
      cpu:     1
      memory:  8Gi
    Requests:
      cpu:      500m
      memory:   8Gi

```

Here is what my nodeAffinity looks like

```
nodeAffinity:
  requiredDuringSchedulingIgnoredDuringExecution:
    nodeSelectorTerms:
    - matchExpressions:
      - key: es-data
        operator: In
        values:
        - "true"

```

And my tolerations

```
tolerations:
  - key: "es-data"
    operator: "Equal"
    value: "true"
    effect: "NoSchedule"

```

And here is my node tolerance when I describe the node

```
Taints:             es-data=true:NoSchedule

```

I tainted it like

```
kubectl taint nodes <node> es-data=true:NoSchedule

```

According to my calculations based on my understand (which is probably wrong) my data pods are only asking for 8G of memory from a node which has 64G available, and only one pod requesting 8G of memory is already using it. So it should have theoretically 56G of memory left to other pods requesting to be scheduled to it. And even the memory being used shows me it's only 13% used. Why can't it schedule? How can I troubleshoot? Am I misunderstanding how this should work? What else can I tell you which helps troubleshoot this?

Resolution: Based on Hakob's comments, the issue is that I had `nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution` set, which is a hard requirement directing the scheduler to only schedule one to each node. What I needed to do in order to schedule more than one to each node was change it to `nodeSelector`. If you are reaching this, please note Hakob's recommendation for why it is not suggested best practice to do this. I agree with that advice. Though in my case it is a requirement coming from client and did not have the option even after have discussion why they should not be doing this. So please take that into consideration when applying this change.

# Answer
> 3 Insufficient memory, 3 node(s) didn't match pod affinity/anti-affinity, 3 node(s) didn't satisfy existing pods anti-affinity rules

This means that ES trying to find a different node to deploy separately all the pods of ES. But the cause of your node count is not enough to run one pod on each node, the other pods remain pending state.

For more information read here

So from here, you have a 2 choice Neo))

1. Add more nodes until all pods will have been scheduled on the different nodes
2. Minimize your ES sts pod count to 3
> 2 votes

---
Tags: kubernetes
---


# Question
Title: What are the cost effective ways to deploy services using Kubernetes on multiple deployment environments?
We are converting a traditional docker-compose based service stack to a Kubernetes based services

Before the conversion we had three deployment environments

1. Local : Basically this is the developer's working machine.
2. Dev : Where the everyday releases happens and developers who belongs to other projects like web apps and mobile app uses endpoints from this environment to get their daily work done.
3. Staging : Project owners/clients see this space
4. Production : This is where the world facing actual services are living.

So when its come to Kubernetes, we need to specially invest on Kubernetes supported clusters and those are heavily expensive. Since we need three deployment environments , do we need to purchase three Kubernetes cluster groupes? Or how other traditional teams managing this requirement?

# Answer
Based on my experience I'd suggest following scenario:

First you should make your apps deployment process similar for all environments, so same tool and set of files will be responsible for this part. And only difference should goes to parameters.

And second it's always a good practice to have kubernetes cluster per stage. Very common situation when your changes goes not only to kubernetes apps manifests, but to surrounding infrastructure also (LoadBalancer, Object Buckets, Disks Storage, etc).

You cat try to cut costs on kubernetes part by using same cluster for dev/stage but still situations with differences in infra resources could happen. And that could lead to more efforts on support and release deployment level.

In case you will use AWS, you can put kuberentes load to EC2 Spot Instances and that could heavily reduce your monthly bill.

As for toolset for k8s apps manifests, I've tried multiple tools and ways, and from my experience terraform+terragrunt works better here. Single tool for all infra resources, parameterization works easily, common code could be separated to reusable modules.

Other option is kustomize or helm, but those tools could help with kubernetes manifests only, and can't manage external resources.
> 1 votes

---
Tags: kubernetes, deployment, servers, orchestration, environments
---


# Question
Title: Auto create Jenkins jobs for branches and pull request via declarative code in scm
I am looking to auto create jobs in jenkins upon pull request , branches, master push etc similar to what we do in Gitlab. My SCM is butbucket here.

I have so far setup docker based agent integration with Jenkins and butbucket, when I create a job and configure it to use repo it all works fine , but I just want to remove altogether a step of job creation in jenkins and want the workflow like this:

In butbucket source code repo to keep all pipeline configuration for and branch and tag to trigger Jenkins pipeline without touching Jenkins for job creation or any config creation. Just want to drive all via the script in code repo for pipeline .

Any recommendations or help for workflow would be appreciated

# Answer
Use the Bitbucket Branch Source plugin, which is an official plugin developed by the Jenkins developers. This plugin automatically creates jobs for all branches and pull requests of all repos in a project, and if you use Bitbucket Cloud it will also set up webhooks in Bitbucket to ensure pushes and PR updates automatically trigger builds in Jenkins.
> 1 votes

---
Tags: jenkins, continuous-integration, jenkins-pipeline, gitlab, bitbucket
---


# Question
Title: Jenkins downstream pipeline jobs not always triggered as expected
I have three separate Multibranch pipeline jobs set up on Jenkins, which I want to be called sequentially as downstream-jobs. Please note that, all these 3 jobs have **separate** git repositories!

Each job has its own Jenkinsfile.

The first job is always triggered manually by me through the Jenkins UI. At the last stage of the 1st job's Jenkinsfile, the second job is being triggered, as shown below. (The second job triggers the third job, in the same way)

```
stage('Trigger second job')
{
   steps {
       build job: 'second_job_name', propagate: true, wait: false
   }
}

```

Now this step always seems to be executed successfully when the first job ends, no errors or warnings show up on the logs. This should mean that the second downstream job has been called, but unfortunately that's not always the case:

* Sometimes the first job triggers the second job, but the second job never actually triggers the third one, although it seems to pass the corresponding Jenkinsfile stage.
* Sometimes the first job completes but never triggers the second job at all.

The only time this setup works as expected, is when there are **new commits to all 3 git repos**. Then all 3 jobs are triggered sequentially without issues.

Does anyone know the reason behind this inconsistent behavior? Is it maybe because the jobs are based on different git repos? Is there anything I can do to fix this?

Thanks

# Answer
> 1 votes
I suspect this job name is the issue:

```
build job: 'second_job_name'

```

In a Multibranch pipeline, there are typically folders in the job name. For instance, if you have a Multibranch project named "my\_project", a repo in that project named "my\_repo", and a branch within that repo that you wish to build, let's say "master", then your build command would look like:

```
build job: 'my_project/my_repo/master'

```

If you only specify up to the project or repo level in your job name, but don't include the branch, instead of running a build for the branch, Jenkins will scan for updates to all branches within that project/repo, and build any new commits which it hasn't yet built. This results in exactly the symptoms you described:

> The only time this setup works as expected, is when there are new commits to all 3 git repos. Then all 3 jobs are triggered sequentially without issues.

---
Tags: jenkins, jenkins-pipeline, jenkinsfile
---


# Question
Title: how can I make jenkins find 2 jenkinsfile in a repo with multibranch?
I am quite new to DevOps and Jenkins in particular. I have played around and created some pipelines jobs. I wanted now to start practicing with a jenkins in local, and apply CI-CD to an small project I had. I have created a multibranch pipeline. which will have 3 repos (front, back, integration). The first 2 repos have 2 jenkinsfile each in the root folder (Jenkinsfile.ci and Jenkinsfile.cd) but when I scan the multibranch pipeline it says no Jenkinsfile has been found. How would be the right way to do this?

# Answer
You need to update your multibranch pipeline build configuration to use **Jenkinsfile.ci** and **Jenkinsfile.cd** instead of **Jenkinsfile**. And you would have to create two multibranch pipelines, one for Jenkinsfile.ci and other for running Jenkinsfile.cd.
> 4 votes

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: AWS Inspector "High Risk" : port open to the internet with IP whitelisting
We use AWS Inspector to identify risks with our AWS assets. Today I received a finding, with a "high" severity. The finding said that the FTP port was open to the internet. However, after reviewing the security group associated with the instance, I saw that the FTP port is only open to a set of whitelisted IP addresses that are trusted.

Does this really constitute a high severity in your opinion? Is it problematic to open ports to the internet with a specific set of whitelisted IPs?

# Answer
> 2 votes
If it is FTP (and not FTPS) then yes, it is a high severity risk. You are sending unencrypted traffic over the open internet. Even if you are whitelisting specific IP addresses, the traffic is still unencrypted between those specific instances.

In addition, FTP is a very old (created in 1971) and non-secure protocol. It is not meant to be a secure method of transferring files. I recommend you research some of the security concerns with FTP and switch to either FTPS or more preferably, use a more native AWS approach for transferring files between instances. If your organization is under any regulation (FEDRAMP, HIPPA, etc.), using FTP across the open internet will not meet the compliance requirements.

---
Tags: amazon-web-services, security, networking
---


title: How to use terraform import with module

text:
# Question
Title: How to use terraform import with module
Terraform v0.11.11

We have a quite big AWS setup created with Terraform. Different regions are applied separately, and that is why we have most of the things in modules which are imported like this:

```
module "assets" {
  source       = "../modules/aws-assets"
  environment  = "${var.environment}"
  upload_users = "${var.upload_users}"
}

```

We have one s3 bucket created manually, and I try to import it. Terraform import command does not find the configuration file and errors.

```
Error: resource address "aws_s3_bucket.machine-learning" does not exist in the configuration.

```

Before importing this resource, please create its configuration in the root module. For example:

```
resource "aws_s3_bucket" "machine-learning" {
  # (resource arguments)
}

```

terraform apply/plan do find the configuration and wants to create it which is not what we want?

Thanks for you help!

================================

Edit:

now moved configuration to root and ended up with different error...

# Answer
The `terraform import` command uses the "Resource Address" syntax, which is a way to talk about objects in a configuration from outside of that configuration. (This is as opposed to references in the main Terraform language, which are always resolved in the context of a particular module.)

To refer to a resource that is declared in a child module, you can add a module path to the beginning of the address:

```
terraform import module.assets.aws_s3_bucket.machine-learning BUCKET-NAME

```

---

If you aren't sure which address to use for a particular resource instance, you can run `terraform plan` to see which instance address Terraform is proposing to *create*, and then use that same address with `terraform import` instead to tell Terraform to use the existing object.

For addresses that include sequences like `[0]` and `["foo"]` to represent one of multiple instances of a module or resource, you'll need to use escaping or quoting to make sure your shell doesn't interpret those as its own metacharacters, and instead passes them on literally to Terraform:

* On Unix-style shells, use single quotes to make the inner address be taken literally: `terraform import 'aws_instance.example["foo"]'`
* On Windows, from the command interpreter `cmd.exe`, escape any quotes with a backslash:

`terraform import "aws_instance.example[\"foo\"]"`

(PowerShell's escaping rules for external programs are very awkward and so I would not suggest using PowerShell to run a command like this.)
> 50 votes

# Answer
Run terraform plan first and use the names in the output.

Goal here is to import in the resources that are listed in the plan output so next time you run plan they won't be listed for add.

So aws\_s3\_bucket.machine-learning didn't exist in the plan output I bet. It could have been module.aws\_s3\_bucket.machine-learning or even module.aws\_s3\_bucket.machine-learning\[0\] depending on what your aws-assets.tf does.
> 8 votes

# Answer
main.tf

```
module "assets" {
  source       = "./modules"
}

```

modules/aws-assests.tf

```
resource "aws_s3_bucket" "machine-learning" {
  # (resource arguments)
}

```

To install the aws\_s3\_bucket module run the below command

```
# terraform init

```
> 4 votes

---
Tags: terraform
---


# Question
Title: How do I implement user authentication and management in Kubernetes?
I'm trying to architect a micro-service application, that runs in Kubernetes. It consists of several RESTful APIs. I want a central point to implement authentication for these APIs.

In my product, users would be able to sign themselves up on a website. Signing up would have to give them access to the API's.

So what I'm looking for is a/the way to have central authentication in Kubernetes for users accessing the APIs. I would also require to be able to programmatically add users to this auth layer, so I can have users sign themselves up for access.

I've seen ingress controllers that do central auth, but I haven't found if they support programmatically adding accounts. I've also seen API gateway software that might do what I want, but I'm not sure if and if so, which ones can run in Kubernetes.

Does anyone have experience in such a setup, that can point me in the right direction?

# Answer
> I'm trying to architect a micro-service application, that runs in Kubernetes. It consists of several RESTful APIs. I want a central point to implement authentication for these APIs.

This is a good idea.

> So what I'm looking for is a/the way to have central authentication in Kubernetes for users accessing the APIs. I would also require to be able to programmatically add users to this auth layer, so I can have users sign themselves up for access.

Just to make it clear: Kubernetes is a container orchestrator, e.g. a way to manage your services. It consists of a *control plane* (think management of apps) and a *data plane* (think your apps and customer traffic). Both parts may require proper authentication and authorization. When it comes to your "business idea" - it is about the *data plane* \- here this problem is orthogonal to the platform (e.g. Kubernetes or Heroku) that you run your apps on. So think how you should solve your problem with Microservice architecture, and not so much about Kubernetes.

> In my product, users would be able to sign themselves up on a website. Signing up would have to give them access to the API's.

The most common protocol for this nowadays is to use OpenID Connect \- that is the protocol behind both e.g. Google Accounts and Microsoft Azure AD. What you need is an *OpenID Connect provider* implementation. The most commonly used open source implementation is Keycloak. Remember, this kind of service is *stateful* and you will need a database as well.

Instead of setting up your own OpenID Connect provider, you could also let the user use an account that he already has, e.g. a Google Account or Twitter Account - it depends on your use case.

> I've seen ingress controllers that do central auth, but I haven't found if they support programmatically adding accounts. I've also seen API gateway software that might do what I want, but I'm not sure if and if so, which ones can run in Kubernetes.

What you talk about here is mostly *validation of authentication* and this is sometimes done using "auth aware ingress controllers". With OpenID Connect protocol, the user send a JWT-token as proof of authentication, and you could redirect unauthenticated users to your *Open ID Connect provider*.
> 1 votes

---
Tags: kubernetes, ingress, authentication, api-gateway
---


# Question
Title: NetworkPolicy: Why are pod selectors used over "service selectors"?
My understanding of containers is that they should be considered to be short-lived. In addition, we can scale containers (and if using a container orchestrator like k8s, via pods) by creating additional instances on one or more machines.

In order to allow for the ephemeral nature of containers, as well as to allow users to access an application from only one address (say, www.myapp.com), we must abstract away the container. From now on, I'm going to assume we're using k8s as the container orchestrator and speak only of pods. In k8s, we do this by using `Service`s to expose pods (either to other pods which comprise the application, or directly to users). `Service`s allow us to load balance and allow for deployment scenarios such as blue/green and zero-downtime deployments while providing a consistent and abstract interface from which to access the underlying pods.

With this in mind, I have a question about the `NetworkPolicy` resource in k8s. When creating a `NetworkPolicy`, a `podSelector` is used to determine to which pod(s) a network policy applies. However, since a `Service` must be created in order to expose a pod, why is it, then, that a `NetworkPolicy` uses a `podSelector` instead of a "`serviceSelector`"? Wouldn't the concept of a "`serviceSelector`" be more general/abstract, and better reflect to what the policy is really allowing access? Is there a reason I'm unaware of for why network policies are "applied" to pods versus services? (Not to say that you would *never* want a `NetworkPolicy` to apply directly to a pod, but, generally speaking, most times we apply them to pods which are exposed by a `Service`—or maybe I'm totally off base here?)

# Answer
> 1 votes
Pods is the deployment unit in Kubernetes and it is also the unit that gets a single unique IP address within the cluster.

You can access Pods without going via a Service. A Service is mostly there to get a "stable" address for an application, where the traffic is then load balanced to any Pod representing the Service. Both Pods and Services gets a DNS name in the cluster.

A Service has a selector to identify what Pods belong to the Service, but a Pod can receive traffic from multiple Services. In the same way a NetworkPolicy has a selector to select the Pods where it is applied.

Since a Pod is the smallest unit in Kubernetes, that gets its own unique network address it also makes sense to add the network security (e.g. Network Policies) at this level.

---
Tags: kubernetes
---


# Question
Title: What tool or type of tool am I looking for?
We have an operations team that looks after our running tenants. They respond to requests and incidents from users by following specific processes, they might for instance restart a container on an instance or undelete a resource a user deleted by accident.

Because these are repeatable actions they should be automated by rolling them into scripts. We want the jobs to be executed centrally so we can be able to track who ran which job and ideally have a central dashboard.

I need to investigate a platform to provide to the Operations team to get this job done.

Being from a dev background Jenkins would be my obvious choice but I feel like this must be a solved problem in the devops/ops world.

I've tried to research this but part of the problem is I don't know the name of the *class* of tool I'm looking for. When I look for automation platforms I find CI systems but they don't really fit with my problem.

Which tool or type of tool is appropriate to my needs?

# Answer
> 4 votes
Jenkins can work just fine for this purpose. In my view, Jenkins is a general-purpose automation tool, not necessarily a CI/CD tool specifically.

I'm on an Ops team and we use both Jenkins and Rundeck. We generally prefer Jenkins because we find it difficult to navigate the Rundeck UI and find things like job output, but on the other hand it's often easier to write jobs for Rundeck, because you don't need to know Groovy/Pipeline. Here are some examples of what we use each for:

* Install package updates on managed systems (yum/dnf packages) - Rundeck
* Deploy & provision new VMs - Jenkins
* Run Chef on managed systems - Rundeck
* Clean up stale Computer objects in Active Directory - Rundeck
* Clean up stale subscriptions in Red Hat Satellite - Rundeck
* Create & configure new repositories in Bitbucket from templates - Jenkins
* Build new vagrant images - Jenkins
* Backup & upgrade some services that my team manages (AWX, Red Hat Satellite, etc.) - Jenkins or Rundeck depending on the service

I'm not claiming that either of these tools is the best choice for any of these particular examples, just that my team uses them for these purposes and they generally work well. Historically, which tool we use for each job generally has come down to personal preference of the person who created the job.

In terms of the name of this type of tool, we just call them "automation tools" or "automation platforms". I'm not sure if there's a more "industry-standard" term for this type of tool.

---
Tags: jenkins, operations
---


# Question
Title: How can I use a buildah script for image generation using GitHub Actions?
Buildah scripts typically use shell. You can see an example of a script here,

```
#!/bin/sh
ctr=$(buildah from alpine:3)
buildah commit "$ctr" myAlpineImage

```

Let's say I have such a shell script that produces an image "myAlpineImage". How can I use GitHub CI to automate the creation of this image, and preferably uploading it to GitHub Container Registry?

# Answer
> 1 votes
Here is what I did, I create a Makefile that includes my own method of image creation,

```
.PHONY: image clean image-release
image: image-release

image-release:
    cargo build --release
    buildah unshare ./image/distroless.sh release

clean:
    cargo -v clean

```

For this purpose the only thing that matters is the image created by `./image/distroless.sh` matches the action (`redhat-actions/push-to-registry`) in the following flow. We'll continue using the name `myAlpineImage` as in the question,

```
name: Distroless Image Creation

on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

env:
  CARGO_TERM_COLOR: always

jobs:
  build:

    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v2
    - name: Download and create image.
      run: make image-release
    - name: Push To Registry
      uses: redhat-actions/push-to-registry@v2.1.1
      with:
        image: myAlpineImage
        tags: latest
        registry: ghcr.io/evancarroll/project
        username: evancarroll
        password: ${{ secrets.GHCR_TOKEN }}

```

You can see here we make the call to `make image-release` which calls the buildah script and makes the image rootless.

Then I added the secret for `GHCR_TOKEN` to my repo. The token is generated in the "Settings / Developer settings / Personal access" tokens on github..

---
Tags: containers, github, github-actions
---


title: yaml files in jenkins pipeline

text:
# Question
Title: yaml files in jenkins pipeline
Just started out working on jenkins, among other things I'm trying to understand the role of yaml in pipelines. I understand that pipelines or declarative pipeline (do other type of pipelines exist in jenkins?) are defined using a syntax that is based on an extension of groovy and that yaml can be used along with some plugins or extended library to generate an on the fly groovy-based pipeline definition. Is my understanding correct?

# Answer
The blog post you linked to in your comment which describes using YAML to generate Pipelines is about an unofficial 3rd-party Jenkins plugin which adds this feature; neither native Jenkins nor official plugins support using YAML to generate Pipelines.

---

> pipelines or declarative pipeline (do other type of pipelines exist in jenkins?) are defined using a syntax that is based on an extension of groovy

There are two types of natively-supported Pipelines: Scripted Pipelines (which are pure Groovy scripts) and Declarative Pipelines (which use a Groovy-based DSL).
> 1 votes

# Answer
In most case in jenkins yaml is not used all command are written in grove, on the fly you can create some snippets using jenkins pipeline syntax generator in order to understand better check documantation https://www.jenkins.io/doc/book/pipeline/getting-started/#snippet-generator it is not like in you run one command like kubectl create deployment and whole manifest is defined also you can use blue ocean editor which is much better for creating pipeline. there are some implementation of yaml pipelines but they are not widely used check this posts

https://plugins.jenkins.io/pipeline-as-yaml/

https://medium.com/wolox/dynamic-jenkins-pipelines-b04066371fbc
> 0 votes

---
Tags: jenkins, jenkins-plugins, yaml
---


# Question
Title: Jenkins: Re-use repository with reference option and do changes on it
I would like to setup a Jenkins cluster where the slaves are building an Android ROM with some parameters like branch (e.g. `lineage-16.0`) and some optional patches. The whole repository will consume around 100GB.

Let's say i want to build branch `lineage-16.0` with patch-A and patch-B and in parallel build branch `lineage-16.0` with patch-A only and in parallel build branch `lineage-17.1` with no patches and so on.

I just read about the `--reference` option in `git` (which can also be used in `repo`). Is it possible to use it this way or will changing the branch and applying patches modify the reference clone as well as it only uses pointers?

I need to save disk space and network traffic somehow. If this is not possible is there any other way without duplicating the 100GB for each jenkins job? I'm also open for other free tools than Jenkins.

# Answer
> If this is not possible is there any other way without duplicating the 100GB for each jenkins job?

To avoid frequent clones of the repository, you can put a single clone of the repository on-disk on your executor nodes outside of the normal workspace path. Then, in your Jenkins job, you can cd to that directory with the dir step, update the repo, and checkout the branch you want, e.g.:

```
dir('/path/to/repo') {
  sh('git fetch --all')
  sh('git checkout origin/lineage-16.0')
  // your build steps here
}

```

However, this has the major drawback that the one repo will be shared across all of your jobs, which means you cannot run those jobs in parallel without risking corruption. It sounds like running jobs in parallel was one of your requirements, so this may be a no-go.

You could probably get parallel builds to work by using a copy-on-write filesystem so that the repo appears to be in multiple independent directories but only actually has to be cloned once, but I've never tried something like this, so I can't say how well it would work.
> 0 votes

# Answer
Why is your "source repo" 100GB?

Admittedly, the article is several years old, but MS reported the windows code base repo was the largest known repo at the time and was 300GB.

Is your **source** really that big? Your Git repo should NOT contain derived artifacts (.class, .jar/war, .o, .dll, .exe, etc), nor 3rd-party libraries. There are other tools for that. You can use a .gitignore file to exclude these file types from being checked into your repo.

If you check in a binary file, Git will generally store a complete copy (generally non-compressible) of every copy you check in. I have seen developers check-in not just 3rd-party OSS jars in multiple locations in the same repo and same jars across multiple repos, resulting in massive duplication and wasted space, but then keep checking new versions. I've seen both parts of the Weblogic installer ( 2x 600 MB+) checked-in. And found one which was continuously checking in the Docker image he built which included 3party installers, making his repo almost 1GB larger every check-in!

You don't mention language, but the standard tools for Java libraries has been Nexus and Artifactory. They now handle as extensive range of language artifact types as well.

These tool also create local caches of the artifacts they retrieve so you no longer have to go to the remote host across the network if they have not changed.

Git-LFS is an excellent extension to handle large, non-derived binary files (audio samples, videos, datasets, and graphics).

Look at using BFG to remove unwanted artifacts from history and compress the repo.

As documented in the "largest repo ever" article, MS have contributed the "VFS for Git" to deal with really large repos if that really is the only solution available to you.

This is not an extensive how-to, just advice, "I think you're doing it wrong and that's why it's a PITA", plus some pointers to tools. You need to review your entire approach. That may include other tools like Ansible or Puppet to manage your infra configuration as well.

**Summary:** Manage your infrastructure as code and keep only source code in your source code repositories. Or just get a faster network.
> -1 votes

---
Tags: jenkins, jenkins-pipeline, git
---


# Question
Title: How many DB instances to use? When to switch to container services?
I'm looking for some insights into cloud management. I won't go into much detail, I just need some general information/resources that I can follow, about these topics:

1. When to use container services like Fargate, ACI, App Service? If the project isn't big enough, is it worth to migrate from more traditional approach (so VM instances with docker-compose running inside)?
2. Managed DBs - I totally get their advantages and I'm a huge fan. But they are expensive. How many instances should be deployed? Let's assume that I have 5 independent projects and each project has dev/staging/prod environments. Should I have 15 DB servers (instances) running? Or can I perhaps run one DB instance and serve dev/staging environments from there and keep prod instances seperate? Important: I mean database server instances, not databases.

I've been trying to find some info online but most docs don't go into that much detail. Could you give me some more insight on how to run this type of infra? Mind you, the projects are not big, the databases are 1-2 GB in size (dev/staging DBs much less than that). What approach would you choose and why?

# Answer
The choice of the approach is a risk assessment, if one of the DB client (application) goes wild and bring your server instance down, what is the cost in term of human hours lost during the outage ?

If you run the DB yourself instead of managed you have to include your time costs to update it, backup it, monitoring it, etc. is it really cheaper ?

All this is very company specific and need to take into account the human cost, does a staging environment down block 1 or 80 devs ? Are the devs junior or senior ? The cost per hour of outage change a lot from these simple two metrics.

At the end of the day, that's a judgement on accepted risk of outage and capacity to restart versus administration time to avoid the outage for which there's no magical mathematics formula.
> 1 votes

# Answer
> Managed DBs - I totally get their advantages and I'm a huge fan. But they are expensive. How many instances should be deployed? Let's assume that I have 5 independent projects and each project has dev/staging/prod environments. Should I have 15 DB servers (instances) running? Or can I perhaps run one DB instance and serve dev/staging environments from there and keep prod instances seperate? Important: I mean database server instances, not databases.

I agree, this would give you too much overhead and it will be expensive. I would go for a single regional database server instance per environment. It would be good if a DBA could maintain it and setup proper backup and disaster recovery.

Remember, *microservices* should all have their own database. But this is *logical database*, e.g. typically a *schema* or *logical database* inside the instance. Each *microservice* should probably has its own dedicated *database user* as well.
> 0 votes

---
Tags: docker, azure, databases, infrastructure-as-code
---


# Question
Title: Spinnaker keeps showing disabled stages in Jenkins pipeline UI
I have a pipeline with a few disabled stages:

```
{
  "enabled": false,
  "completeOtherBranchesThenFail": true,
  "continuePipeline": false,
  "failPipeline": false,
  "parameters": {
    "FOO": "bar"
  },
  "type": "jenkins"
}

```

The doc states that disabled stages are hidden from the UI \[1\]:

> `enabled` \- Determines if stage will be available in the UI.

However, those stages are still present in the UI.

When I go to `Configure Pipeline` \> `Edit stage as JSON`, the `enabled` flag is set to `false`, so the UI has the right value.

1. https://spinnaker.io/guides/operator/custom-job-stages/#creating-a-custom-job-stage

Spinnaker version 1.23.5

# Answer
> 1 votes
I was not able to find how to *hide* a stage, however I found one can skip it using `stageEnabled` \[1\].

It takes a Spring Language Expression like so:

```
{
  "stageEnabled": {
    "expression": "false", // you can also use SpEL, e.g. ${someBoolean}
    "type": "expression"
  },
  "completeOtherBranchesThenFail": true,
  "continuePipeline": false,
  "failPipeline": false,
  "parameters": {
    "FOO": "bar"
  },
  "type": "jenkins"
}

```

Or from the UI editor: `Execution Options` \> `Conditional on Expression` \> Enter `false` \[2\].

1. https://docs.armory.io/docs/spinnaker-user-guides/using-dinghy/#stage-fields
2. https://spinnaker.io/guides/user/pipeline/expressions/#dynamically-skip-a-stage

---
Tags: spinnaker
---


# Question
Title: Simple service to check infrastructure status
I have a very simple setup here (solo IT guy): we have a on-prem server which runs some services. They should produce e.g. a daily logfile. I'd like to get a notification if this doesn't happen for some reason. I make a script, activated from cron, that checks this and sends me an email if it doesn't happen. However, this obviously doesn't work if e.g. the entire server goes down, my email script has a bug, etc, so I'd like to have the converse: have a daily check and get an email if it did NOT ran. I do have this with www.duplicati-monitoring.com: I get an email even if my on-prem server is down. I'd like to extend this to more infrastructure tests.

How is such a system called? Are there websites (preferrably free / open source) offering this? I looked at eg nagios, but it seems far too big, and it seems that I need to host it myself.

# Answer
> 2 votes
This is conventionally called a watchdog, deadman, or health check. Bruce's comment is good, including the part about a raspberry pi. You could also spin up something simple like monit in a container someplace like GCE or AWS.

Other than setting up an external service or server, there are also a bunch of phone apps that will do the pinging and/or http fetch and HTML parsing for you, right from your phone, providing you with a notification and/or audio alert when a condition isn't met. Since you're a one-man shop, using your own phone isn't substantially worse in terms of being a single point of failure, though you'll want to have a migration plan for when and if you do grow.

The thing I would probably do myself is keep the more complex monit configs on-site, summarize those on a simpler monit config from a stable IP at GCE, with a simple config on my (dynamic IP) phone watching the GCE monit. That lets firewall rules keep things relatively private other than that summary query from the phone, and when you add a new team member they just need to add that phone app with the one query. If you're comfortable with leaving the firewall rules more open, then you could skip the GCE step and just have your phone hit a summary statistic on your own server directly.

Back in the depths of time I used to use SMTP mail for this sort of thing a lot; these days I'd prefer the phone app for a more timely notification.

---
Tags: monitoring
---


# Question
Title: ansible & jinja2: How to reference JSON structure items whose keys are variables?
I am using ansible's `community.aws.iam_server_certificate_info` module to get information about a server certificate on aws ec2, and registering it as a variable **"server\_cert\_info\_result"**

Unfortunately, this module returns a JSON structure with a high level key that is the *actual value of the "name" of the server certificate*, as opposed to the more standard format of returning a list of dictionaries, one of which would be {name: \<actual\_name\_of\_cert\>}. I have the actual name of the cert as a variable **"server\_ssl\_cert\_name"** in my playbook. This is what the module's return value looks like, with the offending key highlighted:

```
TASK [aws_full_deployment : Debug server cert return info] ************************************************************************************************
ok: [localhost] => {
    "server_cert_info_result": {
        "ansible_module_results": {
            **"v6-full-deployment-SSL-Certificate"**: {
                "arn": "arn:aws-us-gov:iam::xxxxxxxxxxxxxxxxxxxxxxOBFUSCATED",                                         [truncated.....]

```

I want the **arn** value, in order to create an application elastic load balancer with and https listener. How do I retrieve the value of the **arn**, using jinja2 filters?

Things I've tried (which failed):

1. use the jinja builtin `attr` filter, which cannot accept the variable name of the cert inside it's parameter space, and throws an error. It only accepts strings as parameters, such that trying expand a playbook variable with the normal {{ }} will not work.

```
- name: Debug jinja2 attr filter
  ansible.builtin.debug:
    msg: "{{ server_cert_info_result.ansible_module_results | attr({{ server_ssl_certificate_name }} | attr('arn') }}"
  tags: list_server_certificate_info

```

which returns:

```
TASK [aws_full_deployment : Debug jinja2 attr filter] ***********************************************************************************************************************
fatal: [localhost]: FAILED! => {"msg": "template error while templating string: expected token ':', got '}'. String: {{ server_cert_info_result.ansible_module_results | attr({{ server_ssl_certificate_name }} | attr('arn') }}"}

```
2. `community.builtin.json_query`:

```
- name: Create and debug a query for json_query
  vars:
    cert_arn_query: "'{{ server_ssl_certificate_name }}.arn'"
  ansible.builtin.debug:
    var: cert_arn_query
  tags: list_server_certificate_info

```

Returns:

```
TASK [aws_full_deployment : Create and debug a query for json_query] ********************************************************************************************************
ok: [localhost] => {
    "cert_arn_query": "'v6-full-deployment-SSL-Certificate.arn'"
}

```

and

```
- name: Debug the json_query result of the above query
  vars:
    cert_arn_query: "'{{ server_ssl_certificate_name }}.arn'"
  ansible.builtin.debug:
    msg: "{{ server_cert_info_result.ansible_module_results | community.general.json_query(cert_arn_query) }}"
  tags: list_server_certificate_info

```

Which returns:

```
TASK [aws_full_deployment : Debug the json_query result of the above query] *************************************************************************************************
ok: [localhost] => {
    "msg": "v6-full-deployment-SSL-Certificate.arn"
}

```

which obviously fails when I try to create the load balancer, since the json\_query didn't actually retrieve the arn. Any ideas? I may have to resort to using `ansible.builtin.command` and using the raw `awscli` call, which I believe returns the data in a format that jinja could handle.

# Answer
Don't nest moustaches. Once you're in a Jinja context, you should reference variables by name without any further delimiters.

Using standard attribute access syntax can be a lot easier to read than the `attr()` filter (and actually works for this situation, which `attr()` doesn't):

```
- name: Debug jinja2
  ansible.builtin.debug:
    msg: "{{ server_cert_info_result.ansible_module_results[server_ssl_certificate_name].arn }}"
  tags: list_server_certificate_info

```
> 2 votes

# Answer
I think you don't need to do anything special:

```
"{{ server_cert_info_result.ansible_module_results[server_ssl_cert_name].arn }}"

```
> 0 votes

---
Tags: ansible, amazon-ec2, aws-iam, jinja, ansible-galaxy
---


# Question
Title: Ansible dynamic inventory using AWS plugin -- how to obtain internal DNS records or private IP?
Seeking help from those who've worked with dynamic inventory in Ansible.

I have Ansible set up with the aws\_ec2 plugin (not using the python script as it's been too problematic). Currently, I'm only using an aws\_ec2.yaml file as my inventory and variable source of truth.

When I run ansible-inventory --graph, Ansible returns my instances as public AWS DNS hostnames. I want either private DNS hostname or IP address so I SSH will against the hosts (we have firewall rule blocking ssh against the public AWS hostname).

How can I instruct the aws\_ec2 module to do this via the plugin? Thanks.

# Answer
The `hostnames` parameter controls what the plugin uses as the host name. It accepts a list of attributes, and will use the first one that the instance has defined:

```
hostnames:
  - private-dns-name
  - private-ip-address

```

You can also set the `ansible_host` variable using `compose`. This changes the connection target without modifying the inventory hostname:

```
compose:
  # Note the underscores; this functionality uses the Ansible variable names instead of the raw AWS attributes.
  ansible_host: private_ip_address

```
> 2 votes

---
Tags: amazon-web-services, ansible, amazon-ec2, configuration-management, ansible-inventory
---


# Question
Title: Can large scale services use a public cloud managed database service?
Theoretical question:
Is it possible for a very large scale service to use a major public cloud managed database service?
For example, a service with a scale of Netflix and a service like Amazon Aurora.
If not, which solutions are available for very larges scale services?

# Answer
> -2 votes
My conclusion after additional research and Preston Martin answer:
It depends.
If the workloads doesn't require more resources than the maximum supported by the service:
maximum horizonal and vertical (instance type) scale of the servers.
If it does, the data should be distributed across multiple managed database clusters.
If that doesn't meet the requirements, this solution should help:
Vitess

# Answer
> 3 votes
Yes, it is certainly possible, but it entirely depends on the architectural design of the entire system. Separating application services into manageable components (i.e. microservices) is one approach to this. While Netflix as a whole is one application in terms of the user experience, it is comprised of many applications/services that provide a small fraction of the functionality needed to run Netflix as a whole.

In terms of a database, the same principal can be applied. Netflix doesn't manage one database for the entire application, each component utilizes it's own scalable database service.

---
Tags: cloud, autoscaling
---


# Question
Title: Gitlab CI/CD: deploy docker image to existing kubernetes cluster
I have node.js application that I need to deploy to exising kubernetes cluster.

The cluster is setup using `kops` on AWS.

I have created `.gitlab-ci.yml` file for building docker images.

So, whenever a change is pushed to either `master` or `develop` branch. It will build the docker image.

Now, I have to deploy it to exisitng kubernetes cluster. I have already followed steps mentioned here to add existing cluster.

For now, I am using shared runner.

*How can I integrate kubernetes deployment after image is built with gitlab ci/cd to deploy on aws (cluster is created with kops)?*

*For registry I am using gitlab's container registry not docker hub.*

I have tried below `gitlab-ci.yaml` file,

*gitlab-ci.yaml*

```
stages:
  - docker-build
  - deploy

docker-build-master:
  image: docker:latest
  stage: docker-build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:prod" .
    - docker push "$CI_REGISTRY_IMAGE:prod"
  only:
    - master

deploy-prod:
  stage: deploy
  image: roffe/kubectl
  script:
    - kubectl apply -f scheduler-deployment.yaml
  only:
    - master

docker-build-dev:
  image: docker:latest
  stage: docker-build
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:dev" .
    - docker push "$CI_REGISTRY_IMAGE:dev"
  only:
    - develop

```

I am getting below error.

```
Executing "step_script" stage of the job script
00:01
Using docker image sha256:c8d24d490701efec4c8d544978b3e4ecc4855475a221b002a8f9e5e473398805 for roffe/kubectl with digest roffe/kubectl@sha256:ba13f8ffc55c83a7ca98a6e1337689fad8a5df418cb160fa1a741c80f42979bf ...
$ kubectl apply -f scheduler-deployment.yaml
error: unable to recognize "scheduler-deployment.yaml": Get http://localhost:8080/api?timeout=32s: dial tcp [::1]:8080: connect: connection refused
Cleaning up file based variables
00:00
ERROR: Job failed: exit code 1

```

File `scheduler-deployment.yaml` is a valid yaml file and it works when I apply manually.

Below is the `deployment` file.

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: sheduler-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: scheduler
  template:
    metadata:
      labels:
        app: scheduler
    spec:
      containers:
        - name: scheduler
          image: registry.gitlab.com/user/schedulingapis:dev
          env:
            - name: REDIS_HOST
              value: "redis://redis-master "
            - name: RABBITMQ_HOST
              value: "amqp://rabbitmq"
      imagePullSecrets:
        - name: registry-credentials

```

If someone has any better idea to automate deployment to a k8s cluster with gitlab.

*Pease do suggest. Any link or reference will also work.*

# Answer
> 1 votes
Your error is most likely from referencing the wrong directory in your script block of your job.

When GitLab executes your job in a container, it mounts the project directory to the `/builds` directory in the container. You can reference this either by specifying `/builds/scheduler-deployment.yaml` or by using the predefined environment variable `$CI_PROJECT_DIR` with `$CI_PROJECT_DIR/scheduler-deployment.yaml`.

---
Tags: docker, kubernetes, amazon-web-services, continuous-integration, gitlab
---


# Question
Title: How to deploy loki-stack helm chart with CRIO?
I am trying to configure loki-stack using a helm chart to my k8s cluster.

Since my k8s cluster uses CRIO and not docker (which is common nowdays) I need to change the `pipeline_stages:` config map to use `cri: {}` instead of `docker: {}` to parse the logs properly.

How do I change the helm chart so that `cri: {}` is used by default instead of `docker: {}` in all the `pipeline_stages:`? many thanks.

# Answer
Managed to solve this by changing the `values.yaml` in the `loki-stack` helm chart to the following:

```
promtail:
  enabled: true
  pipelineStages:
  - cri: {}

```

Enjoy.
> 1 votes

---
Tags: kubernetes, helm, logging, grafana, kubernetes-helm
---