# Question
Title: Can I use docker compose to run application container and its dependencies on google cloud or digital ocean, the same as locally?
I have a spring boot application, angular front end app, couchbase, and redis. They are all defined in a docker compose yaml file. I can make the application up and running locally. Is there a way to use docker compose tools/commands to do the same in google cloud or digital ocean?

# Answer
If you don't want/need platform-specific integrations and transforming your services to a cloud I guess GCP Compute Engine is for you -- it's like a usual virtual machine that you can start, SSH to it and have all privileges to install and run software the same way you did it on your dev machine.
> 2 votes

# Answer
If you want to avoid Kubernetes, then the simplest of the GCP options for a docker container is probably their Container-optimized VM images?

That lets you put the parameters to your `docker run ...` command in the VM config as per https://cloud.google.com/compute/docs/containers/deploying-containers#gcloud

This assumes you are able to publish your image to a repo that is accessible to your GCP compute VMs.
> 0 votes

# Answer
I don't know about Google Cloud but **yes on Digital Ocean**, I do it all the time! Just make sure you have Docker Compose installed on the droplet and then run the commands the same as you do locally.
> 0 votes

---
Tags: docker, docker-compose, google-cloud-platform
---


# Question
Title: How to use Vagrant and Docker-for-Windows on a Hyper-V system?
Back in the day, I was using VirtualBox (VB) and Vagrant (V) to run VMs. Since I installed docker-for-windows (DfW) and run V, VB complains about Hyper-V (HV).

```
[user@localhost ~]$ vagrant up
Stderr: VBoxManage.exe: error: VT-x is not available (VERR_VMX_NO_VMX)
VBoxManage.exe: error: Details: code E_FAIL (0x80004005), component
ConsoleWrap, interface IConsole

```

Although this link indicates that V supports HV, the above error message is contradictive. Why does VB look incompatible with HV, why V says that it supports HV and how to prevent that HV needs to be disabled in order to use V again.

```
| Product    | Version |
|:-----------|--------:|
| Vagrant    |   2.0.1 |
| VirtualBox |   5.2.4 |
| Windows    |      10 |

```

**Discussion**

The suggestion to switch Hyper-V on and off is not the preferred solution as it is time consuming.

If it would be possible to run V and DfW on the same system by replacing VB this could be an option.

# Answer
Confirmed by this answer on SO referencing Microsoft technet once Hyper-V is activated it mask the VT-X instructions of the processor to the OS on top of it and windows run like a guest.

Docker for Windows need hyper-V to work, so disabling it is not really an option.

There's no way to have virtual box running on top of Hyper-V and that's quite normal, you should not run an hypervisor within an hypervisor, they'll fight to orchestrate hardware acces.

The best workaround in my opinion is to create hyper-V virtual machine from vagrant with the hyper-V provider instead of virtual box machines, this way you can create VM or container on the same session.

Side note, vagrant already have a note about this in the hyper-v provider documentation page (found while searching the link to add above):

> **Warning**: Enabling Hyper-V will cause VirtualBox, VMware, and any other virtualization technology to no longer work. See this blog post for an easy way to create a boot entry to boot Windows without Hyper-V enabled, if there will be times you will need other hypervisors.
> 9 votes

# Answer
A few years later this thread is still the first result in Google, but running Docker+Vagrant (**with VirtualBox**) on Windows isn't a challenge anymore. You can easily run them both, as described here

If for some reason you want to use Vagrant with VirtualBox as a provider you should update to VB 6.x - after that, you can run VirtualBox, Vagrant, and Docker at the same time. Just configure your VM to use Hyper-V as Paravirtualization Interface. You can do it by VirtualBox GUI (VM's Settings -\> System -\> Acceleration -\> Paravirtualization Interface) or by using Vagrantfile:

```
Vagrant.configure("2") do |config|
  # your config (…)
  config.vm.provider "virtualbox" do |vb|
    vb.customize [
      "modifyvm", :id,
       " - paravirtprovider", "hyperv"
    ]
  end
  # your config (…)
end

```

This will work not only for Windows Enterprise/Pro/Edu but also for Home Edition (with WSL2 as Docker Backend). This solution shouldn't break your VirtualBox's machine, so you **don't have to switch hypervisorlaunchtype** over and over again.
> 5 votes

# Answer
Alongside Vagrant, you could use the old school Legacy 'Docker Toolbox' instead of 'Docker for Windows'. That means Docker will be slow since its running in a Linux VM I believe. https://docs.docker.com/toolbox/toolbox\_install\_windows/ Other than that, yes I have been restarting my computer everything i need HyperV for Docker or remove it when i want vagrant.
> 0 votes

---
Tags: windows, vagrant, virtualbox
---


# Question
Title: Add item(s) to yaml array only if defined (ansible)
How do I write a yaml file such that it defines an array with a list of values, and it skips those values from being added to the yaml array if the given value is undefined?

Specifically, I'm trying to define an array of IP addresses to which I want a service (sshd) to bind to, including the ipv4 and ipv6 addresses. If there is both an ipv4 & ipv6 address, then the array should contain both values. If there is only an ipv4 address or only an ipv6 address, then it should contain one of those values.

This doesn't work, as it will produce an error `"msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'address'"}`

```
    ssh_listen_to:
      - "{{ ansible_default_ipv4.address }}"
      - "{{ ansible_default_ipv6.address }}"

```

This also doesn't work, as it will end-up adding a line `ListenAddress`  with no argument, that breaks the sshd server (this may be a shortsighted design of my ansible role, but that's out-of-scope in this question).

```
    ssh_listen_to:
      - "{{ ansible_default_ipv4.address | default('') }}"
      - "{{ ansible_default_ipv6.address | default('') }}"

```

How can I update the above yaml block such that it will skip adding the values to the array if the value isn't defined? Note that this is being done in my playbook's `vars:` section, so no tasks-based solutions please.

# Answer
> 1 votes
Use `select()` with the `defined` test.

```
ssh_listen_to: "{{ [ansible_facts.default_ipv4.address, ansible_facts.default_ipv6.address] | select('defined') | list }}"

```

As for the linked post about matching dictionary keys in JMESPath, I wouldn't use JMESPath at all for that in Ansible. As I've grown more accustomed to Jinja I've found `json_query()` less and less necessary.

```
ssh_listen_to: "{{ ansible_facts | select('match', '.+_ipv[46]$') | map('extract', ansible_facts, 'address') | select('defined') | list }}"

```

`ansible_facts | select('match', '.+_ipv[46]$')` filters the keys using a regex. Iterating over a dictionary returns its keys, which are then matched against the provided regex.

Passing this filtered list of keys to `map('extract', ansible_facts, 'address')` extracts their values and looks up 'address' in each value. You could do this in two steps as `map('extract', ansible_facts) | map(attribute='address')`, if that makes things clearer.

`select('defined')` gets rid of any undefined values, then `list` converts the generator returned from `select()` into a list (this generator unrolling happens automatically in ansible-base/ansible-core \>= 2.10, but a lot of people are still on Ansible 2.9.)

# Answer
> 1 votes
Create the list in Jinja2 and convert it, e.g.

```
- hosts: localhost
  gather_facts: true
  tasks:
    - set_fact:
         ssh_listen_to: "{{ _ssh_listen_to|from_yaml }}"
      vars:
        _ssh_listen_to: |
          {% if ansible_default_ipv4.address is defined %}
          - {{ ansible_default_ipv4.address }}
          {% endif %}
          {% if ansible_default_ipv6.address is defined %}
          - {{ ansible_default_ipv6.address }}
          {% endif %}
    - debug:
        var: ssh_listen_to|type_debug
    - debug:
        var: ssh_listen_to

```

gives

```
TASK [debug] ********************************************************************
ok: [localhost] =>
  ssh_listen_to|type_debug: list

TASK [debug] ********************************************************************
ok: [localhost] =>
  ssh_listen_to:
  - 10.1.0.27

```

---

If you put the name of the variables into the list, e.g.

```
    vars_addr:
      - default_ipv4
      - default_ipv6

```

the task below gives the same result

```
    - set_fact:
         ssh_listen_to: "{{ vars_addr|
                            map('extract', ansible_facts, 'address')|
                            select('defined')|
                            list  }}"

```

# Answer
> 0 votes
This doesn't answer the question directly, but I found that I could query the entire ansible facts dictionary as a json document using the `json_query` filter and a jmespath expression.

For example, the following produces a list of all the ip addresses.

```
ssh_listen_to: "{{ ansible_facts | json_query('*.address') }}"

```

You can make this safer by just taking the default ipv4 & ipv6 addresses like so:

```
ssh_listen_to: "{{ ansible_facts | json_query('[@.default_ipv4, @.default_ipv6][].address') }}"

```

And you can even further check the sanity of the options by removing anything from the list that's not an IP address

```
ssh_listen_to: "{{ ansible_facts | json_query('[@.default_ipv4, @.default_ipv6][].address') | ansible.netcommon.ipaddr }}"

```

---
Tags: ansible, yaml
---


# Question
Title: How to get CPU and Memory usage of nodes in a specific Kubernetes instance group?
I have a couple separate instance groups, and `kubectl top nodes` will display the CPU and Memory usage of all nodes. I'd like to be able to just ask for the information for a specific instance group if possible.

# Answer
Do your instance groups have labels identifying the groups they belong to? If so, you can filter to them using the `-l`/`--selector` option, like so:

```
kubectl top nodes --selector instance-group=group-1

```

This assumes that the label name is `instance-group` and the label value for the group you want to query is `group-1`.
> 2 votes

---
Tags: kubernetes, kubectl
---


# Question
Title: Why do so many circleCI configs use `version: 2` for their workflows?
I keep seeing this pattern around, and I haven't been able to figure out *why* people do it. I figure there's some dusty tome somewhere explaining why, or maybe just my google skills are off today, but can anyone tell me *why* so many people have `workflows` at version 2?

e.g. the example here, excerpt below

```
version: 2.1

executors:
  my-executor:
    docker:
      - image: buildpack-deps:jessie
    working_directory: /tmp

jobs:
  flow:
    executor: my-executor
    steps:
      - ...

  downstream:
    executor: my-executor
    steps:
      - ...

workflows:
  version: 2

  btd:
    jobs:
      - flow
      - downstream:
          requires:
            - flow

```

I figure I should probably be cargo-culting this in until I know *why* people do it, but I dislike doing that without any understanding :)

Thanks!

# Answer
The Workflows `version` field was used to issue warnings for deprecation or breaking changes during v2 Beta, and it is deprecated as of CircleCI v2.1.

2 is the only valid number besides 1
> 1 votes

---
Tags: circleci
---


# Question
Title: Cross-Platform Deployment Tasks in Bamboo
Currently most of my deployments are using a dedicated agent and using Robocopy (windows) or SSH (linux/osx). A common scenario for my deployments are copying .dlls from a .net project to a server.

I am trying to consolidate capabilities to allow either the windows or linux/osx agents to deploy (using Bamboo).

Is there a simple shared protocol to copy files regardless of the OS or am I correct in assuming I need to setup a solution like OpenSSH on the Windows environment?

# Answer
There isn't a common protocol that exists on all the platforms for deployment. However, if you get an SSH server onto the windows platforms then you can use the same mechanisms in Bamboo to do the deployment, and potentially even use the same keys to secure it all.

Windows SSHD - There are a few alternatives (https://serverfault.com/questions/8411/what-is-a-good-ssh-server-to-use-on-windows). Freesshd you can get administrator level access - been successfully using it with thousands of machines as old as WinXP for years.

The Bamboo SCP task can be used to do the staging to the remote servers - once you have SSH remote access to all your targets.

> You can use the Bamboo SCP task to upload files from Bamboo directly to a remote server as part of a Bamboo job. The SCP task is able to copy multiple files and preserves the directory structure for the copied files.

And Bamboo SSH task can then do post deployment install steps, running scripts relevant to the platform.

> You can use the Bamboo SSH task to execute a SSH command on a remote computer as part of a Bamboo job.
> 1 votes

# Answer
In Jenkins one has to create a private key and add it to the master. If an agent is called and a build is started, then this key is used. This means that one does NOT have to create a key for every agent. I assume that this should be possible in Bamboo as well. At least that is what I personally expect from a CI/CD system.
> 0 votes

---
Tags: continuous-deployment, bamboo
---


# Question
Title: Should the master node's hardware be the same as the worker node's hardware for a Kubernetes setup?
I'm new Kubernetes and am setting up a raspberry pi cluster at home and I want to put Kubernetes on it. I have 4 new raspberry pi 4s with 8gb each and one old raspberry pi 2+. There are several hardware differences between the two models but I think technically I can make the old pi2 the master node to manage the pi4s. I'm wondering if this is a good idea. I don't really understand what the consequences of this would be. For instance if I was to expand my little cluster to 8 or 16 pi4s in the future, would my one pi2 be overloaded in managing the workers? I'm really trying to understand the consequences of putting lower grade hardware in control of higher grade hardware in the master/worker node relationship in Kubernetes.

There are three main goals that I have for this hardware. I want to recreate an entire development environment, so some VMs that would host a testing environment, a staging environment, a dev environment, and then a small production environment for hosting some toy website, and then I want to be able to host some services for myself in Kubernetes like a nas storage service, a local github repo, an externally facing plex media server, stuff like that. I think I can use k8s to containerize all of this, but how will having a pi2 as master limit me? Thanks in advance.

# Answer
> Should the master node's hardware be the same as the worker node's hardware for a Kubernetes setup?

The control plane (master nodes) and the data plane (worker nodes) are completely different workloads. You can use very different hardware or virtual machines if you want.

You can use some with x86, some other nodes with arm64 or even Windows nodes are supported as worker nodes. But you probably need groups for your nodes, since the application typically only run on a specific type of nodes and it is easier to understand e.g. metrics if the application run its instances on the same type of nodes.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Docker tags for multi-stage builds for series of Docker images out of one Dockerfile
Thanks to Docker multi-stage builds, we can compile some artefacts in one environment and put them for execution in other one.

As result, there are two images, where the final corresponding to the last stage and provided tag gets tagged as defined; there is no tag for intermediate (ones), but they remain there as well.

Question: is it possible to create a series of tagged images for all defined stages using multi stage build out of one Dockerfile?

# Answer
You can build and tag any stage of a multi-stage build. Just use the `--target` option to `docker build`. E.g.

```
docker build --target jdk -t myapp-jdk:v1 .
docker build --target jre -t myapp:v1 .

```

With the layer caching, the the earlier steps will be reused from the cache and not rerun from scratch each time, so there's little downside to running the build to different targets multiple times.

If you absolutely need to run this from a single build command, you can parse the output of the build, get the image id of the intermediate step, and tag that image id:

```
docker tag <some_image_id> myapp-jdk:v1

```
> 15 votes

# Answer
I stumbled upon a solution which I thought was quite elegant, and only requires you to build once:

Use LABEL feature in your multistage builds, and then after building, you can tag based on label.

Advantages:

* only pull together the build context once
* fast command to tag after the build has been done

The example was discussed in https://forums.docker.com/t/tag-intermediate-build-stages-multi-stage-build/34795

> Using the LABEL command in my dockerfile I added a label to the build stage I’m interested in, then filter for it later:
>
> ```
> FROM node as builder-stage
> LABEL builder=true
>
> FROM node as app-stage
> LABEL builder=false
>
> ```
>
> Now to filter, sort newest first, cut out the date, and take the top row:
>
> ```
> docker images --filter “label=builder=true” --format ‘{{.CreatedAt}}\t{{.ID}}’ | sort -nr | head -n 1 | cut -f2
>
> ```
>
> Note that it is important to remove the label in the next build stage.

This approach works really well for me
> 1 votes

---
Tags: automation
---


# Question
Title: Auto build and upload to AWS new docker image after commit in repo
I'm new at DevOps, and I looking for solution to my problem.

I have a private code repository where I have a some Flask apps in one of many folders. Example of structure:

```
- Project
-- Django Apps
--- django1_dir
--- django2_dir
--- django3_dir
-- Flask Apps
--- flaskApp_dir1 (dockerfile inside)
--- flaskApp_dir2 (dockerfile inside)
--- flaskApp_dir3 (dockerfile inside)

```

And I have a configured cluster on kubernetes on AWS instance. And a private docker repo. When I want to load new version of my flaskApp to cluster I need:

1. Someone modify flaskApp on repo
2. I need pull repo
3. Locally build a new image based on new dockerfile
4. Push app to repo
5. Manually push docker image with new APPVersion to AWS trough ecr
6. Then kubernetes change image to new version

My question is:

how can I automate this process? I have an script to push dockerimage to AWS through ecr. It is possible to create some kind of trigger that would work in such a way:

1. Someone do a commit
2. Script building a new docker image and push it to AWS automaticly

I read something about it, but as I said I'm totaly new in DevOps so any hint will be awesome. Now I have configured kubernetes, that aws instances and repo for docker images. I read about git hooks, flux and some other solutions but it's quite unclear for me so I looking for best practice for that problem. I wouldn't particularly want to install huge new tools either.

So maybe any hits or keywords I could get hooked on?

# Answer
> how can I automate this process? I have an script to push dockerimage to AWS through ecr.

You are on the right track. Yes, this should be an automated process from Git push or when someone merge a Pull Request.

An important aspect of this is that this "automated process" should be on a dedicated system - such that it is *repeatable* in a *consistent* way. Typically this is called a "Continuous Integration Server" or just "Build Server".

Some popular choices for this kind of process:

* Jenkins \- old but popular
* Tekton \- Kubernetes native - but still in development
* GitHub Actions \- probably most interesting if your Git repo is on GitHub
* Bitbucket Pipelines \- probably most interesting if your Git repo is Bitbucket
* AWS CodeBuild \- limited but may be interesting if you use AWS
* GitLab CI \- probably most interesting if your Git repo is on GitLab

What you need to do is to configure your Git repository system (e.g. GitHub, Bitbucket, Gitlab or AWS CodeCommit) to initiate Pipelines on the Build system (a product of those listed above)
> 2 votes

---
Tags: docker, kubernetes, automation
---


# Question
Title: What minimal IAM permissions are needed by AWS Chatbot so that it can show logs?
The AWS service called Chatbot allows CloudWatch Alarms to be relayed into a Slack channel. The message includes two buttons: **Show logs** and **Show error logs**.

However, when I press either of these buttons, I receive an error message like this:

> ❗ I can't get the logs for you because the role **arn:aws:iam::123412341234:role/my-chatbot** in account **123412341234** is missing necessary permissions related to commands.
>
> If you want to get logs, go to the AWS Chatbot console and choose or create a role that includes **Read-only command permission** for this channel.

However, the recommended way of accomplishing this (the way automatically set up by AWS Console) grants Chatbot the `ReadOnlyAccess` policy which is **way too permissive** for my tastes. I don't want Chatbot to be able to read any object from any of our S3 buckets!

What is the minimal IAM policy that enables Chatbot to read CloudWatch Logs for Lambda functions?

# Answer
> 2 votes
AWS Chatbot uses CloudWatch Logs Insights, so the IAM policy for Chatbot must include these two statements:

```
{
    "Sid": "QueryLogs",
    "Effect": "Allow",
    "Action": [
        "logs:StopQuery",
        "logs:StartQuery",
        "logs:DescribeQueries"
    ],
    "Resource": "arn:aws:logs:::log-group:/aws/lambda/*"
},
{
    "Sid": "GetQueryResults",
    "Effect": "Allow",
    "Action": "logs:GetQueryResults",
    "Resource": "*"
}

```

With these statements in place, pressing the logs button will yield an initial message from Chatbot like this:

> I ran this CloudWatch Logs Insights query to fetch logs from **2020-11-18T09:02 UTC** to **2020-11-18T09:07 UTC**.

... as well as several follow-up messages containing a plaintext dump of the logs.

# Answer
> 0 votes
I solved this issue with these IAM permissions,

```
{
    "Sid": "query",
    "Effect": "Allow",
    "Action": [
        "logs:StopQuery",
        "logs:StartQuery"
    ],
    "Resource": "arn:aws:logs:us-east-1:*:log-group:/aws/lambda/my-lambda-function-name*:*"
},
{
    "Sid": "logstream",
    "Effect": "Allow",
    "Action": [
        "logs:GetQueryResults",
        "logs:DescribeLogGroups"
    ],
    "Resource": "arn:aws:logs:us-east-1:*:log-group::log-stream:*"
}

```

(you need to change the region `us-east-1` and `my-lambda-function-name` to your lambda function name)

# Answer
> 0 votes
Just add the managed policy **CloudWatchReadOnlyAccess** as proposed in the documentation.

---
Tags: amazon-web-services, chatops
---


# Question
Title: Azure Release Pipeline Fails Using Expand-Archive Command
Sometime in the last week, our pipeline started using `Expand-Archive` to open the `.zip` file during the "IIS Web App Deployment" step instead of 7-Zip. It does not appear that this was done by our team, so I'm presuming this may have been changed in DevOps.

We are deploying to Server 2012R2.

Several of our apps are okay with this, however **one** of them fails to expand. It *seems* as if this might be related to path length (this app uses some very deep folder nesting with Angular) since I can *manually* run the command successfully if I shorten the destination path.

The command that fails to runs on the server is:

```
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NoProfile -NonInteractive -Command "Expand-Archive -Path "C:\azagent\A3\_work\r1\a\thisistheazurejobname\drop\thisisourappsnamex.zip" -DestinationPath "C:\azagent\A3\_work\_temp\temp_web_package_8157628602055582" -Force"

```

If I reduce it to:

```
C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe -NoLogo -NoProfile -NonInteractive -Command "Expand-Archive -Path "C:\azagent\A3\_work\r1\a\thisistheazurejobname\drop\thisisourappsnamex.zip" -DestinationPath "C:\azagent\A3\_work\_temp\short" -Force"

```

...it appears to work fine.

Obviously, I am speculating that it's a path-length issue, but evidence does seem to support it. Also, I reconstructed what *would have been* the 7-Zip command and it worked as expected:

```
C:\azagent\A3\_work\_tasks\IISWebAppDeploymentOnMachineGroup_1b467810-6725-4b6d-accd-886174c09bba\0.184.0\node_modules\webdeployment-common-v2\7zip\7z.exe x -oC:\azagent\A3\_work\_temp\temp_web_package_8157628602055582 C:\azagent\A3\_work\r1\a\thisistheazurejobname\drop\thisisourappsnamex.zip

```

I looked at the advice here and found we are already using the setting: https://igorpuhalo.wordpress.com/2019/08/29/overcoming-long-path-problem-in-powershell/

My hope is that someone knows what may have changed, can suggest a straightforward way to go *back* to using 7-Zip, or a way to make `Expand-Archive` to work.

# Answer
> 1 votes
This was answered via StackOverflow by Kevin Lu:

https://stackoverflow.com/a/66739852/1224614

In short, the second suggestion he made to update to PowerShell 7 and adding the environment information worked. I also had to run the script to reregister the Azure agent on the target server.

---
Tags: azure-devops, windows, azure-pipelines, release
---


# Question
Title: How to create a local network mapping in linux
I'm deploying nextcloud via docker, however the way the configuration is set up, it creates a subnetwork at 10.10.10.0/24 on the system it's running on instead of running it on 127.0.0.1. Then the only way to access it is by navigating to 10.10.0.5 on the host machine.

However, I want to be able to access this image from every machine on my local network.

I already have a few things running, so I need to map the port as well.

Is there an intuitive way to map the network such that navigating to 127.0.0.1:2287 maps it to 10.10.0.5? I can't seem to find anything indicating how to do this.

I would think this would have to be a bidirectional mapping since it needs to send data back since it's nextcloud, but I'm unsure about this.

I did try setting up the docker container to report to 127.0.0.1 but it didn't work out. It would only report on 172.19.0.5

If you wanted to see my docker setup, it's as follows:

1. Create the subnetwork for docker: `docker network create --driver=bridge --subnet=10.10.10.0/24 --gateway=10.10.10.1 dockernet`
2. Run docker-compose

```
version: '3'

services:
  nextclouddb:
    image: mariadb
    container_name: nextcloud-mariadb
    volumes:
      - /home/username/data/mysql:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    networks:
      default:
        ipv4_address: 10.10.10.4
    environment:
      - MYSQL_ROOT_PASSWORD=STONGPASSWORD
      - MYSQL_PASSWORD=ANOTHERSTONGPASSWORD
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    restart: unless-stopped

  nextcloudapp:
    image: nextcloud:latest
    container_name: nextcloud-app
    networks:
      default:
        ipv4_address: 10.10.10.5
    volumes:
      - /home/username/data/html:/var/www/html
      - /home/username/data/config:/var/www/html/config
      - /home/username/data/apps:/var/www/html/custom_apps
      - /home/username/data/data:/var/www/html/data
      - /home/username/data/themes:/var/www/html/themes
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped

networks:
  default:
    external:
      name: dockernet

```

# Answer
Do you have network constraints to create a new subnet or what is the reason? if it's not useful at all, you can just map the port and open the firewall of your PC so that the other machines can see the application.

```
version: '3'
services:
  nextclouddb:
    image: mariadb
    container_name: nextcloud-mariadb
    volumes:
      - /home/username/data/mysql:/var/lib/mysql
      - /etc/localtime:/etc/localtime:ro
    environment:
      - MYSQL_ROOT_PASSWORD=STONGPASSWORD
      - MYSQL_PASSWORD=ANOTHERSTONGPASSWORD
      - MYSQL_DATABASE=nextcloud
      - MYSQL_USER=nextcloud
    restart: unless-stopped

  nextcloudapp:
    image: nextcloud:latest
    container_name: nextcloud-app
    volumes:
      - /home/username/data/html:/var/www/html
      - /home/username/data/config:/var/www/html/config
      - /home/username/data/apps:/var/www/html/custom_apps
      - /home/username/data/data:/var/www/html/data
      - /home/username/data/themes:/var/www/html/themes
      - /etc/localtime:/etc/localtime:ro
    restart: unless-stopped
    ports:
      - 2287:80

```
> 1 votes

---
Tags: docker, docker-compose, containers, networking
---


# Question
Title: Directory structure - how do I upload it as-is to an artifactory?
I am new to Artifactory, and I wonder how to upload a directory structure containing many different binaries -- keeping the structure intact -- to an artifactory?

# Answer
This can be done through the Jrog CLI. The CLI has a recursive option that looks like it will do what you hope.

```
jfrog rt upload --recursive artifactory-mirror/* artifact-repo/dir-struct/

```

The CLI also has some performance improvements for uploads that doing it through curl or a browser don't seem to get. Also, scriptable!

To include full directory structure you will want to use the flag --flat=false

```
jfrog rt upload --flat=false --recursive artifactory-mirror/* artifact-repo

```
> 6 votes

# Answer
Since the original question is also tagged with Nexus this is how to do bulk upload with Nexus 3:

```
find <directory name>/ \! -type d -print0 | xargs -P10 -0 -I@ curl -v --user <user>:<password> --upload-file ./@ http://localhost:8081/repository/<repository name>/@

```
> 1 votes

# Answer
Adding on @sysadmin1138's answer, to push the content of a folder to another folder while keeping the directory structure starting from the source folder, you can use:

```
(cd source-folder && jfrog rt upload --detailed-summary --flat=false --recursive ./ artifact-repo/target-folder/

```

The `--detailed-summary` will tell you the details of all the files uploaded.
> 1 votes

---
Tags: maven, artifactory, nexus
---


# Question
Title: AWS SNS-SQS: Checking a subscription already exists
I've found it's a useful strategy to create relevant queues whenever a queue worker service boots. For instance, on service boot, it checks whether its queues exist already in AWS and creates them if they don't.

Something I'm struggling with now is how to ensure that an SNS subscription for the queue exists. Checking whether the queue exists and creating it if not is easy, subscribing a queue to a particular SNS topic on boot is also easy - but I can't seem to find a sensible way of checking whether a particular queue is already subscribed to a topic!

If it helps, this will be using the Ruby SDK - but I think that the clients across the languages are pretty much all the same. If I can find a sensible way with the cli client it will most likely be available in the ruby sdk.

The only thing I've come up with so far is to list all the subscriptions and then filter them, but this seems a really silly way of doing it as there could be tonnes of subscriptions - I'd rather not have to deal with paging responses on worker boot.

It would be great if there were a way to retrieve whether a subscription is active, based on parameters of the queue ARN and the topic ARN.

# Answer
In the end I decided to go another way.

I don't think it was actually a good idea to modify things in that way when the worker boots. It can cause problems because if you have to do some emergency modification of the configuration those changes will be reverted the next time your worker instances restart.

Instead, on boot the worker checks if the queue exists. If the queue *does not* exist, it will create the queue and all its corresponding configuration including SNS subscriptions etc. It deliberately will only create the SNS subscription for the queue if also creating the queue.

This way, if some configuration has to be changed as an emergency in production, those changes won't be "reset" if the worker spins up new instances.

It is a bit annoying, but also safer I think.

Obviously, this is entirely personal opinion and what I went with in the end - but it seems to have served me relatively well so far. If anyone else has any better ideas or disagrees please do let me know.
> 1 votes

---
Tags: amazon-web-services, aws-cli, amazon-sqs
---


title: How to get VPC id in Terraform module?

text:
# Question
Title: How to get VPC id in Terraform module?
I have a module structure like

```
module "vpc" {
source = "./modules/vpc"
}

module "prod_subnets" {
source = "./modules/vpc/modules/subnets/production"
}

```

when running am getting an error like

```
[0m on modules/vpc/modules/subnets/production/production.tf line

    162, in resource "aws_subnet" "prod-pub-1b":
    14:22:55 162: vpc_id = [4maws_vpc.production_vpc[0m.id
    14:22:55 [0m
    14:22:55 A managed resource "aws_vpc" "production_vpc" has not been declared in
    14:22:55 prod_subnets.

```

can anyone please help ?

# Answer
Without seeing the contents of your two modules I'm guessing a bit, but it looks like you have an AWS VPC declared in your `vpc` module and some subnets declared in your `prod_subnets` module and you are asking how the configuration of the subnets can get access to the VPC ID.

If so, the answer is that the `vpc` module must export the VPC ID as an output value and then the `prod_subnets` module must accept the VPC ID as an input variable.

In your `vpc` module, you can declare a `vpc_id` output value like this, for example in a file `modules/vpc/outputs.tf`:

```
output "vpc_id" {
  value = aws_vpc.production_vpc
}

```

In your `prod_subnets` module you can declare a `vpc_id` input variable, for example in a file `modules/vpc/modules/subnets/production/outputs.tf`:

```
variable "vpc_id" {
  type = string
}

```

Then in your existing `modules/vpc/modules/subnets/production/production.tf` file, on line 162, change the `vpc_id` argument for the subnet to refer to that variable:

```
  vpc_id = var.vpc_id

```

Finally, you must then edit the top-level file whose source code you shared in your question to pass the value between the two modules, like this:

```
module "vpc" {
  source = "./modules/vpc"
}

module "prod_subnets" {
  source = "./modules/vpc/modules/subnets/production"

  vpc_id = module.vpc.vpc_id
}

```

`module.vpc.vpc_id` means to take the value of the `vpc_id` output value from the `vpc` module.
> 3 votes

# Answer
It should be written as:

```
output "vpc_id" {
  value = aws_vpc.production_vpc.id
}

```
> 1 votes

---
Tags: terraform, aws-vpc
---


# Question
Title: Make a job easily searchable in Azure Runbook
I have a number of different jobs in my Azure Runbook. However, I don't have an easy way of differentiating between jobs when I look at the Overview of the Runbook. I am forced to click on each individual job to see which one it is. Is there a way I can "label" my job so that it is easily identifiable from the Overview screen?

Thanks!

# Answer
AFAIK currently there is no way to label job so that it is easily identifiable from the Overview screen. If interested, you may raise feature request in this UserVoice / feedback forum. I see this related feature request already raised so you may even upvote it. In general, Azure feature team would check feasibility of a feature request, triage it, prioritize against existing feature backlog, add in roadmap as appropriate and would announce and/or update the related Azure document once a feature request is addressed.
> 1 votes

---
Tags: azure-devops
---


# Question
Title: Jenkinsfile update using version control
I've a Jenkinsfile in the `master` branch of the repository with following code

```
pipeline {
    agent any
    stages {
        stage('A') {
            steps {
                sh 'echo "Step A"'
            }
        }
    }
}

```

Now I want to add a new stage B for which I created a git branch `update/jenkinsfile` and in that branch the Jenkinsfile is now

```
pipeline {
    agent any
    stages {
        stage('A') {
            steps {
                sh 'echo "Step A"'
            }
        }
        stage('B') {
            steps {
                sh 'echo "Step B"'
            }
        }
    }
}

```

How do I make Jenkins to use Jenkinsfile from branch B not from master in order to testing the change?

# Answer
It sounds like you want a `when` statement.

As I understand it, you want to execute stage B **when** you are on branch `update/jenkinsfile`. In order to do this, you need to add a condition to stage B:

```
pipeline {
    agent any
    stages {
        stage('A') {
            steps {
                sh 'echo "Step A"'
            }
        }
        stage('B') {
            when {
              branch 'update/jenkinsfile'
            }
            steps {
                sh 'echo "Step B"'
            }
        }
    }
}

```

Note that you also need your jenkins to discover this new branch - this will depend on how you have configured the job. The above assumes that you have a multibranch pipeline job which is configured to include that branch.
> 1 votes

---
Tags: jenkins, jenkins-pipeline, jenkinsfile
---


# Question
Title: Can I keep a container started with --rm option?
I started my docker container with `--rm` and `-it` options. I need to keep that container now because it has some important data. Can I cancel the `--rm` switch using the `docker update` command?

This answer from SO says it is possible, but I it doesn't say how exactly!

# Answer
No, docker will delete the container when it stops, and there's no update option to change that behavior. You can see this in the help output:

```
$ docker container update --help

Usage:  docker container update [OPTIONS] CONTAINER [CONTAINER...]

Update configuration of one or more containers

Options:
      --blkio-weight uint16        Block IO (relative weight), between 10 and 1000, or 0 to disable (default 0)
      --cpu-period int             Limit CPU CFS (Completely Fair Scheduler) period
      --cpu-quota int              Limit CPU CFS (Completely Fair Scheduler) quota
      --cpu-rt-period int          Limit the CPU real-time period in microseconds
      --cpu-rt-runtime int         Limit the CPU real-time runtime in microseconds
  -c, --cpu-shares int             CPU shares (relative weight)
      --cpus decimal               Number of CPUs
      --cpuset-cpus string         CPUs in which to allow execution (0-3, 0,1)
      --cpuset-mems string         MEMs in which to allow execution (0-3, 0,1)
      --kernel-memory bytes        Kernel memory limit
  -m, --memory bytes               Memory limit
      --memory-reservation bytes   Memory soft limit
      --memory-swap bytes          Swap limit equal to memory plus swap: '-1' to enable unlimited swap
      --pids-limit int             Tune container pids limit (set -1 for unlimited)
      --restart string             Restart policy to apply when a container exits

```

Trying to change the restart policy will result in an error:

```
$ docker run --rm --name test-rm -d busybox tail -f /dev/null
14a7d9bfebc3356652e3c3f060e92d9cebb9f3eb4490873540c9eb1c272f6612

$ docker update --restart=unless-stopped test-rm
Error response from daemon: Cannot update container 14a7d9bfebc3356652e3c3f060e92d9cebb9f3eb4490873540c9eb1c272f6612: Restart policy cannot be updated because AutoRemove is enabled for the container

```

You can try to copy your data out of the container with:

```
docker container cp $container_name:/path/in/container /path/on/host

```

If you don't know which files to save, list which files have changed in the container with:

```
docker container diff $container_name

```
> 3 votes

# Answer
Even though this is a year old...

If the containers important data isn't in a volume mount already, you could always try copying the important content out of the still running container and mount it into your more permanent container.

```
docker cp <container>:/path/in/container /path/on/host
docker run -it -v /path/on/host:/path/in/container ....

```
> 3 votes

---
Tags: docker
---


# Question
Title: AWS ElastiCache Redis DNS error - Name or service not known
## The Problem & The Question

I'm trying to create an AWS ElastiCache Redis-flavored cluster and connect to it from an instance in the same VPC. When I create the cluster initially, I can connect to it just fine via `redis-cli`. If I wait a day or two and then try to connect to it again via `redis-cli`, I get the following error:

```
$ redis-cli -h <aws-elasticache-cluster-primary-endpoint>
Could not connect to Redis at <aws-elasticache-cluster-primary-endpoint>:6379: Name or service not known
not connected>

```

I should be able to connect again with no issue. Why would I be getting DNS errors the next day when it worked initially? Nothing has changed about the cluster since I created it. How could I fix this?

## Steps to Reproduce

1. ElastiCache Dashboard -\> Redis -\> Create.
2. Set the following options:

| Option | Value |
| --- | --- |
| Cluster Engine | Redis |
| Location | Amazon Cloud |
| Engine version compatibility | 6.x |
| Port | 6379 |
| Parameter group | default.redis6.x |
| Node type | cache.t3.micro |
| Number of replicas | 0 |
| Multi-AZ | false |
| Subnet group | default |
| Security groups | default |
| Encryption at-rest | false |
| Encryption in-transit | false |
3. Create.
4. After creating, verify you can connect to the cluster with:

`redis-cli -h <aws-elasticache-cluster-primary-endpoint>`
5. Disconnect.
6. Try reconnecting with the same `redis-cli` command a few days later. You should see the errors shown in the section above.

## Additional Information

### `nping`

```
$ sudo nping --tcp -p 6379 <aws-elasticache-cluster-primary-endpoint>
Failed to resolve given hostname/IP: <aws-elasticache-cluster-primary-endpoint>.  Note that you can't use '/mask' AND '1-4,7,100-' style IP ranges
Cannot find a valid target. Please make sure the specified hosts are either IP addresses in standard notation or hostnames that can be resolved with DNS

```

### `nslookup`

```
$ sudo nslookup <aws-elasticache-cluster-primary-endpoint>
Server:         75.75.75.75
Address:        75.75.75.75#53

** server can't find <aws-elasticache-cluster-primary-endpoint>: REFUSED

```

### Reachability Analyzer

I followed AWS' instructions for testing connectivity from my EC2 instance to the ElastiCache cluster. The results I got back for the connectivity were:

| Reachability Status | State |
| --- | --- |
| Reachable | Succeeded |

### Cluster Metrics

| Metric | Value |
| --- | --- |
| CPU Utilization | ~1.000% |
| Engine CPU Utilization | ~0.283% |
| Database Memory Usage Percentage | ~1.093% |

### Local Wireguard configuration

```
[Interface]
PrivateKey = <value>
ListenPort = 21841
Address = 10.0.0.2/32
DNS = 9.9.9.9 # 75.75.75.75 75.75.76.76

[Peer]
PublicKey = <value>
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = <public-IP>:51820

```

# Answer
It could be due to a "VPC DNS throttling", you can find more information on the official documentation.

https://aws.amazon.com/premiumsupport/knowledge-center/vpc-find-cause-of-failed-dns-queries/
> 1 votes

---
Tags: amazon-web-services, vpn, redis
---


# Question
Title: Having trouble with building Docker image
I'm trying to build a docker image for a software using `docker build -t cytools .`

Here is the output that I'm confused about:

```
[+] Building 0.0s (1/2)
 => [internal] load build definition from Dockerfile                       0.0s
 => => transferring dockerfile: 2B                                         0.0s
failed to solve with frontend dockerfile.v0: failed to read dockerfile: open /var/lib/docker/tmp/buildkit-mount346931491/Dockerfile: no such file or directory

```

Does anyone know what this means? I am new to all this, so sorry if I didn't use the correct terminology or give the full picture!

# Answer
> 2 votes
Looks like the Dockerfile is not in the path, '.' means the current directory you are in.

---
Tags: docker, dockerfile
---


# Question
Title: Azure Devops Pipeline variables in Powershell
I've tried to figure this out and I can't see why my ipaddress is not getting set.

I've checked code samples and they all suggest I am doing everything correct.

`ip` variable defined here:

This is my powershell script:

```
# Write your PowerShell commands here.

Write-Host "AGENTIPADDRESS variable is set to ($env:AGENTIPADDRESS)"

$ipaddress = Invoke-RestMethod http://ipinfo.io/json | Select -exp ip
Write-Host "ipaddress variable is set to ($ipaddress)"
Write-Host "ip pipeline variable to ($ip)"

Write-Host "##vso[task.setvariable variable=IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=env:IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=$env:IP;]$ipaddress"
echo "##vso[task.setvariable variable=IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=IP;]333"
Write-Host "##vso[task.setvariable variable=IP;]$ipaddress"
echo "##vso[task.setvariable variable=IP]999"
echo "##vso[task.setvariable variable=IP;]888"

Write-Host "##vso[task.setvariable variable=IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=IP;]$ipaddress"
Write-Host "##vso[task.setvariable variable=IP]114"
Write-Host "##vso[task.setvariable variable=IP;]123"

Write-Host "ip environment variable to ($ip)"
Write-Host "env:IP environment variable to ($env:IP)"

Write-Host "Set environment variable to ($env:AGENTIPADDRESS)"

```

And here is my output:

Why can't I set & read the `ip` variable?

# Answer
The issue is that you can't reference the changed value in the same task.

I created a new Powershell task and just called `Write-Host "env:IP environment variable set to ($env:IP)"` which output the IP address correctly.
> 2 votes

# Answer
Pipeline Variables are handed to PowerShell as environment variables, which means that `$ip` exists only with the script while you want to use `$env:ip` to pick the environment variable value.

**Note** Secret variables are **not** passed automatically: you must add an explicit `env:` in YAML or as arguments in Classic Editor. The latter means that the PowerShell code must have a `param` declaration (e.g. `param($ip)`)and the Task use *Arguments* field (e.g. `-ip $(ip)`).
> 3 votes

---
Tags: azure-devops, azure-pipelines, environment-variables, powershell
---


# Question
Title: Ansible - Variable expansion inside hostvars declaration?
I have a hardcoded hostvars line, like so:

`node1_hostname={{ hostvars['192.168.0.162']['node1_hostname'] }}`

I have a `set_fact` variable from a previous play that I'd like to expand in the host section of the hostvars declaration, instead of the IP address. Can hostvars take expansion inside those brackets and single quotes?

Thanks.

# Answer
No, you cannot nest jinja2 expansion i.e. the following **won't work**:

```
# Warning ! Those will fail !
some_var: "{{ some_dict[{{ some_dynamic_key }}] }}"
other_var: "{{ {{ dynamicaly_prefixed }}_var }}"
# Did I forget to say the above examples will fail ?

```

But you can use your var without any problem, just as in almost any other programming language dealing with list and indexes. There is actually an ansible FAQ entry on this particular subject

```
some_inventory_hostname: 192.168.0.162
node1_hostname: "{{ hostvars[some_inventory_hostname].node1_hostname }}"

```
> 3 votes

---
Tags: ansible, linux, configuration-management, yaml, variables
---


# Question
Title: Problem with Ansible Asynchronous actions
I'm trying to run two simple tasks

`Task A` starts an Ipref3 server and waits for a client to connect
`Task B` starts to run from *host-1* until the given timeout

once `Task B (host-1)` is over another `Task B` run from host-2

`Task B` starts to run from *host-2* until the given timeout

What is a good approach to orchestrating this use-case?

**This is what I tried**

`Task B` runs the client from multiple hosts (host-1, host-2, ..)

```
- name: Run iperf3 client
  win_command: iperf3 -c {{ iperf_server }} -t {{ iperf_time }}
  args:
    chdir: "{{ iperf_dest }}"
  register: iperf3_out

- debug:
    var: iperf3_out.stdout

```

`Task A` run the server

```
- name: Start iperf3 server
  shell: iperf3 --server
  async: "{{ iperf_async }}" # iperf_async = iperf_time + 10
  poll: "{{ iperf_poll }}"   # iperf_poll  = 0

```

Is there a way to know when to confirm `Task A` is finished before I move to the next task.

NOTE: Order of execution is important Server (`Task A`) needs to run before the client (`Task B`)

# Answer
> 1 votes
> Order of execution is important Server (Task A) needs to run before the client (Task B)

Then why are your running Task A async in that case ? Is it just because `iperf --server` is not falling in the background and dying as soon as the shell task finishes ? A quick look at the documentation shows we can probably work around that problem with something like:

```
iperf3 --server --daemon --pidfile /path/to/iperf3.pid --logfile /path/to/log

```

Notes:

* I added the `logfile` option since I guess it can be useful but it is not mandatory for the current issue
* I took for granted you are running the server on a linux platform. Running it on Windows would apparently slightly change the below scenario. See Running as a service on windows

From what I understand the last unsolved issue is running the client tasks sequentially on each client host. This can be done using the `serial` keyword on your play.

Here is an untested pseudo ansible setup summarizing what I understand from your requirements. Adapt to your real use-case.

**`inventory/iperf_hosts.yml`**

```
---
iperf_servers:
  vars:
    iperf_pid_file: /tmp/iperf.pid
    iperf_log_file: /tmp/iperf.log
  hosts:
    server1.local:
iperf_clients:
  vars:
    iperf_time: 1
    iperf_dest: some\win\directory
  hosts:
    host1.local:
    host2.local:
    host3.local:

```

**`playbook.yml`**

```
---
- name: Start the iperf server
  hosts: iperf_servers

  tasks:
    - name: Fire the start command
      command: "iperf3 --server --daemon --pidfile {{ iperf_pid_file }} --logfile {{ iperf_log_file }}"

- name: Launch clients one after the other.
  hosts: iperf_clients
  serial: 1

  tasks:
    - name: Run iperf3 client
      win_command: "iperf3 -c {{ hostvars[group['iperf_servers'][0]].inventory_hostname }} -t {{ iperf_time }}"
      args:
        chdir: "{{ iperf_dest }}"
      register: iperf3_out

    - debug:
        var: iperf3_out.stdout

- name: Now we are over with clients, kill the server
  hosts: iperf_servers
  # We already gathered facts at first play so we disable here
  # You might want to disable in all other plays if you don't need facts at all
  gather_facts: false

  tasks:
    - name: Slurp pid from file
      slurp:
        path: "{{ iperf_pid_file }}"
      register: slurped_pid

    - name: kill the iperf3 process
      command: "kill {{ slurped_pid.content | b64decode }}"

```

---
Tags: ansible, automation
---


# Question
Title: if the bottle neck is the number of connections, is it better to scale redis by sharding or by replication?
# Background

last week we had a major outage, we identified the bottle neck to be redis number of connections:

Our quick fix was to scale redis from cache.m4.4xlarge to chache.m4.10xlarge (vertical scaling)

However this scaling is quite expensive:

we would like to scale horizontally:  Unfortunately, the application code we are using now only allows us to either create a redis cluster with multiple shards, or using a single shard with multiple replicas (see discussion here).

# Question

considering that our bottle neck is the connection limit, would it be getter to scale by clustering and increasing the shard number, or by having a single shard and replicating it?

# Answer
Redis Cluster could be considered as an OOTB solution for the Scalability requirements. There is another interesting open source library known as Redis Shard for sharding implementations on Redis. Please find the GitHub repository here.
> 1 votes

# Answer
Just for the sake of completeness: Redis documentation https://redislabs.com/ebook/part-3-next-steps/chapter-10-scaling-redis/ suggest a general rule of thumb - add replicas for reads, add shards for writes and memory capacity. And it does make sense.

The replica can have a complete copy of its shard. The more shards the smaller they are. Also writing to one shard does not limit other ones.

If the only concern is reading, then replicas are the simplest solution.

If you need both, just combine.
> 0 votes

---
Tags: redis, scalability, cache
---


# Question
Title: Redirect to a custom page on 503 error in Ingress
When our site is getting deployed, the Ingress gets created and before I could get any response I get 503 for 5-6 minutes (time taken to startup).

We need to configure our ingress to redirect to a static page that is custom-made instead of the 503 error. Can I add an annotation to achieve this? If yes, please give me an example for redirect config using the status code.

# Answer
> 2 votes
There is no built-in configuration/annotation to handle this.

Although, you can use the server-snippet annotation to create a custom configuration that intercepts the error 503, proxying the request to a service that is serving your custom error page. Example:

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/server-snippet: |
      proxy_intercept_errors on;
      error_page 503 = @errorpages;

      location @errorpages {
        proxy_set_header X-Code $status;
        proxy_pass http://errors-svc.default.svc.cluster.local;
      }
spec:
  rules:
  - host: ameba.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: nginx
            port:
              number: 80

```

---
Tags: amazon-web-services, kubernetes, load-balance, ingress
---


# Question
Title: Advantages of using Code-pipeline and Codebuild for CI/CD over Bitbucket pipeline?
What are the advantages of using AWS Codepipeline/code build for our pipeline over the bitbucket pipeline. ?

Some of the factors that I can think of:

1. Pricing?
2. Pipeline for a serverless application
3. integration with AWS
4. services
5. Speed of execution

# Answer
> 1 votes
A bit too long for a comment, so converting to an answer:

That's super broad and highly depends on your final needs IMHO, if you're full AWS and ok with being vendor locked down, then code commit/code build/code pipeline is great as it's nicely integrated now.

If you have needs outside pure AWS services, that may get harder, for exemple if you plan on stretching on multiple cloud vendors in a always on DRP (Disaster Recovery Plan), that's probably not the better tool.

My take is that for a full AWS things, AWS products are good and *usually* soften the edges, but if you need to interact with things out of AWS, there's solution more aimed at CI/CD like Bitbucket, as you said but that'd be my last choice, CircleCi, Travis, Github Actions and Gitlab.

The CI/CD cost is rarely a concern, even if you have a super large company with dozens of products, the code repository and user management costs will probably out-stand the pipeline costs by a large factor unless you run thousands of pipelines per day.

My *personal* preference goes to Gitlab for its versatility in pipelines, how easy internal runners are to setup and to avoid vendor lock down on the hosting.

---
Tags: amazon-web-services, continuous-integration, continuous-deployment, bitbucket-pipelines, aws-codepipeline
---


# Question
Title: Container Image Promotion across environments in ECR/ECS
We have started doing some initial work on deploying containers on ECS Fargate service.

However we are not yet completely figured out how to manage our image promotion across different environments.

In our case, developers managed their own environment so they can deploy the docker image specifying just the tag of something they have built.

We only have UAT and PROD environments , As of now whatever we have in `develop` branch got baked in as image and pushed to ECR as `develop` tag and that gets deployed to UAT.

we release the code from develop to master as part of our release process.

During the release process ( from develop to master merge ) we create another container image this time on merge to master and that gets pushed to ECR as `master` tag which gets deployed to PROD.

I think this process is not ideal and it can be improved by a lot margin.

1. We shouldn't need to create image on each stage and should promote the same image tag in across environments ( i.e whatever we have in UAT will get QA/TESTED and released to PROD as artifact ), how are you managing this image promotion across environment ? what are other release process look like ?
2. We are currently tagging our image as branch tag i.e develop or master , How are you managing these tags creation ? Are you using GIT SHA ? or Sem ver ?

Any other improvements or general comments can be really helpful as well.

# Answer
> 1. We shouldn't need to create image on each stage and should promote the same image tag in across environments ( i.e whatever we have in UAT will get QA/TESTED and released to PROD as artifact ), how are you managing this image promotion across environment ? what are other release process look like ?

Yes, rebuilding images is considered an anti-pattern in containerized world. While Bruce's solution with re-tagging in a sibling answer is neat, it doesn't solve rollbacks.

Recommended approach for production is to pin your images to their sha256 digests. So image definition in task would be something like *myregistry/myimage@sha256:shadigestgoeshere*. This ensures maximum degree of control what image gets deployed.

In some cases people update those tags manually, which is pretty tedious and brittle. However, there exists specialized tooling which keeps track of versions and routes what images should go there (dislaimer: I'm working on a tool of this sort - Reliza Hub).

> 2. We are currently tagging our image as branch tag i.e develop or master , How are you managing these tags creation ? Are you using GIT SHA ? or Sem ver ?

It is common to use version, git sha and branch as image tags. It is fine to have same image tagged with all of them, so in example you may have all of the following for same image:

* myregistry/myimage:mybranch
* myregistry/myimage:0.1.0
* myregistry/myimage:mygitsha

This can be achieved very easily by just adding multiple -t flags on docker build command. Again, remember that images under same tag can mutate (unless you use some special constraints on docker registry side which is rare) - so the only good way to ensure exact image getting deployed is using sha256 digests in production manifests.

> Any other improvements or general comments can be really helpful as well.

It is recommended to label images with version, branch, git commit, some ci metadata. This way when you do image inspect, you know right away what it is. See my toy project here for an example how to do labelling via build args - https://github.com/taleodor/mafia-vue .
> 1 votes

# Answer
Although difficult to give a canonical answer, I think I understand your frustration. My opinion is that you are focussing on promoting the wrong thing - the image.

I would start with the concept of "Dev-Prod parity" (or in your case, UAT/Prod parity). This concept says:

> Keep development, staging, and production as similar as possible

The simplest way to do this is to use the *same* image in the two environments. The difference between them comes from the configuration and backing services. These are set at run time, not at build time - *ie* they are changed when you start the container not when you build it.

So, your process should look like:

1. Build the image, tag it with `uat`
2. UAT environment discovers new image, new image gets deployed
3. Run UAT against new deploy, they pass
4. **tag that same image with `latest`**
5. Prod discovers new image, gets deployed.

This process is extremely lightweight and you will have certainty in what is deployed.

TL;DR instead of promoting images and having to rebuild them, just shift the tag used in prod.
> 0 votes

---
Tags: amazon-web-services, continuous-integration, containers, continuous-deployment, amazon-ecs
---


# Question
Title: How do I include my internal CA certificate to validate SSL exchanges in Chef?
We do use an internal Certificate Authority to create server certificates in my company.

We also have to deal with a transparent proxy doing SSL interception (MITM).

I regularly encounter SSL validation errors due to Chef not knowing the CA certificate, and sometimes it's the tooling around chef itself (berkshelf, knife; even chef client itself when talking to the server for the first time since chef 12 enables SSL by default).

Question is: How do I make Chef aware of my CA certificate to get valid SSL exchanges ?

# Answer
There's a couple of way to achieve the result:

1. Chef has a `trusted_dir` to allow adding certificate to the trusted list. the documentation has a lot of details about it. Adding your CA certificate to this directory would solve the problem. `knife` has it also in a slightly different path as per it's own documentation
2. Chef use its own CA certiticate list in `/opt/chef/embedded/ssl/certs/cacert.pem`. You can add your CA certificate at end of this list to trust it.

The second option has an advantage of allowing you to export the environment variable `SSL_CERT_FILE` pointing to chef `cacert.pem` to allow most of the tools using `openssl` library to know your CA certificate.

For the case of a self signed certificate on the chef server (or another server used as target in a recipe), `knife ssl_fetch` would allow all knife commands to work.

To add the server certificate to the cacert.pem for the case 2. above, you can use the following command:

```
# For a self signed CA certiticate
openssl s_client -showcerts -connect <YOUR_CHEF_SERVER>:443 </dev/null 2>/dev/null|openssl x509 -outform PEM >> /opt/chefdk/embedded/ssl/certs/cacert.pem

# For an internal CA signed certificate:
openssl s_client -showcerts -verify 5 -connect <YOUR_CHEF_SERVER>:443 </dev/null 2>/dev/null | awk '/BEGIN/,/END/{if(/BEGIN/){a++}; certs[a]=(certs[a] "\n" $0)}; END {print certs[a]}' >> /opt/chefdk/embedded/ssl/certs/cacert.pem

export SSL_CERT_FILE=/opt/chefdk/embedded/ssl/certs/cacert.pem

```

The openssl command is included in chef-dk, so this can be done under windows also. Change the path to `c:\opscode\` instead of `/opt/`. To export the environment variable use `set SSL_CERT_FILE=...` (with `/P` to add it permanently to your environment) in your command.
> 15 votes

# Answer
In my case, I was doing basically the same thing, because the internal certificate path was different than the typical certificate path, so I had to add my internal certificates. In my case, adding the SSL\_CERT\_FILE didn't work. I needed to copy my certificates, which I had exported as \*.cer and then made \*.pem versions using

. ./openssl x509 -inform der -in C:\dev\apps\certificates\org\_root.cer -out C:\dev\apps\certificates\org\_root.pem

I found the openssl tool here:

C:\opscode\chefdk\embedded\bin

Then I copied them to this location (which the location in the documentation above doesn't clarify very well) on the target machine:

C:\Users\\\[username\]\\.chef\trusted\_certs

I had to create the "trusted\_certs" folder on my target machine where I wanted to run the chef client. After copying the certificates there, it began working and trusting the internal certificate chain without SSL errors (in my case self-signed certificate error).

Also, on the ChefDK machine if you're using kitchen or something like that, I had to take the contents of my internal certificates' \*.pem files and copy the contents of the content to the tail of the "C:\opscode\chefdk\embedded\ssl\certs\cacert.pem" file or the cacert.pem file referenced by the SSL\_CERT\_FILE environment variable. In my case, I was copying an internal intermediate cert as well as a internal root cert and I believe they recommend that order also (ending with the root in the chain). This requirement on the ChefDK machine is due to the ChefDK communicating with a client machine using "remote\_file" and the solution I'm providing appears to relate to this Chef bug, which may be fixed in newer versions of Chef.

https://github.com/chef/chef/issues/5944
> 0 votes

---
Tags: chef, ssl
---


# Question
Title: mysqld: [ERROR] Found option without preceding group in config file /root/.my.cnf at line 1!
I'm trying to create a `.my.cnf` file for my MySQL docker-container that would allow me to prefill the user and password for the MySQL client. To achieve this I created the following dockerfile:

```
FROM mysql:5.7
RUN touch ~/.my.cnf && echo -e "[client]\nuser=root\npassword=$(echo $MYSQL_ROOT_PASSWORD)" >> ~/.my.cnf

```

But when starting the container I always get the errors:

```
mysql5.7-database    | 2021-04-01 18:38:51+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 5.7.33-1debian10 started.
mysql5.7-database    | 2021-04-01 18:38:51+00:00 [ERROR] [Entrypoint]: mysqld failed while attempting to check config
mysql5.7-database    |  command was: mysqld --verbose --help --log-bin-index=/tmp/tmp.5imGsNRXXC
mysql5.7-database    |  mysqld: [ERROR] Found option without preceding group in config file /root/.my.cnf at line 1!
mysql5.7-database    | mysqld: [ERROR] Fatal error in defaults handling. Program aborted!

```
* The environment variable `$MYSQL_ROOT_PASSWORD` is successfully set with the correct password.
* I can login with the password in the `$MYSQL_ROOT_PASSWORD`

The result of `echo -e "[client]\nuser=root\npassword=$(echo $MYSQL_ROOT_PASSWORD)"` is:

```
[client]
user=root
password=<my-generated-password>

```

When I create the `.my.cnf` file manually in the running docker container, it works like a charm. Any advise?

# Answer
> 1 votes
The default Dockerfile shell is `sh`. I don't think echo there has the `-e` modifier for backslash. Newline & other special symbols will work by default. In your command, resulting `.my.cnf` would contain a litteral `-e`.

Remove `-e` **or** use the original command in Dockerfile in a different shell, for instance:

```
RUN /bin/bash -c '<your-command>'

```

---
Tags: docker, mysql
---


# Question
Title: Terrafrom to declare both infra and application deployment in one state
Are there any advantages of maintaining both infrastructure (Infra-as-code) and application/service (helm chart deployment using helm\_release provider of terraform ) under one state file?

I don't like the idea of maintaining both infra and application under one state, because for some reason it could be possible that our intention was just to change application however as there is one state and it could detect some other infra related change, and it will deploy them, which could have unintended consequences on the infrastructure.

however I would like to be open and see suggestion, if there are any advantages which maintaining one state file provides.

# Answer
> 2 votes
I tend to prefer infra code living separately from application deployment code. Multiple reasons for this:

* You want your infra team to make infra changes. You want your app team to make application changes (including application configuration). Putting them separately lets you specify separate guard rails on them (for example, only the infra team is allowed to push changes to your infra repo).
* You don't necessarily want to trigger a redeployment of the app if you make infra changes, and you rarely want to trigger a redeploy of your infrastructure if you trigger a new app build.
* You don't want a bad build of your app preventing you from making any infrastructure changes until it's fixed, and vice-versa.

It actually makes sense to put your app deployment code (i.e. Helm chart) inside the app repo itself.

Then, you can actually package up application config changes along with the corresponding release of your application. You also empower developers to do it themselves, without running the risk of breaking production more than they already can.

It will also allow your CI/CD tool to deploy both at once.

And since there's a strict mapping of app IaC and app itself in each GitHub branch, this actually lets you very easily create feature branch environments (i.e. a developer creates a new branch, your CI/CD picks it up, and automatically deploys it to a kube environment).

---
Tags: terraform, helm
---


# Question
Title: Fluentd unable to read logs from all namespaces
I have a cluster installed on Openshift and used the Cluster Logging Operator and Elasticsearch Operator to enable logging on the cluster.

On Kibana, I am only able to see the logs from certain namespaces and not all the namespaces I have. How do I configure Fluentd to read logs from all pods in all namespaces?

I do not have a Fluentd service account (It was not created with the installation of the operator).

Also, the namespaces from which I don't see any logs have a node selector mentioned in them. Could that be a reason why the logs are not available?

# Answer
Million dollar question - are you running your fluentD as a DaemonSet? If not, there's a chance that fluentd simply isn't running on the node where some containers live.

Otherwise, my best guess is, it's missing permissions to read from all namespaces, especially since you mentioned you haven't attached a ServiceAccount to it. If you used a public FluentD Helm chart such as this one, try making sure you're deploying it with rbac enabled, i.e.:

```
rbac:
  create: true

serviceAccount:
  create: true

```

It'll also help if you post some logs from your fluentd container.
> 2 votes

---
Tags: openshift, kibana, fluentd
---


# Question
Title: Untracable glassfish docker container image or clean installation?
For a customer I need to create a glassfish container, containing the glassfish 5.0 release.

I have been looking for standard glassfish images on docker hub, and other places, but can't manage to find a recent image containing a recent glassfish release. For some reason, I also can't find, glassfish isn't available on dockerhub anymore. Only thing I could find is a reference to glassfish server on dockerhub, but that's also gone for some unclear reason.

Now am I correct to just use a clean docker image and create a new one where I install glassfish myself?

# Answer
> 1 votes
Look into Payara. It's a Glassfish fork/drop-in replacement that's better in most ways, including support.

There is a Payara 5 docker image available right on docker hub:

https://hub.docker.com/r/payara/server-full

---
Tags: docker, docker-registry, dockerhub
---


title: Google K8s Mirror offline?

text:
# Question
Title: Google K8s Mirror offline?
Heys dudes,

More someone got error when trying update cache of the Google K8s repositories? I have been got the error about **NO\_PUBKEY**

```
Err:10 https://packages.cloud.google.com/apt kubernetes-xenial InRelease
  The following signatures were invalid: EXPKEYSIG 6A030B21BA07F4FB Google Cloud Packages Automatic Signing Key <gc-team@google.com> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B57C5C2836F4BEB

```

and

```
W: An error occurred during the signature verification. The repository is not updated and the previous index files will be used. GPG error: https://packages.cloud.google.com/apt kubernetes-xenial InRelease: The following signatures were invalid: EXPKEYSIG 6A030B21BA07F4FB Google Cloud Packages Automatic Signing Key <gc-team@google.com> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B57C5C2836F4BEB
W: Failed to fetch http://apt.kubernetes.io/dists/kubernetes-xenial/InRelease  The following signatures were invalid: EXPKEYSIG 6A030B21BA07F4FB Google Cloud Packages Automatic Signing Key <gc-team@google.com> The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 8B57C5C2836F4BEB
W: Some index files failed to download. They have been ignored, or old ones used instead.

```

Something was changed?

# Answer
As per compute/docs/troubleshooting/known-issues:

> On Debian and Ubuntu based systems, including your local workstations, you might encounter an error similar to the following example: \[...\] The following signatures were invalid: XXXX \[...\]

To resolve this error, get the latest valid `apt-key.gpg` key file from `https://packages.cloud.google.com` by running the following command:

```
curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -

```
> 1 votes

---
Tags: kubernetes
---


title: How to create pipeline with jenkins?

text:
# Question
Title: How to create pipeline with jenkins?
I followed this tutorial and when i get to 29:32 the option "Build Pipeline View" is not displayed in the list i only get list view and my view.What can be the cause of this ? ( i followed every step carefully)

# Answer
This is a plugin that you can install.

Plugin Name: Build Pipeline

Plugin Web page: https://plugins.jenkins.io/build-pipeline-plugin/
> 1 votes

---
Tags: jenkins, continuous-integration, cicd
---


# Question
Title: Jenkinsfile credential as paramter unable to use it
I am trying to use the credentials parameter for a git clone. But i am getting the error that the variables are not found

Param definition

```
  credentials (credentialType: 'Username with password', defaultValue: 'fcb2d7c3-4b35-4ef2-bdf0-24fc4ff1137c', description: 'Git credential', name: 'git_credential', required: true)

```

Usage in stage

```
withCredentials([usernamePassword(credentialsId: params.git_credential, passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
sh 'git reset --hard; git clone https://${DOCKER_USERNAME}:${DOCKER_PASSWORD}@repo_url/scm/${params.repository}.git --branch master'

```

> Error: Wrong variable used

# Answer
withCredentials publish environment variables, and in sh to access the environment variables, you have to do it like this: "${env.JOB\_BASE\_NAME}".

Try this:

```
withCredentials([usernamePassword(credentialsId: params.git_credential, passwordVariable: 'DOCKER_PASSWORD', usernameVariable: 'DOCKER_USERNAME')]) {
   sh 'git reset --hard; git clone https://${env.DOCKER_USERNAME}:${env.DOCKER_PASSWORD}@repo_url/scm/${params.repository}.git --branch master'
}

```
> 1 votes

---
Tags: jenkins, jenkins-pipeline, jenkinsfile, groovy, credentials
---


# Question
Title: Build image from inside a pod or container
It seems pretty easy to build image when I just mount the docker.sock in a container. It's same as what I do on hosts machine. Simply `docker build . -t image/name:sometag`.

But when I need to do the same from a pod/container in K3s installation, I don't find a way of doing it. Can someone please guide me to something? It has to happen without mounting docker.sock.

# Answer
> 1 votes
If you simply need to build a docker image, I recommend using Kaniko. It allows you to build a docker image without having to expose the docker.sock or running your pods in privileged mode, which introduces potential security vulnerabilities. Per the Kaniko documentation, your spec file would look similar to:

```
apiVersion: v1
kind: Pod
metadata:
  name: kaniko
spec:
  containers:
  - name: kaniko
    image: gcr.io/kaniko-project/executor:latest
    args:
    - "--dockerfile=<path to Dockerfile within the build context>"
  restartPolicy: Never

```

# Answer
> 0 votes
You can deploy docker on same host where you have k3s and then use same technique with docker sock via mounting volume. I.e. this is part of the spec we did for Jenkins on k3s:

```
apiVersion: v1
kind: Pod
metadata:
  labels:
    run: jenkins-slave
spec:
  containers:
  - name: dind
    image: docker:19.03.12-dind
    command:
    - cat
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: dockersock
    tty: true
  volumes:
  - name: dockersock
    hostPath:
      path: /var/run/docker.sock

```

---
Tags: kubernetes, builds, images
---


# Question
Title: Jenkins return "Error validating repository information. Credentials ok." when validating repo url using Github App authentication
This is the error shown in UI

```
Error validating repository information. Credentials ok.
java.io.FileNotFoundException: https://api.github.com/repos/xxxxx/xxxxxxxx
    at org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$OkHttpURLConnection.getInputStream(ObsoleteUrlFactory.java:488)
    at org.kohsuke.github.extras.okhttp3.ObsoleteUrlFactory$DelegatingHttpsURLConnection.getInputStream(ObsoleteUrlFactory.java:1201)
    at org.kohsuke.github.GitHubHttpUrlConnectionClient$HttpURLConnectionResponseInfo.bodyStream(GitHubHttpUrlConnectionClient.java:197)
    at org.kohsuke.github.GitHubResponse$ResponseInfo.getBodyAsString(GitHubResponse.java:326)
    at org.kohsuke.github.GitHubResponse.parseBody(GitHubResponse.java:91)
    at org.kohsuke.github.Requester.lambda$fetch$1(Requester.java:74)
    at org.kohsuke.github.GitHubClient.createResponse(GitHubClient.java:461)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:412)
Caused: org.kohsuke.github.GHFileNotFoundException: https://api.github.com/repos/xxxxx/xxxxxxxx {"message":"Not Found","documentation_url":"https://docs.github.com/rest/reference/repos#get-a-repository"}
    at org.kohsuke.github.GitHubClient.interpretApiError(GitHubClient.java:492)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:420)
    at org.kohsuke.github.GitHubClient.sendRequest(GitHubClient.java:363)
    at org.kohsuke.github.Requester.fetch(Requester.java:74)
    at org.kohsuke.github.GHRepository.read(GHRepository.java:117)
    at org.kohsuke.github.GitHub.getRepository(GitHub.java:549)
    at org.jenkinsci.plugins.github_branch_source.GitHubSCMSource$DescriptorImpl.doValidateRepositoryUrlAndCredentials(GitHubSCMSource.java:2033)
    at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:710)
    at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
    at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
    at org.kohsuke.stapler.interceptor.RequirePOST$Processor.invoke(RequirePOST.java:77)
    at org.kohsuke.stapler.PreInvokeInterceptedFunction.invoke(PreInvokeInterceptedFunction.java:26)
    at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
    at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
    at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:536)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:281)
    at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
    at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
    at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
    at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
    at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:763)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1631)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
    at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:76)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at com.atlassian.bitbucket.jenkins.internal.applink.oauth.serviceprovider.auth.OAuth1aRequestFilter.doFilter(OAuth1aRequestFilter.java:109)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
    at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
    at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
    at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:153)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
    at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:119)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
    at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
    at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
    at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
    at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
    at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:51)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
    at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
    at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
    at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:36)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1618)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:549)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:143)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:578)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:235)
    at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1610)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1369)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:489)
    at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1580)
    at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1284)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127)
    at org.eclipse.jetty.server.Server.handle(Server.java:501)
    at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:383)
    at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:556)
    at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:375)
    at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:272)
    at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311)
    at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
    at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171)
    at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129)
    at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:806)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:938)
    at java.base/java.lang.Thread.run(Thread.java:834)

```

# Answer
> 4 votes
I encountered the same issue when using the GitHub Branch Source Plugin in Jenkins as part of a multibranch pipeline. The documentation for the credentials configuration states:

> Credentials used to scan branches and pull requests, check out sources and mark commit statuses.
>
> Note that **only "username with password" credentials are supported**. Existing credentials of other kinds will be filtered out. This is because Jenkins uses the GitHub API, which does not support other ways of authentication.
>
> If none is given, only the public repositories will be scanned, and commit status will not be set on GitHub.
>
> If your organization contains private repositories, then you need to specify a credential from a user who has access to those repositories. **This is done by creating a "username with password" credential where the password is GitHub personal access tokens**. The necessary scope is "repo".

After creating a new Personal Access Token (PAT) in GitHub with 'repo' scope, I added the PAT into the Jenkins Credentials manager as type 'Username and Password' (username was the GitHub account username for which the PAT was created). I then went back to my job configuration in Jenkins and ensured that my new credentials were being used for the GitHub branch source, which validated successfully.

I'm unsure if this answer applies to the context where you saw this issue (as there isn't too much detail in your question) but hopefully this answer will be useful for someone else in a similar position - as I didn't find many documented instances of this error.

---
Tags: jenkins, github
---


# Question
Title: Ansible Playbook fails start Apache Cassandra
I have been tried ran a Ansible Playboo, but after change the `cassandra.yml` when the service tries start I received this error bellow:

```
● cassandra.service - LSB: distributed storage system for structured data
   Loaded: loaded (/etc/rc.d/init.d/cassandra; bad; vendor preset: disabled)
   Active: failed (Result: exit-code) since sex 2021-04-02 01:06:05 UTC; 15min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 5052 ExecStop=/etc/rc.d/init.d/cassandra stop (code=exited, status=1/FAILURE)
  Process: 4883 ExecStart=/etc/rc.d/init.d/cassandra start (code=exited, status=0/SUCCESS)
 Main PID: 4972 (code=exited, status=3)

abr 02 01:05:37 ip-10-5-10-195.ec2.internal cassandra[4883]: Starting Cassandra: OK
abr 02 01:05:37 ip-10-5-10-195.ec2.internal systemd[1]: Started LSB: distributed storage system for structured data.
abr 02 01:05:40 ip-10-5-10-195.ec2.internal systemd[1]: cassandra.service: main process exited, code=exited, status=3/NOTIMPLEMENTED
abr 02 01:05:40 ip-10-5-10-195.ec2.internal runuser[5063]: pam_unix(runuser:session): session opened for user cassandra by (uid=0)
abr 02 01:05:40 ip-10-5-10-195.ec2.internal cassandra[5052]: Shutdown Cassandra: kill: sending signal to 4972 failed: No such process
abr 02 01:05:40 ip-10-5-10-195.ec2.internal runuser[5063]: pam_unix(runuser:session): session closed for user cassandra
abr 02 01:06:05 ip-10-5-10-195.ec2.internal cassandra[5052]: ERROR: could not stop cassandra
abr 02 01:06:05 ip-10-5-10-195.ec2.internal systemd[1]: cassandra.service: control process exited, code=exited status=1
abr 02 01:06:05 ip-10-5-10-195.ec2.internal systemd[1]: Unit cassandra.service entered failed state.
abr 02 01:06:05 ip-10-5-10-195.ec2.internal systemd[1]: cassandra.service failed.

```

In my *cassandra.yml* I just change the mount point/path where Cassandra stores the **data** and sure that this works.

Looking at Cassandra `system.log` I found:

```
ERROR [main] 2021-04-05 16:57:34,074 CassandraDaemon.java:822 - Exception encountered during startup
java.lang.ExceptionInInitializerError: null
    at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:240)
    at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:676)
    at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:800)
Caused by: java.lang.IllegalStateException: Failed to mkdirs /hints; unable to start server
    at org.apache.cassandra.config.DatabaseDescriptor.createAllDirectories(DatabaseDescriptor.java:1395)
    at org.apache.cassandra.db.commitlog.CommitLog.<init>(CommitLog.java:95)
    at org.apache.cassandra.db.commitlog.CommitLog.construct(CommitLog.java:78)
    at org.apache.cassandra.db.commitlog.CommitLog.<clinit>(CommitLog.java:65)
    ... 3 common frames omitted

```

It's weird because I create the directory at new mount point with `cassandra:cassandra` permitions:

```
drwxr-xr-x  2 cassandra cassandra   6 abr  5 14:46 commitlog
drwxr-xr-x  2 cassandra cassandra   6 abr  5 14:46 data
drwxr-xr-x  2 cassandra cassandra   6 abr  5 16:53 hints
drwxr-xr-x  2 cassandra cassandra   6 abr  5 16:53 saved_caches

```

I have been spend long hours trying solve but without success, someone my can help me?

# Answer
> 1 votes
I found my fault. It's necessary define a `cassandra.service` to Systemd manages Cassandra:

*path*: `/etc/systemd/system/cassandra.service`

And adjust like you installation

```
[Unit]
Description=Cassandra Cluster Node Daemon

[Service]
Type=forking
User=cassandra
ExecStartPre=/usr/bin/echo "Starting Cassandra Daemon"
ExecStart=/usr/sbin/cassandra
ExecStartPost=/usr/bin/echo "Cassandra Daemon Running"

ExecStopPost=/usr/bin/rm -rf /cassandra/saved_caches;/usr/bin/echo "Cassandra Daemon Stopped"

SuccessExitStatus=143

[Install]
WantedBy=default.target

```

---
Tags: ansible, cassandra, systemd
---


# Question
Title: Jenkins - Active Choice DynamicReferenceParameter not returning list
I have been trying unsuccessfully to get DynamicReferenceParameter of the Active Choice plugin to show previously selected values as list, however it either shows them on one line or it treats each symbol as a new element in the list.

```
hosts_script = '''
return['host1', 'host2', 'host3']
'''

properties([
    parameters([
      [$class: 'ChoiceParameter',
        choiceType: 'PT_MULTI_SELECT',
        description: 'Select one or more hosts',
        filterLength: 1,
        filterable: true,
        name: 'hosts',
        randomName: 'choice-parameter-5631314439613978',
        script: [
            $class: 'GroovyScript',
            fallbackScript: [
                classpath: [],
                sandbox: true,
                script:
                    'return[\'Could not get hosts\']'
            ],
            script: [
                classpath: [],
                sandbox: true,
                script: hosts_script,
            ]
        ]
    ],
    [$class: 'DynamicReferenceParameter',
        choiceType: 'ET_UNORDERED_LIST',
        description: 'These are the selected hosts',
        name: 'Selected hosts',
        omitValueField: false,
        randomName: 'choice-parameter-5693388460832175',
        referencedParameters: 'hosts',
        script: [
            $class: 'GroovyScript',
            fallbackScript: [
                classpath: [],
                sandbox: true,
                script:
                    'return[\'Could not get hosts\']'
            ],
            script: [
                classpath: [],
                sandbox: true,
                script:
                    """
                    //def list = hosts.replaceAll( '"','\\'' )
                    //List list = hosts.collect{ '\\'' + it + '\\','}
                    //def list1 = new ArrayList(Arrays.asList(hosts))
                    List list = hosts.collect{ '\\'' + it + '\\','}
                    //String list = hosts.inspect()

                    return list
                    """
              ]
          ]
       ]
    ])
 ])

```

I can get this  `• host1,host2`

Or this:

```
 •'h',
 •'o',
 •'s',
 •'t',
 •'2',

```

Where I want this:

```
 • host1
 • host2

```

At the very end you will see what I have been playing with ... the biggest problem for me is that is impossible to use `println` to troubleshoot.

# Answer
> 1 votes
I figured out what was going on. The return from `hosts` var was a single string with comma separated hostnames `host1, host2`, I had to convert it to List using the comma as delimiter. The following post How to convert comma-separated String to List? helped me with that task.

Basically the last section now reads

```
                script:
                """
                List<String> list = Arrays.asList(hosts.split("\\\\s*,\\\\s*"));
                return list
                """

```

---
Tags: jenkins, groovy, jenkins-plugins
---


# Question
Title: Service to service communication - through proxy or direct?
A common scenario is two or more services behind a reverse proxy, typically each deployed in a container:

```
| service 1 |          | service 2 |
           \            /
            \          /
             \        /
             | Proxy  |

```

Now the two services want to talk to each other. What are the pros/cons of:

1. Talking directly to each other?
2. Going through the reverse proxy?

Is there any standard best practice? Any potential security concerns?

# Answer
The question is: why use a proxy to communicate both applications? What is the point or target (support SSL, security, availability between services?

In any case, you need 2 proxies, one in each app, in front of each application, it is the best way to use it.

You can also use tunnels, like SSH tunnel or Stunnel that support TLS
> 1 votes

# Answer
Your challenges are:

* Service discovery (usually DNS-based, but there are other approaches)
* Load balancing for multiple instances of services (server-side with a load balancer, or client-side with smart service discovery)
* Encryption/TLS
* Retry and circuit breaker

There are few patterns

* API Gateway routing traffic for API calls not only for external traffic but also internally. Sometimes useful, but there are situations when you expose a "public" API through the gateway only, but not the "private" APIs that are internal-only
* A service mesh (Istio, LinkerD) - a sidecar is added to each service that is technically a reverse proxy (Envoy) and will handle mTLS/encryption, authentication, tracing, logging, circuit breaker, load balancing etc. The control plane of a service mesh keeps all proxies in sync.
* Point-to-point communication with client-side load balancing (or none in a simplest situation)
> 1 votes

---
Tags: containers, networking, microservices
---