# Question
Title: Applying ansible playbook to a docker container on a remote host
I have automated the setup of several docker containers with ansible. This works pretty well but there are certain recurring tasks that have to be executed within the docker containers.

It is possible to implement this using the docker connection driver for ansible but this only seems to work when ansible is running on the host that is the docker runner. This is not the case and I don't want to run ansible on multiple hosts if possible. Is there a better way to use ansible to execute playbooks within containers?

Notes:

* I know how I can execute commands in a docker container from the host system but I don't only want to execute commands. I want to be able to use Ansible for docker containers as if they where standalone hosts.
* I tried it out of curiosity and it works pretty well using ansible directly on the docker runner,... anyway not my favorite solution.

# Answer
With the help of @Levi (refered to Stackoverflow) I managed to find a way to connect straigt into the docker containers using ansibles docker connection driver and the remote API capability of Docker.

First of all you have to expose the API which is by default not the case. Simply add `-H tcp://0.0.0.0:1337` to the `ExecStart` constant of the systemd script. Then reload the start script and restart the docker service. (From this blogpost)

Then you can list the containers in the inventory file like this:

```
[containers]
container-name ansible_connection=docker ansible_docker_extra_args="-H tcp://1.2.3.4:1337"

```

And execute any playbook like your used to.

```
ansible-playbook -i docker_inventory playbooks/my_playbook.yml

```

The upside of this way of implementing it this way is that you can mix up "real" hosts/vms that are maintained through SSH and Docker containers. One negative thing I have to mention is that the docker API is much slower than using SSH as a control tunnel.

**This setup is of cause horribly insecure and its only purpose is to explain the concept. Before using it in production you should read into the security features and limit access to the daemon as far as possible.**
> 14 votes

# Answer
I have achieved the same goal but a bit more securely, by forwarding the unix socket (but not a tcp socket). However it is a bit convoluted. The following example contains everything from starting an example container, forwarding the socket and tearing down everything afterwards.

In this scenario I have 3 "hosts" (in the sense of ansible hosts), localhost, the server and the container. Docker-cli must be present on localhost.

My inventory looks like this, I've explicitly specified the forwarded docker socket for the connection to the container:

```
---
all:
  hosts:
    server:
    container:
      ansible_connection: docker
      ansible_docker_extra_args: "-H unix:///tmp/remote_docker.sock"
  vars:
    docker_sock_path: "/tmp/remote_docker.sock"

```

The playbook consists of multiple plays, as it needs to apply to different hosts.

1. At first an ssh-forwarding of the unix-socket is setup. This is done on localhost (however it's not a separate play, just delegated
2. The docker container is started (this is done on the server). The container is started with `interactive: true` so it stays alive without anything to do.
3. Tasks are run in the Container
4. The SSH Process is killed and the socket file is deleted (this is done on localhost)

```
---
- name: "Preparing SSH, starting the container"
  hosts: server
  tasks:
  - name: Create SSH tunnel for the unix docker socket
    command:
      cmd: "ssh -fnNT -L {{ docker_sock_path }}:/var/run/docker.sock {{ ansible_user }}@{{ ansible_host }}"
    delegate_to: localhost
  - name: create connection container
    community.docker.docker_container:
      name: container
      image: python:3.12-slim-bookworm
      state: started
      interactive: true
      entrypoint: "/bin/sh"

- name: "Executing steps in docker container"
  hosts: container
  tasks:
    [...]
- name: "Shutting down proxy container"
  hosts: server
  tasks:
  - name: stop connection container
    community.docker.docker_container:
      name: server
      state: stopped
- name: "Shutting down socket forwarding"
  hosts: localhost
  connection: local
  tasks:
  - name: get process ids
    shell: ps xf
    register: all_pid
  - name: extract pid of our ssh process
    ansible.builtin.set_fact:
     ssh_pid: "{{ item | trim | split | first }}"
    loop: "{{ all_pid.stdout_lines }}"
    when: item is match("^.*{{docker_sock_path }}:/var/run/docker.sock.*$")
  - name: kill ssh process
    ansible.builtin.command:
      cmd: "kill {{ ssh_pid }}"
  - name: Remove socket file
    ansible.builtin.file:
      path: "{{ docker_sock_path }}"
      state: absent

```
> 0 votes

---
Tags: docker, ansible
---


# Question
Title: Prometheus/BlackBox Exporter probe_success 0 but service still running
I'm currently utilizing Prometheus with Blackbox Exporter for monitoring our services. However, I've encountered an issue where one of our external services is showing probe\_success as 0, despite being accessible when tested with curl.

***All my other services are being monitored correctly.***

I would appreciate any assistance in troubleshooting and resolving this discrepancy. Thank you. prometheus config

```
  - job_name: 'ExternalPartners_ICMP'
    scrape_interval: 15s
    metrics_path: /probe
    params:
      module: [http_2xx]  # Look for a HTTP 200 response.
    static_configs:
      - targets:
        - https://*/*.asmx  #GazProm
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: 127.0.0.1:9115  # The blackbox exporter's real hostname:port.

```

blackbox config

```
http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions:
      - HTTP/1.1
      - HTTP/2
      - HTTP/2.0
      valid_status_codes: [200]
      method: GET
      no_follow_redirects: false
      fail_if_ssl: false
      fail_if_not_ssl: true
      preferred_ip_protocol: "ip4"
      ip_protocol_fallback: true
      tls_config:
        insecure_skip_verify: true

```

# Answer
maybe `tls_config` setting needs to be set

```
 scheme: https
    tls_config:
      insecure_skip_verify: true

```
> 0 votes

---
Tags: prometheus
---


# Question
Title: kubernetes access nodeport fails, master cannot reach workers
I've got an on premise kubernetes cluster with a NodePort service running on port 32001. But in 80% of the cases requests fail:

```
curl localhost:32001

```

times out and sometimes it works, results are returned

Als starting an nginx pod (ng) in the cluster and calling

```
wget -O- http://ng:80

```

from a busybox does not work ever. While in another cluster it works.

We use flannel CNI so I checked logs of flannel pods and kube-proxy but can't find anything weird.

Any ideas what to check for? What can cause a sometimes timeout and sometimes success?

# Answer
> 1 votes
Found the cause, maybe someone will find it useful in the future.

This happened because flannel is not correctly set up for an on premise kubernetes environment.

I installed the default kube-flannel.yaml but two things need to be changed first in on premise environments.

1. The specific ip of the network interface must be passed to flanneld as a commandline argument. Otherwise flannel takes the first network interface.

```
 command:
 - /opt/bin/flanneld
 args:
 - --ip-masq
 - --kube-subnet-mgr
 - --iface=enp0s8

```
2. The default config has "backend type" VXLAN which causes the vxlan.go package code to be triggered in flannel. This works in azure, gcloud and AWS etc. but not on premise. Type host-gw must be configured so the hostgw code is run which tries to connect to other machines with layer2 routing.

net-conf.json: | { "Network": "10.244.0.0/16", "Backend": { "Type": "host-gw" } }

Checking out flannel's source code for hostgw:

https://github.com/flannel-io/flannel/blob/master/pkg/backend/hostgw/hostgw.go

every backend is added with it's "constructor" New function during startup so in the case of host-gw where ip address of extIface is the one passed with --iface option.

```
func init() {
    backend.Register("host-gw", New)
}

func New(sm subnet.Manager, extIface *backend.ExternalInterface) (backend.Backend, error) {
    if !extIface.ExtAddr.Equal(extIface.IfaceAddr) {
        return nil, fmt.Errorf("PublicIP differs from interface IP probably on NAT is not supported by host-gw")
    }

    be := &HostgwBackend{
        sm:       sm,
        extIface: extIface,
    }
    return be, nil
}

```

The HostgwBackend is then instantiated in main.go and RegisterNetwork is called.

```
bm := backend.NewManager(ctx, sm, extIface)
be, err := bm.GetBackend(config.BackendType)
bn, err := be.RegisterNetwork(ctx, &wg, config)

```

And in RegisterNetwork the function is set up which communicates the pod routes to the "pod-network-cidr" (10.244.0.0/24) of the cluster to the kernel with the netlink package when it calls AcquireLease and Lease. Then to retrieve the routes it uses netlink again to query for the routes.

```
    n.GetRoute = func(lease *lease.Lease) *netlink.Route {
        return &netlink.Route{
            Dst:       lease.Subnet.ToIPNet(),
            Gw:        lease.Attrs.PublicIP.ToIP(),
            LinkIndex: n.LinkIndex,
        }
    }

```

So after initialization 'ip route' shows all the created routes.

ip route show

```
10.244.1.0/24 via 10.203.32.120 dev ens160
10.244.2.0/24 via 10.203.32.98 dev ens160
10.244.3.0/24 via 10.203.32.111 dev ens160
10.244.4.0/24 via 10.203.32.99 dev ens160
10.244.5.0/24 via 10.203.32.110 dev ens160
10.244.6.0/24 via 10.203.32.121 dev ens160

```

---
Tags: kubernetes, networking, containerd
---


# Question
Title: 502 Bad Gateway when HTTPS listener is configured on Nexus Private Registry
**Context:**
In order to pull custom images I made for AWX execution environment, I deployed a Sonatype Nexus Repository on my K8s cluster using this Helm Chart.

I've simply modify it with my values:

```
nexus:
  docker:
    enabled: true
    registries:
      - host: awx-private-repo.mydomain.com
        port: 8080
        secretName: nginx-ingress-tls-secret

ingress:
  enabled: true
  ingressClassName: nginx
  annotations:
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
  hostPath: /
  hostRepo: my-nexus.mydomain.com
  tls:
    - secretName: nginx-ingress-tls-secret
      hosts:
        - my-nexus.mydomain.com

```

My configuration is pretty basic, an ingress-nginx is configured to expose the Nexus UI, this works pretty smoothly and easily.

In a later stage, I created a docker(hosted) private registry from Nexus UI with an HTTP or HTTPS listener on port 8080 (will be detailed into Problem: section below) Naturally, the configuration beyond create a service + an ingress.

From an K8s-UI perspective, my deployment looks like this

From a Nexus-UI perspective, docker private registry is simply configured like this :

**Problem:**

* if my repo is configured with a 8080 **HTTPS** listener, it's not accessible and nginx raised a `502 BAD GATEWAY` error
* if I configure a 8080 **HTTP** listener, repository is accessible and I can perform a docker login command.

Furthermore, I saw on Nexus UI below HTTP/HTTPS listener  `Create an HTTP connector at specified port. Normally used if the server is behind a secure proxy.` and `Create an HTTPS connector at specified port. Normally used if the server is configured for https.`

**My question are :**

Does ingress-nginx is considered as a "secure proxy" ?

What should be the best practices with Nexus repository ? create an HTTP repo on port 8080 or an HTTPS one ? in which case HTTP/HTTPS should be used in a repository context ?

# Answer
> 1 votes
This is most likely due to the ingress not being configured to passthrough SSL.

Add this annotation to the ingress:

```
nginx.ingress.kubernetes.io/ssl-redirect: "true"

```

Additionally, make sure that the ingress controller was started with `--enable-ssl-passthrough`

To answer your questions: Considering ingress-nginx as a "secure proxy" is for your security organization to decide based on specific deployment, and access means. However it is considered secure in many organizations

You can decide on whatever port you want, but the difference between HTTP & HTTPS is where the SSL termination will occur. Using nginx as ingress will by default encrypt traffic from client to Nginx, but from nginx to Nexus will be in the clear. With the annotations given above, traffic will be encrypted end-to-end.

---
Tags: kubernetes, nexus, ingress-nginx
---


# Question
Title: EC2 instances cannot connect over Elastic IPs
I have 2 EC2 instances and I am trying to get them to connect over port 44874 (it's a layer 1 cryptocurrency like `dogecoind`).

* Both are running Ubuntu 22.04 AMI
* Both are in the same subnet
* Both are in the same VPC
* Security Group:
    + `0.0.0.0/0` incoming on port `44874` is open
    + `0.0.0.0/0` and `::/0` outbound are open on all ports
* `sudo ufw status` is showing "Inactive" for both
* On my personal computer, I can SSH into both of them from their elastic IPs (22 is open in the SG on my IP)

I am trying to get them to connect with their respective Elastic IP addresses. They will not talk to each other, but they will talk to my wallet when I hit their elastic IPs from my personal computer.

Does anyone know what the problem might be? I have about 10 other servers in this VPC and they seem to all be doing this as well. No problems talking to the outside world, but they cannot talk to each other with their elastic IPs.

Any ideas?

# Answer
> 1 votes
To troubleshoot your network configuration, consider utilizing the VPC reachability analyzer, a tool provided by AWS. This tool helps analyze reachability and visibility within your VPC. You can find more information about it here -https://aws.amazon.com/blogs/aws/new-vpc-insights-analyzes-reachability-and-visibility-in-vpcs/.

The potential issues might stem from incorrect ACL rules or misconfigured routing tables. It's advisable to carefully review and verify these configurations to ensure proper communication between your EC2 instances.

---
Tags: amazon-web-services, amazon-ec2
---


# Question
Title: MAUI build failing with "Microsoft.Android.Sdk.Tooling.targets(20,5): error XA0031: Java SDK 11.0 or above is required when using .NET 6 or higher"
We have an existing Azure DevOps build pipeline that was working fine until today, 20th November 2023. We have noticed that the previous successful build was using MSBuild version 17.7.3. Today it's using 17.8.3 that I assume means it is on a server with the new VS image on it.

```
MSBuild version 17.8.3+195e7f5a3 for .NET
Build FAILED.
C:\Program Files\dotnet\packs\Microsoft.Android.Sdk.Windows\33.0.95\targets\Microsoft.Android.Sdk.Tooling.targets(20,5): error XA0031: Java SDK 11.0 or above is required when using .NET 6 or higher. Download the latest JDK at: https://aka.ms/msopenjdk`

```

Looks like this is a breaking change for Xamarin / MAUI builds in the latest VS version (.NET 8 official, although our project is .NET 7). Any ideas how to fix it?

# Answer
> 0 votes
Fixed it by adding a Java Tool Installer step before the dotnet build step, to use Java 11 explicitly

Settings:

---
Tags: azure-devops, dotnet
---


# Question
Title: Beginner Uptime Calculations Based on ELB?
Is there a clear way to calculate starting uptime metrics from AWS ELB?

I need to start running uptime calculations. I'd like to start by just using AWS services for monitoring the uptime and then grow from there as I determine what I'd like that to mean.

I have the service behind an ELB as a target group. I trust the instance health check to indicate service uptime. I'd like to use the metrics on the AWS ELB as received by AWS CloudWatch to calculate the uptime over 30 days. If the `HealthyHostCount` metric is 0 for 10 seconds, that's 10 seconds of downtime. Otherwise, we have uptime. Then I'd like to have a percentage of time up over a period (30 days) to the hundredth of a percent.

I've been playing with CloudWatch metrics to try to figure this out and finding that I can do math and have been getting results that seem in the right neighborhood, but I'm not confident that I'm properly thinking about the statistic period and the graph period. I'm also wondering if I should be using the PR statistic instead of doing my own math.

Here's what I have right now. I'm grabbing the ELB's `HealthyHostCount as m1` and `UnhealthyHostCount` as m2. Both are summed up for 30 days. Then I have an expression `100*(m1/m1+m2)` to calculate percent up counts over the total counts taken for that 30 days. I have the chart set to a relative 30 days period.

The problem is that this calculation, as soon as I have some value greater than 0 in m2, seems to result in numbers in the thousands or millions instead of reducing the percent down from 100.

How would you calculate service percent uptime from AWS ELB metrics?

# Answer
I decided to approach the calculations this way.

Success = HealthyHostCount \> 0

Fail = HealthyHostCount \<= 0

Time = HealthyHostCount + UnhealthyHostCount

Uptime = 100\*(Success/(Success+Fail))

That ended up with a dashboard widget in CloudWatch with a source like this.

```
{
    "metrics": [
        [ { "expression": "100*(succeed/(succeed+fail))",
            "label": "Dev %Up", "id": "e2",
            "region": "us-west-1", "period": 10 } ],
        [ { "expression": "IF(hhc > 0, 1, 0)",
            "label": "Success Count", "id": "succeed",
            "visible": false, "region": "us-west-1" } ],
        [ { "expression": "IF(uhc > 0, 1, 0)",
            "label": "Fail Count", "id": "fail",
            "visible": false, "region": "us-west-1" } ],
        [ "AWS/ELB",
          "UnHealthyHostCount",
          "LoadBalancerName",
          "internal-dev-XXX",
          { "id": "uhc", "visible": false,
            "region": "us-west-1" } ],
        [ ".", "HealthyHostCount", ".", ".",
          { "id": "hhc", "visible": false,
            "region": "us-west-1" } ]
    ],
    "view": "gauge",
    "stacked": false,
    "region": "us-west-1",
    "stat": "Sum",
    "period": 10,
    "title": "Dev Uptime",
    "yAxis": {
        "left": {
            "min": 0,
            "max": 100
        }
    },
    "setPeriodToTimeRange": true,
    "sparkline": false,
    "trend": false,
    "singleValueFullPrecision": false
}

```
> 0 votes

---
Tags: amazon-ec2, amazon-cloudwatch
---


# Question
Title: Creating a cmd task with all the private ip address details collected from gather_facts
I am running an Ansible playbook from my local laptop to AWS connecting on the public IP address. The infra is being spun up by terraform so I don't know the private IP address before hand. I have around 8 hosts. I want to create a command which needs to have all the private IPs of all 8 hosts.

I have tried `gather_facts` method but I don’t know how to get values from all 8 hosts assigned to some variable, so that I can use it at another task.

```
    - name: initialise cluster
      command:
        cmd: /opt/splunk/bin/splunk splunk init shcluster-config -auth admin:changed -mgmt_uri https://{{ private ip address of host1 }}:8089 -replication_port 34567 -replication_factor 2 -conf_deploy_fetch_url https://{private ip address of host2}:8089 -secret mykey -shcluster_label shcluster1
      become: true
      become_user: root
      ignore_errors: yes
      async: 10
      poll: 0

- name: set cluster
      command:
        cmd: /opt/splunk/bin/splunk splunk bootstrap shcluster-captain -servers_list "{{private ip of 1st host}}:8089,{{ private ip of 2nd host }}:8089,{{ private ip of 3rd host }}:8089,etc..." -auth <username>:<password>
      become: true
      become_user: root
      ignore_errors: yes
      async: 10
      poll: 0

```

How do I run a `set_facts` command which will get the private ip from each hosts and assign it to a cmd task?

# Answer
> 2 votes
Q: ***"Run a set\_facts command which will get private ip from each host."***

A: You can either run a command, for example,

```
  - shell: 'ifconfig em0 | grep inet | cut -w -f3'
    register: out

```

and use the registered variable

```
  cmnd_ip: "{{ dict(ansible_play_hosts|
                    zip(ansible_play_hosts|
                        map('extract', hostvars, ['out', 'stdout']))) }}"

```

, or you can run the module *setup* and use the facts, for example

```
  fact_ip: "{{ dict(ansible_play_hosts|
                    zip(ansible_play_hosts|
                        map('extract', hostvars, ['ansible_em0', 'ipv4']))) }}"

```

Notes:

* Fit the command to your needs.
* When you use *setup* the format of facts may vary among the operating systems.
* There might be more addresses assigned to an interface. In this case, you'll have to decide which one to use.
* Create the list of the servers. For example,

```
  servers_list: "{{ cmnd_ip.values()|
                    product(['8089'])|
                    map('join', ':')|
                    join(',') }}"

```

gives

```
  servers_list: 10.1.0.51:8089,10.1.0.52:8089,10.1.0.53:8089

```

---

<sup>

Example of a complete playbook for testing

```
- hosts: all

  vars:

    fact_ip: "{{ dict(ansible_play_hosts|
                      zip(ansible_play_hosts|
                          map('extract', hostvars, ['ansible_em0', 'ipv4']))) }}"

    cmnd_ip: "{{ dict(ansible_play_hosts|
                      zip(ansible_play_hosts|
                          map('extract', hostvars, ['out', 'stdout']))) }}"

  tasks:

    - name: Get facts
      block:

        - setup:
        - debug:
            var: fact_ip
          run_once: true

      tags: fact

    - name: Run command
      block:

        - shell: 'ifconfig em0 | grep inet | cut -w -f3'
          register: out
        - debug:
            var: cmnd_ip
          run_once: true

      tags: cmnd

```

given the inventory

```
shell> cat hosts
[test]
test_01
test_02
test_03

```

gives (abrdiged)

```
PLAY [all] ***********************************************************************************

TASK [setup] *********************************************************************************
ok: [test_01]
ok: [test_03]
ok: [test_02]

TASK [debug] *********************************************************************************
ok: [test_01] =>
  fact_ip:
    test_01:
    - address: 10.1.0.51
      broadcast: 10.1.0.51
      netmask: 255.255.255.255
      network: 10.1.0.51
    test_02:
    - address: 10.1.0.52
      broadcast: 10.1.0.52
      netmask: 255.255.255.255
      network: 10.1.0.52
    test_03:
    - address: 10.1.0.53
      broadcast: 10.1.0.53
      netmask: 255.255.255.255
      network: 10.1.0.53

TASK [shell] *********************************************************************************
changed: [test_01]
changed: [test_02]
changed: [test_03]

TASK [debug] *********************************************************************************
ok: [test_01] =>
  cmnd_ip:
    test_01: 10.1.0.51
    test_02: 10.1.0.52
    test_03: 10.1.0.53

```
</sup>

---
Tags: ansible, ansible-inventory
---


title: unzip | caution: filename not matched

text:
# Question
Title: unzip | caution: filename not matched
I'm trying to unzip multiple files which has a prefix in common **service1.zip** **service-2.zip** **service\_3.zip** and so on...

Now, when I try to unzip all those files at once using the command `unzip service*.zip` I'm getting an error which says `caution: filename not matched: service.zip`

# Answer
> 0 votes
This is because there are special characters in the file names and bash misunderstands while expanding that command.

To avoid this error:

**solution-1**: use the command within **''** `unzip 'service*.zip'`
**solution-2**: pass the file name with **\** to escape special characters `unzip service\\*.zip`

---
Tags: linux, ubuntu, bash
---


# Question
Title: How to aggregate logs and metrics from multiple modules accross different servers?
We are thinking about setting up a solution for aggregating logs and metrics from multiple modules accross different servers. The problem is that it needs to run in a hospital and the logs may contain patient data when we debug a problem. So all external solutions are not possible. Also it needs to run as a windows service, as hospitals here usually only use Windows.

The modules are a normal Spring Boot application. The modules can connect to central server but not the other way around, so it needs to push the data to a central server.

Any suggestions?

# Answer
> 1 votes
I recommend looking into using Grafana, Loki, and Promtail on-premises. Because they are implemented in Go, cross-compiling to run in Windows is possible. For the past several years, I have been extremely pleased using Grafana in private, on-prem settings disconnected from external world.

Additionally, consider time series database systems for metrics. In your case, InfluxDB might be a more suitable choice than Prometheus because of push vs pull models.

Check out shared dashboards you can download and install.

---
Tags: monitoring, java
---


# Question
Title: validate service connection: validate CRI v1 image API for endpoint
Today when I use this crictl `crictl version v1.28.0` command to pull image in CentOS 7.9:

```
[root@k8sslave01 ~]# crictl pull registry.k8s.io/pause:3.6
WARN[0000] image connect using default endpoints: [unix:///var/run/dockershim.sock unix:///run/containerd/containerd.sock unix:///run/crio/crio.sock unix:///var/run/cri-dockerd.sock]. As the default settings are now deprecated, you should set the endpoint instead.
ERRO[0000] validate service connection: validate CRI v1 image API for endpoint "unix:///var/run/dockershim.sock": rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing: dial unix /var/run/dockershim.sock: connect: no such file or directory"
E1122 16:47:19.799530   31559 remote_image.go:180] "PullImage from image service failed" err="rpc error: code = DeadlineExceeded desc = failed to pull and unpack image \"registry.k8s.io/pause:3.6\": failed to resolve reference \"registry.k8s.io/pause:3.6\": failed to do request: Head \"https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6\": dial tcp 142.250.157.82:443: i/o timeout" image="registry.k8s.io/pause:3.6"
FATA[0030] pulling image: rpc error: code = DeadlineExceeded desc = failed to pull and unpack image "registry.k8s.io/pause:3.6": failed to resolve reference "registry.k8s.io/pause:3.6": failed to do request: Head "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6": dial tcp 142.250.157.82:443: i/o timeout

```

I have checked the `/etc/containerd/config.toml` and make sure the cri is enabled and look like this :

```
#   Copyright 2018-2022 Docker Inc.

#   Licensed under the Apache License, Version 2.0 (the "License");
#   you may not use this file except in compliance with the License.
#   You may obtain a copy of the License at

#       http://www.apache.org/licenses/LICENSE-2.0

#   Unless required by applicable law or agreed to in writing, software
#   distributed under the License is distributed on an "AS IS" BASIS,
#   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
#   See the License for the specific language governing permissions and
#   limitations under the License.

#disabled_plugins = ["cri"]

#root = "/var/lib/containerd"
#state = "/run/containerd"
#subreaper = true
#oom_score = 0
[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

#[grpc]
#  address = "/run/containerd/containerd.sock"
#  uid = 0
#  gid = 0

#[debug]
#  address = "/run/containerd/debug.sock"
#  uid = 0
#  gid = 0
#  level = "info"

```

this is the containerd version info:

```
[root@k8sslave01 ~]# containerd -v
containerd containerd.io 1.6.24 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523

```

Am I missing something? what should I do to fixed this issue? I have already tried to restart the containerd:

```
systemctl restart containerd

```

# Answer
Finally I found I should add configuration `/etc/crictl.yaml`:

```
runtime-endpoint: unix:///var/run/containerd/containerd.sock
image-endpoint: unix:///var/run/containerd/containerd.sock
timeout: 10
#debug: true
debug: false

```
> 0 votes

---
Tags: kubernetes
---


# Question
Title: How do Argo CD track the resources it deploys?
How do Argo CD track the resources it deploys? I see no label or annotation being put on the resources, or any other modification for that matter. So how do Argo CD determine which Kubernetes resources belongs to a certain Argo CD Application or not?

# Answer
Argo CD by default uses the label **app.kubernetes.io/instance** to track resources it creates, but it can also be configured to use a custom label and/or annotation.

See https://argo-cd.readthedocs.io/en/latest/user-guide/resource\_tracking/ for full details.
> 0 votes

---
Tags: kubernetes, argocd
---


# Question
Title: What's the penalty for dropping a Google Cloud commitment?
Google Cloud is offering me discounts if I commit to a service for three years. I'm having a hard time trying to find out what's the penalty should I leave early.

The discount is tempting, but I'm updating my infrastructure all the time. What if I find a cheaper provider or if I decide to change my infra and suddenly I don't need a service any more?

# Answer
> 1 votes
No penalties will be incurred in terms of money. If you choose not to use it, you can block the attached card for payments. The only consequence will be the suspension of your account.

---
Tags: gcp
---


# Question
Title: Why is the containerd still pulling from the network even though the server contains an image
Today I found the kubernetes pod shows error like this:

```
Failed to create pod sandbox: rpc error: code = DeadlineExceeded desc = failed to get sandbox image "registry.k8s.io/pause:3.6": failed to pull image "registry.k8s.io/pause:3.6": failed to pull and unpack image "registry.k8s.io/pause:3.6": failed to resolve reference "registry.k8s.io/pause:3.6": failed to do request: Head "https://us-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.6": dial tcp 64.233.187.82:443: i/o timeout

```

Because of the network policy, I can not access the `registry.k8s.io`. So I load the pause image into the server like this:

```
ctr -n=k8s.io image import /root/pause.tar

```

this is the image list:

```
[root@k8sslave01 ~]# ctr -n=k8s.io image list|grep "pause"
registry.k8s.io/pause:3.6                                                                                application/vnd.docker.distribution.manifest.v2+json      sha256:79b611631c0d19e9a975fb0a8511e5153789b4c26610d1842e9f735c57cc8b13 669.8 KiB linux/amd64                                                                   io.cri-containerd.image=managed
[root@k8sslave01 ~]#

```

it worked for weeks but today it seems like it did not pull from local first. is it possible to fix this issue?

# Answer
There 2 are places that defines pause image tag, if you run kubernetes using containerd.

1. On containerd configuration file.

You can check running configuration by running

```
 containerd config dump | grep sandbox

```

You should modify this part on /etc/containerd/config.toml, so it similar as below:

```
    [plugins]
       [plugins."io.containerd.grpc.v1.cri"]
         sandbox_image = "yourregistryserver/pause:3.6"

This part defines default pause image.

```
2. kubelet configuration on /var/lib/kubelet/kubeadm-flags.env.

This part defines pause image used by current kubernetes version. You will find configuration like this:

```
 KUBELET_KUBEADM_ARGS=".... --pod-infra-container-image=yourregistryserver/pause:3.8"

```

As far as I recall, containerd configuration relevant only during initial initialisation. You can modify the value in both places if you want to be sure.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Automatic builds based on commit and deploy
I am trying to figure out how to get my Ubuntu docker server to react to code commits on a git-repo and build and deploy an image based on that.

I have a git repo in Azure Devops that contains an Angular project (with a working dockerfile).

I found that Docker does have automatic builds but it's a paid feature.

I have previously created build pipelines in Azure which work well (albeight very slowly), I am wondering if it is possible to get my Ubuntu VPS Docker host to react to git commits on the main branch and pull the latest code and build the docker image and replace it on the (local) server.

Just to be clear, I want to do all this on my own VPS. Any guides or hints to keywords that I should be looking for are much appreciated.

# Answer
> 0 votes
As far as I can see, you can only use Webhook on an actual repository that will call some service on your docker host. Docker host must be aware somehow.

That is how to set up webhook. https://learn.microsoft.com/en-us/azure/devops/service-hooks/services/webhooks?view=azure-devops

So flow is something like:

1. create microservice on docker host
2. create webhook on docker Azure repo
3. 2 URL will be 1 endpoint

1 can be run as docker user, so you have permission to execute docker build and so on.

I would do it this way.

# Answer
> 0 votes
Probably you can do the opposite, make github call the docker server. Use self hosted github runners. Configure and run the github agent on your server. An then create an action to do what you want on the docker server. That action can be auto fired for example with a commit or a merge,

---
Tags: docker
---


# Question
Title: Why does every image I build have its own repository?
What does "repository" mean to the build process? Why does every image need a repository?

I guess I should take this question up with whoever architect or team made this naming choice but why is there a new "repository" created whenever I build a new image for the same project? I think of repositories a service which stores multiple versions of multiple projects. I don't see why running 'docker build -t localBind9.v02 .' should result in any kind of repo. I had a 'actual' docker repository running at some point but to use that I had to deliberately push an image into it.

# Answer
I'm going to share my opinionated definitions that do not match the "official" definitions...

One stores one or more Docker *images* in a Docker *repository* (or more precisely, this would be known as *repositories* in the official definition), locally. I like to call this "the repository" rather than "repositories" when talking about the list of images that appear locally when I do `docker images` at the command line because "repositories" (plural) sounds confusing from that perspective to a lot of users. I think of the list of Docker images as being more analogous to a Maven repository, rather than some list of individual, separate Git repositories.

This local Docker repository may pull images from a public registry, such as Docker Hub, or a private registry, such as on a private network, for ease of sharing. Additionally, one can push an image to one of these registries.

Each *container* is an instance of the *image* (analogous to an *object* being an instance of a *class*). There is probably someone out there that thinks this definition is grossly incorrect, but it works for me.

Related:

1. Difference between Docker registry and repository
2. What's the Difference Between Docker Images and Containers?
> 1 votes

---
Tags: docker
---


# Question
Title: helm trouble creating pod using bitnami/mongodb chart
Here is my `My Chart.yaml` file:

```
apiVersion: v2
name: pixer-microservice-infra
description: A Helm chart for Kubernetes
type: application
version: 0.1.0
appVersion: "1.16.0"
dependencies:
  - name: mongodb
    version: ~14.3.1
    repository: "@bitnami"
    alias: mongodb

```

After running commands:

```
helm dependency build .
helm upgrade <chart-name>

```

my mongodb pod has the status: CrashLoopBackOff

Checked pod logs: `kubectl logs pod/<pod-name>`

`/opt/bitnami/scripts/libos.sh: line 346: 154 Illegal instruction "$@" > /dev/null 2>&1`

`uname -m`: arm64

Is there any compatible issue between this chart and the CPU architecture?

# Answer
> 1 votes
Yes, mongodb 5 on x86 requires a cpu with avx, check your cpu flags of the kubernetes node with lscpu | grep avx

---
Tags: kubernetes, helm, kubernetes-helm
---


# Question
Title: Microk8s Cluster Node not ready on Ubuntu with "cni plugin not initialized: failed to load cni config"
I'm trying to start a microk8s kubernetes instance on an ubuntu server, however the node becomes not ready.

```
~$ microk8s status
microk8s is not running. Use microk8s inspect for a deeper inspection.

```

Inspect returns the following error message:

```
less /var/snap/microk8s/6103/inspection-report/snap.microk8s.daemon-containerd/journal.log
  level=error msg="failed to load cni during init, please check CRI plugin status before setting up network for pods" error="cni config load fa
    iled: no network config found in /var/snap/microk8s/6103/args/cni-network: cni plugin not initialized: failed to load cni config"

```

The node is stuck with a similar message:

```
microk8s kubectl describe node
Name:               xxx
Roles:              <none>
Labels:             beta.kubernetes.io/arch=amd64
                    beta.kubernetes.io/os=linux
                    kubernetes.io/arch=amd64
                    kubernetes.io/hostname=xxx
                    kubernetes.io/os=linux
                    microk8s.io/cluster=true
                    node.kubernetes.io/microk8s-controlplane=microk8s-controlplane
Annotations:        node.alpha.kubernetes.io/ttl: 0
                    volumes.kubernetes.io/controller-managed-attach-detach: true
CreationTimestamp:  Wed, 08 Nov 2023 08:42:48 +0000
Taints:             node.kubernetes.io/not-ready:NoSchedule
Unschedulable:      false
Lease:
  HolderIdentity:  xxx
  AcquireTime:     <unset>
  RenewTime:       Wed, 08 Nov 2023 09:19:13 +0000
Conditions:
  Type             Status  LastHeartbeatTime                 LastTransitionTime                Reason                       Message
  ----             ------  -----------------                 ------------------                ------                       -------
  MemoryPressure   False   Wed, 08 Nov 2023 09:15:07 +0000   Wed, 08 Nov 2023 08:42:48 +0000   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   Wed, 08 Nov 2023 09:15:07 +0000   Wed, 08 Nov 2023 08:42:48 +0000   KubeletHasNoDiskPressure     kubelet has no disk pressure
  PIDPressure      False   Wed, 08 Nov 2023 09:15:07 +0000   Wed, 08 Nov 2023 08:42:48 +0000   KubeletHasSufficientPID      kubelet has sufficient PID available
  Ready            False   Wed, 08 Nov 2023 09:15:07 +0000   Wed, 08 Nov 2023 08:42:48 +0000   KubeletNotReady              container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

```

Since it seems potentially relevant: The host is behind a corporate proxy with no internes access except for the proxy configured in /etc/environment:

```
http_proxy=http://xxx:3128/
https_proxy=http://xxx:3128/
no_proxy=10.0.0.0/8,192.168.0.0/16,127.0.0.1,172.16.0.0/16

```

Could anyone explain what the problem in this case is?

# Answer
> 0 votes
If you are using AWS EKS, you need to install CNI Plugin.

Also, could you please share the output of sudo microk8s inspect

---
Tags: kubernetes, ubuntu
---


# Question
Title: How to organize Gitflow releases with multiple environments for web application?
We use gitflow branching strategy for different reasons, one of them being complicated bureaucracy around releases.

We have environments: `dev`, `qa`, `preprod`, `prod`.

Once the developers finish their work in `dev` branch, we create `release` branch and trigger deployment to `qa`. After `qa` process is done, we trigger deployment of the same branch to `preprod`. After tests in `preprod` are finished, we deploy the same branch to `prod`.

Issues arise when business asks to remove/add some features from `release` branch, for example, after deploying to `preprod` but before deploying to `prod`. In that case, if its a simple commit or several of them, we can cherry-pick. But that gets complicated very soon.

Is there a general solution for this? And, is the solution technical or is it on the planning side like establishing a consensus: `release` branch doesn't get deployed to `production` until all the features that went into `release` branch initially are ready?

# Answer
If you need to remove features from something that has been deployed to `preprod`, the new code should go through the process of promotion from `dev` to `qa` to `preprod` before it reaches `prod`. Depending on your amount and quality of automated testing, this could be a very fast process, but it doesn't make sense to short-circuit the process and put potentially untested or undertested changes into production.

In the absence of something like feature flags or keystone interfaces, Levi's suggestion of feature branches and squashing merges to make it easier to revert all aspects of a change is good. However, it doesn't negate the need for good testing since later changes could rely on the earlier (now-reverted) change. However, I would also invest time in understanding the root causes for needing to revert the change in the first place and addressing those so this becomes an exceptional case.
> 1 votes

# Answer
Have your developers do their work in feature branches and do a single feature per branch. Then merge their work into your `dev` branch. Now when you need to drop a feature from your release branch it's easier. If you want to make it really easy, when you merge from a feature branch to `dev` use the squash setting.
> 1 votes

---
Tags: git
---


# Question
Title: Dockerhub says Hello World causes CVE-2023-5363 and CVE-2023-5678
Dockerhub says I have the vulnerabilities: CVE-2023-5363 and CVE-2023-5678, which is part of the openssl package. See this image:

Does anyone know how I can resolve this issue? I've simplified my nodejs project to just `console.log('hello world');` and I removed all possible npm packages, but I still get this vulnerability. My `hello` project only consists of 4 files and they are:

`~/src/index.js`

```
console.log('hello');

```

`~/package.json`

```
{
  "name": "test",
  "version": "1.0.0",
  "main": "src/index.js",
  "scripts": {},
  "dependencies": {},
  "devDependencies": {}
}

```

`~/docker-compose.yml`

```
version: "3.8"
services:
  api_image:
    image: johndev/api:test2
    build:
      context: ./
      dockerfile: ./make.dockerfile

```

`~/make.dockerfile`

```
FROM node:21-alpine

WORKDIR /api

COPY ./package*.json ./

RUN mkdir ./src
COPY ./src/ ./src/

RUN npm install
ENTRYPOINT [ "echo", "'All Done'" ]

```

To push an image to dockerhub, I just run these commands:

```
docker-compose up --build -d;
docker push johndev/api:test2;

```

What did I do wrong? What in my code is causing the vulnerabilities CVE-2023-5363 and CVE-2023-5678?

# Answer
> 1 votes
CVS-2023-5363 and CVE-2023-5678 are both vulnerabilities in OpenSSL. They don't come from any of the code that you wrote. Your base image - `node:21-alpine` \- probably has a vulnerable version of OpenSSL by default. By adding the commands to update installed packages, you are installing a patched version of OpenSSL that no longer contains the vulnerabilities.

# Answer
> 0 votes
I added this line to my `make.dockerfile`

```
RUN apk update && \
    apk upgrade && \
    apk add --no-cache openssl

```

So now my code looks like:

```
FROM node:21-alpine

WORKDIR /api
RUN apk update && \
    apk upgrade && \
    apk add --no-cache openssl

COPY ./package*.json ./

RUN mkdir ./src
COPY ./src/ ./src/

RUN npm install
ENTRYPOINT [ "echo", "'All Done'" ]

```

---
Tags: docker-compose, node.js, dockerhub, npm
---


# Question
Title: What is the best practice for a multibranch pipeline configuration?
I'm messing around with a multibranch pipeline in jenkins. Currently we have zero pipelines in production and everything is using freestyle jobs which is very sloppy. I have my own multibranch pipeline using my name to get my branches, checking which ones have a Jenkinsfile. I'm sure there's no one great answer to this, but what is the general use of a multibranch pipeline? Do people create one per user? What is your current or your favorite setup for this?

If this isn't the place to ask a "best practice" question, please point me to a devops forum or somewhere that I might be able to ask this.

# Answer
When using the forking git workflow, each engineer has their own separate server-side repository. In Jenkins, to work with this setup, you'll need to create a separate multibranch pipeline for each engineer. This is because Jenkins looks at one repository at a time along with its branches and open pull requests when making a multibranch pipeline.

So, in a nutshell, with the forking git workflow, Jenkins needs individual multibranch pipelines for each engineer to handle their repositories, branches, and pull requests separately (if you're using the forking workflow). That's why you were able to create a multibranch pipeline with your own repo.
> 0 votes

# Answer
With traditional free-style jenkins jobs, you're limited to building only what is explicitly checked out by the build, which is typically main (or trunk if using svn). If you make a branch then you could duplicate the main job and change the checkout to get your branch instead of main. If you're branching infrequently then this might not be an issue but otherwise gets quite tedious!

With a multibranch pipeline, if a new branch has been created then a job will automatically be created for it and then run. This saves the overhead of manually creating them.

An excellent resource is the Jenkins book: https://www.jenkins.io/doc/book/pipeline/
> 1 votes

# Answer
Multi branch pipelines work great when your are using the git flow method: https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow

We define one multi branch pipeline per application. There is one pipeline documentation for all branches. Based on the branch name different artifacts are build in the end.

* Main branch builds release artifacts. This could be replaced with builds on git Tag events. The release artifacts will be deployed by fixed version numbers to the test and production environment.
* Develop branch builds Snapshot artifacts. Those will be deployed by using latest to the develop environment.
* Feature branches build also Snapshot artifacts which can be deployed to the dev environment as well.

All commit events on all branches run lint, unit test, SAST, dependency scan checks before the artifact is pushed to the artifactory. This guarantees that you don’t have to remediate a lot of findings after merging into the develop or main branch. Using different artifactories helps you reduce storage costs because you can just delete all Snapshot artifacts after 30 days.

Feel free to ask for specific details.

Cheers!
> 0 votes

---
Tags: jenkins, jenkins-pipeline, continuous-integration, continuous-deployment, continuous-delivery
---


# Question
Title: Conflicts while building AWX-EE custom image
I'm currently building AWX EE custom image with the latest available but it seems latest versions got some conflicts with pip modules. My custom image was working great in July (awx-ee version 22.X 23.X)

I used to set collections/modules I need in requirements files as below. However, I got some conflicts with pip module I didn't set (so I assume there are "internal" or managed by AWX-EE base image) and I didn't know how to get rid off them.

I use ansible-builder for all of this task.

Here's my execution-environment.yml :

```
---
version: 1

build_arg_defaults:
  EE_BASE_IMAGE: 'quay.io/ansible/awx-ee:latest'

dependencies:
  galaxy: requirements.yml
  python: requirements.txt

additional_build_steps:
  prepend: |
    RUN cat /etc/os-release

  append:
   RUN whoami

```

Here's the content of requirements.yml

```
---
collections:
-  name: nutanix.ncp
-  name: community.hashi_vault
-  name: arubanetworks.aos_switch
-  name: community.general

```

and requirements.txt :

```
hvac

```

The error message I got is the following. I read some pip modules are in conflicts but I don't know where those modules have been configured/asked to be installed. To get a workaround, I set hvac module to a very lower version than 1.2.X but error message is the same.

```
The user requested requests>=2.4.2 and ~=2.26.0
210.3     azure-core 1.28.0 depends on requests>=2.18.4
210.3     awxkit 13.0.0 depends on requests
210.3     hvac 1.2.1 depends on requests<3.0.0 and >=2.27.1
210.3
210.3 To fix this you could try to:
210.3 1. loosen the range of package versions you've specified
210.3 2. remove package versions to allow pip attempt to solve the dependency conflict
210.3
210.3 ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts
------
Dockerfile:54
--------------------
  52 |     COPY _build/requirements.txt requirements.txt
  53 |     RUN $PYCMD /output/scripts/introspect.py introspect --sanitize --user-pip=requirements.txt --write-bindep=/tmp/src/bindep.txt --write-pip=/tmp/src/requirements.txt
  54 | >>> RUN /output/scripts/assemble
  55 |
  56 |     # Final build stage
--------------------
ERROR: failed to solve: process "/bin/sh -c /output/scripts/assemble" did not complete successfully: exit code: 1

```

Also, I tried to remove azure-core cause I didn't need it ,directly in execution-environment.yml but it fails cause module isn't installed yet in the prepend part.

```
RUN python3 -m pip uninstall azure-core

```

About the solution provided by pip, **I don't know how I can loosen/remove a package version that I didn't ask for in my requirement file.**

```
210.3 1. loosen the range of package versions you've specified
210.3 2. remove package versions to allow pip attempt to solve the dependency conflict

```

I also tried to set a requirements.txt empty but still got the same kind of error.

My question is : How can I set version of pip module that seems to be embedded in AWX-EE image ?

EDIT : Finally found out the root cause : nutanix.ncp collection got a requirements.txt file that is responsible for asking 'requests' ~=2.26.0

I'll try to override that or found a workaround.

**EDIT 2:** After looking at ansible-runner debug and execution-environment.yml it seems that this file is used because collections installed are exactly the same (+ those I set in my execution-environment.yml file) but I have no idea how to override it. (for instance, delete all azure\_\* collection which I don't need)

# Answer
nutanix.ncp creates conflicts cause it's the only collection I use that specify python module version `requests~=2.26.0`

I finally create a new AWX-EE custom image without nutanix.ncp in requirements.yml =\> Building is ok.

And simply add nutanix.ncp as a requirements into my Ansible projects, collection is installed at project execution without any conflicts. Weird but working workaround.
> 1 votes

---
Tags: docker, ansible, python, awx
---


# Question
Title: Azure DevOps won't make description field read-only
I added rules to make some fields of my backlog items read-only for everyone who hasn't got a specific role. That works for all fields except the description field. The description field remains editable. Why does my rule not work on the description field? The screenshot below shows the configuration of the read-only rule.

# Answer
> 0 votes
I deleted the rule and recreated it. That fixed the issue.

---
Tags: azure
---


# Question
Title: Use buildx in Jenkinsfile for platform specific build
I'm currently building my docker image declaratively in a `Jenkinsfile` by:

```
app = docker.build("...")

```

Now I need to generate a platform specific build for ARM64. For this purpose

```
docker buildx build --platform linux/arm64 ...

```

must be used.

I there a way to keep using the `docker.build()` syntax and still generate a platform specific image?

Or do I have to rewrite the syntax, if then how?

# Answer
> 2 votes
The Jenkins Docker Pipeline plugin does not support `buildx` (yet?). Voting may help, so add yours to the issue if so inclined.

Jeremie Drouet wrote a great post "Multi-arch build and images, the simple way" which covers the underlying ways you can do it by building images first, then create a manifest to link them together.

bgulla on Github has a gist that does as much of the build as possible with pipeline using `docker.withRegistry()`. Then uses `sh` to create and push the manifest.

# Answer
> 0 votes
Here's how to do it in Jenkins only using the plugin for image naming:

```
  script {
    docker.withRegistry('my.registry.com', 'my_creds') {
      def image = docker.image("my_image:my_tag")
      sh "docker buildx create --use --name multiarch"
      sh """
      docker buildx build \
        --platform linux/amd64,linux/arm64 \
        -t ${image.imageName()} \
        --push .
      """
    }
  }

```

For this to work the Jenkins agent needs to have the QEMU binary formats registered with binfmt so the builder it creates will have all the emulators it needs. This docker container will do it:

```
docker run --rm --privileged multiarch/qemu-user-static --reset -p yes

```

This command needs to be be run on every reboot of the agent. In AWS EC-2 you can put it in an executable script in `/var/lib/cloud/scripts/per-boot/` and see the output in `/var/log/cloud-init-output.log`.

---
Tags: docker, jenkins
---


# Question
Title: How do I include a file in a helm config map
I have a helm deployment that includes a several configmaps.
I would like to add a few rather large files to it and I thought I coud put them in a an external directory

```
my-app/values.yaml
my-app/conf/my-large-file1.conf
my-app/conf/my-large-file2.conf
my-app/templates/configmaps.yaml

```

I thought I could use include but this did not work

```
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: example1
data:
  index.html: {{ include "/conf/my-large-file1.conf" . }}

```

Error:

```
template: my-app/templates/configmaps.yaml:23:17: executing "my-app/templates/configmaps.yaml" at <include "/conf/my-large-file1.conf" .>: error calling include: template: no template "/conf/my-large-file1.conf" associated with tem\
    plate "gotpl"

```

I then tried different things with `{{ $.Files.Get "conf/my-large-file1.conf" }}` but got weird scoping errors.

So my hope is that someone can tell me how to do it correctly

```
$ helm version
version.BuildInfo{Version:"v3.11.1+6.el8", GitCommit:"66bfc44f827aea6eb8e001150914170ac0d49e2d", GitTreeState:"clean", GoVersion:"go1.18.9"}

```

# Answer
This is how I ended up solving it

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-cm
data:
  app.properties: |-
    {{- tpl ($.Files.Get "conf/app.properties.tmpl") . | nindent 4 }}

  app-users.properties: |-
    {{- tpl ($.Files.Get "conf/app-users.properties.tmpl") . | nindent 4 }}

```
> 0 votes

# Answer
Try with `Files.Glob` , `Files.AsConfig` and with indentation. Similar pattern works for me -\>

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: example1
data:
{{ (.Files.Glob "conf/my-large-file1.conf").AsConfig | indent 2 }}

```

`Files.Glob` \- reference

`AsConfig` \- reference.
> 1 votes

---
Tags: helm
---


# Question
Title: Do Kubernetes cron jobs have readiness and/or liveness probes?
If I am starting a container in Kubernetes and I want to run it as a cronjob, is there a way that I can tell that the containers is ready to be ran? I have a collection of short-lived containers instead of background services. I'd like something that'd test it, prior to run, similar to how readiness probes work for services.

# Answer
The "readiness probe" is for determining whether the pod can receive traffic from the cluster. Cronjobs aren't intended to receive traffic from the cluster, so a readiness probe is pointless.

I'm having trouble correlating your subject line with the content of your question.

It is pointless to check if a container is "ready to run" for a cronjob. The container is not created until the cronjob instance is run according to the schedule.
> 0 votes

---
Tags: kubernetes
---


# Question
Title: Unable to install nodejs 20 in AWS codebuild image
I need to install nodejs 20 as part of a build process running on codebuild.

Steps to reproduce

```
- sudo apt purge nodejs -y
- sudo apt-get install -y ca-certificates curl GnuPG
- export NODE_MAJOR=20
- sudo mkdir -p /etc/apt/keyrings
- curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
- echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list
- sudo apt-get update && sudo apt-get install -y nodejs=20.8.0-1nodesource1
- echo "node version= $(node --version)"
- npm install -g npm

```

I am following the instructions described in nodejs distribution and specifying the nodejs version I want. Despite this, I keep installing nodejs 18 (from the source that comes with the image). I am using AWS image standard 7.0

Output from codebuild:

```
Preparing to unpack .../nodejs_20.8.0-1nodesource1_amd64.deb ...
Unpacking nodejs (20.8.0-1nodesource1) ...
Setting up nodejs (20.8.0-1nodesource1) ...

[Container] 2023/10/04 16:48:09 Running command echo "node version= $(node --version)"
**node version= v18.16.1**

[Container] 2023/10/04 16:48:11 Running command sudo npm install -g npm
npm ERR! code EBADENGINE
npm ERR! engine Unsupported engine
npm ERR! engine Not compatible with your version of node/npm: npm@10.2.0
npm ERR! notsup Not compatible with your version of node/npm: npm@10.2.0
npm ERR! notsup Required: {"node":"^18.17.0 || >=20.5.0"}
npm ERR! notsup Actual:   {"npm":"9.5.1","node":"v18.16.1"}

```

# Answer
You can use buildspec.yml in the configuration use, you can use as follows

```
phases:
  install:
    runtime-versions:
      nodejs: 18. x
    commands:
      - n 20

```
> 1 votes

---
Tags: node.js, ubuntu, aws-codepipeline
---


# Question
Title: AWS don't gather properly logs in non-blocking mode
I have faced a situation when some of the logs don't gather by awslogs driver.

Awslogs driver with mode = **non-blocking** and maximum buffer size = **25mb** (I have experimented with these values by looking at https://github.com/moby/moby/issues/45999). My logs' throughput even don't have rate \>25mb/s, but I lose logs when payload size is \>0.25MB (this isn't a concrete number, but sounds true). I don't understand why this happens. In the code I have 5 log statements: "args", "built request", "got response" and "successfully got available hotels", on the third one "got response" log payload body can be up to 1MB (but this statement is logging when payload body \<0.2 MB). What shall I suppose to do?

# Answer
> Log event size : 256 KB (maximum). This quota can't be changed.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch\_limits\_cwl.html
> 0 votes

---
Tags: amazon-web-services, logging
---


# Question
Title: Terraform conditions : How to have a multiple values in case of False condition?
I need to have two urls in case if my environment is nonprod(sit,uat,dev) and my application is Techno, otherwise only one url. In order to do that, I have made every string into a list here, so that the returned value is consistent. Otherwise I get `The true and false result expressions must have consistent types` error.

Now, with this, I am getting this error :

`Inappropriate value for attribute callback_urls: element 0: string required.`

How to achieve this ?

```
callback_urls = [var.target_infra == "prod" ? (var.application_name == "techno" ? ["https://${local.domain_name}.${local.pe_subdomain_name[var.target_infra]}/auth"] : ["https://${local.domain_name}.${local.pe_subdomain_name[var.target_infra]}/setup/auth"]) : (var.application_name == "techno" ? ["https://${local.domain_name}.${var.environment}.${local.pe_subdomain_name[var.target_infra]}/auth", "http://localhost:3000/auth"] : ["https://${local.domain_name}.${var.environment}.${local.pe_subdomain_name[var.target_infra]}/setup/auth"])]

```

# Answer
ChatGPT answers :)

```
callback_urls = concat(
  var.target_infra == "prod" ?
    (var.application_name == "techno" ?
      ["https://${local.domain_name}.${local.pe_subdomain_name[var.target_infra]}/auth"] :
      ["https://${local.domain_name}.${local.pe_subdomain_name[var.target_infra]}/setup/auth"]
    ) :
    (var.application_name == "techno" ?
      [
        "https://${local.domain_name}.${var.environment}.${local.pe_subdomain_name[var.target_infra]}/auth",
        "http://localhost:3000/auth"
      ] :
      ["https://${local.domain_name}.${var.environment}.${local.pe_subdomain_name[var.target_infra]}/setup/auth"]
    )
  )

```
> 0 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: Terraform - No Infrastructure Changes Detected
I have deployed a Cloudwatch Log Group community module to my terraform stack (sourced from https://github.com/cloudposse/terraform-aws-components) but I have spent more than a day trying to resolve this, unsuccessfully.

When I run terraform plan, the new component is not showing as a new piece of infrastructure.

```
$ terraform plan

```

```
No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration and found no differences, so no changes are needed.

```

Could anyone please just scan my configuration to see if there are things that could be missing or misconfigured that would cause terraform from ignoring the new component. Getting terraform to run without error has been a struggle which I have just resolved but this in a new level that doesn't make sense. My configuration is set as follows:

**test-stage-eu-west-1.yaml**

```
    ec2_session_manager_log:
      component: cloudwatch_logs
      vars:
        context:
          enabled: true
          name: "ssm-logs"
        enabled: true
        name: "ec2-ssm-log-name"
        retention_in_days: 7
        kms_key_arn: "arn:aws:kms:eu-west-1:12345667890:key/xxxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxx"
        principals: "ec2.amazonaws.com"
        additional_permissions: []
        stream_names: [ "app1", "app2" ]
        region: eu-west-1

    cloudwatch_logs:
      vars:
        context:
          enabled: false

```

**cloudwatch-logs module: ./module/cloudwatch-logs/main.tf**

```
locals {
  enabled = module.this.enabled
}

data "aws_caller_identity" "current" {
  count = local.enabled ? 1 : 0
}

data "aws_partition" "current" {
  count = local.enabled ? 1 : 0
}

module "logs" {
  source  = "cloudposse/cloudwatch-logs/aws"
  version = "0.6.5"

  name                   = var.name
  stream_names           = var.stream_names
  retention_in_days      = var.retention_in_days
  principals             = var.principals
  additional_permissions = var.additional_permissions
  kms_key_arn            = var.kms_key_arn
}

module "kms_key_logs" {
  source  = "cloudposse/kms-key/aws"
  version = "0.12.1"

  description             = "KMS key for CloudWatch Logs"
  deletion_window_in_days = 10
  enable_key_rotation     = true
  alias                   = "alias/${module.this.id}"
  policy                  = join("", data.aws_iam_policy_document.kms.*.json)

}

data "aws_iam_policy_document" "kms" {
  count = local.enabled ? 1 : 0

  statement {
    sid    = "EnableRootUserPermissions"
    effect = "Allow"

    actions = [
      "kms:Create*",
      "kms:Describe*",
      "kms:Enable*",
      "kms:List*",
      "kms:Put*",
      "kms:Update*",
      "kms:Revoke*",
      "kms:Disable*",
      "kms:Get*",
      "kms:Delete*",
      "kms:Tag*",
      "kms:Untag*",
      "kms:ScheduleKeyDeletion",
      "kms:CancelKeyDeletion"
    ]

    resources = [
      "*"
    ]

    principals {
      type = "AWS"

      identifiers = [
        "arn:${join("", data.aws_partition.current[*].partition)}:iam::${join("", data.aws_caller_identity.current[*].account_id)}:root"
      ]
    }
  }

  statement {
    sid    = "Allow CloudWatch to Encrypt with the key"
    effect = "Allow"

    actions = [
      "kms:Encrypt*",
      "kms:Decrypt*",
      "kms:ReEncrypt*",
      "kms:GenerateDataKey*",
      "kms:Describe*"
    ]

    resources = [
      "*"
    ]

    principals {
      type = "Service"

      identifiers = [
        "logs.${var.region}.amazonaws.com",
      ]
    }
  }
}

```

**Root: ./main.tf:**

```
module "cloudwatch-logs" {
  source  = "./modules/cloudwatch-logs"
  for_each = local.cloudwatch_logs_components

  name                   = try(each.value.vars.name, null)
  stream_names           = try(each.value.vars.stream_names, null)
  retention_in_days      = try(each.value.vars.retention_in_days, null)
  aws_service_principal  = try(each.value.vars.principal, null)
  additional_permissions = try(each.value.vars.additional_permissions, null)
  region                 = try(each.value.vars.region, null)
  kms_key_arn            = try (each.value.vars.region, null)
  context                = try(each.value.vars.context, null)

}

```

# Answer
Your locales or data objects probably evaluate to nothing. So when you run your for\_each statement, it is like running a loop on an empty list, nothing happens. I would check the local.cloudwatch\_logs\_components to see if there is data in it.
> 0 votes

---
Tags: amazon-web-services, terraform, yaml, aws-cli, terraform-module
---


# Question
Title: How do I "Dockerise" a client/server setup using socket.io?
Forgive the vague question title, I'm a complete beginner and I lack the terminology to pose my question more specifically.

I've been developing a simple application that consists of a game server (using socket.io, so WebSockets) and a game client, which is just an HTTP server. I've finished developing locally and would now like to host it on the servers of my student organisation; seeing as that machine already has Docker installed I figured that would be the easiest solution.

Concretely, in local development I've set the WebSockets server to use port 3141 and the HTTP server to use port 5173. I tried my hand at using `docker compose` and came up with the following configuration:

```
services:
  server:
    build: .
    ports:
      - 3141
  frontend:
    build: ./kwispel-client
    ports:
      - 5173

```

Using `docker inspect kwispel-server-server-1` I can then manually figure out the IP address of the HTTP server and connect to it. However, container naming resolution doesn't work; I need that for

* telling the client side of socket.io where to connect to, and passing in `http://server:3141` doesn't work;
* telling the server side of socket.io which CORS headers to allow, and passing in `http://frontend:5173` doesn't work Locally the project works fine using `localhost` instead of `server` or `frontend`.

If I try defining a shared network, like such:

```
services:
  server:
    build: .
    ports:
      - 3141
    networks:
      kwispel-server_gedeeldNetwerk:
        ipv4_address: 172.16.238.10
  frontend:
    build: ./kwispel-client
    ports:
      - 5173
    networks:
      kwispel-server_gedeeldNetwerk:
        ipv4_address: 172.16.238.20
networks:
  kwispel-server_gedeeldNetwerk:
    driver: bridge
    ipam:
      config:
        - subnet: "172.16.238.0/24"

```

so that I can have fixed IP addresses, I can't even connect to the HTTP server on that address.

Am I missing something obvious? Do I need to pass an extra flag to `docker-compose up` to expose the ports I need? I'm at a complete loss. Any pointers in the right direction are much appreciated.

# Answer
Looks like you need a `Dockerfile` to build your image. What I like to do is build the image, first, then reference that image in `docker-compose.yml` such that I am only using docker-compose for running the containers.

Here is a good tutorial with an example of a `Dockerfile` that can help: https://dockerize.io/guides/node-socket-giude

Also, have a look at existing Docker images that have socket.io support: https://hub.docker.com/search?q=socket.io
> 1 votes

---
Tags: docker, docker-compose
---


# Question
Title: Why does my Docker composer not work with volume?
I'm trying to create a volume to be shared among the containers, and this volume binds to a location of my host, the problem is that every time I run the docker-compose up gives the following error:

```
In file '. \ docker-compose.yml', service 'volumes' must be a mapping not an array.

```

and so far I didn't understand why. Following is the compose file:

```

version : '3.4'

services:
    testando-volume-compartilhado-a:
      image: carloshenriquecarniatto/teste:latest

    volumes:
       - D:\App:/app
    ports:
        - "10001:80"
        - "44378:443"

```

# Answer
It seems there is a spacing issue in your yaml

You should change the compose file to

```
version : '3.4'

services:
    testando-volume-compartilhado-a:
      image: carloshenriquecarniatto/teste:latest

    volumes:
        - D:\App:/app  # <- extra space here
    ports:
        - "10001:80"
        - "44378:443"

```
> 3 votes

# Answer
It is a spacing issue, but because of indentation.

`volumes` and `ports` should be at the same level of image

```
version : '3.4'

```

services: testando-volume-compartilhado-a: image: carloshenriquecarniatto/teste:latest

```
  volumes:
     - D:\App:/app
  ports:
      - "10001:80"
      - "44378:443"

```

I am not sure about the Windows volume mapping (Linux only here)...but the indentation was off for sure!
> 0 votes

---
Tags: docker, windows, volume
---


# Question
Title: How do I get a secret into a property file with openshift and kubernetes?
I am new to openshift and kubernetes.

I have a secret in openshift and made it avaliable to the pod as an environment variable.

What is the *best practice* to get that value into a properies file (java) before the containers service starts?

Is there a better way than using `sed` on the propertyfile before startup?

# Answer
> 1 votes
There are two ways you can manage this :

* Like you said, make the secret data available trough environnement variables and run `sed` or `envsubst` on the file before startup.
* Or, you could create a secret that would hold your whole PropertyFile content, and then use that secret trough a `volume` to mount its content as a file in your Pod (see this example in the documentation for more informations). But since you only have one secret data in your PropertyFile, this might be overkill, depending on your setup.

I wouldn't know what option is the best in your situation, but these the one I would suggest.

---
Tags: kubernetes, openshift, secret-management
---


# Question
Title: Kubernetes worker node not able to reach POD network CIDR after reboot
Really looking for some advice and help, I have run out of ideas. Basically everything was working fine, 1 master and 3 workers. One Worker node was turned off to increase CPU and RAM resources and once turned back on, PODs running on this worker were not able to reach any other PODs and also DNS resolving was not working, so no external network connection also was working, e.g. google.com etc. Here is a list of POD cidrs

```
$ kubectl get nodes -o json | jq -r '.items[].spec.podCIDR' | sort
Master - 10.244.0.0/24
Worker1 - 10.244.1.0/24
Worker2 - 10.244.2.0/24
Worker3 - 10.244.5.0/24

```

The one which was restarted and stopped working is **Worker3**. When running dnsutils container on this node and trying some debuging

```
root@dnsutils:/# nslookup kubernetes.default
;; connection timed out; no servers could be reached

```

Firewalld has been disabled and turned off. Also when trying to ping all the networks from Worker3, only 10.244.1.0 and 10.244.5.0 (itself) is pinging

```
root@dnsutils:/# ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1043ms

root@dnsutils:/# ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=63 time=0.721 ms
64 bytes from 10.244.1.0: icmp_seq=2 ttl=63 time=0.947 ms
^C
--- 10.244.1.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1041ms
rtt min/avg/max/mdev = 0.721/0.834/0.947/0.113 ms
root@dnsutils:/# ping 10.244.2.0
PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
^C
--- 10.244.2.0 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1024ms

root@dnsutils:/# ping 10.244.5.0
PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
64 bytes from 10.244.5.0: icmp_seq=1 ttl=64 time=0.116 ms
64 bytes from 10.244.5.0: icmp_seq=2 ttl=64 time=0.142 ms
^C
--- 10.244.5.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1038ms
rtt min/avg/max/mdev = 0.116/0.129/0.142/0.013 ms

```

I believe the issue is because it cant reach coredns service/pods, which both are currently on 10.244.0.35 and 10.244.0.36 IPs

```
NAME                                                                 READY   STATUS    RESTARTS         AGE     IP              NODE        NOMINATED NODE   READINESS GATES
calico-kube-controllers-7b564c7d85-rkc98                             1/1     Running   0                278d    10.244.0.30     Master       <none>           <none>
canal-5cffv                                                          2/2     Running   4 (378d ago)     630d    10.203.8.167    Worker3      <none>           <none>
canal-p822m                                                          2/2     Running   2 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
canal-scdrs                                                          2/2     Running   4 (351d ago)     630d    10.203.40.166   Worker2      <none>           <none>
canal-xkx7f                                                          2/2     Running   0                5h16m   10.203.8.174    Worker4      <none>           <none>
coredns-cc4c6799f-4tjbd                                              1/1     Running   0                11h     10.244.0.36     Master       <none>           <none>
coredns-cc4c6799f-bkq8n                                              1/1     Running   0                11h     10.244.0.35     Master       <none>           <none>
etcd-Master                                                          1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
kube-apiserver-Master                                                1/1     Running   6 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
kube-controller-manager-Master                                       1/1     Running   10 (265d ago)    392d    10.203.8.165    Master       <none>           <none>
kube-proxy-9n4wc                                                     1/1     Running   1 (474d ago)     630d    10.203.8.165    Master       <none>           <none>
kube-proxy-rtccq                                                     1/1     Running   2 (351d ago)     630d    10.203.40.166   Worker2      <none>           <none>
kube-proxy-vtr55                                                     1/1     Running   0                5h16m   10.203.8.174    Worker4      <none>           <none>
kube-proxy-w9wkk                                                     1/1     Running   2 (378d ago)     630d    10.203.8.167    Worker3      <none>           <none>
kube-scheduler-Master                                                1/1     Running   326 (265d ago)   630d    10.203.8.165    Master       <none>           <none>
kube-state-metrics-8945855d-pvqqt                                    1/1     Running   0                139d    10.244.2.89     Worker3      <none>           <none>
metrics-server-7cf8b65d65-qpnzm                                      1/1     Running   2 (6h43m ago)    139d    10.244.2.83     Worker3      <none>           <none>

```

Running dnsutils on Worker2 for example

```
root@dnsutils2:/# nslookup kubernetes.default
Server:     10.96.0.10
Address:    10.96.0.10#53

Name:   kubernetes.default.svc.cluster.local
Address: 10.96.0.1

root@dnsutils2:/# ping 10.244.0.0
PING 10.244.0.0 (10.244.0.0) 56(84) bytes of data.
64 bytes from 10.244.0.0: icmp_seq=1 ttl=63 time=0.787 ms
64 bytes from 10.244.0.0: icmp_seq=2 ttl=63 time=0.835 ms
^C
--- 10.244.0.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1061ms
rtt min/avg/max/mdev = 0.787/0.811/0.835/0.024 ms
root@dnsutils2:/# ping 10.244.1.0
PING 10.244.1.0 (10.244.1.0) 56(84) bytes of data.
64 bytes from 10.244.1.0: icmp_seq=1 ttl=64 time=0.129 ms
64 bytes from 10.244.1.0: icmp_seq=2 ttl=64 time=0.152 ms
^C
--- 10.244.1.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1021ms
rtt min/avg/max/mdev = 0.129/0.140/0.152/0.016 ms
root@dnsutils2:/# ping 10.244.2.0
PING 10.244.2.0 (10.244.2.0) 56(84) bytes of data.
64 bytes from 10.244.2.0: icmp_seq=1 ttl=63 time=1.03 ms
64 bytes from 10.244.2.0: icmp_seq=2 ttl=63 time=0.933 ms
^C
--- 10.244.2.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 0.933/0.984/1.036/0.060 ms
root@dnsutils2:/# ping 10.244.5.0
PING 10.244.5.0 (10.244.5.0) 56(84) bytes of data.
64 bytes from 10.244.5.0: icmp_seq=1 ttl=63 time=0.787 ms
64 bytes from 10.244.5.0: icmp_seq=2 ttl=63 time=0.995 ms
^C
--- 10.244.5.0 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.787/0.891/0.995/0.104 ms

```

Cluster was initialized running below command

```
kubeadm init --apiserver-advertise-address=10.203.8.165 --pod-network-cidr=10.244.0.0/16

```

And for network cni plugin flannel-canal was used

```
kubectl apply -f https://projectcalico.docs.tigera.io/manifests/canal.yaml

```

I would really appreciate any help on where to start debugging this. Basically this Worker3 was working just fine before a restart and admins say that nothing was changed from network side or security groups etc. This is running on premises on VMware VMs.

I have also tried `kubeadm reset` and then `iptables -F && iptables -t nat -F && iptables -t mangle -F && iptables -X` , rebooting the node and then `kubeadm join` but still the issue persists. No idea why this specific Worker3 wont ping anything besides 10.244.1.0 and 10.244.5.0 (itself)

I am confused about why wouldnt node be able to ping 10.244.2.0 network for example, but is able to ping 10.244.1.0 although there is iptables

```
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT

```

and also routes

```
# route -n
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
0.0.0.0         10.203.8.254    0.0.0.0         UG    100    0        0 ens192
10.203.8.0      0.0.0.0         255.255.255.0   U     100    0        0 ens192
10.244.0.0      10.244.0.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.1.0      10.244.1.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.2.0      10.244.2.0      255.255.255.0   UG    0      0        0 flannel.1
10.244.5.2      0.0.0.0         255.255.255.255 UH    0      0        0 cali952544557fa
10.244.5.3      0.0.0.0         255.255.255.255 UH    0      0        0 calic15b507754a
10.244.5.4      0.0.0.0         255.255.255.255 UH    0      0        0 calif987de568c4
10.244.5.7      0.0.0.0         255.255.255.255 UH    0      0        0 cali969530ca43d
10.244.5.8      0.0.0.0         255.255.255.255 UH    0      0        0 calib3c61c3cba9

```

# Answer
> 0 votes
So after deleting all canal PODs and letting them recreate, everything started to work again as it should. Seems like the recreation of those PODs somehow recreated routes and added the Worker node. Not sure why it didnt work right after a worker reboot, later when removing and re-joining this worker to the cluster, it got assigned a new CIDR, after canal POD recreation it started to work.

# Answer
> 0 votes
Do you have vxlan in canal.yaml? Then change that to host-gw and see if it fixes the problem. flannel will communicate new routes to the kernel with netlink sockets then.

```
  net-conf.json: |
    {
      "Network": "10.244.0.0/16",
      "Backend": {
        "Type": "vxlan" # CHANGE TO host-gw
      }
    }

```

More info here.

kubernetes access nodeport fails, master cannot reach workers

and on the flannel github.

https://github.com/flannel-io/flannel

If it still doesn't work, try adding the specific network interface flannel must work on. If you dont it will claim the first interface it finds.

```
 - name: kube-flannel
          image: quay.io/coreos/flannel:v0.9.1
          command: [ "/opt/bin/flanneld", "--ip-masq", "--kube-subnet-mgr" ]

```

add --iface=iface\_you\_want to "command"

---
Tags: kubernetes
---


# Question
Title: Permission Issue when creating an ECS Service with CloudFormation
I am trying to create an ECS service in a cloud formation json. Here is the role I created (I know that it has additional permissions, I am using one role for now and will break them up later)

```
 "LambdaBrownlowRole": {
      "Type": "AWS::IAM::Role",
      "Properties": {
        "RoleName": "LambdaBrownlowRole",
        "AssumeRolePolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Principal": {
                "Service": [
                  "lambda.amazonaws.com",
                  "events.amazonaws.com",
                  "ecs-tasks.amazonaws.com",
                  "ecs.amazonaws.com"
                ]
              },
              "Action": "sts:AssumeRole"
            }
          ]
        },
        "Policies": [
          {
            "PolicyName": "LambdaBrownlowPolicy",
            "PolicyDocument": {
              "Version": "2012-10-17",
              "Statement": [
                {
                  "Effect": "Allow",
                  "Action": [
                    "s3:GetObject",
                    "s3:PutObject",
                    "s3:ListBucket",
                    "s3:DeleteObject"
                  ],
                  "Resource": [
                    "arn:aws:s3:::afl-game-data/*",
                    "arn:aws:s3:::afl-game-data"
                  ]
                },
                {
                  "Effect": "Allow",
                  "Action": [
                    "dynamodb:PutItem",
                    "dynamodb:GetItem",
                    "dynamodb:Query",
                    "dynamodb:UpdateItem",
                    "dynamodb:Scan",
                    "dynamodb:BatchWriteItem",
                    "dynamodb:BatchGetItem"
                  ],
                  "Resource": "*"
                },
                {
                  "Effect": "Allow",
                  "Action": [
                    "logs:CreateLogGroup",
                    "logs:CreateLogStream",
                    "logs:PutLogEvents",
                    "logs:DescribeLogStreams"
                  ],
                  "Resource": "*"
                },
                {
                  "Effect": "Allow",
                  "Action": [
                    "ecs:CreateService",
                    "ecs:DescribeServices",
                    "ecs:UpdateService"
                  ],
                  "Resource": "*"
                }
              ]
            }
          }
        ]
      }
    },

```

Here is the json snippet for the resources in question.

```
 "ECSBrownlowTaskDefinition": {
      "Type": "AWS::ECS::TaskDefinition",
      "Properties": {
        "Family": "afl-brownlow-scrape",
        "ExecutionRoleArn": {
          "Fn::GetAtt": [
            "LambdaBrownlowRole",
            "Arn"
          ]
        },
        "Memory": "512",
        "Cpu": "256",
        "TaskRoleArn": {
          "Fn::GetAtt": [
            "LambdaBrownlowRole",
            "Arn"
          ]
        },
        "RequiresCompatibilities": [
          "FARGATE"
        ],
        "NetworkMode": "awsvpc",
        "ContainerDefinitions": [
          {
            "Name": "BrownlowScrape",
            "Image": {
              "Ref": "ECRRepositoryUriParameter"
            },
            "Essential": true,
            "Environment": [
              {
                "Name": "YEAR_TO_QUERY",
                "Value": {
                  "Ref": "YearToQueryParameter"
                }
              },
              {
                "Name": "BUCKET_NAME",
                "Value": {
                  "Ref": "BucketNameParameter"
                }
              },
              {
                "Name": "DATA_PATH",
                "Value": {
                  "Ref": "DataPathParameter"
                }
              }
            ],
            "LogConfiguration": {
              "LogDriver": "awslogs",
              "Options": {
                "awslogs-group": {
                  "Ref": "AFLScrapeLogGroup"
                },
                "awslogs-region": {
                  "Ref": "AWS::Region"
                },
                "awslogs-stream-prefix": "ecs"
              }
            }
          }
        ]
      }
    },
    "AFLScrapeLogGroup": {
      "Type": "AWS::Logs::LogGroup",
      "Properties": {
        "LogGroupName": "/ecs/afl-brownlow-scrape"
      }
    },
    "ECSServiceScrapeAFL": {
      "Type": "AWS::ECS::Service",
      "Properties": {
        "Cluster": "afl-brownlow-cluster",
        "ServiceName": "ServiceECSBrownlow",
        "TaskDefinition": {
          "Ref": "ECSBrownlowTaskDefinition"
        },
        "LaunchType": "FARGATE",
        "DesiredCount": 0,
        "Role": {
          "Fn::GetAtt": ["LambdaBrownlowRole", "Arn"]
        }
      }
    },
    "ECSScrapeScheduledRule": {
      "Type": "AWS::Events::Rule",
      "Properties": {
        "Name": "ecs-brownlow-scrape-weekly",
        "Description": "Scheduled rule for ECS brownlow-scrape",
        "RoleArn": {
          "Fn::GetAtt": [
            "LambdaExecutionRole",
            "Arn"
          ]
        },
        "ScheduleExpression": "cron(0/10 * ? * * *)",
        "State": "ENABLED",
        "Targets": [
          {
            "Arn": {
              "Fn::GetAtt": [
                "ECSServiceScrapeAFL",
                "Arn"
              ]
            },
            "Id": "ecs-brownlow-scrape",
            "EcsParameters": {
              "TaskDefinitionArn": {
                "Ref": "ECSBrownlowTaskDefinition"
              },
              "TaskCount": 1,
              "LaunchType": "FARGATE"
            }
          }
        ]
      }
    }

```

On the creation of ECSServiceScrapeAFL, I get the following error:

Resource handler returned message: "Invalid request provided: CreateService error: Access denied (Service: AmazonECS; Status Code: 400; Error Code: AccessDeniedException; Request ID: XXX; Proxy: null)" (RequestToken: XXX, HandlerErrorCode: InvalidRequest).

Does anyone know what I am missing?

# Answer
According to https://docs.aws.amazon.com/AmazonECS/latest/APIReference/CommonErrors.html

AccessDeniedException You do not have sufficient access to perform this action.

that means that is is your role (or the role which Cloudformation is running under) which does not have the permissions. You can go to Cloudtrail and look up your request and there will be more details (which exact action did fail with which parameters). You should be able to figure out what is missing. Sometimes it is about some failing conditions. E.g. if your environment enforces resources to have particular tags. Would be good if you can update the question with the cloudtrail output.
> 0 votes

---
Tags: amazon-web-services, amazon-ecs, aws-cloudformation
---


# Question
Title: How to get the actual memory usage of a container
I run `docker stats` and get:

```
CONTAINER ID   NAME      CPU %    MEM USAGE / LIMIT     MEM %   NET I/O           BLOCK I/O         PIDS
b5ebb4b467f6   haproxy   0.06%    13.84MiB / 78.57GiB   0.02%   35.2MB / 59.2MB   12.1MB / 6.12MB   33

```

So this tells me that the haproxy container consumes 13MB.

But when I run `ps` on the host, I see 2 haproxy processes which occupy 13MB and 282MB each:

```
$ ps -eo size,command | grep -E 'SIZE|haproxy'
  SIZE COMMAND
 13772 haproxy -W -db -f /usr/local/etc/haproxy/haproxy.cfg
282764 haproxy -W -db -f /usr/local/etc/haproxy/haproxy.cfg

```

My 2nd test with `docker stats` is this:

```
CONTAINER ID   NAME            CPU %   MEM USAGE / LIMIT     MEM %    NET I/O           BLOCK I/O         PIDS
c3fbca04c3a4   R.processing.3  0.22%   248.8MiB / 78.57GiB   0.31%    3.33MB / 1.73MB   18.5MB / 0B       36
88b39d761182   R.processing.2  0.20%   243.9MiB / 78.57GiB   0.30%    3.33MB / 1.73MB   5.98MB / 0B       42
b5065a2f3ffd   R.processing.1  0.25%   239.7MiB / 78.57GiB   0.30%    3.32MB / 1.72MB   1.53MB / 0B       42

```

Which means that the 3 `R` containers consume 244MB + 243MB + 239MB. But on the host I get 4.7GB + 4.7GB + 4.7GB:

```
$ ps -eo size,command | grep -E 'SIZE|R'
   SIZE COMMAND
4708864 /opt/R/4.3.2/lib/R/bin/exec/R --no-save
4707860 /opt/R/4.3.2/lib/R/bin/exec/R --no-save
4707680 /opt/R/4.3.2/lib/R/bin/exec/R --no-save

```

So which one do I believe? Which is correct?

# Answer
`SIZE` is a bit misleading. Quotes are taken from `man ps`:

> SIZE approximate amount of swap space that would be required if the process were to dirty all writable pages and then be swapped out. This number is very rough!

So this only counts writable pages, which may not be really the size taken when the pages are copy on write.

You may use `ps -eo rss,command` for something a bit more accurate

> RSS resident set size, the non-swapped physical memory that a task has used (in kilobytes). (alias rssize, rsz).

For comparison you can do a `ps -eo SIZE,RSS,VSZ` and check the differences between each value (Swappable, really used, potentially adressable/virtual memory)

With a small caveat:

> The SIZE and RSS fields don't count some parts of a process including the page tables, kernel stack, struct thread\_info, and struct task\_struct. This is usually at least 20 KiB of memory that is always resident.

I don't have a satisfactory answer to give about your docker info vs ps. I'm not sure if the docker commands only takes the first container command (entrypoint) or if it should report the whole cgroup usage. (This is assuming you have only one haproxy container running)
> 2 votes

---
Tags: docker
---


# Question
Title: How to modify a Jenkins pipeline job's property programmatically?
We set up a Jenkins job in multiple folders that has the following "Choice Parameter" property.

How can I modify the property programmatically to add more choices? For example, I want to add a `feature` and `devops` choice. I know I can just got to each individual job and add them, but that's just not DevOpsy. Thanks!

# Answer
A common approach is to define your pipelines as code. That way they will always be up-to-date with the code they build, and much easier to maintain.

You will end up with a Jenkinsfile in your repository that represents your pipeline along with all the parameters, steps, agent requirements, environment variables, triggers, and other elements.

There are several built-in types of parameters, with many others are available as plugins.

The `choices` parameter would look like this, with the first element of the list being the default value:

```
parameters
    {
        choice name: 'ENVIRONMENT',
            choices: [
                'sandbox',
                'staging',
                'production',
                'feature',
                'devops'
            ],
            description: 'The environment to deploy the application to'
    }

```

If you want to use a single Jenkinsfile for multiple jobs, you can either do the same but manually checkout the desired repository in the first stage, or explore the abilities provided by shared libraries. Both options have an obvious drawback - if the application code requires changes to the pipeline, you will have to make them in the pipeline repository first. And to apply them gradually (e.g. modify the apps one by one if required), you will need to maintain two or more pipeline versions simultaneously and make sure they always work correctly.

Note that unlike other CI servers, Jenkins is unable to automatically pick up the pipeline parameters changes until the start of the next build, so you may need to implement some workaround for that, i.e. adding a `stage` that aborts the build if some dummy parameter value is changed.
> 0 votes

---
Tags: jenkins, jenkins-pipeline, groovy
---


# Question
Title: Azure B2C - Restrict access for default domain after creating custom domain
I have created an Azure B2C and we have a few user flows. These user flows use the default domain of `b2cinstance.b2clogin.com`. This domain doesn't go through a WAF so I decided to create an Azure Front Door with WAF Policy to set up a custom domain for my Azure B2C instance.

I have set this all up and it is working as expected. We now have the domain `login.mywebsite.com`, which forwards to origin `b2cinstance.b2clogin.com`. for our user flows which is working as expected and can see the requests in the Azure Front Door.

My question if I try to use the domain `b2cinstance.b2clogin.com`, the user flow still works. How do I restrict use of this domain

# Answer
> 0 votes
You need to put in place a custom Azure B2C policy to block access to the original domain

https://learn.microsoft.com/en-us/azure/active-directory-b2c/custom-domain?pivots=b2c-custom-policy#optional-block-access-to-the-default-domain-name

---
Tags: azure
---


title: Kubernetes Ingress route path pod name

text:
# Question
Title: Kubernetes Ingress route path pod name
I have a Docker container that is used in Kubernetes, but I've encountered a task to implement routing using the pod name. Here are my Deployment and Service. There's also an Ingress, but it's not working. Pay attention to the replicas. I need each replica (pod) to have individual access like this: http://IP\_INGRESS/POD\_NAME/api -\> PROXY\_PASS -\> POD\_IP:3303 and http://IP\_INGRESS/POD\_NAME/novnc -\> PROXY\_PASS -\> POD\_IP:8060.

```
apiVersion: v1
kind: Service
metadata:
  name: app-service
  labels:
    app: app-pod
spec:
  selector:
    app: app-pod
  ports:
    - protocol: TCP
      name: port-6080
      port: 6080
      targetPort: 6080
    - protocol: TCP
      name: port-3033
      port: 3033
      targetPort: 3033
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
  labels:
    app: app-pod
spec:
  replicas: 2
  selector:
    matchLabels:
      app: app-pod
  template:
    metadata:
      labels:
        app: app-pod
    spec:
      containers:
        - name: app-container
          image: xxx/xxxx
          resources:
            limits:
              memory: "2Gi"
          ports:
            - containerPort: 6080
            name: novnc
            - containerPort: 3033
            name: api
      imagePullSecrets:
        - name: dockerhub-secret
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  namespace: default
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
    kubernetes.io/ingress.class: "nginx"
spec:
  rules:
    - host: ""
      http:
        paths:
          - path: /(.*)/api
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 3033
          - path: /(.*)/novnc
            pathType: Prefix
            backend:
              service:
                name: app-service
                port:
                  number: 6080

```

# Answer
> 0 votes
According to the documentation, if you want to use regular-expression rewriting in your paths, you you be using `pathType: ImplementationSpecific` rather than `pathType: Prefix`. In fact, if I submit your manifests locally (against k8s 1.27), I get the warning:

```
Warning: path /(.*)/api cannot be used with pathType Prefix
Warning: path /(.*)/novnc cannot be used with pathType Prefix

```

You want:

```
spec:
  rules:
  - host: ""
    http:
      paths:
      - backend:
          service:
            name: app-service
            port:
              number: 3033
        path: /(.*)/api
        pathType: ImplementationSpecific
      - backend:
          service:
            name: app-service
            port:
              number: 6080
        path: /(.*)/novnc
        pathType: ImplementationSpecific

```

With that one change, the Ingress appears to work as expected. Here are my test manifests (you can find these online at https://github.com/larsks/devops-example-18709):

```
apiVersion: v1
kind: Service
metadata:
  labels:
    app: so-example
  name: app-service
spec:
  ports:
  - name: port-6080
    port: 6080
    protocol: TCP
    targetPort: 6080
  - name: port-3033
    port: 3033
    protocol: TCP
    targetPort: 3033
  selector:
    app: so-example
  type: NodePort
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: so-example
  name: app-deployment
spec:
  replicas: 2
  selector:
    matchLabels:
      app: so-example
  template:
    metadata:
      labels:
        app: so-example
    spec:
      containers:
      - env:
        - name: WHOAMI_PORT_NUMBER
          value: "6080"
        - name: WHOAMI_NAME
          value: novnc
        image: docker.io/traefik/whoami
        name: app-novnc
        ports:
        - containerPort: 6080
          name: novnc
      - env:
        - name: WHOAMI_PORT_NUMBER
          value: "3033"
        - name: WHOAMI_NAME
          value: api
        image: docker.io/traefik/whoami
        name: app-api
        ports:
        - containerPort: 3033
          name: api
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/use-regex: "true"
  labels:
    app: so-example
  name: my-ingress
spec:
  rules:
  - host: ""
    http:
      paths:
      - backend:
          service:
            name: app-service
            port:
              number: 3033
        path: /(.*)/api
        pathType: ImplementationSpecific
      - backend:
          service:
            name: app-service
            port:
              number: 6080
        path: /(.*)/novnc
        pathType: ImplementationSpecific

```

With these deployed in my local environment, http://cluster.example.com/something/api connects to port 3033 in the `app-deployment` pod, and http://cluster.example.com/something/novnc connects to port 6080.

---
Tags: kubernetes
---