title: Setting up Keycloak with Kong v5.1

text:
# Question
Title: Setting up Keycloak with Kong v5.1
I found a tutorial how to setup kong with two open source plugins to get keycloak working with kong, but unfortunately it is 2 years old and I ran in troubles.

Most of the steps still worked and I was able to install the plugin versions in the tutorial via `luarocks`. unfortunately, the kong did not restart after configuring it to work with the plugins:

`plungins = bundeld,oidc,jwt-keycloak`

running in the error, that the `kong.plugins.base_plugin` cannot be found:

```
Error: /usr/local/share/lua/5.1/kong/tools/utils.lua:708: error loading module 'kong.plugins.oidc.handler':
/usr/local/share/lua/5.1/kong/plugins/oidc/handler.lua:1: module 'kong.plugins.base_plugin' not found:No LuaRocks module found for kong.plugins.base_plugin
    no field package.preload['kong.plugins.base_plugin']

```

**What I tried**

* I tried to install a `base_plugin` but lua does not find any.
* I was googling a lot, but did't quite found the same issue.

Well, I am quite stuck and was wondering if there is an other way (maybe directly with the oAuth plugin?), or how I can fix this issue. Also I don't really know how debug that issue and am happy for ideas!

# Answer
> 1 votes
The root cause is BasePlugin is deprecated and removed from kong 3.x,see it here.The jwt-keycloak plugin imported BasePlugin in handler.lua, so you have to do the migration by yourself,it's easy,just fork its repo,change code,install it from source in Dockerfile。

---
Tags: api-gateway, keycloak
---


title: Persistent Volume with Kubernetes

text:
# Question
Title: Persistent Volume with Kubernetes
I have an application which consists of 2 containers which are running on an on-premise system. The responsibility of the application is to accept data from other on-premise systems and upload it to cloud. First container exposes a `store` service which accepts data and writes the data to a volume mounted on the on-premise system. There is another `upload` service which gets triggered whenever a new file is available in this volume and uploads the data to S3 in the background.

The problem I am trying to solve is to host these containers in a `K3S` cluster to provide HA capabilities. I do not want to combine `store` and `upload` into a single POD as failure detection and recovery becomes complex. I want to host each container in a separate POD. In such a scenario how do I ensure that data is not lost whenever one container fails and new POD is started? For example, lets say `store` service received 100 files and written it to the volume. `Upload` started uploading but crashed while uploading 50th file. Now K3S will ensure that new `upload` POD is started, but how do I ensure that it mounts the same volume which had 50 pending files?

# Answer
> 0 votes
To ensure that the new upload pod mounts the same volume as the previous pod and continues uploading the remaining files, you can use a persistent volume claim (PVC) to claim the volume and mount it to the new pod.

A PVC is a resource in Kubernetes that allows a pod to request and consume a persistent volume (PV) in the cluster. When you create a PVC, you can specify the size of the volume and the access mode (e.g. read-write). The PVC is then bound to a PV that meets the specified criteria.

To use a PVC to claim the volume in your scenario, you can do the following:

1. Create a PV to represent the volume on the on-premise system. The PV should specify the path to the volume on the on-premise system, as well as the access mode (e.g. read-write).
2. Create a PVC that references the PV. The PVC should specify the same size and access mode as the PV.
3. Mount the PVC to the upload pod using a volume mount in the pod's spec.

Here is an example of how you can create a PV, PVC, and pod that mounts the PVC:

```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: my-pv
spec:
  capacity:
    storage: 10Gi
  accessModes:
  - ReadWriteMany
  hostPath:
    path: /path/to/volume

---

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: my-pvc
spec:
  accessModes:
  - ReadWriteMany
  resources:
    requests:
      storage: 10Gi
  volumeName: my-pv

---

apiVersion: apps/v1
kind: Deployment
metadata:
  name: upload
spec:
  replicas: 1
  selector:
    matchLabels:
      app: upload
  template:
    metadata:
      labels:
        app: upload
    spec:
      containers:
      - name: upload
        image: upload-image
        volumeMounts:
        - name: my-volume
          mountPath: /path/to/volume
      volumes:
      - name: my-volume
        persistentVolumeClaim:
          claimName: my-pvc

```

This configuration will create a PV and PVC that represent the volume on the on-premise system and claim it for the upload pod. The pod will then mount the volume to the specified path using the PVC.

If the upload pod fails and is restarted, the PVC will still be bound to the PV and the new pod will be able to mount the same volume and continue uploading the remaining files.

I hope this helps!

# Answer
> 0 votes
I'm picturing a use-case like a log-exporter on network with a very flaky external internet connection. I take it you want to ensure that even if there are considerable delays accessing S3, your client applications are not blocked, and you also do not risk data loss from local failures (of cluster nodes or storage drives).

You will need a *Persistent Volume Claim*, which both of your pods can then mount.

Since you are using k3s you may be limited in what underlying types of persistent volume are supported, unless you install additional components.

---
Tags: docker, kubernetes, high-availability, k3s
---


# Question
Title: nginx reverse proxy on docker - problem with setup custom ip for each domain
I an have nginx reverse proxy running in a docker container. On the host server I have a service failover ip.

I want set for each domain on the nginx server to a custom ip, beacause at the moment all domains are using the main ip of my server.

How I can do that?

# Answer
First, I would argue that what you have right now is the most common configuration: your reverse proxy (nginx/traefik/haproxy/etc) listens on a single address and then routing requests to appropriate backend hosts by name. Configuring a separate external ip for each domain adds a level of complexity that may not be necessary.

However, if you really want to do that, start by adding multiple ip addresses to your host. For example, on my host, I have:

```
$ ip addr show eth0
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    altname eno2
    altname enp0s31f6
    inet 192.168.1.200/24 brd 192.168.1.255 scope global noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.201/24 brd 192.168.1.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.202/24 brd 192.168.1.255 scope global secondary noprefixroute eth0
       valid_lft forever preferred_lft forever
    inet 192.168.1.175/24 brd 192.168.1.255 scope global secondary dynamic noprefixroute eth0
       valid_lft 55548sec preferred_lft 55548sec

```

I have a sample configuration in which I am running nginx as the frontend with two different backends (named `backend1` and `backend2`).

I'm going to configure nginx so that connections to `192.168.1.201` go to `backend1` and connections to `192.168.1.202` go to `backend2`. We do that by:

1. Having nginx listen on multiple ports (one port for each backend), and then
2. Publishing these ports on different ip addresses

We start with the `nginx` configuration:

```
server {
    listen       8000;
    server_name  localhost;

    location / {
        proxy_pass http://backend1/;
    }
}

server {
    listen       8001;
    server_name  localhost;

    location / {
        proxy_pass http://backend2/;
    }
}

```

This maps connections on port 8000 to `backend1` and connections on port 8001 to `backend2`. To expose these on port 80 on different host addresses, we can create a `docker-compose.yaml` like this:

```
services:
  frontend:
    image: docker.io/nginx:mainline
    volumes:
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf
    ports:
      - 192.168.1.201:80:8000
      - 192.168.1.202:80:8001

  backend1:
    image: docker.io/containous/whoami:latest

  backend2:
    image: docker.io/containous/whoami:latest

```

This means that connections to `http://192.168.1.201` will get mapped to port 8000, and hence to `backend1`, while connections to `http://192.168.1.202` will get mapped to port 8001, and hence to `backend2`.

---

After running `docker-compose up`, I can confirm this configuration:

```
$ curl -s 192.168.1.201 | grep Host:
Host: backend1
$ curl -s 192.168.1.202 | grep Host:
Host: backend2

```
> 0 votes

---
Tags: docker, nginx
---


# Question
Title: How to verify the time zone change is correct on AWS EC2 using Ansible?
We can use the following code to set a new timezone on AWS EC2 using Ansible.

```
---
- hosts: linuxservers proxmoxservers
  remote_user: ansible
  become: yes

  tasks:

  - name: 'Set timezone to Buenos Aires'
    timezone:
      name: America/New_York

  - name: Ansible date fact example
    debug:
      msg="{{ lookup('pipe','date +%Y-%m-%d-%H-%M-%S') }}"

```

After this,

* How do we verify if the current timezone is set to the expected value?
* How to validate the timezone has been changed correctly (to the desired value)?

# Answer
**How to validate that ... has been changed correctly (to the desired value)?**

Since Ansible is a Configuration Management Tool with which you set a Desired State, such validation is part of the modules in general.

For `timezone` module – Configure timezone setting

> *It is recommended to restart `crond` after changing the timezone, otherwise the jobs may run at the wrong time.*

After that you may have a look into one the following

> *For Linux it can use `timedatectl` or edit `/etc/sysconfig/clock` or `/etc/timezone` and `hwclock`.*

or better (re-)gather Ansible Facts

```
---
- hosts: localhost
  become: false
  gather_facts: true
  gather_subset:
    - "!all"
    - "!min"
    - "date_time"

  tasks:

  - name: Show Facts date_time
    debug:
      msg: "{{ ansible_date_time }}"

```

since that will contain the timezone settings or updated timezone settings.

**Further Readings**

Please take note about
> 1 votes

---
Tags: amazon-web-services, ansible, amazon-ec2
---


# Question
Title: Does AWS CloudFront work with a Network Load Balancer?
I am aware that it is straightforward to use `AWS CloudFront` with an AWS **Application** Load Balancer (Layer 7) origin (using a certificate in each for E2E encryption). However, is it possible to use `AWS CloudFront` (with certificate) and an AWS **Network** Load Balancer (Layer 4) origin to distribute HTTP over non-standard ports (in this case, TCP 4000 and TCP 3000)?

I have implemented CloudFront with a Network Load Balancer origin. When queried, a 504 is generated by CloudFront (TCP 443 \> TCP 3000), which suggests a timeout.

I have confirmed that there are no issues with the security group and network load balancer configuration using `nc` (for example, `nc -zv lb-rswvin-XXXXX.elb.XXXXX.amazonaws.com 4000` returns `Connection to lb-rswvin-XXXXX.elb.XXXXX.amazonaws.com (X.X.X.X) 4000 port [tcp/*] succeeded!`)

I am open to alternative solutions/suggestions (e.g. a dedicated EC2 instance acting as a reverse proxy) but prefer to use a native AWS service/feature.

All infrastructure was deployed using IaC (Terraform).

# Answer
> 0 votes
Given that the Cloudfront is configured to receive external traffic on 443 and send to port 3000, and that the LB responds to a netcat socket connect the problem is on the application server

The application that is on a server in the backend group for the LB is not running correctly. Maybe it is responding really slowly. This is what a 504 usually suggests. If it was a security group then that would be 502 bad gateway, if the backend group was missing instances then that would be 503 service unavailable.

As the server is able to be a member of the backend group the health check http path must be working correctly on the server and talking back to the LB.

Look in your application server logs for 200 from the healthcheck and for any trace of the 504 requests.

There is probably an issue in the application server configuration or a bug in the under lying software

---
Tags: amazon-web-services, terraform, amazon-cloudfront
---


# Question
Title: Enable/disable resource in terraform using count
I added a piece of code to support enable/disable to the feature - I used the `count`. Now I have the following problem:

```
ERROR:
on modules/eventhub/main.tf line 110, in resource "azurerm_eventhub" "events":

 110:   namespace_name      = azurerm_eventhub_namespace.eventhub.*.name
    |----------------
    | azurerm_eventhub_namespace.eventhub is tuple with 1 element

Inappropriate value for attribute "namespace_name": string required.

# Create EventHub namespace
resource "azurerm_eventhub_namespace" "eventhub" {
  count               = var.enable_eh ? 1 : 0
  name                  = "${var.name}-ns"
  location              = var.location
  resource_group_name   = var.resource_group
  sku                   = var.sku
  capacity              = var.capacity
  dedicated_cluster_id = azurerm_eventhub_cluster.eventhub[count.index].id #for eventHub Cluster
  auto_inflate_enabled     = var.auto_inflate != null ? var.auto_inflate.enabled : null
  maximum_throughput_units = var.auto_inflate != null ? var.auto_inflate.maximum_throughput_units : null

    dynamic "network_rulesets" {
    for_each = var.network_rules != null ? ["true"] : []
    content {
      default_action = "Deny"
      dynamic "ip_rule" {
        for_each = var.network_rules.ip_rules
        iterator = iprule
        content {
          ip_mask = iprule.value
        }
      }

      dynamic "virtual_network_rule" {
        for_each = var.network_rules.subnet_ids
        iterator = subnet
        content {
          subnet_id = subnet.value
        }
      }
    }
  }

  tags = var.tags
}

resource "azurerm_eventhub_namespace_authorization_rule" "events" {
  for_each            = local.authorization_rules
  name                = each.key
  namespace_name      = azurerm_eventhub_namespace.eventhub.*.name
  resource_group_name = var.resource_group

  listen = each.value.listen
  send   = each.value.send
  manage = each.value.manage

}

resource "azurerm_eventhub" "events" {
  for_each            = local.hubs
  name                = each.key
  **namespace_name      = azurerm_eventhub_namespace.eventhub.*.name  <------ problematic line**

  resource_group_name = var.resource_group
  partition_count     = each.value.partitions
  message_retention   = each.value.message_retention

}

```

I don't know what the issue is.

# Answer
I believe you should be able to reference the namespace name like so:

`namespace_name = azurerm_eventhub_namespace.eventhub[0].name`

However, if the namespace\_name field is a required parameter on the resources you are passing it to, you might run into other issues while you are trying to set enabled to false, as the resources depend on the output of the resource you are setting to count\[0\]. If you want them all to be conditional on the creation of the namespace, i would recommend continuing down your approach with the for\_each over the local values.

If you need to the other resources to not be created unless the namespace is created you can create a condition on your local value like this (assuming map):

```
locals{
    hub = var.enable_eh ? {foo = "bar", baz = "qux"} : {}
}

```

The for\_each will omit the resources that are empty.
> 1 votes

# Answer
I think you should migrate from count to for\_each, here is a wider context and howto:

Terraform provides a 'for\_each' iterator (aka set/map or key named list) and a 'count' iterator (aka indexed list with strict ordering) which allows you to loop over elements of a list and perform multi resource deployments. What many people fail to realize is that when using indexed list with the 'count' iterator, any change to the list ordering (like removing an item) causes replacement of all items in the list, those can be important cloud resources like databases, instances, buckets etc.

High level overview here: https://www.hashicorp.com/blog/new-terraform-tutorials-on-count-and-for-each

With ‘count’, even when one of the list items needs just reconfiguration of substantial attributes, terraform might plan to destroy and/or replace the entire list of resources, making this update a clear 'no-go' in production environments. Those resources would have better support (yet not perfect) for updating its items, if the list would have been implemented using the 'for-each' iterator, simply because the difference between a simple indexed list and a set or a map is that the item names (keys) are all guaranteed to be unique, and also do not have any particular ordering.

But many times, for various reasons, one would already implement a 'count' iterator in production and face the need to migrate it later to 'for each' iterator without destroying any item and while keeping terraform states clean and synchronized with the cloud.

here is a stable workaround for anyone facing such requirement, i think this one is not well documented yet.

terraform init -var-file=./tfvars/.tfvars -reconfigure terraform state list → to grab the count list items

Change ‘count’ instructions to ‘for-each’ instructions (example):

resource "google\_container\_node\_pool" "nodes" {

* count = length(var.node\_pools)

* for\_each = {for i, v in var.node\_pools: i =\> v}

* initial\_node\_count = var.node\_pools\[count.index\].initial\_node\_count

* initial\_node\_count = each.value.initial\_node\_count

etc ….

Now, obviously at this point the terraform plan would show it's intention to destroy and re-create everything under that list of resources.

terraform plan -var-file=./tfvars/.tfvars → not looking good

Here is the golden transformation steps (example):

terraform state mv 'google\_container\_node\_pool.nodes\[0\]' 'google\_container\_node\_pool.nodes\["0"\]'

In this example we manually, yet safely, transformed one of the items from numbered (indexed) list to named item in the list, keeping the old ordering as needed (although will not be important post transformation it is important during transformation).

After doing the above on all items in the list you can initiate plan again and watch the results:

terraform plan -var-file=./tfvars/.tfvars → “infrastructure matches the configuration”

Now you can safely commit, merge and apply your new terraform infrastructure code and continue working with a new key/named list, with the ‘for-each’ iterator handling item updates much better (although - nor perfect)
> 0 votes

---
Tags: terraform, azure, cloud, terraform-module, variables
---


title: Debug Docker caching issues

text:
# Question
Title: Debug Docker caching issues
I am trying to improve CI/CD performance and part of that is improving the caching behaviour of Docker builds. I find myself trying to debug what exactly invalidates the cache. Are there any ways to query the Docker build process to get more information than "cached" and "not cached" ? For example, at some point, I add a large directory, which has not changed as far as I can tell, but busts the cache. I would like to find out what is causing that, exactly.

# Answer
> 3 votes
The docker cache needs the following:

* Build has been run on the same docker host previously (or you explicitly added flags to trust a pulled image) and
* The previous image needs to still be on the build host (not pruned) and
* The same previous layer and either
* The same command being run (including the same environment/args) or
* The same hash on the files being copied

If you see the cache being busted on a `COPY` or `ADD` command, and not the previous step, then you need to look at the hash being generated. All the files need to be identical, same file names, cases, file permissions, file owners, and the contents need to be bit for bit identical. You can look at the resulting checksum for this hash by running a `docker image history` command, e.g.:

```
$ docker image history golang:alpine
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
6b21b4c6e7a3        3 weeks ago         /bin/sh -c #(nop) WORKDIR /go                   0B
<missing>           3 weeks ago         /bin/sh -c mkdir -p "$GOPATH/src" "$GOPATH/b…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV PATH=/go/bin:/usr/loc…   0B
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOPATH=/go               0B
<missing>           3 weeks ago         /bin/sh -c set -eux;  apk add --no-cache --v…   344MB
<missing>           3 weeks ago         /bin/sh -c #(nop)  ENV GOLANG_VERSION=1.12.7    0B
<missing>           3 weeks ago         /bin/sh -c [ ! -e /etc/nsswitch.conf ] && ec…   17B
<missing>           3 weeks ago         /bin/sh -c apk add --no-cache   ca-certifica…   551kB
<missing>           3 weeks ago         /bin/sh -c #(nop)  CMD ["/bin/sh"]              0B
<missing>           3 weeks ago         /bin/sh -c #(nop) ADD file:0eb5ea35741d23fe3…   5.58MB

```

You can include the `--no-trunc` option to get a full line. The key piece of data relevant to your question from the above image is in the `ADD` step where you see `file:0eb5ea35741d23fe3…`. If that checksum changes between two image builds, the cache will bust and you'll start a new set of layers.

# Answer
> 0 votes
According to Docker documentation search for: "Leverage build cache", there is checksum calculated on the directory - please see the article for details.

What I will suggest is a multistage build.

If we have static content of directory we could create container/image with its content. Then instead of using Docker copy, we could use FROM as a build stage and copy that directory into a new layer, which in fact will reuse that layer. That is based on a fact that we don't operate on a directory (when a file could get a changed attribute like last accessed time).

## Base Dockerfile

```
FROM alpine:latest
COPY /myDirectory .

```

Then you could tag this container as storage-base O

## MultiStage Dockerfile

```
FROM storage-base AS storage

FROM alpine:latest
RUN a command
WORKDIR /root/
COPY --from=storage /myDirectory .

```

Please check more references here: Docker Multistage Build

## Edit

1. the image/container created in step 1 shall be only recreated when directory content will change (by a trigger that knows that the data changed), so that means it will not be created on every build.
2. then we use the image that was created once (and re-use that layer)

I just created a simple example to help express my idea - Git repo \- run build-docker.sh script

I found that on reboot Docker is re-creating cache instead of reusing, but it takes just a few minutes and it is not critical.

# Answer
> 0 votes
I have to deal with that same issue and the main problem is not the cache itself is the time that takes to make a build but also the problem of being sure you have the right changes on the image you are building.

I can recommend you some different options you can make separately or all together:

* Make cronjob that cleans your docker Daemon of old images periodically, for example, once a week. With that, you are sure to have fresh images every week and have a good performance with the cache for the daily build, the only one that takes a little bit more time is the first build after the flush.
* Make your own base image with all the content you know you don't change too often, for that you can have more than one Dockerfile on your project and launch a build process of the base image after the flush on the previous step to be sure you have the latest changes on a fresh image
* Use multi-stage build or build patter to have a good size image at the end of the build.

# Answer
> 0 votes
Did you try

```
docker inspect TAG

```

Also try the manual binary search method in the answer by Timmay to Server Fault StackExchange question "How to debug Docker cache invalidation?"

# Answer
> 0 votes
As I've discussed in this blog post, I found the following reasons for (unexpected) cache misses in Docker:

* If the problem happens in a *CI pipeline*, and if you use *multiple* build agent machines (or "ephemeral" machines), it might happen that build job #1 was executed on agent #A, but build job #2 was executed on agent #B, which has a different local cache than agent #A. Consequently, when you look at your CI pipeline output, be sure to check on which agent the jobs are executed. To prevent these kinds of cache misses, you can use a remote cache, storing caching information in a remote image registry. There are two implementation approaches: *inline* caching (where the image builder embeds caching meta-data into the image it builds), or using a separate *registry cache* (where a separate image is pushed that contains only cache blobs). The usage details of remote caching depend on your image builder tool. For instance, `docker build` supports only inline caching (see here), `docker buildx` (or when using BuildKit directly) supports both approaches (see here), Buildah and kaniko only support the registry cache.
* If you use `ARG` in your Dockerfile, it is easy to accidentally break the cache invalidation. Whenever the value of some `ARG` is different between two `docker build` executions, the second execution won’t be able to reuse the previously cached layer for a `RUN` or `ENV` command that **uses** the `ARG`'s value. This then also invalidates all follow-up layers. See here for background information. If you use *multi-stage builds*, and if you run `docker build` *several* times (for different *targets*), make sure you **always** provide the same *ARG* values to *all* docker build calls!
* Sometimes the entire image is rebuilt whenever a new **base** image has been released (that you reference in a `FROM` statement). This particularly happens if you use `docker build --pull`. You need to closely look at the builder’s output of the *first* layer, which includes the SHA-256 checksum of the base image. If it keeps changing frequently, there is no real "fix". Your image *should* be rebuilt, to include the most recent security fixes of the base image. However, if the base image is rebuilt *very often* (e.g. multiple times per day), you may want to stop using the `--pull` flag, and instead have a different approach that only runs `docker pull <base image>` (or *delete* the base image) more rarely, e.g. once per day.
* Layers for `COPY` or `ADD` statements are rebuilt "unexpectedly" whenever files change that you did not have on your radar (which you did not include into your `.dockerignore` yet). This could be the `.git` folder, or files created during building/testing (e.g. unit test report files, or log files). This typically happens when running `COPY . .`, because then your entire project directory is copied from the build context into the build container, which increases the chance that you missed excluding some superfluous files (that do not belong into the container anyway) via `.dockerignore`. To make it easy to address this issue, I developed a small CLI tool called Directory Checksum. It recursively computes the checksum of the contents of a directory, and prints the checksums up to a depth you can specify.

---
Tags: docker
---


# Question
Title: Why should we apply `flannel" after installing Kubernetes?
In most of the Kubernetes installation instructions on Linux machines, there is such a commands:

```
kubectl apply -f https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml

```

What is the above command and what does it do? Why there is no official repository for that?

Also like to know about the below command:

```
kubectl taint nodes --all=true node-role.kubernetes.io/master:NoSchedule-

```

What happens if we run this commands after installation and what if we don't?

# Answer
> 1 votes
Flannel is Container Network Interface (CNI)

Basically, CNI will help your pod or container in your cluster can communicate each other. You can read more at the link above.

About kubectl taint command. This command mean remove all taint node-role.kubernetes.io/master:NoSchedule from all nodes in your cluster. This taint is always added to Master node to disallow schedule the application pod on Master node. In short, you can schedule your pods on Master node when you run this command. You can refer to this link: https://kubernetes.io/docs/concepts/scheduling-eviction/taint-and-toleration/#concepts

---
Tags: kubernetes, repository, installation
---


# Question
Title: Setting the network using docker-compose
Do you think these commands are valid?

```
wget https://raw.githubusercontent.com/input-output-hk/cardano-wallet/master/docker-compose.yml
NETWORK=testnet docker-compose up

```

The first command downloads the `.yml` file. However, when I try to run the second command, node complains about `NETWORK=testnet` not being a valid. Here is the error printout:

Is there anybody here who could point out what I am not understanding and make it work? The link to this particular set of commands can be found here: https://developers.cardano.org/docs/get-started/cardano-wallet-js/

# Answer
> 1 votes
> `NETWORK=testnet docker-compose up`

This syntax is assuming a bash or similar shell. It defines an environment variable and runs the command with that variable exported. It's similar to (but not the same as) running:

```
export NETWORK=testnet
docker-compose up

```

Since you appear to be running this on Windows, I believe the equivalent there is:

```
set NETWORK=testnet
docker-compose up

```

---
Tags: docker, docker-compose
---


# Question
Title: Resolving _class not in classpath_ error without doing it manually in a k8s Jenkins instance
I have installed a k8s cluster (v1.25.4+k3s1) and I have deployed a jenkins image to that cluster (using this guide) which means that I have the following installed:

| Name | Version |
| --- | --- |
| Jenkins | v2.375.1 |
| jvm | 11.0.17+8 |
| Kubernetes plugin | 3743.v1fa\_4c724c3b\_7 |

When I then try to set up workers on this setup using another guide, I get this error (when starting the job):

> `Jan 10, 2023 7:47:16 PM WARNING org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper watchCloud`
>
> `failed to set up watcher on kubernetes`
> `io.fabric8.kubernetes.client.KubernetesClientException: No httpclient implementations found on the context classloader, please ensure your classpath includes an implementation jar`
> `at io.fabric8.kubernetes.client.utils.HttpClientUtils.getHttpClientFactory(HttpClientUtils.java:164)`
> `at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:145)`
> `at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:56)`
> `at org.csanchez.jenkins.plugins.kubernetes.KubernetesFactoryAdapter.createClient(KubernetesFactoryAdapter.java:147)`
> `at org.csanchez.jenkins.plugins.kubernetes.KubernetesClientProvider.createClient(KubernetesClientProvider.java:59)`
> `at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.connect(KubernetesCloud.java:523)`
> `at org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper.watchCloud(Reaper.java:220)`
> `at org.csanchez.jenkins.plugins.kubernetes.pod.retention.Reaper.preLaunch(Reaper.java:122)`
> `at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:296)`
> `at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)`
> `at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)`
> `at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)`
> `at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)`
> `at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)`
> `at java.base/java.lang.Thread.run(Thread.java:829)`
>
> `Jan 10, 2023 7:47:16 PM WARNING org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch`
>
> `Error in provisioning; agent=KubernetesSlave name: kube-slave-tq3z1, template=PodTemplate{id='282a0649-b2c7-4a9b-9ee2-0721a8da7ed7', name='kube-slave', namespace='devops-workers', label='kube-slave'}`
> `io.fabric8.kubernetes.client.KubernetesClientException: No httpclient implementations found on the context classloader, please ensure your classpath includes an implementation jar`
> `at io.fabric8.kubernetes.client.utils.HttpClientUtils.getHttpClientFactory(HttpClientUtils.java:164)`
> `at io.fabric8.kubernetes.client.utils.HttpClientUtils.createHttpClient(HttpClientUtils.java:145)`
> `at io.fabric8.kubernetes.client.DefaultKubernetesClient.<init>(DefaultKubernetesClient.java:56)`
> `at org.csanchez.jenkins.plugins.kubernetes.KubernetesFactoryAdapter.createClient(KubernetesFactoryAdapter.java:147)`
> `at org.csanchez.jenkins.plugins.kubernetes.KubernetesClientProvider.createClient(KubernetesClientProvider.java:59)`
> `at org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud.connect(KubernetesCloud.java:523)`
> `at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:120)`
> `at hudson.slaves.SlaveComputer.lambda$_connect$0(SlaveComputer.java:298)`
> `at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:48)`
> `at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:82)`
> `at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)`
> `at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)`
> `at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)`
> `at java.base/java.lang.Thread.run(Thread.java:829)`
>
> `Jan 10, 2023 7:47:16 PM INFO org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate`
>
> `Terminating Kubernetes instance for agent kube-slave-tq3z1`
>
> `Jan 10, 2023 7:47:16 PM INFO hudson.slaves.NodeProvisioner update`
>
> `kube-slave-tq3z1 provisioning successfully completed. We have now 2 computer(s)`

To experiment I tried to run this script (inspired by this answer) in the Script Console:

```
@Grab(group='io.fabric8', module='kubernetes-client', version='6.3.1')
@Grab(group='io.fabric8', module='kubernetes-client-api', version='6.3.1')
@Grab(group='io.fabric8', module='kubernetes-httpclient-jdk', version='6.3.1')

import io.fabric8.kubernetes.client.KubernetesClient;
import io.fabric8.kubernetes.client.KubernetesClientBuilder;

KubernetesClient client = new KubernetesClientBuilder().build()

```

Which gives the same error:

> `io.fabric8.kubernetes.client.KubernetesClientException: No httpclient implementations found on the context classloader, please ensure your classpath includes an implementation jar`

I would prefer not to have to make manual steps to inject jar files into a ready image, so am I missing another Jenkins plugin, or can I have missed something else?

# Answer
> 3 votes
Upgrade jenkins kubernetes plugin to 3802 and it should work. At least the `No httpclient implementations found on the context classloader` is addressed by that release.

References:

---
Tags: kubernetes, jenkins
---


# Question
Title: Kubernetes internal network and addresses not working for all internal names for same pod/service
I am setting up a k8s cluster using k3s. Currently I have a Jenkins pod running, set up according to this guide.

I have run into an issue setting up the kubernetes cluster for Jenkins workers I can't seem to get my head wrapped around:

If I go to my Jenkins pod and start the console to test that the JNLP is working, not all the different names I can use work:

`curl localhost:50000` ✔ Works
`curl `cat /etc/hostname`:50000` ✔ Works
`curl jenkins-service:50000` ❌ Times out

**BUT !!!**

`curl localhost:8080` ✔ Works
`curl `cat /etc/hostname`:8080` ✔ Works
`curl jenkins-service:8080` ✔ Works

I then went back to the YAML and took a look at it. Appart from there being more references to 8080 than 50000 (8080 is a part of several urls and liveliness checks and so on), I do not see anything amiss

```
      containers:
        ...
        ports:
        - containerPort: 8080
          name: httpport
          protocol: TCP
        - containerPort: 50000
          name: jnlpport
          protocol: TCP

```

I have looked at these defects and the issues seem to not be the same, causing me to think that it must be Kubernetes related.

# Answer
> 0 votes
I found a detailed guide which lead me to the answer...

In the service section only port 8080 was forwarded to 32000.

So I changed the service.yaml to be

```
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: devops-tools
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector:
    app: jenkins-server
  type: NodePort
  ports:
    - name: httpport
      port: 8080
      targetPort: 8080
      nodePort: 32000
    - name: jnlpport
      port: 50000
      targetPort: 50000

```

and executed `kubectl apply -f service.yaml` which seems to have had an immediate effect.

---
Tags: kubernetes, jenkins
---


# Question
Title: Does OpenStack Horizons expose iops numbers for instances and volumes?
OpenStack has the ability to set iops quotas with Teraform using `openstack_blockstorage_qos_v3` which seems to correspond to Block Storage API 3 in OpenStack.

My question is on setting the `qos_specs`

```
specs = {
  read_iops_sec  = "40000"
  write_iops_sec = "40000"
}

```

Can I use Horizons (the web interface) to find peak `iops` for a running instance? If not, is this metric recorded somewhere that I can get with the API? I'd like to find his information out for a volume, and and an instance (with all the volumes backing it).

# Answer
> 1 votes
It will depend on the provider of the underlying Openstack infrastructure.

You will need Monasca, Ceilometer to gather the data for you.

https://www.openstack.org/software/project-navigator/openstack-components#operations-tooling

---
Tags: terraform, openstack
---


title: kubectl uses wrong IP

text:
# Question
Title: kubectl uses wrong IP
My kubectl suddenly stopped working. Any commands results in "no route to host".

For example, getting logs `kubectl logs mypod-746cfd667b-tdrnt`

```
Error from server: Get "https://hades:10250/containerLogs/default/mypod-746cfd667b-tdrnt/mypod": dial tcp 10.0.0.10:10250: connect: no route to host

```

That's no wonder, because my server is located at `10.0.0.102`, and for some reason, it is going to `10.0.0.10`. Why?

* I tried removing kubectl (with purge) and reinstalling it. No difference.
* I tried deleting the `~/.kube` folder. No difference.
* I CAN manage my cluster using Kubernetes Lens, which uses the same `~/.kube/config` file. However, when trying to get the logs it fallbacks to using kubectl and I get the same issue.

```
# ping hades

PING hades (10.0.0.102) 56(84) bytes of data.
64 bytes from hades (10.0.0.102): icmp_seq=1 ttl=64 time=0.199 ms

```

Any idea how I can force kubectl to refresh its DNS cache?

Any idea how I can force kubectl to use a specific IP address, bypassing the DNS entirely?

My `~/.kube/config` file is below. Note that "hades" isn't even mentioned anywhere.

```
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: <<<long secret string>>>
    server: https://10.0.0.102:16443
  name: microk8s-cluster
contexts:
- context:
    cluster: microk8s-cluster
    user: admin
  name: microk8s
current-context: microk8s
kind: Config
preferences: {}
users:
- name: admin
  user:
    token: <<<short-secret-string>>>

```

# Answer
> 1 votes
The issue isn't on this machine at all. When making calls such as 'logs', kubectl contacts the node which in turns looks up which ip it should talk to. In this case, the node has a faulty `/etc/hosts` file with the last digit `2` dropped:

```
hades 10.0.0.10

```

All that is needed is to ssh into the machine and update the hosts file there. There never was an issue on the client machine.

---
Tags: kubernetes, kubectl
---


# Question
Title: Do docker layers work at file level or block level?
I'm considering using a Docker image as way to snapshot a MySQL database after fixtures have been loaded, in order to quickly restore the snapshot before each acceptance test.

In order for this to be fast enough, the layer filesystem should operate at block level, not file level: if every (potentially large enough) database file has to be copied before being written to, this would be quite a bottleneck.

**Do Docker layers operate at file level, or block level?** Does this depend on the storage driver?

# Answer
> 2 votes
The answer depends on the storage driver you are using with Docker. The documentation on selecting a storage driver indicates that the *overlay2*, *aufs* and *overlay* drivers all work at the file level rather than the block level. These are the most commonly used and best supported storage drivers.

*devicemapper*, *btrfs* and *zfs* are mentioned as block-level storage drivers with better performance for write-heavy workloads.

---
Tags: docker, containers
---


# Question
Title: SSL issue while reaching kubernetes dashboard
I'm configuring a kubernetes cluster (using microk8s) and cert-manager. Certificates work fine to traefik dashboard and to other sites but I have an issue with kubernates dashboard as it's already server via SSL (service is on port 443) and I currently expose a Let'sEncrypt certificate.

When I try to reach the dashboard the dashboard's pod logs:

```
2021/06/17 07:43:20 http: TLS handshake error from 213.215.191.83:39484: remote error: tls: bad certificate
2021/06/17 07:43:23 http: TLS handshake error from 213.215.191.83:39500: remote error: tls: bad certificate

```

The `IngressRoute` I'm using is:

```
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: dashboard-k8s
  namespace: kube-system
spec:
  entryPoints:
    - websecure
  routes:
    - match: "Host(`k.example.com`)"
      kind: Rule
      services:
      - name: kubernetes-dashboard
        port: 443
  tls:
    secretName: k8s-dashboard-http-cert

```

I have no clear idea of how should SSL work in this scenario. Would it be safe to have dashboard served on 80 and SSL terminated on traefik?

# Answer
There are multiple ways to expose Dashboard through traefik.

1. Traefik terminates HTTPS from client and as client opens new HTTPS request to Dashboard.

```
     +-------------+       HTTPS       +-------------+        HTTPS      +-------------+
     |  Client     -----(session 1)-----  Traefik    ------(session 2)---- Dashboard   |
     +-------------+                   +-------------+                   +-------------+

```

This is what you are using. The problem with your configuration is, on second session, Traefik will verify Dashboard's TLS certificate.
When Dashboard certificate is not signed by any of CA in Traefik list, it closes connection, then Dashboard raises error message.
If you want to use this scenario, you need to skip Dashboard certificate verification, or add Dashboard certificate to Traefik list of trusted CA.
Below configuration works by skipping Dashboard certificate verification.

```
    apiVersion: traefik.containo.us/v1alpha1
    kind: ServersTransport
    metadata:
      name: mytransport
      namespace: kube-system
    spec:
      serverName: "k.example.com"
      insecureSkipVerify: true
    ---
    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRoute
    metadata:
      name: dashboard-k8s
      namespace: kube-system
    spec:
      entryPoints:
        - websecure
      routes:
        - match: "Host(`k.example.com`)"
          kind: Rule
          services:
          - name: kubernetes-dashboard
            port: 443
            serversTransport: mytransport
      tls:
        secretName: k8s-dashboard-http-cert

```
2. Traefik passes the TLS traffic to Dashboard without any processing.

```
     +-------------+                   +-------------+                   +-------------+
     |  Client     -------HTTPS-------Traefik(passthrough)---------------- Dashboard   |
     +-------------+                   +-------------+                   +-------------+

```

In this scenario, Traefik does not do any TLS termination, the traffic passes as it is to Dashboard.
This works for me.

```
    apiVersion: traefik.containo.us/v1alpha1
    kind: IngressRouteTCP
    metadata:
      name: dashboard-k8s
      namespace: kube-system
    spec:
      entryPoints:
        - websecure
      routes:
        - match: HostSNI(`k.example.com`)
          services:
          - name: kubernetes-dashboard
            namespace: kubernetes-dashboard
            port: 443
      tls:
        passthrough: true

```
3. Traefik terminates HTTPS from client and open new plain HTTP request to Dashboard.

```
     +-------------+                   +-------------+                   +-------------+
     |  Client     -------HTTPS---------  Traefik    ---------HTTP-------- Dashboard   |
     +-------------+                   +-------------+                   +-------------+

```

In this scenario, Dashboard need to provide plain HTTP termination.
I am not sure whether Dashboard support it.
> 7 votes

# Answer
There is 4th way:

* Instruct Kube dashboard to use your CA cert (not self-generated ones): Delete kube dashboard own certs:

```
kubectl -n kubernetes-dashboard delete secret kubernetes-dashboard-certs

```
* Add cert from your CA: (note that the one of the dns name must be kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local )

```
kubectl -n kubernetes-dashboard create secret tls kubernetes-dashboard-certs --cert=<your cert open key pem> --key=<your cert private key>

```
* Edit kube dashboard deployment file for using your key:

```
kubectl edit -f https://raw.githubusercontent.com/kubernetes/dashboard/v2.6.0/aio/deploy/recommended.yaml

```
* add tls-cert-file/tls-key-file to container spec as shown below:

```
      spec:
        containers:
        - args:
          - --auto-generate-certificates
          - --namespace=kubernetes-dashboard
          - --tls-cert-file=/tls.crt
          - --tls-key-file=/tls.key

```
* Instruct Traefik to use your CA to check Kube dashboard certs (should go to kube-system namespace with key name ca.crt).

```
kubectl -n kube-system create secret generic <yourca> --from-file=ca.crt=<your ca open pem key>

```
* Modify ServerTransport

```
---
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: mytransport
  namespace: kube-system
spec:
  serverName: kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local
  rootCAsSecrets:
  - <yourca>

```
* Use that transport in IngressRoute

```
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: kube-dashboard
  namespace: kubernetes-dashboard
spec:
...
routes:
...
    services:
    - name: kubernetes-dashboard
      namespace: kubernetes-dashboard
      port: 443
      serversTransport: mytransport

```
> 1 votes

# Answer
If you want to use `PathPrefix` to distinguish services, beware that you may need to strip the prefix.

```
apiVersion: traefik.containo.us/v1alpha1
kind: ServersTransport
metadata:
  name: skip-verify
  namespace: kube-system
spec:
  insecureSkipVerify: true
---
apiVersion: traefik.containo.us/v1alpha1
kind: Middleware
metadata:
  name: kubernetes-dashboard-strip-prefix
  namespace: kube-system
spec:
  stripPrefix:
    prefixes:
      - /kubernetes-dashboard
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: kubernetes-dashboard
  namespace: kube-system
spec:
  entryPoints:
    - websecure
  routes:
    - match: PathPrefix(`/kubernetes-dashboard`)
      kind: Rule
      services:
        - name: kubernetes-dashboard
          port: 443
          serversTransport: skip-verify
      middlewares:
        - name: kubernetes-dashboard-strip-prefix

```
> 0 votes

---
Tags: kubernetes, traefik
---


# Question
Title: How can I set a Route53 record as an alias for EKS load balancer?
I set an EKS cluster using Terraform. I try to set Route53 record to map my domain name, to the load balancer of my cluster.

I set my EKS cluster:

```
resource "aws_eks_cluster" "main" {
  name     = "${var.project}-cluster"
  role_arn = aws_iam_role.cluster.arn
  version  = "1.24"

  vpc_config {
    subnet_ids              = flatten([aws_subnet.public[*].id, aws_subnet.private[*].id])
    endpoint_private_access = true
    endpoint_public_access  = true
    public_access_cidrs     = ["0.0.0.0/0"]
  }

  tags = merge(
    var.tags,
    {
      Stack = "backend"
      Name  = "${var.project}-eks-cluster",
    }
  )

  depends_on = [
    aws_iam_role_policy_attachment.cluster_AmazonEKSClusterPolicy
  ]
}

```

And I have created the following k8s service:

```
apiVersion: v1
kind: Service
metadata:
    name: backend-service
spec:
    selector:
        app: dashboard-backend
    type: LoadBalancer
    ports:
        - protocol: TCP
          port: '$PORT'
          targetPort: '$PORT'

```

**As far as I know, once I deploy a k8s service, AWS automatically generates an ALB resource for my service.** So, I set this route53 sources:

```
resource "aws_route53_zone" "primary" {
  name = var.domain_name

  tags = merge(
    var.tags,
    {
      Name = "${var.project}-Route53-zone",
    }
  )
}

data "kubernetes_service" "backend" {
  metadata {
    name = "backend-service"
  }
}

resource "aws_route53_record" "backend_record" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "www.api"
  type    = "A"
  ttl     = "300"

  alias {
    name                   = data.kubernetes_service.backend.status.0.load_balancer.0.ingress.0.hostname
    zone_id                = ??????
    evaluate_target_health = true
  }
}

```

I did get the load balancer host name using `data.kubernetes_service.backend.status.0.load_balancer.0.ingress.0.hostname`, but how can I get its zone ID to use in `zone_id` key?

# Answer
> 1 votes
There is no direct way to get the zone\_id of the load balancer from the `kubernetes_service` data source. It can only provide you the `ip` and `hostname`. Refer: https://registry.terraform.io/providers/hashicorp/kubernetes/latest/docs/data-sources/service#ingress

You can get the aws elb hosted zone id using the data source `aws_elb_hosted_zone_id`.

```
data "aws_elb_hosted_zone_id" "elb_zone_id" {}

resource "aws_route53_record" "backend_record" {
  zone_id = aws_route53_zone.primary.zone_id
  name    = "www.api"
  type    = "A"
  ttl     = "300"

  alias {
    name                   = data.kubernetes_service.backend.status.0.load_balancer.0.ingress.0.hostname
    zone_id                = data.aws_elb_hosted_zone_id.elb_zone_id.id
    evaluate_target_health = true
  }
}

```

---
Tags: kubernetes, amazon-web-services, terraform
---


# Question
Title: Issue (401 unauthorized) adding kubernetes cluster to existing Jenkins server
I want to add a kubernetes cluster to an existing jenkins server (and migrate jobs from docker to kubernetes using named agents).

I set up the kubernetes cluster and added a serviceaccount like for a Jenkins instance running on the same kubernetes (which I find much easier to figure out).

#service-account.yml

```
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins

---
kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: jenkins
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create","delete","get","list","patch","update","watch"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get","list","watch"]
- apiGroups: [""]
  resources: ["events"]
  verbs: ["watch"]
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: jenkins
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: jenkins
subjects:
- kind: ServiceAccount
  name: jenkins

```

I execute `kubectl create -f service-account.yml`
then
`kubectl -n devops-tools create token jenkins-admin`
the output of which I copy to the relevant Jenkins credential

Then I set up a job and I always get 401 unauthorized. Do I need to add something else to the serviceaccount when i want to use the token from outside the cluster?

> Caused: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: https://10.10.0.247:6443/api/v1/namespaces/devops-tools/pods. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(\_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}).

# Answer
This assumes that the namespace is `devops-tools`.

## Creating an authentication token for an external Jenkins (ie. not deployed inside the kubernets cluster)

1. Create the serviceAccount
`kubectl create serviceaccount jenkins -n devops-tools`
2. Create the role yaml file:
```
 ---
 kind: Role
 apiVersion: rbac.authorization.k8s.io/v1
 metadata:
   name: jenkins
 rules:
 - apiGroups: [""]
   resources: ["pods"]
   verbs: ["create","delete","get","list","patch","update","watch"]
 - apiGroups: [""]
   resources: ["pods/exec"]
   verbs: ["create","delete","get","list","patch","update","watch"]
 - apiGroups: [""]
   resources: ["pods/log"]
   verbs: ["get","list","watch"]
 - apiGroups: [""]
   resources: ["secrets"]
   verbs: ["get"]

```
3. Apply the role: `kubectl apply -f jenkins-role.yaml -n devops-tools`
4. Create the role yaml file:
```
 ---
 apiVersion: rbac.authorization.k8s.io/v1
 kind: RoleBinding
 metadata:
   name: jenkins
   namespace: devops-tools
 roleRef:
   apiGroup: rbac.authorization.k8s.io
   kind: Role
   name: jenkins
 subjects:
 - kind: ServiceAccount
   name: jenkins

```
5. Apply the role: `kubectl apply -f jenkins-rolebinding.yaml -n devops-tools`
6. Create the token in the same namespace (you may have clashes in names - serviceaccoutns created in e.g. the `default` namespace):
`kubectl create token jenkins -n devops-tools` (copy this jwt token)
7. Create a new credential in your external Jenkins of type `secret text` with the token as copied from the last step

## Verification of the token

The token is a JWT so you *can* base64-decode it or use https://jwt.io to verify it.

Here is a curl example (linux) to test it out (replace the namespace and url as appropriate):

```
export JENKINS_TOKEN=PASTE.TOKEN.HERE

curl --location --request GET 'https://localhost:6443/api/v1/namespaces/devops-tools/pods' \
--header 'Authorization: Bearer $JENKINS_TOKEN'

```

## References
> 0 votes

---
Tags: kubernetes, jenkins
---


# Question
Title: With kubectl, I'm getting Unable to connect to the server: x509: certificate signed by unknown authority
When I run something with k3s, like `kubectl apply`, I'm getting

> Unable to connect to the server: x509: certificate signed by unknown authority

# Answer
This probably means you've got a dangling `~/.kube/config` left over from a prior installation where you followed instructions like these to set up `~/.kube/config` and you've since reinstalled k3s and left the old kubeconfig. You simply need to generate a new copy of your `~/.kube/config` in your home directory.

You can do this like this,

```
# if not set, set it to the right location
# export KUBECONFIG=~/.kube/config

sudo k3s kubectl config view --raw > "$KUBECONFIG"

```
> 2 votes

---
Tags: kubectl, k3s
---


# Question
Title: Jenkins and Kubernetes CLI plugin and GKE
I'm trying to use Jenkins Kubernetes CLI plugin from https://plugins.jenkins.io/kubernetes-cli/ to let deploy artifacts from Jenkins to GKE cluster.

## Expected Behavior

Get cluster info on `kubectl cluster-info`

## Actual Behavior

Error out as `Unable to connect to the server: x509: certificate signed by unknown authority`

## Steps to Reproduce the Problem

Output of generated `./kube/config`

```
+ kubectl config view
apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: DATA+OMITTED
    server: https://xxx.xxx.xxx.yyy
  name: my-cluster
contexts:
- context:
    cluster: my-cluster
    user: jenkins-pipeline
  name: my-cluster-jenkins
current-context: my-cluster-jenkins
kind: Config
preferences: {}
users:
- name: jenkins-pipeline
  user:
    username: jenkins-pipelin

```

When I run from a command line I need to do

`export GOOGLE_APPLICATION_CREDENTIALS=path/to/key.json`

Where that part goes in that plugin?

# Answer
> 1 votes
Put it inside the sh?

```
withKubeConfig(...) {
  sh “””
    export ...
    kubectl ...
  “””
}

```

# Answer
> 0 votes
Whatever machine you're running `kubectl` on is reading a kube config (likely `~/.kube/config`) with a `client-certificate-data` or `client-key-data` that doesn't correspond to the server you're communicating with.

You likely want to remove `~/.kube/config` and use the one you download from Google.

---
Tags: jenkins, kubernetes, jenkins-plugins, google-kubernetes-engine
---


title: How does Krew compare to Helm?

text:
# Question
Title: How does Krew compare to Helm?
I've use Helm before, but I see some projects now are supporting a new plugin manager called Krew. Specifically, minio, provides a Krew package.

What is the difference between Krew and Helm? How do they compare?

# Answer
> 2 votes
They do different things,

* Krew is a **plugin manager for `kubectl`** the client-side cli tools for Kubernetes. For example, minio uses Krew becuase it provides `kubectl minio` which is the MinIO Kubernetes plugin.
* Helm is a **package manager for Kubernetes**. This is allows you to install software (mostly through Kubernetes operators) onto your cluster.

---
Tags: helm
---


# Question
Title: Spring Boot Microservices cannot run on Kubernetes (java.net.SocketTimeoutException: connect timed out)
I have a problem about running Spring Boot Microservices on Kubernetes. After I installed minikube, I started it and open its dashboard.

Here is the commands to open dashboards.

```
1 ) minikube start
2 ) minikube dashboard

```

Next, I run all services through this command.

```
kubectl apply -f k8s

```

After waiting for a certain amount of time, I got this issue shown below.

```
15:22:37.395 [main] ERROR org.springframework.boot.SpringApplication - Application run failed
org.springframework.cloud.config.client.ConfigClientFailFastException: Could not locate PropertySource and the resource is not optional, failing
 at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.doLoad(ConfigServerConfigDataLoader.java:197)
 at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.load(ConfigServerConfigDataLoader.java:102)
 at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.load(ConfigServerConfigDataLoader.java:61)
 at org.springframework.boot.context.config.ConfigDataLoaders.load(ConfigDataLoaders.java:107)
 at org.springframework.boot.context.config.ConfigDataImporter.load(ConfigDataImporter.java:128)
 at org.springframework.boot.context.config.ConfigDataImporter.resolveAndLoad(ConfigDataImporter.java:86)
 at org.springframework.boot.context.config.ConfigDataEnvironmentContributors.withProcessedImports(ConfigDataEnvironmentContributors.java:116)
 at org.springframework.boot.context.config.ConfigDataEnvironment.processWithProfiles(ConfigDataEnvironment.java:311)
 at org.springframework.boot.context.config.ConfigDataEnvironment.processAndApply(ConfigDataEnvironment.java:232)
 at org.springframework.boot.context.config.ConfigDataEnvironmentPostProcessor.postProcessEnvironment(ConfigDataEnvironmentPostProcessor.java:102)
 at org.springframework.boot.context.config.ConfigDataEnvironmentPostProcessor.postProcessEnvironment(ConfigDataEnvironmentPostProcessor.java:94)
 at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEnvironmentPreparedEvent(EnvironmentPostProcessorApplicationListener.java:102)
 at org.springframework.boot.env.EnvironmentPostProcessorApplicationListener.onApplicationEvent(EnvironmentPostProcessorApplicationListener.java:87)
 at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
 at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
 at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
 at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:131)
 at org.springframework.boot.context.event.EventPublishingRunListener.environmentPrepared(EventPublishingRunListener.java:85)
 at org.springframework.boot.SpringApplicationRunListeners.lambda$environmentPrepared$2(SpringApplicationRunListeners.java:66)
 at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:120)
 at org.springframework.boot.SpringApplicationRunListeners.doWithListeners(SpringApplicationRunListeners.java:114)
 at org.springframework.boot.SpringApplicationRunListeners.environmentPrepared(SpringApplicationRunListeners.java:65)
 at org.springframework.boot.SpringApplication.prepareEnvironment(SpringApplication.java:344)
 at org.springframework.boot.SpringApplication.run(SpringApplication.java:302)
 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1306)
 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1295)
 at com.microservice.orderservice.OrderServiceApplication.main(OrderServiceApplication.java:15)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.base/java.lang.reflect.Method.invoke(Method.java:566)
 at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
 at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
 at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
 at org.springframework.boot.loader.JarLauncher.main(JarLauncher.java:65)
Caused by: org.springframework.web.client.ResourceAccessException: I/O error on GET request for "http://config-server-svc:9296/ORDER-SERVICE/default": connect timed out; nested exception is java.net.SocketTimeoutException: connect timed out
 at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:785)
 at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:711)
 at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:602)
 at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.getRemoteEnvironment(ConfigServerConfigDataLoader.java:303)
 at org.springframework.cloud.config.client.ConfigServerConfigDataLoader.doLoad(ConfigServerConfigDataLoader.java:118)
 ... 35 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
 at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
 at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:412)
 at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:255)
 at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:237)
 at java.base/java.net.Socket.connect(Socket.java:609)
 at java.base/sun.net.NetworkClient.doConnect(NetworkClient.java:177)
 at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:508)
 at java.base/sun.net.www.http.HttpClient.openServer(HttpClient.java:603)
 at java.base/sun.net.www.http.HttpClient.<init>(HttpClient.java:276)
 at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:375)
 at java.base/sun.net.www.http.HttpClient.New(HttpClient.java:396)
 at java.base/sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:1253)
 at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect0(HttpURLConnection.java:1187)
 at java.base/sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:1081)
 at java.base/sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:1015)
 at org.springframework.http.client.SimpleBufferingClientHttpRequest.executeInternal(SimpleBufferingClientHttpRequest.java:76)
 at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48)
 at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:66)
 at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:776)
 ... 39 common frames omitted

```

Here is my deployment.yaml file shown below.

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: auth-service-app
spec:
  selector:
    matchLabels:
      app: auth-service-app
  template:
    metadata:
      labels:
        app: auth-service-app
    spec:
      containers:
        - name: auth-service-app
          image: noyandocker/authservice
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 7777
          env:
            - name: CONFIG_SERVER_URL
              valueFrom:
                configMapKeyRef:
                  name: config-cm
                  key: config_url
            - name: EUREKA_SERVER_ADDRESS
              valueFrom:
                configMapKeyRef:
                  name: eureka-cm
                  key: eureka_service_address
            - name: DB_HOST
              valueFrom:
                configMapKeyRef:
                  name: mysql-cm
                  key: hostname

---

apiVersion: v1
kind: Service
metadata:
  name: auth-service-svc
spec:
  selector:
    app: auth-service-app
  ports:
    - port: 80
      targetPort: 7777

```

Here is the config map yaml file

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: config-cm
data:
  config_url: "config-server-svc"

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: eureka-cm
data:
  eureka_service_address: "http://eureka-0.eureka:8761/eureka"

---

apiVersion: v1
kind: ConfigMap
metadata:
  name: mysql-cm
data:
  hostname: "mysql-0.mysql"

```

Here is the config-server-deployment.yaml file

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: config-server-app
spec:
  selector:
    matchLabels:
      app: config-server-app
  template:
    metadata:
      labels:
        app: config-server-app
    spec:
      containers:
        - name: config-server-app
          image: noyandocker/configserver
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 9296
          readinessProbe:
            httpGet:
              path: /actuator/health
              port: 9296
            initialDelaySeconds: 20
            timeoutSeconds: 10
            periodSeconds: 3
            failureThreshold: 2
          livenessProbe:
            httpGet:
              path: /actuator/health
              port: 9296
            initialDelaySeconds: 30
            timeoutSeconds: 2
            periodSeconds: 8
            failureThreshold: 1
          env:
            - name: EUREKA_SERVER_ADDRESS
              valueFrom:
                configMapKeyRef:
                  name: eureka-cm
                  key: eureka_service_address

---

apiVersion: v1
kind: Service
metadata:
  name: config-server-svc
spec:
  selector:
    app: config-server-app
  ports:
    - port: 80
      targetPort: 9296

```

I thought all the services will start simultaneously. Config Server is the Dependent Service for all other Serivces like auth service and this Auth service should not start until Config Server service is up and running.

How can I do that?

Here is my repo : Link

Here is my docker hub : Link

Here is git backend system : Link

# Answer
Reading the config-server-deployment.yaml you deploy a service under the name `config-server-svc`. `config-server-svc` maps port `80` to `9296` of pods port with label `config-server-app`.
K8s documentation for svc definition
Your error states:

```
GET request for "http://config-server-svc:9296/ORDER-SERVICE/default

```

Your `svc` serves to port `80` and not to port `9296` which is the pod's port.
Change the port you perform the request. At github spring app you pass an environment variable called `CONFIG_SERVER_URL`.
If it's empty, you use the `localhost:9296`. It seems that you pass `CONFIG_SERVER_URL` environmental variable that spring app will read with wrong port.
When provide a value to CONFIG\_SERVER\_URL you have to override also the port. Try to add port 80 at config-maps.yaml so the line `config_url: "config-server-svc"` will become `config_url: "config-server-svc:80"`
> 1 votes

---
Tags: kubernetes, java, minikube, spring-boot
---


# Question
Title: GitLab Container Registry: errors: denied: requested access to the resource is denied [...] error parsing HTTP 401 response body
When I run `podman push` (the equiv of `docker push`) to get my image up to GitLab's container registry, I'm getting the following error.

> errors: denied: requested access to the resource is denied \[...\] error parsing HTTP 401 response body: unexpected end of JSON input: ""

You can see it all here,

```
❯ podman push localhost/acme-web-release registry.gitlab.com/evancarroll/acme-backend
Getting image source signatures
Error: trying to reuse blob sha256:1e05dc5a6784f6e7375fe1f73dc2b02f7d184bc53a0150daf06062dcbfde02d4 at destination: checking whether a blob sha256:1e05dc5a6784f6e7375fe1f73dc2b02f7d184bc53a0150daf06062dcbfde02d4 exists in registry.gitlab.com/evancarroll/acme-backend: errors:
denied: requested access to the resource is denied
error parsing HTTP 401 response body: unexpected end of JSON input: ""

```

I've confirmed that I've `podman login` (the podman analog of `docker login`) with a Personal Access Token that grants `write_registry`).

# Answer
A personal access token to push to the registry does not *just* require `write_registry` but **ALSO** `read_registry` or `read_api` (more general).
> 0 votes

# Answer
firstly, you can just make `docker logout registry.gitlab.com`, and login again `docker login registry.gitlab.com`, it's wrong because old token have not access for new registry
> 0 votes

---
Tags: docker, gitlab, podman
---


# Question
Title: Does GitLab support searching logs across jobs for specific strings?
I need to find all jobs that failed in a specific fashion, a few methods are viable

* One method that would be viable is finding where GitLab stores its log files, and searching them with something like Grep. I asked that question here.
* Another method would be using native GitLab search functionality to search the logs. Does such functionality exist?

# Answer
This functionality doesn't yet exist in the UI, though internally job logs are upload as a job artifact. And you can search them with a little bit of work. The backing store is configured in your, `/etc/gitlab/gitlab.rb`.

* If `artifacts_object_store_enabled` is enabled and you're using the Object Storage, the artifacts will store in whatever the value is of `artifacts_object_store_remote_directory` under a file called `job.log`. Assuming that value is `artifacts`, and you're using something like minio, you can search across jobs for a specific failure with

+ `mc find` and `mc cat`,

    ```
    # In the example below, I'm looking for failures that contain Client.Timeout
    mc find minio/artifacts \
      --name 'job.log'      \
      --exec 'sh -c "mc cat {} | grep Client\.Timeout | xargs -r echo {time} {}; exit 0"' \
      | tee /tmp/client_timeout.log

    ```

    In my experience, this is just too slow as `mc find` isn't indexed it's just looking for any bucket with the name `job.log`.
    + A faster method is to lean on the database. Gitlab ships a PostgreSQL client wrapper called `gitlab-psql` which allows you to connect to underlying PostgreSQL datastore. After you connect with `gitlab-psql` can query for all `file.log` and get the relative path to the bucket from the result. You can easily tool this by creating a batch file of commands for s5cmd log,

    ```
    \t
    \a
    \o /tmp/pull_logs.s5cmd
    SELECT FORMAT(
      'cp s3://artifacts/%s /tmp/job_%s.log',
      to_bucket_path(ci_job_artifacts),
      id
    )
    FROM ci_job_artifacts
    WHERE file = 'job.log'
    AND expire_at IS NULL
    AND file_store = 2;

    ```

    Which will generate a file you can run with something like s5cmd to pull down all of your logs from the Gitlab Object store. You'd run it by setting up your `~/.aws/credentials` and running (example with minio).

    ```
    s5cmd --endpoint-url https://minio.acme.net run /tmp/pull_logs.s5cmd

    ```
* You can also use NFS. If you're using NFS for artifact storage, it's as easy as going to the mount point and searching with ripgrep.
> 0 votes

---
Tags: gitlab, logging
---


# Question
Title: How to enable ContainerInsights (setting) on an ECS cluster created from a Batch compute environment using Terraform?
I have an AWS Batch compute environment defined in my Terraform files. The creation of the compute environment creates an ECS Cluster, where the containers are being run.

The issue I have with Terraform, is that while the `aws_batch_compute_environment` resource is managed by Terraform, then the `aws_ecs_cluster` which is created from it is not.

I want to enable ContainerInsights on this cluster, but because it is not managed I am struggling to understand how to do it.

This is how my Batch compute environment is currently defined:

```
resource "aws_batch_compute_environment" "fargate-batch-compute" {
  compute_environment_name = "fargate-batch-compute"

  compute_resources {
    max_vcpus          = 100
    security_group_ids = [...]
    subnets            = [...]
    type               = "FARGATE"
  }

  type = "MANAGED"
}

```

This is how to easily enable ContainerInsights on an ECS cluster using Terraform:

```
resource "aws_ecs_cluster" "foo" {
  name = ""

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

```

**Notes:**

* I checked and it seems that it's not possible to pass an existing ECS cluster to the creation of the compute environment - Reference

# Answer
The only solution I found is to import the created resource in AWS to a new configured resource in your Terraform configuration.

If the newly created ECS Cluster is named `AWSBatch-foo-123456` than append to your Terraform configuration the following resource:

```
resource "aws_ecs_cluster" "foo-cluster" {
  name = "AWSBatch-foo-123456"

  setting {
    name  = "containerInsights"
    value = "enabled"
  }
}

```

and then execute `terraform import aws_ecs_cluster.foo-cluster AWSBatch-foo-123456`.

Now, the ECS cluster will be managed by Terraform and you can apply any setting on it.
> 0 votes

---
Tags: terraform
---


# Question
Title: Creating a set of of kubenertes pods from a list of arguments
I have a list of 35 services that need to be continually monitored/exercised. I've got a container that can monitor any one of them and can check an environment variable to see which one of them it should monitor.

I'd like to tell Kubernetes to please deploy 35 pods of that container, each of them identical except for the environment variable. I could do this by creating 35 nearly identical manifest files and running `kubectl create deployment` in a loop, but I suspect there's probably a better way. How do I best tell Kubernetes "please take this list of N strings and deploy one pod for each of them, each being assigned one of them?"

For example, if I've got a list like:

```
baseball
football
soccer
tennis

```

I'd like create a bunch of deployments that are identical, except for their name (preferably I'd include the name from the list as part of the name) and one environment variable telling the container which sport to monitor.

# Answer
Identical resources that have just such small differences are subject to templates.
In Kubernetes, helm is one tool that can help you solve the issue.
Create a `helm chart` that will deploy the needed deployment and template the fields relate to the sports.
Place a for loop and iterate over the list of sports provided in the values.yaml.
Example of the idea:

```
# values.yaml
sports:
  - baseball
  - football
  - soccer
  - tennis

# templates/pod.yaml
{{- range .Values.sports }}
apiVersion: v1
kind: Pod
metadata:
  name: {{ . }}-pod
spec:
  containers:
  - name: {{ . }}-container
    image: my-image
    env:
    - name: SPORT
      value: {{ . }}
{{- end}}

```
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Where does GitLab omnibus store ci log files?
When viewing a raw CI log file, such as this one

```
https://acmeco.net/acmeco/acmeco/angular-ui/-/jobs/334040/raw

```

I can see content such as

```
net/http: request canceled (Client.Timeout exceeded while awaiting headers) (manager.go:235:15s)

```

I want to find all ci jobs that produced the same error. Where are these logs stored?

# Answer
Depends on the setting in your `gitlab.rb` file,

For more information see
> 0 votes

---
Tags: gitlab
---


# Question
Title: Using kubernetes secret env var inside another env var
I have a secret being used as env var in another env var as follows:

```
- name: "PWD"
  valueFrom:
    secretKeyRef:
      name: "credentials"
      key: "password"
- name: HOST
  value: "xyz.mongodb.net"
- name: MONGODB_URI
  value: "mongodb+srv://user:$(PWD)@$(HOST)/db_name?"

```

When i exec into the container and run `env` command to see the values of env i see -

```
mongodb+srv://user:password123
@xyz.mongodb.net/db_name?

```

The container logs show error as authentication failure. Is this something that is expected to work in kubernetes ? There docs talk about dependent env vars but do not give example using secrets. Did not find clear explanation on this after extensive search. Only found this one article doing something similar.

Some points to note -

* The secret is a sealed secret.
* This is the final manifest's contents, but all this is templated using helm.
* The value is being used inside a spring boot application

Is the new line after `123` expected ? If this evaluation of env from a secret in another env is possible then what am I doing wrong here ?

# Answer
> 0 votes
The issue was with the command used to encode the secret - `echo "pasword" | base64`. The echo adds a newline character at the end of the string. Using `echo -n "password" | base64` fixed the secret. Closing the issue.

---
Tags: kubernetes, kubernetes-helm, environment-variables
---


# Question
Title: kubernetes ingress config for jenkins says service doesn't exist
I have a Jenkins setup which is inspired by this guide.

I wanted to add an url, so I added An ingress.yaml. jenkins is still available at \<node\_url\>:32000 but not at http://jenkins.localdomain (where /etc/hosts has ) been updated with the IP of the node running Jenkins. What Am I missing?

## $ cat ingress.yaml

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: devops-tools
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: 'websecure'
    traefik.ingress.kubernetes.io/router.tls: 'true'
spec:
  rules:
    - host: jenkins.localdomain
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: jenkins-service
                port:
                  number: 32000

```

## $ cat service.yaml

```
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: devops-tools
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector:
    app: jenkins-server
  type: NodePort
  ports:
    - name: httpport
      port: 8080
      targetPort: 8080
      nodePort: 32000
    - name: jnlpport
      port: 50000
      targetPort: 50000

```

## EDIT #1

I tried changing the port to 8080 as suggested, but portainer apparently doesn't like it (and it doesn't work)

How it looks in the select service part of portainer's ingress page:
How it looks in the select service part of portainer's service page:

# Answer
Try change the port number in `ingress.yaml` to `8080` as that is the port on which the *jenkins-service* service is available within the cluster. The *NodePort* is a port on the cluster nodes themselves which would hairpin the ingress traffic. *NodePort*s are meant for use with loadbalancing solutions external to the cluster.

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: jenkins-ingress
  namespace: devops-tools
  annotations:
    traefik.ingress.kubernetes.io/router.entrypoints: 'websecure'
    traefik.ingress.kubernetes.io/router.tls: 'true'
spec:
  rules:
    - host: jenkins.localdomain
      http:
        paths:
          - pathType: Prefix
            path: /
            backend:
              service:
                name: jenkins-service
                port:
                  number: 8080

```
> 1 votes

---
Tags: kubernetes, ingress
---


# Question
Title: Can you include an Azure DevOps wiki inside an existing repository?
Azure DevOps has a built in wiki feature that allows either a Provisioned Wiki (a single wiki for the project), or multiple wiki repositories.

My question is with regards to the repository wikis, and if you can add the wiki to an existing repository so that both the code and wiki documentation are co-located?

So rather than having these repos;

* backend
* backend-wiki
* frontend
* frontend-wiki

There are 2 code repositories and within each is the wiki markdown code?

The idea behind this is so that if a code change is made within the repository, then documentation changes can be made within the same branch so will reflect in the Pull Requests.

# Answer
> 2 votes
the short answer is yes You can use any \*.md files in a code repo as wiki, you simply go to the Project, Wiki, select "Publish Code as Wiki", point it to the Git Repo and select the Folder you want as root for your wiki. You can do that with as many Git Repos as you need.

In your case, if I understood you right you would add the wiki md files to the main branch of the backend/frontend repo, then go to the Project, Wiki, select "Publish Code as Wiki" and select the existing gitrepo and select the main branch and the folder with the wiki files in the options.

---
Tags: azure-devops
---


# Question
Title: Ansible playbook fails on Windows server
```
---
# This is a basic playbook for testing
  - hosts: WindowsSandbox
    #gather_facts: no
    vars:
      ansible_remote_tmp: c\temp
      remote_tmp: c\temp
    tasks:
    - win_shell: cd c:\temp

Here is the error:

fatal: [WindowsServer1]: UNREACHABLE! => {"changed": false, "msg": "Failed to create temporary directory.In some cases, you may have been able to authenticate and did not have permissions on the target directory. Consider changing the remote tmp path in ansible.cfg to a path rooted in \"/tmp\", for more error information use -vvv. Failed command was: ( umask 77 && mkdir -p \"` echo c\\temp `\"&& mkdir \"` echo c\\temp/ansible-tmp-1661983610.31-13-149784338894642 `\" && echo ansible-tmp-1661983610.31-13-149784338894642=\"` echo c\\temp/ansible-tmp-1661983610.31-13-149784338894642 `\" ), exited with result 1", "unreachable": true}

Why is ansible using Linux commands for my Windows server?

```

# Answer
> 1 votes
Add the below lines in the host file(not config file).

```
[windows:vars]
remote_tmp = C:\Users\ans_user\Tmp
become_method = runas
ansible_shell_type = powershell
shell_type = powershell

```

---
Tags: ansible
---


title: What is a GitLab locked artifact?

text:
# Question
Title: What is a GitLab locked artifact?
In the table `ci_job_artifacts` there is a column called `locked`. What does this column do? It's not documented in the Job Artifacts API? It's currently a `smallint` and it has 3 possible values: 0, 1, 2.

```
SELECT locked, count(*)
FROM ci_job_artifacts
GROUP BY locked;
 locked | count
--------+--------
      0 |  18140
      1 |   2252
      2 | 203191
(3 rows)

```

Even though I guess it should be?

# Answer
It refers to the Locable mixin,

```
# `locked` will be populated from the source of truth on Ci::Pipeline
# in order to clean up expired job artifacts in a performant way.
# The values should be the same as `Ci::Pipeline.lockeds` with the
# additional value of `unknown` to indicate rows that have not
# yet been populated from the parent Ci::Pipeline
enum locked: {
  unlocked: 0,
  artifacts_locked: 1,
  unknown: 2
}, _prefix

```
> 0 votes

---
Tags: gitlab
---


# Question
Title: conditionals in module providers meta-argument
I am provisioning a bunch of resources via a module.

```
module "test" {
  source = "../modules/my-module"

  vpc_id = module.vpc.vpc_id

  subnet_ids = module.subnets.ids
}

```

Now, in the calling module ( the root module ), I use the alias of the provider that the child module should use. I would use the module `providers` meta-argument :

```
provider "my_provider" {
  alias = default

  region = "default"
  ...
}

provider "my_provider" {
  alias = alias1

  region = "region1"
  ...
}

module "test" {
  source = "../modules/my-module"

  providers = {
    my_provider = my_provider.alias1
  }

  vpc_id = module.vpc.vpc_id

  subnet_ids = module.subnets.ids
}

```

Now, I want to provision the same module, but based on a condition :

```
module "test" {
  source = "../modules/my-module"

  providers = {
    my_provider = var.region == "region1" ? my_provider.alias1 : my_provider.default
  }

  vpc_id = module.vpc.vpc_id

  subnet_ids = module.subnets.ids
}

```

And I get thrown `The providers argument requires a provider type name, optionally followed by a period and then a configuration alias.`

Then I tried treating the providers meta-argument as a normal argument ( I knew it would not work, but still tried anyway ) :

```
module "test" {
  source = "../modules/my-module"

  providers = var.region == "region1" ? { my_provider = my_provider.alias1 } : { my_provider = my_provider.default }

  vpc_id = module.vpc.vpc_id

  subnet_ids = module.subnets.ids
}

```

This time, the error was more logical - `A static map expression is required.`

---

It does seem like the `providers` meta-argument for modules are always supposed to be static ( essentially hard-coded ) just like the `backend` argument.

Would there be a native way of re-using the same module declaration to switch between 2 different providers?

Or is the only way to achieve this would be code duplication and the `count` meta-argument based on the condition?

# Answer
The relationship between a resource and its provider configuration is an important part of building the dependency graph, and Terraform must build the dependency graph before it can evaluate any expressions, so there is no way to use arbitrary expressions to describe the relationship between a resource and its provider configuration.

The `providers` argument in a `module` block isn't *directly* specifying the relationship between the resource and its provider configuration, but it is still data used by the algorithm that ultimately resolves those connections and so it too needs to be resolved before it's possible to evaluate expressions.

Although you cannot dynamically choose which provider configuration to assign to a resource, you *can* make the provider configuration itself include dynamic expressions:

```
provider "my_provider" {
  alias = "for_test_module"

  region = var.region == "region1" ? "region1" : "default"
}

module "test" {
  source = "../modules/my-module"
  providers = {
    my_provider = my_provider.for_test_module
  }

  vpc_id     = module.vpc.vpc_id
  subnet_ids = module.subnets.ids
}

```

In this example the module is always associated with the same provider configuration object but the arguments inside that block will vary depending on the value of `var.region`, thereby achieving the same effect.
> 2 votes

---
Tags: terraform, terraform-module
---


# Question
Title: What is the GitLab proj_id_hash for the bucket structure in the object store?
In this document covering the Feature Bucket, it says, the Bucket Structure for the Job Artifacts is

```
/artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>

```

In the above, what is the `<proj_id_hash>`? There is only a `ci_project_mirrors` table (no `ci_projects`), and it does not have a hash column it only has 3 columns for `id`, `project_id`, and `namespace_id`.

# Answer
> 0 votes
# sha256sum of the Project ID

The `proj_id_hash` is the hash of the project id. It doesn't exist in the database. If your project id `4242` you would

```
echo -n 4242 | sha256sum

```

---
Tags: gitlab
---


# Question
Title: How can I create a path into GitLab Object Storage from the ci_job_artifacts table?
Gitlab has an internal PostgreSQL database. There is a client wrapper for `psql` called `gitlab-psql` that I connect to it with. This allows me to query job artifacts through the table `ci_job_artifacts`. This table schema looks like this,

```
gitlabhq_production=# \d ci_job_artifacts
                                          Table "public.ci_job_artifacts"
      Column       |           Type           | Collation | Nullable |                   Default
-------------------+--------------------------+-----------+----------+----------------------------------------------
 project_id        | integer                  |           | not null |
 file_type         | integer                  |           | not null |
 size              | bigint                   |           |          |
 created_at        | timestamp with time zone |           | not null |
 updated_at        | timestamp with time zone |           | not null |
 expire_at         | timestamp with time zone |           |          |
 file              | character varying        |           |          |
 file_store        | integer                  |           |          | 1
 file_sha256       | bytea                    |           |          |
 file_format       | smallint                 |           |          |
 file_location     | smallint                 |           |          |
 id                | bigint                   |           | not null | nextval('ci_job_artifacts_id_seq'::regclass)
 job_id            | bigint                   |           | not null |
 locked            | smallint                 |           |          | 2
 original_filename | text                     |           |          |
 partition_id      | bigint                   |           | not null | 100
 accessibility     | smallint                 |           | not null | 0

```

How do I convert this to a path that I can use to find the artifact in GitLab Object Storage.

# Answer
> 0 votes
Aside from customer-facing documentation there is a document in the source that details how the path is created called *"Uploads guide: Adding new uploads"* currently at `doc/development/uploads/working_with_uploads.md`. This document shows the magic sauce for the bucket structure everywhere in GitLab. Specifically for the job artifacts, the structure is

```
/artifacts/<proj_id_hash>/<date>/<job_id>/<artifact_id>

```

The `proj_id_hash` is the sha256 is the `project_id` you can read more about that on this question,*"What is the GitLab proj\_id\_hash for the bucket structure in the object store?"*

To calculate the rest, I've create a helper function which should greatly ease this process,

```
CREATE EXTENSION pgcrypto; -- needed for `digest(bytea, text) `

CREATE OR REPLACE FUNCTION generate_job_artifact_bucket_path(project_id bigint, created_at timestamp with time zone, job_id bigint, id bigint, file text)
RETURNS text
AS $$
  SELECT FORMAT(
    '%s/%s/%s/%s/%s/%s/%s',
    substring(proj_sha256 FROM 1 FOR 2),
    substring(proj_sha256 FROM 3 FOR 2),
    proj_sha256,
    to_char(created_at, 'YYYY_MM_DD'),
    job_id,
    id,
    file
  )
  FROM encode(digest(project_id::text::bytea, 'SHA256'), 'hex') AS proj_sha256
$$ LANGUAGE SQL IMMUTABLE;

COMMENT ON FUNCTION generate_job_artifact_bucket_path
IS $$Generates the relative bucket to `gitlab_rails['object_store']['objects']['artifacts']['bucket']` for more information see https://devops.stackexchange.com/q/17298/18965$$;

```

Now you can query for the path like this,

```
SELECT generate_job_artifact_bucket_path(project_id, created_at, job_id, id, file)
FROM ci_job_artifacts;

```

Alternatively you can do something like this,

```
CREATE OR REPLACE FUNCTION to_bucket_path(t ci_job_artifacts)
RETURNS text
AS $$
  SELECT FORMAT(
    '%s/%s/%s/%s/%s/%s/%s',
    substring(proj_sha256 FROM 1 FOR 2),
    substring(proj_sha256 FROM 3 FOR 2),
    proj_sha256,
    to_char(t.created_at, 'YYYY_MM_DD'),
    t.job_id,
    t.id,
    t.file
  )
  FROM encode(digest(t.project_id::text::bytea, 'SHA256'), 'hex') AS proj_sha256
$$ LANGUAGE SQL IMMUTABLE;

CREATE OR REPLACE FUNCTION to_bucket_path(t ci_pipeline_artifacts)
RETURNS text
AS $$
  SELECT FORMAT(
    '%s/%s/%s/pipelines/%s/artifacts/%s/%s',
    substring(proj_sha256 FROM 1 FOR 2),
    substring(proj_sha256 FROM 3 FOR 2),
    proj_sha256,
    t.pipeline_id,
    t.id,
    t.file
  )
  FROM encode(digest(t.project_id::text::bytea, 'SHA256'), 'hex') AS proj_sha256
$$ LANGUAGE SQL IMMUTABLE;

```

This will provide a polymorphic convention to implement a `to_bucket_path` for multiple tables..

```
SELECT to_bucket_path(ci_job_artifacts) FROM ci_job_artifacts;
SELECT to_bucket_path(ci_pipeline_artifacts) FROM ci_pipeline_artifacts;

```

---
Tags: gitlab
---


# Question
Title: What is `file_store` in the `ci_job_artifacts`?
Where is this field defined and what does it mean? I can't find any information about it.

# Answer
> 0 votes
The `file_store` field in `ci_job_artifacts` corresponds to the code in `app/uploaders/object_storage.rb`,

Currently it only has two values,

```
module Store
  LOCAL = 1
  REMOTE = 2
end

```

This can be used to determine if the artifact exists in a Object Storage backend, or local storage.

---
Tags: gitlab
---


# Question
Title: Jenkins restart without interrupting currently running jobs
There has a chance to restart Jenkins with stops the job or once job is completed. But is there any command or process to restart Jenkins without interrupt running job.

# Answer
Jenkins jobs are not in interrupted if you run them on Jenkins agents and not a controller.

What you look for is `safeRestart` which:

> Puts Jenkins into the quiet mode, wait for existing builds to be completed, and then restart Jenkins

Read more here and Jenkins docs.
> 1 votes

---
Tags: jenkins
---


# Question
Title: Building a docker container in a gitlab ci job
I am trying to build a docker image as part of a gitlab ci job. I work on my own gitlab instance with a separate machine running ci jobs. I have tried and failed with many different configurations for using Docker in Docker. My current try is the method from this blog post: https://medium.com/@tonywooster/docker-in-docker-in-gitlab-runners-220caeb708ca

When executing a CI job on this runner with the docker command I get the following error:

```
ERROR: Failed to remove network for build
ERROR: Preparation failed: Cannot connect to the Docker daemon at tcp://gitlab-dind:2375. Is the docker daemon running? (docker.go:745:240s)

```

Why is that and how can I fix it? I am also happy with any other working method to build and push a docker container in my CI job. I am happy to provide any further information, but I am not sure what would help and don’t want to bloat the post.

As requested my gitlab CI file looks like this:

```
image: docker

stages:
  - package
  - test
  - push

build docker image:
  stage: package

  tags:
    - docker

  script:
    - cat /proc/self/cgroup
    - docker info
    - docker build -t $CI_REGISTRY_IMAGE .

test docker image:
  stage: test

  tags:
    - docker

  script:
    - docker run $CI_REGISTRY_IMAGE python testscript.py

push docker image:
  stage: push

  tags:
    - docker

  script:
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin
    - docker push $CI_REGISTRY_IMAGE

```

# Answer
To build a docker image in Gitlab runner, I recommend using kaniko instead of "docker in docker". Please read the official document of kaniko with Gitlab runner.

https://docs.gitlab.com/ee/ci/docker/using\_kaniko.html
> 0 votes

# Answer
I wrote how to perform Docker build in GitLab Self-Hosted Runners Demo article. It uses docker in docker approach.

`.gitlab-ci.yml` is below:

```
stages:          # List of stages for jobs, and their order of execution
  - build

services:
  - name: docker:20.10.16-dind
    command: ["--mtu=1300"]

build:
  tags: [docker-on-prem]
  image: docker:20.10.16
  stage: build
  variables:
    DOCKER_HOST: "tcp://docker:2376"
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: "1"
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  script:

    - docker build -t test .

```

Sample repository with Dockerfile which you can fork and try the pipeline is at https://gitlab.com/warrior7089/gitlab-self-hosted-runners-demo/
> 0 votes

---
Tags: docker, gitlab, gitlab-ci-runner, docker-in-docker
---