# Question
Title: How to use uriRegexRewrite in Istio VirtualService?
I was waiting on the Regex Rewrite based on Capture Groups for a long time and it was finally merged in 1.19.4. I cannot find much documentation regarding how to use the uriRegexRewrite and my attempts to make it work all failed so far.

I need the VirtualService to capture something like `/captcha-srv(/|$)(.*)` and rewrite it to `/$2`

My current Virtual Service looks like this:

```
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    app.kubernetes.io/instance: authserver-webapi
  name: authserver-webapi
  namespace: istio-full
spec:
  gateways:
  - authserver-webapi
  hosts:
  - auth.example.com
  http:
  - match:
    - uri:
        regex: /captcha-srv(/|$)(.*)
    rewrite:
      uri: /
    route:
    - destination:
        host: captcha
        port:
          number: 80
  - route:
    - destination:
        host: authserver-webapi
        port:
          number: 80

```

This setup rewrites whatever you write after `captcha-srv` to my microservice's `/` however, I need it to send `/captcha-srv` to `/` and e.g. `/captcha-srv/v1` to `/v1`. In Nginx Ingress, this is easily done, but apparently, this feature was recently added to Istio, and using the documentation wasn't much help except that the directive used to achieve this is `uriRegexrewrite`.

Any help?

# Answer
> 0 votes
To answer my question, to achieve my goal the correct way of writing a `virtualservice` according to the Istio reference documentation is as below:

```
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    app.kubernetes.io/instance: authserver-webapi
  name: authserver-webapi
  namespace: istio-full
spec:
  gateways:
  - authserver-webapi
  hosts:
  - auth.example.com
  http:
  - match:
    - uri:
        regex: /captcha-srv(/|$)(.*)
    rewrite:
      uriRegexRewrite:
        match: /captcha-srv(/|$)(.*)
        rewrite: /\2
    route:
    - destination:
        host: captcha
        port:
          number: 80
  - match:
    - uri:
        prefix: /
  - route:
    - destination:
        host: authserver-webapi
        port:
          number: 80

```

It's essential to know how to read and understand the reference manuals in such cases, e.g. in this case `HTTPRewite` is a subsection of `virtualservice` and therefore has an indentation, then, if you use the `uriRegexRewrite` you'll have to refer to the `RegexRewrite` section and use the `match` and `rewrite` directives with an indentation.

---
Tags: kubernetes, networking, istio
---


# Question
Title: Is it possible to loop through EC2 resources created with a count variable, while in the same module?
I have an EC2 module, which determines the number of instances via a count variable, like so

```
resource "aws_instance" "this" {
  count = var.instance_count

```

I would like to subsequently create EBS volumes in the same module, and attach them to each of the instances created. The problem I encountered with this though, is that I need a way to pass the id of each volume to all of the `volume_attachment` resource, in order to attach each EBS volume to each EC2 instance, so I'd have to have a way to loop through each ec2 id, to then attach all volumes to it.

Which leads to my question: Is there a way to loop through each EC2 instance id, and attach all the EBS volumes to it?

```
resource "aws_volume_attachment" "this" {
  for_each    = toset(var.ebs_volume_attachments)
  device_name = each.name
  volume_id   = aws_ebs_volume.this.id
  instance_id = module.ec2.id
}

resource "aws_ebs_volume" "this" {
  for_each          = toset(var.ebs_volume_attachments)
  availability_zone = each.availability_zone
  size              = each.size
  encrypted         = each.encrypted
  snapshot_id       = each.snapshot_id
  type              = each.type
  kms_key_id        = each.kms_key_id

  tags = local.tags
}

```

# Answer
What would the value of `var.ebs_volume_attachments` be?

Wouldn't this work?

```
resource "aws_volume_attachment" "this" {
  count       = var.instance_count
  device_name = "/dev/sdh"
  volume_id   = aws_ebs_volume.this[count.index].id
  instance_id = aws_instance.this[count.index].id
}

resource "aws_ebs_volume" "this" {
  count             = var.instance_count
  ...
}

```
> 0 votes

---
Tags: terraform, terraform-module, amazon-ebs
---


# Question
Title: Run dependencies only once in muliple plays in Ansible
I have a collection of roles that all depend on a common role:

```
# roleA/meta/main.yaml
dependencies:
 - role: common

# roleB/meta/main.yaml
dependencies:
 - role: common

# roleC/meta/main.yaml
dependencies:
 - role: common

```

I have a playbook that then installs stuff

```
 - name: Setup all the servers
   hosts: my_servers
   roles:
     - roleA

 - name: Setup additional things on special server
   hosts: special_servers
   roles:
     - roleB

 - name: Setup additional things on special server
   hosts: other_servers
   roles:
     - roleC

```

`special_servers` and `other_servers` are subsets of `my_servers`. The problem is, when the play for `special_servers` and `other_server` is run, it's re-running the `common` role which was already run in the `Setup all the servers` play. This is redundant and is causing my entire playbook to take 3x as long as it should.

How do I tell Ansible "Hey, `common` was already run, go ahead and skip"?

# Answer
* Use the task *import\_role* and put all roles into the single playbook

```
- hosts: all
  tasks:
    - import_role:
        name: roleA
      when: inventory_hostname in groups.my_servers
    - import_role:
        name: roleB
      when: inventory_hostname in groups.special_servers
    - import_role:
        name: roleC
      when: inventory_hostname in groups.other_servers

```

See Ansible only executes role dependencies once

---

* The 'deduplication' doesn't work if you run the roles in multiple plays

```
shell> cat pb.yml
- hosts: my_servers
  roles:
    - roleA

- hosts: special_servers
  roles:
    - roleB

- hosts: other_servers
  roles:
    - roleC

```

In this case, you can skip the tasks in the role *common* if already been completed. For example, create a control variable

```
shell> cat roles/common/tasks/main.yml
- block:
    - include_tasks: tasks.yml
    - set_fact:
        role_common_completed:  true
  when: not role_common_completed|d(false)

```

```
shell> cat roles/common/tasks/tasks.yml
- debug:
    msg: "{{ ansible_role_name }} running on {{ inventory_hostname }}"

```

Test it. Given the project

```
shell> tree .
.
├── ansible.cfg
├── hosts
├── pb.yml
└── roles
    ├── common
    │   └── tasks
    │       ├── main.yml
    │       └── tasks.yml
    ├── roleA
    │   ├── meta
    │   │   └── main.yml
    │   └── tasks
    │       └── main.yml
    ├── roleB
    │   ├── meta
    │   │   └── main.yml
    │   └── tasks
    │       └── main.yml
    └── roleC
        ├── meta
        │   └── main.yml
        └── tasks
            └── main.yml

12 directories, 11 files

```

```
shell> cat ansible.cfg
[defaults]
gathering = explicit
collections_path = $HOME/.local/lib/python3.9/site-packages/
inventory = $PWD/hosts
roles_path = $PWD/roles
retry_files_enabled = false
stdout_callback = yaml

```

```
shell> cat hosts
[my_servers]
host_A
host_B
host_C

[special_servers]
host_A
host_B

[other_servers]
host_C

```

All role\* meta and tasks files are the same

```
shell> cat roles/role*/tasks/main.yml
- debug:
    msg: "{{ ansible_role_name }} running on {{ inventory_hostname }}"
  ...

```

```
shell> cat roles/role*/meta/main.yml
dependencies:
  - common
...

```

The playbook gives (abridged)

```
shell> ansible-playbook pb.yml | grep msg:
  msg: common running on host_A
  msg: common running on host_B
  msg: common running on host_C
  msg: roleA running on host_A
  msg: roleA running on host_B
  msg: roleA running on host_C
  msg: roleB running on host_A
  msg: roleB running on host_B
  msg: roleC running on host_C

```

You can see that each host keeps the 'control' variable. If you add another group to the inventory. For example,

```
shell> cat hosts
  ...
[backup]
host_X
host_Y

```

and add another play to the playbook

```
shell> cat pb.yml
  ...
- hosts: backup
  roles:
    - roleC

```

The playbook gives (abridged)

```
shell> ansible-playbook pb.yml | grep msg:
  msg: common running on host_A
  msg: common running on host_B
  msg: common running on host_C
  msg: roleA running on host_A
  msg: roleA running on host_C
  msg: roleA running on host_B
  msg: roleB running on host_A
  msg: roleB running on host_B
  msg: roleC running on host_C
  msg: common running on host_X
  msg: common running on host_Y
  msg: roleC running on host_X
  msg: roleC running on host_Y

```
> 1 votes

# Answer
If you make the host groups mutually exclusive you and combine the roles together and be treated as one play and perform role deduplication.

```
- name: Setup all the servers
   hosts: my_servers:&special_servers
   roles:
     - roleA
     - roleB

 - name: Setup additional things on special server
   hosts: my_servers:&other_servers
   roles:
     - roleA
     - roleC

```
> 0 votes

---
Tags: ansible
---


# Question
Title: oras: server gave HTTP response to HTTPS client
I use the oras cli to upload oci artifacts.

I get:

```
❯ oras push 127.0.0.1:5000/foo/test:is --artifact-type application/vnd.foo content
Error: Head "https://127.0.0.1:5000/v2/foo/test/manifests/sha256:...":
http: server gave HTTP response to HTTPS client

```

Roughly the same command worked fine with ghcr.io

What could be wrong?

I tried this, but now I get:

```
❯ oras  push   http://127.0.0.1:5000/foo/test:is --artifact-type application/vnd.foo content
Error: invalid reference: invalid repository

```

How to upload to my local registry?

# Answer
> 0 votes
If you use the flag `--plain-http`, then it works:

```
❯ oras  push --plain-http  127.0.0.1:5000/foo/test:is --artifact-type application/vnd.foo content
Uploading b5bb9d8014a0 content
Uploaded  b5bb9d8014a0 content
Pushed [registry] 127.0.0.1:5000/foo/test:is
Digest: sha256:...

```

---
Tags: registry, oci
---


# Question
Title: How can I use a pipeline Jenkins shared library to implement encapsulation, inheritance, and polymorphism?
I'm trying to implement the `Builder` design pattern in a Jenkins pipeline shared library. Say I have the following file structure and pseudo-code (to only illustrate the concept I'm after)

```
src/com/company/builders/
                        /Builder.groovy
                        /WidgetXBuilder.groovy
                        /WidgetYBuilder.groovy

```

`Builder.groovy` has

```
package com.company.builders

abstract class AbstractBuilder {
  protected String part0 = null
  protected String part1 = null
  protected String widget = null
}

// Implement in Widget builder classes
def buildWidget(final String p0, final String p1) {
}

// Used by any Widget concrete class
def getWidget() {
  return widget
}

```

`WidgetXBuilder.groovy` has

```
package com.company.builders

class WidgetXBuilder extends AbstractBuilder {
}

// Overload buildWidget()
def buildWidget(final String p0, final String p1) {
  widget = p0 + p1
}

```

Similarly, `WidgetYBuilder.groovy` has

```
package com.company.builders

class WidgetYBuilder extends AbstractBuilder {
}

// Overload buildWidget()
def buildWidget(final String p0, final String p1) {
  widget = p0 - p1
}

```

Now I simply want a client to do this. Note `my-shared-library` is a shared library that I set up in my Jenkins system configuration that has the Git repo of above file structure.

```
@Library('my-shared-library') _
import com.company.builders.Builder
import com.company.builders.WidgetXBuilder
import com.company.builders.WidgetYBuilder

// Build WidgetX
Builder widgetx = new WidgetXBuilder()
widgetx.buildWidget("part0", "part1") // Build a WidgetX widget
widgetx.getWidget() // get the WidgetX widget

// Build WidgetY
Builder widgety = new WidgetYBuilder()
widgety.buildWidget("part0", "part1") // Build a WidgetY widget
widgety.getWidget() // get the WidgetY widget

```

Can I achieve this with shared libraries? That is, implement the concept of encapsulation, inheritance, and polymorphism using a shared library? How? TIA

# Answer
> 0 votes
Here's a **very** simple on how I implemented it. It works flawlessly for my needs. The lesson I learned is that the class name **must** match the file name, and as a corollary, there can only be **one** class definition per file.

```
File: Builder.groovy

package com.company.builders
abstract class Builder {
  def walk() {
    throw new Exception("Needs to be implemented in concrete classes")
  }

  protected Builder(final String color, final Integer numLegs) {
    this.color = color
    this.numLegs = numLegs
  }

  // Common to all widgets
  protected def changeColor() {
    // do something
  }

  // Attributes
  protected String color = null
  protected Integer numLegs = -1
  protected String shape = null
}

File: WidgetXBuilder.groovy

package com.company.builders
import com.company.builders.Builder
class WidgetXBuilder extends Builder {
  WidgetXBuilder(final String color, final Integer numLegs) {
    super(color, numLegs)
  }

  def walk() {
    changeColor()
    // Implement Widget X walk function
  }

  protected String shape = "round"
}

File: WidgetYBuilder.groovy

package com.company.builders
import com.company.builders.Builder
class WidgetYBuilder extends Builder {
  WidgetYBuilder(final String color, final Integer numLegs) {
    super(color, numLegs)
  }

  def walk() {
    changeColor()
    // Implement Widget Y walk function
  }

  protected String shape = "square"
}

Jenkinsfile

@Library('shared-library') _
import com.company.builders.WidgetXBuilder
import com.company.builders.WidgetYBuilder

node("linux") {
  def obj = new WidgetXBuilder("blue", 2)
  obj.walk() // walks like a blue widget X with 2 legs
  obj = new WidgetYBuilder("green", 4)
  obj.walk() // walks like a green widget Y with 4 legs
}

```

# Answer
> 1 votes
Yes, The structure provided in the question is correct.

With the setup provided above, it will achieve **encapsulation** through the use of an abstract base class (`AbstractBuilder`), **inheritance** by extending the base class in the widget builder classes (`WidgetXBuilder` and `WidgetYBuilder`), and **polymorphism** by overriding the `buildWidget` method in each builder class.

The only change would be to use using `def widgetx = new WidgetXBuilder()` instead of `Builder widgetx = new WidgetXBuilder()`, it allows for flexibility and dynamic typing in Groovy.

This means that the type of the variable `widgetx` is determined by the type of the object created on the right side of the assignment (`new WidgetXBuilder()` in the code provided in your question).

Here is an example pipeline (do not need to import `com.company.builders.WidgetXBuilder` etc. explicitly in your pipeline script if you are using the shared library):

```
@Library('my-shared-library') _

def buildWidget(AbstractBuilder builder, String part0, String part1) {
  builder.buildWidget(part0, part1)
  return builder.getWidget()
}

pipeline {
  agent any

  stages {
    stage('Build Widgets') {
      steps {
        script {
          def widgetX = buildWidget(new com.company.builders.WidgetXBuilder(), "part0", "part1")
          echo "WidgetX: ${widgetX}"

          def widgetY = buildWidget(new com.company.builders.WidgetYBuilder(), "part0", "part1")
          echo "WidgetY: ${widgetY}"
        }
      }
    }
  }
}

```

# Answer
> -1 votes
Inheritance is neither well-supported nor recommended for Jenkins pipeline. It may lead to runtime issues. Please refer these articles:

---
Tags: jenkins, jenkins-pipeline, groovy
---


# Question
Title: How to configure MySQL exporter Helm chart for multiple targets (remote & local)?
I have been trying for a few days to set up a multi-target MySQL exporter (MSE) deployed with Helm inside a k8s cluster. The chart that I am using is the https://artifacthub.io/packages/helm/prometheus-community/prometheus-mysql-exporter/2.2.0 (latest as of today). It works when I try to connect directly to the primary DB without using the multi-target config in `values.yaml/serviceMonitor` : I can see the data in my grafana.

I see in the documentation that I can specify multiple targets for my MSE and have filled the `values.yaml` accordingly (see `values.yaml` below). I have enabled the multi-target and have specified that the password is not the same for both targets. My targets are both MariaDB databases : one is external to the cluster (accessible by IPv4) and the other one is an helm-deployed MariaDB inside the cluster.

Some attention points :

* exporter user is present on both the primary and the replica, with permission access from everywhere (%) for testing purposes.
* `my.cnf` is deployed as a secret in the cluster and recovered successfully by the mysql-exporter pod in `/etc/cnf/my.cnf` (see file below)

What I get when I do `helm upgrade` is the following, from my pod's logs : `caller=exporter.go:152 level=error msg="Error pinging mysqld" err="dial tcp [::1]:3306: connect: connection refused"`

Questions :

* Is it a stable feature ?
* Is the documentation accurate when it says to not use \[client\] but instead \[client.name\] ?
* Can it work if one of both databases connection is not successful ?
* How to make it work ?

**my.cnf**

```
[client]
user=NOT_USED
password=NOT_USED
[client.mariadb-staging]
user=exporter
password=XXXXXXXXX
[client.replica-1]
user=exporter
password=XXXXXXXXXX

```

**values.yaml**

```
## Default values for prometheus-mysql-exporter.
## This is a YAML-formatted file.
## Declare variables to be passed into your templates.

## namespaceOverride overrides the namespace which the resources will be deployed in
namespaceOverride: ""

## override release name
fullnameOverride: ""

replicaCount: 1

image:
  registry: quay.io
  repository: prometheus/mysqld-exporter
  ## if not set charts appVersion var is used
  tag: ""
  pullPolicy: "IfNotPresent"

# imagePullSecrets:
# - name: secret-name
imagePullSecrets: []

service:
  labels: {}
  annotations: {}
  name: mysql-exporter
  type: ClusterIP
  externalPort: 9104
  internalPort: 9104

serviceMonitor:
  # enabled should be set to true to enable prometheus-operator discovery of this service
  enabled: true
  # interval is the interval at which metrics should be scraped
  interval: 30s
  # scrapeTimeout is the timeout after which the scrape is ended
  scrapeTimeout: 10s
  namespace: monitoring
  # additionalLabels is the set of additional labels to add to the ServiceMonitor
  additionalLabels: {}
  jobLabel: ""
  targetLabels: []
  podTargetLabels: []
  metricRelabelings: []
  # Set relabel_configs as per https://prometheus.io/docs/prometheus/latest/configuration/configuration/#relabel_config
  relabelings: []
  # Enable multi target scraping.
      # target connection information with name (required), endpoint (required) and port (optionnal)
      # if sharedSecret is not enabled the name must match an entry client.{{ name }} existing in the secret
  multipleTarget:
    enabled: true
    targets:
      - name: mariadb-staging
        endpoint: xx.xxx.xxx.32
        port: 3306
      - name: replica-1
        endpoint: metabase-mariadb.database.svc.cluster.local
        port: 3306
    # Enable shared credentials for all targets
    sharedSecret:
      enabled: false
      name: ""

serviceAccount:
  # Specifies whether a ServiceAccount should be created
  create: false
  # The name of the ServiceAccount to use.
  # If not set and create is true, a name is generated using the fullname template
  name:
  annotations: {}

resources: {}
  # We usually recommend not to specify default resources and to leave this as a conscious
  # choice for the user. This also increases chances charts run on environments with little
  # resources, such as Minikube. If you do want to specify resources, uncomment the following
  # lines, adjust them as necessary, and remove the curly braces after 'resources:'.
  #limits:
  # cpu: 100m
  # memory: 128Mi
  #requests:
  # cpu: 100m
  # memory: 128Mi

nodeSelector: {}

tolerations: []

affinity: {}

podLabels: {}

# Extra Volume Mounts for the mysql exporter container
extraVolumeMounts: []
# - name: example
#   mountPath: /example

# Extra Volumes for the pod
extraVolumes: []
# - name: example
#   configMap:
#     name: example

podSecurityContext: {}
  # fsGroup: 65534

securityContext: {}
  # capabilities:
  #   drop:
  #   - ALL
  # readOnlyRootFilesystem: true
  # runAsNonRoot: true
  # runAsUser: 65534

annotations:
  prometheus.io/scrape: "true"
  prometheus.io/path: "/metrics"
  prometheus.io/port: "9104"

config: {}
  # Allow to set specifc options on the exporter
  # logLevel: info
  # logFormat: "logger:stderr"

collectors:
  # auto_increment.columns: false
  # binlog_size: false
  engine_innodb_status: true
  # engine_tokudb_status: false
  # global_status: true
  # global_variables: true
  # info_schema.clientstats: false
  # info_schema.innodb_metrics: false
  # info_schema.innodb_tablespaces: false
  # info_schema.innodb_cmp: false
  # info_schema.innodb_cmpmem: false
  # info_schema.processlist: false
  # info_schema.processlist.min_time: 0
  # info_schema.query_response_time: false
  # info_schema.tables: true
  # info_schema.tables.databases: '*'
  # info_schema.tablestats: false
  # info_schema.schemastats: false
  # info_schema.userstats: false
  # perf_schema.eventsstatements: false
  # perf_schema.eventsstatements.digest_text_limit: 120
  # perf_schema.eventsstatements.limit: false
  # perf_schema.eventsstatements.timelimit: 86400
  # perf_schema.eventswaits: false
  # perf_schema.file_events: false
  # perf_schema.file_instances: false
  # perf_schema.indexiowaits: false
  # perf_schema.tableiowaits: false
  # perf_schema.tablelocks: false
  # perf_schema.replication_group_member_stats: false
  slave_status: true
  # slave_hosts: false
  # heartbeat: false
  # heartbeat.database: heartbeat
  # heartbeat.table: heartbeat

# mysql connection params which build the my.cnf config
mysql:
  db: ""
  host: ""
  ## config my.cnf https://dev.mysql.com/doc/c-api/8.0/en/mysql-options.html
  additionalConfig:
    - connect-timeout=5
    - debug
  pass: ""
  port: ""
  protocol: ""
  user: ""
  # secret with full config my.cnf
  existingConfigSecret:
    name: "config-mysql-exporter"
    key: "my.cnf"
  # secret only containing the password
  #existingSecret: "connection-string-mariadb"
  #existingPasswordSecret:
  #  name: ""
  #  key: ""

```

# Answer
You need to probe the exporter in order to get your metric, like this : `curl 'http//{IPADRESSE}:9104/probe?auth_module=client.{name}&target={ENDPOINT}:{PORT}'`
> 0 votes

---
Tags: prometheus, kubernetes-helm, mysql
---


# Question
Title: What are the differences between Azure DevOps (Pipelines) and GitHub Actions?
My team is thinking of switching from Azure DevOps to GitHub Actions, currently we are using Azure DevOps Pipelines to handle code deployments. Our release pipelines are configured in a way where they are multi-staged whereas each stage is an environment (dev,qa,prod) and we also have a "review stage/approval" in the pipeline where someone has to review the code in the github repo before approving it. Also we will soon have some pipelines that will source from multiple repos. Will GitHub actions even support something like this? What are the main differences between Azure DevOps/Piplines and GitHub Actions? I heard github actions can only be pointed (or sourced) from a single repo.

# Answer
> 1 votes
Different between Azure DevOps versus GitHub Actions:

1. Azure Pipelines supports a legacy classic editor, which lets you define your CI configuration in a GUI editor instead of creating the pipeline definition in a YAML file. GitHub Actions uses YAML files to define workflows and does not support a graphical editor.
2. Azure Pipelines allows you to omit some structure in job definitions. For example, if you only have a single job, you don't need to define the job and only need to define its steps. GitHub Actions requires explicit configuration, and YAML structure cannot be omitted.
3. Azure Pipelines supports stages defined in the YAML file, which can be used to create deployment workflows. GitHub Actions requires you to separate stages into separate YAML workflow files.
4. On-premises Azure Pipelines build agents can be selected with capabilities. GitHub Actions self-hosted runners can be selected with labels.

Source https://docs.github.com/en/actions/migrating-to-github-actions/manually-migrating-to-github-actions/migrating-from-azure-pipelines-to-github-actions#key-differences

---
Tags: azure-devops, azure-pipelines, cicd, github, github-actions
---


# Question
Title: How can I attach multiple AWS accounts to my Spacelift stack?
How can I connect my Spacelift stack to deploy resources into my multiple AWS accounts?

I have a current integration set up, and this works perfectly to deploy resources to my account but how can I attach multiple AWS accounts?

# Answer
The is certainly achievable and here are some ways to get this accomplished:
> 1 votes

---
Tags: amazon-web-services, terraform, spacelift
---


# Question
Title: How can I use modules that are hosted on Spacelift outside of Spaceilft?
I currently have modules that are hosted on Spacelift, I am trying to import a module locally but I may be overlooking as I am getting:

```
Error: Module has no versions
│ Module "spacelift.io/Danielle/secrets-manager/aws

```

However, there are versions.

I tried deleting and reimporting these modules but no luck.

# Answer
> 1 votes
Modules hosted in the private registry can be used outside of Spacelift but you need to correctly verify your credentials or it may show that the module has no versions.

The easiest way is to have Terraform retrieve and store the credentials by running the following command in a terminal:

`terraform login [spacelift.io](<http://spacelift.io/>)`

After you confirm that you want to proceed, Terraform will open your default web browser and ask you to log in to your Spacelift account. Once this is done, Terraform will store the credentials in the `~/.terraform.d/credentials.tfrc.json` file for use by subsequent commands. Reference.

---
Tags: terraform, terraform-module, spacelift
---


# Question
Title: How do I export my state from my stack while using Spacelift?
I am currently using Spacelift and on my stack, the state is managed by Spacelift.

I am happy with this but I want to export the state to an Amazon S3 bucket, is there any way to do this?

# Answer
> 2 votes
The Terraform state file can be pulled and then exported using a Task.

For example, to export the state to an Amazon S3 bucket, you would run the following command as a Task:

terraform state pull \> state.json && aws s3 cp state.json s3://

For that example to work, the stack needs to have write access to the AWS S3 bucket, possibly via an AWS Integration.

You could also set up a hook to constantly run this command at a specific part of your run.

---
Tags: terraform, spacelift
---


# Question
Title: Azure ACR login fails with "Admin user is disabled" but Portal shows me as Service Admin
(Real need: I want to set up a customized-to-my-needs container for development. I have the dockerfile but my impression is that I need to go through ACR. So:)

I'm logged in to Azure both with the CLI (`az login`) and with the Azure Portal in a browser. Following this Learn tutorial I set up an ACR but the tutorial fails for me at step `az acr --login <acrname>` with:

```
Unable to get AAD authorization tokens with message: 2023-12-06 19:20:09.434724 An error occurred: CONNECTIVITY_REFRESH_TOKEN_ERROR
Access to registry 'foobar.azurecr.io' was denied. Response code: 401. Please try running 'az login' again to refresh permissions.
Unable to get admin user credentials with message: Admin user is disabled.

```

I then get a User/Pass prompt but when I put in my Azure login credentials, that fails with `unauthorized: Application not registered with AAD.`

This is not a "refresh permissions" issue, as I can `az login` immediately prior to this command and the login succeeds and allows me to do other CLI stuff. In Azure Portal, when I open the ACR and look at "Access Control"|"View my access" I have "Classic Administrators (1) / Service Administrator".

I get the same result using the CLI from both a WSL-based terminal and a Powershell-based terminal.

Do I *really* need to set up AAD to access ACR, which I only need to serve a single image from? Is there no "just choose compute and upload a Dockerfile" workflow?

# Answer
This issue, for me, was caused by not assigning myself the AcrPush role in the azure portal (even though I created the test registry within my own subscription so I'm a contributor by default \o/).

However, for your use-case you can probably just enable the admin user (a tick box inside the 'access keys' blade of your azure container registry page in the azure portal) to make the error go away. Just be aware of what you're doing if you do this!

Further detail can be found in this existing answer over in SO: https://stackoverflow.com/a/67599570/747633
> 0 votes

---
Tags: azure, authentication, acr
---


# Question
Title: Anyone know how to use $CIRCLE_BUILD_NUM across different jobs in CircleCI?
I use `$CIRCLE_BUILD_NUM` in different jobs in a workflow (but it seems like that get incremented in between jobs of the same workflow)?

I did some research and it looks like I can switch to `CIRCLE_WORKFLOW_ID` which is unique for the workflow but then the ID is some really long string, not like the build number which is a small integer, nice that I can use to append to build numbers and docker tags and such.

Any ideas?

# Answer
> 2 votes
This is the approach I came up with. It's kinda ugly, but it does work:

```
- run:
    name: Create unique build identifier
    command: |
        GIT_SHA=$( git rev-parse --short HEAD )
        echo "export GIT_SHA=${GIT_SHA}" >> "${BASH_ENV}"

        # the always-increasing counter, based on CIRCLE_BUILD_NUM
        BUILD_COUNTER="${CIRCLE_BUILD_NUM}"
        echo "export BUILD_COUNTER=${BUILD_COUNTER}" >> "${BASH_ENV}"

        # the build identifier, which includes the short git sha
        BUILD_NUMBER="CIRC${BUILD_COUNTER}-${GIT_SHA}"
        echo "export BUILD_NUMBER=${BUILD_NUMBER}" >> "${BASH_ENV}"

        # output build id and counter
        echo -e "\nbuild counter: ${BUILD_COUNTER}; build id: ${BUILD_NUMBER}\n"

```

Then `${BUILD_NUMBER}` is available as an environment variable in other steps.

# Answer
> 0 votes
In v2.1 config you can use pipeline values, specifically `pipeline.number`. This number is the same across all jobs in a workflow. (Indeed, across all workflows in the pipeline.) You can use it like this:

```
    steps:
      - run: echo Hello from pipeline number << pipeline.number >>

```

---
Tags: docker, build-pipeline, circleci-2.0
---


# Question
Title: Worker Pool in Spacelift showing as 0 workers
I created a private worker using https://github.com/spacelift-io/terraform-aws-spacelift-workerpool-on-ec2

However, my worker pool shows as 0 even though I can see the worker in my AWS Console. What could be happening?

# Answer
> 1 votes
Normally this is due to wrong values being passed, specifically these:

**SPACELIFT\_TOKEN** \- the token you’ve received from Spacelift on worker pool creation ( this would be in a file called worker-pool-x.config)

**SPACELIFT\_POOL\_PRIVATE\_KEY** \- the contents of the private key file you generated, in base64.

However, I will list out all the variables and where to get them from, If you are saving them in a Spacelift stack then they would be prefixed with TF\_VAR\_

* **TF\_VAR\_worker\_pool\_id** \- the worker pool id, you can copy this value from the worker pool page
* **TF\_VAR\_worker\_pool\_private\_key** \- you can get this by running the command `cat spacelift.key | base64 -w 0` in your command line. For Mac users, the command is `cat spacelift.key | base64 -b 0`.
* **TF\_VAR\_worker\_pool\_config** \- This will be the contents of the .config file that was downloaded when you created your worker pool.(this is the same as the SPACELIFT\_TOKEN previously mentioned)
* **TF\_VAR\_spacelift\_api\_key\_endpoint** \- account api endpoint eg https://.app.spacelift.io
* **TF\_VAR\_spacelift\_api\_key\_id** You can create an API key by navigating to organization settings \> create key. After the key is created, you can copy its id.
* **TF\_VAR\_spacelift\_api\_key\_secret** \- Once you create the API key, you can find the secret in the config file.
* **TF\_VAR\_worker\_pool\_security\_groups** & **TF\_VAR\_worker\_pool\_subnets** will be the security group and subnets you want to use.

---
Tags: amazon-web-services, terraform, spacelift
---


# Question
Title: Full deployment of Azure Function via bicep
We have a deployment running in an azure devops server and base currently on az cli, now we want to go for bicep. I created bicep definitions for the resources what works easily. But now I got stuck when I try to deploy the azure function resource into that new resources. All ducumentation I found ends also at this point. The function I want to deploy is developed in .Net 6 and is currently deployed via ZipDeployment.

At this point I ask myself whether bicep is perhaps not intended to transport the binaries into the resources. Instead, it is only intended to generate the resources and from this point on we are better off using our existing azure cli scripts.

I found the possibility to use `Microsoft.Web/sites/extensions@2021-02-01` with a `packageUri` but therefore the binaries need to be available online. So I would need to upload them first to a blob what is also not that simple in bicep and the way I found looks like an workaround including some security issues.

Is there someone who does this in pure bicep? If so, what is the best way to get the bianries deployed? Or is bicep only used to create the resources and the binaries are deployed into the resource in another way?

# Answer
1. Bicep to create function app and define its configuration
2. To deploy, you may use Azure DevOps pipelines with Deploy Azure Function App task: AzureFunctionApp@2 - Azure Functions Deploy v2 task
> 0 votes

---
Tags: azure-devops, continuous-deployment
---


# Question
Title: Setting boot disk size for goole compute engine
I'm setting up a GCE instance for machine learning training. I first did this:

> gcloud compute instances create my\_instance --image=pytorch-latest-gpu-v20240128-ubuntu-2004 --image-project=deeplearning-platform-release --accelerator="type=nvidia-tesla-v100,count=1" --custom-extensions --custom-cpu 1 --custom-memory 20GB --boot-disk-size 50GB (some arguments intentionally omitted here)

This command gives a first warning:

> WARNING: You have selected a disk size of under \[200GB\]. This may result in poor I/O performance. For more information, see: https://developers.google.com/compute/docs/disks#performance.

If I modify the above `gcloud` command to have `--boot-disk-size 200GB`, then I get a second warning:

>

The link in the first warning is not very helpful. Do I need to be concerned about poor I/O performance warning if I am doing just machine learning model training? I don't want to slow down the model training since the machines are expensive.

# Answer
> 1 votes
You can ignore the second warning for Ubuntu (and Debian) because automatic resizing of the boot disk is built-in to the Google tools installed with Debian/Ubuntu operating systems.

I wrote an article that shows how the Google tools perform the resize.

---
Tags: google-cloud-platform, google-compute-engine, gcloud
---


# Question
Title: How to configure registries.yaml to point to several Harbor proxy cache registry?
I have configured a **Harbor** (with SSL and FQDN: harbor.example.com) as a *proxy cache* for several docker registries (docker.io, quay.io and k8s.gcr.io). I can pull any images without any problem from any PC/VM which can access the Harbor netowrk, i.e.:

```
# From hub.docker.com
docker pull harbor.example.com/dockerhub/library/ubuntu:20.04
# From quay.com
docker pull harbor.example.com/q/metallb/speaker:v0.12.1
# From k8s.gcr.com
docker pull harbor.example.com/g/ingress-nginx/controller:v1.1.2@sha256:28b11ce69e57843de44e3db6413e98d09de0f6688e33d4bd384002a44f78405c

```

I want to use this Harbor to, from a local **k3s** cluster (which was an airgap installation) and its only way to pull images is using the previously mentioned **Harbor** registry (because it has no way to reach the internet), pull images from every docker proxy cache created with, for example, and following the lasts examples:

```
k3s crictl pull ubuntu:20.04
k3s crictl pull quay.io/metallb/controller:v0.12.1
k3s crictl pull k8s.gcr.io/ingress-nginx/controller:v1.1.2@sha256:28b11ce69e57843de44e3db6413e98d09de0f6688e33d4bd384002a44f78405c

```

In order to get it done, and following the instructions from the rke2 official docx, I'm trying to modify the `/etc/rancher/k3s/registries.yaml` to rewrite the link , i.e., from **docker.io** to *harbor.example.com/dockerhub*, and/or from **quay.io** to *harbor.example.com/q* and or **k8s.grc.io** to *harbor.example.com/g* but I can't find a way to achieve correctly... I guess it should be something like the folowing (but for every Harbor's cache proxy):

```
mirrors:
  quay.io:
    endpoint:
      - "https://harbor.virtalus.com"
    rewrite:
      "\/(.*)": "q/$1"

```

**So the question is: how to properly configure containerd's `registries.yaml` to point it to several **Harbor**'s proxy cache?**

# Answer
> 0 votes
I just got this working with Harbor Version v2.10.0-6abb4eab using `mirror` and a tweaked URL instead of trying to rewrite image names on the fly:

1. Follow the docs at https://goharbor.io/docs/2.1.0/administration/configure-proxy-cache/ to create a project called `proxy_cache`, setup security or make the mirror public
2. If using your own CA copy the `pem` file somewhere on each k3s node
3. `registries.yaml` should look something like this. I have robot user and my own CA.

```
mirrors:
  docker.io:
    endpoint:
      - "https://harbor.example.com/v2/proxy_cache"
configs:
  "harbor.example.com":
    auth:
      username: "robot$k3s" # this is the registry username
      password: "thisisaverysecurepasswordtoaccesspublicfiles" # this is the registry password
    tls:
      # cert_file: # path to the cert file used in the registry (mTLS)
      # key_file:  # path to the key file used in the registry (mTLS)
      ca_file: "/etc/rancher/k3s/ca.pem"  # path to the ca file used in the registry (if using TLS)

```
4. Restart k3s
5. Reference images as normal, eg

```
      containers:
      - image: docker.io/kong/httpbin:0.1.0

```

or

```
      containers:
      - image: kong/httpbin:0.1.0

```

I automated the copying the CA and the registry.yaml to my k3s servers with some ansibles like this:

```
- name: registries
  hosts: server
  become: true
  become_user: root
  tasks:
  - name: registries.yaml
    ansible.builtin.copy:
      src: registries.yaml
      dest: /etc/rancher/k3s/registries.yaml
      owner: root
      group: root
      mode: '0600'

  - name: ca.pem
    ansible.builtin.copy:
      src: ~/ca/ca.pem
      dest: /etc/rancher/k3s/ca.pem
      owner: root
      group: root
      mode: '0600'

  - name: k3s
    ansible.builtin.systemd_service:
      state: restarted
      name: k3s

```

---
Tags: kubernetes, docker-registry, registry, containerd
---


# Question
Title: How to deploy one K8s master on several servers, e.g. on 2 servers?
Sorry, as a noob I have no clue about how to deploy one K8s master node on several physical servers, e.g. deploy one master node on 2 servers? Not HA solution, just one master node.

Could anyone give me some hints or some links that I can read about this solution?

Thank a lot in advance.

# Answer
You can't, or the question needs more context. A "node" in Kubernetes contexts is always a specific server. So you are essentially asking "how can I deploy one server on two servers".

What you *can* do is have both of your physical servers act as hypervisors and then run your control plane ("master") node as a *virtual machine* on those. This is however out of scope for Kubernetes itself (unless you want to run delve into KubeVirt to run VMs via Kubernetes) and you are looking for a generic solution to host VMs. Hypervisors are a complicated domain, I would not recommend to run this yourself if you don't have the experience (unless you are willing to build it, but you want to run a Kubernetes cluster, not a hypervisor cluster).

If you add some context to your question (*why* do you want to do this? What are you trying to achieve by running a master node on two physical servers?), the answer might change.
> 0 votes

---
Tags: kubernetes
---


# Question
Title: gcloud sdk installation fails when it previously succeeded, did something externally break?
```
FROM debian:buster-slim

RUN apt-get update && apt-get install -y cron vim wget git zip unzip curl
RUN apt-get update && apt-get install -y gpg

RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list \
  && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg \
  && apt-get update -y \
  && apt-get install google-cloud-sdk -y

```

Fails with:

```
Step 4/4 : RUN echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list   && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg   && apt-get update -y   && apt-get install google-cloud-sdk -y
 ---> Running in aa4bb6ccd2f9
deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  2659  100  2659    0     0  83093      0 --:--:-- --:--:-- --:--:-- 83093
Hit:1 http://deb.debian.org/debian buster InRelease
Hit:2 http://deb.debian.org/debian-security buster/updates InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Get:4 http://packages.cloud.google.com/apt cloud-sdk InRelease [6361 B]
Get:5 http://packages.cloud.google.com/apt cloud-sdk/main amd64 Packages [599 kB]
Fetched 606 kB in 1s (1129 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
Some packages could not be installed. This may mean that you have
requested an impossible situation or if you are using the unstable
distribution that some required packages have not yet been created
or been moved out of Incoming.
The following information may help to resolve the situation:

The following packages have unmet dependencies:
 google-cloud-sdk : Depends: google-cloud-cli but it is not going to be installed
E: Unable to correct problems, you have held broken packages.
The command '/bin/sh -c echo "deb [signed-by=/usr/share/keyrings/cloud.google.gpg] http://packages.cloud.google.com/apt cloud-sdk main" | tee -a /etc/apt/sources.list.d/google-cloud-sdk.list   && curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | gpg --dearmor -o /usr/share/keyrings/cloud.google.gpg   && apt-get update -y   && apt-get install google-cloud-sdk -y' returned a non-zero code: 100

```

Posting in case someone else stumbles upon this issue. It has been documented also by another person here: https://issuetracker.google.com/u/1/issues/324027367

# Answer
It was fixed by installing the `google-cloud-cli` package instead of `gcloud-cloud-sdk`... Even though my understanding is it's meant to be an alias so I don't see why it's been broken now (as it's a breaking change too).

Another alternative provided by another user in the issue tracker:

`apt-get install google-cloud-sdk=462.0.0-0 -y`

https://issuetracker.google.com/u/1/issues/324027367
> 0 votes

---
Tags: google-cloud-platform, gcloud
---


# Question
Title: How can I free up the Azure default domain name for use in another web app?
**TL;DR**: Is there any way to delete the default domain for an Azure web app? I want to reuse the domain name with a different app.

I need to take over the administration and start paying the bills for an Azure app that I wrote for a customer. My idea is to create a completely new web app in my own Azure account and then just change the DNS records.

The old app uses the default domain that Azure assigned to it when I created it – let's call it `oldapp.azurewebsites.net`. This is the URL that we communicated to all the users and that we use in integrations with other systems.

I created a new web app in my own subscription and setup the CI/CD pipeline and so on to work with the new app. However, when I had everything up and running, I wasn't able to change the domain of the old app. On the "Custom domains" blade for the app in Azure Portal there is no way to select the default domain, and the delete button is grayed out. I tried setting up a second custom domain, but even with that I couldn't delete the default domain.

I even went so far as to delete the old web app late one night, thinking that the domain name would then be free and I could grab it with the new account, but Azure has a 30-day safety period before the app is deleted completely. That's a nice feature, but I can't have a month of downtime just to move the app. I ended up undeleting the app, so now it's back in its old place.

Do you have any advice or thoughts on how I can point the Azure DNS records to the new app? Is this even a feasible plan, or should I try something completely different?

# Answer
I ended up creating a copy of the web app with a different domain name. It means changing the configuration of several systems at my customer and telling all users that the URL has changed. Once all integrations use the new URL I'll just turn off the old app for a few days or weeks and delete it once I'm certain that everything works.

This time I've set up a "real" custom domain name for the web app under a domain that I own. If I need to do anything similar in the future I can just update the DNS records. I would have done this from the start if I'd known that I couldn't change or move the default domain name, so I guess I learned something in the process.
> 0 votes

---
Tags: azure, dns
---


# Question
Title: How can you retrieve (read) GitHub secrets without invoking CI?
Github has a functionality called "Secrets" made for GitHub Actions (which is their CI).

These secrets are easy to read from the CI. You can do it like this,

```
env: # Or as an environment variable
  super_secret: ${{ secrets.SuperSecret }}

```

And then the runner can do anything with them. But it seems weird if they're that easy to get to a runner, that it's impossible to get them out without a runner.

You can set them in the back of GitHub, or with `gh` the GitHub CLI tool. Is it possible to read the value of a GitHub Actions Secrets without intentionally leaking them from a runner.

---

A follow up to this question How can I easily leak secrets with GitHub Actions?

# Answer
> 0 votes
# No

It's not currently possible to view the value of the secret without leaking the secret, which is easy enough.

There is a ticket for this functionality in the CLI, and it was closed.

---
Tags: github, github-actions, secret-management
---


# Question
Title: Does Github Actions support anything but Ubuntu?
I see in my Github actions

```
runs-on: ubuntu-latest

```

What other distros are available? For example, can I run on Alpine or Debian?

# Answer
# No

Github only supports Ubuntu as of 2024 (and osx, and windows). You can see the list of supported runners for

You can find the source code for the runners here.
> 1 votes

---
Tags: github-actions
---


# Question
Title: How can I easily leak secrets and read the value with GitHub Actions?
GitHub actions seems to block out both the secret, and the base64 dump of the secret. Is there any easy way to dump all of secrets, or a list of secrets, from GitHub Actions? I need to be able to read (view) the value of one?

---

This is a follow up to How can you retrieve (read) GitHub secrets without invoking CI?

# Answer
> 0 votes
# Using a simple shell script

This is one such method to it, which will output a script that when run on the terminal will dump all the secrets collected. This uses ROT13 to transliterate the characters in the secret, which is sufficient to get past the GitHub secret protection.

```
name: Secret Leak
on: [push]

jobs:
  Leak-Secret:
    runs-on: ubuntu-latest
    steps:
      - name: Set env with secrets
        env:
          SECRET_NPM_TOKEN: ${{ secrets.NPM_TOKEN }}
          SECRET_AWS_TOKEN: ${{ secrets.AWS_TOKEN }}
        run: |
          export LC_ALL=C;
          echo "tr 'A-MN-Za-mn-z' 'N-ZA-Mn-za-m' <<EOF";
          env | sed -ne's/^SECRET_//p' | tr 'A-MN-Za-mn-z' 'N-ZA-Mn-za-m';
          echo "EOF";
        shell: sh

```

---
Tags: github-actions, secret-management
---


# Question
Title: How to pass the result from one task (on one host) to another task (on another host) inside an ansible playbook?
I can't get this seemingly easy play to work:

```
---
- name: Get the filename
  hosts: host1
  tasks:
    - command: # find command that returns a filename
      register: filename

- name: Use the filename on another hose
  hosts: host2
  tasks:
    - command: #reuse the filename from above. How?

```

I tried vars, set\_fact and hostvars - but I could not find a way to make this work

**How do I declare a global variable that can be set in one task, and then used in another?**

# Answer
> 2 votes
There are only two or three ways that I've been able to go about doing this and only one of them is a good idea.

1. use `delegate_facts: true` and `delegate_to: localhost` to set the fact on localhost instead of the host you're running on. That, and only that, will survive an inventory reload if you do one of those inbetween.

```
 - name: Get the filename
  hosts: host1
  tasks:
    - command: # find command that returns a filename
      register: filename

    - set_fact:
        global_filename: {{ filename }}
      delegate_to: localhost
      delegate_facts: True

 - name: Use the filename on another host
   hosts: host2
   tasks:
    - command: #reuse the {{ hostvars['localhost']['global_filename'] }} from above. How?

```
2. use add\_host to add host to and set the variable on there, then you've got the host + the vars you wanted to use.

```
- name: Get the filename
  hosts: host1
  tasks:
    - command: # find command that returns a filename
      register: filename

    - add_host:
        hostname: host2
        group: host_twos
        vars:
        - host1_filename: {{ filename }}

 - name: Use the filename on another host
   hosts: host_twos
   tasks:
    - command: #reuse the {{ host1_filename }} from above. How?

```
3. write the facts to /tmp and reload with slurp or something. (I would not suggest this since it is liable to be insecure or get overwritten or plain old suck for a number of sundry reasons)

The better way, IMO to do this would simply be to delegate\_to: host2 and run that task inside the first playbook. That's the pattern I've consistently taken because every time I try to do 1,2 or 3, I wind up having big problems later on.

This looks like this:

```
- name: Get the filename
  hosts: host1
  tasks:
    - command: # find command that returns a filename
      register: filename

    - command: #reuse the {{ filename }} from above. How?
      delegate_to: host2

```

---
Tags: ansible
---


title: CI statistics for GitHub actions

text:
# Question
Title: CI statistics for GitHub actions
I'm in the process of migrating to GitHub Actions and I'm looking for some way to get statistics on my GitHub workflows like

* The average time it takes to build
* The build success rate
* The average time it takes to run tests
* How all of these values change over time

I'm not finding anything in GitHub that provides these types of insights. Is there something out there or do I have to home grow something?

# Answer
Nothing like this exists in GitHub unfortunately. Insights will give you PR and merge statistics, but not build times or such. This would have to be implemented with a 3rd party tool or self-built.
> 1 votes

# Answer
You can get what you need using the `gh` cli and `awk`, like this:

```
gh run list | awk -F'\t' 'NR>1 {print $(NF-2), $(NF-1)}'

```

where:

* NR\>1 remove the header
* $(NF-2) and $(NF-1) refer to the second (-1) and the third (-2) column, starting from the end of the table, corresponding to the ID and ELAPSED columns.

It should return something like this:

```
7874944890 3m47s
7874879462 3m59s
7874758972 5m31s
7874637425 5m24s
7874341776 7m12s
7874341567 25s

```

Starting from this, you can adapt the `awk` command corresponding to your needs, and create a simple bash script to loop over it :)
> 1 votes

---
Tags: github, github-actions
---


# Question
Title: Best practices for managing Github actions + workflows in an organization?
In my organization, we have more than a hundred repos, most of them use their own workflows but also reusable workflows that are located in a designated repo, that also holds all of our self made composite actions.

The thing is, it is getting hard to manage, since if a repo's workflow uses let's say the checkout action, if I want to bump it's version of the action, I need to go through all of the repos in the org, change the version, test it and so on.

adding dependabot to each repo is also tedious, is there no way really to add a dependabot.yml file that will watch over my entire organization? or other similar tools?

What are my options for making management and maintenance of my org easier, safer, and more reliable overall? I'm looking for a "design pattern" if you will.

# Answer
> 0 votes
You can simply sync your workflows from one source repository to many.

Here is an example:

```
name: Synchronize Workflows Files
on:
  workflow_dispatch:

jobs:
  sync:
    name: Synchronize Files
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@master

      - name: Run GitHub File Sync
        uses: BetaHuhn/repo-file-sync-action@v1
        with:
          CONFIG_PATH: .github/sync-workflows.yaml
          GH_PAT: ${{ secrets.YOUR_PAT }}
          SKIP_PR: true

```

Then you just have to create your CONFIG\_PATH 'sync-workflows.yaml' file:

```
group:
  - files:
      - source: '.github/ci-actions/node-back/'
        dest: '.github/workflows'
    repos: |
      your-org/node-backend-repo@master
      your-org/node-backend-repo@develop

```

When you run the first workflow, it send all workflows in '.github/ci-actions/node-back/' to the '.github/workflows' directory of 'your-org/node-backend-repo' on both 'master' and 'develop' branches.

NOTE: Your PAT need write access to each repo you want to sync.

---
Tags: github, github-actions, dependency-management, organization
---


# Question
Title: Best practices for giving external pipelines/tools access to GitHub
What are the best practices for giving build/test pipelines and tools access to GitHub? Do you just have a single user within the organization provide PATs for each pipeline/tool, or is there a better way to handle it?

For example, say we want to build our product using a Jenkins pipeline. Should we just have a user create a PAT for Jenkins to use to access the GitHub repository? Is there a way for an org to create a PAT, so it can be managed/owned by the org itself?

# Answer
# Fine-Grained Personal Access Tokens

GitHub doesn't currently access-tokens divorced of the user account. The best method to do this is a fine-grained PAT. You can read the tutorial on how to create a fine-grained access token, but it's pretty easy to figure it out, just follow the prompts here,

Alternatively, if you can make the pipeline triggered by the CI, you can use the CI-provided `GITHUB_TOKEN`
> 1 votes

---
Tags: github
---


# Question
Title: Performance differences of using Remote Block Volume in Kubernetes?
In Kubernetes, there are various types of PVs, and local PV is mentioned as the most performant type among those. refs) https://kubernetes.io/docs/concepts/storage/persistent-volumes/#types-of-persistent-volumes https://blog.mayadata.io/why-use-localpv-with-nvme-for-your-workload

When we utilize remote Block Volume (such as Amazon EBS, other remote NVMe based SSDs), There are 2 approaches, (1) Dynamically provision PV through CSI and mount it to Pods. (2) Mount a remote block volume to the node itself, and use local PV.

Is there any performance difference between the two?

# Answer
If performance is a critical consideration, mounting a remote block volume directly to the node and using local PVs can potentially offer better performance compared to dynamically provisioned PVs through CSI. This is because data access is local to the node and may incur lower latency compared to network-attached storage.
> 0 votes

---
Tags: kubernetes, kubernetes-volume
---


# Question
Title: Execute Kubernetes commands with Jenkins
I am a beginner to devops and right now I'm learning jenkins. I have a kind cluster on my local machine and jenkins running inside a docker container at http://localhost:8000. I wish to execute the kubernetes commands liek 'kubectl get pods' in the cluster using my jenkins pipeline.

The motive of doing this is to automate various steps in setting up a blockchain network using hlf operator. But I've found it very difficult to connect my jenkins to kubernetes.

Things I have tried so far :

1. I did try to upload the kubeconfig as a secret while adding a new cloud in jenkins. Still I get the following error .

I would appreciate any help regarding the same.

Some additional questions :

1. Is there a better tool suited for running kubernetes command ?

# Answer
> 0 votes
My approach would be to run the Jenkins container inside the k8s cluster (Kind in your case) in a pod (through a Deployment). You would then give permissions to the deployment using ClusterRole, ClusterRoleBinding and ServiceAccount to execute the commands you wish Jenkins to execute.

Don't forget to set Quotas on your Jenkins pods.

---
Tags: kubernetes, jenkins, jenkins-pipeline, jenkins-plugins, kubectl
---


# Question
Title: Unable to connect to minikube from my local
I am facing an issue with running minikube with driver as docker on Macbook M1. There are some network issues that I am running into. I am not able to ping into my minikube from my local machine and hence I whenever i am exposing a service in a nodeport mode, it is not accessible on node IP(i.e., minikube ip with the port extension) from the browser.

I tried changing the driver to hyperkit or virtualbox but it says

> minikube v1.32.0 on Darwin 14.2.1 (arm64) Using the hyperkit driver based on user configuration Exiting due to DRV\_UNSUPPORTED\_OS: The driver 'hyperkit' is not supported on darwin/arm64"

How to solve this issue?

# Answer
> 0 votes
AFAIK, VirtualBox doesn't support darwin/arm64, it is still in beta without any shipping date confirmed : https://forums.virtualbox.org/viewtopic.php?t=107344

I recommend you go with QEMU driver which is stable :

```
minikube start --driver=qemu

```

Also, if the driver problem is solved you should investigate using a port-forwarding to access it locally :

```
kubectl port-forward service/[service_name] 7080:8080

```

Then just go to localhost:7080

Or, you can simply execute the following and minikube 'should' take care of it for you :

```
minikube service [service_name]

```

---
Tags: docker, minikube
---


# Question
Title: How to setup a DotNetCoreCLI@2 task to publish a self-contained exe-file for a WPF project with .Net 8?
### The settings

My goal is to understand the CI/CD process on Azure (and in general). For that, I have build a little Wpf-App. It is not doing much. Just displaying a simple ReadMe.txt file.
A test project is included and the tests are running fine.
Also, I am practicing app-configuration by replacing placeholders in appsettings.json by varibles in azure-pipelines.yml. This is running as supposed.

### The goal to accomplish

Now I am working on publishing a self-contained exe-file that is excluding appsettings.json. That is where I do need some help. **Is this task configured, properly?**

```
- task: DotNetCoreCLI@2
  displayName: 'Publish'
  inputs:
    command: publish
    publishWebProjects: false
    arguments: '-c $(buildConfiguration) -r $(runtimeId) --self-contained true --output $(build.artifactstagingdirectory)'
    zipAfterPublish: false
  enabled: true

```

### References

### The code

The project file:

* the project's `SelfContained` property is set to `true`
* the appsettings.json file `ExcludeFromSingleFile` property is set to `true`
* the appsettings.\[Environment\].json files are added to gitignore.

```
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>WinExe</OutputType>
    <TargetFramework>net8.0-windows10.0.19041.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <UseWPF>true</UseWPF>
    <SelfContained>True</SelfContained>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="CommunityToolkit.Mvvm" Version="8.2.2" />
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging" Version="8.0.0" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\DevOpsWpf.Library\DevOpsWpf.Library.csproj" />
  </ItemGroup>

  <ItemGroup>
    <None Update="appsettings.Development.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="appsettings.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <ExcludeFromSingleFile>True</ExcludeFromSingleFile>
    </None>
    <None Update="appsettings.Production.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="appsettings.Staging.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="Documentations\AzureDevOpsSolution.docx">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="Documentations\ReadMe.txt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="Properties\launchSettings.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
  </ItemGroup>

</Project>

```

The azure-pipeline file:

* task VSBuild@1 is currently disabled
* instead task DotNetCoreCLI@2 is enabled

I can not figure out which arguments in DotNetCoreCLI@2 to use to accomplish my goal.

```
trigger:
- main

pool:
  vmImage: 'windows-latest'

# Check replacetokens-task and appsettings.json
# the following variables are used in appsettings.json
# applicationHomeDirectory
# applicationName
variables:
  solution: '**/*.sln'
  buildPlatform: 'Any CPU'
  runtimeId: 'win-x64'
  buildConfiguration: 'Release'
  applicationHomeDirectory: 'C:\DevOpsSolution'
  applicationName: 'Azur DevOps Solution (azure-pipelines.yml)'

# not sure, if this is needed
steps:
- task: DeleteFiles@1
  displayName: 'Delete files in artifactstagingdirectory'
  inputs:
    SourceFolder: '$(build.artifactstagingdirectory)'
    Contents: '**'

# working as intended
- task: NuGetToolInstaller@1
  displayName: 'Use NuGet 6.8.0'

# working as intended
- task: NuGetCommand@2
  displayName: 'Restore NuGet Packages'
  inputs:
    restoreSolution: '$(solution)'

# working as intended
- task: qetza.replacetokens.replacetokens-task.replacetokens@5
  displayName: 'Replace tokens in appsettings.json'
  inputs:
    targetFiles: '**/*.json'

# disabled task for testing
- task: VSBuild@1
  displayName: 'Build solution'
  inputs:
    solution: '$(solution)'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
  enabled: false

# testing this task instead of VSBuild; in artifact there are tons of files, instead of one exe-file and one json-file
- task: DotNetCoreCLI@2
  displayName: 'Publish'
  inputs:
    command: publish
    publishWebProjects: false
    arguments: '-c $(buildConfiguration) -r $(runtimeId) --self-contained true --output $(build.artifactstagingdirectory)'
    zipAfterPublish: false
  enabled: true

# working as intended
- task: VSTest@2
  displayName: 'Test solution'
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

# working to publish it to SolutionDrop; not sure if this is all that needs to be done here
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: SolutionDrop'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'
    ArtifactName: SolutionDrop
  condition: succeededOrFailed()

```

The published artifact is \>200 MB and has a ton of files. That can not be correct. Here you see just the top:

# Answer
Finally, I have received the result as it was intended to be.

This answer was the final clue. And I did find the reference in Microsoft documentation.

These properties are set in the project file:

```
    <PublishSingleFile>true</PublishSingleFile>
    <IncludeAllContentForSelfExtract>true</IncludeAllContentForSelfExtract>

```

There was a change in the publish process by Microsoft. If PublishSingleFile is set to true, Self-Contained is implied automatically.
When the above properties are set in the project file, they do not need to be listed as arguments in the command line.

The ***DotNetCoreCLI@2*** task looks like this:

```
steps:
- task: DotNetCoreCLI@2
  displayName: 'Publish *.csproj'
  inputs:
    command: publish
    publishWebProjects: false
    projects: '**/DevOpsWpf.csproj'
    arguments: '-c Release -r win-x64 --output $(build.artifactstagingdirectory)'
    zipAfterPublish: false

```

The ***project file*** looks like this:

```
<Project Sdk="Microsoft.NET.Sdk">

  <PropertyGroup>
    <OutputType>WinExe</OutputType>
    <TargetFramework>net8.0-windows10.0.19041.0</TargetFramework>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
    <UseWPF>true</UseWPF>
    <PublishSingleFile>true</PublishSingleFile>
    <IncludeAllContentForSelfExtract>true</IncludeAllContentForSelfExtract>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="CommunityToolkit.Mvvm" Version="8.2.2" />
    <PackageReference Include="Microsoft.Extensions.Configuration" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.DependencyInjection" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.Hosting" Version="8.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging" Version="8.0.0" />
  </ItemGroup>

  <ItemGroup>
    <ProjectReference Include="..\DevOpsWpf.Library\DevOpsWpf.Library.csproj" />
  </ItemGroup>

  <ItemGroup>
    <None Update="appsettings.Development.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="appsettings.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
      <ExcludeFromSingleFile>True</ExcludeFromSingleFile>
    </None>
    <None Update="appsettings.Production.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="Documentations\ReadMe.txt">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
    <None Update="Properties\launchSettings.json">
      <CopyToOutputDirectory>Always</CopyToOutputDirectory>
    </None>
  </ItemGroup>

</Project>

```
> 0 votes

---
Tags: azure-devops, azure-pipelines, azure-artifacts
---


# Question
Title: Azure DevOps Build Validation of other repo's pipeline: "while loading the YAML build pipeline. Object reference not set to an instance of an object."
This (nearly empty) Azure DevOps pipeline yaml fails when initiated because of a PR from another repo's Build Validation:

**azure-pipelines.yaml** of "App A":

```
pool:
  name: default

steps:
- bash: |
    echo "##[debug]Because"
  displayName: 'log stuff'

```

Normal build validation of a pipeline pointing at the same repo works ("Common Gated" as this is a policy for the "Common" repo), but running App A's pipeline as a Build Validation, it errors out immediately with little detail. App A's pipeline works fine when triggered manually. (App B is all exactly the same)

> An error occurred while loading the YAML build pipeline. Object reference not set to an instance of an object.
> 2125292

In case I'm at the wrong tree, what I'm trying to do is a gated build of both App A and App B when attempting to update Common's main branch. App A and App B each have git-submodules of Common embedded in them - an experiment to avoid publishing packages and versioning of Common without sacrificing much overall code/build stability. Ultimately, I think I'll need to switch the gated build's Common submodule branch to `$(System.PullRequest.SourceBranch)` before this check will make any sense, but I'm not that far along. Anyone succeeded in such a setup?

# Answer
That is possible if your yaml does not exist in the source or target gat branch. The build tries to run but can not fetch the yaml in the downloaded source code. If you create your yaml only on master ... do not forget to merge it through other branches.
> 0 votes

---
Tags: azure-devops, git, azure-pipelines, build-pipeline, tfs
---


# Question
Title: How do I get real client IP inside docker container for logging to the database
I have a following docker compose file:

```
version: "3.8"

services:
    postgres:
        image: postgres:11
        volumes:
            - myapp_postgres_volume:/var/lib/postgresql/data
            - type: tmpfs
              target: /dev/shm
              tmpfs:
                size: 536870912 # 512MB
        environment:
            POSTGRES_DB: elearning_academy
            POSTGRES_USER: myapp
            POSTGRES_PASSWORD: myapp123
        networks:
            - myapp_network

    pgadmin:
        image: dpage/pgadmin4:5.4
        volumes:
            - myapp_pgadmin_volume:/var/lib/pgadmin
        environment:
            PGADMIN_DEFAULT_EMAIL: myapp@cse.iitb.ac.in
            PGADMIN_DEFAULT_PASSWORD: myapp123
        ports:
            - 8080:80
        networks:
            - myapp_network

    redis:
        image: redis:6.2.4
        volumes:
            - myapp_redis_volume:/data
        networks:
            - myapp_network

    wsgi:
        image: wsgi:myapp3
        volumes:
            - /myapp/frontend/static/
            - ./wsgi/myapp:/myapp
            - /myapp/frontend/clientApp/node_modules
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro
        depends_on:
            - postgres
            - redis
        ports:
            - 9090
            - 3000:3000
            - 8000:8000
        environment:
            C_FORCE_ROOT: 'true'
            SERVICE_PORTS: 9090
        networks:
            - myapp_network
        deploy:
            replicas: 1
            update_config:
                parallelism: 1
                delay: 10s
            restart_policy:
                condition: on-failure
                max_attempts: 3
                window: 120s

    nodejs:
        image: nodejs:myapp3
        volumes:
            - ./nodejs/frontend:/frontend
            - /frontend/node_modules
        depends_on:
            - wsgi
        ports:
            - 9000:9000 # development
            - 9999:9999 # production
        environment:
            BACKEND_API_URL: http://0.0.0.0:3000
        networks:
            - myapp_network

    nginx:
        image: mydockeraccount/nginx-brotli:1.21.0
        volumes:
            - ./nginx:/etc/nginx/conf.d:ro
            - ./wsgi/myapp:/myapp:ro
            - myapp_nginx_volume:/var/log/nginx/
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        networks:
            - myapp_network

    haproxy:
        image: haproxy:2.3.9
        volumes:
            - ./haproxy:/usr/local/etc/haproxy/:ro
            - /var/run/docker.sock:/var/run/docker.sock
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        depends_on:
            - wsgi
            - nodejs
            - nginx
        ports:
            - 9763:80
        networks:
            - myapp_network
        deploy:
            placement:
                constraints: [node.role == manager]

volumes:
    myapp_postgres_volume:
    myapp_redis_volume:
    myapp_nginx_volume:
    myapp_pgadmin_volume:

networks:
    myapp_network:
        driver: overlay

```

As you can see I have a nodejs app and a django (wsgi) app. I have written django middleware to log incoming IP to the database. However, it logs the IP different from the actual IP. After reading online, I came to know that this might be due to how the docker network is configured (`overlay` as can be seen in last line of above docker compose file). I read online that I need to configure docker network in `host` mode. So I tried adding `network_mode: host` to each service and removing `networks` section from above file.

```
version: "3.8"

services:
    postgres:
        image: postgres:11
        volumes:
            - myapp_postgres_volume:/var/lib/postgresql/data
            - type: tmpfs
              target: /dev/shm
              tmpfs:
                size: 536870912 # 512MB
        environment:
            POSTGRES_DB: elearning_academy
            POSTGRES_USER: myapp
            POSTGRES_PASSWORD: myapp123
        network_mode: host

    pgadmin:
        image: dpage/pgadmin4:5.4
        volumes:
            - myapp_pgadmin_volume:/var/lib/pgadmin
        environment:
            PGADMIN_DEFAULT_EMAIL: myapp@cse.iitb.ac.in
            PGADMIN_DEFAULT_PASSWORD: myapp123
        ports:
            - 8080:80
        network_mode: host

    redis:
        image: redis:6.2.4
        volumes:
            - myapp_redis_volume:/data
        network_mode: host

    wsgi:
        image: wsgi:myapp3
        volumes:
            - /myapp/frontend/static/
            - ./wsgi/myapp:/myapp
            - /myapp/frontend/clientApp/node_modules
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro
        depends_on:
            - postgres
            - redis
        ports:
            - 9090
            - 3000:3000
            - 8000:8000
        environment:
            C_FORCE_ROOT: 'true'
            SERVICE_PORTS: 9090
        network_mode: host
        deploy:
            replicas: 1
            update_config:
                parallelism: 1
                delay: 10s
            restart_policy:
                condition: on-failure
                max_attempts: 3
                window: 120s

    nodejs:
        image: nodejs:myapp3
        volumes:
            - ./nodejs/frontend:/frontend
            - /frontend/node_modules
        depends_on:
            - wsgi
        ports:
            - 9000:9000 # development
            - 9999:9999 # production
        environment:
            BACKEND_API_URL: http://0.0.0.0:3000
        network_mode: host

    nginx:
        image: mydockeraccount/nginx-brotli:1.21.0
        volumes:
            - ./nginx:/etc/nginx/conf.d:ro
            - ./wsgi/myapp:/myapp:ro
            - myapp_nginx_volume:/var/log/nginx/
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        network_mode: host

    haproxy:
        image: haproxy:2.3.9
        volumes:
            - ./haproxy:/usr/local/etc/haproxy/:ro
            - /var/run/docker.sock:/var/run/docker.sock
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
        depends_on:
            - wsgi
            - nodejs
            - nginx
        ports:
            - 9763:80
        network_mode: host
        deploy:
            placement:
                constraints: [node.role == manager]

volumes:
    myapp_postgres_volume:
    myapp_redis_volume:
myapp_nginx_volume:
myapp_pgadmin_volume:

```

When I run `docker stack deploy -c docker-compose.yml my_stack`, it outputs `Ignoring unsupported options: network_mode.` I tried to check how the network is created:

```
$ docker network ls
NETWORK ID     NAME                       DRIVER    SCOPE
tp6olv2atq06   myapp_default              overlay   swarm

```

So it ended up being configured in the overlay mode only and not in host mode.

Further reading online revealed that host network mode is not available for swarm started with `docker stack deploy`. (Possible related open issue: github link).

For development, I simply connect vscode to nodejs and wsgi containers and run the apps in debug mode. So nginx is not involved at during development. However, in the deployment we do use nginx, which is also deployed as a docker container.

Now I have following questions:

**Q1.** Is configuring nginx to set `X-Real-IP` header with actual IP only solution?
**Q2.** Cant I do some docker config or changes in docker compose file to obtain real IP address inside docker container (django in my case), without doing any changes to nginx? Say somehow configuring those containers to be on `host` network?
**Q3.** If somehow I configure nginx to set `X-Real-IP` header, will it work even while development, given that nginx is not involved while development?
**Q4.** The comments to this answer seem to suggest that if the nginx itself is running as a docker container, then configuring it to add `X-Real-IP` header will not work. Is it so?

# Answer
> 1 votes
Although you've set the `network_mode` to `host`, I believe you need to set the `ports` -\> `mode` to `host` to get the real client IP.

For example:

```
ports:
  - mode: host

```

Hopefully this works for you.

# Answer
> 1 votes
If your trying to do what I think you are it's not done that way, atleast not currently.

**Answer.**

If you are trying to put a single service into "Host" mode you have to give it a network that has Host mode enabled. To do that you can do something like this:

```
networks:
  rajsNetwork:
    external: true

```

And then all you have to do is add the network to a service like so:

```
 wsgi:
        image: wsgi:myapp3
        volumes:
            - /myapp/frontend/static/
            - ./wsgi/myapp:/myapp
            - /myapp/frontend/clientApp/node_modules
            - /etc/timezone:/etc/timezone:ro
            - /etc/localtime:/etc/localtime:ro
            - /var/run/docker.sock:/var/run/docker.sock:ro
        depends_on:
            - postgres
            - redis
        ports:
            - 9090
            - 3000:3000
            - 8000:8000
        environment:
            C_FORCE_ROOT: 'true'
            SERVICE_PORTS: 9090
        networks:
          - rajsNetwork
        deploy:
            replicas: 1
            update_config:
                parallelism: 1
                delay: 10s
            restart_policy:
                condition: on-failure
                max_attempts: 3
                window: 120s

```

Then the container will be exposed to the host and use it's network.

If you haven't changed any of the default configurations for Docker I assume you have a network bridge and that is why you are getting weird IP addresses that start like 10.n.n.n or 172.n.n.n or 192.168.n.n.

These are IP address given by the Docker bridge and therefore not use the hosts IP address.

**But! Word of warning.**

You don't want to put all your containers in host mode.

You should explicitly define your containers networks in the docker-compose.yaml file so you know what they are explicitly connecting to.

Also, taking it off the Docker bridge network and giving it host mode will essentially mean your other devices will not be able to communicate with the container as they are on a different networks.

**One solution.**

What you could do is add 2 networks like so:

```
networks:
  rajsNetwork:
    name: rajsNetwork
    driver: bridge
  rajsNetworkHost:
    name: rajsNetworkHost
    external: true

```

This way you can connect your other containers to 'rajsNetwork' and then your Django Middleware container to 'rajsNetwork' & 'rajsNetworkHost' networks.

This way it can still communicate with the other containers and still be on "Host" mode.

**Answers to your questions in your post** Q1. No, there is multiple like using a reverse proxy like Traefik or Haproxy.

Q2. Yes, It's above.

Q3. Well, if your serving via Node for dev then no it will not.

Q4. I'm not too sure about this one, probably not as this is an essential feature to block troublesome IP addresses with something like Fail2Ban.

What I like to do is set NGinx config to use the Systemd/Journald logger and not a log file or any type of Syslog app. Instead what I do is add the logging driver to the container and then all the logs from your NGinx container will be inside the hosts Journald daemon, which to view is "journalctl -xe". You can then setup something like Fail2Ban to monitor the hosts journal and get the IP address from there.

A big benefit of doing it this way is you only need to install Fail2Ban on 1 container/machine/host/vm and as long as the containers who are exposed to the internet are logging IP's and have the journal logging driver enabled, means you only need Fail2Ban installed on the host machine to protect all the containers on your system.

Rather than installing in each container. TO add logging to a container, add this:

```
    logging:
      driver: journald
      options:
        tag: "{{.Name}}"

```

Make sure its the same indentation amount as the 'image', 'network' or 'volumes' keywords.

The `tag: "{{.Name}}"` like means that each entry in the Docker hosts Journal will have the .Name of the container with the log entry.

---
Tags: docker, docker-compose, nginx, docker-swarm, network-monitoring
---


# Question
Title: How to update GitHub issue commit links after migrating code and issues from another repository?
We are an organization with multiple repositories in GitHub which we are merging into a monorepo. We have no issues with the mechanics of how to do this:

1. Use something similar to the method outlined at https://blog.jdriven.com/2021/04/how-to-merge-multiple-git-repositories/ to merge repositories into the target monorepo
2. Use the GitHub CLI `gh issue transfer` tool to transfer the issues from the source repos to the target monorepo

The problem is that we make heavy use of issue IDs in our Git commit messages in order to link issues in GitHub to specific commits, and those links still refer to the commit in the original repository after we transfer them. An example will illustrate the problem:

* Let's say I have a bug which is documented in GitHub issue #123 in repository `RepoA`
* Let's say I make commit `abcd123` in repository `RepoA` which fixes the bug, and in the commit message I put something like "Closes #123"
* In the GitHub issue #123 in `RepoA`, I will now see a message in the timeline along the lines of "UserABC closed this as completed in abcd123 yesterday", and the "abcd123" will be a link to the commit in question. My action will also close the issue because I used the keyword "Closes".
* At some point in the future, I merge `RepoA` into my `monorepo` repository (also hosted on GitHub under the same organization) and transfer its issues into the `monorepo` using the techniques described in (1) and (2) above. The method I use to merge preserves commit IDs, so now I have commit ID `abcd123` in the `monorepo` as well, and will have the GitHub issue (though with a different ID) in `monorepo` too.
* The message in the timeline in the transferred issue in the `monorepo` which previously said "UserABC closed this as completed in abcd123 yesterday" has changed, however: it now explicitly points to the original repository and says "UserABC closed this as completed in RepoA@abcd123 yesterday", and the link is to the original commit in `RepoA`, not the identical commit in the new `monorepo`.

I understand why this happens, and it's arguably even correct behavior, but it's undesirable for us because one of our goals is to ultimately delete the original repositories completely once we know the `monorepo` migration is correct and complete. When you do this (we tried it on a couple of test repos) the message changes to "UserABC closed this as completed yesterday", with no commit ID, so we would lose this useful linkage between issues and commits, and even if we don't delete the source repositories the links are less useful because they take you outside of the `monorepo`.

Is there a way to update/edit these GitHub issues so they point to the migrated/merged commit in the `monorepo`, instead of to the original commit in the source repository? Or some other way to solve this conundrum?

# Answer
You might have to create a script to automate changing the ID's is the only answer I can think of.

Unfortunately Git isn't designed to be used like this. It's version tracking software so it tracks all changes including the previous repos commits & pointing to the previous repo.

But at least a relatively small script could achieve this though.
> 0 votes

---
Tags: git, github, migration
---


# Question
Title: Postgresql container automatically dropping database after some time in Docker Compose setup
I have a Docker Compose setup for a PostgreSQL container like below:

```
version: '3'
services:

  #PostgreSQL Service
  um-db:
    image: postgres:15
    container_name: um-db
    tty: true
    ports:
      - "5432:5432"
    environment:
      POSTGRES_DB: user_management
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      PGDATA: /var/lib/postgresql/data/pgdata
      SERVICE_TAGS: dev
      SERVICE_NAME: postgresql
    volumes:
      - um-dbdata16:/var/lib/postgresql/data
    networks:
      - um-app-network

#Volumes
volumes:
  um-dbdata16:
    driver: local

```

After running `docker-compose up -d --build`, the database gets built successfully. However, after some time, it automatically gets dropped. Here's a snippet of the log:

```
docker logs um-db
2024-02-11 07:20:09.510 GMT [161] ERROR:  cannot drop the currently open database
2024-02-11 07:20:09.510 GMT [161] STATEMENT:  DROP DATABASE postgres;
2024-02-11 07:20:09.639 GMT [22] LOG:  checkpoint starting: immediate force wait
2024-02-11 07:20:10.449 GMT [165] FATAL:  password authentication failed for user "postgres"
2024-02-11 07:20:10.449 GMT [165] DETAIL:  Connection matched pg_hba.conf line 100: "host all all all scram-sha-256"

```

The same setup is working fine on my Ubuntu server locally, but when deployed to a VPS, this issue occurs. I have thoroughly searched the internet for a solution but couldn't find any useful resources. Please point me in the right direction so that I can solve this issue.

# Answer
The problem is `2024-02-11 07:20:10.449 GMT [165] FATAL: password authentication failed for user "postgres"`.

* Are you sure its the correct password?
* Your Volume is different from your Ubuntu machine to the VPS so check they are identical to rule out that issue.
* Check your `pg_hba.conf` on the line mentioned. You might have editited a line above or below and therefore the wrong permissions are set.
* You've not declared a `network:` section, is it on the same network as the thing your trying to connect to?

If these don't fix the issue, add your response & any other logs/errors and we will see fi we can get it sorted.
> 0 votes

---
Tags: docker, docker-compose, postgresql
---


# Question
Title: containerd: Unable to overwrite sandbox image
**System**

Ubuntu 22.04.01

kubeadm v1.25.4

containerd 1.6.10

**Description**

I'm trying to setup k8s 1.25.4 with a private docker repository and the containerd runtime.

I followed the docs to overwrite the sandbox image in `/etc/containerd/config.toml`:

```
...
[plugins."io.containerd.grpc.v1.cri"]
  sandbox_image = "my-private-registry:5000/registry.k8s.io/pause:3.8"
...

```

Afterwards, the containerd service, and later the machine, was restarted.

However, when trying to initialize the k8s cluster with kubeadm (`kubeadm init --config cluster-config.yml`), the initalization failed with `journalctl -exu containerd` showing the following output:

```
Dec 14 11:04:00 myhost containerd[866]: time="2022-12-14T11:04:00.911273848Z" level=info msg="trying next host" error="failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.6\": dial tcp 34.107.244.51:443: i/o timeout" host=registry.k8s.io
Dec 14 11:04:00 myhost containerd[866]: time="2022-12-14T11:04:00.927516068Z" level=error msg="RunPodSandbox for &PodSandboxMetadata{Name:myhost,Uid:c4ebaf5ac4e21e7bc99cda96857ec39e,Namespace:kube-system,Attempt:0,} failed, error" error="failed to get sandbox image \"registry.k8s.io/pause:3.6\": failed to pull image \"registry.k8s.io/pause:3.6\": failed to pull and unpack image \"registry.k8s.io/pause:3.6\": failed to resolve reference \"registry.k8s.io/pause:3.6\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.6\": dial tcp 34.107.244.51:443: i/o timeout"

```

This indicates to me that the sandbox image overwrite was unsuccessful. `cluster-config.yml` is configured to use the private registry and kubeadm can succesfully pull the k8s images (including the sandbox image):

```
...
imageRepository: my-private-registry:5000
...

```

What am I missing here?

# Answer
> 0 votes
I am facing the same issue, for me, I add the `version = 2` config in `/etc/containerd/config.toml` and it works:

```
version = 2
[plugins."io.containerd.grpc.v1.cri".registry]
   config_path = "/etc/containerd/certs.d"

[plugins."io.containerd.grpc.v1.cri"]
  #sandbox_image = "k8s.m.daocloud.io/pause:3.6"
  sandbox_image = "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6"

```

check the new sandbox image address:

```
[root@k8sslave01 ~]# crictl info|grep pause
    "sandboxImage": "registry.cn-hangzhou.aliyuncs.com/google_containers/pause:3.6",

```

this is the official config doc: https://github.com/containerd/cri/blob/master/docs/config.md

---
Tags: kubernetes, kubeadm, containerd
---