# Question
Title: How to add an AWS Auto Scaling Group health check on a non-web instance
# Background

Adding health checks to our web ec2 instances on AWS is easy, we simply have to create an endpoint that always returns a 200 http response when it's healthy.

However, we also have other non-web instances in our Auto Scaling Groups (ASG) where this model doesn't apply:

# Problem

The problem we have is that our non-web instances are failing due to various reasons (ie disk running out of space, sometimes the instances are not reachable via ssh even for us to investigate what went wrong). When this happens, the ASG is unaware of that and so doesn't automatically attempt to recycle the instance.

# Question

How can we add a health check on a non-web instance (without actually turning it into one, ie by installing NGINX server etc)

# Answer
Related StackOverflow question: https://stackoverflow.com/questions/43975456/how-to-implement-custom-health-checks-for-ec2-instances-without-using-an-elb

Basic suggestion: use a cron job to push health info to AWS using the CLI. I've not encountered a better solution in recent/limited research unfortunately.
> 0 votes

# Answer
An Amazon Web Services (AWS) Auto Scaling Group is a service that automatically adjusts the number of instances in a group based on user-defined policies. To add a health check on a non-web instance in an Auto Scaling Group, you can follow these steps:

```
Determine the type of health check to use: There are two types of health checks you can use with an Auto Scaling Group: EC2 and ELB. The EC2 health check checks the health of each instance in the group by running a script or command on the instance. The ELB health check checks the health of each instance behind an Elastic Load Balancer (ELB) by sending HTTP or HTTPS requests to the instance.

Create a health check script or command: If you choose to use the EC2 health check, you will need to create a script or command that returns a response of 0 if the instance is healthy, and a response of 1 if the instance is not healthy. The script or command can be anything that checks the health of your application or service running on the instance.

Configure the Auto Scaling Group: In the AWS Management Console, navigate to the Auto Scaling Groups page and select the group you want to add the health check to. Click the "Edit" button, and then select the "Health check type" option. Choose either EC2 or ELB depending on the type of health check you want to use.

Configure the health check settings: If you choose to use the EC2 health check, enter the script or command you created in step 2 into the "Health check grace period" field. If you choose to use the ELB health check, select the ELB from the dropdown list and configure the health check settings for the ELB.

Save the changes: Click the "Save" button to apply the health check settings to the Auto Scaling Group.

```

By following these steps, you can add a health check to a non-web instance in an AWS Auto Scaling Group. This will help ensure that the instance is always running and healthy, and will trigger the Auto Scaling Group to replace any unhealthy instances as needed.
> 0 votes

---
Tags: amazon-web-services, autoscaling, healthcheck
---


# Question
Title: Is it possible to deploy a self hosted agent from Azure DevOps?
I need to deploy a self hosted agent from Azure DevOps to on premise Hyper-V hosts. I have VMM managing the hosts. I need to complete the following:

* Deploy the VM from a base template
* Install Pre Requisites for the agent and Docker
* Docker will be available for the pipeline tasks
* Install the agent and connect to DevOps

The VM deployment could be something I kick off from VMM. Can I please have some guidance on how to approach this?

Any advice on a better approach also appreciated.

# Answer
you may download the build agent here: https://github.com/microsoft/azure-pipelines-agent/releases

Then use: Unattended config
> 0 votes

---
Tags: azure-devops, builds
---


# Question
Title: Is there anyway to upgrade AzureDevops 2019 on premise to Azure Devops 2022 on premise
My question is simple.

I was wondering about if is there any way to upgrade?

When I read this article, I thought I can upgrade 2019 to 2022.

https://learn.microsoft.com/en-us/azure/devops/migrate/migration-overview?view=azure-devops-2022

But for now, I haven't find any way to do it yet.

# Answer
Yes. you can updates... the steps will be same as here: Upgrade Azure DevOps Server to 2020
> 0 votes

---
Tags: azure-devops
---


# Question
Title: Using Azure Icons and Boxes on Azure DevOps with Mermaid
I have the following mermaid diagram, which I have validated on mermaid.live and intend to use it on the Wiki section of Azure DevOps:

```
sequenceDiagram
    box Container;
    participant A as Container A;
    participant B as Container B;
    end;

```

I know that the wiki for Azure DevOps has support for mermaid diagrams, but for some reason it doesn't render if I include the `box` block above. Is this a version problem? If so, is there a way to update it mermaid to the latest version?

I also would like to know if it is possible to use Azure Architecture icons within these diagrams.

# Answer
> 0 votes
I found that a lot of syntax for mermaid is not supported in Azure DevOps. See here where is stated:

> Not all syntax in the content linked below for diagram types works in Azure DevOps. For example, we don't support most HTML tags, Font Awesome, flowchart syntax (graph used instead), or LongArrow ----\>.

I've tested your syntax as well and I can confirm it wouldn't load here as well, only without the box.

---
Tags: azure-devops
---


# Question
Title: Can terraform re-create a vm that was manually deleted
I have a terraform managed infrastructure with a set of 4 virtually identical VMs. One of them has been deleted in VSphere by mistake. I had expected `terraform refresh` to re-create it, but I misunderstood the purpose of the command. Now the VM doesn't show up even in the `tfstate` file. Is there a way to make terraform recreate it without having to re-deploy the entire infrastructure?

# Answer
> 2 votes
It sounds like you have bigger problems.

From the documentation:

`terraform refresh`:

> The terraform refresh command reads the current settings from all managed remote objects and updates the Terraform state to match.

It goes on to say:

> **You shouldn't typically need to use this command**, because Terraform automatically performs the same refreshing actions as a part of creating a plan in both the terraform plan and terraform apply commands. This command is here primarily for backward compatibility, but **we don't recommend using it because it provides no opportunity to review the effects** of the operation before updating the state.

Meaning that the command is working as expected -- but not as you want it to.

If you have the "proper" (*i.e.* **desired**) state described in a Terraform statement (*e.g.* `main.tf`), then you should be able to converge back to the desired state by performing another `plan`/`apply` loop:

`terraform plan` --\> should show the drift in state, *i.e.* there was a resource deleted and needs to be created. `terraform apply` --\> the change will be applied and your state will converge to the declaration as specified in the Terraform file.

# Answer
> 0 votes
Posting the exact bit that worked for me. Terraform (in my case at least) compared what was demanded by the tf files to what was in the state and noted that the VM was missing. It properly re-created it, checking that the planned actions were correct. YMMV.

```
terraform apply -state=project.tfstate -var-file=project.tfvars.json

```

---
Tags: terraform
---


# Question
Title: In Gitlab is there a way to automatically change a tag on a ticket when an MR is created and/or when a deployment is completed
In Gitlab we have a flow where our tickets go from "In Progress" to "Code Review" when an MR is created and from "Code Review" to "Ready for Testing" when a deployment to our QA environment is performed. However, the developers have to update the tickets by hand. Each of those statuses (In Progress, Code Review, Ready for Testing) are just tags on the ticket

Is there a way to automate this flow?

# Answer
Sure, utilize GitLab API within your CI/CD pipeline to update the ticket status automatically:

1. Set up a GitLab CI/CD pipeline to run automated tests whenever a new merge request is created or updated. If the tests pass, the pipeline can automatically update the associated ticket status to "Code Review".
2. Configure GitLab webhooks to trigger another pipeline when a deployment is made to the QA environment. This pipeline will run automated tests again, and if they pass, the associated ticket can be updated to "Ready for Testing".

Top it off with some GitLab permissions to restrict access to changing ticket status to authorised users only.
> 1 votes

---
Tags: gitlab, cicd
---


title: Performance of newly created Azure VM

text:
# Question
Title: Performance of newly created Azure VM
Our team copied a VM from one Azure subscription onto a new Azure subscription. This VM acts as a Clickhouse client. Both the original and the copied VMs share the same characteristics: **Standard B4ms (4 vcpus, 16 GiB memory)**.

The newly created VM behaves properly, however I noted that it does not perform as well as the original VM. For instance, when trying to run a simple Clickhouse query, this is what both VMs return:

```
Original VM: Processed 2.45 million rows, 9.79 MB (125.86 million rows/s., 503.44 MB/s.)
New VM:  Processed 2.45 million rows, 9.79 MB (62.26 million rows/s., 249.03 MB/s.)

```

Do you have any idea what may cause this difference in performance?

# Answer
From Microsoft documentation, quoting:

> The B-series provides you with the ability to purchase a VM size with baseline performance that can build up credits when it is using less than its baseline. When the VM has accumulated credits, the VM can burst above the baseline using up to 100% of the vCPU when your application requires higher CPU performance

Standard\_B4ms has base performance of 90% and max of 400%. Also there are variable of processors that may be used.

Truth could be in credits. Credits should be checked in new and old VM in order to verify if it can burst or not.
> 1 votes

---
Tags: azure, performance, vm
---


# Question
Title: Configure Azure Kubernetes user context for on-premise resource access
My question is hopefully simple.In the period where you have both on-premise and Azure based resources can jobs running in an Azure Kubernetes cluster be configured to use an AD user context such that things like network folders can be accessed or SQL Server connections made using trusted connections? If not, what is the safest way to proceed as I'm not a fan of UID/PWD connection strings as the details tend to get shared where they shouldn't.

# Answer
> 0 votes
You cannot take the Azure AD-based managed identities and grant them access to on-premises resources. One option would be to move your AKS cluster on-premises via Azure Arc service.

That being more infrastructure required may not be ideal. The best option is to move those resources you need on-premises into the cloud because it lessens the latency that will be incurred trying to retrieve unknown data size between Azure and your on-premises network.

* need network files, have application re-coded to talk to Storage Account Servies (file share, blob, etc.)
* need database, migrate to Azure VM or Azure SQL DB/MI that support managed identity authentication

---
Tags: kubernetes, azure, authentication
---


# Question
Title: Prometheus scrape config for kubelet node role not auth'ing
I have a kubernetes (not minikube) cluster (1.25) installed on a dedicated machine (1-node cluster). I installed Prometheus. I am using a ServiceAccount, Role, ClusterRoleBinding like this:

```
apiVersion: v1
kind: Namespace
metadata:
    name: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups:
  - extensions
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs:
  - "/metrics"
  verbs: ["get"]
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
  namespace: monitoring
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: monitoring

```

with my Deployment resource containing `serviceAccountName: prometheus`.

I am getting some metrics (from Prometheus and some Istio scrape configs), but am not able to get the k8s node metrics (like CPU, etc). I tried this:

```
      - job_name: 'kubelet'
        kubernetes_sd_configs:
        - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt

```

I get an error in my Prometheus targets dashboard (http://192.168.0.121:30000/targets?search= of:

`Get "https://192.168.0.121:10250/metrics": x509: cannot validate certificate for 192.168.0.121 because it doesn't contain any IP SANs`

Adding `insecure_skip_verify: true` turns this error into:

`server returned HTTP status 401 Unauthorized`

My KubeletConfiguration:

```
% kubectl describe configmap kubelet-config -n kube-system
Name:         kubelet-config
Namespace:    kube-system
Labels:       <none>
Annotations:  kubeadm.kubernetes.io/component-config.hash: ...

Data
====
kubelet:
----
apiVersion: kubelet.config.k8s.io/v1beta1
authentication:
  anonymous:
    enabled: false
  webhook:
    cacheTTL: 0s
    enabled: true
  x509:
    clientCAFile: /etc/kubernetes/pki/ca.crt
authorization:
  mode: Webhook
  webhook:
    cacheAuthorizedTTL: 0s
    cacheUnauthorizedTTL: 0s
cgroupDriver: systemd
clusterDNS:
- 10.96.0.10
clusterDomain: cluster.local
cpuManagerReconcilePeriod: 0s
evictionPressureTransitionPeriod: 0s
fileCheckFrequency: 0s
healthzBindAddress: 127.0.0.1
healthzPort: 10248
httpCheckFrequency: 0s
imageMinimumGCAge: 0s
kind: KubeletConfiguration
logging:
  flushFrequency: 0
  options:
    json:
      infoBufferSize: "0"
  verbosity: 0
memorySwap: {}
nodeStatusReportFrequency: 0s
nodeStatusUpdateFrequency: 0s
resolvConf: /run/systemd/resolve/resolv.conf
rotateCertificates: true
runtimeRequestTimeout: 0s
shutdownGracePeriod: 0s
shutdownGracePeriodCriticalPods: 0s
staticPodPath: /etc/kubernetes/manifests
streamingConnectionIdleTimeout: 0s
syncFrequency: 0s
volumeStatsAggPeriod: 0s

```

My kubeadm-config is:

```
% kubectl describe configmap kubeadm-config -n kube-system
Name:         kubeadm-config
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Data
====
ClusterConfiguration:
----
apiServer:
  certSANs:
  - <my tailscale ip address>
  extraArgs:
    authorization-mode: Node,RBAC
  timeoutForControlPlane: 4m0s
apiVersion: kubeadm.k8s.io/v1beta3
certificatesDir: /etc/kubernetes/pki
clusterName: kubernetes
controlPlaneEndpoint: 192.168.0.121:6443
controllerManager: {}
dns: {}
etcd:
  local:
    dataDir: /var/lib/etcd
imageRepository: registry.k8s.io
kind: ClusterConfiguration
kubernetesVersion: v1.25.3
networking:
  dnsDomain: cluster.local
  podSubnet: 10.244.0.0/16
  serviceSubnet: 10.96.0.0/12
scheduler: {}

```

which was created by

`kubeadm init --control-plane-endpoint=192.168.0.121 --pod-network-cidr=10.244.0.0/16 --apiserver-cert-extra-sans=<my tailnet ip address>`

Any insight would be greatly appreciated!

# Answer
> 1 votes
You need to send bearer token as well, like this:

```
- job_name: 'kubelet'
        kubernetes_sd_configs:
        - role: node
        scheme: https
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
        bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token

```

---
Tags: kubernetes, prometheus, kubelet
---


# Question
Title: Terraform: Is there a concise syntax for specifying multiple tags for a resource?
To attach multiple tags to a resource, I currently use multiple `tag` blocks:

```
resource "aws_autoscaling_group" "instance" {
  ...

  tag {
    key                 = "Name"
    value               = "${var.cluster_prefix}"
    propagate_at_launch = true
  }

  tag {
    key                 = "Owner"
    value               = "${var.tag_Owner}"
    propagate_at_launch = true
  }

  tag {
    key                 = "Project"
    value               = "${var.tag_Project}"
    propagate_at_launch = true
  }
}

```

It works, but I wonder if there is a more concise syntax. It is also a bit error prone as you have to specify `propagate_at_launch` for each tag.

**Update:** The question was written before Terrafrom v0.12 came out. Since then the support has improved with Dynamic Nested Blocks (see my answer below).

# Answer
> 9 votes
For an `auto-scaling-group` this is the most concise syntax available.

For most other resources you use the `tags` syntax which looks like:

```
tags {
  Key1 = "value1"
  Key2 = "value2"
}

```

# Answer
> 6 votes
Terraform v0.12 added support for Dynamic Nested Blocks. The following example is derived from their blog post about the new features (see section Dynamic Nested Blocks):

```
locals {
  standard_tags = {
    Name    = var.cluster_prefix
    Owner   = var.tag_Owner
    Project = var.tag_Project
  }
}

resource "aws_autoscaling_group" "example" {
  # ...

  dynamic "tag" {
    for_each = local.standard_tags

    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

```

# Answer
> 2 votes
Another option is to use the `terraform-null-label` module. It supports passing a variable called `tags` as a standard Terraform map. The module then emits an output called `tags_as_list_of_maps` which contains the tags in the format you want. But an even better reason to use this module is to generate a consistent set of resource names that follow a fixed convention.

For example, you can do this:

```
module "example" {
  source     = "git::https://github.com/cloudposse/terraform-null-label.git?ref=master"
  namespace  = "eg"
  stage      = "prod"
  name       = "bastion"
  tags       = {
                 "BusinessUnit" = "XYZ"
                 "Snapshot" = "true"
               }
}

```

So from your example, we can then write something like this:

```
resource "aws_autoscaling_group" "instance" {
  ...
  name = "${module.example.id}"
  ...
  tags = "${module.example.tags_as_list_of_maps}"
  ...
}

```

**NOTE:** I added the `name` field to show how to generate co

# Answer
> 1 votes
The `aws_autoscaling_group` does support a list of tags now (https://www.terraform.io/docs/providers/aws/r/autoscaling\_group.html#tags). But this looks a bit different to the syntax of other terraform resources:

```
tags = [
  {
    key                 = "explicit1"
    value               = "value1"
    propagate_at_launch = true
  },
  {
    key                 = "explicit2"
    value               = "value2"
    propagate_at_launch = true
  },
]

```

This also allows to dynamically build tags via interpolation.

# Answer
> 0 votes
Here is the best way to do now

To be able to

1. Attach the same common tags to all resources
2. While still being able to add custom tags to individual resources.

You can add the default tag section in terraform provider section

```
# Configure the AWS Provider
provider "aws" {
  region  = var.aws_region
  profile = var.aws_profile

  default_tags {
    tags = {
      environment = var.tag_environment
      created_by  = var.tag_created_by
    }
  }

}

```

You can check the read the docs here

---
Tags: terraform
---


title: Best dockerized VPN clients?

text:
# Question
Title: Best dockerized VPN clients?
I'm trying to set up a VPN client in a Docker service with Docker Compose, but I'm running into some issues.

ExpressVPN: I tried this ExpressVPN container, but it just keeps restarting constantly. Here's my code:

```
expressvpn:
  container_name: expressvpn
  image: polkaned/expressvpn
  environment:
    - ACTIVATION_CODE=<code>
  cap_add:
    - NET_ADMIN
  devices:
    - /dev/net/tun
  stdin_open: true
  tty: true
  command: /bin/bash
  privileged: true
  ports:
    - 6881:6881 # QBittorrent
  restart: unless-stopped

```

OpenVPN: I tried installing this DockOVPN client, but the instructions say to run this command to set up/configure:

```
docker run -it --rm --cap-add=NET_ADMIN \
    -p 1194:1194/udp -p 80:8080/tcp \
    -e HOST_ADDR=$(curl -s https://api.ipify.org) \
    --name dockovpn alekslitvinenk/openvpn

```

When I run that I get this error:

> WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested

> standard\_init\_linux.go:219: exec user process caused: exec format error

I also tried this linuxserver.io OpenVPN client, but this one is also restarting constantly.

Long story short, I'm just trying to find any dockerized VPN client that works. Can anyone either tell me what I'm doing wrong or recommend an alternate solution?

# Answer
You are trying to run an x86 image on a non-x86 processor. That will not work.

You need to find or build an image for the linux/arm64/v8 platform, and use that. You can read more about multi platform images at https://docs.docker.com/build/building/multi-platform/
> 1 votes

---
Tags: docker, docker-compose
---


# Question
Title: Kubernetes repeating job with start and end date?
I am looking to see if there is a way of running a task on a repeat interval with an explicit start and end date.

In the ecosystem I have found that Cron Job task type lines up very well with a repeat task. However I've been having trouble finding a Cron Job scheduler that has a start and end date.

Most only allow a cron expression to provided to tune when a task is run. So far the only way I've found to enforce a start and end date is to make the logic of the task itself either exit early if before the start date or delete the job after the end date.

I was wondering if there were any schedulers that implemented start and end date behavior for recurring tasks.

(So far I've found the Kubernetes CronJob and Argo CronJob)

# Answer
> 0 votes
```
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: my-cronjob
spec:
  schedule: "*/15 * * * *"
  startTime: 2023-03-07T10:00:00Z
  jobTemplate:
    spec:
      activeDeadlineSeconds: 1800 #end time specification
      template:
        spec:
          containers:
          - name: my-container
            image: my-image
            command: ["/bin/sh"]
            args: ["-c", "echo Hello, World!"]

```

---
Tags: kubernetes, argocd
---


# Question
Title: How to define rollback or rescue in ansible folder structure?
I would like to define rescue or rollback executions within my Ansible structure. I am using this recommended structure:

```
roles
  rolename
    files
    tasks
    templates
    handlers
    ...

```

I know that I can set blocks and within them a rescue section. But this seems a little clumsy to me and I would like to define all the rollback/rescue options within its own folder like

```
roles
  rolename
    ...
    tasks
    rollback/rescue
    handlers
    ...

```

This would make much more sense to me. I am speaking here about rollback, because I would like to be able to rollback a setup that I have created once with this role. I understand that the concept of rescue is a little bit different because it aims to fire when there is an error approached in executing a role. But I am hoping that I could use the concepts together while defining an anti-step to each step that I have defined in the role. And these anti-steps should have their own place and not included in the tasks files.

# Answer
For example, enable *backup* and *register* the results. As a last task create a record in the *journal* (see the declaration of the variables in the details under the line) and keep it in the cache

```
shell> cat roles/test_rollback/tasks/config.yml
- name: config_01
  template:
    src: test.ini.j2
    dest: /tmp/test.ini
    backup: true
  register: config_01

- name: Write journal
  set_fact:
    journal: "{{ journal|d({})|combine({timestamp: journal_config_dict}) }}"
    cacheable: true
  vars:
    timestamp: "{{ '%Y-%m-%d_%H_%M_%S'|strftime }}"

```

If you want to rollback the changes set the *rollback\_journal\_record*

```
shell> cat roles/test_rollback/rollback/config.yml
- copy:
    remote_src: true
    src: "{{ item.value.backup_file }}"
    dest: "{{ item.value.dest }}"
  loop: "{{ journal[rollback_journal_record]|dict2items|selectattr('value.changed') }}"
  loop_control:
    label: "{{ item.key }}"

```

Let's test it. Given the file on the remote host

```
shell> ssh admin@test_11 cat /tmp/test.ini
test = X

```

Run the playbook

```
shell> ansible-playbook pb.yml
  ...
TASK [test_rollback : config_01] *************************************************************
changed: [test_11]

TASK [test_rollback : Write journal] *********************************************************
ok: [test_11]
  ...

```

The file changed

```
shell> ssh admin@test_11 cat /tmp/test.ini
test = A

```

and the record was added to the journal

<sup>
```
shell> cat /tmp/ansible_cache.json/ansible_facts_test_11
{
    "journal": {
        "2023-03-08_02_33_48": {
            "config_01": {
                "backup_file": "/tmp/test.ini.50045.2023-03-08@01:33:48~",
                "changed": true,
                "checksum": "6676ed8ea4caa804d3c5f0247c03ece8b7a80b0b",
                "dest": "/tmp/test.ini",
                "diff": [],
                "failed": false,
                "gid": 0,
                "group": "wheel",
                "md5sum": "0740a0962b43fa7bedf3dcd97e96697f",
                "mode": "0644",
                "owner": "root",
                "size": 9,
                "src": "/home/admin/.ansible/tmp/ansible-tmp-1678239225.046748-1753646-178984183659153/source",
                "state": "file",
                "uid": 0
            }
        }
    }
}

```
</sup>

See the records in the journal

```
shell>ansible-playbook pb.yml -t info -e debug_journal_records=true
  ...
TASK [test_rollback : debug] *****************************************************************
ok: [test_11] =>
  msg: |-
    Journal records:
    - 2023-03-08_02_33_48
  ...

```

See the changed file(s) in the record

```
shell> ansible-playbook pb.yml -t info -e debug_journal_record=2023-03-08_02_33_48
  ...
TASK [test_rollback : debug] *****************************************************************
ok: [test_11] =>
  msg: |-
    Journal changes of 2023-03-08_02_33_48:
    config_01:
        backup_file: /tmp/test.ini.50045.2023-03-08@01:33:48~
        changed: true
        checksum: 6676ed8ea4caa804d3c5f0247c03ece8b7a80b0b
        dest: /tmp/test.ini
        diff: []
        failed: false
        gid: 0
        group: wheel
        md5sum: 0740a0962b43fa7bedf3dcd97e96697f
        mode: '0644'
        owner: root
        size: 9
        src: /home/admin/.ansible/tmp/ansible-tmp-1678239225.046748-1753646-178984183659153/source
        state: file
        uid: 0
  ...

```

Rollback the changed file(s)

```
shell> ansible-playbook pb.yml -t rollback -e rollback_journal_record=2023-03-08_02_33_48
  ...
TASK [test_rollback : copy] ******************************************************************
changed: [test_11] => (item=config_01)
  ...

```

The file was restored from the backup

```
shell> ssh admin@test_11 cat /tmp/test.ini
test = X

```

---

<sup>

Example of a complete project for testing

```
shell> tree .
.
├── ansible.cfg
├── hosts
├── pb.yml
└── roles
    └── test_rollback
        ├── defaults
        │   └── main.yml
        ├── rollback
        │   ├── config.yml
        │   └── main.yml
        ├── tasks
        │   ├── config.yml
        │   ├── info.yml
        │   ├── main.yml
        │   └── rollback.yml
        └── templates
            └── test.ini.j2

6 directories, 11 files

```

Enable caching

```
shell> cat ansible.cfg
[defaults]
gathering = explicit
collections_path = $HOME/.local/lib/python3.9/site-packages/
inventory = $PWD/hosts
roles_path = $PWD/roles
ansible_managed = {file}
deprecation_warnings=true
remote_tmp = ~/.ansible/tmp
retry_files_enabled = false
stdout_callback = yaml
# fact_caching
fact_caching = jsonfile
fact_caching_connection = /tmp/ansible_cache.json
fact_caching_prefix = ansible_facts_
fact_caching_timeout = 86400

```

```
shell> cat hosts
test_11
test_12
test_13

```

```
shell> cat pb.yml
- hosts: test_11

  roles:
    - test_rollback

```

The role

```
shell> cat roles/test_rollback/tasks/main.yml
- import_tasks: info.yml
  tags: info

- import_tasks: rollback.yml
  when: rollback_journal_record is defined
  tags: rollback

- import_tasks: config.yml
  tags: config

```

```
shell> cat roles/test_rollback/tasks/info.yml
- debug:
    msg: |
      Journal records:
      {{ journal.keys()|list|to_nice_yaml }}
  when: debug_journal_records|d(false)|bool

- debug:
    msg: |
      Journal changes of {{ debug_journal_record }}:
      {{ journal[debug_journal_record]|dict2items|
         selectattr('value.changed')|items2dict|to_nice_yaml }}
  when: debug_journal_record is defined

```

```
shell> cat roles/test_rollback/tasks/rollback.yml
- import_tasks: rollback/main.yml

- meta: end_play

```

```
shell> cat roles/test_rollback/tasks/config.yml
- name: config_01
  template:
    src: test.ini.j2
    dest: /tmp/test.ini
    backup: true
  register: config_01

- name: Write journal
  set_fact:
    journal: "{{ journal|d({})|combine({timestamp: journal_config_dict}) }}"
    cacheable: true
  vars:
    timestamp: "{{ '%Y-%m-%d_%H_%M_%S'|strftime }}"

```

```
shell> cat roles/test_rollback/rollback/main.yml
- debug:
    msg: |
      Rollback {{ rollback_journal_record }}:
      {{ journal[rollback_journal_record]|dict2items|
         selectattr('value.changed')|
         map(attribute='value.dest')|to_nice_yaml }}
  when: debug|d(false)|bool

- import_tasks: rollback/config.yml

```

```
shell> cat roles/test_rollback/rollback/config.yml
- copy:
    remote_src: true
    src: "{{ item.value.backup_file }}"
    dest: "{{ item.value.dest }}"
  loop: "{{ journal[rollback_journal_record]|dict2items|selectattr('value.changed') }}"
  loop_control:
    label: "{{ item.key }}"

```

```
shell> cat roles/test_rollback/defaults/main.yml
# Journal
journal_config_varnames: "{{ q('varnames', '^config_*') }}"
journal_config_vars: "{{ q('vars', *q('varnames', '^config_*')) }}"
journal_config_dict: "{{ dict(journal_config_varnames|zip(journal_config_vars)) }}"

```

```
shell> cat roles/test_rollback/templates/test.ini.j2
test = A

```
</sup>
> 1 votes

---
Tags: ansible
---


# Question
Title: Migrating from dockershim to containerd - kubectl continues to show docker runtime
Following directions from here. End goal is upgrading cluster to 1.24 and beyond.

At the end when I run `kubectl get nodes -o wide` seems like I would see 'containerd' as the runtime (link) but instead I see **docker://23.0.1**. Granted - the other nodes are showing **docker://1.13.1** so its an improvement?

On the node I see many instances of `/usr/bin/containerd-shim-runc-v2` whereas on other nodes I see `/usr/bin/docker-containerd-shim-current` so its definitely *different*.

Anyway - what step(s) did I miss?

# Answer
First check what kubelet says

```
journalctl -u kubelet

```

What worked for me was that in the plugins section in /etc/containerd/config.toml that systemdcgroup had to be true

```
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
            SystemdCgroup = true

```

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

systemctl restart containerd after that of course

On another cluster the logs said something like "server http response to https client" so containerd did not end up pulling images.

The solution for that was to turn on TLS skip verify

```
 [plugins."io.containerd.grpc.v1.cri".registry.configs."<MASTERIP>:5000".tls]
          insecure_skip_verify = true

```

and most important setting grpc.mirrors endpoint as http.

```
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."<MASTERIP>:5000"]
  endpoint = ["http://<MASTERIP>:5000"]

```

So one cluster on k8s V1.26 has the systemdcgroup as true and https enabled and the other on k8s V1.23 has systemdcgroup as true and http enabled. Only after those kubectl get nodes showed a CRI of containerd on both
> 1 votes

---
Tags: docker, kubernetes, containerd
---


# Question
Title: GitLab CI/CD: How do I generate a change log between pipeline runs?
I'm currently using Gitlab CI/CD combined with Fastlane on our build machine to perform nightly builds of an iOS app, and push those builds to TestFlight on completion. So far, so good. However, I've been asked to generate a change log of the new commits that have gone into each night's build.

Is it possible to get not just the `CI_COMMIT_SHA` of the current pipeline run, but also the SHA of the previous commit that the pipeline last ran for? That way, I could generate a change log of what's new in the current pipeline run? After searching the docs on Gitlab.com, I'm coming up empty so far.

As part of the pipeline, we also generate a build manifest that includes the current commit SHAs used for the build, and save that as an artifact. So, if it was possible to download the most recent artifacts, I could also maybe just look at that manifest, but again, I'm not seeing a way to do that either.

# Answer
Note: the answer is rather speculative, based on docs only, I didn't actually use GitLab pipelines yet.

You may be able to use the Pipelines API. More specifically you could get a pipeline's variables:

> Get variables of a pipeline
>
> ```
> GET /projects/:id/pipelines/:pipeline_id/variables
>
> Attribute Type            Required    Description
> id            integer/string  yes         The ID or URL-encoded path of the project owned by the authenticated user
> pipeline_id   integer         yes         The ID of a pipeline
>
> ```
>
> Example of response
>
> ```
> [
>   {
>     "key": "RUN_NIGHTLY_BUILD",
>     "variable_type": "env_var",
>     "value": "true"
>   },
>   {
>     "key": "foo",
>     "value": "bar"
>   }
> ]
>
> ```

I'm suspecting that in the response you'd find the variables shown in a pipeline creation response example, which include these which could be of interest to you:

> ```
>   "id": 61,
>   "sha": "384c444e840a515b23f21915ee5766b87068a70d",
>
>   "before_sha": "0000000000000000000000000000000000000000",
>
> ```

I'm uncertain if the `before_sha` for the current pipeline is what you're looking for, if not you'd just get the `sha` for the previous pipeline ID, which you could derive from the current pipeline ID (via the `CI_PIPELINE_ID` environment variable, see Predefined environment variables). I'm assuming that the pipeline IDs are consecutive.
> 0 votes

# Answer
As you suggested, having a build artifact to save the current commit SHA for the build would be ideal. Subsequent runs of the same workflow can download the artifact using the dawidd6/action-download-artifact action, which allows downloading artifacts generated from another workflow run.

Snippet for uploading artifact:

```
- name: Upload artifacts
  uses: actions/upload-artifact@v2
  with:
    name: nightly-build
    path: ./COMMIT_SHA

```

Snippet for downloading the latest artifact from another workflow run:

```
- name: Download artifacts
  uses: dawidd6/action-download-artifact@v2
  with:
    workflow: nightly.yml
    name: nightly-build
    path: .

```
> 0 votes

# Answer
This may not be appropriate to your specific case, but I think it's worth chiming in that a general approach to "generate a changelog" -- especially if it's supposed to be something *human readable* would be to use semantic release.

This relies on a formal syntax in commit messages, which are then processed by `semantic-release` during the pipeline execution.

The changelog plugin can be used to generate the changelog, using a Lodash template for the formatting. You have access to the relevant commit data in `result` object returned by semantic release.

You can find a Gitlab CI example at their CI recipes page:

```
# The release pipeline will run only if all jobs in the test pipeline are successful
stages:
  - test
  - release

before_script:
  - npm install

node:10:
  image: node:10
  stage: test
  script:
    - npm test

node:12:
  image: node:12
  stage: test
  script:
    - npm test

publish:
  image: node:12
  stage: release
  script:
    - npx semantic-release

```

If you just want to generate a changelog and not a full release object with artifacts and all, you can use the changelog plugin:

```
{
  "plugins": [
    "@semantic-release/commit-analyzer",
    "@semantic-release/release-notes-generator",
    "@semantic-release/changelog",
    "@semantic-release/git"
  ]
}

```
> 0 votes

---
Tags: gitlab, gitlab-ci-runner
---


# Question
Title: Automated preview environments from microservices in multiple repositories?
I have multiple microservices (with Dockerfiles) in GitHub repositories.

Although I do unit testing on each microservice, I want to spin up all the services as a preview environment for E2E testing (review environment in Heroku lingo).

Locally, I have a `docker-compose.yml` file that can create this environment. However, I would like to run this in an automated manner for each PR in each repository.

What CI pipelines/tools are there to enable this workflow that supports:

* Creating a new preview environment for each new PR (and destroying it when PR is closed)
* Supports running E2E tests
* Can post test results on the GitHub pull request
* As simple as possible. The services already have self-contained Docker images, so only configuration is needed through environment variables.
* Big bonus if the environment can be accessed with a public URL

# Answer
> 0 votes
I understand you're looking for a way to create automated preview environments for your microservices in multiple repositories. It's great that you're already doing unit testing on each microservice, and now you're looking to run end-to-end (E2E) tests in a preview environment for each new pull request (PR) opened in each repository.

To achieve this, you can use a container orchestration tool like Kubernetes or Docker Swarm to manage the deployment and scaling of your microservices. You can then use a continuous integration (CI) tool like GitHub Actions, GitLab CI/CD, CircleCI, or Jenkins to trigger the creation of preview environments when a new PR is opened.

Your CI pipeline can be configured to build and deploy the microservices using their Docker images and environment variables, and then run E2E tests against the preview environment. Test reporting tools like pytest or JUnit can be used to generate test reports that can be uploaded to the PR on GitHub for easy review.

It's important to store environment variables securely using a secret management tool like HashiCorp Vault or AWS Secrets Manager to prevent sensitive data from being exposed in the code repository or preview environment.

For accessing the preview environment with a public URL, you can use a tool like ngrok or a cloud provider like AWS Elastic Beanstalk or Google App Engine to expose the environment. This makes it easy for developers and stakeholders to access the preview environment for review and testing.

By automating as much as possible using tools that integrate well with each, you can create a streamlined workflow for creating and managing preview environments for your microservices that can help you catch issues early in the development process.

---
Tags: continuous-integration, microservices
---


# Question
Title: How does GitLab handle stranded (orphaned) artifacts in Object Store?
If I have a two minio clusters,

* one running in production and,
* one running as a hot-backup streaming with `mc mirror`,

And, if I have a point-in-time-backup running `gitaly-backup`. What happens if new artifacts are added *after* `gitaly-backup` is run? Are they effectively stranded? Does GitLab prune them by default? And if not, can I trigger a rpruning?

# Answer
# `gitlab:cleanup`

In order to clean up artifact storage post-restoration you'll need to run the following from the gitlab:cleanup, (you may want to try without `DRY_RUN=false` first).

```
sudo gitlab-rake gitlab:cleanup:orphan_job_artifact_files DRY_RUN=false

```

In the same light, you also remove uploaded files after the PITR backup,

```
sudo gitlab-rake gitlab:cleanup:project_uploads

```
> 0 votes

---
Tags: gitlab, backup
---


# Question
Title: How can I deploy a Kubernetes cluster onto OpenStack with Terraform?
*I know OpenStack has a supported method of installing a Container Container Orchestration Engine (COE) like Kubernetes using Magnum. I can not use Magnum because I am not an administrator on OpenStack and we do not have the "Service Endpoint" for Magnum installed.*

---

Terraform has the ability to provision resources on OpenStack: I have this working. Is there any easy way to provision multiple resources and to configure a Kubernetes cluster on top of OpenStack from a non-admin level of permissions? I'm open to using other tools like Ansible to get this job done. I'm just wondering if this is possible as a regular non-admin of OpenStack to easily set up a Kubernetes cluster from the resources I *can* provision?

# Answer
Most likely the best path forward is to provision the VMs using Terraform, and the rest with Ansible.

OpenStack can be set up a number of ways, but I believe the only hook in the terraform provider that would allow you to execute code on the node would be via cloud-init user data (bash script).

What I would do is provision nodes with terraform and have it provide outputs, have python generate Ansible hosts and vars from those outputs, and then run Ansible.

What flavor of k8s are you looking at? k0s, k3s, microk8s, etc?
> 0 votes

# Answer
Terraform + Ansible could be the best path forward.

Terraform also has local and remote exec provisioners when provisioning VMs. Local exec runs scripts from the local machine you're deploying from and remote exec spins up an on demand SSH server using a local public key for authentication to run commands on the deployed VM for configuration (eg setup.sh)

https://developer.hashicorp.com/terraform/language/resources/provisioners/remote-exec

```
resource "aws_instance" "web" {
  # ...

  # Establishes connection to be used by all
  # generic remote provisioners (i.e. file/remote-exec)
  connection {
    type     = "ssh"
    user     = "root"
    password = var.root_password
    host     = self.public_ip
  }

  provisioner "remote-exec" {
    inline = [
      "puppet apply",
      "consul join ${aws_instance.web.private_ip}",
    ]
  }
}

```
> 0 votes

---
Tags: kubernetes, openstack
---


title: Unable to deploy grafana using helm

text:
# Question
Title: Unable to deploy grafana using helm
I'm new to grafana and prometheus, started working on it from last few days. I'm having some issues. I build a new grafana image and I am using the official helm repo to install it. On using the official image its wotking but on using my image the Pod is showing error. `Failed to pull image "my-repository/grafana:9.4.1": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/my-repository/grafana:9.4.1": failed to resolve reference "docker.io/my-repository/grafana:9.4.1": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed` On checking I understood that I need to add imagePullSecret but after add that too i'm having the issue. This is where I have added image pull secret

```
#Overrides the Grafana image tag whose default is the chart appVersion
 tag: "9.4.1"
 sha: ""
 pullPolicy: IfNotPresent
#Optionally specify an array of imagePullSecrets.
 #Secrets must be manually created in the namespace.
 #ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
 #Can be templated.
 #pullSecrets:
 #- myRegistrKeySecretName
imagePullSecrets:
-name: secretname

```

What might be the issue here, any insights will be helpful

Tried adding docker secret with few indentation but still having same issue.

# Answer
In the YAML you have pasted the `pullSecrets` line and the `- myRegistrKeySecretName` lines are commented (with `#`); if you have used exactly what you have pasted, that would be the source of the issue.

Otherwise, I would check the following:

1. you have actually pushed the image to docker.io with the same name and tag
2. the ImagePullSecret is actually valid
3. You have named it correctly in the spec (e.g., `myRegistrKeySecretName` seems to have a typo)
> 0 votes

---
Tags: docker, kubernetes, helm, grafana, kubernetes-helm
---


# Question
Title: Are webhooks needed in Gitlab for triggering a build on tags push
I would like to trigger a Build on Gitlab when a Git Tag is pushed to the repository. At the same time I would like to increment the version number on a file located on the repository before building it on a runner. My question is . Is it a webhook needed for the purpose ?

# Answer
You don't need a webhook.

You can use the `tags` keyword with the `only/except` condition syntax for running jobs in a pipeline.

You then need to implement the job of course, which will do what you need:

1. take the new tag
2. update a file with it
3. commit the new file and push it back to the repo
4. ... other builds steps ...
> 0 votes

---
Tags: gitlab, gitlab-ci-runner
---


# Question
Title: What are the core differences between DevOps and Agile ? And Is both two different approach to solve the similar problem?
I read some articles related to DevOps and Agile and found both are two different methodologies for software development. My questions is, If both methodology is solution for same problem then what is advantages of one over another.

# Answer
**Agile** is a methodology to organize (or group) a set of changes to software components in a give period (= *sprint*), so think about it as a planning tool or so. Whereas the stake holders are the scrum master, the PO, the actual developers, etc.

**DevOps** is (mostly) about shipping modified components from one environment (e.g. unit testing) to another one (e.g. acceptance testing), and for which things like **CI** and **CD** can be used. Whereas the stake holders are the developers, the testers, operations control people (or SREs if you prefer), etc.
> 1 votes

# Answer
**Agile** is really a simple manifesto about how software should be developed by people working collaboratively with customers. It assumes change will happen (rather than trying to lock down requirements early) and uses demonstrable working software to track progress.

The manifesto was a response to heavyweight software delivery models.

Since 2001 (when the manifesto was written), we have learned a great deal using an agile style of software development. Lots of different methods (XP, Scrum, Disciplined Agile Delivery, and so on) were created under the Agile umbrella.

**DevOps** was originally a very simple idea about getting development teams and operations teams to collaborate. One of the main ideas here was to align their goals to remove an almost universal conflict between Dev and Ops that was caused by asking:

* Developers to delivery more stuff, more often
* Operations to keep everything stable

Over the past decade, DevOps has grown to include Lean, Agile, and Continuous Delivery (the modern version of XP) - thanks to a great deal of research by the folks at DORA (DevOps Research and Assessment).

If you look at the DevOps model, it includes culture, product development, technical practices, Lean management, and lots of other very specific ways of improving your performance. It doesn't require you to do all of them, it's like a list of ideas that can help solve problems. There's a predictive relationship that links the capabilities together with business outcomes.
> 0 votes

---
Tags: deployment, release-management
---


# Question
Title: Why is the development of a REST-API Webserver Application with less expected changes after its short term initial build a DevOps project?
I am thinking about setting up a project where a one-year development / built phase of a rest-API Server is planned. Providing an initial version in three months and deploying updates every month during the first year should be enough.

Further information to consider:

* After the services are built, they should be relatively stable for a long time. It is not expected to change the code every week. Probably one time in three months.
* There are only 10 people affected if the API does not work. And there will be no major business impact from being down.
* Code is not affected as a whole. Changes in one web service API will not affect others.

To set up a DevOps structure in order to ship updates fast and continuously is questionable since it takes some effort to set it up with no real benefit. Since there are not many changes expected during the lifecycle, there is no real benefit of DevOps.

Is this view right? what else should I consider?

A friend recommended me to use DevOps procedures anyway. Does this makes any sense?

# Answer
> 0 votes
DevOps isn't just linked to frequent high-quality deployments, it is also linked to many other outcomes. The capability model for DevOps that is published by DORA allows you to adopt the techniques and practices that are relevant to your context. You don't need to tick every box in the model.

The 2022 State of DevOps report used clustering to identify different performance levels that aligned to different product development stages.

If you aren't going to be making frequent changes to the software:

* Starting (making sense of the problem by experimenting to a solution)
* Flowing (active development of a competitive software product)
* Slowing (less frequent changes, customers like the product)
* Retiring (planning to retire software)

It sounds like the goals of your API align to that "slowing" state, where you don't deploy often but can keep a reasonable level of reliability.

You would apply "just enough DevOps" to make your life easier. For example, it may be tempting to do manual deployments because you won't do them often, but automating this will mean they work (if you don't deploy often, you forget the manual steps and make mistakes).

So, yes - I'd still "do DevOps" - though not every single capability in the big diagram. Applying Continuous Delivery is a good starting point for your use case.

---
Tags: continuous-integration, continuous-deployment, continuous-delivery, project-management, the-phoenix-project
---


# Question
Title: I activated all three API Keys on my Etherscan.io account, was I only suppose activate one (1), not three (3)?
I am new to web3, cloud, C++, CLI, Meta and Stack. This is all a new world a new platform for me. I have no one on one teacher I started off being a service (gig) worker to Content Creator-to-Developer in matter of the last six months. I learn from trial and error. I signed my new business up with Walmart Marketplace last week and I was denied access to the platform because it said I had too many API accounts or something to that affect. Don't hold me to the exact wording and terminology though, I'm behind on task so I haven't dealt with he issue yet.

However, all that said, it raises my first question above. Basically how many API's could I activate on one platform? Is one enough, or, more equals workflow efficiency?

# Answer
Organizations who offer an API usually have in place a mechanism to limit your use. For example, they will offer tiers of service at different prices, like:

* Free (for up to 100 requests a day)
* Business (up to 1,000 requests a day)
* Enterprise (unlimited, usually "call us for pricing")

When someone creates multiple accounts, it can look like you are trying to bypass the limits, for example by attempting to use ten free accounts to get 1,000 requests a day.

If you use more requests than you pay for, you will normally get an HTTP 409 response, which means "too many requests". When you create several accounts, you are likely to find they get locked.
> 1 votes

---
Tags: azure-devops, api, devsecops
---


# Question
Title: What are the good metrics to collect for frontend application?
Context: We have some features that rarely calls any backend service and the logic stays in the frontend.

Most of the tools we have can calculate SLI/SLO status based on latency and error code return from the server side, but right now since the applications is frontend-heavy, we're mostly tracking bugs based on user reports, manually.

We're trying to add some services to track the end user interaction and hope that we can collect some metrics like number of clicks for example.

The main question is: what are the good metrics to collect for a frontend application?

I think analyzing bug tickets, user feedback, and surveys are good methods also but I'm looking for some "black box" metrics so based on that we can set up some kind of alerts to notify our team to send someone to investigate before bug tickets arrive.

# Answer
> 0 votes
There is a crossover here with some analytics tools as you are probably interested in:

* Front end errors
* Rage clicks / dead clicks
* Time from interaction to delivery (i.e. how long to do something after a user clicks)

If you monitor these, you can set up automatic alerts when there's an anomaly, such as a sudden increase in front-end errors that may have been caused by a change in the latest version of your app.

Rage clicks and dead clicks indicate usability problems.

You'd expect the timings to be super-fast if the whole app runs in the front-end without communicating with back end services, but this will certainly offer surprises if there's an operation that takes longer than a few milliseconds!

## Business metrics

Additional to this would be tracking the business metrics that relate to the app. Are you hoping people will do something as a result of the app? How many / what proportion of users do the thing you wanted?

A new version of the app might be technically brilliant, but if users stop doing the business thing you hoped for, it might need to roll back to keep your business operating.

As an example, if you ran a hotel bookings site and released a change to the sorting of the results, it could reduce the number or value of hotel bookings, which could put you out of business if you don't go back to the old sorting.

---
Tags: monitoring, sre, metrics
---


title: Release rollback approaches?

text:
# Question
Title: Release rollback approaches?
I've got an old monolithic project using MariaDB and PHP that I'm trying to put on CI/CD. The project has DB migrations to update the DB state. My current CD is built on top of AWS CodeCommit + CodePipeline + CodeBuild + CodeDeploy pushing a Docker image into ECR and then making AWS ECS redeploy the service. The container itself applies the DB migrations before starting the web server.

New deployments are ok, but I can't figure out how to rollback properly.

If I try to just deploy the older version of the image it won't have the migrations that need to be reverted and the DB state will differ from the one expected by the code.

I couldn't find any AWS guidance on the rollback with the special handling of the DB migrations, so it looks like it is up to me to come up with some solution.

My current idea is to update the build process so it would look for migration files that exist in the currently deployed image (the one being rolled back) but not in the upcoming deployment (some previous app version), and store them in a special directory. When the container will start it will revert all migrations found in the special directory.

This approach has a big flaw: new migrations are not guaranteed to work with the old code and/or configuration. Same time, the reverted code is not guaranteed to work correctly at all, hence I can't use the rolled-back image to take down its changes.

What would be the correct implementation of the rollback in this case?

# Answer
> 2 votes
Databases are definitely the most challenging area for rollbacks.

The binary approach to rolling back a database is to restor a backup from before the upgrade. You then have to work out how you deal with any data that changed after the backup.

Instead, we normally apply database refactoring patterns to carefully introduce non-breaking changes using a series of small steps.

The following are "safe changes":

* Adding a table
* Adding a column with a default value

Any previous version of your application would work after these changes as it simply wouldn't know the new items existed.

If you perform these:

* Adding a required columns
* Renaming anything
* Deleting anything

The database is no longer compatible with older versions of the software. This is where the refactorings help, as they provide a path that lets the change be divided into smaller safe steps.

At the end of a database refactoring, you still have to track compatibility as there will be a "minimum application version" supported by the database.

Without database refactoring, you'll be tackling the big data issue every time there is a problem (unless you can roll-forward the application instead). With database refactoring, you should be able to avoid the restore-from-backup scenario.

---
Tags: amazon-web-services, deployment, continuous-deployment, databases
---


title: Static Analysis for Feature Flag Safety

text:
# Question
Title: Static Analysis for Feature Flag Safety
In a CD environment with new features hidden behind feature flags, there must be a way to ensure that code being pushed to production is effectively zero change?

Particularly in a change management controlled environment, it would be ideal to not treat code deployment as a change - only the feature switch on. To be able to do this, is there a way to prove that all code changes are protected behind feature flags? And that the effective flow is zero change. Feels like something that could and should exist. But maybe others solve this differently?

# Answer
You've highlighted the great conundrum of feature flags!

I don't believe it would be possible to check this with static-analysis alone as it requires some knowledge of application behaviour (even if the understanding is "what is a feature flag", as they often look like things that aren't feature flags).

In order to prove everything is hidden by feature flags, you'd need to be able to run a test suite aginst the application with all flags switched off.

Whether you also need to test "on" states and combinations of "on" states depends a lot of how you are using feature flags. When you use them to hide a feature in development it's very different to when you make a system user-configurable with flags, or have multiple values per flag.

More flags and flag values = way more combinations. It quickly becomes possible to only test "selected combinations".
> 0 votes

---
Tags: continuous-delivery, feature-flags, feature-toggles
---


# Question
Title: DevOps & Development Teams Processes, Procedures, Communication & Coordination to ensure tha latest supported technologies are used in development
Like many IT departments, we have numerous Developer teams, and DevOp teams.

Recently, our .NET Development Team just completed an application based on .NET 3.1 that took around a year to complete.

We were excited to coordinate with the DevOps team in question in order to deploy said application to the UAT environment.

Unfortunately, as we were making progress with deploying said application, the DevOps team Management Stopped us because they said that .NET 3.1 is No longer supported by Microsoft. Therefore, they said that inspectors/auditors would Raise serious concerns about having Unsupported technologies deployed to production environment.

The politics behind said incident seemed to sort of suggest that the DevOps team put the blame on our .NET Development Team for failing to know that the .NET 3.1 is No longer supported by Microsoft.

As a member of the .NET Development team, I just felt like we took too much of the blame.

What kind of management techniques/processes/procedure could have been put in place in order to prevent said setback incident?

# Answer
> 1 votes
Part of the problem is that you do not have a DevOps mindset. You don't have "DevOps teams" - there's no such thing. DevOps is a set of values and principles that build upon Agile Software Development. A key principle of Agile Software Development is that "business people and developers must work together daily throughout the project". DevOps expands this to tear down silos that existed between development teams and operations teams and to move operational considerations earlier in the system development life cycle, which is sometimes referred to as "shift left".

I can tell that you don't have a DevOps mindset because you didn't find an operational concern until before deployment to a UAT environment. If the use of outdated and unsupported tools and technologies is a risk, I would expect that to be moved earlier in the process, even into requirements. You would have specific requirements around deploying only supported dependencies, and perhaps have accepted deviations in place if there are technical reasons why it is not possible. Part of the DevOps culture is a culture change to move toward automation, so I would expect static analysis and dependency analysis tools to be in place and warn if there are third-party dependencies that contain known vulnerabilities or are end-of-life to allow the teams to take appropriate action.

So if you really want to prevent this type of problem, I would recommend actually looking at the DevOps values and principles. Get your operations specialists involved in your system requirements and find ways to build quality into your product, likely through automation.

# Answer
> 1 votes
There are some DevOps capabilities that would really help your situation.

First a foremost, the original idea behind DevOps is that developers and operations collaborate the whole time. If you have a development team working in isolation of the operations team, it's definitely not DevOps even if they have been named "DevOps".

Let's get into some capabilities. I'm going to hand pick some specific techniques and practices that I think would really help future work in your organization.

## Culture: Westrum organizational culture

Quite a lot of your original question involves blame. DevOps really needs a generative culture. That means a blameless culture with high psychological safety. You need to build trust, maximize learning, and aim to have people who are satisfied they can do good work.

This is one of the top-predictors of successful outcomes.

You may not be in a position to re-wire the organizational brain, but you can definitely make an impact with how you respond to your situation. Rather than looking for someone to blame, work out how you can collaborate with those other teams to get better results next time. A bit of empathy and relationship building will go a long way.

## Lean product development: Work in small batches

If there is one universal lesson that came out of phased software delivery, Agile, Lean, and DevOps... it's *work in small batches*.

Imagine you had created a minimal .NET application in the first week. It just has one API endpoint that returns "hello world". If you delivered that version of the application you might have found out about the .NET version issue a year earlier.

Imagine the difference in finding out a year ago, rather than when you are about to go live. The situation wouldn't have blown up as the pressure to get things live is adding pressure.

Working in small batches would also mean having a deployment pipeline that you and the operations team had confidence in as you would have been deployment every week for a year.

## Technical capabilities: Continuous Delivery

This is really a number of capabilities all bundled together - but they are highly inter-related. Frequent integration of code into the main branch and an automated deployment pipeline that make your code "always ready to deploy" are a great way to remove the stress from software delivery.

In your case, building the deployment pipeline would have meant collaborating with the ops team earlier and finding out what kind of governance, risk, and compliance needs they might have.

A good build process and automated deployments will boost your levels of joy. It will also enhance confidence in your team if you can make this area pain free.

## Capabilities everywhere!

You can explore more capabilities on the DORA research site. There's a lot of research and analysis that give us confidence in the relationships in the model.

## .NET Note

Upgrading the .NET version shouldn't be too tricky. I've updated .NET Core MVC sites to .NET 6 and it none of the code needed to change. I tend to update regularly, so I'm now using .NET 7, but you may want to stick with .NET 6 as the even numbers have 3 years of support. The odd numbers have only short term support.

I hope you can get your API updated and get it launched. Now is the time to introduce some kind of continuous improvement process to make things better going forward. Once the dust settles and the crisis is over, people will be less willing to change the status quo.

Good luck!

---
Tags: project-management, process
---


# Question
Title: Why is Ansible task getting stuck for so long?
I'm new to Ansible, I've made a playbook with only two tasks as shown below:

```
- name: Installing Kubernetes for the master node
  connection: ssh
  hosts: x.x.x.x
  become_method: su

  tasks:

    - name: installing curl
      ansible.builtin.shell:
        cmd: sudo apt install --yes apt-transport-https curl
      vars:
        ansible_persistent_log_messages: True
      register: update_curl_install_output

    - name: update output
      ansible.builtin.debug:
        var: update_curl_install_output.stdout_lines

```

My ansible.cfg:

```
[ssh_connection]

ssh_args = -o ServerAliveInterval=100 -o ControlMaster=auto -o ControlPersist=2m
enable_task_debugger = True

```

And running the playbook with the following command:

```
ANSIBLE_ENABLE_TASK_DEBUGGER=True ansible-playbook -vvv master-node.yaml -i inventory --extra-vars "ansible_sudo_pass=managed_host_pass"

```

The remote host is easily getting connected, but the task is getting stuck for a long while. The logs below show nothing atypicality.

```
<192.168.1.15> (0, b'', b'')
<192.168.1.15> ESTABLISH SSH CONNECTION FOR USER: adil
<192.168.1.15> SSH: EXEC sshpass -d11 ssh -o ServerAliveInterval=100 -o ControlPersist=2m -o
'User="adil"' -o ConnectTimeout=10 -o ControlPath=/root/.ansible/cp/7bf699d458 -tt
 192.168.1.15 '/bin/sh -c '"'"'/usr/bin/python3 /home/adil/.ansible/tmp/ansible-tmp-
 1679218469.1440735-1171472-85023283189053/AnsiballZ_command.py && sleep 0'"'"''

```

And after that it shows no logs, completely stuck. Any ideas?

# Answer
Answering those who could face the same problem. I solved the problem, it was due to putting the escalation method in the playbook.

```
become_method: su

```

Rather than it should be this

```
become: true

```
> 1 votes

---
Tags: ansible, ubuntu
---


# Question
Title: How to generate a list of every external URL accessed during docker build?
Is there a straight forward way to generate a `.txt` file containing each URL (or corresponding IP) visited during a docker build process?

### Example

An example list might look like this:

```
# urls.txt
https://hub.docker.com/
https://www.npmjs.com/

```

(but a real case using an extensive Dockerfile may be more complex and contain many more URLs/IPs).

### Things I tried so far

**1. Installing and using an intrusion detection engine**

IDSs I attempted to install and use were feature rich and probably more complex than required for what I'm attempting. Configuring IDS engines is not trivial (especially for someone with little network knowledge), so I'd like to explore simpler solutions, if possible.

**2. Collecting and parsing docker build logs**

I captured build logs with this:

```
docker build --no-cache --progress=plain  . &> build.log

```

and extracted lines containing a URL with this regular expression:

```
(http|ftp|https):\/\/([\w_-]+(?:(?:\.[\w_-]+)+))([\w.,@?^=%&:\/~+#-]*[\w@?^=%&\/~+#-])

```

A lot of URLs are found, but... i) it doesn't find URLs which aren't explicitly stated in the build logs, and ii) it finds URLs which aren't actually accessed but which are simply stated in logs (i.e. logs along the lines of "For more info, visit https://www.example.com".

### Question

How can I generate a nice clean list of each external URL visited during a `docker build` process?

# Answer
One thing that pops my mind is using either of Wireshark, tshark, or tcpdump for capturing network packets originating at docker daemon while docker build is running.

After recording the traffic, need to filter & process the captured data.
> 3 votes

# Answer
Maybe you could use some local redirecting proxy service and patch your *localhosts* file to get all requests caught there?
> 1 votes

# Answer
You could start with something like this (`tcpdump`):

```
tcpdump -q ip and ip6 and dst port 80 or 443 > tcpdump.log

```

This should capture all outgoing ipv4 and ipv6 traffic, to destiation ports 80 and 443 (http and https). You can add `-i eth1` to capture data on a specific interface only. You can play with filters. You should get something like this (results from curling google and facebook) - notice that you won't get actual **google.com** domain, but the results could still help to see which domains are blocked by your firewall:

```
19:07:58.915115 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.915125 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.947639 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.947869 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:07:58.948406 IP pmsbuild02.4752 > bud02s38-in-f4.1e100.net.http: tcp 0
19:08:04.608537 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.610415 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.610494 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 80
19:08:04.656677 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.656887 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0
19:08:04.657372 IP pmsbuild02.10146 > edge-star-mini-shv-02-frt3.facebook.com.http: tcp 0

```

You can then easily filter out the domains with awk (get the 5th section, delimited by spaces) - note that there's still the http/https appended to the end of the domain, and the colon sign:

```
awk '{print $5}' tcpdump.log

```

Result:

```
edge-star-mini-shv-01-vie1.facebook.com.https:

```
> 1 votes

---
Tags: docker, networking, network
---


# Question
Title: Can I have a GitHub hosted Github Enterprise instance running under my own domain?
I'm trying to figure out whether the following is possible, but I might be trying the wrong search queries or I'm not understanding some concepts around GitHub Enterprise, because so far I came up with no clear answer.

So

* I'd like to have a GitHub Enterprise instance running under my own domain (reachable like so: github.mydomain.com)
* But I'd like to have GitHub manage the instance for me (upgrades, backups, etc., i.e. a GitHub Enterprise Cloud instance)

What I found so far is that I could run my own instance on my own servers (or on an AWS EC2 instance) with a GitHub Enterprise Server instance, but I think that would mean that I'd be responsible for all the maintenance related to running that instance. The other option (GitHub Enterprise Cloud) seems to be to have my organization be hosted on the main GitHub portal (like github.com/myorg), and I'd like to have more, clearer separation than that.

Does anyone know whether I missed something obvious, or is what I'm trying to do not supported?

# Answer
If You want to separate from GitHub.com, ask GH Team for a GitHub Enterprise Managed Users trial. It is hosted on GH.com but as a "separate instance," so like with GH Server, all users will be forced to have separate accounts with read-only access to public projects (outside of Your organization). Main pros:

* separation from public GH.com
* no maintenance is needed from Your side
* no public projects/repositories/organizations available to create

**https://docs.github.com/en/enterprise-cloud@latest/admin/identity-and-access-management/using-enterprise-managed-users-for-iam/about-enterprise-managed-users**
> 1 votes

---
Tags: github
---


# Question
Title: How do you deploy a container to AWS Lambda?
I am working on this little project.

https://github.com/jamesgeddes/itsbeginningtolookalotlikechristmas

It's currently a work-in-progress - don't judge!

As you can see, I have configured CircleCI to build, scan and publish the container. That, ideally, should be the end of the road for the Continuous Integration, CI, tool, as Continuous Deployment, CD, should take over, so I would like Docker Hub Webhooks to trigger the deployment of that container into AWS Lambda.

Unfortunately, I am struggling to find any dedicated CD tools.

I have seen,

* Octopus Deploy, but their minimum is £10 per month - more than I would like to pay for such a small project.
* Harness CD, but their £0 plan only allows you to deploy to Kubernetes

Everything else appears to be,

* a CI tool
* a CI tool that insists that it can do CD as well
* self-hosted - which I can't be bothered to do for such a small project

How should I go about deploying my container to AWS Lambda while maintaining the best-practice separation of concerns between CI and CD?

# Answer
> -1 votes
The community edition of Octopus is free to self-host, so that might be an option if you have somewhere you can run it. You can have 5 targets and 5 projects.

You could run a free cloud trial to test it out and then think about self-hosting once you've worked out the details.

Once you have it, you could use it to deploy other stuff you have, too. I'm using Octopus Cloud community edition to deploy a little monitoring tool I wrote (UpDn) that checks my websites are up, their certificates aren't about to expire, and that my DNS hasn't been messed with. It easily saves me $10 of effort each month :)

---
Tags: amazon-web-services, deployment, continuous-deployment, continuous-delivery, aws-lambda
---


title: Load balancing while deployment

text:
# Question
Title: Load balancing while deployment
How does a `L4` or `L7` load balancer knows how to route traffic during deployments. Say, In `Canary` deployment strategy, how does LB route only 10% traffic to a certain server or in case of `Blue-Green` deployment strategy, how does LB knows that the updated application is available on another set of servers and I should be routing the traffic to them.

Where should we make those configuration ideally ?

# Answer
> 1 votes
Your load balancer usually offers this as configuration. Using HAProxy as an example (this is similar for most load balancers) you can change:

1. Whether a server is in balance or not
2. What proportion of traffic it should receive
3. What strategy to use to divide traffic between servers

Common use cases for this would be:

* Taking a server out of balance (and allowing existing connections to drain) before running a deployment, then warming it back up with a slower rate of traffic.
* Splitting a small proportion of traffic for a canary release, which you might also use "session pinning" for, to make sure a canary user *stays* on the canary server, rather than flip-flopping between the old and new version

With HAProxy, you can perform some operations via the command line to take servers in and out of balance. For the load balancing strategy, you'd normally change the config and reload it (which, again, you could automate via the command line).

It's not uncommon for people to set up HAProxy with "redispatch" configured. They can then surprise it by deploying to a server without telling it. HAProxy will receive the failure and re-send the request to a different server. The user has no idea there was any issue. You need to make sure this doesn't cause a domino effect in your scenario.

Nginx is pretty similar. Other load balancers offer things like REST APIs for these operations.

---
Tags: continuous-deployment, nginx, load-balance, high-availability
---


# Question
Title: What's the best practice for managing / deploying multi-environments in production app?
I'm spinning up a production site and wanted to get advice on the latest best practices to manage and deploy multiple environments. At a high level, I'd like to have a good balance between keeping prod from being broken vs dev speed.

This is my best high-level guess at what I should strive for:

1. Have devs try to ship small changes + small branches into trunk branches to avoid overhead.
2. Have multiple environments aside from prod, say a "dev" environment.
3. Have the "dev" environment use CI/CD so that whenever branches merge into trunk, the "dev" environment will build and deploy the latest trunk
4. Dogfood / QA "dev" environment frequently, and whenever we feel good about it and want to ship to prod new features that made into "dev", we manually deploy the current trunk to "prod" environment.

Thoughts on this approach? If this seems reasonable, I was wondering about tips on the details:

* To prevent bad trunk merges from breaking prod, does this imply deploying to "prod" environments is entirely manual w/o any CI / CD? Is this a common practice?
* We're using github. Would it make sense for both "prod" and "dev" environments to deploy from the master branch (aka trunk branch), one automatically vs one manually? Or would it make more sense for a separate prod release branches like "prod-release-v1", "prod-release-v2", etc. where we just cut from from the right commit in trunk branch whenever we want to deploy to prod?

Other context:

* We're a lean startup, 10 people atm, and looking to scale a bit more.
* We have prod website up and running and enough customers that it's worth investing in this to avoid too much breakage.
* AWS is most of our stack, but also comfortable with adding any other devops tech that's well recommended in the industry.

Appreciate any help or pointers I can get as a noob devops eng! Thank you!

# Answer
You've picked the right time to ask these questions (well, I can see you asked a *long* time ago, but I figured people will find this question who are experiencing the same situation - sorry the answer comes a bit late for you!)

Sorting out your deployment pipeline before you scale up makes it far easier to co-ordinate the required changes. Later on, you'd have to deal with different teams who grown their own deployment pipelines and might not want to change them.

## The research

The research is on your side for most of your plans... it has found a strong predictive relationship between the following techniques and improved software delivery performance *and* stability.

1. Work in small batches
2. Integrate continuously into trunk

You hare already identified that automated checks are vital to this. You need a five-minute build process that runs tests that catch problems early. If you have a longer set of tests that need to run on an environment, you should deploy to your "edge" environment automatically and kick off those long-running tests.

The goal is to ensure no human even looks at a software version unless it passes these checks.

## After the automation

If you have manual review of software versions, you may want to allow the people using that environment to pull new versions, so they can ensure an update doesn't occur in the middle of a test session.

This should be a push-button operation that deploys the software to their test environment. Ideally, they should also have a push-button operation to reset their test data to a known state (it sounds like hard work, but the investment really pays off!)

When they are done, one more push-button to say it's good to go.

## Production

It is possible to set up a continuous deployment pipeline that runs off the automated checks and deploys to production after the checks complete successfully.

Your organization may have *policy constraints* that mean a human has to approve the release. This is fine - you'll be in continuous delivery mode. Continuous Delivery really has all the same automation as Continuous Deployment, but it also has additional "manual intervension" steps, such as sign-offs and approvals.

In some rare cases, you may even want to schedule deployments to occur at specific times, so another manual intervention is needed - but the deployment itself should still be automated. You want the automated deployment process to be re-used for all environments as this makes deployments reliable and repeatable.

You can use a deployment automation tool to manage environments, approvals, deployments, and the different varaibles you need to use for dev, test, and production. You should also be able to set up things like self-service deployments and operations runbooks in your deployment automation tool, so you could make it easy to reset test data, clear out caches, or empty log file folders.

## Agile, Lean, and DevOps (oh my!)

DevOps as defined in the Accelerate State of DevOps report is the best way humanity has come up with to deliver software (up to now). It's worth keeping tabs on the annual report to see what comes up. You can use the DevOps capability model to get ideas for things that may improve your process (but don't try and adopt them all at once!)

Hope this helps some folks!
> 0 votes

---
Tags: continuous-integration, deployment, continuous-deployment, github
---


# Question
Title: Install gitlab on baremetal cluster using helm chart
I'm trying to install gitlab using helm on my kubernetes cluster running on vbox.

I have already installed traeffik, metallb and argocd.

I have tried to install gitlab using helm with this command:

```
helm upgrade \
  --install gitlab gitlab/gitlab\
  --timeout 600s \
  --set certmanager-issuer.email=me@example.com  \
  --set postgresql.image.tag=13.6.0 \
  -n gitlab

```

but I have several pods in `CrashloopBack`  due to PVC, PV and others issues.

I have read the documentation, but it is so big that I am lost.

So my question is:

Can I use helm chart to install gitlab on my baremetal cluster or it's only for designed to run on Cloud provider ?

# Answer
You probably do not have a default StorageClass.
I had similar problem.
Here is a piece of my own code in ansible provisioning nfs dynamic provider.

```
  - name: helm install nfs provisioner
    kubernetes.core.helm:
      atomic: true
      chart_ref: nfs-subdir-external-provisioner
      chart_repo_url: https://kubernetes-sigs.github.io/nfs-subdir-external-provisioner
      create_namespace: true
      name: nfs-provisioner
      namespace: nfs-provisioner
      state: present
      wait: true
      values:
        nfs:
          server: 10.1.1.20 (replace with your own)
          path: /srv/nfs/kubernetes/development
        storageClass:
          name: default-nfs
          defaultClass: true
          archiveOnDelete: false

```

You need to have nfs server installed on some server (in my case it is the same machine)
> 1 votes

# Answer
You can use Helm to install GitLab on a bare-metal Kubernetes cluster, as well as on a cloud-based Kubernetes cluster. Helm is a package manager for Kubernetes that makes it easy to install and manage applications on a Kubernetes cluster.

To install GitLab on a bare-metal cluster using Helm, you will first need to install Helm on your cluster. Then, you can use the GitLab Helm chart to deploy GitLab to your cluster. The Helm chart includes all of the necessary resources and configurations to run GitLab on Kubernetes.

To install GitLab using the Helm chart, you will need to run the helm install command and specify the GitLab chart repository, as well as any additional configuration options that you want to set. For example:

```
    helm install gitlab gitlab/gitlab \
      --set global.hosts.domain=example.com \
      --set certmanager-issuer.email=your@email.com

```

This will install the GitLab chart and deploy all of the necessary resources to your cluster. Once the installation is complete, you will be able to access GitLab from your cluster's external URL.

Keep in mind that running GitLab on a bare-metal cluster may require additional configuration and resources compared to running it on a cloud-based cluster. You will need to ensure that you have sufficient resources and network connectivity to support the GitLab installation and any additional workloads that you plan to run on your cluster.
> 0 votes

---
Tags: kubernetes, gitlab, helm, kubernetes-helm, kubernetes-on-premises
---


title: Pause renovatebot for single PR

text:
# Question
Title: Pause renovatebot for single PR
We have configured renovate for few of our repositories. It generates PRs after approval via dependency dashboard and automerges patch releases.

But there is one update which fails the CI - it look like there is an error in the library itself. While we continue to work on the master branch renovate rebases it's update branches which again triggers the CI.

Can I pause renovate to work on that single PR without closing it until we have have fixed this dependency so that it dos not start the CI over and over? (It will fail anyway)

# Answer
There is a configuration option named `stopUpdatingLabel`. With this feature one can add a label to the pr to stop renovate from updating it like so:

```
{
  "stopUpdatingLabel": "take-a-break-renovate"
}

```

The feature is enabled by default and listens on the label `stop-updating`.

It's documented over here: https://docs.renovatebot.com/configuration-options/#stopupdatinglabel
> 1 votes

# Answer
Maybe you could resolve this by pinning the version of the problem library temporarily so it doesn't update it to the "erroring version"?

You could use the `packageRules` configuration to set the allowed version to the known good one, preventing the problematic update.

```
"packageRules": [
  {
    "matchPackageNames": ["@angular/router"],
    "allowedVersions": "<15.0.0"
  }
]

```
> 0 votes

---
Tags: continuous-integration
---


# Question
Title: Plugin for Jenkins truncates Terraform Plan output
The Blue Ocean plugin for Jenkins truncates the output of a Terraform plan at around 1700 lines, and shows a button "Show Complete Log" for the full thing.

When I open the full log, Terraform plans that have colors are now turned into special funky characters that make it impossible to read the plans.

I'm looking to increase the amount of lines (or bytes) that can be shown in the nice interface (with colors) to make it easy to read.

Example of special characters:

```
[2023-03-23T17:48:53.254Z] [0m[1mmodule.mymodule.module.application.module.aws_something.data.file.policy: Reading...[0m[0m
...
[32m+[0m [0m[1m[0meffect[0m[0m    = "Allow"
[2023-03-23T17:48:53.280Z]           [32m+[0m [0m[1m[0mresources[0m[0m = [
[2023-03-23T17:48:53.280Z]               [32m+[0m [0m"arn:aws:sqs:ap-xxxxxxx-2:11212121212:xxxxxxxxxxxxxxxxxxx",
[2023-03-23T17:48:53.280Z]             ]
[2023-03-23T17:48:53.280Z]           [32m+[0m [0m[1m[0msid[0m[0m       = "PermitTopic"

```

Any help?

# Answer
> 1 votes
You can configure this with the `hudson.consoleTailKB` system property (default value is 150), documented on this page. How/where to set system properties depends on how you installed Jenkins and how you start your Jenkins service (for instance, I configure mine by setting the `JENKINS_JAVA_OPTIONS` variable in `/etc/sysconfig/jenkins`).

Alternatively - those look like ANSI escape codes to me, so you could try the ansicolor plugin or load your logs into a viewer/editor which speaks ANSI color codes.

---
Tags: jenkins, jenkins-pipeline, jenkins-blueocean
---


# Question
Title: Upload pages artifact without overwriting previous data
I am trying to create a GitHub workflow that would essentially deploy multiple static websites to GH Pages. The sites would live in their own subfolders beneath the (website) root, and the catch is that they need to be deployed independent of one another.

For this to work I need to upload a new website build into its subfolder without changing the said folder's siblings. I'm using `actions/upload-pages-artifact@v1.0.7` but have not found a way of doing this; `github-pages` environment is always cleared first and only the last thing I upload remains.

Is there a way to either do "partial writes" through a workflow into the `github-pages` environment, or, alternatively, can I download my current environment data, append files to it and then upload the whole thing? Or should I just back off and have my workflow manage the files in a branch?

# Answer
> 0 votes
If you are using GitHub addresses for the page, it naturally creates subfolders for additional repositories. You can only choose one repository to be the root.

Example

* Repo: website `<username>.github.io`
* Repo: blog `<username>.github.io/blog`
* Repo: projects `<username>.github.io/projects`

And so on.

## Custom domains

If you want to use custom domains, you would need to "front door" requests to map each subfolder as I don't think there is way to use the above set up with a custom domain.

Alternatively, you could push the files to a static site hosting location / file storage (i.e. Azure Static Site, Amazon S3, etc). Each push would overwrite files, but you may need a strategy to handle deletions as you're not clearing everything out on each publish.

---
Tags: github, github-actions
---


title: docker-compose healthcheck for rabbitMQ

text:
# Question
Title: docker-compose healthcheck for rabbitMQ
I'm trying to run `rabbitMQ` using `docker-compose`, but the service is always starting or unhealthy.
rabbit is running fine, so I suspect there is something wrong with my health check.

Running the healthcheck command locally does return a value.

```
> curl -f http://localhost:5672
AMQP    %

```

But `docker-compose ps` always says the service is unhealthy (or starting, before it runs out of time).

```
> docker-compose ps
docker-entrypoint.sh rabbi ...   Up (unhealthy)   15671/tcp

```

Here is what my docker-compose.yml file looks like.

```
# docker-compose.yml
version: '2.3' # note: I can't change this version, must be 2.3

volumes:
  rabbit-data:

services:
  rabbit:
    hostname: 'rabbit'
    image: rabbitmq:3.8.5-management
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:5672"]
      interval: 30s
      timeout: 30s
      retries: 3
    ports:
      - '5672:5672'
      - '15672:15672'
    volumes:
      - 'rabbit-data:/var/lib/rabbitmq/mnesia/'
    networks:
      - rabbitmq

networks:
  rabbitmq:
    driver: bridge

```

I have also tried using `nc` instead of `curl` in the healthcheck, but got the same result.

```
healthcheck:
  test: [ "CMD", "nc", "-z", "localhost", "5672" ]

```

From https://github.com/docker-library/rabbitmq/issues/326

# Answer
> 42 votes
You could use the command `rabbitmq-diagnostics -q ping` in case you just need a basic check.

```
healthcheck:
  test: rabbitmq-diagnostics -q ping
  interval: 30s
  timeout: 30s
  retries: 3

```

More information on how to run more advanced health checks could be found here

# Answer
> 8 votes
If `rabbitmq-diagnostics -q ping` not working for you, check this

```
healthcheck:
  test: rabbitmq-diagnostics check_port_connectivity
  interval: 1s
  timeout: 3s
  retries: 30

```

In my example I also reduced interval.

# Answer
> 6 votes
I just want to add to above answers about `rabbitmq-diagnostics`, which worked great, however there was one missing part.

```
    depends_on:
      rabbitmq:
        condition: service_healthy

  rabbitmq:
    image: rabbitmq:3.8.17-management
    healthcheck:
      test: rabbitmq-diagnostics check_port_connectivity
      interval: 30s
      timeout: 30s
      retries: 10

```

With that my dependent service waits exactly what is needed to.

# Answer
> 2 votes
Your solution is good if you do not own / control the RabbitMQ Compose code but still have to check the service presence and your RabbitMQ image contains the management plugin.

First of all you are using curl which sends an HTTP request which on the other hand rabbitmq is not HTTP.

To fix this you can modify you command as follows:

```
# Note the port 15672 is the RabbitMQ Management UI port
# It responds with 200 OK and the HTML page which is acceptable for a healcheck
test: ["CMD", "curl", "-f", "http://localhost:15672"]

```

Note also that `localhost` may not work. Instead you should most likely use the RabbitMQ service name declared externally as host name (e.g. `http://rabbitmq-service:15672`) and ensure your service containing health check is on the same external network with the RabbitMQ server.

As well you have to have `curl` or `netcat` package installed on the Docker image your run the container from. This would require to construct a custom dockerfile.

Other answers show lots of approaches the rabbitmq team suggests in case the RabbitMQ service is defined in your Docker Compose file.

# Answer
> 1 votes
I will go with this, I want to raise issues like *"Publishers will be blocked until this alarm clears"*:

```
    healthcheck:
        test: rabbitmq-diagnostics -q status && rabbitmq-diagnostics -q check_local_alarms
        interval: 60s
        timeout: 30s
        retries: 3

```

---
Tags: docker, docker-compose, rabbitmq
---


# Question
Title: How to get Dockerfile or last CMD/ENTRYPOINT of a kubernetes deployment image?
So I have a deployment like this:

```
apiVersion: v1
kind: Pod
metadata:
  name: yy
spec:
  containers:
    - name: yy
      image: gcr.io/xx/yy
  imagePullSecrets:
    - name: gcr-json-key

```

That `CrashLoopBackOff` on Minikube, I know the issue, that the `ENTRYPOINT` need to be overriden before use, how can I know what's the original `CMD`/`ENTRYPOINT` of that image?

# Answer
If you're running docker, pull the image and inspect it with:

```
docker pull gcr.io/xx/yy
docker image inspect gcr.io/xx/yy

```

Without docker, some registries allow you to explore the images. And there are online tools like https://explore.ggcr.dev (drill down into the config digest).

There are also commands to interact with registries, including crane (from Google), skopeo (from Redhat), and regclient/regctl (from me). Here's the regctl example:

```
$ regctl image inspect ghcr.io/regclient/regsync:latest
{
  "created": "2023-02-26T02:37:17Z",
  "architecture": "amd64",
  "os": "linux",
  "config": {
    "User": "appuser",
    "Env": [
      "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ],
    "Entrypoint": [
      "/regsync"
    ],
    "WorkingDir": "/",
...

```
> 1 votes

# Answer
I found out how:

```
minikube image save gcr.io/xx/yy /tmp/a.tar

```

then open that `/tmp/a.tar` using any decompressor and look for the `.json` file to know the entrypoint or cmd.

Not sure how to do this with real remote kubernetes though.
> 0 votes

---
Tags: docker, kubernetes, minikube
---


# Question
Title: How to use Chef cron resource to run in minute intervals
Is it possible to set cron to run per minute? I have tried two ways below with no luck. I have reviewed the documentation: https://docs.chef.io/resources/cron. The way the documents reads to me is that I can only run cron in a specific minute within a specific hour. Like 8:05. Rather than every single 5 minutes.

```
  cron 'Check-In to Chef Manage every minute' do
      minute '*'
      command 'chef-client'
      action :create
    end

```

```
  cron 'Check-In to Chef Manage every minute' do
      minute '0'
      command 'chef-client'
      action :create
    end

```

# Answer
The correct answer is in the comments by Tensibai. The following worked:

```
cron 'Check-In to Chef Manage every 5 minutes' do
  minute '*/5'
  command 'chef-client'
  action :create
end

```
> 0 votes

---
Tags: chef
---


# Question
Title: Builds triggered by change of arbitrary artifacts?
I am wondering about this:
we are all used to continuous build. I.e. every push of source code to a version control system triggers a build job on a build system (Jenkins, GitLab CI/CD, whatever, ...) that builds the source code. Builds produce artifacts, usually libraries (jar files, dll files, whatever ...) that are pushed to an artifact storage (Nexus, artifactory, GitLab package registry, whatever ...).
Then builds trigger other builds of components that depend on the just created artifacts. This in turn triggers further builds up the software dependency hierarchy.

**Q**: why do only updates of source code trigger builds jobs, but updates of artifacts in an artifact store don't? Is there a specific reason?

At least the environment I'm used to (Jenkins, GitLab CI/CD, Nexus, GitLab package registry) works like that.
**Q**: are there other environments (build automation systems, artifact stores, whatever, ...) that include job triggers by any artifact update?

# Answer
> 0 votes
Just to make sure I understand, you have the following situation...

1. You change the code for a component
2. You commit the code
3. An automated build runs
4. The artifact is published

And you are thinking, why stop there? You have projects that depend on this component, so can't it "cascade" onwards, so the application that depends on the component gets the updated dependency.

This *is* something that is emerging. GitHub has Dependabot, which can submit pull requests for updated dependencies. You need to manually review them, but that's *close* to what you're talking about.

Because Dependabot looks at all your dependencies, it may update interal and external ones. The pull request process helps you review the changes it makes. You can set it up to achieve your desired noise-level (you can run it once a week, for example), so it's not quite "on artifact change".

More tools are starting to appear in this space as software supply chain security is a hot topic.

I'm sure this will concept will develop further.

---
Tags: build-pipeline, builds, artifacts
---