# Question
Title: How can I retrieve a lost login token for KubeApps?
How can I retrieve the root token from a KubeApps deployment?

# Answer
> 0 votes
When you set up a KubeApps deployment you must install a Kubernetes secret, currently the installation instructions detail it like this,

```

kubectl create clusterrolebinding kubeapps-operator --clusterrole=cluster-admin --serviceaccount=default:kubeapps-operator

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Secret
metadata:
  name: kubeapps-operator-token
  namespace: default
  annotations:
    kubernetes.io/service-account.name: kubeapps-operator
type: kubernetes.io/service-account-token
EOF

kubectl get --namespace default secret kubeapps-operator-token -o jsonpath='{.data.token}' -o go-template='{{.data.token | base64decode}}' && echo

```

So basically it's a service-account-token installed in the default namespace. You can list these like this,

```
 kubectl get --namespace default secrets

```

You specifically want the `kubeapps-operator-token` and you can get it with the last line,

```
kubectl get --namespace default secret kubeapps-operator-token -o jsonpath='{.data.token}' -o go-template='{{.data.token | base64decode}}'

```

The  `-o go-template='{{.data.token | base64decode}}'` is basically just piping it to `base64 -d` in Go (like this).

```
kubectl get --namespace default secret kubeapps-operator-token -o jsonpath='{.data.token}' | base64 -d

```

---
Tags: secret-management, authentication, kubeapps
---


# Question
Title: Is it possible to have multiple gitlab-ci files in a single repo?
I have a single repo handling integration tests for 4 different software projects. As those projects share common resources, I'd like to create a **generic file** in which all those common features are set properly for preparing the CI/CD job.
The project organization looks like the following:

```
project_root:
  > folder_1
  > folder_2
  > folder_3
  > folder_4

```

From GitLab's documentation on the `include` keyword, I understand that it is possible to include a file that would handle the common features of all those projects so I don't have to write those command lines in the 4 differents `gitlab-ci.yml`. So I thought I could do the following:

```
project_root:
 common-features-handler.yml
  > folder_1
   gitlab-ci.yml
  > folder_2
   gitlab-ci.yml
  > folder_3
   gitlab-ci.yml
  > folder_4
   gitlab-ci.yml

```

The thing is, the documentation is unclear on whether it is possible to have multiple `gitlab-ci.yml` files in the same repo, regardless of them being in separate folders. It just says that this file should be "placed at the root of \[the\] repository", so I guess this, in itself, excludes the possibility of having several `gitlab-ci.yml` files in the same repo. I tried including the `common-features-handler.yml` file in the `gitlab-ci.yml` from folder\_1 but the job didn't even launched. My guess is that, as it was not at the root of the project, it simply wasn't seen by the CI pipeline. This makes me think that it's not possible for one repo to handle several gitlab-ci files, but I couldn't find anything about it in the doc or online to confirm or infirm this hypothesis.

So is it possible to have multiple gitlab-ci files in a single repo in one way or another? If not, is there a way for avoiding writing the same setup command lines four times?

# Answer
> 14 votes
No, you can't have multiple gitlab-ci files per repository. You can see more information in the following links:

Now, there are some work in progress about this, but it's not available yet.

For example, you can see here that they are working on pipeline which can include other (complete) pipelines.

The `include` keyword is more about reducing the size of your main gitlab-ci file.

# Answer
> 7 votes
You are not able to create multiple `.gitlab-ci.yml` but you can manage to have what you want.

You currently have multiple software in the same repository with the same CI/CD Pipeline or jobs for your softwares.

You can use `include` in order to include local files from your repository, so you would get

```
include:
  - local: 'my_folder/.gitlab-ci.yml'

```

You can reach the generic way you want by using Gitlab Variables and extends.

Basically, you would end up with something like the following:

```
## my_folder/build.yml
# the `.` (dot) before build means that we don't want it run alone, we must
# extends to run this part
.build:
  variables:
    SOFTWARE_ROOT: ""
  script:
    - build ${SOFTWARE_ROOT}

## .gitlab-ci.yml
include:
  - local: 'my_folder/build.yml'

my_first_build:
  extends: .build
  variables:
   SOFTWARE_ROOT: "first_software/"

my_second_build:
  extends: .build
  variables:
    SOFTWARE_ROOT: "second_software/"

```

You can also put everything in the same file.

This way you only declare your build job once for all.

# Answer
> 0 votes
I think „child/parent“ pipelines are what you are looking for: https://docs.gitlab.com/ee/ci/pipelines/pipeline\_architectures.html#child--parent-pipelines

---
Tags: continuous-integration, gitlab, gitlab-ci-runner, yaml
---


# Question
Title: Why does php-fpm show nginx's IP while they are on different containers?
I am trying to set up an Nginx server for the purpose of load balancing between 4 Php-fpm servers. You can access my docker configuration files from this Github repository.

My containers' network configurations are as below:

```
"Containers": {
            "17e57fd1126d99bcd0cba4dde930181b0b240cea54e6d7f9b31bce3dbda416a6": {
                "Name": "my_app_container_3",
                "EndpointID": "d71c4f7965c51f1202c518172f8ef04b8d6304200000c61b8651b8d72b7a78ed",
                "MacAddress": "02:42:ac:12:00:04",
                "IPv4Address": "172.18.0.4/16",
                "IPv6Address": ""
            },
            "256e89ae5bcd4acaed1c621075fa4541af781731300dfc742b1a5f9a719dd87a": {
                "Name": "my_app_container_2",
                "EndpointID": "1491bec10a1f56a146ddb78da186c59bebda7d8bcd2a580414827987ce3d1964",
                "MacAddress": "02:42:ac:12:00:03",
                "IPv4Address": "172.18.0.3/16",
                "IPv6Address": ""
            },
            "814a7eb51e9a8ed7b2231dc6e33cb23f497224228d3cb9abfa36ae1eb8766ae7": {
                "Name": "phpmyadmin_container",
                "EndpointID": "c6db79bd7775e1fc684ffc083d92ffe0d503695333dff5cd265af567a4ce625f",
                "MacAddress": "02:42:ac:12:00:07",
                "IPv4Address": "172.18.0.7/16",
                "IPv6Address": ""
            },
            "83daac9add189960af858bf7a16fc2b6b1592268f10da37ae87babe67c36b9d5": {
                "Name": "container_mysql",
                "EndpointID": "6bb1f812020a445d3210611d66febccb45b49b887e8c1f176310fe7ef01c3d18",
                "MacAddress": "02:42:ac:12:00:02",
                "IPv4Address": "172.18.0.2/16",
                "IPv6Address": ""
            },
            "ac0f65cc4948578a7a056d2a27e4771bccb94ce08da87f262edba423a1d3caa0": {
                "Name": "nginx_container",
                "EndpointID": "4fb3adf6ad5a1f6ad0ee8af816386fe731087230a651140f6b714071e3df74ee",
                "MacAddress": "02:42:ac:12:00:08",
                "IPv4Address": "172.18.0.8/16",
                "IPv6Address": ""
            },
            "cb17b123ff02035f23fcf44895db0dfea8d1337bf3278a71392abd153c0a07a1": {
                "Name": "my_app_container_4",
                "EndpointID": "e03291091e859e08c7c5a7c78ae188e0a3cac117abe8cf351f81fbe8d63d12e0",
                "MacAddress": "02:42:ac:12:00:06",
                "IPv4Address": "172.18.0.6/16",
                "IPv6Address": ""
            },
            "f7fcf98941d12cd5d7fda36f77413d8278e129d5769daa1e6cdc400f342a44b1": {
                "Name": "my_app_container_1",
                "EndpointID": "bc82184499842d089473049dcd0e22bdb8b777bc38e11072f58efdff73407946",
                "MacAddress": "02:42:ac:12:00:05",
                "IPv4Address": "172.18.0.5/16",
                "IPv6Address": ""
            }
        },

```

In my Php file, I have a simple line of code to show the IP address of the server:

```
<h1>This request is handeled by the server working on <?php echo $_SERVER['SERVER_ADDR'] ?></h1>

```

Now that my PHP interpreters are running on different containers, I expect to see different IP addresses whenever I refresh the page. Interestingly though, it always prints out the IP of the Nginx server despite the fact that the code is being interpretted in another container.

Could you explain to me why it is so?

# Answer
The `SERVER_ADDR` is a FastCGI variable that is being set inside the `fastcgi_params` file as

```
fastcgi_param  SERVER_ADDR        $server_addr;

```

The `$server_addr` nginx variable description says it contains an address of the server which accepted a request, which is obviously an nginx server address. If you want to determine an address of upstream server that is chosen to handle the request, you can try to pass an additional FastCGI variable to your PHP-FPM backend using the `$upstream_addr` nginx internal variable value, e.g.

```
fastcgi_param  UPSTREAM_ADDR  $upstream_addr;

```

and check its value via the `$_SERVER['UPSTREAM_ADDR']` array item value (I didn't check if this would work, so if you'd check it, any positive or negative feedback is welcome).

Alternatively you can check this StackOverflow answer.
> 2 votes

---
Tags: docker, networking, nginx, php, ip
---


title: What is DataOps?

text:
# Question
Title: What is DataOps?
This site already has great answers for questions like What is DevOps and what is SecOps.

Over time, I have come to be of the opinion that DevOps was born when the philosophy of the Agile methodology was applied to Operations. Similarly so with SecOps.

What then would be DataOps - applying agile methods to Big Data & Data Analytics look like? How is this similar to agile software development and DevOps and where might it be different?

# Answer
> Over time, I have come to be of the opinion that DevOps was born when the philosophy of the Agile methodology was applied to Operations. Similarly so with SecOps.

Wikipedia indicates that Agile software development is about "early delivery", "adaptive planning" and "rapid and flexible response to change". In order to deliver early, automation is key in DevOps.

> What then would be DataOps - applying agile methods to Big Data & Data Analytics look like?

According to Wikipedia, DataOps tries to incorporate DevOps best practices to enhance data analytics.

> How is this similar to agile software development and DevOps and where might it be different?

**Similar**

* Quick action on change
* Fail fast
* Adaptation
* Tools & platforms: ansible, terraform, AWS, GCP, AKS, EKS

**Differences**

* Focus on BigData
* Knowledge about NoSQL Hadoop, Hbase, Cassandra, MongoDB needed
* How to tune relational database like postgres, mysql that contains large amount of data
* More emphasize on bigdata tools like Spark
> 2 votes

# Answer
The ultimately goals of **agile** and **DevOps** are the same - to improve the speed and quality of software development see the following articles which give a better understanding -

**Atlassian :**DevOps vs Agile

**PhoenixNAP:** Head to head comparison
> -1 votes

---
Tags: terminology, data
---


# Question
Title: Including vault-encrypted variables in a task
I just recently started to use Ansible, and am trying to deploy a configuration file that contains a secret key and email (for credentials to use with an API). I encrypted them with `ansible-vault encrypt_string ...`, and added them to playbook:

```
- name: generate dehydrated dns-hook config file
  template:
    src: etc/dehydrated/dns-hook.sh.j2
    dest: /etc/dehydrated/dns-hook.sh.j2
  vars:
    token: !vault |
              $ANSIBLE_VAULT;1.2;AES256;dns-hook
              66326635666264386238316238373135626233393437633937626534623931636365336231363462
              6131333637663337363162653561393962626661313762300a383633303136633564346366383935
              65333265306162386564363937636531346265633635656536646533393961343935313161643262
              6132336430373235380a313364323438393765613131356535373862306337306434653237316566
              34333965356663626231626533656661663364313737663036343161663638373836366235643730
              3965323463643231666361356330343663653536306634646438
    email: !vault |
              $ANSIBLE_VAULT;1.2;AES256;dns-hook
              64336666396139636636386536373531343133303964646631346135633033356162626562333561
              3338336534333236613330336532353861393265613766380a376632616665366463313237656166
              30633135303431336138303535343962663965363536616139393631626534353432313466633064
              3466363063666663360a343233376333643931323738356435643930623765306537313437356638
              3332396164356236663133613432313063373130653632323432643230646237366

```

However, whenever I run the playbook, I get the following error:

```
TASK [generate dehydrated dns-hook config file] *****************************************************
[WARNING]: There was a vault format error: Vault format unhexlify error: Odd-length string

fatal: [sherlock.server.com]: FAILED! => {"changed": false, "msg": "AnsibleVaultFormatError: Vault format unhexlify error: Odd-length string"}

```

The only thing I can find about this error is that I might have newlines that end in `\r\n` instead of just `\n`, but I've verified that is not the case. Everything else I've seen online seems to imply that this format should work, but I'm unable to get it to run successfully.

# Answer
> 5 votes
Robuster and more practical solution is an encrypted file. For example,

```
shell> cat vault1.yml
my_email: admin@example.com
my_tag: tag123

```

```
shell> ansible-vault encrypt vault1.yml
Encryption successful

```

```
shell> cat vault1.yml
$ANSIBLE_VAULT;1.1;AES256
33376431383930313965356364356136306338383238303032363165633962636366663939373237
3433383262306631643431346236653534316331643466660a326562663633346662656233353733
63373561636432653535666437656537326635363935366261666237353136313939323535336665
3333666466656664610a633430313138393238653065623231393165383162656262646139353730
63653231363465643237666465613631646539366262656537323932346530386364353132326234
3230623232393630396361333462343862323231323733376665

```

Then the encrypted file can be used in various places. For example,

```
shell> cat playbook.yml
- hosts: localhost
  tasks:
    - include_vars: vault1.yml
    - debug:
        msg: "email: {{ my_email }} tag: {{ my_tag }}"

```

gives

```
    "msg": "email: admin@example.com tag: tag123"

```

**Including vault-encrypted variables in a task**

It's possible to limit the scope of the variables to a task. This limits the exposure of the sensitive variables and makes the solution safer. For example, the task below gives the same result.

```
    - debug:
        msg: "email: {{ my_vault.my_email }} tag: {{ my_vault.my_tag }}"
      vars:
        my_vault: "{{ lookup('file', 'vault1.yml')|from_yaml }}"

```

# Answer
> 1 votes
Just ran into the same problem myself. Turns out, the error message is dead on: I made a copy-paste error.

And sure enough, in the OP's case, the variable `email` has an odd number of chars. I think this gets across better when piped through `xxd`:

```
$ cat > odd.txt
64336666396139636636386536373531343133303964646631346135633033356162626562333561
3338336534333236613330336532353861393265613766380a376632616665366463313237656166
30633135303431336138303535343962663965363536616139393631626534353432313466633064
3466363063666663360a343233376333643931323738356435643930623765306537313437356638
3332396164356236663133613432313063373130653632323432643230646237366
$
$ xxd odd.txt | tail -3
00000160: 3331 3330 3633 3337 3331 3330 3635 3336  3130633731306536
00000170: 3332 3332 3334 3332 3634 3332 3330 3634  3232343264323064
00000180: 3632 3337 3336 360a                      6237366.

```

---
Tags: ansible, ansible-vault
---


# Question
Title: How to update nested arrays in mongodb database
I have this mongodb schema:

```
{
    "bsonType": "object",
    "required": [
        "group_wallet_id", "categories",
        "total_donations", "created_at"
    ],
    "properties": {
        "group_wallet_id": {
            "bsonType": "objectId",
            "description": "Id of the parent group"
        },
        "total_donations": {
            "bsonType": "decimal",
            "description": "Total donations"
        },
        "created_at": {
            "bsonType": "date",
            "description": "Wallet creation date"
        },
        "updated_at": {
            "bsonType": "date",
            "description": "Wallet update latest date"
        },
        "categories": {
            "bsonType": "array",
            "items": {
                "bsonType": "object",
                "required": [
                    "category_id", "donor_info",
                    "c_created_at", "c_updated_at"
                ],
                "properties": {
                    "category_id": {
                        "bsonType": "objectId",
                        "description": "Category unique id"
                    },
                    "donors_info": {
                        "bsonType": "array",
                        "items": {
                            "bsonType": "object",
                            "required": [
                                "user_id", "total_balance", "d_created_date",
                                "d_updated_date"
                            ],
                            "properties": {
                                "user_id": {
                                    "bsonType": "objectId",
                                    "description": "User id"
                                },
                                "total_balance": {
                                    "bsonType": "decimal",
                                    "description": "Amount donated"
                                },
                                "d_created_at": {
                                    "bsonType": "date",
                                    "description": "The date the donation was made"
                                },
                                "d_updated_at": {
                                    "bsonType": "date",
                                    "description": "The latest update date"
                                }
                            }

                        }
                    },
                    "amount": {
                        "bsonType": "decimal",
                        "description": "The total amount donated for per category"
                    },
                    "c_created_at": {
                        "bsonType": "date",
                        "description": "The date the donation was made"
                    },
                    "c_updated_at": {
                        "bsonType": "date",
                        "description": "The latest update date"
                    }
                }
            }

        }
    }
}

```

I am trying to update the **categories array** and after that the **donors\_info** nested array. I have this:

```
donor_3 = collection_one.update_one(
                                {"group_wallet_id": ObjectId(data["group_wallet_id"])},
                                {"$set": {"update_at": now}},
                                {"$inc": {"total_donations": data["amount"]}},
                                {"$push": {"categories": {
                                    "category_id": category_id,
                                    "categories.$.donors_info": {
                                        "user_id": ObjectId(user_id),
                                        "total_balance": convert_to_decimal128(data["amount"]),
                                        "d_created_at": now,
                                        "d_updated_at": now
                                    },
                                    "c_created_at": now, "c_updated_at": now,
                                    "amount": convert_to_decimal128(data["amount"]),
                                }}}, session=session)

```

I have tried this and unfortunately I am unable to do this. I using the pymongo lib in my app to access the monogodb atlas instance.

# Answer
I was able to figure out how to update those records. This how the update\_one command should be written:

```
collection_one.update_one(
                           {"group_wallet_id": ObjectId(data["group_wallet_id"])},
                                {
                                    "$set": {"update_at": now},
                                    "$inc": {"total_donations": data["amount"]},
                                    "$push": {"categories": {
                                        "category_id": category_id,
                                        "categories.$.donors_info": {
                                            "user_id": ObjectId(user_id),
                                            "total_balance": convert_to_decimal128(data["amount"]),
                                            "d_created_at": now,
                                            "d_updated_at": now
                                        },
                                        "c_created_at": now, "c_updated_at": now,
                                        "amount": convert_to_decimal128(data["amount"]),
                                    }}}, session=session)

```

All the operations should be wrapped in a **{}** as seen above.
> 0 votes

---
Tags: mongodb
---


# Question
Title: Does GitLab support assigning a reviewer based on the contributor?
My company has the notion of a Senior Tier of developers. These developers are dispersed amongst development teams. Then we assign the Senior Developers from one team to assess the contribution from a different team they were assigned.

Let's say I have a project `foo` and for `foo`, I want

* `SeniorBarGroup` to review PRs to `BazGroup`, and
* `SeniorBazGroup` to review PRs to `QuzGroup`

Is this workflow possible in GitLab?

# Answer
> 0 votes
I don't believe there is any native functionality to help assist in this, and that this relationship will have to be modeled external to GitLab. I've opened this issue for clarity,

https://gitlab.com/gitlab-org/gitlab/-/issues/365887

---
Tags: gitlab, pipeline, code-review
---


# Question
Title: How can I make host docker images available to k8s Deployment?
I have a k8s pod that is running docker. My issue is that for the docker images on my host machine to be available within the pod I either have to run the image as a pod or run a docker pull script at the beginning of my deployment.

Docker's pull API rate limits after some time and these images are not meant to be run without inputs, so I would like to know if there's another way to either expose the host's docker or get access to those images.

I am currently trying to expose the local docker socket, but that does not appear to be provide access to the Deployment from the local docker instance. I'm testing by running `docker images` inside the Deployment and on the host to see if the images match which they don't.

```
volumeMounts:
    - mountPath: /var/run/docker.sock:ro
      name: docker-sock
...
volumes:
    - name: docker-sock
    hostPath:
        path: /var/run/docker.sock

```

I'm also mounting with minikube like so

```
minikube start --memory='5000MB' --mount --mount-string='/:/k8s' --cpus='4'

```

# Answer
As the README describes, you can reuse the Docker daemon from Minikube with `eval $(minikube docker-env)`.

So to use an image without uploading it, you can follow these steps:

1. Set the environment variables with `eval $(minikube docker-env)`
2. Build the image with the Docker daemon of Minikube (eg `docker build -t my-image .`)
3. Set the image in the pod spec like the build tag (eg `my-image`)
4. Set the `imagePullPolicy` to `Never`, otherwise Kubernetes will try to download the image.

**Important note:** You have to run `eval $(minikube docker-env)` on each terminal you want to use, since it only sets the environment variables for the current shell session.
> 1 votes

---
Tags: docker, kubernetes
---


# Question
Title: When and who should set a feature to "done" in Azure DevOps?
We use Azure DevOps with the Scrum template to manage our product backlog. For each change request our customer asks for, we create a new feature with underlying PBIs (product backlog items) and tasks. Our developers set the PBIs to "done" as soon as the PBI passes the pull request.

At the moment we don't have a standard that defines

1. When the feature should be set to "done". That could be

* as soon as all PBIs of the feature are set to "done"
* as soon as the feature is deployed to the production environment
* etc.

2. Who is responsible for changing the status. That could be

* the product owner
* the developer that executes the deployment
* etc.

How do you manage the feature status in your teams?

# Answer
Answers will be somewhat opinion-based, but where I work it's typically the responsibility of the person who raised the feature to close it down. Only they can decide if the development team has met all of the acceptance criteria.

QA may say "we've tested it, we're happy to ship it", but there may be some child items under the feature which are follow-up activities (documentation, customer comms, website updates, support might want to do deployment tests into a cloud platform...) which need to be completed before the feature is "done".
> 0 votes

---
Tags: azure-devops
---


# Question
Title: AWS API Gateway: 403 forbidden with disabled API key; error from cloudfront
How can happen that with a Lambda/API Gateway combination, if no explicit Cloud Front configuration has been done, in an error response header they mention CloudFront?

```
header: x-amzn-ErrorType: ForbiddenException
header: x-amz-apigw-id: P3mmbT_DREF8bg=
header: X-Cache: Error from cloudfront

```

Actual problem: with a disabled API key, API keeps responding "forbidden".

UPDATE. As it seems the problem is not related to the Edge API/CloudFront, as the problem with disabled key persists also on changing API type to regional.

UPDATE. I can now disable API key checking as suggested by @Harish i.e. it's not about disabling the API key itself but telling the API whether it's required or not.

What is yet open though is why if created by Zappa, there is no option to change this setting in the UI, because there are not single methods.

# Answer
> When you deploy an edge-optimized API, API Gateway sets up an Amazon CloudFront distribution and a DNS record to map the API domain name to the CloudFront distribution domain name. Requests for the API are then routed to API Gateway through the mapped CloudFront distribution.

Source: API Gateway documentation — Edge-optimized custom domain names.

---

To allow calls to a method of a resource in your API without API key, set its `API Key Required` setting to `false`:
> 3 votes

# Answer
Besides setting `API Key Required = false`, I was missing something in `API Gateway` `>` `Custom domain names`:

The API Mapping had no stage selected. When I selected `dev`, the endpoint worked.
> 5 votes

---
Tags: amazon-web-services, amazon-cloudfront
---


# Question
Title: bind mount from host not appearing in docker container when using compose
I have a bind mount at /home/quant/mounted\_folder with a fstab like this:

```
/mnt/d/some_folder /home/quant/mounted_folder none bind

```

I can confirm that the folder mounts correctly on the host:

```
DESKTOP-NHBC022@quant: ~/mounted_folder $ ls
foo.txt bar.txt

```

**Reproducing the problem:**

Take this `docker-compose.yml` file:

```
version: "3.3"
services:
  env:
    image: debian:latest
    volumes:
      - "/home/quant:/root/host"

```

The contents of `/root/host/code` appears empty inside the container:

```
DESKTOP-NHBC022@quant: ~ $ docker compose run env bash -c 'ls -al /root/host/code'
total 28
drwxr-xr-x  2 1000 1000  4096 Jun 18 12:28 .
drwxr-xr-x 17 1000 1000 24576 Jun 18 12:56 ..

```

**Workaround:**

If I skip the docker-compose and use just docker run, it seems to work:

```
DESKTOP-NHBC022@quant: ~ $ docker run -v /home/quant/:/root/host debian:latest bash -c 'ls -al /root/host/code'
total 24
drwxrwxrwx  1 1000 1000  4096 Jun 18 08:59 .
drwxr-xr-x 17 1000 1000 24576 Jun 18 13:00 ..
-rwxrwxrwx  1 1000 1000     0 Feb 11  2021 foo.txt
-rwxrwxrwx  1 1000 1000     0 Jun 18 08:59 bar.txt

```

How can I make sure that mounted paths on the host correctly appear inside the docker container if they're subdirectories of paths mounted in the container?

I'm running docker 20.10.1 on windows 10 WSL.

# Answer
> 0 votes
My guess is different defaults to bind mount propagation between `docker run` and `docker compose`. Try setting it to rslave or shared in your compose file:

```
version: "3.3"
services:
  env:
    image: debian:latest
    volumes:
      - "/home/quant:/root/host:rslave"

```

---
Tags: docker, docker-compose
---


# Question
Title: Why are all volumes deleted when 1 EC2 instance is deleted via Terraform?
I am using this template to create AWS resources.

My count is 2, and everything is getting created as per plan, I am attaching a 2nd EBS volume of 1 GB to both machines, which is also happening well, but the only issue is when I try to delete one EC2 instance by below cmd, both of the 2 EBS volume of 1GB are getting destroyed. I checked, and they are connected on separate instances.

```
$ terraform destroy -target=aws_instance.jumpserver[1]
aws_vpc.main_vpc: Refreshing state... (ID: vpc-06b59734024ad6adc)
aws_key_pair.ProdKeypair: Refreshing state... (ID: ProdKeypair)
aws_security_group.sg_internet_facing: Refreshing state... (ID: sg-05a2739733f4f8a32)
aws_subnet.public_subnet: Refreshing state... (ID: subnet-0a8c6ea2718a44224)
aws_instance.jumpserver[1]: Refreshing state... (ID: i-05646d53baa34a988)

An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
  - destroy

Terraform will perform the following actions:

  - aws_ebs_volume.vol_generic_data[0]

  - aws_ebs_volume.vol_generic_data[1]

  - aws_instance.jumpserver[1]

  - aws_volume_attachment.generic_data_vol_att[0]

  - aws_volume_attachment.generic_data_vol_att[1]

```

This is the main.tf

```
# Define webserver inside the public subnets
resource "aws_instance" "jumpserver" {
  count                       = "${var.num_of_instances}"
  ami                         = "${var.ami}"
  instance_type               = "t2.micro"
  key_name                    = "${aws_key_pair.ProdKeypair.id}"
  subnet_id                   = "${aws_subnet.public_subnet.id}"
  vpc_security_group_ids      = ["${aws_security_group.sg_internet_facing.id}"]
  associate_public_ip_address = true
  source_dest_check           = false
  #   user_data = "${file("install.sh")}"

  root_block_device = {
    volume_type           = "gp2"
    volume_size           =  "8"
    delete_on_termination = "${var.delete_on_termincation}"
  }

  tags {
    Name = "${format("jump-%01d",count.index+1)}"
  }

  provisioner "remote-exec" {
    inline = ["sudo apt-get  -y install python"]

    connection {
      type        = "ssh"
      user        = "ubuntu"
      private_key = "${file(var.private_key_path)}"
    }
  }
}

resource "aws_ebs_volume" "vol_generic_data" {
  size              = "1"
  count             = "${var.num_of_instances}"
  type              = "gp2"
  availability_zone = "${element(aws_instance.jumpserver.*.availability_zone, count.index)}"

 tags = {
    Name = "${format("jump-%01d",count.index+1)}"
  }
}

resource "aws_volume_attachment" "generic_data_vol_att" {
  device_name = "/dev/xvdf"
  volume_id   = "${element(aws_ebs_volume.vol_generic_data.*.id, count.index)}"
  instance_id = "${element(aws_instance.jumpserver.*.id, count.index)}"
  count       = "${var.num_of_instances}"
}

# Define webserver inside the private subnet
resource "aws_instance" "backendserver" {
  ami                         = "${var.ami}"
  instance_type               = "t2.micro"
  key_name                    = "${aws_key_pair.ProdKeypair.id}"
  subnet_id                   = "${aws_subnet.private_subnet.id}"
  vpc_security_group_ids      = ["${aws_security_group.sg_backend.id}"]
  associate_public_ip_address = false
  source_dest_check           = false
  user_data                   = "${file("install.sh")}"

  tags {
    Name = "backendserver"
  }
}

```

# Answer
> 2 votes
The issue is with the `availability_zone` on the `aws_ebs_volume` resource. it depends on the `aws_instance`, so when Terraform destroys the EC2, it thinks that your EBS can't live without it. Even though it's an EC2 independent variable.

For more info check a very detailed description: https://github.com/hashicorp/terraform/issues/30614#issuecomment-1058769588

If you have a reference to your VPC, use that. For me it's `module.vpc-prod.azs[0]`

---
Tags: amazon-web-services, terraform
---


# Question
Title: Questions about Ansible connection to remote hosts
I want to ask some questions about working of Ansible.

**1** \- Which shell Ansible uses when logging to remote host ?

https://askubuntu.com/questions/937354/track-logins-made-via-non-interactive-shells/937716?noredirect=1#comment1486867\_937716

**2** \- If I have 10 tasks in a playbook, then ansible makes connection 10 times to remote host OR uses a single connection to perform all these 10 tasks ?

**3** \- If I am running 10 tasks on 2 hosts,

It will do first task on host 1 and then on host 2 then second task on host 1 and then on host 2, .. . so on,

.

So, here also the connection persists to both the hosts?

OR

does it firstly connect to host 1, runs task 1 and then disconnects AND then connects to host 2, runs task 1 and then disconnects ?

# Answer
Can't answer 1 (I always thought it just uses the default /bin/sh unless specified otherwise?).

2. It will make the connection more than 10 times. A single task will typically have an SCP or SFTP connection to copy the taskfile that'll get remotely executed, then another connection to trigger the script. You can monitor this happening by running your playbook with -vvv, such as:

```
ansible-playbook deploy_app.yml -u maplebird -vvv

```

3rd-level verbosity shows all connections to client.
3. By default, Ansible will execute tasks concurrently on all hosts up to the max configured number of forks. So, it'll run task 1 on both hosts, then it'll run task 2 on both hosts, etc. Forks are defined in ansible.cfg, and default to 5. Change this variable to a higher number or comment it out:

```
forks = 5

```

Optionally, you can also do a rolling batch when running playbooks by specifying the `serial` option in the playbook. This will only concurrently execute tasks for however many hosts you've defined.

Say you're running a playbook against 5 hosts, and have the serial option set. With serial = 1, it will run the full playbook 1 host at a time. With serial = 2, it will run hosts 1 & 2, then hosts 2 & 3, then host 5. Example:

```
name: deploy to all webservers
hosts: webservers
serial: 2
roles:
   - deploy_application

```

More reading:

Serial (rolling) playbook runs:

Forks:

Hope this helps.
> 7 votes

# Answer
*(1) Which shell does Ansible use?*

Ansible uses `/bin/sh` by default. On many \*nix systems including RHEL / CentOS, `/bin/sh` is `bash` \- however, on Ubuntu/Debian it's `dash` which is much more basic.

### Making Ansible use bash

It should be possible to change this with the `executable = /bin/bash` config option in ansible.cfg and possibly also setting ansible\_shell\_type in the inventory.

But in practice (on Ansible 2.2.2 and higher) I and others have found that setting `executable` in `ansible.cfg` doesn't work (whether in `become` mode or not).

An approach that works well, but does mean a bit more code per task, is to write this:

```
- shell:
    echo hello from $0
  args:
    executable: /bin/bash

```

This should say `hello from /bin/bash`, indicating the shell is correct.

If you need to use `rvm`, `rbenv`, `pyenv` or similar tools that require a previous `source ~/.bash_profile` (as with an interactive shell), you will need to use a wrapper script or invoke bash within the shell task.
> 3 votes

# Answer
1. Shell module use /bin/sh by default, but you can set executable in global config or locally in playbook.

http://docs.ansible.com/ansible/latest/shell\_module.html
> 2 votes

# Answer
```
/*sudo apt-get update
sudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg -y

curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg REDIRECT /dev/null
** Replace REDIRECT with chevron for greater than (YT doesn't like chevrons!) **

AZ_REPO=$(lsb_release -cs)
echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | sudo tee /etc/apt/sources.list.d/azure-cli.list
sudo apt-get update
sudo apt-get install azure-cli
sudo apt install ansible
sudo apt install python3-pip
pip3 install ansible[azure] --user
az login
az ad sp create-for-rbac --name ansible
az role assignment create --assignee appID --role Contributor
az account show --query '{tenantId:tenantId,subscriptionid:id}';
az ad sp list --display-name ansible --query '{clientId:[0].appId}'
vi ~/.azure/credentials
ansible localhost -m azure_rm_resourcegroup -a "name=test1234 location=uksouth"
ansible localhost -m azure_rm_resourcegroup -a "name=test1234 location=uksouth state=absent"
mkdir azure
cd azure
vi azure_vm.yml
ansible localhost -m azure_rm_resourcegroup -a "name=Testing location=uksouth"
ansible-playbook azure_vm.yml
cat azure_vm.yml
ssh youtubedemo@IP_ADDRESS*/

```
> 0 votes

# Answer
```
Sudo apt-get update
sudo apt-get install ca-certificates curl apt-transport-https lsb-release gnupg -y

curl -sL https://packages.microsoft.com/keys/microsoft.asc | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/microsoft.gpg REDIRECT /dev/null

```

Replace REDIRECT with chevron for greater than (YT doesn't like chevrons!)

```
AZ_REPO=$(lsb_release -cs)

echo "deb [arch=amd64] https://packages.microsoft.com/repos/azure-cli/ $AZ_REPO main" | sudo tee /etc/apt/sources.list.d/azure-cli.list

sudo apt-get update
sudo apt-get install azure-cli
sudo apt install ansible
sudo apt install python3-pip
pip3 install ansible[azure] --user
az login
az ad sp create-for-rbac --name ansible

az role assignment create --assignee appID --role Contributor

az account show --query '{tenantId:tenantId,subscriptionid:id}';

az ad sp list --display-name ansible --query '{clientId:[0].appId}'

vi ~/.azure/credentials

ansible localhost -m azure_rm_resourcegroup -a "name=test1234 location=uksouth"

ansible localhost -m azure_rm_resourcegroup -a "name=test1234 location=uksouth state=absent"

mkdir azure
cd azure
vi azure_vm.yml
ansible localhost -m azure_rm_resourcegroup -a "name=Testing location=uksouth"

ansible-playbook azure_vm.yml

cat azure_vm.yml

ssh youtubedemo@IP_ADDRESS

```

Links:

https://azure.microsoft.com/en-gb/free/?ref=microsoft.com

https://docs.microsoft.com/en-us/azure/developer/ansible/install-on-linux-vm

https://docs.microsoft.com/en-us/cli/azure/install-azure-cli-linux?pivots=apt

https://docs.microsoft.com/en-us/azure/developer/ansible/create-ansible-service-principal

https://docs.ansible.com/ansible/latest/scenario\_guides/guide\_azure.html
> 0 votes

---
Tags: ansible
---


# Question
Title: When OnPrem with Kubernetes, what is the recommended way to do file storage buckets?
When you are running kubernetes on GKS, AKS, etc, you use your cloud provider specific storage bucket such as Amazon S3, google cloud storage, azure blob storage, to store your raw binary content in a super fast, easy to use manner.

But when you are deploying Kubernetes behind the firewall, you do not have a storage bucket SDK from your cloud provider.

What is the alternative to this when you are using, for example, OpenShift? Do we have to just use persistent volumes?

# Answer
You have many, many options. The following all provide an S3-compatible object storage API:

* If you are using OpenShift, you can install OpenShift Data Foundation, which includes the Noobaa object storage service. This provides `ObjectBucket` and `ObjectBucketClaim` resources that are analgous to `PersistentVolume` and `PersistentVolumeClaim` resources.
* You could run MinIO, which can be deployed on Kubernetes via an operator or a helm chart.
* You can run your own Ceph cluster, which includes the object gateway.
* You could run OpenStack Swift.
> 1 votes

---
Tags: kubernetes, amazon-s3, storage, google-cloud-storage
---


# Question
Title: Arguments in docker_compose.yml throwing error, but not with docker run
I am in some desperate help. I am atempting to translate this docker run command:

```
docker run -d -p 5434:5432 --name postgres-2 livingdocs/postgres:14.4 standby -d "host=host.docker.internal port=5433 user=postgres target_session_attrs=read-write"

```

which works flawlessly, but when I attempt to convert it to docker\_compose.yml:

```
  postgres-2:
    image: livingdocs/postgres:14.4
    container_name: postgres-SLAVE
    ports:
      - "5434:5432"
    volumes:
      - postgres2:/tmp/data
    entrypoint: "/scripts/entrypoint standby"
    command:
      - host=host.docker.internal
      - port=5433
      - user=postgres
      - target_session_attrs=read-write"

```

It throws:

```
pg_basebackup: error: too many command-line arguments (first is "host=host.docker.internal")

Try "pg_basebackup --help" for more information.

```

After I save the yml file and execute: `docker-compose up -d --remove-orphans`

I've tried the online conversion tools and about 300 variations, but I cannot get to function the same as docker run.

# Answer
Why in your `docker-compose.yml` are you setting `entrypoint`, when your `docker run` command does not have a `--entrypoint` option? You've also removed the `-d` option from the invocation, and you're passing the remaining options as individual arguments rather than the single string you're using in the `docker run` invocation.

The equivalent to your `docker run` command line would be:

```
  postgres-2:
    image: livingdocs/postgres:14.4
    container_name: postgres-SLAVE
    ports:
      - "5434:5432"
    command:
      - standby
      - -d
      - "host=host.docker.internal port=5433 user=postgres target_session_attrs=read-write"

```

I've removed the `volumes` stanza, since you're not using any volumes on the `docker run` command line.
> 0 votes

# Answer
I got it working with:

```
  postgres-2:
    image: livingdocs/postgres:14.4
    container_name: postgres-SLAVE
    ports:
      - "5434:5432"
    volumes:
      - postgres2:/tmp/data
    entrypoint:
      - standby
    command: "--host host.docker.internal --port 5433 --user postgres"

```
> 0 votes

---
Tags: docker, docker-compose
---


# Question
Title: How to incorporate feedback from pull request
I am new to bitbucket. I tried to google it but unable to find exact answer for my question.

I created a feature branch. Made few changes in the feature branch then I pushed/committed all the changes from my local machine to Bitbucket feature branch on bitbucket.org. Then I created a pull request and added a reviewer.

My senior reviewed the pull request and added his code reviews/comments in the pull request.

Now I am stuck at this point how to incorporate changes which are mentioned by the pull request reviewer. Should I create another branch? I am totally stuck what to do next.

Please help me.

# Answer
> 1 votes
The pull request is from your feature branch (the version on Bitbucket) into some other branch. The most common workflow here is to make the changes on your local feature branch, commit those, and push those commits to Bitbucket. That would cause the pull request to reflect those changes and give the reviewer an opportunity to review them. Depending on team norms and how people have notifications set up, you may also want to leave a comment in the pull request to let reviewers know about the changes.

# Answer
> 0 votes
The next step would be to merge the accepted pull request into your desired target branch. By accepted I mean, that all changes necessary for the pull request to be deemed complete have to be part of your current branch. In case you only performed a local review of the pull request, you should push the changes on your current branch to your Bitbucket Server.

---
Tags: git, bitbucket
---


# Question
Title: How to purge / prevent using the prometheus local storage when using remote_write
I configured Prometheus to push metrics using `remote_write` but the local storage is still in use and keeps growing

How could Prometheus be configured to stop storing locally and only push metrics or to minimize the retention policy from the local storage?

# Answer
Prometheus can run in agent mode. In this mode it doesn't store data locally - it just sends the scraped samples to the configured remote storage systems. See this article for details.

P.S. see also the project I work on - vmagent \- this is lightweight Prometheus-compatible agent, which understands Prometheus scrape configs. It usually requires lower amounts of memory and CPU comparing to Prometheus when scraping the same targets.
> 0 votes

---
Tags: prometheus
---


title: Group authentication in kubernetes

text:
# Question
Title: Group authentication in kubernetes
I am trying to implement group based authentication in k8s with RBAC. My requirement is as below:

There is a linux group named "readonly". There are 3 users reader1, reader2 and reader3 in readonly group.

How can I implement the RBAC to give readonly access to all the users who are part of that group and all the future users who will become part of that group ? Readonly access means, access to list, get and watch resources using kubectl.

I came across .subjects\[0\].kind: Group in clusterrolebinding. But not sure how can that be used or if that can even be used for the above scenario

Thanks.

# Answer
This is something I have implemented recently.

***NOTE: I'm assuming you are aware ho to add users. or users in group in this case.***

here is what I have applied:

```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: readonly_role
rules:
- apiGroups:
  - ""
  resourceNames:
  - ""
  resources:
  - "pods"
  - "pods/logs"
  - "deployments"
  verbs:
  - get
  - list
  - watch
- apiGroups:
  - authorization.k8s.io
  resources:
  - selfsubjectaccessreviews
  - selfsubjectrulesreviews
  verbs:
  - watch
  - get
  - list

---

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: bind_readonly_dev_role
subjects:
- kind: Group
  name: myorg:readonly
  apiGroup: rbac.authorization.k8s.io
roleRef:
  kind: ClusterRole
  name: readonly_role
  apiGroup: rbac.authorization.k8s.io

```

You can use either `ClusterRole` & `ClusterRoleBinding` or else just `Role` & `Rolebinding`, depending on your requirement. In case you are not looking for cluster scoped access, instead just a namespace, I would strongly suggest going with the `Role` & `Rolebinding` kind of object.

**What I'm doing/creating in the above?**

I'm creating one RBAC Role (`readonly_role`) scoped to Cluster. and in that I'm specifying I want to let access `pods`, `logs` (i.e `pods/logs`), and `deployments`. additionally you. can also add things like `services` or else if you just want to give access to all specify `""` instead.

in the 2nd part, I'm creating a ClusterRoleBinding Object named `bind_readonly_dev_role`  which ensures only users in group `myorg:readonly` are having this role or privileges in the role we just created (`readonly_role` ).

NOTE: Incase you want to allow proxy access to certain applications, like `kubectl proxy`. you might have to specify them separately in the `ClusterRole` rules.

I do like this for one of my svc:

```
- apiGroups:
  - ""
  resourceNames:
  - dashboard:5000
  resources:
  - services/proxy
  verbs:
  - '*'

```

at last but not least, you have to test if the user has those privileges correct, you can test like this:

```
kubectl auth  can-i get pods --as myorg:readonly

```

in the above replace the pods with the object you want the users of the group to have access to.

Hope this helps!
> 1 votes

# Answer
The title of the question is "Group authentication in kubernetes" and the original poster asked in the comments "Do you have any link on how to create k8s user group and add user to group". So it could be helpful to answer:

**How do you add kubernetes users into groups?**

Kubernetes doesn't store usernames and groups in its local database, in a way that's analogous to most other systems. Instead, it depends on being provided the 'username' or 'group' by whatever your preferred authentication mechanism is. Depending upon which method you are using, the way to specify the group differs.

That means you first must determine your authentication method. For example, "I'm using X509 Client Certs" or "I'm using a service account token". That answer can't be "Probably I am just using the default built-in kubernetes users and groups" because unfortunately that's something which doesn't exist.

Based on the authentication method, here are alternative ways to set the 'group'.

(Most of this is taken verbatim from https://kubernetes.io/docs/reference/access-authn-authz/authentication/ with some editing and rearranging for clarity).

**X509 Client Certs:**

Place the group names in the certification signing request:

```
openssl req -new -key jbeda.pem -out jbeda-csr.pem -subj "/CN=jbeda/O=app1/O=app2"

```

This would create a CSR for the username "jbeda", belonging to two groups, "app1" and "app2".

**Static Token File:**

The API server reads bearer tokens from a file when given the --token-auth-file=SOMEFILE option on the command line. The token file is a csv file with a minimum of 3 columns: token, user name, user uid, followed by optional group names.

Note: If you have more than one group the column must be double quoted e.g.

```
token,user,uid,"group1,group2,group3"

```

**Bootstrap Tokens:**

The authenticator authenticates as system:bootstrap:Token ID. It is included in the system:bootstrappers group. The naming and groups are intentionally limited.

**OpenID Connect Tokens:**

When configuring the API Server to support OpenID Connect Tokens, add this flag

```
--oidc-groups-claim JWT claim to use as the user's group. If the claim is present it must be an array of strings.   Example: groups Required: No

```

The identity provider would include 'groups' in the token.

**Webhook Token Authentication:**

The API server is configured to communication with a remote authentication service. That external service should return a validation in json format that may include an optional group membership:

```
 # Optional group memberships
      "groups": ["developers", "qa"],

```

**Authenticating Proxy:**

Configure the API server with this flag:

```
--requestheader-group-headers 1.6+. Optional, case-insensitive. "X-Remote-Group" is suggested. Header names to check, in order, for the user's groups. All values in all specified headers are used as group names.

```

Send an http request with X-Remote-Group included in the headers.

```
GET / HTTP/1.1
X-Remote-User: fido
X-Remote-Group: dogs
X-Remote-Group: dachshunds
X-Remote-Extra-Acme.com%2Fproject: some-project
X-Remote-Extra-Scopes: openid
X-Remote-Extra-Scopes: profile

```

**Anonymous requests:**

When enabled, requests that are not rejected by other configured authentication methods are treated as anonymous requests, and given a username of system:anonymous and a group of system:unauthenticated.

**User impersonation:**

A user can act as another user through impersonation headers. The following HTTP headers can be used to performing an impersonation request:

```
Impersonate-User: The username to act as.
Impersonate-Group: A group name to act as. Can be provided multiple times to set multiple groups. Optional. Requires "Impersonate-User".

```

**client-go credential plugins:**

Interacts with an external service (LDAP, Kerberos, OAuth2, SAML, etc.).

During the authentication process, the "external service verifies the signature on the token and returns the user's username and groups."

---

source information from https://kubernetes.io/docs/reference/access-authn-authz/authentication/
> 0 votes

---
Tags: docker, kubernetes
---


# Question
Title: How to set the Virtual Machine name using Azure Image Builder?
According to this manual one can build an image on Azure using Azure Image Builder. When running the first example, the following error is shown in the customization log:

```
PACKER ERR ==> azure-arm: ERROR: -> RequestDisallowedByPolicy :
Resource '1f43d6s8hd4fr5e' was disallowed by policy
"Virtual Machine Naming Policy"

```

The question is how to set the Virtual Machine image name using Azure Image Builder.

**Attempts to solve the issue**

1. Asked the team that is responsible for the policy to remove are change the *Virtual Machine Naming Policy*
2. Tried to change the VM image name by modifying some keys in the json file.
3. Created the following GitHub issue.

# Answer
> 0 votes
Use the following configuration key:

```
.
.
"temp_compute_name": "<desired_name>",
.
.

```

Note: Try to keep name short.

---
Tags: azure, packer, vm
---


# Question
Title: Pass variables form current shell environment to the node app
Please help me somebody with passing variables to node app. Given: node app on ubuntu package.json contains

```
scripts {
    start: node index.js
}

```

The app is started with command npm run start

Problem: the app requires a ton of environment variables the app runs on the server so i am not using .env file variables are exported in the shell so i can verify they are available in the shell before i do npm run start like so echo $MY\_VAR works

Currently in the sell that has all variables exported when i start the app i get from process.env.MY\_VAR is undefined

PS i’d always been using .env file before but now that i want it to run on the server i have no idea what magic is needed to pass variables form current shell environment to the node app.

# Answer
Ok i figured it out. I simply forgot to export the variables that were declared in the current shell.
> 0 votes

---
Tags: node.js, environment-variables
---


# Question
Title: HorizontalPodAutoscaler scales up pods but then terminates them instantly
So I have a `HorizontalPodAutoscaler` set up for my backend (an fpm-server and an Nginx server for a Laravel application).

The problem is that when the `HPA` is under load, it scales up the pods but it terminates them instantly, not even letting them get into the **Running** state.

The metrics are good, the scale-up behavior is as expected the only problem is that the pods get terminated right after scaling.

What could be the problem?

**Edit:** The same `HPA` is used on the frontend and it's working as expected, the problem seems to be only on the backend.

**Edit 2:** I have `Cluster Autoscaler` enabled, it does it's job, nodes get added when they are needed and then cleaned up so it's not an issue about available resources.

**deployment.yaml**

```
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fpm-server
  labels:
    tier: backend
    layer: fpm
spec:
  replicas: {{ .Values.replicaCount }}
  selector:
    matchLabels:
      tier: backend
      layer: fpm
  template:
    metadata:
      labels:
        tier: backend
        layer: fpm
    spec:
      {{- with .Values.imagePullSecrets }}
      imagePullSecrets:
        {{- toYaml . | nindent 8 }}
      {{- end }}
      containers:
        - name: fpm
          image: "{{ .Values.fpm.image.repository }}:{{ .Values.fpm.image.tag }}"
          ports:
            - name: http
              containerPort: 9000
              protocol: TCP
          env:
          {{- range $name, $value := .Values.env }}
          - name: {{ $name }}
            value: "{{ $value }}"
          {{- end }}
          envFrom:
            - secretRef:
                name: backend-secrets
          resources:
            {{- toYaml .Values.resources | nindent 12 }}

```

**hpa.yaml**

```
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
  name: fpm-server-hpa
  labels:
    tier: backend
    layer: fpm
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: fpm-server
  minReplicas: {{ .Values.autoscaling.minReplicas }}
  maxReplicas: {{ .Values.autoscaling.maxReplicas }}
  metrics:
    {{- if .Values.autoscaling.targetCPUUtilizationPercentage }}
    - type: Resource
      resource:
        name: cpu
        targetAverageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }}
    {{- end }}
    {{- if .Values.autoscaling.targetMemoryUtilizationPercentage }}
    - type: Resource
      resource:
        name: memory
        targetAverageUtilization: {{ .Values.autoscaling.targetMemoryUtilizationPercentage }}
    {{- end }}

```

# Answer
Seems that the problem was with the `replicas: {{ .Values.replicaCount }}` definition. It seems that, if you are using HPA, `replicas` can't be used. I removed this line and the HPA started scaling.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Does an AWS service automatically assume a needed IAM role?
Suppose I create a role with `AssumeRolePolicyDocument` allowing an AWS service (e.g. `s3.amazonaws.com`) to assume the role - do I need to in some way tell the service to assume the role, or will it automatically assume the role if it needs a permission that the role grants?

The motivation for this question is S3 Inventory, where according to the docs the S3 principal is what's accessing resources: https://docs.aws.amazon.com/AmazonS3/latest/userguide/configure-inventory.html

# Answer
> 1 votes
I think you may have this a little backwards.

S3 is a service that you may want to access with a role. S3 would not be accessing anything, things access s3.

So, if you made IAM role `ABC`, you could set up policy to allow it to list and write to a specific S3 bucket, for example.

You could also make role `ABC` assume-able by another role. In that case, you may have a server with default IAM role (instance profile) `XYZ` and the assume role policy can state that `XYZ` can assume `ABC`, which would then let it access S3.

Any entity is just one role at a time. So, once your server assumed `ABC` from `XYZ`, it effectively is just `ABC`.

You can also assume roles from IAM users - but IAM users are generally bad practice as they are very losable.

Assuming a role is a very explicit operation, you have to do it on purpose. Some programs may "seem" like they do it easily by configuration, but in reality if they are assuming a role, they are running very specific code for it, similarly to what you would do to assume the role from a command line.

## Example Tutorial

Here is a decent AWS tutorial for assuming roles in a CLI to get familiar with it.

https://aws.amazon.com/premiumsupport/knowledge-center/iam-assume-role-cli/

The final command looks like this:

```
aws sts assume-role --role-arn \
"arn:aws:iam::123456789012:role/example-role" \
--role-session-name AWSCLI-Session

```

But that assumes role `arn:aws:iam::123456789012:role/example-role` is set up to allow your current role (whatever it is) to assume it. Note that role assumption within an account only requires the target role to allow it, but role assumption between two AWS accounts requires the source and target role to be set upt o allow it.

# Answer
> 0 votes
There is a variety of ways how different services access an S3 bucket. What John Humphreys answer describes is the scenario where the access is performed by some user controlled code running in the same AWS account. Some other access patterns include:

* Delivering content through Cloudfront: There is something called an "Origin Access Identity" - a sort of Cloudfront specific role that can access the S3 bucket. This role cannot be assigned any policies itself, so you have to use the S3 bucket policy as an overlapping security layer to allow the policy access to the bucket (on top of any other access that exists). An additional example would be a certain Wordpress plugin I have worked with that in the past required the S3 bucket and objects to be public. In this case the policy would have to be configured to be "nothing but the Origin Access Identity"
* You could imagine a SaaS service that delivers something to your bucket, but without wanting to be restricted to how they do that. They may use EC2 now, but want to be free to deliver the data later by using Lambda or even S3 bucket replication. In this case, you would set up a policy that allows anything in the other AWS account to have write access to your bucket.
* Some AWS Services have a default named role, for example if you use EC2 Lifecycle Manager, a role is created in your account called AWSDataLifecycleManagerDefaultRole. For this role to use a KMS key, the key needs to have a policy to allow access for this role.
* While I have not used S3 inventory myself, if you look at the example policy linked from the service description: https://docs.aws.amazon.com/AmazonS3/latest/userguide/example-bucket-policies.html#example-bucket-policies-use-case-9

```
  {
        "Version": "2012-10-17",
        "Statement": [
          {
              "Sid": "InventoryAndAnalyticsExamplePolicy",
              "Effect": "Allow",
              "Principal": {
                  "Service": "s3.amazonaws.com"
              },
              "Action": "s3:PutObject",
              "Resource": [
                  "arn:aws:s3:::destinationbucket/*"
              ],
              "Condition": {
                  "ArnLike": {
                      "aws:SourceArn": "arn:aws:s3:::sourcebucket"
                  },
                  "StringEquals": {
                      "aws:SourceAccount": "111122223333",
                      "s3:x-amz-acl": "bucket-owner-full-control"
                  }
              }
          }
      ]
  }

```

In this case the policy allows the S3 service itself (in some undefined capacity) to write to your target bucket and it could even be an inventory of an S3 bucket in another account.

So, the S3 service does not "pick the right role", it operates without a specific(ally known) role.

---
Tags: aws-iam
---


# Question
Title: How to list all users with specific role in GCP
`gcloud projects get-iam-policy [PROJECT-ID]` lists all users with their roles for specific project. There are different filters and formatters available but I can't seem to find the right way to just filter only by specific role.

```
$ gcloud projects get-iam-policy MY_PROJECT

bindings:
- members:
  - serviceAccount:12345678-compute@developer.gserviceaccount.com
  - user:alice@foobar.com
  role: roles/editor
- members:
  - user:you@yourdomain.com
  - user:someoneelse@yourdomain.com
  role: roles/owner
etag: ARBITRARY_ETAG_HERE
version: 1

```

What's the `--filter=` expression that I need to use to list only users with the role `roles/owner`?

# Answer
That can be achieved using another `gcloud` command:

```
gcloud beta asset search-all-iam-policies --query policy:"roles/owner" --project $your_project_id --flatten="results[].policy[]" --format="csv(bindings.members[0])"

```
> 3 votes

# Answer
Ran into this requirement recently as well. You can achieve this without the use of the `assets` API.

```
$ gcloud projects get-iam-policy $PROJECT --flatten="bindings[].members" --filter="bindings.role:roles/owner" --format='table(bindings.members)'

```
> 2 votes

---
Tags: google-cloud-platform, gcp
---


# Question
Title: Rebooting node at end of Jenkins pipeline
I have a pipeline that runs on a dedicated bare metal node for automated performance benchmarking. Using bare metal is necessary due to the nature of the project.

Near the end of the pipeline, the results are sent to users (simply seeing a pass or fail status is not sufficient).

To provide each instance of the pipeline a clean slate, the node is rebooted at the end of the pipeline. This makes the pipeline appear stuck in Jenkins. Several minutes after Jenkins is able to reconnect to the node, Jenkins realizes the pipeline that was running ended abruptly, and marks it failed:

```
+ sudo reboot
Cannot contact baremetal: hudson.remoting.ChannelClosedException: Channel "hudson.remoting.Channel@35ca6b21: baremetal": Remote call on baremetal failed. The channel is closing down or has closed down
wrapper script does not seem to be touching the log file in /home/jenkins/workspace/bare-metal-benchmarks@tmp/durable-e96e9611
(JENKINS-48300: if on an extremely laggy filesystem, consider -Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=86400)
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code -1
Setting status of *** to FAILURE with url *** and message: ' '
Using context: Bare Metal Benchmarks
Finished: FAILURE

```

The next instance of this pipeline can then start running on the rebooted bare metal node.

Is there a way to safely reboot a node at the end of a pipeline so Jenkins does not mark the pipeline status as failed?

The basic structure of the scripted pipeline is:

```
node('baremetal') {
    checkout scm

    try {
        stage('Benchmarks') {
            // run benchmarks
        }

        stage('Process data') {
            // process data and send results to users
        }
    }
    finally {
        stage('Clean up') {
            sh 'sudo reboot'
        }
    }
}

```

# Answer
> 1 votes
Quick answer:

Changing `sudo reboot` to `sudo shutdown -r +m5` or what ever time delay needed, would prevent the pipeline from failing. It forks the reboot process to the system and lets the Jenkins agent finish this pipeline cleanly.

Longer (preventing potential conflicts answer):

Depending on how the system is configured, it should prevent new logins to the server while the reboot is scheduled.

If the agents are configured to launch via ssh,

and configured to be brought offline if not in use,

Provided the agent times out, Jenkins should be unable to start a new pipeline on that agent until the reboot completes.

From the man page for shutdown (on one of my servers)

```
If the time argument is used, 5 minutes before the system goes down the
/run/nologin file is created to ensure that further logins shall not be
allowed.

```

# Answer
> 0 votes
Some ideas to get you started:

* Create a separate job to shut down the node and call that with a build step from your pipeline with "wait: false", e.g.: `build(job: "shutdown_host", wait: false)`. The "shutdown\_host" job would probably need to have a sleep statement or something before shutting down the node to avoid killing the node before the parent job has a chance to exit.
* Execute a short shell script that forks & detaches a new process to shut down the host. Again you would probably need a sleep statement in the forked process to avoid shutting down the node before the Jenkins job exits. Also, in this case, you would need to disable the Process Tree Killer to avoid the forked process being killed.
* Execute a step on the Jenkins master to ssh into your node to shut it down. But this would depend on the Jenkins master having ssh access to your node and administrative privileges to shut it down.

I'm sure there are some other clever and creative ideas that I'm missing as well.

---
Tags: jenkins, jenkins-pipeline, linux
---


# Question
Title: AWS- How to estimate a server configuration for nginx load balancer?
What kind of parameters can be used to estimate how much memory and the number of CPU cores are required for a load balancer?
I have a requirement to setup LB based on Nginx to route the traffic to other nodes via TCP/IP as on other nodes that have PHP-FPM installed. SSL will terminate at Nginx and traffic to nodes is to be based on just HTTP. Max timeout to be five minutes.

I am unable to understand how many requests this Nginx LB can swiftly serve while running on a c6gn.medium (1vCPU,2GB Memory, network bandwidth up to 25GBPS) which is CPU optimized and a t4g.medium which is a burstable instance having 2 vCPU and 4GB memory. Cheers!

# Answer
> 0 votes
There are 2 ways I can think of to solve this problem:

* You can use your own data points from your own experience or others: One datapoint I can provide is ~70,000 sessions per month, probably 50 connections per page request, 1MB page size and maybe 1.5-2 pages per session was running on Nginx 1.14 on a t3.medium. I don't recall ever getting charged for CPU credits, so the CPU usage can't have been ever above 20% over 24h. I do recall upgrading from t3.small at some point, so it must have been memory usage. ~200 vhosts and their TLS certificates.
* You can use Route53 weighted routing and distribute the traffic to say a c6gn.xlarge (to be super save) and a t4g.medium and then slowly dial up the percentages and watch CPU, memory, network and disk throughput until you notice bottlenecks.

---
Tags: amazon-web-services, amazon-ec2, nginx, load-balance, loadbalancing
---


# Question
Title: What does "Does not have minimum availability" in k8s mean?
What does "Does not have minimum availability" mean?

A GitHub discussion was found, but it is not clear to me what the error message means.

# Answer
> 27 votes
As @Tensibai indicated in one the comments, this could be caused as there is insufficient CPU or memory, but that is not always the case.

For example, a helm chart was just deployed, it failed and the workload in GCP indicated that:

> Pod errors: CrashLoopBackOff

Based on the comment of @Tensibai the first impression was that there were insufficient resources, but further analysis using `kubectl describe pod <pod-name>` indicated that in this case the livenessProbe check failed:

```
Liveness probe failed: Get http://10.16.0.13:80/: dial
tcp 10.16.0.13:80: getsockopt: connection refused

```

In summary, the `Does not have minimum availability` message is generic. Multiple issues could trigger this and more in dept analysis is required to find the actual error.

# Answer
> 4 votes
Just want to throw out an example of it not being the CPU / mem.

I remade a cluster, which included a new node pool, nodes, etc. Then tried re-deploying some of the pods that were on the previous cluster. 3 of them were working, and one kept showing "Does not have minimum availability".

It turns out, the pod in question's deployment file had `spec.template.spec.nodeName` hard-coded to one of the nodes from the previous cluster. In the new cluster, that node does not exist, so it failed.

Review your k8s deployment file (.yaml or .json) if you are running into this issue and the other solutions don't work. Maybe even service files as well.

# Answer
> 3 votes
I have faced a similar error for my Gitlab runner deployment on the GKE cluster. As you can see from the error, something is wrong with our pods. Why shouldn't we go and check the pod status?

```
kubectl describe pods -n namespace

```

You will get a fair idea about what's the real issue behind the error “*Does not have minimum availability*”.

In my case, the node was unable to pull the image for my deployment from the google container registry (GCR). I checked the permission for the service account used for k8s-cluster to pull the images from GCR.

I provided the correct roles and then it worked. Roles:

* storage
* object
* viewer

# Answer
> 1 votes
If your persistent volumes aren't creating, it'll cause this error. So add DISK to CPU/Mem/NodeSpec requirements.

# Answer
> 0 votes
Chinmay Mandal answer is legend. Basically this saved me a lot of time `kubectl describe pods -n namespace`. Just read through the file and in my case I had the k8s "taints" wrong, and for some reason was impacting on how the pods gather resources from node.

# Answer
> 0 votes
I installed a helm chart ok. Then uninstalled it and installed it again. Then I got this error on some nodes. For me the solution was to delete the cluster and recreated it. It's not a great solution but could be worth a try.

---
Tags: kubernetes
---


# Question
Title: Why does the Rancher Security Group use TCP Port 10256?
What does TCP 10256 do and where is to documented. According to the *"Rancher AWS EC2 Security Grouplink"* it's open. But I don't see any documentation on port 10256 elsewhere.

It's also documented on the similar *"Azure machine config"*

# Answer
According to this page it is used for `kubeproxy`.

> The Kubernetes network proxy runs on each node. This reflects services as defined in the Kubernetes API on each node and can do simple TCP, UDP, and SCTP stream forwarding or round robin TCP, UDP, and SCTP forwarding across a set of backends. Service cluster IPs and ports are currently found through Docker-links-compatible environment variables specifying ports opened by the service proxy. There is an optional addon that provides cluster DNS for these cluster IPs. The user must create a service with the apiserver API to configure the proxy.
>
> \[...\]
>
> `--healthz-bind-address ipport Default: 0.0.0.0:10256`
>
> The IP address with port for the health check server to serve on (set to '0.0.0.0:10256' for all IPv4 interfaces and '\[::\]:10256' for all IPv6 interfaces). Set empty to disable. This parameter is ignored if a config file is specified by --config.

**References**
kube-proxy
> 1 votes

---
Tags: amazon-web-services, rancher, ports
---


# Question
Title: AWS-Terraform VPC difference between aws_route_table and aws_route
I'm struggling to understand the difference between the following Terraform Resources:

Example 1:

```
resource "aws_route_table" "public_1" {
  vpc_id = aws_vpc.test.id
  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.main.id
  }
  tags = {
    Name = "test-route"
  }
}
resource "aws_route_table_association" "test_pub_route" {
  subnet_id      = aws_subnet.test_pub.id
  route_table_id = aws_route_table.public_1.id
}

```

Correct me if I'm wrong but in this example I'm just creating a route and a route table with `public_1` resource.
In the second part I'm just associating the public\_1 route table subnet called test\_pub.

**Question 1:** I don't understand what does the block route do in `public_1` The documentation here doesn't make much sense to me.

Example 2:

```
resource "aws_route_table" "public" {
  vpc_id = aws_vpc.main.id
}
resource "aws_route" "public" {
  route_table_id         = aws_route_table.public.id
  destination_cidr_block = "0.0.0.0/0"
  gateway_id             = aws_internet_gateway.main.id
}
resource "aws_route_table_association" "public" {
  subnet_id      = aws_subnet.test_pub.id
  route_table_id = aws_route_table.public.id
}

```

**Question 2(main)** I'm trying to understand if these two examples are actually the same. And that the only difference is that in the first example we have route block which is equivalent to resource aws\_route in the second example.
Even the required variables are the same just name is little bit different.

# Answer
> 2 votes
These are two ways of accomplishing the same thing with limitations.

Note that you cannot use both methods for the same table. In other words if you create routes in the `aws_route_table` you cannot associate routes created with `aws_route`

> Terraform currently provides both a standalone Route resource and a Route Table resource with routes defined in-line. At this time you cannot use a Route Table with in-line routes in conjunction with any Route resources. Doing so will cause a conflict of rule settings and will overwrite rules.

Q1. The route block in `aws_route_table` is creating a route inside the routing table without the need to create a separate route using `aws_route`.

Q2. Yes, they accomplish the same result.

---
Tags: amazon-web-services, terraform, aws-vpc
---


# Question
Title: Newly installed k3s cluster on fresh OS install can not resolve external domains or connect to external resources?
I'm following the k3s tutorial for troubleshooting DNS, after step 2 *"Add `KUBECONFIG` for user."*, if run this command,

```
kubectl run -it --rm --restart=Never busybox --image=busybox:1.28 -- nslookup www.google.com

```

I get this error,

```
nslookup: can't resolve 'www.google.com'
pod "busybox" deleted
pod default/busybox terminated (Error)

```

However, I'm running k3s. It's a single node cluster, and on the same machine that k3s is installed I can run `nslookup www.google.com`, and everything works. The tutorial doesn't say where to go from there? What could cause DNS failures for external resolution inside of k3s, but not ousdie of k3s?

My core DNS logs show,

```
[ERROR] plugin/errors: 2 google.com. AAAA: read udp 10.42.0.6:40115->1.1.1.1:53: i/o timeout
[ERROR] plugin/errors: 2 google.com. A: read udp 10.42.0.6:54589->1.1.1.1:53: i/o timeout

```

And when I run curl on an external server, I get

> command terminated with exit code 6

While this was the first symptom for me, it turns out that I also can't ping or curl/wget external websites by IP. For these reasons I think the problem is even more complex, and perhaps involves IP tables.

I uploaded my `iptables -L` information here

# Answer
> 0 votes
My problem was basically that I had multiple default routes, from `ip route`

```
default via 172.16.42.1 dev ens5 proto dhcp src 172.16.42.135 metric 100
default via 172.16.42.1 dev ens3 proto dhcp src 172.16.42.95 metric 100
default via 10.2.64.1 dev ens4 proto dhcp src 10.2.67.51 metric 100
10.2.64.0/19 dev ens4 proto kernel scope link src 10.2.67.51
169.254.169.254 via 172.16.42.2 dev ens5 proto dhcp src 172.16.42.135 metric 100
169.254.169.254 via 172.16.42.2 dev ens3 proto dhcp src 172.16.42.95 metric 100
169.254.169.254 via 10.2.64.11 dev ens4 proto dhcp src 10.2.67.51 metric 100
172.16.42.0/24 dev ens5 proto kernel scope link src 172.16.42.135
172.16.42.0/24 dev ens3 proto kernel scope link src 172.16.42.95

```

The cause of this was not using `no_gateway = true` in my Terraform stanza,

```
resource "openstack_networking_subnet_v2" "subnet_project" {
  name       = "subnet_project"
  network_id = openstack_networking_network_v2.net_project.id
  cidr       = "172.16.42.0/24"
  ip_version = 4
}

```

Without `no_gateway = true`. I can fix this by lowering the metric on the default route on the host,

```
sudo ip route replace default via 10.2.64.1 dev ens4 metric 90

```

Which will add a new route to this,

```
default via 10.2.64.1 dev ens4 metric 90

```

Now running `nslookup google.com` as in the question will work fine, I can re-break it with

```
sudo ip route del default via 10.2.64.1 dev ens4 metric 90

```

---

# Other diagnostics

These are taken before bringing up k3s. My `ip -o addr` shows,

```
1: lo    inet 127.0.0.1/8 scope host lo\       valid_lft forever preferred_lft forever
1: lo    inet6 ::1/128 scope host \       valid_lft forever preferred_lft forever
2: ens3    inet 172.16.42.95/24 brd 172.16.42.255 scope global dynamic ens3\       valid_lft 40534sec preferred_lft 40534sec
2: ens3    inet6 fe80::f816:3eff:fecd:6722/64 scope link \       valid_lft forever preferred_lft forever
3: ens4    inet 10.2.67.51/19 brd 10.2.95.255 scope global dynamic ens4\       valid_lft 24333sec preferred_lft 24333sec
3: ens4    inet6 2620:0:28a4:4140:f816:3eff:fed2:72c7/64 scope global dynamic mngtmpaddr noprefixroute \       valid_lft 2591979sec preferred_lft 604779sec
3: ens4    inet6 fe80::f816:3eff:fed2:72c7/64 scope link \       valid_lft forever preferred_lft forever
4: ens5    inet 172.16.42.135/24 brd 172.16.42.255 scope global dynamic ens5\       valid_lft 40534sec preferred_lft 40534sec
4: ens5    inet6 fe80::f816:3eff:fe3d:cde8/64 scope link \       valid_lft forever preferred_lft forever

```

And `ip link` shows

```
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:cd:67:22 brd ff:ff:ff:ff:ff:ff
3: ens4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:d2:72:c7 brd ff:ff:ff:ff:ff:ff
4: ens5: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1492 qdisc fq_codel state UP mode DEFAULT group default qlen 1000
    link/ether fa:16:3e:3d:cd:e8 brd ff:ff:ff:ff:ff:ff

```

---
Tags: dns, k3s
---


# Question
Title: Azure DevOps: How to pass a parameter from a build to a release?
I have an azure devops build pipeline that builds some docker images and a release pipeline that deploys the docker images using some artifacts I publish in the build.

In the build I have some parameters that allow the user to change which docker images are built. Defined like this:

```
parameters:
  - name: docker1
    displayName: Build docker image 1?
    type: boolean
    default: true
  - name: docker2
    displayName: Build docker image 2?
    type: boolean
    default: true

```

I now want to make jobs in my release stages dependent on these variables as well. So something like this:

Now the config in this picture doesn't work.

My question is how does the syntax for this work if this is just a problem of syntax or how do I make these parameters available to the release if this is more of an issue of variable scope?

What I've tried:

* Just accessing the parameters with the syntax shown above `eq('${{ parameters.docker1 }}', true)`
* Specifying the Build like one would with predefined variables of the build scope `eq('${{ Build.parameters.docker1 }}', true)`
* looked at variable groups vaguely based on this post Azure DevOps Build and Release pipeline artifact name variable . Variable groups seem to be immutable from the build pipeline and also shared between multiple potentially simultaneous builds so this doesn't seem to work for this usecase.

I can image that one way to solve this would probably be to write all my parameters to a file, upload this as a pipeline artifact and then write variables from this imported artifact in the release stage. I want to avoid this if possible since it feels like a very clunky workaround.

# Answer
I found another solution that works even without text files just by using the Azure REST API to get the build parameters.

In my release pipeline I execute this Powershell tasks first to extract any build pipeline parameters:

```
function Get-ObjectMember {
  [CmdletBinding()]
  Param(
      [Parameter(Mandatory=$True, ValueFromPipeline=$True)]
      [PSCustomObject]$obj
  )
  $obj | Get-Member -MemberType NoteProperty | ForEach-Object {
      $key = $_.Name
      [PSCustomObject]@{Key = $key; Value = $obj."$key"}
  }
}

$url = "https://dev.azure.com/$(account)/$(project)/_apis/build/builds/$(Build.BuildId)?api-version=6.0"
$build = Invoke-RestMethod -Uri $url -Headers @{
    Authorization = "Bearer $env:SYSTEM_ACCESSTOKEN"
}
$params = $build.templateParameters
if ($params) {
  $params | Get-ObjectMember | foreach {
    Write-Host $_.Key : $_.Value
    echo "##vso[task.setvariable variable=$($_.Key);]$($_.Value)"
  }
}

```

My build pipeline contained the parameter: `RELEASE_VERSION` and after I executed the above code, I can use `$(RELEASE_VERSION)` in my other release tasks.
> 1 votes

# Answer
In case anyone ever finds this question: I didn't find a satisfying answer. My workaround was to write my variables to a text file in the build and then publish it, download it in the deployment and publish the values as output variables to a bash task. Looks something like this:

```
source file_containing_your_vars.txt
echo "##vso[task.setvariable variable=var1;isoutput=true]$var1"

```

the output variables parameters of this bash would need some reference name, for example job\_out and you would then be able to reference the variables from other jobs in that stage like this:

```
variables['job_out.var1']

```

If you are running a yaml deployment pipeline (instead of one made in the portal) you can even use job output variable between stages.

For portal pipelines that want to share variables between stages this exists and seems to have helped some people but I didn't get it to work: https://stackoverflow.com/questions/53042605/how-to-modify-azure-devops-release-definition-variable-from-a-release-task/53060141#53060141
> 1 votes

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: Why is my docker composer volume not working?
My volume isn't being created with the containers, this volume binds with the host, every time I try to attempt to run it or go to an online validator for yaml, it shows the correct validation on the validator but fails to run:

> ERROR: In file './docker-compose.yaml', service 'image' must be a mapping not a string.

and I can't understand what's causing this when the online validator says it's a valid yaml.

```
version: "3.7"
services:
 image:  postgres
 environment:
   - POSTGRES_DB=postgres
   - POSTGRES_USER=postgres
   - POSTGRES_PASSWORD=postgres

 build: .
 command: python manage.py runserver 0.0.0.0:8000
 volumes:
   - .:/app
 ports:
   - "8000:8000"
 depends_on:
   - db

```

# Answer
> 1 votes
You need to specify the container name (which is a variable, called CONTAINER\_NAME here), you might want to call it postgres\_server or something like that:

```
version: "3.7"
services:
  CONTAINER_NAME:
   image:  postgres
   environment:
     - POSTGRES_DB=postgres
     - POSTGRES_USER=postgres
     - POSTGRES_PASSWORD=postgres

   build: .
   command: python manage.py runserver 0.0.0.0:8000
   volumes:
     - .:/app
   ports:
     - "8000:8000"
   depends_on:
     - db

```

source : https://stackoverflow.com/questions/56733841/service-image-must-be-a-mapping-not-a-string

---
Tags: docker-compose
---


# Question
Title: Switching to multi-part cloud-init, getting: SyntaxError: invalid syntax
I used to have

```
user_data  = data.template_file.user_data.rendered

```

I changed that to,

```
user_data  = data.template_cloudinit_config.master.rendered

```

And I added,

```
data "template_cloudinit_config" "master" {
  gzip          = true
  base64_encode = true

  # get common user_data
  part {
    filename     = "cloud-init.cfg"
    content_type = "text/part-handler"
    content      = data.template_file.user_data.rendered
  }
}

```

But now I'm getting this error, in my `cloud-init.log`

```
2022-07-12 15:19:21,514 - util.py[DEBUG]: Writing to /var/lib/cloud/instances/1a6e9475-5d46-4b3f-b7bc-642055614e22/handlers/part-handler-000.py - wb: [600] 16701 bytes
2022-07-12 15:19:21,514 - util.py[WARNING]: Failed at registering python file: /var/lib/cloud/instances/1a6e9475-5d46-4b3f-b7bc-642055614e22/handlers/part-handler-000.py (part handler 0)
2022-07-12 15:19:21,514 - util.py[DEBUG]: Failed at registering python file: /var/lib/cloud/instances/1a6e9475-5d46-4b3f-b7bc-642055614e22/handlers/part-handler-000.py (part handler 0)
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/cloudinit/handlers/__init__.py", line 144, in walker_handle_handler
    mod = fixup_handler(importer.import_module(modname))
  File "/usr/lib/python3/dist-packages/cloudinit/importer.py", line 15, in import_module
    __import__(module_name)
  File "/var/lib/cloud/instances/1a6e9475-5d46-4b3f-b7bc-642055614e22/handlers/part-handler-000.py", line 3
    users:
         ^
SyntaxError: invalid syntax

```

However, nothing changed in that file. What's going on?

# Answer
> 0 votes
The problem here was I had

```
content_type = "text/part-handler"

```

I should have had,

```
content_type = "text/cloud-config"

```

---

For more information:

---
Tags: terraform, cloud-init
---


title: GitLab "Groups" for permissions only?

text:
# Question
Title: GitLab "Groups" for permissions only?
I have a many repos all of which can be edited by a different "Senior Review Group" group. I've detailed this idea here. Should I use the GitLab Group functionality to assign reviewers are say who can merge? Or should I create external tooling around this?

A big concern I have is that a Group in GitLab seems to be focused on Development. But for example a "Senior Review Group" will never produce code.

# Answer
I have followed this pattern in the past for LDAP synchronisation, where a "permission" group was synched with LDAP and then invited into a "projects" group with a specific role.

This used the Gitlab ldap group synch feature.

There are a few benefits over adding users directly:

1. you can manage identities exclusively in the IdP (*e.g.* Active Directory) - onboarding/offboarding/privilege escalation all done from the IdP.
2. You make simple declarative statements about which (user) groups need to be in which (project) groups, with given roles.
3. This makes it easier to think about the authn/authz model and indeed makes it feasible to write declarative infrastructure as code for the configuration of the projects themselves -- which group can merge, which branches are protected, *etc*.

On the topic of a "senior review" group which will never produce code - I agree this is a tricky permission to model. You want someone who can approve merge requests and perhaps merge them (thereby effectively making a commit and therefore "producing code" however). However, it's called privilege *escalation*, when what you want is true role-based authorization. I'm not sure you can do that yet with Gitlab.
> 1 votes

---
Tags: gitlab
---


title: Azure Web Apps Serves Old Files

text:
# Question
Title: Azure Web Apps Serves Old Files
I am using an Azure DevOps release pipeline to deploy code to an Azure Web App. It pulls my .war file from my most recent build pipeline to deploy. However, it serves old code. For example, the app has referenced an old application.properties file, in the past. It also will reference previously deleted files at runtime, even when the code has no reference to those files.I have attempted to restart, stop and start the app, to no avail. It’s running on a Tomcat 9.0 server.

In my current position, I'm limited to using TFVC so resorted to using the DevOps pipeline UI, instead of yaml. For my release pipeline, I use the "Azure App Service deploy" task.

I have seen this previously posted question, but I am confident it is not a personal caching issue. First, I have cleared my browser's cache, and also have used a different browser, which hadn't cached the old site. Second, and more indicative of the issue, I can push code through the Eclipse Azure plugin, which then causes the web app to serve the most recent code. However, I don't get similar functionality from the DevOps release pipeline

I have snooped around on kudu and still can't find the issue. Similar to the previously linked question, I discover that the code appears to be the most recent. I have also manually deleted the code from webapps folder, then redeployed from the DevOps pipeline, with no luck. I'm trying to discover what the Azure Eclipse does differently in deployment that I'm missing in my DevOps pipeline. Is Tomcat caching content? If so, is there a way to automate a clear cache using Azure Pipelines' UI?

# Answer
> 0 votes
I discovered two issues that hindered the pipeline deploying properly.

First, it appears that the Azure plugin for Eclipse creates an app.war file in the wwwroot folder of the app. Azure, apparently, prioritizes picking up this war, instead of any that might be in wwwroot/webapps.

Previously I had manually deleted app.war, but then the app would just 404. This leads to the second issue. I was attempting to deploy using the default package name "...SNAPSHOT"; however, Tomcat looks for ROOT.war. I added the following lines to my pom.xml.

```
<build>
       <finalName>ROOT</finalName>
</build>

```

This delivered the appropriately named file, which the "Azure App Service deploy" task places in wwwroot/webapps (where Tomcat looks for it). I then deleted the app.war file and any leftover ...SNAPSHOT folders in webapps, and restarted the app. It booted up with the war from the pipeline, as intended.

I will not be using the Azure plugin for Eclipse any longer, as it appears incompatible with Azure DevOps pipelines.

---
Tags: azure-devops, azure, azure-pipelines, java, tomcat
---


# Question
Title: How to configure AWS Incident Manager to call from same number?
I have an Android device with call screening and want to all my incident escalation phone calls to ring my phone, but so far all calls from AWS arrive from different numbers.

Is it possible to request AWS to use the same number for its calls?

I'm using AWS Incident Manager with a contact with a configured phone number.

# Answer
With help from a redditor, there is an Amazon virtual contact card available that contains all the phone numbers Amazon uses to make these calls. Adding this contact to my phone allows these pages to skip my call screener.

https://docs.aws.amazon.com/incident-manager/latest/userguide/contacts.html#contacts-details-file
> 0 votes

---
Tags: amazon-web-services
---


# Question
Title: How to fetch azure secret if exist in KV using terraform
I am using below terraform code for fetch azure secret and this is working fine when secret is exist in the azure KV.

Getting error when secret is not available in KV.

```
data "azurerm_key_vault_secret" "win_admin_pass" {
    name         = "${var.secret_name}"
    key_vault_id = "${data.azurerm_key_vault.keyvault.id}"
}

```

In my case, this secret may available or may not available.

How can we ignore error for this particular task when secret not available, or how can we check if secret exist or not, based on this condition we can fetch and ignore set of code?

# Answer
> 1 votes
This looks like a case for a conditional resource.

First, look up the secrets with a `data "azurerm_key_vault_secrets"` block:

```
data "azurerm_key_vault_secrets" "example" {
  key_vault_id = data.azurerm_key_vault.keyvault.id
}

```

and then a condition on the lookup:

```
data "azure_rm_key_vault_secret" "win_admin_pass" {
  count        = contains(data.azurerm_key_vault_secrets.win_admin_pass.names, var.secret_name) ? 1 : 0
  name         = var.secret_name
  key_vault_id = data.azurerm_key_vault.keyvault.id

}

```

If the secret doesn't exist, the data will look up zero resources.

---
Tags: terraform, azure, cloud, terraform-module, secret-management
---


title: Filtering AWS SQS Tags using JQ

text:
# Question
Title: Filtering AWS SQS Tags using JQ
I'm having doubts when trying to run a query with JQ. I have an SQS queue with Tags applied and when I run

`aws sqs list-queue-tags --region sa-east-1 --queue-url <my-url-queue> --output json | jq`

returns the output:

```
{
  "Tags": {
    "owner": "foo",
    "Name": "bar-queue"
  }
}

```

So...I just filter using like this:

`aws sqs list-queue-tags --region sa-east-1 --queue-url <my-url-queue> --output json | jq '.Tags[] | [.Name]'`

or

`aws sqs list-queue-tags --region sa-east-1 --queue-url <my-url-queue> --output json | jq '.Tags[].Name'`

and other methods without success, always output: `jq: error (at <stdin>:6): Cannot index string with string "Name"`

Maybe I'm having a noobie hahaha, but someone can help me?

# Answer
> 1 votes
Take heart, I've been struggling with JMESPath query expressions for the past couple of years! It's been a long time getting used to them.

In your example output, `Tags` is a JSON object rather than a list, so the `[]` in your JMESPath expression is causing the error. Try this as your `jq` command:

```
jq '.Tags.Name'

```

You'll get:

```
"bar-queue"

```

You can use the `-r` option to `jq` to get the output without quotes.

However I think you can do better by having the `aws` command do this filtering for you. Try the `--query` option:

```
aws sqs list-queue-tags --region sa-east-1 --queue-url <my-url-queue> --query 'Tags.Name'

```

Yes, the expression for the `aws` command doesn't have the leading `.`. If you don't want the quotes around the output, add the option `--output text`.

---

*Edited to add these notes about JMESPath expressions for jq vs. aws:*

The leading `.` character isn't the only difference between expressions that `jq` likes and the ones the `aws` command likes. I don't have an exhaustive list, but I have needed to use different quotes around path names for one program than I used for the other.

There are pretty good tutorials on query expressions for each of these programs:

And a website that's useful for interactively trying out expressions, once you've figured out how the site works: https://jmespath.org . It includes a link to another, more general tutorial.

---
Tags: aws-cli, amazon-sqs
---


# Question
Title: Azure pipelines Docker@2 build command does not pass through build args
I'm running an azure build pipeline for a dockerized application set up using an `azure-pipelines.yml` file. The problem is that we require a build arg passed through to the docker build command, but the arg is blank in all circumstances (even if no build arg is provided and a default is specified in the `Dockerfile`).

Snippet of `azure-pipelines.yaml` (sensitive data changed only):

```
- task: Docker@2
  displayName: build
  inputs:
    containerRegistry: our_container_registry
    repository: $(Build.Repository.Name)
    command: build
    Dockerfile: '**/Dockerfile'
    tags: $(Build.BuildNumber)
    arguments: --build-arg BuildNumber=$(Build.BuildNumber)

```

Further details:

* We are using the VM image of `ubuntu-latest` to run the job
* The build arg appears to be being passed through correctly as it does appear in the printout of the command that is run
    + e.g. if we pass `--build-arg BuildNumber=1.2.3` as part of the `arguments` property of the yaml file, we'll see `--build-arg BuildNumber=1.2.3` on the printout of the command run.
* As can be seen from the above snippet we're not running the `buildAndPush` command that I am aware has issues with build args (publish is it's own step we're not even getting to yet).
* The `Dockerfile` contains the command `RUN echo Build Number: ${BuildNumber}` which prints out `Build Number:`  during the build.
* Running the same build command printed out by the build step locally builds the container with no errors, and prints out the build number appropriately in the above mentioned command

I'm looking for an explanation as to how I can get the build args passed through correctly to our docker build, or if there is something we're not aware of preventing this then recommendations of other alternatives that would achieve the same ends.

# Answer
I found the solution by trying various problems, and my issue was that the scope of the `ARG` command was wrong.

I had in my `Dockerfile`:

```
FROM base_image_name AS build
ARG BuildNumber=0.1.0.0

# ...

FROM build AS publish

# ...

RUN echo Build Number: ${BuildNumber}

```

instead, what fixed the issue on the Azure was:

```
FROM base_image_name AS build

# ...

FROM build AS publish
ARG BuildNumber=0.1.0.0

# ...

RUN echo Build Number: ${BuildNumber}

```

I do not know why this built on my local machine, but failed on Azure, but with the scoping fixed it now builds.
> 0 votes

---
Tags: docker, azure-pipelines
---