# Question
Title: How can I transfer an Azure DevOps project from one organization to another?
In my DevOps organization, I've created a project for a client with git repo assets, boards, pipelines, etc. I'd like to transfer the project ownership from my organization to my client's organization.

How can I do this?

It's easy enough to switch ownership for the entire organization under "Organization Settings." But that's not what I want. I am looking, however, for that same workflow (if it exists) for a single project. The project ultimately needs to show up in their Organization space instead of mine.

If this isn't as simple as switching organizational ownership, is the save data an option?

I read here that I can save project data. But, there are no instructions on re-importing that data elsewhere. Also, I get the impression that I'm not getting a full project.

After the project is transferred they need to be able to continue to use it as it functioned in my organizational space.

# Answer
I feel this thread with this answer is relevant

> There is no official document to support this function.
> We can use this extension OpsHub Visual Studio Online Migration Utility to migrate source code, work items.
> In addition, there are Copy or clone work items and move git repositories between team projects features.

Seems worth mentioning that, in addition to the above, AzDO Boards migration tools lists:
Azure DevOps Migration Tools \[marketplace link\] as an expert tooling option.
> 2 votes

# Answer
moving an entire project currently isn't possible, and I don't foresee it being implemented. The git repo you can download, and then upload which can have your build and deploy scripts/configuration but will require some tuning. This is because things like subscriptions may not exist in the target organization for deployments.

Trying to migrate the project boards and work items is not possible because there might be different (or even custom) work item types, layout, process (agile vs scrum in AzDOs project settings) and expecting Microsoft to be able to map those automatically is unrealistic. Also, things like users, iterations, and teams are additional items that require a lot of attention to migrating boards. There is a public API for boards and tasks that can be programmed against if you wish to do a migration.
> 1 votes

---
Tags: azure-devops
---


# Question
Title: net.sf.json.JSONException: Invalid JSON String error on jenkins
```
Warning: API call to create metrics set failed:
[Pipeline] echo
net.sf.json.JSONException: Invalid JSON String
at net.sf.json.JSONSerializer.toJSON(JSONSerializer.java:143)
at net.sf.json.JSONSerializer.toJSON(JSONSerializer.java:103)
at net.sf.json.JSONSerializer.toJSON(JSONSerializer.java:84)
at org.jenkinsci.plugins.pipeline.utility.steps.json.ReadJSONStepExecution.doRun(ReadJSONStepExecution.java:86)
at org.jenkinsci.plugins.pipeline.utility.steps.AbstractFileOrTextStepExecution.run(AbstractFileOrTextStepExecution.java:32)
at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

```

I'm basically using a script on jenkins piepline to send data to grafana I'm facing this issue on jenkins version 2.235.1

The same script is working on another jenkins instance and it doesn't display this error

I think it's caused by plugin deprecated version or jenkins itself

I'm not sure if there is a workaround or which plugin to update

# Answer
It is possible for `readJSON` to fail due to whitespaces, leading and/or trailing. Appending `trim()` to the string could prevent this issue.

```
def my_json_string = readFile("my_json_file.json").trim()
def my_json = readJSON text: my_json_string

```
> 0 votes

---
Tags: jenkins, jenkins-pipeline, jenkins-plugins
---


# Question
Title: How can I configure OpenStack authentication for Terraform?
I want to use some company terraform files. They all rely on authentication to OpenStack which is the resource they're provisioning from. How can I configure this?

# Answer
> 0 votes
It depends on your proivder stansa,

```
provider "openstack" {
  # stuff
}

```

It should look something like the above. If it has,

* `auth_url` defined, then it also will very likely have `application_credential_name` and `application_credential_secret` defined.
* `cloud` defined, then it's looking for your OpenStack `clouds.yaml`. (Note this is distinct from `cloud.cfg`) Your `clouds.yaml`
    + is documented in OpenStack
    + can be exported from Horizons, the web interface,
    + typically resides in `$HOME/.config/openstack`, and applies for the user. But can also be in the local directory, or installed globally in `/etc/openstack`.
* Has neither `auth_url` nor `cloud`, in which case Terraform will try to authenticate with environmental variables which are documented here

---
Tags: terraform, openstack, authentication
---


# Question
Title: How can I get my clouds.yaml file from OpenStack Horizons?
OpenStack can export a configuration file called clouds.yaml. How do I export this file from the Horizon's interface?

# Answer
Follow these steps,

1. Log into Horizons
2. On the left bar click **Identity**.
3. Under Identity, click **Application Credentials**.
4. Click the **\+ Create Application Credentials** button at the top
5. You must at least give a name.
6. Click **"Create Application Credential"** at the bottom of the page.
7. Click **Download clouds.yaml**

And you'll typically want to put this file either,

* `$HOME/.config/openstack/clouds.yaml`
* `/etc/openstack/clouds.yaml`
> 0 votes

---
Tags: terraform, openstack, authentication
---


# Question
Title: Is it possible to download a "clouds.yaml" file for an Application Credential that already exists?
When I log into OpenStack Horizons, under **Identity****Application Credentials** I can see the name of the credential but I do not have the `clouds.yaml` file anymore. Can I download the `clouds.yaml` for this?

# Answer
> 0 votes
# No

You can not download the `clouds.yaml` file for an Application Credential that already exists. You'll have to delete it and create a new Application Credential..

---
Tags: openstack, authentication
---


# Question
Title: Where does Ansible DSL get translated to either Powershell or Python?
When running Ansible scripts, where does the translation from Ansible DSL to Python(Linux) or Powershell(Windows) happen? I assume this is on the Ansible node, that is, where the Ansible command is executed?

**NOTE**: I'm NOT asking how to use Ansible itself, or how it works from a tooling perspective - I've been using it for years so I know how it works from that point of view.

TIA

# Answer
The "translation" happens in the Ansible task executor, which takes information about a module and pushes executable code to the target.

From the documentation:

> Ansible works by connecting to your nodes and pushing out scripts called “Ansible modules” to them.

So to answer your question:

> I assume this is on the Ansible node, that is, where the Ansible command is executed?

**yes**, this happens on the controller (the node where the `ansible` command is executed).

Based on the configuration of the controller, it knows where to look for modules. This is set by the `ANSIBLE_LIBRARY` environment variable, or relevant section in the configuration file.

The `_execute()` method of the Ansible task executor class is the place to look for how this happens.

In terms of what happens in detail, from what I can tell from the executor class, Ansible

1. Creates a task queue manager with the list of tasks (handling loops, etc)
2. for each task creates a task executor script based on the module type, pulling in all of the relevant context for that task (module variables, connection variables, magic variables)
3. Creates a connection to the remote target see the `connection.py` library
4. Creates a pickle of the executable and writes it to the remote target (see`write_to_file_descriptor()`)

Part of the connection is determining the remote interpreter (python/powershell, etc) and the task executor takes care of ensuring that the remove interpreter for the module's execution environment is actually present (if, not, an error is raised).
> 1 votes

---
Tags: ansible
---


# Question
Title: When running terraform from Gitlab CI, terraform asks me for the "The address of the REST endpoint"
When I run `terraform apply` using the Terraform files in my GitLab CI Pipeline (I didn't write them) I get the following,

```
Initializing the backend...
address
  The address of the REST endpoint

```

Can these files be run in isolation of GitLab CI, on a workstation with vanilla terraform?

# Answer
This seems to be a part of the provider, `backend "http"`, which is like this somewhere

```
terraform {
  backend "http" {
  }
  ...

```

This is how you configure Terraform State as a backend in GitLab CI/CD. This is need so your tfstate file doesn't remain on the runner. This is what's responsible for updating that tfstate file remotely.

If running them outside of GitLab CI just delete the backend provider.
> 0 votes

---
Tags: terraform, gitlab, gitlab-ci-runner
---


# Question
Title: I am looking for a production alternative to kubectl port-forward
I want to know what are production alternatives to make http://192.168.49.2:30001 this accessible http://host\_machine\_ip:8007 here

I have set up minikube cluster with mobile-api service on it.

```
minikube service mobile-api-service --url this returns http://192.168.49.2:30001

```

curl http://192.168.49.2:30001 returns

```
{"mobileApi/dashboard":"http://192.168.49.2:30001/mobileApi/dashboard/",
"mobileApi/notification":"http://192.168.49.2:30001/mobileApi/notification/",
"mobileApi/feedback":"http://192.168.49.2:30001/mobileApi/feedback/"}

```

so my API is accessible on http://192.168.49.2:30001 from the host machine.

but i want to make it accessible http://host\_machine\_ip:8007.

when I do

```
kubectl port-forward --address 0.0.0.0 service/mobile-api-service 8007:8007

```

it makes http://host\_machine\_ip:8007 work as I want but I am sure this is not a production-like case.

## Update

```
apiVersion: v1
kind: Service
metadata:
  name: mobile-api-service
spec:
  selector:
    app: mobile-api
  type: LoadBalancer
  ports:
    - protocol: TCP
      port: 8007
      targetPort: 8007
      nodePort: 30001

```

this is service it is loadBalancer

# Answer
> 0 votes
Either make `service/mobile-api-service` of LoadBalancer type or use Kubernetes Ingress to expose the workload to external connectivity. Using LoadBalancer service would be the easiest. However, your cloud provider needs to support provisioning LoadBalancers.

I described a demo of Kubernetes Ingress. The post also talks about Kubernetes Load Balancer service limitations.

To play with minikube loadbalancers use `minikube tunnel` command.

# Answer
> 0 votes
You can try and experiment with metallb and use Loadbalancer service Another option is to use node port with external Loadbalancer.

---
Tags: docker, kubernetes, networking, kubectl, minikube
---


# Question
Title: Digital Ocean vs AWS EC2 for Laravel based application
I'm working on a project based on Microservice architecture in Laravel, which means I'll have many small apps and Databases, I'm confused weather I should buy AWS EC2 or Digital Ocean's hosting. My concern is pricing, In numbers I'm having a 100% price difference and getting Digital Ocean's hosting in 50% of the price with more features then what I'm getting on AWS.

Please suggest me what would be the better option.

# Answer
We faced exactly the same choices for our live service that deploys a laravel API and multiple react apps.

There is no generic correct answer to a question "what tool/provider is better?" as it depends on your circumstances. Cost is a particularly tricky question. Are we just talking about the cost of getting something live, or the cost of running an elastically scalable service, or the cost of running a highly available elastic service that spans data centre availability zones such that it isn't taken down by an outage in a single data centre zone?

Digital Ocean ("DO") is very highly promoted in the Laravel community so it was "the default" for the development team when I joined a startup about two months from go-live. When I asked "what about data centre availability zones" the answer at the time for Digital Ocean was that they didn't support it and we would have to set-up in two different DO data centres and work on our own mechanism (e.g. DNS) to switch between them if DO had a major outage in one place. Yet on AWS they have availability zones and also Elastic Load Balancer so its far easier to get a setup going that is "future ready" if you intend to be running a very successful large scale service a year after launch.

In the end we went live on openshift.com which is a shared kubernetes service running on AWS. So our live DNS points at the Elastic Load Balancer of a shared Kubernetes cluster were we rent space for our containers. The cluster we are on has containers by other companies and runs across many AWS EC2 virtual machines. Yet they are not dedicated to us so we only pay a small fraction of the costs. Yet if any one virtual machines crashes where our containers are running openshift kubernetes automatically starts a replacement container on another virtual machine. So we have automated high availability and crash recovery but only pay-as-we-go for the memory and CPU we currently use. We can scale up more containers anytime we want by using changing the replica count. Openshift has internal load balancing so that a single internal service IP with an internal DNS name automatically points at however many containers we want to run of a particular type. That solves discovery between our services. If you deploy a redis instance from the built-in catalog it is instantly available on a private IP with a DNS entry only viable in your project. That sort of ”discovery” feature is really important when you are running microservices at scale.

With our setup, we get a great low-cost price for a very professional, elastic, highly scalable, fully automated infrastructure-as-code setup, that has all the advantages of AWS ELB, and all the advanced features of kubernetes for microservices.

**Edit:** I wrote a tutorial for my experiences running laravel backend with a reactjs frontend on openshif.com that is over at https://gist.github.com/simbo1905/d27ba7218a2fd71fcb4f0fcede82fd6d
> 5 votes

# Answer
There are a couple of additional considerations to take into account when you compare your service between AWS and Digital Ocean.

If you know your service needs to be hosted on instances and you can estimate your performance needs well then you can buy reserved instances which give you between 30 and 50% off the listed price.

If you are in the early stages of a project you should also be able to run your service for free for 12 months. After that you should be able to estimate what reserved instances to use.

Keep in mind that various AWS services can significantly reduce your service costs while costing barely anything themselves such as API gateway caching, Cloudfront and Lambda@Edge.

Also AWS autoscaling features should allow you to use peaks and troughs in traffic to optimise your costs. You do need some basic scale for that though.

But more importantly Lambda should be your service of choice for microservices. It has a billing granularity of 100ms and 128Mb memory and a permanent free tier for a certain amount of "grains". It therefore can work like auto scaling, but with a startup and shutdown time of 100ms eliminating all costs where your service is actually not used.
> 1 votes

# Answer
This is kind of a subjective question but I will try to answer it as objectively as I can. Naturally you can list the advantages and disadvantages of the possible choices. Both choices offer about the same availability if I understand correctly. AWS is never cheapest in the price comparisons that I read. So that gives DigitalOcean an advantage.

For reasons from my management, I use AWS all the time but if I would start a project today then I would go for DigitalOcean because I also have seen price comparisons where AWS is never cheapest. Competition is good for everybody and therefore it is good to buy from competitors so that AWS feels a competition.

If features and pricing is your main concern and if DigitalOcean can do it, then you have convincing arguments to use DigitalOcean.

If complete automation with something like Cloudformation is your goal and if you already know AWS very well, then you might want to go with AWS instead because you already know it.
> 0 votes

# Answer
I would suggest to use AWS as this platform is far more mature than DO. You start with one app, but I assume more apps will be deployed. If you deploy and investigate everything yourself it costs also a lot of money. Imagine that you want to deploy a HA postgres cluster in DO, good luck with that. In AWS it is just a matter of a couple of clicks.
> 0 votes

# Answer
If your main concern is pricing, try Linode - an AWS alternative. Linode is a cloud service provider recently purchased by Akamai.

It has great support, docs and you can create Kubernetes cluster there very fast comparing to other cloud providers, where you can deploy your Laravel app.

Disclaimer, I wrote the linked article.
> 0 votes

---
Tags: amazon-ec2, microservices, digital-ocean
---


# Question
Title: How can I use non-TF environmental variables in Terraform?
Let's say I want to read `$HOME` in Terraform. I could create a variable called `HOME` and set it with `TF_HOME`, ie.,

```
TF_HOME=$HOME terraform apply

```

But is there a way to read `$HOME` directly in Terraform?

---

I know about `pathexpand` which solves this issue for `HOME`, I'm just wanting more information as to whether or not Terraform can access any environmental variable and used it as an example.

# Answer
> 0 votes
# Workaround (official solution)

You can not access the environmental variables through an official API. However, you can access the output of a script as a workaround. To access environmental variables external to the `TF_` prefix, you must

* Create a script to output the variable, you can wrap that output in JSON.
* Run that script as an external data source
* Read from the output of the script.

For more information on this approach check out,

# Answer
> 1 votes
From Terraform's perspective, environment variables are just another kind of data from outside of Terraform and so if you want to read from them then you'd do so using a data source designed for that purpose.

Using data sources for external data means that there's a well-defined model for when Terraform will access the data and a guarantee that Terraform will access it only once per plan and apply, either during the plan phase if possible or during the apply phase if the data source configuration depends on information not available until the apply step.

There is not an official provider for accessing environment variables because in most cases a Terraform configuration should not directly depend on anything about the specific system where it's running -- a typical Terraform configuration is for managing remote objects via network APIs -- but there are some third party providers in the registry whose documentation suggests that they might give you what you need:

I have not directly used either so I cannot specifically recommend either of these. I suggest reviewing the documentation and the source code to see if they can do what you need.

---
Tags: terraform
---


# Question
Title: Why is ansible loop adding square brackets ("[]") into my role?
I'm writing a role where I need to iterate over a list using the `loop` option. I'm using the ZFS module aisbergg.zfs.

The role:

```
    - hosts: localhost
      vars:
        zfs_pool_name: tank
        zfs_first_fs: subtank
        zfs_clients:
          - mom
        clients_fs_size: 1T

      tasks:
        - name: Create the clusters's filesystem
          zfs:
            name: "{{ zfs_pool_name }}/{{ zfs_first_fs }}/{{ zfs_clients }}"
            state: present
            extra_zfs_properties:
              volsize: "{{ clients_fs_size }}"
          loop: "{{ zfs_clients }}"
          loop_control:
            pause: 5

```

When I run the playbook I got the following error:

> TASK \[Create the clusters's filesystem\] \********************************************************************************************************************** failed: \[localhost\] (item=mom) =\> {"ansible\_loop\_var": "item", "changed": false, "item": "mom", "msg": "cannot create 'tank/subtank/\[mom\]': invalid character '\[' in name\n"}

Meaning that somehow the square brackets sneak into my variable, which I got from a list.

**Any idea on how can I get my variable without the square brackets?**

# Answer
> *Meaning that somehow the square brackets sneak into my variable, which I got from a list.*

Right, that

```
cannot create 'tank/subtank/[mom]': invalid character '[' in name

```

is the expected behavior according the code snippet you've provided.

> *Any idea on how can I get my variable without the square brackets?*

To get the list element in your loop you need just to reference it via `item`.

```
- name: Create the clusters's filesystem
  zfs:
    name: "{{ zfs_pool_name }}/{{ zfs_first_fs }}/{{ item }}"
    state: present
    extra_zfs_properties:
      volsize: "{{ clients_fs_size }}"
  loop: "{{ zfs_clients }}"
  loop_control:
    pause: 5

```

A minimal example which shows it more focused

```
- name: Loop over sample list
  debug:
    msg: "{{ item }}"
  loop: ['A', '1', 'B', '2']
  loop_control:
    extended: true
    label: "{{ ansible_loop.index0 }}"

```

will result into an output of

```
TASK [Loop over sample list] ******
ok: [localhost] => (item=0) =>
  msg: A
ok: [localhost] => (item=1) =>
  msg: '1'
ok: [localhost] => (item=2) =>
  msg: B
ok: [localhost] => (item=3) =>
  msg: '2'

```

**Further Documentation**
> 1 votes

---
Tags: ansible
---


# Question
Title: Simple CICD workflow for small-scale deployments?
I work for a small startup. We have 3 environments (Production, Development, and Staging) and GitHub is used as VCS.

All env runs on EC2 with docker.

Can someone suggest me a simple CICD solution that can trigger builds automatically after certain branches are merged / manual trigger option?

Like, if anything in merged into dev-merge, build and deploy to development, and the same for staging and pushing the image to ECR and rolling out docker update.

We tried Jenkins but we felt it was over-complicated for our small-scale infra.

GitHub actions are also evaluated (self-hosted runners), but it needs YAMLs to be there in repos.

We are looking for something that can give us the option to modify the pipeline or overall flow without code-hosted CICD config. (Like the way Jenkins gives the option to either use Jenkins file or configure the job manually via GUI)

Any opinions about Team City (Self-Hosted Community Version)?

PS: We are looking for free and self hostable options only!

# Answer
I would recommend reconsidering configuration-as-code (what you call code-hosted CICD config) this is considered best practice all over the place. The advantage is that you set up your tools to point to your repo and it "just works" - this provides a single point of truth as well as "easy" (read: easier than having to reconfigure all jobs manually) disaster recovery.

The only downside is that you have issues if you update the pipeline without wanting to push code changes to your prod branch, but only hits you if you want to do roll-backs. Besides - you can always pull the file to your production branch with `git checkout feature/otherbranch -- Jenkinsfile`

# Jenkins:

I use Jenkins for some of my own projects and have a setup that almost sounds like what you want (Except I build on all pushes, and deploy based on separate config in a separate repo.) these comments are based on that I even have a demo project on github you might want to gather inspiration from?.

You can always put your pipeline config in a separate repo and write code to manually check out the code, but this means that you have to either have THAT job polling the first branch often (to emulate triggers that work out of the box if you keep to a single repo)

Alternatively you can eliminate the second repo by manually configuring a pipeline in Jenkins, but you have to create one for each branch (you miss out on the goodness of multi-branch pipelines - again I'd advise you to reconsider the configuration-as-code approach)
> 1 votes

# Answer
In OneDev the Pipeline can be configured in a GUI. See here.

But I'm not sure if this is the right tool for you, because it is a git Server with CI/CD included.

But maybe it is possible to set up a mirror (which might complicate things).

Or you completely switch to onedev, which might be risky.
> 0 votes

---
Tags: docker, cicd
---


title: Argo CD + Operators = Overkill?

text:
# Question
Title: Argo CD + Operators = Overkill?
I intend to build a local Kubernetes cluster to host services in my home lab and share them with family and friends. If I'm already planning to deploy Argo CD as a GitOps solution and it'll already have a reconciliation loop for all my services, is it overkill to also use Operators for these same services? E.g. to deploy GitLab Operator through Argo CD. Would that be two tools doing the same thing, or did I get the Operator concept wrong?

# Answer
> 1 votes
I only used Fluxcd in the past so I'm not 107% confident it will work in exactly the same way in Argos but what I needed to do was to deploy those deployments in stages. In the first stage I've deployed an operator application itself, together with its CRDs. One of the CRDs would be e.g an `IstioOperator` or `GitLab` in your case.

After that stage was deployed to the cluster (especially after the CRDs got applied), there was the next stage depending on the first one, which was deploying a single manifest file like e.g.:

```
apiVersion: apps.gitlab.com/v1beta1
kind: GitLab
metadata:
  name: gitlab
spec:
  chart:
    version: "X.Y.Z" # https://gitlab.com/gitlab-org/cloud-native/gitlab-operator/-/blob/0.8.1/CHART_VERSIONS
    values:
      global:
        hosts:
          domain: example.com # use a real domain here
        ingress:
          configureCertmanager: true
      certmanager-issuer:
        email: youremail@example.com # use your real email address here

```

As you can see, the manifest is a simple "recipe" monitored by the operator application which would deploy all the required GitLab resources according to the manifest above.

From my experience, the first stage is responsible for keeping the operator application installed and running on its proper version release etc. The second stage is keeping the "recipe" file synced and reconciling its content, in case somebody e.g. changed its values directly in the Kubernetes cluster. On the other hand, the Operator agent will take care of keeping the deployed resources right according to the "recipe" manifest.

In general, I believe Argos needs to behave in a similar way as you can't apply `GitLab` manifest before you created the CRD defining it etc.

---
Tags: kubernetes, argocd
---


# Question
Title: How frequently is AWS Route 53 DNS Latency-Based Routing (LBR) recalculated for a given user?
Does anybody know how frequently one could expect Route 53's latency-based routing data to change for a given user?

For example, if I have Route 53 LBR setup with CloudFront sitting in front of 2 separate regions of my application, and R53 sends my user to the West Coast, how long is that good for? Another way of putting it is: ***how sticky is that decision?***

Do we run the risk of the user being switched a few minutes later to the East Coast even though West Coast is still running, merely because of some minor routing change on the public internet?

Or is the decision more abstract, and rather based on IP ranges rather than directly on the user's actual point to point latency between them and the region's border? Meaning that the user is sent to the region that "should" have the least latency, but that minor real-time changes in latency will be ignored?

# Answer
> 0 votes
I received this answer directly from an AWS Solutions Architect:

> Generally speaking, Route53 uses latency data from AWS points-of-presence to our end users location over a long-tail period to determine the fastest path from AWS to their known location.
>
> Because latency measurements are performed and stored over a period of time, I wouldn’t anticipate Route53 picking up volatile changes in latency that would lead to a user ‘flip-flopping’ between regions. From our docs:
>
> "Latency-based routing is based on latency measurements performed over a period of time, and the measurements reflect these changes. A request that is routed to the Oregon region this week might be routed to the Singapore region next week."

So while this isn't an exact answer, it does give general insight that is good enough for some determinations. For example, will we see the user switching back and forth between regions based on some large scale routing issue on the internet backbones? The answer is no. This is enough information to not need to worry about synching session information between the two regions in most application use cases, for example.

Hope this helps others.

---
Tags: amazon-web-services, dns, amazon-cloudfront
---


title: strip all comments from helm package

text:
# Question
Title: strip all comments from helm package
I have a lot of comments in my helm chart in my repository. I want to deliver my helm chart to others without these comments.

I am hoping to be able to do this during my CI pipeline. I have a step in my pipeline which performs the `helm package` command. I am hoping to have this package completely stripped of comments (in the templates, values, etc)

I have not found a way to do this directly with helm.

Is there a simple solution to my problem?

I have already considered some ways to solve it but I feel like I am missing a simple solution here. I know I can parse the values files and re write them using something like `yq` which will strip all comments. But that won't work for the template files as they are not valid yaml with the helm/go templating in them.

# Answer
> 1 votes
If you running Linux-based agents/runners, then you may be able to integrate the following command into your CI pipeline:

```
find path/to/helmchart/directory/ -type f -exec sed -i '' '/^#/d' {} \;

```

In the above command we use the `find` command to list out all the files in the helm chart directory and then we use the `exec` switch to run a `sed` to delete the lines which begin with a `#` using the regex pattern `^#`.

The above command can be modified to include other regex patterns as defined by the use case.

Also note that only the comments which begin with a `#` would be removed from the files in the helm directory. Helm chart templates have comments in the form of `{{/* ... */}}`. These will not be stripped. But supplying another regex will be able to do this job too. I hope this helps.

---
Tags: kubernetes, continuous-integration, helm, kubernetes-helm
---


# Question
Title: How to Control Ansible Loop or Task Default Output
I am trying to get my playbook to print readable debug output for stat on a number of directories when executing. The below shows the **with\_items** and **loop** way and both work but the real problem is the output of Ansible itself printing the entire dictionary or list of dictionaries for stat with each task, eg:

**ok: {hostname} =\> "multi-line spanning string of more information than wanted"**

The loop version is slightly more readable in that in prints a longer list of dicts only once, but requires modification every time I add a new directory to mydirdict. with\_items does not, but it prints the stat dict for each item in mydirdict each time and is far less legible.

I see that under other circumstances, having this print is useful but not in this case. Is there a way to control this?

```
vars:
  mydirdict:
    - { id: 1, name: /tmp/foo }
    - { id: 2, name: /tmp/bar }
tasks:
  - name: check dirs
    stat: path="{{ item.name }}"
    with_items: "{{ mydirdict }}"
    register: stats

  - name: check dir stat debug
    debug:
      msg:
       - "path {{ item.item.name }} exists? {{ item.stat.exists }}"
    with_items:
      - "{{ stats.results }}"

  - name: check dir stat debug
    debug:
      msg:
        - "path {{ item[0].item.name }} exists? {{ item[0].stat.exists }}"
        - "path {{ item[1].item.name }} exists? {{ item[1].stat.exists }}"
    loop:
      - "{{ stats.results }}"

```

# Answer
> 1 votes
Answering my own question based on understanding after Zeitounator pointed out my mistake. First off, I made a mistake that was hard to see even though I read the loop documentation. I was effectively doing this:

```
- debug:
    msg: "{{ item }}"
  loop:
    - "{{ [1,2,3] }}"

```

Which is different than both:

```
- debug:
    msg: "{{ item }}"
  loop: "{{ [1,2,3] }}"

- debug:
    msg: "{{ item }}"
  loop:
    - 1
    - 2
    - 3

```

But the real answer was adding a label as suggested:

```
- name: check dir stat debug
  debug:
    msg:
      - "path {{ item.item.name }} exists? {{ item.stat.exists }}"
  loop: "{{ stats.results | flatten(1) }}"
  loop_control:
    label: "{{ item.item }}"

```

which gives me just the list I started with on stdout and the message. I had mistakenly though label would mean supplying only part of the data structure but really it is about the stdout display only.

Lastly, there is some control one can have on the deeper parts of ansible output with the community.ansible.diy callback module as applied to Ansible config but not through a playbook.

---
Tags: ansible
---


# Question
Title: What would be the best questions to ask to assess technical skill on Kubernetes for an interview?
I have to conduct interviews on a specific position which requires somewhat probing expertise on Kubernetes.

I am not very familiar with it. What would be the best questions that would allow me to assess one's ability on this system ? Those questions can be both technical or generic.

# Answer
> 0 votes
It depends on the level of Engineer and questions. Which one do you need? I would ask what's the difference between pod and replicaset, then between replicaset and deployment, then between deployemnt and statefulset and daemonsets. What Kind of services exist and usecases. Again, you have to know kuberntes, before you ask this questions, or you may make wrong conclusions and decisions. AND AGAIN, it depends on the level of Engineer.

---
Tags: kubernetes
---


title: Get private key from terraform state?

text:
# Question
Title: Get private key from terraform state?
I used the terraform resource tls\_private\_key to generate a key and push that into AWS, but it didn't give me the private key to use to access the related resources. How can I recover that key?

# Answer
If you need to use a value like this private key for later steps, the common answer is to return it as a root module output value. Since the private key is sensitive, you can mark it as such to prevent Terraform from showing it on the screen by default:

```
output "private_key" {
  value     = tls_private_key.example.private_key_pem
  sensitive = true
}

```

When you run `terraform apply`, Terraform will redact the value because it's sensitive. However, you can still access it from scripts by running `terraform output -raw private_key`, because the value is still stored in the state and the sensitive option only prevents it from showing in human-oriented output. (The `-raw` option is intended for use in scripts.)
> 10 votes

# Answer
The private key is stored in the terraform state. Typically you would show the resources with the `terraform state` command, but sensitive values are not shared on the screen. If you use the -json option, this restriction is removed, dumping the entire state in json format.

Then you can use `jq` to parse the output to find the value. This value will have quotes around it and escaped newlines that you'll need to clean up in order to get a properly formatted PEM key. The `-r` option to jq will handle that well by dumping the value in raw format.

```
terraform show -json | \
jq -r '.values.root_module.resources[].values | select(.private_key_pem) |.private_key_pem' \
> ~/.ssh/terraform_private_key.pem

```

In the above `jq` command, three are three sections. The first filters down to the values of each resource. The second selects only value sets with a `privatey_key_pem` key in them. The third prints the value of those specific key(s). The select prevents the printing of a null line for each value set without a `private_key_pem` key/value pair to show.
> 4 votes

# Answer
# Using `private_key_openssh`

Another option is to use

```
terraform show -json |
  jq -r '.values.root_module.resources[] | select(.address=="tls_private_key.ssh").values.private_key_openssh'

```

This command also uses `root_module.resources` specifically, because that's where the key is created. This address corresponds to this in a `main.tf`,

```
resource "tls_private_key" "ssh" {
  algorithm   = "ECDSA"
  ecdsa_curve = "P384"
}

```

In your case the key may have a different name and resource. By any means, I'd rather not write it explicitly, you can find out what resource it's in with,

```
terraform show -json | jq -r | less

```

I choose not to use PEM because OpenSSH has its own key format which is different in some cases. For example, as of now OpenSSH can not read ed25519 keys in PKCS#9 (PEM).
> 1 votes

---
Tags: terraform, tls
---


# Question
Title: What is the ip address of the OpenStack metadata server, and how can I query it?
Where is the OpenStack metadata server located? What can I do query it?

# Answer
> 0 votes
OpenStack by default has its metadata hosted at `169.254.169.254`, and you can get to it from any resource provisioned on OpenStack with,

```
curl http://169.254.169.254/latest/meta-data

```

You can also substitute any date for `latest`. The list returned will look something like this,

```
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
reservation-id

```

You can then follow up by picking something like

```
curl http://169.254.169.254/latest/meta-data/public-hostname

```

---

You can find more information here,

---
Tags: openstack
---


# Question
Title: Getting "(sensitive value)" from `terraform state show`, how can I uncensor the value?
When I run I try to use `terraform state show` on variables flagged sensitive on some addresses I get

```
private_data = (sensitive value)

```

How can I get around this problem?

# Answer
> 7 votes
Unfortunately you can not. There is already a filed issue on this #32439.

# Workaround

Until that's resolved you'll have to use

```
terraform show -json | jq QUERY

```

And learn how to query with jq to get what you want.

---
Tags: terraform
---


# Question
Title: Does OpenStack meta data carry the remote-exec provisioner stanza?
I have a Terraform HCL file with a `remote-exec` provisioner that looks like this..

```
resource "openstack_compute_instance_v2" "sandboxvm" {
  provisioner "remote-exec" {
    inline = [<<EOF
      echo "start of sandboxvm provision..."
    EOF
    ]
  }
}

```

I **know** `user-data` gets stored on the OpenStack metadata server. I learned that the hard way when I wrote this answer. Does the OpenStack metadata server have access to the script used by this provisioner?

# Answer
# No

Having just done tested the `remote-exec` provisioner it does not..

```
$ curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
instance-action
instance-id
instance-type
local-hostname
local-ipv4
placement/
public-hostname
public-ipv4
public-keys/
reservation-id

```
> 0 votes

---
Tags: terraform, openstack
---


# Question
Title: Multiple shell commands not executing with shell module
For some reason this fails:

```
- name: "Get GPG key from the Jenkins portal"
  get_url:
    url: https://pkg.jenkins.io/debian-stable/jenkins.io.key
    dest: /usr/share/keyrings/jenkins.io.key
- name: "Delete existing keys and de-arm"
  become: true
  shell: |
     "rm -rf /usr/share/keyrings/jenkins.gpg"
     "gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key"

```

This is a part of a tasks file that gets called from another file. I do not see why the first should fail. The failure reason is:

```
fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": "\"rm -rf /usr/share/keyrings/jenkins.gpg\"\n\"gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key\"\n",
    "delta": "0:00:00.021351",
    "end": "2023-02-07 12:00:17.975367",
    "invocation": {
        "module_args": {
            "_raw_params": "\"rm -rf /usr/share/keyrings/jenkins.gpg\"\n\"gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key\"\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "msg": "non-zero return code",
    "rc": 127,
    "start": "2023-02-07 12:00:17.954016",
    "stderr": "/bin/sh: 1: rm -rf /usr/share/keyrings/jenkins.gpg: not found\n/bin/sh: 2: gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key: not found",
    "stderr_lines": [
        "/bin/sh: 1: rm -rf /usr/share/keyrings/jenkins.gpg: not found",
        "/bin/sh: 2: gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key: not found"
    ],
    "stdout": "",
    "stdout_lines": []
}

```

I don't know what `not found` means in this context. I can run these very commands in sequence on a real shell and they work.

```
- name: "Get GPG key from the Jenkins portal"
  get_url:
    url: https://pkg.jenkins.io/debian-stable/jenkins.io.key
    dest: /usr/share/keyrings/jenkins.io.key
- name: "Delete existing key"
  become: true
  shell: "rm -rf /usr/share/keyrings/jenkins.gpg"
- name: "De-arm the keys"
  become: true
  shell: "gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key"
  register: jenkinskey

```

Since these get processed via the shell, they should be fine I believe.

Any hints?

# Answer
> 1 votes
> *I do not see why the first should fail.*

It looks like the task has a syntax error

```
- name: "Delete existing keys and de-arm"
  become: true
  shell: |
     "rm -rf /usr/share/keyrings/jenkins.gpg"
     "gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key"

```

According the error message

```
    "stderr": "/bin/sh: 1: rm -rf /usr/share/keyrings/jenkins.gpg: not found\n/bin/sh: 2: gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key: not found",
    "stderr_lines": [
        "/bin/sh: 1: rm -rf /usr/share/keyrings/jenkins.gpg: not found",
        "/bin/sh: 2: gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key: not found"

```

The lines are processed as one single command and if I interpret it correctly. Furthermore, no quotes are necessary.

> *I don't know what `not found` means in this context.*

It means that the command as a whole or parts of it wasn't found, a syntax error and is the expected behavior.

**Possible Solution**

You could try with

```
  - name: Exec sh script on Remote Node
    shell:
      cmd: |
        rm -f /usr/share/keyrings/jenkins.gpg
        gpg --dearmor -o /usr/share/keyrings/jenkins.gpg /usr/share/keyrings/jenkins.io.key
    register: result

```

or even with

```
    shell: "rm -f /usr/share/keyrings/jenkins.gpg && gpg --dearmor -o /usr/share/keyrings/jenkins.gpg"

```

**Similar Q&A**

**Further Recommendation**

Instead of using the `shell` module implement the commands with Ansible modules

---
Tags: ansible
---


# Question
Title: Should I turn off copilot for text files containing secrets?
I use VSCode and have recently started using GitHub Copilot for my projects (programming and devops related). I understand that Copilot will send parts of my text to a remote server to get suggestions etc. Some of those text is ansible inventory files which may contain URLs, paths, passwords and other secrets.

Is there a common consensus about this topic? Is this risky? Do people exclude specific file types with sensitive data from being read by copilot?

# Answer
> 3 votes
Cutting off copilot for files that contain secrets is a good idea. The risk that github gets hacked may be small, but it is still a risk that can be easily avoided so why take a chance?

Keep in mind that the more fundamental issue is that you should avoid having secrets in text files. These are things that should be stored in vault and only retrieved for as long as they are needed. Storing them on disk in any way should be avoided. Presumably you are keeping your ansible code in a github repo and we have multiple recent examples of organizations that have had breaches of their repos.

---
Tags: ansible, security, github, vscode
---


# Question
Title: GitLab CI runner: remote: You are not allowed to download code from this project
Inside of one of my runner's logs, when I try to replay it, I'm getting the following error,

```
Reinitialized existing Git repository in /builds/users/repo/.git/
remote: You are not allowed to download code from this project.
fatal: unable to access 'https://acme.net/users/repo.git/': The requested URL returned error: 403

```

How can I resolve this error?

# Answer
> 3 votes
This problem is caused when the person (admin) that tried to execute the pipeline replay had insufficient access to the repository they replayed it on. This can happen when the for example the pipeline is private and they're not member, but an admin hits the replay button. The admin's replay does not have the ability to download from the cache. **This bug is reported as issue 21700 upstream.**

For a temporary workaround, you may opt to make the repository public if possible. Then the admin can replay the job.

---
Tags: gitlab, gitlab-ci-runner
---


title: Enterprise Ready Prometheus

text:
# Question
Title: Enterprise Ready Prometheus
I am new to Prometheus and have been reading about its popularity as a tool for monitoring applications, containers and K8s objects. However, most of the blogs that I have come across have at best done a very rudimentary introduction to using Prometheus for mission critical applications and I am trying to compile a list of everything Prometheus that an SRE should be aware of when setting Prometheus as the tool of your choice for monitoring.

I am listing out my current understanding, so feel free to correct if I may have got something wrong. But would really like to hear from the community on how best have they used Prometheus.

a) Installation of Prometheus -\> Where is the ideal location to install Prometheus? Some of the blog posts talk about installing this on the same K8s cluster that you have deployed your applications on. To me, that sounds counterintuitive, since now the monitoring stack eats up into your application resources. OR Should we deploy prometheus on a separate K8s cluster of its own. In this case, how does the configuration file changes to automatically scrape say two clusters which have two different applications deployed.

b) What all do you scrape using Prometheus? So far, I have gathered these

1- cAdvisor \[ for collecting resource and usage data for running containers\]

2- kube-state-metrics \[ for gathering stats about state of K8s objects\].

3- Node Exporters \[ For gathering node related metrics\]

4- Custom Exporters \[ For your applications not emitting metrics in prometheus format\]

Would be interested to hear, what else should one be monitoring ??

c) How does prometheus config.yaml file look like, when monitoring these above plugins/exporters. Do you manually add them to the scrape target? How would automatic service discovery work in the above cases? Can someone share a sample \<kubernetes\_sd\_config\> for their production scenarios \[ Obviously obfuscating the application or specific details\]

d) What is the ideal backend storage to be used for storing this time series data and how long should one be storing these data for? I think some of it may be driven from the compliance requirements of your application/organization, so in case where long term storage is needed, what are the practices adopted for metrics retention.

e) How is the DR strategy planned for Prometheus? I understand that since you are now monitoring your application using Prometheus, how do you ensure that this system has an effective DR plan? How would one design one?

I would appreciate inputs from anyone/everyone who has experience with Prometheus and how has this tool helped you in your monitoring pursuits.

# Answer
My current client is looking at embedding Prometheus in the application cluster, along with EFK, Vault, OPA, Consul, MongoDB and RedPanda. We call that the *core* architecture. We will have one cluster per 100 tenants. We will have a dev and a few test clusters too, each based on the same core. There will also be a pipeline cluster to build and validate source-code, which will have the same core tooling, plus tekton and argocd, and a container registry and nexus. I'd like to put the git repo in there too, but we're addicted to co-pilot and code-spaces, so we're going for github.

I think we'll eventually need to aggregate all this information in a single pane of glass, so we will need to create a cluster just for prom/graf (and probably efk too) and forward cluster-specific metrics to it along with some app specific stuff. But, we'll still go to an individual cluster's graf in order to view most of that cluster's application-specific stuff.
> 0 votes

---
Tags: prometheus
---


# Question
Title: Bandwidth utilization by pod on Kubernetes
Can someone tell me how I can get bandwidth usage per pod in Kubernetes from CLI? Ideally, I need something that will extract bandwidth usage for every pod.

# Answer
> 1 votes
Prometheus (and Thanos or Cortex, usually with some Grafana), is what you are looking for.

You have a Kubernetes (vanilla, OpenShift, ...), then kubelet/cadvisor should already be exposing some metrics, which you may scrape with Prometheus, and visualize in Grafana

See cadvisor docs in GitHub: https://github.com/google/cadvisor/blob/master/docs/storage/prometheus.md

Sample cadvisor scraping job in prometheus, discovering your nodes/targets integrating with Kubernetes API:

```
- job_name: cadvisor
  bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
  kubernetes_sd_configs:
  - role: node
  metrics_path: /metrics/cadvisor
  relabel_configs:
  - action: replace
    source_labels: [__meta_kubernetes_node_name]
    target_label: kubernetes_node_name
  scheme: https
  scrape_interval: 30s
  scrape_timeout: 10s
  tls_config:
    ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    # if ca won't work, add this:
    # insecure_skip_verify: true

```

Grafana dashboards for Prometheus/Cadvisor: https://grafana.com/grafana/dashboards/?dataSource=prometheus&search=cadvisor

... And, you were asking about CLI ... I'm not sure I have an answer here. While scripting directly on top of cadvisor is obviously possible: it could take you some time to figure out what to query. Keeping in mind you would be exclusively working with instantenous values / as opposed to Prometheus, which has functions working with metrics over an interval of time.

If possible, I would still look into Prometheus. And once you have one: this you can query, getting bandwidth consumption per pod, container, ... Even in CLI / there's an API. Maybe writing a kubectl plugin, extending get pods/adm top pods

# Answer
> 0 votes
Kubecost is a good tool to get the overall usage metrics and associated costs, please take a look here https://blog.kubecost.com/blog/network-egress-traffic-costs/

---
Tags: kubernetes, networking, kubectl, network-monitoring
---


# Question
Title: How do I use the remote-exec provisioner with Terraform?
I am using Terraform 0.14 and AWS. I am trying to write a .tf file that will invoke a remote-exec command via Terraform's provisioner.

This code does not work:

```
resource "aws_instance" "example" {
  ami           = "ami-1234"
  instance_type = "t2.large"
  key_name      = "foobar"
  user_data     = local.cloud_config_config

  provisioner "remote-exec" {
    inline = ["sudo yum -y install coolprogram"]
  }
}

```

The above will create the EC-2 instance. But it won't run the inline command. After I run "terraform apply" with the above content in my .tf file, I see "Error: host for provisioner cannot be empty"

I tried this .tf file variation, but I get an error about the variable:

```
resource "aws_instance" "example" {
  ami           = "ami-1234"
  instance_type = "t2.large"
  key_name      = "foobar"
  user_data     = local.cloud_config_config

  provisioner "remote-exec" {
    inline = ["sudo yum -y install coolprogram"]
  }
  connection {
    host     = "${var.host_name}"
  }
}

```

The hostname will be created dynamically. Terraform's documentation does not mention a hostname. I have tried different variables.tf files.

But I always get something like this:

> aws\_instance.example (remote-exec): Connecting to remote host via SSH... aws\_instance.example (remote-exec): Host: example aws\_instance.example (remote-exec): User: root aws\_instance.example (remote-exec): Password: false aws\_instance.example (remote-exec):
> Private key: false aws\_instance.example (remote-exec): Certificate: false aws\_instance.example (remote-exec): SSH Agent: false aws\_instance.example (remote-exec): Checking Host Key: false aws\_instance.example (remote-exec): Target Platform: unix

How do I use the remote-exec provisioner on an EC-2 server?

# Answer
here is a complete example:

```
resource "aws_key_pair" "my_key" {
 key_name   = "my_key"
 public_key = file(pathexpand("~/.ssh/id_rsa.pub"))
}

resource "aws_instance" "example" {
  ami = my_ami
  instance_type = "t2.micro"
  key_name = aws_key_pair.my_key.key_name
  ebs_block_device {
    device_name = "/dev/sda1"
    volume_size = 50
  }
  provisioner "remote-exec" {
   inline = ["my_command"]
  }
  connection {
   host        = coalesce(self.public_ip, self.private_ip)
   agent       = true
   type        = "ssh"
   user        = "My_user_name"
   private_key = file(pathexpand("~/.ssh/id_rsa"))
  }
}

```

so I assumed that you have your public and private key located in your home directory in `.ssh` otherwise you need to hard code the path. and you need to make sure that your IP is allowd in the AWS security group.
> 2 votes

# Answer
Terraform Provisioners are a last resort, so I'd encourage you to think about other options first.

In this case, your example suggests that you're already using cloud-init and so if at all possible I'd suggest modifying that cloud-init configuration to include a request to install the package you want to install. If you are using the cloud-config YAML format then you can use the Package Update Upgrade Install module to get a similar effect to your `yum install` command line:

```
packages:
 - coolprogram

```

---

However, if you've eliminated all other options except provisioners then the typical way to use remote-exec with an EC2 instance is to pass one of the instance's own IP addresses as the hostname, which you can do by using the special `self` object in the `connection` block to refer to the attributes of the object that the provisioner is running against, like this:

```
resource "aws_instance" "example" {
  ami           = "ami-1234"
  instance_type = "t2.large"
  key_name      = "foobar"
  user_data     = local.cloud_config_config

  connection {
    host = self.private_ip
  }

  provisioner "remote-exec" {
    inline = ["sudo yum -y install coolprogram"]
  }
}

```

The above example uses `self.private_ip`, which will work only if the host where you are running Terraform is running in or able to connect to the VPC where you're deploying this instance. Another option is to use `self.public_ip` to use the public IP address, but that will work only if your VPC has an Internet Gateway and your instance belongs to a security group that can accept incoming SSH connections from the internet.

In addition to supplying a suitable IP address, you'll also need to generate and assign an allowed SSH public key to the EC2 instance and provide the corresponding private key in the `connection` block, using the `private_key` argument. (Or one of the other authentication strategies that a `connection` block supports.)

These connection and authentication complexities are, as you hopefully saw in the Terraform documentation I linked earlier, one of the main reasons why provisioners are a last resort. Using cloud-init instead will avoid this because cloud-init runs automatically on system boot and retrieves the `user_data` string using a private API accessible from the instance's private network interface, and so it doesn't need any outside connectivity, to start any listening services, or to generate any new credentials.
> 1 votes

# Answer
As it was pointed out by Martin, remote-exec provisioner may not be able to connect to the target EC2 instance because of networking or security issues.

If the EC2 instance is SSM managed, you can use AWS Systems Manager Run Command to execute command on the instance.

Resource ssm\_command from 'pbobov/ssm' provider can be used to run SSM commands by Terraform. It was specifically designed as a workaround for using remote-exec provisioner.
> 0 votes

---
Tags: amazon-web-services, terraform, amazon-ec2
---


# Question
Title: Docker.Core.HttpBadResponseException: {"message":"2 errors occurred:\n\t* provisioning docker WSL distros: deploying
2 years cannot fix error install Docker on Windows

```
Docker.Core.HttpBadResponseException:
{"message":"2 errors occurred:\n\t* provisioning docker WSL distros: deploying \"docker-desktop\": importing WSL distro \"docker-desktop\": deploying WSL distro \"docker-desktop\" (\"C:\\\\Program Files\\\\Docker\\\\Docker\\\\resources\\\\wsl\\\\wsl-bootstrap.tar\" -\u003e \"C:\\\\Users\\\\Administrator\\\\AppData\\\\Local\\\\Docker\\\\wsl\\\\distro\"): exit status 1\n\t* checking if isocache exists: CreateFile \\\\wsl$\\docker-desktop-data\\isocache\\: The filename, directory name, or volume label syntax is incorrect.\n\n"}

   at Docker.Core.GoBackend.GoBackendClient.<PostNoBodyWithError>d__19.MoveNext() in C:\workspaces\4.16.x\src\github.com\docker\pinata\win\src\Docker.Core\GoBackend\GoBackendClient.cs:line 226
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.Engines.WSL2.LinuxWSL2Engine.<DoStartAsync>d__11.MoveNext() in C:\workspaces\4.16.x\src\github.com\docker\pinata\win\src\Docker.Engines\WSL2\LinuxWSL2Engine.cs:line 54
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.ApiServices.StateMachines.TaskExtensions.<WrapAsyncInCancellationException>d__0.MoveNext() in C:\workspaces\4.16.x\src\github.com\docker\pinata\win\src\Docker.ApiServices\StateMachines\TaskExtensions.cs:line 29
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.ApiServices.StateMachines.StartTransition.<DoRunAsync>d__6.MoveNext() in C:\workspaces\4.16.x\src\github.com\docker\pinata\win\src\Docker.ApiServices\StateMachines\StartTransition.cs:line 91
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at Docker.ApiServices.StateMachines.StartTransition.<DoRunAsync>d__6.MoveNext() in C:\workspaces\4.16.x\src\github.com\docker\pinata\win\src\Docker.ApiServices\StateMachines\StartTransition.cs:line 118
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.ApiServices.StateMachines.EngineStateMachine.<StartAsync>d__15.MoveNext() in C:\workspaces\4.16.x\src\github.com\docker\pinata\win\src\Docker.ApiServices\StateMachines\EngineStateMachine.cs:line 72
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Docker.Engines.Engines.<StartAsync>d__23.MoveNext() in C:\workspaces\4.16.x\src\github.com\docker\pinata\win\src\Docker.Engines\Engines.cs:line 109

```

# Answer
Follow the below steps to resolve your issue

Uninstall Docker Desktop

Open Local Disk C

Navigate to Users\<user name\>\AppData\Roaming\Docker

Delete all files in the Docker folder

Reinstall the Docker application 4.15.0 version
> 0 votes

# Answer
you could run `wsl --update` then on the cmd as admin `netsh winsock reset` then restart the machine worked for me
> 0 votes

---
Tags: docker, windows
---


# Question
Title: After removing Docker container, why is it automatically re-created?
I'm using ubuntu-16.04 desktop and installed `Docker-1.5-1` and `docker-engine-17.05.0~ce-0~ubuntu-xenial`, in which I installed `openmedicus/phplist`. My aim is to remove the container, and the image as well.

The issue is that the container is removed successfully but after some time the container is automatically created with a different container ID.

I even tried to remove images also but faced the same issue.

Trying to remove images:

```
$ docker images | grep phplist
openmedicus/phplist             latest              24d3de5279cb        4 months ago        694MB

$ docker rmi openmedicus/phplist:latest --force
Untagged: openmedicus/phplist:latest
Untagged: penmedicus/phplist@sha256:6ac98b2f624eb8cd61b64c4b398650975a876224d5925175c58ff0c1b05a2593

$ docker images | grep phplist    ## Delete Sucessfully

```

Now trying to remove container :

```
$ docker ps
b02b641560e8        24d3de5279cb             "/bin/sh -c /usr/s..."   2 minutes ago       Up 2 minutes        0.0.0.0:8080->80/tcp
$ docker stop b02b641560e8
b02b641560e8

$ docker rm b02b641560e8 --force   # Remove Sucessfully
b02b641560e8

```

Again, the container is created automatically :

```
$ docker ps
2c78199dc40c        openmedicus/phplist      "/bin/sh -c /usr/s..."   29 seconds ago      Up 28 seconds       0.0.0.0:8080->80/tcp                                 phplist

```

How can I remove the Docker container and image permanently?

# Answer
Check the following:

Something like watchtower maybe running and monitoring your host machine on which the containers are running. Watchtower can re-pull your images from a registry and re-launch a container for you. There are some tools similar to watchtower which do this.
> 1 votes

# Answer
This can also com from swarm. A look here to find some more
> 0 votes

---
Tags: docker, containers, linux
---


# Question
Title: Trigger different environments in one pipeline
I have a dev, uat, and prod environment for my ado project, and I just completed a pipeline deploy to dev.

Is it better practice to have a second pipeline that triggers on the uat branch, or is another stage in the same pipeline a better approach?

I'm leaning towards having a single pipeline, but am unsure how to set up a PR with a manual approval for uat.

Is this even an approach I should be investigating?

# Answer
maybe you can go in to a bit more details on the branching strategy you follow for example git flow as that will impact how you setup your CI/CD Pipelines.

For us we have one Pipeline for CD that will build the Development Branch (Git Flow) to deploy to our UAT Environment but automatically skip the Prod environment stage if the branch is called Develop\* or Feature\*

If the Branch starts with Release/\* it will go throe the build stage, the UAT Deployment stage, The Automated Testing stage (Against UAT) and then the Production Stage, see below graphic to illustrate.

What I like on this approach is that you use the exact same code package (artifacts) on UAT and Prod, in your case that would be Dev, UAT and Prod.

In our case the CI Pipeline is a separate Pipeline and uses the same yaml templates for the build but only deploys to the Automated Test environment and is triggered by a Pull Request Policy
> 0 votes

# Answer
It depends on your specific requirements and constraints, but having a single pipeline that handles multiple environments can simplify the overall CI/CD process and make it easier to track changes and deployments across different environments.

If you want to use a single pipeline, you can add a stage for UAT that is triggered only when a pull request is made to the UAT branch, and requires manual approval before proceeding to the next stage. To set this up, you can use the following steps:

1. Create a separate stage for UAT in your pipeline and configure it to run only when a pull request is made to the UAT branch.
2. Add a manual approval step in the UAT stage, so that someone must approve the deployment to UAT before it proceeds.
3. Use a feature such as approval rules in your version control system, such as GitHub, to ensure that only approved changes can be merged into the UAT branch.
4. Add a stage to your pipeline that handles the production deployment, and ensure that it only runs after the UAT stage has been approved and completed.

This approach can provide a more controlled and auditable deployment process, as well as give you the ability to test changes in the UAT environment before deploying to production. However, if your requirements are different, you may need to use a different approach that better suits your needs.
> 0 votes

---
Tags: azure-pipelines, yaml
---


# Question
Title: Best approach to toggle app availability in azure?
I have an azure webapp based on a docker image that implements an image processing API and is expensive to leave active 24/7. I'm wondering if there's an easy-to-manage way to auto-scale from 0 to 1 instances when we need it, and auto-shutdown after some period of inactivity. AKS seems like one option but maybe it's overkill?

I also considered azure functions but since we're using a custom docker image it wasn't clear to me if I can provision an on-demand instance.

The API app runs in the background to respond to requests from a thin Flask app, and the Flask app has a background Celery task thread, so I could take on the task of toggling API availability from the Celery thread. This seems doable but also feels a lot like I'm reinventing the wheel.

Is there a better approach?

# Answer
> 1 votes
You are looking for Serverless. The app will be auto-scaled up if there's a demand.

---
Tags: azure-devops, azure-kubernetes-service, flask
---


# Question
Title: Can't change ownership of folders and files in Docker containers
I am having these two dockerfiles which in both I am adding the command `RUN chown -R www-data:www-data /var/www/html` to change the ownership to www-data user. The containers seems to build successfully with no errors, however when I ssh for example into the running php container and run `ls -la` on the workingdir I can still see all files and folders are owned by root. If I do `ls -la` outside of the running container on my local env I can see them owned by my username and staff.

```
// Apache
FROM httpd:alpine

COPY ./docker/apache/httpd-vhosts.conf /usr/local/apache2/conf/extra/httpd-vhosts.conf

WORKDIR /var/www/html/
COPY ./public /var/www/html/public

RUN chown -R www-data:www-data /var/www/html

EXPOSE 80
CMD ["httpd", "-D", "FOREGROUND"]

-----

// PHP
FROM php:8.1-fpm-alpine

# Install any additional dependencies like pdo, xdebug etc
RUN apk add --no-cache --update linux-headers \
    ${PHPIZE_DEPS} \
    openssl-dev \
    && docker-php-ext-install pdo pdo_mysql

# Set up the php configuration in the container
COPY ./docker/php/php.ini /usr/local/etc/php/php.ini
COPY ./docker/php/conf.d/*.ini /usr/local/etc/php/conf.d/

# Copy the codebase in the working dir of the container
WORKDIR /var/www/html
COPY . /var/www/html

RUN chown -R www-data:www-data /var/www/html

EXPOSE 9000
CMD ["php-fpm", "-y", "/usr/local/etc/php-fpm.conf", "-R"]

```

What do I need to do to make sure the files & folders are owned by www-data user within the containers?

# Answer
One thing you must do is ensure the `www-data` user exist take this example here:

https://github.com/docker-library/httpd/blob/master/2.4/alpine/Dockerfile

so you need to add something akin to:

```
# ensure www-data user exists
RUN set -x \
    && adduser -u 82 -D -S -G www-data www-data
# 82 is the standard uid/gid for "www-data" in Alpine

```

or better you can temporarily debug things and add these lines to list to see what you already have in there before adding the above:

```
# Update package list and install grep
RUN apk update && apk add --no-cache grep

# see a list of www-data members
RUN grep ^www-data /etc/group
#see if www-data exists
RUN grep ^www-data /etc/password

```
> 0 votes

---
Tags: docker, dockerfile
---


# Question
Title: Is it possible to create a tls kubernetes secret using Azure Key Vault data resources in Terraform?
I have a certificate file and a private key file that I am using to implement tls encrypted traffic for several different k8s pods running under an NGINX ingress load balancer. This works fine (i.e. the web apps are visible and show as secure in a browser) if I create the kubernetes.io/tls secret in either of these ways:

1. Use kubectl: `kubectl create secret my-tls-secret --key <path to key file> --cert <path to cert file>`.
2. Reference those files locally in terraform:

```
resource "kubernetes_secret" "my_tls_secret" {
  metadata {
    name = "my-tls-secret"
  }

  type = "kubernetes.io/tls"

  data = {
    "tls.crt" = file("${path.module}/certfile.cer"),
    "tls.key" = file("${path.module}/keyfile.key")
  }
}

```

However, neither of these methods are ideal because for #1, it turns my `terraform plan/apply` steps into 2-step processes and for #2, I don't want to commit the key file to source control for security reasons.

So, my question is: is there a way to do this by using some combination of Azure Key Vault data resources (i.e. keys, secrets or certificates)?

I have tried the following:

1. Copy/pasting the cert and key into key vault secrets (have also tried this with base64 encoding the values before pasting them into the key vault and using base64decode() around the tls.crt and tls.key values in the Terraform):

```
data "azurerm_key_vault_secret" "my_private_key" {
  name         = "my-private-key"
  key_vault_id = data.azurerm_key_vault.mykv.id
}

data "azurerm_key_vault_secret" "my_certificate" {
  name         = "my-certificate"
  key_vault_id = data.azurerm_key_vault.mykv.id
}

resource "kubernetes_secret" "my_tls_secret" {
  metadata {
    name = "my-tls-secret"
  }

  type = "kubernetes.io/tls"

  data = {
    "tls.crt" = data.azurerm_key_vault_secret.my_certificate.value,
    "tls.key" = data.azurerm_key_vault_secret.my_private_key.value
  }
}

```
2. Tried importing the cert as an Azure key vault certificate and accessing its attributes like so:

```
data "azurerm_key_vault_certificate_data" "my_certificate_data" {
  name         = "my-certificate"
  key_vault_id = data.azurerm_key_vault.mykv.id
}

resource "kubernetes_secret" "my_tls_secret" {
  metadata {
    name = "my-tls-secret"
  }

  type = "kubernetes.io/tls"

  data = {
    "tls.crt" = data.azurerm_key_vault_certificate_data.my_certificate_data.pem,
    "tls.key" = data.azurerm_key_vault_certificate_data.my_certificate_data.key
  }
}

```

which results in an error in the NGINX ingress log of: `[lua] certificate.lua:253: call(): failed to convert private key from PEM to DER: PEM_read_bio_PrivateKey() failed, context: ssl_certificate_by_lua*, client: xx.xx.xx.xx, server: 0.0.0.0:443`

Both of these attempts resulted in failure and the sites ended up using the default/fake/acme kubernetes certificate and so are shown as insecure in a browser.

I could potentially store the files in a storage container and wrap my `terraform` commands in a script that pulls the cert/key from the storage container first and then use working method #2 from above, but I'm hoping there's a way I can avoid that that I am just missing. Any help would be greatly appreciated!

# Answer
> 1 votes
Method #1 from the original post works - the key point I was missing was *how* I was getting the cert/key into Azure KeyVault. As mentioned in the post, I was copy/pasting the text from the files into the web portal secret creation UI. Something was getting lost in translation doing it this way. The right way to do it is to use the Azure CLI, like so:

```
az keyvault secret set --vault-name <vault name> --name my-private-key --file <path to key file>

az keyvault secret set --vault-name <vault name> --name my-certificate --file <path to cert file>

```

---
Tags: kubernetes, terraform, azure, ssl, secret-management
---


# Question
Title: How do I find out the attributes of a terraform object?
I'm creating a Terraform file, and something is obviously wrong,

```
Error: Incorrect condition type

  on ssh.tf line 47, in locals:
  47:   is_generated = data.local_sensitive_file.public_key ? true : false
    ├────────────────
    │ data.local_sensitive_file.public_key is object with 4 attributes

The condition expression must be of type bool.

```

How do I find out what the attributes of `data.local_sensitive_file.public_key` are though?

# Answer
Since `local_sensitive_file` here is a data source, you can assume that its attributes are the attributes defined by the provider that this data source belongs to, which is `hashicorp/local`.

At the time of writing the documentation for `local_sensitive_file` says that it has the following attributes:

* `filename`
* `content`
* `content_base64`
* `id`

Provider documentation is typically the best place to find out about attributes of a resource type because it'll also tell you what these attributes represent, in addition to their names.

Not all objects you'll encounter in Terraform will originate in providers though, so a more general answer that will work for any object you can refer to in an expression is to run `terraform console` and then ask it for the type of the value in question:

```
> type(data.local_sensitive_file.public_key)
object({
    content: string,
    content_base64: string,
    filename: string,
    id: string,
})

```

Unfortunately for a resource (either a `resource` or `data` block) it won't yet have a known type until it's been created or read for the first time, and so if your configuration isn't valid then you would need to make it valid enough to run `terraform apply` successfully before you could use this technique. But for objects that are defined within the Terraform configuration itself, rather than by external operations that haven't run yet, the console is a useful way to inspect data and data types.
> 4 votes

# Answer
You can find the keys with

```
terraform show -json |
  jq '.values[].resources[] | select(.address=="data.local_sensitive_file.public_key").values | keys'

```

Which should return,

```
[
  "content",
  "content_base64",
  "filename",
  "id"
]

```

Alternatively you can look up `local_sensitive_file` in the docs.
> 2 votes

---
Tags: terraform
---


# Question
Title: How can I read a file if it exists, but if it doesn't do not throw a fatal error?
Terraform give us the ability to a read a local file, which has a secret..

```
data "local_sensitive_file" "foo" {
  filename = "/tmp/foo"
}

```

But, what is the right way to handle a file that may not exist? In the above, if `/tmp/foo` doesn't exist, you get this error on compilation:

> Error: Read local file data source error
>
> ```
> with data.local_sensitive_file.foo,
> on ssh.tf line 12, in data "local_sensitive_file" "foo":
> 12: data "local_sensitive_file" "foo" {
>
> ```
>
> The file at given path cannot be read.
>
> +Original Error: `open /tmp/foo: no such file or directory`

Seems as if setting the filename to `null` also doesn't work.

# Answer
> 2 votes
In this particular case where the path is well-known I would typically use the `file` function instead of the `local_sensitive_file` data source, since the data source is primarily intended for reading files that are generated dynamically while Terraform is running.

```
locals {
  filename = "${path.module}/whatever.txt"
  example  = fileexists(local.filename) ? sensitive(file(local.filename)) : null
}

```

However, if in your real configuration this *is* a file that comes from outside of your Terraform configuration or is generated dynamically by your configuration then you can get a similar effect by telling Terraform that there should be no instances of this data resource if the file doesn't exist:

```
locals {
  filename = "${path.module}/whatever.txt"
}

data "local_sensitive_file" "example" {
  count = fileexists(local.filename) ? 1 : 0

  filename = local.filename
}

locals {
  file_content = one(data.local_sensitive_file.example[*].content)
}

```

That last expression using `one` is a concise way to take either the one value of `content` or to produce `null` if there are no instances of this resource. Therefore `local.file_content` would be either the content of the file or `null` if there is no file.

---

Although I suspect this won't be a problem in your case due to this being temporary local files on disk rather than long-lived objects in a remote system, I suggest caution when trying to declare rules like "use this only if it exists". If applying this Terraform configuration could potentially *change* whether this file exists then you risk declaring something contradictory that will cause Terraform to exhibit unspecified behavior, because it'll make assumptions during the planning phase that won't necessarily hold during the apply phase.

In general I'd recommend instead being explicit about whether you expect the file to exist or not, so that Terraform can tell you immediately if your assumption is incorrect, rather than relying on your assumption and then acting strangely if it turns out not to be true.

For example, you can declare an input variable whose presence explicitly represents that the file ought to exist, but that can be omitted when it need not exist:

```
variable "filename" {
  type    = string
  default = null
}

data "local_sensitive_file" "example" {
  count = var.filename != null ? 1 : 0

  filename = var.filename
}

locals {
  file_content = one(data.local_sensitive_file.example[*].content)
}

```

This is essentially the same as my previous example except now the predicate for deciding whether to read the file is whether a filename was specified at all, and so Terraform can therefore give quick feedback if the caller of this module makes a typo when writing the filename, or otherwise assumes a file to exist that doesn't actually exist.

# Answer
> 1 votes
In Terraform, you can use the `fileexists` function to check if a file exists before attempting to use it. This function returns a boolean value indicating whether the specified file exists or not.

https://developer.hashicorp.com/terraform/language/functions/fileexists

in your case since you don't give to much logic behind what you want to do

```
filename = fileexists("/tmp/foo") ? "/tmp/foo" : "/tmp/whateverfile_you_want_if_foo_doesn't_exist"

```

`filename` will have `/tmp/foo` if file exists and `/tmp/whateverfile_you_want_if_foo_doesn't_exis` if file doesn't exists

---
Tags: terraform
---


title: Create service per pod in statefulset

text:
# Question
Title: Create service per pod in statefulset
I have a statefulset app which can have any number of replicas. The pods produced should be able to communicate to external comptures. Those pods must act statefully when clients communicate with them. I tried to create headless service and nodeport services, but it still has only one service to communicate with and also couldn't make it to communicate externally. How can I make it allocate external ip and port per each port?

# Answer
The above answer is incorrect. You can use pod name labels, cf: https://stackoverflow.com/questions/68123751/kubernetes-service-for-a-subset-of-statefulset-pods

For example:

```
apiVersion: v1
kind: Service
metadata:
  name: my-sts-1
spec:
  ...
  selector:
    statefulset.kubernetes.io/pod-name: my-sts-1
  ...

```

You will have to manually create the services when you scale your statefulset up or down. Alternatively, you could create a helm chart or a CRD + Operator.
> 4 votes

# Answer
Just to clarify if I got you right.

Do you want to have a separate service for every pod/replica in your statefulSet?

If so - this is not how K8s sees services being used. The service is used for grouping and load balancing purposes.

In your case, if you want to have 1-on-1 connection between an external client and a pod, I suggest deploying your statefulSet a number of times with `replicas=1` and exposing a service (as NodePort/LB) every time.
> 2 votes

---
Tags: kubernetes, openshift
---


# Question
Title: Terraform's OpenStack provider: Error: One of 'auth_url` or 'cloud` must be specified
When I run `terraform apply` (but not `terraform validate`) I get the following error,

> Error: Invalid provider configuration
>
> Provider "registry.terraform.io/terraform-provider-openstack/openstack" requires explicit configuration. Add a provider block to the root module and configure the provider's required arguments as described in the provider documentation.

Immediately followed by,

> Error: One of `auth_url` or `cloud` must be specified
>
> ```
>  with provider["registry.terraform.io/terraform-provider-openstack/openstack"],
>  on <empty> line 0:
>    (source code not available)
>
> ```

How can I resolve this problem?

# Answer
The problem I had to generate this was my provider had an alias,

```
provider "openstack" {
  alias       = "acmecorp"
  tenant_name = "admin"
  cloud       = "openstack"
}

```

That `alias` caused this problem, and removing it resolved the error. But looking at the specific `terraform-provider-openstack/openstack` provider documentation `alias` isn't mentioned. I pulled the alias convention from AWS and it seems to not be supported with OpenStack.

I opened #1480 for a better error message
> 0 votes

---
Tags: terraform, openstack
---


title: Using docker-swarm with Jenkins

text:
# Question
Title: Using docker-swarm with Jenkins
I have a cluster which is being used as a docker swarm - 1 manager, and 4 workers. The manager node is a pure manager node, set via

```
docker node update --availability drain <manager-id>

```

The manager has the `/usr/lib/systemd/system/docker.service` set to start with

```
ExecStart=/usr/bin/dockerd -H fd:// -H tcp://0.0.0.0:2375 --containerd=/run/containerd/containerd.sock

```

Now I am trying to use this cluster/swarm as a Jenkins cloud provider, so that all jobs run in docker containers in this swarm. In Jenkins, under `Configure Clouds`, I have create a cloud provider of type "docker", and provided the manager IP address as the docker host uri, eg. `tcp://a.b.c.d:2375`. There is no error, but when I start a job, it creates a container on the manager itself.

How can I make it so that each container is deployed on a non-manager container?

# Answer
> 0 votes
Docker Swarm is a layer over Docker and it manages the creation of containers on nodes given docker service definitions.

As I understand it, Jenkins does not know docker swarm, and simply asks the target node to create containers. Docker swarm, and its orchestration is consequently not involved - everything gets created and scheduled on the target node only.

---
Tags: jenkins, docker-swarm
---


# Question
Title: I cannot exec into a docker container running in ECS
I'm attempting to login to a container (from ECR) deployed into an AWS ECS cluster. For what it's worth I'm using the docker compose/ecs integration to deploy this cluster. My docker compose file is very minimal. The container needs a GPU so I'm deploying it to a GPU instance (g4dn.12xlarge) with an ecs optimized ami, ami-03d0d75de9d82f509 (amzn2-ami-ecs-gpu-hvm-2.0.20221230-x86\_64-ebs).

I'm trying to exec into this container but am unable to login.

I've executed this command to attempt login:

```
aws ecs execute-command --cluster apptest --task 36fd9d835ad24b4ca188e40c59768cee --container apptest --interactive --command "/bin/sh"

```

I'm getting the following error:

```
The Session Manager plugin was installed successfully. Use the AWS CLI to start a session.

An error occurred (TargetNotConnectedException) when calling the ExecuteCommand operation: The execute command failed due to an internal error. Try again later.

```

I would really appreciate any additional info on why I might be getting this error and what I should check/test.

I've run the amazon-ecs-exec-checker script which gives the following output (I have removed some account info). Exec is enabled for the task and I believe all the correct permissions are in place (SSM policy permissions, etc). When I searched for similiar errors on google I saw that this was an issue on older ami's but it should have been fixed.

```
    Prerequisites for check-ecs-exec.sh v0.7
-------------------------------------------------------------
  jq      | OK (/usr/bin/jq)
  AWS CLI | OK (/usr/local/bin/aws)

-------------------------------------------------------------
Prerequisites for the AWS CLI to use ECS Exec
-------------------------------------------------------------
  AWS CLI Version        | OK (aws-cli/2.9.9 Python/3.9.11 Linux/5.10.149-133.644.amzn2.x86_64 exe/x86_64.amzn.2 prompt/off)
  Session Manager Plugin | OK (1.2.398.0)

-------------------------------------------------------------
Checks on ECS task and other resources
-------------------------------------------------------------
Region : us-east-1
Cluster: apptest
Task   : arn:aws:ecs:us-east-1:****:task/apptest/36fd9d835ad24b4ca188e40c59768cee
-------------------------------------------------------------
  Cluster Configuration  | Audit Logging Not Configured
  Can I ExecuteCommand?  | arn:aws:iam::****:user/***
     ecs:ExecuteCommand: allowed
     ssm:StartSession denied?: allowed
  Task Status            | RUNNING
  Launch Type            | EC2
  ECS Agent Version      | 1.67.2
  Exec Enabled for Task  | OK
  Container-Level Checks |
    ----------
      Managed Agent Status
    ----------
         1. STOPPED (Reason: Received Container Stopped event) for "apptest_ResolvConf_InitContainer" - LastStartedAt: null
         2. RUNNING for "apptest"
    ----------
      Init Process Enabled (apptest-apptest:14)
    ----------
         1. Disabled - "apptest_ResolvConf_InitContainer"
         2. Disabled - "apptest"
    ----------
      Read-Only Root Filesystem (apptest-apptest:14)
    ----------
         1. Disabled - "apptest_ResolvConf_InitContainer"
         2. Disabled - "apptest"
  Task Role Permissions  | arn:aws:iam::****:role/apptest-apptestTaskRole-12KZYNKIW0B65
     ssmmessages:CreateControlChannel: allowed
     ssmmessages:CreateDataChannel: allowed
     ssmmessages:OpenControlChannel: allowed
     ssmmessages:OpenDataChannel: allowed
  VPC Endpoints          | SKIPPED (vpc-020109*** - No additional VPC endpoints required)
  Environment Variables  | (apptest-apptest:14)
       1. container "apptest_ResolvConf_InitContainer"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined
       2. container "apptest"
       - AWS_ACCESS_KEY: not defined
       - AWS_ACCESS_KEY_ID: not defined
       - AWS_SECRET_ACCESS_KEY: not defined

```

# Answer
> 1 votes
Check if you can use AWS Session Manager to connect to the instance. I was having the same issue and I found it was because the SSM agent on the instance could not connect due to having restricted egress network access.

---
Tags: docker, amazon-web-services, amazon-ecs, ecs
---


# Question
Title: Is there a way to have user stories pass on their tags to nested tasks in Azure Dev Ops?
I have multiple user stories under a single feature.

When I assign sprints to those user stories it passes it to any task that I create under it, but when I add a new tag to that user story it will not set the tasks underneath it to inherit that new tag.

Is there a way to have tasks that are already nested under user stories to inherit a new tag?

# Answer
> 1 votes
No. You can only add the same tags for all child work items through Edit one or more fields. You may edit it in queries and backlogs.

---
Tags: azure-devops, azure
---


# Question
Title: How to enable kernel modules by default on google cloud kubernetes node os image?
I'm having an issue with Google Cloud kubernetes node os image (container optimized image).
Trouble is that one of the pods on the node is NFS server and node does not have nfs and nfsd kernel modules enabled on startup.
So whenever the node restarts I need to ssh into it and run

```
sudo modprobe nfs
sudo modprobe nfsd

```

to enable required kernel modules.

How can I make this modules enabled by default when node restarts?

Thank you for your suggestions :-D

# Answer
> 0 votes
For any future stumblers...

I solved the issue by creating initContainers block which allows any container to run before the pod's main container starts.

Since we are actually running commands on host os we need to run `initContainers` container in `privileged` mode (I may have added extra privilege arguments.. it works, I don't care).

Also we need to mount host os `/lib/modules` directory to container in order for `modprobe` command to modify host modules.

While experimenting, `initContainers` busybox container would fail and to check its fail output I would run this command:

```
kubectl logs --container='load-nfs' [YOUR_POD_FULL_NAME_HERE]

```

Here's my test yaml that works:

```
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: ch1
  name: deployment-test
  namespace: default
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: test
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: test
    spec:
      containers:
        - image: strm/helloworld-http
          imagePullPolicy: Always
          name: test
          ports:
            - containerPort: 3000
              protocol: TCP
          resources:
            limits:
              cpu: "1"
              memory: 1Gi
            requests:
              cpu: 200m
              memory: 256Mi
      initContainers:
        - name: load-nfs
          image: busybox
          command: ["sh", "-c", "modprobe nfs && modprobe nfsd"]
          volumeMounts:
            - mountPath: /lib/modules
              name: host-modules
          securityContext:
            runAsUser: 0
            privileged: true
            allowPrivilegeEscalation: true
            capabilities:
              add:
                - SYS_ADMIN
      volumes:
        - name: host-modules
          hostPath:
            path: /lib/modules
            type: Directory

```

---
Tags: kubernetes, google-cloud-platform
---


# Question
Title: Run GitLab job only if variable is provided
I have two jobs in my gitlab ci pipeline 'job a' and 'job b'. I want only 'job a' to run if variable x is provided and run 'job b' if variable b isn't provided. Something like this.

```
job a:
  rule:
    - if: variable X is provided
  script:
    - do something

job b:
  rule:
    - if: variable X is not provided
  script:
    - do something

```

# Answer
> 2 votes
If you are OK with considering an empty variable value equivalent to it not being "provided", you can do this:

```
if: $x == ''

```

for "not provided" and

```
if: $x != ''

```

for "provided".

---
Tags: gitlab, cicd
---


# Question
Title: GCP IAM Role and Deny Rule On Organisation Folders
I have a GCP organisation of a handful of projects laid out in folders. I also have a handful of humans that need access of various types to the estate.

My first class of humans are those which need full read-only access to just about all of GCP.

My second class of users need read-only access to one or more folders of projects. However, I'd like to deny them access to anything in Secrets Manager or Billing.

My first thought was to create the first class of users at the top level, and the second class at the folder level, with a deny rule at the project level. However, since they all use `roles/viewer`, that doesn't work.

I could also solve this by denying each user in the second class, but not the first. Since I'm permissioning everyone with Terraform, I could just deny the second class of users. This could work, but it means something special for each user - not a terrible problem, but seems wrong.

The last method I've looked at is to create a custom role. The first class of users just use `roles/viewer` as they do now, but the second class use `roles/restricted-viewers` or whatever. My problem with that is that I'd need to somehow keep up with permissions in the `viewer` role and apply them to my custom role.

Are there any other (better?) ways to do this? Can we guess at how Google intended us to handle situations such as this?

# Answer
> 0 votes
After some battling, I've found that ultimately this isn't possible to achieve (at least, not with a Deny policy).

The good news is you can attach a deny policy to a project, folder or organisation (See: https://cloud.google.com/iam/docs/deny-access#attachment-point).

The bad news is you cannot deny access to billing - it's not a supported permission (see: https://cloud.google.com/iam/docs/deny-permissions-support). If your use case is covered by supported permissions then it actually all works quite well though.

For anyone stumbling across this, the Terraform to try this out looks something like this:

```
terraform {
  required_providers {
    google = {
      version = "~> 4.53.1"
    }
  }
}

provider "google" {
  project = "myProject"
  region  = "europe-west1"
}

resource "google_iam_deny_policy" "deny" {
  provider     = google-beta
  parent       = urlencode("cloudresourcemanager.googleapis.com/folders/1234567890")
  name         = "my-deny-rule"
  display_name = "Deny Rule for trying it out"
  rules {
    description = "Deny Access to confidential resources"
    deny_rule {
      denied_principals  = ["principal://goog/subject/ralph@example.com"]
      denied_permissions = ["dns.googleapis.com/managedZones.list"]
    }
  }
}

```

Something that tripped me up were the permissions required to apply this. I *thought* I was an admin of everything, but still had to add the required permissions (see: https://cloud.google.com/iam/docs/troubleshooting-deny#before-you-begin).

In terms of solving the original problem (ie. broad viewer access, except some specific areas), I'm none the wiser. I think my best bet is just permission the restricted users with a series of `role/<something>.viewer` roles that allow them to view the things they need and nothing more. Figuring out what they need is going to take some time, and it's bound to be wrong at some point in the future, but at least they won't be able to see things they shouldn't.

---
Tags: google-cloud-platform
---