title: AWS SES - Sending limits on shared IP

text:
# Question
Title: AWS SES - Sending limits on shared IP
I am looking to use AWS SES for SMTP Traffic however I can't find an answer anywhere as to what Amazon deems as "High Volume" and therefore would benefit from a dedicated IP Rather than their shared ones.

We plan to send 120,000 a month ( Varying by 40k or so with 100k growth in the next year ).

Up to 100 a second + - ( busy times )

Am I being blind or do they not have conventional thresholds

# Answer
> 1 votes
TL;DR - Based on your calculations you are considered a high volume sender and would be a candidate for a dedicated IP address in AWS SES

I was unable to find a source where AWS specifically enumerates what they consider to be high volume e-mail sending. Other sources vary (or are vague) about what they consider high volumes (see references below). From those sources I think you can safely say 100k e-mails a month is the high volume threshold.

Note that there are several pros and cons to using a dedicated IP address that you should consider as well.

**Pros**

* Good for high volume senders
* Your reputation will be your own
* Your IP is more likely to be safe-listed by ISP's
* You can isolate your reputation by e-mail type, recipient, etc.
* The IP address will never change

**Cons**

* It requires additional setup and time to build reputation (and keep it)
* You manage your own reputation
* Not good for unpredictable sending patterns
* Additional monthly cost

Your dedicated IP address needs to warm up before it will be accepted. You'll also need to keep a steady volume to maintain that reputation.

> For each ISP with which you want to cultivate a reputation, you should send several hundred emails within a 24-hour period at least once per month.

Finally a couple of caveats I saw in the AWS SES Developers Guide:

> If you don't plan to send large volumes of email on a regular and predictable basis, we recommend that you use shared IP addresses. If you use dedicated IP addresses in situations where you're sending low volumes of mail, or if your sending patterns are highly irregular, you might experience deliverability issues.

And:

> Although there's no minimum commitment, we recommend that you lease more than one dedicated IP address in each AWS Region where you use Amazon SES....When you lease more than one dedicated IP address, we distribute those addresses as evenly as possible across the Availability Zones in the AWS Region that you specified in your request. Distributing your dedicated IP addresses across Availability Zones in this way increases the availability and redundancy of your dedicated IP addresses.

**References**
AWS SES Developers Guide
Dedicated vs Shared IP addresses - How they affect email delivery
How to Send High Volume Email
Dedicated vs. Shared Email IP

---
Tags: amazon-web-services, email
---


# Question
Title: How to manage 10+ team environments with Helm and Kubernetes?
I'm looking for feedback on how we can improve our team environment and staging deployment process. We currently have 10 agile teams, and set up our Kubernetes deployments through Helm as follows:

All our helm charts are in a single git repo. At the top layer of the repo we have a collection of bash scripts (`deploy.sh`, `buildSecrets.sh`, and `buildConfigMaps.sh`) and some json configuration files. (`namespaceCharts.json`, `secrets.json`, `configmaps.json`)

The two main configuration files are a long list of secrets, which are encrypted, and a long list of config map environment variables, 80% of which are shared between helm charts. The `namespaceCharts.json` defines which charts need to be deployed into each namespace grouping.

For each environment that we deploy into, we clone the repo, change the values of the `secrets.json` and `configmaps.json` and then run the scripts passing in our base namespace name. (For example, the team name)

This results in us basically creating a series of namespaces such as `teamA-frontend` and `TeamA-backend` each of which contains the right microservices with the connection strings to the correct DB etc.

We are currently hitting a few main roadblocks.
1. It's hard to deploy to multiple clusters or new clusters.
2. It's hard to get a single source of truth for the configurations.
3. It's hard to set up a brand new Team environment because we have to go through the entire configuration file to see what needs to be changed or kept.

I have come up with three ways of moving forward, all of which "feel wrong" to me.
1. Store all the environment configurations in separate "configurations" git repo.
2. Create a configuration service with a DB table defining each environment, and create a pub/sub feature to dynamically grab configuration at runtime, as well as dynamically grab configurations in the .sh scripts.
3. Use ansible or something similar to store the scripts and configurations for each environment.

Are we doing this all wrong? Are we missing some obvious best practice to do this better? Are one of those three ideas actually the right approach?

# Answer
> 1 votes
My 2 cents. We have similar requirement like you do. We end up having multiple values files in the form of `values-<namespace>.yaml` and it should come with the defaults set to `values.yaml`

We also have some shell scripts to deploy to the namespace. Something like this deploy.sh --namespace app.

It is not without its problem, the number of files are steadily growing. But so far we are ok with it.

# Answer
> 1 votes
Checkout flux(https://github.com/weaveworks/flux/), it introduces a new kubernetes crd called HelmRelease, in which you can include your helm release info(like the chart repo, name, version, namespace, and values. You can even pull your charts from a git repo).

The flux controller monitors a git repo of your choice(you can monitor a path inside the repo as well, which I think might best suit your requirements). And applies all the yaml/json files(including HelmRelease definitions) found in that path periodically.

When a new HelmRelease definition is created, this other component of flux called helm-operator takes over and try to create/update your helm release to the desired state.

The nice thing about this is you can define the desired cluster state using various yaml/json files and flux will try to keep it in sync. If you want to crate a new cluster, just copy the files to a new folder, deploy your cluster, configure flux on it with the new folder path and it'll do the rest of the work for you. Mind you flux is not a full ci/cd solution, so you can't run your shell scripts in it, you will have to find a different way to encode that information into yaml files.

Take a look here(https://github.com/stefanprodan/gitops-helm) for a minimal git repo.

# Answer
> 1 votes
I have made the scripts and tools that we use to solve these problems opensource on GitHub calling it OCD. You mention that you have thought the following feels wrong:

> * Store all the environment configurations in separate "configurations" git repo
> * Use ansible or something similar to store the scripts and configurations for each environment.

With OCD we use Helmfile which makes it easy to manage many helm release in either one of a small number of simple yaml file. If you are familiar with Ansible then it gives all the benefits of that tool but it only manages helm releases. It makes total sense to put the `helmfile.yml` configuration into Git. These act as the playbooks to setup your environment. They shouldn't live in the application repos it should be in one or more "environments repo(s)".

We have created a generic chart to install secrets and another to install configmaps. Your folder structure might look like:

```
|____config
| |____nexmo.yaml.secret // <- gpg encrypted secret
| |____reepay.yaml.secret
| |____backoffice-db.yaml.secret
| |____helmfile.yaml // <- hemlfile manages 3 secret releases and inline configmaps
|____backoffice-app
| |____helmfile.yaml // <- hemlfile manages 1 app
|____frontoffice-app
| |____helmfile.yaml // <- hemlfile manages 1 app

```

In the above example, we have three helmfiles. There is one that manages shared configuration and one each for two apps that use the shared config. Since shared config is going to be multipe secret and configmaps it just uses the two generic charts. It uses the generic secret chart to stamp out many separately encrypted secrets using different values files. Likewise, one generic configmap chart is used to stamp out many configmaps by feeding it different values.

With OCD you setup webhooks on your config repo to point at an adnanh/webhook. When you push a change to the config repo the webhook fires. The webhook runs a script to checkout the config repo, decrypt the secrets, then does `helmfile apply` in each folder that has a `helmfile.yaml`. That does a helm diff on every release in each helmfile and so will upgrade any secrets, configmaps or application deployments where the yaml has changed.

We have a wizard script to install the webhook into a new environment. This means that a team can spin up a new environment using the config in git quickly. Since it is loading the config from git it is an accurate process. Here is a video of setting up a new environment in six minutes which would be quicker if I was not making a video.

Helmfile reduces the burden of having to go through all the files to try to find all the settings to change for a new team environment. The `helmfile.yaml` captures whatever you are overriding over the chart defaults. All the configmaps and secrets are organised to be easily found in one place. Helmfile has quite a lot of capabilities that make managing config and managing many helm releases easier.

We happen to be using openshift kubernetes so our application deployment chart for microservices uses openshift specific yaml. Yet the secrets and configmaps charts are vanilla k8s. So the OCD approach and scripts to manage config should work with any k8s distro. We plan to have a 1.0 release of OCD in the next couple of weeks. I was considering doing a demo of running OCD on vanilla k8s to manage configuration. So I would be really interested in your thoughts and perhaps we can collaborate. Just raise an issue on the ocd-meta github asking about running it on the version of k8s you are using.

# Answer
> 0 votes
I would recommend using something like ArgoCD \- a declarative, continuous delivery for Kubernetes manifests.

With ArgoCD you can:

* structure the Kubernetes objects into separate repositories instead of one huge repo and therefore assign permissions as needed,
* set up RBAC permissions around ArgoCD projects, so each team can control only "their" objects,
* orchestrate deployments into multiple clusters from a central place if desired.

ArgoCD works with plain Kubernetes manifests, Helm charts, Kustomize dynamic manifests ... just to scratch the surface.

---
Tags: kubernetes, configuration-management, helm
---


title: Using helm to manage an "environment?"

text:
# Question
Title: Using helm to manage an "environment?"
I'm new to Kubernetes, and hence helm, so I've been reading a lot, and teaching myself by setting up a cluster, services, deployments, and pods. In my case I use AWS EKS.

I now at least understand each piece from a technical point of view. But now I want to put it all together in managing an "environment." I believe this is the idea behind GitOps.

Say I'm a QA person and am testing a "staging" environment. This environment comprises a number of microservices with their own respective versions. So logically I can represent it this way

```
/staging-2021-03-06-4 # 4th build of staging environment on 2/06/2021
- microservice1:1.1.0
- microservice2:1.2.0
- microservice3:1.3.0

```

When we build the next staging, e.g, `staging-2021-03-06-5`, anyone of those microservices versions can change.

But now I, the QA guy, find an issue with above `staging-2021-03-06-4` build, and I ask a developer to look at it. I want to be able to just tell the developer to "check out" the `staging-2021-03-06-4` environment so he/she can have **exactly** the environment that I have to reproduce the issue.

I'm thinking I can set up this idea of "environment" with `helm`, since the "environment" is nothing but a package of packages? Is this the correct utility of `helm`? If so, do I just set up a `staging-2021-03-06-4` helm chart that has dependencies of other charts, and those other charts are for each microservice?

I'm also reading How to manage 10+ team environments with Helm and Kubernetes? since it applies to me also, but that describes a different (additional) problem I'm trying to solve with `helm`.

**Update**: I think I'm essentially wanting GitOps, so I'll read up on that.

# Answer
> 1 votes
Helm charts should not be built for a specific environment, IMHO. You can see that many of the official helm charts have a set of default values and they are configurable from outside. You also can create a helm chart like that and declare your deployment + environment-specific values with another tool like Helm Operator or ArgoCD. In this way, you can both simplify your build pipeline and increase dev/prod parity. You can use that one chart and **promote** the same build into different environments so you can be sure that different environments are identical except the configuration.

Example ArgoCD application manifests:

**production-application.yaml:**

```
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: production-yourservice
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourorganisation/helm-charts
    path: yourservice
    targetRevision: master
    helm:
      values: |
        environment: production
        replicas: 10
  destination:
    server: https://kubernetes.default.svc
    namespace: production-yourservice

```

**staging-application.yaml:**

```
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: staging-yourservice
  namespace: argocd
spec:
  project: default
  source:
    repoURL: https://github.com/yourorganisation/helm-charts
    path: yourservice
    targetRevision: master
    helm:
      values: |
        environment: staging
        replicas: 1
  destination:
    server: https://kubernetes.default.svc
    namespace: staging-yourservice

```

# Answer
> 0 votes
1. You could set up the "environments" as Chart of charts using Helm
2. or (my prefered way) utilize ArgoCD with Kustomize \- having "default" manifest for each service, creating the "environment" with a Kustomize file, referencing the "default" manifests and overriding needed values.

(also see my other answer on how to use ArgoCD)

---
Tags: kubernetes, helm, kubernetes-helm
---


# Question
Title: What happens when two jobs produce artifacts with conflicting locations?
In my GitLab CI log I pipeline with three stages

1. `build` creates a distribution
2. `semantic-release` which bumps the version
3. `publish`

The problem occurs in `publish`. Sometimes I see the version unbumped. You can see here that `semantic-release` has the higher number 148288 and it's running BEFORE `build` which overwriting the files. This means this job results in failure,

```
Downloading artifacts for semantic-release (148288)...
Downloading artifacts from coordinator... ok        id=148288 responseStatus=200 OK token=xFzfAzW6
Downloading artifacts for build (148285)...
Downloading artifacts from coordinator... ok        id=148285 responseStatus=200 OK token=zxTTXZoA

```

But sometimes the same pipeline elsewhere is succeeding,

```
Downloading artifacts for build (113579)...
Downloading artifacts from coordinator... ok        id=113579 responseStatus=200 OK token=Cfqjv8bG
Downloading artifacts for semantic-release (113665)...
Downloading artifacts from coordinator... ok        id=113665 responseStatus=200 OK token=GAyN8S9M

```

Notice on this one the `build` job comes before `semantic-release`. That means the older artifacts are replaced. What happens if the artifact in `semantic-release` and `build` conflict by path? Is this defined?

# Answer
## Bug #324412

With the bug, basically the rule of thumb is this,

> A job can rely on an artifact produced by a previous job only so long as no jobs previous to that one produced the same artifact.

The bug report suggests using `dependencies` to only download specific artifacts.
> 0 votes

---
Tags: gitlab
---


# Question
Title: Is there a way to install a private key for a user with cloud-init?
I have a user that needs to authenticate against a company source repository when using `git clone`. To set this up for the user I need to specify a users private key (not the host private key in `/etc`). Is there a method to do this?

The user it configured with `system_info` in cloud-init, which doesn't have a mechanism to install the user's private key.

Note: Let's say you're provisioning a a new machine and adding a user bob on it. How do you install a private key for a bob such that he can authenticate with something using ssh?

# Answer
> 0 votes
# Update

Oops, **Don't do this**. It was pointed out to me that this was wholly insecure as `curl http://169.254.169.254/latest/user-data` will show you any unprivileged user the private keys. The data gets saved as `/run/cloud-init/instance-data.json`

# Original post

There is no module to make this easier, and there is no argument under `system_info` (how you add and configure the user) to ease the ability to configure the user's SSH keys. The way I went about this was adding something like this in my `main.tf` to populate the variable `ssh_keys_user`

```
ssh_keys_user = {
  write_files = [
    {
      path        = "/home/ecarroll/.ssh/id_rsa"
      content     = file("./ssh/user/cp-terraform-user-id_rsa")
      owner       = "ecarroll:ecarroll"
      permissions = "0600"
      defer       = true
    },
    {
      path        = "/home/ecarroll/.ssh/id_rsa.pub"
      content     = file("./ssh/user/cp-terraform-user-id_rsa.pub")
      owner       = "ecarroll:ecarroll"
      permissions = "0644"
      defer       = true
    },
    {
      path        = "/home/ecarroll/.ssh/id_ecdsa"
      content     = file("./ssh/user/cp-terraform-user-id_ecdsa")
      owner       = "ecarroll:ecarroll"
      permissions = "0600"
      defer       = true
    },
    {
      path        = "/home/ecarroll/.ssh/id_ecdsa.pub"
      content     = file("./ssh/user/cp-terraform-user-id_ecdsa.pub")
      owner       = "ecarroll:ecarroll"
      permissions = "0644"
      defer       = true
    },
    {
      path        = "/home/ecarroll/.ssh/id_ed25519"
      content     = file("./ssh/user/cp-terraform-user-id_ed25519")
      owner       = "ecarroll:ecarroll"
      permissions = "0600"
      defer       = true
    },
    {
      path        = "/home/ecarroll/.ssh/id_ed25519.pub"
      content     = file("./ssh/user/cp-terraform-user-id_ed25519.pub")
      owner       = "ecarroll:ecarroll"
      permissions = "0644"
      defer       = true
    }
  ]
}

```

Then what I did was wired it into my cloud-init like this,

```
write_files:
${ yamlencode( ssh_keys_user.write_files ) }

```

I generated these files with a Makefile like this,

```
user/cp-terraform-user-id_ecdsa:
        -mkdir user 2> /dev/null;
        ssh-keygen -C "User key for SSH authentication to repos" -N "" -b 521 -t ecdsa -f "$@";
        touch "$@";

user/cp-terraform-user-id_ed25519:
        -mkdir user 2> /dev/null;
        ssh-keygen -C "User key for SSH authentication to repos" -N "" -t ed25519 -f "$@";
        touch "$@";

user/cp-terraform-user-id_rsa:
        -mkdir user 2> /dev/null;
        ssh-keygen -C "User key for SSH authentication to repos" -N "" -b 4096 -t rsa -f "$@";
        touch "$@";

```

This works fine. Then I just added the `.pub` files to BitBucket and GitLab.

# Answer
> 0 votes
If you don't mind the keys being generated on boot, you could potentially use the phone-home module to send the newly generated pubkeys wherever you need them to go:

The current example for this module might be a good starting point.

```
    url: http://example.com/$INSTANCE_ID/
    post:
        - pub_key_dsa
        - pub_key_rsa
        - pub_key_ecdsa
        - pub_key_ed25519
        - instance_id
        - hostname
        - fqdn
    tries: 5

```

I haven't done this before, but if "keys generated on boot" fits your requirements, something like this *should* work.

---
Tags: terraform, ssh, terraform-module, cloud-init
---


# Question
Title: Is it possible to split traffic by fixed ratio among two Deployments using only Service
Context: Kubernetes (1.23), 2 Deployments of the same app with different version/config/etc., 1 Service

Let's say I have a web application deployed in Kubernetes using a Deployment. I want to add a new (canary, A/B testing) version of the same app as a new separate Deployment, so we're not touching the original Deployment. I want to set fixed ratio of traffic between these two Deployments.

Is it possible to achieve this using Service, or other Kube native objects? (Not by utilizing a service mesh).

# Answer
> 0 votes
Closest solution I found so far is creating those two deployments with such number of pod replicas, that corresponds to the required ratio (or close to it). I.e. if the required ratio is 7:3, Deployment A would have 7 pods, Deployment B would have 3. Service being simple round-robin, should ensure the requests spread according to the ratio.

It's not a very elegant solution though, this might be quite a big waste of resources in some cases.

---
Tags: kubernetes, deployment, load-balance
---


# Question
Title: How can I set up Deployment to run at least one pod on each node?
I need to set my deployment to have at least one pod on each node so that the system will continue uninterrupted in case the node goes down. I can set this to run one pod per node with daemonset or podAntiAffinity. But what I really want is that there is at least one pod in each node and all pods can be scheduled if there are more replicas than the number of nodes.

How can I do that? Thank you for your answers

# Answer
> 2 votes
You can use `topologySpreadConstaints` for spreading Deployment pods evenly across nodes, regardless of the number of pods. Kubernetes documentation contains multiple examples, as well as following article - Kubernetes: Evenly Distribution of Pods Across Cluster Nodes.

The `.spec.teplate.spec` would look something like:

```
    spec:
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              type: dummy

```

---
Tags: kubernetes, deployment, high-availability
---


# Question
Title: Growing local development environment issues
Where I work we've been adding microservices for different purposes and the local development environment is becoming difficult to setup. Services have too many environment variables to configure and usually there's not enough memory avaiable to run them.

We plan to fix these issues. I understand it's a matter of architecture and DevOps mostly. One way we've thought of is to create a proper service registry that allows easier setup and opens the door to, for example, have some services running locally and others in the cloud. All wired together with the service registry.

Another option could be to stub some of the dependencies with something like https://wiremock.org/ but it seems too limited and difficult (?).

I wanted to ask, what other strategies are there to manage growing development environments?

# Answer
I recommend running all the dependent microservices on cloud and deploy only the microservice that is under active development in your laptop.
> 3 votes

# Answer
There are several options for handling this and they are not exclusive of each other:

1. Giving your developers machines with lots of memory and CPU, and probably GPU, is a quick way to solve problems if you have money to throw at things.
2. Use a VPN to let users access resources in the cloud. Hopefully you have some sort of pre-prod (aka staging) environment so they can develop against something that isn't needed by production.
3. Shrink your containers. Many folks start with containers that are based on an OS. This will eat lots of memory in production and development spaces. If you can build lighter containers it will save you money on developer machines and cloud resources. There's tons of info on the web about ways to get your container down to what it only needs.

I'm also a fan of inventories like service catalogs, but unless the catalog helps you make cloud services available to local devs I don't see how it is going to solve your memory contention issues.
> 1 votes

---
Tags: environments
---


# Question
Title: Should a pipeline fail if early termination is desired?
Normally a pipeline breaks down into stages such that each stage has one or more jobs. In order to stop the pipeline from running conditionally, you simply fail that stage of the pipeline.

For example let's assume I have a pipeline with four stages,

```
[Build] -> [Test] -> [Version Bump] -> [Publish]

```

Now, let's assume the `Version Bump` may or may not bump the version number which means the `Publish` stage may or may not run. In order to model this in GitLab we would normally fail the `Version Bump` stage at the point that we know the `Publish` should not run. But one of my co-workers does not like the term "fail" here.

Given the above circumstance, is it a normal flow to have a stage pass unconditionally, and to have a subsequent stage conditionally execute?

Or, is the pipeline failing just the way in which a pipeline should conditionally terminate?

# Answer
I'm going to take an opinionated approach based on the 12 factor app pattern

Typically, stages should execute a function, with predictable inputs and outputs.

If stage 3 executes the function "compute next version" and the output of that function is "The next version is the same as the previous version", then even if there is no version change, the stage has successfully executed it's function. Again, the inputs and outputs are predictable.

Similarly for the fourth stage (Publish). If you have built the software, but the version hasn't changed, you still have a new artifact. It has a different **build**, but the same **version**. This could happen if for example you change the environment or configuration it's deployed into. As the 12factor app puts it:

> The twelve-factor app uses strict separation between the build, release, and run stages. https://12factor.net/build-release-run

So your publish function *should* deliver a new artifact, with the **same** version, but different build metadata (*ie* pipeline execution number, date, *etc*.)

In either case, no you do not need to fail the pipeline (nothing has broken, no changes are necessary) and yes you should execute the last stage, which should implement the actual deployment strategy you use.
> 1 votes

---
Tags: gitlab
---


# Question
Title: Can I create a pipeline stage which unconditionally passes but one in which a subsequent stage conditionally runs?
This is a follow up to Should a pipeline fail if early termination is desired?. In that question, I'm asking about best practices and whether this request is sound. In this, I just need to know how best to get it done.

---

Normally a pipeline breaks down into stages such that each stage has one or more jobs. In order to stop the pipeline from running conditionally, you simply fail that stage of the pipeline.

For example let's assume I have a pipeline with four stages,

```
[Build] -> [Test] -> [Version Bump] -> [Publish]

```

Now, let's assume the `Version Bump` may or may not bump the version number. Is there a way to have the `Version Bump` pass *regardless* of whether the number was bumped, and to have Publish run conditionally if the version was bumped?

# Answer
> 0 votes
To paraphrase:

> Is there a way to have *(a job)* pass regardless of whether the *(something has changed)*, and to have *(another job)* run conditionally if *(something was actually changed)*?

The first part (run a job whether something changes or not) can be done by either changing the script so that it returns a zero exit code (success), even if something hasn't changed (which can be interpreted as "fail"), or you can continue the pipeline regardless of whether that job has failed or not. This is typically done with the `allow_failure` keyword on gitlab jobs.

If you're using `semantic-release`, the decision to cut a new release is done by default with the `commit-analyzer` plugin, which analyses every commit in the change to determine, according to your release policy, whether a new version is required.

In your particular case you're dealing with version numbers. So the version changes, publish. Else, do not publish. So a common trick is to compare the `nextVersion` (the version you have computed in this pipeline) with the existing tags on the repo or published versions of the artifact to determine whether `nextVersion` \> `latestVersion`.

If `nextVersion` \> `latestVersion` -\> publish

else exit 0

---
Tags: gitlab
---


# Question
Title: How to run a container from within another container while also mounting a volume from the first container?
I'm running a build in Jenkins which is running inside Kubernetes.

When the job starts, a container starts in the cluster and the build runs inside it.

A step in the build requires building an RPM using rpmbuild and for that matter I use this docker image.

So what happens is that when the job starts, it fires up a jenkins slave jnlp dind container with a volume mount of `/var/run/docker.sock:/var/run/docker.sock`, it then pulls the code from git and running a Makefile to build the code.

When the build process finishes, I want to fire up another container (the one containing rpmbuild), so from within the jnlp dind container I run:

```
docker run -e WORKSPACE=/home/jenkins/agent/workspace/product-Provisioner-3.17 -e PROV_NAME=product-provisioner -v ${WORKSPACE}:/srv --privileged -t registry/rpmbuild:latest /usr/bin/rpmbuild -v -bb --clean --define '_workspace /srv' /srv/provisioner/SPECS/${PROV_NAME}.spec

```

But for some reason the volume is not mounted and when I do `kubectl exec -it jnlp-container sh`, I see that the `/srv` directory doesn't contain the files in the ${WORKSPACE}.

Running the rpmbuild container just with --help works so the docker inside docker inside docker works, just the volume isn't mounted properly.

As far as I know, running a container from within another container should be doable.

What could be the reason that `/srv` is not mounted properly?

# Answer
So this happens to me often and I'm sure there is a better solution, such as using docker volumes but this is what I do.

So first the reason that it doesn't work as expected. Is because we think of the relationship between the first container and the second container being "nested" like this...

```
k8 -> jenkins -> rpmbuild

```

Right?

Thats what I thought was going on... but its actually like this:

```
k8 -> jenkins
k8 -> rpmbuild

```

So they are not "nested", they actually run in parallel to each other side by side.

So when you try to mount a path from inside the jenkins container like `/var/jenkins_home/some_workspace/rpmbuild` that gets sent back to the docker daemon running on the K8. And it's like... I dont have that path on me.

So what I do on my non-k8 setup, is I mount the jenkins home to someplace like `/tmp/jenkins_home/`, but dont use tmp this is just an example.

Jenkins = `docker run -v /tmp/jenkins_home:/var/jenkins_home`

Now when I want to start the rpm container:

rpmbuild = `docker run -v /var/jenkins_home/some_workspace/rpmbuild`

Now your rpmbuild will have the files like you expected it to. I'm not sure how you will implement this on K8's but what I did was pass the location on the host where I'm mounting the jenkins home directory to the jenkins container as an environment variable like `-e DIND_JENKINS_HOME=/tmp/jenkins_home`. Then for the pipeline logic to create the path to pass to the rpmbuild container:

```
    List parts = env.WORKSPACE.split('/workspace/')

    rpmWorkspace= "${env.DIND_JENKINS_HOME}/workspace/${parts[1]}/product-Provisioner-3.17"

```

Then just start rpmbuild with `docker run -v ${rpmWorkspace}:/srv`

This took me a long time figure out and it was incredibly frustrating. Googling did not really help. So I hope this clears things up for you and bravo for you coming forward to ask this question.
> 1 votes

---
Tags: docker, jenkins, kubernetes, kubernetes-volume
---


# Question
Title: Building a pipeline to recognize changes in a terraform template compared to the git commit before and having it apply changes using specified command
Looking to get pointed in the right direction:

I am trying to build a pipeline that will recognize changes in a remote terraform template that will recognize changes compared to the previous git command before mine and have it only apply those changes by using specified commands:

```
terraform get -update
terraform plan

```

I also want it to display to the user and have user confirmation to proceed and have 2 min for total user response

Still green with Jenkins but so far:

I have the git fetch stage to grab the branch on the remote repository

A terraform init stage to initialize terraform

**Can anyone point me to the right direction to detect the changes and apply them with the commands?**

# Answer
> 0 votes
Your question is really vague and depends a lot of information that you didn't supply.

Some of things you didn't mention were what git branching strategy you are using.

Typically you are not going to run `terraform apply` on every single commit. Instead you are going to have people open a PR and on that PR you will run linting and other static code checks like `tfsec`, `terraform validate` etc.

Then once that PR is merged to the `master/main` branch of your repo, you will run `terraform apply`.

This is still a beginners pipeline though because it has lots of problems still, like what happens when you merge to master but your deployment fails.

You need to talk to your team about how they want to build and test there software so that you can pick a branching strat and then design the pipeline around that.

---
Tags: jenkins, jenkins-plugins, bitbucket, bitbucket-pipelines, jenkins-blueocean
---


# Question
Title: how to specify the dependency in the deployment?
This is a design question. I am not sure this is the right place. We have multiple microservices deployed using a blue/green pattern in Azure DevOps pipelines.

What I am trying to achieve is I want to deploy (trigger pipeline) services based on the service version.

As an example, let's say we have three services `A`, `B` and `C` which have an internal dependency graph, and different versions, *i.e.* `A@1.2`

If I want to deploy, say service `A@1.2` and for that to work we want service `B@2.1` and service `C@1.1`.

So when I decide I want to deploy service `A@1.2` automatically I want to trigger the pipeline for service `B@2.1` and service `C@1.1`.

So, we want to deploy exact version of those services on a set of events, such as upgrading the underlying k8s cluster, or similar changes.

I tried to trigger pipeline from one pipeline to other(pipeline trigger), but I am not able to pass the exact same version what the dependency service needs to be deployed

# Answer
> 1 votes
If I understand correctly this question is based on the problem I initially outlined here - https://worklifenotes.com/2020/03/04/microservices-combinatorial-explosion-of-versions/

So we've been working on this for the past 2 years. Key things in the approach that we find performing well so far are the following:

1. All components should be built independently. Triggering one pipeline from another pipeline is an anti-pattern - which usually results in a messy process.
2. You need an integration layer on top of individual builds to mix and match components together and record the resulting metadata. In many cases taking always latest is ok. In the simplest case, you can make it work with a commit to a dedicated git repository to capture working set of inter-version links. In some cases (regulated environments) more is needed - say you are required to pin some component at a certain version or have more automation or better audit in place. This is where specialized tools like the one we are building (Reliza Hub) or alternatives (i.e. DeployHub) may be used for the purpose.
3. Regarding the deployment - at deploy time your instance would query source of truth which holds version permutations (again, in the simplest case - this would be a dedicated git repository - see GitOps, or in more complex case - a specialized tool would be in the mix). Essentially, here you are establishing an authoritative source of what version goes to which environment based on some rules.

---
Tags: azure-devops, deployment, azure-kubernetes-service, bluegreen
---


# Question
Title: "The plain HTTP request was sent to HTTPS port" issue with nginx-ingress in rancher/k8s
I'm running Rancher v2.6.3 and I have a cluster with a Django workload built all on top of AWS. The deployment has a cluster ip port set up, I have an ingress created with a prefix rule pointing at my deployment. I've installed the nginx-ingress helm chart with the following service annotations:

```
service.beta.kubernetes.io/aws-load-balancer-protocol: http
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: my-cert-arn

```

Neither of these annotations are present but I've tried adding and removing them both together and individually at various points. I believe one is deprecated:

```
backend-protocol: "HTTP"
secure-backends: "false"

```

All of my infrastructure in AWS looks good and HTTP works just fine, however when I try to visit the site using SSL, I get a 400 Bad Request error showing "The plain HTTP request was sent to HTTPS port". The SSL certificate on that page is valid.

What am I doing wrong regarding HTTPS traffic?

# Answer
I found a resolution to my issue. I deleted the installed nginx-ingress app (chart). Went back to the marketplace to reinstall it and when I was given the opportunity to edit my configuration, I edited the service block with the following values (leaving everything else the same):

```
  service:
    annotations: {
      service.beta.kubernetes.io/aws-load-balancer-ssl-cert: my-cert-arn,
      service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "http",
      service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https",
      service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: '3600'
    }
    targetPorts:
      http: http
      https: http

```

Everything is behaving correctly now.

This link provided the details I needed to make the proper changes: https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx#aws-l7-elb-with-ssl-termination

It's for the ingress-nginx repo, however the helm chart for nginx-ingress in Rancher v2.6.3 provides kubernetes-ingress. It seems to follow the same structure at the moment.
> 3 votes

---
Tags: kubernetes, amazon-web-services, ingress, ssl, rancher
---


title: Terraform get list index on for_each

text:
# Question
Title: Terraform get list index on for_each
Terraform newbie here. I'd like to iterate a list using `for_each`, but it seems like the key and value are the same:

```
provider "aws" {
  profile = "default"
  region  = "us-east-1"
}

variable "vpc_cidrs" {
  default = ["10.0.0.0/16", "10.1.0.0/16"]
}

resource "aws_vpc" "vpc" {
  for_each             = toset(var.vpc_cidrs)
  cidr_block           = each.value
  enable_dns_hostnames = true
  tags                 = { Name = "Company0${each.key}" }
}

```

I'd like the tag Name to be `"Name" = "Company01"` and `"Name" = "Company02"` but according to terraform apply,

I get: `"Name" = "Company010.0.0.0/16"` and `"Name" = "Company010.1.0.0/16"`

What am I missing?

# Answer
Found an easy solution using the index function:

```
tags = { Name = "Company0${index(var.vpc_cidrs, each.value) + 1}" }

```
> 3 votes

# Answer
Terraform offers two resource repetition mechanisms: `count` and `for_each`. The main difference between these is how Terraform will track the multiple instances they create:

* When using `count`, each of the multiple instances is tracked by a number starting at 0, giving addresses like `aws_vpc.vpc[0]` and `aws_vpc.vpc[1]`.
* When using `for_each`, each of the multiple instances is tracked by a string which could be obtained either from a key in a map or from an element in a set, giving addresses like `aws_vpc.vpc["10.0.0.0/16"]` and `aws_vpc.vpc["10.1.0.0/16"]`.

The most common reason to choose `for_each` rather than `count` is to ensure that the order of whatever collection of values is driving the repetition is not significant. In your specific case, that would produce the same result regardless of whether `var.vpc_cidrs` is `["10.0.0.0/16", "10.1.0.0/16"]` or `["10.1.0.0/16", "10.0.0.0/16"]`, because the instances would be tracked by the CIDR block string and the index in the list is totally ignored.

With that all said, the reason why the index isn't directly available when using `for_each` is that this would defeat the main purpose of using `for_each`: it would make the order significant again. In your specific situation, if you used the index in the tag Name then changing the order of the list would cause Terraform to plan to swap the tagging on those two VPCs:

```
 # aws_vpc.vpc["10.0.0.0/16"] will be updated in-place
 ~ resource "aws_vpc" "vpc" {
     ~ tags = {
         ~ Name = "Company01" -> "Company02"
       }
     # ...
   }

 # aws_vpc.vpc["10.0.0.0/16"] will be updated in-place
 ~ resource "aws_vpc" "vpc" {
     ~ tags = {
         ~ Name = "Company02" -> "Company01"
       }
     # ...
   }

```

Perhaps that is fine for your case, if those numbers are just there to create some uniqueness and it doesn't matter if they get reassigned over time as CIDR blocks are added, removed and reordered in the input variable. If that's true, then we can derive a value that contains the original index and use that as part of `for_each` like this:

```
resource "aws_vpc" "vpc" {
  for_each = { for i, b in var.vpc_cidrs : b => { index = i } }

  cidr_block           = each.key
  tags = {
    Name = format("Company%02d", each.value.index + 1)
  }
}

```

If you consider the position in the list to be truly significant though -- for example, if you'd want swapping two elements in the list to change the `cidr_block` values on the two existing objects rather than to re-tag them, it would be more appropriate to use `count` so that Terraform tracks them by the indices itself:

```
resource "aws_vpc" "vpc" {
  count = length(var.vpc_cidrs)

  cidr_block           = var.vpc_cidrs[count.index]
  tags = {
    Name = format("Company%02d", count.index + 1)
  }
}

```
> 4 votes

# Answer
You could use without index:

```
resource "aws_vpc" "vpc" {
  for_each = { for i, b in var.vpc_cidrs : i => b }

  cidr_block           = each.value
  tags = {
    Name = format("Company%02d", each.key + 1)
  }
}

```

each.key = index number
> 0 votes

---
Tags: terraform
---


# Question
Title: How to figure out optimum location for server for least latency to a target service?
Given a service URL / IP, how can we find the optimum location to spin up a server at? I suppose generally if we assume the service is on AWS, google cloud or azure, setting up the server in the same region as the target service should get us least latency?

So in cases of these public clouds, how can you find out which region the target service is hosted at? I have thought like we could try to get the IP from DNS, and then get a location with reverse IP geolocation and use the region containing or near the location we got from the reverse IP geolocation. But I am not sure how accurate would the result be from these random reverse IP geolocation services.

And in cases of not public clouds, generally how would you figure out which region to place your server at?

# Answer
> 0 votes
I think the geolocation of the IP address is generally quite accurate - this is from 15 years of my own anecdotal experience. I can only recall ever having seen an IP address having moved to th "wrong" location once some time in the 2000s.

In terms of what region is "nearest" - it seems geography is secondary to geopolitics and how fibre-cables are layed.

This github project: https://github.com/turnkeylinux/aws-datacenters maintains a list of fibre cables and AWS locations, I've extended it before to look at the latest AWS regions and so it can be used for other public cloud providers as well.

Some (maybe) surprising latency results I found working with people:

* Norway seems to have better latency to North America than to central Europe (feedback from in-country technical person)
* Israel better latency to Frankfurt than to Bahrain or Mumbai

Also given this story and others as part of fallout from the Snowden releases https://www.dw.com/en/call-me-a-german-satellite-and-internet-company-wants-answers-from-the-nsa/a-17571811 I would assume that for low latency to most of Northern Africa and Western Middle East, you are best to choose a server location in Germany or Italy.

---
Tags: amazon-web-services, azure, google-compute-engine
---


# Question
Title: How to use the Shared Workspace plugin in a Jenkins pipeline?
I want to use the Shared Workspace plugin, so I set up the following key/value pair in my `Manage Jenkins -> Configure System -> Workspace Sharing`

```
Name: sandbox-workspace
Repository URL: git@bitbucket.org:mycompany/myrepo.git

```

Now I want to share the above workspace between two separate pipeline jobs `Job1` and `Job2`, where `Job1` triggers `Job2` so they're guaranteed to never run concurrently. Each job may/can run on different nodes. What is the pipeline syntax for using the above shared workspace?

# Answer
> 1 votes
Considering that plugin hasn't been updated in 7 years, it almost certainly doesn't have Pipeline support - Pipelines were likely either nonexistent or not widely used last time that plugin received an update.

You can still try to use it in a Pipeline job. Jenkins has a snippet generator feature to automatically generate Pipeline code using Freestyle job steps in the web UI. However there is no guarantee that the Pipeline code output by this tool will work identically to the equivalent Freestyle configuration.

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: Docker cloud hosting with persistent storage
I need to find a cloud provider to host a simple docker service but with a persistent storage attached.

I know I can achieve this using GCE (VM + docker + volume mounted from VM) but this looks a bit too complicated (not complex though), and I was wondering if someone here knew a cloud provider providing this kind of solution, in a very simple way.

Thanks

# Answer
I'm not aware of any cloud providers which do this as you describe. I'd wager that it's due to the simplicity of a user setting it up on their own, compared to the headache of them taking that task on themselves and lack of demand for non-kubernetes docker management.

Docker setup is super simple and broadly documented, so I'd still recommend a VM of your choice from a reputable host. Once logged into the VM, you can have docker up and running in less than 1 minute.
> 0 votes

---
Tags: docker, cloud, persistent-volumes
---


# Question
Title: While pulling windows docker image from private registry, Docker trying to download some layers from internet
I created a Windows based docker image (asp.net web project) and pushed it to my private image registry.

when i pull the image from another windows server (offline and there is not a proxy), i'm getting the following error. As you can see the first two lines (system fs layer) tried to download many times, but it couldn't.

If I create an image and push the private registry, why does the pulling process try to download some layers from the internet?

I checked the ip address (204.79.197.219). It belongs to Microsoft.

How to resolve the issue?

Thank you for help

```
4612f6d0b889: Retrying in 1 second
1bd78008c728: Retrying in 1 second
5fd81474115b: Download complete
b887704f8144: Download complete
b0ba1d0b65f3: Download complete
0e8416bc92db: Download complete
3cd07ed78b88: Download complete
14ed468ea106: Download complete
f695358ea9e2: Download complete
5d1118c7c751: Download complete
f66d3542ac00: Download complete
0afb205dafe5: Download complete
4f64ca1251ee: Download complete
3093e4c70a9a: Download complete
93742327496f: Download complete
7602cf05342b: Download complete
3e18c730e090: Download complete
c870f86c0865: Download complete
28442d8cdc02: Download complete
4519d453c4e2: Download complete
8399eeba9588: Download complete
e1d4cea1955e: Download complete
c3ad37fcb432: Download complete
a3b2cb480d5a: Download complete
493303eb4a29: Download complete
1e3cc68c5708: Download complete
0e458143e888: Download complete
d2ca9661d1fb: Download complete
700fd2d72ce3: Download complete
dial tcp 204.79.197.219:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.

```

# Answer
this could be the reason - https://docs.docker.com/registry/recipes/mirror/

Configure the Docker daemon Either pass the --registry-mirror option when starting dockerd manually, or edit /etc/docker/daemon.json and add the registry-mirrors key and value, to make the change persistent.

More details in the link
> 1 votes

---
Tags: docker, windows, docker-registry
---


# Question
Title: Does the AWS Admin user have the eks:AccessKubernetesApi permission?
I am logged in to AWS as the AWS Admin, with Administrator access.

How do I find out if this user has the `eks:AccessKubernetesApi` IAM permission?

# Answer
> 1 votes
There are a few ways to perform the action you're looking for. I'll demonstrate how to accomplish this task from the command line and the console. Using the SDK's (e.g. boto3) are an option too.

From the command line you can use:

```
aws iam simulate-principal-policy --policy-source-arn <arn of role or user> --action-names eks:AccessKubernetesApi --query 'EvaluationResults[].EvalDecision'

```

It will return the following if you have access :

> \[ "allowed" \]

From the console use the IAM Simulator
Pick the Role/User/Group that you wish to test:

Then choose the service:

And the action(s):

Click `Run Simulation` and the results will be published:

**References**
Simulate Principle Policy
Access Policies Testing Policies

---
Tags: amazon-web-services, aws-iam, eks
---


# Question
Title: How to run a gitlab child pipeline only on changes in merge requests?
According to the documentation, parent-child pipelines should work well with `rules` \- so i'm trying to setup a parent-child pipeline, where the child pipelines only run when something has actually changed in the sub-service. Using the following setup:

main .gitlab-ci.yml:

```
trigger_service_a:
  trigger:
    include: service_a/.gitlab-ci.yml
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - service_a/*
      when: always

```

service\_a/.gitlab-ci.yml:

```

workflow:
  rules:
    - if: '$CI_PIPELINE_SOURCE == "merge_request_event"'
      changes:
        - service_a/*
      when: always

service_a:
  script:
    - echo "run"

```

However, when i push in a merge request without changes inside service\_a, the pipeline fails with : `trigger_service_a - failed - (downstream pipeline can not be created, Pipeline filtered out by workflow rules.)`

Even if i change something in the service\_a, i still get the same error.

How can i trigger child-pipelines in branches and merge requests only when something has changed in the sub-service?

# Answer
> 0 votes
Why are you using the same rules for both parent and child pipelines? You are just triggering the child service `if anything changes on files inside the child service directory` So, just define rules on child pipeline... It will automatically trigger if any changes happen on `service_a/*`

---
Tags: gitlab, gitlab-ci-runner
---


# Question
Title: Terraform Aws S3 - deny all users except for a specific user
I have a bucket which I need to restrict to a specific user, I have written the following script but it still seems to allow all users to operate on the bucket.

```
resource "aws_s3_bucket" "vulnerability-scans" {
  bucket = "vulnerability-scans"
}

resource "aws_s3_bucket_policy" "vulnerability-scans" {
  bucket = aws_s3_bucket.vulnerability-scans.id
  policy = data.aws_iam_policy_document.vulnerability-scans.json
}

data "aws_iam_policy_document" "vulnerability-scans" {
  statement {
    principals {
      type = "AWS"
      identifiers = [
        aws_iam_user.circleci.arn,
      ]
    }

    actions = [
      "s3:PutObject",
      "s3:GetObject",
      "s3:ListBucket",
    ]

    resources = [
      aws_s3_bucket.vulnerability-scans.arn,
      "${aws_s3_bucket.vulnerability-scans.arn}/*",
    ]
  }
}

```

# Answer
First, let's understand how roles and policies work on AWS. In order for an user to be able to access a bucket, we can allow it in 3 ways:

1. Allow it using an IAM policy attached to the role the user is assuming;
2. Allow it using a bucket policy;
3. The group of the user has the policy attached to it or there is a policy directly attached to the user which allows access to the bucket.

These are explicit `Allow` policies. The user will have access if there is at least on policy from above granting him/her access.

What is important is that an explicit `Deny` takes precedence of an explicit `Allow`. So, if we want to deny access to a specific user, we would want to create a bucket policy with an explicit `Deny`. In order to do this, we can use `NotPrincipal`.

Bucket policy example:

```
{
    "Id": "bucketPolicy",
    "Statement": [
        {
            "Action": "s3:*",
            "Effect": "Deny",
            "NotPrincipal": {
                "AWS": [
                    "arn:aws:iam::1234567890:user/alloweduser"
                ]
            },
            "Resource": [
                "arn:aws:s3:::examplebucket",
                "arn:aws:s3:::examplebucket/*"
            ]
        }
    ],
    "Version": "2012-10-17"
}

```

Terraform code for this policy:

```
data "aws_iam_policy_document" "vulnerability-scans" {
  statement {
    not_principals {
      type = "AWS"
      identifiers = [
        aws_iam_user.circleci.arn
      ]
    }

    effect = "Deny"

    actions = [
      "s3:*"
    ]

    resources = [
      aws_s3_bucket.vulnerability-scans.arn,
      "${aws_s3_bucket.vulnerability-scans.arn}/*",
    ]
  }
}

```
> 2 votes

---
Tags: amazon-web-services, terraform, amazon-s3
---


# Question
Title: Which CI/CD tool(s) should a DevOps newbie learn?
I'm new to DevOps and I'm learning that:

1. The adoption of containerization and GitOps have been clear trends
2. Some CI/CD tools (e.g. Jenkins X) are more geared toward this trend than others (e.g. Jenkins)
3. While some CI/CD tools are limited to CI (e.g. Jenkins) or CD (e.g. ArgoCD) but can be extended to cover both, others inherently cover both (e.g. Jenkins X)

In light of all that, I've been trying to answer these questions:

1. To which extent is it advantageous to have one tool for CI & CD?
2. Which tool(s) would you recommend someone new to start with?

# Answer
Just some freeform thoughts from your question.

For perspective, I run a large enterprise DevOps team that has automated build & deployment for a wide range of large scale services (which run over k8s/AWS primarily). That doesn't necessarily mean I'm good at things, haha (always check your sources) - but I think we're in a very good spot personally =). But the advice comes from practical experience and lots of time spent iterating at least.

* When we hire people we really don't care what CI/CD tool they knew, or what "automation" tool they knew. We just care that they know one well and can explain some pipelines and their challenges in depth.
    + Note that if you're not in k8s, deployment usually involves terraform, cloud formation, ansible, or one of many other tools on top of your CI/CD engine.
* So, rather than overthinking which tool to learn (they evolve constantly), my recommendation is pick a mainstream one and focus on a "hard use case" so that you have to do something complex in it that forces you to learn.
* We use GitLab CI, and previously Jenkins. People in the industry also seem to like Spinnaker, CircleCI, and the equivalent cloud tools (AWS and Azure have all their own). GitHub Actions is also huge now.
* I don't think tools just do CI or CD. That's a very arbitrary line. CD just means your pipeline pushed to production (in a hopefully validated and safe manner). Jenkins can do that, GitLab can do that, any of those tools can. They're free form and you tell them what to do.
* Having everything in one tool is revolutionary and makes life 1000 times easier. E.g. GitLab, GitHub, etc are going all in to do code, wikis, testing, container and artifact registries, deployment, monitoring, even things like incident management. It's amazingly helpful and goes far beyond "CI and CD". More tools = more integration issues and more sprawl and a greater chance you miss or get lazy about something / waste time coding integrations.
* For getting started, anything really works again. I might recommend GitHub actions, GitLab, or Jenkins. The first 2 may be a little better as they have more tools in them. With Jenkins you'll need external artifact registries and such. But Jenkins is good as it is super common still.

FYI, I don't know much about Jenkins vs JenkinsX, so ignoring that part :).
> 3 votes

---
Tags: jenkins, cicd, toolchain, jenkins-x
---


# Question
Title: terraform apply - how to use wildcard or range for indexes?
I have 10 virtual machines to be re-created. I want to do it in 2 groups, first 5 machines \[0-4\] and second 5 machines \[5-9\]

```
terraform apply -replace=module.ci.azurerm_windows_virtual_machine.vm[0-4]

```

But it does not work, says:

```
│ Index key must be followed by a closing bracket.

```

Is there any way to use a wildcard, or point a group of machines in general in command above?

# Answer
> 1 votes
I don't know that you can specify ranges. On Linux, you can add a bash function such as from here.

A slight modification should work,

```
terraform-replace () {
    sed 's/\x1b\[[0-9;]*m//g' | grep -o '# [^( ]* ' | grep '\.' | sed " s/^# /-replace '/; s/ $/'/; "
}

```

Then use it like this,

```
terraform plan | terraform-replace | grep 'azurerm_windows_virtual_machine.vm\[[0-4]\]' | xargs -r terraform apply -auto-approve

```

---
Tags: terraform
---


# Question
Title: Why is one not allowed to use a Boolean in a docker-compose.yml?
Defining a boolean in a docker-compose.yml file:

```
environment:
  SOME_VAR: true

```

and running `docker up` results in:

```
contains true, which is an invalid type, it should be a string, number, or a null

```

**Attempts to solve the issue**

1. If true is changed to True the issue persists.
2. Using `'true'` is not accepted by the code itself (a play framework app is started using the `./target/universal/stage/bin/APPNAME -Dplay.evolutions.db.default.autoApply=`, i.e. either `-Dplay.evolutions.db.default.autoApply=true` or `-Dplay.evolutions.db.default.autoApply=false` parameter):

VAR has type STRING rather than BOOLEAN
3. Using `yes` or `no` as a variable results in:

contains true, which is an invalid type, it should be a string, number, or a null
4. Using `yes` and using a script that transforms `yes` to true works

**Discussion**

According the docs `Any boolean values; true, false, yes no, need to be enclosed in quotes to ensure they are not converted to True or False by the YML parser`:

> **Environment**
>
> Add environment variables. You can use either an array or a dictionary. Any boolean values; true, false, yes no, need to be enclosed in quotes to ensure they are not converted to True or False by the YML parser.
>
> Environment variables with only a key are resolved to their values on the machine Compose is running on, which can be helpful for secret or host-specific values.
>
> ```
> environment:
>   RACK_ENV: development
>   SHOW: 'true'
>   SESSION_SECRET:
>
> environment:
>   - RACK_ENV=development
>   - SHOW=true
>   - SESSION_SECRET
>
> ```

**Question**

Why is it not allowed?

# Answer
> 31 votes
This come from a design choice of YAML language about booleans

Every unquoted value matching this "regex":

```
 y|Y|yes|Yes|YES|n|N|no|No|NO
|true|True|TRUE|false|False|FALSE
|on|On|ON|off|Off|OFF

```

Will be converted to `True` or `False`. Note that from YAML 1.2 and onwards it appears that only `true` and `false` will be interpreted as boolean values.

This start causing a problem when your code will test an environment value to be yes or no for example taking this script (other examples in the PR discution):

```
if [ "$SOME_VAR" == "yes" ];
then
  echo "Variable SOME_VAR is activated"
else
  echo "Variable SOME_VAR is NOT activated"
fi

```

And setting in your compose file

```
environment:
  SOME_VAR: yes

```

Will result in `SOME_VAR` being `True` when the script run, hence taking the wrong case as it is not equal to `yes`.

So the choice has been made to disallow boolean to prevent unwanted behaviors hard to debug when you're not aware of the YAML rule.

I see two way to get over the problem:

1. Using an `env_file` instead, they are not parsed IIRC and should prevent the conversion.
2. As you already said, use a wrapper script around your launcher to define the value instead before launching the app, something along the line of this should do:

```
 AUTOAPPLY=false
 if [ "$SOME_VAR" == "true" ]
 then
     AUTOAPPLY=true
 fi

 ./target/universal/stage/bin/APPNAME -Dplay.evolutions.db.default.autoApply=$AUTOAPPLY

```

# Answer
> 11 votes
That's YAML. It interprets `true` as a boolean. Envars must be strings, hence the requirement to make the type explicit via quotes.

Test this out with https://www.json2yaml.com/

# Answer
> 4 votes
double quotes worked for me. Example:

```
environment:
  MY_VAR: "true"

```

---
Tags: docker-compose
---


title: Use S3 To Store ConfigMap

text:
# Question
Title: Use S3 To Store ConfigMap
I have a ConfigMap which stores Nginx configs and It is getting `size exceeded`. There are many site configs in there so I can't create multiple ConfigMaps.

I thought of using S3 to store ConfigMap, is it an option? If it is, how can one achieve it.

# Answer
> 3 votes
ConfigMaps in Kubernetes are limited to 1Mib. If your file could not be split or shrunk, I feel you should use a volume to store it using the storage class that your cloud vendor/subsystem offers you.

S3 is a good object store, but you can't store ConfigMaps in it. You can't use it either to store volumes. You could use EBS and EFS (since we are talking about AWS products) based on access pattern that you want to achieve for this filesystem.

---
Tags: kubernetes, amazon-s3
---


# Question
Title: docker command substitution fails on gitlab
I am trying to run the following command as part of a deployment in Gitlab.

```
TIMESTAMP=$(date +%s) docker stack deploy --with-registry-auth --compose-file $(`printf "docker-compose.%s.unimark.yaml" "$SRV_DEPLOY_ENV"`) unimark`

```

I am getting the following error. I am not sure what is going wrong here.

```
/bin/sh: eval: line 332: docker-compose.demo.unimark.yaml: not found
"docker stack deploy" requires exactly 1 argument.
See 'docker stack deploy --help'.
Usage:  docker stack deploy [OPTIONS] STACK
Deploy a new stack or update an existing stack

```

I also tried

```
DOCKER_COMPOSE_DEPLOY_FILE=`printf "docker-compose.%s.unimark.yaml" "$SRV_DEPLOY_ENV"`

TIMESTAMP=$(date +%s) docker stack deploy --with-registry-auth --compose-file $($DOCKER_COMPOSE_DEPLOY_FILE) unimark`

```

the same error saying docker-compose file cannot be found. I checked and the file is there in the same directory. Not sure if it has to do with some `bash` vs `sh` issue.

# Answer
I think this is a problem with bash command substitution.

```
$(`printf "docker-compose.%s.unimark.yaml" "$SRV_DEPLOY_ENV"`)

```

or

```
DOCKER_COMPOSE_DEPLOY_FILE=`printf "docker-compose.%s.unimark.yaml"
"$SRV_DEPLOY_ENV"`
$($DOCKER_COMPOSE_DEPLOY_FILE)

```

To bash is interpreted the same: please execute the command that is the result of this nested command.

Removing the backtick in the first instance or removing the parenthesis in the second one should do the trick.
> 0 votes

---
Tags: gitlab, docker-swarm, bash
---


# Question
Title: Unable to use GCP Cloud Armor due to insufficient quota
I'm trying to use "Cloud Armor".

However when trying to create some config (I believe it was the security police) in the console I get an error of:

> Operation type \[insert\] failed with message "Quota 'SECURITY\_POLICIES' exceeded. Limit: 0.0 globally."

When I try to increase the quota:

> Based on your service usage history, you are not eligible for quota increase at this time. Please make use of the current available quota, and if additional resources are needed, please contact our Sales Team (https://cloud.google.com/contact/) to discuss further options for higher quota eligibility.

Upon doing a request for a quota increase I got an email back stating:

> Dear Developer,
>
> We have reviewed your request for additional quota. Unfortunately we cannot grant your request.
>
> While evaluating quota requests, we consider several factors, including resources that most legitimate customers use, a customer’s previous usage and history with Google Cloud Platform. In addition to this, we may deny a request for additional quota when there are outstanding issues with a billing account for billing quota or paid services related quota requests.

So now what do I do? Any ideas? Why is using this so restrictive while cloudflare is much less trouble to setup...

# Answer
Asking for a quota increase is asking for a credit line increase. Google has declined your request to use the Cloud Armor resource.

There are several options:

1. Wait until you have some payment history.
2. Prepay your account.
3. Use a different payment instrument that has credit available.
4. Contact Google Cloud sales and establish an account relationship.
> 2 votes

---
Tags: google-cloud-platform
---


# Question
Title: In Ubuntu, how do I authenticate to trigger a Jenkins job remotely using the CLI?
As far as I understand, using the Linux CLI, builds can be triggered remotely using the curl command and the following URL syntax:

\<JENKINS\_URL\>/job/\<JOB\_NAME\>/build?token=\<TOKEN\_NAME\>

Example:

```
$ curl http://localhost:8080/job/MyJob/build?token=my-token

```

However, when I run the command, I receive the following:

```
Authentication required
<!--
-->
</body></html>

```

Is authentication done using the CLI or the Jenkins web interface? How do I authenticate?

# Answer
This is covered in the documentation. In short, if you are triggering via HTTP, you need to generate an authentication token and use it with HTTP Basic auth, for instance:

```
curl -X POST -L --user your-user-name:apiToken https://jenkins.example.com/job/your_job/build

```
> 1 votes

---
Tags: jenkins, authentication
---


# Question
Title: Cost minimization in aws parallel cluster
I'm new to aws cloud service. I have to create a parallel cluster in aws. So, I'm using aws parallel cluster. But the problem is, when creating a cluster the NAT gateway billing me when I'm not using it. So, the cost increases. Is there any way I can minimize or automate the NAT gateway process so that it will cost me less??

PS:If there is any project which gives me on demand NAT gateway hour or Nat gateway instances in parallel cluster that would be very nice.

# Answer
> 0 votes
This is possible, by creating two lambda function , one to check if the NAT getaway is available, if not, handles the provisioning of a new gateway and the creation of the required routes. And the second lambda to check if the lambda is still required and if not, it can deactivate.

Please refer this blog for Code and explanation.

https://adamsnotes.medium.com/a-simple-on-demand-nat-gateway-for-aws-97d4d4b388f3

---
Tags: amazon-web-services, amazon-ec2
---


# Question
Title: How come Maven's validate and verify phases execute without goals bound to them?
Maven's documentation states the following: "If a build phase has no goals bound to it, that build phase will not execute"

However, according to the same documentation, the *validate* and *verify* phases do not seem to have goals bound to them.

What am I missing?

# Answer
I found the exact same question answered on Stack Overflow and, according to it, ...

* both of the above statements are correct
* the *validate* and *verify* phases don't have goals bound to them **by default**
* the *validate* and *verify* phases will only execute if the POM binds a plugin's goal to them
> 0 votes

---
Tags: builds, maven
---


title: Docker vs Virtualisation

text:
# Question
Title: Docker vs Virtualisation
I am trying to understand the underlining architecture of Docker. The diagram that's been shown everywhere claims that in contrast with Virtualisation technologies such as "VirtualBox", Docker uses the OS of the host directly and only ships the applications together with dependencies such as libraries etc. Now, from what I can see in Docker, every image includes an OS. It starts with a `FROM <os-image>` tag. Isn't this contradictory to what's been claimed? Please advise.

# Answer
> 7 votes
First, Docker is just a company. =)

There are two methods of isolating things,

* Methods that isolate the kernel.
* Methods that do not isolate the kernel.

For all intents and purposes, methods that do not isolate the kernel are called "containerization", while those that do are called "virtualization". In industry almost 100% of the use cases of "containerization" refer to Linux Containerization. It's for the most part correct to say that containers are a Linux thing. One more point of confusion, many non-Linux systems that support "native" containerization do so with a virtual machines which means that you have the native kernel (like Darwin/BSD) running on the host, and a Linux kernel running in a virtual machine which hosts just the container environment. As a rule of thumb,

* Containerization is always less secure: vulnerable to kernel-level exploits.
* Containerization is always faster: less context switching and hypervisor overhead.

It's not true that just because something does not virtualize the kernel, that it's not isolated from the host. While it's true Linux containers are *just* processes, and thus,

* Visible from the host
* Subject to any kernel level resource optimizations, like memory deduplication

Containerized processes must

* Run in a different namespaces which, barring a kernel-level exploit, isolate them from other process on the machine
* (Usually) run in isolated cgroups subject to different quotas and limits.

As a last point, just to drive it home, because

* Containerization typically refers to the implementation in Linux.
* Linux has no native concept of containerization itself, only providing cgroups (resource control) and namespaces (isolation)
* A container is just a native process

Then we tend to say any process on Linux that makes use of namespaces is running in a container, moreso if it's using cgroups.

As a final point, typically when you hear "Docker Image" people mean an OCI Compliant Image, which is what everyone uses.

---

When you see

```
FROM <os-image>

```

In a dockerfile, what you're actually saying is that you want to in git-parlance clone a working set of stuff and build on top of it. This stuff does not include a kernel. But it will include **everything** else because you will not have access to the system's stuff. (The container is in a different namespace and isolated). For example, a container must include a copy of glibc if needed, and a container of Debian must include apt and other Debian based utilities that constitute a "core" system.

# Answer
> 0 votes
run `uname -r` inside a docker image and then on your base os, you'll notice it's the same kernel

```
bash-4.4# uname -r
3.10.0-1160.59.1.el7.x86_64
bash-4.4# exit
[petturn@h-mybox-2 ~]$ uname -r
3.10.0-1160.59.1.el7.x86_64

```

that's probably the biggest distinction between virtualization and containerization. A virtual machine is running it's own kernel, a container runs inside the space of your base OS. You see the processes and everything. What neither of them can (or should) be able to do is interact with the base OS.

When you build a container "from ubuntu" or whatever, I believe that has more to do with specifying the toolchains you're building from.

If you build a container "from scratch" (I've never tried this, but it seems like fun), I think that'd become apparent!

---
Tags: docker, containers, virtualization
---


# Question
Title: gitlab project registry - docker push fails with invalid reference format error message
I have create a personal access token for my project. in my gitlab-ci.yml I'm trying to create a docker image and then save it in my project's repository.

I'm able to login - I see an explicit message saying Login Succeeded. Then it builds the image but when trying to push is where I see this error:

```
12 exporting layers
#12 exporting layers 7.8s done
#12 writing image sha256:5337a378e31dcb6f8ed2cae0ab82188273947b395d52908eb07e4777b028a8a4 done
#12 naming to docker.io/library/widgetsimage done
#12 DONE 7.9s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
$ docker push https://mygitserver.org/jj/pipelinetests/widgetsImage
invalid reference format

```

**Code**

```
stages:
  - build
#  - unit-test

BuildDockerImage:
  stage: build
  environment:
    name: development
  image: docker
  services:
    - docker:dind
  script:
    - docker login -u myPersonalAccessTokenUserName -p asdfasdfQn https://mygitserver.org/JJ/pipelinetests
    - docker build -t widgetsimage .
    - docker push https://mygitserver.org/jj/pipelinetests/widgetsimage
  tags:
    - mac-pipelines-shell

```

I do see the Packages & Registries option in my project - which I guess indicates its enabled? But the list of packages is empty, of course. Any suggestions would be appreciated.

**EDIT 1**

I've tried changing the permissions / visibility on the project from private to Internal - everyone with access should now be able to do mostly everything on this test repo. But it didn't fix the issue.

Also in case it's related ... The URL for the project in gitlab is:

```
 https://mygitserver.org/JJ/pipelinetests

```

aka. I have Uppercase in my path. But the docker login command never complains about the lower/uppercase. I've tested with both.
it always says it was successful:

```
Executing "step_script" stage of the job script
$ docker login -u myPersonalAccessTokenUserName -p asdfasdfQn  https://mygitserver.org/jj/pipelinetests
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded
$ docker build -t widgetsimage .
#1 [internal] load build definition from Dockerfile

```

**EDIT 2**

I've also tried to use the url that I see when i navigate to the packages in the Gitlab GUI for the docker push command, like so:

```
https://mygitserver.org/jj/pipelinetests/-/packages/widgetimage

```

But still no dice.

**EDIT 3**

So I tried to change my code like this:

```
- docker login -u myPersonalAccessTokenUserName -p asdfasdfQn mygitserver.org
- docker build -t mygitserver.org/jj/pipelinetests/widgetsimage .
- docker push mygitserver.org/jj/pipelinetests/widgetsimage

```

But I'm basically getting a 404 / permissions error.

```
12 exporting layers 8.2s done
#12 writing image sha256:756f9fc8559519e6893f8a28f35a60c6cc28cb50be09351bdba33baafd4db4f7 done
#12 naming to mygitgserver.org/jj/pipelinetests/widgetsimage done
#12 DONE 8.2s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
$ docker push mygitgserver.org/jj/pipelinetests/widgetsimage
Using default tag: latest
The push refers to repository mygitgserver.org/jj/pipelinetests/widgetsimage]
8704fdba6e23: Preparing
7621ddf6521a: Preparing
27e96e15eec9: Preparing
1e773440db1a: Preparing
fecad1a1ae13: Preparing
6c882dfb600e: Preparing
fd6a5c73e987: Preparing
6c882dfb600e: Waiting
a45a618792f0: Waiting
6c946b32cfe2: Waiting
7dd4a3e7d836: Waiting
error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE html>\n<html>\n<head>\n  <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n  <title>The page you're looking for could not be found (404)</title>\n  <style>\n    body {\n      color: #666;\n      text-align: center;\n      font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n      margin: auto;\n      font-size: 14px;\n    }\n\n    h1 {\n      font-size: 56px;\n      line-height: 100px;\n      font-weight: 400;\n      color: #456;\n    }\n\n    h2 {\n      font-size: 24px;\n      color: #666;\n      line-height: 1.5em;\n    }\n\n    h3 {\n      color: #456;\n      font-size: 20px;\n      font-weight: 400;\n      line-height: 28px;\n    }\n\n    hr {\n      max-width: 800px;\n      margin: 18px auto;\n      border: 0;\n      border-top: 1px solid #EEE;\n      border-bottom: 1px solid white;\n    }\n\n    img {\n      max-width: 40vw;\n      display: block;\n      margin: 40px auto;\n    }\n\n    a {\n      line-height: 100px;\n      font-weight: 400;\n      color: #4A8BEE;\n      font-size: 18px;\n      text-decoration: none;\n    }\n\n    .container {\n      margin: auto 20px;\n    }\n\n    .go-back {\n      display: none;\n    }\n\n  </style>\n</head>\n\n<body>\n  <a href=\"/\">\n    <img src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRoIGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo=\"\n       alt=\"GitLab Logo\" />\n  </a>\n  <h1>\n    404\n  </h1>\n  <div class=\"container\">\n    <h3>The page could not be found or you don't have permission to view it.</h3>\n    <hr />\n    <p>The resource that you are attempting to access does not exist or you don't have the necessary permissions to view it.</p>\n    <p>Make sure the address is correct and that the page hasn't moved.</p>\n    <p>Please contact your GitLab administrator if you think this is a mistake.</p>\n    <a href=\"javascript:history.back()\" class=\"js-go-back go-back\">Go back</a>\n  </div>\n  <script>\n    (function () {\n      var goBack = document.querySelector('.js-go-back');\n\n      if (history.length > 1) {\n        goBack.style.display = 'inline';\n      }\n    })();\n  </script>\n</body>\n</html>\n"
Cleaning up project directory and file based variables
00:00
ERROR: Job failed: exit status 1

```

I googled around a bit and I guess it's still possible that the registry option in general for Gitlab needs to be configured. Is this true? Like I need to assign a specific port for the registry? wondering how I can check? I'm not a gitlab admin. I don't have access to the server config. But wondering if there's anything I can try / test that would conclusively prove that the registry hasn't been set up globally?

# Answer
An image reference in docker doesn't contain the schema, so there's no `https://`. For a registry login, you login to the server only, not to the repository. And when building, the tag either needs to include the full image reference you want to later push, or you need to retag the image with `docker tag` to have the name you want to push. So your steps should be:

```
    - docker login -u myPersonalAccessTokenUserName -p asdfasdfQn mygitserver.org
    - docker build -t mygitserver.org/jj/pipelinetests/widgetsimage .
    - docker push mygitserver.org/jj/pipelinetests/widgetsimage

```
> 0 votes

---
Tags: docker, gitlab, gitlab-ci-runner, docker-registry
---


# Question
Title: GitLab tag removal through the web interface does not actually remove the tag from Git
When I hit the delete button in the Tags interface I can verify the tag does not show, but when semantic-release tries to run, I still `git tag` on the runner I still get

```
fatal: tag 'v5.2.3' already exists
    at makeError (/usr/local/lib/node_modules/semantic-release/node_modules/execa/lib/error.js:60:11)
    at handlePromise (/usr/local/lib/node_modules/semantic-release/node_modules/execa/index.js:118:26)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at async tag (/usr/local/lib/node_modules/semantic-release/lib/git.js:224:3)
    at async run (/usr/local/lib/node_modules/semantic-release/index.js:191:5)
    at async module.exports (/usr/local/lib/node_modules/semantic-release/index.js:260:22)
    at async module.exports (/usr/local/lib/node_modules/semantic-release/cli.js:55:5) {
  shortMessage: 'Command failed with exit code 128: git tag v5.2.3 b49dff6c91a03a326e6afde67c32a5c788914c25',
  command: 'git tag v5.2.3 b49dff6c91a03a326e6afde67c32a5c788914c25',
  escapedCommand: 'git tag v5.2.3 b49dff6c91a03a326e6afde67c32a5c788914c25',
  exitCode: 128,
  signal: undefined,
  signalDescription: undefined,
  stdout: '',
  stderr: "fatal: tag 'v5.2.3' already exists",
  failed: true,
  timedOut: false,
  isCanceled: false,
  killed: false
}.map

```

What's going on here?

# Answer
I'm not sure what GitLab is thinking here, but the Tags API and tags UI represent tags in the web interface, but they do not disruptively remove the tags from the underlying repo. This seems like a **major** but which I'll file later, or update this with a link to at a later point in time.

You can force the removal from the upstream repo **which will also remove the tag from the UI** using vanilla Git.

```
git push upstream :refs/tags/<tag to delete>

```

Like this

```
git push upstream :refs/tags/v5.2.3

```
> 0 votes

---
Tags: gitlab, gitlab-ci-runner
---


# Question
Title: gitlab container registry: how to test that it's set up and working?
I'm trying to push for the first time to a gitlab project container registry. I am not a gitlab administrator so I can't change global settings. But as far as my project is concerned I've made sure that I have a personal access token created for the project.
When I try to run my pipeline, it fails with a 404 error. I've recreated the steps just on the commandline to simplify things. here's what the output looks like:

```
PS /Users/me/src/pipelinetests> docker login -u widgetsProjectAccessToken -p asdf mygitserver.org/myusername/pipelinetests

WARNING! Using --password via the CLI is insecure. Use --password-stdin.
Login Succeeded

PS /Users/me/src/pipelinetests> docker build -t mygitserver.org/myusername/pipelinetests:widgetsimage .

[+] Building 190.4s (12/12) FINISHED
 => [internal] load build definition from Dockerfile                                                                                0.0s
 => => transferring dockerfile: 37B                                                                                0.0s
 => [internal] load .dockerignore                                                                                0.0s
 => => transferring context: 2B                                                                                0.0s
 => [internal] load metadata for mcr.microsoft.com/dotnet/sdk:6.0.202-bullseye-slim-amd64                                                                                0.2s
 => [internal] load build context                                                                                0.0s
 => => transferring context: 21.82kB                                                                                0.0s
 => [1/7] FROM mcr.microsoft.com/dotnet/sdk:6.0.202-bullseye-slim-amd64@sha256:1ee95620598b9afa3046d356bb7f7863e1602075dd407c790c34d0a9833acc73                                                                                0.0s
 => => resolve mcr.microsoft.com/dotnet/sdk:6.0.202-bullseye-slim-amd64@sha256:1ee95620598b9afa3046d356bb7f7863e1602075dd407c790c34d0a9833acc73                                                                                0.0s
 => CACHED [2/7] RUN apt update && apt-get install vim -y                                                                                0.0s
 => [3/7] COPY . /builds                                                                                0.1s
 => [4/7] RUN curl -sL https://aka.ms/InstallAzureCLIDeb | bash                                                                                93.2s
 => [5/7] RUN apt-get install curl && curl -sL https://deb.nodesource.com/setup_12.x | bash -                                                                                25.7s
 => [6/7] RUN apt-get install nodejs                                                                                13.0s
 => [7/7] RUN npm install -g azure-functions-core-tools@4 --unsafe-perm true                                                                                48.4s
 => exporting to image                                                                                9.7s
 => => exporting layers                                                                                9.6s
 => => writing image sha256:2f30a1f69f875df9e4645bea95247e7f8678da89d2da04524d0fc570e3e1ce4e                                                                                0.0s
 => => naming to mygitserver.org/myusername/pipelinetests:widgetsimage                                                                                0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
PS /Users/me/src/pipelinetests> docker push mygitserver.org/myusername/pipelinetests:widgetsimage

The push refers to repository [mygitserver.org/myusername/pipelinetests]
0ee173286353: Preparing
4c70ac952b79: Preparing
b00534d89ac6: Preparing
b7482490dd9f: Preparing
ea13d436bb8a: Preparing
120468cee518: Waiting
fd6a5c73e987: Waiting
e4869730bdcc: Waiting
c782777c8a5d: Waiting
a45a618792f0: Waiting
6c946b32cfe2: Waiting
7dd4a3e7d836: Waiting
20147d2db13c: Waiting
9c1b6dd6c1e6: Waiting
error parsing HTTP 404 response body: invalid character '<' looking for beginning of value: "<!DOCTYPE html>\n<html>\n<head>\n  <meta content=\"width=device-width, initial-scale=1, maximum-scale=1\" name=\"viewport\">\n  <title>The page you're looking for could not be found (404)</title>\n  <style>\n    body {\n      color: #666;\n      text-align: center;\n      font-family: \"Helvetica Neue\", Helvetica, Arial, sans-serif;\n      margin: auto;\n      font-size: 14px;\n    }\n\n    h1 {\n      font-size: 56px;\n      line-height: 100px;\n      font-weight: 400;\n      color: #456;\n    }\n\n    h2 {\n      font-size: 24px;\n      color: #666;\n      line-height: 1.5em;\n    }\n\n    h3 {\n      color: #456;\n      font-size: 20px;\n      font-weight: 400;\n      line-height: 28px;\n    }\n\n    hr {\n      max-width: 800px;\n      margin: 18px auto;\n      border: 0;\n      border-top: 1px solid #EEE;\n      border-bottom: 1px solid white;\n    }\n\n    img {\n      max-width: 40vw;\n      display: block;\n      margin: 40px auto;\n    }\n\n    a {\n      line-height: 100px;\n      font-weight: 400;\n      color: #4A8BEE;\n      font-size: 18px;\n      text-decoration: none;\n    }\n\n    .container {\n      margin: auto 20px;\n    }\n\n    .go-back {\n      display: none;\n    }\n\n  </style>\n</head>\n\n<body>\n  <a href=\"/\">\n    <img src=\"data:image/svg+xml;base64,PHN2ZyB3aWR0aD0iMjEwIiBoZWlnaHQ9IjIxMCIgdmlld0JveD0iMCAwIDIxMCAyMTAiIHhtbG5zPSJodHRwOi8vd3d3LnczLm9yZy8yMDAwL3N2ZyI+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTVsMzguNjQtMTE4LjkyMWgtNzcuMjhsMzguNjQgMTE4LjkyMXoiIGZpbGw9IiNlMjQzMjkiLz4KICA8cGF0aCBkPSJNMTA1LjA2MTQgMjAzLjY1NDhsLTM4LjY0LTExOC45MjFoLTU0LjE1M2w5Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTIuMjY4NSA4NC43MzQxbC0xMS43NDIgMzYuMTM5Yy0xLjA3MSAzLjI5Ni4xMDIgNi45MDcgMi45MDYgOC45NDRsMTAxLjYyOSA3My44MzgtOTIuNzkzLTExOC45MjF6IiBmaWxsPSIjZmNhMzI2Ii8+CiAgPHBhdGggZD0iTTEyLjI2ODUgODQuNzM0Mmg1NC4xNTNsLTIzLjI3My03MS42MjVjLTEuMTk3LTMuNjg2LTYuNDExLTMuNjg1LTcuNjA4IDBsLTIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+CiAgPHBhdGggZD0iTTEwNS4wNjE0IDIwMy42NTQ4bDM4LjY0LTExOC45MjFoNTQuMTUzbC05Mi43OTMgMTE4LjkyMXoiIGZpbGw9IiNmYzZkMjYiLz4KICA8cGF0aCBkPSJNMTk3Ljg1NDQgODQuNzM0MWwxMS43NDIgMzYuMTM5YzEuMDcxIDMuMjk2LS4xMDIgNi45MDctMi45MDYgOC45NDRsLTEwMS42MjkgNzMuODM4IDkyLjc5My0xMTguOTIxeiIgZmlsbD0iI2ZjYTMyNiIvPgogIDxwYXRoIGQ9Ik0xOTcuODU0NCA4NC43MzQyaC01NC4xNTNsMjMuMjczLTcxLjYyNWMxLjE5Ny0zLjY4NiA2LjQxMS0zLjY4NSA3LjYwOCAwbDIzLjI3MiA3MS42MjV6IiBmaWxsPSIjZTI0MzI5Ii8+Cjwvc3ZnPgo=\"\n       alt=\"GitLab Logo\" />\n  </a>\n  <h1>\n    404\n  </h1>\n  <div class=\"container\">\n    <h3>The page could not be found or you don't have permission to view it.</h3>\n    <hr />\n    <p>The resource that you are attempting to access does not exist or you don't have the necessary permissions to view it.</p>\n    <p>Make sure the address is correct and that the page hasn't moved.</p>\n    <p>Please contact your GitLab administrator if you think this is a mistake.</p>\n    <a href=\"javascript:history.back()\" class=\"js-go-back go-back\">Go back</a>\n  </div>\n  <script>\n    (function () {\n      var goBack = document.querySelector('.js-go-back');\n\n      if (history.length > 1) {\n        goBack.style.display = 'inline';\n      }\n    })();\n  </script>\n</body>\n</html>\n"

```

**What I've Done so far**

I've made sure that my project visibility and the settings are correct:

I've also created the personal access token - which the successful login proves.

**Questions**

In googling, it seems that some people found their container registry wasn't actually set up properly. I'm not sure how to test that if I'm not a global admin.
So my first question is do you know of any other way I can test to make sure that it's a config issue?

If we assume it's actually set up correctly, could it just be that I haven't specified the correct port in my push command? Like if the admin chose a non standard port, could that be the issue?

Are there other trees I should be barking up?

**EDIT 1**

These are the options under Packages & Registries

# Answer
It looks to me like you're not pushing to the right port.

When you go to the project space, and then go to your container registry `/container_registry` (or click under **Packages & Registries****Container Registry**

Or, like this if you prefer to click on **Packages & Registries**

On this page you should see a bunch of links, one of them should have something like this,

```
docker push domain:5050/myusername/pipelinetests

```

You want to be sure you're including that `:5050`. That's the defualt port for the GitLab Container Registry. It looks like from that error, that you're targeting the actual project page.
> 0 votes

---
Tags: docker, gitlab, docker-registry
---


title: Cloud-Init Script Won't Run?

text:
# Question
Title: Cloud-Init Script Won't Run?
I am working through this Terraform tutorial, which boots up a Go web app on an EC2 instance, using Terraform and cloud-init. This is my Terraform config file:

```
terraform {
  required_version = ">= 0.13"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

provider "aws" {
  region = var.region
}

data "aws_ami" "ubuntu" {
  most_recent = true

  filter {
    name   = "name"
    values = ["ubuntu/images/hvm-ssd/ubuntu-*20*-amd64-server-*"]
  }

  filter {
    name   = "virtualization-type"
    values = ["hvm"]
  }

  owners = ["099720109477"] # Canonical
}

resource "aws_vpc" "vpc" {
  cidr_block           = var.cidr_vpc
  enable_dns_support   = true
  enable_dns_hostnames = true
}

resource "aws_internet_gateway" "igw" {
  vpc_id = aws_vpc.vpc.id
}

resource "aws_subnet" "subnet_public" {
  vpc_id     = aws_vpc.vpc.id
  cidr_block = var.cidr_subnet
}

resource "aws_route_table" "rtb_public" {
  vpc_id = aws_vpc.vpc.id

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = aws_internet_gateway.igw.id
  }
}

resource "aws_route_table_association" "rta_subnet_public" {
  subnet_id      = aws_subnet.subnet_public.id
  route_table_id = aws_route_table.rtb_public.id
}

resource "aws_security_group" "sg_22_80" {
  name   = "sg_22"
  vpc_id = aws_vpc.vpc.id

  # SSH access from the VPC
  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 80
    to_port     = 80
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  ingress {
    from_port   = 8080
    to_port     = 8080
    protocol    = "tcp"
    cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }
}

data "template_file" "user_data" {
  template = file("../scripts/add-ssh-web-app.yaml")
}

resource "aws_instance" "web" {
  ami                         = data.aws_ami.ubuntu.id
  instance_type               = "t2.micro"
  subnet_id                   = aws_subnet.subnet_public.id
  vpc_security_group_ids      = [aws_security_group.sg_22_80.id]
  associate_public_ip_address = true
  user_data                   = data.template_file.user_data.rendered

  tags = {
    Name = "Learn-CloudInit"
  }
}

output "public_ip" {
  value = aws_instance.web.public_ip
}

```

My cloud-init script looks like this:

```
#cloud-config
# Add groups to the system
# Adds the ubuntu group with members 'root' and 'sys'
# and the empty group hashicorp.
#groups:
  - ubuntu: [root,sys]
  - hashicorp

# Add users to the system. Users are added after groups are added.
users:
  - default
  - name: terraform
    gecos: terraform
    shell: /bin/bash
    primary_group: hashicorp
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    lock_passwd: false
    ssh_authorized_keys:
      - ssh rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCMT6uYfhWx8jOmiGR9ryIPKcWy2ceqvyZ4
Q4+q5QTiZtlbWxbP37YZnT8uQhyjB4QRR1cjOyvGKC3Zu0Isy0eHIx2lGm/7B04b
soWWAUqhJmYWMZlivnHtJJJ4P5gnvXiRNmFg9iK07C7ClggNBAQZZHUeA5wcnvvH
T/pDkGUjMUqgLvmWRJqJM9qLT717e229F1Fyh+sYtAj08qmcFF1JCs2D33R46RQ8
YBMpQqmWLfjuJDUrjdvMu7Mv3aPpaeUWuYoC90iHR9XMeNonrtRlx21nY3CoMZ0A
OpeNl999UzyMJrsvN4qm6byK2Pc6jrEyKr9jI8SvMEGdSWgqr/Hd

# Downloads the golang package
packages:
  - golang-go

# Sets the GOPATH & downloads the demo payload
runcmd:
  - sudo su terraform
  - sudo mkdir /home/terraform/go
  - sudo chown terraform:hashicorp /home/terraform/go
  - export GOPATH=/home/terraform/go
  - go get github.com/hashicorp/learn-go-webapp-demo

```

After successfully applying the Terraform config, I obtain the following error when I try to SSH into this instance from the CLI using my OpenSSH private key:

*Permission denied (publickey,gssapi-keyex,gssapi-with-mic).*

To further investigate, I have SSH'd into the instance with AWS Direct Connect, from which I found none of the users, groups or directories had been created and the packages hadn't been installed - so it seems like the cloud-init script did not run at all. I'm guessing I can't SSH into my instance because the public key wasn't uploaded to it.

I assumed that this could be an issue with Ubuntu, so I changed the AMI to Linux 2, which resulted in the same error. Am I missing something obvious?

# Answer
> 3 votes
There are a few issues with this cloud-init script.

* `#groups` is commented out, so the list beneath it makes it invalid YAML.
* Even if we uncomment groups,  `- ubuntu: [root,sys]` is invalid because the default user already has a group specification. If you look near the bottom of `/etc/cloud/cloud.cfg` on a launched instance, you should see something like:

```
system_info:
   # This will affect which distro class gets used
   distro: ubuntu
   # Default user name + that default users groups (if added/used)
   default_user:
     name: ubuntu
     lock_passwd: True
     gecos: Ubuntu
     groups: [adm, audio, cdrom, dialout, dip, floppy, lxd, netdev, plugdev, sudo, video]
     sudo: ["ALL=(ALL) NOPASSWD:ALL"]
     shell: /bin/bash

```

So you either need to remove the `ubuntu` line from your groups definition, or remove `- default` from your user definition and define the `ubuntu` user manually.

* Your ssh key is also pasted as invalid YAML. Keep it as a single line
* In your runcmd, `sudo su` doesn't work that way in a script. Instead you should `sudo -u <user>` for an individual user to run an individual command. Additionally, `sudo` isn't needed for root permissions as all user scripts run as root already.

Fixing all of these issues, we should get a resulting cloud-init script that looks something like this:

```
#cloud-config

# Add the empty group hashicorp.
groups:
  - hashicorp

# Add users to the system. Users are added after groups are added.
users:
  - default
  - name: terraform
    gecos: terraform
    shell: /bin/bash
    primary_group: hashicorp
    sudo: ALL=(ALL) NOPASSWD:ALL
    groups: users, admin
    lock_passwd: false
    ssh_authorized_keys:
      - ssh rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCMT6uYfhWx8jOmiGR9ryIPKcWy2ceqvyZ4Q4+q5QTiZtlbWxbP37YZnT8uQhyjB4QRR1cjOyvGKC3Zu0Isy0eHIx2lGm/7B04bsoWWAUqhJmYWMZlivnHtJJJ4P5gnvXiRNmFg9iK07C7ClggNBAQZZHUeA5wcnvvHT/pDkGUjMUqgLvmWRJqJM9qLT717e229F1Fyh+sYtAj08qmcFF1JCs2D33R46RQ8YBMpQqmWLfjuJDUrjdvMu7Mv3aPpaeUWuYoC90iHR9XMeNonrtRlx21nY3CoMZ0AOpeNl999UzyMJrsvN4qm6byK2Pc6jrEyKr9jI8SvMEGdSWgqr/Hd

# Downloads the golang package
packages:
  - golang-go

# Sets the GOPATH & downloads the demo payload
runcmd:
  - mkdir /home/terraform/go
  - chown terraform:hashicorp /home/terraform/go
  - sudo GOPATH=/home/terraform/go -u terraform go install github.com/hashicorp/learn-go-webapp-demo@latest

```

# Answer
> 0 votes
Maybe it does not matter but also try to add "${}" around terraform's file function before rendering:

```
data "template_file" "init_script" {
  template = "${file("../scripts/add-ssh-web-app.yaml")}"
}

```

---
Tags: amazon-web-services, terraform, infrastructure-as-code, ssh, cloud-init
---