# Question
Title: How would you build docker images in Kubernetes while distributing its layer caching among all pod builders at scale?
We are using Azure DevOps as our CICD platform.

We have our own self hosted Linux agents which we integrated into our CICD Azure Kubernetes Cluster .

We use these agents to build our Docker images within the cluster, meaning that we have N agents represented by N pods.

While working, we might scale up and down the amount of concurrent agents depending on different parameters.

Since we have started with this approach, we have encountered a huge issue in leveraging docker's cache.

We have tried to:

1. Use kubernetes StatefulSets. But this limits us from having the cache shared between all pods (one PV/PVC per pod)
2. Use Google's Kaniko \- Unfortunately we have found that in many cases it is slower than just using Docker's BuildKit without cache.
3. We tried to use Azure Files with SMB and NFS as we thought that a PV with ReadWriteMany will solve the problem. Unfortunately we have ended up with a fatal error.

Currently we are a bit confused as we assume that this is a common industry issue and therefore we might have got it wrong. We are looking to understand how to leverage docker's cache while using our current setup and without moving to another platform.

How should we do that? What are the best practices for this issue?

# Answer
Docker allows you to pull a cache from anywhere with a --cache-from command. See dockumentation here: https://docs.docker.com/engine/reference/commandline/build/#specifying-external-cache-sources

The easiest approach is to build your images with the `--build-arg BUILDKIT_INLINE_CACHE=1` flag which stores the metadata needed to build from cache. You can store these images in two places, either a file system or a registry you already have. If you want to keep the registry of images used for deployment separate from the ones with metadata, you could separate with tags or separate registries.

If you want to store the files somewhere closer you can attach volumes directly to the nodes using pvc's or shared volumes. See this stack overflow on more details. https://stackoverflow.com/a/52564314/8954290.

Here is someone who uses the same approach. https://medium.com/swlh/fast-docker-build-in-kubernetes-f52088854f45

NOTE: The reason that CI providers do this is to do this reliably is hard. You must handle race conditions of which images to store as cache and which ones to pull from in the event of multiple images being built at the same time. CI providers use workflow engines, queus, replicas etc to handle this which is not easy to build. This is why most people use services like circle ci, codefresh etc.
> 1 votes

---
Tags: docker, kubernetes, azure-devops, continuous-integration
---


# Question
Title: Modularizing CI/CD pipelines in Jenkins or in other tools?
I have the following case:

There are a couple of repositories (well, two-digit number). Each of them has some sort of (declarative) Jenkins pipeline - it would be one of three types:

* run tests -\> build docker image -\> deploy to test environment;
* run tests -\> create package -\> upload to package repository (i.e. PyPI);

Now, currently, each repository has a Jenkinsfile of one of the above types, and a Jenkinsfile which runs only the tests and does nothing more. These Jenkinsfiles are basically the same across the repositories - same code, same environment variables. The only thing that changes are the image/package names (and, of course, there are differences between building images and creating packages). If I have to make a change to any one Jenkinsfile, I have to make a change to all of them, go through Pull Requests etc., which is obviously too much unneeded work.

Not only this, but the project pipeline involves adding even more tasks performed by Jenkins - like running integration tests, deploying to higher environments, etc. Most of these integrations would require certain combinations of steps - i.e. today I want to build an image and deploy to environment A; tomorrow I want unit-test, deploy to environment B and integration-test; etc. As you see, these pipelines would have many steps in common. Following the current practice, this will lead to code duplication and maintenance costs.

If all this is implemented as separate Jenkinsfile for each pipeline, this will lead to explosion of Jenkinsfiles which are almost the same within a repository and across repositories.

What I want to do is to abstract away the common steps (like test a repository; build an image; create a package; run some integration tests; deploy to an environment) in a separate repository which will hold all such CI/CD code. Then I want to create *parameterized* (Jenkins) pipelines in each of the repositories, which just run these generic steps in any combination I wish with the configuration I have given for a concrete repository.

Three options I have researched:

* create a separate Jenkins job with a corresponding Jenkinsfile for each generic step; have a pipeline in each repository that just calls a combination of these generic steps providing parameters. This has the overhead of having to manually specify git-clone operations for the repository which needs to be built/tested in each generic step. This answer seems to point in this direction, but I am still concerned with the possible overhead.
* call a combination the generic Jenkinsfiles from within a pipeline providing the necessary parameters. This has the overhead of rather low maintainability (calling Jenkinsfile from Jenkinsfile doesn't seem that straightforward and includes more lines of code).
* call abstracted-away shell scripts. As I see it, this leads to even harder maintenance and even lower abstraction. It constrains the user to script everything in shell; loses all possible Jenkins plugins and other tools. While possible, it seems to me that it won't provide the level of abstraction I hope to achieve. Please correct me if I am wrong.

Now the question is: how can I achieve this using (declarative) Jenkins pipelines with the least amount of overhead?

Even more importantly - are there tools that are better-suited for this kind of work?

# Answer
> 1 votes
The built-in tool that Jenkins provides for this situation is called shared libraries. You can define common Pipeline steps within a shared library and then include those steps from within your Pipelines. Just to lend some anecdotal support - since there are so many tasks in common across my team's build pipelines, the vast majority of our Pipeline code actually lives in shared libraries; the individual Jenkinsfiles in our repos mainly just call steps from the shared libraries.

---
Tags: jenkins, cicd, build-pipeline, toolchain
---


# Question
Title: Change kubernetes resource requests after pod/job has started
I'd like to change my pod's CPU request value after a pod has started. Is this possible?

In our shared kubernetes system there is a requirement that our actual resource usage be "reasonably close" to the resource request value in the yaml, otherwise we limit a users ability to deploy further jobs until they correct their usage requests.

Therefore it's valuable to me to be able to adjust the resource request value after the job started, if that's possible. Presumably such a request would only succeed if the node has the resources (CPU, Memory) available.

I noticed there's a `kubectl patch` option to made dynamic updates, but it's not clear that it could accomplish what I'm looking for.

# Answer
`LimitRange` for the rescue: by setting both `Max` limits and `LimitRequestRatio` for pods users **must** set limits that are lower than Max or request\*ratio or apiserver would return `403 Forbidden`.
> 1 votes

---
Tags: kubernetes, kubectl
---


# Question
Title: How to copy files recursively to a Kubernetes pod?
I want to copy files recursively to a Kubernetes pod

I tried `kubectl cp -r`

I got: `error: unknown shorthand flag: 'r' in -r`

What are the best ways to transfer whole directories recursively into a pod.

# Answer
`kubectl cp` by default does recursive copies when given a directory, although it seems to be picky about trailing slashes. If `foo/bar` is the directory you'd like to copy, simply run

```
kubectl cp /path/to/foo/bar <pod-id>:/path/in/container/foo/

```
> 35 votes

# Answer
For Windows to Linux, the following commands worked for me (this will not accept backslash, only forward-slash worked):

```
kubectl cp C:/myfiles/azureCLI/WantToCopy my-pod:/mnt/data/something/

```
> 1 votes

# Answer
According to the help menu, the recursive option does not seem to exist.

```
user@localhost ~ $ kubectl cp --help
Copy files and directories to and from containers.

Examples:
  # !!!Important Note!!!
  # Requires that the 'tar' binary is present in your container
  # image.  If 'tar' is not present, 'kubectl cp' will fail.

  # Copy /tmp/foo_dir local directory to /tmp/bar_dir in a remote pod in the default namespace
  kubectl cp /tmp/foo_dir <some-pod>:/tmp/bar_dir

  # Copy /tmp/foo local file to /tmp/bar in a remote pod in a specific container
  kubectl cp /tmp/foo <some-pod>:/tmp/bar -c <specific-container>

  # Copy /tmp/foo local file to /tmp/bar in a remote pod in namespace <some-namespace>
  kubectl cp /tmp/foo <some-namespace>/<some-pod>:/tmp/bar

  # Copy /tmp/foo from a remote pod to /tmp/bar locally
  kubectl cp <some-namespace>/<some-pod>:/tmp/foo /tmp/bar

Options:
  -c, --container='': Container name. If omitted, the first container in the pod will be chosen

Usage:
  kubectl cp <file-spec-src> <file-spec-dest> [options]

Use "kubectl options" for a list of global command-line options (applies to all commands).

```

In order to copy files recursively, all files could be put in a directory and when this folder is copied to the pod, all files were copied:
> 0 votes

# Answer
`-r` is not needed to copy the folder. The below command will work. I tested this on `kubectl` version 1.21.

```
kubectl cp <pod_id>:/home/test_folder test_folder_localhost

```
> 0 votes

---
Tags: kubernetes
---


# Question
Title: What is the best way to monitor job status?
I would like to monitor for recent spike in failed application jobs at my company.

Job statuses are stored in a MySQL table. We have on order of 50 different types of jobs, and sometimes, one of those jobs will start to fail, either because of a change in code or a change to a contract. In Production we'd like to visualize the spike in failures and alert on it.

We have ELK, grafana, opsgenie, and python, bash scripting capabilities. ELK doesn't seem to work because it parses logs, and our logs are not well standardized that we can detect a spike in failures. Grafana seems to work better for metrics like CPU and memory. We've done montioring in the past, by scripting, but surely there must be a better way. Customers are hit with the problem long before we know about it.

Is there something in ELK, Grafana, or another open source tool that can help us here?

# Answer
> 2 votes
There could be multiple ways to solve this.

1. Create a python based cronjob that queries the DB every T secs (scraping interval) for the failed jobs in the last T secs (delta). Push the relevant data into a specific index like `failed-job` in ES. Then use that index to set up alerts in Grafana/opsgenie.(You can also use watcher here) Don't need to read logs.
2. Use prometheus,(if already using), create a new custom prometheus exporter.Exporter would be a Flask web server that provides an endpoint to run the query against the db and return results in a prometheus compatible format to be stored. Configure job in prometheus to scrape this endpoint every T secs. Again, use Grafana for visualization.
3. Use a component called , prometheus-pushgateway, if possible, either push the job status into this component using the python script as in option 1. or modify the Job itself to push the data directly. Again , can be connected to Prometheus/Grafana.

If you are trying to solve only a single use case as described here, option 1 would be the better choice. For a larger more complex setups that covers a lot of other use cases, Prometheus might be worth looking into.

Alternatively, you can use the same design to push metrics into any datasource.(Like the one you are using for storing CPU/mem metrics, as specified.)

# Answer
> 0 votes
Not sure if it checks all your boxes but Rundeck is great for having an interface for your jobs . It alerts you if a job fails , has a quick summary for the jobs. here you can a look for details.

# Answer
> 0 votes
Maybe you can connect Grafana to the MySQL database and query for number of failures? Grafana also has a basic alerting capabilities, so you could send alerts to the interested parties.

At my company, we have a similar approach to jobs - the statuses are kept in the database. What we do to check for failures is we use Nagios and a custom query to check for problems and send alerts - but you might want to stick with Grafana and not install another tool.

---
Tags: monitoring
---


# Question
Title: how to change a rancher UI installed prometheus server config
I am learning rancher (still) and after deploying the monitoring chart (prom, grafana etc) i need to add more targets to the prometheus config (such as other bare-metal server and some QEMU too), I know I can update the config by entering the pods via shell but these changes would be lost after redeploy or upgrade of the chart.

I tends to think that the chart uses configMap to handle the prometheus config but (and i'm feeling noobish here) I cannot find a way to update this configMap, well I just cannot find this specific configMap.

I went through the helm quickstart and chart update docs but this did not help, same goes for the rancher docs.

Any hint will be highly apreciated.

For example in the cattle-monitoring-system namespace I can see these configmaps :

```
pierre@latpopo:~$ k get cm -n cattle-monitoring-system
NAME                                                   DATA   AGE
grafana-nginx-proxy-config                             1      2d23h
kube-root-ca.crt                                       1      2d23h
prometheus-nginx-proxy-config                          1      2d23h
prometheus-rancher-monitoring-prometheus-rulefiles-0   26     2d22h
rancher-monitoring-crd-manifest                        1      2d22h
rancher-monitoring-grafana                             1      2d23h
rancher-monitoring-grafana-config-dashboards           1      2d23h
rancher-monitoring-grafana-datasource                  1      2d23h
rancher-monitoring-grafana-test                        1      2d23h
rancher-monitoring-prometheus-adapter                  1      2d23h

```

None of them seems to be hosting the prometheus config file

In the describe for what I think is the actual prom server deployment I can see there is a use of a configMap :

```
k describe statefulsets prometheus-rancher-monitoring-prometheus -n cattle-monitoring-system
[...]
--config.file=/etc/prometheus/config_out/prometheus.env.yaml
[...]
    Mounts:
      /etc/prometheus/config from config (rw)
[...]

```

But still, I can't find how to change in a persistent manner

EDIT : Ok by reviewing again the rancher docs i finally saw :

> For information on configuring custom scrape targets and rules for Prometheus, please refer to the upstream documentation for the Prometheus Operator.

But following that documentation path led me to the prometheus-operator which seems to be the correct place for editing the prometheus config, but this also seems to be only usable for kubernetes based resources: *Looks like this is not the correct place to add external targets and services monitoring*.

So the question still stand:

**How can I add new targets (external to kubernetes) to the prometheus "chart deployed" ?**

# Answer
> 1 votes
If you could let us know what helm chart you have used to install prom-operator, that might help refine the answer.

But in case of kube-prometheus-stack, in the values.yaml of the , use the `additionalScrapeConfigs` section to describe jobs that are external to k8s system and re-deploy.

Or use serviceMonitors as described in this article

---
Tags: kubernetes, helm, rancher
---


# Question
Title: Purge old docker images, except images that have been used in the past 4 weeks (docker image prune --filter until=672h)
How can I tell docker to `prune` old images, except any images that have been used in the past 4 weeks?

To prevent the oft-cited issue of docker filling our disks in prod, we have a script that runs the following command (among others):

```
time nice docker image prune --force --all --filter until=672h

```

This should, by our interpretation, delete only images that have been unused for 672 hours = 4 weeks.

Sadly, we stopped the image temporarily during a change the other day, and our `docker ... prune` cron kicked-off right in the middle of our change. The result: An hour later when we finished the change, the image had been deleted.

So even though the image had been used in a container \<24 hours ago (that containter itself was *started* 5 weeks ago, but *stopped* \<24 hours ago), the filter deleted it because it was \[a\] currently unused and \[b\] the image was created \>4 weeks ago.

How can I do `docker image prune` cleanups such that it doesn't delete images unless the image hasn't been used by a container for over 4 weeks?

# Answer
> 1 votes
It seems that this is still an ongoing issue. No metadata is available (contained in the image itself) that provides information about last usage.

You can try to implement your own script/tool that monitors `docker events` and keeps track of image usage and prunes older ones. For example, `docker events --since 24h --filter event=start --format '{{ .Time }} {{ .From }}'` will return timestamp and image name of started containers for the past 24h.

It also depends on what's your criteria for an "used image". If using the events log, however, keep in mind that **`Only the last 1000 log events are returned.`**, so that should also be taken care of.

---
Tags: docker
---


# Question
Title: AWS: Restoring database snapshot to different environment
I've a webapp hosted on AWS. This has Electric Beanstalk, with EC2 and then a RDS Postgres database at the bottom.

There's prod, UAT and staging environments and databases.

I want to take a recent snapshot of the prod database (well I've done that) and restore it on the staging database, as the current staging environment is a bit out of data wise.

Yet when I go to restore snapshots in the staging database, I can only restore those which were taken on the staging database.

Is there any way to restore the prod snapshot which I've made to the staging database?

# Answer
Restoring a snapshot simply creates a new RDS instance. So if you restore the prod DB snapshot to staging then you will be creating a new RDS instance with the restored data in it.

Ps: Another option can be, using mysqldump to take backup from prod and then restoring it to the staging instance
> 1 votes

---
Tags: amazon-web-services, amazon-ec2, elastic, amazon-rds
---


# Question
Title: Cloudformation issue with target groups and ASG
I am trying to use cloudformation to setup a ASG and I have a target group which I would like to associate with the ASG. The target group is created okay but when I go to attach it to the ASG I get an error message. I'd like to be able to attach this as I want a ALB in front of the ASG

Error message:

```
Value of property TargetGroupARNs must be of type List of String

```

Cloudformation code

```
TargetGroup:
    Type: AWS::ElasticLoadBalancingV2::TargetGroup
    Properties:
      VpcId:
        Fn::ImportValue:
          !Sub "${Prefix}-VpcId"
      TargetType: instance
      Port: 443
      Protocol: HTTPS
AutoScalingGroup:
    Type: AWS::AutoScaling::AutoScalingGroup
    Properties:
      AutoScalingGroupName: Autoscaling Group Alpha
      Cooldown: 120
      DesiredCapacity: 1
      LaunchConfigurationName: AlphaLaunchConfiguration
      MaxSize: 1
      MinSize: 1
      Tags:
        -
          Key: Name
          Value: Alpha ASG
          PropagateAtLaunch: 'true'
      TargetGroupARNs:
        Ref: TargetGroup

```

# Answer
> 2 votes
```
Resources:
    TargetGroup:
        Type: AWS::ElasticLoadBalancingV2::TargetGroup
        Properties:
          VpcId:
            Fn::ImportValue:
              !Sub "${Prefix}-VpcId"
          TargetType: instance
          Port: 443
          Protocol: HTTPS
    AutoScalingGroup:
        Type: AWS::AutoScaling::AutoScalingGroup
        Properties:
          AutoScalingGroupName: Autoscaling Group Alpha
          Cooldown: "120"
          DesiredCapacity: "1"
          LaunchConfigurationName: AlphaLaunchConfiguration
          MaxSize: "1"
          MinSize: "1"
          Tags:
            -
              Key: Name
              Value: Alpha ASG
              PropagateAtLaunch: true
          TargetGroupARNs:
            - Ref: TargetGroup

```

As the error message states, TargetGroupARNs value should be of type List. https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html#cfn-as-group-targetgrouparns

# Answer
> 0 votes
Changing the value of "TargetGroupARNs:" from "Ref: TargetGroup" to "- !Ref TargetGroup" should solve the problem.

Reference :https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-as-group.html

---
Tags: autoscaling, aws-cloudformation
---


# Question
Title: Bitbucket triggers builds on deleted branches
I configured Bitbucket webhook to trigger on Repository push. After each merge (which also deletes the source branch) it triggers build, which fails in Codefresh because of deleted branch.

Does anyone have the same experience? How do you deal with it?

# Answer
> 1 votes
There are some issues with the way you have implemented your CI workflows:

* It sounds like CodeFresh is attempting to do a build on the branch that is being merged and deleted. Instead, you should be building on your integration branch (e.g., Develop, master).
* If you choose to adopt a Pull Request workflow, you'll still have this issue if the code is merged and the source branch is deleted before the branch build completes.

Consider implementing a workflow where you only build a branch if there is a Pull Request and the branch is updated. Otherwise, stick to compiling just the mainline integration branch. This will ultimately speed up your CI process as well.

---
Tags: bitbucket, builds, codefresh
---


title: HashiCorp Vault roles

text:
# Question
Title: HashiCorp Vault roles
It's quite easy to create roles for policies and also list them in HashiCorp Vault:

```
vault write auth/jwt/role/myrolename ...
...
list auth/jwt/role

```

However, I can't find anything regarding how to display role details. Strange enough, the UI does not contain the section "roles".

Is it possible to access role details for an existing role?

# Answer
Yes, it's possible as following:

```
vault read -format=json auth/jwt/role/ROLENAME

```
> 0 votes

---
Tags: hashicorp-vault, role-based-access-control
---


# Question
Title: Using a conditional environment{ } directive in Jenkinsfiles
My Jenkinsfiles use the environment{} directive. I've been trying to set a condition where I invoke different variables depending on the GIT branch being builded.

I've tried something like e. g. :

```
switch(branch_name) {
   case 'dev' :
    Var1 = x;
   case 'master' :
    Var1 = y;
}

```

That would allow me skip using different Jenkinsfiles for each branch on a repo. But groovy syntax seems not to work inside this environment{} directive.

Is there a way tackling it? Or would you suggest another approach for handling different global variables for each branch?

# Answer
I often have an extra `Prepare` stage to calculate these logics:

```
stage('Prepare') {
   steps {
      script {
         switch(branch_name) {
            case 'dev' :
               env.VAR_1 = x;
            case 'master' :
               env.VAR_1 = y;
         }
      }
   }
}

```

`env.` will set the global environment for you.
> 5 votes

---
Tags: jenkins, git, jenkinsfile
---


# Question
Title: Using counted data source in non-counted provider block in Terraform
I'm new to `Terraform`, I need suggestion about handling one case where I need to use data from `counted` `data-source` to `un-counted` `provider`.

I need to fetch the `gke-cluster` context and pass it to `provider helm/kubernetes` to provision resources on it . There can be multiple `gke -clusters`, so I'm using `count` on the `data-source google_container_cluster` and then I was trying to get the `endpoint` from it . Like below :

```
data "google_container_cluster" "my_cluster" {
    count      = var.enabled ? length(var.gke_clusters) : 0
    location   = lookup(var.gke_clusters[count.index], "cluster_location")
    name   = lookup(var.gke_clusters[count.index], "cluster_name")
}
provider "helm" {
  kubernetes {
    host  = "https://${data.google_container_cluster.my_cluster.endpoint}"
    token = data.google_client_config.provider.access_token
    cluster_ca_certificate = base64decode(
    data.google_container_cluster.my_cluster.master_auth[0].cluster_ca_certificate,
    )
  }
}

```

But this does not work because when I'm referring to `my_cluster` I have to specify the `index` , and if I specify the index `my_cluster[count.index]`, I get an error that `Error: Reference to "count" in non-counted context`.

Thanks in advance.

# Answer
> 1 votes
If I have read this correctly you are trying to target different clusters with your provider so you can then use a helm\_release.

You need to have distinct providers for each cluster and differentiate them with an `alias` so.

```
provider "helm" {
  kubernetes {
    host  = "https://my_endpoint1"
    token = "provider.access_token1"
    cluster_ca_certificate = "base64encodedcerthere1"
  }
}

provider "helm" {
  alias = "cluster2"
  kubernetes {
    host  = "https://my_endpoint2"
    token = "provider.access_token2"
    cluster_ca_certificate = "base64encodedcerthere2"
  }
}

```

the first provider will be used where no alias is provided and the second used only where an alias is used for the provider so for example:

```
resource "helm_release" "example" {
  name       = "my-bob-release"
  ...
}

```

the above uses the default un-named provider

```
resource "helm_release" "example" {
  provider   = helm.cluster2
  name       = "my-bob-release"
  ...
}

```

---
Tags: terraform, terraform-module, google-kubernetes-engine
---


# Question
Title: Terraform import and using ip address as attribute name
I am building terrform.tfstate files from Terraform managed Fortigate devices. After producing output JSON files using Terraform output and a custom script to produce tf.address.tf.json, I wish to import the objects using the attribute name as outputted if possible.

With a tf.address.tf.json file structured like this

```
{
    "resource": {
        "fortios_firewall_address": {
            "11.0.0.0/8": {
                ...
            }
        }
    }
}

```

I perform the following

`$ terraform import fortios_firewall_address.11_0_0_0/8 "11.0.0.0/8"`

which produces the error

```
Error: Attribute name required
│
│   on <import-address> line 1:
│    1: fortios_firewall_address.11_0_0_0/8
│
│ Dot must be followed by attribute name.

```

I suppose I need to alter the attribute name to something like "ADDRESS-11\_0\_0\_0M8" but want to ask if there is another way to use the name as "fortios\_firewall\_address.11\_0\_0\_0/8" in terrform import.

# Answer
The problem you've encountered here is not specific to `terraform import` but rather a general Terraform language rule: the name of a resource must be a valid identifier, and valid identifiers can start only with letters and underscores, not with digits.
> 1 votes

---
Tags: terraform
---


# Question
Title: Kubernetes pods scaling doesnt increase throughput
I have a simple Django web application running on AWS EC2 instance (2 vCPU, 4Gi). For that type of deployment I have a performance (measured with Hey)

```
$ hey -t 30 -z 1m https://mydomain.com/my/endpoint/

```

about `30 reqs/s`

When deployed with Kubernetes onto EKS cluster (same docker image) I have about `15 reqs/s` for 1 pod with limits

```
    resources:
      requests:
        cpu: "500m"
        memory: "2Gi"
      limits:
        cpu: "1"
        memory: "4Gi"

```

I expect that scaling pods would double the performance but nothing happens (request rate stays the same)

```
$ kubectl scale deploy/myapp --replicas=2

```

I'm running Django app behind `gunicorn` like this (adjusting number of workers did not help)

```
gunicorn --config gunicorn.py myapp.wsgi:application
# gunicorn.py
import os

# tried different values from 1 to 5
# for 1 vCPU allocated for the pod it should be 3 as per docs
# (2 x $num_cores) + 1
# https://docs.gunicorn.org/en/stable/design.html#how-many-workers
workers = int(os.getenv('GUNICORN_WORKERS_APP', default=4))
daemon = False
bind = '0.0.0.0:8080'
max_requests = 2000

```

So couple of questions here:

* how to properly debug this type of issue and what's the toolset I can use?
* how `service` does load balancing to pods and is there any overhead introduced?
* is there an overhead for resource management (cpu allocation/throttling)?
* how performance tuning is done for k8s-driven applications?

# Answer
What does 'kubectl top node' return for your node usage?

Kubernetes is pretty low overhead and can certainly handle more than 15req/s. Does your application rely on a database? If so, where is that database living? Is that single node also running any other pods?

As others noted, an APM would be a helpful tool to diagnose unexpected performance issues.
> 1 votes

---
Tags: kubernetes, load-balance, performance
---


# Question
Title: Common permissions for EC2 usage and VPC Security Groups in enterprise use
**Context:** Looking to create some software that acts as middleware to a user's AWS account, where the user will provide their cloud credentials, and have the software use their EC2 instances.

**Question:** Trying to figure out any permission concerns for my target users. In an enterprise environment if I'm a data scientist that uses EC2 instances regularly to run models etc. and I have permissions to create, and destroy EC2 instances within my VPC, would I typically also have permissions to create and modify security groups? Or is it common to have security set up where the data scientists can use compute, but not permissions to modify or change SG?

From using EC2 it looks like they go hand in hand, e.g., during the EC2 instance creation process one step is to configure security groups as needed, so I'm led to believe the above question is yes, but unsure.

Thanks

# Answer
> 1 votes
Not technically, I understand your need to create and destroy EC2 instances, whether it's done via automation or not. About having permissions at the security group level, this can often be limited to viewing only the resource, as it is a networking firewall resource, usually who manages these objects are SDN or NaaS

---
Tags: amazon-web-services, amazon-ec2, security, aws-vpc, aws-security-group
---


# Question
Title: Spring boot prometheus - difference between system_cpu_usage and process_cpu_usage
Spring boot reports two statistics when Prometheus in place.

```
    # HELP system_cpu_usage The "recent cpu usage" for the whole system
    # TYPE system_cpu_usage gauge
    system_cpu_usage 0.029126213592233007

    # HELP process_cpu_usage The "recent cpu usage" for the Java Virtual Machine process
    # TYPE process_cpu_usage gauge
    process_cpu_usage 0.0

```

What is the difference between `system_cpu_usage` and `process_cpu_usage` ?

# Answer
As far as I understand,

`system.cpu.usage` is host’s cumulative CPU usage in nanoseconds this includes user, system, idle CPU modes. (the sum of the /proc/stat CPU line)

`process.cpu.usage` is the cpu usage for the JVM process aka CPU time used by the JVM process

CPU usage aka CPU time is measured by counting the seconds that a CPU spends processing instructions from an application or from OS.

Both metrics are of type **Gauge** because their values that can go up or down as their showing a resource usage .
> 1 votes

---
Tags: prometheus, spring-boot
---


# Question
Title: Using cached dependencies for pipelines in Azure Devops
I want to avoid maven downloading all dependencies from a remote repository in all build

I configured a cache, but all executions still download all dependencies.

azure-pipelines.yml

```
trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

steps:
- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'clean package'
    codeCoverageToolOption: JaCoCo
    sonarQubeRunAnalysis: false
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Pipeline.Workspace)'
    publishLocation: 'pipeline'
- task: Cache@2
  inputs:
    key: 'maven | "$(Agent.OS)" | **/pom.xml'
    restoreKeys: |
      maven | "$(Agent.OS)"
      maven
    path: $(MAVEN_CACHE_FOLDER)
  displayName: Cache Maven local repo

- script: mvn install -B -e

```

# Answer
I forgot to add a MAVEN\_OPTS variable in mavenOptions:
`mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'`

My pipeline.yml stayed like this:

```
trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

variables:
  MAVEN_CACHE_FOLDER: $(Pipeline.Workspace)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

steps:

- task: Cache@2
  inputs:
    key: 'maven | "$(Agent.OS)" | **/pom.xml'
    restoreKeys: |
      maven | "$(Agent.OS)"
      maven
    path: $(MAVEN_CACHE_FOLDER)
  displayName: Cache Maven local repo

- task: Maven@3
  inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m $(MAVEN_OPTS)'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.8'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'clean package'
    codeCoverageToolOption: JaCoCo
    sonarQubeRunAnalysis: false
- task: PublishPipelineArtifact@1
  inputs:
    targetPath: '$(Pipeline.Workspace)'
    publishLocation: 'pipeline'

- script: mvn install -B -e

```
> 1 votes

# Answer
I made it work like this:

```
variables:
  MAVEN_CACHE_FOLDER: $(HOME)/.m2/repository
  MAVEN_OPTS: '-Dmaven.repo.local=$(MAVEN_CACHE_FOLDER)'

trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

steps:
  - task: Cache@2
    displayName: 'Cache Maven local repo'
    inputs:
      key: '"funcs" | maven | "$(Agent.OS)" | pom.xml'
      restoreKeys: |
      path: $(MAVEN_CACHE_FOLDER)

  - task: Maven@3
    displayName: 'Build and Test'
    inputs:
      mavenPomFile: 'pom.xml'
      mavenOptions: '-Xmx2048m'
      javaHomeOption: 'JDKVersion'
      jdkVersionOption: '1.11'
      jdkArchitectureOption: 'x64'
      publishJUnitResults: true
      testResultsFiles: '**/surefire-reports/TEST-*.xml'
      goals: '-B package'

  - task: PublishCodeCoverageResults@1
    displayName: 'Publish code coverage'
    inputs:
      codeCoverageTool: 'JaCoCo'
      summaryFileLocation: '$(System.DefaultWorkingDirectory)/target/site/jacoco/jacoco.xml'
      pathToSources: '$(System.DefaultWorkingDirectory)/src/main/java'
      additionalCodeCoverageFiles: '$(System.DefaultWorkingDirectory)/target/jacoco.exec'

```

It has:

* caching of the Maven repository
* Run maven with `-B` to avoid download texts in the output
* Use Java 11
* Publish test reports
* Publish Jacoco code coverage
> 1 votes

---
Tags: azure-pipelines, maven, repository
---


# Question
Title: How is my branch is one commit behind master
I have a branch and I merged it into the master branch (via Pull Request).Then I un-ticked the box to not delete the branch. Now when I look at the branches this particular branch shows as 1 commit behind the master branch.

I don't understand this, how can it be a commit behind when the commit was what was merged from my branch into master? Is there a way to fix this so that it knows it's the same commit?

# Answer
From what you've described, it seems the "one commit" is a merge commit. This is due to how git works: depending on the steps you performed to get your code merged, it will or will not create a merge commit in the target branch. Check your git history to confirm it.

If that is the case, you can safely ignore it. In a typical git workflow, your "feature branches" are short-lived, getting deleted when merged, so you won't even notice this difference.

If you don't want a merge commit, merging with fast-forward won't produce one, but you may need to rebase (or squash, depending on your preference) the source branch before the merge, which will change your branch history to allow a fast-forward.

Since you've tagged `Azure DevOps` in your question, you can avoid the merge commit using the "Squash merge" or "Rebase and fast-forward" PR merge types.

P.S.: Also, if you are new to git, be extra careful when rewriting history. Things can get messy if you don't know what you are doing. I strongly recommend getting to know git better to get the most out of it.
> 2 votes

# Answer
First, from your description, it is clear that you are using a graphical engine for git. That can sometimes present issues.

Second, this message means that you have not pushed your commit and someone has made an additional commit in your current branch (and pushed their changes) after you last pulled and comitted.

To resolve the issue:

1. Push your commit. How to do this in your graphical engine we cannot speak to because you have not provide details about what you are using
2. If you have conflicts, you will need to merge or resolve them. That can be difficult if there is a collision (wherein you are editing the same sections of code)
3. To prevent this in the future, be sure to push your commits and pull every time you begin to make changes.
> 1 votes

---
Tags: azure-devops
---


# Question
Title: How to have multiple git repos in an Azure DevOps project?
My organisation uses Azure DevOps and we want to use git as our repo.

But, Azure projects seem to only allow one git repo per project.

This is bizarre to me! Projects and repos are usually separate. One project may involve development on several tools, and a tool may be used by several projects.

How can I get projects to engage with multiple git repos?

# Answer
> 28 votes
Following works for me:

* Click on your Project and then \[Repos\] in the left hand menu.
* Along the top, you'll see the bread-crumbs: `Organization / Project / Repos / Files / {NameOfCurrentRepo}`
* Click the down arrow next to that and you'll see the option to add a `new Repository`

If that's not available, then it must be a permissions setting that I'm not aware of.

# Answer
> 2 votes
As an alternative to DrGriff's answer..

* Go to the \[Project Settings\] for your project and select \[Repos\] / \[Repositories\] on the left-hand side
* There should be a "Create" button on the top right (below your profile button)

If you don't see the "Create" button your user almost certainly does not have the permission to create repositories for this project.

Looking at *ataraxia's* comment, I had the same issue. I was both a project and collection administrator but could not create a new repository. The reason being that I had explicitly set "Deny" on the "Create repository" permission for "Contributors". An explicit Deny on one of the groups you belong to seems to override your Allow privileges. So please go over your group member privileges, perhaps that solves your problem?

# Answer
> 2 votes
If you have alread added one repository to your project then you should do the following:

1. On left hand side panel click on `repos`
2. On the top panel you will see `organization-name`/`project name`/`Repos`/`Files`/`current repository` \- click on `current repository` which has a git icon next to it and a down-arrow to its right
3. Press on `Import repository`
4. Enter the details of your repository on the pop-up pannel

---
Tags: azure
---


# Question
Title: Terraform AWS AZ error - manually specify AZ?
I'm fairly new to terraform and when creating AWS infrastructure I use the following to fetch the 'available' availability zones for my region (us-east-1) where I'm trying to create an EKS cluster.

```
data "aws_availability_zones" "available" {}

```

However when I run a 'terraform apply', it generates the following error because us-east-1a is at capacity.

```
aws_eks_cluster.demo: error creating EKS Cluster (test-cluster):
UnsupportedAvailabilityZoneException: Cannot create cluster 'test-cluster' because us-east-1a, the targeted availability zone, does not
currently have sufficient capacity to support the cluster. Retry and
choose from these availability zones: us-east-1b, us-east-1c, us-east-1d

```

Is there any way I can get terraform to handle this automatically or specify that I only want to use 1b,1c,1d etc.

EDIT:

I've realised this is tied with where I create my subnets, which I do using this line:

```
 availability_zone = "${data.aws_availability_zones.available.names[count.index]}"

```

So I assume I somehow need to skip us-east-1a by doing something here?

Thanks in advance.

# Answer
> 1 votes
If anyone else has this issue, here's how I got around it.

Specify a variable with the availability zones you want to target

```
variable "zones" {
  default = ["us-east-1b", "us-east-1c", "us-east-1d"]
}

```

When creating your subnets, use this line

```
availability_zone = "${var.zones[count.index]}"

```

# Answer
> 0 votes
You could use AWS ClI for consult availability zones:

```
aws ec2 describe-availability-zones

```

With this result you could prepare a script to take the first result.

# Answer
> 0 votes
I know this is an old question but I came across this same issue, and solved it in a different way. Rather than hardcoding the availability zones like the accepted answer, I think it is better to filter out those availability zones that do not support EKS. This approach makes it easier to move to a new region because the data is coming from aws, and not a hardcoded list.

https://registry.terraform.io/providers/hashicorp/aws/latest/docs/data-sources/availability\_zones#exclude\_names

```
data "aws_availability_zones" "available" {
    # List of availability zones that do not support EKS
    exclude_names = ["us-east-1e"]
}

```

---
Tags: amazon-web-services, terraform
---


title: no route to host container registry

text:
# Question
Title: no route to host container registry
After installing docker (ubuntu) the default 172 ip was replaced with 10.200.x.x

```
sudo ip addr add dev docker0 10.200.200.90/24
sudo ip addr del dev docker0 172.17.0.1/16

```

The machine's ip is 10.203.32.90

Then running a private container registry with the basic command:

```
docker run -d -p 5000:5000 --restart=always --name registry registry:2

```

But then push to the registry fails with "no route to host"

```
docker push 10.203.32.90:5000/myimage

```

What have I done wrong or forgotten?

# Answer
Solved it:

> docker network ls

47ebbf46f26d bridge bridge local

> docker network inspect bridge

the bridge ip was still 172.17.0.0/16

So alter the bridge ip in daemon.json

> cat /etc/docker/daemon.json

```
{
    "bip": "10.200.200.90/24",
}

```

restart docker daemon and bridge is working and we can push to the registry
> 1 votes

---
Tags: docker
---


# Question
Title: GitLab Maintain versions of the code in GitLab CI/CD pipeline for the develop branch
I would like to ensure that code merged via merge request in a GitLab from the feature branch to our develop branch has always a new unique version (gitflow strategy).

Version currently stored as the part of the code and depends on the project language: (ex: JavaScript it's package.json)

Version suppose to be used for:

* Docker image tag
* Package/code/library bundle
* Displayed to the user and added to the release notes.

I am having a few strategies in mind

**Scenario 1**

* Add checklist template to the merge request so reviewer has to mark checbox that version was increased.
* Additionally to the previous manual step I would like to have CI/CD step for the validation: Ex: need to check previous version of commit with a new one to ensure that pending merge request has greater version than previous. Any clues whether it's possible to determine?

**Scenario 2**

* CI/CD on a develop branch should automatically increase version of the package and commit. Not sure whether this possible and whether ad-hoc commit will no trigger CI/CD again. (can a problem to handle hotfixes)

I would appreciate any suggestions and your ideas. BR

# Answer
While I generally do not recommend GitFlow these days, fully automated way to do what you ask is possible to achieve with an external system. Below I will describe how to do it with Reliza Hub (which is the one I'm developing - but you may find other options).

Here is the idea -

* You create your project in Reliza Hub and choose versioning schema, i.e. semver.
* You may also choose custom schema for feature branches (default is Branch.Micro).
* Next, you generate a project token from Reliza Hub and set it to your CI system.
* Then inside your CI, add a retrieve version step, which would call Reliza Hub for the next version. See samples of this on GitHub - https://github.com/relizaio/reliza-hub-integrations (note that for Jenkins there is now a dedicated plugin).

After doing this, version will be auto-resolved on every run and every merge. In any case it is guaranteed that every version is unique and there are no conflicts between branches.

If you want to adopt automatic increments of major version component - versus minor component - versus patch component, use Conventional Commits - https://www.conventionalcommits.org/en/v1.0.0/

Basically, if you start your commit message with *fix*, it would know to increment patch; if you start it with *feat*, it would know to increment minor.

You can also put what's called *Version Pin* on a specific branch - i.e. you can set it to *1.3.Micro* \- then regardless of anything else, inside this branch version will only be incremented within Major 1 and Minor 3.
> 1 votes

---
Tags: gitlab, python, versioning, javascript
---


# Question
Title: How to add a wait condition when creating IAM entities
I'm working on a template that allows to create a user, tag it, and add it to any groups, add any roles, managed policies and in-line policies if needed:

```
## =================== VERSION ===================
AWSTemplateFormatVersion: '2010-09-09'

## =================== DESCRIPTION ===================
Description: >-
  Creates an IAM User if none is present, optionally manage to which IAM group(s) the user belongs to
  Optionally embed AWS managed policies, customer managed policies and inline policies in the user

## =================== METADATA ===================
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      -
        Label:
          default: Set a one-time password
        Parameters:
          - paramOneTimePassword
    ParameterLabels:
      paramOneTimePassword:
        default: ' '

## =================== PARAMETERS ===================
Parameters:
  paramOneTimePassword:
    Description: The user will have to pick a new password upon first sign in
    Type: String
    NoEcho: 'true' # mask the parameter value as asterisks (*****) to prevent it from being displayed in the console, CLI, or API
    ConstraintDescription: Password must be between 8 and 32 characters, start with lowercase or uppercase letter, and can be alphanumeric with the following special characters !@#$%&

## =================== MAPPINGS ===================
Mappings:
  mapUserData:
    UserName:
      AWS: testuser
      Real: 'Test User'
    Groups:
      Group: BackendDev
    Tags:
      product: testProduct
  mapManagedPolicies:
    CustomerManagedPolicies:
      ARN: None

## =================== CONDITIONS ===================
Conditions:
  hasManagedPolicy:
    !Not [!Equals [!FindInMap [mapManagedPolicies, CustomerManagedPolicies, ARN], 'None'] ]

## =================== RESOURCES ===================
Resources:
  myInlinePolicyForChangeOwnPassword:
    Type: 'AWS::IAM::Policy'
    Properties:
      PolicyName: IAM-ChangeOwnPassword
      PolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Sid: ViewAccountPasswordRequirements
            Action: 'iam:GetAccountPasswordPolicy'
            Resource: "*"
          - Effect: Allow
            Sid: ChangeOwnPassword
            Action:
              - 'iam:GetUser'
              - 'iam:ChangePassword'
            Resource: !Join ['', [arn:aws:iam::*:user/,!FindInMap [mapUserData, UserName, AWS]]]
      Users:
        - !FindInMap [mapUserData, UserName, AWS]
  myUser:
    Type: 'AWS::IAM::User'
    Properties:
      UserName: !FindInMap [mapUserData, UserName, AWS]
      LoginProfile:
        Password: !Ref paramOneTimePassword
        PasswordResetRequired: true
      Groups:
        - !FindInMap [mapUserData, Groups, Group]
      ManagedPolicyArns: # list of ARNs of IAM managed policies that you want to attach to the user
        - !If [ hasManagedPolicy, !FindInMap [mapManagedPolicies, CustomerManagedPolicies, ARN], !Ref "AWS::NoValue" ]
      Tags:
        - Key: name
          Value: !FindInMap [mapUserData, UserName, Real]
        - Key: product
          Value: !FindInMap [mapUserData, Tags, product]
  WaitCondition:
    Type: AWS::CloudFormation::WaitCondition
    DependsOn: myUser
    Properties:
      # Handle: !Ref WaitHandle
      Timeout: 120
      Count: 1

## =================== OUTPUT ===================
Outputs:
  outputArn:
    Description: User ARN
    Value: !GetAtt myUser.Arn
  outputName:
    Description: User name
    Value: !Ref myUser

```

Thing is that there's a race condition between the moment the user creation is triggered, and the moment CF tries to create/attach the in-line policy to the user: if the user haven't finished creating, the stack deployment will fail.

I read about using `WaitCondition` but I'm not really sure how to employ it here with IAM resources. I'd greatly appreciate any feedback that can point me into the right direction.

Thanks

# Answer
> 1 votes
It turns out I was doing it wrong due to me being new to CloudFormation.

For anyone else looking into how to attach both managed and inline policies to a user at the moment of user creation, this is how I do it which *incidentally* is pretty well documented by the AWS folks:

```
## =================== VERSION ===================
AWSTemplateFormatVersion: '2010-09-09'

## =================== DESCRIPTION ===================
Description: >-
  Creates an IAM User if none is present, optionally manage to which IAM group(s) the user belongs to
  Optionally embed AWS managed policies, customer managed policies and inline policies in the user

## =================== METADATA ===================
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      -
        Label:
          default: Set a one-time password
        Parameters:
          - paramOneTimePassword
    ParameterLabels:
      paramOneTimePassword:
        default: ' '

## =================== PARAMETERS ===================
Parameters:
  paramOneTimePassword:
    Description: The user will have to pick a new password upon first sign in
    Type: String
    NoEcho: 'true' # mask the parameter value as asterisks (*****) to prevent it from being displayed in the console, CLI, or API
    ConstraintDescription: Password must be between 8 and 32 characters, start with lowercase or uppercase letter, and can be alphanumeric with the following special characters !@#$%&

## =================== MAPPINGS ===================
Mappings:
  mapUserData:
    UserName:
      AWS: testuser
      Real: 'Test User'
    Groups:
      Group: BackendDev
    Tags:
      product: testProduct
  mapManagedPolicies:
    CustomerManagedPolicies:
      ARN: None

## =================== CONDITIONS ===================
Conditions:
  hasManagedPolicy:
    !Not [!Equals [!FindInMap [mapManagedPolicies, CustomerManagedPolicies, ARN], 'None'] ]

## =================== RESOURCES ===================
Resources:
  # myInlinePolicyForChangeOwnPassword:
  #   Type: 'AWS::IAM::Policy'
  #   Properties:
  myUser:
    Type: 'AWS::IAM::User'
    Properties:
      UserName: !FindInMap [mapUserData, UserName, AWS]
      LoginProfile:
        Password: !Ref paramOneTimePassword
        PasswordResetRequired: true
      ManagedPolicyArns: # list of ARNs of IAM managed policies that you want to attach to the user
        - !If [ hasManagedPolicy, !FindInMap [mapManagedPolicies, CustomerManagedPolicies, ARN], !Ref "AWS::NoValue" ]
      Policies:
        - PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Sid: ViewAccountPasswordRequirements
                Action: 'iam:GetAccountPasswordPolicy'
                Resource: "*"
              - Effect: Allow
                Sid: ChangeOwnPassword
                Action:
                  - 'iam:GetUser'
                  - 'iam:ChangePassword'
                Resource: !Join ['', [arn:aws:iam::*:user/,!FindInMap [mapUserData, UserName, AWS]]]
          PolicyName: IAM-ChangeOwnPassword
      Groups:
        - !FindInMap [mapUserData, Groups, Group]
      Tags:
        - Key: name
          Value: !FindInMap [mapUserData, UserName, Real]
        - Key: product
          Value: !FindInMap [mapUserData, Tags, product]

## =================== OUTPUT ===================
Outputs:
  outputArn:
    Description: User ARN
    Value: !GetAtt myUser.Arn
  outputName:
    Description: User name
    Value: !Ref myUser

```

---
Tags: aws-cloudformation
---


title: Team City Server issue after upgrade

text:
# Question
Title: Team City Server issue after upgrade
Recently we have upgraded TC to the latest version (TeamCity Professional 2021.1.1 (build 92714)).

After upgrade builds run fine when a new VCS root is created, but after a push is made to the repository, VCS root falls over when trying to detect the changes. Based on the Stack trace it looks like there is a problem when retrieving free memory. Here is the stack trace:

```
java.lang.NumberFormatException: For input string: "18446744073709551615"
    at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
    at java.lang.Long.parseLong(Long.java:592)
    at java.lang.Long.parseLong(Long.java:631)
    at jdk.internal.platform.cgroupv1.SubSystem.convertStringToLong(SubSystem.java:159)
    at jdk.internal.platform.cgroupv1.SubSystem.getLongValue(SubSystem.java:153)
    at jdk.internal.platform.cgroupv1.Metrics.getMemoryLimit(Metrics.java:410)
    at sun.management.OperatingSystemImpl.getFreePhysicalMemorySize(OperatingSystemImpl.java:99)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitServerUtil.getFreePhysicalMemorySize(GitServerUtil.java:541)
    at jetbrains.buildServer.buildTriggers.vcs.git.ProcessXmxProvider.getDefaultStartXmx(ProcessXmxProvider.java:151)
    at jetbrains.buildServer.buildTriggers.vcs.git.ProcessXmxProvider.<init>(ProcessXmxProvider.java:63)
    at jetbrains.buildServer.buildTriggers.vcs.git.FetchCommandImpl.fetchInSeparateProcess(FetchCommandImpl.java:100)
    at jetbrains.buildServer.buildTriggers.vcs.git.FetchCommandImpl.fetch(FetchCommandImpl.java:89)
    at jetbrains.buildServer.buildTriggers.vcs.git.CommitLoaderImpl.doFetch(CommitLoaderImpl.java:126)
    at jetbrains.buildServer.buildTriggers.vcs.git.CommitLoaderImpl.loadCommits(CommitLoaderImpl.java:162)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitCollectChangesPolicy$FetchContext.fetchIfNoCommitsOrFail(GitCollectChangesPolicy.java:238)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitCollectChangesPolicy.ensureRepositoryStateLoadedFor(GitCollectChangesPolicy.java:128)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitCollectChangesPolicy.lambda$collectChanges$0(GitCollectChangesPolicy.java:82)
    at jetbrains.buildServer.buildTriggers.vcs.git.RepositoryManagerImpl.runWithDisabledRemove(RepositoryManagerImpl.java:256)
    at jetbrains.buildServer.buildTriggers.vcs.git.GitCollectChangesPolicy.collectChanges(GitCollectChangesPolicy.java:76)
    at jetbrains.vcs.api.services.collectChanges.CollectRepositoryChangesPolicyFactory$3.collectChanges(CollectRepositoryChangesPolicyFactory.java:5)
    at jetbrains.vcs.api.services.collectChanges.CollectChangesServiceProvider$1.collectChanges(CollectChangesServiceProvider.java:10)
    at jetbrains.buildServer.buildTriggers.vcs.LoadChanges$CollectChangesForCheckoutRules.lambda$call$1(LoadChanges.java:3)
    at jetbrains.buildServer.serverSide.impl.BaseAccessChecker.runWithDisabledChecks(BaseAccessChecker.java:3)
    at jetbrains.buildServer.serverSide.impl.SecondaryNodeSecurityManager.executeSafe(SecondaryNodeSecurityManager.java:10)
    at jetbrains.buildServer.buildTriggers.vcs.LoadChanges$CollectChangesForCheckoutRules.call(LoadChanges.java:5)
    at jetbrains.buildServer.buildTriggers.vcs.LoadChanges$CollectChangesForCheckoutRules.call(LoadChanges.java:11)
    at jetbrains.buildServer.vcs.VcsOperationProgressProviderImpl.runWithProgress(VcsOperationProgressProviderImpl.java:29)
    at jetbrains.buildServer.buildTriggers.vcs.LoadChanges.collectChangesWithProgress(LoadChanges.java:15)
    at jetbrains.buildServer.buildTriggers.vcs.LoadChanges.collectChangesForCheckoutRules(LoadChanges.java:18)
    at jetbrains.buildServer.buildTriggers.vcs.LoadChanges.collectChangesByCombinedRule(LoadChanges.java:9)
    at jetbrains.buildServer.buildTriggers.vcs.LoadChanges.collectChanges(LoadChanges.java:66)
    at jetbrains.buildServer.buildTriggers.vcs.VcsRootChangesLoader$RunLoadChanges.run(VcsRootChangesLoader.java:55)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
    at jetbrains.buildServer.buildTriggers.vcs.VcsRootChangesLoader.loadChanges(VcsRootChangesLoader.java:114)
    at jetbrains.buildServer.vcs.impl.VcsChangesFetcher$LoadChangesForRoot.run(VcsChangesFetcher.java:26)
    at jetbrains.buildServer.vcs.impl.ImmediateFutureExecService$2.call(ImmediateFutureExecService.java:2)
    at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:63)
    at jetbrains.buildServer.serverSide.impl.ImmediateFuture.get(ImmediateFuture.java:73)
    at jetbrains.buildServer.vcs.impl.VcsChangesLoader.waitForTasksToComplete(VcsChangesLoader.java:38)
    at jetbrains.buildServer.vcs.impl.VcsChangesLoader.doLoadChanges(VcsChangesLoader.java:92)
    at jetbrains.buildServer.vcs.impl.VcsChangesLoader.tryLoadChanges(VcsChangesLoader.java:139)
    at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction$1.run(VcsModificationChecker.java:18)
    at jetbrains.buildServer.util.NamedThreadFactory.executeWithNewThreadName(NamedThreadFactory.java:76)
    at jetbrains.buildServer.serverSide.impl.VcsModificationChecker$CollectChangesAction.run(VcsModificationChecker.java:5)
    at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

```

Any hints towards what could be the cause?

# Answer
> 1 votes
Just in case someone stumbles on to this question, maybe this will save some time. The issue was with JRE version. Even though Team City documentation states that it supports Java 8. The issue was caused by the JRE version. Upgrading to JDK (and JRE) 11 fixed the problem with number format conversion.

---
Tags: linux, java, teamcity
---


# Question
Title: Using access tokens on Dockerfile to clone repository: RUN command error
In a Dockerfile, I'm trying to clone several Gitlab repositories that are specified in a file, one per line, using an access token passed as an environment variable set in `docker-compose.yml`:

```
COPY my-repos.txt my-repos.txt
RUN 'cat my-repos.txt | while read REPO; do git clone https://oauth2:$GITLAB_ACCESS_TOKEN@gitlab2.<domain>.com:16443/$REPO.git; done'

```

But I'm getting the following error:

```
/bin/sh: 1: cat my-repos.txt | while read REPO; do git clone https://oauth2:$GITLAB_ACCESS_TOKEN@gitlab2.<domain>.com:16443/$REPO.git; done: not found
The command '/bin/sh -c 'cat my-repos.txt | while read REPO; do git clone https://oauth2:$GITLAB_ACCESS_TOKEN@gitlab2.<domain>.com:16443/$REPO.git; done'' returned a non-zero code: 127

```

Running without quotes gives another error:

```
Cloning into 'ML_tools'...
remote: HTTP Basic: Access denied
fatal: Authentication failed for 'https://gitlab2.<domain>.com:16443/<correct_repo>.git/'
The command '/bin/sh -c cat my-repos.txt | while read REPO; do git clone https://oauth2:$GITLAB_ACCESS_TOKEN@gitlab2.<domain>.com:16443/$REPO.git; done' returned a non-zero code: 128

```

The thing is, if I comment out the RUN line, build the image and enter the container, running manually the command works perfectly. The environment variable is correctly set. Is maybe some variable out of scope at build time?

# Answer
> 1 votes
The problem was that I was declaring:

```
ARG GITLAB_ACCESS_TOKEN

```

BEFORE the FROM statement and it has to be done AFTER it.

---
Tags: docker, docker-compose, dockerfile, bash
---


# Question
Title: What is a good strategy to reliably and automatically perform remote system tests of a web service?
In a company I work for, we have deployed many services: a major part of them are small and rather rarely used, but an outage would be a problem nonetheless. At some point I realized that manual testing of whether they work properly requires too many resources and it might be a good idea to automate the work.

My first thought was that self-hosting this piece of infrastructure means that I'd have to maintain two separate critical ecosystems: if I run the tests on the same server (or even hosting provider) that my services run on, downtime might bring both systems down and I would get no notification of a failure. Because of that, I decided to choose paid solutions.

I researched the topic and found many tools like Runscope, but most of them were rather basic request-response validation tools: if the server doesn't time out or respond with an error, the tools assume that everything is OK. This doesn't work well when a website is not open to unauthenticated users: crafting a POST login query is in many cases not feasible (e.g. because of CSRF) and means that I need browser-based automated tests, as opposed to query-based ones. I was suggested to expose an unauthenticated "sanity test" endpoint, but this feels too synthetic: ideally, the test would resemble user interaction as much as it's reasonably possible.

My current solution is to use Pingdom's transaction/synthetic tests, but I quickly discovered multiple problems:

1. Pricing. Pingdom charges 490,00 € yearly for 10 automated browsed-based test scenarios with failure notifications,
2. Security. The current solution requires me to upload credentials to the tested applications in order to test anything beyond the login wall,
3. "who's watching the watchmen" problem. I solved this by exposing a separate service which keeps track of when it was last contacted and sends a separate set of notifications if Pingdom didn't contact it for some time.

I feel like this is such a common problem, I cannot believe there are no popular/standard solutions or strategies that would address it. Is my approach somehow wrong, or is it something that is usually developed in a project-specific manner?

# Answer
For less trafficked parts of your web service, synthetic transactions are a good option.

There are some trade-offs to be made when choosing either using an *API* to or using a *full browser* style check.

*API* checks require changing the application being tested. I'd try and go this route first, but it is dependant on the functionality you need to test. These can be considered health endpoints. Some example implementations are Beamly S4 or the Hootsuite Health Checks. There are many sites which can hit these endpoints as you mentioned eg. Pingdom, FreshPing or Datadog Synthetics.

Full browser style checks do not require changes to the application, which means they are easy to add. As they use contents of the rendered webpage to navigate around, they can be brittle. Changes to the layout or CSS of the page might break the tests that have been created. Options here include BrowserStack or Datadog Synthetics.

On running this yourself, you can also run into the problem that certain locations, regions, POPs for CDN's might have issues reaching your site. This is why most of these services offer many locations to hit your site from, so you can catch those issues which might only affect some of your customers. So I'd definitely recommend going the hosted route.

From a security perspective, these systems will definitely need credentials. If you can't lock down the account used by these services in your application to read only, or only provide permissions to certain areas on your app, an option might be to rotate credentials. For example, create job that runs daily that changes the service accounts, and updates the provider with the new password. For example, using the Datadog Synthetics Terraform resource.

For "watching the watcher": As you will have regular load on your application now based on the synthetic transactions, you could instrument your application or parse the logs to ensure the transactions are coming through.
> 4 votes

---
Tags: automation, monitoring, testing
---


# Question
Title: Best Practice for Kubernetes Development Environment
When using kubernetes with minkube locally, what is best practice for a development environment?

When using docker, in development I have mounted a volume with the source code, so I can modify code and see the changes in real-time without rebuilding the image. In production I bake the app code into the image for obvious reasons.

What do people do when using k8s/minikube for development. Same idea? Mount a volume pointing to where the app code lives on the host machine so they don't have to push images? If so isn't editing the deployment image the main way the pods update?

# Answer
> 1 votes
It's really up to your personal preference, but for minikube you can certainly follow the same standards that you would for production.

If your production environment will use some sort of volume mount for your application code, then follow that same workflow for local dev and just have a pod running that mounts to your local code.

If your production environment expects that application code inside of the container, then set up a local registry in minikube, and build/push images into that registry.

# Answer
> 0 votes
I'm looking at tilt, and so far it's awesome for this.

# Answer
> 0 votes
I would say the reason you are wanting to develop against Kubernetes locally is to see how the application works with within the pod, and how it interacts with other dependencies inside the Kubernetes instance.

In this case you want to go for speed of getting the code into the container. Here you want something like Tilt Live Update which will run your build of your code locally, and then sync it into the running container. This avoids needing to build a container and deploy a new resource each time.

---
Tags: docker, kubernetes, environments
---


# Question
Title: Using ansible to update remote Ubuntu servers that have no external internet access
I am quite new to Ansible, and have to been able to use it to install packages on 80 remote Ubuntu servers that have no external internet access.

Is it possible to use Ansible to update/patch these Ubuntu servers that have no external internet access? Something similar to running "sudo apt upgrade" on these servers if they had internet access. Is using Ansible a viable solution/option?

I have researched online but no joy. Any assistance or guidance on how to use Ansible to patch/upgrade remote Ubuntu servers that have no external internet access?

# Answer
If I understand the question correctly, the problem isn't really related to ansible at all. What you need is a package repository on the local network that can cache or mirror public packages.

There are a number of deb specific options like rsyncmirror or apt-mirror. There are also generic options that hold many different types of binaries and package feeds, like artifactory or sonatype nexus. On top of being able to get your packages into your private network they offer varying levels of control over what packages are made available within the environment. Once you have a local repository on the network, you could use ansible to update the apt sources.list to pull from the local url instead of the internet based repos.

If you don't need any control over what packages are available, making internet repos available through a proxy is also an option, but the above solutions scale better long term.
> 1 votes

---
Tags: ansible, ubuntu
---


# Question
Title: Access denied when pulling private registry image using helm with gitlab runner helm chart and ci job
I have a kubernetes cluster with 1 master and 2 workers. All nodes have their IP address. Let's call them like this:

* master-0
* worker-0
* worker-1

The network pod policy and all my nodes communication are setting up correctly, all works perfectly. If I specify this infrastructure, it's just to be more specific about my case.

Using helm I have created a chart which deploy a basic nginx. It's a docker image that I build on my private gitlab registry.

With the gitlab ci, I have created a job which used two functions:

```
# Init helm client on k8s cluster for using helm with gitlab runner
function init_helm() {
  docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" "$CI_REGISTRY"
  mkdir -p /etc/deploy
  echo ${kube_config} | base64 -d > ${KUBECONFIG}
  kubectl config use-context ${K8S_CURRENT_CONTEXT}
  helm init --client-only
  helm repo add stable https://kubernetes-charts.storage.googleapis.com/
  helm repo add incubator https://kubernetes-charts-incubator.storage.googleapis.com/
  helm repo update
}

# Deploy latest tagged image on k8s cluster
function deploy_k8s_cluster() {
  echo "Create and apply secret for docker gitlab runner access to gitlab private registry ..."
  kubectl create secret -n "$KUBERNETES_NAMESPACE_OVERWRITE" \
    docker-registry gitlab-registry \
    --docker-server="https://registry.gitlab.com/v2/" \
    --docker-username="${CI_DEPLOY_USER:-$CI_REGISTRY_USER}" \
    --docker-password="${CI_DEPLOY_PASSWORD:-$CI_REGISTRY_PASSWORD}" \
    --docker-email="$GITLAB_USER_EMAIL" \
    -o yaml --dry-run | kubectl replace -n "$KUBERNETES_NAMESPACE_OVERWRITE" --force -f -
  echo "Build helm dependancies in $CHART_TEMPLATE"
  cd $CHART_TEMPLATE/
  helm dep build
  export DEPLOYS="$(helm ls | grep $PROJECT_NAME | wc -l)"
  if [[ ${DEPLOYS}  -eq 0 ]]; then
    echo "Creating the new chart ..."
    helm install --name ${PROJECT_NAME} --namespace=${KUBERNETES_NAMESPACE_OVERWRITE} . -f values.yaml
  else
  echo "Updating the chart ..."
    helm upgrade ${PROJECT_NAME} --namespace=${KUBERNETES_NAMESPACE_OVERWRITE} . -f values.yaml
  fi
}

```

The first function allow the gitlabrunner to login with docker, init helm and kubectl. The second to deploy on the cluster my image.

All the process works well, e-g my jobs are passed on the gitlab ci, no error occurred except for the deployment of the pod.

**Indeed I have this error**:

```
Failed to pull image "registry.gitlab.com/path/to/repo/project/image:TAG_NUMBER": rpc error: code
= Unknown desc = Error response from daemon: Get https://registry.gitlab.com/v2/path/to/repo/project/image/manifests/image:TAG_NUMBER: denied: access forbidden

```

To be more specific, I am using gitlab-runner helm chart and this the config of the chart:

```
## GitLab Runner Image
##
## By default it's using gitlab/gitlab-runner:alpine-v{VERSION}
## where {VERSION} is taken from Chart.yaml from appVersion field
##
## ref: https://hub.docker.com/r/gitlab/gitlab-runner/tags/
##
# image: gitlab/gitlab-runner:alpine-v11.6.0

## Specify a imagePullPolicy
## 'Always' if imageTag is 'latest', else set to 'IfNotPresent'
## ref: http://kubernetes.io/docs/user-guide/images/#pre-pulling-images
##
imagePullPolicy: IfNotPresent

## The GitLab Server URL (with protocol) that want to register the runner against
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-register
##
gitlabUrl: https://gitlab.com/

## The Registration Token for adding new Runners to the GitLab Server. This must
## be retrieved from your GitLab Instance.
## ref: https://docs.gitlab.com/ce/ci/runners/README.html#creating-and-registering-a-runner
##
runnerRegistrationToken: "<token>"

## The Runner Token for adding new Runners to the GitLab Server. This must
## be retrieved from your GitLab Instance. It is token of already registered runner.
## ref: (we don't yet have docs for that, but we want to use existing token)
##
# runnerToken: ""
#
## Unregister all runners before termination
##
## Updating the runner's chart version or configuration will cause the runner container
## to be terminated and created again. This may cause your Gitlab instance to reference
## non-existant runners. Un-registering the runner before termination mitigates this issue.
## ref: https://docs.gitlab.com/runner/commands/README.html#gitlab-runner-unregister
##
unregisterRunners: true

## Set the certsSecretName in order to pass custom certficates for GitLab Runner to use
## Provide resource name for a Kubernetes Secret Object in the same namespace,
## this is used to populate the /etc/gitlab-runner/certs directory
## ref: https://docs.gitlab.com/runner/configuration/tls-self-signed.html#supported-options-for-self-signed-certificates
##
# certsSecretName:

## Configure the maximum number of concurrent jobs
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
concurrent: 10

## Defines in seconds how often to check GitLab for a new builds
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
checkInterval: 30

## Configure GitLab Runner's logging level. Available values are: debug, info, warn, error, fatal, panic
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-global-section
##
# logLevel:

## For RBAC support:
rbac:
  create: true

  ## Run the gitlab-bastion container with the ability to deploy/manage containers of jobs
  ## cluster-wide or only within namespace
  clusterWideAccess: true

  ## Use the following Kubernetes Service Account name if RBAC is disabled in this Helm chart (see rbac.create)
  ##
  serviceAccountName: default

## Configure integrated Prometheus metrics exporter
## ref: https://docs.gitlab.com/runner/monitoring/#configuration-of-the-metrics-http-server
metrics:
  enabled: true

## Configuration for the Pods that that the runner launches for each new job
##
runners:
  ## Default container image to use for builds when none is specified
  ##
  image: ubuntu:16.04

  ## Specify one or more imagePullSecrets
  ##
  ## ref: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/
  ##
  imagePullSecrets: ["namespace-1", "namespace-2", "default"]

  ## Specify the image pull policy: never, if-not-present, always. The cluster default will be used if not set.
  ##
  # imagePullPolicy: ""

  ## Specify whether the runner should be locked to a specific project: true, false. Defaults to true.
  ##
  # locked: true

  ## Specify the tags associated with the runner. Comma-separated list of tags.
  ##
  ## ref: https://docs.gitlab.com/ce/ci/runners/#using-tags
  ##
  tags: my-tag-1, my-tag-2"

  ## Run all containers with the privileged flag enabled
  ## This will allow the docker:dind image to run if you need to run Docker
  ## commands. Please read the docs before turning this on:
  ## ref: https://docs.gitlab.com/runner/executors/kubernetes.html#using-docker-dind
  ##
  privileged: true

  ## The name of the secret containing runner-token and runner-registration-token
  # secret: gitlab-runner

  ## Namespace to run Kubernetes jobs in (defaults to the same namespace of this release)
  ##
  # namespace:

  # Regular expression to validate the contents of the namespace overwrite environment variable (documented following).
  # When empty, it disables the namespace overwrite feature
  namespace_overwrite_allowed: overrided-namespace-*

  ## Distributed runners caching
  ## ref: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/docs/configuration/autoscale.md#distributed-runners-caching
  ##
  ## If you want to use s3 based distributing caching:
  ## First of all you need to uncomment General settings and S3 settings sections.
  ##
  ## Create a secret 's3access' containing 'accesskey' & 'secretkey'
  ## ref: https://aws.amazon.com/blogs/security/wheres-my-secret-access-key/
  ##
  ## $ kubectl create secret generic s3access \
  ##   --from-literal=accesskey="YourAccessKey" \
  ##   --from-literal=secretkey="YourSecretKey"
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  ##
  ## If you want to use gcs based distributing caching:
  ## First of all you need to uncomment General settings and GCS settings sections.
  ##
  ## Access using credentials file:
  ## Create a secret 'google-application-credentials' containing your application credentials file.
  ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-cache-gcs-section
  ## You could configure
  ## $ kubectl create secret generic google-application-credentials \
  ##   --from-file=gcs-applicaton-credentials-file=./path-to-your-google-application-credentials-file.json
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  ##
  ## Access using access-id and private-key:
  ## Create a secret 'gcsaccess' containing 'gcs-access-id' & 'gcs-private-key'.
  ## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-cache-gcs-section
  ## You could configure
  ## $ kubectl create secret generic gcsaccess \
  ##   --from-literal=gcs-access-id="YourAccessID" \
  ##   --from-literal=gcs-private-key="YourPrivateKey"
  ## ref: https://kubernetes.io/docs/concepts/configuration/secret/
  cache: {}
    ## General settings
    # cacheType: s3
    # cachePath: "cache"
    # cacheShared: true

    ## S3 settings
    # s3ServerAddress: s3.amazonaws.com
    # s3BucketName:
    # s3BucketLocation:
    # s3CacheInsecure: false
    # secretName: s3access

    ## GCS settings
    # gcsBucketName:
    ## Use this line for access using access-id and private-key
    # secretName: gcsaccess
    ## Use this line for access using google-application-credentials file
    # secretName: google-application-credential

  ## Build Container specific configuration
  ##
  builds:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi

  ## Service Container specific configuration
  ##
  services:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi

  ## Helper Container specific configuration
  ##
  helpers:
    # cpuLimit: 200m
    # memoryLimit: 256Mi
    cpuRequests: 100m
    memoryRequests: 128Mi
    image: gitlab/gitlab-runner-helper:x86_64-latest

  ## Service Account to be used for runners
  ##
  # serviceAccountName:

  ## If Gitlab is not reachable through $CI_SERVER_URL
  ##
  # cloneUrl:

  ## Specify node labels for CI job pods assignment
  ## ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/
  ##
  nodeSelector: {}
    # gitlab: true

## Configure resource requests and limits
## ref: http://kubernetes.io/docs/user-guide/compute-resources/
##
resources:
  # limits:
  #   memory: 256Mi
  #   cpu: 200m
  requests:
    memory: 128Mi
    cpu: 100m

## Affinity for pod assignment
## Ref: https://kubernetes.io/docs/concepts/configuration/assign-pod-node/#affinity-and-anti-affinity
##
affinity: {}

## Node labels for pod assignment
## Ref: https://kubernetes.io/docs/user-guide/node-selection/
##
nodeSelector: {}
  # Example: The gitlab runner manager should not run on spot instances so you can assign
  # them to the regular worker nodes only.
  # node-role.kubernetes.io/worker: "true"

## List of node taints to tolerate (requires Kubernetes >= 1.6)
## Ref: https://kubernetes.io/docs/concepts/configuration/taint-and-toleration/
##
tolerations: []
  # Example: Regular worker nodes may have a taint, thus you need to tolerate the taint
  # when you assign the gitlab runner manager with nodeSelector or affinity to the nodes.
  # - key: "node-role.kubernetes.io/worker"
  #   operator: "Exists"

## Configure environment variables that will be present when the registration command runs
## This provides further control over the registration process and the config.toml file
## ref: `gitlab-runner register --help`
## ref: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
##
envVars:
  - name: RUNNER_EXECUTOR
    value: kubernetes

```

As you can see, I created a **secret** on my ci job, no error occurred here too. In my chart, I declare this same secret (by his name) in `values.yaml` file, which allow `deployment.yaml` to use it.

Notes:

* If it was the registry, e-g my image repository `registry.gitlab.com/path/to/repo/project/image:TAG_NUMBER` , I will get an error like my image does not exist or something like this. To be more precise, just notice that the var **TAG\_NUMBER** is set in my jobs, and I recover the right value. So I don't think my problem comes from the image repository url.
* I have tested my secret with a simple pod deployment. All works well with these credentials, the pod is ok (I have use the `kubectl create -f <filename>.yaml` command. I juste read this post about kubernetes and gitlab: https://docs.gitlab.com/ee/topics/autodevops/#private-project-support. Maybe it links with my issue.

# Answer
> 1 votes
The error you are getting is from your Kubernetes cluster being unable to pull container image from GitLab.

Providing a temporary credential to pull the containers as deploy time is not a great solution. For example, say your Kubernetes cluster needs to scale up by adding another node, the node will come up and try and pull the container, but the secret will be invalid so you won't be able to scale.

I'd create a GitLab static deployment token which has permission to pull images from the GitLab registry: https://docs.gitlab.com/ee/user/project/deploy\_tokens/#read-container-registry-images

You need to provide this secret to Kubernetes, for that see this document: https://kubernetes.io/docs/tasks/configure-pod-container/pull-image-private-registry/

This should allow you to just do a `helm install` or `helm upgrade` successfully.

---
Tags: docker, kubernetes, gitlab, helm, gitlab-ci-runner
---


# Question
Title: Strategies for sharing Ansible vaults with root/administrator credentials?
Historically we've been a one man devops team and stored all our passwords in Ansible vaults - including things like domain admin passwords and root account passwords.

However we're now growing and adding new team members and this creates a challenge. If we give all members of the devops team the vault and vault password they all suddenly have domain admin and root credentials - which we're hesitant to do. Some members of our team are more junior and we're trying to avoid handing out proverbial keys to the kingdom to everyone on the team.

However the Ansible scripts still need these credentials to function - to do things like join a windows active directory domain.

Is there a way with Ansible, Ansible tower or awx to grant a devops person the ability to use a credential without being able to decrypt and read it in clear text?

If not what other strategies do their teams use to address this situation?

# Answer
This sounds like you are applying Ansible playbooks locally on peoples workstations.

The best way to protect this would be applying the playbooks using a remote system, like AWX or a CI system such as GitHub Actions or TeamCity. The CI system would have the ability to decrypt the vault and run the playbooks.

Mostly the CI system can expose the vault password to the job using an environment variable (that is hidden from end users and from the logs). You can then write a simple script which provides the password to the vault. Details here: https://docs.ansible.com/ansible/latest/user\_guide/vault.html#storing-passwords-in-third-party-tools-with-vault-password-client-scripts

For example:

```
#!/usr/bin/env python

import os
import sys

if "ANSIBLE_VAULT_PASS" in os.environ:
  sys.stdout.write('%s\n' % os.environ['ANSIBLE_VAULT_PASS'])
  sys.exit(0)

sys.exit(1)

```
> 1 votes

---
Tags: ansible, security, passwords
---


# Question
Title: Should i stop native apache and mysql when running docker?
I have installed MySQL and apache using brew on my local machine and have been using them for several php project environments. Now i want to start learning docker to containerize a new Laravel environment (example) . Before i start using docker on my local machine, should i stop the httpd and mysql service ? will the 'php' and 'mysql' command have a conflict ?

# Answer
> 1 votes
The only issue you might have running the same services in containers as locally on your machine is a port conflict. Let's take httpd as an example:

* You try and start the container and you get `listen tcp 0.0.0.0:80: bind: address already in use`. This is because a HTTP server is already running on your machine.
* Tell docker to bring the container up using a different local port: `docker run -p 80:8080 <your container, commands etc>`. This will tell Docker to bind port `80` in the container to port `8080` on your machine.
* You can now access both `http://localhost:80` (your machine) and `http://localhost:8080` (the container).

---
Tags: docker
---


# Question
Title: Sorting out automated testing in CI pipeline and getting conflicting information related to building, unit testing and Docker images
I seem to get a unique response for every person that I ask, or every article that I read, which is just confusing me even more.

Basically, I'm trying to setup automated testing in the CI pipeline which I'm pretty new to. The stack is basically these things:

* Django API service
* React Admin Portal service
* React Client Portal service
* PostgreSQL
* Redis
* Azure Kubernetes Services
* Azure DevOps Pipelines with Microsoft Hosted Agent

I was originally under the impression thanks to advice here, which was pretty consistent among respondents, that basically my CI pipeline should be the following:

1. Build the services.
2. Run unit tests.
3. If passing, `COPY` the build into the Docker image.
4. Push Docker image to Container Registry.
5. Deploy images to production-like Kubernetes (using Kind, Minikube, etc.) in a pipeline VM ("Hosted Agent" in Azure DevOps terms).
6. Run integration tests.
7. Run E2E/Selenium tests.
8. If passing, merge feature branch to production.

Started talking to someone else about it and they basically said:

> Unit \> integration \> e2e. Test as much as you can at each step without pissing the devs off. Testing in parallel is better because it lessens the time taken. If a failure in any test type means the pipeline breaks the only thing that controls when you run the test is if you can do it before the app is running (unit, functional) or after compile and deployment (integration/e2e).
>
> All of that aside: **DON'T COPY COMPILED CODE INTO A CONTAINER**. Such an anti-pattern and bad practice. The Dockerfile should include the entire build process and remain immutable once built. Copying code into your container is the equivalent of passing it to a coworker and assuming it's going to run on their laptop.
>
> Multi-stage builds and development containers containing sdks/build-kits/whatever are the first step, runtime containers that contain only the necessary pieces to run are the second. If they aren't available from the platform provider - .net, java, python etc. (they are) - you can build them both out of the same base container which standardizes both the build and runtime environments.
>
> Use multi-stage Dockerfiles and combine to reduce steps so that containers stay lean.

In other words:

1. Build the services in a Docker image.
2. Run unit tests in the the Docker image.
3. If passing, deploy images to production-like Kubernetes (using Kind, Minikube, etc.) in a pipeline VM ("Hosted Agent" in Azure DevOps terms).
4. Run integration tests.
5. Run E2E/Selenium tests.
6. If passing, push to Container Registry and merge feature branch to production.

Which, unless I'm misinterpreting this, contradicts some other statements I've received like this:

> **Never, ever build in docker images**, because it means we have to install dev-kits in the container. No docker container in production should have any of these kits. For security.

I can see the value of both approaches.

I'm basically just looking for more clarification on this process regarding building, unit testing and Docker images. Suggestions?

# Answer
> 1 votes
You've missed the part of the 1st quote about multi-stage builds and other methods which avoid exactly the problem mentioned in the 2nd quote: the resulting image would consist of it's base image and artifacts from previous stages while being run in the same *build* step of a pipeline.

---
Tags: continuous-integration, azure-pipelines, testing, integration, unit-testing
---


# Question
Title: Best Practice for Container Image Versioning
I'm looking for some guidance on best practices for container image versioning and despite reading a number of the best practice guides they leave some ambiguity.

I understand that most container images use Stable versioning and that the version tag will stay the same when things like frameworks, packages, libraries, etc are updated, but I'm unclear how this strategy ensures that the containers deployed from an image are consistent.

If I build an image today with the tag set to `v1.0.0` and then next month the image is updated to include updates to frameworks, packages, libraries, etc but the application remains unchanged then my understanding of the stable versioning strategy means the new image will keep the tag set to `v1.0.0`.

With this strategy how do I ensure the same version of the image is deployed consistently? Developers, test environments, and production could all believe they are using the same version of the image as the tag is still set to `v1.0.0` but changes in underlying frameworks, packages, and libraries could introduce instability which everyone is unaware of between environments. Presumably this would be even worse if there is a cluster running multiple container instances of a image.

One other dimension is that the frameworks, packages and libraries might be in a parent image and with no changes in the application image itself.

Whilst I could include a build number in the tag for the application images and any parent images that contain the frameworks, packages and libraries, this feels like a huge maintenance overhead as I'd need to create a new dockerfile for the application image every time the parent image is updated due to the parent image tag now including a build number e.g. `from parent-image:v1.0.0-build123`.

Is there a better strategy to solve this problem, or is it simply better to accept that there will be some differences between images using the same stable version tag? I'm not using Kubernetes at the moment but its something that we're likely to use in the future and I'm keen to pick a versioning strategy that will work for Kubernetes and ensure that there aren't "hidden" differences in the container instances deployed on the nodes from the same image version.

# Answer
First a couple key points:

1. Tags are mutable in most registries, so v1 today may be different from v1 a month ago. However clients do need to pull this tag again to see the updates.
2. Multiple tags may point to the same image. So `stable` and `v1` could point to the same image.
3. Pointing to a digest ensures the image is immutable, however you will not receive updates, and if that image becomes untagged, it's possible for some registries to delete it as part of their garbage collection.

With that said, the best practice depends on how you intend to use the image. For something that can be rebuilt in the future and receive security updates, you typically want to depend on a mutable but stable tag. This is particularly important when you don't have some way to automatically update your Dockerfiles with new image tags. However if you control the various ends of the build pipeline, you may want to depend on a digest for relatively reproducible builds and automatically inject that digest when your base image pipeline finishes.

In general, semver in docker tags is handled by building a persistent tag that the image creator intentionally doesn't change. So when `1.0.0` is released and tagged in git, that git tag is not changed, and the resulting image is `v1.0.0` and it is also never changed (by policy). And with the ability to push multiple tags, that same build would also push `v1.0` and `v1`. Then when `1.0.1` is tagged in git, the image `v1.0.1` is built and the tags for `v1.0` and `v1` are modified to point to that new image.

That semver strategy allows for others to depend on the version they need in their Dockerfile with `FROM base-image:v1` so that they will get minor updates, bug fixes, and security patches, without any modification to their Dockerfile, they just need to periodically rebuild and pull updated base images.

When upstream makes a breaking change, they release `2.0.0` in git, tag their image `v2.0.0`, `v2.0`, and `v2`. And anyone that depends on `v1` will not be impacted by that breaking change until they update their Dockerfile to use the new major release.
> 4 votes

# Answer
The most stable way to assure you're deploying a consistent image is to use the digest instead of tags. The digest is like the git sha of a container image. This assures that if someone uploads a new image and tags it with an existing version tag, either accidentally or maliciously, you will continue to pull the known good image. The drawback is digests are not user-friendly.
> 0 votes

---
Tags: docker, kubernetes, containers, versioning
---


# Question
Title: Build a app.exe file. Where to release it?
I have a specific question about Azure DevOps Pipeline CI/CD. I'm use to create CI and CD pipeline for IIS web application deployments. Sometime, I also do Click Once deployments.

This time, I'm using Azure DevOps Pipeline for creating and saving a very simple app.exe. This is a console application I want to make available as a downloadable tool for my company. I plan to create more small similar small app in future. with .NET Core I can easily integrate all my dependencies in the executable package and create a unique exe that can be used by some people from my company.

My question is how can I make this app.exe accessible as a downloadable file. Do I need to create my own FTP system to upload the app.exe and create a web page to give the minimum of information to my users? Or, can I use something from Azure directly? Can I deploy all my app.exe versions as a feed in the Artifact section?

It seems on some Git website like Github or Sourceforge there is a tab that keep all files. I'm looking for something similar on Azure DevOps

example: https://github.com/Argent77/NearInfinity/releases https://github.com/zarunbal/LogExpert/releases

# Answer
A good approach for distributing .Net Core console applications is to make them a dotnet tool and publish them to a Nuget feed. It should be as simple as adding some config to the csproj and run `dotnet pack` This assumes you have a private Nuget feed for your company which Azure DevOps can provide if you don't.
> 3 votes

---
Tags: azure-devops, azure-pipelines, release-management
---


title: Accelerate autoscaled VM boot time

text:
# Question
Title: Accelerate autoscaled VM boot time
We're using Google Cloud Platform with an autoscaled pool of VM behind a google's load-balancer.

When a new VM is created in the pool it starts from a template then executes a startup-script. The startup script is pulling multiple docker images and it takes some precious time.

I'd like to reduce this time. From what i saw on the internet some people prefer to boot from a snapshot but this wouldn't work in my case since the snapshot would give the same ip for all my VM.

Do you have any good practice on how to pull these docker images faster or before creating the new VM ?

# Answer
> 1 votes
To increase the start up time, you need to "bake" (install) more of what is in your start up script into the image.

Snapshotting is *one* method, but I would avoid manually creating instances and then taking snapshots.

A better approach is to automate the creation of these images. Hashicorp Packer is the most common tool used for doing this. You can find the GCP "builder" here

In your case for example, you could install Docker and pull the containers that you want to be running as part of the Packer build.

This build is saved in your GCP account as an image, and your autoscaling group can use this image as its base.

Once the instance comes up, the startup script just applies some basic configuration and tries to start the image which has already been cached.

---
Tags: docker, autoscaling, gcp
---