# Question
Title: Can I persist a Yarn install through different stages in a GitLab pipeline?
I was thinking about a pipeline with the following steps:

1. Install image dependencies (such as java, node, yarn and openssl)
2. Install node packages needed for the build
3. Install PMD
4. Do a code Analysis
5. Build
6. Deploy
7. Run tests on the server
8. Clean up

But the issue here is that I require the tools installed on steps 1 and 2 for the rest of the build. There's a package that is used to run steps 5 to 7, for example, and OpenSSL is required to decrypt a server key in the repository (so GitLab can connect to the server).

I can't seem to find a way to persist Yarn's install through different stages. When the pipeline moves from stage 1 to stage 2, it will use yarn commands and will fail with the 127 error code (command not found), like this:

```
$ yarn global add sfdx-cli
/bin/sh: eval: line 81: yarn: not found

ERROR: Job failed: exit code 127

```

Is it possible to persist the "environment setup" through all the stages? Can `artifacts` be used for this? If so, will I need to setup the commands for every stage? For example: Yarn might be installed in a folder which will be cached, but I need to point the 'yarn' command to the correct binary, right?

# Answer
We install things in the `before_script` section of the `.gitlab-ci.yml` that way on every run of the runner we know what packages exist for each stage.
> 1 votes

---
Tags: gitlab, cache
---


# Question
Title: Is it possible to pass environment variables to multibranch pipeline job in a time of generating it via DSL?
I want to pass some environment variables in the time of generating the multibranch pipelines, so upon the build start, it will already have these variables.

The only thing I found was this envInject plugin which is not supported by multibranch DSL generation.

https://jenkinsci.github.io/job-dsl-plugin/#plugin/envinject

Does anyone know the way to do that?

# Answer
> 1 votes
You can use Jenkins DSL and pass the environment variable as mentioned in below links:

https://jenkins.io/doc/book/pipeline/syntax/#environment https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#withenv-set-environment-variables

# Answer
> 1 votes
This might help you with your problem. In the normal pipeline or any Jenkins job, we can define parameters which can be accessed via say `${env.SOME_VARIABLE}`

My solution for using environment variables in the Jenkins Multibranch Pipeline. Scenario. Say you have a variable named VARIABLE whose value is `123456789`.

**Create a secret text with id and description.**

* Secret Text: 123456789

**NOTE**: In Jenkins, the secret text will be completely hidden.

* Id: VARIABLE

**NOTE**: You can define it as you want. If left empty a random value will be assigned to it in the form as45f2sf-43rs-4sdf-s3f3-329f9bc9ae269

* Description: VARIABLE

**NOTE**: Description for the variable.

Once this is created in your multibranch pipeline Jenkinsfile, you can add the following to access those variables whenever required.

```
pipeline {
  agent any
    environment {
      SECRET_VARIABLE = credentials("VARIABLE")
    }
    stages {
      stage('stage') {
        steps {
          echo "This is a secret variable: $SECRET_VARIABLE"
        }
      }
   }
}

```

\**PS: The variables will appear in a format `****` \**

---
Tags: jenkins, job-dsl
---


# Question
Title: Deploying different versions of the same application stack in the same system
We are a small team with few resources and we have a product which is near completion. The stack comprises of a few Java Spring Applications, a Python Application and a Windows Application and using a Postgresql DB. We have a single physical system where till now, we used to deploy all of these application and test them.

Now we are in a situation where we would need to have multiple versions of this stack running at the same time. For eg. there should be a stable running version and at least 2 versions under testing. We want to achieve this using the single Windows system we have. Is there a way to achieve this? How viable is using multiple Docker images?

# Answer
First of all, I would **not** recommend to use a single system for running stable and testing version both on the same system. There are many reasons for this such as:

* Your `testing` version can cause cpu and memory spikes on system which will ultimately affect your `stable` version and end users.
* Secondly, it will be difficult to manage the dependencies and resources for multiple versions in a single system.
* It can also take a lot of debugging time if the monitoring is not properly setup.

Ideally, you should use 2 different system. One for running the `stable` version and other `testing` versions. It will be easy to manage and always a good practice to keep it on different stack.

Still, if you want to use a single system, then yes, you can use docker which will avoid a lot of conflict issues which can arise. Also, the dependencies and shared libraries can be managed easily with different docker images and running docker containers.

If the team is proficient in docker, you can also use `docker swarm` to setup clusters. You can setup different clusters for `stable` and `testing` versions.

I hope you chose the best possible option and implement it.
> 1 votes

---
Tags: docker, deployment
---


# Question
Title: Sub-subdomain Domain name for different environment
If we have two components in a API

`api` `front-end`

and we have three environments `staging` `qa` `development`

and a domain name `superproduct.com`

What would be the most logical domain spacing:

```
http://development.front-end.superproduct.com
http://development.api.superproduct.com

http://qa.front-end.superproduct.com
http://qa.api.superproduct.com

http://staging.front-end.superproduct.com
http://staging.api.superproduct.com

```

or the other way around:

```
http://front-end.development.superproduct.com
http://api.development.superproduct.com

http://front-end.qa.superproduct.com
http://api.qa.superproduct.com

http://front-end.staging.superproduct.com
http://api.staging.superproduct.com

```

# Answer
Both are completely correct. It can differ organisation by organisation. Also, once you start using one of the domain namespace type, you will gradually remember the format and no issues will come. However, I can see one issue here which is related to SSL.

If you go with any one of the above mentioned namespaces, you need to generate multiple SSL certificates such as `*.api.superproduct.com` and `*.front-end.superproduct.com`.

If you go with other namespace type then SSL certs will be required for `*.staging.superproduct.com`, `*.qa.superproduct.com` and `*.development.superproduct.com`.

To avoid this, I would suggest you to have the domains in either of the following formats.

1. `<component-name>-<env-name>.superproduct.com`

Examples are: `api-qa.superproduct.com`, `api-staging.superproduct.com` etc.

2. `<env-name>-<component-name>.superproduct.com`

Examples are: `qa-api.superproduct.com`, `staging-api.superproduct.com` etc.

In this case, only one SSL certificate for `*.superproduct.com` will be required.

You can go ahead with the best possible option as per your requirements and analysis.
> 2 votes

---
Tags: dns
---


# Question
Title: Compound calculation of GCP GCE SLAs based on number of instances?
GCP publishes a 99.99% uptime SLA for GCE.

How would I calculate an estimated SLA if my application used services deployed on 1,000 GCE instances? Assuming that the services were dependent on each other, would the estimated downtime per year of my application be 1,000 * 0.0001 = 0.10 or 10% per year?

# Answer
> 0 votes
99.99% SLA uptime roughly translates to 52.5 minutes a year of downtime (calculator here). Generally if the service is down, it is down for everything in that geographical area. So if you had 1000 servers in the same geographical area and that area goes out of service, your downtime would be tied together.

However, most of the "big" cloud providers (e.g. Google) provide their services across different regions and zones. By spreading your infrastructure out across different areas, you can make your network more resilient to cloud provider outages.

---
Tags: google-cloud-platform, gcp, sla
---


title: Deploy dist file in docker file

text:
# Question
Title: Deploy dist file in docker file
How to write a docker file to deploy a dist file of an Angular Application In a server. I see most of them have used nginx as a server. Why is it so? Is there any specific reason

I want it to be deployed in Apache using alpine as a base image

# Answer
There are lots of good tutorials for NGINX and it is an easy server to setup with the Angular router. You can still accomplish this with an Apache server, but you may need more steps in your Dockerfile. As a starting place check out the answer on Stack Overflow Deploy Angular 2 to Apache Server. These types of server modifications will need to be baked into your Dockerfile.
> 1 votes

---
Tags: dockerfile, nginx, apache-httpd
---


# Question
Title: Mitigating Docker Compliance Issues - Can I put everything in the `daemon.json`?
I am going through compliance tasks to harden our Docker setup and I am seeing many things like

* Favor using `docker run` with `docker run --pids-limit 100` to prevent forkbombs
* Favor using `docker run` with `docker run --security-opt=no-new-privileges` to be safe

etc...

I want to know if I can apply these settings across the board by, say, putting

```
...
"pids-limit": "100",
"security-opt": "no-new-privileges",
...

```

in `/etc/docker/daemon.json`

I prefer to just have a hardened config on all my machines I am hardening.

Tips and tricks?

Thanks.

# Answer
> 1 votes
Check out the dockerd options. This will apply to all containers managed by the Docker Daemon.

Docker also has quite a bit of documentation on how to enforce different standards and compliance here.

---
Tags: docker, security, devsecops
---


# Question
Title: Why could the encrypted Ansible files not be decrypted although the vault password file has been set?
Follow-up to this Q&A.

According to this and this documentation it should be possible to define a default vault password file by specifying `vault_password_file = ~/.vault_pass.txt` in one of the following files:

> ```
> * ANSIBLE_CONFIG (an environment variable)
> * ansible.cfg (in the current directory)
> * .ansible.cfg (in the home directory)
> * /etc/ansible/ansible.cfg
>
> ```

Although the password file exists and the log indicates that the ansible.cfg is used that contains the definition, the decryption fails:

```
user@host$
Using /etc/ansible/ansible.cfg as config file
ERROR! Decryption failed on /path/to/ansible/group_vars/production/vault

```

**Discussion**

* > Is the pass correct for this vault?

Yes. When ansible is run using `--vault-password-file=~/.vault_pass.txt` the decryption succeeds.

* > Just so that we're clear - the name of the environment variable is `ANSIBLE_VAULT_PASSWORD_FILE`, correct?

No. When this variable is used it works, but the aim is to set it in the ansible.cfg by setting `/etc/ansible/ansible.cfg`.

* > What Ansible version is used?

```
user@host:/dir$ ansible --version
ansible 2.2.2.0
    config file = /etc/ansible/ansible.cfg
    configured module search path = Default w/o overrides

```
* > Does using a full path in the config works?

The decryption failes as well when the full path instead of `~`, i.e. `/home/user/.vault_pass.txt` was specified.

# Answer
> 1 votes
It is unclear why it failed, but setting `vault_password_file = ~/.vault_pass.txt` in `/etc/ansible/ansible.cfg` was sufficient.

# Answer
> 2 votes
I found an issue with Ansible/Python causing a failure to decrypt.

Specifically, Ansible 2.7.10 and 2.7.13 under Python 3.6.8 on Ubuntu 18.04 would not decrypt (using `ansible-vault edit -vvvvv`) a vault file that was identical to file that decrypted OK on macOS 10.13 using Python 3.7.2 with same Ansible versions.

Upgrading to Python 3.7.3 on Ubuntu (using `apt-get install python3.7` on 18.04) fixed this.

Your issue may be different, but if on Python 3.6 or lower, I would try upgrading to 3.7.

# Answer
> -1 votes
According to the docs,

> You can also set ANSIBLE\_VAULT\_PASSWORD\_FILEenvironment variable, e.g. ANSIBLE\_VAULT\_PASSWORD\_FILE=~/.vault\_pass.txtand Ansible will automatically search for the password in that file.

Ensures that you are using the exact same case.

---
Tags: ansible-vault
---


# Question
Title: Jenkins Email notification for multiple builds
I need suggestion for Jenkins project (multi-conf or pipeline) and plugins that will fit my work.

I have 10 "flavors" of the product, so I must build 10 times every time I commit to the repository (all in the same workspace). Today I have 10 jobs (freestyle) and a "master" job that trigger the rest. I tried to add Email notification (using Email Extension Plugin) but I want only one Email report for all the builds, not 10 Emails.

I understand that I should change to one multi-configuration project or one pipeline project that will handle all the builds, so it will be easier to trigger only one Email, but what is the best practice to get only one Email report on multiple builds?

# Answer
The other option you could try is to add email notification as part of your build script. I use ANT build script and call the email notification target from my jenkins job instead of using the email plugin
> 1 votes

# Answer
If you had one e-mail job that was a downstream of all the other jobs finishing via the Join Plugin, the e-mail job should only fire off after everything is completed.

It seems the "normal" build trigger will cause an issue. https://stackoverflow.com/questions/25912639/how-to-build-a-jenkins-project-only-when-two-other-projects-have-successfully-be
> 1 votes

---
Tags: jenkins
---


# Question
Title: helm - cannot get resource "namespaces" in API group
I installed Kubernetes, helm on the Ubuntu 16.04.6 Box, I want to deploy kubernetes-dashboard using helm but getting below error while installing it

```
$ helm install stable/kubernetes-dashboard --name dashboard-demo
Error: release dashboard-demo failed: namespaces "default" is forbidden: User "system:serviceaccount:kube-system:default" cannot get resource "namespaces" in API group "" in the namespace "default"

$ helm list
Error: configmaps is forbidden: User "system:serviceaccount:kube-system:default" cannot list resource "configmaps" in API group "" in the namespace "kube-system"

```

Tools Version:

```
$ kubectl version
Client Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:11:31Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.1", GitCommit:"b7394102d6ef778017f2ca4046abbaa23b88c290", GitTreeState:"clean", BuildDate:"2019-04-08T17:02:58Z", GoVersion:"go1.12.1", Compiler:"gc", Platform:"linux/amd64"}

$ helm version
Client: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}
Server: &version.Version{SemVer:"v2.13.1", GitCommit:"618447cbf203d147601b4b9bd7f8c37a5d39fbb4", GitTreeState:"clean"}

$ kubectl get pods --all-namespaces
NAMESPACE     NAME                                       READY   STATUS             RESTARTS   AGE
kube-system   calico-kube-controllers-58dc446999-zrwz6   1/1     Running            0          4h41m
kube-system   calico-node-wjczq                          1/1     Running            0          4h41m
kube-system   coredns-fb8b8dccf-fp9gx                    1/1     Running            0          4h41m
kube-system   coredns-fb8b8dccf-q574t                    1/1     Running            0          4h41m
kube-system   etcd-ubuntu1                               1/1     Running            1          4h40m
kube-system   kube-apiserver-ubuntu1                     1/1     Running            1          4h40m
kube-system   kube-controller-manager-ubuntu1            1/1     Running            2          4h40m
kube-system   kube-proxy-ztmms                           1/1     Running            1          4h41m
kube-system   kube-scheduler-ubuntu1                     1/1     Running            2          4h41m
kube-system   tiller-deploy-864c6f45bb-2585b             1/1     Running            0          6m55s

```

helm updated to latest version.

I followed this : https://stackoverflow.com/questions/46672523/helm-list-cannot-list-configmaps-in-the-namespace-kube-system but still facing same issue, how can we fixed it ?

# Answer
> 4 votes
Check https://github.com/helm/helm/issues/3130, this might help. I followed the instruction in the post:

```
kubectl --namespace kube-system create serviceaccount tiller

kubectl create clusterrolebinding tiller --clusterrole cluster-admin --serviceaccount=kube-system:tiller

helm init --service-account tiller --upgrade

```

This works for me.

# Answer
> 1 votes
By default, tiller stores release information in ConfigMaps in the namespace where it is running.

You can refer to the official Helm RBAC documentation for more information on setting up different RBAC scenarios for Tiller.

Kubernetes authorizes API requests using the API server. It evaluates all of the request attributes against all policies and allows or denies the request. All parts of an API request must be allowed by some policy in order to proceed. This means that permissions are denied by default.

For more details see: kubernetes authorization

---
Tags: docker, kubernetes, containers, helm
---


title: Loop chmod with Ansible

text:
# Question
Title: Loop chmod with Ansible
I'd like to remove the execute bit from some ubuntu motd (Message of the day) files with ansible to disable some ad-like Ubuntu news. The module should only act if files exist and permissions are wrong.

If there are no files the tasks are running fine, but as soon as a file exists, I'm getting this:

**Error Message:**

```
failed: [ip] (item={'_ansible_parsed': True, u'stat': {u'isuid': False, u'uid': 0, u'exists': True, u'attr_flags': [... redacted the rest of the file attributes from stat ...]

```

Mayboe someone can help?

Here the files of my role:

**defaults/main.yml**

```
---
motd_path: '/etc/update-motd.d'
unwanted_motd_files:
  - 10-help-text
  - 50-motd-news

```

**tasks/main.yml**

```
---
- name: Register unwanted motd files
  stat:
    path: "{{motd_path}}/{{item}}"
  register: filecheck
  with_items: "{{ unwanted_motd_files | default([]) }}"

- name: Check if unwanted motd files are executable and remove the executable bit
  file:
    path: "{{motd_path}}/{{ item }}"
    state: touch
    mode: u-x,g-x,o-x
  with_items: "{{ filecheck.results }}"
  when: item.stat.exists == true

```

# Answer
Ok, solved it :)

Here the task for everyone who intends to do the same.

Only the skip/changed console output could be replaced by a nicer message.

```
- name: Register unwanted motd files
  stat:
    path: "{{motd_path}}/{{item}}"
  register: filecheck
  with_items: "{{ unwanted_motd_files | default([]) }}"

- name: Check if unwanted motd files are executable and remove the executable bit
  file:
    path: "{{motd_path}}/{{item.item}}"
    state: touch
    mode: u-x,g-x,o-x
  with_items: "{{ filecheck.results }}"
  when: item.stat.exists == true and item.stat.executable == True

```
> 2 votes

---
Tags: ansible
---


title: Docker not found in circleci config

text:
# Question
Title: Docker not found in circleci config
Why is `docker` not found? Do I need to use a different docker image than `debian:latest`? I want the latest debian image so does that mean I need to manually install Docker?

```
orbs:
  aws-ecr: circleci/aws-ecr@6.2.0
version: 2.1

workflows:
  # Build every commit on every branch without pushing to ECR
  build_image:
    jobs:
      - build_image

jobs:
  build_image:
    docker:
      - image: debian:latest
    steps:
      - aws-ecr/build-image:
          account-url: AWS_ACCOUNT_URL
          repo: node

```

# Answer
> 1 votes
Try adding,`- steup_remote_docker` before `- aws-ecr/build-image:`

---
Tags: circleci-2.0, circleci
---


# Question
Title: What's the benefit of using Nginx to serve a Flask API on AWS?
I want to deploy a Flask API on AWS using Docker. A common way seems to be using a combination of Nginx + app server (e.g Gunicorn) + Flask app.

The reason to use app server seems obvious to me. However, since I plan to use AWS ALB I'm not sure if there's a need for Nginx anymore? From my understanding Nginx is used for connection handling and serving static resources. In my case I don't need any of these: the connection is handled already by ALB (AFAIK) and I won't have to serve static resources.

Are there any other advantages of using Nginx for a low latency API service?

# Answer
> 5 votes
Unless you are using the AWS API Gateway, you'll still need some way of "serving" your Flask API. You will also need to provision an EC2 instance and run something to serve your API. Instead of running a windows EC2 instance with IIS you can save about 50% operating cost and use Nginx on a linux EC2 instance. Additionally, Nginx can be used as a reverse proxy, and supports caching which you may find beneficial.

However, because you are using an Application Load Balancer and are looking into Gunicorn to serve the API you probably don't *need* Nginx, and it would just require more overhead for deployments if you are not going to setup any of the features in Nginx.

---
Tags: amazon-web-services, python, nginx, amazon-ecs, flask
---


title: GCP compute quota hard caps?

text:
# Question
Title: GCP compute quota hard caps?
Are there published upper limits or hard caps for the GCE CPU or Managed Instance Group Quota per account? I know that you can request to increase your Quota, but I have not been able to find what the limits are for these requests.

# Answer
There is a Quotas page in the GCP console. This Google help doc will tell you more and how to go about identifying your limits and increasing your quotas: https://cloud.google.com/compute/quotas
> 1 votes

---
Tags: google-cloud-platform, gcp
---


# Question
Title: How to set a shared-library globally using the JCAC plugin?
This Q&A is related to this one and is slightly different.

# Answer
https://github.com/jenkinsci/configuration-as-code-plugin/blob/master/demos/jenkins/jenkins.yaml

It should be possible by configuring it like this:

```
unclassified:
  globalLibraries:
    libraries:
      - name: "some-lib"
        retriever:
          modernSCM:
            scm:
              git:
                remote: "https://github.com/some-project.git"
                credentialsId: 'some-credentials'

```
> 0 votes

---
Tags: jenkins, jcac
---


# Question
Title: Best way to create a Ci/CD pipeline to reduce bugs and facilitate refactor
I started a project with a small team (3 people ) and we had to hack a lot and built a big application with no unit tests, relying only on manual. Now we have a huge technical debt and we started implementing CI/CD. Our goal is to start doing cypress tests everywhere, then some unit tests and refactoring the code as well as implementing static analysis tools on the Ci/CD pipeline:

Our idea is that we will not have any local environment but just server-based. So developer push code to Gitlab, Gitlab runner in Digital Ocean push the repository to staging server, Developer debugs in Digital Ocean using Local Cypress Runner and his own modifications, if he breaks something or forgets to do a full regression test a command-line test regression runner is triggered BEFORE the second deployment so it doesn't allow the code to pass to the deployment/production server.

Questions 1: do we need to separate deployment environment from staging in 2 droplets? can we achieve this configuration without major headaches in the same droplet?

2: do we need to use docker? if not how it should be done

3: what setup can we do to reduce manual work? our problem is not needing approvals but automating as much as we can since we don't have IT/DevOps guy and neither QA.

Note: the first stage is to implement full regression, the second stage is to implement code quality tools (static analyzers) such as sonarqube, phpStan, etc.

Stack: LAMP stack with PHP Codeigniter and vanilla JS.

# Answer
> 3 votes
It might not be obvious now, but usually as projects advance full regression costs (resources/time) grow much faster than static analysis ones.

You'll also find that static analysis alone isn't a sufficiently solid bug gate, it must be complemented with regression testing as well.

So I'd place the full regression **after** the static analysis in the CI/CD pipeline - I wouldn't want to waste full regression costs on code which doesn't even pass static analysis.

Down the road you'll probably find that running full regression and/or full static analysis in earlier pipeline stages can easily become bottlenecks:

* if you block other commits while waiting for the run results you'd slow down the commit rate
* if you allow other commits while a run is in progress they'll change the repository context potentially invalidating the earlier run results (for example you can't immediately fix a runtime bug uncovered by the run if the subsequent commit that happened in the meantime causes a build time breakage).

Addressing these bottlenecks usually means splitting such runs across 2 major functional stages (each one can actually consist of multiple sub-stages and/or steps):

* the **(pre-)integration** stage(s), which execute **subsets** of these runs in the earlier pipeline stages:

+ shorter - to maintaining a sufficiently high commit rate and avoiding bottlenecks
    + the target quality level is below deployment, but high enough to keep development flowing
    + fixing regressions detected in these stages has a high priority.
    + it's possible to use such runs to gate commits/merges in order to **prevent** rather than **detect and fix** regressions, in which case these would actually be **pre-integration** stages
* the **post-integration** stages, executing the longer/full regression and/or static analysis runs:

+ the goal here would simply be increasing the quality to deployment levels (i.e. blocking the actual deployment if not met)
    + regressions detected at this stage aren't blocking for development, fixing them is a lower priority

# Answer
> 0 votes
Please below where I address specific questions.

> 2: do we need to use docker? if not how it should be done

By using Docker you can automate the configuration of your server images as well as application images to use to deploy to staging and into production.

Docker use standard images that you build on by adding your custom software installation and commands into the image. As Docker is mean to run one service per container (a running image), you are also having the benefit of beginning to decouple your applications into microservices.

Take a look at Docker "images", "containers" and "Dockerfiles", to get an understanding of the principles and architecture.

The benefit of automating your infrastructure with Docker images that are created with Dockerfiles are many: \- Your setup becomes less prone to manual mistakes. \- Faster spin-up of services (your app or software) in staging, and also in production. You can batch the spinning up of containers. \- Simpler overview of the tasks involved to setup your servers. As you begin to write Dockerfiles you are taken down the thought process of both development and operations, because you have to write sequential commands for the setup of the servers. That leads to a deeper understanding of your own pipeline - And how it can be improved.

> 3: what setup can we do to reduce manual work? our problem is not needing approvals but automating as much as we can since we don't have IT/DevOps guy and neither QA.

Try to embrace the KISS principle - "Keep It Simple, Stupid". Start small with making a list of the manual tests that you perform on a routinely basis.

Are these tests related to testing REST endpoints and/or load testing? Then take a look at tools such as JMeter to codify your tests that can be saved as files and triggered in for instance Jenkins (with the Performance plugin). You can load your tests in CSV format and call JMeter from within Jenkins.

These are just a couple of suggestions (from many possibilities). I hope this inspires you for your CI/CD setup.

---
Tags: continuous-integration, continuous-deployment, cicd, gitlab-ci-runner
---


title: How to enable shallow clone using JCAC?

text:
# Question
Title: How to enable shallow clone using JCAC?
Another question that is related to this one. We try to configure the `shallow clone` using Jenkins Configuration As Code (JCAC). Although a configuration section is available, `shallow clone` seems to be omitted:

Gitter

# Answer
According to the Jenkins Job DSL documentation `shallow` has to be enabled:

```
shallow(true)

```
> 1 votes

---
Tags: jenkins, jcac
---


# Question
Title: Git Release branching strategy for Continuous integration and deployment
We have tried multiple branching strategies to adopt CI/CD.

1. Developers develop on the feature branch and then merging to master which get deployed to QA for testing and then master is deployed to production, so in short whatever goes out to QA/UAT goes out to production unless you put a create a PR/patch to fix the bug - Least hassle of merging, syncing etc.
2. Developers develop on the feature branch and then merge to master which gets deployed to QA/UAT for testing and based on the testing outcome, spin out a release branch for things which needs to be deployed to Production.

What are the recommendations in case of organizations with 50 odd devs and spread across ~8 scrum teams?

Remember, we always have feature flags to toggle the releases and features pushed out.

# Answer
> 4 votes
You need to separate the 2 concepts a bit:

* the integration part - how are the changes integrated into your master branch (or some other integration branch, that's also possible), which may be continuous or not. Technically neither approach you describe is actually continuous integration unless your feature branches have **a very short lifespan** \- typically less than 1 day. But this part doesn't seem to be the main focus of your post.
* the delivery/deployment part - how are the **integrated** changes from an integration branch (*master* in your case) being delivered/deployed. In other words your release branch strategy. This appears to be the point of your post, I'll focus on it below.

Both approaches you describe are valid release branch strategies, I've seen both being used. Each with their own advantages and disadvantages.

1. This approach is inline with the continuous delivery/deployment strategy. The CI/CD pipeline will have the commits to the master branch (either feature branch merges or point fixes) as entry point: each such commit will trigger the pipeline verification, attempting to progress into it as far as possible. The QA/UAT is one of your pipeline steps, it that passes then it goes into production. Effectively your releases are simply selected (typically tagged) commits from the master branch, they aren't really branches.

Advantages:

* as you noticed - the least amount of branch activities
    * everyone is on the same page - developers, QA, etc all focus on the same branch context
    * lowest cost (but only if the project scale allows it to be effective, see also the disadvantages section)

Disadvantages:

* you can't have actively **maintained** releases - you only have one "current" release out there, each additional fix will become another release. Which isn't necessarily a bad thing, such release model can actually be desired (for example in the case of SaaS products).
    * the release cadence is not actually predictable. It may be difficult to get releases out - if a problem is detected, by the time the fix for it is merged into the master branch some other changes may be merged in and other problems detected. This gets progressively worse as the rate of branch commits goes higher (larger scale projects) and may actually be non-convergent (the project may not reach stability at all). Doing CI/CD at scale is not a trivial matter.
2. This approach is a more traditional release strategy, but technically it's **not really** continuous delivery/deployment.

Advantages:

* more predictable release cadence. The spinned-off release (aka throttle) branches allow reducing the rate of commits for a particular release. Typically a release branch would only receive fixes necessary for that particular release, fixes for future releases (or release trains) would go into the master branch instead. The reduced commit rate allows reducing the risks of new problems being introduced that would require additional commits, the branch is more stable. Typically the release branches are pulled from the master/integration branch commits that passed all (or most) of the CI/CD pipeline verification stages (for the master branch). In your particular example those branches wouldn't expect subsequent commits (they passed QA/UAT), but if say a problem is detected after deployment in production it would be possible to just commit that particular hotfix into the branch and re-deploy, without having to pick up any other commit that went into master since the branch was pulled, reducing the risk of additional regressions.
    * it supports multiple simultaneous/overlapping actively maintained releases, with potentially complex release policies in place (long-term/short term/rolling/etc, like OpenSUSE's Lifetime for example).
    * it supports multi-level releases (like `major.minor.increment`, for example) - aka release "trains": a major release branch could itself have multiple minor release children branches, etc.
    * each release branch could actually use the CI/CD methodology at its own level, independently of the other release branches - if it **never** gets upstream/bulk merges from its parent (i.e. effectively becoming its own "master")!

Disadvantages:

* everyone is no longer on the same page. Each release branch has its own context which may come with its own policies, processes, tools, necessary resources, etc. Each requires its own development/testing/validation. This drives development/maintenance costs up, often significantly.
    * tracking issues and propagating fixes across multiple releases may not be trivial. Merging back release branches into master (or their parent branches) appears an inviting approach, but IMHO it's a bad practice, see How to get rid of develop branch for simplified Git flow .
    * context switching becomes a major problem.

# Answer
> 0 votes
You didn't mention the tools you are using for CI/CD. I worked on Azure DevOps and used visual studio 2017 for these activities. If you are using it for a large number of the developer, I would suggest you use visual studio 2017 or later version. Visual studio has facilities to pull the code on the local repository. Then create a decentralized sub-branch with Master branch and perform the changes. After completing the work, you can use merge or rebase technique to merge it with the master branch by clicking on the button. This will not only store the history but also save the person identity who modifies it. Push that to the Azure DevOps and rest will be done by Azure itself. I worked on it and it is pretty amazing stuff.

---
Tags: continuous-integration, git, continuous-deployment, release-management, release
---


# Question
Title: Can/should I use Jenkins to monitor whether a website is up and send out an email if it's down?
I'm trying to set up a way to constantly monitor a website that is built and deployed via Jenkins. A simple curl command would tell me if everything is OK, but how I would set up a job to do this every ~10 minutes?

Or would this be a bad idea?

# Answer
> 0 votes
You can use Build Periodically functionality under Build Triggers in Jenkins FreeStyle Job. (1)

At the Bottom of the Job Configuration, there is also a BUILD Menu with "Add Build Step" and you can execute shell. (2)

There is a POST BUILD Menu which you can use to send notifications once the build is done. (3)

Depending on how you want things done, you can have the whole script in (2) instead of splitting it into (2) and (3).

https://stackoverflow.com/questions/47758319/build-triggers-in-jenkins

# Answer
> 1 votes
It's possible and pretty simple to do this with Jenkins. Here is a sample pipeline:

```
pipeline {
    agent any

    triggers {
        cron('*/5 * * * *')
    }

    stages {
        stage('Health checker') {
            steps {
                sh 'curl https://mywebsite.com'
            }

            post {
                failure {
                    mail to: 'notify-list@example.com',
                        from: 'jenkins@example.com',
                        subject: "${env.JOB_NAME} - Failed",
                        body: "Job Failed"
                }
            }
        }
    }
}

```

---
Tags: jenkins
---


# Question
Title: How to set git-lfs as an additional checkout behavior using the JCAC plugin?
We are using the Jenkins Configuration As Code (JCAC) plugin to automate the jenkins configuration. One of the challenges is to set `git-lfs` as an additional checkout behavior.

`GitLFSPull` has to be configured, but it does not seem to be included in the DSL:

The README of the JCAC github repository indicates that support questions should be asked on Gitter

According to this Q&A, `GitLFSPull` should be an extension:

```
extensions: [[$class: 'GitLFSPull']]

```

However, querying for this on the Job DSL page shows that GitLFSPull seems to be omitted:

# Answer
> 1 votes
It seems that it is only possible by using a workaround as depicted in this answer.

After setting gitLFS in the UI, subsequently inspecting the `/var/lib/jenkins/jobs/some-job/config.xml` file it became obvious that the XML looked as follows:

```
<traits>
   <jenkins.plugins.git.traits.GitLFSPullTrait>
      <extension class="hudson.plugins.git.extensions.impl.GitLFSPull"/>
   </jenkins.plugins.git.traits.GitLFSPullTrait>
</traits

```

After defining `jenkins.plugins.git.traits.GitLFSPullTrait`:

```
jobs:
  - script: >
      multibranchPipelineJob("example") {
        branchSources {
          ...
        }
        configure { node ->
          node / sources / data / 'jenkins.branch.BranchSource' / source / traits {
            'jenkins.plugins.git.traits.GitLFSPullTrait'()
          }
        }
      }

```

Git-lfs was enabled.

---
Tags: jenkins, jcac
---


# Question
Title: How do I get the UUID of a build Mac from Azure DevOps?
I'm trying to set up an Azure DevOps pipeline to build a macOS app. I think I have all the necessary steps in place, and it runs as far as I expect it to for now.

However, I now have reached the step where, in order to sign the product, the build machine device must be included in our provisioning profiles, which means we have to add its UUID to the list of devices in Apple Developer.

How do I get this pipeline to use this profile? I imagine a solution might involve one of these:

* There's a way to find the UUIDs of every Mac which might run the pipeline
* There's a way to restrict that pipeline to always use the same Mac with the same UUID
* There's a way to specify/spoof the UUID of the build machine
* There's an API or something to register the build machine with Apple Developer at build time
* Azure DevOps has a special way to circumvent this part of the build process while still allowing signing
* Apple has some special way I don't know about to allow an arbitrary build machine to sign without knowing its UUID beforehand

I've looked around for all these and can't find definitive info for any of them. **Does anyone know the best practice for signing an Xcode project pipeline in Azure DevOps using a specific set of provisioning profiles?**

---

Our certs and profiles are stored in Azure's "Secure files" section:

And referenced in the pipeline's build steps:

# Answer
> 1 votes
These steps in the Azure DevOps docs may be helpful: Sign Your Mobile App.

Basically you should be able to generate your p12 and provisioning profile on a mac you control, and then using the "Install an Apple certificate/provisioning profile" tasks you can install the certs onto the mac agent for building. No UUID should be needed for the agent. Here's a guide that may be useful (it's targeting iOS, not macOS but the concept is the same).

The way I set this up on a project was to go through Microsoft's AppCenter which had tooling to automatically handle the provisioning profile madness: Deploy Documentation.

---
Tags: azure-devops, azure-pipelines, macos
---


# Question
Title: When should an application be copied to a new region?
I'm taking my first course For AWS cloud practitioner and I am curious:

**The case**: I have a single, client-facing application hosted on an EC2 instance. Let's keep it simple and say it's brochureware, the database is in a volume attached to the instance. It's using us-west-2a as its availability zone.

**The conundrum**: I understand the use of CDNs as they essentially store a copy of the application in the edge locations so clients can be delivered a static copy quickly without using up EC2 compute time.

I also understand regions but only to the extent that it is an initial starting point for building out your application.

**What would be the nature of cases that would require me to duplicate my EC2 instance to a new availability zone and run the two exact same applications simultaneously?**

A few things pop-up for me initially:

* Computational availability (ex: the application client needs to request specific data, like a user defined request)
* Zone based version control (ex: an incremental rollout of a feature)
* And if I'm using this term right, fault tolerance (graceful degradation?). If one zone is down we can fall back to another.

Are these assumptions correct?

# Answer
**First a primer on regions and availability zones**

Regions are physical places where the AWS resources are located. As such different regions may offer different resources. This is also important for latency-sensitive operations and data sovereignty laws. Having an EC2 instance in the us-east-1 region means all of my instance resources are in that region of the United States.

Availability Zones, while in the same geographic area, may be backed by one or multiple data centers. However, if you have an EC2 instance running in two availability zones, they will run in separate data centers. So an EC2 instance in us-east-1a and us-east-1b will be in two separate data centers.

Imagine you have a mission critical application that needs high availability. By setting up a load balancer (controlled by the region and not the availability zone) you can help scale your application across multiple availability zones and redirect traffic if something happens to an availability zone. This helps create a highly available infrastructure.

To answer your three examples:

* Computational availability is not really a factor. You could always throw a larger instance (or multiple instances in the same availability zone) to decrease load.
* While you could use multiple zones for rolling out upgrades, this can also be done within the same availability zone.
* Your third point here is the main draw to the availability zones. If a data center catches on fire, floods, gets hit by a meteor and all your EC2 instances are tied to that center then you would be unable to access whatever application you had running.
> 1 votes

---
Tags: amazon-web-services, amazon-ec2, high-availability
---


# Question
Title: Expose metrics for Kotlin using JMX_Exporter to prometheus
I am trying to use JMX\_Exporter for my kotlin code to expose metrics to prometheus in order to show it in Grafana. I have gone through many articles and tried to understand how could it be done. I have found below two links useful and trying to achieve using those.

https://github.com/prometheus/jmx\_exporter https://www.openlogic.com/blog/monitoring-java-applications-prometheus-and-grafana-part-1 What i did so far is, created a folder 'prometheus-jmx' in root directory and added mentioned JAR and config.yml file in that folder. Then, i added below parameter to my dockerfile.

```
CMD java -Xms128m -Xmx256m -javaagent:./jmx_prometheus_javaagent-0.12.0.jar=8080:./config.yml -Dconfig.file=config/routing.conf -cp jagathe-jar-with-
    dependencies.jar:./* com.bmw.otd.agathe.AppKt

```

My prometheus is running in my OpenShift cluster along with my application. I could scrape metrics for my other applications/deployments like Jenkins, SonarQube etc without any modifications in deployment.yml of Prometheus.

My application is running properly now on OpenShift and from application's pod, I can scrape metrics by using below command.

```
curl http://localhost:portNumber

```

But on promethus UI, I cant see any JVM or JMX related metric.

Can someone please tell me where and what am i doing wrong? Any kind of help would be appreciated.

# Answer
> 1 votes
After trying many things, I came to know that I needed to expose the port of my application's container in order to let Prometheus or other deployments to know. After exposing the port, I could see my application under targets on Prometheus and I could scrape all JMX and JVM metrics. Hope this would help someone in future...

---
Tags: openshift, prometheus, grafana
---


title: Developing in Docker for Windows Basics

text:
# Question
Title: Developing in Docker for Windows Basics
Starting with Docker for Windows and C# applications.

I've been able to do some basic building and running of dockerfiles. However, I've run into the first application that doesn't run right out of the gate, and am uncertain how I would begin to troubleshoot it.

What are some of the techniques used when moving an app into a container (Windows image) for the first time?

# Answer
The first thing you'll want to do is look for errors. It sounds like you have a Dockerfile that is creating an image, but when you run the image it appears to not do anything. There are two immediate things you can do to troubleshoot this:

1. Do not use the -d flag when running the image. This flag runs the container in detached mode and will not display container output to the console.
2. Examine the logs from the container. Even though the container does not appear to be running, it is most likely crashing on start. You can verify this by using `docker ps -a` which will show you all containers (including ones that have stopped due to crashes).

Once you know what the error is you can look at whether this is a problem with your startup script or the Dockerfile. Based on what you've described so far this sounds like you could be missing files in the right location, or the Entrypoint or CMD is incorrect.
> 2 votes

# Answer
Review the containers' output in your terminal. If it was run in detached mode (with `-d` flag) then the you can review its output using `docker container logs <container name>`.

If the container doesn't behave as planned but also doesn't exit, you may want to diagnose parts of your application from inside the container. You can do this by creating an interactive shell inside it with the command: `docker container exec -it <container name> sh`.

Another way to interact with the container's file system would be to mount some of its content on the host machine, then you will have it available to any tools you might want to use from your machine. This can be done by adding the option `-v /path/on/host:/path/in/container` when running the image.

For a more focused answer, it will help if you provide an example of what Error or behavior occurs. Good luck!
> 0 votes

---
Tags: docker
---


# Question
Title: Docker: PHP deployment in apache, couldn't acces container URL in local browser; shows blank page
I'm trying to deploy PHP files in Apache container using Dockerfile, Dockerfile on console shows successfully built and tagged but after running the container, i couldn't access the container IP or Container URL in local browser

**Dockerfile**

FROM ubuntu:bionic
RUN apt-get update
RUN apt-get install -y software-properties-common && \add-apt-repository ppa:ondrej/php && apt-get update
RUN apt-get -y upgrade && DEBIAN_FRONTEND=noninteractive apt-get -y install \
apache2 php7.0 php7.0-mysql libapache2-mod-php7.0
RUN sed -i "s/short_open_tag = Off/short_open_tag = On/" /etc/php/7.0/apache2/php.ini
RUN sed -i "s/error_reporting = .*$/error_reporting = E_ERROR | E_WARNING | E_PARSE/" /etc/php/7.0/apache2/php.ini

ENV APACHE_RUN_USER www-data
ENV APACHE_RUN_GROUP www-data
ENV APACHE_LOG_DIR /var/log/apache2
ENV APACHE_LOCK_DIR /var/lock/apache2
ENV APACHE_PID_FILE /var/run/apache2.pid

EXPOSE 80
ADD www /var/www/site
ADD apache-config.conf /etc/apache2/sites-enabled/000-default.conf
ADD hosts /etc/hosts
CMD /usr/sbin/apache2ctl -D FOREGROUND ```
</code></pre>

# Answer
> 1 votes
You need to expose the container ports locally using either:

* `docker run --rm -it -p 8080:80 myapp`
* or a `docker-compose.yml` file:

```
services:
  myapp:
    image: myapp:tag
    ports:
    - 8080:80

```

Then, you'll be able to `curl localhost:8080`.

# Answer
> 0 votes
A side-note (sorry, I haven't got enough reputation to post direct comments yet): I see you are basing your image off of Ubuntu (`FROM ubuntu:bionic`) and then installing Apache... I would suggest using an image that's already tailored for this function, such as the Apache HTTP Server.

---
Tags: docker, docker-compose, dockerfile
---


# Question
Title: Kubernetes nginx ingress controller - one pr. environment?
I have a kubernetes cluster in Azure. In this i have three namespaces:

1. development
2. staging
3. production

I have installed nginx ingress controller into its own namespace, ingress-nginx. I am kind of new to the ingress part, and I cant seem to find anywhere any advice if I should have a controller pr namespace, or share one controller with all.

Should I have one that is shared or install one pr. namespace? What is the pattern here?

# Answer
I would suggest at least one for production and one for npe. The nginx-ingress helm chart appears to only support setting a scope to all namespaces or just one namespace (a frustrating limitation not in other ingress controller options):

https://github.com/helm/charts/tree/master/stable/nginx-ingress

`controller.scope.namespace`
> 2 votes

---
Tags: kubernetes, nginx, ingress
---


# Question
Title: Kubernetes expose an app via a DNS name in minicube
I have a Minikube installation in which I created a simple hello-world deployment like this:

```
kubectl create deployment hello-node \
    --image=gcr.io/hello-minikube-zero-install/hello-node

```

I exposed the deployment via a service in the following way:

```
kubectl expose deployment hello-node --type=LoadBalancer --port=8080

```

Now If I call: `http://<local cluster ip>:8080` it prints "Hello World!" as expected.

**What I want to achieve:**

I want to expose different deployments in the same cluster to different sub-domains of the cluster. For instance, deployment `hello1` to `hello1.my-k8-cluster.com`, `hello2` to `hello2.my-k8-cluster.com`.

I want to test this locally because later I will do the same on a real cluster.

**Question:** How to test DNS configurations of services locally? How to define sub-domains in services?

**What I tried so far:** I went through the how-to guides here and the documentation which though didn't bring me a clear picture on how to configure what I want.

# Answer
> 1 votes
The approach I've seen work for this is to use Nginx as the LoadBalancer and defining the Ingress with type NodePort for the different services. You can see a tutorial that shows the first steps for setting this up here.

Additionally, the official docs DNS for Service and Pods and Customizing DNS Service are good resources.

---
Tags: kubernetes, minikube
---


# Question
Title: What is the difference between DevOps and GitOps?
I am hearing nowadays a lot about GitOps. What is the difference between DevOps and GitOps?

Is it just the SCM does the build and deploy when there is a commit instead of the CI/CD tool exclusively?

# Answer
> 13 votes
GitOps is a technique of using Git to manage infrastructure provisioning and software deployments. This technique uses many of the features of Git such as Pull Requests to manage and trigger deployments, and allows "diffing" (looking at the difference between commits) to view the differences between deployments.

DevOps is more of a culture or philosophy. It is not tied to the specific tool Git. Instead, DevOps is the joining of Development (Dev) and System Operations (Ops). Often continuous integration and continuous deployment are big focuses of companies that invest in a DevOps culture.

A company that has a DevOps culture may choose to use a technique like GitOps to manage their infrastructure. They may choose to use other tools though.

Note: DevOps is not a process. It is a culture. GitOps is a specific System Operations (ops) process tied to a specific tool (Git). A company that uses GitOps may not have a DevOps culture. Conversely, a company that has a DevOps culture may not use GitOps.

---
Tags: continuous-integration, git, terminology, continuous-delivery, scm
---


title: Does Kubernetes Ingress use NodePort?

text:
# Question
Title: Does Kubernetes Ingress use NodePort?
When using a Kubernetes `LoadBalancer` or `Ingress`, is the normal `NodePort` used as the underlying connection point? I.e. does the external-to-kubernetes load balancer implementation (used even when `Ingress` is in play) communicate with the `Service` in the cluster via `NodePort` or is there some other "magic" way that is used?

I ask because in reading about kubernetes, I have found some sources that have almost implied that `NodePort` is somehow unreliable ("if you don't need your services to be available all the time"), and if `NodePort` is the mechanism underlying other load balancer implementations, then it should be safe to use, given an understanding of its particularities.

# Answer
> 2 votes
**For your first question:**

> When using a Kubernetes LoadBalancer or Ingress, is the normal NodePort used as the underlying connection point?

Technically it depends on the cloud network for LoadBalancers (e.g. Google Compute Engine does not technically require this although one may be created). However, I believe a NodePort and ClusterIP are generally created.

**For your question on the usage of NodePorts:**

The short answer is yes, they are safe to use if you know what you are doing.

The longer answer is that NodePorts are great for debugging and development but have some issues in production:

* You can only have one service on a port.
* HTTP/HTTPS traffic may be routed through a non-standard port.
* The port will be opened on *all* nodes.

However, NodePorts do provide some nice flexibility (from Kubernetes documentation):

> Using a NodePort gives you the freedom to set up your own load balancing solution, to configure environments that are not fully supported by Kubernetes, or even to just expose one or more nodes’ IPs directly.

---
Tags: kubernetes, ingress
---


# Question
Title: How do I use only an encrypted version of my AWS RDS password in my terraform code?
Is there a way to only have an encrypted version of my RDS password in my Terraform code and for AWS to decrypt it for RDS so you use the decrypted password to login to the RDS?

So set my rds password variable value to the encrypted password and when terraform builds the RDS I login in with is the unencrypted password?

# Answer
> 3 votes
If I am understanding your question correctly, what you are describing sounds like a perfect use case for SSM. You can store your RDS password as a secure string parameter encrypted by KMS, then reference it in you terraform file as a data source with decryption. With this method, you only have to reference your decrypted password, and never directly store the encrypted or plain text in your tf file.

```
data "aws_ssm_parameter" "foo" {
  name = "foo"
  with_decryption = true
}

```

---
Tags: amazon-web-services, terraform, encryption, amazon-rds
---


# Question
Title: How to automate control of synchronization between Maven artefact version and Git tag?
Given the following situation:

* in a POM file in the repo root (Atlassian BitBucket) I can define that this POM has e.g. the version `0.0.1-SNAPSHOT`
* then I can tag the codebase whatever I want like `v.0.1`.

This would be then inconsistent versioning. Is there a better/established way than some custom scripting to control sync of this?

# Answer
> 2 votes
> then I can tag the codebase whatever I want like v.0.1

One could use semantic versioning, i.e. X.Y.Z. instead of v.0.1

# Answer
> 1 votes
I would recommend to use the maven-release-plugin wich will take care of the sync and keep the version and tag equal. As you can read here

> The `developerConnection` contains the URL of the Source Control Management system pointing to the folder containing this pom.xml This URL is prefixed with `scm:[scm-provider]` so the plugin can pick the right implementation for committing and tagging. The Maven SCM-page contains an overview all the supported SCMs, per SCM you can see how the URL should look like.

---
Tags: git, maven
---


# Question
Title: How do Sensu handler notification plugins determine the path of their JSON config files?
So, we're transitioning from Slack to MSTeams. I'm tasked with updating our Sensu configuration so that alerts get sent to MSTeams channels instead of Slack channels.

In the `/etc/sensu/conf.d/handlers.json`, we have Slack notifications configured as follows:

```
...
"slack" : {
    "type": "pipe",
    "command": "handler-slack.rb -j slack",
    "filters": [
        "occurrences"
    ]
},
....

```

That `-j slack` tells the handler to load a JSON config file which is located at `/etc/sensu/conf.d/slack.d/slack.json`.

The Teams handler is configured similarly:

```
...
"microsoft-teams" : {
    "type": "pipe",
    "command": "handler-microsoft-teams.rb -j microsoft-teams",
    "filters": [
        "occurrences"
    ]
},
...

```

But when Sensu tries to pass a notification to the Teams handler, I get an error that leads me to believe that the JSON config wasn't loaded:

```
/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-microsoft-teams-2.0.0/bin/handler-microsoft-teams.rb:104:in `get_setting': undefined method `[]' for nil:NilClass (NoMethodError)
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-microsoft-teams-2.0.0/bin/handler-microsoft-teams.rb:28:in `payload_template'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-microsoft-teams-2.0.0/bin/handler-microsoft-teams.rb:108:in `handle'
    from /opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-2.7.1/lib/sensu-handler.rb:89:in `block in <class:Handler>'

```

`payload_template` is the first setting that handler-microsoft-teams.rb attempts to load. It is not specified in my JSON, and the code is written to treat it as optional.

So, my question is this: how does the Slack plugin go from the string passed to the `-j` argument (`slack`) to the path `/etc/sensu/conf.d/slack.d/slack.json`? And how do I determine the equivalent path magic for the Teams plugin? I don't know if this magic is inherent in Ruby (in which I'm a non-expert), Sensu, sensu-plugins, or something else. I've been digging in code for a while now and I'm coming up empty.

(Yes, I'm aware of Sensu Go and would like to upgrade; that's not an option on my current timeline.)

# Answer
Okay, so it turns out this wasn't my problem, but I know the answer to the question I asked:

The file `/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugins-microsoft-teams-2.0.0/bin/handler-microsoft-teams.rb` defines a function `get_setting`, which looks like this:

```
def get_setting(name)
  settings[config[:json_config]][name]
end

```

`config[:json_config]` is the string passed with the `-j` switch.

On to `config`: the file `/opt/sensu/embedded/lib/ruby/gems/2.4.0/gems/sensu-plugin-2.7.1/lib/sensu-handler.rb` requires `sensu-plugin/utils`, which defines the following:

```
module Sensu
  module Plugin
    module Utils # rubocop:disable Metrics/ModuleLength
      def config_files
        if ENV['SENSU_LOADED_TEMPFILE'] && File.file?(ENV['SENSU_LOADED_TEMPFILE'])
          IO.read(ENV['SENSU_LOADED_TEMPFILE']).split(':')
        elsif ENV['SENSU_CONFIG_FILES']
          ENV['SENSU_CONFIG_FILES'].split(':')
        else
          ['/etc/sensu/config.json'] + Dir['/etc/sensu/conf.d/**/*.json']
        end
      end

      def load_config(filename)
        JSON.parse(File.open(filename, 'r').read)
      rescue
        {}
      end

      def settings
        @settings ||= config_files.map { |f| load_config(f) }.reduce { |a, b| deep_merge(a, b) }
      end
...

```

So `settings` is going to load all JSON files nested anywhere under `/etc/sensu/conf.d/` and deep-merge them. Beyond that, the path is not relevant to Sensu's ability to load it using the string `slack`; that's the name of the entry in the JSON payload. This mechanism is common to all Sensu config, not just handler plugins. Documentation is here: https://docs.sensu.io/sensu-core/1.8/reference/configuration/.

I looked at my log files, and it does seem to be finding and loading the files I had optimistically placed under `/etc/sensu/conf.d/teams.d/`, but it's not working.

Before I found the answer documented here, I took a stab and renamed the directory from `teams.d` to `microsoft-teams.d`, and the configuration object defined within from `teams` to `microsoft-teams`. I don't know why that would have had any effect, but it's working now. Maybe I had fat-fingered or copypasta'd something and resolved that inadvertently in the process of adding the `microsoft` prefix.
> 2 votes

---
Tags: sensu
---


title: On-Prem and Cloud Deployments

text:
# Question
Title: On-Prem and Cloud Deployments
If I have a mix of cloud and "on-prem" (across different networks) systems to deploy to is there a common tool, technique, or software available? I've used Kubernetes for cloud deployments, but am unsure of what approach to take when On-Prem installs/upgrades are mixed in.

Is is considered best practice to attempt to standardize the deployment or work with the ops team to develop two separate deployment strategies?

# Answer
If you can keep one common deployment strategy across both that would be simpler and less effort from the ops side of things. Kubernetes seems to be a great choice for the cloud orchestration layer at this point in time. Does Kubernetes make sense for the on-prem side though? If you're not planning on each on-prem install including 5-10 servers to spread Kubernetes out over then it probably doesn't make sense to use it there. All hope is not lost. :) You could take the same Docker containers that you ran under Kubernetes and run them yourself for on-prem. You're going to be stuck dealing with service discovery, but if you're only installing on one server that can be solved in some old school way like picking and assigning ports to specific services.
> 1 votes

# Answer
(Kublr CTO) If you can use Kubernetes as the deployment target for your applications, it is a great tool to standardize delivery and operations. The only problem is, as you correctly pointed, a consistent approach to deployment and operations of Kubernetes clusters themselves.

The best practice in my opinion is definitely to centralize and unify Kubernetes management; but whether it is actually worth to do depends on the expected scale of operations (how many clusters you want to manage, and how often changes are implemented), and on availability of Kubernetes management tools that fit your specific needs.

Generally speaking here are some links where you can start gathering information on various tools in this area:

* A place to start when selecting a Kubernetes management solution that suites your needs: https://kubernetes.io/docs/setup/ \- it includes a list of products and tools suitable for managing cluster fleets in production.
* Another good curated list of tools and solutions related to Kubernetes, including deployment and management tools, can be found here: https://github.com/ramitsurana/awesome-kubernetes
* One more reference point for research is the Kubernetes conformance testing results maintained by CNCF (Cloud Native Computing Foundation) on their GitHub account: https://github.com/cncf/k8s-conformance ; you may find conformance testing results there from different Kubernetes management tools and see which of them maintain conformance with relatively current versions of Kubernetes, e.g. here is the K8s 1.15 conformance testing results: https://github.com/cncf/k8s-conformance/tree/master/v1.15

Although the full list of tools and vendors looks huge, if you filter it though the "production readiness requirements" list, there may actually be very few solutions left that fit the bill.

Here is the list, in no particular order, of considerations that may be important in case like yours:

* the tool is mature enough for production use, and deploys production quality (secure, reliable, self-healing) clusters,
* deployed clusters compatibility and conformance - whether upstream Kubernetes is used or a custom build,
* configurability - whether and how the tool limits deployed Kubernetes clusters configuration options; e.g. can you adjusts Kubernetes API server, controller manager, or kubelet configuration options
* whether support is available, for the tool itself, for the clusters deployed through the tool, and for the "borderline" problems - problems for which you cannot be sure whether it is your application running on the cluster, or the cluster itself, or a specific configuration of this cluster,
* cost,
* whether the tool is SaaS or can be installed in your environment, cloud or DC;
* whether the deployment and management are centralized, or each cluster is separate; and whether a "single pane of glass" for your clusters and a single API for cluster management are available;
* management and "day 2 operations" of clusters - updates, upgrades, recovery etc
* if the tool provides centralized features, then how access control management for cluster operations looks like (this is different from Kubernetes RBAC, which is implemented by Kubernetes itself),
* integrations with enterprise IDM for SSO and user federation,
* support for different and heterogeneous target environments - e.g. AWS, GCP, Azure, on-prem, vSphere; and support for different target OS
* uniformity of Kubernetes clusters deployed into different environments: e.g. if your Kubernetes clusters in AWS are deployed using AWS EKS, and Google cloud clusters are GKE, then you may run into a situation when these clusters cannot achieve feature parity due to different set of supported versions, addons etc,
* availability of operations tools like log and metrics collection, visualization, alert and notification management etc.

In cases where you know for sure that you will stay in one specific cloud, the corresponding cloud provider's managed Kubernetes solution would probably be the first candidate - such as AWS EKS, GCP GKE, and Azure AKS; although it should still be run through the list of your specific requirements.

In other cases, cloud-agnostic tools should be considered.

Another consideration to keep in mind is how entangled or integrated your infrastructure management, Kuberletes cluster management, and application operations are going to be with different tools. Different tools may put different emphasis on different parts of the stack; e.g. some Kubernetes distributions may also provide integrated and/or opinionated application lifecycle management components, or integrated infrastructure/cost management features.

While integrated solutions may be better in terms of time to market and learning curve; at the same time they may limit your ability to benefit from the latest developments in DevOps/CI/CD technologies.

I personally lean towards treating Kubernetes as a convenient infrastructure abstraction layer, on top of which not only your applications, but also your DevOps/CI/CD tools, and various managed services (e.g. cloud native storage solution, databases, etc) can run. We have some presentations published on our web site about that: https://kublr.com/register/all-things-kubernetes-online-meetup/

Talking about specific vendors, I must put a disclaimer here - I am a CTO at Kublr - https://kublr.com/ \- an enterprise Kubernetes management solution built around the requirements listed above. It can be found in the lists of Kubernetes tools mentioned above; you can also schedule a demo on the web site, or get an unlimited non-production version at https://kublr.com/deploy/
> 0 votes

---
Tags: docker, kubernetes, continuous-deployment
---


# Question
Title: See which Roles contain a GCLOUD Permisson
I'd like to see wich `Roles` in `iam` yield a certain permission. In other words, for a given permission (like `billing.accounts.updateUsageExportSpec`) I want to know a list of `Roles`, that satisfy this permission.

# Answer
> 1 votes
Looks like only the `roles/billing.admin`: https://cloud.google.com/iam/docs/understanding-roles#billing-roles

# Answer
> 0 votes
If under unix, with gcloud SDK you can execute the following script:

```
export expected_permission=billing.accounts.updateUsageExportSpec

for role in $(gcloud iam roles list --format='value(NAME)');
do permissions=$(gcloud iam roles describe $role --format='value(includedPermissions)')
if [[ $permissions =~ $expected_permission  ]]; then echo "-------------------------------------------------" &&  echo $role && echo "-------------------------------------------------"; fi
done

```

Either way, there's this reference in official docs where permission/roles mapping can be shown

---
Tags: google-cloud-platform, permissions
---


# Question
Title: Deploying a Kubernetes cluster for production on multiple clouds
I want to install a **highly available kubernetes cluster** (multiple master nodes) on a public cloud (AWS, Azure, GCP and maybe others).
I'm looking for a non-vendor specific option to do it.

I'm aware of several options like

* Kops
* Juju
* Other Github projects I found...

I was wondering if anyone has a good automated example of doing this with such tools like Terraform, Ansible or others.

The **HA** is a must as this if for production use!

# Answer
> 6 votes
At work we are currenly using a multi AZ Kubernetes cluster on AWS and we're using `kops` along with Terraform (`kops` generates the Terraform configuration files) to provision the cluster. What is not clear to me is if your intention is to run a single multi-cloud cluster or if you want to run multiple clusters in multiple clouds.

Anyway, our current setup is a multi-master highly-available multi-AZ Kubernetes cluster. I'll try to explain step by step.

The first thing in order to create the cluster is to generate the Terraform configuration with `kops` (you could directly apply the changes in AWS by directly using `kops`, but in our case we think it's best to keep the Terraform files versioned in git, to be able to change details about this configuration and inspect them, also). For instance, this could be the command we used:

`kops create cluster \
--name=my_new_cluster \
--ssh-public-key=k8s.pub \
--dns-zone=<your-name>.<your-domain> \
--cloud=aws \
--master-size=t2.medium \
--node-size=t2.medium \
--vpc=<vpc-id> \
--zones=us-east-1a,us-east-1b,us-east-1c \
--master-zones=us-east-1a,us-east-1b,us-east-1c \
--out=my_new_cluster \
--target=terraform`

This creates the Terraform configurations (in the directory specified by the `--out` flag) for a Kubernetes cluster which nodes and masters are distributed along 3 Availability Zones (us-east-1a, us-east-1b and us-east-1c). This implies creating:

* all necessary subnets, security groups and route tables
* IAM roles
* AWS Auto Scaling Groups to manage the cluster nodes and masters (it's the magic of all this, they're the ones that maintain the high availability of the cluster by launching extra nodes as traffic increases, and replaces faulty ones)
* all EC2 instances and EBS volumes
* Route53 DNS records to refer to the cluster API and internal DNS for k8s to communicate with pods.

`kops` uses `kubectl` internally to create the cluster and all necessary configurations in Kubernetes.

After that, you could `terraform apply` the configurations and watch it all go up in AWS. It's quite simple, really, but you have to learn how Terraform works. I've found it very intuitive.

---

Later on, you may need to edit you cluster. There's a command: `kops edit cluster <cluster-name>` that let you do that. It's all configurations in `yaml` format. I won't get into specifics but after some changes you may need to `kops rolling-update <cluster-name>` to apply them.

Hope I've been clear.

Regards.

# Answer
> 1 votes
(Kublr CTO) Take a look at Kublr - https://kublr.com/

It is designed exactly for a use-case like this: centralized management and operations of multiple Kubernetes clusters in multiple and/or heterogeneous target environments. Currently it supports AWS, Azure, Google Cloud, vSphere, vCloud Director, and on-prem/BYOI (Bring Your Own Infrastructure mode.

In this response - https://devops.stackexchange.com/a/9281/17283 \- I described in more details which requirements we considered important when designing Kublr.

---
Tags: kubernetes
---


# Question
Title: Unable to run the playbook in remote machine using Ansible job template. Connection Failed
**Playbook.yml**

```
---
- name: connectionTest
  hosts: Windows
  tasks:
  - name: "Windows Connect"
    win_command: hostname
    register: wincmd_out

```

Variable declared in Inventories `Project/inventories/XXX_Local/`

```
[Windows]
XXVMxxxxxxx

```

Error Message:

```
fatal: [XXVMxxxxxxx]: UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug1: Control socket \"/tmp/awx_1636443_S7HHdB/cp/6b18046463\" does not exist\r\ndebug2: resolving \"XXVMxxxxxxx\" port 22\r\nssh: Could not resolve hostname XXVMxxxxxxx: Name or service not known",
    "unreachable": true
}

```

# Answer
> 1 votes
Looking at your code it seems you try to connect to a Windows host. Usually you connect windows via `WinRM` and not via ssh

The user guide for this can be found here: https://docs.ansible.com/ansible/latest/user\_guide/windows\_winrm.html

> For Ansible to communicate to a Windows host and use Windows modules, the Windows host must meet the following requirements:
>
> * Ansible’s supported Windows versions generally match those under current and extended support from Microsoft. Supported desktop OSs include Windows 7, 8.1, and 10, and supported server OSs are Windows Server 2008, 2008 R2, 2012, 2012 R2, 2016, and 2019.
> * Ansible requires PowerShell 3.0 or newer and at least .NET 4.0 to be installed on the Windows host.
> * A WinRM listener should be created and activated. More details for this can be found below.

Then you need to tell ansible to connect wia `winrm`

```
...
ansible_connection=winrm
...

```

Checkout the details here: https://www.ansible.com/blog/connecting-to-a-windows-host

---
Tags: ansible, windows
---


title: Jenkins Git Client Plugin Issue

text:
# Question
Title: Jenkins Git Client Plugin Issue
Git Client plugin previously was version was version:3.0.0-rc(beta) which has security vulnerabilities so upgraded to latest version:2.8.6 (latest). But unfortunately all the dependency plugins were still looking for 3.0.0 and believing that 2.8.6 is older version (which was a bug in plugin) are still using older version instead of newer version which caused issue during the upgrade. So tried to revert back but there is not revert back to beta version from 2.8.6 to 3.0.0.

No Option for revert back to older version: And there is no such version in https://updates.jenkins-ci.org/download/plugins/git-client/ for manual upgrade of Git Client as before one was beta version.

```
Note this problem I faced in my project. So if someone else is also facing similar issue this might help them to resolve.

```

# Answer
> 2 votes
Solutions:

You have a few choices to resolve the security issue in git client plugin 3.0.0-rc.

1) Install git plugin 3.12.1 and git client plugin 2.8.5 from the Jenkins update center
2) Upload git plugin 3.12.1 and git client plugin 2.8.6 into your Jenkins server from the "Advanced" tab of the "Manage Plugins" page inside "Manage Jenkins".
3)Install git plugin 4.0.0-beta11 and git client plugin 3.0.0-beta11 from the Jenkins experimental update center.
4)Upload git plugin 4.0.0-beta11 and git client plugin 3.0.0-beta11 into your Jenkins server from the "Advanced" tab of the "Manage Plugins" page inside "Manage Jenkins"

If you choose git plugin 3.12.1, you are choosing the released git plugin and the most stable.

If you choose git plugin 4.0.0-beta11, you are choosing the most recent pre-release git plugin with more bug fixes than are available in git plugin 3.12.1.

Reference: Git Client Plugin

GitHub Link

Plugin manual download

Related Issue : https://issues.jenkins-ci.org/browse/JENKINS-59336

---
Tags: jenkins, git, jenkins-plugins
---


# Question
Title: How to handle availability zones and failover with ansible? Is it even within the scope of ansible?
I am only getting started with ansible.

I want to set up a new development infrastructure with the infrastructure as code principle. First thing I'll have to prove is a Jenkins instance with a few nodes for a Java dev team.

I want to do it correct right from the start, so I want to include some redundancy. Let's assume that in this scenario there are 3 sites with bare metal servers. I want to set up a Jenkins master in all 3 zones which form a cluster.

But I don't want to think about that myself. I just want to provide the info to ansbile what server is in which availability zone, and ansible should make sure to set up a Jenkins master in each availability zone and connecting it to the cluster.

Maybe this is already out-of-scope for ansible and I need another tool on top of ansible?

Any thoughts?

# Answer
> 5 votes
What your are describing can typically be addressed with a correct inventory design.

```
[zone1]
serverA
serverB
serverC

[zone2]
server1
server2
server3

[zone3]
serverX
serverY
serverZ

[jenkins_master]
serverA
server1
serverX

[jenkins_slave]
serverB
serverC
server2
server3
serverY
serverZ

```

From there you can easily:

* create plays that will target a group or a pattern and that you can limit to an other group or pattern at runtime, e.g. target `jenkins_master` and run the playbook on all of them or limit it to `zone1`.
* create inventory vars for your hosts and groups so that e.g. servers in `zone1` will know what is the uri of the master in that zone.

This is a super-quick overlook. I suggest you read in depth working with inventory to get a better understanding of all the inventory concepts and pay attention to the part concerning the organization of variables in the inventory for groups and hosts

---
Tags: jenkins, ansible, high-availability
---


# Question
Title: Can Terraform S3 remote state be kept in a different AWS account?
I am currently using Terraform with S3 remote state. The IAM user that I'm using to run Terraform exists in the same AWS account as the S3 bucket holding the remote state files. This is all working fine.

However, I have IAM users in other AWS accounts (all part of the same AWS organization) and I want to grant them access to use terraform in the main account. It seems like I should be able to do this with IAM roles. I've created an IAM Role in the main account, and I've configured the `assume_role` attribute of the `aws` provider to assume a role in the main AWS account.

However, this approach fails because the user in the other AWS account doesn't have access to the remote state files in S3. I've tried granting access to the remote state files to the IAM role, but terraform apparently tries to fetch state *before* the `aws` provider is loaded and the new IAM role is assumed.

Is there any way to grant access to the remote state files in S3? Or more generally, is there any way to allow users across multiple AWS accounts to perform terraform operations across the same multiple AWS accounts, even though sometimes the S3 remote state will be located in a different account?

# Answer
> 1 votes
You can have two approaches:

1. Assume the role that has access to the AWS bucket. Assume this role in every account (don't forget to set up trust relationship). This is your approach. I think the issue might be some objects (state files) were created in a different way. Other culprit can be an account-specific KMS key that is used to create the objects. Don't forget, KMS keys are region-specific.
2. You grant access to other “users” via the S3 UI. (At the time of writing it is not supported by the Terraform AWS Provider.)

1. Go to the S3 bucket -\> Permissions -\> Access Control List -\> Access for other AWS accounts -\> Add account
    2. Find the canonical ID for the account you want to give permission to.
    3. Set the bucket-level and object-level permissions. Beware, the Principal (owner) of the object will be different, so the account that actually hosts the bucket might not have access to it.

---
Tags: amazon-web-services, terraform, amazon-s3, aws-iam
---


# Question
Title: can't access my instances by public IP in a VPC (aka: what is wrong with my terraform)?
As stated in the title of the question, I have a Terraform file (attached below), but the infrastructure it builds doesn't allow me to connect by SSH (I get "Connection timed out"). So either the packets aren't getting to the EC2 instances or the responses aren't getting out. Any ideas? For the purposes of this question the DNS and ALB can be ignored (the former works fine, the latter almost certainly doesn't yet).

```
provider "aws" {
  region = "us-west-2"
}

resource "aws_vpc" "vpc" {
  cidr_block = "10.0.0.0/16"
}

resource "aws_internet_gateway" "gw" {
  vpc_id = "${aws_vpc.vpc.id}"
}

resource "aws_subnet" "subnet_a" {
  vpc_id = "${aws_vpc.vpc.id}"
  cidr_block = "10.0.1.0/24"
  availability_zone = "us-west-2a"
}

resource "aws_subnet" "subnet_b" {
  vpc_id = "${aws_vpc.vpc.id}"
  cidr_block = "10.0.2.0/24"
  availability_zone = "us-west-2b"
}

resource "aws_route_table" "rt" {
  vpc_id = "${aws_vpc.vpc.id}"

  route {
    cidr_block = "0.0.0.0/0"
    gateway_id = "${aws_internet_gateway.gw.id}"
  }
}

resource "aws_route_table_association" "rt_a" {
  subnet_id = "${aws_subnet.subnet_a.id}"
  route_table_id = "${aws_route_table.rt.id}"
}

resource "aws_route_table_association" "rt_b" {
  subnet_id = "${aws_subnet.subnet_b.id}"
  route_table_id = "${aws_route_table.rt.id}"
}

resource "aws_security_group" "alb_sg" {
  vpc_id = "${aws_vpc.vpc.id}"

  ingress {
    from_port = 0
    to_port = 80
    protocol = "tcp"
  }

  ingress {
    from_port = 0
    to_port = 443
    protocol = "tcp"
  }
}

resource "aws_alb" "alb" {
  internal = false
  load_balancer_type = "application"
  security_groups = ["${aws_security_group.alb_sg.id}"]
  subnets = ["${aws_subnet.subnet_a.id}","${aws_subnet.subnet_b.id}"]
}

resource "aws_route53_zone" "primary" {
  name = "REDACTED.REDACTED"
}

resource "aws_route53_record" "alias_route53_record" {
  zone_id = "${aws_route53_zone.primary.zone_id}"
  name    = "REDACTED.REDACTED"
  type    = "A"

  alias {
    name                   = "${aws_alb.alb.dns_name}"
    zone_id                = "${aws_alb.alb.zone_id}"
    evaluate_target_health = true
  }
}

resource "aws_key_pair" "kp" {
  key_name = "kp2019y09m25d001i"
  public_key = "ssh-rsa REDACTED"
}

resource "aws_instance" "instance_a" {
  ami = "ami-08ba47c10a42c61be"
  instance_type = "t2.micro"
  availability_zone = "us-west-2a"
  subnet_id = "${aws_subnet.subnet_a.id}"
  associate_public_ip_address = true
  key_name = "${aws_key_pair.kp.key_name}"
}

resource "aws_instance" "instance_b" {
  ami = "ami-08ba47c10a42c61be"
  instance_type = "t2.micro"
  availability_zone = "us-west-2b"
  subnet_id = "${aws_subnet.subnet_b.id}"
  associate_public_ip_address = true
  key_name = "${aws_key_pair.kp.key_name}"
}

output "instance_a_ip" {
  value = "${aws_instance.instance_a.public_ip}"
}

output "instance_b_ip" {
  value = "${aws_instance.instance_b.public_ip}"
}

```

# Answer
> 1 votes
Your default security groups and newly created security groups include default rules that **do not enable you to access your instance from the Internet**. Add this:

```

resource "aws_security_group" "allow-ssh" {
  vpc_id = "${aws_vpc.vpc.id}"

  ingress {
    from_port = 0
    to_port = 22
    protocol = "tcp"
  }
}

```

Then add the line `vpc_security_group_ids = [aws_security_group.allow-ssh.id]` to the instance definitions.

---
Tags: amazon-ec2, terraform, aws-vpc
---


title: How do I convert xml to a Jenkins Job?

text:
# Question
Title: How do I convert xml to a Jenkins Job?
I want to restore a deleted Jenkin's job. I had the JobConfig history plugin installed, but it does't have a restore option for that job. It only shows the xml file. I'm wondering how I can restore a Jenkin's job from that xml file.

# Answer
> 2 votes
One possibility would be via the Jenkins REST API. I guess the recipe below would be dependent on the particular Jenkins version, so you may need to tweak it a bit. The version I looked at was 2.176.1

Go the Jenkins page for the folder in which you want the job created/restored. On the page's footer you'll see a `REST API` link, go there.

On the `REST API` page you have all sorts of links, customized for your particular Jenkins server and job folder. You're looking for this section:

> Create Job
>
> To create a new job, post `config.xml` to this URL with query parameter name=JOBNAME. You need to send a `Content-Type:
>  application/xml` header. You will get a `200` status code if the creation is successful, or `4xx`/`5xx` code if it fails. `config.xml` is the format Jenkins uses to store the project in the file system, so you can see examples of them in the Jenkins home directory, or by retrieving the XML configuration of existing jobs from `/job/JOBNAME/config.xml`.

---
Tags: jenkins
---


# Question
Title: How do I do a dry run with docker-compose?
I want to know what docker-compose is going to do before I do it.

Is there a tool or flag which I can use to see what docker-compose would do without actually stopping containers and starting new ones?

# Answer
> 7 votes
A dry run for `docker-compose` seems to be an open feature request.

---
Tags: docker, docker-compose
---


# Question
Title: How to secure persistent user data with docker on client location?
I'm working on a system where a dockerized application must run on client side machines (Ubuntu PC), with some sensitive data that must persist between runs that we want to limit access to. In prototyping I just stored the data in the host file system and used a bind mount to map that folder into the application container.

I want to avoid situtations where someone with access to the machine can just copy out some user data, or even look at it.

What would be a good solution in my case?

What I have read from google this sounds to be a bit of a different use case as to where user data/docker volumes are securely hidden away between the (fire)walls of server rooms and cloud instances and I haven't therefore been able to find a good solution on how to limit access to said user data so that no one (no proccess, nothing) except the running container has access to that data, even with physical access to the machine.

I am also very new to DevOps and deploying systems so I would appreciate any insight on how to solve that matter!

# Answer
> 1 votes
> I want to avoid situtations where someone with access to the machine can just copy out some user data, or even look at it.

You cannot stop someone with local access to the machine from reading data on the machine. Full stop.

# Answer
> 0 votes
You could save the data in a database. That database can be also a container (deploy them both with docker-compose maybe). The database is password protected and it can persist its encrypted data on the disk between runs.

Also you can use Hashicorp Vault (see https://docs.docker.com/samples/library/vault/).

---
Tags: docker
---


# Question
Title: Why does Jenkins safeRestart not wait for job to finish?
I have a simple pipeline job with a single stage:

```
timestamps {
    node ("jenkins-slave-small") {
            stage('Stage 1') {
                sh "echo 'stage2 echo'; sleep 500s"
            }
    }
}

```

I am testing whether or not /safeRestart waits for my pipeline to finish executing, but it seems like it's not.

Logs:

```
[Pipeline] { (Stage 1)
[Pipeline] sh
+ echo stage2 echo
stage2 echo
+ sleep 500s
Resuming build at Thu Sep 26 18:14:18 UTC 2019 after Jenkins restart
Waiting to resume part of test-folder » test-pipeline-no-parallels #6: Finished waiting
Ready to run at Thu Sep 26 18:14:55 UTC 2019

```

Am I misunderstanding how /safeRestart is supposed to behave or is that the intended behaviour?

# Answer
Jenkins pipeline jobs are designed to be serializable to disk, to increase durability (their words not mine). Therefore they are considered safe to be suspended. The only things you can do is either wrap your stages in functions with @NonCPS decorator or using properties make the jobs non-resumable after restart, but this will cause them to not be resumed after restart.

Also, restarting jenkins is not really a thing that happens all the time, so it might not be worth the effort to protect against.
> 0 votes

---
Tags: jenkins
---


# Question
Title: Setting up CI/CD for an AWS CDK app using AWS CodeBuild/Deploy/Pipeline
I'm trying to setup a CI/CD pipeline for a dotnet app which uses AWS Lambda and AWS CDK for infrastructure. The source is on github and we have a Jenkins pipeline which runs the tests and publishes the artifacts. I want to use the artifact and deploy (or better use Code Deploy)

Can I use CodePipeline to run cdk deploy?
How can I use CodeDeploy to do a `dotnet test` and `dotnet publish`? and then pass on the artifact to CodePipeline

# Answer
It may be helpful to look at the different AWS developer services as you are trying to achieve your goal in the wrong services:

* Running the specific dotnet testing and publish steps are not done in CodeDeploy. Rather, they would be done in AWS CodeBuild, in the AWS Pipeline, or directly on your Jenkins Server.
* AWS CodeDeploy is for taking your build artifacts and deploying them "to a variety of compute services such as Amazon EC2, AWS Fargate, AWS Lambda, and your on-premises servers".
* AWS CodePipeline is for modeling an automated workflow. This service integrates with Jenkins and GitHub.

With these services in mind I think what you are looking for is to start with a new CodePipeline. This will allow you to start building a model that specifies where you are running your dotnet tests (Jenkins or on AWS) and what to do with your artifacts (publish through Jenkins or AWS CodeDeploy). This can be accomplished with AWS CDK.
> 3 votes

---
Tags: cicd, dotnet, aws-codedeploy, aws-codepipeline
---


# Question
Title: How does the creation of an AMI on AWS work and why is the creation time unpredictable?
Creation date: `September 23, 2019 at 9:22:06 PM UTC+2`

`21:48`: still waiting for completion

`21:53`: completed

Sometimes it takes 15 minutes and sometimes longer. How does a creation of an AMI work and why is the creation time unpredictable?

# Answer
When creating an AMI, you are making a copy of your EBS volumes. This utilizes block storage and will only need to process the diffs of your EBS snapshots. If you changed a lot of data in your EBS or don't have a previous snapshot, it will take longer to create your AMI.

Here is a quote from the documentation at https://docs.aws.amazon.com/AWSEC2/latest/WindowsGuide/Creating\_EBSbacked\_WinAMI.html

> Depending on the size of the volumes, it can take several minutes for the AMI-creation process to complete (sometimes up to 24 hours). You may find it more efficient to create snapshots of your volumes prior to creating your AMI. This way, only small, incremental snapshots need to be created when the AMI is created, and the process completes more quickly (the total time for snapshot creation remains the same).
> 7 votes

---
Tags: amazon-web-services, amazon-ec2, amazon-ami
---


# Question
Title: Route53 Routing Policies - Best Policy for US-based Ecommerce?
When I look over the documentation for Route 53 Routing Policies they list 7 different policies. How should I pick the right one if I am building out new infrastructure in AWS for a US-based e-commerce website? Here is the list of offerings:

* Simple
* Failover
* Geolocation
* Geoproximity
* Latency-based
* Multivalue Answer
* Weighted

These different routing policies are pretty confusing. I would of course always want failover for my DNS. Why would I ever choose Simple? Is there some major difference between Geolocation and Geoproximity? What do they mean by "Traffic Flow Only" next to Geoproximity?

Because of the confusion around all of this I am for now choosing latency-based routing. That seems simple and intuitive. I have ELBs to handle failover for me, and with my latency routing I have associated health-checks, so it *seems* like my bases are covered.

Notes:

* We have a handful of ELBs but they are mostly in us-west-2 region, some in us-east-1.
* These ELBs are serving the same site out of multiple regions.

# Answer
To answer your question, ill first try to give simple explanations for each policy to help with the confusion (https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/routing-policy.html).

* **Simple**: a basic routing policy that randomly sends a user to one of your addresses.
* **Failover**: You have a primary (active) route and a secondary (passive) route. When the primary fails, it routes to the passive. If passive fails, you have an issue.
* **Multivalue**: Multiple routes that each have health checks. It will randomly select an address (Simple routing policy) from the healthy routes. A more complex failover policy.
* **Geolocation**: Route users dependent on where they are located. If a user is in England and gets better latency in Germany, they will still be routed to England.
* **GeoProximity**: A more fine grained Geolocation. You would be able to route people located on the border of France and Germany (based on longitude) to a specific route. "Traffic Flow Only" is to specify that you must use the Route 53 traffic flow to configure this policy. It is the most complicated of the route 53 policies
* **Weighted**: This policy allows percentage based balancing. Routing 30% of people to route-1, 20% to route-2 and 50% to route-3.
* **Latency**: Route to the address that gives the user the best latency.

Now for your architecture, your current implementation would handle instance failures with the ELBs and the latency based routing (with health checks) would handle the routing failures. If you don't require routing your users to the lowest latency address, use the multivalue policy instead.
> 1 votes

---
Tags: amazon-web-services, dns
---


# Question
Title: How to properly manage mupltiple environmental variables with Gitlab CI and Docker Swarm?
In our project we are using Docker and we have multiple services with a pile of environmental variables (mostly credentials to various dbs, mail, samba, 30+ entries in total). It all worked well with `docker_compose.yml` and `docker-compose up -d --build`. The advantage of this method was an extensive usage of env files. For example, I had mupltiple env files and I could use them in `docker-compose.yml`:

```
env_file:
  - ./envs/common.env
  - ./envs/mail.env
  - ./envs/mysql.env

```

We set up a Gitlab CI and are planning to use Docker Swarm, so no more manual creation of production env files, use CI/CD. And the primary way in Gitlab is to create variables one by one. We could create file type variable and put the contents of env file in it, but it will be hard to manage those inner-variables.

So how do I have to work with env vars now?

* Create a ton of variables in Gitlab UI and aggregate them in env files?
* Use file type variables?
* Pass them to services in `docker-compose.yml` file like this:

environment:

+ VAR1
    + VAR2 ...
    + VAR20

Or should we also use Docker secrets or some configuration management tool?

# Answer
There is stages in gitlab ci and on 'docker build' stage I don't need ALL env variables. Usually its only a REACT\_APP\_SERVER\_URI variable. Later I use env file in compose.yml like you are.

```
env_file:
  - ./envs/common.env

```
> 0 votes

---
Tags: docker, docker-compose, gitlab, docker-swarm
---


# Question
Title: How to see previous version of a Docker image?
I am looking for a netframework Docker image with .net 4.7.2 and netcoreapp2.2. I searched on the main docker website: DockerHub, .NET Framework By Microsoft without success. I also check the Microsoft website just in case.

Is there a way to see previous version of a Docker image? (Especially for the .NET Framework)

Something to allow me to fill this kind of table:

```

| docker image                         | framework | core  | aspcore |
|--------------------------------------|-----------|-------|---------|
| microsoft/dotnet-framework:4.8-sdk   | 4.8       | 3.0   | ?       |
| microsoft/dotnet-framework:4.7.2-sdk | 4.7.2     |  2.1? | ?       |
|                                      | 4.7.1     | ?     | ?       |
|                                      | 4.7       | ?     | ?       |
|                                      | 4.6.2     | ?     | ?       |
|                                      | 4.6.1     | ?     | ?       |
|                                      | 4.6       | ?     | ?       |
|                                      | 4.5.2     | ?     | ?       |
|                                      | 4.5.1     | ?     | ?       |
|                                      | 4.5.0     | ?     | ?       |

```

# Answer
`docker image inspect` \- will show available information on the container (docs)

`docker image history [your OPTIONS] imagename` \- Shows history of the container (docs)
> 2 votes

---
Tags: docker, dotnet
---