# Question
Title: kubernetes allocating pods with the wrong IP
I'm running kubernetes with the supposed pod network cidr of `10.244.0.0/16`. Everything seems to agree with that:

```
# history | grep kubeadm
kubeadm init --control-plane-endpoint "guide:6443" --upload-certs --pod-network-cidr=10.244.0.0/16

$ kubectl get nodes -o kubectl get nodes -o jsonpath='{.items[*].spec.podCIDR}'
10.244.0.0/24 10.244.1.0/24 10.244.2.0/24 10.244.3.0/24 10.244.4.0/24 10.244.5.0/24

# cat /run/flannel/subnet.env
FLANNEL_NETWORK=10.244.0.0/16
FLANNEL_SUBNET=10.244.0.1/24
FLANNEL_MTU=1450
FLANNEL_IPMASQ=true

```

However all the pods I create get IPs in the 172.17.0.0/16 range.

```
$ kubectl get pods -o wide
NAME                                   READY   STATUS             RESTARTS   AGE   IP            NODE         NOMINATED NODE   READINESS GATES
busybox1                               1/1     Running            1          13m   172.17.0.11   k8s-node-1   <none>           <none>
debain-shell-547f8fc54b-v8p99          0/1     ImagePullBackOff   0          27m   172.17.0.3    k8s-node-1   <none>           <none>
nginx-deployment-ex-54f57cf6bf-k55nn   1/1     Running            1          27m   172.17.0.5    k8s-node-1   <none>           <none>
nginx-deployment-ex-54f57cf6bf-q2khx   1/1     Running            1          27m   172.17.0.9    k8s-node-1   <none>           <none>
nginx-deployment-ex-54f57cf6bf-r8b7v   1/1     Running            1          27m   172.17.0.7    k8s-node-1   <none>           <none>

```

I can't talk to coredns (which is how I found this condition) and I can't seem to use my ingress service anymore (though I've restarted enough things by now I may have just temporarily broken that).

What is causing this, or alternatively, where should I look for more info?

# Answer
This was caused by missing `--network-plugin=cni` on the end of kubelet. Added that to the args and it all works now.
> 2 votes

---
Tags: docker, kubernetes
---


# Question
Title: What is the difference between EKS and EC2 installed kubernets?
I'm newbie in AWS usage.
I don't understand the difference between EKS and EC2 installed kubernets?

> Amazon EKS makes it easy for you to run Kubernetes on AWS without needing to install and operate your own Kubernetes clusters.

Does this means that we could install Kubernets to EC2 and use EC2 instead of EKS?

# Answer
> 2 votes
You absolutely could install Kubernetes to EC2 and setup the configuration yourself. However, EKS will handle the bulk of the setup, configuration, maintenance, and monitoring for you. Check out the feature page to determine if it is worth it to you.

---
Tags: amazon-web-services, kubernetes, amazon-ec2
---


title: How should GCP projects be organized?

text:
# Question
Title: How should GCP projects be organized?
I have recently inherited the previous director of devops duties in managing a transition from Heroku/AWS to GCP. I'm wondering what is best practice for organizing projects in GCP.

Should the be organized by production, staging, and QA environments? Should they be organized by each individual self contained front-end project or middleware project? If so, where do databases live in that function? The middleware project?

You can see how many core questions can lead to adjacent questions.. does anyone have best practices here about how they organize their GCP projects in the enterprise?

Corollary question: how does your project organization affect cost or inter-networking?

Thanks!

# Answer
Speaking primarily from the Google App Engine (GAE) perspective.

My preference is for environment-driven project split. This might be of interest: Advantages of implementing CI/CD environments at GAE project/app level vs service/module level?

I wouldn't go for architecturally-driven project split unless there are serious reasons for doing so (if you feel you have such reasons you should ask with details):

* probably difficult (if not impossible) to avoid inter-project bandwidth charges
* added configuration complexity for inter-operation between projects (GCP pretty much considers them completely isolated/standalone, except maybe for shared IAM permissions). Possibly relevant: Google app engine access cross project resources

Depending on the GCP products used there may be sufficient support for architectural splits inside the same project. At least GAE does IMHO.

My GAE apps are all standalone and do not use inter-networking at all, so no costs from this perspective. There are still networking (bandwidth) costs for accessing various GCP-provided infra/services, but those are irellevant/unavoidable from the project split perspective.
> 5 votes

# Answer
Google's recommended best practice is to create an Organisation within GCP so that you get access to folders, then model the hierarchy of your organisation within GCP:

Some guidance/rules:

1. there should only be one organisation that represents the entire organisation; configuring directory sync is very painful otherwise.
2. folders should represent business units
3. projects should represent instances of an app or component
4. use different projects for each environment, because each environment is generally an instance of an app or component
5. fully qualify the names of each folder or project, because GCP often doesn't show the full path to the project. Consider there are intentionally two applications named `mgmt-prod` in the above diagram. However, because they are called `amido-ops-pmo-mgmt-prod` and `amido-people-mgmt-prod` it's easy to tell just from the project name what they are.

The significant advantage of this model is that you can create a group called `TechOps` and give it audit or read permission at the Organisation level. Those people will have access to the entire estate. Equally, you can also grant granular permissions at the folder and project level based upon business requirements.
> 3 votes

# Answer
We have two projects: production and staging. Later runs dev, qa also. And for billing purpose, we tag resources accordingly.
> 2 votes

---
Tags: architecture, gcp
---


# Question
Title: PowerShell command to automate Azure Linux VM
I am looking for some PowerShell command which can check if any update/patch need to be applied to existing Azure Linux VM and then apply the patch if required. I will then use this PowerShell command to automate the process.

There was an extension but that appears to be outdated now.

# Answer
> 3 votes
Apologies for not directly answering your question, but by the sounds of your question, you are trying to build an **update management** system to mitigate the threats of vulnerable operating system components and tools having unpatched vulnerabilities which expose you to risk.

This functionality is actually already available for both Windows and Linux via Azure Automation in the form of the unimaginatively named Update Management solution.

Essentially you will need to do the following:

1. Configure Azure Monitor and Log Analytics to provide information about vulnerable packages on your machines,
2. Ensure that any network security groups and firewalls are configured to allow access to/from the various components of the system,
3. Enable Change Tracking and Inventory,
4. Enable Update Management,
5. Onboard your machines.

I would also highly recommend employing Azure Security Center as this provides a clearer overview of vulnerabilities across your estate, not just as applied to specific virtual machines.

---
Tags: azure-devops, linux, automation
---


# Question
Title: Deploying and testing features on a server by more than one developer
I have 4 environments (prod/preprod/SIT/DEV), also two main branches (Master & Develop), two developers work on features (One & Two).

Developer One creates a feature (Feature A), he then merges the feature into the Develop branch.

Developer Two also creates a feature (Feature B) and he does the same, he merges the created feature into the Develop.

Now, the Develop branch will have two features (A & B), when the business owner decides to go with one feature ONLY (feature A), what should be the right approach to follow?

what is the best GIT workflow that we should follow? especially when implementing DevOps solutions.

P.S. I only mentioned two developers with two features, while in real life, I might have many more at the same time, hence I need to know the best approach to follow.

# Answer
> 3 votes
Since you have a single released version product one thing to consider would be keeping the Develop branch or not:

* keeping it would have the advantage that at any moment you can bring in a hotfix in master even if Develop isn't stable enough for a merge into master. But this means you'd need to have 2 focus points instead of a single one - both branches need to be kept stable. There will be additional efforts for constant merges back and forth between the 2 branches to keep them from diverging. The overhead of context switching between them can be significant.
* getting rid of it (see How to get rid of develop branch for simplified Git flow) and doing the integration directly in master would reduce the overhead of carrying together 2 branches. The releases would simply be tagged versions of the master branch. But getting in a hotfix for the current release version would require the branch to be stable enough (virtually ready for a new release) at any moment. Which would be desired anyways and is achievable (with some effort).

In the context of continuing the practice of developing features in isolation (on separate, potentially long-lived feature branches) and going through the pains of integration hell one thing that could be improved would be a better communication between the business owner and the development team: deciding to drop feature B **after** it was integrated into the product is practically equivalent with adding a new feature C (late in the game).

Another (much better IMHO) option would be to switch to the Continuous Integration practice: features would be developed directly in master (using very short lived branches carrying tiny incremental changes much easier to handle, ideally merged back into master in less than 1 day), behind feature toggles/feature flags:

* during development the features would be disabled by default, making master virtually equivalent to the current release version, ready to receive hot fixes if/when needed.
* the flags can be individually enabled in various environments to determine the respective feature progress/readiness for release. This can be done simultaneously for virtually any number of features (or feature combinations), regardless of their size or development stage or duration, as different verification steps of the automated CI system.
* whenever a feature is ready and the business owners decide to accept it the respective flag is turned on in a tiny commit and a new tag "brings" the feature into the next release. No massive branch merges, no risk of surprises, everything should be just business as usual - all kinks should have been already ironed out. A much more predictable (and less stressful) development process.
* keeping the flag supporting code in place for a while also allows staging the dropping of the feature in a subsequent release by disabling the flag in a CI system verification stage. But eventually a cleanup of the feature flag and the supporting code can/should be done as a background code base maintenance task, making the feature permanent.

IMHO such approach is the only reasonable option in larger scale projects where many features are being developed simultaneously.

---
Tags: git, bitbucket, branch
---


# Question
Title: How can I convince someone that 100% reliability is not the right target for anything?
It's a fundamental principle of DevOps and SRE that failure is normal and setting a goal of perfect reliability is misguided. But sometimes IT execs and business leaders push back on the idea. They believe the business needs 100% reliability. What are some good way to persuade them to change their thinking?

# Answer
I was wondering whether anyone in this forum had new arguments they have heard or tried in person to convince executives of the need to set service level objectives at a level below 100%, hence the original question.

Here are some approaches I've seen or taken in the past to tackling this problem. *The Site Reliability Workbook* has a few suggestions here, which include:

* 100% is not a reasonable target since every component of any system has a non-zero probability of failure.
* External components (such as an ISP) sitting between the customer and the target system aren't 100% reliable.
* 100% reliability implies you can never change the system, since all change introduces risk.
* 100% reliability implies you spend all your time reacting to reliability problems and have no time for anything else.

In addition to the above, I've also gathered the following talking points from various in-person discussions that might be useful to persuade organizational leaders:

* If an IT exec sets a target of 100% reliability, they are basically encouraging their engineers to lie to them and hide problems, which means they will discover those problems in the most expensive possible way.
* Nature has never evolved a 100% reliable system. Gene replication is unreliable. The human heart is not 100% reliable. Life continues and thrives in the face of imperfect reliability, and so can technology systems and their users.
* No group of humans has ever engineered a system that's 100% reliable in the long run.

We also have to acknowledge that for certain dimensions of reliability, such as data loss prevention, some regulated industries require levels of reliability so high and over long periods of time that they are virtually indistinguishable from 100%, like 99.999999%. For other dimensions like availability and latency, three or four nines of reliability are often more than adequate, and some of the arguments above may help persuade IT leadership that this is true.
> 4 votes

# Answer
If your company/manager/etc is arguing for 100% reliability, please ask them to name **just one** human-engineered system that has achieved that mark over any non-trivial period of time. I'm not aware of any -- and I've looked *a lot*. Pacemakers don't even have 100% reliability.

Remove the humans. You'd be hard pressed to find anything in nature that achieved that goal, either.

A goal of perfection is deeply counterproductive and will eventually slow your innovation asymptotically towards 0.

Here's a fuller explanation.

You will ship much greater reliability to your customers if you concentrate less on reducing errors, and focus more on detecting them and mitigating them quickly. The smaller the blast radius, the faster you can go.

The sin is not in the failing, but in the failing to notice.
> 2 votes

# Answer
Why do you think 100% reliability is not a good idea? Can you support your beliefs with numbers, facts and logical explanations?

Is the needed hardware too expensive? Development time too long? Maybe you can show that the probability of success of a chain of components is the multiplication of the individual parts, some of which are not under your control?

If you can support your claims then present those to your managers, if not then your managers might be right.
> 1 votes

# Answer
This is actually quite easy, if you know how much lost means the system being down.

That is if you take eBay or Amazon they would have considerable loss even for one second of downtime in a year. So they strive to have a 99.9999... reliability/uptime.

So this is not about 100% but how many nines does it make sense to invest for you to come close to 100%.
> 1 votes

---
Tags: sre
---


# Question
Title: Ansible playbook fails with: no test named 'success'
I have an Ansible Playbook that is used to configure WSL on my Windows Laptop, having rebuilt my laptop recently I needed to run the playbook again, however it is failing with the following message:

```
The conditional check 'aptitude_installed is success' failed. The error was: no test named 'success'
  line 1

The error appears to have been in '/mnt/c/source/richardslater/workstation-setup/wsl/plays/wsl.yml': line 15, column 5, but may\nbe elsewhere in the file depending on the exact syntax problem.
The offending line appears to be:

  - name: ensure aptitude is installed
    ^ here

```

The context from the playbook is:

```
tasks:
- name: check if aptitude is installed
  shell: dpkg-query -W -f='${Status}' aptitude | grep 'install ok installed'
  register: aptitude_installed
  failed_when: no
  changed_when: no

- name: ensure aptitude is installed
  command: apt-get -y install aptitude warn=False
  when: aptitude_installed is success

```

The output of `dpkg-query -W -f='${Status}' aptitude | grep 'install ok installed'` is:

```
install ok installed

```

and `ansible --version`

```
ansible 2.2.1.0
  config file = /mnt/c/source/richardslater/workstation-setup/wsl/plays/ansible.cfg
  configured module search path = Default w/o overrides

```

and `pip show jinja2`

```
Name: Jinja2
Version: 2.8
Summary: A small but fast and easy to use stand-alone template engine
written in pure python.
Home-page: http://jinja.pocoo.org/
Author: Armin Ronacher
Author-email: armin.ronacher@active-4.com
License: BSD
Location: /usr/lib/python2.7/dist-packages
Requires: MarkupSafe

```

I'm not entirely sure how it ever worked, either that or something changed since I built this playbook, any ideas on how to fix?

# Answer
> 1 votes
You use pretty old ansible, maybe worth to try a new one. Although tests "succeeded" and "success" should be in 2.2 too.

But main problem I think it's `failed_when: no` in your task. It makes test `when: aptitude_installed is success` completely useless, it will be always `true` because `failed_when: no` marks all tasks as successful.

If you don't want to stop on task, but still to have an idea how it went, you should use `ignore_errors: true`, not `failed_when: no`. With `ignore_errors: true` you can both continue your playbook and inspect a real tasks results.

# Answer
> 0 votes
I do not have any clue of how your code once ran and produced the expected result. The `when` clause cannot be executed (as you found out from the error) and would anyway produce the reverse result you are expecting if I follow the underlying logic (install the package only when it is already installed). Moreover, your are in this simple case far from good ansible practice.

You should avoid using `shell` and `command` when there is an existing module that already does the job and manages idempotency for you. In present case, you can use the agnostic `package` module or the specific `apt` (which I will prefer in my example since your playbook targets an Ubuntu install on wsl and a debian/ubuntu package anyway).

```
tasks:
  - name: Ensure aptitude is installed
    apt:
      name: aptitude
      state: present

```

The above will simply install `aptitude` unless it is already there in a simple easy single task.

# Answer
> 0 votes
I got this error also and ended up here, reason is I think I was using another version of ansible on another computer so I changed the syntax to `when: sometest is success` because I got warnings before. Now on this computer where I run version `2.2.1.0` apparently this does not work at all, so I fell back on `when: sometest|success` instead and now it works.

---
Tags: ansible, jinja
---


# Question
Title: Gitlab runner, docker registry and self-signed certs
I've set up a Gitlab registry with self-signed certificate. After that I had to convince docker app to trust my CA, I did this by putting CA.crt into

```
/etc/docker/certs.d/registry.gitlab.yourdomain:5000

```

But you have to convince docker executor and dind service in Gitlab Runner too, I've done this by passing cert through volumes in `config.toml`:

```
volumes = ["/cache", "/certs/client", "path/to/cert/ca.crt:/etc/docker/registry.gitlab.yourdomain:5000/ca.crt"]

```

See the problem? Gitlab Runner uses old volume specification, so `:5000` part will break it. I know about existence of `pre_clone_script`, so that you can map your cert into some folder and then move it. But that will not solve the dind service problem - `pre_clone_script` will only be executed in 'main' container.

# Answer
> 1 votes
Oh well, the answer was simple: why do you need to use `volume` only for file transfer? Just `volume` the directory, containing problematic folder with cert in it.

---
Tags: docker, gitlab-ci-runner
---


# Question
Title: What are DevOps & Ops user stories, or elements in scrum context?
What type of user stories should Operations write down while being asked to be a stakeholder in a scrum project?

Following this template:

"As Operations engineer, I want to.. in order to.."?!

My first idea is for example to come up with for example expectations to quality standards i.e. *acceptance criteria* to deliverables, like "as an operations engineer, I want to get deliverables accompanied with installation and validation automation scripts in order to enable fast and transparent deployments", but is there more to it?

# Answer
> 7 votes
1. Make sure that your stories are not centered on Operations. Remember that DevOps is a culture not a role. Is an "Operations Engineer" really the stakeholder for the story?
2. Think of the features and business value they provide. As a **stakeholder** I want **feature** so that **business value**. If you are struggling to identify the business value why are you doing it?
3. Stories are supposed to be "vertical slices" as opposed to "horizontal slices". This means that stories should focus on an entire feature, not a layer. This fits very well in a DevOps culture as your stories can cut across the traditional Development and Operations layers.

You already have some interesting features such as the quality standards. Now you just need to identify who the end stakeholder is, what the business value of this would be, and ensure that you write the story in such a way that people with development, operations, and possibly QA skill sets can work on it.

---
Tags: architecture, operations, agile
---


title: How to restore default Jenkins plugins?

text:
# Question
Title: How to restore default Jenkins plugins?
I do a lot of testing in my Jenkins VM which I have on my local laptop. I found this time-consuming to remove Jenkins and installing again from scratch. It would be better if I have a simple way to restore my Jenkins.

Is there a way to remove all the plugins that I have installed manually and keep the plugins that were installed during Jenkins installation?

# Answer
One could use ansible geerlingguy.jenkins role. When this role is applied, a Jenkins system will be created without any plugins. Subsequently, one could install plugins manually, but also define them in ansible, i.e. plugins as code.
> 2 votes

# Answer
One could also use docker Jenkins LTS. If one wants to start from scratch, just remove the mounted Jenkins home folder and there will be a clean Jenkins in no time.

https://github.com/jenkinsci/docker/blob/master/README.md
> 3 votes

# Answer
One possible approach, although a bit tedious (and a bit dumb), would be to:

* make a fresh Jenkins installation of the exact same version on another (maybe scrappable) machine
* get the list of the (default) plugins it ends up with
* add/remove/update the plugins in the original Jenkins installation to match that list
> 2 votes

# Answer
I find it strange that nobody mentioned the Configuration as Code plugin. Our solution to this problem is to maintain a base Docker image for jenkins, provisioned with two config files:

* jenkins.yml
* plugins.txt

The plugins are installed by the `install-plugins.sh` script provided by the base Jenkins image:

```
COPY casc_configs/plugins.txt /usr/share/jenkins/ref/plugins.txt
COPY casc_configs/jenkins.yml /usr/share/jenkins/ref/jenkins.yml
RUN /usr/local/bin/install-plugins.sh < /usr/share/jenkins/ref/plugins.txt

```

and Jenkins itself is configured using the `jenkins.yml`.

The plugin file is generated from a template:

```
{% for plugin in jenkins.plugins %}
{{ plugin.name }}:{{ plugin.version }}
{%endfor%}

```

where the variable `jenkins.plugins` is a big list of dicts:

```
jenkins:
  plugins:
    - name: configuration-as-code
      version: 1.14

```

Doing things this way makes it possible to create and destroy fully configured jenkins instances on the fly. If you want, you can keep the `jenkins.yml` and `plugins.txt` on a mount that Jenkins can read when it starts.
> 2 votes

---
Tags: jenkins, jenkins-pipeline, jenkins-plugins, jenkins2, jcac
---


# Question
Title: How to run Ansible file in Jenkins using Ansible plugin
Please look into this error trying to implement the deploy.yml file with ansible plugin which is installed in the jenkins.

In ansible console I can implement the app successfully to the target machine but while same deploy.yml file using in jenkins getting error .

Tried :

1. ssh connection from target and master machine able to connect
2. Add the rsa key/finger print key to the target machine as well as in master controller .
3. From terminal i can able to access target machine as well as can able to deploy any app to any VM but while switched to jenkins ansible plugin getting error.

# Answer
> 1 votes
Got the solution :

in the inventory file (`/etc/ansible/hosts`) :

```
[host]
192.XXX.XXX.XXX ansible_user=ubuntu1(target hostname) ansible_become_password=XXXXXXXX(target pwd)

```

and in the play we have to set

```
- hosts: host
  become: yes
  tasks:
    -name : Update XYZ app
     script: update.py

```

I have already written a python script for my process so I have giving same script here in .yml

# Answer
> 3 votes
Simple one. You should either add `ansible_ssh_extra_args="-o StrictHostKeyChecking=no"` to inventory or `disableHostKeyChecking: true` into Jenkins pipeline script (inside ansiblePlaybook block).

Like this:

```
ansiblePlaybook(
  colorized: true,
  installation: 'ansible27'
  inventory: '[hostsfile]'
  playbook: '[playbook]'
  disableHostKeyChecking: true
)

```

---
Tags: jenkins, ansible, jenkins-plugins
---


title: S3 bucket policy errors

text:
# Question
Title: S3 bucket policy errors
I have the following bucket policy for Amazon S3, which is giving the following error:

> Error: Invalid bucket policy syntax.

Either it is a JSON syntax error or a Policy error.

This is the policy I am trying and failing with:

```
{
    "Version": "2012-10-17",
    "Statement": [{
        "Effect": "Allow",
        "Action": [
            "s3:ListBucket",
            "s3:GetBucketLocation",
            "s3:ListBucketMultipartUploads"
        ],
        "Resource": "arn:aws:s3:::<my bucket>",
        "Principal": {
            "AWS": [
                "arn:aws:iam::560184616970:user/<my username>"
            ]
        },
        "Statement": [{
            "Effect": "Allow",
            "Action": [
                "s3:AbortMultipartUpload",
                "s3:DeleteObject",
                "s3:DeleteObjectVersion",
                "s3:GetObject",
                "s3:GetObjectAcl",
                "s3:GetObjectVersion",
                "s3:GetObjectVersionAcl",
                "s3:PutObject",
                "s3:PutObjectAcl",
                "s3:PutObjectAclVersion"
            ],
            "Resource": "arn:aws:s3:::<my-bucket>/*",
            "Principal": {
                "AWS": [
                    "arn:aws:iam::560184616970:user/<my-username>"
                ]
            },
            "Statement": [{
                "Effect": "Allow",
                "Action": "s3:ListAllMyBuckets",
                "Resource": "*"
            }]
        }]
    }]
}

```

Can anyone give me guidance here?

# Answer
It looks like you tried to compile multiple policies into one.
Each original had a statement with its array of statements and you nested them.

This (untested) should work with each statement block within the array of Statement of the policy:

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:ListBucketMultipartUploads"
      ],
      "Resource": "arn:aws:s3:::<my bucket>",
      "Principal": {
        "AWS": [
          "arn:aws:iam::560184616970:user/<my username>"
        ]
      }
    },
    {
      "Effect": "Allow",
      "Action": [
        "s3:AbortMultipartUpload",
        "s3:DeleteObject",
        "s3:DeleteObjectVersion",
        "s3:GetObject",
        "s3:GetObjectAcl",
        "s3:GetObjectVersion",
        "s3:GetObjectVersionAcl",
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:PutObjectAclVersion"
      ],
      "Resource": "arn:aws:s3:::<my-bucket>/*",
      "Principal": {
        "AWS": [
          "arn:aws:iam::560184616970:user/<my-username>"
        ]
      }
    },
    {
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    }
  ]
}

```

An alternative could be to use aws bucket policy generator
> 1 votes

---
Tags: amazon-web-services, amazon-s3, json
---


# Question
Title: Why is developing Ansible Playbooks using Roles a best practice?
It's a best practice to develop Ansible Playbooks with reusable roles.

From what I've seen, people end up putting all their roles into a single complex files anyways. So it doesn't really fix the problem there. I've read they can be split into multiple files, but now there's more to manage and it's no better than just writing multiple playbooks.

Especially considering you can reuse playbooks and tie them together with an Ansible Tower Workflow, I'm not seeing the benefit of including an additional layer of abstraction and complexity for roles.

Can someone tell me if I'm crazy for thinking this?

**EDIT:**

I've had some more thoughts after reading Bruce's answer.

1. I don't want to be converting Ansible-Galaxy roles into playbooks, or make people I share with turn playbooks into roles.
2. Not everyone I share or collaborate with has access to Tower.
3. Testing

# Answer
There are many reasons that using roles is better than using long playbooks, and very few reasons that using a single long playbook is better than using a role. These are almost always clear only at a certain scale. Team size, workload number and complexity and frequency of execution all come into play here and the larger any of them get, the more obvious it becomes that organising things **hierarchically** and **modularly** into roles is the right thing to do.

From the Ansible docs, there is an emphasis on *re-usability*:

> Roles are ways of automatically loading certain vars\_files, tasks, and handlers based on a known file structure. Grouping content by roles also allows easy sharing of roles with other users.

This is also why Ansible Galaxy is potentially useful. One can summarise the *effect* of a long list of tasks, into a single concept - a role. Although the role may be considered as simply a long list of tasks -- and so not that different from a playbook -- the main benefit comes from the way it scopes variables. Out of the several analogies which may be applied, one may use the Object Oriented analogy. A role is like a class. The *usage* of that role, applied in a playbook, is like the *instantiation* of a class into an *object*. If you consider object-oriented programming useful, you will consider object-oriented infrastructure provisioning useful.

For me, however it comes down to **testing**. A playbook is *tactical* and is useful in *specific situations*. A good role is *strategic* and is written to solve general problems. Since *good* roles are modular, they should contain unit tests for various scenarios to prove to the downstream user that they do indeed solve the more general problem, as well as making it clear what is actually covered in the tests.

Typically roles should be **maintained**, with proper documentation:

* what variables do they use ?
* what scenarios are they written for?

and wide coverage:

* several base operating systems
* several deployment scenarios (docker, AMIs, bare-metal, *etc*)

and organise tasks cleanly:

* use tags
* dynamically import tasks based on execution conditons and user-provided variables

*etc*.

In short, I wouldn't consider it a fair comparison to compare a bunch of playbooks with the usage of different roles. When it comes time to refactor your playbooks into something maintainable, you will need roles.
> 6 votes

---
Tags: ansible, configuration-management
---


# Question
Title: How much memory/cpu to allocate per pod?
This is a question that's been bothering us quite a while. How can you determine how much to allocate per pod? What if I don't know the program, maybe it needs a lot of memory at peak times.

What are some general strategies to finding out how much to allocate? cpu/memory

# Answer
It's a severe problem to solve if you don't know the performance characteristics of the process you are containerising. There are a couple of approaches you could take that will get you an answer, although you will need to monitor it carefully in production and an adjust the resource limits as required.

* **Approach #1: Educated Guess** \- take a look around at what other people have done to host a specific process, GitHub is an excellent resource for finding Dockerfiles and their associated Helm Charts. Equally, if someone has done some tests for Virtual Machine sizing, you may be able to derive an answer that is acceptable in the short term.
* **Approach #2: Testing** \- my personal preference is to test the component in question:
    + The Apache Bench is excellent for quick performance tests against HTTP(S) endpoints
    + Tools such as the MySQL Benchmark tool allow you to test MySQL, many applications have dedicated benchmarking tools.
    + If the software is more complicated you can use tools such as Artillery or Gattling to create a performance test suite that you can use to slam the container and find out how it performs.

Whichever approach you take, you will need to make sure that you have a proper monitoring and dashboarding solution working so that you can actually analyse the behaviour of the containers under load and adjust accordingly.
> 3 votes

# Answer
To expand on Richard's excellent answer, load testing and logging can help as well:

* Logging can help with monitoring, dashboards, and alerts. However, to properly set this up your services will need to properly log when resources are becoming an issue. If you already have this then you can make a more informed guess at the resources that will be needed.
* Load testing can help you determine if your limits are reasonable for the load you anticipate. While you may be able to run with a small set of users, load testing can help determine the cost/benefit of increasing your cpu/memory. Please note that if you are using a cloud provider you often need to contact them before performing this type of testing (e.g. see AWS's policy).

By using these two items you can hopefully make a better educated guess and perform more targeted testing.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Secure deployments of on-prem artifacts to cloud
Right now, we have locally hosted (across multiple Datacenters) artifactory and Bitbucket. We're at a halfway point between Jenkins and Bamboo (with Jenkins beginning to make more traction against Bamboo).

However, we're currently going through the task of building out a hybrid cloud platform, which is essentially going to be K8s on top of a cloud provider such as AWS or Azure. However, I've no idea how to approach ***securely*** deploying code or artifacts over the internet. Nor can I seem to find a 'best practice' cheat sheet.

Any pointers?

# Answer
> 1 votes
Both Azure and AWS will have any container images built off their network pushed to their repositories using https which will encrypt the transfer (e.g. AWS ECR encryption). Both of these providers offer ways to encrypt the transfer of data into their non-container storage solutions (often via https) and then offer ways to encrypt the data at rest. Once it is on the cloud provider deploying to k8s can all be done securely within their servers.

---
Tags: jenkins, deployment, artifacts, bamboo
---


# Question
Title: How to setup Docker mirror proxy Repository in Nexus Repository ManagerOSS 3.19.1-01?
So,

We are going to setup local proxy repository. Which directs to further docker hub and our another remote Docker Hosted repository hosted on different location.Which is requested through Docker(group) in Nexus.

I want to create a mirror for our remote repository which we are accessing through docker proxy in Nexus Repository manager in our local repository and caching its contents when somebody requests a image that is hosted in remote repository and next time when somebody fetches it he is served through local caching.

> So if anybody pushes updated docker image on our remote repository it should be automatically mirrored in local repository

unlike when somebody requests updated image then local cache is invalidated and request again goes to remote repository for fetching the updated image.

Bottom line.

> How to do docker repository mirroring between two repository manager based on Nexus version i mentioned in question or any other way?

Any help appreciated.Thank you

# Answer
> 5 votes
In Nexus, go to the Admin Backend.

Create a blob store if you haven't already. Guidance can be found here: https://help.sonatype.com/repomanager3/configuration/repository-management#RepositoryManagement-BlobStores

Once you have a blob store, continue like so:

Choose "Repositories", hit "Create Repository". Choose "docker (proxy)" as the recipe.

Give it any name (for example "docker-hub").

Under "Remote storage" enter `https://registry-1.docker.io`.

As "Docker Index" select `Use Docker Hub`.

Select a blob store for it (which you have created above).

Hit "Create repository". No more configuration is needed for the proxy repo.

Next up: A group repository, from which you will pull:

Go back to "Repositories" and hit "Create repository" again. This time select "docker (group)".

Call it anything you like (for example "docker-group").

For "HTTPS" enter any port you like. (we have 10500)

For blob store select any blob store (for example the one you have created above).

Under "Group" select for "member repositories" the repository you have created above.

You can also add a "docker (hosted)" repository as well if you have one set up. (I will explain later)

Hit "Create repository".

Now open your Docker cli and do a pull of some Docker Hub image but don't pull it from Docker Hub, pull it through the HTTPs endpoint of your docker (group) repo that you have created above like so:

```
$ docker pull my.repository.com:10500/phpmyadmin/phpmyadmin

```

This will create a pull request to your Nexus OSS, which will proxy the request to Docker Hub. The image from Docker Hub will be cached in your Nexus and will be delivered to you.

After pulling browse your docker (proxy) repo that you have created above and check the images inside. You will find a phpmyadmin/phpmyadmin inside (or any other image you have checked out) with a current timestamp.

Hope that helps!

---
Tags: docker, nexus, docker-registry
---


# Question
Title: How to patch AssemblyInfo in TeamCity for .net core 2.1 projects?
In the .net core 2.1 project, there is a `*.csproj` file, that contains assembly information:

```
<PropertyGroup>
    <AssemblyVersion>2019.1.15341.0</AssemblyVersion>
    <FileVersion>2019.1.15341.0</FileVersion>
    <Version>2019.1.15341</Version>
</PropertyGroup>

```

In Standard **.net** projects there were Properties/AssemblyInfo.cs which contained the Assembly information, but in the .net core, there is no AssemblyInfo.cs, all Assembly information are in the csproj.

How can I change the Assembly Version, File Version in Assembly Info ***Patcher*** in ***TeamCity*** for the **.net** core projects?

# Answer
You can use `File Content Replacer` build feature with predefined templates for .NET Core

https://www.jetbrains.com/help/teamcity/file-content-replacer.html#.NET+Core+csproj+templates
> 4 votes

# Answer
Using `/p` switch didn't work for my. However `-p` works just fine

```
dotnet build -p:Version=1.2.3.4

```

You can also apply the same switch to the dotnet pack

```
dotnet pack Yourproject.csproj -p:Version=1.2.3.4

```

look at this page https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-build#msbuild for details.
> 2 votes

# Answer
Just in case anyone misses it, Rajesh commented above that *if using the .NET CLI Runner* you can simply add this to the **Command line parameters** option for the build step:

```
/p:Version=%build.number%

```

..and it'll version all the binaries for you. This worked for me and seems simpler than configuring the `File Content Replacer` as above.

I've added this as it seems to be a perfectly acceptable answer, and because Rajesh's comment didn't give full details (pointed me in the right direction though)
> 1 votes

---
Tags: teamcity
---


title: Ansible resetting swap linux

text:
# Question
Title: Ansible resetting swap linux
I am newbie to Ansible, i failed using ansible via command that way :

```
ansible host -a "/sbin/swapoff -a && /sbin/swapon -a"

```

Only this worked

```
ansible host -a "/sbin/swapoff -a
ansible host -a "/sbin/swapon -a

```

I am thinking of making a playbook, starting maybe with this:

```
- name: Turn off swap
  shell : "swapoff -a"

- name: Turn on swap
  shell : "swapon -a"

```

I would like to **reset swap space** with a condition, only when the swap used space is **greater** than **100mb**. Is it possible to do that ?

Many thanks for your help

# Answer
You can certainly do this through a handler or task, but first you need to generate the trigger conditions. The swap space is in the Ansible facts that are returned when `setup:` is run. From the Ansible docs:

> Facts
>
> These are variables that contain information pertinent to the current host (inventory\_hostname). They are only available if gathered first.
>
> ansible\_facts Contains any facts gathered or cached for the inventory\_hostname Facts are normally gathered by the setup module automatically in a play, but any module can return facts.
>
> ansible\_local Contains any ‘local facts’ gathered or cached for the inventory\_hostname. The keys available depend on the custom facts created. See the setup module for more details.

As an example:

```
ansible -i localhost, all -c local -m setup -a filter=*swap*
localhost | SUCCESS => {
    "ansible_facts": {
        "ansible_swapfree_mb": 437,
        "ansible_swaptotal_mb": 979
    },
    "changed": false
}

```

In your playbook your task to reset swap should then have the conditional like:

`when: ansible_swap_free_mb < 500`

In your case, however you want to trigger on swap used, so you would have to compute the difference between total and free.
> 2 votes

# Answer
I have a solution on that way :

```
---
- hosts: hostnametest
  tasks:
   - name: we take swap space used (megabytes)
     shell : free -m | grep Swap | awk '{print $3}'
     register: swap_used

   - name: Turn off swap
     shell: "swapoff -a"
     when: (swap_used.stdout_lines[0] | int) > 100

   - name: Turn on swap
     shell: "swapon -a"
     when: (swap_used.stdout_lines[0] | int) > 100

```
> 1 votes

---
Tags: ansible, linux
---


# Question
Title: DC Jenkins server to AWS servers connection
We are trying to automate the deployment process for our application, our application has two components (one hosted on DC, one hosted on AWS). Now, we are not sure how do we connect the jenkins from our DC to the AWS environment hosted servers using VPN or is there any way by which the connection can be established.

P.S: We cannot install a new jenkins in the EC2 instance directly and do the deployment there, as we would need to pull the source code for the build to the cloud layer which is not acceptable by our compliance team.

# Answer
> 3 votes
Depending on your use-case, you should look either:

---
Tags: jenkins, amazon-web-services, vpn
---


# Question
Title: Last Question on CI-CD edx.org Final Exam
**The problem:**

An organization is developing a solution that contains

```
a web application project
class library projectS

```

You manage a continuous integration pipeline for the application.

Each time code is checked in, you must

```
build both debug and release configurations in parallel

```

*The most recent build has compile errors for the release configuration but not for the debug configuration. Developers confirm they have tested the release configuration correctly works on their devices.*

**QUESTION: What is the reasons for the build issues? (SELECT ONLY ONE)**

(i know "what IS the reasonS is just unfair, but it's as it is!!)

**o** The parallel builds caused contention issues for shared solution files.

**o** The web application references assemblies from the debug folder of the class libraries debug.

**o** A developer set their configuration to debug in Visual Studio.

---

For me is B.

NOT A, because you can use MSBuild Task's BuildInParallel option.

NOT C, because developerS confirm THEY have tested the RELEASE config (ON THEIR DEVICES)

---

Can you help me, please?

# Answer
I would suggest that the answer is most likely to be B, although it may be A under some circumstances.

* **A**: You can and should use `BuildInParalell` however sometimes this causes a race condition, for example, it could be that a developer omits the `$(Configuration)` in an output path meaning that both the `Debug` and the `Release` build output a file to the same location, then it just becomes a fight to see which build actually gets and locks that file first.
* **B**: If a developer was referencing binaries in the `Debug` folder **and it was a clean checkout** then those assemblies wouldn't exist, although if they are being built in parallel on the same machine they, in fact, might exist.
* **C**: If a developer sets their configuration to Debug (or anything else for that matter) it will impact builds on **their** machine assuming you use the `/p:Configuration=Release` option on the `msbuild` command line on the CI/CD environment.

It's entirely possible that Azure DevOps or any other CI/CD tool could be configured to rebuild each configuration in a clean container, thus meaning Release and Debug are actually being compiled in their own sandboxed filesystem thus can't possibly conflict with each other.

This isn't a great question for an exam because there is not quite enough information to differentiate between options A and B. Some exams do intentionally do this though, and often your best bet is to select the **least wrong** answer in which case I would select B.
> 0 votes

---
Tags: continuous-integration, azure-devops, configuration, builds
---


# Question
Title: What are the security risks of CD in an open source project?
In the processes of setting up continuous deployment for an open source project with Travis I came across a predicament, can the encrypted private key be easily exfilitrated? For context, the deployment workflow is:

1. In a Travis build job, build and test code
2. If it passes, deploy the code
3. Decrypt the encrypted private SSH key and start the SSH client
4. Push to the Git remote on the production server

Using `travis encrypt-file deploy_rsa --add`, the private key is encrypted and only decryptable within the Travis build job. Am I correct in the assumption that someone forking the repo and creating a Travis build job will not be able to decrypt the key? In addition, the **only attack vector** would be for someone with push rights to modify the `.travis-ci` script to decrypt the key and send it to themselves?

# Answer
You essentially have three threats here:

1. Someone could modify `.travis-ci` and use that to exfiltrate the unencrypted key material.
2. You could accidentally check-in or publish the decrypted key material as part of your release process.
3. Someone could attempt to attack the encrypted key-material directly; this is very unlikely given modern practice and current computing power.

The challenge of secret management is common in open source projects; one approach I have taken in the past is to have a second "deploy" stage in a private repository and pipeline. A private repository and pipeline, then ring-fences the sensitive data from malicious actors entirely, protecting yourself from 1 and 3. You can safeguard against two by whitelisting files that can be checked in to decrease the probability of making a mistake.

Finally, an approach that can be used to increase the detectability of an attack is to store the key in AWS CloudHSM, Google Cloud HSM or Azure KeyVault with the goal of marking the private key as *non-exportable* and then handing off any encrypt, wrap or sign requests to the cloud. This means even if the credentials for the HSM/KeyVault do get exfiltrated you know every time they are used to sign something and can effectively revoke access to the key entirely. Unfortunately, there doesn't seem to be any way of doing this with SSH keys at the moment, however, it works will with code-signing certificates.
> 1 votes

---
Tags: continuous-deployment, travis-ci, open-source
---


# Question
Title: How to incorporate containers into a standardized dev environment?
I'm currently working at a small company where we've had a lot of issues being on the same page with entity framework migrations and local databases. I want to start incorporating containers in hopes of standardizing everyone's development environment. Ideally, I'd like a disposable database that can be modeled using Entity Framework code first, seed data into it, then disposed of once the testing is done. What resources are there to help get me started and what other things should I know about standardizing the development environment.

# Answer
Docker is an awesome tool for creating a database, seeding data into it, and disposing of it.

For an example check out Microsoft's SQL Server with Docker. The following would pull and run a sql server 2017 instance:

```
docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD="
   -p 1433:1433 --name sql1
   -d mcr.microsoft.com/mssql/server:2017-latest
```
After running this you now have a developer ready SQL instance that you can then connect to SQL Management Studio (or any other GUI management tool), or you can even connect via the command line.

If you are not using SQL you can run a container for the desired database type (check out the options on Dockerhub).

Once you have your database up and running, run the scripts to set it up, seed the data, and test without worrying about messing up your production database.
> 1 votes

---
Tags: docker, databases, sql
---


# Question
Title: What is self hosted version control system?
What is self hosted version control system and how is it different from github? I mean if I use self-hosted VCS then where does my code go? I mean: does it get uploaded to the server, or it is saved on my local machine? If it is uploaded to the server, then if I reset the computer, will I able to access my code again?

# Answer
Let's take it word by word:

* **Self**: You or your organisation owns this thing. By contrast, a cloud-based offering by GitHub.com, Azure, GitLab.com, etc. (see this list on Wikipedia for a more comprehensive list) is not owned by you, but rather the service provider in question. Do not confuse ownership of the *code* (dictated by it's license – typically owned by you), with ownership of the service for performing version control on the code.
* **Hosted**: You or your organisation are responsible for *operating* and *maintaining* this thing. In some cases, this is exactly the same software (e.g. one can self-host GitLab or run GitHub Enterprise “on premise” in your own datacentre). This means keeping it secure, making any changes to it, serving user requests for change, etc. You are also responsible for some part of the infrastructure necessary to provide this service. This may vary, depending on who you have service provision agreements with, from everything down to the power and networking, or only the last layer of the service (here, the version control system itself), or anything in between.
* **"version control system"** (or VCS): anything (software or otherwise, but most likely software) which provides the functionality necessary to control versions. Note there is a subtle difference between Version Control and Version Control Software, although the meaning is often conflated.

The term VCS can refer to any version control system, even when it is not applied to software, however in our context, it can be assumed that you are referring to a version control system **applied to software**.

In other words, a “self-hosted VCS” can be thought of as:

> A service and all of its components which is under your own ownership and control, acting as a version control system for software.

Often, this can take the form of:

* base infrastructure (network/power/server),
* operating system,
* authentication and authorisation mechanism, and
* the VCS server itself (Git, SVN, etc.).

The second part of your question is about how it is different from GitHub. The following contrast can be made:

1. Functionally there may be differences between what you can do with GitHub an what you can do with your self-hosted VCS. GitHub, as is the case for almost all VCS hosted by third parties (i.e., **not** hosted by you) goes a few steps further than simply doing version control. There are collaboration tools, identity management, issue tracking, etc. built into their service. While this is strictly speaking not part of VCS, it greatly improves the experience of using one, almost to the point where doing version control without them feels impossible.
2. GitHub owns and operates the service. You may make suggestions to change their service, API, etc., and perhaps even contributions to their open source products, but you do not own the service. It is provided to you based on the SLA for the offering that you choose. You cannot make your own changes to it. This is overall a very good thing, unless you are in the business of providing the VCS itself – but chances are that you just want to *use* the service, not compete with it.

Update to answer the edited question:

> does it get uploaded to the server, or it is saved on my local machine? If it is uploaded to the server, then if I reset the computer, will I able to access my code again?

You are conflating two issues here: self-hosting and **distributed version control**. Of course, if you self-host *on your own computer*, your run the risk of losing everything if the machine dies. However, of *someone else* hosts your code in only one place, then you *still* risk losing everything if *their* machine goes down.

To address this (and several other issues related to centralisation of source control), **distributed version control** was invented. A good discussion is in the git book.
> 23 votes

# Answer
Assume you have your own IT infrastructure, eg a Linux server or a good old mainframe, on which you are running (hosting) your VCS software of choice, e.g GIT or ChangeMan ZMF. In such case you have a self hosted hosted VCS. As compared to GitHub, which is a web-based hosting service for software development projects using Git.
> 5 votes

# Answer
An example will be maybe worth a thousand words:

```
$ sudo apt install git
$ ssh 192.168.0.110
kub@host:~$ git init --bare repo1
Initialized empty Git repository in /home/kub/repo1/
kub@host:~$ logout
Connection to 192.168.0.110 closed.

```

Voila, We already posses a self-hosted version control system. Let's use it as an example client:

```
$ git clone 192.168.0.110:repo1 /tmp/r
Cloning into '/tmp/r'...

```
> 5 votes

# Answer
> What is self hosted version control system

A version control system hosted in a location you control and administer yourself.

Git is designed as a "distributed version control system", this means that by default every user has a complete copy of the history. It's not strictly necessary to even have a central server at all, though it is generally very inconvenient not to.

At a minimum a shared git repo can simply be a shared directory that you all have access to on a unix-like system you can all ssh into.

Most people however would like something a bit fancier than that, they may want the ability to let people perform some write actions on the repo without giving them full access to do anything they like to it. They may want a web interface to browse the code. They may want issue-tracking that integrates with the repo, they may want a pull-request system.

> and how is it different from github?

Github mostly operates as a cloud-service\*, your code is stored on servers owned and controlled by github. They provide hosting for git repos, along with a bunch of related services like issue tracking and pull-requests. Initially their model was that public repos were free and private ones cost money. Since the microsoft takeover they have started offering limited private repos for free, but if you want to do serious private collaboration you are going to have to pay them.

The advantage of cloud-services is that you don't have to expend any effort on maintaining the service. The downside is the operator of the cloud service can screw you over at any time and there is little you can do about it.

With git the risk is somewhat less than with other cloud services because as I mentioned earlier the actual version control history is kept by every client, so you can set up a new git server, switch your clients over and as far as the code itself is concerned things can continue as before. However you still risk losing the other stuff like the issue tracker, pull requests, permission setups etc.

If you want to self-host a git service with integrated web interface, issue-tracking and pull-requests then the main option i'm aware of seems to be gitlab.

* They do offer a self-hosted option for "Github Enterprise", but their website says nothing about how much it costs.....
> 2 votes

# Answer
A "self hosted version control system" is one that is used for tracking its own source code.

When a new version control system is developed, initially no version control is used for it, or some existing version control.

Eventually, a major milestone is reached when the version control system is good enough for handling its own source code. At that point, the developers are motivated to actually switch to it; and thereby it has become self-hosted.

This is a similar concept to a self-boostrapping compiler: a compiler written in its own language. You need that compiler to compile that compiler, just like you need a running version of a self-hosting version control system in order to check out its own sources.

Every serious version control system is self-hosted, because if it isn't, it shows that the authors don't have confidence in it; would you use Git if the development of Git relied on something else, like Mercurial?

**This is not to be confused with a "self-hosted repository" which is just a version control repository that you host yourself on a machine that you control instead of using someone else's hosting service like gitlab.**
> 1 votes

---
Tags: github, distributed-systems
---


# Question
Title: Azure DevOps Artifact.zip folder structure for .NET Core 3.0 application
I have an ASP.NET Core 3.0 application for which I have a full CI/CD setup in Azure DevOps. The Application is hosted on a Ubuntu 18.04 server machine.

In the Build Pipeline, the solution is built and afterward published to the *ArtifactStagingDirectory*.

```
- task: VSBuild@1
  inputs:
    solution: '$(solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactStagingDirectory)\MyProject.$(build.BuildNumber).$(Build.BuildId).zip"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: PublishBuildArtifacts@1
  inputs:
    PathtoPublish: '$(Build.ArtifactStagingDirectory)'
    ArtifactName: 'drop'
    publishLocation: 'Container'

```

This zip contains the complete folder structure of the server, which has IMHO nothing to do with the project. For me this is noise. For example:

> /drop/MyProject.20191028.2.49/Content/d\_C/a/1/s/MyProject/obj/Release/netcoreapp3.0/PubTmp/Out

Is it possible to reduce this folder structure to something like this or even less?

> /drop/MyProject.20191028.2.49/Release/netcoreapp3.0/

---

At the moment, I am working around this in the release stage with a bash script.

```
drop=$(find /MyProject/Release/drop/MyProject.$(build.BuildNumber).$(Build.BuildId)/ -name "Out")

ln -s $drop /MyProject/Release/Current

```

In general, I am fine with this, but a little cleaner solution would be nice :)

\*The application will be ported to .NET Core 3.1 as soon as it gets released. If this is relevant.

# Answer
I'm looking this fact:

> I have an ASP.NET Core 3.0 application for which I have a full CI/CD setup in Azure DevOps. The Application is hosted on a Ubuntu 18.04 server machine.

Therefore I can safely assume that you are developing ASP.NET Core 3.0 app to be hosted in Ubuntu. Any .NET Core 3.0 (or later) application means that you should rely on the `dotnet build` instead of using VSBuild.

Also you had stated that you will host the app on Ubuntu 18.x, then you should also run the build on Azure DevOps agent that runs on Ubuntu. This means you should only use `dotnet build` in `DotNetCoreCLI@2` task, because `VSBuild` task only runs on Windows based agent, not Ubuntu and it is intended to compile .NET Framework and other platform other than .NET Core.

Please consult the official doc of DotNetCoreCLI@2 task at https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops

After you use `dotnet build`, use `dotnet publish` to publish your artifact. It is much easier and it's also the best way to publish .NET Core apps.

I have sample usage of `dotnet build` and `dotnet publish` of this on my repo: https://github.com/eriawan/dotnetcore-cicd-samples/blob/master/yaml-samples/sample\_pipelines\_check.yml
> 3 votes

---
Tags: azure-devops, linux, artifacts
---


# Question
Title: Problem with Connecting Jenkins to SonarQube
I have a Jenkins pipeline that periodically pull from gitlab and build different repos, build a multi-component platform, run and test it. Now I installed a sonarqube server on the same machine (Ubuntu 18.04) and I want to connect my Jenkins to sonarqube.

In Jenkins:

* I set up the sonarqube scanner at `Global Tool Configuration` as below:
* I generated a token in sonarqube and in Jenkins at `configuration` I set up the server as below BUT I couldn't find any place to insert the token (and I think this is the problem):
* In the jenkins pipeline this is how I added a stage for sonarqube:

```
stage('SonarQube analysis') {
    steps{
        script {
            scannerHome = tool 'SonarQube';
        }
        withSonarQubeEnv('SonarQube') {
            sh "${scannerHome}/bin/sonar-scanner"
        }
    }
}

```

But this fails with below logs and `ERROR: script returned exit code 127`:

```
[Pipeline] { (SonarQube analysis)
[Pipeline] script
[Pipeline] {
[Pipeline] tool
Invalid tool ID
[Pipeline] }
[Pipeline] // script
[Pipeline] withSonarQubeEnv
Injecting SonarQube environment variables using the configuration: SonarQube
[Pipeline] {
[Pipeline] sh
+ /var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarQube/bin/sonar-scanner
/var/lib/jenkins/workspace/wws-full-test@tmp/durable-2c68acd1/script.sh: 1: /var/lib/jenkins/workspace/wws-full-test@tmp/durable-2c68acd1/script.sh: /var/lib/jenkins/tools/hudson.plugins.sonar.SonarRunnerInstallation/SonarQube/bin/sonar-scanner: not found
[Pipeline] }
WARN: Unable to locate 'report-task.txt' in the workspace. Did the SonarScanner succeeded?
[Pipeline] // withSonarQubeEnv
[Pipeline] }
[Pipeline] // stage

```

And when I check my jenkinstools on the disk sonnar plugin is not there:

```
$ ls /var/lib/jenkins/tools/
jenkins.plugins.nodejs.tools.NodeJSInstallation

```

Can someone please let me know how I can connect Jenkins to sonarqube?

# Answer
Create and add token to be able to connect to SonarQube.
You have create project in SonarQube and use it as a parameter:

```
sh """
   ${scannerHome}/bin/sonar-scanner \
   -Dsonar.projectKey=your_project_key_created_in_sonarqube_as_project \
   -Dsonar.sources=. \
"""

```

---

Copied from \[StackOverflow\] Problem with Connecting Jenkins to SonarQube.
> 1 votes

---
Tags: jenkins, jenkins-pipeline, jenkins-plugins, sonarqube
---


# Question
Title: What is the term for a deployment designed to make a future cutover seamless?
I'm currently working on a new major version (v2) of an application meant to replace the current production version (v1). This redesign carries with it changes to the front-end, public API layer, internal API layer, and database — the whole kit and kaboodle.

While it is easy to have active v1 and v2 front-ends and public API (which I'll call the "front half" for brevity's sake) simultaneously in our lower environments, for various reason, this is NOT possible for us to do with the internal API and database (which I'll call the "back half").

The v2 redesign necessitates changes to the back half that cause breakage in the v1 front half. However, in this particular instance, by making a minor change in the v1 front half, this breakage can be avoided. As such, it is possible for us to deploy this v1 "shim" along with the changes to the back half to production. When the time comes, we will be able to cut over to the v2 front half without the back half needing any further deployments.

Is there a name for this particular type of deployment, where a shim is deployed to an existing app to make it work with an updated back end, with the goal of making a future cutover seamless?

Taking my definition of blue/green deployment from this informative blog post, I believe this is a distinctly separate kind of deployment.

# Answer
> 2 votes
> ... by making a minor change in the v1 front half, this breakage can be avoided.

The above excerpt of this queston reminds me of something similar from around the late 90s, say 1996-1999 (when zillions of ITers were preparing for Y2K ...). In those days, new software releases of huge software vendors would come with release notes that included things like either of these:

* *"this release is to make this software **Y2K tolerant** "* (= allows the software to contnue to work as of Jan 1, 2000, at least for a while, though it would keep showing dates with 2 digits for the year, like 00, and not yet 4 digits, like 2000).
* *"this release is to make this software **Y2K ready** "* (= allows the software to continue to work as of Jan 1, 2000, without any (known) limitation because of the change of century from 19 to 20).

Y2K tolerant softwares would typically also include some sort of magical cut-off value, e.g.: years from 70 (when even MS DOS did not exist ...) to 99 were assumed to be from 1970 to 1999, while anything from 00 to 69 were assumed to be from 2000 to 2069. Such assumption was also why such softwares were required to be superseeded by some Y2K compliant release later on (in which those assumptions were obsoleted).

With the above in mind, the term you are asking for could be something like a "***V2 tolerant release***", which at some point in the future would then be followed by a "***V2 ready release***" (= your actual V2 release).

# Answer
> -2 votes
"Compatibility layer" sounds good to me.

# Answer
> -2 votes
Perhaps "Zero-downtime live migration"

---
Tags: deployment, terminology, release-management
---


title: docker hub repo

text:
# Question
Title: docker hub repo
I have created a repo on my docker hub account. While trying to download it it shows errors. I have tried the following commands:

* `docker id saurabhgore65`
* `docker pull <saurabhgore65>/subash`
* `docker pull <saurabhgore65>/subash:latest`
* `docker pull <saurabhgore65>/subash:sagoreimage`

As you told there is nothing in the docker hub repository. What are subash and sagore a

# Answer
> 0 votes
Try below cmd to pull the images from dockerhub:

`sudo docker pull username/reponame:latest[tag]`

*ie.*

`sudo docker pull iakshaykr/dochub:tom`

above: \- iakshaykr (username for docker hub account) \- dochub (repo created on my dochub) \- tom is image which i have pushed ,in your case it is tag which you wants to pull

I looked into your repo and found that you haven't pushed any image till now into your repo ,so you can't pull. First push an image from your local repo then you can able to pull.

`docker pull saurabhgore65/subash:[tag]`

# Answer
> 0 votes
First make sure that you have entered `docker login` in the cli. After that it really depends on the error you are getting. You may need to use an elevate permissions (i.e., Admin powershell console or `sudo` prefixed to the pull commands).

As stated in another answer there is currently nothing in your docker hub repository. To fix this you should first build an image `docker build . -t myimagename` from your dockerfile. Next you will need to push the image: `docker push saurabhgore65/subash`. If you do not specify a tag then it will default tag as "latest". Once something is in your repository you should be able to pull it.

---
Tags: docker, docker-registry
---


# Question
Title: Adding default external network in docker-compose
I am trying to learn docker and understanding docker-compose

As I was trying out the external network section:

```
 networks:
  default:
    external:
      name: my-pre-existing-network

```

I understand that 'my-pre-existing-network' needs to be created.

Is it possible to create a new default external network from within the compose file itself?

This is more from a learning/understanding perspective. Thanks.

# Answer
> 1 votes
No that is not possible , it is called `external` because is created from outside the current `docker-compose.yml` you are working on

---
Tags: docker, docker-compose, dockerfile
---


# Question
Title: Accessing AWS Secrets Manager from Docker
All the tutorials I've seen for AWS Secrets Manager have secrets being retrieved by the CLI. How is this done in production?

Can I mount a set of secrets to a docker container as a file?

# Answer
> 1 votes
> Access to AWS Secrets Manager requires AWS credentials. Those credentials must have permissions to access the AWS resources that you want to access, such as your Secrets Manager secrets. The following sections provide details on how you can use AWS Identity and Access Management (IAM) policies to help secure access to your secrets and control who can access and administer them.

so you need to provide the `aws` credentials to docker which has access to the manger

you can follow on of these methods:

providing the credentials with `run` command:

```
docker run -e AWS_ACCESS_KEY_ID=XXXX -e AWS_SECRET_ACCESS_KEY=XXXX myimage

```

or providing the file `~/.aws/credentials` and then create your container:

```
docker-machine create --driver amazonec2 --amazonec2-open-port 8000 --amazonec2-region us-west-1 aws-sandbox

```

see this

---
Tags: docker, amazon-web-services
---


# Question
Title: Check connectivity of 5000 + servers from ansible and get list of hosts which are connectable and which are not
I need to check connectivity of hosts on port 43. I can manually login and check connectivity by:

```
nc -v `hostname` 43
nc -v `hostname_1` 43

```

Is there any ansible script that goes through all the servers and saves the output of connectable hosts in one file and not connectable in another file?

# Answer
Maybe a simple shell script would be more useful. If must to use ansible, it could be something like that (assuming you have all servers in group `testall`):

```
- hosts: testall
  connection: local
  gather_facts: false
  tasks:

    - wait_for:
        timeout: 2
        port: 43
        host: '{{ (ansible_ssh_host|default(ansible_host))|default(inventory_hostname) }}'
      register: waitfor
      ignore_errors: true

    - copy:
        content: |
          {% for host in groups['testall'] %}
          {% if hostvars[host].waitfor.failed %}{{ hostvars[host].inventory_hostname }} {% endif %}
          {% endfor %}
        dest: /tmp/bad_file
      delegate_to: localhost

    - copy:
        content: |
          {% for host in groups['testall'] %}
          {% if not hostvars[host].waitfor.failed %}{{ hostvars[host].inventory_hostname }} {% endif %}
          {% endfor %}
        dest: /tmp/good_file
      delegate_to: localhost

```

hosts that have this port opened will be in `/tmp/good_file`, others are in `/tmp/bad_file`
> 2 votes

---
Tags: ansible, automation, networking
---


# Question
Title: Why apt command stuck with Waiting for headers in docker?
I have an Ubuntu 19.04 based docker container:

```
docker run -d -v /home/adr:/root/adr -p 880:80 --name ubuntu-19.04-nginx ubuntu:19.04 tail -F /var/log/alternatives.log

```

on which I compiled an nginx and installed these packages:

```
apt-get update
apt-get install apt-file net-tools htop iputils-ping git curl wget build-essential libcanberra-gtk-module jq zlib1g-dev libssl-dev luajit luarocks libgeoip-dev libperl-dev libxslt1-dev libgd-dev libluajit-5.1-dev lua-cjson nano mlocate mc psmisc apt-utils telnet dnsutils iproute2

```

I then committed the image and created another container:

```
docker run --net albums-webapp -d -v /home/adr:/root/adrx -p 880:80 -h nginx --name nginx-lua --network-alias=nginx adrhc/ubuntu-19.04-nginx-lua tail -F /var/log/alternatives.log

```

on which I tried to install:

```
apt install mysql-client

```

which yields:

```
root@nginx:~# apt install mysql-client
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  libaio1 libnuma1 mysql-client-5.7 mysql-client-core-5.7 mysql-common
The following NEW packages will be installed:
  libaio1 libnuma1 mysql-client mysql-client-5.7 mysql-client-core-5.7 mysql-common
0 upgraded, 6 newly installed, 0 to remove and 1 not upgraded.
Need to get 9459 kB of archives.
After this operation, 65.1 MB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Err:1 http://archive.ubuntu.com/ubuntu disco/main amd64 libnuma1 amd64 2.0.12-1
  Connection failed [IP: 91.189.88.162 80]
Err:2 http://archive.ubuntu.com/ubuntu disco/main amd64 libaio1 amd64 0.3.111-1
  Connection failed [IP: 91.189.88.31 80]
Ign:3 http://archive.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client-core-5.7 amd64 5.7.27-0ubuntu0.19.04.1
Err:4 http://archive.ubuntu.com/ubuntu disco/main amd64 mysql-common all 5.8+1.0.5
  Connection failed [IP: 91.189.88.162 80]
Err:3 http://security.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client-core-5.7 amd64 5.7.27-0ubuntu0.19.04.1
  Connection failed [IP: 91.189.88.174 80]
Ign:5 http://archive.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client-5.7 amd64 5.7.27-0ubuntu0.19.04.1
Ign:6 http://archive.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client all 5.7.27-0ubuntu0.19.04.1
Err:5 http://security.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client-5.7 amd64 5.7.27-0ubuntu0.19.04.1
  Connection failed [IP: 91.189.88.31 80]
Err:6 http://security.ubuntu.com/ubuntu disco-updates/main amd64 mysql-client all 5.7.27-0ubuntu0.19.04.1
  Connection failed [IP: 91.189.88.174 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/n/numactl/libnuma1_2.0.12-1_amd64.deb  Connection failed [IP: 91.189.88.162 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/liba/libaio/libaio1_0.3.111-1_amd64.deb  Connection failed [IP: 91.189.88.31 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.7/mysql-client-core-5.7_5.7.27-0ubuntu0.19.04.1_amd64.deb  Connection failed [IP: 91.189.88.174 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/m/mysql-defaults/mysql-common_5.8+1.0.5_all.deb  Connection failed [IP: 91.189.88.162 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.7/mysql-client-5.7_5.7.27-0ubuntu0.19.04.1_amd64.deb  Connection failed [IP: 91.189.88.31 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/m/mysql-5.7/mysql-client_5.7.27-0ubuntu0.19.04.1_all.deb  Connection failed [IP: 91.189.88.174 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

```

but despite this issue the telnet command works on those ip & port:

```
root@nginx:~# telnet 91.189.88.174 80
Trying 91.189.88.174...
Connected to 91.189.88.174.
Escape character is '^]'.
^]
telnet> Connection closed.
root@nginx:~# telnet 91.189.88.162 80
Trying 91.189.88.162...
Connected to 91.189.88.162.
Escape character is '^]'.
^]
telnet> Connection closed.

```

Also ping to various servers (e.g. google, ubuntu) works fine.

Why `apt install mysql-client` doesn't work and what should I do in order to make it work?

# Answer
> 2 votes
Ok, it seems that the default docker MTU (1500) is not acceptable for me because I use a PPPoE Internet connection on the host on which I run the docker service:

```
ip addr show eth0
48: eth0@if49: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:12:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.18.0.2/16 brd 172.18.255.255 scope global eth0
       valid_lft forever preferred_lft forever

```

The solution is to create the network with an appropriate MTU:

```
docker network create --opt com.docker.network.driver.mtu=1492 albums-webapp
docker network inspect albums-webapp | grep -i mtu

```

Be aware that while no container is running the command below will still show the 1500 MTU:

```
ip link show | grep 4d725814c237 | grep mtu

```

Be also aware that newly created network must be allowed to go outside - so check it also with the firewall.

Also see the related issue: https://github.com/moby/moby/issues/28314

---
Tags: docker
---


# Question
Title: AWS RDS- Determine what Database Tables aren't interacted with?
Is it possible to have AWS-RDS (or another aws service) generate a report on databases that haven't been interacted with (write/read) after a certain period of time?

My background is in QA and I'm relatively new to the DevOps game, sorry if this is a basic question, couldn't find it in search.

# Answer
> 1 votes
I don't think any database services "internal traffic" is available to AWS services directly.

That being said, you should be able to dump all sql queries to a log file on S3 and also extract all database/table combinations to a text file as well.

It then becomes a text processing exercise that you can possibly solve with AWS Athena

---
Tags: databases, amazon-cloudwatch, amazon-rds
---


title: AWS Cloudwatch for unused accounts

text:
# Question
Title: AWS Cloudwatch for unused accounts
Is there any possibility for getting a report on unused user accounts using cloudwatch ? Requirement : Need to get a report on the accounts which is unused for more than 30 days.

# Answer
> 1 votes
You can get all used User accounts with Cloudtrail, it would then be a text processing exercise to find the unused ones.

---
Tags: amazon-cloudwatch, aws-iam
---


# Question
Title: Is there a Jenkins plugin for downloading source code?
Although *our own* sources are checked out from a revision-control systems, the third-party software we depend on needs to be downloaded from their respective sites.

The plugin I'm looking for would offer a "Source Management" (or "Pre-Build") step, where I could register one (or more) sources for the same file. For example:

and its digest (such as SHA256).

At build- (or at a prebuild-) time, it would check its cache and, if the file is not present there, download and verify it, and make it available to the subsequent build-steps.

I'm sure something like this, or very similar, exists already, but I cannot find it. Sonatype's Nexus seems to offer this, but only for files already found in one's Nexus repository, not for arbitrary URL's.

Any help?

# Answer
> 1 votes
Perhaps you are looking for the Job Cacher plugin?

To quote the plugin page:

> **Summary**
>
> Enables caching of files on transient executors for use on subsequent builds to improve performance
>
> **Features**
>
> * Item storage extension point supporting on master storage and AWS S3
> * Cache Wrapper for free style jobs that manages the cache Arbitrary File Cache where user specifies paths to be cached
> * UI on Job page to review the job's caches
> * Supports Pipeline jobs with a cache directive
> * Cache Extension Point for other plugins to provide opinionated caching capability such as Gradle caches
> * Incremental storing of cache into storage to minimize data movement

However, these are for files **created** by the jobs, not **needed** by them. In order to use it in your case, you might need a few dummy jobs specifically for downloading the content from the arbitrary URLs, and then caching them in your Jenkins cache. I guess you'd have to write the script for that.

To be frank, I've never used this plugin, and what I describe above doesn't seem to be an elegant solution. Your question specifically ask for a Jenkins plugin, however I believe the *general* problem is **how to cache job dependencies** in Jenkins. If you agree to this wider interpretation, there are some other options.

I have often solved the problem you mention using a combination of managed scripts and persistent storage which Jenkins and workers have access to, in Jenkins. If you need really *arbitrary* dependency management, a tool like Spack or EasyBuild may be useful. Although these have evolved in the HPC domain, they do quite well at caching dependencies for workloads.

However, since you mention just a few dependencies (specific sqlite versions *e.g.*), I think it may be worth your time to write a script which does the caching for you and, together with globally-available persistent storage, do the caching and retrieval that way.

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: How to make sure the specified Azure vm size available?
I am trying to create a VM with a playbook like this:

```
  tasks:

  ...

  - name: Create VM
    azure_rm_virtualmachine:
      resource_group: AnsibleResourcEGroup
      name: AnsibleVM
      vm_size: Standard_D4s_v3

      ...

```

When I run the playbook, this "Create VM" task fails:

```
fatal: [localhost]: FAILED! => {
  "changed": false,
  "msg": "Error creating or updating virtual machine AnsibleVM - Azure Error: SkuNotAvailable\nMessage: The requested size for resource '/subscriptions/some-subscription-id/resourceGroups/some-resource-group-name/providers/Microsoft.Compute/virtualMachines/AnsibleVM' is currently not available in location 'region-name' zones '' for subscription 'some-subscription-id'. Please try another size or deploy to a different location or zones. See https://aka.ms/azureskunotavailable for details."
}

```

However, when I use `az` cli to enquery for the available vm size, the size `Standard_D4s_v3` seems available:

```
$ az vm list-skus --location region-id --size Standard_D --output table

ResourceType     Locations    Name                Zones    Capabilities                                                                                Restrictions
---------------  -----------  ------------------  -------  --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------  --------------

...

virtualMachines  region-id    Standard_D2s_v3     1,2,3    ['MaxResourceVolumeMB=16384', 'OSVhdSizeMB=1047552', 'vCPUs=2', 'HyperVGenerations=V1,V2', 'MemoryGB=8', 'MaxDataDiskCount=4', 'LowPriorityCapable=True', 'PremiumIO=True', 'vCPUsAvailable=2', 'ACUs=160', 'vCPUsPerCore=2', 'CombinedTempDiskAndCachedIOPS=4000', 'CombinedTempDiskAndCachedReadBytesPerSecond=33554432', 'CombinedTempDiskAndCachedWriteBytesPerSecond=33554432', 'CachedDiskBytes=53687091200', 'UncachedDiskIOPS=3200', 'UncachedDiskBytesPerSecond=50331648', 'EphemeralOSDiskSupported=True', 'AcceleratedNetworkingEnabled=False', 'RdmaEnabled=False', 'MaxNetworkInterfaces=2']                                           None
virtualMachines  region-id    Standard_D4s_v3     1,2,3    ['MaxResourceVolumeMB=32768', 'OSVhdSizeMB=1047552', 'vCPUs=4', 'HyperVGenerations=V1,V2', 'MemoryGB=16', 'MaxDataDiskCount=8', 'LowPriorityCapable=True', 'PremiumIO=True', 'vCPUsAvailable=4', 'ACUs=160', 'vCPUsPerCore=2', 'CombinedTempDiskAndCachedIOPS=8000', 'CombinedTempDiskAndCachedReadBytesPerSecond=67108864', 'CombinedTempDiskAndCachedWriteBytesPerSecond=67108864', 'CachedDiskBytes=107374182400', 'UncachedDiskIOPS=6400', 'UncachedDiskBytesPerSecond=100663296', 'EphemeralOSDiskSupported=True', 'AcceleratedNetworkingEnabled=True', 'RdmaEnabled=False', 'MaxNetworkInterfaces=2']                                         None
virtualMachines  region-id    Standard_D8s_v3     1,2,3    ['MaxResourceVolumeMB=65536', 'OSVhdSizeMB=1047552', 'vCPUs=8', 'HyperVGenerations=V1,V2', 'MemoryGB=32', 'MaxDataDiskCount=16', 'LowPriorityCapable=True', 'PremiumIO=True', 'vCPUsAvailable=8', 'ACUs=160', 'vCPUsPerCore=2', 'CombinedTempDiskAndCachedIOPS=16000', 'CombinedTempDiskAndCachedReadBytesPerSecond=134217728', 'CombinedTempDiskAndCachedWriteBytesPerSecond=134217728', 'CachedDiskBytes=214748364800', 'UncachedDiskIOPS=12800', 'UncachedDiskBytesPerSecond=201326592', 'EphemeralOSDiskSupported=True', 'AcceleratedNetworkingEnabled=True', 'RdmaEnabled=False', 'MaxNetworkInterfaces=4']                                    None

...

```

which does not seems right.

So which part of my understanding wrong? How should I make sure that a VM size is available in my region?

# Answer
> 2 votes
There are a couple of possibilities based upon my experience:

1. You may not have specified `region-id` in Ansible, which means it will default to the region of the Resource Group. Thus you may have a Resource Group in a region that does in-fact not support `Standard_D4s_v3`.
2. From time to time specific instance types are unavailable in a region due to resource constraints from Azure, this is usually a temporary state, so trying again in a few hours will work.
3. limits or quotas on your subscription might be preventing you from creating the Virtual Machine, although I would expect that you would get a different, more descriptive error message if this were the case.

---
Tags: ansible, azure, vm
---


# Question
Title: How to create slave jenkins given master jenkins?
Below code snippet, taken from here creates master jenkins that runs in a container:

```
# Create Docker Group with GID
# Set default value of 497 if DOCKER_GID set to blank string by Docker Compose
RUN groupadd -g ${DOCKER_GID:-497} docker

# Used to control Docker and Docker Compose versions installed
# NOTE: As of February 2016, AWS Linux ECS only supports Docker 1.9.1
ARG DOCKER_ENGINE=1.10.2
ARG DOCKER_COMPOSE=1.6.2

# ......

# Install Docker Engine
RUN apt-key adv --keyserver hkp://pgp.mit.edu:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D && \
    echo "deb https://apt.dockerproject.org/repo ubuntu-trusty main" | tee /etc/apt/sources.list.d/docker.list && \
    apt-get update -y && \
    apt-get purge lxc-docker* -y && \
    apt-get install docker-engine=${DOCKER_ENGINE:-1.10.2}-0~trusty -y && \
    usermod -aG docker jenkins && \
    usermod -aG users jenkins

```

---

Above container is dind that launches task(each container) on **single** EC2 host.

---

To regulate load across **multiple** EC2 host and launch slave jenkins instance(on each).

We would like to have `master` jenkins generate `build_id` for each build that can run on two slave jenkins instances.

To have master & slave jenkins run within each container, across EC2 hosts.

How to approach on making slave container and take `build_id` from master jenkins?

# Answer
You could use the EC2 plugin or the ECS plugin if you want to provision agents dynamically. These would provide some environment variables to use when constructing `build_id` \- *e.g.* :

* `BUILD_TAG`:
> String of "jenkins-${JOB\_NAME}-${BUILD\_NUMBER}". All forward slashes ("/") in the JOB\_NAME are replaced with dashes ("-"). Convenient to put into a resource file, a jar file, etc for easier identification.
* `NODE_NAME`:
> Name of the agent if the build is on an agent, or "master" if run on master

You would have to configure the clouds in Jenkins, assign a label to the templates you want to use as agents, and then request those in your pipeline or job configuration.
> 2 votes

---
Tags: docker, jenkins, docker-compose, dockerfile
---


# Question
Title: How to specify specific subnet in Terraform when using for each
I am trying to create an ec2 instance that will hold my Jenkins server. I want this to be in a private subnet which I created using a for each loop. Below is my subnet for each loop and my ec2 instance resource. I have also included the error message. Idea is I want it in the first private subnet.

ec2 instance

```
resource "aws_instance" "jenkins" {
  ami           = "${var.ubuntuAMI}"
  instance_type = "t3.micro"
  availability_zone = "us-east-1a"
  key_name = "me"
  monitoring = true
  vpc_security_group_ids = [aws_security_group.ssh_access.id]
  disable_api_termination = true
  subnet_id = "${aws_subnet.private[each.key]}"

  tags = {
    Name = "Jenkins"
  }
}

```

subnet resource

```
resource "aws_subnet" "private" {
  for_each = var.subnet_numbers_private

  vpc_id            = aws_vpc.Main_VPC.id
  availability_zone = each.key
  cidr_block        = cidrsubnet(aws_vpc.Main_VPC.cidr_block, 8, each.value)
  tags = {
      Name = "Private-${each.key}"
  }
}

```

variable used by the subnet loop

```
variable "subnet_numbers_private" {
  description = "Map for private subnets"
  default     = {
    "us-east-1a" = 1
    "us-east-1b" = 2
    "us-east-1c" = 3
  }
}

```

error message seen when doing a `terraform plan`

```
The "each" object can be used only in "resource" blocks, and only when the
"for_each" argument is set.

```

# Answer
> 1 votes
When you want to refer to a specific instance of a resource that has multiple instances due to using `for_each`, you need to include the specific key of the instance you want in your references:

```
  subnet_id = aws_subnet.private["us-east-1a"].id

```

If you just want to select any one subnet from the list, without needing to specify its name directly, you can use an expression like the following to select a single key by sorting the keys lexically and taking the first one:

```
  subnet_id = aws_subnet.private[keys(aws_subnet.private)[0]].id

```

However, beware that if you do the above then any later changes to `var.subnet_numbers_private` that affect which subnet sorts first will cause a different subnet id to be selected, which will then in turn cause Terraform to plan to replace your instance to move it into a new subnet.

---

From the way you wrote the question it sounds like you want just one instance in a single subnet rather than one instance *per* subnet, but if you did want an instance per subnet then you can achieve that by putting `for_each` in the `aws_instance` resource too, using the subnet resource itself (represented in expressions as a map) as the repetition expression:

```
resource "aws_instance" "jenkins" {
  for_each = aws_subnet.private

  # ...
  subnet_id = each.value.id
  # ...
}

```

In this latter case, the `aws_instance.jenkins` instances will also be identified by the keys you selected for the subnets, so you'd have instances with addresses like `aws_instance.jenkins["us-east-1a"]` and `aws_instance.jenkins["us-east-1b"]` so both you and Terraform can see which instance belongs to which subnet.

---
Tags: terraform
---


# Question
Title: Module directory modules/queues.tf does not exist or cannot be read
I am trying to setup terraform with modules and when doing `terraform init` after creating a module I have ran into an issue where I am told it does not exist or can't be read. I've checked many times and tried many paths to it and it does exist where I think it does.

Not sure in that case why it can't be read. I've included my main.tf file as well as the modules/queues.tf file

```
main.tf
  | main.tf
  | modules
     | queues.tf

```

main.tf

```
# This file brings together all of the modules
variable "environments" {
  type = list(string)
}

module "pae_queues" {
    source = "./modules/queues.tf"

    environments = "${var.environments}"
}

```

queues.tf

```
variable "environments" {
  type = "list(string)"
}

resource "aws_sqs_queue" "email_queue_${var.environments[count.index]}" {
  count = length(${var.environments})
  name                      = "email_queue_${var.environments[count.index]}"
  delay_seconds             = 0
  max_message_size          = 2048
  message_retention_seconds = 86400
  receive_wait_time_seconds = 10
  redrive_policy            = "{\"deadLetterTargetArn\":\"${aws_sqs_queue.terraform_queue_deadletter.arn}\",\"maxReceiveCount\":3}"

  tags = {
    Environment = "${var.environments[count.index]}"
  }
}

```

# Answer
A Terraform module is represented by a whole directory, not an individual file. If you create a directory `./modules/queues` and move `./modules/queues.tf` to `./modules/queues/queues.tf` then you can call that module like this:

```
module "pae_queues" {
  source = "./modules/queues"

  environments = var.environments
}

```
> 4 votes

---
Tags: terraform, terraform-module
---


# Question
Title: "Error: forwarding ports: Upgrade request required" Error in helm of a kubernetes cluster
I have a kubernetes cluster built using `kubespray` and imported to Rancher.

The nodes are configured with

* CentOS Linux 7 3.10.0-957.12.1.el7.x86\_64
* Docker version : 18.9.5
* Kubelet version : v1.14.1
* Tiller version : v2.14.1 ( got this version from the tiller pod's image `gcr.io/kubernetes-helm/tiller:v2.14.1` )

All the tiller resources are working fine:

```
$ kubectl get all -n kube-system | findstr tiller

pod/tiller-deploy-57ff77d846-frtb7             1/1       Running   0          12d

service/tiller-deploy          ClusterIP   10.233.49.112   <none>        44134/TCP                16d

deployment.apps/tiller-deploy             1         1         1            1           16d

replicaset.apps/tiller-deploy-57ff77d846             1         1         1         12d
replicaset.apps/tiller-deploy-69d5cd79bb             0         0         0         16d

```

But when I run the `helm` commands, I am getting this error:

```
$ helm version

Client: &version.Version{SemVer:"v2.11.0", GitCommit:"2e55dbe1fdb5fdb96b75ff144a339489417b146b", GitTreeState:"clean"}
Error: forwarding ports: error upgrading connection: Upgrade request required

$ helm ls

Error: forwarding ports: error upgrading connection: Upgrade request required

```

**I tried:**

* The tiller version is 2.14.1. So, upgraded the helm client to version 2.14.1 from 2.11.0. But that doesn't solves the issue.

Can someone help me to solve this error?

# Answer
See my answer here:

> Ran into this today when trying to use Garden.io for a cluster running in Jelastic.
>
> Found the solution in this Github comment:
>
> First acquire a local binary for Tiller (server-version of Helm), either by compiling or by downloading it from the release page.
>
> Then run:
>
> ```
> $ export HELM_HOST=":44134"
> $ tiller -listen ${HELM_HOST} -alsologtostderr >/dev/null 2>&1 &
>
> ```
>
> This will run a local version of the Kubernetes Helm Server. Now try your original command again, kubectl, that will delegate to this local Helm instead and manage to connect.
> 1 votes

---
Tags: kubernetes, helm, rancher, upgrade, centos
---


# Question
Title: How do I bootstrap an alpine image to work with a private repo proxy behind TLS?
I am in a corporate environment behind a gateway that does DPI and often replaces the TLS certificate of most online resources with our corporate certificate. Many of our developers use images based on alpine linux to do their work and at peak times the network gets overloaded, builds crash because of failed `apk add` commands and whatnot. So I set up an instance of Nexus with a caching proxy for alpine apk packages.

However, the Nexus is served behind TLS with a certificate signed by our internal CA which is obviously not in the list of default trusted CAs. Without doing `apk add ca-certificates` I cannot do an `update-ca-certificates` and without trusting our internal CA I cannot do `apk add`. I would like to avoid exposing Nexus via insecure HTTP.

I can do a `wget our.cert.server.com/CA.crt`, but `/etc/ssl/certs` is empty and behavior doesn't change when I create `/etc/ssl/certs/ca-certificate(s).crt` and copy the certificate into there.

How can I (or our developers) bootstrap alpine images for work in our network without too much of a change to our workflow?

# Answer
It turns out that `/etc/ssl/cert.pem` is the bundle scanned by `apk` to verify a repo behind HTTPS, so the following works:

```
wget our.cert.server.com/CA.crt
cat CA.crt >> /etc/ssl/cert.pem
sed -i 's,http://dl-cdn.alpinelinux.org/alpine,https://nexus-endpoint.company.com/repository/alpine-proxy,' /etc/apk/repositories
apk update

```
> 2 votes

---
Tags: ssl, provisioning, repository, alpine
---