# Question
Title: Kubernetes: Custom second scheduler container unable to start
I tried to add a new scheduler to the master node as a static POD. I create the yaml file and save it. The pod as been created and his state is running but the container is unable to start.

the kubectl output:

```
NAME                                   READY   STATUS    RESTARTS   AGE
my-scheduler                           0/1     Running   5          28m

```

I tried to see reason in the events and saw the reason `cannot join network of a non running container` and the last event with error:

```
Startup probe failed: Get "https://127.0.0.1:10251/healthz": http: server gave HTTP response to HTTPS client

```

I tried to send GET request to http URL and got `OK` answer but https answer was a error:

```
curl: (35) error:1408F10B:SSL routines:ssl3_get_record:wrong version number

```

also I tried to get the scheduler log and got:

```
    I1016 18:47:15.779350       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.779753       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.791161       1 registry.go:173] Registering SelectorSpread plugin
    I1016 18:47:15.791191       1 registry.go:173] Registering SelectorSpread plugin
    W1016 18:47:15.794300       1 authorization.go:47] Authorization is disabled
    W1016 18:47:15.794518       1 authentication.go:40] Authentication is disabled
    I1016 18:47:15.794695       1 deprecated_insecure_serving.go:51] Serving healthz insecurely on [::]:10251

```

scheduler file:

```
apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: null
  labels:
    component: my-scheduler
    tier: control-plane
  name: my-scheduler
  namespace: kube-system
spec:
  containers:
  - command:
    - kube-scheduler
    - --authentication-kubeconfig=/etc/kubernetes/scheduler.conf
    - --authorization-kubeconfig=/etc/kubernetes/scheduler.conf
    - --bind-address=127.0.0.1
    - --kubeconfig=/etc/kubernetes/scheduler.conf
    - --leader-elect=false
    - --scheduler-name=my-scheduler
    - --port=10251
    - --secure-port=0
    image: k8s.gcr.io/kube-scheduler:v1.19.0
    imagePullPolicy: IfNotPresent
    livenessProbe:
      failureThreshold: 8
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    name: my-scheduler
    resources:
      requests:
        cpu: 100m
    startupProbe:
      failureThreshold: 24
      httpGet:
        host: 127.0.0.1
        path: /healthz
        port: 10251
        scheme: HTTPS
      initialDelaySeconds: 10
      periodSeconds: 10
      timeoutSeconds: 15
    volumeMounts:
    - mountPath: /etc/kubernetes/scheduler.conf
      name: kubeconfig
      readOnly: true
  hostNetwork: true
  priorityClassName: system-node-critical
  volumes:
  - hostPath:
      path: /etc/kubernetes/scheduler.conf
      type: FileOrCreate
    name: kubeconfig
status: {}

```

what am I doing wrong? What is the right way to config the scheduler?

# Answer
> 2 votes
In your manifest, you should set the `spec.containers[0].livenessProbe.httpGet.scheme` to `HTTP` not `HTTPS`.

This is because you use the option `--port=10251`. As the documentation said, you should not use the deprecated `--port` option anymore, because it expose the API insecurely.

You should set the following options: `--port=0 --secure-port=10251` and then you would be able to set the `spec.containers[0].livenessProbe.httpGet.scheme` to `HTTPS`.

---
Tags: kubernetes
---


# Question
Title: '[[0;1;31mFAILED[0m] Failed to start' when running cloud-init?
I am trying to contextualize a docker container using `cloud-init`.
Why do I see `[[0;1;31mFAILED[0m] Failed to start` in the OpenStack log?
What is a likely cause of this error?

# Answer
> 0 votes
From my rather limited experience, a likely reason is that the command `sudo docker image build` *does not find the docker file*.
(But a virtual machine may have been successfully created.)

---
Tags: docker, cloud-init, openstack
---


# Question
Title: Ansible - merging inventories to share variables across?
Is it possible to merge inventories so I can share variables across them? Let's say I have two inventory files. Inventory A and Inventory B. They are both in a directory called inventories.

Variables are defined in B and I want to reference them in inventory A. Is this possible? I've heard of merging inventories so share variables across but I haven't gotten this to work.

Here's are the files in my inventories directory: /opt/ansible/inventories/aws\_ec2.yaml /opt/ansible/inventories/secrets.yaml

contents of `aws_ec2.yaml`:

```
---
plugin: aws_ec2

aws_access_key: "{{ aws_access_key }}"
aws_secret_key: "{{aws_secret_key}} "
aws_security_token: "{{aws_security_token}}"

regions:
  - us-west-2

keyed_groups:
  - key: tags
    prefix: tag
  - key: placement.region
    prefix: aws_region

groups:
  TEST: "'prod' in (tags:list)"

compose:
  ansible_host: private_ip_address

```

Here is my `secrets.yaml` file:

```
---

aws_access_key: xxxxx
aws_secret_key: xxxxx
aws_security_token: xxxxx

```

When I run my playbook, referencing both inventories, inventory does not seem to occur and I get these errors. It looks like the variables aren't being referenced and expanded?:

```
me@workstation:/opt/ansible/inventories$ ansible-playbook -i /opt/ansible/inventories/aws_ec2.yaml -i /opt/ansible/inventories/secrets.yaml /opt/ansible/playbooks/test.yaml
[WARNING]:  * Failed to parse /opt/ansible/inventories/aws_ec2.yaml with aws_ec2 plugin: Failed to
describe instances: An error occurred (AuthFailure) when calling the DescribeInstances operation:
Credential must have exactly 5 slash-delimited elements, e.g. keyid/date/region/service/term, got '{{'
[WARNING]: Unable to parse /opt/ansible/inventories/aws_ec2.yaml as an inventory source
[WARNING]: Unable to parse /opt/ansible/inventories/secrets.yaml as an inventory source
[WARNING]: No inventory was parsed, only implicit localhost is available
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost
does not match 'all'

```

Or is my syntax for variable assigning and referencing wrong? Am I on the right path here? my `/etc/ansible/ansible.cfg` file, under `[defaults]` is as such:

```
[defaults]
inventory = /opt/ansible/inventory/aws_ec2.yaml

```

# Answer
> 1 votes
While I haven't worked with `plugin: aws_ec2`, here's some general Ansible information that might help you:

The order of inventory files matter; the files given first will be read first; the ones read last will take precedence. In your execution, you're referencing `secrets.yaml` last, so when `aws_ec2.yaml` is read, the secrets are still undefined.

You're trying to define a variable based on another variable with the same name. That's asking for trouble; rename the reference variables, as that will help you debugging.

When defining variables for the inventory, it's generally better to not do that directly on the inventory itself. See the link below on how to do that:

https://docs.ansible.com/ansible/latest/user\_guide/intro\_inventory.html#organizing-host-and-group-variables

Finally, `keyed_groups`, `regions` and all these other keywords are specific to `aws_ec2`, and they'd work because you have `plugin: aws_ec2` at the top of the first file. You have no such definition on `secrets.yaml`, so Ansible would probably treat it as a normal inventory file, where that structure would not be accepted. Your approach here is probably wrong, then.

Again, I never used that inventory plugin. It may be the case that the inventory plugin does not allow for variables at that point.

# Answer
> 0 votes
Notice that what you're trying to do is not merging inventories. Merging inventories happens when you have two fully defined inventories and you try to, well, merge them. So, you have an inventory with a set of defined hosts, groups and variables and you combine that with another inventory with its own hosts, groups and variables.

In your example, what you're trying to do is to split the definition of a single inventory. The `aws_ec2` plugin requires some information so it goes to Amazon and say, 'hey take this info and give me back my inventory'. You want part of that information in one file, part in another one.

Once you're using a plugin, the inventory's parsing will be controlled by the plugin, so I don't think there are any guarantees that stuff that works elsewhere (like variables) will work here.

You can test that; run `ansible-playbook` without secrets.yaml and set each of the variables as extra vars using `-e`:

```
ansible-playbook \
 -i /opt/ansible/inventories/aws_ec2.yaml  \
 -e aws_access_key=xxx \
 -e aws_secret_key=xxx \
 -e aws_security_token=xxx \
 /opt/ansible/playbooks/test.yaml

```

If that works, then you have validated that you can use Ansible variables in that place, and you just need to find a proper place to put them. I'm not sure where exactly to put that, though.

If you don't mind to put that data elsewhere, the plugin accepts those values you're trying to separate as environment variables:

```
ansible-doc -t inventory aws_ec2

- aws_access_key
    The AWS access key to use.
    (Aliases: aws_access_key_id)[Default: (null)]
    set_via:
      env:
      - name: EC2_ACCESS_KEY
      - name: AWS_ACCESS_KEY
      - name: AWS_ACCESS_KEY_ID

    type: str

```

---
Tags: ansible, amazon-ec2, configuration-management, ansible-inventory
---


title: Playbook without relative path to vars

text:
# Question
Title: Playbook without relative path to vars
I had already bunch of playbooks, so I started to organize them in *playbooks* dir. It cause there must be relative paths to vars files.

**Actual playbook:**

```
---
- hosts:  aws_instance.jenkins-agents
  user:   ec2-user
  vars_files:
    - ../../../vars/main.yaml
    - ../../../vars/vault.yaml
  ...

```

Is there any way to keep the dir structure, but avoid the lochness family (`../../../`)?

I know for roles You can specify path to them, but I did not find anything similar for *vars\_files*

**Project conditions:**

1. We need to run the playbooks from the root of the project
2. Both *vars* and *playbooks* dir are in root of the project
3. This project doesn't have static inventory/hosts, but it is generated on the fly when You run some playbook

# Answer
You'll notice that the official recommendations for directory layout put all of the playbooks at the root level. This is intentional, as Ansible doesn't handle other schemes well.

You *can* put playbooks in a subdirectory, as you've started to do, but that will require (as you've already found) the use of relative paths from the playbooks to any other resources they use; Ansible uses the playbook path to start its search, that's just how it's written. For the most part, that's just what you get to deal with when you make the choice to move where playbooks live.

However, you seem to have more nesting than you should:

```
---
- hosts:  aws_instance.jenkins-agents
  user:   ec2-user
  vars_files:
    - ../../../vars/main.yaml
    - ../../../vars/vault.yaml
  ...

```

Why are you navigating up *three* directories? I'd expect a layout that's like the official one, but with one subfolder for playbooks; then you'd end up with just

```
---
- hosts:  aws_instance.jenkins-agents
  user:   ec2-user
  vars_files:
    - ../vars/main.yaml
    - ../vars/vault.yaml
  ...

```

which is a much more minor change.

---

Secondly, it's pretty rare that you actually want to use `vars_files`. Most variables used in Ansible either vary based on the host (and thus should go in `group_vars`/`host_vars` in the project root) or are role-specific (and thus should go in `vars`/`defaults` in the role directory). A few documentation links:

Personally, I think it's best to have as little as possible in your playbooks, but delegate everything out to roles. Here's an example entire playbook in your setup:

```
- hosts: aws_instance.jenkins-agents
  roles:
    - ../roles/jenkins_agent

```

This gives you far more flexibility for re-use.

And if you find yourself not wanting to use relative paths for roles, you can override `roles_path` in an `ansible.cfg` in the root directory where you run your Ansible commands from.
> 7 votes

# Answer
Since I can't comment on 030's answer yet (just opened an account), I'll answer here.

Group vars directories are relative to the Ansible path.

You don't have to run Ansible from /etc/ansible. You can run it from ~/.hidden/directory/stuff if you wish. Just make sure to keep a directory structure. I.e.:

```
/pipeline/   # ansible playbooks are here
/pipeline/roles
/pipeline/group_vars

```

Then you can run something like

```
cd /pipeline
ansible-playbook deploy_app.yml

```

Which will read group vars from the pipeline/group\_vars folder. Better to cd to your ansible root folder first since it looks for paths from the current folder you're in if it can't find them in default location (/etc/ansible), such that you can put your ansible.cfg in /pipeline without having to specify other command line arguments. You can also optionally define specific paths inside ansible.cfg as well.

You can take it a step further and set up separate group vars for each environment/location/etc:

```
/pipeline/prod/group_vars/all.yml
/pipeline/dev/group_vars/all.yml

```

And run as follows using include-vars.

ansible-playbook deploy\_app.yml -i prod
> 2 votes

# Answer
To avoid such issues, that is why there are Hosts and Groups in Ansible.

> ```
> /etc/ansible/group_vars/raleigh # can optionally end in '.yml', '.yaml', or '.json'
> /etc/ansible/group_vars/webservers
> /etc/ansible/host_vars/foosball
>
> ```
> 1 votes

# Answer
Ansible usually takes relative path according to `inventory_dir` & `playbook_dir`:

```
- debug: var=inventory_dir
- debug: var=playbook_dir

```
> 1 votes

---
Tags: ansible
---


# Question
Title: Create dynamic resources using 'for_each' or 'for'
I want to create multiple resources using this:

```
resource "group" "list" {
 g_name      = group_name
 p_name      = project_name
}

```

The input local variable is nested:

```
> local.input
{
  "group" = [
    {
      "name" = "group1"
      "projects" = [
        "project1",
        "project3",
      ]
    },
    {
      "name" = "group2"
      "projects" = [
        "project18",
        "project31",
      ]
    },
  ]

```

I am able to transform it into a better form:

```
> flatten([for grp_key, group in local.input.group: [for project in group.projects: { group_name=group.name, project_name=project} ]])
[
  {
    "group_name" = "group1"
    "project_name" = "project1"
  },
  {
    "group_name" = "group1"
    "project_name" = "project3"
  },
  {
    "group_name" = "group2"
    "project_name" = "project18"
  },
  {
    "group_name" = "group2"
    "project_name" = "project31"
  },
]

```

I can not use `for_each` because it is not `map` neither `set`.

How could I create multiple resources using the `for` like:

```
resource "group" "list" {
  for item in ... :
    g_name=item.group_name
    p_name=item.project_name
}

```

# Answer
> 0 votes
you need to use `count`, and with it you should be able to access item by index: https://www.terraform.io/docs/configuration/resources.html#the-count-object

---
Tags: terraform
---


# Question
Title: How to use git repositories in Azure Devops with Visual Studio
I am working on Visual Studio projects using Visual Studio Community. I have created local git repositories and remote repositories on https://dev.azure.com.

I want to work with Team Foundation Version Control. I need "Microsoft Team Foundation Server". It seems this product is now called "Azure DevOps Server". Is it the same thing as https://dev.azure.com?

What should I do in order to work with Team Foundation Version Control instead of git?

# Answer
> 2 votes
Q: "*What should I do in order to work with Team Foundation Version Control?*"

A: TL'DR: Use the TFS command line client, `tf`.

? But why use TFVC ?

Azure DevOps is confusingly several products. There is the cloud offering: DevOps Services, formerly known as Visual Studio Team Services, formerly, Visual Studio Online, formerly Team Foundation Service. There is also an On-Premise offering: Azure DevOps Server, formerly known as Team Foundation Server (now 2020, formerly 2019, 2017, 2015, 2013, 2012, 2010, 2018, 2005). They do share a common REST API, albeit the on-prem lags behind.

Aside from the delay in getting features into the on-prem variant, the questions of choosing on-prem or cloud have nothing to do with choosing TFVC vs Git. It likely depends more on cost, infrastructure and people resources, geographic and network constraints.

You can see most of the Product Announcements from the Program Manager here.

From TFS 2005 to 2012, the version control engine was TFVC, a centralized version control tool. Team Foundation Server 2013 added native support as a centralized Git remote host (similar to Github/Gitlab/BitBucket,but on-prem), albeit in separate team projects. TFS 2015 Update 1 (clumsily) allowed hosting both in the same team project.

But, let's get back to today. Since TFS 2015, Microsoft has put very little effort into improving the TFVC side of their offering. It would seem today, they have essentially abandoned it. Team Explorer Everywhere is no longer maintained. In 2016, they open-sourced it. Admittedly, in 2017 they did reintroduce it after a large outcry. The Eclipse client was last updated for Neon release (2016-06). The tfs-plugin for Jenkins has not had a significant release since Mar 2018. On 2020-11-06 (Nov 6, 2020), the Visual Studio Code plugin will be removed from the VS Code Marketplace; users will have to rely on the `tf` command line.

The Program Manager's first post detailed TFS Usage within Microsoft. Those were last updated in 2013. A few years later, they are now boasting of having the world's largest Git repo and then acquired GitHub.

The basic question you should be asking is Choosing the right version control for your project? Do you want a Centralized model or a Distributed model of version control?

I have a lot (decades) of experience managing Centralized and Distributed version control systems. As MS documents, each have their strengths, weaknesses and trade-offs. But, **why would you choose a centralized version control tool that is provided by a vendor that evidently no longer supports it?**

---

PS: Most people visualize Centralized Version Control like this:  (from: https://scmquest.com/wp-content/uploads/2014/05/central\_vcs.jpg)

and Distributed Version Control like this:  (from: https://scmquest.com/wp-content/uploads/2014/05/distributed\_vcs.jpg)

But that is really just a "disconnected, centralized model". Git allows for very complex distributed workflows, such as show below. You do not get that flexibility using centralized tools. But it can also be a total nightmare to keep track of what's where.

(from: https://howtodoinjava.com/vcs/how-distributed-version-control-system-works/)

# Answer
> 0 votes
First, Azure DevOps is the next release of Team Foundation Server 2018 (on premise) and Visual Studio Team System (on cloud). Therefore, Azure DevOps has two flavors, Azure DevOps Service (on cloud) and Azure DevOps Server (on premise). The latest release of Azure DevOps Server is Azure DevOps Server 2020 that released in this October.

Second, you should not mix TFVC/Git with TFS. TFVC is a centralized source control, was released during the initial release of Team System 2005 (older name of Team Foundation Server). TFVC is also copyright and trade mark of Microsoft. There are other samples of centralized source control such as SVN, CVS, Rational Clear Case.

Git is a distributed source control. Git is open source and it's not owned by any company.

If you want to continue work using TFVC, you can still use TFVC within Azure DevOps Service or Azure DevOps Server. Both of these Azure DevOps products still support TFVC.

For more information, please visit this official docs: https://docs.microsoft.com/en-us/azure/devops/user-guide/source-control?view=azure-devops

---
Tags: azure-devops, tfs
---


# Question
Title: Gitlab CI error during connect: Get http://docker:2375/v1.40/containers/json?all=1: dial tcp: lookup docker on xx.xx.xx.x:xx: no such host
I've recently created a new repository in gitlab.com, started a new Ubuntu instance in digitalocean, installed docker and gitlab-runner in the ubuntu instance. Also, did gitlab-runner register and passed the correct host and token from gitlab.com ci/cd settings. Pushed a branch, and the expected pipeline job runs but fails, presenting the error:

```
error during connect: Get http://docker:2375/v1.40/containers/json?all=1: dial tcp: lookup docker on 67.207.67.2:53: no such host

```

Just followed the basic steps and expected it to work perfectly. By looking for the error when google search, there's no info.

The config.toml file, which is created automatically on gitlab-runner register:

```
concurrent = 1
check_interval = 0

[[runners]]
  name = "digital ocean runner"
  url = "https://gitlab.com/"
  token = "xxxxxxxxxxxxxxxxx"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "ruby:2.1"
    privileged = true
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]

```

The gitlab ci file:

```
image: docker:latest
services:
  - docker:dind
stages:
  - test

test-build:
  stage: test
  script:
    - echo "Fine!"
    - docker info
  tags:
    - docker

```

Steps to reproduce

```
open a gitlab.com account
Install Ubuntu latest
Install gitlab-runner
do `register` and should have the config.toml above

```

What causes this issue?

The error log for one one of the tests, not for the job above:

```
Running with gitlab-runner 12.3.0 (a8a019e0)
  on foobar laptop szWcjfZg
Using Docker executor with image foobar/docker-bash ...
Starting service docker:dind ...
Authenticating with credentials from /Users/foobar/.docker/config.json
Pulling docker image docker:dind ...
Using docker image sha256:5768e15eefd175c1ba6969b616cfe827152556c5fe691b9258cb57d1a5c37e9d for docker:dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-szWcjfZg-project-14670943-concurrent-0-docker-0 probably didn't start properly.

Health check error:
service "runner-szWcjfZg-project-14670943-concurrent-0-docker-0-wait-for-service" timeout

Health check container logs:

Service container logs:
2019-10-05T23:18:52.128774700Z Generating RSA private key, 4196 bit long modulus (2 primes)
2019-10-05T23:18:53.209639200Z .......................................................................................................................................................................................++++
2019-10-05T23:18:53.694383300Z .......................................................................................++++
2019-10-05T23:18:53.694784400Z e is 65537 (0x010001)
2019-10-05T23:18:53.710661300Z Generating RSA private key, 4196 bit long modulus (2 primes)
2019-10-05T23:18:53.789938500Z ............++++
2019-10-05T23:18:54.926850200Z ..............................................................................................................................................................................................................++++
2019-10-05T23:18:54.927064600Z e is 65537 (0x010001)
2019-10-05T23:18:54.953296700Z Signature ok
2019-10-05T23:18:54.953354500Z subject=CN = docker:dind server
2019-10-05T23:18:54.953422500Z Getting CA Private Key
2019-10-05T23:18:54.967240700Z /certs/server/cert.pem: OK
2019-10-05T23:18:54.970126300Z Generating RSA private key, 4196 bit long modulus (2 primes)
2019-10-05T23:18:55.244959900Z .................................................++++
2019-10-05T23:18:55.317443900Z ...........++++
2019-10-05T23:18:55.317858100Z e is 65537 (0x010001)
2019-10-05T23:18:55.339564700Z Signature ok
2019-10-05T23:18:55.339581700Z subject=CN = docker:dind client
2019-10-05T23:18:55.339671300Z Getting CA Private Key
2019-10-05T23:18:55.356469400Z /certs/client/cert.pem: OK
2019-10-05T23:18:55.359632000Z mount: permission denied (are you root?)
2019-10-05T23:18:55.359866500Z Could not mount /sys/kernel/security.
2019-10-05T23:18:55.359910700Z AppArmor detection and --privileged mode might break.
2019-10-05T23:18:55.361037300Z mount: permission denied (are you root?)

*********

Authenticating with credentials from /Users/foobar/.docker/config.json
Pulling docker image foobar/docker-bash ...
Using docker image sha256:2ecf555077c9d0a7df0cc45188094a84fbbae4ce4c28cac37a652e9765bf3157 for foobar/docker-bash ...
Authenticating with credentials from /Users/foobar/.docker/config.json
Running on runner-szWcjfZg-project-14670943-concurrent-0 via Foobar-MacBook-Pr.local...
Authenticating with credentials from /Users/foobar/.docker/config.json
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/foobar/foobar-project/.git/
Checking out 4f9789a7 as review-ci-test...

Skipping Git submodules setup
Authenticating with credentials from /Users/foobar/.docker/config.json
Authenticating with credentials from /Users/foobar/.docker/config.json
Authenticating with credentials from /Users/foobar/.docker/config.json
$ echo "INFO - Clear exited containers" # collapsed multi-line command
INFO - Clear exited containers
INFO - List all services running...
error during connect: Get http://docker:2375/v1.40/containers/json?all=1: dial tcp: lookup docker on 192.168.65.1:53: no such host
Authenticating with credentials from /Users/foobar/.docker/config.json
ERROR: Job failed: exit code 1

```

# Answer
> 9 votes
I might have skipped the optional installation of docker in my Ubuntu instance, so if the gitlab runner is going to use Docker, remember to install it:

```
curl -sSL https://get.docker.com/ | sh

```

I got back to this issue and seem to have partially fixed it by checking it the gitlab runner is running

```
gitlab-runner status

systemctl is-enabled gitlab-runner

systemctl is-enabled docker

```

I do have a different issue to solve, but it doesn't seem to be related. That is:

```
error during connect: Post http://docker:2375/v1.40/images/ [...] dial tcp: lookup docker on xx.xxx.xx.x:xx: no such host

```

Then I found the solution, there's a problem in Gitlab CI that is reported here ( https://gitlab.com/gitlab-org/gitlab-runner/issues/4566#note\_199261985 ); And there are a lot of ways to fix this but my solution follows:

1) Set the `.gitlab-ci.yml` services to use an older dind version:

```
services:
  - docker:18.09.7-dind

```

2) Check the toml file `/etc/gitlab-runner/config.toml`, set the DOCKER\_HOST just in case, to point to the right place, make sure it runs in `privileged` mode. My working version is:

```
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "xxxxxx xxxxxxxx"
  url = "https://gitlab.com/"
  token = "xxxxxxxxxx"
  executor = "docker"
  pre_build_script = "export DOCKER_HOST=tcp://docker:2375"
  [runners.custom_build_dir]
  [runners.docker]
    tls_cert_path = ""
    tls_verify = false
    image = "alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

```

Hope this helps someone else in the future!

# Answer
> 2 votes
I want to share the solution!

just add `image = docker:stable`  and `privileged = true`

and that's all! enjoy!

---
Tags: docker, gitlab, gitlab-ci-runner, docker-in-docker
---


# Question
Title: Docker compose scale, what is it good for?
I have a docker-compose with 4 containers deployed in production.
I wanted to use docker `scale` for the nodejs part. (with nginx LB)
If the docker-compose `scale` is done on the host machine, how does it suppose to improve performance, I did not limit the resources for that container, it means that it can burst as much as the host allows it.

What docker-compose scale is good for anyway?

Thanks

# Answer
> 2 votes
Scaling is increasing the number of containers for any specific service. This is primarily helpful for a cluster, **not** on a single host setup like yours. When you scale your application up in a cluster, multiple containers of the same service get spread across the cluster (Based on their availability (Resource or anyway)). This makes the distribution of load across containers an available benefit. In a single host environment, with limited resources, scaling won't do you much good as the processes are going to be the slave of the same schedular, so you won't have that parallel servicing.

So now your question, why does scale in docker-compose exist? From the top of my head in no specific order of value:

* You can minimize your services' downtime (going as low as none) while upgrading with docker-compose's scale functionality.
* If an app can handle only so many inputs, in other words, if there's a queue of inputs, you can essentially increase the available queue space by scaling the container up and have multiple containers handle a portion of the inputs, reducing input loss.
* The more obvious one, if one container goes down, there will still be at least one replica left.

---
Tags: docker-compose
---


# Question
Title: Jenkins multi-agent build switching directories behind the scenes preventing AWS CodeDeploy plugin from zipping built files
I've narrowed my problem down to a very specific cause but am not sure how exactly how to fix it.

A Jenkins server set up using Jenkins-configuration-as-code in a docker container runs a pipelineJob script that sets up a listener to a github repo and whenever there's a new commit pulls a jenkinsfile from the repo and uses that to run a job using the declarative pipeline plugin. At the end of the job there's a post-build action using the AWS CodeDeploy plugin to push the built app out to an autoscaling group.

Previously there were two separate jobs for the frontend and backend of the app that deployed each piece separately and everything was fine. The whole system worked end-to-end without any human interaction required.

Now though, after merging the two repos and their declarative pipelines into just one using multiple docker agents, the job still builds and if I check the Jenkins server it says it was successful but if I check an ec2 instance CodeDeploy pushed the update to all I find is the original source files, no built files at all.

After a bunch of digging and head banging I've found that under the hood Jenkins is running git clone and the codedeploy plugin in /var/jenkins\_home/workspace/FOLDER but building the backend using a golang docker agent and the frontend using a node docker agent in /var/jenkins\_home/workspace/FOLDER@2. That means when CodeDeploy zips it all up it's missing everything that was actually built.

My question is how do I stop/fix that?

Pipeline Jenkinsfile:

```
pipeline {
    agent any

    stages {
        stage('Clone git') {
            steps { git 'https://github.com/REPO' }
        }

        stage('Backend') {
            agent { docker {
                image 'golang:latest'
                args '-v "$PWD"/backend:/test -w /test'
            } }

            environment { GOCACHE = "${WORKSPACE}" }

            steps {
                dir('backend') {
                    sh 'make install'
                    sh 'make build'
                }
            }
        }

        stage('Frontend') {
            agent { docker { image 'node:latest' } }

            steps {
                dir('frontend') {
                    sh 'make install'
                    sh 'make build'
                }
            }
        }

    }

    post ('Deploy') {
        success {
            step([
                $class: 'AWSCodeDeployPublisher',
                applicationName: 'server_app',
                awsAccessKey: '',
                awsSecretKey: '',
                deploymentGroupAppspec: false,
                deploymentGroupName: 'server_group',
                deploymentMethod: 'deploy',
                excludes: '',
                iamRoleArn: '',
                includes: '**',
                proxyHost: '',
                proxyPort: 0,
                region: 'us-west-2',
                s3bucket: 'server-bucket',
                s3prefix: '',
                subdirectory: '',
                versionFileName: '',
                waitForCompletion: false
            ])
        }
    }
}

```

Job console output (note where it switches from FOLDER to \`FOLDER@2):

```
Checking out git https://github.com/REPO into /var/jenkins_home/workspace/FOLDER@script to read pipeline.jenkinsfile
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
 > git rev-list --no-walk 263260386f964b2d7bc8e9921f3b3bb7214ea09a # timeout=10
[Checks API] No suitable checks publisher found.
Running in Durability level: MAX_SURVIVABILITY
[Pipeline] Start of Pipeline
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/FOLDER
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
[Checks API] No suitable checks publisher found.
[Pipeline] }
[Pipeline] // stage
[Pipeline] withEnv
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Clone git)
[Pipeline] git
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git rev-parse refs/remotes/origin/master^{commit} # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (refs/remotes/origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
 > git branch -a -v --no-abbrev # timeout=10
 > git branch -D master # timeout=10
 > git checkout -b master 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
[Checks API] No suitable checks publisher found.
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Backend)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/FOLDER@2
[Pipeline] {
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO+refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
[Checks API] No suitable checks publisher found.
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . golang:latest
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 3ff4f947dfec2937398c540976187282866705cfe05a4a98255e57566e43f6a9
$ docker run -t -d -u 1000:1000 -v $PWD/backend:/test -w /test -w /var/jenkins_home/workspace/FOLDER@2 --volumes-from 3ff4f947dfec2937398c540976187282866705cfe05a4a98255e57566e43f6a9 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** golang:latest cat
$ docker top 69a4b1b98f0a3824bbcb785d9464f5450f4432a2458399e328214981f46e992d -eo pid,comm
[Pipeline] {
[Pipeline] withEnv
[Pipeline] {
[Pipeline] dir
Running in /var/jenkins_home/workspace/FOLDER@2/backend
[Pipeline] {
[Pipeline] sh
+ make install
go get "github.com/julienschmidt/httprouter"
[Pipeline] sh
+ make build
go build -o ../server main.go
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
$ docker stop --time=1 69a4b1b98f0a3824bbcb785d9464f5450f4432a2458399e328214981f46e992d
$ docker rm -f 69a4b1b98f0a3824bbcb785d9464f5450f4432a2458399e328214981f46e992d
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Frontend)
[Pipeline] node
Running on Jenkins in /var/jenkins_home/workspace/FOLDER@2
[Pipeline] {
[Pipeline] checkout
The recommended git tool is: NONE
No credentials specified
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/REPO # timeout=10
Fetching upstream changes from https://github.com/REPO
 > git --version # timeout=10
 > git --version # 'git version 2.11.0'
 > git fetch --tags --progress -- https://github.com/REPO +refs/heads/*:refs/remotes/origin/* # timeout=10
Seen branch in repository origin/master
Seen 1 remote branch
 > git show-ref --tags -d # timeout=10
Checking out Revision 385e449faf2f2316e308d9cc2de07725cfaf3582 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 385e449faf2f2316e308d9cc2de07725cfaf3582 # timeout=10
Commit message: "simplify jenkinsfile"
[Checks API] No suitable checks publisher found.
[Pipeline] withEnv
[Pipeline] {
[Pipeline] isUnix
[Pipeline] sh
+ docker inspect -f . node:latest
.
[Pipeline] withDockerContainer
Jenkins seems to be running inside container 3ff4f947dfec2937398c540976187282866705cfe05a4a98255e57566e43f6a9
$ docker run -t -d -u 1000:1000 -w /var/jenkins_home/workspace/FOLDER@2 --volumes-from 3ff4f947dfec2937398c540976187282866705cfe05a4a98255e57566e43f6a9 -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** -e ******** node:latest cat
$ docker top ce7b3a6c0e4a3a511bcf52e10262618b8c095060bd42f155977f31374cb7dee7 -eo pid,comm
[Pipeline] {
[Pipeline] dir
Running in /var/jenkins_home/workspace/FOLDER@2/frontend
[Pipeline] {
[Pipeline] sh
+ make install
yarn install
yarn install v1.22.5
warning package.json: No license field
warning new@1.0.0: No license field
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.84s.
[Pipeline] sh
+ make build
make: 'build' is up to date.
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
$ docker stop --time=1 ce7b3a6c0e4a3a511bcf52e10262618b8c095060bd42f155977f31374cb7dee7
$ docker rm -f ce7b3a6c0e4a3a511bcf52e10262618b8c095060bd42f155977f31374cb7dee7
[Pipeline] // withDockerContainer
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] stage
[Pipeline] { (Declarative: Post Actions)
[Pipeline] step
Zipping files into /tmp/#2-8971329807001316273.zip
// deleting some aws account info that displays
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[Checks API] No suitable checks publisher found.
Finished: SUCCESS

```

# Answer
> 1 votes
Got it! `reuseNode` needs to be set true for each docker agent:

```
       agent { docker {
            image 'golang:latest'
            reuseNode true
        } }

```

Wish that didn't take a couple hours to learn but que sera sera

---
Tags: docker, jenkins, jenkins-plugins, aws-codedeploy
---


# Question
Title: How to pick a right hardware configuration setup(server) for MERN stack
I'm learning programming(MERN stack) and along with that basic DevOps(linux, containers etc). I've read a lot about software configuration setups(with NGINX, and without, etc), but can't find anywhere regards how well the MERN stack will behave on different server hardware configuration setups.

I will probably host my app on Linode, for various reasons(price, anti-ddos etc), but I guess it will applicable for alike services too(digital ocean etc).

So, is there a way to approximately know before deploying, how much my app will need resources? Are there any assumptions I can make regards MongoDB's usage(of RAM, for example?), for example?

Any info will be valuable for me, thank you.

# Answer
> 1 votes
Since this is for your learning, it shouldn't tax your hardware incredibly. If you are hosting I would with a free student account if you can. But if you have a linux laptop all of this should work well on it. You're not writing apps that are getting 10,000's of connections and transforming complex data. If you had an enterprise app I would indeed split off the DB from the frontend etc.

---
Tags: configuration, serverspec
---


# Question
Title: What part of the Bamboo pipeline should generate a Docker image?
We are using Bamboo to build and deploy a number of Java web applications. Right now, the Build Plans generally produce a WAR file, which is deployed by the Deploy Plans. We are moving our hosting (and building) to docker, and I'm unclear on the best way to handle this in Bamboo. Specifically, I'm unsure of whether the Build Plan or Deploy Plan should be responsible for generating a docker image.

### Option 1

* The Bamboo build plan does a multi-stage Docker build to tag and publish a docker image (~ 300MB)
* The Bamboo deploy plan is responsible for spinning up a container for that image.

### Option 2

* The Bamboo build plan does a Docker build, and attaches the artifact (WAR file ~ 80 MB)
* The Bamboo deploy plan is responsible for creating an image, inserting the WAR file, and spinning up a container

Just wondering if there is a standard or best practice pattern here. Or am I thinking about this wrong? Are there other options I should be considering?

# Answer
I believe both options are viable routes, but my preference is for **Option 1** as I like having the "deploy plan" to be only concerned with starting a container using the built image.
> 1 votes

---
Tags: docker, deployment, build-pipeline, bamboo
---


# Question
Title: Docker update with persistent storage and 0 downtime
I have a docker container running with persistent storage. When I try to update now I kill the container, and rerun the updated image all over again.

I've considered setting up the new container first, then switching traffic over via NginX, but how would this work with persistent storage? I can't bind the same data to two different containers can I?

Most questions and googling I've done suggest moving over to kubernetes, but for a single-host, 3 container setup, it doesn't seem worth it for such a small application.

# Answer
> 0 votes
Docker itself does not provide such rolling update feature. You have to do it by hand which is not an easy task. Even if you want to use a one node Swarm cluster, the scheduler does have downtime.

```
The scheduler applies rolling updates as follows by default:

Stop the first task.
Schedule update for the stopped task.
Start the container for the updated task.

```

If you only have one machine, you can try something like k3s, which is a lightweight Kubernetes distribution. You'll be able to use a Deployment, a Kubernetes object, which provide zero downtime rolling update feature.

---
Tags: docker, kubernetes, persistent-volumes
---


title: od output is confusing

text:
# Question
Title: od output is confusing
This isn't a deploy related question, but I believe it belongs here.

I've found out about the `od` tool yesterday, and began to fiddle with it.

The question is about `utf-8` encoding. See the examples:

```
~ $ echo -n á | od -x
0000000 a1c3
0000002
~ $ echo -n áá | od -x
0000000 a1c3 a1c3
0000004
~ $ echo -n ác | od -x
0000000 a1c3 0063
0000003

# This is the issue
~ $ echo -n cá | od -x
0000000 c363 00a1
0000003

```

So `á` translates to `c3 a1` correctly in each test besides the last, which begins with an `ascii` char. Why isn't the output of the last test `0063 a1c3` (i.e. invert the one before the last) ?

I've tried to read from a `utf-8` file and the result is the same.

Thanks in advance.

# Answer
> 1 votes
Problem solved:

It's because of endianess of output.

There is an option `--endian` which by default is `little`.

The example string was , which translates to `63 C3 A1` `utf-8` encoded:

```
~ $ echo -n cá | od --endian=big -x
0000000 63c3 a100
0000003

```

(The `-n` prevents `\n` in the end of string, making last byte `00`)

So it's exactly the same output, but with bytes inverted:

```
~ $ echo -n cá | od --endian=little -x
0000000 c363 00a1
0000003

```

(`c363` instead of `63c3` and so forth).

# Answer
> 1 votes
Because of endianess and other things I've settled on these two aliases I use all the time. The important thing is `-t x1`

```
alias od1='od -Ax -t x1'
alias od2='od -Ax -t x1 -c'

```

---
Tags: bash
---


# Question
Title: Integrate iOS Xcode Build Node with Jenkins X + Tekton
We are currently investigating in replacing our old QuickBuild CI setup with a modern cloud-native solution that is integrated into our k8s cluster. Jenkins X with Tekton (without classic Jenkins) sounds very promising.

We do not only have k8s applications though. Especially our iOS builds need to be executed on a Mac build node, and thus cannot run in k8s.

(How) is it possible to integrate a Mac build node into Jenkins X + Tekton?

# Answer
I don't think OS X builds can be done with Tekton yet. For builds that need OS X, I would currently consider to use GitHub Actions with support for macOS actions.
> 1 votes

---
Tags: ios, jenkins-x
---


title: Jenkins: how to run task every 30 days

text:
# Question
Title: Jenkins: how to run task every 30 days
Jenkins version is 2.235.5 (I should update it to next LTS version 2.249).

I configured Jenkins to run a job periodically every 30 days: Build Triggers → Build periodically → Schedule:

```
H 3 */30 * *

```

And now I see:

> Would last have run at Saturday, October 31, 2020 3:20:39 AM EDT; would next run at Sunday, November 1, 2020 3:20:39 AM EST.

So it will be started again after 1 day, not 30 days…

How to run a job periodically every 30 days?

Update: history of Jenkins "predictions" for schedule above (job was created October 28):

> Would last have run at Thursday, October 1, 2020 3:20:06 AM EDT; would next run at Saturday, October 31, 2020 3:20:06 AM EDT.

> Would last have run at Saturday, October 31, 2020 3:20:39 AM EDT; would next run at Sunday, November 1, 2020 3:20:39 AM EST.

> Would last have run at Sunday, November 1, 2020 3:20:43 AM EST; would next run at Tuesday, December 1, 2020 3:20:43 AM EST.

# Answer
In short, it's not practically possible. This S/O answer is quite detailed.

> Jenkins used a cron expression, and the different fields are:
>
> ```
> MINUTES Minutes in one hour (0-59)
> HOURS Hours in one day (0-23)
> DAYMONTH Day in a month (1-31)
> MONTH Month in a year (1-12)
> DAYWEEK Day of the week (0-7) where 0 and 7 are sunday
>
> ```

cron syntax simply does not support every `n days`.

An impractical option wuld be to have a daily job with a Conditional BuildStep to calculate if this day is n × 30 days from a given fixed day and abort if not..

But every 30 days doesn't really align w/calendar anyways, so is that really the intent?
> 2 votes

# Answer
Jenkins- scheduled tab change syntax like...

. 30 * * * *

Example for understanding.... similar to cron job
> -1 votes

---
Tags: jenkins
---


# Question
Title: How to see when an AWS IAM role was last used?
In AWS in the GUI you can select a role and then click on delete, then when you are prompted to approve or deny the deletion it shows you when the role was last used.

I would like to be able to know when a role was last used without pretend deleting it via the GUI.Ideally if I could get that data from the AWS CLI or a Boto3 script that would be great. Is there a way to accomplish that?

# Answer
I've been meaning to come back to this. After consulting with AWS support it would seem that this is possible using some functions of the AWS CLI, that I was not aware of.

Specifically they recommended using `aws iam generate-service-last-accessed-details` and `aws iam get-service-last-accessed-details-with-entities`, and they referred me to https://docs.aws.amazon.com/IAM/latest/UserGuide/access\_policies\_access-advisor-view-data.html#access\_policies\_access-advisor-viewing-cli.

For completion's sake here's how I used it:

```
#! /bin/bash

ROLES=$(aws iam list-roles | jq -r .Roles[].Arn)
for ROLE in $ROLES
do
    echo $ROLE
    JOBID=$(aws iam generate-service-last-accessed-details --arn $ROLE | jq -r .JobId)
    echo $JOBID
    NAMESPACES=$(aws iam get-service-last-accessed-details --job-id $JOBID | jq -r .ServicesLastAccessed[].ServiceNamespace)
    for NAMESPACE in $NAMESPACES
    do
        echo $NAMESPACE
        aws iam get-service-last-accessed-details-with-entities --job-id $JOBID --service-namespace $NAMESPACE | jq '.JobCompletionDate,.EntityDetailsList[].EntityInfo.Name,.EntityDetailsList[].EntityInfo.Id'
    done
done
```
> 2 votes

# Answer
Amazon CloudTrail tracks all API use

In CloudTrail create a new trail

You can configure each trail to send log events to CloudWatch: Edit the trail and choose to send Logs to CloudWatch. It will offer you to create a Role for this

https://docs.aws.amazon.com/awscloudtrail/latest/userguide/monitor-cloudtrail-log-files-with-cloudwatch-logs.html

Then, run some calls on your role and wait 5 minutes.

In Cloudwatch, go to your log trail and search for "AssumedRole". Your events will be shown here

There is also documentation on searching Cloudwatch logs with filters through the api https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/SearchDataFilterPattern.html
> 5 votes

# Answer
**This is possible using AWS SDKs.**

I was cleaning my tabs after finishing this task and decided I would write an answer to help other people.

**Versions:**

* aws-cli/2.0.61
* go1.15.3 linux/amd64

---

Let's do this in Golang with the right imports.

```
 import (
     "fmt"
     "github.com/aws/aws-sdk-go/aws"
     "context"
     "github.com/aws/aws-sdk-go-v2/config"
     "github.com/aws/aws-sdk-go-v2/service/iam"
)

```

First, you want to init a session and a client:

```
cfg, err := config.LoadDefaultConfig()
svc := iam.NewFromConfig(cfg)

```

Then you want to get a list of roles, to iterate through all role names. You can also use a specific role name because that's the only thing you will need to get the LastUsedDate:

```
roles_iam, err := svc.ListRoles(context.Background(), &iam.ListRolesInput{
PathPrefix: aws.String("/")})
if err != nil {
    fmt.Println("Error", err)
    return
}

/** For simplicity and less dereferencing: more execution speed **/
roles_list := roles_iam.Roles

```

Finally, iterate through a loop if you need to get LastUsedDate for all roles. Just set input with the right GetRoleInput arguments then call svc.GetRole to get a structure with your info.

```
/** Declare slice 'unused_roles' containing string unused role names **/
var unused_roles []string

for i := range roles_list {
    role_name := *roles_list[i].RoleName
    input := &iam.GetRoleInput{
        RoleName: aws.String(*roles_list[i].RoleName),
    }
    role_info, err := svc.GetRole(context.Background(), input)
    if err != nil {
        fmt.Println("Error", err)
    }

/** Check if role has never been used **/
    if (role_info.Role.RoleLastUsed.LastUsedDate) == nil {
        fmt.Printf("Role %s has never been used\n", role_name)
        unused_roles = append(unused_roles, role_name)
        continue
    }
    last_used_date := *role_info.Role.RoleLastUsed.LastUsedDate
}

```

More info:

https://docs.aws.amazon.com/IAM/latest/APIReference/API\_Role.html

https://docs.aws.amazon.com/IAM/latest/APIReference/API\_RoleLastUsed.html
> 3 votes

# Answer
There currently is no method using SDKs for the AWS CLI to get the last accessed time of an IAM role.

Currently, the only way is to use the AWS Management Console.

1. Select your **IAM role** and explore it
2. Click the "**Access Advisor**" tab.
3. The contents of this tab will display the last access time for each of the various services.
> 2 votes

---
Tags: amazon-web-services, aws-cli
---


title: Kibana slow performance optimization

text:
# Question
Title: Kibana slow performance optimization
I have a Kibana page which takes about 7-10 seconds to load.

Benchmarking same page shows ~1s to complete the request

```
$ ab -n 10 "http://kibana/app/discover#/view/15a67850-6921-11ea-8308-058e42e28132"

```

and the exact same query to elastic takes about 1s to complete as well.

```
$ ab -n 20 -p same-query.json -T application/json "http://kibana:9200/logstash*/_search"

```

It's just 2 simple `match` filters within a request to `logstash*` index. Taking into account benchmarking above I don't think the the issue related to Elasticsearch/Logstash but rather to Kibana itself. So my questions are:

1. Is the a way to tune/optimize performance for Kibana (page loading time)?
2. In case I move Kibana UI to separate instance will that give any benefits to performance? (currently it's ELK stack on the one instance 8 vcpus, 16 GiB memory)
3. How can I additionally narrow down the exact reason of this slow performance?

# Answer
> 3 votes
So basically I've ended up turning these flags off (`/app/management/kibana/settings`)

```
filterEditor:suggestValues
discover:searchOnPageLoad
doc_table:highlight
visualize:enableLabs
telemetry:enabled

```

Other ideas I had in mind:

* adding nginx as a proxy caching all the static files (js for plugins / images)
* warmup script (periodically makes requests to predefined hot pages for Kibana to internally cache these pages) if that would work
* migrate to hosted cloud solution which might be optimized better for the loading time

---
Tags: azure, performance, ubuntu, elk, kibana
---


title: Azure DevOps proxy for Git

text:
# Question
Title: Azure DevOps proxy for Git
Does Azure DevOps 2019 Proxy server (formerly TFS proxy server) support Git repositories? The documentation only mentions TFVC repositories, and it works great, but we are not sure if Git repositories can be configured to use Azure DevOps (TFS) proxy (not to confuse with regular Web proxy). Appreciate any help!

# Answer
> 1 votes
Yes, you are right. Azure DevOps (or TFS) Proxy was designed to support Team Foundation Version Control while TFS did not support GIT in earlier versions. The documentation does not contain any information for GIT, it contains only for TFVC: Configure Team Foundation version control.

Consider using Git Virtual File System: https://github.com/microsoft/VFSForGit

---
Tags: azure-devops
---


title: AWS t4g instance type not eligible

text:
# Question
Title: AWS t4g instance type not eligible
According to the blog post since september/2020 and up to december/2020 instance type t4g will be available. I'm trying to create a machine, on one of the right zones (Oregon) according the FAQ, and using the right AMI, also according to the FAQ

So why I cannot choose those machines?

As you can see on the screenshot, t3a are eligible, but all t4g not:

# Answer
> 2 votes
Solution was pretty simple: you have to choose ARM architecture on the first step, AMI selection. Accustomed to choose x86 architecture, I didn't pay attention to that radio button. Also I didn't bother to hover over the forbidden icon on the first column in the image posted on question, which clearly stated what was the problem:

So solution is to select ARM on the previous step:

---
Tags: amazon-ec2
---


# Question
Title: Terraform Dynamic Block with list intro them
I'm creating a AWS Cloudfront module and some troubles are happening because have some Cloudfront parameters that are "dynamics/optional" too.

When I'm trying run the `$ terraform validate` show me the error:

```
on ../../../../../terraform-aws-modules/cloudfront_distribution/main.tf line 24, in resource "aws_cloudfront_distribution" "cdn":
  24:   dynamic "custom_origin_config" {

Blocks of type "custom_origin_config" are not expected here.

```

Following my **module** `main.tf`:

```
resource "aws_cloudfront_distribution" "cdn" {

  enabled               = var.enabled
  comment               = var.comment
  default_root_object   = var.default_root_object
  aliases               = var.aliases
  is_ipv6_enabled       = var.is_ipv6_enabled
  http_version          = var.http_version
  price_class           = var.price_class
  web_acl_id            = var.web_acl_id
  retain_on_delete      = var.retain_on_delete
  wait_for_deployment   = var.wait_for_deployment

  dynamic "origin" {
    for_each  = var.origin
    content {

      domain_name   = origin.value.domain_name
      origin_id     = origin.value.origin_id
      origin_path   = origin.value.origin_path
    }
  }

  dynamic "custom_origin_config" {
    for_each  = var.custom_origin_config
    content {

      http_port                 = custom_origin_config.value.http_port
      https_port                = custom_origin_config.value.https_port
      origin_protocol_policy    = custom_origin_config.value.origin_protocol_policy
      origin_ssl_protocols      = custom_origin_config.value.origin_ssl_protocols
      origin_keepalive_timeout  = custom_origin_config.value.origin_keepalive_timeout
      origin_read_timeout       = custom_origin_config.value.origin_read_timeout

    }
  }

```

My `vars.tf` :

```
variable "origin"   {
    type        = any
    default     = []
    description = "One or more origins for this distribution (multiples allowed)."
}

variable "custom_origin_config" {
    type        = any
    default     = []
    description = "The CloudFront custom origin configuration information. If an S3 origin is required, use s3_origin_config instead."
}

```

**EDIT 1**

A made a little changes `main.tf`

```
 dynamic "origin" {
    for_each  = var.origin
    content {

      domain_name   = origin.value.domain_name
      origin_id     = origin.value.origin_id
      origin_path   = origin.value.origin_path

      dynamic "custom_origin_config" {
        for_each  = var.custom_origin_config
        content {

        http_port                 = custom_origin_config.value.http_port
        https_port                = custom_origin_config.value.https_port
        origin_protocol_policy    = custom_origin_config.value.origin_protocol_policy
        origin_ssl_protocols      = custom_origin_config.value.origin_ssl_protocols
        origin_keepalive_timeout  = custom_origin_config.value.origin_keepalive_timeout
        origin_read_timeout       = custom_origin_config.value.origin_read_timeout

        }
      }
    }
  }

```

and onto `vars.tf`:

```
variable "origin"   {
    type        = any
#    default     = []
    description = "One or more origins for this distribution (multiples allowed)."
}

variable "custom_origin_config" {
    type        = list
#    default     = []
    description = "The CloudFront custom origin configuration information. If an S3 origin is required, use s3_origin_config instead."
}

```

**And now the output error change:**

```
Error: Incorrect attribute value type

  on ../../../../../terraform-aws-modules/cloudfront_distribution/main.tf line 29, in resource "aws_cloudfront_distribution" "cdn":
  29:         origin_ssl_protocols      = custom_origin_config.value.origin_ssl_protocols

Inappropriate value for attribute "origin_ssl_protocols": set of string
required.

Error: Incorrect attribute value type

  on ../../../../../terraform-aws-modules/cloudfront_distribution/main.tf line 43, in resource "aws_cloudfront_distribution" "cdn":
  43:         locations          = geo_restriction.value.location

Inappropriate value for attribute "locations": set of string required.

Error: Incorrect attribute value type

  on ../../../../../terraform-aws-modules/cloudfront_distribution/main.tf line 55, in resource "aws_cloudfront_distribution" "cdn":
  55:       cloudfront_default_certificate  = viewer_certificate.value.cloudfront_default_certificate

Inappropriate value for attribute "cloudfront_default_certificate": a bool is
required.

```

**Someone has been through this problem, managed to solve it?**

# Answer
I'm resolved my question like bellow:

* I discovered that some arguments that I used before not was used when you use the **s3\_origin** and them are used with **custom\_orogin**

Was necessary used too variables `type=list(object({foo=bar})` for my dynamic blocks lists.

Part of my module bellow:

**main.tf**

```
 dynamic "origin" {
    for_each = var.origin
    content {

      domain_name = origin.value.domain_name
      origin_id   = origin.value.origin_id

      s3_origin_config {
        origin_access_identity = origin.value.origin_access_identity
      }
    }
  }

```

**vars.tf**

```
variable "origin" {
  type        = list(object({ domain_name = any, origin_id = string, origin_access_identity = string }))
  description = "One or more origins for this distribution (multiples allowed)."
}

```

Another thing that I learned it's when you set a `default=value` onto variable `type` same this *var* is (optional) you turn them (required).
> 0 votes

---
Tags: amazon-web-services, terraform, terraform-module
---


# Question
Title: Best practices to maintain forks of Git repository in an on-prem Azure DevOps Server
We have an on-premises Azure DevOps Server and like to use the Git workflow "Fork-Clone-Push-PR".

In the moment of forking the system let allow us decide into which project and under which name the repository should get forked.
Forking it in the same project would mean tons of such forks after a while, which looks ugly.

So a dedicated project "Forks" might be the solution to host all these forks. But is this a nice option?

Are there any best practices or tips & tricks on how we should proceed?

# Answer
I think that the best way to answer your question is to take a step back to focus on your goal. Based on the fact that you're using an on-premise Azure DevOps Server, I am assuming that you're working on a closed-source project (I apologize if this assumption is incorrect). I believe that your goal is to control quality via PRs in this environment.

As this Atlassian tutorial states, "The Forking Workflow is most often seen in open source projects". The reason for this is that the fork mechanism was created for a very specific problem, "How do you give complete strangers the ability to contribute to a project without giving the world write permissions to your repository?". Note, I occasionally use the forking mechanism for other fairly advanced purposes as well, but this is its primary use. There is a fantastic book called Git for Teams that describes git workflows and their intents very nicely.

My first experience with git was working in GitLab on a closed-source project. We were a team of ten employees working on an enterprise software application. We were required by the process to fork the repo and create a PR to get our changes back. The result was a big learning curve for the engineers who were new to git and a lot of confusion around setting up remotes and getting the latest code.

Many people erroneously think that the only way to have a pull-request mechanism is to fork the code, probably because their only exposure is through PRs in the open-source community. To quote Microsoft, "Pull requests can come from either **topic branches** or from a branch in a fork of the original repository". A topic branch is just a fancy term for a `bug`/`feature`/etc branch. When I am working with a closed-source group that is new to git, I will often disallow forks in the Project/Repository settings and I will enforce a branching policy of a minimum number of reviewers for a pull-request to the master branch.

Regarding best practices for branching strategies in git, the book I mentioned above is a great first step. There are many standard branching strategies, a couple examples are:

1. gitflow which is great for teams that are new to git and has some awesome tools and many tools incorporate the gitflow pattern, like sourcetree.
2. trunk-based development a very lean development process.

Regarding your original question, I suspect that creating a new Azure DevOps project to store all of those unsightly extra fork repos would just add another level of complexity to your work. You would need to enforce security for both projects, manage users and permissions for the new project, etc. In this case subtraction is likely better than addition, I suggest removing forking and using a good branching strategy for your project (if possible). It would simplify your life significantly.

Please let me know if this did not answer your question or if you have any further questions.
> 1 votes

---
Tags: git, azure-devops
---


# Question
Title: Can I upgrade a docker image/container from buntu 18.04 to ubuntu 20.04?
I have a host ubuntu 20.04 running my software on docker with a container derivated from ubuntu 18.04. I would like to use ubuntu 20.04 for several reasons (like the better software on 20.04). If I run do-release-upgrade on the container terminal I receive the following: "do-release-upgrade Checking for a new Ubuntu release You have not rebooted after updating a package which requires a reboot. Please reboot before upgrading." Rebooting the container or rebooting the host machine does not solve the problem.

# Answer
> 1 votes
Docker doesn't recommend running an apt-get upgrade and dist-upgrade on a docker image as many of the "essential" packages from the partent image cannot upgrade inside an unprevileged container. take a look at the Docker best practices

# Answer
> 0 votes
That is not good design to have docker or any lib update after logging into the container. As mentioned by Andrea Giaime Bodei , you need to update the base image in the Dockerfile and rebuild the image. The docker images are immutable.

```
# Download base image ubuntu 20.04
FROM ubuntu:20.04

# LABEL about the custom image
LABEL maintainer="abcd@me.com"
LABEL version="0.1"
LABEL description="This is custom Docker Image for \
the test"

```

---
Tags: docker, containers, linux, ubuntu
---


# Question
Title: How to get branch hash using git command
I have JenkinsFile where I'm trying to get the hash for my current branch. I saw this post: https://stackoverflow.com/questions/9110478/how-to-find-the-hash-of-branch-in-git

but when I tried to run `def BranchHash = sh "git rev-parse ${BRANCH-NAME}` I got: `fatal: ambiguous argument 'HEAD': unknown revision or path not in the working tree`

I'm looking for the branch hash, and not the last commit

# Answer
> 2 votes
```
git show -s --format="%H" <HEAD or Branch Name>

```

`-s` supressess the diff output

`--format="%H"` specifies that only the hash should be shown.

See `man git-show` for more details.

---
Tags: git
---


title: nginx rewrite with proxy_pass

text:
# Question
Title: nginx rewrite with proxy_pass
I want to proxy requests with nginx to different domain and also rewrite some part of the path. For example

`http://origin.com/` \- \> `http://target.com/_plugin/kibana`

How to accomplish such automated addition of `_plugin/kibana` part to any route that I request and forward to my target server

```
server {
    listen 80;
    server_name _;
    location / {
        proxy_set_header X-Forwared-Proto $scheme;
        rewrite ^(.*)$ /_plugin/kibana/$1 break;
        proxy_pass https://my-es-custer-in4lf3ruqx6shv2ndhju.us-east-1.es.amazonaws.com/;

    }
}

```

Should I have different location blocks or should I have a rewrite directive on a server level for this to work properly?

# Answer
> 1 votes
You do not need to use `rewrite` for your case, since you want to append `_plugin/kibana` to **any routes**.

Assuming the nginx config is for `http://origin.com/`:

```
server {
    listen 80;
    server_name origin.com;
    location / {
        proxy_set_header X-Forwared-Proto $scheme;

        # http://origin.com/req/1/2 will go to http://target.com/_plugin/kibana/req/1/2
        proxy_pass http://target.com/_plugin/kibana;

    }
}

```

---
Tags: amazon-web-services, nginx, kibana
---


# Question
Title: Helm Installation fails with shared config maps
I have a couple of deployments in Kubernetes packaged using Helm. Both the deployments are different but they share the same config map.

When performing installation via Helm the first deployment is successful but the second deployment fails with the below error.

> **Error: rendered manifests contain a resource that already exists. Unable to continue with install: ConfigMap "appconfig" in namespace "testnamespace" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-name" must equal "app2": current value is "app2"**

# Answer
Couple of possible solutions:

1. Control the configmap from a third chart and add the third chart as a dependency in both the charts. This will duplicate the secret and can cause a skew in the values being used both the charts since if they have independent release cycles
2. Create the configmap in one namespace and refer to it using an external reference. You can see several examples of such at https://github.com/helm/charts/search?q=existingSecret&type=code

I personally prefer option 1 since there is explicit dependency specification and gives the ability to deploy the charts individually
> 1 votes

---
Tags: kubernetes, helm
---


# Question
Title: How to navigate to an index.html with nginx?
This builds and runs without apparent error:

```
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# docker exec -ti some-nginx bash
root@2c912aa5ce5c:/#
root@2c912aa5ce5c:/# ls /usr/share/nginx/html/
50x.html  index.html
root@2c912aa5ce5c:/#
root@2c912aa5ce5c:/# cat /usr/share/nginx/html/50x.html
<!DOCTYPE html>
<html>
<head>
<title>Error</title>
<style>
    body {
        width: 35em;
        margin: 0 auto;
        font-family: Tahoma, Verdana, Arial, sans-serif;
    }
</style>
</head>
<body>
<h1>An error occurred.</h1>
<p>Sorry, the page you are looking for is currently unavailable.<br/>
Please try again later.</p>
<p>If you are the system administrator of this resource then you should check
the error log for details.</p>
<p><em>Faithfully yours, nginx.</em></p>
</body>
</html>
root@2c912aa5ce5c:/#
root@2c912aa5ce5c:/# cat /usr/share/nginx/html/index.html

hello world

how do I put this file in a diff dir?

root@2c912aa5ce5c:/#
root@2c912aa5ce5c:/# exit
exit
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# docker container ls
CONTAINER ID        IMAGE                COMMAND                  CREATED             STATUS              PORTS                  NAMES
2c912aa5ce5c        some-content-nginx   "/docker-entrypoint.…"   5 minutes ago       Up 5 minutes        0.0.0.0:8080->80/tcp   some-nginx
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# ls
dockerfile  index.html
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# cat dockerfile
FROM nginx
COPY index.html /usr/share/nginx/html
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# cat index.html

hello world

how do I put this file in a diff dir?

root@mordor:/home/nicholas/docker/nginx#

```

Yet, there's no output when browsing to `localhost`:

```
nicholas@mordor:~$
nicholas@mordor:~$ lynx localhost --dump

Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/
nicholas@mordor:~$
nicholas@mordor:~$ lynx localhost/index.html --dump

Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/index.html
nicholas@mordor:~$
nicholas@mordor:~$ lynx localhost/50x.html --dump

Looking up localhost
Making HTTP connection to localhost
Alert!: Unable to connect to remote host.

lynx: Can't access startfile http://localhost/50x.html
nicholas@mordor:~$

```

Neither am I getting a welcome file from `nginx` as I'd expect.

# Answer
> 1 votes
Did you only try to reach localhost? Or even 0.0.0.0 or 127.0.0.1?

Depending on the operating system, docker config and the internal networking setup, some problem can occurs to reach the docker network.

You can inspect the container to get the network IP, maybe you can go there https://stackoverflow.com/a/20686101/2904965

---
Tags: docker, docker-compose, dockerfile, nginx, servers
---


# Question
Title: How to copy HTML files to include into nginx?
How do I copy an `index.html` file into `nginx`?

```
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# ll
total 12
drwxrwxr-x 2 nicholas nicholas 4096 Nov 13 16:18 ./
drwxrwxr-x 3 nicholas nicholas 4096 Nov 13 16:12 ../
-rw-rw-r-- 1 nicholas nicholas   60 Nov 13 16:18 dockerfile
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# cat dockerfile
FROM nginx
COPY /home/nicholas/html /usr/share/nginx/html

root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# docker build -t some-content-nginx .
Sending build context to Docker daemon  2.048kB
Step 1/2 : FROM nginx
 ---> c39a868aad02
Step 2/2 : COPY /home/nicholas/html /usr/share/nginx/html
COPY failed: stat /var/snap/docker/common/var-lib-docker/tmp/docker-builder726461098/home/nicholas/html: no such file or directory
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# ls /home/nicholas/html/
index.html
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# cat /home/nicholas/html/index.html

index
hello world

nicholas@mordor:~/html$

```

I'm unclear on the source and destination, which might be complicated by `docker` having been installed from `snap`.

---

Looks like `ansible` has slightly clearer, at least for me, syntax:

```
  tasks:
  - name: "Copying files"
    copy:
      src: files/{{ item }}
      dest: /my/folder/
    with_items:
      - file.txt

```

or perhaps I'm doing something incorrectly.

# Answer
`The docker build command builds Docker images from a Dockerfile and a “context”. A build’s context is the set of files located in the specified PATH or URL. The build process can refer to any of the files in the context. For example, your build can use a COPY instruction to reference a file in the context.`

From docker build documentation.

As you run the docker build with the `.` path for the context, it will try to find the folder `home/nicholas/html` from the context `root@mordor:/home/nicholas/docker/nginx`

The COPY command documentation shows some examples
> 1 votes

# Answer
This at least builds and runs:

```
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# ls -al
total 16
drwxrwxr-x 2 nicholas nicholas 4096 Nov 13 17:18 .
drwxrwxr-x 3 nicholas nicholas 4096 Nov 13 16:12 ..
-rw-rw-r-- 1 nicholas nicholas   49 Nov 13 17:18 dockerfile
-rw-rw-r-- 1 nicholas nicholas   56 Nov 13 17:18 index.html
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# cat dockerfile
FROM nginx
COPY index.html /usr/share/nginx/html
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# cat index.html

hello world

how do I put this file in a diff dir?

root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# docker build -t some-content-nginx .
Sending build context to Docker daemon  3.072kB
Step 1/2 : FROM nginx
 ---> c39a868aad02
Step 2/2 : COPY index.html /usr/share/nginx/html
 ---> Using cache
 ---> 07a94375fb55
Successfully built 07a94375fb55
Successfully tagged some-content-nginx:latest
root@mordor:/home/nicholas/docker/nginx#
root@mordor:/home/nicholas/docker/nginx# docker run --name some-nginx -d -p 8080:80 some-content-nginx
2c912aa5ce5c8b864a6fe79a18ee8e833874f85a75fed1c7020163ca7b90bedb
root@mordor:/home/nicholas/docker/nginx#

```

although I can't seem to get anything from navigating to `localhost` so I'll have to look closer at the `container` which is built.
> 0 votes

---
Tags: docker, docker-compose, dockerfile, nginx, ubuntu
---


# Question
Title: How to serve a custom index.html using the stock nginx docker image?
How do I serve static content from the default `nginx` docker image? Given:

```
root@mordor:~#
root@mordor:~# docker image ls nginx
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nginx               latest              c39a868aad02        8 days ago          133MB
root@mordor:~#

```

where and how can a hello world be configured?

In the context of:

https://askubuntu.com/q/1292182/847449

# Answer
> 6 votes
The "official" nginx image has a specific path where it looks for files. You can either copy a file into the image by creating a new image from it or mount your files into this path as a volume.

To copy files, have a folder with your `index.html` and a `Dockerfile` that looks like this:

```
FROM nginx
COPY index.html /usr/share/nginx/html

```

Then build a new image with `docker build -t my-nginx .`

To start a container with your own files as a volume, you can run the container like this:

```
docker run -p 8080:80 -v ${PWD}:/usr/share/nginx/html nginx

```

All the files in your current folder (including the `index.html`) will now be available via `http://localhost:8080/index.html`.

Most of this information is covered in the documentation for the image at https://hub.docker.com/\_/nginx

# Answer
> 1 votes
running and copying files into the container:

```
root@mordor:~#
root@mordor:~# sudo docker run --name docker-nginx -p 80:80 -d nginx
29b4280504c2d46c10b8459de0844ba30bce0c1bc97b70521c3f812f29d17c27
root@mordor:~#
root@mordor:~# docker cp index.html docker-nginx:/indx.html
root@mordor:~#
root@mordor:~# docker exec -it docker-nginx bash
root@29b4280504c2:/#
root@29b4280504c2:/# ls
bin   dev          docker-entrypoint.sh  home       lib    media  opt   root  sbin  sys  usr
boot  docker-entrypoint.d  etc           indx.html  lib64  mnt    proc  run   srv   tmp  var
root@29b4280504c2:/#
root@29b4280504c2:/# cd /usr/share/nginx/html/
root@29b4280504c2:/usr/share/nginx/html#
root@29b4280504c2:/usr/share/nginx/html# ls
50x.html  index.html
root@29b4280504c2:/usr/share/nginx/html#
root@29b4280504c2:/usr/share/nginx/html# mv index.html index.html.bkup
root@29b4280504c2:/usr/share/nginx/html#
root@29b4280504c2:/usr/share/nginx/html# cp /indx.html ./index.html
root@29b4280504c2:/usr/share/nginx/html#
root@29b4280504c2:/usr/share/nginx/html# exit
exit
root@mordor:~#
root@mordor:~# docker container list
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                NAMES
29b4280504c2        nginx               "/docker-entrypoint.…"   2 minutes ago       Up 2 minutes        0.0.0.0:80->80/tcp   docker-nginx
root@mordor:~#
root@mordor:~# docker stop docker-nginx
docker-nginx
root@mordor:~#
root@mordor:~# docker start docker-nginx
docker-nginx
root@mordor:~#

```

so that:

```
nicholas@mordor:~$
nicholas@mordor:~$ lynx http://localhost/ --dump
   hello nginx docker world
nicholas@mordor:~$

```

---
Tags: docker, docker-compose, dockerfile, nginx, servers
---


# Question
Title: How to access a fully qualified domain name through "localhost"?
About nine minutes into a tutorial on `ERPNext` I followed directions as:

```
root@mordor:/home/nicholas/git/frappe_docker#
root@mordor:/home/nicholas/git/frappe_docker# cat .env
LETSENCRYPT_EMAIL=nicholas@mordor.bounceme.net
ERPNEXT_VERSION=edge
FRAPPE_VERSION=edge
MARIADB_HOST=mariadb
MYSQL_ROOT_PASSWORD=admin
SITE_NAME=erp.mordor.bounceme.net
SITES=`erp.mordor.bounceme.net`
DB_ROOT_USER=root
ADMIN_PASSWORD=admin
INSTALL_APPS=erpnext
ENTRYPOINT_LABEL=traefik.http.routers.erpnext-nginx.entrypoints=websecure
CERT_RESOLVER_LABEL=traefik.http.routers.erpnext-nginx.tls.certresolver=myresolver
HTTPS_REDIRECT_RULE_LABEL=traefik.http.routers.http-catchall.rule=hostregexp(`{host:.+}`)
HTTPS_REDIRECT_ENTRYPOINT_LABEL=traefik.http.routers.http-catchall.entrypoints=web
HTTPS_REDIRECT_MIDDLEWARE_LABEL=traefik.http.routers.http-catchall.middlewares=redirect-to-https
HTTPS_USE_REDIRECT_MIDDLEWARE_LABEL=traefik.http.middlewares.redirect-to-https.redirectscheme.scheme=https
root@mordor:/home/nicholas/git/frappe_docker#
root@mordor:/home/nicholas/git/frappe_docker# docker-compose up -d
Creating network "frappe_docker_default" with the default driver
Creating volume "frappe_docker_redis-cache-vol" with default driver
Creating volume "frappe_docker_cert-vol" with default driver
Creating volume "frappe_docker_redis-queue-vol" with default driver
Creating volume "frappe_docker_mariadb-vol" with default driver
Creating volume "frappe_docker_redis-socketio-vol" with default driver
Creating volume "frappe_docker_sites-vol" with default driver
Creating volume "frappe_docker_assets-vol" with default driver
Pulling erpnext-python (frappe/erpnext-worker:edge)...
edge: Pulling from frappe/erpnext-worker
Digest: sha256:45e7fb925187bc6ed71ca8db2bd886a8cbcfb7714197b7f140fb479469bc3c98
Status: Downloaded newer image for frappe/erpnext-worker:edge
Pulling mariadb (mariadb:10.3)...
10.3: Pulling from library/mariadb
6a5697faee43: Pull complete
ba13d3bc422b: Pull complete
a254829d9e55: Pull complete
2ee2cadd29fc: Pull complete
6915a184049d: Pull complete
5ca6ffdb5f56: Pull complete
1537f7bbef8b: Pull complete
5790e54322d1: Pull complete
ea98cb829471: Pull complete
3247072bad5f: Pull complete
b3f16e2db43f: Pull complete
d78c54af82d5: Pull complete
b08843faa954: Pull complete
Digest: sha256:69dec37e0651252108d61ac87874ed8d83b74ead38d196b0bb10d784b7295e8b
Status: Downloaded newer image for mariadb:10.3
Pulling frappe-socketio (frappe/frappe-socketio:edge)...
edge: Pulling from frappe/frappe-socketio
babf97a3f00a: Pull complete
d81eb1809d95: Pull complete
d50e90677ab1: Pull complete
4ca478941ba5: Pull complete
cac0c8c03bb1: Pull complete
55aedfc1b65f: Pull complete
1bee561a211c: Pull complete
fab9a623660f: Pull complete
7b5f157fcbc6: Pull complete
e5f74c3ab32d: Pull complete
6f1760dc7d35: Pull complete
f9389a9f5f4c: Pull complete
Digest: sha256:dfe6dc1d2ffb824f04dc2cd620327f75355620ba657865ee98441c88b8596ed5
Status: Downloaded newer image for frappe/frappe-socketio:edge
Pulling traefik (traefik:v2.2)...
v2.2: Pulling from library/traefik
cbdbe7a5bc2a: Pull complete
f16506d32a25: Pull complete
605303653d66: Pull complete
a9005a35b171: Pull complete
Digest: sha256:ea0aa8832bfd08369166baecd40b35fc58979df8f5dc5182e4e63ee6adbe66db
Status: Downloaded newer image for traefik:v2.2
Pulling erpnext-nginx (frappe/erpnext-nginx:edge)...
edge: Pulling from frappe/erpnext-nginx
bb79b6b2107f: Pull complete
5a9f1c0027a7: Pull complete
b5c20b2b484f: Pull complete
166a2418f7e8: Pull complete
1966ea362d23: Pull complete
3915f6e86363: Pull complete
f6bb7178bcfb: Pull complete
44ac8c313cd6: Pull complete
07bf0268acb1: Pull complete
2b15c776928e: Pull complete
58517104a340: Pull complete
44e719acaf68: Pull complete
f16420799ad5: Pull complete
Digest: sha256:d7031e28b62d0e67eae432937902fe618d32709039a1d63199598881ed5c6459
Status: Downloaded newer image for frappe/erpnext-nginx:edge
Creating frappe_docker_redis-cache_1     ... done
Creating frappe_docker_erpnext-nginx_1  ... done
Creating frappe_docker_mariadb_1        ... done
Creating frappe_docker_erpnext-python_1  ... done
Creating frappe_docker_redis-socketio_1 ... done
Creating frappe_docker_traefik_1                ... done
Creating frappe_docker_redis-queue_1    ... done
Creating frappe_docker_frappe-socketio_1        ... done
Creating frappe_docker_site-creator_1           ... done
Creating frappe_docker_erpnext-schedule_1       ... done
Creating frappe_docker_erpnext-worker-long_1    ... done
Creating frappe_docker_erpnext-worker-default_1 ... done
Creating frappe_docker_erpnext-worker-short_1   ... done
root@mordor:/home/nicholas/git/frappe_docker#

root@mordor:~#
root@mordor:~# docker container list
CONTAINER ID        IMAGE                         COMMAND                  CREATED             STATUS              PORTS                                      NAMES
d150de825277        frappe/erpnext-worker:edge    "docker-entrypoint.s…"   15 minutes ago      Up 15 minutes       8000/tcp                                   frappe_docker_erpnext-worker-short_1
54c716c27275        frappe/erpnext-worker:edge    "docker-entrypoint.s…"   15 minutes ago      Up 15 minutes       8000/tcp                                   frappe_docker_erpnext-worker-default_1
ef73c8514f32        frappe/erpnext-worker:edge    "docker-entrypoint.s…"   15 minutes ago      Up 15 minutes       8000/tcp                                   frappe_docker_erpnext-schedule_1
ce45f0955bac        frappe/erpnext-worker:edge    "docker-entrypoint.s…"   15 minutes ago      Up 15 minutes       8000/tcp                                   frappe_docker_erpnext-worker-long_1
9a3eb0a2f9d2        frappe/erpnext-worker:edge    "docker-entrypoint.s…"   15 minutes ago      Up 15 minutes       8000/tcp                                   frappe_docker_site-creator_1
652280f21336        frappe/frappe-socketio:edge   "docker-entrypoint.s…"   16 minutes ago      Up 15 minutes                                                  frappe_docker_frappe-socketio_1
e6b6c59b01fd        redis:latest                  "docker-entrypoint.s…"   16 minutes ago      Up 16 minutes       6379/tcp                                   frappe_docker_redis-queue_1
c3e0d8c41c08        redis:latest                  "docker-entrypoint.s…"   16 minutes ago      Up 16 minutes       6379/tcp                                   frappe_docker_redis-socketio_1
58cd5b59579c        traefik:v2.2                  "/entrypoint.sh --lo…"   16 minutes ago      Up 15 minutes       0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp   frappe_docker_traefik_1
4a4b4caffbec        frappe/erpnext-worker:edge    "docker-entrypoint.s…"   16 minutes ago      Up 15 minutes       8000/tcp                                   frappe_docker_erpnext-python_1
221dd3e1e419        frappe/erpnext-nginx:edge     "/docker-entrypoint.…"   16 minutes ago      Up 14 minutes       80/tcp                                     frappe_docker_erpnext-nginx_1
6100cc729cb1        redis:latest                  "docker-entrypoint.s…"   16 minutes ago      Up 15 minutes       6379/tcp                                   frappe_docker_redis-cache_1
a31d8a43726c        mariadb:10.3                  "docker-entrypoint.s…"   16 minutes ago      Up 16 minutes       3306/tcp                                   frappe_docker_mariadb_1
root@mordor:~#

```

However:

```
nicholas@mordor:~$
nicholas@mordor:~$ lynx localhost --dump

Looking up localhost
Making HTTP connection to localhost
Sending HTTP request.
HTTP request sent; waiting for response.
HTTP/1.0 302 Found
Data transfer complete
HTTP/1.0 302 Found
Using https://localhost/
Looking up localhost
Making HTTPS connection to localhost

lynx: Can't access startfile http://localhost/
nicholas@mordor:~$

```

How do I configure as above for local usage? Effectively, offline. To my understanding, `nginx` is being used as a reverse proxy here.

I'm getting some result:

```
nicholas@mordor:~$
nicholas@mordor:~$ lynx erp.mordor.bounceme.net --dump

nicholas@mordor:~$

```

but not the welcome screen I'm looking for. The `FQDN` isn't publicly available.

# Answer
> 2 votes
According to your screen dump you have your port 80 mapped to the Traefik container

`58cd5b59579c traefik:v2.2 "/entrypoint.sh --lo…" 16 minutes ago Up 15 minutes 0.0.0.0:80->80/tcp, 0.0.0.0:443->443/tcp frappe_docker_traefik_1`

The listen address of this mapping is `0.0.0.0` on both port `80 HTTP`, and `443 HTTPS` which means it listens on all your computer interfaces. Including the local-loopback interface which has the address of `127.0.0.1`.

Most operating systems have a file called `hosts` which is located at `/etc/hosts` for Linux, MacOS, BSD and at `C:\Windows\System32\drivers\etc\hosts` for Windows 10.

This file maps the DNS names (such as `localhost`) to their IP addresses, like `127.0.0.1`.

For you to have a working `FQDN`, you can temporarily create a new mapping in this file to map your desired name to the local-loopback address of 127.0.0.1.

Edit the `hosts` file and add a line saying:

```
127.0.0.1 erp.mordor.bounceme.net

```

The applications that lookup that name later, such as `lynx`, will now know that `erp.mordor.bounceme.net` is located at `127.0.0.1` IP address. And your Docker Traefik container is listening there (as well) on ports `80` and `443`.

---
Tags: docker, docker-compose, nginx, dns, environment-variables
---


# Question
Title: Is local / staging / production still industry standard?
I've inherited a project that uses local / staging / production.

It is a simple project that lets users create a login and download customized PDF documents. We keep the document templates and the per-user-customisations in S3.

But I've heard that githubflow allows direct deploy-to-production--upon--merge-PR-to-master.

The project comprises a React front-end and a Django backend. The backend is hosted on AWS and administered through Kubernetes. There is also a database and S3 bin on AWS.

The infrastructure was never completed, so I'm in the process of automating deploy-to-{staging/production}, and wondering if I'm being a dinosaur.

My thought is that if deploy-to-production automatically backs up the database and S3, this could give a faster/more-efficient workflow.

# Answer
When you say "Industry Standard" that's very much going to depend on what industry.

You *can* drop code into production however you want (legally restricted industries obviously being the exemption to this), but the question you should probably be examining is if you *should* be doing that.

I've recently worked on products which range from having incredibly test coverage, huge amounts of infrastructure set up to perform every imaginable regression test against any integration point that exists, and even though it was not the done thing with that particular product, after passing all of the automated testing required for a branch to be mergable, I probably would have been pretty comfortable launching straight to production. On the other end of the spectrum I've also worked on projects which arguably were developed in production given that the production instances were so dissimilar to what was actually checked in to source control or running in any test environment and, of course, no test cases to be seen.

The standard approach to making production deployments as safe as possible is to try and ensure that the thing you are planning to deploy is as rigorously tested as possible, and to do that most of the testing will need to be automated, otherwise you'll just end up spending your life testing.

Additionally, most modern systems use standard patterns to make a production roll back an extemely frictionless task, if not to automate it entirely. If you are deploying canaries of your software, or you are running a blue/green deployment and can have built in mechanisms to roll back if certain success conditions aren't met, then it further reduces the risk of a production change breaking something.

Essentially, I would question anyone telling you how to manage and derisk your production releases unless they are just as intimate with the software as you are.
> 4 votes

# Answer
In the software industry, it is popular to have staging environments but in order to understand why it is important to understand the reasons why these environments exist in the first place.

The production environment is where the company's software meets the company's users. Most companies want their users to be happy, and to receive high-quality software to use that meets their expectations.

On the other hand, a development environment is where developers are creating the software. In most cases, the quality of this software when first created is far from good. Even great developers make mistakes from time to time, and they need some safety net to catch these problems so they can fix them.

Improving software quality is an iterative process. Developers are tasked with constantly creating more and more changes, it is their job. These changes need to become high-quality, and the only way of doing that is to test the changes somewhere until all the wrinkles are polished. This is where the QA, Staging, User Acceptance environments come in. These environments are where software can be checked and any problems caught before it is being put in front of customers.

All of the above describes how having multiple environments is used to reduce the risk of bad code in production, while at the same time allowing developers to have a place where they can make mistakes safely and fix these.

Many companies have additional rules that make it practical to have multiple environments. For example, having a tradition to release software on a "special date" for everyone to use as Apple does. They still have beta testers and other testing happening before this release - which can be considered a staging or user-acceptance environment.

In some industries which are heavily regulated, there are restrictions on who can have access to the "production environment". In many cases only designated people are allowed to touch these environments because of security and legal concerns. Where in such a company there are hundreds of other developers who are not designated, they need someplace to integrated and test their code - the staging environment is less strict regarding who is allowed to change things in the environment.
> 1 votes

---
Tags: terminology, build-pipeline
---


# Question
Title: Helm Repo: can it host dockerfiles and other image-building artifacts
I have a service that I want to port to Helm. It's a django application that requires memcached and postgresql and a forward proxy like nginx.

I would ideally like to have one repository with all that stuff (dockerfiles/buildah scripts/charts). I think I can that easily with Helm repository if the images were already created. But they're not.

Is there any convention to include Dockerfile's or buildah scripts inside the Helm repository? It seems weird to have to create another repository to build the images, and then to have a helm repository to tie them together. Is this the way it's done currently?

# Answer
Well that's a legit question IMHO, but there are multiple way of doing things (and no one is worth than other).

## 1- Everything from dev to prod at the same place

If you want to grab everything from dev to production in a single project, it's possible, but you cannot build the docker image from hell command.

Helm is only here to organize, use template in yaml and sync releases following kubernetes file structure.

You can follows helm repository structure and add other scripts, but *you have to execute steps in the right order*: build, push, deploy

## 2- The "regular" way to do

I more often saw a separation of production configuration files and application files.

What I do is:

1. One repository which carry my application source code + everything for the CI (test, build, docker file)
2. One repository which carry my helm.charts for all my infrastructure.

Choose what suits the best for you, but keep in mind that helm is a templating system for kubernetes yaml files, so *it will not build the docker image*.
> 1 votes

---
Tags: docker, kubernetes, containers, helm, images
---


title: Updating a TFS repo with Jenkins

text:
# Question
Title: Updating a TFS repo with Jenkins
I want to checkout a TFS repo, make some changes to a file and send the changeset to TFS.

What are some ways to accomplish this with Jenkins?

# Answer
> 0 votes
I solved this using team-explorer-everywhere. We made a container that clones the code using it edits it an sends it back up. https://github.com/microsoft/team-explorer-everywhere

---
Tags: jenkins, tfs
---


# Question
Title: how can I import python module tkinter in gitpod
I can't import pyton module `tkinter` in my gitpod workspace.

My .gitpod.Dockerfile contains:

```
FROM gitpod/workspace-full-vnc
USER root
RUN apt update && apt install -y python-tk python3-tk tk-dev

```

but when I run my project I get

```
Traceback (most recent call last):
  File "basic-screen-changer.py", line 1, in <module>
    import tkinter as tk
  File "/home/gitpod/.pyenv/versions/3.8.6/lib/python3.8/tkinter/__init__.py", line 36, in <module>
    import _tkinter # If this fails your Python may not be configured for Tk
ModuleNotFoundError: No module named '_tkinter'

```

`python --version` shows me `Python 3.8.6`.
Installed packaged are:

```
itpod /workspace/pydiag $ dpkg -l |grep ^ii |tr -s ' ' |cut -d ' ' -f2,3 |egrep '^python.*tk|^tk'
python-tk 2.7.18-1
python3-tk:amd64 3.8.5-1~20.04.1
tk 8.6.9+1
tk-dev:amd64 8.6.9+1
tk8.6 8.6.10-1
tk8.6-blt2.5 2.5.3+dfsg-4
tk8.6-dev:amd64 8.6.10-1

```

# Answer
That problem occurs because the `workspace-full-vnc` image uses `pyenv` and the module path is different (source). To use python modules, you need to install them using `python2` or `python3 -m pip install ...`.
> 1 votes

---
Tags: cicd, python
---