# Question
Title: What's the relevance of periodicity in CI / CD processes?
Part of what (I think) CI and CD is about, is processes that get repeated over and over again (= the "*continuous*" part of it). However continuous in this context is not exactly (I think) like a river where water keeps going (flowing) all the time. Instead it's rather something happening every X seconds, minutes, hours etc.

So assume some test-environment where every day, eg, from 5 pm to (max) 6 pm all required processes are executed to rebuild all executables that are impacted by the changed dependencies (sources, includes, etc), combined with appropriate CI/CD processes related to them. While before 5 pm and after 6 pm, none of these kinds of processes are allowed to be executed (eg to ensure the stability of the related environments).

Does such kind of periodicity (= between 5 pm and 6 pm, every day) fit with CI/CD or not? Or is it rather that periodicity has nothing to do with CI/CD?

# Answer
I would separate the CI and CD contexts, as the periodicity in one of them is rather loosely coupled to the periodicity in the other one.

That's primarily because CI attempts to produce versions of software *available* for delivery/distribution, see How does continuous integration relate to continuous delivery / deployment? But not all CI executions are successful. CD periodicity will, at best, be equal to the CI one, but typically it will be lower.

In CI the periodicity is tightly coupled to the desired development velocity (and maybe branch stability) and can be driven by different project-specific requirements. Here are just some examples of CI execution patterns, there could be others:

1. launched for every changeset commit - the recommended one, as it offers the shortest time to identify a changeset causing a regression, see Build on each commit - Continuous delivery. Costs vary with the commit rates.
2. launched every `N` (fixed number of) commits - trade culprit identification performance of #1 for slightly lower costs (overall fewer executions)
3. launched every `T` (fixed) time interval

* fixed costs due to predictive amount of executions: no cost hikes for commit activity peaks, a day with twice the number of commits than another will still cost the same
    * culprit identification performance drops vs #1 if multiple changesets are committed within `T`
    * consecutive executions without any changeset committed in between offer measurements of the verification process reliability - different results in such executions could indicate unreliable verifications
4. launched whenever resources for execution are available - if that's where the bottleneck of the process lies. Any of the above categories can appear depending on the pattern of commits relative to the intervals between executions. This maximizes the verification resources ROI.
5. launched with one pattern during working hours and a different pattern overnight - to balance computing resources utilisation, for example.

Another possible reason for periodicity mismatch between CI and CD is the variability in the duration of the CI executions. Caused, for example, by that prohibited build schedule policy you mentioned (see also How to implement a frozen test environment?) Or by builds executed on classes of servers with significantly different performance, thus lasting different amounts of time.

Even for a successful CI execution in which the produced software version qualifies and is available for deployment it doesn't necessarily mean it will also be immediately deployed (or even deployed at all). There can be other criteria or policies preventing it (business reasons, for example).
> 4 votes

# Answer
I would hesitate to describe your once per day model as CI/CD, it sounds more like the "nightly build" model.

It really depends on the nature of your work and processes. If things flow smoothly through building, testing, and preparation of an artifact for deployment and your development model / velocity isn't such that it would benefit from more frequent runs, I would say you have the spirit of CI/CD.
> 4 votes

# Answer
Periodicity will only come into effect if the overall integration and deployment processes take a long time to complete, which is more likely the actual problem. Said differently, in an environment where applications are built using lightweight architectures that are quickly scanned, built, deployed and tested, periodicity would cease to be relevant.

I would say that periodicity prevents a system from being truly continuous, so that is the way in which they are related. PI/PD
> 1 votes

---
Tags: continuous-integration, continuous-delivery, continuous-deployment
---


# Question
Title: how to connect to a MySQL database in a docker container?
starting up the default `mysql` server:

```
root $
root $ docker run --name mysql -d -p 3306:3306 -e MYSQL_ROOT_PASSWORD=password mysql/mysql-server:latest
d2b066e9aa4192b45334e9dada923b6b3dca5703e811733e150333f52849c1a1
root $
root $ docker ps -a
CONTAINER ID        IMAGE                       COMMAND                  CREATED             STATUS                            PORTS                               NAMES
d2b066e9aa41        mysql/mysql-server:latest   "/entrypoint.sh mysq…"   9 seconds ago       Up 6 seconds (health: starting)   0.0.0.0:3306->3306/tcp, 33060/tcp   mysql
root $

```

trying to login to the container, but getting the host `MySQL` instance:

```
nicholas $
nicholas $ mysql -h 172.17.0.2 -P 3306 --protocol=tcp -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 21
Server version: 8.0.21 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| %         | root             |
| %         | user             |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
6 rows in set (0.00 sec)

mysql>

```

I'm trying that explicit `IP` address because:

```
root $
root $ docker inspect mysql | grep IP
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "172.17.0.2",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
                    "IPAMConfig": null,
                    "IPAddress": "172.17.0.2",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
root $

```

although I'm also getting the same result for `localhost` in that I'm still logging into the host instance of `MySQL` rather than the container.

```
root $
root $ docker exec -ti --user root mysql bash
bash-4.2#
bash-4.2# mysql -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 26
Server version: 8.0.21 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| localhost | healthchecker    |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
5 rows in set (0.00 sec)

mysql>

```

which is virgin, not having user `user` configured on the container.

Additionally:

```
root $
root $ netstat -peanut | grep 3306
tcp6       0      0 :::3306                 :::*                    LISTEN      0          25585352   488887/docker-proxy
root $
root $ docker inspect mysql | grep password
                "MYSQL_ROOT_PASSWORD=password",
root $

```

and so I was expecting that by specifying `tcp` for `localhost` from the host I'd be able to connect to the docker container. Yet that's not happening, I'm getting the host's database:

```
nicholas $
nicholas $ mysql -h localhost -P 3306 --protocol=tcp -u root -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 8.0.21 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql>
mysql> SELECT host, user FROM mysql.user;
+-----------+------------------+
| host      | user             |
+-----------+------------------+
| %         | root             |
| %         | user             |
| localhost | mysql.infoschema |
| localhost | mysql.session    |
| localhost | mysql.sys        |
| localhost | root             |
+-----------+------------------+
6 rows in set (0.00 sec)

mysql>

```

I *think* if I uninstall `mysql` server from the host I'll be able to connect to the docker database, but surely there's a better approach.

# Answer
> 1 votes
One way to achieve this is by using the MySQL socket to connect instead of port. For this you would have bind a mount to your container. You would anyway need a host directory bound to container to persist Database data.

* First create a directory on host to mount container `/var/lib/mysql`
* Use `-v` option in `docker run` to attach host path to container
* Connect from client using `-S` and point to `mysql.sock` socket file

Example:

```
mkdir /home/user/mysql-1

```

Then start the container so that `/home/user/mysql-1` will attach to container's `/var/lib/mysql`:

```
docker run --name mysql -d \
-v /home/user/mysql-1:/var/lib/mysql \
-e MYSQL_ROOT_PASSWORD=password \
mysql/mysql-server

```

Once the container is started, you can connect to it with:

```
mysql -h localhost -u user -p -S /home/user/mysql-1/mysql.sock

```

---
Tags: docker, networking, databases, mysql, ports
---


# Question
Title: How to skip rest of hosts when item found in host - Ansible
I have a list of hosts using aws\_ec2 and I want to find the first host where the ec2 instance has a specific docker container.

I created a play book with a list of tasks, which ultimately executes a curl command. I only want to execute it on one host but first I have to find that host based on if the host is running a specific docker image. I'm using a shell task to execute `docker ps | grep my-filter` to see if the host has the docker container named "my-filter", once that container is found on that particular host, I'd like to execute the rest of my tasks on that particular host.

I was looking at with\_first\_found but that seems to only apply to files.

Any help/pointers would be appreciated.

# Answer
Let's test the use-case with 10 remote hosts (test\_01 - test\_10) in the group *test* and user *my-user* in the hosts test\_03 and test\_07. To test the existence of the user, use the shell command

```
shell> cat /etc/passwd|grep my-user

```

For example the playbook

```
shell> cat playbook.yml
- hosts: test
  tasks:
    - shell: cat /etc/passwd|grep my-user
      register: result
      ignore_errors: true
    - set_fact:
        results: "{{ results|default({})|
                     combine({item: hostvars[item].result.rc}) }}"
      loop: "{{ ansible_play_hosts }}"
      run_once: true
    - debug:
        var: results
      run_once: true

```

gives

```
ok: [test_01] =>
  results:
    test_01: 1
    test_02: 1
    test_03: 0
    test_04: 1
    test_05: 1
    test_06: 1
    test_07: 0
    test_08: 1
    test_09: 1
    test_10: 1

```

---

To skip the rest of the hosts, when the first host is found, iterate the list of the hosts and test *rc*. Then add the host to a new group and run the next play with this group. For example,

```
shell> cat playbook.yml
- hosts: test
  vars:
    dresult:
      rc: 1
  tasks:
    - block:
        - shell: cat /etc/passwd|grep my-user
          register: result
          loop: "{{ ansible_play_hosts }}"
          delegate_to: "{{ item }}"
          ignore_errors: true
          when: (result|default(dresult)).rc == 1
        - set_fact:
            my_host: "{{ result.results|json_query('[?rc == `0`].item')|first }}"
        - debug:
            var: my_host
        - add_host:
            name: "{{ my_host }}"
            groups: my_group
      run_once: true

- hosts: my_group
  tasks:
    - debug:
        var: inventory_hostname

```

gives

```
shell> ansible-playbook playbook.yml

PLAY [test] ****

TASK [shell] ****
failed: [test_01 -> test_01] (item=test_01) => changed=true
  ansible_loop_var: item
  cmd: cat /etc/passwd|grep my-user
  delta: '0:00:00.023271'
  end: '2020-09-25 08:20:31.823168'
  item: test_01
  msg: non-zero return code
  rc: 1
  start: '2020-09-25 08:20:31.799897'
  stderr: ''
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
failed: [test_01 -> test_02] (item=test_02) => changed=true
  ansible_loop_var: item
  cmd: cat /etc/passwd|grep my-user
  delta: '0:00:00.017642'
  end: '2020-09-25 08:20:33.386657'
  item: test_02
  msg: non-zero return code
  rc: 1
  start: '2020-09-25 08:20:33.369015'
  stderr: ''
  stderr_lines: <omitted>
  stdout: ''
  stdout_lines: <omitted>
changed: [test_01 -> test_03] => (item=test_03)
skipping: [test_01] => (item=test_04)
skipping: [test_01] => (item=test_05)
skipping: [test_01] => (item=test_06)
skipping: [test_01] => (item=test_07)
skipping: [test_01] => (item=test_08)
skipping: [test_01] => (item=test_09)
skipping: [test_01] => (item=test_10)
...ignoring

TASK [set_fact] ****
ok: [test_01]

TASK [debug] ****
ok: [test_01] =>
  my_host: test_03

TASK [add_host] ****
changed: [test_01]

PLAY [my_group] ****

TASK [debug] ****
ok: [test_03] =>
  inventory_hostname: test_03

PLAY RECAP *****
test_01: ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=1
test_03: ok=1 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0

```
> 2 votes

---
Tags: ansible
---


# Question
Title: hello world with flask and redis using docker networking
The sample from Docker, a "hello world" with `redis` and `flask`, seems to build fine, but, as an exercise can it be broken into two components?

> The redis service uses a public Redis image pulled from the Docker Hub registry.

and

> Compose pulls a Redis image, builds an image for your code, and starts the services you defined. In this case, the code is statically copied into the image at build time.

What I have in mind is to use the default `redis` image as its own container, and then modify the `flask` app so that it connects to the `redis` image. This is reasonable?

# Answer
Yes, it's absolutely reasonable and this is how real apps work. This is the microservice approach which allows to independently scale instances of a service and a DB.

What you have to do is to run your Flask and Redis services in a way so the first one can achieve the second one. If they are on the same machine, you just have to open Redis port used by your Flask app (probably 6379):

```
docker run -d -p 192.168.10.20:6379:6379 your_redis_image

```

And specify IP address and port of the Redis container in your Flask app. This is better to be done with a config file like the following `yaml`:

```
redis:
  host: '172.16.10.20'
  pw: 'YourPassWord'
  port: 6379
  db: 0

```

And use it in the code:

```
import yaml, redis

cfg_file open('config.yaml', 'r')
cfg = yaml.load(cfg_file)
redcfg = cfg['redis']

r = redis.Redis(
    host=redcfg['host'], port=redcfg['port'],
    db=redcfg['db'], password=redcfg['pw'],
)

```

Don't forget to copy this file into container to be accessible from the inside.
> 2 votes

---
Tags: docker, docker-compose, containers, flask, redis
---


# Question
Title: gitlab CI job fails after manual ftp deploy (lftp) command completes successfully
**Scenario**

I'm using LFTP to deploy files via sftp. The transfer completes successfully and all files get to the destination but for some reason the job fails.

**Question**

How can I find out what is causing my `deploy-stage` job to fail?

**Attempts**

* commented out `when` \- no success
* tried a couple variations of `lftp` usage - no success

---

*Gitlab CI .yml*

```
image: node:10.19.0 # https://hub.docker.com/_/node/
# image: node:latest

cache:
  # untracked: true
  key: my-project
  # key: ${CI_COMMIT_REF_SLUG} # per branch
  # key:
  #   files:
  #     - package-lock.json # only update cache when this file changes (not working) @jkr
  paths:
    - .npm/
    - node_modules
    - build

stages:
  - prepare # prepares builds, makes build needed for testing
  - test # uses test:build specifically @jkr
  - build
  - deploy

# before_install:

before_script:
  - npm ci --cache .npm --prefer-offline

prepare:
  stage: prepare
  needs: []
  script:
    - npm install
    - npm run build:test

test:
  stage: test
  needs: [prepare]
  except:
    - schedules
  tags:
    - linux
  script:
    - npm run test:ci-deps
    - npm run test:e2e # runs puppeteer tests @jkr
  artifacts:
    reports:
      junit: junit.xml
    paths:
      - coverage/

build:
  stage: build
  needs: [prepare]
  only:
    - schedules
  before_script:
    - apt-get update && apt-get install -y zip
  script:
    - npm run build:stage
    - zip -r build.zip build
  # cache:
  #   paths:
  #     - build
  #   <<: *global_cache
  #   policy: push
  artifacts:
    paths:
      - build.zip

deploy-stage:
  stage: deploy
  needs: [build]
  tags: [linux]
  only:
    - schedules
  #   # - branches@gitlab-org/gitlab
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    # temporarily using 'verify-certificate no'
    # for more on verify-certificate @jkr: https://www.versatilewebsolutions.com/blog/2014/04/lftp-ftps-and-certificate-verification.html
    # variables do not work with 'single quotes' unless they are "'surrounded by doubles'"
    - lftp -e "set ssl:verify-certificate no; open my-project.com; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye"
  environment:
    name: staging
    url: http://dev.my-project.com/projects/my-project/build
    # url: https://stg2.my-project.org
  when: manual
  # allow_failure: true

deploy-prod:
  stage: deploy
  needs: [build]
  tags: [linux]
  only:
    - schedules
    # - master
  before_script:
    - apt-get update && apt-get install -y lftp
  script:
    - sh deploy-prod
  environment:
    name: production
    url: http://www.my-project.com
  when: manual
  # allow_failure: true

```

*Job Output*

```
Running with gitlab-runner 13.2.2 (a998cacd)
  on docker-auto-scale fa6cab46
Preparing the "docker+machine" executor
00:40
Using Docker executor with image node:10.19.0 ...
Pulling docker image node:10.19.0 ...
Using docker image sha256:01b816051d343a5aaa3b33e165f6b6d620b7e899961bf6e8955a7577a0129873 for node:10.19.0 ...
Preparing environment
00:04
Running on runner-fa6cab46-project-13332271-concurrent-0 via runner-fa6cab46-srm-1596764203-a2a74b3b...
Getting source from Git repository
00:06
$ eval "$CI_PRE_CLONE_SCRIPT"
Fetching changes with git depth set to 50...
Initialized empty Git repository in /builds/my-project/.git/
Created fresh repository.
Checking out d4675c89 as pipeline-deploy-fail...
Skipping Git submodules setup
Restoring cache
00:26
Checking cache for my-project...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/13332271/my-project
Successfully extracted cache
Executing "step_script" stage of the job script
00:25
$ apt-get update && apt-get install -y lftp
Ign:1 http://deb.debian.org/debian stretch InRelease
Get:2 http://security.debian.org/debian-security stretch/updates InRelease [53.0 kB]
Get:3 http://deb.debian.org/debian stretch-updates InRelease [93.6 kB]
Get:4 http://deb.debian.org/debian stretch Release [118 kB]
Get:5 http://deb.debian.org/debian stretch Release.gpg [2410 B]
Get:6 http://security.debian.org/debian-security stretch/updates/main amd64 Packages [546 kB]
Get:7 http://deb.debian.org/debian stretch-updates/main amd64 Packages [2596 B]
Get:8 http://deb.debian.org/debian stretch/main amd64 Packages [7080 kB]
Fetched 7895 kB in 2s (3798 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  lftp
0 upgraded, 1 newly installed, 0 to remove and 65 not upgraded.
Need to get 711 kB of archives.
After this operation, 2235 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian stretch/main amd64 lftp amd64 4.7.4-1 [711 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 711 kB in 0s (5324 kB/s)
Selecting previously unselected package lftp.
(Reading database ... 29960 files and directories currently installed.)
Preparing to unpack .../lftp_4.7.4-1_amd64.deb ...
Unpacking lftp (4.7.4-1) ...
Setting up lftp (4.7.4-1) ...
$ lftp -e "set ssl:verify-certificate no; open my-project.com; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose build/ /var/www/domains/dev/projects/my-project/build/; bye"
Transferring file `asset-manifest.json'
Transferring file `favicon.ico'
Transferring file `index.html'
Transferring file `manifest.json'
Transferring file `precache-manifest.be0a0d937dde3502acb8ad9c21f53a42.js'
Transferring file `service-worker.js'
Making directory `images'
Transferring file `images/content-block-arrow.png'
Transferring file `images/logo-mobile.png'
Transferring file `images/primary-nav-bg.png'
Transferring file `images/secondary-nav-bg.png'
Transferring file `images/social-footer.png'
Transferring file `images/social-media-facebook.png'
Transferring file `images/social-media-instagram.png'
Transferring file `images/social-media-twitter.png'
Transferring file `images/social-media-youtube.png'
Making directory `static'
Making directory `static/css'
Transferring file `static/css/2.e01eb902.chunk.css'
Transferring file `static/css/2.e01eb902.chunk.css.map'
Transferring file `static/css/main.a760ea4c.chunk.css'
Transferring file `static/css/main.a760ea4c.chunk.css.map'
Making directory `static/js'
Transferring file `static/js/2.1b305e72.chunk.js'
Transferring file `static/js/2.1b305e72.chunk.js.LICENSE.txt'
Transferring file `static/js/2.1b305e72.chunk.js.map'
mirror: Access failed: 552 10099096: No space left on device (2.1b305e72.chunk.js.map)
Transferring file `static/js/main.9394eb63.chunk.js'
Transferring file `static/js/main.9394eb63.chunk.js.map'
Transferring file `static/js/runtime-main.76c0dd92.js'
Transferring file `static/js/runtime-main.76c0dd92.js.map'
Making directory `static/media'
Transferring file `static/media/GothamBook.b31b3549.woff2'
Transferring file `static/media/GothamMedium.39e89370.woff2'
Transferring file `static/media/Plantin.0e23fafd.woff2'
Transferring file `static/media/logo.81af0ace.png'
Transferring file `static/media/revicons.04eb8fc5.woff'
Transferring file `static/media/revicons.17629a5d.ttf'
Transferring file `static/media/revicons.2feb69cc.eot'
ERROR: Job failed: exit code 1

```

# Answer
**Thanks to this answer**

I fixed this by adding the following to my lftp command

```
set ftp:use-allo false; set ftp:passive-mode true; set ftp:prefer-epsv false;

```

which made my finished command the following

```
lftp -e "set ftp:use-allo false; set ftp:passive-mode true; set ftp:prefer-epsv false; set ssl:verify-certificate no; open mediajackagency.com; user $LFTP_USERNAME $LFTP_PASSWORD; mirror --reverse --verbose --delete-first build/ /var/www/domains/dev/clients/jhm/cornerstone-mvp/build/; bye"

```
> 0 votes

# Answer
Maybe solving space problem first.

Then, in my opinion, I prefer using rsync for copy file to server.
> 2 votes

---
Tags: continuous-integration, gitlab, continuous-deployment, yaml
---


# Question
Title: How do I make a redis container accessible through bridge networking from the host?
I'd like to be able to connect directly to the `redis` container below:

```
root $
root $ docker images redis
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
redis               alpine              bd71e6db4a54        2 weeks ago         32.2MB
redis               latest              84c5f6e03bf0        2 weeks ago         104MB
root $
root $ docker run --name rd -d -p 99:99 redis:latest
ac4b8b2b032b2c2911f03fd4f6af649f470620cd29eb566b821e5ce738d9ac11
root $
root $ docker exec -ti --user root rd bash
root@ac4b8b2b032b:/data#
root@ac4b8b2b032b:/data# redis-cli
127.0.0.1:6379>
127.0.0.1:6379> exit
root@ac4b8b2b032b:/data#
root@ac4b8b2b032b:/data# exit
exit
root $
root $ docker network ls
NETWORK ID          NAME                    DRIVER              SCOPE
74c94db9195c        basexhttpnetwork        bridge              local
a5efbdf4bc39        bridge                  bridge              local
4b6586d1bc8c        frappe_docker_default   bridge              local
1b875368faa0        helloworld_default      bridge              local
3bc6659535b4        host                    host                local
e877e399bf78        none                    null                local
root $
root $ docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS                          NAMES
ac4b8b2b032b        redis:latest        "docker-entrypoint.s…"   6 minutes ago       Up 6 minutes        0.0.0.0:99->99/tcp, 6379/tcp   rd
root $

```

using the `redis-cli` tool from the host. So I would need to add or attach the `rd` (for `redis`) container to the bridge driver? Then specify an IP address and port from the host?

# Answer
> 1 votes
By default Docker uses the `bridge` network.

On a default Docker installation you can simply map a port to the container's service port. The Redis Docker image exposes its service on port `6379` so you can do:

`$ docker run --name rd -d -p 6379:6379 redis:latest`

From your host you can now run use the `redis-cli` utility:

```
$ redis-cli
127.0.0.1:6379> ping
PONG
127.0.0.1:6379>

```

# Answer
> 0 votes
tentative answer:

```
root $
root $ docker run --name rds -d redis redis-server --appendonly yes
d85ce8736fc33f81edc7e63972391f6eb34586d562b4066f57395f9ea2bafe58
root $
root $ docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
d85ce8736fc3        redis               "docker-entrypoint.s…"   9 seconds ago       Up 6 seconds        6379/tcp            rds
root $
root $ docker exec -it rds redis-cli
127.0.0.1:6379>
127.0.0.1:6379> keys *
(empty array)
127.0.0.1:6379>
127.0.0.1:6379> set foo bar
OK
127.0.0.1:6379>
127.0.0.1:6379> keys *
1) "foo"
127.0.0.1:6379>
127.0.0.1:6379> get foo
"bar"
127.0.0.1:6379>

```

should work as persistence in a microservice architecture. To my understanding.

```
root $
root $ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
e57e2c862b19        bridge              bridge              local
25aba55a5413        host                host                local
ad5c1c2fa1e1        none                null                local
root $
root $ docker inspect rds | grep bridge
                "bridge": {
root $
root $ docker network inspect bridge
[
    {
        "Name": "bridge",
        "Id": "e57e2c862b19e68b470adf5bbd4719b675c15a399e3bd01380a9d7ad35c7ea26",
        "Created": "2020-09-24T17:43:14.098411476-07:00",
        "Scope": "local",
        "Driver": "bridge",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "172.17.0.0/16",
                    "Gateway": "172.17.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "d85ce8736fc33f81edc7e63972391f6eb34586d562b4066f57395f9ea2bafe58": {
                "Name": "rds",
                "EndpointID": "b0afddd9a063d8051021db490342e6627bdf9694b8f526ad5bbd5a665eeafefd",
                "MacAddress": "02:42:ac:11:00:02",
                "IPv4Address": "172.17.0.2/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.bridge.default_bridge": "true",
            "com.docker.network.bridge.enable_icc": "true",
            "com.docker.network.bridge.enable_ip_masquerade": "true",
            "com.docker.network.bridge.host_binding_ipv4": "0.0.0.0",
            "com.docker.network.bridge.name": "docker0",
            "com.docker.network.driver.mtu": "1500"
        },
        "Labels": {}
    }
]
root $

```

So a `flask` container also on the bridge network should be able to access this container.

---
Tags: docker, networking, databases, ports, redis
---


# Question
Title: How to run a single one-time command on a kubernetes instance/pod
Currently I have a setup on AWS with 4 EC2 machines. They provide a web interface that among other things also has a database search. My recent project was to move the database search to AWS Elastic Search. That took a while but in the end it was fine, because to create the initial index I simply had to SSH into one of the machines, run the command to create the index and I was done.

I could not have done it without SSH'ing into it because the production environment (meaning DB and ES instances) is not directly accessible from dev PCs (which makes sense to me).

So, being quite new to Kubernetes my question is: if we were to migrate these EC2 instances and switch to a Kubernetes kind of setup, how would I run the initial command to create the ES index?

I know I can connect to a pod when it's a single container but these would most likely have containers at least for nginx, redis and django (the one I'd want to connect).

# Answer
> 9 votes
A little more information is needed about specific operations you are doing, but generally your options are:

1. For one-time operations: use kubectl exec to get into containers and do commands (works similar to `docker exec`).
2. Also for one-time operations, spawn separate pods with shell access to interact with existing pods, something along the lines:

```
kubectl run -it --rm --restart=Never --image busybox tempbusybox -- sh

```
3. For reproducible (not one-time) operations, you may use kubernetes jobs.
4. In some cases you may use init containers for setup tasks.

---
Tags: kubernetes, ssh
---


# Question
Title: Is it possible/fine to run Prometheus, Loki, Grafana outside of Kubernetes?
In some project there are scaling and orchestration implemented using technologies of a local cloud provider, with no Docker & Kubernetes. But the project has poor logging and monitoring, I'd like to instal Prometheus, Loki, and Grafana for metrics, logs, and visualisation respectively. Unfortunately, I've found no articles with instructions about using Prometheus without K8s.

But is it possible? If so, is it a good was? I know that Prometheus & Loki can automatically detect services in the K8s to extract metrics and logs, but will the same work for a custom orchestration system?

# Answer
I've found out that it is a possible and fine approach. Prometheus support different methods of service discovery, this is explained in the docs: Configuration. As I understood the Loki way of work, it just uses Promtail which sends logs from services, so there is no need of service discovery. An example of Promtail installation is here. Also, more details can be found in the doc: Promtail install.
> 1 votes

# Answer
Yes--absolutely you can skip Kubernetes. Suggest that you still use Docker, but you could even just install the components separately.

I think it's easiest using Docker Compose (need to have package `docker-compose` installed on your host) to manage the containers.

Here is a `docker-compose.yml` that might be helpful:

```
version: '3'

services:
    prometheus:
        image: prom/prometheus:latest
        restart: unless-stopped
        ports:
            - "9090:9090"
        volumes:
            - /home/${USER}/opt/prometheus/data:/opt/prometheus/data:rw
            - /home/${USER}/opt/prometheus/config:/opt/prometheus/config:rw
        environment:
            SERVER_KEY: ${SERVER_KEY}
            SERVER_CERT: ${SERVER_CERT}

    grafana:
        image: grafana/grafana:latest
        restart: unless-stopped
        ports:
            - "3000:3000"
        volumes:
            - /home/${USER}/var/lib/grafana:/var/lib/grafana:rw
            - /home/${USER}/var/log/grafana:/var/log/grafana:rw
            - /home/${USER}/etc/grafana/grafana.ini:/etc/grafana/grafana.ini:rw
            - /home/${USER}/etc/grafana/ldap.toml:/etc/grafana/ldap.toml:rw
        environment:
            SERVER_KEY: ${SERVER_KEY}
            SERVER_CERT: ${SERVER_CERT}
            GF_SECURITY_ADMIN_USER: ${GF_SECURITY_ADMIN_USER}
            GF_SECURITY_ADMIN_PASSWORD: ${GF_SECURITY_ADMIN_PASSWORD}
        depends_on:
            - "prometheus"

```

Can also use Docker Compose for Loki and Promtail.

Docker Compose reference: https://docs.docker.com/compose/compose-file
> 0 votes

---
Tags: kubernetes, prometheus, orchestration, grafana
---


# Question
Title: Extended Choice Parameter: How to load params dynamically from any file for jenkins build job
Though below code works fine individually:

```
extendedChoice description: '', multiSelectDelimiter: ',', name: 'Capablities', quoteValue: false, saveJSONParameterToFile: false, type: 'PT_CHECKBOX',value: 'Chrome_macos10_15, Chrome_win10', visibleItemCount: 30

```

I need Chrome\_macos10\_15, Chrome\_win10 need to be passed from external file.

How can it be achieved?

# Answer
Use the Active Choice Parameter plugin which allows you to run a Groovy script to dynamically load values for your parameters.
> 2 votes

---
Tags: jenkins-pipeline
---


# Question
Title: The ansible_local var is not being populated when using the setup module to gather custom facts on windows
The ansible\_local var is not being populated when using the setup module to gather custom facts on windows.

**Playbook:**

```
---
- name: Testing and Debug PB
  hosts: all
  tasks:

    - name: Gather Facts
      setup:
        fact_path: 'c:\custom_facts'

    - debug:
        msg: "{{ hostvars[inventory_hostname] }}"

    - debug:
        msg: "{{ ansible_local }}"

```

**Contents of "C:\custom\_facts\foo.fact" on remote host:**

```
[general]
asdf=1
bar=2

```

Playbook Output:

```
ansible-playbook -i inv/dev 301_pb_testing.yml -vv
ansible-playbook 2.9.12
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible-playbook
  python version = 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]
Using /etc/ansible/ansible.cfg as config file

PLAYBOOK: 301_pb_testing.yml ************************************************************************************************************************************************1 plays in 301_pb_testing.yml

PLAY [Testing and Debug PB] *************************************************************************************************************************************************
TASK [Gathering Facts] ******************************************************************************************************************************************************task path: /home/alex/git.sigma/SDG-Tech-DevOps/ansible/301_pb_testing.yml:2
ok: [demi-win00]
META: ran handlers

TASK [facts : Gather Facts] *************************************************************************************************************************************************task path: /home/alex/git.sigma/SDG-Tech-DevOps/ansible/roles/facts/tasks/main.yml:14
ok: [demi-win00]

TASK [debug] ****************************************************************************************************************************************************************task path: /home/alex/git.sigma/SDG-Tech-DevOps/ansible/301_pb_testing.yml:15

---------REMOVED SOME HOSTVAR OUTPUT FOR BREVITY AND PRIVACY--------

ok: [demi-win00] =>
  msg: |-
    ansible_architecture: 64-bit
    ansible_bios_date: 03/31/2014
    ansible_bios_version: rel-1.13.0-0-gf21b5a4aeb02-prebuilt.qemu.org
    ansible_check_mode: false
    ansible_connection: winrm
    ansible_date_time:
  ......
    ansible_facts:
   ......
    ansible_kernel: 10.0.17763.0
    ansible_lastboot: '2020-08-29 11:07:46Z'
    ansible_machine_id: S-1-5-21-49790847-4020553596-361474921
   ......

---------REMOVED SOME HOSTVAR OUTPUT FOR BREVITY AND PRIVACY--------

TASK [debug] ****************************************************************************************************************************************************************task path: /home/alex/git.sigma/SDG-Tech-DevOps/ansible/301_pb_testing.yml:18
fatal: [demi-win00]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: 'ansible_local' is undefined

    The error appears to be in '/home/alex/git.sigma/SDG-Tech-DevOps/ansible/301_pb_testing.yml': line 18, column 7, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:

        - debug:
          ^ here

```

```
ansible 2.9.12
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/alex/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.6/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.6.9 (default, Jul 17 2020, 12:50:27) [GCC 8.4.0]

```

The ansible Controller is running debian, The remote is Windows Server 2019

I would expect ansible\_local to be populated once setup is run over a valid custom fact, but It doesn't. What am I doing wrong?

# Answer
According to documentation:

> NOTE - For windows clients, the results will be added to a variable named after the local file (without extension suffix), rather than ansible\_local.

According to your example it should be in `ansible_foo`, but take into consideration that the files in the path must be PowerShell scripts (with extension .ps1) so maybe `foo.fact` wasn't actually executed.

Hope it helps.
> 1 votes

---
Tags: ansible, windows
---


# Question
Title: Include pre- and post_tasks transparently for all Ansible roles
I am looking for a way to execute some actions before and after each role has run. `pre_tasks` and `post_tasks` look like a good candidate, but I have found no way to include them transparently for all defined roles/playbooks. I don't want to add these manually to all playbooks since this seems like a maintenance nightmare.

Another angle to solve this might be to abuse the fact that Ansible logs in via ssh. However I am not aware of any identifying properties of an Ansible login as opposed to a regular user login.

My specific issue is: I want to trigger etckeeper before and after each execution of a role.

If you have any other "out-of-the-box" ideas to solve this, I will be grateful to hear them.

# Answer
> 1 votes
Not tested this, but is include\_role any help? so instead of a playbook like this,

```
 ---
 - name: Example
   hosts: all
   roles:
    - first_role
    - second_role
    - third_role

```

maybe this,

```
 ---
 - name: Example
   hosts: all
   tasks:
    - etckeeper_task
    - include_role:
        name: first_role
    - etckeeper_task
    - include_role:
        name: second_role
    - etckeeper_task
    - include_role:
        name: third_role
    - etckeeper_task

```

alternatively (also untested) you could build a role, let's call it `wrap_role` with `tasks/main.yml` containing,

```
---
# Replace this task with your actual before-role etckeeper task,
- name: etckeeper pre-actions
  meta: noop

- include_role:
  name: {{role_name}}

# Replace this task with your actual after-role etckeeper task,
- name: etckeeper post-actions
  meta: noop

```

Now your top-level playbook can be something like,

```
 ---
 - name: Example
   hosts: all
   roles:
     - { role: wrap_role, role_name: first_role }
     - { role: wrap_role, role_name: second_role }
     - { role: wrap_role, role_name: third_role }

```

not ideal based on your question, but at least it leaves the roles untouched, even if you have to alter top-level playbooks. If I get the time I'll test the above suggestions and update this answer accordingly.

---
Tags: ansible
---


# Question
Title: Unable to connect to Elasticsearch [ Error: Request Timeout after 30000ms ]
I have installed elasticsearch-oss 7.9.2 and kibana 7.9.2 on ubuntu 18.04 when I start elasticsearch service then it's started normally but when I start service kibana it's not starting and shows error '**Unable to connect to Elasticsearch**. Error: Request Timeout after 30000ms'

# Answer
> 1 votes
Put the following setting in `/etc/elasticsearch.yml` and it will work.

```
network.host: 0.0.0.0
discovery.type: single-node

```

---
Tags: elastic, kibana, elasticsearch
---


# Question
Title: How can I manage secrets in .tf and .tfstate?
I would like to use the Terraform MySQL Provider to keep a list of mysql users and grants handy for creating new test environments. The `.tf` and `.tfstate` files both seem to want to store the MySQL passwords in plaintext.

**Concerning .tf:**

It is my understanding that `.tf` files live in revision control and are maintained by a team. How does that practice differ when secrets are in the `.tf`? It is possible to encrypt these values at all?

**Concerning .tfstate:**

I can store the `.tfstate` securely somewhere after running Terraform apply, but it would be preferable for this use case to not store it at all?

# Answer
> 30 votes
Terraform supports adding an additional file with variables during invocation.

documentation: https://www.terraform.io/intro/getting-started/variables.html#from-a-file

We are using that feature to provide a `secrets.tfvars` file on each invocation of Terraform. We also use a script to wrap the command so that its invocation is consistent, and all team members avoid having to make the same mistakes. The wrapper synchronizes `.tfstate` with S3 before an execution, and pushes `.tfstate` back to S3 at the end. I also hear of people doing the same thing with state stored in Consul, even adding a kind of semaphore in consul to prevent two people from starting Terraform at the same time.

When you avoid setting a default value in a `variables.tf` file, it forces the user to input the value. It can be either entered manually or using the `-var-file` command option like described above. Not setting a default on your secrets is a good way to enforce changes that require a change in secrets.

The `secrets.tfvars` file is a symbolic link to one of the files with secrets which are not stored in version control. We have several, one per environment, like so `secrets-prod.tfvars`, `secrets-dev.tfvars`, `secrets-stg.tfvars`, etc...

An even better practice would be to generate these secrets files during the wrapper script based on data in Vault or some other way to *share* secrets. Since currently when the format of secrets changes, or secrets themselves, we need to communicate it to the team outside the version control channel - and this doesn't always work well, to be honest. But secrets do change infrequently.

# Answer
> 11 votes
If you're on AWS, then have a look at "The Right Way to Manage Secrets" by Segment.io on the AWS Blog. We advocate using `chamber` to all of our customers for managing secrets. It works by leveraging the AWS Systems Manager Parameter Store (SSM) together with KMS keys. This ensures secrets are encrypted at rest (and in transit), secured with IAM, auditable with CloudTrails, and only exposed as environment variables at run-time.

After configuring chamber and setting up the KMS key, we write the secrets to the parameter store.

```
chamber write db TF_VAR_DB_USER foobar
chamber write db TF_VAR_DB_PASS secret

```

Then use those secrets when you call terraform.

```
chamber exec db -- terraform plan

```

This assumes you've defined a variable called `DB_USER` and `DB_PASS` in your HCL code.

For example, you could add this to `variables.tf`

```
variable "DB_USER" { }
variable "DB_PASS" { }

```

**NOTE:** `chamber` will always export environment variables in uppercase

We provide a terraform module called `terraform-aws-kms-key` to make provisioning the KMS key easy. Check out our detailed documentation with examples of how to use `chamber` with multiple namespaces as well as how to use chamber with terraform to manage secrets. See our complete reference example for provisioning chamber dependencies.

As for `.tfstate`, you bring up a really good point about the existence of plain-text secrets in the state file. There's really no way around this. In order for terraform to calculate changes to build a plan, it needs to know the "before" and "after" state. For this reason, we recommend using an encrypted S3 bucket with mandatory versioning. Use the `terraform-aws-tfstate-backend` module to provision a bucket and DynamoDB locking table according to best practices.

# Answer
> 8 votes
We avoid terraform handle our secrets. Even if you manage to inject secrets by a var file "secrtes.tfvars" as pointed out above, these secrets will be stored in your terraform (remote-)state.

You can protect remote-state files by using e.g. S3 authorization, or you can gitignore local state files but we decided not to rely on this kind of protection.

# Answer
> 3 votes
To import secrets into .tf files, you can also use an external data source. This could be e.g. a script which decrypts your secrets.

# Answer
> 2 votes
I looked at a few different ways but I particularly liked git-crypt for an adhoc thing to do before implementing something bigger like Vault.

# Answer
> 0 votes
You can utilize the external data source to import secrets to Terraform and encrypt the secret files with `gpg`.

For instance, this what a terraform file would like for creating an rds instance with password `foobarbaz`

```
data "external" "rds" {
  program  = [ "cat", ".secrets/rds.json"]
}

resource "aws_db_instance" "default" {
  allocated_storage    = 20
  storage_type         = "gp2"
  engine               = "mysql"
  engine_version       = "5.7"
  instance_class       = "db.t2.micro"
  name                 = "mydb"
  username             = "foo"
  password             = "${data.external.rds.result.password}"
  parameter_group_name = "default.mysql5.7"
}

```

You can check this post that explains in detail how to do this.

**How to encrypt terraform secrets**

# Answer
> 0 votes
you can store values in the parameter store and use them in terraform resources by fetching values using data resources.

```
data "aws_ssm_parameter" "rds_password" {
  name = "/${var.project_name}/DB_PASSWORD"
}

resource "aws_db_instance" "rds" {
  ...
  password = data.aws_ssm_parameter.rds_password.value
  ...
}

```

---
Tags: terraform, security
---


# Question
Title: why is "docker run" used to connect to redis?
Trying to understand what's happening here:

```
root@mordor:~#
root@mordor:~# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
root@mordor:~#
root@mordor:~# docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
10d38d6f1fa1        bridge              bridge              local
5b5fde2dd7cc        foo                 bridge              local
1a7946d0b10c        host                host                local
0cbbb6d4f486        none                null                local
root@mordor:~#
root@mordor:~# docker run --name rd --network foo -d redis redis-server --appendonly yes
ea782770bcff816a642bc700abd2fadd524e3eeca10820e4c158ae4e72eda139
root@mordor:~#
root@mordor:~# docker container ls
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS              PORTS               NAMES
ea782770bcff        redis               "docker-entrypoint.s…"   42 seconds ago      Up 40 seconds       6379/tcp            rd
root@mordor:~#
root@mordor:~# docker run -it --network foo --rm redis redis-cli -h rd
rd:6379>
rd:6379> ping
PONG
rd:6379>
rd:6379> set foo bar
OK
rd:6379>
rd:6379> get foo
"bar"
rd:6379>
rd:6379> keys *
1) "foo"
rd:6379>

```

I can see how to spin up the community version of `redis`, I only added the `foo` network. But why is the connection with `run` rather than `exec`?

# Answer
In short, `docker exec` is used to operate an already-running container, while `docker run -it` starts a new container interactively.

When you ran `docker run -it --network foo --rm redis redis-cli -h rd` you created a 2nd container, this time using the `redis-cli` image, which was used to query your running `redis` container.

Were you to repeat the steps you took, and then open another terminal and run `docker container ls`, you would see two running containers.

Reference: https://spin.atomicobject.com/2018/10/04/docker-command-line/
> 2 votes

---
Tags: docker, networking, databases, configuration, redis
---


# Question
Title: s3fs to mount S3 bucket with iamrole on non-aws machine
I am trying to mount S3 bucket on my on-premises CentOS-6 server using s3fs as below:

```
[roots@myhost ~]# s3fs bucket-name /mnt/s3mnt -o  uid=1001,gid=1001,allow_other,umask=022,iam_role=iam-role,use_cache=/tmp

```

This gives error :

```
s3fs: unable to access MOUNTPOINT /mnt/s3mnt: Transport endpoint is not connected
debug :
    set_moutpoint_attribute(3293): PROC(uid=0, gid=0) - MountPoint(uid=0, gid=0, mode=40777)
s3fs_init(2597): init
s3fs_check_service(2896): check services.
RequestPerform(1524): ### CURLE_OPERATION_TIMEDOUT
    RequestPerform(1617): ### retrying...
    RemakeHandle(1288): Retry request. [type=-1][url=http://169.254.169.254/latest/meta-data/iam/security-credentials/iam-role][path=]
    RequestPerform(1620): Failed to reset handle and internal data for retrying.
s3fs: Failed to check IAM role name(iam-role).

```

Please let me know if somebody did it successfully. Tried with endpoint parameter and URL parameter but it says:

```
fuse: unknown option `endpoint=ap-northeast-1'

```

Any suggestion is highly appreciated. Thank you.

# Answer
> 1 votes
Have you got AWS credentials configured on that machine?

```
~ $ aws configure

```

And then verify with:

```
~ $ aws sts get-caller-identity

```

Hope that helps :)

# Answer
> 0 votes
Following worked for me - `/usr/bin/s3fs my-bucket /my-mount/ -o uid=500,gid=501,iam_role=MyRole,use_cache=/tmp,endpoint=ap-south-1,url=https://s3.amazonaws.com`

---
Tags: amazon-web-services, amazon-s3, s3fs
---


# Question
Title: Ansible: Detecting in parent role when meta dependency role had change
I would like to make an ansible role that drops some credentials into /etc for a few services that are running. Each service is slightly different and requires different mechanisms for restarting to pick up the new credentials.

Is it possible to have a meta dependency somehow communicate back up to the consumer/parent to trigger a handler to restart the services that depend on the file?

# Answer
> 1 votes
I think you can use `listen` as explained here: https://medium.com/@george.shuklin/listen-feature-for-handlers-in-ansible-29183524c7e1

---
Tags: ansible
---


# Question
Title: iptables rules to permit GitHub Actions from breaking (chains default to DROP)
Assuming my iptables rules default to DROP on the INPUT and OUTPUT chains, what is the bare minimum set of rules that I must add to my chains to prevent a script running in GitHub Actions from stalling indefinitely?

I'm using (free) GitHub Actions for my open-source application's CI/CD infrastructure. When I push changes to github.com, it automatically spins up an Ubuntu 18.04 linux server in Microsoft's cloud that checks-out my repo and executes a BASH script to build my application.

For security reasons, early on in my build script I install and setup some very restrictive `iptables` rules that default to `DROP` on the `INPUT` and `OUTPUT` chains. I poke a hole in the firewall for `127.0.0.1`, `RELATED/ESTABLISHED` on `INPUT`, and only permit the `_apt` user to send traffic through `OUTPUT`.

This works great when I run the build script in a docker container on my local system. But--as I just learned--when it runs with GitHub Actions, it stalls indefinitely. Clearly, the instance itself needs to be able to communicate out to GitHub's servers in order to finish. And I appear to have broken that.

So the question is: what `-j ACCEPT` rules should I add to my `iptables` `INPUT` and `OUTPUT` chains to only permit the bare necessities for GitHub Actions executions to proceed as usual?

For reference, here's the snippet from my build script that sets-up my firewall:

```
##################
# SETUP IPTABLES #
##################

# We setup iptables so that only the apt user (and therefore the apt command)
# can access the internet. We don't want insecure tools like `pip` to download
# unsafe code from the internet.

${SUDO} iptables-save > /tmp/iptables-save.`date "+%Y%m%d_%H%M%S"`
${SUDO} iptables -A INPUT -i lo -j ACCEPT
${SUDO} iptables -A INPUT -s 127.0.0.1/32 -j DROP
${SUDO} iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} iptables -A INPUT -j DROP
${SUDO} iptables -A OUTPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j ACCEPT
${SUDO} iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} iptables -A OUTPUT -m owner --uid-owner 100 -j ACCEPT # apt uid = 100
${SUDO} iptables -A OUTPUT -j DROP

${SUDO} ip6tables-save > /tmp/ip6tables-save.`date "+%Y%m%d_%H%M%S"`
${SUDO} ip6tables -A INPUT -i lo -j ACCEPT
${SUDO} ip6tables -A INPUT -s ::1/128 -j DROP
${SUDO} ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} ip6tables -A INPUT -j DROP
${SUDO} ip6tables -A OUTPUT -s ::1/128 -d ::1/128 -j ACCEPT
${SUDO} ip6tables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} ip6tables -A OUTPUT -m owner --uid-owner 100 -j ACCEPT
${SUDO} ip6tables -A OUTPUT -j DROP

# attempt to access the internet as root. If it works, exit 1
curl -s 1.1.1.1
if [ $? -eq 0 ]; then
        echo "ERROR: iptables isn't blocking internet access to unsafe tools. You may need to run this as root (and you should do it inside a VM)"
        exit 1
fi

```

# Answer
This can be achieved by running your build script in a docker container and applying your iptables rules inside that container, which won't affect the host `runner`'s connectivity.

For example, if the below script is executed in a GitHub Actions job (in the Ubuntu 18.04 GitHub shared runner), it will run the build script (`docker_script.sh`) in a debian docker container that has no internet connectivity, except from the `_apt` user.

```
#!/bin/bash
set -x

###################
# INSTALL DEPENDS #
###################

apt-get -y install docker.io

##################
# DOWNLOAD IMAGE #
##################

# At the time of writing, Docker Content Trust is 100% security theater without
# explicitly adding the root public keys to the $HOME/.docker/trust/ directory
#
#  * https://github.com/BusKill/buskill-app/issues/6#issuecomment-700050760
#  * https://security.stackexchange.com/questions/238529/how-to-list-all-of-the-known-root-keys-in-docker-docker-content-trust
#  * https://github.com/docker/cli/issues/2752

docker -D pull debian:stable-slim

#################
# CREATE SCRIPT #
#################

tmpDir=`mktemp -d`
pushd "${tmpDir}"

cat << EOF > docker_script.sh
#!/bin/bash
set -x

# SETTINGS #
SUDO=/usr/bin/sudo

# DEPENDS #
${SUDO} apt-get update
${SUDO} apt-get install iptables curl

# IPTABLES #

# We setup iptables so that only the apt user (and therefore the apt command)
# can access the internet. We don't want insecure tools like `pip` to download
# unsafe code from the internet.

${SUDO} iptables-save > /tmp/iptables-save.`date "+%Y%m%d_%H%M%S"`
${SUDO} iptables -A INPUT -i lo -j ACCEPT
${SUDO} iptables -A INPUT -s 127.0.0.1/32 -j DROP
${SUDO} iptables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} iptables -A INPUT -j DROP
${SUDO} iptables -A OUTPUT -s 127.0.0.1/32 -d 127.0.0.1/32 -j ACCEPT
${SUDO} iptables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} iptables -A OUTPUT -m owner --uid-owner 100 -j ACCEPT # apt uid = 100
${SUDO} iptables -A OUTPUT -j DROP

${SUDO} ip6tables-save > /tmp/ip6tables-save.`date "+%Y%m%d_%H%M%S"`
${SUDO} ip6tables -A INPUT -i lo -j ACCEPT
${SUDO} ip6tables -A INPUT -s ::1/128 -j DROP
${SUDO} ip6tables -A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} ip6tables -A INPUT -j DROP
${SUDO} ip6tables -A OUTPUT -s ::1/128 -d ::1/128 -j ACCEPT
${SUDO} ip6tables -A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
${SUDO} ip6tables -A OUTPUT -m owner --uid-owner 100 -j ACCEPT
${SUDO} ip6tables -A OUTPUT -j DROP

# attempt to access the internet as root. If it works, exit 1
curl 1.1.1.1
if [ $? -eq 0 ]; then
        echo "ERROR: iptables isn't blocking internet access to unsafe tools. You may need to run this as root (and you should do it inside a VM)"
        exit 1
fi

# BUILD #

# ...
# <DO BUILD HERE>
# ...

exit 0
EOF
chmod +x docker_script.sh

##############
# DOCKER RUN #
##############

docker run --rm --cap-add "NET_ADMIN" -v "${tmpDir}:/root/shared_volume" debian:stable-slim /bin/bash -c "cd /root/shared_volume && docker_script.sh"

# exit cleanly
exit 0

```

Note that:

1. You have to execute the `docker run` command manually rather than just specify the `container:` in the GitHub Actions yaml file in order to add the `NET_ADMIN` capability. See Also https://stackoverflow.com/questions/63959783/how-to-run-script-in-docker-container-with-additional-capabilities-docker-exec/64045054#64045054
2. This is a security risk unless you pin the root signing keys before calling `docker pull`. See also https://security.stackexchange.com/questions/238529/how-to-list-all-of-the-known-root-keys-in-docker-docker-content-trust
3. The above script should be executed as root. For example, prepend `sudo` to it in the `run:` key for the step in the GitHub Actions workflow.
> 1 votes

---
Tags: continuous-integration, networking, github
---


# Question
Title: Ansible fails in Jenkins pipeline when run from docker
I'm trying to deploy a build via Jenkins pipeline using `agent` `docker` and Ansible playbook but it fails on `Gathering Facts` stage as shown below:

```
TASK [Gathering Facts] *********************************************************
fatal: [destination.box.local]: UNREACHABLE! => {"changed": false, "msg": "argument must be an int, or have a fileno() method.", "unreachable": true}

```

Similar Jenkins pipeline using `agent` `any` and Ansible not from docker (local installation) will do the job w/o any hiccups.

Agent section from Jenkins pipeline looks like:

```
pipeline {

    agent {
       docker {
          image 'artifactory.devbox.local/docker-local/myrepo/jdk8:latest'
          args '-v $HOME/.m2:/root/.m2 -v /etc/ansible:/etc/ansible -v $HOME/.ansible/tmp:/.ansible/tmp -v $HOME/.ssh:/root/.ssh'
       }
    }

   parameters {
      string(
              name: 'PROFILE',
              defaultValue: 'dev',
              description: 'Select profile to build?'
      )
      string(
              name: 'HOST',
              defaultValue: 'dev',
              description: 'Which Outbound Host(s) group to deploy to?'
      )
   }

   stages {

      stage('Artifactory configuration') {
         steps {
            rtMavenDeployer(
                    id: "MAVEN_DEPLOYER",
                    serverId: 'Artifactory',
                    releaseRepo: "libs-release-local",
                    snapshotRepo: "libs-snapshot-local"
            )

            rtMavenResolver(
                    id: "MAVEN_RESOLVER",
                    serverId: 'Artifactory',
                    releaseRepo: "libs-release",
                    snapshotRepo: "libs-snapshot"
            )
         }
      }

      stage('Maven build') {
         steps {
            withEnv(['JAVA_HOME=/usr/lib/jvm/java-1.8-openjdk', 'MAVEN_HOME=/usr/share/maven']) {
               rtMavenRun(
                       pom: 'pom.xml',
                       goals: 'clean',
                       deployerId: "MAVEN_DEPLOYER",
                       resolverId: "MAVEN_RESOLVER"
               )
            }
         }
      }

      stage ('Publish build') {
         steps {
            rtPublishBuildInfo (
                    serverId: 'Artifactory',
            )
         }
      }

      stage('deploy') {
         when{
            anyOf {
               branch 'master'
               branch 'develop'
            }
         }
         steps {
            ansiblePlaybook(
                    playbook: 'playbook.yml',
                    extraVars: [
                            profile: "${params.PROFILE}",
                            host: "${params.HOST}"
                    ]
            )
         }
      }

   }
}

```

After adding `ansible_ssh_common_args='-o StrictHostKeyChecking=no'` to the Ansible inventory (or setting `host_key_checking = False` in the config) I have got that error:

```
TASK [Gathering Facts] *********************************************************
fatal: [destination.box.local]: UNREACHABLE! => {"changed": false, "msg": "'getpwuid(): uid not found: 700'", "unreachable": true}
fatal: [ansible_ssh_common_args=-o StrictHostKeyChecking=no]: UNREACHABLE! => {"changed": false, "msg": "[Errno -3] Try again", "unreachable": true}

```

Any thought what I need to add to it to let Ansible run a playbook?

# Answer
> 0 votes
In my case it ended up that Jenkins was running docker agent with specific UID and GID. To get that fixed it required to rebuild that docker image with creating internal Jenkins user with the same UID and GID

For that purpose on top of the Jenkinsfile to crate that docker image I have added:

```
def user_id
def group_id
node {
   user_id = sh(returnStdout: true, script: 'id -u').trim()
   group_id = sh(returnStdout: true, script: 'id -g').trim()
}

```

and then during the build stage I have passed additional arguments to the docker as

```
--build-arg JenkinsUserId=${user_id} --build-arg JenkinsGroupId=${group_id}

```

then in the Dockerfile for that build:

```
FROM alpine:latest

#pick up provided ARGs for the bild

ARG JenkinsUserId
ARG JenkinsGroupId

//do your stuff here

#create Ansible config directory
RUN set -xe \
    && mkdir -p /etc/ansible

#create Ansible tmp directory
RUN set -xe \
    && mkdir -p /.ansible/tmp

#set ANSIBLE_LOCAL_TEMP
ENV ANSIBLE_LOCAL_TEMP /.ansible/tmp

#create Ansible cp directory
RUN set -xe \
    && mkdir -p /.ansible/cp

#set ANSIBLE_SSH_CONTROL_PATH_DIR
ENV ANSIBLE_SSH_CONTROL_PATH_DIR /.ansible/cp

# Create Jenkins group and user
RUN if ! id $JenkinsUserId; then \
    groupadd -g ${JenkinsGroupId} jenkins; \
    useradd jenkins -u ${JenkinsUserId} -g jenkins --shell /bin/bash --create-home; \
  else \
    addgroup --gid 1000 -S jenkins && adduser --uid 1000 -S jenkins -G jenkins; \
  fi

RUN addgroup jenkins root

# Tell docker that all future commands should run as the appuser user
USER jenkins

```

and finally update `docker` `agent` for the main pipeline which had issue:

```
   agent {
      docker {
         image 'artifactory.devbox.local/docker-local/myrepo/jdk8:latest'
         args '-v $HOME/.m2:/root/.m2 -v $HOME/.ssh:/home/jenkins/.ssh  -v /etc/ansible:/etc/ansible -v $HOME/.ansible/tmp:/.ansible/tmp -v $HOME/.ansible/cp:/.ansible/cp'
      }
   }

```

---
Tags: docker, jenkins, ansible
---


title: Convert AnsibleUnsafeText to String

text:
# Question
Title: Convert AnsibleUnsafeText to String
I am using the ansible shell module to compare the chain of 2 SSL certificate with openssl (they have the same chain):

```
openssl x509 -issuer -noout -in Certiticate1
openssl x509 -issuer -noout -in Certiticate2

```

I register the output in variables chain\_from\_certificate1 and chain\_from\_certificate2 then i compares there standard output:

```
- name: Compare the Chain
    fail:
      msg: Error The chains aren't the same
    when: chain_from_certificate1.stdout !=  chain_from_certificate2.stdout

```

Ansible tell me that the chains are not similar (and it's false because they are).

I think that the issue came from the comparison operator because the type of the variable are "AnsibleUnsafeText" and not string.

My question how to compare 2 AnsibleUnsafeText ? and it's necessary to convert them to string before comparison ?

Thank you in advance.

# Answer
> 0 votes
I believe this is due to non existing variable attribute.
Most likely `chain_from_certificate1` and `chain_from_certificate2` don't have `stdout`

Did you try debug by printing the variables?
With `stdout`:
Did this throw similar failure?

```
    - name: print variable
      debug:
        msg: "{{ chain_from_certificate1.stdout }}"

```

Without `stdout`:
What is output of this?
Will this print same output as of manually execute `openssl x509 -issuer -noout -in Certiticate1`?

```
    - name: print variable
      debug:
        msg: "{{ chain_from_certificate1 }}"

```

My assumption is that the script set the value of `stdout` in `chain_from_certificate1` and `chain_from_certificate2` rather than the `stdout: <value>`. Therefor `stdout` do not exist as attribute.
Or `chain_from_certificate1` and `chain_from_certificate2` contain another data structure.

The key is to print them out.

---
Tags: ansible
---


title: How to increase jenkins logger size?

text:
# Question
Title: How to increase jenkins logger size?
I use Jenkins log recorders as described in docs: https://www.jenkins.io/doc/book/system-administration/viewing-logs/
It works, but log is only about 255 lines long, which is too short for my processing.
Newest lines are shown, I cant reach older logs, so I dont know what happened in the beginning.
How do I increase this size? I would like to keep at least 1MB of data.

# Answer
> 3 votes
I'm pretty sure this Jenkins system property will do the trick:

> ```
> hudson.util.RingBufferLogHandler.defaultSize
>
> ```
>
> tuning
>
> Since: 1.563
>
> Default: 256
>
> **Description:** Number of log entries in loggers available on the UI at /log/

You can also custom Configure Loggers for Jenkins or create a logger in Jenkins for troubleshooting and diagnostic information that capture specifics you are looking for.

Additional references:
JENKINS - Logging
JENKINS - Logger Configuration

Or, if you are the admin, you should be able simply to scan the logs on the filesystem. Review what log rotation options and retention you may have.

---
Tags: jenkins, logging
---


# Question
Title: How to Automate VM provisioning and setup using code?
Here is what I want to achieve. I write code(Java, NodeJs, Go) to tell the program

* which Cloud Provider to choose- AWS/Azure
* write code to create a instance , lets say AWS - EC2 instance under which region, security pair, ports, volumes.
* Then once the VM is created , write code to ssh into the system , do yum update , install docker, do other stuff and fetch certain containers. Basically ready the system for future use but mostly I would want to deploy Docker containers on these VMs later.

My question is - to achieve the above scenario which tools should be used . I understand for provisioning of the VMs in AWS , I can use the SDK, so applies for other vendors as well , but for the setup part what would be the preferred mechanism ?

# Answer
First and foremost, there are several tools for better managing and provisioning of infrastructure such as Terraform.

You could create a pipeline to create things automatically as needed (based on triggers or manual start) on Jenkins or Gitlab, your code could be created using any language using the defined cloud provider SDK or API, or the. Terraform code as previously said. I'd use the Terraform since it's way easier to manage what is there, and sincronize the current cloud state.

Also, Ansible is great to setup machines, but Terraform can do it too.
> 1 votes

---
Tags: amazon-ec2, automation, provisioning
---


# Question
Title: How to exclude storage.bucket.list permission on public gcp storage bucket
Google Cloud Platform Storage Buckets can be directly exposed via HTTP. The recommended configuration for this is to have `allUsers` with the `Storage Object Viewer` role.

The Storage Object Viewer role has got storage.bucket.list permission. I tried creating a custom role based on this role and removed the storage.bucket.list permission.

I updated the IAM permissions by getting permissions with gcutil, editing, then I tried to update.

When I tried updating I used the following command

`gsutil iam set permissions.txt gs://example.com/`

But I got the following error

> BadRequestException: 400 Role roles/CustomStorageObjectViewer is not supported for this resource.

Is there any way to configure a public gcp storage bucket without the storage.bucket.list permission?

# Answer
I am not entirely sure what configs you were using, but here is a process using CLI (gcloud and gsutil). The end product is a new custom role replacing the current standard roles/storage.objectViewer on allUsers for a GCP Bucket.

Recommend using GCP Cloud Cloud Shell in your Project: https://cloud.google.com/shell

1. Create a new Role with the Custom Permissions you would like.

Create yaml file to hold the new config:

```
vi custom_role.yaml

```

Add role configurations to new custom\_role.yaml:

```
title: "storageobjectviewer.nolist"
description: "Storage Object Viewer Role without source objects list"
stage: "ALPHA"
includedPermissions:
- resourcemanager.projects.get
- storage.objects.get

```

Create new Role in GCP:

```
gcloud iam roles create storageobjectviewer.nolist --project=[YOUR PROJECT] --file=custom_role.yaml

```
2. Pull current permissions for \[YOUR BUCKET\] locally:

> gsutil iam get gs://\[YOUR BUCKET\] \> perms.txt

3. Update Permissions with the new role.

Should only be this section:

> ```
> {
>   "members": [
>     "allUsers"
>   ],
>   "role": "projects/[YOUR PROJECT]/roles/storageobjectviewer.nolist"
> }
>
> ```

4. Update the allUsers to the new role on \[YOUR BUCKET\]

> gsutil iam set perms.txt gs://\[YOUR BUCKET\]

The new configuration should be viewable in CLI:

```
gsutil iam get gs://[YOUR BUCKET]

```

Also in the UI: Bucket Details -\> Permissions

and IAM -\> Roles  Hopefully this helps!

Additional Info:
> 2 votes

---
Tags: gcp
---


# Question
Title: Access private DockerHub repo in Docker Swarm
I am trying to deploy a docker swarm using Digital Ocean's docker-machine driver.

This is currently a single node that contains several public DockerHub images, and one private.

I am having difficulties in configuring it to accept the private image properly.

I have SSHed into the node, and used `docker login`. After that, I was able to properly `docker pull` the private image.

But, when trying to run `docker stack deploy -c myapp.yml myapp`, I am getting this warning:

```
image me/myapp:latest could not be accessed on a registry to record
its digest. Each node will access me/myapp:latest independently,
possibly leading to different nodes running different versions of the image.

```

What do I need to do to properly login to a private registry in docker swarm mode?

# Answer
You need the `--with-registry-auth` flag, e.g.:

```
docker stack deploy -c myapp.yml --with-registry-auth myapp

```

From the `docker stack deploy --help` output:

```
--with-registry-auth     Send registry authentication details to Swarm agents

```
> 3 votes

# Answer
first you need to do docker login to repo then you do stack deploy with --with-registry-auth flag.
> 0 votes

---
Tags: docker, docker-swarm, docker-registry, docker-machine
---


# Question
Title: Git feature branching strategy allows unapproved 'tag along' features
## Context

Our current operations consists of three servers: QA is a playground for devs to try out new features; Platform is the live customer facing application; Mock is a copy of Platform and functions as a staging area for new releases to be tested before release to Platform.

Our branching strategy follows these three servers. There is a QA branch for the QA server, Mock branch for Mock, Master for Platform. When a new feature is requested, a feature branch is branched off Master to be developed on a devs local machine. Once the feature is complete, the feature branch is merged into the QA branch for testing/approval. When release time comes around, we pick out all the approved feature branches and merge them into the Mock branch for final release testing. After release testing the Mock branch is tagged as the next version and merged into Platform.

## Symptom

The main issue that we have been dealing with using this strategy is that devs who finish a feature will accidentally branch off the newly finished (or sometimes still in progress) feature branch instead of the Master branch when moving on to the next feature. This means that when approved features are added into Mock for final release review, they can often carry along unapproved features as they sit in the same feature branch.

## Question

Is there an automated way to go through each feature branch to check that no other feature branches are in its history since the last release tag? I can do this manually (which I do occasionally) but that becomes cumbersome. I can also make my own script to see if any branches share the same commit since the last git tag if such automation doesn't exist.

*I am not trained in DevOps - if you see a glaring issue with this dev cycle please point it out to me!*

# Answer
> 1 votes
Technically I guess something like this would print out any branchs between any particular branch and your master branch:

```
for hash in $(git log --format=%h master..); do if git branch --contains=$hash | grep -qv $(git name-rev HEAD | sed -e 's#HEAD ##g'); then git branch --contains=$hash | grep -v $(git name-rev HEAD | sed -e 's#HEAD ##g'); fi; done

```

But fundamentally there is probably an underlying problem here, I'm not intimate with your work flow or the teams you work with, but it seems unlikely to me that developers would accidentally create a new feature branch rooted on another feature branch unless they actually wanted some of the code in that other feature branch, perhaps to reuse in delivering whatever the new feature might be.

It seems that you might make a lot of improvement to your process quite quickly and easily by simply ensuring that the all features that are currently being worked on to completion are included in the next release and thus cN be merged into a pre-release branch as soon as they are approved. From there any branch taken from that pre-release branch should only contain features intended to be released,but it would mean that if a developer wrote code they thought was applicable to more than one problem,then they would almost immediately be able to reuse it once merged.

Although I'm not entirely sure that I've understood the issue you are facing, it seems to me that the existence of code in two branches isn't actually a problem, it's just code reuse. If one of those branches is pulled from the release then that's fine, the code that is being reused would still need to stay?

---
Tags: git
---


# Question
Title: micrometer exposing actuator metrics vs kube-state-metrics vs metrics-server to set pod request/limits
micrometer exposing actuator metrics to set request/limit to pods in K8svs metrics-server vs kube-state-metrics -\> K8s Mixin from kube-promethteus-stack Grafana dashboad It's really blurry and frustrating to me to understand why there is such big difference between values from the 3 in the title and how should one utilize K8s Mixin to set proper request/limits and if that is expected at al. I was hoping I can just see same data that I see when I type kubectl top podname --containers to what I see when I open K8s -\> ComputeResources -\> Pods dashboard in Grafana. But not only the values differ by more than a double, but also reported values from actuator differ from both. When exposing spring data with micrometer the sum of jvm\_memory\_used\_bytes is corresponding more to what I get from metrics-server (0.37.0) rather then what I see on Grafana from the mixin dashboards, but it is still far off. I am using K8s: 1.14.3 on Ubuntu 18.04 LTS managed by kubespray. kube-prometheus-stack 9.4.4 installed with helm 2.14.3. Spring boot 2.0 with Micrometer. I saw the explanation on metrics-server git that this is the value that kubelet use for OOMKill, but again this is not helpful at all as what should I do with the dashboard? What is the the way to handle this?

# Answer
> 0 votes
Based on what I see so far, I have found the root cause, renamed kubelet service from old chart to new that can get targeted by serviceMonitors. So for me the best solution would be grafana kube-state-metrics + comparing what I see in the jvm dashboard

---
Tags: kubernetes, prometheus, java, grafana
---


# Question
Title: Reset Gitlab Project to Repository Content
I had some unwanted files in a repo which is going open-source soon. I removed the files using git-filter-branch. I noticed the project size in Gitlab growing with every such removal. Some research suggested that Gitlab has another backup layer, where the data is still in the project, which can be accessed by exporting the project.

Is there a way to just reset the Gitlab storage to what the repo actually contains when being cloned? I'd much prefer not to do these operations again on the project file, since it is prone to data loss.

# Answer
> 1 votes
The solution I found in the end is to just create a blank new project, add that to your git remotes and then push to it.

You will have to manually carry over your Issues and probably some other stuff as well. Note that issues can be exported and imported as well, but they loose their comments in the process.

---
Tags: gitlab
---


title: Populate Database

text:
# Question
Title: Populate Database
I am using Terraform Cloud to build the infrastructure. One of the resources is a postgresql database.

What is the best way to populate the database by creating tables, multiple schema etc after the database gets created.

# Answer
> 1 votes
I ended up using the `cyrilgdn/postgresql` provider which allowed be to create/destroy db as needed

```
terraform {
  required_providers {
    hashiaws = {
      source  = "hashicorp/aws"
      version = "~> 3.4.0"
    }
    postgresql = {
      source  = "cyrilgdn/postgresql"
      version = "1.7.2"
    }
  }
  required_version = ">= 0.13"
}

provider "postgresql" {
  host             = local.db_url
  port             = local.db_port
  database         = local.db_name
  username         = local.db_usr
  password         = local.db_pwd
  sslmode          = "disable"
  connect_timeout  = 15
  expected_version = local.db_eng_ver
}

```

The only downside to this solution is that the server you are running TF on will need to have access to the RDS manage instance

# Answer
> 1 votes
For RDS, the simplest would probably be to have a template RDS snapshot that you create instances off of, just like you would an AMI.

However if later updates to the database structure will be frequent/complex enough that the added complexity of a CD migration tool will be worthwhile, then you may be better off starting with that for the initial deployment as well. I don't know how mature they are in the postgres world, but https://dbmstools.com/categories/version-control-tools/postgresql might be a good place to start.

---
Tags: amazon-web-services, terraform, databases
---


# Question
Title: docker build fails on one machine but succeeds on another
On my own machine (Windows 10, Docker Desktop 2.1.0.3) the build succeeds and runs. On another machine (Ubuntu 18.04, docker CE 19.03.13) the build fails with the error shown.

Why does the build work on one machine but fail another?

dockerfile:

```
FROM ubuntu:latest AS toolchain
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
  build-essential \
  g++-5

```

error:

```
Package g++-5 is not available, but is referred to by another package.
This may mean that the package is missing, has been obsoleted, or
is only available from another source
However the following packages replace it:
  gcc-10-test-results gcc-9-test-results gcc-8-test-results gcc-7-test-results

Package 'g++-5' has no installation candidate
The command '/bin/sh -c apt-get update && apt-get install -y   build-essential   g++-5' returned a non-zero code: 100

```

# Answer
The problem seems to be that the ubuntu:latest image is stale. Changing to ubuntu:bionic fixes the issue:

```
FROM ubuntu:bionic AS toolchain
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get update && apt-get install -y \
  build-essential \
  g++-5

```

I don't know why an old image is being used, but that seems like a separate question.
> 0 votes

---
Tags: docker, ubuntu
---


# Question
Title: Start only one pod at a time in kubernetes
Is it possible to tell kubernetes, that when performing a deployment of multiple pods that it should start only one at a time and only create the next pod if the previous has completely started up?

It would be useful in our usecase: Our application automatically migrates/upgrades the database when starting up. But when there are multiple pods starting up at the same time, it could happen that multiple pods try to upgrade the database at the same time which could corrupt the database.

But if kubernetes would wait with starting up the second pod until the first pod has fully started up, it would mean that the database has been correctly upgraded by then.

If kubernetes does not have such functionality, how would you handle such a case?

# Answer
> 5 votes
Use a combination of Readiness Probes and a proper deployment strategy.

1. Configure your Deployment spec like so:
```
spec:
  replicas: 4    # or as many as you need to have
  strategy:
    rollingUpdate:
  maxUnavailable: 0
  maxSurge: 1

```
`maxUnavailable` = 0 means that there should always be a number of pods available equal to the `replicas` count. `maxSurge` = 1 means that kubernetes will create the new pods one at a time, and destroy the old ones one-at-a-time as well.
2. Implement a `readinessProbe` which should return a successful response when the database migration has completed. You'd have to write the logic for that yourself, but if you do it right, kubernetes will create a pod with the new Deployment and wait for it to complete and only after that, evict 1 of the old ones.

PS: It's worth noting that this approach will cause your node to temporarily contain more pods than your `replicas` config implies, namely `replicas + 1` for a short period of time while deploying. You need to make sure that your node can handle the little extra load untill the rollout is complete.

# Answer
> 2 votes
Here you could create a lock table inside your database, then the first pod would take the lock and then run the update. Other pods startup scripts would wait for the lock to be released before starting the application. A simpler solution is to run a pod inside a job for upgrading the database before launching the application.

---
Tags: kubernetes
---


# Question
Title: Get all Deployments referencing given ConfigMap by the ConfigMap's name
Suppose I have a `ConfigMap`:

```
apiVersion: v1
kind: ConfigMap
metadata:
  name: london-tz-config
data:
  SERVER_TIMEZONE: "Europe/London"

```

I also have about 50 distinct Deployments. Let's say that 20 of them use the `london-tz-config` ConfigMap to set their environments, like this

```
  template:
    metadata:
      labels:
        app: echo-app
    spec:
      containers:
        - name: echo-app-container
          image: ealen/echo-server:latest
          ports:
            - containerPort: 8080
          env:
            - name: SERVER_TIMEZONE
              valueFrom:
                configMapKeyRef:
                  name: london-tz-config    # Reference our ConfigMap here
                  key: SERVER_TIMEZONE

```

I don't know which Deployments use which ConfigMaps at any given time without manually inspecting the cluster.

Is there a way to ask Kubernetes to show me all Deployments that contain references to a given ConfigMap by its `metadata.name` attribute?

Either `kubectl` or the Kubernetes API would be okay.

# Answer
You can use `kubectl get -o json` and parse the output using `jq` to iterate over the deployments, matching any desired `configMapKeyRef.name` and returning a `uniq` list of deployments.

This should work:

```
kubectl get deployments --all-namespaces -o json | jq -r '.items | map(select(.spec.template.spec.containers[]?.env[]?.valueFrom.configMapKeyRef.name == "london-tz-config" ) | .metadata.name) | .[]' | uniq

```
> 3 votes

---
Tags: kubernetes, kubectl
---


title: Pass Ansible parameters to include_role

text:
# Question
Title: Pass Ansible parameters to include_role
I have a role that contains this task:

```
- name: Download something via Git
  include_role:
    name: git_cached
  vars:
    params:
      repo: https://somerepo
      dest: /some/path

```

I would really like to then have the git\_cached role look like this:

```
- git: {{ params }}

```

I don't want to have to duplicate all of the parameters like this:

```
- git:
    repo: "{{ repo }}"
    dest: "{{ dest }}"

```

Because then, I have to do it for all the parameters, even the ones I don't typically use. I just want to basically "override" certain parameters that I pass into the include\_role task.

Is what I'm trying to do possible?

# Answer
Q: **"*Override certain parameters that pass them into the include\_role task.*"**

A: The parameter *tasks\_from* serves this use-case. For example, create the task file

```
shell> cat roles/git_cached/tasks/git.yml
- git:
    repo: "{{ params.repo }}"
    dest: "{{ params.dest }}"
    recursive: "{{ params.recursive|default(ommit) }}"
    depth: "{{ params.depth|default('1') }}"

```

Then "call" this task

```
    - name: Download something via Git
      include_role:
        name: git_cached
        tasks_from: git.yml
      vars:
        params:
          repo: https://somerepo
          dest: /some/path

```

You might want to complete the task with all git parameters:

* The parameters *repo* and *dest* are required.
* In some parameters, like *recursive*, the default values are defined. You can either omit the parameter, in which case the default value will be used, or define your own default.
* Some parameters, like *depth*, don't have defaults. Also here you can either omit the parameter, in which case the parameter won't be used or define your own default.
* It's a good idea to "namespace" the variables.
* It's a good idea to put the default variables into roles/git\_cached/defaults/main.yml
> 3 votes

---
Tags: ansible
---


# Question
Title: Pass information between Jenkins Job builds
Please excuse my non-fluent terminology..

There is a Jenkins server, master and a few slaves, there is a pipeline job.. let's call it **playground**

I run the job once, I want to save some data somewhere. Run the job again, I want to get the data from previous build.

* Given that the Jenkins server is bounded to a VPN
* Given that the Job may run on master or any of the slaves
* Given that the Job deletes the workspace with every build

I couldn't find a way to save and load the data..

One plugin that actually do this is the git plugin, that saves the commit id and reads it in the next build for the changelog

I went through the git plugin code.. I am very confused and cannot figure out how or where it is done!

Can someone point me in the right direction.

# Answer
There's a simple two step process...

In your originating job, tick step `[ X ] Archive the artifacts` and set the files to archive.

The files are copied to the master, stored in the jobs folder tree, under the builds directory for each build, along with the log. (Delete the build and it deletes the artifacts that came from it).

Install the Copy Artifact plugin. Configure the job accordingly.

Despite the plugin's screenshot example, this is actually a bad practice for artifacts like jars. Those should be properly managed using an artifact repository like Nexus or JFrog.

See my response regarding keeping the the Jenkins space under control, managing the build and artifact retention soundly.
> 3 votes

---
Tags: jenkins, jenkins-pipeline, git, jenkins-plugins, github
---


# Question
Title: Is gitlab-runner suit for production use
Is the GitLab CI/CD tools suit/reliable for production use?

As more details, I have more than two servers and they've gitlab-runner inside then connected as runners to my GitLab instance. In this case, I wanna make sure that is these runners are reliable and stable if using the server as some test and production server?

Thanks in advance.

# Answer
Is your infrastructure in a cloud provider? If so, any architecture that allows for auto-remediation should suffice to make this production-ready. Bonus points for auto-scaling.

If you're in AWS, for example, check out these documents:
> 0 votes

---
Tags: gitlab, gitlab-ci-runner
---


# Question
Title: How do I run opendistro elasticsearch in docker on HTTP
I'm trying to run the docker image of the AWS opendistro elasticsearch on my laptop for tesing

https://opendistro.github.io/for-elasticsearch-docs/docs/install/docker/

When I run it, it expects an HTTPS connection. How do I run it on HTTP?

I'm using this command to run:

`docker run -d --name elasticsearch-od -p 9200:9200 -p 9600:9600 -e "discovery.type=single-node"`

# Answer
in your docker-compose.yml you need change this parametres:

environment: ELASTICSEARCH\_URL: https://odfe-node1:9200 ELASTICSEARCH\_HOSTS: https://odfe-node1:9200

For this:

environment: ELASTICSEARCH\_URL: http://odfe-node1:9200 ELASTICSEARCH\_HOSTS: http://odfe-node1:9200

And after in your file config for elasticsearch put this:

https-enabled = false

Try and enjoy ;)
> 1 votes

---
Tags: docker, elasticsearch
---


# Question
Title: best Kubernetes platform for deploying apps while learning
I want to practice deploying app on real environment while I am learning Kubernetes. Do you know any affordable cloud platform for this purpose?

# Answer
I am use this website, with this are possible create a k8s with N nodes ;) this is perfect for learning k8s and make any deployment, each session is for 4h, but this are sufficient.

https://labs.play-with-k8s.com/
> 1 votes

---
Tags: kubernetes
---


title: Spread pods over cluster

text:
# Question
Title: Spread pods over cluster
Current using AWS's auto-scale policy to start EKS worker nodes (min+desired 10, max 15) on a fixed time every day. There are 120 pods and 30 deployments.

Obviously, only one node will be up firstly. Then the second node. After that last node will be up. For that reason, Kubernetes add all pods in the first nodes. Is there any way so that pods will be evenly distributed in the scenario?

# Answer
> 1 votes
Take a look at the Descheduler. This project runs as a Kubernetes Job that aims at killing pods when it thinks the cluster is unbalanced.

The `LowNodeUtilization` strategy seems to fit your case:

> This strategy finds nodes that are under utilized and evicts pods, if possible, from other nodes in the hope that recreation of evicted pods will be scheduled on these underutilized nodes.

---

Another option is to apply a little of chaos engineering manually, forcing a Rolling Update on your deployment, and hopefully, the scheduler will fix the balance problem when pods are recreated.

You can use the `kubectl rollout restart my-deployment`. It's way better than simply deleting the pods with `kubectl delete pod`, as the rollout will ensure availability during the "rebalancing" (although deleting the pods altogether increases your chances for a better rebalance).

# Answer
> 2 votes
You should look into affinity and anti-affinity as that allows you to control what pods go on which node. You can have it so that with anti-affinity there is only 1 pod of each deployment on a node. This is a bit overkill and wouldn't work for you exactly though IIRC you can have multiple pods on a node just with a limit.

https://kubernetes.io/docs/tasks/configure-pod-container/assign-pods-nodes-using-node-affinity/

---
Tags: amazon-web-services, kubernetes, autoscaling, eks
---


title: GitHub Payload in Jenkins

text:
# Question
Title: GitHub Payload in Jenkins
I'm using the GitHub plugin with Jenkins, and I'm trying to figure out how I can access the webhook payload. I'm aware that the Generic Webhook Trigger has this ability, as well as the GitHub Pull Request plugin, but this plugin seems more suitable to an automated workflow.

On that note, what I'm looking at is importing the class and parsing the payload, similar to what is done in the troubleshooting section here.

This sample works, but only prints the class:

```
import org.kohsuke.stapler.StaplerRequest
import org.jenkinsci.plugins.github.webhook.GHEventPayload
import org.jenkinsci.plugins.github.webhook.GHEventPayload.PayloadHandler

payload = new PayloadHandler()
println(payload)

```

This sample wants more info, but I'm not handy enough with Java to know what it's asking for:

```
import org.kohsuke.stapler.StaplerRequest
import org.jenkinsci.plugins.github.webhook.GHEventPayload
import org.jenkinsci.plugins.github.webhook.GHEventPayload.PayloadHandler

payload = new PayloadHandler()
println(payload.parse())

```

According to the source code, that's the method I need, but it produces the following error:

```
hudson.remoting.ProxyException: groovy.lang.MissingMethodException: No signature of method: org.jenkinsci.plugins.github.webhook.GHEventPayload$PayloadHandler.parse() is applicable for argument types: (java.lang.Class) values: [interface org.jenkinsci.plugins.github.webhook.GHEventPayload]
Possible solutions: parse(org.kohsuke.stapler.StaplerRequest, java.lang.annotation.Annotation, java.lang.Class, java.lang.String), parse(org.kohsuke.stapler.StaplerRequest, org.jenkinsci.plugins.github.webhook.GHEventPayload, java.lang.Class, java.lang.String), use([Ljava.lang.Object;), use(java.lang.Class, groovy.lang.Closure), wait(), grep()

```

Has anyone tried something like this before?

# Answer
What I wanted to do is solvable with https://github.com/jenkinsci/pipeline-github-plugin
> 0 votes

# Answer
GitHub plugin doesn't pass payload data to any variables available from inside the pipeline.

Both Jenkins tracker and GitHub plugin project have a number of open and cancelled feature requests requesting this, for instance, https://issues.jenkins-ci.org/browse/JENKINS-24291.
> 2 votes

---
Tags: jenkins, jenkins-pipeline, github, jenkinsfile
---


# Question
Title: Jenkins transferring file using publish over SSH plugin but file not show in remote development server
I have a running a Jenkins job by Jenkins pipeline. i wants to send an environmental (`.env`) file from my local system to my server using Publish Over SSH Plugin.

Here is my Jenkins snippet:

```
stage('Pull In Development') {
        when { branch 'development' }
        steps {
            script {
                sshPublisher(publishers: [
                    sshPublisherDesc(configName: 'developmment-server',
                        transfers: [
                            sshTransfer(
                                cleanRemote: false,
                                excludes: '',
                                execCommand: '',
                                execTimeout: 120000,
                                flatten: false,
                                makeEmptyDirs: false,
                                noDefaultExcludes: false,
                                patternSeparator: '[, ]+',
                                remoteDirectory: '/var/www/sockets',
                                remoteDirectorySDF: false,
                                removePrefix: '',
                                sourceFiles: '**/*.env'
                            )
                        ],
                        usePromotionTimestamp: false,
                        useWorkspaceInPromotion: false,
                        verbose: false
                    )]
                )
            }
        }
    }

```

Then after transferring, it shows 1 file transferred.

```
SSH: Connecting from host [pasha-pc]
SSH: Connecting with configuration [stage-server-v1] ...
SSH: Disconnecting configuration [stage-server-v1] ...
SSH: Transferred 1 file(s)

```

But when i enter the server and go to `/var/www/sockets` no `.env` file here. So am I make any mistake on SSH code.

NB: `.env` on git ignored file

# Answer
> 3 votes
Please try with double `//` before `var` in `remoteDirectory: '/var/www/sockets'`.

---
Tags: jenkins, jenkins-pipeline
---