# Question
Title: How to specify 'os' for free gitlab runners ('runs-on' equivalent)
How can I specify the operating system in the `.gitlab-ci` file to indicate which underlying OS on which the `gitlab-runner` should be running when using the 2,000 minutes of free execution in with gitlab.com shared runners in Google Cloud Platform?

For context, Github's free account supports 2,000 free minutes per month in "Github-hosted runners" for running Github Actions in Azure and MacStadium. You can specify whether you want the build to happen in Linux, Windows, or MacOS via the `runs-on` key in the github workflow yaml file.

What is the equivalent method for specifying that I want to run a CI/CD job in a specific OS on GCP in the `.gitlab-ci` file?

# Answer
> 1 votes
You can specify that you want gitlab to execute your CI job on a windows machine using tags.

Gitlab made Windows Shared Runners available as Beta in Jan 2020. Their announcement shows how to use the `tags` key in your `.gitlab-ci` file to specify Windows machines. You can also run in a docker container in windows:

For example, I was able to execute a simple "hello world" powershell script with the following job in my `.gitlab-ci` file:

```
windows:
  script: "build/windows/buildExe.ps1"
  tags:
    - shared-windows
    - windows
    - windows-1809

```

And here's the `buildExe.ps1` powershell script reference above:

```
Write-Host "listing contents of C drive root"
Get-ChildItem -Path C:\ -Force

Write-Host "listing contents of cwd"
Get-ChildItem -Force

```

The above config produced the following output in my CI job:

```
[0KRunning with gitlab-runner 12.9.0 (4c96e5ad)
[0;m[0K  on windows-shared-runners-manager 6QgxEPvR
[0;msection_start:1590842482:prepare_executor
[0K[0K[36;1mPreparing the "custom" executor[0;m
[0;m[0KUsing Custom executor with driver autoscaler dev (64a348d)...
[0;mCreating virtual machine for the job...
Virtual machine created!
section_end:1590842624:prepare_executor
[0Ksection_start:1590842624:prepare_script
[0K[0K[36;1mPreparing environment[0;m
[0;mRunning on PACKER-5E557E8E via
runner-6qgxepvr-wsrm-87cf700a3349de62a18e...
section_end:1590842640:prepare_script
[0Ksection_start:1590842640:get_sources
[0K[0K[36;1mGetting source from Git repository[0;m
[0;m[32;1mFetching changes with git depth set to 50...[0;m
Initialized empty Git repository in C:/GitLab-Runner/builds/maltfield/cross-platform-python-gui/.git/
[32;1mCreated fresh repository.[0;m
[32;1mChecking out 91f0d9e3 as master...[0;m
From https://gitlab.com/maltfield/cross-platform-python-gui
 * [new ref]         refs/pipelines/151176370 -> refs/pipelines/151176370
 * [new branch]      master                   -> origin/master
git-lfs/2.8.0 (GitHub; windows amd64; go 1.12.2; git 30af66bb)

[32;1mUpdating/initializing submodules...[0;m
git-lfs/2.8.0 (GitHub; windows amd64; go 1.12.2; git 30af66bb)
section_end:1590842661:get_sources
[0Ksection_start:1590842661:restore_cache
[0K[0K[36;1mRestoring cache[0;m
[0;msection_end:1590842670:restore_cache
[0Ksection_start:1590842670:download_artifacts
[0K[0K[36;1mDownloading artifacts[0;m
[0;msection_end:1590842677:download_artifacts
[0Ksection_start:1590842677:build_script
[0K[0K[36;1mRunning before_script and script[0;m
[0;m[32;1m$ build/windows/buildExe.ps1[0;m
listing contents of C drive root

    Directory: C:\

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d--hs-       12/11/2019   2:47 AM                $RECYCLE.BIN
d--hs-        2/25/2020   8:52 PM                Boot
d--hsl       12/11/2019  10:29 AM                Documents and Settings
d-----        2/25/2020   8:16 PM                Git
d-----        5/30/2020  12:44 PM                GitLab-Runner
d-----        2/25/2020   8:16 PM                GitLFS
d-----        2/25/2020   8:34 PM                Go
d-----        9/15/2018   7:12 AM                PerfLogs
d-r---        2/25/2020   8:35 PM                Program Files
d-----        2/25/2020   8:27 PM                Program Files (x86)
d--h--        2/25/2020   8:51 PM                ProgramData
d--hs-        2/25/2020   8:10 PM                Recovery
d--hs-       12/11/2019  10:28 AM                System Volume Information
d-----        2/25/2020   8:32 PM                tools
d-r---        5/30/2020  12:43 PM                Users
d-----        2/25/2020   8:38 PM                vcpkg
d-----        2/25/2020   8:51 PM                Windows
-arhs-        2/25/2020   8:45 PM         408834 bootmgr
-a-hs-        9/15/2018   7:09 AM              1 BOOTNXT
-a-hs-        5/30/2020  12:41 PM     1073741824 pagefile.sys
listing contents of cwd

    Directory: C:\GitLab-Runner\builds\maltfield\cross-platform-python-gui

Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d--h--        5/30/2020  12:44 PM                .git
d-----        5/30/2020  12:44 PM                build
d-----        5/30/2020  12:44 PM                src
-a----        5/30/2020  12:44 PM            534 .gitlab-ci.yml
-a----        5/30/2020  12:44 PM          35787 LICENSE
-a----        5/30/2020  12:44 PM           1171 README.md
-a----        5/30/2020  12:44 PM              6 requirements.txt

section_end:1590842686:build_script
[0Ksection_start:1590842686:archive_cache
[0K[0K[36;1mSaving cache[0;m
[0;msection_end:1590842694:archive_cache
[0Ksection_start:1590842694:upload_artifacts_on_success
[0K[0K[36;1mUploading artifacts for successful job[0;m
[0;msection_end:1590842701:upload_artifacts_on_success
[0K[32;1mJob succeeded
[0;m

```

As for MacOS, gitlab currently doesn't support it natively. If this changes, updated answers are welcome.

Other answers showing how to run MacOS inside a VM (eg using vagrant and libvirt) on a gitlab shared runner are very welcome :)

---
Tags: continuous-integration, gitlab, github, gitlab-ci-runner, workflow
---


# Question
Title: Docker convenience script stuck after "Scanning linux images"
I have a Raspberry Pi 2B and a fresh install of Raspbian.

I am trying to install Docker, and following the instructions on the site (https://docs.docker.com/engine/install/debian/#install-using-the-convenience-script), I used the convenience script just like it's written there.

This is all the output I get:

```
# Executing docker install script, commit: 26ff363bcf3b3f5a00498ac43694bf1c7d9ce16c
+ sh -c apt-get update -qq >/dev/null
+ sh -c DEBIAN_FRONTEND=noninteractive apt-get install -y -qq apt-transport-https ca-certificates curl >/dev/null
+ sh -c curl -fsSL "https://download.docker.com/linux/raspbian/gpg" | apt-key add -qq - >/dev/null
Warning: apt-key output should not be parsed (stdout is not a terminal)
+ sh -c echo "deb [arch=armhf] https://download.docker.com/linux/raspbian buster stable" > /etc/apt/sources.list.d/docker.list
+ sh -c apt-get update -qq >/dev/null
+ [ -n  ]
+ sh -c apt-get install -y -qq --no-install-recommends docker-ce >/dev/null
Scanning processes...
Scanning linux images...

```

I let it run for more than a half hour and nothing changes.

How do I resolve this?

# Answer
Just found this question from 5 days ago because I ran into the same thing on a RPi 4, also using the convenience script.

Just waiting a bit did not seem to help. So I killed the process after a while. And Docker is running just fine, now.
> 1 votes

---
Tags: docker
---


# Question
Title: How do I expire users credentials for Kubernetes in Azure?
I am looking for information on how to forcefully expire users credentials in Azure Kubernetes Service (AKS). The reason is to better understand Security models and protecting Kubernetes deployments from unauthorized access.

Scenario: You are an Administrator and your notebook is lost/stolen. Your Kubernetes credentials have been cached because you used `az aks get-credentials ...`. The one who now has your notebook has been able to extract your `~/.kube/config` file. They now have your Admin access!

Thankfully this has not happened to me. However, imagine the nightmare! Now, how do we prevent unauthorized access with this scenario?

One thing I have not yet found is how/where to forcefully expire credentials after X hours. Thus requiring users to get fresh credentials every day or (X hours). What should I be looking for or where can I find documentation that talks of this?

Google searches did not render results to answer this.

# Answer
**First** How to check Check certificate expiration

You can use the `kubeadm alpha certs check-expiration` command to check when certificates expire on a given cluster.

**Simplest remedy** : you can choose to upgrade the control plance which will renew the certs and then you are good and secure as before loosing your laptop.

Automatic certificate renewal

> Note : kubeadm renews all the certificates during control plane upgrade.

This feature is designed for addressing the simplest use cases; if you don’t have specific requirements on certificate renewal and perform Kubernetes version upgrades regularly (less than 1 year in between each upgrade), kubeadm will take care of keeping your cluster up to date and reasonably secure.

> Note: It is a best practice to upgrade your cluster frequently in order to stay secure.

**Another option is to perform** Manual certificate renewal

> Note: The default Kubernetes certificates normally reach their expiration date after one year.

Refer more on `kubeadm alpha certs renew` command usage.

You probably also need to explore the beta feature for Certificate Rotation
> 2 votes

# Answer
You would normally use an Azure AD object like a Service Principal to get the credentials for AKS using the `az aks get-credentials` command.

To expire the aks credentials, you need to set a policy for token expiry on the Service principal which you use.

Once you set the expiry time, kubectl will ask you to reauthenticate when the token expires.

**Refer:** https://docs.microsoft.com/en-us/azure/active-directory/develop/active-directory-configurable-token-lifetimes

This github issue can help understand as well: https://github.com/Azure/AKS/issues/133
> 1 votes

---
Tags: kubernetes, security, azure-kubernetes-service, credentials
---


# Question
Title: Jenkins Docker Behind Reverse Proxy Adds ":80" to the URL
The Jenkins container redirects to `URL:80`, which is causing an error. When I remove the `:80`, everything is working as it should.

I have tried the proxy with a simple web app and there were no issues.

This is the `docker-compose.yml` https://pastebin.com/djJTuBZL.

The cert is correctly issued and the connection is secured. The host I'm using is live and correctly configured with ports 80 and 443 open.

The container exposes `8080` and this is the same as the `VIRTUAL_PORT`.

Also, I would have assumed that accessing an address with `:80` wouldn't have caused any issues this being the standard HTTP port eg `www.google.com:80`; unless I'm missing something.

If this helps this is the logs when I run `docker-compose up` https://pastebin.com/NUbDL7PC. Here are the logs when I try to do the first login, the space is before I remove the `:80` then load the page https://pastebin.com/0wxGDBzi.

How can I resolve this problem?

# Answer
If it's a brand new Jenkins, you can try creating an SSH tunnel directly into the container with port 8080 and checking if the problem also appears there. That will at least allow you to check if it's Jenkins redirecting you or Nginx.

There you can also try adjusting "Jenkins URL" in "Manage Jenkins" -\> "Configure System".

Alternatively, check `jenkins_home/jenkins.model.JenkinsLocationConfiguration.xml`. jenkinsUrl tag is also present there.
> 1 votes

---
Tags: docker, jenkins, docker-compose
---


# Question
Title: Conversion tool for converting kubernetes yaml files from 1.15 to 1.17?
we are on kubernetes 1.15.\* and our yaml files are incompatible with the latest 1.17 installation.

Is there an easy way of converting our yaml files with a tool?

# Answer
You can try the kubeyaml validator. It doesn't convert the yamls but it does show the exact errors for incompatible yaml for each version.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: What is the purpose of AssumeRolePolicyDocument in IAM?
I'm confused as to what the purpose of `AssumeRolePolicyDocument` is.
It seems like to create a role all you need is a parent name that you can attach policies to.

It seems like `AssumeRolePolicyDocument` is that parent name but somehow there's more to it.

What is this doing?

```
"AssumeRolePolicyDocument": {
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": ["lambda.amazonaws.com"]
            },
            "Action": ["sts:AssumeRole"]
        }
    ]
}

```

# Answer
> 16 votes
The purpose of the `AssumeRolePolicyDocument` is to contain the trust relationship policy that grants an entity permission to assume the role.

In your example it's granting the Lambda service the ability to assume

**References**

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html

https://docs.aws.amazon.com/cli/latest/reference/iam/create-role.html

# Answer
> 13 votes
For anyone who is scratching their head at the naming convention:

**AssumeRolePolicyDocument** (in CloudFormation yaml) = **Trust Relationships** (in AWS Console web GUI)

**Policies** (in CloudFormation yaml) = **Permissions** (in AWS Console web GUI)

# Answer
> 10 votes
The purpose of assume role policy document is to grants an IAM entity permission to assume a role. It is also known as a "role trust policy".

In other words, for given permissions you set, it *allow users from certain AWS account to assume this role and access that account*.

It is recommended that you update the role trust policy to restrict access to only authorized users, otherwise any AWS account could assume the role and access that account.

By specifying *Principal* using Amazon Resource Name (ARN) of the AWS account, IAM user, IAM role, federated user, or assumed-role user, you allow or deny access to assume that role (`sts:AssumeRole`).

# Answer
> 7 votes
I think it's much clearer when considering the user that creates a role isn't necessarily the one attaching it for use somewhere:

* `AssumeRolePolicyDocument` is a restriction placed by the user that creates the role - e.g. possibly an admin. It prevents other users from attaching it to any AWS service / user they like. Also, it tells nothing about *where* it's actually used. It's a protection on the Role-side (treating Role as a resource that should also be protected).
* OTOH, actually assigning a role to a service / user, is done by users that actually need now some service to have permissions. And they're allowed to do it only if the Role's `AssumeRolePolicyDocument` allows it.

Granted, when you're the sole user you might not care much about it. It does, though, add more security and documents (forcibly :) ) how the role is intended to be used.

---
Tags: amazon-web-services, aws-iam
---


# Question
Title: Jenkins 2.164.3 - programmable plugins installations for specific versions
Which tool should I be using in order to be able to control Jenkins plugins from a cli/script in such way that when I have a new installation (e.g. 2.164.3)?

For example, I want to type blueocean - 1.1.15 (older version then current for example) and it will download the dependencies for the given version without messing everything else?

As I see I shouldn't be using jenkins-cli, but https://github.com/jenkinsci/plugin-installation-manager-tool? If so can anyone provide a simple example for the scenario I have given as the documentation is short of simple/clear examples.

The already existing topics are not answering the question in 2020 and I can't solve my scenario with them.

# Answer
Why not use Plugin installation manager tool?

Your alternative would be the install-plugins.sh referenced in the tool readme.md. You can explicitly list all plugins and versions or let it auto-resolve dependencies.

See also answer in S/O re:Jenkins plugins for old versions.
> 1 votes

---
Tags: jenkins-plugins
---


# Question
Title: How to sort displayed tasks in the order they were run rather than by the time it took each task to run?
I've configured ansible.cfg to display the steps it has done during a playbook run and it looks like so:

```
===============================================================================
common : Upgrade all packages --------------------------------------- 407.33s
jenkins-slave : compile git --------------------------------------- 153.26s
jenkins-slave : clone tss repo --------------------------------------- 44.61s
jenkins-slave : remove current git installation --------------------------------------- 24.22s
jenkins-slave : install java 1.8.0 --------------------------------------- 16.61s
jenkins-slave : download git source --------------------------------------- 12.44s
jenkins-slave : Install required packages --------------------------------------- 10.83s
jenkins-slave : clone repo --------------------------------------- 9.02s
common : Install base packages --------------------------------------- 6.74s
jenkins-slave : install nvm --------------------------------------- 5.72s
common : install pip --------------------------------------- 5.21s
common : Install base packages centos 7 --------------------------------------- 3.53s
node-exporter : download node-exporter tarball --------------------------------------- 2.83s
jenkins-slave : create "/opt/devenv/company" --------------------------------------- 1.97s
common : download pip --------------------------------------- 1.80s
Gathering Facts --------------------------------------- 1.69s
common : install pip packages --------------------------------------- 1.66s
jenkins-slave : unarchive git source --------------------------------------- 1.46s
node-exporter : untar node-exporter tarball --------------------------------------- 1.32s

```

ansible.cfg is configured like so:

```
[defaults]
nocows = 1
nocolor = 0
retry_files_enabled   = False
display_skipped_hosts = True
host_key_checking = False
transport = paramiko
bin_ansible_callbacks = True
callback_whitelist = profile_tasks
forks = 25
[ssh_connection]
transfer_method = scp

```

If you see from the output, the order is all mixed up and is sorted by the length of each task's run in descending order.

How can I sort the output so it displays the tasks by the order they were run?

# Answer
You can add an entry in your ini file to set the sort\_order for the callback plugin to none.

```
[callback_profile_tasks]
sort_order = none

```

By default it is set to descending, which is why the tasks are displayed in decreasing order of time taken.

Refer: https://docs.ansible.com/ansible/latest/plugins/callback/profile\_tasks.html#parameter-sort\_order
> 1 votes

---
Tags: ansible
---


# Question
Title: How to protect secrets whilst enabling the ability to amend a pipeline
I'm writing a CI pipeline using GitHub Actions.

The pipeline will build a Docker image, which it will then push to our Docker repository (AWS ECR).

In order to talk to ECR, we'll need to provide a secret (and some other details).

That secret we'll be pulling from Hashicorp Vault... though that itself requires a secret in order to access it, so to some extent we're just offsetting the problem.

The pipeline's code is in the same repository as the code for which it is run (to which our developers have access); though we can hold some actions called by this code in a separate repository (to which only our DevOps team have access) if needed.

Whilst we trust our developers, it's generally good practice to keep things locked down where possible. As such, is there any way we can set things up such that developers can amend the pipeline without being able to (deliberately or otherwise) expose these secrets? Are there any best practices around this sort of thing?

# Answer
> 1 votes
Developers or even admins of your GitHub repository wouldn't be able to see secrets configured there by default. Even if you debug GitHub Actions workflows and accidentally log secrets, GitHub would mark them with \***.

So the only attack vector I can think of is highly deliberate and it would require modifying GitHub Actions workflow code and egressing the secret outside to some pre-configured recipient on the Internet.

If you're worried about such attack vector beyond regular code reviews, you may try exploring CODEOWNERS as a way to protect .github directory - https://help.github.com/en/github/creating-cloning-and-archiving-repositories/about-code-owners.

---
Tags: security, github, hashicorp-vault, secret-management, aws-ecr
---


# Question
Title: How do I setup ssh from jenkinsci/blueocean docker to remote?
I am running `jenkinsci/blueocean` docker locally as per the instructions of running jenkins using a docker on jenkins.io. Everything seems to be working correctly but when I docker exec into the docker, I can't use sudo and I cant create a home directory for the jenkins user with an .ssh directory.

There seems to be apk package manager but if this functionality is not out of the box it makes me wonder if I am doing this wrong.

Does anyone have any experience setting up a basic ssh config on this docker image?

# Answer
> 1 votes
You can run an ssh server in the container as the `ENTRYPOINT`. It can either be started in the foreground :

```
ENTRYPOINT ["/usr/sbin/sshd", "-D","-p","22"]

```

Ensure that you have `openssh` installed and host keys setup.

Perhaps this Ansible role would be illustrative

---
Tags: docker, jenkins
---


title: Use Sqlite database on ECS

text:
# Question
Title: Use Sqlite database on ECS
I have an application which uses SQLite database and want to migrate it to AWS ECS service. It's dockerized and have access to DB file via mounted volume.

Is there a way to make this work with ECS? (I cannot use EBS as my launch type is `Fargate`). Should I go with something like EFS or just get rid of SQLite altogether?

# Answer
> 1 votes
Seems like EFS is now supported on Fargate

---
Tags: amazon-web-services, fargate, ecs
---


# Question
Title: vagrant on windows 10 incorrect network configuration on multi-machine setup
I have a problem with vagrant that is running on windows 10 when I deploy centos 8 machines all the machines have eth0 with the same Ip and they cannot communicate with each other

the ip 10.0.2.15 is always present. this is the vagrant file:

```
Vagrant.configure("2") do |config|
  # The most common configuration options are documented and commented below.
  # For a complete reference, please see the online documentation at
  # https://docs.vagrantup.com.

  # Every Vagrant development environment requires a box. You can search for
  # boxes at https://vagrantcloud.com/search.
  config.vm.define "ansible" do |ansible|
    ansible.vm.box = "bento/centos-8.1"
    ansible.vm.hostname = "ansible"
    ansible.vm.network "private_network", ip: "10.0.2.10"
  end

  config.vm.define "ldap" do |ldap|
    ldap.vm.box = "bento/centos-8.1"
    ldap.vm.hostname = "ldap"
    ldap.vm.network "private_network", ip: "10.0.2.100"
  end

```

and this is the ip configuration on the first machine;

```
[vagrant@ansible ~]$ ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 52:54:00:72:fe:6e brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic noprefixroute eth0
       valid_lft 86077sec preferred_lft 86077sec
    inet6 fe80::5054:ff:fe72:fe6e/64 scope link
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 08:00:27:91:b2:04 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.10/24 brd 10.0.2.255 scope global noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fe80::a00:27ff:fe91:b204/64 scope link
       valid_lft forever preferred_lft forever

[vagrant@ansible ~]$ ip route show
default via 10.0.2.2 dev eth0 proto dhcp metric 100
10.0.2.0/24 dev eth0 proto kernel scope link src 10.0.2.15 metric 100
10.0.2.0/24 dev eth1 proto kernel scope link src 10.0.2.10 metric 101

```

is it getting eth0 by dhcp? and then assigning the ip i want to eth1? is there something i can do with virtualbox configuration on windows?

i need both machines to see each other

# Answer
eth0 is used by vagrant for nat configuration in order to establish portforwarding. eth1 private networks that you have configured should allow you to ping/nslookup from one node to another. You can use the following to customize the ip of the nat but I don't see why would you do that:

```
config.vm.provider "virtualbox" do |vbx|
  vbx.customize ['modifyvm', :id, '--natnet1', '192.168.169.0/24']
end

```

Update: tested your setup it doesn't work, this is some CentOS 8 specifics.
> 2 votes

---
Tags: windows, vagrant, centos
---


# Question
Title: Was AWS Secrets Manager removed from AWS CLI commands?
If I enter:

```
 aws secretsmanager get-secret-value --secret-id  my secret id

```

It says invalid choice.

> aws: error: argument command: Invalid choice, valid choices are:

and doesn't list `secretsmanager` as one of the choices or any variation on `secretsmanager`?

```
aws secretsmanager get-secret-value --secret-id  my secret id

```

was used in a video from 2018

Similarly, this command doesn't work

and AWS help and AWS man don't contain secrets manager?

# Answer
AWS secrets are still accessible via the CLI.

The link you provided goes to an old version of the AWS CLI. Check out the latest version and commands here: https://awscli.amazonaws.com/v2/documentation/api/latest/reference/secretsmanager/index.html
> 4 votes

---
Tags: aws-cli
---


# Question
Title: Should I clean up my local machine before installing Docker?
This is a basic question for which I didn’t see anything on in the docs or forum searches.

I have a development machine which has all kinds of stuff installed to do machine learning, like anaconda, python, etc. I want to containerize all of my development using Docker. Would it be a good practice to uninstall everything of importance on the host machine, since everything is going to be running on containers? I guess the main question I have is whether my docker build can/will use any of these ML from my local host machine when building the container.

# Answer
> 2 votes
Ultimately, as long as you have the disk space and RAM, you can run both in parallel with very few issues.

The biggest problem you may run into is that the default ports of the applications you run in a container may already be in use. For example SQL server install runs on port 1433 by default. If you try to run a SQL server container and do not map a different port on the host machine to 1433 on the container you will run into errors. This is easy enough do though:

```
docker run -e "ACCEPT_EULA=Y" -e "SA_PASSWORD=" -p 1431:1433 -d mcr.microsoft.com/mssql/server:2019-CU3-ubuntu-18.04
```

In the above example, MSSQL server will run on port 1431 and map to port 1433 in the container. On your local machine you can access either your SQL server on port 1433 or the container SQL server on 1431.

---
Tags: docker
---


# Question
Title: simulate post step for dynamically generated Jenkins pipeline stages
The following simplified Jenkinsfile dynamically generates sequentially executed stages; however, I cannot create a `post` step for these dynamically stages, like so:

```
pipeline {
    agent none
    stages {
        stage('Gen Stages') {
            steps {
                script {
                    def stageNames = ["st1", "st2", "st3"]
                    stageNames.each { stageName ->
                        createStage(stageName)
                    }
                }
            }
        }
    }
    post {
        always {
            echo "post > always"
        }
        success {
            echo "post > success"
        }
    }
}

def createStage(String stageName) {
    stage(stageName) {
        echo "Stage: ${stageName}"
    }
    // I want to uncomment and use code below - or something effectively simiarl:
    // post {
    //     always {
    //         echo "${stageName} > post > always"
    //     }
    //     success {
    //         echo "${stageName} > success > always"
    //     }
    // }
}

```

I would like to be able to use the commented-out `post {}` stage, or something effectively similar.

If at all possible, I'd like to use the DSL as much as possible and avoid scripted pipelines.

Suggestions?

# Answer
> 2 votes
You can't use `post` because post is part of the declarative DSL, not scripted, and you're defining your stages in a `script` block. So I'm afraid that as long as you're using a `script` block to define your stages, you're going to have to stick with the scripted method, which is the groovy `try { ... } catch { ... } finally { ... }` construct. For example:

```
def createStage(String stageName) {
    stage(stageName) {
        try {
            echo "Stage: ${stageName}"
            echo "${stageName} > this is executed only on success"
        finally {
            echo "${stageName} > this is always executed"
        }
    }
}

```

---
Tags: jenkins, jenkins-pipeline, jenkinsfile
---


# Question
Title: K8s to Prometheus apiserver_client_certificate_expiration_seconds_count metric, why does it count more seconds than 86400 for a single day?
In the 2 pictures below I am comparing the metric from K8s apiserver\_client\_certificate\_expiration\_seconds\_count{job="apiserver"} for my 3 master nodes on K8s 1.14.3. A single day should have 86400 seconds, how come at the start of the day for a given node Prometheus displays 59565 at 00:00 and at 24:00 it shows 205534 seconds? Does that mean that the metric reported by k8s is broken or my query on the graphical dashboard has something in it that I don't understand? In theory it should mean that many seconds have expired from the api server cert, but the math doesn't match the specifications?

# Answer
Based on community answer to that question it turned out there isn't count of the seconds, it is count of all samples for given histogram. That means I had ~146,000 HTTP connections to that API server in that period.
> 0 votes

---
Tags: prometheus
---


# Question
Title: Docker calls to detect app crashing inside it
Is there a way to know if an app has crashed inside docker using maybe bash script or API call from host container, am implementing CI/CD using **Jenkins** and will be deploying the built app via `ssh` to the main server and want to stop container when it crashes on auto-deploy.

# Answer
> 2 votes
> Is there a way to know if an app has crashed inside docker

If your app is writing logs and is the main process in that container, then yes, just watch them

> am implementing CI/CD and want to stop container when it crashes on auto-deploy

What's your tools used for CI/CD? For example, when you are releasing an app with `kubectl apply` you can wait till automated deployment succeed: `kubectl rollout status`

---
Tags: docker, jenkins, jenkins-pipeline, docker-compose
---


# Question
Title: Initial Configuration: Grafana and Prometheus inside docker and then AWS
I'm a bit of a newbie with this, but I haven't been able to find an explanation. Here's my issue. I'm trying to get Prometheus and Grafana configured on an AWS Fargate instance. I'm testing inside a Docker instance on my desktop. Both environments were coming up, but Grafana was reporting no data.

My Datasource:

```
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    access: direct
    orgId: 1
    url: http://localhost:9090
    isDefault: true

```

I started the container with this:

```
docker run -d \
    --name prometheus \
    --hostname prometheus \
    -p 8110:22 \
    -p 9091:9091 \
    prometheus:latest

```

When I couldn't figure out why Grafana reported no data, I restarted the container, adding -p 9090:9090 so that I could access Prometheus's graphing interface. And suddenly Grafana shows data.

Which has me deeply puzzled. I don't know how Grafana works under the hood, and so far I haven't found an explanation. So I'm wondering why port-mapping fixed the problem.

Do I need to configure Grafana to access Prometheus via an externally-resolved URL (that is, one that resolves in my browser, not from Grafana itself)?

I hadn't planned on exposing Prometheus itself to the world, but maybe I have to.

Additional Info: Within AWS, my Grafana charts all have a red warning indicator for each chart, and the rollover says "Not allowed to request resource". I haven't found any docs yet on what can cause this. Still digging.

# Answer
> 1 votes
Okay, it took time and help from someone else. I was going down the path of installing nginx, but that won't end up being necessary.

```
apiVersion: 1

datasources:
  - name: Prometheus
    type: prometheus
    # <string, required> access mode. proxy or direct (Server or Browser in the UI). Required
    access: proxy
    orgId: 1
    url: http://localhost:9090
    isDefault: true

```

This is the end change. originally I had access: direct. As I understand it, that means access to the metrics is via the browser. Switching it to proxy means that Grafana will look up the data. Which means the URL doesn't need to be externally-useful.

To a newbie, none of this was obvious, so maybe my two days of pain will help someone out in the future.

---
Tags: prometheus, grafana
---


# Question
Title: Terminate single processing server based on SQS queue condition, but keep/ reinstate autoscaling size at 1
I have a processing server that processes requests in an SQS queue, only about 10 per day, each taking around 2 minutes to process.

Sometimes the process gets "stuck" and it would be ideal to terminate the single server in the autoscaling group, but keep the autoscaling size to 1, as the problem is often resolved with a fresh server created by autoscaling.

Can this be done with autoscaling configuration alone, or do I need to create a separate process with step or lambda functions?

# Answer
**Disclaimer**: this is *certainly not the best solution* for backend workers, I'm just trying to be creative within the given constraints.

First, you'll need to be familiar with SQS's visibility timeout (documentation) for this to work.

Let's assume the following, then I'll walk through how it might work:

* We only have 1 request (henceforth "message") coming in at a time (10 per day, hopefully that's feasible)
* We can take up to 12 minutes to process a message in the worst-case scenario

Next, some configuration changes:

* We configure the ASG to scale desired count to 0 when `ApproximateNumberOfMessagesVisible` = 0 for 5 minutes.
* We configure the ASG to scale desired count to 1 when `ApproximateNumberOfMessagesVisible` \>= 1.
* We set all messages `VisibilityTimeout` to 10 minutes.

A failure scenario might work out like this:

1. At 0 minutes, 1 message is being processed by the worker. There are now 0 other messages in the queue, `ApproximateNumberOfMessagesVisible` = 0
2. At 2 minutes we are now assuming that the worker is now stuck processing its in-flight message. No action is taken yet.
3. At 5 minutes our ASG scale-down event triggers, and desired capacity is set to 0 (because there is no pending work)
4. At 10 minutes, SQS automatically re-enqueues the message due to the visibility timeout. `ApproximateNumberOfMessagesVisible` = 1
5. Shortly after, our ASG scale-up event triggers, and desired capacity is set to 1 (because there is pending work)

The 3-minute and 5-minute gaps are somewhat arbitrary, they merely highlight the need for a brief waiting period for such a fickle system.

Reference:

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-visibility-timeout.html

https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/sqs-available-cloudwatch-metrics.html

**HOWEVER...**

The best worker is one that exits its process instead of getting stuck. That would allow for process monitoring on EC2, or for containerization. Were your worker an ECS service, it could scale up when there is work, recover from failures, and scale down afterwards. Refactoring your worker might be less effort than ongoing manual intervention for my hacky solution, and save you money!
> 4 votes

# Answer
Use AWS Automation document on state change to modify autoscaling group to 0 and return to 1 after. It should terminate and recreate.
> 1 votes

---
Tags: autoscaling, amazon-sqs
---


# Question
Title: How to get logged in username of Jenkins in parameter's Groovy script?
In Jenkins parameter, I'm writing Groovy script and in that script I need to pass Jenkins logged in username NOT user. User and username, could be different

Ex - User, that is displayed left to 'log out', could be - Bob Gill and username, used to login - could be - bob

User can be retrieved using `User.current()`. Please tell me how to get username.

**NOTE** I don't need username in pipeline code. I need it in Groovy script of parameter.

# Answer
From the Java API documentation for `hudson.model.User`:

> `String` `getDisplayName()` Returns the user name.
>
> `String` `getFullName()` Gets the human readable name of this user.
>
> `String` `getId()`

It's not clear to me from your question what data you want (I think it's the user ID?), so I'm not sure which of these methods will get you that data (`getId()`?). But in any case, you would call the method like:

```
User.current().getId()

```
> 5 votes

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: How to create GitHub Actions for unit testing in .NET projects?
Could You help me? I use **NUnit** framework to test my .NET project. And I want to run my tests via **GitHub Actions**.

What should be included in the assembly of my project? Maybe there are some standard examples?

# Answer
You do not need to include anything into your assembly to run your tests using GitHub Actions. Just create workflow file in .github/workflows folder with the following content (assuming that you have .NET Core project):

```
---
name: Tests

on: push

jobs:
  tests:
    name: Unit Testing
    runs-on: windows-latest
    steps:
      - uses: actions/checkout@v2.1.0
      - run: dotnet test

```

> dotnet is pre-installed on windows machine but is not pre-installed on macos and ubuntu. So, you have to install dotnet by adding an extra step in case you want to run it on one of these machines. You can use actions/setup-dotnet action for this purpose.
> 3 votes

---
Tags: github, nuget, unit-testing
---


# Question
Title: Is it an antipattern to avoid API Gateway for certain tasks?
I'm creating a platform architecture for my project and I'd like to use an API Gateway of some sort for handling most requests to a Kubernetes cluster (with a few microservices).

There is, however, one task that relies a lot on performance and sheer number of connections (think IoT metrics kind of traffic, a lot of connections sending data often, most likely gRPC). I'd always hit a single service in the cluster. It's also different because it will come always from a single physical device (IoT hub, if we're using that analogy) unlike the rest (being web browser, mobile app, 3rd party access, etc).

Since I'd like to offload the API Gateway from that kind of traffic to provide stability and performance for remaining traffic, would it be an antipattern to have a direct connection for this case? Or maybe there's another solution for it?

# Answer
You could have a separate API gateway for that kind of traffic.

Check out the "*Variation: Backends for frontends*" section of the following article:

https://microservices.io/patterns/apigateway.html
> 1 votes

---
Tags: api
---


# Question
Title: Confluent Kafka on K8s - offset is stuck, lag increases, what can be done?
We are using Confluent Kafka cluster on production K8s. It looks like our consumers are stuck in a loop as offset is static and lag keeps going up. When I restart zookeper/brokers (persistent volume) the offset moves by a small amount and then it is stuck again. I have checked my consumers and only 1 of them has assigned something, that is processed and all assignments finish after. What happens on consumer side is that sometimes we have an error, we tried ignoring the message but Kafka is still affected. What can be done outside of the default config. We are not sure why message are stopped being consumed or it looks like there is some loop state leading to attempting to process same message over and over agian.

# Answer
Things we have checked were what happens with the background io queue and the buffer. It turned out our producers were ok as lag was increasing, but as offset was static on check of consumer handlers it turned out we don't send acknowledgement to Kafka on read, leading to Kafka sending the same message after timeout on each attempt to consume. On java code side we have just made sure to send ack.
> 0 votes

---
Tags: kafka
---


# Question
Title: How do a flaten a list mixed with nested dict in ansible playbook?
I have a variable named packages as showed below, it is not a plain list, I have some nested dict in it like browsers.

```

---
packages:
  - apt-file
  - axel
  - ibus-rime
  - rofi
  - i3
  - ffmpeg
  - xclip
  - cargo
  - httpie
  - ttf-ubuntu-font-family
  - tldr-py
  - glances
  - browsers:
    - brave-browser
    - chromium
  - editors:
    - emacs

```

I'd like to use instructions like below

```
- name: Install a list of packages
    become: yes
    apt:
        name: "{{ packages }}"

```

to install all the pacakges defined in the variable, but I do not know what filter I should use the convert the variable to plain list, I need the items in the nested dictionary.

any help will be appreciated.

# Answer
> 1 votes
I probably should use comment instead group the value in a dict who's value is a list

like

```
packages:
  - apt-file
  - axel
  - ibus-rime
  - rofi
  - i3
  - ffmpeg
  - xclip
  - cargo
  - httpie
  - ttf-ubuntu-font-family
  - tldr-py
  - glances
  # browsers:
  - brave-browser
  - chromium
  # editors
  - emacs

```

---
Tags: ansible
---


# Question
Title: Why does TFS only support 1-to-many Predecessor /Successor relationships?
The TFS task tracker only supports 1-to-many Predecessor/Successor relationships (docs).

I totally do not understand why is so.

For me it's obviously a many-to-many relationship, since a task may be blocked by more than one predecessor tasks.

This is the first and only time I've seen such a restriction in such a tracker (for instance, Meistertask supports many-to-many).

Since I can't think of any technical problem to do that (it should be trivial), I'm pretty sure there is a real reason for that. I'd just like to know that reason: I guess it's public, but I couldn't find it.

**Note: this is not a rant**. This is **a very specific question**, that may very well have a known answer (although it's possible it doesn't). If you think this is a rant either you misread it, or I miswrote it, or a combination of both.

# Answer
Because this link has direction from one side to another. Only one work item can on the start of this direction. However, work item:

1. Can has many successor
2. Can has many predecessor.

This type of link can not create the cycle, like:

1. Wi1 successor to WI2
2. Wi2 successor to WI1
> 1 votes

---
Tags: tfs
---


# Question
Title: Azure Devops Pipelines Triggers not working
I have a Repository in Azure Repos which has branches as master, build, feature. Build Branch and Feature branches have their own pipelines code in the azure-pipelines.yaml in the branch itself respectively. So that they can have their dedicated pipelines

The problem is that When I commit to the Build branch or in the feature branch then, both the pipelines are triggered.

I have written in my azure-pipelines.yml

```
trigger:
  branches:
    include:
        - build
    exclude:
       - feature

```

# Answer
If you have set a CI trigger make sure you have set the filter to the exact branch which you need the build to be triggered and not the entire repository which is the default case.
> 1 votes

# Answer
You might consider using the pipeline trigger YAML override feature.

Go to the pipeline designer/editor view. Next to the "Run" button is the ellipsis. From that menu, select "Triggers". From the "Continuous Integration" section, you can choose "Override the YAML continuous integration trigger from here". Then you can tailor the pipeline triggers very specifically without the need to define them in the YAML.
> 1 votes

---
Tags: azure-devops, azure, azure-pipelines
---


# Question
Title: How do I trigger a build pipeline on a TFVC branch commit to map and get that branch?
I have project in DevOps TFVC not Git. I use branches, they live and die in the order of months. I want to create a build pipeline that is triggered when a commit is done on any of the branches and the agent job must map and get the relevant branch that triggered the pipeline. I have no idea how to do this. The initial setup of the pipeline seems to look like I can only map/cloak and get a static set of folders.

# Answer
You've discovered one of the great pitfalls with TFVC. I have also not found an answer to this issue.

The only possible option I've seen around getting code from TFVC out of different branches is to base your pipeline off the highest point of the source tree possible (i.e. the highest common parent folder of all branches) and always build using source labels. You have to do this otherwise a label based build request will checkout nothing if the labeled path(s) do not fall under the pipeline's source root path.

A side effect of this is that in the build agent working directory the checked out code is going to land in a physical folder path that includes all the parent folders going up to the root specified by the pipeline's source root. If your build tasks are generic enough to tolerate this it might work. However, in my experience, the build tasks need to collect artifacts which will essentially be in an unknown location (based on the arbitrary branch) so you'll need something a bit smarter to get the path from which to collect said artifacts. It gets messy quickly.

Another side effect of this is that you can't do continuous integration builds. In order to support getting code from labeled locations, the pipeline's source location is the base path of all possible branches. A CI triggered build then has no context for which instance of a project to build (i.e. which branch). While you could probably extract the branch of a commit from the commit data, you'd need more complicated build scripts to do so.

I've found the pain of this to be far greater than just migrating to Git which allows for considerably more flexibility for building from arbitrary branches since the pipeline is based on the standard repo root from which all branches are based. Since Microsoft has done all but outright state that TFVC is going away, your time may be better invested in migrating to Git to take advantage of this and the other things it provides.
> 1 votes

---
Tags: tfs, branch
---


# Question
Title: AWS Setting instance to use domain from another account
We're transitioning between accounts and we have an app on our new account but we want to use the domain from our old account. We plan on shutting off the old account soon and transitioning everything over, but there's services running on the old account so we can't do that yet.

The best idea I had is to try and use RAM and share the VPC instance and then set a new load balancer in the old account and have a DNS entry in Route 53 pointing to that load balancer. Whenever I try to share the resource it's giving me and error that I don't have permissions to share with an external account, even when I use a root account, I've tried writing policies to give myself permissions but that didn't work either, but I could have written the permissions incorrectly.

Would anyone know how to fix this permissions issue? Or know of a better way to achieve my goal?

Let me know if you need any other info.

# Answer
If I'm getting this right; Maybe you can just create on your old account's Route53 public hosted zone a CNAME record (if you have a load balancer on your new account) or an A record for the instance on your new account. Just don't use the Alias functionality from Route53 as you wont find the resources from the new account.
> 1 votes

---
Tags: amazon-web-services, amazon-ec2, dns
---


title: Tying playbooks together

text:
# Question
Title: Tying playbooks together
Say, i have written bunch of ansible roles and playbooks, configured my playbooks structure and now i'm trying to tie them to one playbook which will do its thing depending on tags or inventory file provided. Should i strive for that? Or is it better to have bunch of playbooks and call them depending on the tasks/hosts needed? What is the best way to go about this? I can't seem to find any example of tying it all together in one playbook or bunch of them which will do its thing to ones infrastructure depending on arguments provided.

Best practice in ansible docs covers organization of playbook structure but not organization of playbook use on infrastructure. Is there any examples of organizing playbooks use on infrastructure?

# Answer
> 1 votes
What I have done before goes on the same direction as Zeitounator commented; you can have one or more "meta" or master playbooks that import the ones you need to run your entire automation. To add to Zaito's comment; you can use the `when` clause on `import_playbook` or `include` to control the execution of those "inner" playbooks; something like this:

Say you want to execute PlatformB playbooks only for an stg or prod environment using a defined `env` ansible variable:

```
- include: platformA/do_something_on_platformA.yml
- include: platformB/do_something_on_platformB.yml
  when: env == 'stg' or env == 'prod'

```

you could change the value of `env` using `--extra-vars` on execution time or something.

hope it helps, cheers.

---
Tags: ansible, infrastructure-as-code
---


# Question
Title: Does local cache solve the problem with locked files when deploying to Azure App Service?
During the deployments, occasionally I experienced issue with files being in use by App Service and thus not overwritten with newer versions.

There are multiple recommendations to overcome this problem:

* Run your app from the ZIP package directly
* Stop your app or enable offline mode for your app during deployment (using App\_Offline.htm)
* Deploy to a staging slot with auto swap enabled.

I am exploring the local cache option, to fix a different problem, and it is not compatible with running from ZIP feature, as discussed here and here

If I understand correctly, when using local cache, the ZIP with code is deployed to Storage Account, and then is loaded into memory on App Service restart.

Does this mean, the issue with locked files is eliminated by using local cache, since the code is not run from the same place where it's deployed to?

# Answer
The answer is no, **if you are using the recommended setup**, which is:

* App Service should have Staging and Production slots
* Staging slot has Local Cache disabled
* Production slot has Local Cache enabled
* The following settings are configured as "Slot setting" aka "Sticky setting" - WEBSITE\_LOCAL\_CACHE\_OPTION, WEBSITE\_LOCAL\_CACHE\_SIZEINMB
* The code is deployed to Staging slot
* Staging slot is swapped with Production slot

For deployments to Staging, I created the following steps:

1. Shut down Staging slot by introducing App\_Offline.htm (POST to SCM URI or can be part of artifact being deployed)
2. Wait for slot to stop before cleaning (probe the URL of Staging slot)
3. Cleanup wwwroot (post to SCM URI) - this is to clean unused files. Deployments are not incremental, so if it's important to have clean old files, you can introduce this step.
4. Deploy to Staging slot using `az webapp deployment source config-zip`
5. Remove App\_Offline.htm (POST to SCM URI)

For swapping slots:

1. Check Staging slot is up and ready (probe the URL of Staging slot) - this one is useful when triggering deployment environments in Atlassian Bamboo when deployed to Staging. You need to wait for Staging to be up and loaded before swapping, otherwise issues may occur
2. Swap the slots using `az webapp deployment slot swap`

If you deploy directly to Production slot, and restart it right after (because of Local Cache enabled), then yes, there should be no file locks when using this method. But this creates the downtime during deployment, in contrast with deployments to Staging slot and swapping it with Production.
> 1 votes

---
Tags: azure
---


# Question
Title: Trick a docker containers DNS to resolve a different url
I am not sure I know the terminology to describe this correctly, my apologies.

But for a contrived example. Say I have a docker container that runs code that accesses "google.com". But I want all requests that the container sets to "google.com" to actually go to "facebook.com". In other words I want the DNS lookup of one domain name to use another domain when reloving. Is there a way to do this with docker or docker-compose?

My ask is a bit more than this, because I know that often multiple websites are hosted on the same server (same IP), so I need a way for the conversion to happen at the domain name level, not at the IP level.

Thank you

# Answer
> 3 votes
If you want a DNS lookup of one domain to resolve to another one, you have 2 options.

1. Modify the Container /etc/hosts file. By default any DNS lookup is done there. You schould do that in the Dockerfile.

Modify /etc/hosts in a Dockerfile

2. Maintain your own DNS servers with all mapping records. On container runtime just pass the DNS server.

Docker docs

> --dns The IP address of a DNS server.

# Answer
> 2 votes
Is this what you are trying to do?

```
$ docker run --add-host foo:127.0.0.42 alpine ping foo -c 3
PING foo (127.0.0.42): 56 data bytes
64 bytes from 127.0.0.42: seq=0 ttl=64 time=0.045 ms
64 bytes from 127.0.0.42: seq=1 ttl=64 time=0.061 ms
64 bytes from 127.0.0.42: seq=2 ttl=64 time=0.070 ms

--- foo ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.045/0.058/0.070 ms

```

Look for --add-host parameter for docker run or extra\_hosts for docker-compose.

# Answer
> 1 votes
The `extra_hosts` setting in `docker-compose` might help you. Docs

---
Tags: docker, docker-compose, dns
---


title: Monitoring K8s Pods in case of outage

text:
# Question
Title: Monitoring K8s Pods in case of outage
K8s has self-healing builtin features, if a pod crash then it will be replaced with the new pod but its log is gone. So what is the best way to monitor K8s pods in case of outage?

# Answer
> 1 votes
For health monitoring, I would recommend the prometheus-operator. If you are looking for application logs, you should consider a logging stack such as elasticsearch-fluentd-kibana. I personally would recommend the kube-fluentd-operator. Additionally, if you would like to see the kubernetes events I find eventrouter quite useful which can also be wired into your logging infrastructure.

---
Tags: kubernetes, monitoring, logging
---


# Question
Title: How do I list pod name/version across multiple clusters in Kubernetes
I have multiple clusters that I want to query and in each cluster I want to find and list the versions of the pods running in there.

I've written a shell script that sets the namespace and then changes context to each cluster and then runs `kubectl describe pods | grep "/version=" -B1` in each cluster.

This seems to be a very hacky and clumsy approach to this problem. Is there a succinct command in the Kubernetes realm that I can issue that will take a list of clusters and return the name/version of each pod?

I feel that there are 2 problems described here and help with either or both would be appreciated:

1. How do I issue commands like `kubectl describe pods` to multiple clusters at the same time and aggregate those results?
2. What is best way to list names with versions of each pod in a cluster?

Update: I built an assumption into the questions that the answer would be a scripted `kubectl` command but it does not need to be. The objective is to find all the name/version combos of all pods running in listed clusters.

# Answer
The best I have found so far for part 2 is to use the JSON output of `get pods`:

```
kubectl get pods -o json | jq '.items[] | select(.metadata.labels."app.kubernetes.io/name") | { name: .metadata.labels."app.kubernetes.io/name", version: .metadata.labels."app.kubernetes.io/version" }'

```

produces

```
{
  "name": "db",
  "version": "0.0.1"
}
{
  "name": "app",
  "version": "0.0.1"
}

```
> 2 votes

# Answer
For listing versions of each container in each pod I found that the kubectl-images plugin is quite useful.
> 0 votes

---
Tags: kubernetes, kubectl
---


title: golang and circle CI setup

text:
# Question
Title: golang and circle CI setup
Go and devops are both new things for me. I'd like to setup a CI that tests and builds my module, then stores artifacts for download. In addition it'd be good to cache dependencies for next builds.

I don't get how to do it using subdirectory for the go product. I want my git repository to look like that:

```
+ repository_root/
  + go/
    - go.mod
    - go.sum
    + hello/
      - main.go
  + cpp/
  + python/
  + .circleci/

```

go.mod

```
module github.com/{USER}/{REPOSITORY_ROOT}/go

go 1.14

```

Of course {USER} is replace with my github user name and {REPOSITORY\_ROOT} is the repo name.

For now I have a following config.yml that doesn't work.

```
version: 2
jobs:
  build:
    working_directory: ~/project/go
    docker:
      - image: circleci/golang:1.9

    steps:
      - checkout:
          path: ~/project/

      - run: mkdir bin
      - run: go get -v -t -d ./...
      - run: go test -v ./...
      - run: go build -o bin/hello.exe github.com/{USER}/{REPOSITORY_ROOT}/go/hello

      - store_artifacts:
          path: bin/
          destination: hello-app

```

Error on build command:

```
#!/bin/bash -eo pipefail
go build -o ./bin/hello.exe github.com/{USER}/{REPO_ROOT}/go/hello
can't load package: package github.com/{USER}/{REPO_ROOT}/go/hello: cannot find package "github.com/{USER}/{REPO_ROOT}/go/hello" in any of:
    /usr/local/go/src/github.com/{USER}/{REPO_ROOT}/go/hello (from $GOROOT)
    /go/src/github.com/{USER}/{REPO_ROOT}/go/hello (from $GOPATH)

Exited with code exit status 1
CircleCI received exit code 1

```

I know that the module is outside of `GOPATH` but I have the same setup on my local machine and the same build command works just fine.

# Answer
> 2 votes
My approach nowadays, always package your build process into Dockerfile. I.e. here is Dockerfile for one of my open-sourced Go projects:

https://github.com/relizaio/relizaGoClient/blob/master/Dockerfile

*Note*: if you're using go modules (which seems to be the case), replace `RUN go get -d -v ./...` with `RUN go mod download` in the build-stage. BTW, missing `go mod` instruction might be the reason for your build failure in the current setup.

Then in your CI, simply do `docker build` instruction, and then copy your artifact from the container if you need it outside.

*Key benefit*: you can verify build process against same Dockerfile locally and you can port it to any CI tool whatsoever.

---
Tags: cicd, circleci, go
---


title: Whats the point of circleci orbs?

text:
# Question
Title: Whats the point of circleci orbs?
I'm looking at: `CircleCI` `Orb` for `kubernetes`

Whats the point of this? Whats the benefit of an `orb` which is a `CircleCI` specific feature rather than say a `bash` function/script (which is more interoperable/common)?

Seems like unnecessary lockin, but maybe I'm just being naive here?

# Answer
> 1 votes
Let me answer each your question in parts.

# What the point is?

As per Orbs Introduction:

> orbs are shareable packages of configuration elements, including jobs, commands, and executors. CircleCI provides certified orbs, along with 3rd-party orbs authored by CircleCI partners

The advantages of this approach are that an **authorized** package of **jobs**, **commands**, and **executors** is offered.

Assuming hundreds of `orbs`, it looks like a community-approved plugin/addon ecosystem. E.g. `Ansible roles` or `Jenkins plugins` or `Wordpress plugins`

# `orbs` vs shell functions/script

Since `orb` packages:

then it can hardly be replaced by a simple combinations of shell commands/functions.

# Seems like unnecessary lockin?

As for lock-in concerns: you are right.

It looks like lock-in. And it **is** lock-in.

But as I know, there isn't any portable solution than can be easily migrated from one `CI/CD` system to another. E.g. try to convert `.gitlab-ci.yml` to `travis.yml`.

But lock-in's, as usual, tend to give you an advantage in speed or ease of deployment.

# Portability

Since there are four executor types (`docker`, `machine`, `windows` and `macos`), it gives you more portable and cross-platform approach.

# Answer
> 0 votes
orbs can help you saving a lot of time and from lots of configurations hassles in circle ci configuration. orbs provides extra functionality in your build pipeline with its pre-configured properties.

suppose, you want to copy your build artifact to s3 bucket from circle build environment. to achieve that goal without orbs, you need to install awscli and then set environment variables for iam access key id, secret access key, region. but with orbs you can import the s3 copy functionality in build pipeline and provide iam key pairs and region value into that orb function. thus it helps you to use pre-existing templates with certain functionalities in build environment. same goes to k8s deployment by using kubectl from k8s orb without installing kubectl in circle pipeline.

---
Tags: kubernetes, circleci
---


# Question
Title: Run integration test with ansible playbook using terraform provisioning
Ok so I am totally new to `devops`! I am an experienced developer, but at my new job I have been tasked to automate CI/CD. Non-existing so far. Only CI with Travis.

So the ideal workflow is: Push to CI -\> build -\> if successful provision an EC2 instance -\> run integration tests

Our ideal architecture is to use `terraform` for provisioning and `ansible` for automation/config management. Scenarios down the road foresee continuous deployment and more stuff. So I need to get the basic building blocks right. This workflow seems like a good start.

I've searched for tutorials all day and, being beginner, I am totally confused.

As the goal is to fully automate the process, I am naively assuming that the ansible playbook would call a terraform provisioning role/task, and then continue by running all the steps for the integration tests.

But all I have found says to always run `terraform init` first and then `terraform apply` before switching to run a playbook, which seems cumbersome and not that "automated".

Can someone point me to a good resource describing how I can setup my desired workflow? Am I understanding things wrong?

# Answer
> 1 votes
You are right about Terraform having a plan-apply cycle which may not be that conducive to automation. The following has worked for me in similar situations:

1. Use Terraform to "develop" new versions of a base EC2 image.
2. Use Ansible playbooks to provision EC2 instances off a specific version of the image and run tests on them.

# Answer
> 1 votes
This is indeed a tricky thing to do "right", because there are so many moving parts. I have found that it helps to reduce the degrees of freedom by adopting some conventions, and factoring out some parts.

Draw a line between where Terraform is responsible and where Ansible becomes responsible, and separate those stages. You do this so that you can deal with smaller bits and verify the changes.

This way, you can maintain smaller modules of the pipeline, and create deployment or testing scenarios from various well-known starting points.

The next strategy that might help is to "start from the end" and automate the rest. Ie, write a set of assertions on the final state of your pipeline (we do this with Inspec profiles. In this way, you can know whether the application of your Terraform and Ansible has had the desired effect (ie, you can "fail the build"), and roll back.

It's not clear from your post what the complexity of your inventory is, but if it's something like a few machines deployed in a cloud, you could do the following:

1. Terraform the cloud
    1. `terraform init`
    2. `terraform init`
    3. `terraform plan -out $BUILD_NUMBER`
    4. `terraform apply -auto-approve $BUILD_NUMBER`
    5. (ensure that you have an `output.tf` to generate an inventory to target for Ansible in the next steps).
2. Assert correct state of the infrastructure:
    1. `inspec exec cloud_profile`
3. Configure instances
    1. \`ansible-playbook -i playbook.yml
4. Assert the correct state of the converged instances:
    1. `inspec exec instance-profile`

Now you have known-good infrastructure and machines, resulting from an automated pipeline an what is more, you will know clearly (from where the pipeline fails) if there is a problem.

Final thought - you could go one step further in separating the infrastructure from configuration management by building and testing your images in a separate pipeline, with a Packer template. This could use the same Ansible playbook and Inspec profile to generate a known-good image which you can then pass to Terraform at `apply` time.

The feasibility of these approaches depends on how interconnected the resources are in your setup. It might help to draw an abstract dependency graph of the pipeline steps to map actions and tools to stages.

# Answer
> 0 votes
This scenario remembers me how i used to test my ansible playbook with terraform. I would run "remote exec" in terraform to install python and other dependencies of ansible. After that I would retrieve the public ip of the instance and save in the inventory file of ansible in the local. And finally, I ran ansible from my local machine from terraform configuration using "local exec". Of course, before everything you need to configure SSH from local to newly provisioned machine in terraform configuration.

```
resource "null_resource" "ConfigureAnsibleLabelVariable" {
  provisioner "local-exec" {
    command = "echo [${var.dev_host_label}:vars] > hosts"
  }
  provisioner "local-exec" {
    command = "echo ansible_ssh_user=${var.ssh_user_name} >> hosts"
  }
  provisioner "local-exec" {
    command = "echo ansible_ssh_private_key_file=${var.ssh_key_path} >> hosts"
  }
  provisioner "local-exec" {
    command = "echo [${var.dev_host_label}] >> hosts"
  }
}

resource "null_resource" "ProvisionRemoteHostsIpToAnsibleHosts" {
  count = "${var.instance_count}"
  connection {
    type = "ssh"
    user = "${var.ssh_user_name}"
    host = "${element(aws_instance.DevInstanceAWS.*.public_ip, count.index)}"
    private_key = "${file("${var.ssh_key_path}")}"
  }
  provisioner "remote-exec" {
    inline = [
      "sudo yum update -y",
      "sudo yum install python-setuptools python-pip -y",
      "sudo pip install httplib2"
    ]
  }
  provisioner "local-exec" {
    command = "echo ${element(aws_instance.DevInstanceAWS.*.public_ip, count.index)} >> hosts"
  }
}

resource "null_resource" "ModifyApplyAnsiblePlayBook" {
  provisioner "local-exec" {
    command = "sed -i -e '/hosts:/ s/: .*/: ${var.dev_host_label}/' play.yml"   #change host label in playbook dynamically
  }

  provisioner "local-exec" {
    command = "sleep 10; ansible-playbook -i hosts play.yml"
  }
  depends_on = ["null_resource.ProvisionRemoteHostsIpToAnsibleHosts"]
}

```

---
Tags: ansible, continuous-integration, terraform
---


# Question
Title: Use dynamic values for parameter-store in aws buildspec.yml for CodeBuild
For my CodeBuild projects, I have a dev, staging and production project. Each project has an environment variable called $BUILD\_ENVIRONMENT set to dev, staging and production, respectively. Each stage has its own key/value pair in Systems Manager Parameter Store. I want to use one buildspec.yml across all branches, so if I try to merge a branch or checkout a new branch, it does not copy hardcoded values in buildspec.yml. This is what I tried to do:

```
version: 0.2

env:
  parameter-store:
    SECRET_KEY_BASE: /myproject/$BUILD_ENVIRONMENT/secret_key_base

```

But it doesn't work. There is no interpolation done and I get the following error:

> Decrypted Variables Error Message: parameter does not exist: /myproject/$BUILD\_ENVIRONMENT/secret\_base\_key

Since this is such a common use case, I am wondering if anyone found any type of solution to this?

# Answer
I faced the same problem while using different SSH key pairs for different deployment environment. Usually I used to create different ssm variables per environment and then changed the name of ssm per git branch in buildspec.yml. That worked perfectly for me. For example, for deploying to stage environment i used the git branch "dev" which had following buidlspec.yml config

```
env:
  parameter-store:
    SECRET_KEY_BASE: "DEV_SECRET_KEY_BASE"

```

for "release" environment it was,

```
env:
  parameter-store:
    SECRET_KEY_BASE: "RELEASE_SECRET_KEY_BASE"

```

make sure you resolve merge conflicts between inter git branches before you apply.
> 1 votes

---
Tags: amazon-web-services
---


title: components in kubernetes master node?

text:
# Question
Title: components in kubernetes master node?
I'm still a newbie learning the kubernetes and trying to clarify the details.

I've read this document about the components and would like to know the structure of kubernetes master node:

1. is it allowed to deploy the kubernetes control plane components on different master nodes in same cluster? (e.g. deploy "kube-schedulter" on master node A and "kube-controller-manager daemon" on master node B)
2. Is there options to make HA/FT for the master node?
3. is it suggested to deploy all the kubernetes control plane components together on a master node no matter what kind of use cases?
4. does the deployment of master node(s) support both static IP address and DHCP IP address assignment?

# Answer
1. I do not see any obstacles. All communication is done through the apiserver over https.
2. Multiple replicas of the controller and scheduler are allowed, leader election is built in. All you need is load balancing in front of the apiservers.
3. I find this to work best, since usually the failure domain is a node, rather than a component.
4. Yes.
> 3 votes

---
Tags: kubernetes
---


title: Using Azure Devops with Git Workflow

text:
# Question
Title: Using Azure Devops with Git Workflow
I have used Azure Pipeline for CI/CD on various small projects that I have worked and it has been a great experience. I am working now on a decent size project that has a web app, mobile(xamarin) app and API. We plan to use Gitflow along with Azure Devops for the delivery process. But I have got a few questions that is really bothering me, and would like to hear other opinion.

1. I understand that feature branches are created to work on a user story, but when multiple people are working on a user story, do we create a feature branch, and then each developer branch of it and then work on the individual sub-tasks? And when they complete their work, they merge back to the feature branch. Is this a standard practice? Or do developers merge back to develop branch as they finish their sub-tasks?
2. Is it a good practice to do a build in Azure Devops for every PR merge to develop branch?
3. When a feature branch is merged with develop branch, do we immediately sent out a release or do we group and send out as nightly builds to QA?
4. I understand that both build and release pipelines offer a great flexibility. But generally does the build pipeline generates the artifacts or is it the release pipeline that does that?

I appreciate if someone can help me with these questions. Or atleast point me in the right direction

# Answer
Most answers to these questions are going be subjective and influenced by your particular organization's workflows and culture. Little of this response if more than just my own opinion and experience and much is not really specific to Azure DevOps but can be applied to any build and release orchestration tooling.

1. Azure DevOps itself has very little to do with your chosen workflow apart from it being the common remote for the team and the source for pull requests etc. What source workflow you use is highly dependent on your individual situation. The number of people on your team doing simultaneous work, how much possible conflict there is between feature development efforts, your rate of release, complexity of code/projects, and philosophies about release management to name a few. Much has been written about various source management workflows. In the realm of Git, of course there's the over 10 year old GitFlow approach (note the recent update) and GitHubFlow. I won't regurgitate what they or the numerous other writings suggest about what flow to use. Your team needs to research the options and evaluate what is most applicable.
2. It certainly is not a bad practice to do a build for every PR. If developers are following good practices where feature branches originate from the merge target, there is a high probability that any testing/compiling they did in code from those branches will essentially match what will eventually be merged. However, that's never a guarantee so having the ability to do a pre merge PR build is a very helpful feature of ADO. And ADO can be configured to required a passing PR build before merge. You can also deploy PR builds if you want to support initial manual testing and/or more complex integration of a PR prior to merge.
3. This again will be a choice driven by your team and situation. Do ***you*** want every feature merge to result in a build? If you are practicing Agile, you may have work items that require manual testing as part of the definition of done. In that case, getting builds out immediately after code is complete could be imperative to forward progress. I follow a continuous integration approach and have builds trigger immediately upon commits/merges to the "typical" branch of work in progress (often that will be the `development` branch). The sooner you discover a problem, whether that is a compile error of the merged code, a unit test failure, or a manual integration test bug the better. Every minute you wait to fix an issue is compounded into more time resolving it later as the work fades from peoples' minds.
4. I have typically followed the philosophy that "what I deploy to production should be what we tested". While theoretically, if you do 2 builds of the same exact source code you should get the same result, there's no guarantee of that. Particularly now with the use of public packages/modules and the ability to allow flexibility with what versions are consumed, it's not an unreasonable possibility that a later build will result in different artifacts, for example, but pulling in a newly updated version of a dependency. For some very real perspective on this, see this article about how the "internet broke" some years ago. I work in the healthcare industry which has very strict rules that drive our implementation approach. We need to deploy just what was tested and approved to the production environment. If we did a fresh build for production release that introduced a changed library with an unknown security vulnerability, we could get into serious hot water. I follow the practice of having a build pipeline do the actual build of final artifacts which become the singular source for deployments for testing and final promotion to production. ADO makes this very simple by publishing the artifacts into its own internal storage and providing those as an artifact source for a release pipeline. You can then deploy that (and additional) artifacts to each release stage of your pipeline. One key factor to consider is environment configuration management. Depending on your deployment target, you'll need to take into account how to influence the runtime configuration. My organization still deploys .NET applications to traditional IIS web servers using file copy. This requires an explicit step to apply config transformations for the application based on the target environment. This approach can be enhanced by using a separate artifact source for the transformation to better implement the good DevOps practice of separation of concerns (environment agnostic build artifacts vs. env specific configuration; env configuration, including secrets, can be retrieved from a different, more protected source so they are not in the application code repository). If you are deploying to a docker image, your docker runtime environment can drive environment specific settings for the container instances. Deploying to a cloud application host service? Use its instance configuration tooling, such as the app service/slot specific configuration settings in Azure. Applications like React, however, take a different approach that typically dictates that you do a build for the target environment since environment driven values are bundled into the deployed artifacts. I'm not a fan of this as it breaks my core philosophy of "build once, run anywhere". But there are some solutions to help make such builds environment agnostic.
> 4 votes

---
Tags: git, azure-devops, continuous-deployment
---


# Question
Title: How to push to other origins using Azure Pipelines?
Inspired by Necessity of mirroring git repos for small team considering switching to Azure:

How does one use Azure Pipelines to push changes to a secondary/backup origin on push?

Possible use cases:

* Need of a backup, like GitLab as backup for Azure Repos.
* LAN-only Azure DevOps Server makes backups to the cloud.
* Global teams wants lower latency and code hosted everywhere.

# Answer
> 1 votes
While I haven't tried this, I suppose a solution would be to have a regular pipeline (this should work in either build or release pipelines) that would check out your repo (as it's default first step) then some scripted command that adds the desired remote and pushes to it.

Some pitfalls I could see you running into:

* issues with the state of the agent work folder based repo prior to a push; but that should be reconcilable with the appropriate Git commands. If the target remotes (i.e. backups) are never being committed to by anything other than the pipeline, you hopefully won't run into conflicts
* arbitrary branches: I'm not a git expert, but I think there's a way to push all branches to the remote
* authentication: you'll obviously need to provide appropriate authentication to the target remote(s). Credentials shouldn't be stored in the scripts in plain text. Use secured pipeline (or group based) variables
* repo size: a huge repo will need to be cloned into the ADO agent workspace. Not a problem as long as you have the disk to support it.

Given the needs you listed, it seems like you should consider something like AzureDevOps services in the cloud, or private GitHub repos (recently became free), etc. Why manage your own backups if the cloud can do it for you?

---
Tags: git, azure-devops, backup
---


# Question
Title: How to setup priorities between pipelines that might be triggered simultaneously on Azure DevOps pipelines
I am using Azure devOps to build and deploy my application, and within that application I have a project that is being used by other related applications (let's call it project N). Thus I decided to deploy this project as a NuGet package, and consume it during the build.

Project N is likely to change often, thus I am having some trouble in defining a reliable setup that:

1. Creates the NuGet package only when there are changes in Project N and its dependencies
2. Guarantees that, if the NuGet package is to be created, as per 1), the pipeline does that before anything else

The way I have setup this so far, is to have two different pipelines, one that creates the NuGet package and deploys it (NuGet Pipeline), and another one that consumes that NuGet package and then builds and deploys the application (Main Pipeline).

The problem is, when changes happen both in files related with Project N, and in some of the remaining files of the solution, I don't know how to make a rule that says "Run NuGet Pipeline before the Main Pipeline", so for the moment I have to trust in luck for that to happen, and if not, I would have to manually trigger the Main Pipeline after the two build are finished. Because I want to be sure that the new changes introduced in the Project N do not break the Solution.

Is there an elegant way to do that? I could setup a trigger that, whenever the Pipeline NuGet finishes it would trigger the Main pipeline, but by doing it that way I would trigger the Main Pipeline twice whenever changes would be made to the Project N and to the rest of the solution.

# Answer
> 1 votes
It sounds like both of these pipelines are being triggered by the same source commit(s). If that's the case, you should be able to use CI trigger source path filters so that the builds only happen on commits to the relevant code.

If a developer does a single commit that is committing changes to both the main application and the package library (which would still trigger both pipelines regardless of the filter), you should consider promoting better code commit practices. Commits should be generally atomic to an appropriate context (i.e. don't commit to a stand-alone library and an app that consumes it together). Commit \[and push if Git\] the package changes first so they'll build and publish, then commit \[and push\] the main application. This will ensure that the trigger filters do what they should.

I don't know of a feature in ADO that allows you to serialize the pipeline execution order in the way you would like. As you suggested, the alternative of triggering one pipeline from another would kind of work but ultimately doesn't solve the core problem.

Alternatively, considering that "project N" changes a lot, and if it has that tight a relationship with the main project. Perhaps consuming it as a package is not appropriate. If the library is also to be consumed by other applications that have a more loose relationship, perhaps consume the dependency directly as code by the main project, but make its package publishing a part of the main application build instead of its own pipeline. (I've used this model for API applications that publish a data contracts library. The API application itself consumes the project directly as a project reference since the two are very tightly related, but others consume as a published package.)

---
Tags: azure-devops, azure-pipelines, build-pipeline, nuget
---


# Question
Title: Azure DevOps Different Post Deployment script per environment
I have a visual studio database project being built and released through Azure DevOps.

I need to execute a different post deployment for Dev vs. Prod.

Is there a way to achieve this either through the release pipeline or the visual studio project?

# Answer
You can define multiple stages in a release pipeline. Each stage (i.e. Dev, Prod) can have its own set of tasks and/or other factors that differ. If you want the same set of tasks, you can duplicate a stage then change what's needed. Alternatively, you can configure the tasks using pipeline variables. The variables can be defined to have different values based on the stage scope (or applicable to the release scope). This way, each stage is essentially the same set of tasks but at runtime each behaves differently based on the variable values applicable to it at runtime.

See the MS docs on stages based on your pipeline type preference:
> 1 votes

---
Tags: azure-devops, azure-pipelines, sql
---


# Question
Title: Unable to get value of password parameter into another parameter
In Jenkins pipeline parameter say `Branches` of type `Active Choice Reactive Parameter`, I've to auto-populate branch names hence I've following command, which I can execute in parameter's Groovy script -

```
svn ls <url> --non-interactive --no-auth-cache --trust-server-cert --username <username>--password <password>

```

`username` can be retrieved using `User.current().getId()`

I've `Password` parameter of `Password Parameter` type. To use value of this parameter in Groovy script of `Branches`, I've added `Password` parameter in `Referenced parameters` of `Branches`

But I cannot see its value reflected in `Branches` parameter. Can you please suggest how this can be handled?

Or any other way to pass Password value or authentication details so I can execute svn command.

# Answer
> -1 votes
Could credentials binding plugin be the answer?

The idea is to add svn user and password to Jenkins credentials store and then use it in the pipeline.

```
withCredentials([
    usernamePassword(
        credentialsId: 'mySvnLogin',
        usernameVariable: 'USER',
        passwordVariable: 'PASSWORD')]) {
    sh '''
      svn ls <url> --non-interactive --no-auth-cache --trust-server-cert --username $USER
 --password $PASSWORD
    ''''

```

---
Tags: jenkins, jenkins-pipeline
---


title: Github Actions Last Build API

text:
# Question
Title: Github Actions Last Build API
I'm looking for an option similar to Jenkins where I can view the last build, last successful build, last failed build, etc. with a simple api call. The Jenkins example would be:

```
jenkins_url + /job/${job_name}/lastBuild/

```

Does Github Actions offer anything comparable to this?

# Answer
> 2 votes
List workflow runs would be the equivalent API endpoint in Github Actions:

```
GET /repos/:owner/:repo/actions/workflows/:workflow_id/runs

```

You can use the `status` query parameter e.g. to filter for successful runs:

```
GET /repos/:owner/:repo/actions/workflows/:workflow_id/runs?status=success

```

---
Tags: jenkins, github, api
---


# Question
Title: kubernetes How to test if an operator has already been applied
How can I check if an operator has not changed, and then not apply it?

usually, I would apply an operator like this

```
kubectl apply -f openebs.yml

```

In an automated script, I would like to test if Kubernetes is running this operator already - and only apply it if it has changed or even stop if there are changes.

How can I achieve this?

# Answer
Not sure why this is required since Kubernetes is declarative and will not do anything is nothing has changed. Nonetheless, you can run it with the `--server-dry-run=true` flag. What attributes are going to be changed in the resource will not be mentioned but it does mention which resource(s) will be changed in the apply. You will need to parse the output message to identify if something changed or not since the exit code is always 1. Sample outputs:

```
$ kubectl apply --server-dry-run=true -f ./configmap.yaml # First run
secret/my-conf created
$ kubectl apply --server-dry-run=true -f ./configmap.yaml # Re-run without modifications
secret/my-conf unchanged
$ kubectl apply --server-dry-run=true -f ./configmap.yaml # Re-run with modifications
secret/my-conf configured

```
> 1 votes

---
Tags: kubernetes, kubectl
---


# Question
Title: Unable to connect to remote service using AWS VPC Peering
I have a question regarding some AWS infra I'm trying to build. Due to legacy constraints, I have a service running in a remote VPC that I need to access, and I have a new service I'm developing in another VPC. the reason the 2 VPCs are separate is because the newer VPC is managed by Terraform and I am planning to migrate all the services into Terraform but it's going to take a long time, so for the present time I want to connect the 2 VPCs using AWS VPC Peering (I own both VPCs and both services, I can control anything in this infrastructure, except for the remote service code and deployment details which is a black box).

The remote service I believe is deployed as a single EC2 instance, the new service is deployed using ECS Fargate with an ALB on top. There are 3 security groups involved; the remote instance security group, the ECS security group, and the ALB security group.

I looked at the documentation here as well as a bunch of related pages, and did the appropriate steps. In particular, I did:

* Create the VPC Peering connection
* Accept the peering connection
* Add entries to both VPC routing tables where the destination is the other VPC's CIDR block and the destination is the ID of the peering connection
* Add a TCP exception to the remote service security group to allow incoming traffic from the ECS security group (by security group ID), and vice-versa
* Check my Network ACL to make sure I have an Allow rule to accept traffic

For some reason, when I try to make my request over the peering connection, the request gets thrown into the void and never returns. It's not logged on the remote server meaning that I don't think it ever arrives. I spun up an EC2 instance inside the ECS security group and tried to call the remote resource manually (using curl) and that didn't work either.

I've followed all the guides I could find and can't figure this out, but I'm a newbie to AWS so maybe there's something I missed. Does anyone have any other ideas? Thanks.

# Answer
How are you trying to connect to the remote service? - IP address, default internal/external DNS name or custom DNS name private/public hosted zone? Please compare the output of nslookup to the remote service in both the VPCs. If the IPs resolving are different, the traffic is trying to go through the public internet and not the peering connection.

It would most probably be the need to either share the private hosted zone with the peered VPC or share internal hostnames across the peering connection
> 1 votes

---
Tags: amazon-web-services, security, aws-vpc
---


# Question
Title: Can I use AWS Step Functions to launch many different fargate 'runs'?
I am building a service that has two independent steps. The first step is a data transformation step that has to happen and is common to all downstream processes, then the second is starting a few dozen independent parallel processes that use the same input data, save for a few different runtime specified parameters.

I know I can use a step function to set off the data transformation step and wait for it to complete before starting off a single fargate task, but can I get it to start a few dozen (or more) in parallel using a common docker object but pass each of those runs slightly different parameter values.

I want to use fargate because we are a small team and I don't want to have to spend team resources on maintaining compute resources, and I can't use AWS Lambda because we expect the run times will often exceed 15 minutes, and there isn't an easy or obvious way to break the problem down any further to be certain the jobs will be able to execute within 15 minutes.

# Answer
Yes, you can start ECS Fargate tasks from Step Functions in the way you've described.

Step Functions allows you to pass `Parameters` to the underlying service API.

https://docs.aws.amazon.com/step-functions/latest/dg/connect-parameters.html

*\[sidebar: that doc also specifies how to pass state inputs, which you've stated is a requirement\]*

For ECS Fargate, a single state machine with no unique configuration could look something like this:

```
 "States": {
    "${name}": {
      "Type": "Task",
      "Resource": "arn:aws:states:::ecs:runTask.sync",
      "Parameters": {
        "LaunchType": "FARGATE",
        "Cluster": "${cluster}",
        "TaskDefinition": "${task_definition}",
        "NetworkConfiguration": {
          "AwsvpcConfiguration": {
            "Subnets": ["${subnet_1}", "${subnet_2}"],
            "SecurityGroups": ["${security_group}"],
            "AssignPublicIp": "DISABLED"
          }
        }
      },

```

Now, you've asked about passing each task slightly different parameter values...

Thankfully, the ECS `RunTask` API has a helpful `overrides: { containerOverrides: [] }` parameter.

https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API\_RunTask.html

Overriding the `command` (for task execution) or `environment` (for OS variables) should provide the customization you require without needing multiple ECS Task Definitions.

So, add each distinct task to your state machine, each ingesting state inputs and with its own unique override, and that should do it!
> 2 votes

---
Tags: docker, amazon-web-services, fargate, step-functions
---


# Question
Title: How do you simulate "sudo su <user>" in Ansible?
I have a server where my user is able to sudo to the root user without a password, and then the root user can sudo to a third user without a password. However, my user cannot sudo directly to the third user without a password.

Ansible's become directive uses sudo in the traditional manner, i.e. the one that requires prompting for my password.

I've tried putting `become_user: root` on a block and `become_user: <other-user>` on a task inside that block, but it appears Ansible overrides the block's definition, rather than nesting the sudo calls as I hoped.

Note: this question has also been asked on StackOverflow, but since there's no solution I figured I would repost it here, which I think is the better site for it.

# Answer
> 2 votes
I met the same situation. I have `login1` for my machine, passwordless sudo and `login2`, under which I should perform some actions. I did not solve it by ansible means. But I made such a workaround:

```
- name: "Install nvm"
  shell: sudo -u buildkite-agent bash -c "<my commands to be performed on behalf of buildkite-agent>"
  become: true

```

Such a case:

```
- name: "Install nvm"
  shell: sudo -u buildkite-agent <my commands to be performed on behalf of buildkite-agent>
  become: true

```

do not change home to `buildkite-agent`'s home, i.e. uses `/root` as home.

# Answer
> 1 votes
Become super user at the play level and become another user at the task level:

```
- hosts: all
  become: yes
  tasks:

    - file:
        path: /tmp/test
        state: touch
      become_user: www-data

```

Validation:

```
$ ls -l /tmp | grep test
-rw-r--r-- 1 www-data www-data    0 Mär 10 14:08 test

```

For very simple cases, a workaround like the following could also help, but this doesn't really scale, as you cannot use ansible modules with this strategy:

```
- name: test
  command: "sudo -u www-data whoami"
  become: true

```

Output (extract): `"stdout": "www-data"`

# Answer
> 0 votes
What I did that worked for me :

```
- name: launch pg_ctl start
  command: "su <USER> -l -c '<COMMAND>'"
  become: yes
  become_exe: sudo

```

PS : With `su -l`, it automatically load the `.bash_profile` and not the `.bashrc`

# Answer
> -1 votes
For example you want to create a file you can use :

```
 name: Create file
 shell: "touch abc"
 become_user: <username>
 become: yes

```

This is equivalent to :: sudo su username -c "touch abc"

---
Tags: ansible
---