# Question
Title: Where Azure DevOps Services Enviromnent come into play?
I am trying to understand where you actually use the environments you setup in a project. I though I could use them to deploy on specific servers from deployment agent which are hosted on another on premise server.

The example given in the documentation shows that you can define the environment inside a deployment job. But I can't find it inside the UI when defining a release pipeline

Where do you use environments if you're not using a yaml pipeline ?

```
- stage: deploy
  jobs:
  - deployment: DeployWeb
    displayName: deploy Web App
    pool:
      vmImage: 'Ubuntu-16.04'
    # creates an environment if it doesn't exist
    environment: 'smarthotel-dev'
    strategy:
      runOnce:
        deploy:
          steps:
          - script: echo Hello world

```

# Answer
> 1 votes
You need to enable the multi-stage pipelines preview feature. Here is a guide to enable preview features.

---
Tags: azure-pipelines, release
---


title: JFrog on dedicated disk

text:
# Question
Title: JFrog on dedicated disk
Is it possible to install JFrog on a separate disk? For example in an AWS EBS

I find their documentation a bit confusing, so i checked don't have a walk-through for this. I'm looking at this documentation: Installing on Linux Solaris or Mac OS.

# Answer
> 2 votes
It is.

You can install the ZIP installation and unzip it in the EFS. Alternatively, install using Debian/RPM and mount the EFS to `/var/opt/jfrog/artifactory` in advance.

Alternative approach can be to only configure the Filestore to use the EFS.

---
Tags: amazon-ec2, artifactory, amazon-ebs
---


# Question
Title: OpenShift fails to deploy jenkins - connection refused
I am using the GitHub project elio/openshift-vagrant. vagrant up and then the script oc-up.sh gives me a master and two nodes. Ansible provision runs without problems.

After provision I can login to https://master.example.com:8443 as admin. I can create a project. When I try to deploy *jenkins-ephemeral* this fails with message

> error: couldn't get deployment jenkins-1: Get https://172.30.0.1:443/api/v1/namespaces/test2/replicationcontrollers/jenkins-1: dial tcp 172.30.0.1:443: connect: connection refused

Ansible installer status shows no problems in the cluster. Why is this not working? How can I troubleshoot this problem?

# Answer
> 1 votes
Have you set Jenkins ssh key to have access to source? Another point is to validate if the SSH port is open. According to error can be one of these things

---
Tags: jenkins, openshift
---


# Question
Title: How to schedule a Jenkins Job dynamically based on the time value in database
I have a situation where there is a database in which a table consists of scheduled time to build. This schedule time to build may vary from day to day. Say for example,

```
Today the entry in db is 09/01/2020 5:00 PM
Tomorrow the entry can be 10/01/2020 4:00 PM

```

So, based on the above table, I need to schedule a Jenkins job today at 5:00 PM and tomorrow at 4:00 PM Is there any possibility in Jenkins to configure the build based on the above table entries.

Thanks in advance

# Answer
> 1 votes
Jenkins itself isn't very good at using external data sources to drive dynamic job scheduling. The reason for this is that job configuration, including scheduling, is built statically, either manually via the web UI, or with code via a Pipeline script which is run at job execution time. Even if the Pipeline script builds the job configuration in a dynamic way, the configuration itself will be locked in until the next job execution.

The easiest way to have flexible build schedules is use an external source to drive your builds. The hook mentioned in the comments would be ideal, but it sounds like that is not possible for you. Another idea would be to have a cron job on your database server that runs once a day, grabs the build time from the database, and schedules the build for that time. I'm sure this isn't a great solution, but it's just an example.

---
Tags: jenkins
---


# Question
Title: Calling a Terraform object in seperate folder
I need to reference my VNET Gateway residing in an infrastructure folder, in a VPN folder in order to connect a VPN utilising a Local Network Gateway, but these could be any object in Azure for the sake of argument. These folders both use seperate state files hosted in Azure blob storage.

What would be the best way to go about calling this VNET Gateway's ID in another folder? The usual methods of using e.g "module.vnet\_gateway.id" or e.g "data.azurerm\_vnet\_gateway.id" won't work in this scenario.

I've considered some way of referencing it by name, which I know, and pulling the id as an output, but I'm not certain this is the best method.

I've read about using remote-state, which I assume as per the guides on Google would involve creating a data-source pointed to my specific state file for "inf", and then pulling the data back, but I've yet to find a good example on Azure of how to achieve this.

Any pointers or examples StackExchange could provide to get me in the right direction?

# Answer
> 2 votes
Using a remote state file and referencing the state from a different state template would be the way I would go. Would need to add an output to resource that needs to be shared as well as configuring remote state storage, access controls, and security. This URL has example of setting up the remote backend in azureRM. https://www.terraform.io/docs/backends/types/azurerm.html

---
Tags: terraform, azure-devops, azure
---


title: JenkinsFile to get git branch name

text:
# Question
Title: JenkinsFile to get git branch name
I want to get git current checkout BRANCH NAME in my Jenkins file here is the code snippet... Here I am trying to push a file to my current branch I have tried many things could not find it...

```
stage('Push yaml file to bitbucket'){
  steps{
    withCredentials([usernamePassword(credentialsId: '8e2ca0a6-2dc5-4f89-b76a-03076d6b0843', passwordVariable: 'GIT_PASSWORD', usernameVariable: 'GIT_USERNAME')]) {
      sh '''
        echo '#!/bin/sh' > /home/jenkins/git_askpass.sh
        echo "exec echo \"${GIT_PASSWORD}\"" > /home/jenkins/git_askpass.sh
        chmod 775 /home/jenkins/git_askpass.sh
        echo "user: ${GIT_USERNAME}"
        GIT_ASKPASS=/home/jenkins/git_askpass.sh
        export GIT_ASKPASS
        cat /home/jenkins/git_askpass.sh
        pwd
        git status
        git checkout feature/OSCCX-44(here I want branch name )
        git config --global user.email "xyz.com"
        git config --global user.name "xyz"
        git add yaml_dev/04-idm-frontend-depl.yaml
        git commit -m "jenkins commit"
        git push https://${GIT_USERNAME}@bitbucket/repository '''
    }
  }
}

```

# Answer
> 2 votes
If you check the Git plugin documentation, you can see that there is a `GIT_BRANCH` environment variable that you can use in your pipeline.

```
GIT_BRANCH - Name of the remote repository (defaults to origin),
             followed by name of the branch currently being used,
             e.g. "origin/master" or "origin/foo"

```

So, in your pipeline:

```
stage('Push to Bitbucket') {
    steps {
        withCredentials([usernamePassword(credentialsId: '8e2ca0a6-2dc5-4f89-b76a-03076d6b0843',
                                          passwordVariable: 'GIT_PASSWORD',
                                          usernameVariable: 'GIT_USERNAME')]) {
            sh '''
            # Content omitted
            git checkout ${GIT_BRANCH}
            # Content omitted
            '''
        }
    }
}

```

# Answer
> 0 votes
Note that if you're not actually checking out a Jenkinsfile (if you're coding in the Pipeline Script config window), `GIT_BRANCH` will be `null`, and Hedi Nasr's answer will fail (you can access the var with `${env.GIT_BRANCH}` instead to avoid the hard fail, but it will still be `null`).

You don't give any information about how you checked out the project, so I'm going to assume you cloned it at some point during the execution of your Jenkinsfile. In that case, the repository will be in a detached state, even if you cloned it with a target branch:

```
git status
# HEAD detached at 9a92344

```

In this case before you can commit you need to checkout your branch as Hedi Nasr points out. You should know your target branch in advance, and can inject it with envInject's Properties Content if you need to vary by instance.

You can also discover branches with:

```
git branch -a

```

---
Tags: git, jenkinsfile, bitbucket
---


# Question
Title: How to define variable in Jenkins declarative pipeline?
I defined variable in declarative Jenkins pipeline script but having issues with simple variable declaration.

Here is my script:

```
pipeline {
    agent none
    stages {
        stage("first") {
            def foo = "foo" // fails with "WorkflowScript: 5: Expected a step @ line 5, column 13."
            sh "echo ${foo}"
        }
    }
}

```

but it's shows error:

```
org.codehaus.groovy.control.MultipleCompilationErrorsException:
startup failed:
WorkflowScript: 5: Expected a step @ line 5, column 13.
    def foo = "foo"
    ^

```

# Answer
> 26 votes
The variable must be defined in a `script` section.

```
pipeline {
    agent none
    stages {
        stage("first") {
            steps {
                 script {
                      foo = "bar"
                 }
                 sh "echo ${foo}"
            }
        }
    }
}

```

# Answer
> 7 votes
You can also use environment block to inject an environment variable.

(Side note: `sh` is not needed for echo)

```
pipeline {
    agent none
    environment {
        FOO = "bar"
    }
    stages {
        stage("first") {
            steps {
                echo "${env.FOO}"
                // or echo "${FOO}"
            }
        }
    }
}

```

You can even define the env var inside the stage block to limit the scope:

```
pipeline {
    agent none
    stages {
        stage("first") {
            environment {
                FOO = "bar"
            }
            steps {
                // prints "bar"
                echo "${env.FOO}"
                // or echo "${FOO}"
            }
        }
        stage("second") {
            steps {
                // prints "null"
                echo "${env.FOO}"
                // or echo "${FOO}", pipeline would fail here
            }
        }
    }
}

```

---
Tags: jenkins, jenkins-pipeline
---


title: Configure VPC for ECR access

text:
# Question
Title: Configure VPC for ECR access
I am creating my infrastructure with Ansible. I create a VPC and an ECS cluster where I use Fargate to run my docker containers. When the tasks are started I am prompted that the task was not able to pull the Image from the ECR. `STOPPED (CannotPullContainerError: Error response from daem)`

I know for a fact that the URI to the image and the IAM permissions are correct. Also I don't think the issue is in the AMI or docker image, because when I create the same scenario over the AWS UI, it works fine.

Therefore, I assume my VPC is not configured correctly in a way that it can not connect to the ECR.

This is how I create my VPC and Subnet:

```
- name: "2) Create VPC"
  ec2_vpc_net:
    name: "{{vpc_name}}"
    state: present
    cidr_block: 10.10.0.0/16
    region: "{{ region_name }}"
  register: vpc_net

- name: "3) Create VPC Subnet"
  ec2_vpc_subnet:
    state: present
    map_public: yes
    vpc_id: "{{vpc_net.vpc.id}}"
    cidr: 10.10.0.0/24
    tags:
      name: "test-subnet"
  register: vpc_subnet

```

How can I ensure my VPC and subnet is configured correctly so that the docker daemon can connect and pull the image from the ECR repo? Am I missing an important attribute?

# Answer
> 4 votes
This error `CannotPullContainerError` usually occurs when you have no access to the Internet from ECS and thus it doesn't able to pull an image from registry.

Make sure you have networking configured in such a way to have an access to Internet https://stackoverflow.com/questions/48226547/aws-fargate-cannotpullcontainererror-500

Manually you can confirm everything with such steps

1. Find route table. Check whether it belongs to your VPC.
2. Find proper NAT (you need NAT Gateway ID) (**IMPORTANT**: NAT should be launched in PUBLIC subnet)
3. Create a route for that

$ aws ec2 create-route --route-table-id "rtb-037148d7b5967a231" --destination-cidr-block "0.0.0.0/0" --nat-gateway-id "nat-0c137ae8a2b409088"

You can also use internet gateway instead of NAT but the main idea is the same - to create connectivity from service to registry. If you have that in place and verified you can easily rewrite that to ansible tasks.

**Update** To start an ECS task you want to use `ecs_task` module https://docs.ansible.com/ansible/latest/modules/ecs\_task\_module.html

To pull image from repository and start it you can use these tasks

```
- name: docker login
  shell: "$(aws ecr get-login --no-include-email --region {{ default_region }})"
  args:
    executable: "/bin/bash"

- name: pull latest app image
  docker_image:
    name: "{{ ecr_repository }}/myapp:{{ image_tag }}"
    force: yes

- name: run application with docker compose
  docker_service:  # module is called docker_service before Ansible 2.8
    project_name: "myapp"
    project_src: "{{ app_dir }}"  # path to a directory containing a docker-compose.yml
    pull: yes
    recreate: always  # run scheduler from a clean state
    remove_orphans: yes  # remove containers for services not defined in the compose file
    state: present  # specifying present is the same as running docker-compose up
    restarted: yes  # use with state present to restart all containers
  register: output

```

but in case of Fargate the above is not needed as everything is handled by task definition

# Answer
> 1 votes
ECR needs authentication and authorization before you can pull an image.

In your ansible ecs\_taskdefinition, please make sure you have parameter `task_role_arn` or `execution_role_arn` pointing to a role (i.e. you need to create a role `my-task-role1`).

Attach the AWS managed policy `AmazonECSTaskExecutionRolePolicy` to the role (for Fargate tasks, the policy `AmazonEKSFargatePodExecutionRolePolicy` might be a better choice).

This should provide you with the necessary permissions to download your image from ECR.

---

Policy descriptions from AWS Console:

* `AmazonECSTaskExecutionRolePolicy`
> Provides access to other AWS service resources that are required to run Amazon ECS tasks
* `AmazonEKSFargatePodExecutionRolePolicy`
> Provides access to other AWS service resources that are required to run Amazon EKS pods on AWS Fargate

---

Side-note: AWS Console creates `ecsTaskExecutionRole` automatically if you create a Task Definition there.

Description from "*Create new Task Definition*" in AWS Console:

> This role is required by tasks to pull container images and publish container logs to Amazon CloudWatch on your behalf. If you do not have the ecsTaskExecutionRole already, we can create one for you.

---
Tags: ansible, amazon-ecs, aws-vpc, aws-ecr, fargate
---


# Question
Title: File transfer from Application server to Db Server without a shared folder access but using Service Account
I am working as a Business Analyst for an IT firm. My team has stumbled upon a certain feature where they are required to use Service account. We are working for a Bank and they have strong policies of keeping no Shared folder on Db server, for this they have advised us and provided us Service accounts with help of their server team. Naturally my team is unaware on how to use this Service account for transferring the file From App server A to Db server B and thus i will be needing help from you guys.

If you ask me i dont know exactly what a service account is, but judging by the name it is a credentials which allows access within different servers. Can anyone advice on how to use this for file tranfer?

# Answer
There is a general lack of understanding here. I suspect your company doesn't understand your "requirement" and the Server team has provided something that either *won't* do the job or your team don't know *how* to make it work.

You say that you "need" to get a file onto a Database server - my first question would be - "**why**"?

What's *in* this file that you need to put it *anywhere near* the Database server?

Your company has a Policy against doing this through file shares on the Database server. \[Being a DBA\] I completely understand (and *agree*) with this Policy.

On its own, a Service Account isn't going to be much use to you.
You *might* be able to use it with some File Transfer utility to copy the file but, again, *why* do you want to do this?

Take a step back and describe the "Big Picture" problem that you're trying to solve here.
> 1 votes

---
Tags: databases
---


# Question
Title: Azure Windows Container: How to read secrets from an azure vault
I am coming from AWS / Linux and try this in Azure / Windows, now.

**The Question:**

I have a windows container in Azure and want it to get some secrets from an Azure Vault. How can I do that?

**This is how I thought it would work (but didn't):**

I had expected that I could just run

`az keyvault secret show --name "my_secret" --vault-name "my_fault"`

... at least after adding permissions to the container. Unfortunately, I cannot find out how to add permissions to the container to access the vault.

**What I have tried so far:**

I have created the vault and added an "application" under "Access Policies". I do not find any way how to make the container act as the "application".

I have read that items need to get an "identity assigned" , but this seems to be not supported with windows containers . I had tried it out before I found the documentation, and it did not work.

How can I access the azure key vault from an azure container?

# Answer
> 1 votes
Assuming this is transient container then my suggestion to work around the not yet implemented issues would be that whatever creates it does so by passing in secure environment variables for the values you want from the key vault; then your scripts can access the data that way.

I'm not sure what your security considerations are so you'll need to take that into account so people can't login to the container to view the environment variables.

As an example I have a function that uses the .Net Fluent API to create a container group and then sets the environment variables for the container using ".WithEnvironmentVariablesWithSecuredValue()" this means anyone whose looking at the container in the portal won't be able to see the values in plain text.

Then you assign a managed identity to the function and then give that identity get permissions to secrets in the key vault and then you can either access the key vault in code in the function or the easier option is the put them as app settings for your function and then just get them as environment variables in your function - https://docs.microsoft.com/en-us/azure/app-service/app-service-key-vault-references

Hope this helps or gives you an idea on a workaround?

---
Tags: containers, azure
---


# Question
Title: How to create dependency between modules in terraform
I have 2 modules A and B. I want a particular resource in module B to be created only after a particular resource in module A is completed completely.

I was trying to achieve just that with this as output in the moduleA

```
output "alb_arn" {
  value = "${aws_alb.model-server.arn}"
}

```

Then in the main.tf i.e root i added this,, in the moduleB

```
  dependency                          = "${module.common.alb_arn}"

```

Then in the moduleB variable file referenced it like this.

```
variable "dependency" { }

```

And in the moduleB main,tf used it as depends\_on like this

```
resource "aws_alb_target_group" "modelServer" {
  name_prefix  = "rqm-"
  port         = "${local.modelServer_container_port}"
  protocol     = "HTTP"
  vpc_id       = "${lookup(var.vpc, var.env)}"
  health_check = "${list(local.health_check)}"
  lifecycle {
    create_before_destroy = true
  }
  depends_on = [ "${var.dependency}" ]
  tags {
    Name = "rqm-modelServer-${var.env}"
  }
}

```

Error:

```
Error: aws_alb_target_group.modelServer: depends on value cannot contain interpolations: ${var.dependency}

```

# Answer
> 3 votes
What you've described here is the recommended way to pass dependencies between modules, but it relies on features introduced in Terraform 0.12. You appear to be using Terraform 0.11, where unfortunately these features are not available.

For Terraform 0.12 it would be idiomatic to write it using the "first-class expressions" syntax, like this (with these examples spanning the three modules just like you did in your opening example):

```
output "alb_arn" {
  value = aws_alb.model-server.arn
}

```

```
module "common" {
  source = "./modules/common"
}

module "other" {
  source = "./modules/other"

  target_group_depends_on = module.common.alb_arn
}

```

```
variable "target_group_depends_on" {
  type    = any # only the dependencies matter, not the value
  default = null
}

resource "aws_alb_target_group" "modelServer" {
  # ...

  depends_on = [var.target_group_depends_on]
}

```

---

The main missing feature for the above in Terraform 0.11 is that it only supports resources in `depends_on`, and not other kinds of object like variables.

However, it's possible to work around that by introducing an additional do-nothing resource that exists only to be a dependency:

```
output "alb_arn" {
  value = "${aws_alb.model-server.arn}"
}

```

```
module "common" {
  source = "./modules/common"
}

module "other" {
  source = "./modules/other"

  target_group_depends_on = "${module.common.alb_arn}"
}

```

```
variable "target_group_depends_on" {
  type    = "string" # (because "any" isn't supported in 0.11)
  default = ""
}

resource "null_resource" "target_group_depends_on" {
  triggers = {
    # The reference to the variable here creates an implicit
    # dependency on the variable.
    dependency = "${var.target_group_depends_on}"
  }
}

resource "aws_alb_target_group" "modelServer" {
  # ...

  # Marking the null_resource as an explicit dependency
  # means this indirectly depends on everything the
  # null_resource depends on.
  depends_on = ["null_resource.target_group_depends_on"]
}

```

Note that in Terraform 0.11 the `depends_on` argument takes a list of literal strings containing resource addresses, not interpolated values. That was the specific reason for the error message you saw when you tried your example, but a reference like `"var.dependency"` would not have worked there either because only resource addresses are accepted as explicit dependencies in Terraform 0.11.

---
Tags: terraform, terraform-module
---


# Question
Title: What are reasons for not not using Bitbucket-server for storing artifacts?
I'm working with a company setting up a brand new project, and we're talking about what tools to use for what. I was talking about Artifactory or Nexus for storing built artifacts (APKs in this case), and they asked why they can't just use Bitbucket like they will use for the software, to reduce the number of tools they need to install and maintain.

My initial response was "Source goes in source repo and artifacts go in artifact repo", but that's not actually an answer why or why not. In fact Googling around didn't offer any reasoning either.

ADDING TO THIS: This product is for a regulated industry. That means we need a complete versioned, auditable record of the artifacts built. This is one of the reasons we're looking into this. Also, as I said in some comments but will add here, we will install Bitbucket in a private GCP not accessible by the public, so there are no concerns about others accessing it.

Any input into this? Anything that would bite us later on if we store them in Bitbucket?

# Answer
Reasons not to store large binaries in a `git` repository:

* Everbody cloning your repository will download all those binaries, by default. Binaries, if built regularly, tend to consume *massive* amounts of storage, compared to source code - `git` cannot compress them, or calculate deltas, to reduce their size.
* `git` goes to great lengths to make sure history is not lost, it will never delete anything that is part of the history of any `ref` (branches or tags). That also means that if you later decide to delete old, long-useless binaries due to running out of space, you will find that, while not impossible, very hard indeed.
* One point of proper artifact stores is that they help phasing out outdated or compromised parts. `git` cannot really do that for you - you'd have to write your own tooling for that - and you will never get rid of the old versions in history.
* The concept of a "commit" does not really apply to binaries. You need operations like "upload" and "download", as they are regularly involved in the build processes of your software (i.e., `bundler` in the ruby world, or `maven` in Java). Those builders know how to fetch their 3rd party libraries from artifact repositories easily, and how to upload new versions. They may be convinced to work with a `git` repository for downloads, but since `git` has no versioning information, you again need to have your own tooling, specify the commit in which to find that binary manually, or have *all* binaries ever in use checked out locally. And uploading to a `git` repository, again, would use your own tooling (creating spurious commits as well).

In total, using `git` for that is just the wrong tool. If your colleagues are afraid of adding yet another complex tool like Nexus, then at least convince them to use a *simple* tool instead of `git` \- like some arbitrary `(s)ftp`/`scp` repository in a VM somewhere, with `https` access via a simple webserver.

If you *must* use `git`, then at least make sure the build binaries are not committed together with the source code, but in their own part of history. Check out this older answer to see how to create an orphan branch. Those can at least be deleted later, and the binaries themselves removed by garbage collection; and not every client is forced to download all of it all the time.
> 12 votes

# Answer
You can use a Git repository (whether it is hosted on Bitbucket or not) as an artifact repository, but you should be aware that:

* Git was originally made as a version control system *for source code*, not for (large) binary data, and this is still its main concern. There are extensions that allow working with large files in Git efficiently, e.g. Git LFS, but still: It is not built into the core of Git.
* A proper artifact repository has features that a Git based service cannot easily provide. For example you probably want to delete older snapshot artifacts, an operation Git is not designed for, but what is a core feature of artifact repositories. Some more of such features are described in the answers to the question *What is an artifact repository?*.

So while you *can* use Git/Bitbucket as an artifact repository, it's a bit like driving a screw with a hammer instead of a screwdriver.
> 5 votes

# Answer
If you will store apk files inside Bitbucket's repo you need use git or curl it to clone. And if repo private you need have an access to it, size limitation etc.

The Software repository is more comfortable to store binary files. You can create your own mirror of maven, google repos.
> 0 votes

# Answer
You could use bitbucket pipelines to deploy to the download section and keep your artifacts there. Here is an example from their documentation
> 0 votes

# Answer
As others have said, you wouldn't check the binaries into git. But, Bitbucket offers a separate "Downloads" area per-repo. This is a separate area, not part of the git repo. If using Bitbucket Pipelines, you could even automate the deployment to the downloads area, if that suits the workflow.
> 0 votes

---
Tags: artifacts, bitbucket-server
---


title: Docker preserve images policy

text:
# Question
Title: Docker preserve images policy
Can I configure Docker in a such way that it will store only certain amount of tags for the given image e.g. `docker images`

```
REPOSITORY                     TAG                                      IMAGE ID            CREATED             SIZE
ecr.us-east-1.amazonaws.com/myapp   02e56bac4ee3f27a4d5670a6ac0d578e3dd5b2e5   af154c422af8        2 days ago          1.25GB
ecr.us-east-1.amazonaws.com/myapp   d43d33c8e15a5578cf9f6831f94a18f94e9b932f   e9958837c2c5        3 days ago          1.25GB
ecr.us-east-1.amazonaws.com/myapp   3a97c3ac36f6c241fb061199c5d3347b9bbff94a   3e80e7a25667        3 days ago          1.25GB
ecr.us-east-1.amazonaws.com/myapp   12f9a2890c67d42eb050e7d06dd593ec932b1ee7   8c07b05cd756        4 days ago          1.25GB
ecr.us-east-1.amazonaws.com/myapp   7b55fb6a75c73d16cbc731050d29323ae6b90632   426530a5cc66        4 days ago          1.22GB
ecr.us-east-1.amazonaws.com/myapp   c222e88d6b24984f338091cc5600aabb62e0ac72   e65e650d2dd1        2 weeks ago

```

and I want to store locally only N **most recent** images. Can I configure some kind of policy or is there any common script that runs rotation with cron or something like that? Other best practices/suggestions are welcome.

# Answer
> 1 votes
So I ended up using this simple shell script

```
docker images -q --filter "before=ecr.us-east-1.amazonaws.com/myapp:02e56bac4ee3f27a4d5670a6ac0d578e3dd5b2e5" | xargs docker rmi --force

```

First you filter your images to list only those created before the latest one (you either know your recent tag or just use `latest` one)

The you just remove those images with `docker rmi` piping it to xargs

You can invoke this on each deploy or setup a cron to do that periodically.

# Answer
> 3 votes
I do not believe that there is a way to set this as a global policy. However, there is a simple command you can run that will "prune" or remove your unused images.

You will want to use `docker image prune --filter [options]` to achieve this. Check out the official documentation for information on how to best filter. As a best practice you should be tagging your images which will make managing of them easier using tools like this.

---
Tags: docker, shell-script, docker-registry, repository, images
---


# Question
Title: Using gitlab-ci to cache maven repository
We have a server running gitlab and gitlab-runners on same machine. I've defined a runner as below

```
[abc@xyz bin]# gitlab-ci-multi-runner register
Running in system-mode.

Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com/):
http://abc.xyz.com
Please enter the gitlab-ci token for this runner:
9nev5wz3voabcdefgh
Please enter the gitlab-ci description for this runner:
[opera.ishisystems.com]: second-docker-remote-runner
Please enter the gitlab-ci tags for this runner (comma separated):
docker, remote
Whether to run untagged builds [true/false]:
[false]:
Whether to lock Runner to current project [true/false]:
[false]:
Registering runner... succeeded                     runner=9nev5wz3
Please enter the executor: docker, docker-ssh, ssh, docker+machine, docker-ssh+machine, parallels, shell, virtualbox, kubernetes:
docker
Please enter the default Docker image (e.g. ruby:2.1):
node:4.2.2
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

```

and my `.gitlab-ci.yml` is

```
image: maven:3-jdk-7

services:
  - wurstmeister/kafka:0.8.2.1

variables:
  MAVEN_REPO: "$CI_PROJECT_DIR/.m2"

cache:
    key: "$CI_BUILD_STAGE"
    paths:
      - .m2

build:
  script: "mvn -Dmaven.repo.local=$MAVEN_REPO test -B"
  tags:
  - docker

```

What I'm trying to do is setup a **central maven repository** on my gitlab-runner server so that every time a new job is fired **instead of downloading** all the maven dependencies it should **use the cached repository**.

# Answer
> 7 votes
To me the easiest route to this would be to have an instance of an artifactory running. There are several choices out there many of which have an open-source, community or free version that would support maven repositories. Most options will support caching remote repositories (maven central for example) which can then be curated to enforce specific versions of the dependencies through your projects. It will also accept your own dependencies this way you can have a central unique access point for all maven dependencies in your team.

I've personally had success with Sonatype's Nexus (OSS version, or Pro version). There are other choices out there though check them out.

That said, if what you want is to prime the local repository for a new agent you want to provision, you could do so. But frankly, unless you have some real hefty dependencies (say in the GB size range), I would not bother doing so really and just let Maven naturally pull from the local binary repository instead. It makes for a simpler solution that will be more robust in the long run.

# Answer
> 1 votes
GitLab CI includes caching of directories, **relative to working directory**. You can use this to cache maven local repository.

```
myjob:
  image: maven
  variables:
    MAVEN_OPTS: "-Dmaven.repo.local=${CI_PROJECT_DIR}/.repository/"
  cache:
    key: maven
    paths: [.repository/]
  script:
  - mvn clean verify

```

You'll see something like this in job log:

```
Checking cache for maven...
Downloading cache.zip from https://storage.googleapis.com/gitlab-com-runners-cache/project/133221/maven
...
Creating cache maven...
 .repository/: found 948 matching files
 Uploading cache.zip to https://storage.googleapis.com/gitlab-com-runners-cache/project/133221/maven

```

See GitLab caching documentation.

---
Tags: gitlab
---


# Question
Title: Can I run command from a program thats based on the root host from a containerized application?
I'm in a situation where I have a Django application, Nginx webserver and Postgres database that I want to dockerize, and that's no issue I can do that. But the Django application has code that calls a subprocess to run a command from a installed program on the root host, and obviously if I run that command it's not recognized since it's not installed on the docker container.

The program can not be dockerized as far as I know, because it uses docker itself. Is there any way I can make the Django Application be able to run commands from that specific program on the root host, even though the Django application is containerized?

Can this be done through some kind of network setup of the docker container?

I find it a little hard to explain, so please do ask questions if something isn't clear enough.

# Answer
This StackOverflow question has answers that could be useful for you, in particular, mounting parts of the hosts filesystem containing the host application.

But really, it doesn't seem to make much sense to run an application in a container and then try to directly access the host system. It seems to defeat many of the reasons to containerize an application in the first place.

It'd probably much better to put the required application in the container, along with Django. I see no reason why your container wouldn't be able to run docker itself (even though for some CI/CD use cases sibling containers are preferred over docker-in-docker approach).
> 4 votes

---
Tags: docker, docker-compose
---


# Question
Title: Is there any concrete and acceptable solution for running systemd inside the docker container?
I have seen many workarounds for this to run systemd inside docker containers but looks like most of them compromise the security of the container and the host. How are most people here dealing with running systemd specific stuff inside the container.

# Answer
You didn't mention what distribution you're using inside your container (which would have implications w/r/t which version of systemd you're using), but the following will successfully boot a CentOS container running systemd:

```
docker run -it --rm \
  -e container=docker \
  --tmpfs /run \
  --tmpfs /tmp \
  -v /sys/fs/cgroup:/sys/fs/cgroup:ro \
  --cap-add SYS_ADMIN \
  centos /sbin/init

```

This is with Docker 17.05.0-ce; older versions may require additional flags. Using a stock `centos:7` image, your initial environment inside the container looks like this:

```
# systemctl status
    State: running
     Jobs: 0 queued
   Failed: 0 units
    Since: Mon 2017-12-04 17:47:15 UTC; 45s ago
[...]

```

And:

```
# ps -fe
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 17:47 ?        00:00:00 /sbin/init
root        16     1  0 17:47 ?        00:00:00 /usr/lib/systemd/systemd-journald
dbus        26     1  0 17:47 ?        00:00:00 /bin/dbus-daemon --system --address=systemd: --nofork --nopidfile --systemd-activation
root        28     1  0 17:47 ?        00:00:00 /usr/lib/systemd/systemd-logind
root        30     1  0 17:47 console  00:00:00 /sbin/agetty --noclear --keep-baud console 115200 38400 9600 vt220
root        32     0  0 17:47 ?        00:00:00 bash
root        58    32  0 17:48 ?        00:00:00 ps -fe

```

Note that I'm using `--rm` here not because it's necessary but because I'm terrible at cleaning things up after the fact. It's not necessary to get the container to run.

> but looks like most of them compromise the security of the container and the host

Well, running systemd does require privileges beyond those granted to a typical Docker container (hence the `--cap-add`). Whether this has security implications for your environment or not depends on what you're doing.
> 6 votes

# Answer
For example, if you want to run systemd in a Arch Linux container, run this command: `docker run -it --privileged -v /sys/fs/cgroup:/sys/fs/cgroup:ro --name=ArchLinux archlinux /bin/sh -c "if [ -x /etc/docker-start ]; then exec /etc/docker-start; else exec /bin/sh; fi"`

Then run these commands in the container:

```
echo 'Server = https://THE_FASTEST_MIRROR_FOR_YOU/archlinux/$repo/os/$arch' >/etc/pacman.d/mirrorlist
pacman -Sy --noconfirm systemd systemd-sysvcompat
passwd -d root
echo -en '#!/bin/sh\numount /etc/hostname; umount /etc/hosts; umount /etc/resolv.conf; exec /usr/lib/systemd/systemd' >/etc/docker-start
chmod 700 /etc/docker-start
echo -e 'nameserver 8.8.8.8' >/etc/resolv.conf
exit

```

Then start the container and attach to it: `docker start -ia ArchLinux`

You need to run with `--privileged`, and this may cause some security problems. Please try to avoid it if possible.
> 2 votes

# Answer
According to this blog, it is possible to run systemd inside a docker container by building the following dockerfile:

> ```
> FROM fedora:rawhide
> MAINTAINER "Dan Walsh" <dwalsh@redhat.com>
> ENV container docker
> RUN yum -y update; yum clean all
> RUN yum -y install systemd; yum clean all;
> (cd /lib/systemd/system/sysinit.target.wants/; for i in *; do [ $i == systemd-tmpfiles-setup.service ] || rm -f $i; done);
> rm -f /lib/systemd/system/multi-user.target.wants/*;
> rm -f /etc/systemd/system/*.wants/*;
> rm -f /lib/systemd/system/local-fs.target.wants/*;
> rm -f /lib/systemd/system/sockets.target.wants/*udev*;
> rm -f /lib/systemd/system/sockets.target.wants/*initctl*;
> rm -f /lib/systemd/system/basic.target.wants/*;
> rm -f /lib/systemd/system/anaconda.target.wants/*;
> VOLUME [ "/sys/fs/cgroup" ]
> CMD ["/usr/sbin/init"]
>
> ```

using this command:

> ```
> docker build -t httpd_rawhide .
>
> ```

The author indicates that the built docker image could be used as a base image

> ```
> FROM systemd_rawhide
> RUN yum -y install httpd; yum clean all; systemctl enable httpd.service
> EXPOSE 80
> CMD ["/usr/sbin/init"]
>
> ```

and if the container is run

> ```
> docker run –privileged -ti -v /sys/fs/cgroup:/sys/fs/cgroup:ro -p 80:80 httpd_rawhide
>
> ```

systemd will run inside the docker container. It is also possible to run multiple services using systemd. According to the creator of this blog it would be possible to run both mariadb and http inside the same container.

Based on this blog and other articles I read my conclusion is that it is technically possible to run systemd inside a docker container, but I would recommend to avoid running systemd inside a container.

First of all, if systemd is able to run inside a container then this means that it is possible to run multiple services like the author did. From a docker perspective this is not recommended as docker is meant to scale horizontally, i.e. if the load increases of http then addtional docker images should be started.

Second, if such a systemd container will be deployed on an orchestration platform like docker swarm. Will that work? I have some doubts whether that will work.

Third, Running systemd by mounting the cgroup in a privileged container does not look very secure.

In conclusion, although you indicated that the script requires systemd, either rewrite the code or use something else. Running systemd inside a docker container should be avoided in my opinion.
> 0 votes

---
Tags: docker
---


# Question
Title: How to run Ansible command to update imunify360 settings
How can I use Ansible to run the following CloudLinux Imunify360 commands properly on CloudLinux 7 (similar to CentOS7)? I tried different ways to "escape" the colons in the commands and it doesn't run properly, even when a YAML syntax checker showed it's correct.

```
imunify360-agent config update '{"MALWARE_SCANNING": {"rapid_scan": true}}'
imunify360-agent config update '{"MALWARE_SCANNING": {"default_action": "cleanup"}, "MALWARE_SCAN_SCHEDULE":{"interval": "week"}}'
imunify360-agent config update '{"MALWARE_SCANNING": {"enable_scan_inotify": true}}'

```

from: https://blog.imunify360.com/new-default-settings-for-imunify360

Note: Imunify360 has a free trial.

# Answer
> 1 votes
It turns out single quote works where double quote and backslash didn't.

> ```
> - name: Enable rapid scan
>   command: imunify360-agent config update '{"MALWARE_SCANNING"':' {"rapid_scan"':' true}}'
>
> ```

I learned that folding block scalar (\>) is an easy alternative and a good way to avoid escaping.

# Answer
> 0 votes
Assuming that you're already planning on executing each command in a separate task, you can just use a folding block scalar (`>`). While these are typically used for multi-line strings since they replace newlines with a single space, a bonus is that they don't evaluate escape sequences, so nothing needs to be escaped.

However, `>` alone won't strip the trailing newline character which is where `-` comes in.

```
- name: Enable rapid scan
  command: >-
    imunify360-agent config update '{"MALWARE_SCANNING": {"rapid_scan": true}}'

- name: Cleanup scan schedule
  command: >-
    imunify360-agent config update '{"MALWARE_SCANNING": {"default_action": "cleanup"}, "MALWARE_SCAN_SCHEDULE":{"interval": "week"}}'

- name: Enable scan inotify
  command: >-
    imunify360-agent config update '{"MALWARE_SCANNING": {"enable_scan_inotify": true}}'

```

---
Tags: ansible, linux, security
---


# Question
Title: Is GitLab flow (or GitHub flow, etc) anti build-once deploy-everywhere?
Historically, especially in the Java world, build-once, deploy everywhere has been a very big thing for deployments. Artifacts should be built exactly one time when a release is made and stored somewhere like Nexus or Artifactory where they are immutable and can be deployed to all environments, repeatedly and safely.

GitLab flow, on the other hand, seems to recommend having a branch per environment and deploying to that environment as an event upon doing a merge request to the branch. E.g. a master -\> staging merge runs CI/CD on the staging branch, resulting in a deployment there. Merging the staging to the production branch does the same in production (with the possibility of forcing a manual agreement to the run).

So, GitLab flow causes us to rebuild our artifacts for every environment rather than build once deploy everywhere.

**Does this mean we should totally ditch build-once deploy-everywhere when moving to modern tools like GitLab/GitHub? Is that a good thing?**

I guess we can augment GitLab flow by:

* Build artifacts on merge into DEV branch and store in a maven/etc repo.
* Do deployment using those artifacts.
* Have merges to staging/prod skip build/test and only do deploy using the same artifacts.

This seems like forcing GitLab/etc to do something against their model though, so I'm thinking its not a good idea.

---

Note: I also am wondering how building/testing/deploying in every environment separately is possible in some large projects. E.g. Facebook's presto has like an hour of tests in a maven build (or more)... I wouldn't want my production release taking hours; that's the opposite of modern.

**Any thoughts here?**

# Answer
> 2 votes
One possible answer could come from the reason for using multiple branches as basis for environments in the first place.

In some organisations (typically for large scale projects and often for historical reasons) it is desired to allow for cherry-picking hot fixes in each environment to limit the amount of changes (thus the risk) picked up in each iteration on that environment. I'd argue that such approach is not really aligned with continuous integration/delivery at the project level (it may be at each of the branch levels).

In this context the branches will diverge, so build-once, deploy everywhere principle is similarly applicable only at the branch/environment level, it cannot be applied at the project level. The GitHub/GitLab flow could work well in such case.

If, however, there is no cherry-picking between branches and **strictly** straight/fast-forward merges from upstream branch refpoints are used to propagate changes to the downstream branches/environments then the reason for using multiple branches is IMHO questionable. A **single/master** branch can be used instead, the environments being implemented simply as different steps/stages in the CI/CD processing pipeline.

The pipeline would be triggered on the branch tags/refpoints from which the original change propagation would be initiated in the multi-branch scenario (ideally for each and every commit).

The build-once, deploy everywhere principle can easily be enforced in this approach, which aligns very well with continuous integration/delivery/deployment at the entire project level. You just need to consider each build as potential release candidate from the very beginning of your pipeline processing.

Note that nothing in this scenario conflicts with using GitLab/GitHub (or any other modern version control system) itself, you just don't **have to** use all the features they offer ;)

---
Tags: deployment, gitlab, continuous-deployment, cicd, build-pipeline
---


title: Question about the new AWS RDS CA

text:
# Question
Title: Question about the new AWS RDS CA
When updating the database to use the new CA, you get this popup:

> Before scheduling the CA certificate rotation, update client applications that connect to your database to use the new CA certificate. Not doing this will cause an interruption of connectivity between your applications and your database.

Does anyone know if this is required? We didn't have to import the CA originally, so why would we have to now? If it's a matter of the new root CA not being part of the trust chain of older distros, does anyone know if Ubuntu 18.04 has the new CA in its ca-certificates package yet?

Thanks!

# Answer
> 1 votes
I am not completely sure that Ubuntu 18.04 will have this CA in their certificates packages because they announced this CA replacement recently.

If you are using the SSL/TLS to connect to your DB instances, then only this will be required, otherwise you can simply ignore this.

You can find the complete details of this CA certificate rotation and it's impact in the AWS blog

# Answer
> 1 votes
As far as I read, client using that db instance has to update certificate if the client has added the db in it's truststore.

***When you schedule this operation, make sure that you have updated your client-side trust store beforehand.***

**https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/UsingWithRDS.SSL-certificate-rotation.html**

I don't think anything apart from this would be required. Hope this helps!!

---
Tags: amazon-rds
---


# Question
Title: I'm trying to run `vagrant provision` on PowerShell Window for Ansible playbook
Here is what it replies:

`Windows is not officially supported for the Ansible Control Machine.
Please check https://docs.ansible.com/intro_installation.html#control-machine-requirements`

What should I do? Thank you

# Answer
> 1 votes
You have 2 viable options to run ansible on windows 10:

* Install a linux vm with your favorite virtualization tool and run ansible inside that vm.
* Run ansible from WSL

With the first option, you will need to install vagrant inside the vm + all virtualization tools needed.

With the second option, you will then need to install vagrant in WSL as well and configure it to access your windows environment.

---
Tags: ansible, windows, provisioning, powershell
---


title: Unable to get Helm to work with ECR

text:
# Question
Title: Unable to get Helm to work with ECR
I'm using Helm v3 with AWS ECR on a Kubernetes cluster that was creating using Kops. I have checked and the role kops assigned to the nodes does have ECR permissions. When I deploy my helm chart I just get `imagepullbackoff` when doing `kubectl get pods` in the correct namespace.

Below is some of my helm chart code, I hope this is everything that's needed to debug this issue. BTW I have looked around on SO and google, both come up short on ECR with Helm and nothing for Helm 3 - was released recently.

Inside my `values.yaml`

```
image:
  repository: <number>.dkr.ecr.us-east-1.amazonaws.com/apigw
  # tag: latest
  tag: ef86dfe0b703
  pullPolicy: IfNotPresent

```

Note that the tag there is a git commit ID and spinnaker will change that on each deployment so here being static is just to test.

I was following this guide and as a result changed some things that might have effected it.

https://aws.amazon.com/blogs/opensource/deployment-pipeline-spinnaker-kubernetes/

Main one I think that makes a difference is

```
 imagePullPolicy: "{{ .Values.image.pullPolicy }}:{{ .Values.image.tag}}"

```

to

```
imagePullPolicy: {{ .Values.image.pullPolicy }}

```

However... If I change it back I get this error when trying to apply the helm chart

```
Error: Deployment.apps "apigw" is invalid: spec.template.spec.containers[0].imagePullPolicy: Unsupported value: "IfNotPresent:ef86dfe0b703": supported values: "Always", "IfNotPresent", "Never"

```

# Answer
Why are you adding the image tag to the pull policy? you dont need it there, just leave it like this: `imagePullPolicy: {{ .Values.image.pullPolicy }}`
> 1 votes

# Answer
Based on the error message it seems that the yaml is invalid:

```
Unsupported value: "IfNotPresent:ef86dfe0b703": supported values:
"Always", "IfNotPresent", "Never"

```

It seems that `IfNotPresent` contains some tag and that is unsupported.
> 0 votes

---
Tags: docker, amazon-web-services, kubernetes, helm
---


# Question
Title: Getting permission denied with docker pipeline on jenkins
I have jenkins deployed via helm charts on kubernetes cluster Now i want to be able to run helm to deploy helm charts applications on my kubernetes cluster and have the following in my jenkinsfile

```
pipeline {
    agent {
        docker { image 'alpine/helm' }
    }
    stages {
        stage('Deploy') {
            steps {
                sh 'helm ls'
            }
        }
    }
}

```

I also get kubernetes authentication setup on my jenkins so connection to cluster is fine but the error am getting now is the following

```
+ docker inspect -f . alpine/helm

Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/containers/alpine/helm/json: dial unix /var/run/docker.sock: connect: permission denied
[Pipeline] isUnix
[Pipeline] sh
+ docker pull alpine/helm
Using default tag: latest
Warning: failed to get default registry endpoint from daemon (Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Get http://%2Fvar%2Frun%2Fdocker.sock/v1.38/info: dial unix /var/run/docker.sock: connect: permission denied). Using system default: https://index.docker.io/v1/
Got permission denied while trying to connect to the Docker daemon socket at unix:///var/run/docker.sock: Post http://%2Fvar%2Frun%2Fdocker.sock/v1.38/images/create?fromImage=alpine%2Fhelm&tag=latest: dial unix /var/run/docker.sock: connect: permission denied
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

```

What am i missing and how to i get out of this error?

Thanks

UPDATE

below are the additions/changes to default values.yaml file of the `stable/jenkins` helm chart

```
agent:
  enabled: true
  image: "joao29a/jnlp-slave-alpine-docker"
  tag: "latest"
  customJenkinsLabels: []
  # name of the secret to be used for image pulling
  imagePullSecretName:
  componentName: "jenkins-slave"
  #privileged: false
  privileged: true

  resources:
    requests:
      cpu: "512m"
      memory: "512Mi"
    limits:
      cpu: "512m"
      memory: "512Mi"
  # You may want to change this to true while testing a new image
  alwaysPullImage: false
  # Controls how agent pods are retained after the Jenkins build completes
  # Possible values: Always, Never, OnFailure
  podRetention: "Never"
  # You can define the volumes that you want to mount for this container
  # Allowed types are: ConfigMap, EmptyDir, HostPath, Nfs, Pod, Secret
  # Configure the attributes as they appear in the corresponding Java class for that type
  # https://github.com/jenkinsci/kubernetes-plugin/tree/master/src/main/java/org/csanchez/jenkins/plugins/kubernetes/volumes
  # Pod-wide ennvironment, these vars are visible to any container in the agent pod
  envVars: []
  # - name: PATH
  #   value: /usr/local/bin
  volumes:
  # - type: Secret
  #   secretName: mysecret
  #   mountPath: /var/myapp/mysecret
   - type: HostPath
     hostpath: /var/run/docker.sock
     mountPath: /var/run/docker.sock

```

everything else pretty much same except for ingress hostnames and admin user password...above is most important modification to default values.yaml file

# Answer
> 2 votes
You need to add the `jenkins` user to the `docker` group:

```
# run the following command as root
usermod -aG docker jenkins

```

---
Tags: jenkins, kubernetes, jenkinsfile, helm
---


# Question
Title: Ansible ad-hoc command to reboot servers in serial with a pause
How do I go through all hosts in an Ansible inventory in serial with an Ansible ad-hoc and reboot servers with a specified number of seconds in between each reboot?

I am looking for something similar to: `ansible -i inventory nodes -a "/sbin/reboot"`

It should have a pause and run in serial.

# Answer
> 5 votes
@Argyle explained the `--fork 1` trick to achieve a serial run.

I suggest you have a look at the `reboot` module that might be better suited than running a command directly.

A quick example that will reboot all your nodes one by one waiting for each to be fully booted before moving to the next one.

```
ansible -i inventory nodes --forks 1 -m reboot

```

Pass needed parameters to the module depending on your requirements with the `-a` option.

# Answer
> 2 votes
I think this *might* be as easy as adding a sleep with the value of seconds you'd like to wait before rebooting. Also, since Ansible runs in parallel by default, we'll have to specify that we only want it to run as one process (what Ansible calls forks) instead of the default of 5. So, like this:

```
ansible -i inventory nodes -a "sleep 30 && /sbin/reboot" -f 1

```

---
Tags: ansible
---


# Question
Title: Azure DevOps/Pipelines seed job equivalent?
I'm looking at moving to Azure and using Azure Pipelines (I've used Jenkins before now) - I plan to deploy to Azure Kubernetes Service and store my Docker images in an Azure Container Registry. Using Azure DevOps/Pipelines to integrate with Azure's own services would seem to be simplest approach.

I've got a small collection of applications to deploy - about 30 or so. In the Jenkins world I had a Git repository containing my job definitions written in Jenkins Job DSL language and a seed job that would create all 30 Jenkins "pipelines" for me.

Can I do this with Azure DevOps? Is there an equivalent to a seed job process in Azure DevOps?

--

Jenkinsfile equivalent in Azure repos looks to be a similar question (but not identical).

# Answer
> 1 votes
Yes this can be done, Azure Pipelines can ingest a yaml file, configuring the pipeline as code. These yaml files should be source controlled and the pipeline can be set up to build in an automated way.

You can also reuse steps within the pipeline

# Answer
> 1 votes
I have been looking for this as well.

Templating is solved very good, and in case you have a file "azure-pipelines.yml" in the root of your repo, the pipeline will automatically be setup (with the name of the repo).

But if you have a repo with submodules (as in directories, not git submodules) and dedicated azure-pipeline.yml in each submodule, these will not be setup automatically. Maybe also because we are not enabled to define a pipeline name in our yaml definitions?

With a short testing round, I found that using the azure pipeline cli together with python is also no quick fix for really automatically setting up my pipelines for submodules.

---
Tags: jenkins, azure-devops
---


# Question
Title: Are there arguments against using “imagization" tools in the literature and if so, what are the main ones?
Although I never used an "imagization" tool (for example, Docker) I understand a *container* as an instantiated (running or stopped) *image* which in plural are simply put, a server environment and corresponding application/s, cloned from one computerized machine into another computerized machine (likely a virtual machine);
I further understand such cloning operations save time because application/s (and maybe also an OS) shouldn't be installed again and configured again to reach the *state* of the cloned machine, so one could start from the cloned state and make changes from there.

## My problem

When I still used Facebook, I spoke there with some anonymous sysadmin who claimed something like he doesn't use imagization tools as an image could be contaminated with malicious software or hacked somewhere along the process without him knowing it.

I am not sure I enough understand what "imagization" tools are and the subsequent opinion I presented.

## My question

Are there arguments against using “imagization" tools in the literature and if so, what are the main ones?

# Answer
You might have better luck finding answers to your questions if you use the more standard term of "containerization".

This article discusses containerization as well as some pros and cons you might want to consider.
> 1 votes

# Answer
In my opinion, your real problem about this is getting incomplete data that didn't include *image scanning*, *hardening* and *trusted registries*.

Here are some first directions:

1. The well curated Official Docker Images on DockerHub
2. You can setup trusted docker registries and setup docker clients to evaluate these based on content trust.
3. If you are planning to mirror images or publish some yourself based on others, you are able to setup docker image scanning

I only included links to the official docker doc links, but especially with "image vulnerablility scanning" there are a lot of free tools and services available as well.

Various (payed) solutions for the trusted registry are available in the cloud, eg: Azure Container Registry, Google Container registry or the JFrog Container Registry. With the later also available for on premises installation. The vulnerability scanning is as far is I know not included automatically.
> 2 votes

---
Tags: docker, kubernetes, culture, rancher, images
---


# Question
Title: Using Prometheus to monitor Spring Boot Applications in Kubernetes Cluster
I have spring boot powered microservices deployed in my local kubernetes cluster. The microservices are using micrometer and prometheus registry but due to our company policy the actuator is available on another port:

* 8080 for "business" http requests
* 8081/manage for actuator. So, I can access `http://host:8081/manage/prometheus` and see the metrics when running the process locally (without kubernetes).

Now, I'm a beginner in Prometheus and have a rather limited knowledge in kubernetes (I'm coming with a Java developer background).

I've created a POD with my application and succesfully run it in kubernetes. It works and I can access it (for 8080 I've created a service to map the ports) and I can execute "business" level http requests it from the same PC.

But I haven't find any examples of adding a prometheus into the picture. Prometheus is supposed to be deployed in the same kubernetes cluster just as another pod. So I've started with:

```

FROM @docker.registry.address@/prom/prometheus:v2.15.2

COPY entrypoint.sh /
USER root
RUN chmod 755 /entrypoint.sh

ADD ./prometheus.yml  /etc/prometheus/

ENTRYPOINT ["/entrypoint.sh"]

```

*`entrypoint.sh`* looks like:

```
#!/bin/sh
echo "About to run prometheus"
/bin/prometheus --config.file=/etc/prometheus/prometheus.yml \
                --storage.tsdb.path=/prometheus \
                --storage.tsdb.retention.time=3d \
                --web.console.libraries=/etc/prometheus/console_libraries \
                --web.console.templates=/etc/prometheus/consoles

```

My question is about how exactly I should define `prometheus.yml` so that it will get the metrics from my spring boot pod (and other microservices that I have, all spring boot driven with the same actuator setup).

I've started with (`prometheus.yml`):

```
global:
  scrape_interval: 10s
  evaluation_interval: 10s

scrape_configs:

  - job_name: 'prometheus'
    metrics_path: /manage/prometheus
    kubernetes_sd_configs:
      - role: pod
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: keep
        regex: sample-pod-app(.*)|another-pod-app(.*)

```

But apparently it doesn't work, so I've asking for the advices:

* If someone has a working example it would be the best :)
* Intuitively I understand I need to specify the port mapping for my `8081` port but I'm not exactly know how
* Since prometheus is supposed to run on another port, am I supposed to expose a kubernetes service for port 8081 at the kubernetes level?
* Do I need any security related resources in kubernetes to be defined?

As a side note, at this point I don't care about scalability issues, I believe one prometheus server will do the job, but I'll have to add Grafana into the picture.

P.S. Originally asked this question on SO but was suggested to ask this here as well.

# Answer
The best way to deploy prometheus to kubernetes is with the helm chart:

https://github.com/helm/charts/tree/master/stable/prometheus

If you haven't used helm yet, its a very simple install to your laptop/etc and the cluster (and the newest version doesn't even require a cluster side install). It takes like 30 seconds. It's a package manager for kubernetes and it allows you to install, upgrade, list, and delete "charts" which are basically simple wrappers around kubernetes YAML files (with some nice templating). It would even be good to wrap your spring boot apps in a helm chart =).

Once you've got prometheus running well, there is a good tutorial here on setting up actuator's prometheus endpoint in spring-boot and getting it picked up by prometheus by editing its values.yaml.

https://www.callicoder.com/spring-boot-actuator-metrics-monitoring-dashboard-prometheus-grafana/

```
# my global config
global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.
    static_configs:
    - targets: ['127.0.0.1:9090']

  - job_name: 'spring-actuator'
    metrics_path: '/actuator/prometheus'
    scrape_interval: 5s
    static_configs:
    - targets: ['HOST_IP:8080']

```

There's also some info at the end of the tutorial about surfacing all this in Grafana for slick visualization.
> 2 votes

---
Tags: kubernetes, prometheus, java, spring-boot
---


title: Azure Powershell DSC or Ansible

text:
# Question
Title: Azure Powershell DSC or Ansible
I wanted to find out what are you using for Desired State Configuration on your end. I am thinking to start introducing Azure Powershell DSC or Ansible for DSC of our on-prem environment and cloud environment. Would you have any recommendations? I am turning more towards Ansible but it would be nice to get to know other people opinion.

# Answer
> 4 votes
**PowerShell Desired State Configuration:**

*Pro's*

* Great support for Microsoft Technologies (IIS, Exchange, SharePoint, etc.)

*Con's*

* OS and PowerShell Version requirements can become prohibitive.
* WinRM and the LCM can be flakey requiring fiddling around restarting services to get it working.
* Opensource Ecosystem isn't as strong.

**Ansible:**

*Pro's*

* Huge Open-source ecosystem around Ansible, with many ways of doing one task.
* Support for a wide range of technologies, enabling you to use Ansible YAML for vSphere, Azure, AWS, etc.
* Easy to hire people with Ansible experience.

*Con's*

* Poor support for the nitty-gritty of Microsoft Products, however, you can use the win\_dsc module to call DSC from Ansible.
* Ansible is becoming less important with the advent of Cloud, Kubernetes and Terraform.

Personally, I use DSC for my workstation and that's it now. Previously I used it to bootstrap MongoDB and Redis clusters but eventually switched to Ansible because it meant throwing away the hard to maintain DSC and using open-source Ansible modules from Galaxy.

---
Tags: ansible, azure, powershell, vmware
---


# Question
Title: What is the best tool for the release management for Kubernetes in a small startup?
In our small startup (6 developers) and we use Google Kubernetes Engine + CircleCi (which deploys docker images to K8S with simple bash scripts). We have ~10 services, but we are growing. We use a simplified git-flow model. W plan to use in the future: terraform for infrastructure, helm charts for managing deployments. We want to have a possibility:

1. (now) to track our releases in every environment
2. (now) manually rollback / automatically rollback after seeing some alerts in grafana. Rollback must apply Django migrations first and then use the image from the previous deployment.
3. (now) release a set of services in one go
4. (future) use some more advanced development techniques like canary deployment
5. (future) have some metrics regarding deployments (eg. build time)

There are plenty of tools which could be used for this job and it's hard to choose

1. Spinnaker -\> it is said that it needs 3 engineers to maintain it, so probably it's too complex.
2. Jenkins -\> it's more CI/CD tool.
3. Harness/Argo/Google Deployment/Team City/Code Fresh -\> maybe one of those?
4. Write a simple inhouse solution in Django -\> the most flexible one, but also the most error-prone.

What do you recommend?

# Answer
> 1 votes
GitLab is a solid good option for most of this (with the appropriate plan level - I don't think the CE one has everything). It's a single-pane-of-glass for your code, security, CI/CD, metrics, etc. You can create and deploy to kubernetes clusters from it.

* You can have multiple environments and it'll store the secrets for you to customize them.
* It will monitor your kubernetes apps and show deployment marks so you can see if they get better/worse performance wise before and after deployment.
* You can do multi project pipelines and actions in parallel to chain things together and get things done fast.
* It supports review apps and canary deployments.
* It has rollback buttons and tracks every release you do. It also has a good API, so I assume you could hook up prometheus + alert manager and make the API trigger a rollback in response to an alert.

Spinnaker has https://www.armory.io/ if you're willing to pay a bit (not sure how much). But that should make it easy to use / effective.

I'm not amazingly fond of Jenkins, but it can do everything. Its hard to maintain over time / with bigger projects and teams though. But it can be customized to do anything. You may want to look into https://jenkins.io/projects/blueocean/ (the blue ocean plugin) for CD if you're leaning this way - it looks fairly impressive and I've heard good things.

# Answer
> 0 votes
I believe that to manage the builds / deploys follows by the same Jenkins you can use Blue Ocean theme it's more friendly and you can track CI/CD. For infrastructure I suggest use Rundeck (Cron like with web gui) + Ansible + ARA Records Ansible to track the plays.

# Answer
> 0 votes
Depending on where your source code is located and how you are setting up the to be deployed containers, this can also be solved very nice in Azure DevOps: https://cloud.google.com/solutions/creating-cicd-pipeline-vsts-kubernetes-engine

---
Tags: kubernetes, release-management, spinnaker
---


title: How to filter vmware_vm_facts

text:
# Question
Title: How to filter vmware_vm_facts
I'm trying to pull out the "uuid" from facts returned by ansible's vmware\_vm\_facts module. There is a solution to a similar question posted on Reddit, but when I try that, it fails with 'ansible.utils.unsafe\_proxy.AnsibleUnsafeText object' has no attribute 'ip\_address'

Here is my code:

```
  - name: Test vmware_vm_facts
    vmware_vm_facts:
      hostname: "{{ vcenter_hostname }}"
      username: "{{ vcenter_username }}"
      password: "{{ vcenter_password }}"
      validate_certs: no
    delegate_to: localhost
    register: rc

  - name: Debug...
    debug:
      msg: "IP of {{ item.key }} is {{ item.value.ip_address }}and power is {{ item.value.power_state }}"
    with_dict: "{{ rc.virtual_machines }}"

```

Anyone have any ideas? Thanks!

# Answer
Thanks for your response @Zeitounator. I'm running Ansible 2.8 Since posting this I have received an answer that works for me. I realized that these facts are presented as a list of dicts as noted in the doco, but I was still having trouble for some reason (confused myself badly.) The solution hinges on using with\_items instead of with\_dict and using an appropriate conditional like so:

```
  - name: Test vmware_vm_facts
    vmware_vm_facts:
      hostname: "{{ vcenter_hostname }}"
      username: "{{ vcenter_username }}"
      password: "{{ vcenter_password }}"
      validate_certs: no
    delegate_to: localhost
    register: rc

  - debug:
      msg: "IP of {{ item.guest_name }} ({{ item.uuid }}) is {{ item.ip_address }} and power is {{ item.power_state }}"
    with_items: "{{ rc.virtual_machines }}"
    when: item.guest_name is search(inventory_hostname)

```
> 1 votes

---
Tags: ansible, vmware
---


# Question
Title: docker-compose run container before other container stop
Can somehow with docker or docker-compose run container B before container A stop?

docker-compose support trigger event or something like that?

Cheers

# Answer
> 1 votes
If you mean make sure that container A does not execute and then exit then no. Docker containers will stop if the container crashes or if the main process exits.

However, if you are attempting to run container B and it is dependent on container A, I think the depends\_on flag may be what you are looking for. This will allow you to ensure that service B will run after service A is started. However, you may need to add something to service A that ensures that it is ready before attempting to connect to it (e.g. a database).

---
Tags: docker, docker-compose
---


# Question
Title: Can one use lando.dev for a production environment?
OpenSource community developed free lando.dev looks great for a local development platform on one's own machine. I'm certainly looking to evaluate it for my own setup.

But, given there are benefits in having a development environment that matches production, if one is using lando.dev in development, then could one use it in production.

Sure, one might want to go on to use Kubernetes in production, but perhaps this is more than what one needs in the early stages of a project and therefore lando.dev is more suitable and more accessible.

# Answer
> 1 votes
From "the horses mouth":

https://twitter.com/DustinLeblanc/status/1220351066178510848

> We don't recommend Lando in production. The container images we use aren't optimized for that and we don't test an open-to-the-internet scenario. It's exclusively meant for local dev

---
Tags: docker, docker-compose
---


title: localhost Pod to Pod communication k8s

text:
# Question
Title: localhost Pod to Pod communication k8s
I am trying to transfer an old legacy system to k8s.

I have a custom server that client apps connect to in order to communicate with one another. Both the server and clients were originally created to run on the same system and communicate using localhost.

I Dockerized both server and client apps. I tested that they work well in the same pod (everything runs well using localhost). Now I would like to separate the server and apps into their own pods. I would like the apps to automatically connect to the server.

Question: How do I connect the client apps to the server without changing the source code of the client apps or server?

# Answer
> 1 votes
Are you basically saying that it’s hard coded to “localhost” and you need to somehow change the meaning of “localhost” because you cannot change the source code?

If so, it sounds fairly hard to achieve... but maybe you can override the meaning of localhost in the hosts file of the pod (the right way).

https://kubernetes.io/docs/concepts/services-networking/add-entries-to-pod-etc-hosts-with-host-aliases/

I feel like this will *probably* have unintended side effects and may affect kubernetes, but I could be wrong. It’s definitely worth a try.

Note: the overrides can point to services created over the pod deployments if it works :). They can have reliable internal domain names that don’t move around.

# Answer
> 1 votes
You can do this by using Kubernetes and Docker's network options. Specifically check out the Kubernetes service documentation for service discovery.

The idea both inside and outside of Kubernetes is that you can add the Docker containers to a defined network, and then discover the services in the network. Depending on how it is configured you may need to run the containers to point to the service name instead of localhost, but again this will vary depending on how you setup your Kubernetes cluster.

---
Tags: docker, kubernetes, networking
---


title: RDS DB migrations using ECS and Fargate

text:
# Question
Title: RDS DB migrations using ECS and Fargate
I run my app with Docker and ECS. How can I handle database migrations? With fargate there doesn’t seem to be a way to SSH into the container to run any commands so I can’t run my RDS MySQL migrations by going into the container. I have some migrations to run at times, not for all deploys. Any advice?

I used to run my docker container in a EC2 box so I had the option to run a command inside the container once the container is up. I have migrated to ECS and Fargate now and I don't know how to handle it now.

# Answer
> 2 votes
If you already have the script/tool in place to execute the database migrations then what you might need is to run the tool/script once. AWS ECS allows to schedule the executions of tasks (containers), you can set it up in a way that the container can run just only once.

These ECS tasks can be scheduled via a cron-like job setup or on a CloudWatch event response:

https://docs.aws.amazon.com/AmazonECS/latest/developerguide/scheduled\_tasks.html

---
Tags: docker, amazon-web-services, amazon-ecs, databases, fargate
---


# Question
Title: Set additional required approver for pull requests at subfolder lever in Azure Devops
I have a pull request policy on my repo which requires at least one member of the `devs` group reviews and approves any change before it is merged in to master. I'd like to be able to specify additionally that any files changes under the `sql_scripts` folder require a member of `dbadmin` group to review and approve.

Can this be done in Azure Devops using either the UI or the API?

# Answer
Yes you can do this under branch policies.

From the GUI navigate to **Repos**-\> then in the hamburger menu **branches**-\> then select your master branch and then ... on choose **Branch Policies**. Like so

Then from the branch policies page scroll down and find the **Add Automatic Reviewer** button and set a **path filter** based on the folder.
> 3 votes

---
Tags: azure-devops
---


title: Publish pipeline artifact to Slack

text:
# Question
Title: Publish pipeline artifact to Slack
I have a pipeline on Azure DevOps that publishes an artifact (`releasenotes.txt`).

It also has integration with Slack using ( https://docs.microsoft.com/en-us/azure/devops/pipelines/integrations/slack?view=azure-devops ) .

How can I send `releasenotes.txt` to Slack? Is it possible?

# Answer
The stock *Azure Pipelines app for Slack* does not support that workflow of uploading artifacts to Slack. You could use this marketplace task and upload your release notes artifact with it.

Bonus points suggestion, if you are going down the release notes path I would take a look at using Richard Fennell's Generate Release Notes task in combination with the Wiki Updater task.

You can create a generalized Task Group or yaml template that you could pretty much re-use anywhere. For example in the Wiki updater task you can create a release name wiki page formatted like: `$(Build.Repository.Name)\$(Release.EnvironmentName)\$(Release.ReleaseName).md`

And then in your Slack message send a generalized link to your release notes. `Release Notes Wiki link <https://collection-name.visualstudio.com/projectname/_wiki/wikis/Release-Notes?wikiVersion=GBmaster&pagePath=$(Build.Repository.Name)\$(Release.EnvironmentName)\$(Release.ReleaseName)|$(Release.EnvironmentName)\$(Release.ReleaseName)>`

You could push them back into the Repo that the release originated from or a central release notes repository. Your release notes get source controlled and are nicely organized!
> 2 votes

---
Tags: azure-devops, build-pipeline, azure-pipelines, artifacts, slack
---


title: Azure devops, fortnightly build trigger

text:
# Question
Title: Azure devops, fortnightly build trigger
I'm trying to write a build pipeline that runs every second friday (essentially at the end of a sprint). Using the classic editor it appears I can only set a trigger that would run every friday. Is there any way I can achieve what I'm after?

Perhaps it runs every week but stops immediately if it can interpret the when the build was last run (although that doesn't seem to be exposed anywhere). I guess another alternative would be a triggering build that calls to the devops API to figure out if its the last day of the timebox. Neither of theses are ideal though for a fairly simple requirement (imo).

Thanks

# Answer
I don't think you can alter the trigger right now in Azure DevOps. What I do is to create an extra task to start with, and use powershell (or your language of choice) to check if the week number is even or odd and then proceed with the build or not.
> 0 votes

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: Ansible: Backing up variable value doesn't work as expected
Attempting to save a variable's value for backup before modifying it doesn't work.

I have the following inventory file (note the custom ansible\_port and user):

```
all:
    hosts:
            test:
                    ansible_port: 11111
                    ansible_user: user_foo
                    ansible_host: 192.168.56.101

```

And I have the following playbook:

```
- name: Test
  hosts: test
  vars:
          backup_ansible_port: "{{ ansible_port }}"
          backup_ansible_user: "{{ ansible_user }}"
          ansible_port: "{{ tmp_port }}"
          ansible_user: root
  tasks:
          - name: foo
            debug:
                    msg:
                    - "{{ backup_ansible_port }}"
                    - "{{ backup_ansible_user }}"
                    - "{{ ansible_port }}"
                    - "{{ ansible_user }}"

```

Running the following command:

> ansible-playbook bootstrap\_ssh.yml --ask-pass -e 'target=test tmp\_port=22'

The output is the following:

```
...
TASK [foo] *********************************
  ok: [test] => {
      "msg": [
          "22",
          "root",
          "22",
          "root"
      ]
  }
...

```

> Expected output:

```
...
TASK [foo] *********************************
  ok: [test] => {
      "msg": [
          "11111",
          "user_foo",
          "22",
          "root"
      ]
  }
...

```

Why is this not the case?

# Answer
It is a totally expected result. You define the same variables in different places and ansible applies its variable precedence

First of all, remember that variables passed as extra vars always win: you can never change them in any other place or during the run (i.e. `set_fact`).

Now let's decompose what happens. The schema is basically the same for all vars where you don't get the expected result. I'll only give the example for `backup_ansible_port`. You can follow the spaghetti yourself for the others once you got it.

When you hit the debug task to display `"{{ backup_ansible_port }}"`

1. The first jinja2 template pass finds it should be an expansion of `ansible_port` as defined in your play vars.
2. `ansible_port` is defined with value "11111" directly in your inventory as a host var (see #8 in above precedence doc) **But** it is also defined as an expansion of `tmp_port` in your play vars (see #12 in above precedence doc). Higher precedence wins, so the second jinja2 template pass finds it should expand `tmp_port`
3. `tmp_port` has value "22" in your extra vars which will be the final value returned.
> 2 votes

---
Tags: ansible, ansible-inventory
---


# Question
Title: How to arrange builds for staging and production with Azure DevOps?
On Azure DevOps I need to build and deploy a ClickOnce application. Because this application needs to be signed (including config) I must create a build for staging and a build for production.

I decided to create an artifact for my App with 2 tasks:

At the moment I'm using the XML transformation to apply my specific staging or production settings. It is working fine but I would like to use the Variables substitution system build in Azure DevOps. I already use it in Deployment part for all my IIS project deployment. But in Deployment I have "stage" and I can easily tell the system to use my staging variable in staging stage and production one for my production stage. I cannot do that in Build Pipeline section.

What can I do? Should I create 2 separate builds with 2 separate artifact?

# Answer
I think I see what you are after. Build variables created in the Variables tab of the GUI designer are scoped to the entire definition. But you can set a Job Scoped variable from a script within a GUI based pipeline.

So within the context of each of the Agent Jobs you have defined above, you would add a powershell task that would allow you to set the variable(s) that you then use for the automagic variable substitution in your xml.

### YAML is better

If you want to go the YAML definition route you can actually create Job scoped variables directly in yaml. This example below substitutes against a JSON file but the concepts are similar for xml.

This is the Json file it substitutes against `test.json`.

```
{
    "menu": {
        "id": "file",
        "value": "File"
    }
}

```

```
trigger:
- master

pool:
  vmImage: 'ubuntu-latest'

jobs:
- job: Production
  variables:
    - name: menu.id # Substituting this value in test.json
      value: 'productionValue'
  steps:
    - task: FileTransform@1
      displayName: 'File Transform:'
      inputs:
        folderPath: '$(System.DefaultWorkingDirectory)'
        fileType: json
        targetFiles: test.json
    - task: PowerShell@2
      inputs:
          targetType: 'inline'
          script: |
            Get-Content -Path test.json

- job: Staging
  variables:
    - name: menu.id  # Substituting this value in test.json
      value: 'stagingValue'
  steps:
    - task: FileTransform@1
      displayName: 'File Transform:'
      inputs:
        folderPath: '$(System.DefaultWorkingDirectory)'
        fileType: json
        targetFiles: test.json
    - task: PowerShell@2
      inputs:
          targetType: 'inline'
          script: |
            Get-Content -Path test.json

```

### Custom Conditions

Another thing you might want to dig into is that Azure DevOps exposes task level custom conditions. You could add both transform tasks for Stage and Production to the pipeline definition and then only conditionally run the step you want. You could potentially collapse this into just one job and only build what you need to.

You need to decide on what that condition is. For example, you could do it based on the source branch (probably better) or just a variable that you can set at queue time.
> 1 votes

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: Running Jenkins in docker with custom uid / gid
I have the following `Dockerfile`:

```
FROM jenkins/jenkins:latest

```

I'm build a Jenkins container as follows:

```
docker build -t my-jenkins --build-arg uid=1003 --build-arg gid=1003 .

```

I then run the image as follows:

```
docker run -it -p 8080:8080 my-jenkins

```

I go to `localhost:8080` and finish installation (without any plugins selected).

I then create a job with a single shell command:

```
id

```

I run the job and I expect the output of the job to be

```
+ id
uid=1003(jenkins) gid=1003(jenkins) groups=1003(jenkins)

```

The output however, is

```
+ id
uid=1000(jenkins) gid=1000(jenkins) groups=1000(jenkins)

```

which seems to indicate that my `--build-arg`s for `uid` and `gid` are **completely ignored**.

Why is this? The Jenkins Dockerfile should accept uid/gid as guild args.

# Answer
As usual I think I came up with the answers shortly after spelling out the problem carefully here. I would be great if someone could confirm that my understanding is correct.

---

## TL;DR: It's not possible to use `--build-arg` for `FROM` images

The `FROM` command basically tells docker what binary image to start from. The `FROM` image is not built recursively. This obviously means that `--build-arg` won't be taken into account in the `FROM` image.

## So why is `uid` and `gid` even `ARG`s in the first place if I can't use them?

It *is* possible to use them in a reasonable straight forward way. `docker build` accepts git repo URLs when building, so you can indeed kick off a Jenkins docker with custom `uid` and `gid` quite easily as follows:

```
docker build -t my-jenkins --build-arg uid=1003 --build-arg gid=1003 https://github.com/jenkinsci/docker.git
docker run -it -p 8080:8080 --rm my-jenkins

```

## But I need to create my own `Dockerfile` :-(

If you really need to custobmize the Jenkins image *and* need a custom uid / gid, the best way forward is to change the uid / gid of the Jenkins user in your Dockerfile. The uid = 1000 and gid = 1000 is already hardwired when your part of the Dockerfile kicks in.
> 1 votes

---
Tags: docker, jenkins
---


title: Cannot install Ansible

text:
# Question
Title: Cannot install Ansible
So I am creating a packer template for docker with the image *php:7.0-apache*. I added a shell provisioner where I try to install ansible so I can provision the image:

```
{
            "type": "shell",
            "inline": [
                "apt-get -y update",
                "apt-get install -y software-properties-common",
                "apt-add-repository ppa:ansible/ansible",
                "apt-get -y update",

                "apt-get install -y sudo",
                "apt-get install -y ansible"
            ]
        },

```

However, I get this error. Why is that?

**EDIT**: After installing the gnupg package, there still seems to be issues that result in the building process to fail:

# Answer
> 1 votes
This is how I managed to test your scenario manually against the `php:7.0-apache` docker image. Note that this image is based on `debian:stretch`. In your question, you are following the install guide for ubuntu which will fail anyway in debian stretch.

I'm using the correct scenario below. See the ansible install documentation for more details. I also kept the install of sudo which was in your initial provisioning script.

```
apt-get update
apt-get install -y gnupg sudo
apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 93C4A3FD7BB9C367
echo deb http://ppa.launchpad.net/ansible/ansible/ubuntu trusty main >> /etc/apt/sources.list
apt-get update
apt-get install -y ansible
apt-get clean

```

After playing this, result:

```
root@18ab6da8ac92:/var/www/html# ansible --version
ansible 2.9.4
  config file = /etc/ansible/ansible.cfg
  configured module search path = [u'/root/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python2.7/dist-packages/ansible
  executable location = /usr/bin/ansible
  python version = 2.7.13 (default, Sep 26 2018, 18:42:22) [GCC 6.3.0 20170516]

```

Now, the above is installing ansible from deb packages that will run in python 2.7 that has been obsoleted. I much rather prefer the following scenario using pip that will install ansible as well but in python3 and does not need to add a supplementary apt repo neither requires to install gnupg

```
apt-get update
apt-get install -y python3-pip
pip3 install --upgrade pip
pip install ansible
apt-get clean

```

And the result

```
root@e6b73a5f1d59:/var/www/html# ansible --version
ansible 2.9.4
  config file = None
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/local/lib/python3.5/dist-packages/ansible
  executable location = /usr/local/bin/ansible
  python version = 3.5.3 (default, Sep 27 2018, 17:25:39) [GCC 6.3.0 20170516]

```

# Answer
> 3 votes
As it says, `No such file or directory: 'gpg'`, so you probably need to install `gnupg` before adding the PPA repository:

```
apt-get install gnupg

```

---
Tags: docker, ansible, packer, provisioning
---


# Question
Title: Should I use release branch to push my changes?
Should I use a release branch to push changes out to production or release it off master branch - the main pipeline where changes are merged by developers of their feature branches?

Not to mention, there is still some amount of manual testing and that happens on the master branch after developer has completed the changes in feature branch and automation testing, code review is completed and changes are merged to master.

What is the recommended practice?

# Answer
> What is the recommended practice?

The book ”Accelerate” by Forsgen, Humble and Kim documents their research that teams that use fewer long-lived branches are more successful.

The question is then what is the best minimum set of branches for a given team. The answer to that, like many things, is ”it depends”. A team that pushes into production a dozen times a day will prefer tag then release from master. A team that releases once a sprint into an integration test environment for a week before it goes live will generally prefer two branches.

> Should I use a release branch to push changes out to production or release it off master branch

My personal experience is that adding branches later if you need them is easy but removing branches is very hard. So given that we want to minimise the number of branches I recommend that small teams who release often always release from a tag applied to master. It's very easy to create a branch from the tag that was last put live if you do need to create a short-live patch release branch. The point is that you want that to be the exception not the rule give the research in the book named above.

I have seen many teams use a two branch approach. To common ones are:

1. Integrate code into a ”develop” branch and deploy to an integration test environment. Then cherry-pick signed-off work down to master where it is tagged to release.
2. Integrate code into ”master” and deploy into a test environment. Then cherry-pick signed-off onto a long-lived ”release” branch where it is tagged to released.

These approaches are actually identical. The reason to take this “two branches” approach is so that features can be released independently and reordered.

> Not to mention, there is still some amount of manual testing and that happens on the master branch after developer has completed the changes in feature branch and automation testing, code review is completed and changes are merged to master.

There isn’t really enough information in your question about how your team actually works to know whether a one or two branch strategy would be a good fit. Your actual release, testing and support ways-of-working dictate the best strategy for your team.
> 1 votes

# Answer
A common practice is to build and test off of a pull request to master. Your developers create feature branches, and submit a pull request to master when it is complete. This then triggers a build job based on the PR.

For example, if you are using Github and Jenkins, you could use the Github pull request plugin to run a Jenkins job when a PR is submitted to master. If it passes, then perform your code review and manual testing.
> 0 votes

# Answer
It depends on what do you want/expect from your release strategy.

Since you're at peace with using feature branches, it means you're also at peace with potentially complicated branch merges and the risks they bring into your master branch: breakages, delays, etc.

If you have strict release deadlines you'll have a hard time meeting them directly from master, pulling release branches off master after the needed feature branches are merged will give you a chance to stabilize the release codebase by isolating it from changes for future releases which I presume continue to be merged into master. The smaller changes with a lower commit rate should significantly lower the risks compared with the master branch.

Release branches also allow you to provide to your customers release hot fixes and complex release stories: multiple simultaneously supported releases, major/minor/incremental releases, etc. It would be very difficult if not impossible to achieve these while releasing directly from master.

The drawbacks of using release branches:

* your costs will increase as each release branch will require maintenance resources, your teams will have to split focus between the master and release branches (which will diverge), context switching may not be insignificant.
* issues uncovered either on the release branch or on the master branch but also affecting the release branch(es) may have to be separately developed and committed on each of the affected branches (since the branches diverge the fixes may not necessarily be identical in all branch contexts).

If the advantages the release branches offer are of no interest or if their disadvantages are show-stoppers for you and you're fine with an "as-is/take it or leave it" basis for your releases then just use selected master branch refpoints for release candidates and, if they meet your release quality criteria, tag them as releases. If after that any issue is being found with a release - just wait for a subsequent release with the fix (but it may contain other stuff as well).

The major disadvantage of this approach is the difficulty of picking your release candidates in full flow of changes going into master (drinking from the fire hose?). The chances of finding good candidates can be pretty low. What if you don't find any? Attempting to throttle the rate of commits going into master in an effort to stabilize it prior to a release will translate into delayed integration (features staying in their dev branch silos longer than absolutely needed), increasingly complicated branch merges, which will eventually cause compounding delays/slips in all your future releases.

Up to you to chose the lesser evil. This is why I'm a fan of continuous integration - at least the complicated merges are out of the picture. The release strategy considerations still apply with CI - the release strategy kicks in after the integration is done - but the 2 evils are "less bad" :) From my experience significantly less. And with a gating CI system in place it's possible to keep master stable regardless of the commit rate.
> 0 votes

---
Tags: release-management, release, branch
---