# Question
Title: How to define dynamic parallel stages in a Jenkinsfile?
In declarative pipelines, Jenkins allows the definition of parallel stages. It further allows scripted pipeline general purpose scripts to create and manipulate the artifacts of the declarative pipeline. For example, it can create `stages` dynamically.

I would like to know how I can *dynamically* create *parallel executing stages*. Please note I am not talking about *steps* or actions, which are solved here:

How to properly achieve dynamic parallel action with a declarative pipeline?

Below is a simple Jenkinsfile that dynamically adds stages using groovy code. These stages are sequential. I would like a Jenkinsfile that adds stages dynamically as below, but uses the `parallel` construct at the `stage` level. As a result, three stages that run parallel should be generated by the program.

```
pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                addStage('offline','open-source','webgoat')
                addStage('offline','open-source','juice-shop')
                addStage('online','internal','our-website')
            }
        }
    }
}

final void addStage(final String type, final String suite,final String application) {
    script {
        stage("Run ${type} application  '\n${application}' in test suite '${suite}'") {

            runTestOnApplication(type,suite,application)

        }
    }
}

final void runTestOnApplication(final String type, final String suite,final String application) {
    labelledShell label: "Run test shell script for ${type} application '${application}' in test suite '${suite}'", script: 'test.bash'
}

```

# Answer
This is quite complicated to achieve in Jenkins. We had a similar issue and here's how we solved it:

1. We have a shared library where we keep our scripts ins `./vars/someScriptName.groovy`
2. We created a `groovy` script that generates the stages.

Here's a working pipeline code for you to try it out:

```
def generateITParallelStages(body)
{
    def config = [:]
    config.stages = []

    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = config
    body()

    def buildStages = [:]
    config.stages.each { String stage, closure ->
        // default settings, if you want to have any
        def settings = [:]
        settings.type = 'online'; // if you don't define the type, it will be 'online' by default
        settings.suite = 'internal'; // if you don't define the suite it will be 'internal' by default

        closure.resolveStrategy = Closure.DELEGATE_FIRST
        closure.delegate = settings
        closure()

        buildStages.put(stage, prepareStage(stage, settings))
    }

    return buildStages
}

def prepareStage(String stage, Map settings)
{
    return {
        print "Run ${settings.type} application  '\\n${stage}' in test suite '${settings.suite}'"
    }
}

pipeline {
    agent any
    stages {
        stage('Add regression tests') {
            steps {
                script {
                    parallel generateITParallelStages {
                        stages = [
                                webgoat: {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'juice-shop': {
                                    type = 'offline'
                                    suite = 'open-source'
                                },
                                'our-website': {
                                    type = 'online'
                                    suite = 'internal'
                                }
                        ]
                    }
                }
            }
        }
    }
}

```

And here's the result:

This is the only way we have been able to dynamically generate the parallel stages. Everything else will fail (at least at the time of writing this).

Other people seem to be having a hard time figuring this out as well - JENKINS-53032. If you feel the same make some noise :)
> 3 votes

# Answer
I hope this helps. We had a similar issue and solved it using the below method.

Our project contains about 12 or so different components (programs) that we wanted to build in parallel. Because some take longer than others, we decided to break them up into 4 individual "tracks" that would run in parallel. To begin, assume we have an array where each element is a new component object. The object has some properties, like name for the name of the component and track for which track, 1, 2, 3, or 4, that the component is assigned to. I'm not focusing on this part as it's probably not relevant to what you're trying to do.

First we define our `createParallelStage` function which iterates over an ArrayList of components, setting them up to build.

```
def createParallelStage(ArrayList components) {
    return{
        // iterate over list of components objects
        components.each { component ->
            dir(buildFolder) {
                // build the component
                sh "./build.sh ${component}"

            }
        }
    }
}

```

We'll use a list of component objects called BuildComponents, iterate over them, and add them to `parallelGroups[i]` where `i` is the value of the objects `track` property

```
BuildComponents.each { component ->
    parallelGroups[component.track].add(component.name)
    echo "${component.name} is going to parallel track ${component.track}"
    // example: IPCService is going to parallel track 3
    }
}

```

We're now left with iterating over `parallelGroups`, passing each element to `createParallelStage()` while adding it to the `parallelStages` ArrayList.

```
parallelGroups.each{i, component ->
    parallelStages.put(i, createParallelStage(component))
}

```

We then set `failFast` to true, so if any of our components fail, the whole build job will fail, and then pass our list of `parallelStages` to the `parallel()` function.

```
parallelStages.failFast = true
parallel(parallelStages)

```

I hope this was specific to the question you were asking. I wish I could be a little more detailed, but then it might be too specific of a solution to work for you.
> 1 votes

---
Tags: jenkins-pipeline, jenkinsfile
---


# Question
Title: How to deploy a desktop application using Azure DevOps?
I'm using Azure DevOps and I need a way to release an internal desktop application. Right now I have a pipeline ready and generating artifacts, but I don't see anywhere in the documentation how can I provide a release link to my users.

The documentation for how to build a desktop client in Azure DevOps is right here:

Build your .NET desktop app for Windows

However, at the end there is no information on how to proceed - in fact, the "Next steps" section provides links on how to deploy Web Apps or VMs, which are hardly relevant for desktop applications (there is also this issue which states this, but was closed with a "thank you, we've added this to our backlog")

# Answer
You have multiple storage solutions in Azure (i.e. blob, file system). All of these integrate with Pipelines and are good target locations for your build artifacts. Now that you have the build you just need to figure out how people are going to use/install what was built.

There are several possible answers depending on what you are trying to accomplish:

* Even though this is an internal application, do not discount the use of a VM. You can deploy to the VM (following the linked guides they actually have), and from there give your users remote access to it. So basically if someone needs to use the app they can remote in.
* Is your build artifact an installer? If so have your pipeline store it in a place where people can access it (e.g. Azure blob). They can then install it themselves.
* You could copy the build artifact to target machines as build step.
* You could modify the app to have an update functionality that checks a server for a new version. When your build pipeline creates the new files, increment the version number. When the desktop app "polls" for a new version it can download it from the server.
> 3 votes

# Answer
So you want to deploy a desktop application to a Windows machine? Looking through the rest of the documentation, the following links may be helpful:

1. Windows Machine File Copy task
2. Build, test, and deploy .NET Core Apps

I can't tell for sure, but you might just be looking for a way for users to download the build artifacts. If that's the case, this link on Build Artifacts may help.
> 2 votes

---
Tags: continuous-integration, azure-devops
---


# Question
Title: How to grant all resources in an AWS account the ability to assume role?
I have an organization with a master account (account A) and a second account (account B). I want to create a role so all instances in account B will have read permission from everything in account A. For example, EKS in account B will be able to pull from ECR in account A. Or a resource in account B will be able to read data from an RDS or S3 in account A.

How can I create a role and apply it to all current and future resource in account B so they will have complete read assess to resources in account A?

# Answer
> 1 votes
There are several ways to enable access, and your question touches on more than one.

First method, is giving EC2 instances access to do API calls with the AWS API in another account. For this, the instances would need the appropriate IAM credentials. You can assign an IAM Role to an instance. The relationship is then "IAM Policy" -\> "IAM Role" -\> "IAM Instance Profile" -\> "EC2 Instance". This Policy can give permission for this instance to do the `sts:AssumeRole` API call in the account of this IAM/EC2, and assume a role in the second account. https://docs.aws.amazon.com/IAM/latest/UserGuide/id\_roles\_use\_switch-role-ec2\_instance-profiles.html

I have not tested this, but as far as I know, while the above description is correct - it will not work with the `sts:AssumeRole` API since it requires non-temporary (non-session) credentials. And the credentials provided to the EC2 instance are in fact `sts:GetSessionToken` type of credentials provided to it via the EC2 metadata service.

The same applies to worker nodes, and pods, running in EKS. There is a way to enable this for EKS that is described in details in the documentation. https://docs.aws.amazon.com/eks/latest/userguide/security-iam.html

On the other hand, if all you need is to access RDS and talk to your MySQL/PostgreSQL database - you don't need any IAM credentials. Just a clear path through the network where security groups allow these instances to access the RDS database. In EC2 and RDS, security groups allow to include foreign-account security groups as a filter for "source IP addresses". So when you choose to do something like that, you don't really need to specify the IP address of your EC2 instance where the application is executed, but rather just specify which security group is being used by that instance, with the addition of the account number. https://docs.aws.amazon.com/vpc/latest/peering/vpc-peering-security-groups.html

The S3 service does require IAM credentials to write/read data from it, but in this case it also allows to specify a permissions policy on the S3 bucket itself. So while the above method of `sts:AssumeRole` credentials might not work, you can create S3 buckets that do allow access from a peer account. https://aws.amazon.com/premiumsupport/knowledge-center/cross-account-access-s3/

Very similar to this, the ECR service also has an option to allow the `GetLogin` API call from another account. https://aws.amazon.com/premiumsupport/knowledge-center/secondary-account-access-ecr/

And naturally the simplest way, is to provide credentials directly to EC2 instances. This can be done either by writing them into a file. For example the `aws` CLI is using the file `~/.aws/credentials`, and it allows to specify multiple named profiles so you can specify which account needs to be controlled. https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html

Another alternative, is to create a service that is a "Credentials ATM Machine". One excellent tool for doing this is HashiCorp Vault https://www.hashicorp.com/resources/deep-dive-vault-aws-auth-backend. AWS even has a QuickStart CloudFormation template to deploy Vault in your account. https://aws.amazon.com/quickstart/architecture/vault/

---
Tags: amazon-web-services, role-based-access-control
---


# Question
Title: Can Cypress with cucumber replace php unit and regression tests in CI/Cd pipeline?
Or there a language agnostic unit and regresion testing?

My team built an MVP with manual tests only and we got into the inverted icecream antipattern. Now to transform into production we need to fix the problem. The marketplace is fully going into Node and JS frameworks and convincing devs to invest more into php is hard considering the job marketplace. Right now we got into a problem that in order to fix 1 bug 2 more appears because there is no regression. Is there a way to fix the icecream antipattern using a php agnostic unit testing? or do we really need to get into phpunit / codeception?

# Answer
Yes Cypress is capable to run in CI.

https://docs.cypress.io/guides/guides/continuous-integration.html#Record-tests

Most of the time it is just running `cypress run`. In our case we ran the docker cypress.
> 1 votes

---
Tags: cicd, php, regression, unit-testing, cypress
---


title: AWS permissions for Fargate and SSM

text:
# Question
Title: AWS permissions for Fargate and SSM
I'm trying to create some infrastructure for a service I am building on AWS using AWS Fargate. I'm using SSM as a value store for some of my application configuration, so I need both the regular permissions for Fargate as well as additional permissions for SSM. However, after banging my head against this particular wall for a while, I've come to the conclusion that I just don't understand AWS IAM in general or this problem in particular, so I'm here for help.

The basis of my IAM code comes from this tutorial; the IAM code is actually not in that tutorial but rather in this file in the github repo linked to that tutorial. I presume I need to retain that STS permission for something although I'm not entirely sure what.

I've converted the IAM code from the tutorial into a JSON document because I find JSON easier to work with than the Terraform native thing. Here's what I've come up with. It doesn't work. I would like to know why it doesn't work and how to fix it. Please ELI5 (explain like I'm 5 years old) because I know nothing about this.

```
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssm:GetParameters",
        "secretsmanager:GetSecretValue",
        "kms:Decrypt",
        "sts:AssumeRole"
      ],
      "Principal": {
        "Service": ["ecs-tasks.amazonaws.com"]
      }
    }
  ]
}

```

# Answer
IAM Role for Fargate has **two policies**:

1. The first one describes which service can assume the role and its permissions. In this case it will be the `ecs-tasks.amazonaws.com` service (= Fargate) that can call `sts:AssumeRole` to get all the permissions from this Role.
2. When Fargate *assumes* the role it gets the permissions specified within, these are the SSM, KMS and SecretsManager permissions.

If you are using CloudFormation you can do something like this:

```
  FargateRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        # This says that Fargate can assume this role
        Version: '2012-10-17'
        Statement:
        - Action: sts:AssumeRole
          Effect: Allow
          Principal:
            Service: ecs-tasks.amazonaws.com
      Path: /
      Policies:
      - PolicyName: FargatePermissions
        PolicyDocument:
          # This says what permissions will
          # the Fargate container receive
          Statement:
          - Action:
            - ssm:GetParameters
            - secretsmanager:GetSecretValue
            - kms:Decrypt
            Effect: Allow
            Resource: '*'

```

If you are building the role manually in the console be sure to select in the first step that it's an *IAM Role* for *ECS Task*, that will create the equivalent of `AssumeRolePolicyDocument` from the above code snippet.

Then in the next step add the policy *without* the `sts:AssumeRole`:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "ssm:GetParameters",
                "kms:Decrypt"
            ],
            "Resource": "*"
        }
    ]
}

```

That should create the IAM Role you need. Ideally you should also restrict the access only to specific resources instead of using `"Resource": "*"` but for start it's easier to just use the wildcard.

Hope that helps :)
> 1 votes

# Answer
The answer was that my IAM configuration did not include a Task Role, only a Task Execution Role. I needed to add a Task Role which included the permissions I needed to access SSM, and then it worked.
> 0 votes

---
Tags: amazon-web-services, aws-iam, fargate
---


# Question
Title: ansible AWX is connecting through ssh as root instead of user defined in credential
I'm running ansible 2.6.2 and awx 1.0.7.2 on rhel 7.5. AWX is connecting to my ssh proxy as root instead of the defined Credential user.

My playbook contains the following

```
become: yes
become_method: sudo
remote_user: remoteuser
ansible_user: remoteuser

```

ssh.config in my playbook contain the following

```
match host !proxyA,!proxyB,* exec "ssh proxyA nc -w1 %h %p &>/dev/null
  ProxyJump proxyA
match host !proxyA,!proxyB,* exec "ssh proxyB nc -w1 %h %p &>/dev/null
  ProxyJump proxyB

```

My AWX credential user which uses an ssh key and associated with the AWX JOB Template is this (which should be used for ssh connections): remoteuser

I have no defined entries in /etc/ansible/ansible.cfg or any ansible.cfg defined in the current working directory (of the playbook) or in $HOME.

Here is the proxyA/B syslog entries showing the user that AWX is connecting as:

sshd: Failed publickey fro root from 1.2.3.4

And here is the AWX ansible output showing the connection as 'root' instead of the 'remoteuser'

root@proxyA's password: root@proxyA's password: root@proxyA's password:

I can run the playbook through ansible (outside of AWX) without errors.

How do I tell AWX to use the right user name?

# Answer
> 1 votes
According to https://docs.ansible.com/ansible/latest/user\_guide/playbooks\_intro.html

it should be `remote_user:` if it still does not work, run the playbook with `-vvvv` and add the log to the Q&A.

---
Tags: ansible, awx
---


# Question
Title: Change container size in a template from the Marketplace
I am trying to use a template from the Azure Marketplace. Everything works, but it defaults to storage sizes that are by orders of magnitude too large for my usecase.

At the end of the process, I can "kind of" export the template files in text form - at least parameters and variables, but there are no values for the storage here. I assume that those are hard coded inside whatever scripts they use internally to deploy that application. I see no way to have a look at or modify those scripts.

Is there something that can be done in such a case? Should the scripts behind marketplace apps be available somewhere (I just didn't see them)? Or are they usually closed off from the user?

# Answer
According to https://azure.microsoft.com/de-de/blog/container-apps-now-available-in-the-azure-marketplace/ it seems to be possible to define the storage-size when a container-app is chosen.
> -1 votes

---
Tags: azure, azure-marketplace
---


# Question
Title: Continuous integration using bitbucket pipelines failed for OpenEdx
I am trying to set up CI process using bitbucket pipelines for my Openedx site. The script that i am using in my bitbucket-pipelines.yml file is given below. I am trying to just set up the build process on LMS (themes), so that whenever someone make any change in the front end of the site, the build updates paver assets and recompile the assets. The problem is, it is failing on paver update assets giving the error:

> python manage.py lms --settings=devstack\_docker print\_setting STATIC\_ROOT 2\>/dev/null Build failed running pavelib.assets.update\_assets: Subprocess return code: 1

Any solution?

```
image: python:3.5.6
definitions:
  services:
    docker:
      memory: 7168
options:
  size: 2x  # all steps in this repo get 8GB memory

pipelines:
  default:
    - step:
        services:
          - docker
        script:
          # Upgrade Docker Compose to the latest version test
          - python --version
          - export DOCKER_COMPOSE_VERSION=1.13.0
          - curl -L https://github.com/docker/compose/releases/download/${DOCKER_COMPOSE_VERSION}/docker-compose-`uname -s`-`uname -m` > docker-compose
          - chmod +x docker-compose
          - mv docker-compose /usr/local/bin

          - export OPENEDX_RELEASE=hawthorn.master
          - git clone https://github.com/sanjaysample/devstack.git
          - cd devstack
          - git checkout open-release/hawthorn.master
          - make dev.checkout
          - make requirements
          - make dev.clone
          - ls
          - make pull
          - make dev.up
          - sleep 60 # LMS needs like 60 seconds to come up
          - docker cp ../metronic edx.devstack.lms:/edx/app/edxapp/edx-platform/themes
          - docker cp ../pavelib edx.devstack.lms:/edx/app/edxapp/edx-platform
          - wget https://raw.githubusercontent.com/sumbul03/edx-theme/master/lms.env.json
          - docker cp lms.env.json edx.devstack.lms:/edx/app/edxapp/lms.env.json
          - rm lms.env.json
          - docker cp edx.devstack.lms:/edx/app/edxapp/lms.env.json .
          - cat lms.env.json
          - docker ps
          - docker-compose restart lms
          - docker-compose exec -T lms bash -c 'source /edx/app/edxapp/edxapp_env && cd /edx/app/edxapp/edx-platform && git init && git remote add origin https://github.com/edx/edx-platform.git && git fetch origin open-release/hawthorn.master && git checkout -f open-release/hawthorn.master && paver install_prereqs && paver update_assets lms --settings=devstack_docker --debug || true'
```

# Answer
> 1 votes
The author of the Q&A indicated that the issue has been solved since absolute paths were used.

---
Tags: docker, continuous-integration, docker-compose, bitbucket-pipelines
---


# Question
Title: How do I enable Progress report for Test Plans
Sprint 156 update rolled out Progress report in preview (should be an option between Test plans\* and Parameters) but I can't seem to find where to enable it. I have turned on all my previews and still don't see it.

# Answer
According https://docs.microsoft.com/en-us/azure/devops/test/track-test-status?view=azure-devops it should be available by clicking on `Progress report` under `Test Plans`.
> 0 votes

---
Tags: azure-devops
---


# Question
Title: Exit a Makefile target called by parent without throwing an error
Let's say that I have 2 targets:

```
test:
    # ...
    @$(MAKE) dosomething
    @echo test
    # ...

dosomething:
    # ...
    @if [ -z "$(SOMETHING)" ]; then exit 0; fi
    @echo dosomething
    # ...

```

I would like `echo test` to execute even if `$(SOMETHING)` is empty and `echo dosomething` only if `$(SOMETHING)` is not empty (which doesn't work with the example, it's still executed).

`exit 0` of course doesn't work here since `make` ignores non-negative exit in subshells used to execute instructions.

Is there any other way to skip the rest of the target without breaking further execution of `test` target? I could make the rest a single long instruction of else but wondering if there's a different command I'm maybe missing.

# Answer
> 1 votes
**Makefile**

```
all: test dosomething

test:
    @echo "hello"

dosomething:
    @if [ "a" = "a" ]; then\
        echo "world";\
        exit 0;\
    fi

```

Running: `make` will return:

```
hello
world

```

---
Tags: bash
---


# Question
Title: How to create databases, users and grants with terraform mysql and output the created credentials?
For a new deployment I want to create databases, users and grants on a previous created MySQL database on azure.

The following code used inside a module creates the environment as expected and I still struggle how to create the outputs of the module and so I get something like the following, to give it to the operations team:

```
module.test_cluster.mysql_users[cat].password = randompassword1
module.test_cluster.mysql_users[dog].password = randompassword2

```

Code example:

```
mysql_databases = [
  "foo",
  "bar"
]

mysql_users = [
  "stage-cat",
  "stage-dog",
  "stage-snake"
]

mysql_grants = {
  "cat-0" = {
    name     = "stage-cat"
    database = "foo"
    grant    = ["ALL"]
  },
  "cat-1" = {
    name     = "stage-cat"
    database = "bar"
    grant    = ["SELECT", "EXECUTE", "SHOW VIEW"]
  },
  "dog-0" = {
    name     = "stage-dog"
    database = "bar"
    grant    = ["ALL"]
  }
}

```

```
resource "mysql_database" "test" {
  for_each = var.mysql_databases
  name     = each.key
}

resource "mysql_user" "test" {
  for_each           = var.mysql_users
  user               = each.value
  host               = "%"
  tls_option         = "SSL"
  plaintext_password = random_password.test[each.value].result
}

resource "mysql_grant" "test" {
  for_each   = var.mysql_grants
  user       = mysql_user.test[each.value["name"]].user
  host       = "%"
  database   = each.value["database"]
  privileges = each.value["grant"]
}

resource "random_password" "test" {
  for_each = var.mysql_users
  length   = 32
  special  = false
}

```

# Answer
This is my solution how to output usernames and passwords:

In the module:

```
output "mysql_credentials" {
  value = [
    for user in var.mysql_users :
    { "username" = "${user}@${var.cluster_name}" // Hostname is for Azure MySQL service
      "password" = "${random_password.test[user]["result"]}"
    }
  ]
}

```

In the terraform file referencing the module:

```
output "mysql_service_credentials" {
  value = module.cluster.mysql_credentials
}

```

The output:

```
mysql_credentials = [
  {
    "Password" = "OecheeP1Iequ6zeis2aipai5eiyooL8g"
    "Username" = "stage-cat@stage"
  },
  {
    "Password" = "xaed1EiGh4ahbea4ohjeetouw0Geph8o"
    "Username" = "stage-dog@stage"
  },
  {
    "Password" = "Seu8ieciewoh4Ohl6Ut7em1aiVie1Ao0"
    "Username" = "stage-snake@stage"
  },
]

```
> 1 votes

# Answer
To ensure separation and use in other environments, make sure to have an outputs file and set each password as an output. TF Output Docs. I haven't done much with for each as it was just released last time I used terraform. I would imagine a way to iterate over passwords and assign them as outputs.

once you have the output set for the module, I would then have my consuming terraform put those values in a password vault of some kind. You can use Azure's Key Vault. TF Azure Key Vault, or any other solution. Just ensure that the ops team has access to the destination. This way there isn't anything introduced between when/where the keys are generated and them being stored, reducing the chance for human error.
> 2 votes

---
Tags: terraform
---


# Question
Title: Jenkins Pipeline - Clean before checkout - git clean -fdx "failed to remove: Directory not empty"
I have my Jenkins build configured to "Clean before Checkout" for its git source:

```
Job Config > Source > git > Behaviours > Clean before checkout

```

This results in

```
Resetting working tree

git reset --hard # timeout=10
git clean -fdx # timeout=10

```

Before the job begins.

This errors out with

```
stderr: warning: failed to remove web/static/node_modules/.../fonts/google-fonts: Directory not empty

```

If I re-run the job (simply replay), I get the same error.

If I re-run it again, git clean works fine and the job finishes with success in the end.

If I start it again it errors out 2 times again until it works at the third time. This is always the case.

What might be going on here?

# Answer
We didn't find the cause, neither a real solution.

There is a workaround though:

Put a pipeline stage with `rm -rf *` before your `scm checkout` stage. This solves the problem but I will change the accepted answer to whoever posts the real reason and fix for the problem.

**Sidenote:**

Keep in mind that you will erase anything in your Jenkins workspace doing that. So be certain that you have everything you need in Git so it comes back with the checkout or get it back some other way.

If you rely on files in your Jenkins workspace from previous runs then this method is not for you!

(Although you might want to think about adapting your Job so files from previous runs are not needed anymore. Every run should contain the files it needs by itself and not rely on other runs.)
> 0 votes

# Answer
I think this *might* just be a permissions issue. You can test this by manually running `git clean -fdx` as the user running Jenkins. If it doesn't work, try it again with `sudo git clean -fdx`. If it works, then it's probably an issue with permissions or ownership on one or more files/directories within the repo.

As an alternative, you can install the Workspace Cleanup plugin, which provides the `cleanWs()` function which is used to clean the workspace directory. There are different options, so consult the documentation to see what might work best for your environment.
> 0 votes

---
Tags: jenkins, jenkins-pipeline, git
---


# Question
Title: Are there any tools to monitor custom console applications on Windows Server?
Are there any DevOps tools or known practices to monitor custom made console application on an on premises (Windows Server) environment?

We developed an exe that needs to be continuously running, and if it fails we would like to know, so we can re-start it. Maybe there are tools to re-start them automatically?

# Answer
This is probably a question better suited for ServerFault or StackOverflow.

One approach that may work is to Create a Windows Service from an Executable, then set the service to restart on failure/crash.
> 1 votes

---
Tags: monitoring
---


# Question
Title: Run docker applications within a kubernetes cluster
As part of my architecture, I want to create a deployment/service on my kubernetes cluster which reads message queues and open dockerized subscribers and then subsequently relay message to the original subscriber (if its an RPC pattern).

I was wondering what would be the best workflow to ensure minimal latency (esp since some workflows are RPC).

1. I can create kubernetes JOBs but that would mean i read logs for the output of pods which would be relayed and then subsequently delete the job . But I worry this might be an overkill
2. I can volume share the docker socket and my service can then run my images using docker client. But I am not if thats the right way to do so given that i would like to have proper tracking/versioning of my docker images esp since nodes on clusters are disposable
3. Are there existing frameworks to handle my situation.

# Answer
After reading your Q&A I think that you are looking for a message broker like kafka, rabbitmq or Amazon MQ. Please let me know whether this helps.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: How should I run a multicontainer app/platform with nginx behind traefik?
I have Traefik reverse proxy running to allow a couple of docker containerized apps to run on the same domain just to learn about running multiple containers following https://www.smarthomebeginner.com/traefik-reverse-proxy-tutorial-for-docker. Everything works well. I want to install Device Hive to test it as an IOT platform, but it has multiple containers and it's own nginx proxy and docker-compose file https://github.com/devicehive/devicehive-docker/blob/master/rdbms-image/docker-compose.yml.

Is there a way I can add the Device Hive platform to my existing Traefik build or is it more trouble than its worth. I wondering, more or less while making sure I'm not having a port conflict, if there's an easy way to have Traefik point a subdomain(subdomains are used for the other apps) to the 80/443 ports of the nginx proxy, and just have everything work. My other option is to create another VM for the IOT platform.

I may be searching with the wrong terms but I haven't really found an answer. Any help?

Note: I want to use Device Hive on my existing server which is using Traefik. When I look at the docker compose file for Device Hive I don't see how I could change it to leave Nginx out.

# Answer
> 1 votes
Although I agree with @BruceBecker's comment, i.e. running two different rproxies, it is possible to run traefik in front of nginx. At the previous job we configured several servers that were running on nginx behind traefik.

Ensure that each microservice has the correct labels. Note that Traefik 2.0 changed significantly.

---
Tags: docker-compose, traefik
---


# Question
Title: Automate custom docker image deployment on AWS
This question is related to my previous one \- but a bit more descriptive. I am trying to build automate the generation and deployment of a docker container. The docker container is created with packer. After the creation of the docker image, I am using packer's post processing modules to upload it to the AWS ECR:

```
{
    "type": "docker-push",
    "ecr_login": true,
    "aws_access_key": "<snip>",
    "aws_secret_key": "<snipe>",
    "login_server": "https://<snip>.dkr.ecr.eu-west-1.amazonaws.com/"
}

```

This works just fine. The docker image got uploaded to my repository in the ECR. Now, the next step is where I am stuck. I want to run start the container on AWS.

I am thinking a little script that uses the aws-cli commands could be suited for that. I do not want to use the web ui - and I am sure this can be done over the command line, especially when someone wants to integrate this into a pipeline of some sort.

However, I have a few questions, as I am not sure if I understand the big picture correctly. My docker image is available over the ECR. To run it, I need some base instance, right? According to a reply in the previous thread, the are various way of running docker containers, either by using the ECS or by spinning up a EC2 instance. Since this is related to some research project where docker and virtualization is analyzed I am questioning what would be the best option for my case. Probably EC2? It's important that I can automate it so that no user interaction is required.

I read the docs and checked several tutorials but I still am not entirely sure how to everything correctly in the right order. Especially part where the environment is set up and the image is started.

I know that I can list my images with:

`aws ecr list-images --repository-name prototypes`.

How would I fire up a new instance and run my image on it?

If anything is unclear, please let me know so that I can update my question to be as detailed as possible.

# Answer
> 3 votes
I will give it a try and explain all required steps with examples.

Please add comments with questions and improvement suggestions if anything is not well enough explained.

The OP (original poster) refers to `awscli`; while I provide corresponding examples, I also discuss limitations of this approach and give examples of doing same with Python.

# EC2 deployment

The purpose of this answer is to demonstrate that as OP correctly assumed, no Web UI access neither human interaction is mandatory to deploy a container on an AWS EC2 instance.

**Scope and limitations**

In this use case, we want to start just one container manually for demo/research purposes, therefore we explicitly decide to do no .

For a complex and possibly more realistic deployment scenario, this example is therefore not reusable. For same simplicity reasons, we will use EC2 service and create there a single virtual machine.

According to a short documentation research, there seems to be no official AMI with pre-installed Docker daemon. Thus we need to deploy some default EC2 instance and install there Docker daemon, then start the daemon.

While using a private registry, a login there is required; for simplicity purposes, I give also an example to run a container from the public Docker Hub registry.

Also, I am not going to provide a copy-paste automation script but single steps which can be easily adopted towards individual needs. In any case, intermediate results require parsing and passing operations which is then a pure programming activity concern.

A boilerplate automation could be done either with bash using awscli or with Python, using the boto3 SDK for AWS.

Python is worth to consider because you can do better processing of JSON responses you get from AWS. You can also tell Python to execute SSH login and bash commands on the remote host.

**Execution plan**

* A. Identify recent Linux AMI (Amazon Machine Image) ID
* B. Create keypair and instance (further, also referenced as VM)
* C. Connect to the instance
* D. Install and configure Docker daemon
* E. Run container

**A. Identify recent Linux AMI image ID**

Search for a recent and up-to-date Amazon Linux image following the official documentation:

```
$ aws ec2 describe-images --owners amazon --filters 'Name=name,Values=amzn2-ami-hvm-2.0.????????.?-x86_64-gp2' 'Name=state,Values=available' --query 'reverse(sort_by(Images, &CreationDate))[:1].ImageId' --output text
ami-01f14919ba412de34

```

**B. Create keypair and VM**

With `awscli`, you would just say:

```
$ aws ec2 create-key-pair --key-name ec2-docker-test

```

However, the API sends you your public key wrapped in a JSON data message. Therefore, using Python it is way easier to process this data message to a *.pem* file:

```
import boto3
ec2 = boto3.resource('ec2')

# create a file to store the key locally
outfile = open('ec2-keypair.pem','w')

# call the boto ec2 function to create a key pair
key_pair = ec2.create_key_pair(KeyName='ec2-docker-test')

# capture the key and store it in a file
KeyPairOut = str(key_pair.key_material)
print(KeyPairOut)
outfile.write(KeyPairOut)

```

With the AMI ID we have got previously we can now create the VM:

```
instances = self.ec2r.create_instances(
                 ImageId=AMI_ID,
                 MinCount=1,
                 MaxCount=1,
                 InstanceType=instance_type,
                 KeyName='ec2-docker-test'
             )

```

AWS API allows also lookup of instance state and IP. With little bit more Python/*boto3* stanzas you can acquire this data to know when you can proceed.

**Note: Firewall consideration**

To configure firewall access, please create a security group and link it to machines's VPC. For readability reasons, example is not included.

**C. Test SSH connection**

As soon as the instance is up and running, you can do everything you like by means of *remote command execution*.

In a Bash script, you would use `ssh` only or `scp` to upload a local bash script and run it remotely.

```
 ssh -t <REMOTE_IP> -i <KEY_FILE> <COMMAND>

```

With Python, you use further modules like paramiko to establish SSH connection and run commands remotely.

```
client = paramiko.SSHClient()
client.connect(ip, username=username, key_filename=key_filename, port=port,
                           timeout=timeout,
                           auth_timeout=timeout)

stdin, stdout, stderr = ssh.exec_command("whoami")

```

For readability purposes, I list further commands unwrapped. In your script, you might want to use elementary helper subroutines to have calls like `ssh_call(command)`.

**D. Install and configure Docker daemon**

As said above, anything else from here is running remote commands.

That is, the concluding steps are, following an example in the official documentation; proceeding is same as you would do it locally, but you need to wrap it in your script and pass over intermediate results if required:

* install and configure Docker daemon
* login to your Docker registry if needed
* pull image
* run Docker container

```
    sudo yum update -y
    sudo amazon-linux-extras install docker
    sudo service docker start
    sudo usermod -a -G docker ec2-user
    # relogin or continue with sudo, which you shouldn't
    aws ecr get-login --no-include-email --region region

```

**E. Run container**

Pull custom image and run container from a private registry in AWS.

```
    docker pull aws_account_id.REGISTRY.ecr.REGION.amazonaws.com/REPOSITORY/IMAGE:TAG
    docker run --name MYSERVICENAME -d -p PORT_HOST:PORT_CONTAINER IMAGE:TAG

```

Pull an official public image and run another container listening on port 8001 in a single step:

```
    docker run --name nginx -d -p 8001:80 nginx:latest

```

---

# Alternative scenario with ECS

As described in these Python/boto3 examples here and here, it's possible to define an machine cluster with the AWS ECS service and run containers there.

Please note that ECS cluster offers you an abstraction either from AWS EC2 or AWS Fargate to deliver the actual virtual machines.

If we go for EC2, again we need to pick up a recent machine image, but this we need a machine optimized for ECS. In a nutshell, these machines will have an ECS agent; through its configuration, we assign a machine to our target cluster.

A small bonus is also that ECS machines have Docker daemon already preinstalled.

So, we create an ECS cluster and add a virtual machine to it.

```
ecs_client.create_cluster(clusterName=cluster_name)

ec2_client.run_instances(
        ImageId=AMI_ID,
        MinCount=1,
        MaxCount=1,
        InstanceType="t2.micro",
        IamInstanceProfile={
            "Name": "ecsInstanceRole"
        },
        UserData="#!/bin/bash \n echo ECS_CLUSTER=" + cluster_name + " >> /etc/ecs/ecs.config"
    )

```

In the cluster, we can create a task definition of the service which are both abstract of the container concept.

```
ecs_client.register_task_definition(
        containerDefinitions=[
        {
          "name": "<MY_SERVICE>",
          "image": "<MY_IMAGE>",
          "portMappings": [
            {
              "containerPort": 80,
              "hostPort": 80
            }
          ]
        }
        ],
        family="hello_world"
    )

```

Now, possibly you ask yourself, how would the Docker daemon on the instance know about your private registry to pull the specified image? For that, the machine's ECS agent needs additional configuration.

Now, you can launch the service, which will use the previously defined task to tell the previously authorized Docker daemon to pull the previously built and published image from ECR and run the container on the machine you've included in the ECS cluster ~~in the data center house that Jeff built.~~

```
ecs_client.create_service(
        cluster=cluster_name,
        serviceName=service_name,
        taskDefinition=task_name,
        desiredCount=1,
        clientToken='request_identifier_string',
        deploymentConfiguration={
            'maximumPercent': 200,
            'minimumHealthyPercent': 50
        }
    )

```

Configuration of security groups in the ECS context has been highlighted on StackOverlow already. In short, these are further calls to the API parts related to security group management.

# Conclusion

Both Bash and Python implementation options have their benefits and drawbacks.

Seen on long term, Python being a general purpose programming language with large community, allows complex string manipulation and abstractions from a cloud API or a system process.

Without knowing the real goal of the deployment, an end-to end of above steps might not be worth the effort in terms that for deploying a set of related containers which is more typical scenario, other approaches could be more pragmatic.

Before stepping into technical implementation, best is to make well elaborated decisions and choices addressing business goals and budget:

* how many times will automation need to run and in which time period vs. time to implement it end-to-end from the beginning?
* assess whether you need only EC2, or ECS with EC2 either Fargate for stateless tasks, or maybe you need Kubernetes?

Further reading: "*The Benefits of Managed Kubernetes vs. Amazon ECS*" (short argument from a cloud consultant blog as of August 2019).

> A 2018 CNCF survey cites **83%** of organizations use Kubernetes as their container orchestration solution vs. **24%** for ECS.

---
Tags: docker, amazon-ec2, deployment, virtualization, aws-ecr
---


# Question
Title: Software Security Testing in a DevSecOps Pipeline
This is a somewhat general question by design, intended for discussion of process. At my current (and all previous) companies we had issues with releasing insecure software into production. This cut across the divide from mobile, web, API's, internal, external, critical and non-critical applications and systems. If we built it, it had/has security issues... just like *your* company. In addition, we lack support stack (qualified application security engineers who "knew code" and not just glorified pen testers who didn't know JavaScript from Java) so we could not scale services as needed. Hence automated software security scanning in the SDLC and now in the DevSecOps CI/CD pipelines. We face the issues of:

1. Service Level Agreements - our dev teams want/need results in minimal time, we all know that's not happening with either DAST or SAST tools without heavy configuration.
2. Allowing teams to choose their own language and not even having static code support... leaving us with DAST if applicable or, ugh, pen tests, which in my view are essentially useless.

We need to shift software security "left" - from point of inception, not wait until code is in production and hope we detect the vulnerabilities - I am sure someone will find them, most likely *not* the pen test team we are paying huge money for, however.

Automation of static and dynamic tools, properly configured, driven off a threat model seem to be our only options to get ahead of the game. We all know "developer education" is a lost cause.

How is your company/team solving this, assuming they even are or recognize it as a problem to address?

# Answer
> 2 votes
I agree that relying on "developer education" is quite futile, at least when it's the sole course of action. Combine this, however, with automation and you can achieve far greater impact. Both sides, DAST and SAST can be covered with:

1. Unit test authentication: Make sure that you have unit tests that check registration, login, password reset and that it is integrated with your development process (continuous integration). Implement processes that automatically block changes to your code when they fail.
2. Testing against known vulnerabilities: These are automated DAST tests that test your application from outside. Programs that handle this task include BDD and mittn which is inspired by GAUNTLT.

Integrate everything *properly* with a CI server (Jenkins, Gitlab, Travis) and stop having to rely on your programmers' "best effort security"

Note: my opinion of proper CI integration, from a language/platform agnostic view, covers the following

**Tests/examples:** \- unit tests that cover at minimum 80%, better 90% \- integration tests that ensure the modules work with one another \- example programs that show how a user should use the API

**Build automation:** \- Automatically build every pushed commit \- Build library, test and example code \- Build for each platform/configuration \- Run the tests

**Merge/pull requests:** \- Block merges into master branch if any thing fails \- Require code review of at least one other colleague

**Code analysis:** \- Run nightly static code analysis (style, anti-conventions, dangerous constructs) \- Run nightly dynamic code analysis (memory leaks, race conditions, known security flaws)

**Publish results:** \- Publish graphs/trends of test/analysis history \- Talk about changes/targets in weekly meetings \- Act on problem areas with sprints/work packages

# Answer
> 1 votes
This is a little bit of a general question as you said, so my answer is a little general too. We have always made security a part of the development process and we enforce policy to make sure our code is secure. This would include forcing developers to only use languages that can work with your security tools. This needs to be part of the review process. Part of this could be in your automated testing process too.

In my experience many DevOps companies view security as an afterthought because of the fear of slowing down development. This runs counter to the reason companies adopt DevOps in the first place. As you said, security must shift left in the development process. I believe this is in line with DevOps / agile principles. Rather than waiting until a product is complete to think about security. Trying to secure something once it is built or already breached will be far more costly in the long run.

# Answer
> 0 votes
While I completely agree with

> security must shift left in the development process

though I can't completely agree with

> We all know "developer education" is a lost cause.

Because I think of the answer to a Quora question I've seen recently, something like "*What is the hardest requirement to a software engineer?*".

The answer there was like "\*a feature which has highest priority unless it requires too much work".

You probably don't want to do same to your security requirements!

Because many software engineers are very eager to experiment with and learn new things if there is an *exactly defined desired outcome*.

By this I mean to say that the following steps should be undertaken:

* **security design**: what are exactly the security requirements by *DevOps lifecycle phases* coming from your customer/business/regulatory bodies?
* **security implementation**: for every requirement, you will either find an applicable tool you could plugin to the CI/CD or you will need to build a custom solution (which is part of work requirement and task to your developers).

A great pitfall is from my experience to ask for "state of the art security" either "what developers consider best to realized" because this produces a lot of opinionated discussion including in some cases questioning tool reliability either metrics applicability as such.

To have a good implementation strategy, some research will be required. I will try what comes to my mind mostly from the open source world. In a large scale business critical industrial environment, investing many thousands to licensing of specialized tools is not much compared to financial and image damage numbers we know about from media.

* **Requirements**: If written requirements exits, do people responsible for that aka *business analysts* iterate with stakeholders on specific security requirements if such exist? Left shift means also customer should be also aware of the information security and privacy protection as business topic. Which assets are to be protected, which risks are acceptable?
* **Static code analysis**: Sonarqube and underlying tools provide metrics on potential vulnerabilities
* **Packaging**: modern artifact repositories provide dependency management tools to highlight problems with package vulnerabilities and licenses (which is though not for free). Also information that some very old package version, either discontinued open source project could be already a source risk. In Java/Maven, you've got the dependency-check-maven based on the US NIST national vulnerability database. For Docker images, you have got security scanners as well.
* **Deployment** You can grade TLS implementation with the open source Qualys SSL Labs' scanner, there are penetration automation tools from OWASP.

*Side note*. An additional assessment is also a potential conflict between advanced CI automation and existing organizational processes which have implemented secure processes bound to human interaction. Large organizations where software is business critical therefore should connect the lines between their digital transformation programs and the role of CI/CD systems in the value delivery chain.

---
Tags: process
---


# Question
Title: Docker Containers fails to DNS resolve on subnet 10.0.2.0/24
The ultimate problem I am facing is this:

```
[root@527eb70c04c9 my-service]# ping google.com
ping: google.com: Name or service not known

```

Inside my container, I can't resolve public domains, although the network is defined as not internal:

```
networks:
  my-service-network:
    driver: overlay
    internal: false
    attachable: true
    ipam:
      driver: default
      config:
        - subnet: 10.0.2.0/24

```

As soon as I change my subnet from 10.0.2.0 to 10.0.5.0 (really just "NOT" have it be 10.0.2.0) everything works:

```
networks:
  ess-team-service:
    driver: overlay
    internal: false
    attachable: true
    labels:
      - "group=ess-services"
    ipam:
      driver: default
      config:
      - subnet: 10.0.5.0/24

ping google.com
PING google.com (172.217.22.14) 56(84) bytes of data.
64 bytes from fra16s14-in-f14.1e100.net (172.217.22.14): icmp_seq=1 ttl=61 time=25.0 ms
64 bytes from fra16s14-in-f14.1e100.net (172.217.22.14): icmp_seq=2 ttl=61 time=22.7 ms
64 bytes from fra16s14-in-f14.1e100.net (172.217.22.14): icmp_seq=3 ttl=61 time=22.5 ms
^C
--- google.com ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 22.575/23.462/25.083/1.161 ms

```

A colleague referred to the problem as being caused because Docker's own nameserver is on 10.0.2.0 but I can't find information to back this theory up let alone confirm it.

The main problem is: Sometimes it works. Most times it doesn't but sometimes I can ping google inside the container while the subnet IS 10.0.2.0

# Answer
> 1 votes
We have not been able to figure out what the problem is or what a solution could be.

The workaround is actually not using the `10.0.2.0` subnet, so I will add that as an answer as requested in the comments.

If someone finds the real issue and solution here, I will gladly accept that answer though.

---
Tags: docker, docker-swarm, dns
---


title: Ansible SSH best practices

text:
# Question
Title: Ansible SSH best practices
What are the best practices to be followed wrt security when it comes to SSH authentication (password less, key based, etc.) when setting up Ansible in a new environment with 1000s of machines? Which is the best method of using passwordless SSH in an enterprise environment and why?

# Answer
The answer is `Key Based SSH`

Reasons

1. You can SSH multiple host machines using the same key or Different keys.
2. If you reboot your machine while playbook running it again SSH using your provided key so your ansible-playbook runs without occurring any errors.
3. Also, you get authentication using the appropriate user.

use can use **inventory.ini** file

**example**

```
[hostMachine]
ubuntu@xx.xx.xx.xx  ansible_ssh_user=ubuntu ansible_ssh_private_key_file=/path of your pem file

[ABCDMachine]
ABCD@xx.xx.xx.xx ansible_ssh_user=ABCD ansible_ssh_private_key_file=/path of your pem file

```

**Command for run ansible-playbook**

> ansible-playbook -i inventory.ini main.yml
> 1 votes

---
Tags: ansible, security
---


# Question
Title: SQS: What could high number of visible messages mean if messages received & deleted are normal?
I'm new to SQS.

I recently noticed the `ApproximateNumberOfMessagesVisible` in my SQS queue started rising. What was interesting though is that there was no sudden surge in the number of messages received or depression in number of messaged being deleted from the queue that corresponds with this surge.

### `NumberOfMessagesSent`

### `NumberOfMessagesReceived`

### `NumberOfMessagesDeleted`

### `ApproximateNumberOfMessagesVisible`

Increasing the number of SQS workers temporarily drained this queue, but what could be possible reasons for this increase?

# Answer
> 2 votes
> *there was no sudden surge in the number of messages received or depression in number of messaged being deleted from the queue that corresponds with this surge*

Yes, it looks like there was.

But first, be sure you have the correct definitions in mind for these metrics, because your intuitive interpretation might be reversed.

"Sent" means *sent to SQS and added to the queue* (it does not mean sent to consumers) and "received" means *delivered to a consumer* (it does not mean received by SQS).

Beginning at approximately 1600 on January 3 there is an unexplained dip in "received" performance. The consumers slowed down and accepted messages at a slower rate, despite the fact that the available messages did not justify that reduction, suggesting that the consumers were not able to process messages at their typical rate.

Beyond this point, the messages "sent" (enqueued) continued its undulating pattern, but the messages "received" (dequeued) and "deleted" (after processing) assumed a much flatter pattern, despite the fact that they had previously been following the same visual pattern as the "sent" value.

It appears that your consumers have slowed down, for external reasons (probably unrelated to SQS -- something else the consumers communicate with, like maybe a database, is not responding as quickly as usual), and so the consumers are no longer able to keep up with the offered workload, and a backlog is growing. "Visible" messages are the messages in the queue that are waiting for an available consumer to be ready to receive them.

Theoretically, it *could* be that this is an internal performance problem inside SQS, but this seems very unlikely so I'd suggest that your consumers and the components of your stack that they interact with should be the focus of your troubleshooting.

---
Tags: amazon-web-services, monitoring, amazon-sqs
---


# Question
Title: Spin up an EC2 instance from within the application
I am developing the architecture of this application and the way we need to scale is not just stateless API requests, but it's a logical split of responsibilities. In other words, we may only be able to work on data 1-5 while we need another instance to work on 6-10. We will need an undefined number of instances and would like a manager to spin up and down EC2 instances as needed.

While scouring the web I am not able to come up with anything good, are there any good articles or is there a term for this? I have found IAC (Infrastrucure as Code) but from what I see that is simply deploying additional services, I don't see anything as IAC spun up from within the application. Is there a term for this?

# Answer
Use **AWS API** to spin up the instances. There are SDKs (Software Development Kits) for various languages, for example if you're using Python use boto3 library. SDKs for other platforms are also available: Tools to Build on AWS, including the command line tools AWS-CLI for use in shell scripts.

With the SDK of choice you'll simply call something like `ec2.start_instance(...)` to spin up your new EC2.

However I would question whether you really need standalone EC2 instances for your processing? Perhaps ECS Containers, especially Fargate, or even serverless lambda functions triggered from a work queue may be a better choice?

Hope that helps :)
> 3 votes

---
Tags: amazon-web-services, amazon-ec2
---


# Question
Title: Is there a DevOps methodology aimed towards academia and students?
I am still learning about DevOps, I wonder in Academia the workloads can often be quite heavy, I noticed there are variations on the DevOps methodology, e.g. DataOps and DevSecOps.

I wonder if there is something like StudentOps or similar to this especially targetted to the more scientific and computer science like subjects? I have noticed a similar question: DevOps in research and science projects? . But this does not explain why additionally one has not been developed for academia.

# Answer
Is there a DevOps methodology aimed towards academia and students? No.
> 0 votes

---
Tags: culture, academia
---


# Question
Title: How to automatically deploy routers / industrial PC's
I'm facing the problem, to setup Gateways and Industrial PC's (with SSH access) regularly. All have different default settings. Some have a default root password, some have a default user. Some have a initial fixed IP, some use DHCP. Some are ARM architecture, some are MIPS. OS varies between Yocto, OpenWRT, Raspian, etc.

I'm looking for a way to somehow script the deployment and execute it automatically.

There are tools like Ansible, Puppet, Chef, etc. but I'm not sure which is best suited for my uscase described above or which isn't at all. In best case the tool has no requirements on the target device and works purely via SSH.

# Answer
> 2 votes
Regarding *no requirements on the target device*, of the tools you mention, only Ansible does not need an agent on the host. The minimal requirement to use Ansible modules is Python on the host you want to update. If you don't have Python on hosts you could even send raw commands to the hosts.

Wikipedia has a list comparing configuration management software, there you can find other agent-less tools.

Not listed on Wikipedia is Fabric, a simple Python library to execute remote shell commands. It is agent-less as well.

---
Tags: deployment, automation
---


# Question
Title: How to determine which packaged pip to install in order to run the Ansible pip module?
I am struggling to find a reasonable method to detect which of `python-pip` or `python3-pip` I should install depending on the contents of the `ansible_python` fact. This is on Debian and Ubuntu.

Looking at `ansible_python` contents, I did not see anything that would let me correlate the Python version Ansible uses with the package name used to install *that* Python version. So even though I could glean the path to the Python executable from the fact I don't see how to determine the package used to install it.

The rationale is trying to avoid hardcoding stuff into my playbook. But failing everything else I'd also be willing to hardcode these *assumptions* into my playbook. The "best" I've come up with so far was to use the `package_facts` module to collect the information on installed packages, then use `ansible_python.version.major` to pick from either `python-pip` or `python3-pip`, which still contains the assumptions: 1.) that `ansible_python.version.major==2` maps to `python-pip` and `ansible_python.version.major==3` maps to `python3-pip` and 2.) that `ansible_python.executable` does name either of the *packaged* Python versions available in the `python` and `python3` packages respectively.

Is there an objectively better way? Better isn't meant to elicit opinions here, anything getting rid of the implied assumptions qualifies as objectively better.

# Answer
There are two options either you make sure you use the same Python version on each host or you make sure the right packages are installed on each platform.

## Chose same Python interpreter

Provide the python binary to use. Since you have a Debian on each host, the location is always the same. If you know a Python version that is available on each host, go for this.

Either in your inventory `hosts`

```
[webserver]
srv1.example.com
srv2.example.com

[webserver:vars]
ansible_python_interpreter=/usr/bin/python3

```

Or in your Ansible configuration `ansible.cfg`

```
ansible_python_interpreter=/usr/bin/python3

```

See also https://docs.ansible.com/ansible/latest/reference\_appendices/python\_3\_support.html#using-python-3-on-the-managed-machines-with-commands-and-playbooks

## Package per Python version

If you do not know whether either Python 3 or 2 is available on each host, ensure to get the right package for each Python version.

```
- apt:
    name: python-pip
  when: ansible_python.version.major==2

- apt:
    name: python3-pip
  when: ansible_python.version.major==3

```

You may face the same issue when installing Python packages. So you can put your Pythone version specific tasks in a v2 and in a v3 file and conditionally include one for version 2 and the other for version 3.
> 1 votes

---
Tags: ansible, linux
---


title: Why Ansible skip items in inserafter?

text:
# Question
Title: Why Ansible skip items in inserafter?
I have about 10 hosts that I run a playbook for them with `serial:1`. in this playbook there is this code which append output of all commands in one file:

```
    - name: copy the output to a local file
      lineinfile:
       dest: "{{ myshell_output4.stdout }}"
       line: "{{ item }}"
       insertafter: EOF
      with_items:
        - "########################################################"
        - "###############HOSTNAME:{{ inventory_hostname }}################"
        - "{{ myshell_output3.stdout }}"
        - "########################END#############################"
     delegate_to: localhost

```

it is always working for the first host but for others it only insert second and third line and skip first and last line. the out put of file is like this:

```
  ########################################################
  ###############HOSTNAME:12.15.1.1################
  (some output here)
  ########################END#############################
  ###############HOSTNAME:12.15.2.2################
  (some output here)
  ###############HOSTNAME:12.15.3.3################
  (some output here)
  .
  .

```

why this happen and how i can prevent it?

# Answer
> 1 votes
Apparently `delegate_to` and `lininfile` or `blockinfile` do not work properly together, https://github.com/ansible/ansible/issues/28313.

## Module `blockinfile`

It seems redundant lines have not been written as you would expect. Eventually, `blockinfile` without a marker does the job.

```
- name: copy the output to a local file
  blockinfile:
    dest: "{{ myshell_output4.stdout }}"
    marker: false
    block: |
      ########################################################
      ###############HOSTNAME:{{ inventory_hostname }}################
      {{ myshell_output3.stdout }}
      ########################END#############################
  delegate_to: localhost

```

## Concatenate files at the end

Alternatively, you can write to different files and concatenate those files after all tasks have run.

```
- name: copy the output to a local file
  copy:
    content: |
      ########################################################
      ###############HOSTNAME:{{ inventory_hostname }}################
      {{ myshell_output3.stdout }}
      ########################END#############################
    dest: tmp/{{ inventory_hostname }}
  delegate_to: localhost

- name: log files are concatenated
  assemble:
    src: tmp/
    dest: "{{ myshell_output4.stdout }}"
  delegate_to: localhost
  run_once: true

- name: tmp folder is absent
  path:
    src: tmp/
    state: absent
  delegate_to: localhost
  run_once: true

```

In case your local file `myshell_output4.stdout` does exist already and you want to append to an existing file, you assemble the files `tmp/*` into `tmp/assembled` and the append them with `lineinfile`.

```
- name: append assembled to a local file
  lineinfile:
    dest: "{{ myshell_output4.stdout }}"
    line: "{{ lookup('file', tmp/assembled) }}"
  delegate_to: localhost
  run_once: true

```

---
Tags: ansible, yaml
---


title: Load Balancer and EC2 in same subnet?

text:
# Question
Title: Load Balancer and EC2 in same subnet?
I have a custom VPC with 3 public subnets and 3 private subnets. I have three ec2 instances with elastic IP addresses in the public subnets taking traffic on port 80 (security group is configured to allow access on port 80 in public subnets which are behind an Internet Gateway). I created an Auto Scaling Group and linked it with the three public subnets to increase/decrease instances dependent on traffic. I attached a Target Group to the Auto Scaling Group with access to port 80. Now I plan to create an Application Load Balancer and attach that Target Group to it so that the Load Balancer can distribute load using the Auto Scaler.

In the AWS docs 'Application Load Balancer Getting Started', it states the following:

Before you begin: Decide which two Availability Zones you will use for your EC2 instances. Configure your virtual private cloud (VPC) with at least one public subnet in each of these Availability Zones. These public subnets are used to configure the load balancer. You can launch your EC2 instances in other subnets of these Availability Zones instead.

Notice at the end of the paragraph it says 'instead', as if you should not have EC2 instances configured in the same public subnets as your Load Balancer. Am I interpreting this correctly? You should not have EC2 instances and Load Balancers in the same subnet?

# Answer
> 4 votes
The best practice is to have your **web, app and db servers** in **private** subnets and only the actual **front end** \- which is *ALB* in your case - in the **public** subnet. The reason is that web, app and db servers shouldn't be accessible from outside even if by accident (e.g. through security group left too open).

Technically it will still work to have both the ALB and the instances in public subnet but it's not recommended for the above reason. Also make sure to lock down access to the instance port 80 and 443 only from the ALB SG if you don't want any direct traffic.

Hope that helps :)

# Answer
> 1 votes
To add a little extra to @MLu's answer, and to be even more paranoid...

You may wish to set up a tiny server in yet another subnet as a "jump box". Allow SSH access to the jumpbox subnet from the outside (port 22), but only allow SSH access to your app server subnet from this jumpbox subnet.

With this set up the only way to access your app server is by first logging into the jumpbox, and then doing an extra ssh hop to your app server. There is no other path for ssh from the outside world to your app server.

If this becomes a nuisance for initial set up or debugging, you can invest 15 minutes (or more, it's fiddly the first time) setting up twin ssh tunnels to go from your local machine straight through to the app server.

Most of the time you can leave the jumpbox shut down, so there is no ssh path through to the app server. For extra paranoia, you can restrict access to the jumpbox subnet to your specific (whitelisted) IP address.

If you are still concerned about hackers, you can also set up sshd on the jumpbox to listen on other than the default port 22, so a hacker's port scanner is slightly less likely to find it when your jumpbox is running.

Ideally, leave no information on the jumpbox to suggest the IP address of your app server (i.e. no scripts), so a hacker that somehow does get into the jumpbox will have no clues how to proceed further.

---
Tags: amazon-web-services, load-balance
---


# Question
Title: Are there Docker Hub comparable public Docker image repositories?
Historically, you can download official community software from a distributed mirror network.

Even for Java Maven, there are two sites globally.

But what is about Docker Hub? What if Docker, Inc. decides to close the Hub or the site will become unstable?

Beyond just having business critical images which you can rebuild and store locally, Docker Hub is also:

* indicator that a release is official
* often provides Dockerfile which is very valuable configuration knowledge
* provides links to backing GitHub projects
* features like starring and download numbers

Are there other mirrors not hosted by Docker either comparable initiatives so that an open source community would say "ok we can publish our release also there"?

After a short research, I see there are many products to setup your own registry, but not an alternative Hub site.

# Answer
> 3 votes
With Docker following the Open Container Initiative, in the event that Docker were to disappear, people would still have access to images stored outside of Docker Hub.

While there is not a public mirror of Docker Hub, you do have access to other registries that often store the official image.

For example, if you were to pull the official Docker SQL Server image, you would target `mcr.microsoft.com/mssql/server:2019`. So even though the image can be searched on in Docker Hub, the image still exists outside of Docker Hub. The three big cloud providers (Azure, AWS, and Google) all have container registries that allow public access. Many of the cloud registries support storing of images in regions and would allow "mirrored" architectures that are transparent to the public and disaster recovery (i.e. end users only pull from one link and the provider routes to an active available registry).

---
Tags: culture, docker-registry, images, risk
---


# Question
Title: What makes a difference between high-cost cloud providers and low-cost ones?
I need to decide, for a small client and a small budget (around $500 a month) the proper Cloud Provider to have some compute instances to hold a web server, a database and an Apache server. I was wondering, seeing companies such as DigitalOcean, that offer compute instances on a very low price compared to Oracle, Azure, AWS, Google...

Why do companies choose these big corporations instead of the cheaper options when it comes to infrastructure? What is the big difference between such high and low prices?

# Answer
> 3 votes
@jasper, and welcome to DevOps SE!

While your statement can be confirmed by other voices, it's very hard to x-ray price policies of a company. It's a complex gameplay always driven by many factors and decisions and a question about pricing policies of Digital Ocean vs. AWS might be a good and on possibly well on topic question at Economics SE.

For example, one could argue to find facts supporting the evidence of Digital Ocean currently being cheaper that:

* ~~Docker, Inc. is in a situation where it cannot afford losing customers, so maybe they a cautious and prefer lower margin to attract customers.~~
* Amazon is the market's prime with top notch customers from overall industry and government: stability, seriousity and credability are way more important than money, especially given that expenditures on information technology is only a (small) part of industrial budget planning. Also, Amazon has many more data centers and is as whole investing *billions* to AI technology ($0.8b in 2017, almost complete estimated market value of Docker Inc.), custom data center hardware, buying startups etc.

Please consider making your question more specific in the DevOps SE site context to get more answers.

This sort of decision is possibly best to elaborate together with the customer, collect their business requirements if price is not the only one requirement and assess risks.

**Further reading**/related business topic: *due diligence*.

**Additional note**: one might think, AWS would have way more customers than Digital Ocean but at least as of 2017 this was not the case.

# Answer
> 2 votes
Aside from @Peter s great answer are you comparing Apples with Apples? Your comparison only works, if you compare a VPS as a "box" with another VPS, based on soimplistic criteria such as CPU clock speed, memory and disk space. With a cloud provider you can dramatically reduce costs with the following ways:

* run night time instance sizes vs day time instance sizes
* run no night time instances, not every Intranet is 24/7
* use long term commitments to reduce price
* spot price market for instances
* extract compute workload onto cheap services such as CDNs, event driven queues, serverless setups

On top of that are these instances really the same? What about custom CPUs, hardware based Hypervisors and other technologies that only top 5 public cloud providers can access.

These providers also have free tiers for services or free monthly allocations that you can factor into your cost calculations.

Finally, what is the cost for change. How do you change instance sizes on the compared providers when your requirements change?

# Answer
> 1 votes
I can think of two main reasons:

1. Several of the larger companies offer cheap/free entry level tiers. As the company begins to grow and needs to scale, then the providers charge more. However, at this point most people are "locked in" to the cloud provider either through reservations of servers, provider specific services, or in-house expertise.
2. Brand name is important when it comes to cloud computing. While an "off-brand" cloud provider may have better and cheaper services than the big ones, most people are happy to spend more on a brand they have heard of. Some clients are still wary of cloud providers (e.g. worried about security, reliability, relinquishing physical control), and are more likely to trust Amazon, Google, or Microsoft than a provider they have never heard of.

---
Tags: amazon-web-services, cloud
---


# Question
Title: How to deploy docker-compose to a remote host
I am new to the docker ecosystem and I am trying to figure out a proper deployment infrastructure for a system I am working on.

Imagine a following scenario: a system consists of three microservices - DB, core server, and an app (server + client). For each of these I created a docker image. I also linked them together with docker compose to describe the system as a whole and described links between the microservices. During a release of the microservice I tag the image with a proper tag and push it to a private docker repository. So far so good, it works locally.

Now, imagine I have three different production clients, and each has it’s own one rack linux server. I am not sure how should I approach deployment of the system with docker:

1) I want to be able to specify different versions of microservices for different prodction servers. How should I approach this? Should I have one general docker-compose file or one file with specified versions for each production environment?

Where should I keep docker-compose? In a repository? How it is copied then to a remote host? How to deploy docker-compose.yml file to a remote host?

2) How can I deploy new version of a microservice to a remote host? I am looking for a tool so that if I run

```
tool deploy app@2.33 [server1]

```

it would run the app image at version 2.33 on server1. Is there a tool for that?

For instance I was using a tool for meteor deployment meteor-up that sends the meteor app docker image to a remote host and runs it there - I am looking for the same thing but for a random docker image/docker compose.

I can’t find what is the best practice now in the docker ecosystem to deploy an image to a simple server without AWS/Beanstalk or any other cloud environment. Should I just ssh into it, copy docker-compose and run `docker-compose up -d` ?

Any thoughts appreciated, thank you!

# Answer
> 4 votes
Let's take your questions one-by-one:

> I want to be able to specify different versions of microservices for different prodction servers. How should I approach this?

There are many different ways to handle this. One possibility be to "tag" your docker images and have different docker-compose files that use these images.

> Should I have one general docker-compose file or one file with specified versions for each production environment?

Depends. How many environments do you have? What I've often seen is a single docker-compose that references a separate environment file.

> Where should I keep docker-compose? In a repository?

Again this depends. A standard enterprise best practice would be to store this in a repository so that it is naturally version controlled, and can easily be accessed by others.

> How it is copied then to a remote host? How to deploy docker-compose.yml file to a remote host?

For deploying a docker-compose.yml it depends on the target architecture. Since you mentioned Linux you could use scp to copy the file.

> How can I deploy new version of a microservice to a remote host?

The industry leading tool for this right now is Kubernetes. If you go this approach your docker-compose could be morphed into a Kubernetes deployment. Using just docker-compose you will need to somehow detect that a new container is available and pull it. Watchtower or a similar tool may be helpful.

Based on what you've described I would recommend taking a look at the interactive Kubernetes tutorials, and seeing if that is ultimately an easier approach than everything I outlined above.

---
Tags: docker, docker-compose, deployment
---


# Question
Title: What is the most used tool for Cloud Infrastructure deployment Ansible vs Puppet vs Terraform?
I have some time for research at work right now, and I was thinking on expanding my knowledge on my favorite subject, which is DevOps (love you all).

I have some advanced experience with Ansible, as it has always been my choice when preparing and provisioning hardware in my Cloud tenancies. I started working with Ansible because it was the first software that popped up based on a quick SE search, a couple of months ago. However, I'd like to know **how Ansible compares with *Terraform* and/or *Puppet*, in the following aspects:**

1. Which is the most commonly used tool between the three in the DevOps community, more specifically when deploying Cloud infrastructure to customers, and why is that?
2. Which of the three use the simplest syntax (to my knowledge, Terraform is best at this)? I’m looking for a human-readable syntax if possible which makes things easier to remember.
3. Which is most respectable in the DevOps community, with regard to community standards, and why? Do any of these tools have a feature that makes them stand out from a development POV?

Thank you as always for your time.

# Answer
> 2 votes
@jasper welcome!

This really might depend on the context. On long term it's worth to take a deeper look for a moment if you are not after quick shots and go beyond experiments for a sustainable (and large) infrastructure.

So we have got many clouds out there. Deployment automation means, they have some API.

So what does make a deployment automation tool delivering *infrastructure as code* capability?

Now there are some challenges to address human readability and sustainability varying by programming language ecosystems and communities:

* Cloud products being themselves computer infrastructure and network engineering (these are different disciplines!) abstractions are not following some official "cloud domain" ISO/IEEE naming standards as far as I know. That is, all the names are pushed by vendors and might be marketing biased. So either you invent your own terminology or decide which terms are most generic, or a mix of that. There is therefore a good reason that you find also cloud vendor specific deployment languages and tools for example like *Cloud Formation* by AWS.
* *Somebody* needs to maintain related components to support new API versions for *every cloud platform*!

Some deployment tools delegate the *cloud driver* (that's another term for cloud platform specific code bases) to separate projects, which in their term delegate responsibility for API compatibility to one or another boilerplate (=often boring and tedious) API implementations.

Therefore my conclusion is that depending on your purposes the most straight-forward way to write infrastructure as a code is the programming language of your choice using a well-supported SDK which is though vendor-specific!

Still this is the point where you have got greatest flexibility because for a small standard use case you could write your own abstraction layer and your custom logic whatever you want.

Python examples:

* *boto3* for AWS
* *exoscale/cs* for Apache CloudStack
* *python-openstackclient* for Apache CloudStack

Recent Terraform example: its Apache CloudStack driver has backward compatibility problems because it's based on some less-visible CloudStack API SDK written in Golang. I am sure you can run into problems of this sort with any deployment tool; so if you commit to one, bring enough courage and patience to connect with and contribute to the corresponding community (same is for SDKs but they at least typically have less abstraction).

That is, before deciding for a certain cloud deployment automation tool, consider besides the things you see on the surface like syntactic sugar and community support also the following aspects:

* If it's open source, is it written in a programming language where you/your team could create an emergency patch if things go really worse?
* How well supported are those cloud drivers critical to your business: these typically can't be *all* of them, if so, you might run into issues like GitLab with *Docker Machine* (also a specific deployment tool after all), where a significant technical base for a competitive product feature suddenly breaks away because all the responsibility in this context had been factually outsourced to the open source community.

# Answer
> 1 votes
These tools serve slightly different purposes: use a *configuration management* tool like Ansible/Puppet for configuration / customization and orchestration tool like Terraform to provision infrastructure.

Ansible and Puppet are primarily configuration management tools. It is suitable for resource customization and configuration.

Puppet requires Puppet Agent installation on each server. Ansible doesn't require any agents. It uses SSH protocol for performing all operations.
Ansible supports YAML syntax. Ansible is little more popular now than Puppet.

Terraform is an infrastructure orchestration tool that supports many cloud platforms. With Terraform you simply declare resources and how you want them configured and then Terraform will map out the dependencies and build everything for you. I prefer to use it to provision VMs and Cloud services like Databases or Storages.

All three tools have excellent community support.

---
Tags: ansible, terraform, deployment, infrastructure-as-code, puppet
---


# Question
Title: Unable to deploy webapp with `mvn tomcat7:deploy`: 401 Unauthorized
I'm trying to auto-deploy my webapps to tomcat using Maven plugin `tomcat7-maven-plugin`. I took the following three steps:

1. Added a new user with deployment rights to tomcat,
2. Introduced the mentioned tomcat user to maven,
3. Edited my project's pom.xml to use the plugin.

For the first step, I added the following lines to the file `conf\tomcat-users.xml` in tomcat's installation directory:

```
<role rolename="manager-gui"/>
<role rolename="manager-script"/>
<role rolename="manager-jmx"/>
<user username="war-deployer" password="some-password"
    roles="manager-gui,manager-script,manager-jmx" />

```

and then I restarted tomcat.

For the second step, I added the following to the `<servers>` node of `conf\settings.xml` file in `MAVEN_HOME` directory:

```
<server>
    <id>maven-tomcat-war-deployment-server</id>
    <username>war-deployer</username>
    <password>some-password</password>
</server>

```

And finally, to take to third step, I added the following lines to `<plugins>` node of `pom.xml` in my project to use `tomcat7-maven-plugin`:

```
<plugin>
    <groupId>org.apache.tomcat.maven</groupId>
    <artifactId>tomcat7-maven-plugin</artifactId>
    <version>2.0</version>
    <configuration>
        <url>http://localhost:8080/manager/text</url>
        <path>/my-project-url-path</path>
    </configuration>
</plugin>

```

I thought everything would go fine, but when typing the command `mvn tomcat7:deploy` my webapp didn't get deployed, nothing was added to tomcat's `webapps` directory, and the following response was sent back from tomcat to the command line:

```
[INFO] tomcatManager status code:401, ReasonPhrase:
[INFO] <!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd">
[INFO] <html>
[INFO]  <head>
[INFO]   <title>401 Unauthorized</title>
[INFO]   <style type="text/css">
[INFO]     <!--
[INFO]     BODY {font-family:Tahoma,Arial,sans-serif;color:black;background-color:white;font-size:12px;}
[INFO]     H1 {font-family:Tahoma,Arial,sans-serif;color:white;background-color:#525D76;font-size:22px;}
[INFO]     PRE, TT {border: 1px dotted #525D76}
[INFO]     A {color : black;}A.name {color : black;}
[INFO]     -->
[INFO]   </style>
[INFO]  </head>
[INFO]  <body>
[INFO]    <h1>401 Unauthorized</h1>
[INFO]    <p>
[INFO]     You are not authorized to view this page. If you have not changed
[INFO]     any configuration files, please examine the file
[INFO]     <tt>conf/tomcat-users.xml</tt> in your installation. That
[INFO]     file must contain the credentials to let you use this webapp.
[INFO]    </p>
[INFO]    <p>
[INFO]     For example, to add the <tt>manager-gui</tt> role to a user named
[INFO]     <tt>tomcat</tt> with a password of <tt>s3cret</tt>, add the following to the
[INFO]     config file listed above.
[INFO]    </p>
[INFO] <pre>
[INFO] <role rolename="manager-gui"/>
[INFO] <user username="tomcat" password="s3cret" roles="manager-gui"/>
[INFO] </pre>
[INFO]    <p>
[INFO]     Note that for Tomcat 7 onwards, the roles required to use the manager
[INFO]     application were changed from the single <tt>manager</tt> role to the
[INFO]     following four roles. You will need to assign the role(s) required for
[INFO]     the functionality you wish to access.
[INFO]    </p>
[INFO]     <ul>
[INFO]       <li><tt>manager-gui</tt> - allows access to the HTML GUI and the status
[INFO]           pages</li>
[INFO]       <li><tt>manager-script</tt> - allows access to the text interface and the
[INFO]           status pages</li>
[INFO]       <li><tt>manager-jmx</tt> - allows access to the JMX proxy and the status
[INFO]           pages</li>
[INFO]       <li><tt>manager-status</tt> - allows access to the status pages only</li>
[INFO]     </ul>
[INFO]    <p>
[INFO]     The HTML interface is protected against CSRF but the text and JMX interfaces
[INFO]     are not. To maintain the CSRF protection:
[INFO]    </p>
[INFO]    <ul>
[INFO]     <li>Users with the <tt>manager-gui</tt> role should not be granted either
[INFO]         the <tt>manager-script</tt> or <tt>manager-jmx</tt> roles.</li>
[INFO]     <li>If the text or jmx interfaces are accessed through a browser (e.g. for
[INFO]         testing since these interfaces are intended for tools not humans) then
[INFO]         the browser must be closed afterwards to terminate the session.</li>
[INFO]    </ul>
[INFO]    <p>
[INFO]     For more information - please see the
[INFO]     <a href="/docs/manager-howto.html" rel="noopener noreferrer">Manager App How-To</a>.
[INFO]    </p>
[INFO]  </body>
[INFO] </html>

```

Although no error was reported by maven, and I received a `BUILD SUCCESS` at the end. I've been struggling with the problem for several hours, and searched the Web, but I couldn't find the solution to it. What's wrong with the process that I've followed? Please help me solve it. Thank you very much.

# Answer
> 1 votes
That's because you forgot to link the `<configuration>` for the plugin with the `<server>` definition in settings.xml., and the tomcat7 plugin is trying to deploy as un-authenticated because it has not credentials to try.

Add `<server>maven-tomcat-war-deployment-server</server>` in the configuration section of the plugin:

```
<configuration>
    <server>maven-tomcat-war-deployment-server</server>
    <url>http://localhost:8080/manager/text</url>
    <path>/my-project-url-path</path>
</configuration>

```

---
Tags: maven, tomcat
---


# Question
Title: The Phoenix Project and Military service
I have recently been reading "The Phoenix Project". While I can relate to most aspects of the story, I am struggling to understand the frequent references to the military background of many characters (e.g. Bill himself, Stefe, Erik, ...).

Since I am not familiar with US culture and army in particular, I am pondering 2 questions:

**Question 1:** Are these reference to the army background an essential part of the message of the book or just story-telling (e.g. trying to give some depth to the characters by telling their history)?

**Question 2:** If an essential part, how does the way the US army organises its operations relate with a DevOps culture? Should it be *embraced* or *avoided* in the workplace?

# Answer
My take on the military references in the book, as a veteran, has to do with Man, Machine, Method, Measure; and, with drilling.

The military is entirely other compared to civilian operational models. Setting aside the chain of command and the Military Justice systems; consider that the way any service member mops, waxes and buffs a floor is detailed in a procedure outlining what (exact) equipment is to be used, how it is to be calibrated, how the person is to operate said equipment and measures by which to determine compliance with the procedure. The same goes for brewing coffee, making Kool-Aid, or launching aircraft from a carrier deck.

Secondly the military runs "drills" constantly so emergencies aren't chaotic and reactive; they are methodical and responsive. Hence these would be recommended practices in a DevOps context. Practice well-documented and consistently executed procedures often to increase reliability and responsiveness in the face of the unknown.

In Phoenix project they apply both these practices to tame the chaos that Bill and the team have been subject to throughout the book.
> 3 votes

# Answer
Check out a book called *Team of Teams* by General Stanley McChrystal. This is how the military operates today and is really all about putting the power to make decisions into the hands of the people on the ground and out of the hands of senior military leadership. It is not my typical read but is really a fantastic book that dives into not only military scenarios/examples but also areas such as the airline industry and how they have managed to dramatically reduce air crashes over the last decade caused by human error (the recent Boeing disasters aside).
> 3 votes

# Answer
Without input from the authors I doubt we'll get a definitive answer but I'll try my best.

As far as I know none of the authors of The Phoenix Project served in the military, however my understanding of the USA is that serving in the army for a couple of years to help fund college is an extremely common practice. In other countries (such as the UK where I live) someone who enters the military often views it as a career and serves for a longer term. In the US it's much more common that people will have some kind of military background.

**Question 1:** Are these reference to the army background an essential part of the message of the book or just story-telling (e.g. trying to give some depth to the characters by telling their history)?

I don't believe an understanding of military process is required to understand or appreciate The Phoenix Project. So why did the authors include them?

* To illustrate the "rigor and discipline" mentality that Bill started the role with.
* To emphasise the similarities with military practice and repetition which Kaizen values so highly.
* To give small talk for Steve, Bill, and Eric to all talk about when the story needs it
* To give Bill more backstory (see the "opening up" scene)
* To make Bill more relatable to many a large demographic of ex-military personal now working in industry

**Question 2:** If an essential part, how does the way the US army organises its operations relate with a DevOps culture? Should it be embraced or avoided in the workplace?

This is harder to answer as I have never been in the military myself. However I wouldn't worry about it too much, it's not essential for the story and you could equally draw these parallels yourself with other teams/organisations you are more familiar with from sports, religion elsewhere.

However, as an extreme outsider the US military has a reputation for a couple of things which do apply to DevOps:

* The size and complexity of the organisation
* High levels or organisation in a hierarchical manner
* Enforcing standards and processes
* Creating strong teams (especially the Marines - see Simon Senik's Leader's Eat Last)

### TLDR

Is a knowledge of military required to understand and get value from The Phoenix Project? No. The idea of the military is mostly used as an example for Kaizen, a illustration of Bill's starting point, and to add some character to the erm... characters.
> 3 votes

---
Tags: culture, the-phoenix-project
---


# Question
Title: User Data in launch configuration of amazon web services
I was trying to create an EBS volume by writing a shell script in the user data of my launch configuration. I don't think there is anything wrong with my bash script. I kept looking up creating launch configuration and writing user data in AWS websites, but could not find anything more than what I did below. For my launch configuration, I used Amazon Linux 2 AMI which has pre-built AWS CLI. So I just wrote the following code in my user data:

```
    #!/bin/bash
    aws ec2 create-volume \
        --volume-type gp2 \
        --size 100 \
        --encrypted \
        --availability-zone us-east-1c

    aws ec2 create-volume \
        --volume-type gp2 \
        --size 100 \
        --encrypted \
        --availability-zone us-east-1d

```

As you can see I am trying to create EBS volumes in two availability zones, but I just could not find out why it does not create any volume for me?

# Answer
> 2 votes
1. Make sure the EC2 instance has permissions to create the volumes. See https://aws.nz/best-practice/ec2-instance-roles/
2. Check for the errors in `/var/log/cloud-init-output.log` and (much more verbose) `/var/log/cloud-init.log`
3. Be aware that you can't attach volumes from other AZs to your EC2, so not quite sure why you would want to create them from the launch script.
4. *(added based on comments)* Specify region: `aws --region us-east-1 ec2 create-volume ...`

Hope that helps :)

---
Tags: amazon-web-services
---


title: Mail logs with Jenkins

text:
# Question
Title: Mail logs with Jenkins
I am deploying my software on different OS ( Red hat & suse) and I need to know if my latest version of the software will work. In order to check if the installation works it's enough to check some log files generated by the software.

Now I am trying to integrate some CI/CD tools and I am trying Jenkins now. I have created a pipeline which is triggered by every commit in bitbucket and I need a way to get the log files and the build status from the servers and mail them to the rest of the team.

# Answer
> 1 votes
If you're using Jenkins pipeline, you can use the email notification post action, as a way to *send* the message:

```
post {
    failure {
        mail to: 'team@example.com',
             subject: "Failed Pipeline: ${currentBuild.fullDisplayName}",
             body: "Something is wrong with ${env.BUILD_URL}"
    }
}

```

(you would have to configure the server and credentials for your case)

The message itself can be constructed from the job data. The easiest way is perhaps to tee the output of the job to a file and use it as an artifact to attach to the message that gets sent.

---
Tags: jenkins, continuous-integration, jenkins-pipeline
---