# Question
Title: Do I use Filebeat, Ingest or Pipelines to get rid of Logstash in my ELK stack?
I'm fairly new to filebeat, ingest, pipelines in ElasticSearch and not sure how they relate.

In my old environments we had ELK with some custom grok patterns in a directory on the logstash-shipper to parse java stacktraces properly. The logstash indexer would later put the logs in ES.

How do I do this without Logstash? And how do I parse custom log formats?

# Answer
You might be able to use Filebeat directly. Newer versions of filebeat has a concept of "modules", which is custom parsers you can apply. There are built-in ones for stuff like nginx, apache2, redis etc. Not sure how one goes about writing custom modules, but it might be an avenue worth exploring.

That said, more advanced/custom parsing is likely better left to Logstash than any other component in the Elastic stack.
> 2 votes

# Answer
I am not sure if you are open to commercial solutions, but Sumo Logic is an excellent ingest-type solution. I also tried Logly and LogDNA which were nice but not as powerful and feature rich as Sumo. Let me also say am not a shill for this company. I’m just a DevOps guy who hates dealing with crappy and incomplete log collection.

After dealing with the complexity of ELK, which is admittedly is very powerful and flexible so I have read, ElasticSearch is not for the faint of heart, especially when you walk into a shop and their ELK stack is 4 years old. It’s important to know right right off that Sumo is a cloud solution, so as long as you’re okay with shipping your logs and metrics over HTTPS to their servers, it’s a good way to go. Then you can focus on just feature configuration without having to maintain the core service. It offers most of what I remember that Splunk does without being prohibitively expensive. As you might have guessed, I just completed a project where I moved from ELK to Sumo for production log data and metrics visualization and it is a world of difference from Kibana as far as how much easier it is to get useful output. I found statistical capabilities similar as well, but Kibana does seem to be more actuarially adherent. It’s easy to set up alarms, integrate with RESTful APIs for tools like like Slack, perform complex data correlation, and make purdy dashboards for management. It’s comprehensive as it is an entire stack. Also, there is a very well-developed chef cookbook so installation and configuration is very manageable once you get your ingestion configured along with rulesets. That can be time consuming but as far as cookbooks go I’d say it’s medium complexity. Data ingestion is incredibly flexible- I have not hit any limitations with what it can do, but I did give up trying to ingest directly from logstash as getting metadata to map was more trouble than it was worth. While slurping text logs is what they are geared for, pushing logs and metrics is also an option. We continue to use logback for our app logging, but now Sumo collects pretty much all our logs now as well as graphite data. It does Windows Event Logs nicely as well. We are starting to integrate fluentd from k8s but we’re having some challenges keeping the log volume tamed.

They have their own pre-built support setups for a large amount of OSs, app frameworks, and infrastructure elements which can make initial setup that much faster, but some vendors like Fastly even have advanced prebuilt Sumo Logic implementations that are downright fantastic so our CDN logs create a remarkable dashboard giving an excellent picture of site health, load, as well tons of other useful data. There is definitely a learning curve but nothing like ELK. Getting all your regex just right so things like exceptions in app logs are handled properly can take time, but you can get a lot going with very little effort as a lot of tedious problems like automatic timestamp detection and multi line detection are built in if you want them. Ingestion can be as sophisticated as you want it to be. I found set up kind of evolutionary over the last year where I started with a very basic setup where I did all my pattern matching in my queries, but eventually I had ingestion rules that sifted all my fields out for me which made things more user-friendly for other users. Also, support and documentation are very good. As far as paid services go, this one seemed quite worth it unless you’re already and ace with ELK.

Boy, that was probably a lot more info than you bargained for, but it’s a fair breakdown of Sumo Logic. (I hope!)
> 1 votes

---
Tags: logstash
---


# Question
Title: Should compilers be kept in artifactory?
We are setting up our source and build system in AWS. we will have two persistent VMs hosting source and artifactory. We have a bunch of compilers (a few Gigs) we want available via NFS, is this something that should be hosted in artifactory?

I'm looking for the 'best practice' answer. I can't find much info on the thought behind this so any links would be appreciated.

# Answer
In many cases compilers are used as-is, already packaged by 3rd parties. When changing the structure of the build systems changing compilers usually translates into switching to a different version, but which is also pre-packaged. If this is your use case the artifact repository offers a single advantage: availability. Even if the provider decides (for whatever reason) to discontinue the version - you still have your copy in the respository.

But if the compilers receive any updates without version changes or if you do any sort of customisation - even if it's just building them without touching any code - you want to store those particular versions in the artifactory to be able to reliably obtain them at any time.

Side note: how exactly you use the compilers can influence the performance of your builds, especially in very large projects with build times which can take hours. You'd have to analyze your builds' structure and/or usage patterns to figure out if it's worth pursuing optimisations and, if so, in which direction(s).

Directly invoking compilers from NFS is convenient - very little configurations required on the build servers. But performance of massive builds can be very sensitive to the NFS performance, of both the NFS server itself and of the network connecting it to the build server(s). Some artifactories may have direct NFS support. You'd have to measure the impact of switching to such service, in some cases it may still be better to use a dedicated NFS server instead. Or maybe switching to VM images that contain the compilers locally installed.
> 2 votes

# Answer
There are a few main reasons why you would use Artifactory (or any other binary repository manager) over a traditional file storage (in your case NFS).

1. The ability to manage your artifact versions
2. A central location to access artifacts across infrastructure (which you have accomplished with NFS)
3. The ability to download/recreate previous versions of the artifacts (you could also use a git/svn repository for this, but it would be contain MUCH more irrelevant metadata about the binary artifacts than would be useful to a user)
4. You want to control who/what has permissions to access the stored artifacts

If none of these are applicable in your use-case, I would say that Artifactory would be overkill and the effort to create and maintain it would not out way the value you would receive from it. Although I have used it in the past, and it is a great tool for what it aims to accomplish :)
> 3 votes

# Answer
One solution that might fit your scenario are environment modules. Given that you only need the compilers on Linux, and have access via a network share (nfs/Samba). To load a toolchain into your terminal, just type

```
module load gcc

```

and let autocompletion fill in the version number. It is basically Python’s virtualenv for any Linux program/toolchain/SDK. I have used it extensively and can warmly recommend this tool.

Each compiler would have their own folder which is subdivided by version which allows you to have direct access to all versions. Configuration is defined server-side with PATH and other environment variables. User-side, a bash script has to be installed into users’ `.bashrc`
> 1 votes

---
Tags: amazon-web-services, builds, artifacts, artifactory
---


# Question
Title: Ansible does not create directories CentOS7
This my ansible script

```
- hosts: localhost
  tasks:
    - file:
       name: add dir
       path: /etc/ansible/roles/webservers/'{{item}}'
       state: directory
       mode: 0755
       owner: nikola
      vars:
       dir_list:
        - handlers
        - vars
        - defaults
        - meta
      with_items: ' {{ dir_list }} '

```

and this is the output from `--check` option

```
PLAY [localhost] ****************************************************************************************************************************************************************************************************************************

TASK [Gathering Facts] **********************************************************************************************************************************************************************************************************************
ok: [localhost]

TASK [file] *********************************************************************************************************************************************************************************************************************************
ok: [localhost] => (item=handlers)
ok: [localhost] => (item=vars)
ok: [localhost] => (item=defaults)
ok: [localhost] => (item=meta)

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0

```

# Answer
You have two mistakes:

1. `name` as parameter of `file` module. It is an alias of `path`, so you actually try to create directory `add dir`.
2. Unnecessary single quotes in `path`, giving you actual paths like `etc/ansible/roles/webservers/'handlers'`.

Try with:

```
...
- name: add dir
  file:
    path: '/etc/ansible/roles/webservers/{{ item }}'
...

```
> 1 votes

---
Tags: ansible, centos
---


# Question
Title: Nginx reverse proxy setup issues with Docker containers
I have two running docker containers:

1. A Docker container running Nginx that serves Angular app resource files that were copied into the Docker image on docker build
2. A Docker container that is an api that is current listening on port 5007 internally on the container and 5007 externally on the host machine

I am setting up a staging environment and the following external urls:

* app.staging.mysite.com
* account.staging.mysite.com

app.staging.mysite.com hits port 80 and responds with the Angular app resources no problem.

Now I want to get *account.staging.mysite.com* external API URL request to hit my API Docker container listening on port 5007. Requests from *account.staging.mysite.com* will hit port 80 of the host machine.

Nginx will receive the *account.staging.mysite.com* port 80 requests and proxy them to Docker container *127.0.0.1:5007* while at the same time still serve my Angular app file by default for all external urls/ domains that I do not explicitly express for proxy in my Nginx config.

Instead of serving my Angular app I would like to use Nginx to forward the request to port 5007 so the my Account API can response. So I have altered my Nginx config to the following:

```
upstream accountstaging{
    server 0.0.0.0:5007
}
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name _;

    # Main
    location / {
        root   /usr/share/nginx/html;
        index  index.html index.htm;
        try_files $uri$args $uri$args/ /index.html;
    }
    error_page   500 502 503 504  /50x.html;
    location = /50x.html {
        root   /usr/share/nginx/html;
    }
}
server {
    listen 80;
    server_name account.staging.mysite.com;

    location / {
        proxy_pass         http://accountstaging;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
    }
}

```

The thing that really surprises me is now there is no default\_server hits. In other words, *app.staging.mysite.com*, which was working and responding with my Angular app resources is no longer working when I added the second server listening on port 80 specifically for *account.staging.mysite.com*.

So really this question is not that much about Docker containers, this is really about Nginx configuration. Though I am not entirely sure that my Docker containers could be excluded from being part of the problem. So here is my docker-compose.yml for *account.staging.mysite.com*:

```
version: '3'

services:
  apistaging:
    build:
      context: ./
      dockerfile: docker/staging/Dockerfile
    image: tsl.api.account.image
    container_name: tsl.api.account.container
    ports:
      - "5007:5007"
    environment:
      ASPNETCORE_URLS: http://+:5007

```

And here is my docker-compose.yml for app.staging.mysite.com:

```
version: '3'

services:
  frontend:
    build:
      context: ./
      dockerfile: docker/Dockerfile
    image: tsl.web.frontend.image
    container_name: tsl.web.frontend.container
    ports:
      - "80:80"

```

Here is my Docker file for Nginx that serves my Angular app and is hopefully going to act as a reverse proxy as well, you can see that Nginx default\_server servers my Angular app resource files that I copied into this Docker image on docker build:

```
FROM centos:7
MAINTAINER Brian Ogden

# Not currently being used but may come in handy
ARG ENVIRONMENT
ENV NODE_VERSION 6.11.1

RUN yum -y update && \
    yum clean all && \
    yum -y install http://nginx.org/packages/centos/7/noarch/RPMS/nginx-release-centos-7-0.el7.ngx.noarch.rpm \
    yum -y makecache && \
    yum -y install nginx-1.12.0 wget

# Cleanup some default NGINX configuration files we don’t need
RUN rm /etc/nginx/conf.d/default.conf

#############################################
# NodeJs Install
#############################################

#Download NodeJs package
RUN wget -q -O - https://nodejs.org/dist/v$NODE_VERSION/node-v$NODE_VERSION-linux-x64.tar.gz \
    | tar --strip-components=1 -xzf - -C /usr/local

# https://stackoverflow.com/a/35774741/1258525
# use changes to package.json to force Docker not to use the cache
# when we change our application's nodejs dependencies:
COPY ./package.json /tmp/package.json
RUN cd /tmp && npm install
RUN mkdir /app && cp -a /tmp/node_modules /app/

WORKDIR /app
COPY . /app

RUN npm run build-$ENVIRONMENT

RUN cd /app && cp -a dist/* /usr/share/nginx/html
COPY ./docker/conf/frontend.conf /etc/nginx/conf.d/frontend.conf
COPY ./docker/conf/nginx.conf /etc/nginx/nginx.conf

EXPOSE 80

CMD ["nginx"]

```

Is this issue possibly because of 80:80 binding for the Angular app frontend docker container?

# Answer
Thanks to this question and answer here, I was able realize that I had two issues going on:

1. the containers have different default Docker networks because I am using two different docker-compose.yml files, I had envisioned my Ngnix proxy working independently from any of my API containers entirely, including the docker-compose, more on that issue below
2. the second issue is simply when I tried to proxy to 127.0.0.1:5023 that is localhost inside the Ngnix container, not the network outside of the Nginx proxy container

So the different default networks being created by docker-compose for my Nginx proxy docker container and my api docker container are because I am using two different docker-compose.yml files. This is because I have Jenkins builds for many API microservices so the have independent docker-compose files and I needed a Nginx proxy to forward requests on port 80 to each microservice.

To test this out, created a docker-compose.yml for both containers, the API and the Nginx proxy:

```
version: '3'

services:
  reverseproxy:
    build:
      context: ./
      dockerfile: docker/nginxproxy/docker/Dockerfile
    image: tsl.devops.reverseproxy.image
    container_name: tsl.devops.reverseproxy.container
    ports:
      - "80:80"
  apistaging:
    build:
      context: ./
      dockerfile: docker/staging/Dockerfile
    image: tsl.api.example.image
    container_name: tsl.api.example.container
    ports:
      - "5023:5023"
    environment:
      ASPNETCORE_URLS: http://+:5023

```

Yes there was still an issue, the proxy pass to http//:127.0.0.1:5023, that forward remains in the Nginx Docker container and never finds the API running on the Docker host, I simply needed to use the docker-compose.yml service name to get to it:

```
upstream accountstaging {
    server apistaging:5023;
}

server {

    listen 80;
    server_name account.staging.mysite.com;

    location / {
        proxy_pass         http://accountstaging;
        proxy_redirect     off;
        proxy_set_header   Host $host;
        proxy_set_header   X-Real-IP $remote_addr;
        proxy_set_header   X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header   X-Forwarded-Host $server_name;
    }
}

```
> 1 votes

---
Tags: docker, docker-compose, node.js, nginx
---


# Question
Title: How to populate docker volumes in a build step to be used by a different container
I currently have a Docker compose based web app that uses Nginx as a reverse proxy that forwards traffic to a Flask and Grafana docker instance. The task at hand is to integrate a SSG (static site generator) such as Jekyll into the setup.

The most straight forward approach is to:

1. have a Jekyll container continually run and serve the static aspect of the site via Nginx redirects (`jekyll serve`).
2. Another approach would be to run the build command in the Jekyll docker image and have the output be shared via docker volumes with Nginx.
3. Yet another is to generate the site in the Nginx container

The downside with the first two solutions is that they create a Jekyll docker instance that is only used once and then persists. In 1.'s case, the serving can be directly handled by Nginx. In 3.'s case, there is no separation between Jekyll and Nginx. This would be useful since Nginx is also responsible for routing traffic to the other Docker instances.

The question is, is there a way to populate a Docker volume with a Dockerfile by Jekyll so that this can then be mounted by Nginx? Step by step:

1. `docker-compose build`: build the site and populate the *static-site* volume
2. `docker-compose up -d`: Nginx mounts the *static-site* volume and serves it

Does this approach even make sense?

# Answer
> 5 votes
`docker-compose` is just a glorified wrapper around `docker` which gives you a nice textual representation of all the options you would regularly give to `docker build` or `docker run`. Plus, of course, it bundles several docker images/containers together. That said, it has no provisions to run anything else than you would get by calling `docker`.

I don't see how you would use `docker-compose build` to populate a volume. Volumes are, by definition, not living inside docker images/containers, but outside (on the host system). `docker build` never writes to the host system, and you cannot mount volumes during `docker build`. `docker-compose build` basically only runs `docker build` for you, and also has no way to run or write anything on the host.

So, no, `docker-compose build` will not be able to create the static files.

You could make your compose file so that it runs your generator (inside a docker container, writing to a named volume), and runs your server (nginx or whatever) in another container, mounting the same named volume (this would be your case 2, I believe). Then, with `docker-compose up`, it would fire up both those containers. The first one would populate your static files; the second would serve. The first would hang around and do nothing afterwards, which you could just ignore, but which would certainly be ugly. Unfortunately, you cannot use something like `docker-compose up --rm` to automatically the unneccessary jekyll container.

So, your best bet is to start the jekyll container yourself (`docker run --rm jekyll:xyz -v ...:...`), the `--rm` will remove the container after it is finished. Then `docker-compose up` with the actual server(s).

---
Tags: docker, docker-compose, nginx
---


# Question
Title: What is the meaning of source artifacts that must be in a scripted form?
Quote from rom https://www.ibm.com/developerworks/library/a-devops6/index.html

> The key prerequisite to versioning everything is that all source artifacts must be in a scripted form. This goes for the infrastructure, the data, configuration, and the application code. The only exception is for libraries and packages — JAR files and RPM packages, for example — that you use but never modify. After all source artifacts are scripted, you can easily version them.

What do they mean by "*source artifacts must be in a scripted form*"?

# Answer
> 3 votes
IMO these could be possible synonyms for ***scripted***, in the context of your question here:

* in a *textual* format, which you can edit in some editor, like YAML, XML, JSON, PHP, etc.
* NOT in a *binary* format, the result of some build process (like a .EXE file, etc).

And I bet the reason for the "key prerequisite" (as in your quopted text), is that in the end you want to be able to perform some sort of comparison between 2 versions of such files. E.g. like a patch file (if usig GIT). Think about how a compare of 2 versions of a binary looks like, pretty sure every line is updated (or: all old lines are deleted, and all new lines are inserted).

This applies to pretty much any operating system, be it Linux, Windows, ..., or even zOS (aka good old mainframes).

---
Tags: artifacts, versioning
---


title: Have Jenkins trigger GitlabCI

text:
# Question
Title: Have Jenkins trigger GitlabCI
Assuming a user initiates a Jenkins job, passing some input parameters.

Is there a way, at some point to have the Jenkins job trigger a `gitlab-ci`, **and** also pass to it (i.e. the `gitlab` pipeline) some input parameters which the latter will use as environment variables?

# Answer
Note: I didn't use this yet, the answer is based solely on documentation.

A GitlabCI pipeline can be triggered via API, see Triggering pipelines through the API.

> Triggers can be used to force a pipeline rerun of a specific `ref` (branch or tag) with an API call.

You would first need to create a trigger for that pipeline.

Then you can activate that trigger (from your jenkins job in this case):

> To trigger a job you need to send a `POST` request to GitLab's API endpoint:
>
> ```
> POST /projects/:id/trigger/pipeline
>
> ```
>
> The required parameters are the trigger's `token` and the Git `ref` on which the trigger will be performed. Valid refs are the branch and the tag.

Such triggers support variable passing:

> You can pass any number of arbitrary variables in the trigger API call and they will be available in GitLab CI so that they can be used in your `.gitlab-ci.yml` file. The parameter is of the form:
>
> ```
> variables[key]=value
>
> ```

You can use the variables inside `.gitlab-ci.yml` like this:

> ```
> upload_package:
>   stage: package
>   script:
>   - if [ -n "${UPLOAD_TO_S3}" ]; then make upload; fi
>
> ```

And

> You can then trigger a rebuild while you pass the `UPLOAD_TO_S3` variable and the script of the `upload_package` job will run:
>
> ```
> curl --request POST \
>   --form token=TOKEN \
>   --form ref=master \
>   --form "variables[UPLOAD_TO_S3]=true" \
>   https://gitlab.example.com/api/v4/projects/9/trigger/pipeline
>
> ```

This kind of POST requests is what you would to do in your jenkins job. But take a closer look at the triggers, you might find a way to do what you want without jenkins.
> 2 votes

---
Tags: jenkins, continuous-integration, continuous-deployment, gitlab, gitlab-ci-runner
---


# Question
Title: Modelling economic value of risk-reduction DevOPs efforts
A significant part of the activity of software Engineers working on automated testing, continuous delivery, service integration and monitoring, and building SDLC infrastructure strives to reduce operational risks lying on the system they are working on. To estimate the economic value of these efforts require a risk model. As a first approach, it seems reasonable to throw an *ad hoc* model where each risk occurrence is governed by a Bernoulli variable (roughly speaking a coin that can be weighted to fall more or less on a given side). But how to find sensible parameters for this model.

To take a concrete example, if I develop a disaster recovery procedure and run drills each month to prove it working, I would need to estimate the odds of my company going bankrupt because it is unable to recover quickly enough from a disaster, and I was unable to find any sensible data on this.

Is there any sensible risk model, together with its parameters, that can be used to estimate the value of efforts invested in risk-reduction?

# Answer
You are probably unable to find data on this for two main reasons. Firstly, because it varies from system to system. Does an outage cost you money due to contractual obligations? Or only in lost reputation and in user perception? How much does *that* cost? This varies from organization to organization. But I think that might be what you are getting at with the Bernoulli variable...

...Which brings me to the second thing. In order for you to be able to model risk, you need research in which someone has already statistically proved or measured the efficacy of DevOps methodologies (which means you have to define exactly what DevOps is first, a notoriously tricky issue.)

You also need to be able to compare that to old methods - meaning that someone must also have measured data for the Dev/Ops segmented model. I'm not sure that research exists or is settled. Probably because the efficacy and cost of Dev/Ops will also vary widely from organization to organization. In that respect it may provide an opportunity to measure the old methodology and then show the average improvement of changing methodologies if you can find enough research subjects.

In short, it is doubtful that such a formula for risk modeling exists. If it does, it likely exists as paid marketing research to a consulting firm and is unlikely to be available to the public and I would personally be very dubious of the accuracy of it's results due to the subjectivity of the input variables measured.
> 2 votes

# Answer
> Is there any sensible risk model, together with its parameters, that can be used to estimate the value of efforts invested in risk-reduction?

Yes, of course. For non-life-threatening risks (say, your common bug which just induces some cost to fix it): take the chance of the problem happening, multiply by cost, and there you go. Easily done by going through your bug ticket system, where you hopefully have recorded the effort gone into fixing.

Then there are the life-threatening risks where you do not actually need to factor in the cost, but there is no question that you do your utmost to avoid it. For example, if you are a small company with one product, and you lose your source code, then it is not a question about how much that costs. You can just pack in. You will want to have some sort of backup strategy that is, by all means and purposes, unbeatable (for example: the boss takes a current set of physical backup tapes back home each weekend =\> and yes, I have worked in companies where that was done (in addition to all the usual other measures, like snapshotting file systems, distributed git repositories, on-site backups, off-site online backups etc.)).

Nothing about this is particularly related to DevOps. DevOps just happens to bring a few tools to the table (like CI/CD pipelines, relatively easy HA through read-only redundant containers/pods etc.).
> 2 votes

---
Tags: risk
---


# Question
Title: How to get instance's route table id using CLI?
I'd like to find a route table id associated with the given EC2 instance.

How can this be achieved using AWS CLI?

---

I've tried to use `aws ec2 describe-vpcs`, but the route tables are not there.

I'm using `aws-cli/1.11.13`.

# Answer
> 1 votes
You can achieve this by determining which subnet the ec2 instance belongs to, then check which route table the subnet is associated with.

```
#!/bin/bash

instanceId='YOU-INSTANCE-ID'

#finds the subnetId that the instance belongs to
subnetId=$(aws ec2 describe-instances \
           --instance-id $instanceId \
           --query "Reservations[*].Instances[].SubnetId" \
           --output text)

routingTableId=$(aws ec2 describe-route-tables \
                 --query "RouteTables[*].Associations[?SubnetId=='$subnetId'].RouteTableId" \
                 --output text)

echo $routingTableId

```

# Answer
> 1 votes
The route table is associated with VPC, which is associated with the instance.

Given `$id` shell variable has the instance ID, e.g.

```
id=i-0xyz # Replace i-0xyz with the real Instance ID.

```

here is the shell command to get VPCs associated with the instance and assign to `$vpcs` variable:

```
vpcs=$(aws ec2 describe-instances --instance-id $id --query 'Reservations[].Instances[].NetworkInterfaces[].VpcId' --output text)

```

Then to list the route tables associated with VPC, run:

```
aws ec2 describe-route-tables --filters "Name=vpc-id,Values=$vpcs"

```

To get just the route ids, add: `--query "RouteTables[].RouteTableId[]"` parameter.

---
Tags: amazon-web-services, amazon-ec2, aws-cli
---


title: Powershell script in jenkins is hanging

text:
# Question
Title: Powershell script in jenkins is hanging
I'm building a process to automate deploys using Jenkins and I have an issue getting the code on the server

I have a powershell script on the deploy server that all it does is do an hg update command.

```
$ciPwd = ConvertTo-SecureString $env:CI_SERVER_PWD -AsPlainText -Force
$ciCredential = New-Object System.Management.Automation.PSCredential ($env:CI_SERVER_USR, $ciPwd)
Invoke-Command -ComputerName web1-dev,web2-dev -Credential $ciCredential -ScriptBlock { C:\hg\Update.ps1 } -InDisconnectedSession -SessionName srv1,srv2

Write-Host "web1-dev output:"
Receive-PSSession -Name srv1
Remove-PSSession -Name srv1

Write-Host "web2-dev output:"
Receive-PSSession -Name srv2
Remove-PSSession -Name srv2

```

the script always hangs at the 'web1-dev output:' portion.

I'm kinda new to powershell scripting so its possible I'm doing something wrong, and if there's a better way to call a remote powershell script, I'm open to that

# Answer
> 3 votes
I'm guessing that your PowerShell script is hanging because of the Invoke-Command command. My best guess is PowerShell is either prompting you for something, you are passing in an argument incorrectly, or you are executing the script in your script block incorrectly. I'm not sure exactly how Script-Block works when referencing paths to PowerShell scripts, but it could be because C:\hg\Update.ps1 does not exist on the remote server. It could also be because of the commands in your "C:\hg\Update.ps1" script. Some debugging tips that I recommend:

1. Remove your Invoke-Command line and see if the program executes.
2. Replace the ScriptBlock command with a simpler command, like Get-Culture, command and see if it still hangs.
3. Replace your C:\hg\Update.ps1 with another script that performs a simple command.
4. If all of those options still hang, try using the simple script method, but with different args.

And of course as with all PowerShell remoting issues, you should check that your firewall and network settings on the remote server allow connections from Jenkins over ports 5985 and 5986.

---
Tags: jenkins, deployment, powershell
---


# Question
Title: Github commit message when deploying to aws beanstalk in Jenkins
As part of our CI setup I am using AWS Beanstalk deployment plugin to deploy my application through Jenkins.

The issue is I am getting git commit hash in beanstalk application versions description. I want to have the commit message instead. I was able to output the message in Jenkins console output using the execute shell step. but for some reason I can't seem to transfer it to beanstalk. If I'm using eb deploy the description contains the commit message. I am getting a blank description with `${ENV,var="MESSAGE"}` in beanstalk.

Would appreciate any help on how to have the commit message in beanstalk application versions.

# Answer
This gives a short commit message

```
git show -s $GIT_COMMIT --format="format:%s"

```

I'd probably put this in the "Excute shell" command box

Hope this helps!
> 1 votes

---
Tags: jenkins, amazon-web-services, jenkins-plugins, github
---


# Question
Title: Remote starting/stopping Tomcat from Jenkins
I like to stop Tomcat before redeployment, manually clean out the old wars and all the logs, then restart the server. To do that from `Jenkins`, I have been using the `SSH plugin` and call `sudo service tomcat stop/start` and it works... sometimes. Other times the stop works but it fails to restart but the job still erroneously succeeds.

Is there an alternative to restarting Tomcat to SSH, especially when invoked from Jenkins. E.g. can I set up something like `xinetd` to make these admin functins somewhat of web services. Automating SSH connection from Jenkins is sort of a crapchute.

# Answer
> 2 votes
If you install the manager webapp with a default set up of tomcat then it is possible to use this to install and restart applications, see

https://tomcat.apache.org/tomcat-7.0-doc/manager-howto.html#Reload\_An\_Existing\_Application

I feel that you are asking the wrong question however. The problem is not how to restart Tomcat but how to be sure that it has restarted

Set a host, tomcat\_port and check path then use shell code like this

```
while [ "$response_code" != "200" ]
do
        echo "Checking whether tomcat (${host}:${tomcat_port}/$check) is alive..."
        response_code=`curl -sL -w "%{http_code}" "http://${host}:${tomcat_port}/${check}" -o /dev/null`
        echo "Received response code $response_code"
        if [ "$response_code" != "200" ]
        then
                echo "Waiting for a bit..."
                sleep 15s
        fi;
done;
echo "Tomcat is alive!"

```

This will wait until the Tomcat comes back - or the containing Jenkins job will time out eventually

---
Tags: jenkins, tomcat
---


# Question
Title: Central repository hosting various binaries/applications for different platforms/architectures
I would like to improve the automation and performance when managing VMs using Ansible. One of my use cases is to install custom software on a variety of operating systems i.e Red-Hat, Ubuntu, Centos. This software is distributed using single-file installers. For instance, installing a mySQL database on Red-Hat and Centos.

I recently learned about "repository managers" (or something similar). I don't know exactly where to look/learn more about what are the tools out there. I also don't know what the official terminology is.

This repository is ideally straightforward to use i.e, I just put a binary there, classify it with some tags\[?\] maybe like 64-bit, version 3.3, Centos.

Currently, I have this. But I don't really like hosting the binary on an NFS server

```
name: Get mySQL binary
command: "find {{ nfs }} -name \"*mysql*\""
register: mySQL_location

name: Install mySQL
command: "{{mySQL_location.stdout}} --option 1 --option 2
become: yes

```

A typical use case would look like this

* I want to install `Software A` on Centos
* I use Ansible to get the appropriate OS specific version of `Software A` from a "central repository"
* I use an Ansible module and specify version, OS, etc, and that downloads that binary (or binaries) to the VM. This should be fast since it will be over the local network, even for binaries of 100 MB+. Afterwards, Ansible installs it.
* The Ansible role would be configurable to affect the installation settings, etc.
* Next, I want to reuse the same role to easily do the same for, let's say Red-Hat

My goal is to have the Ansible playbooks be easily maintainable when a new version comes out, without, having to change anything in a playbook, just swap/add a new binary in the repository for a new version of `Software A`

I understand that the YUM package manager does this in a way, however, I am wondering what other options (if any) are out there.

# Answer
Your scenario is a perfect use case for a binary repository manager. A BRM's primary feature is to act as a central storage system for your binaries, but there are many other features including:

1. Versioning
2. Tagging
3. Permission management
4. Logging/auditing

Using a BRM is more lightweight than a typical source code repository system such as Git or SVN. Those systems will have much more metadata than is necessary for binary files

I have personally used Artifactory by JFrog, but I have also heard many good things about Nexus.
> 2 votes

# Answer
Another option if you are packaging your binaries (eg, as .deb or .rpm files) is to use a system like Foreman/Katello and the Anisble plugin. Katello was originally developed as a plugin for Foreman, and then Merged into the Foreman project. It is backed by Red Hat and re-branded as Red Hat Satellite 6 if you wish to use their licensed and supported version. While the whole system collectively is for management of the Lifecycle of machines (both physical and virtual), it can also manage Docker containers too.

But for package management, Katello will allow you to import your packages after they are built. You are also able to create environments and content views that allow you to version and control which servers get what when. Essentially, this gives you all of the benefits of a WSUS server for managing the release of your updates.

As an added bonus, Foreman can also create, destroy and build VMs for you with your cloud provider, or in your virtualization stack (if you do this in-house) using kickstart scripts.
> 0 votes

---
Tags: ansible
---


# Question
Title: Jenkins managing a pool of servers across restarts
I have a pool of servers to test a provisioning process my team owns. One server is needed per branch or PR. The process requires reboots. Once the process is run, we run some tests. A Jenkins pipeline automates the provisioning process and test.

So my problem is how to manage the pool of servers so that

1. Any available pool server can be selected for a build
2. A server claimed for a build is unavailable for other builds
3. Upon successful build, the server is made available again
4. Failed builds are not made available so they retain the failed state for investigation (with manual return to the pool after)

I can't just use the executor on the slave node because we reboot the server and that would kill the job.

My current solution is:

1. Pool servers are setup as Jenkins slaves with one executor each and a common label
2. A pipeline stage is assigned to the common label, choosing one of the slaves with a free executor--this stage moves `slave.jar` to protect against Jenkins restarting the assigned slave (which it annoyingly will do)
3. The next pipeline stage uses `node.getComputer().disconnect()` to disable the slave--it seems this must run on master so must be in a different stage
4. A `post { success { ... } }` uses ssh to move `slave.jar` back into place on the server and `node.getComputer().connect(true)` to allow it to be picked again

This mostly works and is kinda cool, but it adds a ton of noise to the pipeline and worse, suffers from a race condition between steps 2 & 3. When jobs queue up for the pool, a new one can be assigned to the same node just before a prior jobs disconnects the slave. When this happens, things break badly.

Does anyone else use Jenkins to manage a pool of servers where you need to support reboots in the pool and how do you do it?

Any ideas for ways to manage the pool outside Jenkins, but somehow get the assigned node into Jenkins for the test run? Plugins I've missed, etc.

I've examined the few results google returns for the slave disconnect problem and the most promising one was https://groups.google.com/forum/#!topic/jenkinsci-dev/ch2lQZvZdkw -- the suggestion there failed for me because `OfflineCause.UserCause` class could not be resolved in my pipeline. I only know enough Java and Groovy to be dangerous!

# Answer
> 4 votes
I found the missing piece in the Lockable resource plugin.

```
lock(label: 'server-pool', quantity: 1) {
   ...
}

```

Now the servers to be used for testing don't need to be managed as slaves at all and that greatly simplifies things.

# Answer
> 3 votes
To me, but correct me if I am wrong, the Jenkins servers should not be rebooted, but they should create VMs and reboot them. When I read the question I get the impression the slaves are rebooted, i.e. `I can't just use the executor on the slave node because we reboot the server and that would kill the job.` To me it seems that the jenkins slave should create a VM, run the tests, reboot the system, run some additional tests and finally remove the VMs.

From a production perspective, but again correct me if I am wrong the team is not using Jenkins to provision the systems and subsequently reboot the nodes? It could be possible that they use Jenkins to start the provision step, but they will not reboot the Jenkins systems right?

I would recommend to start with the Production perspective, e.g. How does the team provision the systems at the moment? E.g. Chef? If that is the case, how to they run that? If they would run chef apply then this could be run by a Jenkins slave, but I would not reboot the jenkins slave but automate the process of creating a VM, provision it, reboot it and kill it. The latter is only applicable for testing purposes.

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: Fastlane Screenshots with Authentication
I'm currently using Fastlane to capture screenshots of my application to be used in both the iOS and Android app store. My problem is that Fastlane is only able to capture the loading screen and the username/password authentication screen. Is there a way to pass credentials (possibly test creds with limited access) into Fastlane to allow it to capture other screenshots of my application? If not, what other methods could I use?

# Answer
Please check in Fastlane docs Launch Arguments under Advanced Snapshot section https://docs.fastlane.tools/getting-started/ios/screenshots/ I think it can help

> Launch Arguments
>
> You can provide additional arguments to your app on launch. These strings will be available in your app (eg. not in the testing target) through `NSProcessInfo.processInfo().arguments`. Alternatively, use user-default syntax (`-key value`) and they will be available as key-value pairs in `NSUserDefaults.standardUserDefaults()`.
>
> ```
> launch_arguments([
>   "-firstName Felix -lastName Krause"
> ])
>
> name.text = NSUserDefaults.standardUserDefaults().stringForKey("firstName")
> // name.text = "Felix"
>
> ```
>
> *snapshot* includes `-FASTLANE_SNAPSHOT YES`, which will set a temporary user default for the key `FASTLANE_SNAPSHOT`, you may use this to detect when the app is run by *snapshot*.
>
> ```
> if NSUserDefaults.standardUserDefaults().boolForKey("FASTLANE_SNAPSHOT")
>
> ```
>
> { // runtime check that we are in snapshot mode }
>
> Specify multiple argument strings and snapshot will generate screenshots for each combination of arguments, devices, and languages. This is useful for comparing the same screenshots with different feature flags, dynamic text sizes, and different data sets.
>
> ```
> # Snapfile for A/B Test Comparison
> launch_arguments([
>   "-secretFeatureEnabled YES",
>   "-secretFeatureEnabled NO"
> ])
>
> ```
> 1 votes

---
Tags: ios
---


# Question
Title: Change Ansible module based on variable or group
There are many Ansible Network modules (for example: ios\_user, asa\_user, nxos\_user) which share the same (or similar enough) interfaces. Currently, my playbook has to conditionally execute tasks based on the host's group or variables. This leads to a lot of duplicated code as the only real difference between the tasks is the module name.

**Is there a way to select the module to be run based on group or variable values** without adding extra tasks, roles, or playbooks?

*For example, this is how the playbook currently looks:*

```
- hosts: ios
  tasks:
    ios_user:
      provider: "{{ cli }}"
      ...
- hosts: asa
  tasks:
    asa_user:
      provider: "{{ cli }}"
      ...
- hosts: nxos
  tasks:
    nxos_user:
      provider: "{{ cli }}"
      ...

```

*I'd love to have something like this:*

```
- hosts: all
  tasks:
    {{ module_name }}:
      provider: "{{ cli }}"
      ...

```

I swear I saw some feature like this in passing, but I'm having a hard time finding it again. Along with version 2.5 making netconf\_cli a first-class connection type, this would solve a large number of problems and inconveniences in our org.

# Answer
Elaborating on @030's answer, here's how you can use variables for modules:

1. You need to group the target hosts in groups, for the example I'm using `server1.example.com` as member of `hosts1` group and `server2.example.com` as member of `hosts2` group.
2. You create group variables:

* group\_vars/hosts1
    * group\_vars/hosts2

Both files contain a variable named `module` with different value, in the test I use `stat`.

3. Your playbook will look like this:

`test.yml`

```
---
 - hosts: all
   user: me
   tasks:
     - name: test module variables
       action: "{{ module }} path=/home/me/myfile"

```

The result of the play:

```
[me@myserver]$ ansible-playbook test.yml -i test

PLAY [all] ********************************************************************************

TASK [Gathering Facts] ********************************************************************
ok: [server1.example.com]
ok: [server2.example.com]

TASK [test module variables] **************************************************************
ok: [server1.example.com]
ok: [server2.example.com]

PLAY RECAP ********************************************************************************
server1.example.com        : ok=2    changed=0    unreachable=0    failed=0
server2.example.com        : ok=2    changed=0    unreachable=0    failed=0

```
> 4 votes

# Answer
It is possible to declare group and host vars http://docs.ansible.com/ansible/latest/playbooks\_best\_practices.html

> ```
> group_vars/
>    group1                 # here we assign variables to particular groups
>    group2                 # ""
> host_vars/
>    hostname1              # if systems need specific variables, put them here
>    hostname2              # ""
>
> ```
> 2 votes

---
Tags: ansible
---


# Question
Title: What git command could be used to check when a remote branch was created?
Git is used at work. The number of branches is growing and growing. The aim is to remove branches that are older than X weeks.

**Attempt 1**

Running `git branch -h` will probably indicate what to run to check when a branch has been created.

```
user@localhost $ git branch -h
usage: git branch [<options>] [-r | -a] [--merged | --no-merged]
   or: git branch [<options>] [-l] [-f] <branch-name> [<start-point>]
   or: git branch [<options>] [-r] (-d | -D) <branch-name>...
   or: git branch [<options>] (-m | -M) [<old-branch>] <new-branch>
   or: git branch [<options>] [-r | -a] [--points-at]

Generic options
    -v, --verbose         show hash and subject, give twice for upstream branch
    -q, --quiet           suppress informational messages
    -t, --track           set up tracking mode (see git-pull(1))
    --set-upstream        change upstream info
    -u, --set-upstream-to <upstream>
                          change the upstream info
    --unset-upstream      Unset the upstream info
    --color[=<when>]      use colored output
    -r, --remotes         act on remote-tracking branches
    --contains <commit>   print only branches that contain the commit
    --abbrev[=<n>]        use <n> digits to display SHA-1s

Specific git-branch actions:
    -a, --all             list both remote-tracking and local branches
    -d, --delete          delete fully merged branch
    -D                    delete branch (even if not merged)
    -m, --move            move/rename a branch and its reflog
    -M                    move/rename a branch, even if target exists
    --list                list branch names
    -l, --create-reflog   create the branch's reflog
    --edit-description    edit the description for the branch
    -f, --force           force creation, move/rename, deletion
    --merged <commit>     print only branches that are merged
    --no-merged <commit>  print only branches that are not merged
    --column[=<style>]    list branches in columns
    --sort <key>          field name to sort on
    --points-at <object>  print only branches of the object

```

**Attempt 2**

This post https://stackoverflow.com/questions/3184555/cleaning-up-old-remote-git-branches was found, but this is not an option as it removes branches without checking the age.

# Answer
> 7 votes
*Note: this answer shows how to find branches that have been \*updated\* a long time ago, not branches that have been \*created\* (i.e., spliced off of some parent) a long time ago. I believe this is what the OP actually wanted (as opposed to the title he chose for the question), as this would lead to good candidates for deletion.*

You have to recall that a branch in git is nothing "physical". It is, by definition, and literally, *only* a simple text file in `.git/refs` which only has its file name (which is the branch name) and the hash of the tip/head commit. It has no date information or *anything* else.

So `git branch` will not help you here.

The object type that has dates associated with it are commits, in git. So you have to list commits. This is done with `git log`.

So a working command would be:

```
git log --remotes --before 2018-01-01 --no-walk --decorate

```

You can read up the details on https://www.git-scm.com/docs/git-log , but in short, it means

* `--remotes`: start with any and all remote branches (substitute `--branches` for local branches)
* `--before ...`: only list commits older than the date given
* `--no-walk`: only list the very first commit for each branch, don't walk back the history
* `--decorate`: display all tag and branch names associated, just in case

This will give you all branch heads that are older than your given date.

---
Tags: git
---


title: How to track unused resources in AWS?

text:
# Question
Title: How to track unused resources in AWS?
I have been using AWS for a while now. I always have the difficulty tracking AWS resources and how they are interconnected. Obviously, I am using Terraform but still, there is always ad-hoc operations that cut down my visibility.

Since I have been charged multiple times for resources/services that are present but not used by me.

Unused services include resources that are not pointing to other services but present in the AWS environment.

Tools suggestions are also welcome.

# Answer
There are a few ways to approach deleting unused resources, perhaps tagging items that are part of some trusted or known group, as some have suggested here and then manually monitoring for resources that isn’t tagged appropriately.

You could then extend this to a python script that you run every so often to ID and delete these resources.

However, there are other (potentially less intensive) options.

1) AWS already provide a way for us to monitor our resources, identify unused ones and terminate them with CloudWatch alarms. You can start by configuring some alarming based on some set of checks that you might already be doing manually. Once you are comfortable that the alarming is accurate, you can add a termination step to the chain.

2) Alternatively, our friends at Netflix made this chill tool called JanitorMonkey to monitor and delete unused Instances, EBS Volumes, EBS Snapshots and Autoscaling groups. It is part of the original so-called “Simian Army” open source pack from Netflix and written in Java. Even thought it’s a few years old now, this thing is highly configurable and well worth a look.

Both options here allow you to configure warnings and business rules that fit your needs. Additionally, using CloudWatch might also be a good opportunity to learn more about AWS logs and anomaly identification.
> 7 votes

# Answer
If you wish to do it purely with AWS provided tools, try this:

1. figure out a tagging scheme that works for you
2. tag all the resources you provision, make sure nothing is left untagged
3. for stuff you provision outside of the terraform and are not always tagged correctly by your tagging scheme, use AWS Config service to enforce the tagging scheme

After you have all of your resources tagged, use AWS Budgets to monitor and tracked down changes in you spend and get alerts when it crosses the budget threshold you define.

Also, create and periodically review some reports within the AWS Costs Explorer to find daily changes in region spend / service spend / *etc*.

Using a strict and meaningful tagging scheme helps maintain a healthy account with less money going to waste.
> 3 votes

# Answer
I created such a tool, it is called **AWS Inventory** and eventually, it will cover all available resources in your account. While it is still a work in progress, should be very easy to add the items you might be missing just by adding the API name to a list.

The tool is just a single HTML file that uses `aws-sdk.js` to query all the different `list...` and `describe...` APIs of CORS-enabled services.

The project is MIT license and open source at https://github.com/devops-israel/aws-inventory.
> 3 votes

# Answer
That would be one way of doing, using just the AWS console.

I have been using TotalCloud for many other things but this is very easily possible on the platform. It seems to be free for now. I have been using it with two large AWS accounts for 3 months now.

It shows everything in a single view with visual topological view which makes it fairly easy to understand the relationships between resources/services. The grouping can definitely help you find the resources as needed.
> 2 votes

---
Tags: amazon-web-services, amazon-cloudwatch
---


# Question
Title: From a security perspective, to which location should one copy an app inside a docker image?
I use docker to deploy a lot of apps since a couple of years. Today I ask myself, should I copy my app to a different location in the docker image? I used to copy the file to the root, e.g. `COPY . /AppName`, but I wonder whether this is secure.

In order to find an answer to this question, this tutorial was read, but it does not clarify where to copy a app inside a docker image and why.

# Answer
I don't see a point.

An external attacker (who has access to the host) can easily find where the files are - he only has to look at the last file system layer and will have them presented right there for him.

An internal attacker (who gains access to the container through a misconfigured part of the app or whatever) will also rather easily find them by looking through the process table at `/proc` and noting the CWDs.

Attacking the image might be easier with a fixed location, as the attacker only has to add another FS layer, overwriting that directory with his own application. But then, if you did not secure your image store, it does not matter anyways what else you do.

So... I'd say the convenience of a common location is good, and little to no security would be gained; and if, then only security by obscurity, which does not count.
> 6 votes

---
Tags: docker, security
---


# Question
Title: What is a difference between a Walking Skeleton and an MVP?
I've recently stumbled upon the "Walking skeleton" term and got a bit confused about the terminology.

Is it the same or a variation of a Minimum Viable Product, or how does it relate to MVP in general?

# Answer
The walking skeleton reflects the rough structure/shape of the product, but it may be entirely stubbed out (initially), meaning it may not be actually functional. The skeleton cannot move without the muscles.

The MVP however must meet a certain minimum functionality level to be considered viable.

As development progresses the walking skeleton typically morphs into an MVP and eventually into a final product.
> 4 votes

# Answer
Dan is close. I think the easy way to describe the difference is that the two have different customers. The customer of the walking skeleton is the dev (and ops) team, in that it implements the simplest thing from each of the elements of the architecture and strings them together in a working way. For example, it has a front end, it has a backend, it has a database, and it can move null betwween the three layers of the arhitecture. It can be deployed in production; but from a user perspective it doesn't really *do* anything.

In contrast, the customer of the MVP is, well, the end user of the product. It's the simplest slice of functionality that can satisfy the core payoff/value for the user. The MVP is then the smallest amount of value that can be delivered that demonstrates that the application is worthwhile. Often this is completely the opposite of a walking skeleton. A common approach is a 'concierge' MVP, where the steps are manually performed behind say a simple landing page or google form, with the results sent to the user over email.
> 4 votes

# Answer
Based on the following two comments to the referenced answer in the question:

@Adrian:

> I guess it's the delivery pipeline equivalent of a minimum viable product.

@DaveSwersky

> This does sound similar to minimum viable product, but at a more granular level- "minimum viable component" perhaps. Returning 200 from a service just to get it "running" sounds like a stub to me.

it looks like that a `Walking skeleton` is not identical, but a smaller `MVP`.
> 2 votes

---
Tags: terminology, practices, patterns
---


title: Docker In Kubernetes Deployment

text:
# Question
Title: Docker In Kubernetes Deployment
I am using a third party library that creates sibling docker containers via:

```
docker run -d /var/run/docker.sock:/var/run/docker.sock ...

```

I am trying to create a Kubernetes deployment out of the above container, but currently getting:

> Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

This is expected because I am not declaring `/var/run/docker.sock` as a volume in the deployment yaml.

The problem is I don't know how to do this. Is it possible to mount `/var/run/docker.sock` as a volume in a deployment yaml?

If not, what is the best approach to run docker sibling-containers from within a Kubernetes deployment/pod?

# Answer
> 22 votes
Unverified as it sounds brittle to me to start a container outside of k8s supervision, but you should be able to mount `/var/run/docker.sock` with a hostPath volume.

Example variation from the documentation:

```
apiVersion: v1
kind: Pod
metadata:
  name: test-pd
spec:
  containers:
  - image: gcr.io/google_containers/test-webserver
    name: test-container
    volumeMounts:
    - mountPath: /var/run/docker.sock
      name: docker-sock-volume
  volumes:
  - name: docker-sock-volume
    hostPath:
      # location on host
      path: /var/run/docker.sock
      # this field is optional
      type: File

```

I think a simple mount should be enough to allow communication from docker client within the container to docker daemon on host but in case you get a write permission error it means you need to run your container as privileged container using a securityContext object like such (just an extract from above to show the addition, values taken from the documentation):

```
spec:
  containers:
  - image: gcr.io/google_containers/test-webserver
    securityContext:
      privileged: true
    name: test-container

```

# Answer
> 3 votes
Although this is a working solution (I use it myself), there some drawbacks for running Docker in a Kubernetes pod by mounting `/var/run/docker.sock`

Mostly the fact you are working with Docker containers outside the control of Kubernetes.

Another suggested solution I found is using a **side-car** container in your pod. See A Case for Docker-in-Docker on Kubernetes. There are two parts to it where the proposed solution is in part 2.

I hope this helps.

---
Tags: kubernetes, docker-in-docker
---


# Question
Title: Monitoring immutable infrastructure virtual machines
Currently I am using the HashiCorp stack (Packer, Terraform) and Ansible, to generate Immutable Infrastructure and Infrastructure as Code.

This builds a Virtual Machine from a base image, provisions it and applies it to some VSphere servers.

One of the issues I'm having is knowing when a VM is down or unresponsive.

What would be a good way of monitoring VMs?

I've looked a bit at Consul and Prometheus and am thinking that that is the way forward?

# Answer
> 5 votes
> I've looked a bit at Consul and Prometheus and am thinking that that is the way forward?

In short, yes.

Consul is designed exactly for this - it will run health checks against all of your services that you define and can detect when things go down. However, what it does not do it alert when things do go down, it is more designed for automatically taking things out of circulation when they are unhealthy. But it does give you a single point you can query for the health of your infrastructure which you can then alert on manually somehow or combine it with consul-alerts to do this for you.

Prometheus is geared towards metrics and can monitor the health of your applications in ways that consul is not really well suited. It has its only able to alert when defined events occur and you can tune it to monitor exactly what you require.

You might also want to consider external checks like what services such as panopta offer (this is just an example, there are many alternatives out there) to check public endpoints to make sure they are still working from an external perspective.

---
Tags: monitoring, immutable-servers
---


title: Service check with api in Icinga2

text:
# Question
Title: Service check with api in Icinga2
I am trying to do add service of client using icinga2 api. I have followed https://github.com/saurabh-hirani/icinga2-api-examples. Here is what I have done so far.

First I created zone and endpoint in master as follows:

**Endpoint**

```
Object Endpoint "client.com" {
}

```

**Zone**

```
Object Zone "client.com" {
    endpoints = [ "client.com" ]
    parent = "master.com"
}

```

Restarted icinga2

Then in client server

```
#!/bin/bash -e

. icinga2_vars

echo "CREATING HOSTGROUP"

curl -s -u $ICINGA2_API_USER:$ICINGA2_API_PASSWORD \
    -H 'Accept: application/json' -X PUT \
    -k "https://$ICINGA2_HOST:$ICINGA2_API_PORT/v1/objects/hostgroups/client_hostgroup" \
    -d '{ "attrs": {
            "display_name": "client host group"
} }' | python -m json.tool

echo "CREATING HOST"

curl -s -u $ICINGA2_API_USER:$ICINGA2_API_PASSWORD  \
    -H 'Accept: application/json' -X PUT \
    -k "https://$ICINGA2_HOST:$ICINGA2_API_PORT/v1/objects/hosts/client.com" \
    -d '{ "templates": [ "generic-host" ],
        "attrs": {
        "address": "127.0.0.1",
        "check_command": "hostalive",
        "groups": ["client_hostgroup"],
        "vars.notification['mail']": {
            "groups": ["client_hostgroup_users"]
            }
} }' | python -m json.tool

echo "CREATING SERVICE"

curl -s -u $ICINGA2_API_USER:$ICINGA2_API_PASSWORD  \
    -H 'Accept: application/json' -X PUT \
    -k "https://$ICINGA2_HOST:$ICINGA2_API_PORT/v1/objects/services/client.com!load" \
    -d '{
        "templates": [ "generic-service" ],
        "attrs": {
            "display_name": "check load",
            "check_command" : "load",
            "command_endpoint": "client.com",
            "host_name": "client.com"
        }
}' | python -m json.tool

```

It is working but checking service of master itself. It should check service of client server.

# Answer
> 1 votes
You should add the correct zone attribute for the host and the service, so they will propagate to the client node.

Add `"zone": "client.com"` to the api requests.

---
Tags: monitoring
---


# Question
Title: One Checkout When Running Parallel Stages In Jenkins
## Introduction

We have a multibranch pipeline project. The build configuration is defined by a jenkinsfile. The system consists of one master and a couple of agents.

The stages are run in parallel, and for each one we do a checkout of the repository.

See attached file below.

## The Problem

is that the size of the workspace accumulates greatly as the amount of branches increases.

## The Solution #1

Periodic cleanup of the workspace. A custom script will be written. Something like this one:

```
// Check if a slave has < 10 GB of free space, wipe out workspaces if it does

import hudson.model.*;
import hudson.util.*;
import jenkins.model.*;
import hudson.FilePath.FileCallable;
import hudson.slaves.OfflineCause;
import hudson.node_monitors.*;

for (node in Jenkins.instance.nodes) {
    computer = node.toComputer()
    if (computer.getChannel() == null) continue

    rootPath = node.getRootPath()
    size = DiskSpaceMonitor.DESCRIPTOR.get(computer).size
    roundedSize = size / (1024 * 1024 * 1024) as int

    println("node: " + node.getDisplayName() + ", free space: " + roundedSize + "GB")
    if (roundedSize < 10) {
        computer.setTemporarilyOffline(true, new hudson.slaves.OfflineCause.ByCLI("disk cleanup"))
        for (item in Jenkins.instance.items) {
            jobName = item.getFullDisplayName()

            if (item.isBuilding()) {
                println(".. job " + jobName + " is currently running, skipped")
                continue
            }

            println(".. wiping out workspaces of job " + jobName)

            workspacePath = node.getWorkspaceFor(item)
            if (workspacePath == null) {
                println(".... could not get workspace path")
                continue
            }

            println(".... workspace = " + workspacePath)

            customWorkspace = item.getCustomWorkspace()
            if (customWorkspace != null) {
                workspacePath = node.getRootPath().child(customWorkspace)
                println(".... custom workspace = " + workspacePath)
            }

            pathAsString = workspacePath.getRemote()
            if (workspacePath.exists()) {
                workspacePath.deleteRecursive()
                println(".... deleted from location " + pathAsString)
            } else {
                println(".... nothing to delete at " + pathAsString)
            }
        }

        computer.setTemporarilyOffline(false, null)
    }
}

## The Solution #2 ##
One checkout per job and node. Thus each parallel stage will execute one executable instance of the same repository. For me personaly this is the preferred method, although more complex than solution #1.

pseudocode:

    stage("Build") {
        try {
            parallel builds
        } catch (err) {
        }
    }

    def build(predef) {
        return {
            node() {
                checkout_repository(${env.NODE_NAME})
                Execute build
                ...
            }
        }
    }

    def checkout_repository(node){
        No checkout for this node
            Flag that we are checking out a repository on node (think ~semaphore)
            Cheking out ..
            Flag that repository exist on node
    }

```

## The Question

1. What are the cons in solution #2
2. Does it follow the jenkins way of doing it. Or this to much freestyle?

The job:

# Answer
> 1 votes
Your difficulty stems from encouraging potentially long-lived branches, which is deviating from the CI methodology - which is supposedly why you'd be using jenkins - a CI tool - in the first place.

Normally you shouldn't be running the tests for those branches in the Jenkins pipeline for the trunk - you want to measure the quality of the integrated work, not that of the work done in the silos.

I do understand the desire to run such tests before merging into trunk, so maybe it's OK to use jenkins to do it if you want to, but with separate jobs/pipelines, not tied into the trunk pipeline. And keep in mind that the results of such tests aren't necessarily conclusive: just because they pass over there doesn't mean the same tests will also pass in the trunk, you'll need *very* strict conditions to make that always true.

Personally I'd go for a fully automated patcing/merging solution based on pre-commit verifications which can enforce the strict conditions necessary to prevent post-merge trunk quality regressions (but I'm biased, I built such solutions). See for example How to ensure that git subtrees are kept up to date?

Anyways, back to the question. Since the code for the various pipeline segments that you seek to perform in parallel comes from different branches you won't be able to use **the same** workspace, you have to pull separate workspaces from the respective, different branches. Which makes the question kinda moot.

Final note: assuming the same workspace *could* be used in multiple builds, your #2 approach requires a pristine build structure. It's not uncommon for less-than-perfect builds running in parallel in the same workspace to interfere with each-other: race conditions creating/deleting directories, overwriting artifacts, too high peaks of resource usage, etc - typically hard to repro intermittent failures. These could easily happen if, for example, the builds would be for the same product but for different CPU architectures, or builds including code shared across multiple products. Just be aware of the possibility.

---
Tags: jenkins, jenkinsfile
---


# Question
Title: Docker Compose V3 app container can't access couchdb container
I am trying to connect to couchdb through the container but my app is not seeing it.

However if I use the external address `123.456.789.123:12345` (`server_ip:external_port`) it works but I need a connection through network with the internal port `5986`.

**Docker-Compose.yml**

```
version: '3'
    services:
      couchdb:
        image: couchdb:latest
        restart: always
        volumes:
          - couchdb-data-volume:/usr/local/var/lib/couchdb
        env_file: .env.docker
        ports:
          - "12345:5986"
        networks:
          app-net:
            aliases:
              - couchdb
      app:
        build: .
        image: app
        restart: always
        depends_on:
          - couchdb
        env_file: .env.docker
    networks:
      app-net:
        driver: bridge
    volumes:
      couchdb-data-volume:

```

**ENV FILE**

```
NODE_ENV=production
COUCHDB_SERVER=http://user:pass@couchdb:5986
COUCHDB_USER=user
COUCHDB_PASSWORD=pass

```

I Also tried links instead of networks but the same issue.

```
 links:
 - couchdb

```

Any Ideas ?

# Answer
Your mistake is that you are did not register the *app* service on the *app-net* network. To allow containers to connect with one another, they have to share a network and then use the image name as the domain name. Docker provides a network internal DNS service.

```
version: '3'
services:
  couchdb:
    image: couchdb:latest
    restart: always
    volumes:
      - couchdb-data-volume:/usr/local/var/lib/couchdb
    env_file: .env.docker
    ports:
      - "12345:5986"
    networks:
      app-net:

  app:
    build: .
    image: app
    container_name: app
    restart: always
    depends_on:
      - couchdb
    env_file: .env.docker
    networks:
      - app-net

volumes:
  couchdb-data-volume:

networks:
  app-net:

```

The URL from the *app* to the *couchdb* service would be `http://couchdb:5986`. To test this, open bash inside the *app* service with `docker exec -it app`. This depends on the *container\_name* setting.
> 1 votes

---
Tags: docker, containers, docker-compose
---


# Question
Title: How to determine actual cost of specific spot instance
Is there a way to know how much a specific spot instance is actually costing? My only idea is to give it a specific tag and wait to see it show up in the billing report. But what I really want is to to know in "real time" how much each spot instance is costing.

# Answer
From Amazon EC2 Spot Instances Pricing:

> The following table displays the Spot price for each region and instance type (updated every 5 minutes).

So one (crude) option would be to parse that page to get the price matching your particular instance type and region.

I don't know if the equivalent info is available via the AWS Price List API:

> Offer files do not include information about expiring free tier offers or Amazon EC2 Spot Instances.

* the spot instace pricing isn't explicitly declared as unsupported in the Query API, so there *might* be a chance for another option here, that'd be better than parsing pages. But I didn't try - I'm not an AWS user yet.
> 2 votes

---
Tags: amazon-web-services
---


# Question
Title: Six repositories as a single deployment Docker+CircleCI
I need some help and be pointed in the correct direction.

I have 6 git repositories that together combine a bigger application (3 Node.js API, 2 React web apps and one repository for Ansible and templates for config files for the other repos).

Currently we are building it only with Ansible and CircleCI but the current build process is not that great and I think of moving to Docker, docker-compose and CircleCI 2.0.

The thing is: **how should I dockerize it?**

All the databases are remote. How should I approach nginx?

I thought about writing Dockerfiles for each application and then using Ansible to connect to host, install Docker, pull sources and start building containers from Dockerfiles.

Nginx should be configured by Ansible on host or can it be made into a Docker container somehow?

# Answer
> 1 votes
With docker compose (or docker swarm if you want to distribute the containers) create a container for each Git repository. This should map well to docker’s philosophy of each container being used for a **single purpose**.

As far as examples go, I can recommend you have a look at veleda (shameless self-promotion) which combines 3 databases, a flask app, a ruby app (Grafana), Jekyll and nginx. It also includes unit tests that are integrated into Travis-ci and should be transferable to circle-ci.

Regarding the nginx aspect, 030 already mentioned that there are finished images to make your life easier. Even ones that include automated Let’s Encrypt services.

Ansible can be integrated well into the mix. If you used it to install dependencies and network configurations, this is now handled by docker. For SSH-based tasks, ansible remains useful. Check out this comparison of tools and their scope’s of responsibility.

---
Tags: docker, ansible, docker-compose, circleci-2.0
---


title: Why Run Concourse CI in Kubernetes

text:
# Question
Title: Why Run Concourse CI in Kubernetes
I've just started exploring Kubernetes and Concourse CI. Learning still, so I'm still understanding all the concepts. Forgive me if this question sounds naive.

Why would we deploy Concourse CI in Kubernetes. Is it just for the ease of deployment? I mean concourse has its own way for scaling workers which itself if I've understood is through containers. So why run something that can already scale and handle containers(I mean concourse) inside Kubernetes?

# Answer
In our case yes, it is just ease of deployment/operations. We have existing skills with k8s and that's what we're most comfortable operating.

I guess you're right that its somewhat odd to run Concourse's scheduling inside of k8s scheduling, and it would be great to see Concourse being able to leverage Kubernetes directly at some point.
> 2 votes

---
Tags: continuous-integration, kubernetes, containers, configuration
---


# Question
Title: Update production server (apache, wordpress, mysql) on docker containers, from staging server
For our web servers running on Docker containers (Apache, mysql, wordpress, etc) we have a production and a staging virtual machines.

DEVs and content team work on staging and then I tar ball the files from the wordpress container and dump the mysql DB and scp them to production server. Once in production I untar the files to the wordpress container and "restore" the DB. (in fact we use a server in the middle, but I'm skipping that here to keep it short)

I was thinking on an internal docker registry, but that's only for images... so I will need to start/restart the images to container.

Or some kind of internal "GIT"? where I can push/pull versions?

what will be a better way to update production server?

# Answer
> 2 votes
My thoughts based on your described scenario are:

1. Your tarball strategy, as described, might *sound* safe but is prone to risk in my opinion. A better approach is to version-control the WP files you will modify, while keeping the core intact and within the Docker container. Here's a reference I have used in the past that talks about version-control strategies for WP. You would of course need to spend some time and understand which folders/files to version-control for your specific site.
2. The version-controlled files would be checked out on the *host machine* (which runs Docker) and be volume-mounted within your Docker container.
3. Everytime your changes are validated in `Stage` they get pushed to Git. Getting these changes into `Prod` would essentially be a `git pull`.
4. If you wish you might also use a tool like composer as part of your deployment workflow. A dependency manager should be part of any deployment worklow.

I would also recommend tying all this together with a tool like Ansible, i.e. use Ansible to

* update local git repo
* run the composer bits
* ensure docker container is running

Disclaimer: Although I've worked on DevOps automation for Wordpress sites, I am by no means an expert on Wordpress, so suggestions and clarifications welcome.

Hope this helps.

# Answer
> 1 votes
> Or some kind of internal "GIT"? where I can push/pull versions?

What might be of interest to you, without using immutable Docker packages, is Git LFS (Large File Storage). It is an effective approach to using Git for large binaries (your TARs). This has the following benefits:

1. Use the familiar Git interface
2. Versioning that allows easy rollbacks
3. Guards against data corruption

Installing the client is very straight forward although the server requires a few more steps. There are quite a few hosted options available, but self-hosted should be the most cost-effective variant.

# Answer
> 1 votes
I’ve been using a private project on Gitlab to push/pull. I’m using Docker Compose to launch the containers in development and production.

Although I am needing to customize the WordPress image now, so I’m using the built in private container registry at gitlab too and thinking of baking all of the updates into the image as well.

# Answer
> 0 votes
`Once in production I untar the files to the wordpress container and "restore" the DB`. One of the advantages of docker that these are immutable packages. If one untars packages in production then this is a contradiction. In my opinion one should pull docker images as is that are immutable. Imagine that a newer version of the docker image, e.g. 1.0.1 is broken then one should be able to revert to the last stable image, e.g. 1.0.0. If one would follow your suggested approach then this is not possible.

---
Tags: docker, continuous-delivery
---


# Question
Title: What are the reasons Docker should not be used for databases?
I'm having a discussion with a friend about use cases for Docker. One guy in the team wants to use Docker for everything - like a kind of universal unix process wrapper. The other thinks that Docker should only be used for stateless applications like Microservices and AWS Lambda style apps.

We've engineered proof of concepts for both. On our docker cluster we have a shared drive that gets mounted when the Docker host is mounted, and if a Database in a container is mounted, it simply mounts a volume to the shared drive.

My friend still sticks to his position, despite being shown the contrary evidence. (He also argues that Docker adds unnecessary risk by adding complexity to the stack.)

I'm trying to listen and understand his point of view, both in an act of empathy, but also to better reason with him. (We all get on quite well - so this is a mix of in-jest and serious discussion).

Kind of the question behind the question is: are databases cattle? This comment suggests that a good automated backup and retrieval strategy for your database is indistinguishable from a cattle server.

My question is: **What are the reasons Docker should not be used for databases?**

**EDIT:** People have asked me to clarify my terminology. I was assuming that the database application was in the container, and the storage was in the volume. What I meant was, the RDBMS is in the container, and the database storage is in the volume.

Some commentators have suggested that the docker volume drivers aren't going to work with database writes very well. (Or something to that effect). Could you please expand on that?

# Answer
When people talk about running a database in Docker, they do not mean to store the data in a container; they are talking about having a docker image with the DB software, and mounting the data as a volume (a bind volume, not a container volume).

Volumes are an essential part in Docker, and are not something that is flakey or just tacked on. Docker is *not* just made for stateless (micro)services.

Wish as I might, I cannot find a *technical* reason not to run a database in a Docker, so unfortunately I'll pick the other side of the argument and hence maybe not give you the answer you are looking for.

(I'm using Oracle as an example because I'm familiar with it, both bare metal and dockerized, and because it's quite a notorious beast for being just a bit non-trivial to operate if you go past default settings.)

* Packaging up the DB software itself in a container gives you the usual benefits - having the same version everywhere, avoiding dependency/shared library issues, being able to spin up the exact same DB on developer laptops or wherever you need it.
* It is a *snap* getting it to run anywhere; updating is trivial, and so on. All the Docker benefits apply. There is an Oracle image on Dockerhub which allows you to spin up a working DB in a minute or three (and for the others as well, of course).
* People did do performance tests and found no I/O differences between volumes and bare metal (https://www.percona.com/blog/2016/02/11/measuring-docker-io-overhead/, https://stackoverflow.com/questions/21889053/what-is-the-runtime-performance-cost-of-a-docker-container).
* Under the hood, it's not like Docker somehow intercepts all I/O, anyway. It just gets creative with standard Linux tools (bind mounts in this case, mangling of the internal kernel tables that make the Docker-fu possible at all).
* Obviously that does not mean that you can run two instances of the DB and just have them work on the same files, but nobody is implying that. Docker does not give you automatic simultaneous and magically race-free access to volumes, and did never pretend to do so. The rest of the benefits still apply. If your DB itself does not detect conflicts like this, you better supply a CMD script to the image which refuses spinning up a second container when the volume is already in use.
* You have to be a little more careful spinning up/shutting down the container (just as you would not simply switch off a bare metal DB server), but that should be quite manageable.

Now, depending on circumstances, there *may* be soft reasons not to do it:

* Oracle (the company), for example, might not support you if you run their RDBMS in a Docker container on production systems (in 2021, 3 years after writing this answer, it is unclear to me if that is still true). But maybe you are using dockerized Oracle RDBMS images only for your developers and the testing environment, where you would not need their support in any case, reserving it for a bare metal production server. (But don't forget to pay your licenses...).
* If the ops guys are unfamiliar with Docker, it might just be a bit easier to accidently kill everything, destroy your data files etc..
* If you have big dedicated metal DB machines already, with large amounts of very fast dedicated SAN storage, and running nothing else anyways, then there would just be no point in using Docker to containerize *those* as you will *never* just spin another server up when there are 100s of GB or even TB of data. After all, for production, a RDBMS like Oracle is very, very advanced in all the replication, data integrety, no-downtime failover, etc. aspects. Note that this argument just says "you do not *need* to containerize your RDBMS". It does not say "you *should not* do it" - maybe you want to do it because you wish to roll out database software upgrades through containers or for whatever other reason you could imagine.

So there you go. By all means *do* dockerize your DB, at the very least for your developers (who will be eternally thankful) and your testing environments. On the production, it will come down to taste, and *there* at least, I would also prefer the solution that sits best with the specialized DBA/Ops - if they have decades of experience working bare metal DB servers, then by all means trust them to continue so. But if you are a startup who has all IT in the cloud anyways, then a Docker container would just be one further piece of onion in the whole picture.
> 41 votes

# Answer
I wrote about this in depth but here's the summary:

* Preventing split brain (electing more than one master node) needs to be solved. Failure to do so can be catastrophic
* There are no production ready shared storage solutions to enable databases to be shutdown on one instance and brought up on another without losing all your data.
> 7 votes

# Answer
When you say that the data is mounted into a docker container, would it not be more correct to say that the "database" is mounted into the docker container? If your are persisting your data outside the container then you are doing the "correct" thing of not putting your database in a container.

Sure, go to town putting a DBMS in a container a letting it manage data that you store outside, personally I think that's just good design because it keeps a clean separation between logic and data. But once you put your data into a container them you're potentially playing with fire.

Although container storage drivers have come a long way, I'm personally not yet willing to dive in and leave my data tangled up in a container.
> 0 votes

---
Tags: containers, databases, cattle-vs-pet, risk
---


# Question
Title: Dockerized Flask: Connection reset by peer
Trying to connect to a dockerized Flask app fails with error 104, 'Connection reset by peer' using this minimal example:

app.py:

```
from flask import Flask
app = Flask(__name__)

@app.route("/")
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run()

```

Dockerfile:

```
FROM python:alpine
RUN pip install flask
COPY . /src/
EXPOSE 5000
ENTRYPOINT ["python", "/src/app.py"]

```

docker-compose.yml:

```
…
test:
    build: .
    ports:
        - 127.0.0.1:5000:5000

```

Flask app seems to be running as expected:

```
$ docker logs test
 * Running on http://127.0.0.1:5000/ (Press CTRL+C to quit)

```

Trying to connect from outside fails:

```
$ http http://127.0.0.1:5000/
http: error: ConnectionError: ('Connection aborted.', ConnectionResetError(104, 'Connection reset by peer')) while doing GET request to URL: http://127.0.0.1:5000/

```

Any ideas, why I can't get to see "Hello World!" here?

# Answer
> Trying to connect from outside fails

Are you actually connecting from *outside*? Flask is binding to localhost (127.0.0.1) and that will only be reachable from within the container. If you're on your local machine, you'll need flask to bind to all IP's:

```
from flask import Flask
app = Flask(__name__)

@app.route('/')
def hello():
    return "Hello World!"

if __name__ == "__main__":
    app.run(host='0.0.0.0')

```

That works for me successfully.
> 27 votes

---
Tags: docker, python
---


# Question
Title: How to get the latest stable version of Firefox?
How to get the latest version of firefox? This URL was found, but is there no API? I would like to get the latest version, e.g. `58.0.2` so this could be used for other purposes. Just `latest` is insufficient as I require the version in one of my ansible roles to lookup the checksum.

---

At the moment I am considering to parse https://www.mozilla.org/en-US/firefox/releases/

# Answer
You do not want to be parsing HTML in a programmatic way. Avoid it at all costs, if you can. It would be a nightmare to support on-going. There's actually a JSON output that Mozilla provides of all the versions:

https://product-details.mozilla.org/1.0/firefox\_versions.json

**Edit:** Here is a simple bash script example that will always return the latest Firefox version. Make sure you have `jq` & `curl` installed on your system:

```
curl -s https://product-details.mozilla.org/1.0/firefox_versions.json | jq -r .LATEST_FIREFOX_VERSION

```

This returns:

`58.0.2`
> 3 votes

# Answer
I pulled this from the Firefox Chef cookbook (in particular this .rb file). This an example of the request they use to get the latest version based on OS and language.

```
https://download.mozilla.org/?product=firefox-latest#&os=win#&lang=#en-US

```

It adheres to the following format:

```
https://download.mozilla.org/?product=firefox-{VERSION_NUMBER}#&os={OS_ABBREVIATION}#&lang=#{LANGUAGE_ABBREVIATION}

```

I searched around for official api documentation, but couldn't find anything. If I do, I will post a link to it in this answer.
> 3 votes

---
Tags: automation, api
---


# Question
Title: How to avoid output being truncated when running AWS SSM command?
I've got the following AWS CLI commands to run the PowerShell's `Invoke-WebRequest` command:

```
instanceId=i-0xyz # Change this.
cmdId=$(aws ssm send-command --instance-ids "$instanceId" --document-name "AWS-RunPowerShellScript" --query "Command.CommandId" --output text --parameters commands='"While ($i -le 10) {Invoke-WebRequest -UseBasicParsing -Uri http://example.com; $i += 1}"')
sleep 1
aws ssm list-command-invocations --command-id "$cmdId" --details --query "CommandInvocations[*].CommandPlugins[*].Output[]" --output text

```

However, the output is truncated:

```
...
StatusCode        : 200
StatusDescription : OK
Content           : <!doctype html>
                    <html>
                    <head>
                        <title>Example Domain</title>

                        <meta charset="utf-8" />
                        <meta http-equiv="Content-type" content="text/html;
                    charset=utf-8" />
                        <meta name="viewport" conten...
------------------Output truncated-------------------

```

How I can run `aws ssm list-command-invocations` command, so the output won't be truncated?

# Answer
According to this ssm tutorial, SSM send-command output is truncated at 2500 characters. To work around that, you must send your output to an S3 bucket using the `--output-s3-bucket-name` parameter.
> 5 votes

# Answer
Here is a helper Bash script which uses `aws ssm send-command` with `--output-s3-bucket-name` parameter to run the command and the result is stored in the S3 bucket, then displayed to the standard output.

```
#/usr/bin/env bash -xe
# Script to run PowerShell script on the Windows instance, then uploads the output to S3 bucket.
instanceId="$1"
bucketName="$2"
bucketDir="Output"
[ $# -le 2 ] && { echo "Usage: $0 instance_id bucket_name command"; exit 1; }
aws s3 ls ${bucketName} > /dev/null
cmdId=$(aws ssm send-command --instance-ids "$instanceId" --document-name "AWS-RunPowerShellScript" --query "Command.CommandId" --output text  --output-s3-bucket-name "$bucketName" --output-s3-key-prefix "$bucketDir" --parameters commands="'${@:3}'")
while [ "$(aws ssm list-command-invocations --command-id "$cmdId" --query "CommandInvocations[].Status" --output text)" == "InProgress" ]; do sleep 1; done
outputPath=$(aws ssm list-command-invocations --command-id "$cmdId" --details --query "CommandInvocations[].CommandPlugins[].OutputS3KeyPrefix" --output text)
aws s3 ls s3://${bucketName}/${outputPath}/stderr.txt && aws s3 cp --quiet s3://${bucketName}/${outputPath}/stderr.txt /dev/stderr
aws s3 cp --quiet s3://${bucketName}/${outputPath}/stdout.txt /dev/stdout

```

Example:

```
./run_ec2_ps_cmd_s3.sh i-0xyz my-bucket-name 'While ($i -le 10) {(Invoke-WebRequest -UseBasicParsing -Uri http://example.com).Content; $i++}'

```
> 4 votes

---
Tags: aws-cli
---


# Question
Title: What is the biggest Docker image on Docker Hub and how to find it?
The lowest boundary of a Docker image can be very small i.e. under 1 MB (busybox) or even zero bytes (scratch).

Now for research purporses, I need to reference the biggest Docker image on the Docker Hub.

How to find it?

# Answer
> 5 votes
If you're looking for a master list of all images and sizes, kind of a 'statistics' pages, Docker Hub doesn't have that. You'll need to use their HTTP API to:

1. get all the repositories, stepping through the paginated results. API reference: https://docs.docker.com/registry/spec/api/#listing-repositories
2. pull the Image Manifest for each repo. API reference: https://docs.docker.com/registry/spec/api/#pulling-an-image-manifest
3. Store those manifests in local database so you can sort them based on image size

You'll specifically need to pull the newest `Image Manifest Version 2, Schema 2`, which apparently not all repos support yet. This newer manifest has the image sizes in it: https://docs.docker.com/registry/spec/manifest-v2-2/

All in all, this is much faster than trying to actually download all the images from Docker Hub, but it will still take quite a long time and a lot of processing. The database for storing this all might get rather large and difficult to process, so I'm not sure how much you really want to pursue this route for a simple research project. That's up to you.

You may want to take @Vish's advice and simply go for obvious large images such as the Microsoft or Java ones.

---
Tags: docker, dockerhub
---


# Question
Title: How to avoid branchageddon with large organisations?
How do you avoid a branchageddon situation when working with large organisations?

We work with a number of large financial organisations whose approach is to not take updates to software, but instead only high/critical security patches and bespoke functionality. These organisations will only take patches and custom release in between major updates. Major updates can be years apart and carry high costs. This approach causes us (the software house) to have a branch of our code per major customer, which carries all the costs and inefficiencies of long term branching.

My questions to the community are:

* Have you experienced similar update acceptance approaches from your customers?
* What suggestions do you have to help work with this approach?
* What suggestions do you have to help change organisations approaches to taking software updates?

# Answer
As Michael mentioned, offer a standard solution based on release versions/numbers, with a reasonably long lifespan for your industry (maybe interleaved with one or more shorter lifespan intermediate versions, if it makes sense for your typical customers).

Give your customers the option to embark on this standard release track, maybe with a decent migration deadline in place.

If they insist on a full-custom branch support strategy just charge them accordingly, to properly cover all your extra costs for offering such full-custom support - it only makes business sense. Some customers will migrate to reduce their costs (which will help you reduce the number of custom branches), some won't.

Variable support billing, growing progressively with the age of the release versions from which the custom branches originate can also be an incentive for customers to migrate to newer branches faster, helping with faster closing of the older custom branches. This can help reduce the number of custom branches per customer - if you have customers that simultaneouly run multiple versions of your software.

Make sure you don't fall into the trap of doing full branch merges from/to any of the release branches (both standard and custom), all changes to them should be either individually developed or cherry-picked merged fixes.

Because each of these branches will gradually diverge from each-other, the number of hotfixes requiring customisation/individual development will grow exponentially (plain cherry-pick merging will fail). You need to take the development cost for these into account.

With no (significant) branch merges in the picture you can (and should, I cannot stress its importance enough) build fully automated CI/CD pipelines for these branches, accompanied with a good hotfix tracking/management system in place, making the hotfix delivery just routine (or almost).
> 3 votes

# Answer
Maybe if you maintained branches per versions instead of per customers it could help reduce their number?

Otherwise the only way to really get away from it is to be able to host the software yourself and switch to a SaaS model where you would be able to maintain only one version of it.
> 1 votes

---
Tags: release, process, branch
---