# Question
Title: What do we know about how open source software becomes eligible for the Docker Hub library?
Official community distros of well established open source software can be found in Docker Hub at the "special" place, the *library*, explicitely tagged as "official".

https://hub.docker.com/u/library/

How does it get there - by decision of Docker Inc., the company behind Docker, or is there some community application process?

For example, there is no such image for LaTeX.

Answers count referencing facts (no opinions).

# Answer
From https://docs.docker.com/docker-hub/official\_repos/

> Docker, Inc. sponsors a dedicated team that is responsible for reviewing and publishing all content in the official repositories. This team works in collaboration with upstream software maintainers, security experts, and the broader Docker community. While it is preferable to have upstream software authors maintaining their corresponding Official Repositories, this is not a strict requirement.
> 2 votes

---
Tags: docker, culture, open-source, images
---


# Question
Title: Strategies to work around 30 sec timeout
I received a timeout when trying to perform 7000 * 3 inserts into MariaDB in 30 seconds with AWS Python Lambda. I have three ideas how to get around the 30 second API Gateway timeout:

1. Convert the data to a file which MariaDB can import using `LOAD_DATA_INFILE` and let the Lambda do that.
2. Write the data in a file to S3 and let another function write the data from S3 to MariaDB which then will not have the API Gateway timeout limit.
3. Let the web client convert the file in JavaScript.

I am leaning towards the first strategy, to try and use `LOAD_DATA_INFILE` from the Lambda function. Do you agree?

# Answer
A variation of method 2 is infinitely scalable:

* write the data to s3.
* from s3 trigger lambda to break s3 file into sqs queue events of good size
* from sqs queue trigger lambdas to insert
* sqs can be fifo queue if it matters.
> 2 votes

---
Tags: databases, aws-lambda
---


# Question
Title: Tools used in enterprise level to transfer build artifacts from CI server to live server
I recently learnt some basic devops and set my server on Digitalocean droplet with travis CI. As there is no travis deploy provider supports for Digitalocean, I have to write my own deploy script. However I have more than 50000 files i.e near about 1 GB of data. I tried scp, ftp, rsync but I feel it takes a lot of time.

What are the transfer tools that is used in enterprise level ? Do they also use RSYNC or SCP. Or there are other tools that is better than these which are being used by medium level enterprises which I am not aware of.

Apart from it I have another question is there any deployment script/tool that can parallely transfer my files over ssh ?

# Answer
Try LFTP that has commands to mirror or parallel copy. LFTP can run over many protocols.

The likely problem you have is that you are transfering a lot of data from a build service to a single vm over the Internet. Enterprises use the same tools and protocols but they pay for more bandwidth else ensure that their builds and deploy happens within the same cloud provider. They then deploy to many servers in parallel using standard protocols like SCP with tools like Ansible. The networks within a single cloud provider (Azure, AWS, GCP) are prioritised. This means of you want fat deployment bandwidth, and to run on Digital Ocean, you should build on Digital Ocean.

It may be the case that a very large amount of the 50k of files dont change on each build. So what you really want is a tool that looks at what files are already deployed. We can expect such tools to use file name, timestamps, file sizes and checksums to check whether a file needs uploading. If your build is stateless and creates 1G of fresh files each time their timestamps are going to be newer that the identical files already deployed. LFTP has an "ignore time stamp" option to avoid transfers of newer but identical files. See this question and answer https://serverfault.com/a/311670
> 4 votes

# Answer
Not sure this helps you directly, but one option is to build the application within your CI build and push the (versioned) binaries/packages into an artifact repositorie like Sonatype Nexus or JFrog Artifactory. Your build would then, in a later step pull these binaries from the target machine/from within the droplet.

There are many packaging formats, depending on your technology stack, like e.g. NPM packages, jars, nuget packages, gems, Docker images and so on. Typically, you can also define your own package formats.

As I said, this is not a direct answer to the question about "enterprise level file transfer tools", but it is a common way to decouple building and pushing the application and pulling/deploying it.
> 1 votes

---
Tags: continuous-integration, deployment, continuous-deployment, travis-ci, cicd
---


title: GitHub merge access

text:
# Question
Title: GitHub merge access
I am trying to create a GitHub workflow (not GitHub flow), where specific people will have merge only access to a branch in repo. I tried a few things, and I found that I can restrict users to read and write to repo. But read access wont have merge permission, and write will have too much , like committing directly to the branch.

Is there any better way to do that?

# Answer
> 2 votes
It’s not clear exactly what level of trust you have towards contributors. You could update your question to explain a bit more about whether people are all on one team or might be strangers.

One feature that is likely to help is “branch protection settings” on this protected branches article. You can state that only code reviewed PRs are allowed into a branch. See also About Code Owners where some users can be named as the code owners on a particular branch. Only they can accept changes to the branch.

With those controls you can set it up so that anyone can contribute to a repository but the branch(es) where release are taken from are strongly controlled.

You might consider not having people write directly to the repo if they are not trusted team members. It’s normal to have untrusted people fork the repo, add the main repo as “git remote add main $URL” and do their work on their own repo. They can “git remote pull main” to refresh from the main repo and “git push origin” to update their own fork. When they are done they can go to the web interface of their fork and creating a pull request. It will automatically be directed back to the main repo they forked. Then only writers to the main repo can merge the pull request. That is much cleaner if you really want to limit who can write to your main repo.

You can also enable “squash and merge” of pull requests so that when the PR is accepted all the commits in the PR are squashed into a single commit on the branch they are merged into.

GitHub access controls seem very permissive as “write whole repo” until you see that everyone can create forks and send PRs. When you do only have one team using a private repo then letting them push their own branches into the repo, then integrated onto a projected branch, gives good controls over what is released.

---
Tags: continuous-integration, git, github
---


# Question
Title: Should local Ansible playbooks be loaded each day/week with cron?
I ssh into a remote 5$ `debian:stable` IaaS machine with Putty. The machine has only `debian:stable` and Ansible installed on it and I run an Ansible playbook locally on it with my current working user (that has sudo rights).

I aim to establish a LAMP server environment on this machine.

`playbook.yml`:

```
 - name: Establish a Debian-LAMP with some extras
  hosts: 127.0.0.1
  connection: local
  become: no
  tasks:

  - name: update the apt package index i.e. apt update
    apt: update_cache=yes

  - name: Install ufw
      apt: package=ufw state=present ## Install only if package-index is already present in the apt package-cache.

  # etcetra

```

$ `ansible-playbook playbook.yml`

That single Ansible playbook is an orchestration and deployment of my entire server environment (Installation and setup of ufw, `unattended-upgrades`, SSHGuard, Apache, MySQL, PHP, `ssmtp` and further tools like `curl`, `wget`, `zip`, `unzip` `tree` and so forth).

When I will learn enough Ansible I hope to add directives that will also transduce all my virtual-host files of Apache from say Apache 2.4 to 3.4 to 4.4 to 5.4 ecetra so I won't have to manually change all virtual host files in case I need to.

Given the playbook is filled with `apt upgrade package1 package2 package3 etcetera` I assume I should run this playbook (or at least parts of it) **each time anew**, if Ansible won't do it itself.

Should I run this Ansible playbook daily/wwekly with cron or Ansible will recursively run it itself by some internal scheduling mechanism of its own?

# Answer
> 1 votes
I would discourage you from using Ansible to run Ansible (though this can be done) as you will run the risk of breaking idempotent rule of Ansible. Instead, choose a singular starting point for running your playbooks; either command line, cron or some other program.

With that said, I would use cron to run an Ansible playbook, however I would also setup your `ansible.cfg` file for logging and check the log periodically if there is breakage in your playbook.

Ansible by default doesn't do logging so I would set this up in `ansible.cfg`, just in case:

```
[defaults]
log_path = ./ansible.log

```

Also, make sure to check your email regularly to make sure your cron/Ansible job ran without any problem.

# Answer
> 0 votes
I'm going to assume that the question is about keeping the server secure once deployed.

> Should I run this Ansible playbook daily/wwekly with cron

This can't hurt, but **it is not enough to keep the system secure.** It is not guaranteed that upgrading a piece of software will be enough to keep your system secure. For example, security updates to the kernel require a reboot, and some security updates may require configuration changes and/or service restarts. If you are responsible for such a server you should subscribe to the relevant security updates (usually a mailing list or web feed) and respond to fixes as they arrive.

In more general terms there is no way to deploy a server exposed to the Internet and have it be secure for an indefinite period of time.

---
Tags: ansible
---


# Question
Title: How can container user and host user both have access to same files in volume?
I'm using Docker namespaces in an effort to allow files created within a volume to be arbitrarily read and written to by both the container user and the host user (please let me know if this is even a valid approach to what I'm trying to do!).

My host username is `aryzing`. So I'm running the docker deamon as

```
dockerd --userns-remap=aryzing

```

The volume maps my host's `~/workspace/app` to the volume's `/usr/src/app`.

From the host system I have no issue modifying the files, but from within the container, I have access errors. Specifically, I'm running NPM's install command from within the container, `npm install`, and it's unable to install the packages due to a permissions error:

```
Error: EACCES: permission denied, access '/usr/src/app/node_modules'

```

As per this post I've also tried running the container with the `--user` flag, and ran into the same issue.

Any suggestions on how to achive the functionality I'm after?

# Answer
I was tripping myself up: when I was trying the `--user` flag, I still had `--userns-remap` set. Running the daemon without `--userns-remap` did the trick.
> 2 votes

---
Tags: docker, linux, npm
---


# Question
Title: Unable to mount docker folder into host using docker-compose
Here I created docker container which use the `mariadb` image and created three volumes as below.

### docker-compose.yml

```
version: '2.0'

services:
  mariadb:
  image: mariadb:latest
  restart: always
  container_name: mariadb
  environment:
    - MYSQL_ROOT_PASSWORD=root
    - MYSQL_DATABASE=testdb
  ports:
   - 3307:3306
  volumes:
   - ./database:/var/lib/mysql
   - ./_conf/mariadb.cnf:/etc/mysql/my.cnf:ro
   - ./logs:/var/log/mysql

```

First two volume works successfully but I can't able to find mariadb-logs files into `logs` folder. I seems `logs` folder showing blank on host as well as on container(/var/log/mysql). I think host folder override into docker file system.

If I remove this volume(`./logs:/var/log/mysql`) from docker-compose then logs files are showing on container.

My plan is to mount `/var/log/mysql/` folder to local machine.

Thanks !

# Answer
If you use CentOS 7 (as I have ). You may have SELinux turned on by default.check https://www.rootusers.com/how-to-enable-or-disable-selinux-in-centos-rhel-7/ or mount like this:

```
  volumes:
   - ./database:/var/lib/mysql:Z
   - ./_conf/mariadb.cnf:/etc/mysql/my.cnf:Z
   - ./logs:/var/log/mysql:Z

```
> 5 votes

# Answer
If you're running a RedHat-based distribution e.g. Fedora or CentOS, you may have SELinux turned on by default. You can automatically permit your containers to access files on the hosts by mounting with the `:Z` option, like so:

```
  volumes:
   - ./database:/var/lib/mysql:Z
   - ./_conf/mariadb.cnf:/etc/mysql/my.cnf:Z
   - ./logs:/var/log/mysql:Z

```
> 3 votes

# Answer
If `/var/log/mysql:/var/log/mysql` is defined as a volume then the content that reside in the container's `/var/log/mysql/` will be stored in the `/var/log/mysql/` folder on the host.

It could be possible that it is not possible to mount the `/var/lib/mysql` as SElinux or app armor is preventing this.

https://hub.docker.com/\_/mysql/

> Note that users on host systems with SELinux enabled may see issues with this. The current workaround is to assign the relevant SELinux policy type to the new data directory so that the container will be allowed to access it:
>
> ```
> $ chcon -Rt svirt_sandbox_file_t /my/own/datadir
>
> ```
> 1 votes

---
Tags: docker, docker-compose
---


title: Install minimal dependencies with apt

text:
# Question
Title: Install minimal dependencies with apt
I have a simple Dockerfile which looks like the following

```
FROM debian:stretch-slim as builder
COPY . my_project/
RUN apt-get update && apt-get install -y build-essential libpqxx-dev libtbb-dev

```

Running this with `docker build -t test_proj .` I was dismayed to see that apt is trying to install a huge amount (583MB unpacked) of extra libs, including things like python and sqlite (which sound completely unnecessary for pqxx and tbb). I ran apt-rdepends over each and sure enough neither python nor sqlite were in there, so there are surely more 'spurious' libs being installed by apt

Why is this happening and how can I avoid this? I plan to use a multistage to ditch the build tools by the final image anyway, but this is slowing down builds so I'd like to fix it

# Answer
Have you heard about the distroless? See this link too.
> 1 votes

---
Tags: docker, linux
---


# Question
Title: how to stop and start kubernetes cluster
I have launched kubernetes cluster using kubeadm with 1 master and 2 worker node. Is there any safe way to stop and start the entire cluster without disturbing deployments.

If I restart the master VM directly, It shows most of system pods containers in a stopped state.

# Answer
You can maintain your data while stopping the master if you have persistent volumes attached, this is especially the case when using GKE google Kubernetes engine. But if you wanted to keep your services running while completely stopping a master that would require High Availability, Allowing one master node to take over the responsibility of another master which is clustering.
> 1 votes

---
Tags: docker, kubernetes
---


# Question
Title: Can a 5 US $ VPSE support VAD stack with 2 relatively-small CMSs?
I'm new to DevOps and quite troubled financially but I desire to put websites that I now host on a shared server environment (CetnOS 12.x.x) on some cloud like DigitalOcean or Lindoe (but without using their droplets with ready OSs - I want to install the OS myself with Vagrant).

To have full automation of every upgrade possible (even OS itself say Ubuntu 18.04 to 20.04 or Apache 2.4 to Apache 3.4, PHP 7.4 to PHP 8.4 and so forth) I desire to use the VAD stack: Vagrant-Ansible-Docker.

DigitalOcean's cheapest plan for example is this 5$ plan:

```
25 GB SSD
1 virtual CPU
1 GB RAM
1 TB monthly data-transfer

```

I use the Drupal CMS (\>=8) for generally all my websites (which are generally small-scale perceptional websites for small business of free-professions).

Each VPSE will contain up to 2 separate (non multi-stie) Drupal apps and each such app will usually take no more than 256 MB RAM, if I'm not exaggerating).

## My question

Can I base these such a X2 drupals installment on such 5$ VPSE with a VAD stack in decent performance? Or there's no way I could handle both VAD and X2 such Drupals on one such small VPSE?

# Answer
In this scenario, the cpu cycles required to read from the database and render to PHP will all be handled by a single core, ie they will be handled one after the other. This will be about the slowest way to deliver website traffic out of your server.

Each time a request is made, a certain amount of memory is occupied by PHP. You can test the actual amount, but you can only ever be sure it is not greater than memory\_limit. Based on a use of no more than 256MB per Drupal site request, you can divide your 1GB of memory by that number (minus some memory for apache, MySQL and the OS). The result are 3 requests fit into memory only. You certainly do not want to allow more, because either your system runs out of Ram or it will use swap disk space.

The use of swap disk space is even worse, as it will slow down requests even more and consumes cpu itself.

3 requests can be used up quickly by some Ajax requests in the page or your Admin backend, either one likely bringing the server to its knees. It would be best to check in Chrome inspector how many PHP requests the site makes on frontend and backend.

Your situation will improve somewhat if you can dial down your PHP memory limit to say 64mb, but you still have the single core as the bottleneck.

So the key to keep this stable is to reduce the number of requests allowed either in apache or php-fpm to a number multiplied by PHP memory limit that stays smaller than your 1gb memory.

Your best option would be to copy out your Drupal pages to a static website every time a change is made. A transparent way for this is to put a cdn in front of your website. However you need to be sure.

Another non technical practical solution is to quadruple your hosting budget and find a sponsor or side gig to pay the charges.
> 1 votes

# Answer
I'm happy to presume jdog is right about his performance advice for PHP. PHP is not something I've ever tried to make perform. But one point that wasn't made is that you can:

# test it yourself

1. Fire up a VirtualBox instance on your desktop with the specs you're aiming for.
2. Put the software in there and see how it runs.
3. Run a performance test and see if it can handle a load that is useful to you.
4. If not, then adjust memory or CPU and try again. If it works, then profit.

This seems like a lot of work to some and really easy to other people, but it isn't something that somebody else can do for you. Even if we put the same software on the same spec'd box we might install it differently. Or we might put different content in, or test it differently. If you want a definitive answer for your work load then go and try it out. This site and ServerFault and such can help you find out how to do that in more detail, but we can't predict perfectly how it will work out for you.
> 0 votes

---
Tags: docker, ansible, continuous-integration, vagrant, performance
---


# Question
Title: Release based on restricted timeline with CI/CD
There is possibility that client may want certain functionality or feature to be available based on certain season or specific date/time/zeo location. This could grow more complex based on code logic or database changes spreading or impacting to other screens/logical layers.

How can a team handle above situation with CI/CD or DevOps as a process?

# Answer
> 1 votes
Branch-by-abstraction and / are the typical techniques allowing development of any combinations of features in various degrees of readiness in the same branch. For a proper CI/CD approach that should be the `main`/`master` branch. Each feature being developed can be tested independently by turning the respective flag on or off. "Releasing" the feature in that branch simply means that the respective flag is enabled in the source code

To be effective, all the relevant combinations of the feature flags' values should be subjected to the respective CI/CD QA verifications in this branch. The complication, of course, is the number of such combinations which can cause cost/time "explosions" in these verifications, especially in large scale projects with many features.

Grouping multiple feature flags under a single "super-flag" is an effective method of reducing the number of such relevant combinations. Turning on (in the code base) these super-flags are often time-driven according to a release schedule. Which can be either periodical or based on certain specified/seasonal time/dates, it doesn't really matter (as long the respective features meet the expected quality level by that time, of course, but that's a different discussion).

Turning on the super-flags is all that's needed for the software products that are continuously deployed. But for the products which can't be continuously deployed (like OSes or most of the embedded software products) release branches in which only the super flag(s) (or directly feature flags) applicable to the respective releases are enabled is the typical way to keep the release time/costs under control. They can also serve as the CI/CD branch vehicles for subsequent hot fixes, service packs and/or minor releases.

Release branches would be pulled off the main/master branch (or parent major release branch) whenever the quality level is good enough for the very next release and trouble to support the delta to the 2nd next release remains acceptable (as soon as the branch for the next release is pulled the parent branch can diverge, heading towards the 2nd next release).

And no, release branches aren't (necessarily) evil: they are not feature branches and they can completely avoid any massive branch sync/reparent/merge operation, thus remaining true to the CI principles.

---
Tags: continuous-deployment, cicd
---


# Question
Title: One time database update process as part of applying a patch
I have a PostgreSQL database that requires a server side script to be executed (client prefers php) to:

* Perform some intelligent data modifications to records that will be identified only during the update.
* Update specific functions for a dynamically identified list of schemas.
* Generate a static rollback script/data based upon what is being modified.

I am hoping to integrate it with gitflow like process to allow for some control and visibility as in the past these types of changes were done manually. I've been looking for something similar to libraries like phpunit to avoid reinventing.

Are there any scripting libraries or tools that can serve as a starting point.

# Answer
This is usually called a migration, try yii2, Laravel or doctrine

Mostly these frameworks will manage the schema in yml, XML or PHP and automatically calculate schema changes and apply them.

For data changes, you would package up a number of database queries or even program code into a migration, which usually has a forward/up step and a backward/down step.

Migration status is usually stored as a single value in a separate database table
> 0 votes

---
Tags: continuous-integration, postgresql
---


# Question
Title: How do you instruct AWS FARGATE to provision a t2.small via ECS CLI instead of the default micro?
I am trialling AWS ECS for the first time. I've run through the tutorials and now I'm trying to deploy an existing Docker Compose suite which I've been using on Digital Ocean for some time.

I have four containers in my composition, an ExpressJS app, NGinx, MemcacheD, and a Python Flask API. These work OK on a Digital Ocean 1GB RAM host.

I've been following the guide and specs here: https://github.com/aws/amazon-ecs-cli#launching-an-aws-fargate-task and other general ECS CLI guides.

I've got ECS to make the cluster+service from the docker YAML (with a few tweaks), but when I try to UP the service, I get an error that the t2.micro EC2 which Fargate boots isn't big enough:

```
INFO[0016] (service CI) was unable to place a task because no container instance met all of its requirements.
The closest matching (container-instance ...) has insufficient memory available.
For more information, see the Troubleshooting section of the Amazon ECS Developer Guide.

```

So I'd like to force it to boot an t2.small (2GB RAM) or higher.

I see that you cannot directly tell Fargate to use a specific instant type from ECS - that setting is only for "EC2" direct, but I want to use Fargate to automate the instance management.

I have tried using the `task_size` memory parameter (which I believe to be the correct way of informing Fargate of your application's requirements?) but this doesn't affect the choice of EC2 instance type which Fargate boots - it still boots the default t2.micro.

I added this YML file with `--ecs-params`:

```
version: 1
task_definition:
  task_size:
    cpu_limit: 512
    mem_limit: 2GB

```

When I rebuild my cluster from scratch with this definition, I can see the 2GB is correctly configured under the taskDefinition page (`console.aws.amazon.com/ecs/home#/taskDefinitions/...`) - so the configuration is provided OK, but when I view the instance it's booted, it's still a t2.micro with 1GB RAM.

Am I misunderstanding the task\_size.memory parameter?

EDIT:
I was mistaken in thinking the 2GB setting would influence the host instance size. This is not the case: `ecs-cli up` brings up the cluster and instance long before the docker-specified services are created by `ecs-cli compose service up` which reads those task params.

The task size parameter is per TASK which in Docker Compose terms are "services" (I think of them as containers plus exec params). So as I have four services, there will be four ECS tasks. To get the four tasks to fit inside a 1GB host therefore, task size should be 256MB (not 2GB!)

task\_definition: task\_size: cpu\_limit: 512 mem\_limit: 0.5GB

This now gets further creating the services, without the memory error. I don't quite understand why (as 0.5GB x 4 is still too big).

So while this has avoided the error, I still want to know how to create a larger instance when performing `ecs-cli up` with Fargate.

# Answer
> 1 votes
The short answer is "you don't have to".

The long answer is I was doing it wrong. Here's what was wrong and what I learned.

After working through tutorials, RTFMing and re-building my ECS configuration I found that I actually **wasn't launching via Fargate**. I had made a mistake during the initial `configure` command, so the default config was using EC2 launch type. When making the cluster it created an Autoscaling Group template with t2.micro in it, and then it was using this to launch the initial instance. I didn't think this was unusual, as I read that Fargate managed your EC2's for you, which I presumed was what was happening. No, this was "traditional" EC2 creation via a scaling template.

I figured out the mistake by viewing the config file. I noticed `ecs-cli` doesn't have any "view config" commands - they only write to the config, so it feels a bit mysterious when you are issuing later commands, especially as you are encouraged to set lots of defaults (presumably to simplify the CLI commands which get long and cumbersome).

```
less ~/.ecs/config

version: v1
default: MY-TRIAL-CONFIG
clusters:
  MY-TRIAL-CONFIG:
    cluster: MY-TRIAL-CLUSTER
    region: ap-southeast-2
    default_launch_type: ""      <= this was the problem
  tutorial:
    cluster: tutorial
    region: ap-southeast-2
    default_launch_type: FARGATE

```

So I then tried the EC2-only parameter:

```
ecs-cli up --instance-type "t2.small" --capability-iam

```

and this changed the Autoscaling Group and indeed launched a t2.small. So I now know how to do it the EC2 way.

To do it properly as I had originally intended with Fargate, I needed `default-launch-type FARGATE`:

```
ecs-cli configure --cluster MY-TRIAL-CLUSTER --region ap-southeast-2 --default-launch-type FARGATE --config-name MY-TRIAL-CONFIG

```

Now the config looks like this:

```
less ~/.ecs/config

version: v1
default: MY-TRIAL-CONFIG
clusters:
  MY-TRIAL-CONFIG:
    cluster: MY-TRIAL-CLUSTER
    region: ap-southeast-2
    default_launch_type: FARGATE
  tutorial:
    cluster: tutorial
    region: ap-southeast-2
    default_launch_type: FARGATE

```

After doing this, `ecs-cli compose service up` works very differently from the observed ECS launch flow.

1. it doesn't create any Autoscaling stuff
2. it doesn't launch any EC2 instances (I thought Fargate did, but managed it for you, but you don't get to see the container hosts)
3. It created a Cloudformation Stack instead (interestingly with a t2.micro EcsInstanceType parameter)
4. It's very clearly labelled on the Cluster home page: there's a row for EC2 launch types below the FARGATE row.

e.g.

```
MY-TRIAL-CLUSTER >
FARGATE
1          0             1
Services   Running tasks Pending tasks

EC2
0        0             0                                              0
Services Running tasks Pending tasks CPUUtilization MemoryUtilization Container instances

```

It's very clear now I see it. There are no instances for Fargate services.

You can also watch Fargate re-start the task-containers if they fail, either in the control panel, or with `ecs-cli compose service ps` and `ecs-cli logs --task-id XXX --follow`, to get a feel for the magic and how the Docker Compose features are being performed.

---
Tags: amazon-web-services, docker-compose, amazon-ecs
---


# Question
Title: Azure DevOps Pipeline Conditional Flow with Manual Intervention
I am wondering if Azure DevOps has the ability to do conditional flow as part of a release definition.

***My use case:*** I have a PowerShell script that deploys a SQL Server DACPAC to create or update a database. Sometimes this deployment will fail because a change was made that could potentially cause data loss. Instead of absolutely failing the deployment, I would like to trigger a manual intervention for a DBA to review the changes. If they resume, I would like to attempt the deployment again, but this time with a flag that will ignore potential data loss issues.

Finally, if that second db deployment attempt is successful, I would like the flow to continue back to the rest of the deployment tasks.

***Here is a general flow diagram:***

***Other requirements:***

* If the db deployment fails, but the manual intervention resumes and succeeds, I want the entire deployment to be marked as succeeded.
* I want the flow after resuming an intervention to merge back into existing tasks that would be run on db deployment success. Corollary: I do not want to maintain dual flows with identical tasks.

I am using the Designer (Web UI) and not YAML for all my current builds/releases, but I am also interested if YAML has a solution for this.

Azure DevOps has a section for Conditions, but I did not find that particularly helpful. The number of conditions seems limited. It also looks like variables can be set on jobs, but this appears to be only available with YAML.

Is there a way to set up the conditionals at the end of phases/jobs or tasks to achieve this branching behavior?

# Answer
> 2 votes
@Shayki's answer was a large portion of what I was looking for (thanks!), but I was eventually able to figure out a solution that met all of my requirements. It involves a less than ideal use of the Release Description to propagate state between jobs.

The updated flow diagram looks like this:

I was originally thinking of the conditional flow as an `if else` statement, but it is really more of multiple `if` statements where the condition is the same. If flag is set, do X; if flag is set, do Y; etc.

## Details

Instead of running the agentless job when a previous condition fails (which would result in the entire deployment failing, a violation of one of my requirements), I created a custom condition that the agentless job runs as:

If all the previous tasks have succeeded, and if the Release.ReleaseDescription variable contains the string defined in the \_dbDeployErrors variable, then this job will execute (and the manual intervention task inside of it).

The \_dbDeployErrors variable isn't too important, I set it to `;DB_DEPLOY_ERRORS_$EnvironmentName`. The main idea is that, at least with the Web Designer, there is no way to pass information from an agent job to an agentless job. So I wrote a powershell script which uses the Get Release and Update Release Azure DevOps REST API to update the Release Description parameter on the release. This allows me to pass information to the next job in the deployment.

The basic steps for setting this flag are:

1. Run the db deployment
2. If the deployment has errors, do not fail task, but instead set an agent variable
3. Next task determines if the variable is set, and if so calls the Update Release API to append the flag to the release description
4. Future jobs and tasks can now analyze the state of the release description variable

*Note: I set an agent variable instead of making the API call from the db-deploy script so that I can keep my deploy script clean and maintainable. I move the responsibility of calling the API to a dedicated script instead.*

If the Manual Intervention is resumed, then the third job executes as an agent job. It will attempt to deploy the db again, but this time it sets a `-Force` flag on the script to ignore possible data loss (it was approved by the one who performed the intervention). If this deployment succeeds, then a quick cleanup is run to call the Update Release API and clear the flag from the release description. Finally, the other tasks can resume as normal.

Here is what the tasks in my final release definition looks like:

# Answer
> 1 votes
Maybe is not a full resolution for you, but you can start with this:

After the "Critical Task" add an "Agentless job"

Add a "Manual Intervention" task

In the "Additional options" in the job specify "Run this job - Only when a previous job has failed"

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: How can you pass private environment variables to FARGATE tasks specified from a Docker Compose config
I am using an existing docker-compose.yml to migrate an app to ECS. The config references several $ENV vars which I used to pass in simply as environment variables when running `docker-compose up`. One of these is a password, which used to be specified on the Docker host. In ECS you are advised not to pass sensitive stuff along with the env-vars, so I'd like to improve this and use the best practises.

In FARGATE, I can't see how to pass these secure parameters into to the `ecs-cli compose service up` without sending them into the docker-compose used by ecs-cli (and then they end up displayed in the console, I guess possibly logged, etc).

The ECS CLI documentation explains to use SSM

> `secrets` allows you to specify secrets which will be retrieved from SSM Parameter Store. See the ECS Docs for more information, including how reference AWS Secrets Managers secrets from SSM Parameter Store. value\_from is the SSM Parameter ARN or name (if the parameter is in the same region as your ECS Task). name is the name of the environment variable in which the secret will be stored.

But under the ECS Docs

> "This feature is not yet supported for tasks using the Fargate launch type."

I presume the "yet" means there will be one day.

I wonder if there's an current alternative for Fargate, or if we have to be less secure while they add support for it?

# Answer
> 2 votes
This is a well-written and up-to-date article which summarises the problem I've hit and confirms there isn't really a perfect solution at the moment in Fargate.

https://hackernoon.com/secrets-management-within-aws-ecs-1b6975819ccd \- Connor Beardsmore

The alternatives listed are:

1. Bake into Docker - not an option with an open source repo
2. ECS Environment Variables - as I mentioned less secure
3. Docker Fetches from SSM Parameter Store - secure but "architecturally dirty"
4. Wait for Fargate SSM to be able to inject sensitive data!

Re. option 2. the problem is the environment variables are visible in the AWS console. But I wonder: if an attacker has access to your AWS console, you're pretty screwed already? So isn't this in fact an practically acceptable solution?

---
Tags: docker, amazon-web-services, docker-compose, security, amazon-ecs
---


# Question
Title: How/where are programs installed in Docker containers?
Apologies for being such a basic question but all the resources I've found seem to take this knowledge for granted.

I'm brand new to docker and containerization in general.

Today when opening a program for a project i.e. RStudio, Jupyter, Notebook, Spyder or an IPython Console I would run a bash script in anaconda3/bin, either from a custom menu item or Anaconda Navigator or just the terminal.

How does this work with Docker? If I'm using a Docker container for my project and I install Anaconda and the other packages I use, is there a folder within the container containing the bash scripts or does it link to bash scripts installed on my system or something else?

# Answer
> 4 votes
Hi and welcome to DevOps SE and Docker!

Now the concept of Docker is to create isolated virtual environments, actually it's just a process or more but wrapped in its own environment.

So at first hand you could take any Docker image and the scripts inside would be there where the creators of the image placed them.

If you decide to create your own Docker images, and I believe you will sooner or later, then it is up to you how layout the system.

Additionally, you could link - proper term is here *mount* \- your system's folders with those of the container but still the execution will take place inside the container.

For example to install the RStudio package in an Anaconda3 environment, I have tried the following.

```
$ docker run -it continuumio/anaconda3 bash
(base) root@7162c3acc9e3:/# conda install -y -c r rstudio
(base) root@7162c3acc9e3:/# which rstudio
(base) root@7162c3acc9e3:/# /opt/conda/bin/rstudio

```

As you can see, if installed as root user which is default, the path is `/opt/conda/bin/rstudio`. Most packages not using special system capabilities would considerably not "know" whether they are in a Docker environment or not, so in many cases you could consider a Docker container to be just a (fluid) environment you have the root access too.

Additional note on the terms:

* *image* is a blueprint for a container defined by a Dockerfile. "Installing" something in an image means you add instructions to its Dockerfile. Own images can be derived from others by the statment FROM.
* *container* is basically process isolation in its own OS environment. If you install something in the container during its uptime, this change will be gone with the container.

# Answer
> 3 votes
> How/where are programs installed in Docker containers?

They are stored inside an image.

In a nutshell, what Docker does for you is this:

* It takes a prepared "image" (which you name on the command line `docker run ...`).
* For all intents and purposes, you can consider this image as simply a volume stored somewhere in `/var/lib/docker`, temporarily mounted somewhere on the host temporarily for you.
* Docker then executes an executable within that image as usual, but fakes its environment such that the new process thinks that the volume mounted in the previous step is its `/` directory. Also, the new process is pretty much encapsulated from the host, i.e., it shares nothing (other processes, files, networking etc.) unless specifically opened up by Docker for you.

Apart from that, everything is as usual. There is a lot of fakery going on, but technically, at the end of the day, it's just a normal Linux process. The magic happens due to namespacing, but it is mostly Linux magic, which is simply used by the Docker demon for you. Docker is like `chroot` on steroids.

What Docker does is manage all the tiny bits and pieces for you - i.e., the image management (creating, deleting, listing etc.), the container management (running, stopping, inspecting etc.), virtual networking, the "onion" filesystems which you probably heard about and all such.

> If I'm using a Docker container for my project and install anaconda and the other packages I use, is there a folder within the container containing the bash scripts

Yes, exactly. You acquire an image with anaconda, prepared by someone or yourself. The image is like a `.tar` archive (in a different format), and mounted during runtime. Everything lives in there. (Nomenclature: the folder is within the image, which is mounted. The "container" is just the ephemeral runtime - the process, networking etc.).

---
Tags: docker
---


# Question
Title: Understanding contents and workwise of a Terraform plan
If you export a Terraform plan, it is a binary file.

Now, calls to a cloud management interface are HTTP calls following corresponding REST APIs.

Does this mean this internal Terraform format somehow encodes these calls through some enumerations?

Like, given in the plan it would be, "B#3$1$2" which then translate "call the API function we have given number 3 passing values 1 and 2 as arguments"?

How does it work?

# Answer
Terraform use protobuf to store the plans according to its code

> // writeTfplan serializes the given plan into the protobuf-based format used
> // for the "tfplan" portion of a plan file.

You can find here the code generating the plan file.

So in brief that's a binary file with a serialized version of what to do in a portable format (protocol buffers) for which you find more information here
> 1 votes

---
Tags: terraform
---


# Question
Title: What is the best way to achieve DevOps in any environment, or it is always environment dependent?
I want to know that is there possibility to make DevOps generic, and not environment specific, so that can be used with any project with same cases OR we can cover common cases with one time code. Though language and environment changes but the logic remains same for most of the Test cases based on..

I know every organization follows different methodology, I am assuming here that We will use one standard methodology and One standard Development culture.

1. Run environment
2. Development Culture, etc.

I want to achieve Common DevOps environment that can be used for any project for common cases. Only special cases need to be handled separately.

Have anyone achieved this?

I know this is little broad question, but we this is DevOps related only.

# Answer
> 3 votes
The answer to this question about what is DevOps?

DevOps both involves Strategy and Tactics.

Strategy embraces building a culture centered around collaboration in your software development, technical operations and any other group or organization that has a stake in the deliverables whether it's software or service.

Examples of DevOps strategy may be breaking down silos, continuous delivery, continuous integration, etc.

Tactics are the actual implementation of the strategies.

This is the level where the use of software tools, technology and management methods are found as well as the actual "how to". For instance, to foster "breaking down of silos" as a tactic, you build smaller "agile" teams of 5-7 "stakeholders" let's say programmers, systems admins, QA engineer, and a project manager. To communicate among the team members, you use Slack. Programmers and sysadmins "cross train" on each others tasks where it makes sense. Sysadmins may assist in developing test harnesses for the programmers, use Selenium and python for testing. Programmers learn how to use Ansible and assist the Sysadmins on what they need in the development and production environment.

Where you sound like your at is looking at all these cool tactics and tools that DevOps uses such as Docker, Ansible, Chef but your missing the understanding of Strategy...building of the culture, the "why" you want to use these tactics and tools. That's ok, we're here to learn.

There is no "right" answer on what exactly DevOps is or how you implement it, but in all the talks and examples of DevOps shops and organization it boils down to the strategy of building a collaborating organization and focusing on a purpose then put it into action as tactics.

---
Tags: culture
---


# Question
Title: Access S3 files from Datacenter over VPN to VPC
I am trying to access S3 files from non-EC2 machine. non-EC2 machine is connected to VPC over VPN, also can reach to other hosts in VPC.

I am able to download (`wget`) file from S3 to EC2 in the VPC. When I try to do same (`wget`) from non-EC2 (which is in Data Centre), I get timeout.

Anyone know what I need to do to get this working ?

# Answer
You need to use...

* the Internet, from your data center, to access S3 directly (not via the endpoint) or
* an AWS DirectConnect connection to access S3 directly (not via the endpoint) or
* an HTTP proxy server inside this VPC to access S3 (via the endpoint)

...because S3 endpoints are not designed to be used across VPN connections. They are only accessible from instances in the VPC where they exist.

> Endpoint connections cannot be extended out of a VPC. Resources on the other side of a VPN connection, VPC peering connection, AWS Direct Connect connection, or ClassicLink connection in your VPC cannot use the endpoint to communicate with resources in the endpoint service.
>
> https://docs.aws.amazon.com/vpc/latest/userguide/vpce-gateway.html#vpc-endpoints-limitations
> 4 votes

---
Tags: amazon-web-services, amazon-s3, aws-vpc
---


title: Azure Equivalent for CodeDeploy

text:
# Question
Title: Azure Equivalent for CodeDeploy
Right now, using AWS services we have a CI and deployment pattern that works well for us.

For every commit pushed to github, Travis-CI builds the project and runs tests, if tests pass, Travis uses CodeDeploy to deploy to staging (an AWS EC2). On tagged commits, if tests pass, deploy to production.

We are exploring using Azure, and we can't quite seem to figure out how to get the same pattern. In particular, we can't find the equivalent to CodeDeploy for Azure.

Does anyone have a similar pattern of CI and deployment that deploys to an Azure Virtual Machine?

According to Microsoft's AWS to Azure services comparison, Visual Studio Team Services correspond to CodeDeploy, but it seems like a very different tool and it is not clear to me if it can be put in our current pipeline.

# Answer
> 5 votes
Azure has a robust CI/CD pipeline which is much easier to use than CodeDeploy on AWS. It is called Build/Release pipelines under Azure DevOps. It allows you to configure multiple environments with build and release (deployment) stages for each. It also allows for easy configuration and interoperability with Azure Webapps/VMs and other resources available on Azure. Having recently switched from Azure to AWS, I definitely miss the ease of configuration and the intuitive UI on Azure DevOps.

Azure DevOps integrates a version control repository system, a agile style work items/boards which can also be used to configure your backlogs and iteration items. There is even a dashboard to integrate and monitor your work items and deployments across the team. So it is definitely much bigger than just a devops tool or CodeBuild.

---
Tags: azure, travis-ci, aws-codedeploy
---


# Question
Title: Kafka from client to server vs. "router" in the DC
I want to use Kafka as a message broker on the server side of a distributed application. We have clients that can be located around the world, and that send frequent messages to the server. Right now those messages are just JSON over ZMQ. I am considering changing the connection between client and server to Kafka (since I'm going to use Kafka in the server side anyway) but have the following confusion:

* Is Kafka even meant to be used this way (distributed clients connecting to the server with it)? I don't see that in the docs.
* If I don't use Kafka, I'll have a single choke point at my server/DC entry. What's the efficient alternative to get information from the software gateway (which is now ZMQ) into kafka, without having that software gateway be a bottleneck?

# Answer
Kafka doesn't care how you use it; it does nothing except transporting your arbitrary messages for you. Yes, you can use it that way, and actually I do not see anything specific enough about it to make it necessary to be mentioned in the docs.

Obviously you need to take a little care regarding security (SSL or SASL...) but aside from that, there should be no particular problem.
> 1 votes

---
Tags: kafka
---


# Question
Title: How do I return just the full path of a file found with the find module in Ansible?
This has been reworded, expanded upon and clarified via an update late on 11/9/18. I request it be upvoted now that it has enough code.

I want to find a softlink (to another file) in the $PATH environment variable on Linux managed nodes.

I have an Ansible playbook that uses a find module. I have a string of potential directories to look in. This string is the value assigned to the variable "result\_from\_prev". My playbook looks like this:

```
 - debug:
     msg: "{{ item }}: {{ lookup('env', 'PATH') }}"
   with_dict: {a, 1}
   register: prevar
 - set_fact:
     midvar: "{{ prevar | dictsort | join('') }}"
 - set_fact:
     result_from_prev: "{{ midvar.split(':') }}"
 - find:
     paths: "{{ item }}"
     patterns: "goodfile"
     file_type: link
   register: files_found
   loop: "{{ result_from_prev }}"
 - set_fact:
     jsonvar: "{{ files_found | to_json }}"
 - set_fact:
     var: "{{ files_found | list }}"
 - debug:
     msg: "{{ item }}"
   loop: "{{ files_found | json_query('files[*].path' }}"

```

I want to extract the path of the file name for the file that is found. How do I do this?

The above gives me an error about invalid data being passed to loop. A list must be passed. I think that files\_found is a dictionary. How would I use wantlist=True here? I tried, but I had no luck.

I tried using jsonvar instead of "files\_found" in the loop. But I got the same error about invalid data being passed to 'loop.'

I tried using listvar instead of files\_found. But I got the same error about invalid data being passed to 'loop.'

I tried using "with\_items" instead of "loop". But this prints out nothing for the msg.

In the JSON that the jsonvar variable can print out, there is a "matched" key. The value is "0" for each directory path that the file was not found and the value is "1" if the file was found. But I do not know how to extract just the full path of the file found. I'd like it to be clean like this:

> /tmp/subdirectory/goodfile

The potential locations of where the file will be will change. I have a framework that works: I have the potential files in a list in a variable named "result\_from\_prev".

I have tried using the "when" conditional and the item.matched notation to analyze the value of specific keys. These efforts do not work for me. I have tried putting the JSON into a file or using the "join" feature to create a long string. But I have not had luck parsing this long string. I have also tried using the "lookup" keyword with set\_fact to assign a variable to that directory path that was found with the find module. I tried to start over completely with the stat module instead of the find module. But I found this to be difficult to use a list of potential paths whereas the searching by the file's name is supported with the find module. I want to avoid using the "shell" module. Nothing I have tried works.

I expect the find module to return only successfully found files with a supported syntax that makes extracting the directory path relatively easy. This is not what I have experienced. How do I return just the full path of the file named "goodfile" that is matched with the find module above?

# Answer
> 1 votes
Answers to some parts of your question are:

* "I want to find a softlink (to another file) in the $PATH environment variable on Linux managed nodes"

Use "shell" module instead of lookup.

> Lookup occur on the local computer, not on the remote computer.
* "I want to extract the path of the file name for the file that is found. How do I do this?"

Use "files\_found.files" to see the complete path. Checkout the return values section in find module documentation.
* As far as i understood, you want to do something like this

Try out the following:

```
- name: finding files matching "goodfile" pattern
  find:
     paths: "{{ some_path }}"
     patterns: "goodfile"
     file_type: link
  register: files_found

- name : get the complete path
  set_fact:
    files_found_path: "{{ files_found.files }}"

 - debug:
     var: files_found_path

```

---
Tags: ansible, linux
---


# Question
Title: Take automatic screenshots of a website and send it to WhatsApp or Facebook Messenger
I am trying to automate the process of "sending the screenshots of a monitoring tool dashboard to my WhatsApp number" at regular intervals.

So far I have achieved the following:

1. Automated the screenshot taking using pageres-cli
2. Integrated the same with SMTP server using postfix

Now I also want to send these screenshots to my Whatsapp group or Facebook messenger group. How can I achieve this?

# Answer
In case the requirement for Whatsapp/Facebook is not mandatory, you could consider using a solution offering an API for such things and targeted specifically for IT teams.

For example, Slack channels allow uploading files through Slack's API and sharing files in channels.

Documentation: https://api.slack.com/methods/files.upload

For a ready CLI stub, consider for example slackcat.
> 0 votes

# Answer
> I am trying to automate the process of "sending the screenshots of a monitoring tool dashboard to my WhatsApp number" at regular intervals.

IMO you're approaching this from a very difficult direction, far more than necessary. Sending screenshots of your monitoring dashboard is presumably so you can check and see when things break. But that's what alerting is for.

There are plenty of alerting tools, and if your monitoring db is a standard one then most of them will have hooks into it. The alerting software will run checks, and when the checks fail notify you by phone, slack, PagerDuty, whatever. This will be far easier to manage than a headless UI tool, will be something other future employees are more likely to have experience with, and most importantly, you won't get pager fatigue because it'll only be notifying you when things are *wrong*.
> 0 votes

---
Tags: automation
---


# Question
Title: Ansible: Difference between the apt module and command: apt-get -y install X (module)
Consider these YAML tasks which are based on two different parts of some locally-executed Ansible playbook I read here:

```
- name: Update the apt package-index cache i.e. apt-get update
  apt: update_cache=yes

- name: Ensure aptitude is installed
  command: apt-get install -y aptitude

```

As an Ansible newbie, I recognize two different Ansible modules here: `apt` and `command`.

Why did the OP use both `apt: update_cache=yes` and `command: apt-get install -y` instead using the `apt` module in both tasks?

Note: I myself no longer use `apt-get` but `apt` even in regular Bash scripting.

# Answer
> In other words, why not both be done with an apt module?

We can't say for certain, but likely it's just a mistake coming from ignorance. The latter command can be fully implemented using the Ansible apt module, and you should always prefer that over `command`; `command` exists as an escape for when there isn't a module for what you want to do.
> 1 votes

# Answer
> TL;DR: Ansible modules like `apt` can be mostly implemented with `command` module, but you lose error checking. Using command is usually sign of a lack of experience with ansible.

To answer this question it is best to look at the code implementing the apt module in ansible. What you can see is that most of the features of the module are implemented by constructing an appropriate `apt-get` command and executing it, then handling error conditions and presenting them in ansible data structures. Sometimes depending on options the command can be quite complex. In two exceptions for upgrade, in case it is *full-upgrade* or *safe-upgrade* it would use `aptitude` command instead of `apt-get`.

Of course, the implementation of the module can change going forward, but you should be able to rely on it doing essentially the same thing. On the other hand, if `apt-get` changes in some way or returns new unhandled type of errors, you will need to change your ansible code in case you use `command` module directly. While if you use `apt` module, it is likely the maintainer of the module will implement the new error checking for you.

So in general it is better to use `apt` to `command: apt-get` unless the module won't do something that you absolutely need to do and there is no other way around it.

To do the 2nd task in ansible using apt module do:

```
- name: Ensure some basics
  apt:
    name: aptitude

```
> 1 votes

---
Tags: ansible
---


# Question
Title: What Ansible does in cases there's a conflict between OS version and some software Ansible manages?
Say I use an IaaS cloud on which I installed Ubuntu 18.04 manually via the cloud company's OS installation tool and I ought to establish an all-ansible LAMP environment that will include some more stuff besides Apache, MySQL, and PHP (like \`php-cli, php-curl php-mbstring, php-mcrypt, php-gd, Certbot, ssmtp **and so forth**) -- all installed with Ansible as well, of course.

So far so good, but what if after 4 or 8 or 12 years the particular Ubuntu rlease (18.04) becomes too old for a newer version of **whatever software** I installed with Ansible and also and constantly being upgraded by Ansible?

How Ansible itself deals with such a rare (?) conflict, does it usually do some "rollback" to a "previous state" (just like an autosave and autoload for video games) and also stops to work (and maybe also prompts me) until I myself decide if I want to remove this particular software (or softwares) that cause trouble and then start the Ansible process again?

# Answer
Ansible is at its core a programming language - a meta programming language written in python compiling itself into python and shell scripts mostly. It will do whatever you program it to do. It all depends on how your roles and playbooks are written. It certainly can be done - almost anything can be done - your install roles could track dependencies on package versions or ranges, obviously those might need to be updated over the years, but you can be safe if you put the work into it.

It is probably unreasonable to plan further than the End of Life of the LTS distribution though, as you will fail to receive new security updates and that instantly invalidates the system as production ready.
> 1 votes

# Answer
If you try to install an unsupported version of a package using Ansible, it will do the exact same thing as if you tried to install it yourself. Ansible is merely automating your actions - it doesn't add in special packages or versions or anything.
> 0 votes

---
Tags: ansible, linux
---


# Question
Title: Should I run several applications in Docker in a single server?
I have a server,32Gb ram, i7 processor and 2TB hard disk, with Docker engine running in it. I have like 100 development application to be migrated to this docker server.

So is it a good practice to run all your application in Docker in a single server?

# Answer
> 0 votes
Yes, sure. As long as your resources are enough (i.e., you're not running out of RAM, HDD, CPU, networking), and as long as you are fine with one gigantic single point of failure, there is no reason whatsoever to not run many docker containers on one machine.

Keep in mind that the overhead of docker is really small - it's all just namespacing on the host kernel, it's not like you're spawning copious amounts of VMs or anything like that.

# Answer
> 0 votes
I hope you don't.

Let's assume that cost is paramount. There is no money for a second server. Then the question doesn't apply as you cannot use more than one server. So the cost of a second server must be okay.

Then why wouldn't you take the opportunity to automate the setting up and administration of more than one machine? Then use the development environment with multiple machines as a test place to practice high availability, DR and how to run systems spanning multiple machines.

Are you intending to run more than one server in production with automatic failover when machines die but are not going to set that up in development? How do you know that automation will every work in production if you haven't practised in development?

With devops having development, or test, not behave like production always leads to problems. Nothing quite work first time in production so isn't automated, and production is never "fixed" as we don't feel the pain in development, so we just release less often to space out the pain, which isn't at all the spirit of devops.

Have the ambition to learn everything that you will do in production by practising making development like production, feel the pain of that, and change production until there is no pain doing the same things in development. You don't have to have the entire scale of production in development as that would be expensive, but at a minimum two servers that you can easily shift things between, or run a service across as highly available, is a huge leap over having a single server.

# Answer
> 0 votes
Best practice? No, it's better to have a bunch of smaller nodes so you can make your services better distributed for availability reasons. But will it be any *worse* than what you have currently? No.

---
Tags: docker, docker-compose
---


# Question
Title: What methods are available for connecting many-to-many containers?
I have two containers (Docker):

* TensorFlow container
* NodeJS container used as a frontend to the TensorFlow container.

Both containers have TCP listeners on a port specified using environment variables.

If I want to scale the NodeJS and TensorFlow containers, for example 2 instances of NodeJS and 4 instances of TensorFlow, what are my options for connecting them to evenly spread the load across the 4 instances of TensorFlow?

Currently I'm using the TensorFlow container name inside the NodeJS container, but I'm pretty sure that would only take advantage of a single TensorFlow container... Or??

# Answer
> 1 votes
The classic answer to balancing load is, of course, a load balancer. There are many different methods of doing this, but we're interested in ones that work well with dynamic containers.

This is one of the things an orchestration layer will address. In Kubernetes, for instance, you would create a Tensor Flow service, and scale it to however many pods you need (or use autoscaling!). Kubernetes then provides an ip for the service that kube-proxy load balances across the pods.

There are also service mesh options that replace this layer and perform a similar function.

# Answer
> -1 votes
You should assign a specific port to any containers.
For this configuration you can use `docker-compose` file such as following example:

```
version: '3'

services:
  TensorFlow-1:
      image: gcr.io/tensorflow/tensorflow:latest-gpu
      container_name: first-tensor
      ports:
        - "9001:8888"

  TensorFlow-2:
      image: gcr.io/tensorflow/tensorflow:latest-gpu
      container_name: second-tensor
      ports:
        - "9002:8888"

```

Then in the `docker-compose.yml` path, on command-line do it:

```
sudo docker-compose up

```

Or

```
sudo docker-compose up -d

```

**NOTE**: `-d` option is for run above command in the background.

Finally you have two tensorflow containers with `9001` and `9002` ports.

---
Tags: docker, containers
---


# Question
Title: Giving a meaningful name to Dockerfile's VOLUME
In Dockerfile we can use `VOLUME /some/folder` to let docker always create a volume for the specified folder when calling `docker run`, without having to explicitly use the `-v` parameter. The volume created will have a random (unreadable) alphanumerical name.

**Question:** is it possible to specify a readable name for the volume on container creation?

# Answer
You do not specify a volume source when you define a volume in an image. Doing so would be a potential security exploit where an image could mount a private volume with a common name and send the contents of that volume over the network. To create your volumes with known names, and have that defined in a file, you would do that at runtime with a docker-compose.yml file when the container is created. E.g.

```
version: '3'
volumes:
  - app_data
services:
  app:
    image: your_image:1.0
    volumes:
      - app_data:/data

```
> 2 votes

---
Tags: docker, dockerfile
---


# Question
Title: Is it possible to build docker images using docker-compose concurrently?
When one runs `docker-compose up --build` then the docker images are created sequential. Is it possible to build these images concurrently using docker-compose?

There are multiple issues including this, but it is unclear whether it is possible or not.

# Answer
Yes, it's possible. You can use `--parallel` option since docker-composer 1.23.2 version.

Version 1.23.2 also correct 1.23.0 bug:

> Reverted a 1.23.0 change that appended random strings to container names created by docker-compose up, causing addressability issues. Note: Containers created by docker-compose run will continue to use randomly generated names to avoid collisions during parallel runs.

https://github.com/docker/compose/releases/tag/1.23.2

You can use it simply doing:

```
docker-compose build --parallel

```
> 10 votes

# Answer
Currently it is not possible to make concurrent builds with Docker Compose.

There are issues about that but the latest tests indicate that concurrent builds take actually longer than the sequential ones.

The issue seems to be related to the storage drivers.
You can track this issue to get updates: https://github.com/moby/moby/issues/9656
> 3 votes

---
Tags: docker-compose
---


# Question
Title: Is this okay to use double proxy-server setup (on host and in container) on one machine?
One nginx server on host machine with renewable certificate (via certbot). And another nginx server inside docker container. First one should redirect all traffic to the second one.

Currently I have one nginx service inside container and I connect certificates that stored on host to that server via volumes. But with this kind of setup I can't renew certificates automatically...

# Answer
Sure, you put in as many proxies as you need. You don't want to over-complicate things, but it seems you have a good reason for your architecture.
> 1 votes

# Answer
Production configurations often involve multiple levels of proxies. Each should provide a place for gathering stats. Proxies can also hide backend failures if you have a pool of nodes behind the proxy, which doesn't seem to be what you're going for here.

Your comment that you wouldn't be able to renew certificates with extra proxies is not correct. You can update certs via DNS auth then deploy to your proxies through whatever mechanism you're already happy with.
> 1 votes

---
Tags: docker, nginx
---


title: RDS IOPS utilizing fully

text:
# Question
Title: RDS IOPS utilizing fully
I have upgraded RDS mysql to 1000 IOPS from 120 IOPS, But same database operations utilizing more than 1000 IOPS now. Is there any way to implement RDS cluster or Is it possible to make use of lambda to reduce the usage of IOPS and load?

Newbie here, If anything wrong please suggest the right way. Thank You!

# Answer
First try to review current caching parameter values. Then check list in AuroraMySQL.Reference.ParameterGroups And try to tune/increase caching parameters like *query\_cache\_size*

Alternative solution is to use architecture redesign with AWS CloudFront to reduce load on primary system (probbly web?).
> 2 votes

---
Tags: amazon-web-services, aws-lambda, amazon-rds
---


title: What does mapping in yml file means?

text:
# Question
Title: What does mapping in yml file means?
I am not sure what mapping in YAMl file means, can someone advise what does mapping in YAML file means?

# Answer
YAML is a very simple, text/human-readable annotation format that can be used to store data. YAML keeps data stored as a map containing keys and values associated to those keys. This map is in no particular order, so you can reorder it at will. Each pair is in the format KEY: VALUE When the value is a text string enclose the value with quotes. Some examples below:

```
KEY1: 'HELLO'
HOST_NUMBER: 1

```
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Where would DevOps Engineers mostly look for their next opportunity and what would it be?
Where do DevOp Engineers find their next opportunity? Users Groups, Associations, Job Boards?

What do DevOps look for in their next role?

# Answer
> 1 votes
Understand that the term **DevOps** refers to both a *Strategies* and *Tactics*.
For instance, the *strategies* of Continuous deployment and Consistent development/production environment.
The *tactics* may be using Kubernetics for deployment to the cluster with Docker for development/production environment.
With that said, a DevOps shop will likely use Kubernetics and Docker as their tactical tools and would look for people with those skillsets.

**ALSO**, they would look for people who are a CULTURAL fit who would work well in a collaborative environment. If your background is let's say operational and your not open to learning programming skills(or learn what the programmers need), don't take ownership of a project or work well in a team...you will **NOT** be a fit for the company (*you won't get hired*).

As far as the best place to find a DevOps related job?

I would look at going to a DevOps Meetup, go to trainings/seminars where DevOps tactics and tools are taught (i.e. Docker, Vagrant, Ansible, Kubernetics, etc.), look for certain keywords in the posts such as SRE (Site Reliability Engineer), Systems Automation, etc... and do some research on companies that are moving to or have embraced DevOps Strategy.

If you understand these things, I believe it will give you a great advantage at landing a job in one of those companies.

# Answer
> 1 votes
It can happen that your next DevOps job finds *you* if you have a well elaborated LinkedIn profile and patient enough to deal with and filter/process requests.

Though, my market observations are that many offerings are like:

* DevOps Engineers seen as "full stack developer 2.0", that is as an addon to specific developer skill set you can also setup automation pipelines and deploy to production.
* DevOps Engineers seen as "admin 2.0" to build up automated operational infrastructures
* DevOps Engineers seen as "teams glue" i.e. a company resource pool used to fire up slow/less successfull teams, here at least there is an understanding of how important beyond tech also culture/human communication is, too.
* Sometimes you get also a weird skill set list where nobody really knows where it comes from and what exactly the role is expected to deliver. Buzz wording detected.

Please consider also this post: Why shouldn't I try to hire a 'DevOps Engineer'?

# Answer
> 0 votes
Companies either have straight-away openings for DevOps Engineers, either under this specific title or (since the title is a bit misplaced if you really think about what it means in the opinion of some people); or you can look for positions that have some other title but mention DevOps and similar things (for example, "automation", "tooling", "CI/CD" etc.) in the description.

Or in other words - they find jobs the same way everybody else does. It's not *that* magical of a job description...

What they look for, in my experience from plenty of interviews, is simply working with all the tooling/automation we have these days, and most times with a focus on doing as little straightaway programming as possible (i.e., programming in the non-DevOps way; domain-based programming). This separates them mainly from developers who *also* can do the "DevOps" stuff.

---
Tags: azure-devops
---


# Question
Title: How to find interdependencies between pods in a Kubernetes cluster?
Two Pods run in a Kubernetes cluster. One is a simple Wordpress application and the other a Mysql database. The Wordpress Pod communicates with the Mysql database.

The aim is to find dependencies between pods. Is there any kubectl command or any tool like prometheus that is able to find interdependence between Pods?

# Answer
> 2 votes
Have a look to solutions implementing a service mesh.

For example ISTIO, Consul, Ambasador + Envoy...

# Answer
> 2 votes
Weave Scope is an observability tool that provides a dashboard that shows all network connections between pods.

https://github.com/weaveworks/scope

# Answer
> 1 votes
One way to find interdependencies is to check the services page when `kubectl proxy` has been issued and one navigates to http://localhost:8001. For example, if one deploys wordpress pods then a MySql pod and a wordpress pod will be created. As wordpress has to use a database like MySql one could see that the wordpress connects to the Mysql pod in the services page.

The help command of kubectl was consulted whether there is a subcommand that shows interdependencies, but that does not seem to be available.

# Answer
> 1 votes
Another option is to use a trace-based monitoring tool, eg something OpenTracing compatible. These will show requests as they flow through the entire system, allowing you to visualize dependencies.

---
Tags: kubernetes, prometheus
---


# Question
Title: Why am I unable to push docker image in gce (gcr.io)?
I created a new VM instance on my GCE compute engine (I have Owner permissions). On that instance, I installed git, docker, and gcloud - everything is done under `sudo su`.

I was hoping I can use that VM instance to deploy images to GCR, but whatever I do - **I cannot seem to have sufficient permissions to push** the images from that instance.

1. I granted "Storage Admin" along with the "Editor" permissions to the VM Instance (autogenerated) IAM service-account.
2. When I run `docker login eu.gcr.io` I receive `Login Succeeded` with no problems.
3. I copied both a p12 key and a .json key to the .ssh folder (I generated them on the IAM console) - but that had no use too.
4. To tag and push the docker image, I use this::

docker tag \[SOURCE\_IMAGE\] \[HOSTNAME\]/\[PROJECT-ID\]/\[IMAGE\]

and

```
docker push [HOSTNAME]/[PROJECT-ID]/[IMAGE]

```

Notice: It seems that at this stage (2018) `gcloud docker -- push` is obsolete (and it didn't work for me as well...)

Notice: The bucket is NOT created yet, so setting permissions on the Storage console is out of the question at this point.

What could be the reason for the Permission Denied problem ?

# Answer
> 1 votes
From the Before you begin section of the instructions you referenced:

> Make sure that you:
>
> 3. Have access to the registries which you will be pushing to and pulling from

This is specified in more details in Using Container Registry with Google Cloud Platform:

> To push private Docker images from a Compute Engine instance, your instance must have `read-write` or `full-control` permission to the image's bucket.

So the GCS bucket corresponding to `gcr.io` (or whichever GCR domain you want to use) and the desired cloud project must already be created and your GCE instance's service account must have the necessary role/permissions for push operations.

It's true, the instructions you referenced mention in Push the tagged image to Container Registry:

> When you push an image to a registry with a new hostname, Container Registry creates a storage bucket in the specified multi-regional location.

So I believe you'd have to:

* first create create the bucket by executing a push to the desired GCR domain and project, but:
    + from your own computer (or some machine which is not a GCE instance, but YMMV)
    + using the credentials of an acceptable identity, not the GCE instance service account's ones. I'd try with a real user credentials, I'm not sure if a service account's ones would cut it.
* after the bucket is created add to its access control the GCE instance service account's role/permissions for push operations
* then perform pushes from the GCE instance

# Answer
> 1 votes
I faced this same problem when I was trying to automate GCP kubernetes deployment using Jenkins by putting image in GCR and I created one service account giving admin permission in gcp IAM.

I launched one VM giving same IAM service account access.

I installed gcloud sdk using here follows below command.

$gcloud init

$gcloud auth configure-docker

$cd my\_api && docker build --no-cache -t my\_api .

$docker tag my\_api asia.gcr.io/firebase-mytalk/my\_api:latest

$gcloud docker -- push asia.gcr.io/firebase-mytalk/my\_api:latest

---
Tags: docker, google-compute-engine, registry
---


# Question
Title: Best practices for CI for embedded platforms (ESP32/Arduino)
Development processes for embedded software always lags behind that for web development. When looking at unit testing, automated builds and code analysis there are many limitations to popular tools.

From experience I have already dismissed the Arduino IDE to using platformIO with VS Code since, in regards to CI, it provides improved dependency installation and build configuration automation.

The question is what tools and processes already exist and which can be repurposed to be used within the Arduino framework to enable CI as with web development.

Meta: the next most relevant stack exchange group would be IoT, but not a single post mentions continuous integration

# Answer
> 6 votes
Since "deployment" to the IoT device, especially in bulk, will not be done using the CI system. Then the purpose of the CI system is mostly to make sure that it *will* work okay.

This means that you mostly want to run some tests using the CI. Automated testing is definitely possible, both for unit tests and for integration tests.

The most important thing for unit tests is usually very short runtime and a short feedback loop. For a large code base, if possible to compile parts of it using the native compiler of the host and run unit tests there - it is a great option. If you must, you can also cross-compile to the target platform and run the unit tests using some emulator to have the necessary speed. All of this can happen during a CI "build+test" job. Compiling for the target and running unit tests on the target will just be painfully slow if done properly, but can also be done - just make sure to only flash the smallest possible piece of code and have it report back via the Serial that it worked on the device.

For integration tests, if possible to install your code on an actual device and the have some chatter with the device using the serial interface - you can ensure that the code actually works on a target device. If for some reason the code fails or works incorrectly, maybe draws too much power and/or restarts then the integration tests will catch that. You can also have another test device performing measurements and reporting via the Serial interface to the CI job. Can use an Arduino, or another ESP to maybe measure voltage and/or amperage during the execution of your code parts and checking that against known good values. You don't expect the device to pull 400mA when in a deep-sleep mode, for example, this is easy to test with a "test harness".

All of these unit tests and integration tests can be triggered completely automatically on the CI server. Naturally, you will probably have this "server" running on-premises and not rely on a cloud solution that might or might not support your latest integration testing.

With hardware, smoke tests are more "real", since you can actually hook-up the device and see the magic smoke come out. Usually, you don't even need to run code before it burns since the problem is often in the hardware.

Since I mentioned testing, there is also integration testing which is done on the hardware level. If you have a good test harness that checks that the device works well by probing it, you can use the same test harness to do your QC cycle in the factory where they produce the devices later. There is quite a lot of information about making these testbeds. One example for this is here - https://www.sparkfun.com/tutorials/138.

# Answer
> 2 votes
From your comment

> I was hoping for unit tests of target code, some analysis à la valgrind all the way to CD

I had an answer to the Arduino StackExchange question that was mentioned, which addresses some of this. But since writing that answer I've added some memory-checking features to `arduino_ci` \[of which, full disclosure, I am the author\].

Although it doesn't do static analysis, it can detect certain kinds of memory problems while running unit tests:

```
==2284==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000000280 at pc
0x0001032b987e bp 0x7ffeec94c0b0 sp 0x7ffeec94c0a8
READ of size 1 at 0x603000000280 thread T0
    #0 0x1032b987d in Adafruit_WS2801::show() Adafruit_WS2801.cpp:203
    #1 0x1032b84b1 in Strip::init(unsigned char, unsigned char) Strip.cpp:10
    #2 0x1032baaff in test_set_strip_values::task() strip.cpp:10
    #3 0x1032bd900 in Test::test() ArduinoUnitTests.h:165
    #4 0x1032bcfd4 in Test::run(Test::ReporterTAP*) ArduinoUnitTests.h:137
    #5 0x1032bcb79 in Test::run_and_report(int, char**) ArduinoUnitTests.h:155
    #6 0x1032bcab8 in main strip.cpp:43
    #7 0x7fff769d0114 in start (libdyld.dylib:x86_64+0x1114)

```

More context in the pull request that fixes this problem

---
Tags: continuous-integration, testing, configuration-management
---


title: Build Design - Docker vs Fedora's Mock

text:
# Question
Title: Build Design - Docker vs Fedora's Mock
**This post is migrated from a reddit post. Parts in bold are to adopt it to SE's format.**

I'm talking with companies to help streamline/combine their build process. They must do it together and their current processes are scary. We've convinced them to go with AWS/jenkins/bitbucket/artifactory.

Now one company wants to use fedora's mock for their build environment deps **and** build artifact management. Another wants to use docker for their build environment but **does not** have a build artifact solution yet.

I suggested a combination of docker (for build env deps) and mock (for product related artifacts and deps) to both. They scoffed at it. BTW one of the products is a custom rhel .iso.

* What are the pros/cons for mock vs docker in the way the two companies are debating using them?
* What about pros/cons for my proposal.. or if it is even a smart thing to do?
* Any other suggested tools/build flow that may work better?

**Ultimately I am interested in what is the best practice/most adopted method for doing this (build env deps mgmt/product artifact dep mgmt) in the industry that will have continued support and advancements?**

# Answer
Docker is going to be far more well-known than Mock. I consistently see Docker used during build in companies that I work for. I think I have seen Mock before, but only because I am a Fedora user.

However, Docker is not a build tool in of itself, so you can still use Mock inside a container. This is a great way to avoid the "snowflake" build server. By using Docker containers, your builds are consistent no matter what build tool you are using. You can standardise on Docker as a platform for builds, and leave developers to choose the best build tool (whether that's Mock or NPM).

The only other suggestion I'd have is to run the builds on a CI server, and Docker makes this really easy.

Here's one example of a company using Docker for builds (as well as tests and deploys).
> 2 votes

---
Tags: docker, docker-compose, builds, dependency-management
---


# Question
Title: Access service in remote Kubernetes cluster using ingress
I'm attempting to access a service in an existing kubernetes cluster deployed in a remote machine. I've configured the cluster to be accessible through `kubectl` from my local mac.

```
$ kubectl cluster-info
Kubernetes master is running at https://192.168.58.114:6443
KubeDNS is running at https://192.168.58.114:6443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy

```

The ingress configuration for the service I want to connect is:

```
kind: Ingress
apiVersion: extensions/v1beta1
metadata:
  name: gw-ingress
  namespace: vick-system
  selfLink: /apis/extensions/v1beta1/namespaces/vick-system/ingresses/gw-ingress
  uid: 52b62da6-01c1-11e9-9f59-fa163eb296d8
  resourceVersion: '2695'
  generation: 1
  creationTimestamp: '2018-12-17T06:02:23Z'
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"kubernetes.io/ingress.class":"nginx","nginx.ingress.kubernetes.io/affinity":"cookie","nginx.ingress.kubernetes.io/session-cookie-hash":"sha1","nginx.ingress.kubernetes.io/session-cookie-name":"route"},"name":"gw-ingress","namespace":"vick-system"},"spec":{"rules":[{"host":"wso2-apim-gateway","http":{"paths":[{"backend":{"serviceName":"gateway","servicePort":8280},"path":"/"}]}}],"tls":[{"hosts":["wso2-apim-gateway"]}]}}
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/affinity: cookie
    nginx.ingress.kubernetes.io/session-cookie-hash: sha1
    nginx.ingress.kubernetes.io/session-cookie-name: route
spec:
  tls:
    - hosts:
        - wso2-apim-gateway
  rules:
    - host: wso2-apim-gateway
      http:
        paths:
          - path: /
            backend:
              serviceName: gateway
              servicePort: 8280
status:
  loadBalancer:
    ingress:
      - ip: 172.17.17.100

```

My list of services are:

My `/etc/hosts` file looks like below:

```
##
# Host Database
#
# localhost is used to configure the loopback interface
# when the system is booting.  Do not change this entry.
##
127.0.0.1   localhost
255.255.255.255 broadcasthost
::1             localhost

172.17.17.100 wso2-apim-gateway wso2-apim wso2sp-dashboard

```

What is the URL I should use to access this service from my local browser? Should I do any more configurations?

# Answer
> 3 votes
`https://wso2-apim-gateway/` is the URL that will point to the `gateway` service according to your configuration. This is defined by the `host: wso2-apim-gateway` part of your configuration. Note that your load balancer (`172.17.17.100`) is a private IP address, so you will need to be on the same network to access it.

I can't see any further configuration required, except perhaps TLS certifications - if you receive a 5xx response, you may want to check the logs of the ingress-nginx deployment/replicaset.

---
Tags: kubernetes
---


# Question
Title: Push Jenkins Build status to Github and Trigger Jenkins build from Github Webhook
FYI - New to DevOps. Please bear with me. Thanks

I have a webhook created by Admin on Github Repo to trigger Jenkins build. I created a token from my Github(I am not admin on the repo) and used it in Github Connection in Jenkins Configure.

Now if I commit when the token credentials are selected in Github in Jenkins Config, the Jenkins build is not triggered automatically. When I click on Build now the job is triggered and job status is pushed to Github.

If I don't select any credentials in Github in Jenkins Config, the jenkins build is triggered on commit but the build status is not reflecting in Github.

There is a manage hooks options also in Github in Jenkins Configure but that probably would require a token from Admin on repo?

Can you guys guide me on what should I do now to 1. Trigger the Jenkins Build on commit. 2. Push the build status to Github.

Thanks

# Answer
Triggering Jenkins using GitHub webhook should be a straight forward process. This article would help you.
> 0 votes

# Answer
Found the issue. Certificates were needed to be installed which was blocking connection to Github Enterprise. Once I installed those, the setup is working fine now.

Thanks Everyone. :)
> -1 votes

---
Tags: jenkins, jenkins-plugins, github
---


# Question
Title: Why can the certificate not be found by the Ingress Controller, while the certificate has been issued?
kubectl logs nginx-ingress-controller-X -n nginx-ingress

```
W1222 06:52:04.575115       8 controller.go:1058] Error getting SSL
certificate "app/app-tls": local SSL certificate app/app-tls was not
found. Using default certificate
W1222 06:52:09.955475       8 backend_ssl.go:48] Error obtaining X.509
certificate: no object matching key "app/app-tls" in local store

```

kubectl describe certificates nexus-tls -n cert-manager

```
Message:               Certificate issued successfully
Reason:                CertIssued

```

**Discussion**

It is unclear to me where to deploy the certmanager. I have deployed in a separate namespace, the cert has been created successfully, but the certificate cannot be found.

# Answer
> 2 votes
https://github.com/jetstack/cert-manager/issues/482#issuecomment-449664935

> Once I followed this https://github.com/jetstack/cert-manager/blob/master/docs/tutorials/quick-start/index.rst#step-6---configure-lets-encrypt-issuer and deployed the issuers in the namespace of the app, it works

---
Tags: kubernetes, cert-manager
---


# Question
Title: Best practice for Postgres setup with docker-compose
I have a docker-compose script which starts a Postgres container and an app container like so

```
db:
  image: postgres:10.6
  restart: always
  ports:
    - 5432:5432
app:
  image: test_app:latest
  restart: always
  depends_on:
    - db

```

This all works fine and 'app' can talk to 'db'. However, the postgres server is in a basically unusable state as it has no configuration with regards to having databases, users, etc. setup (apart from the defaults). It appears that the recommended way of fixing this is to create your own 'db image' pulling from postgres and running some shell script to take care of business (and then your composition uses this image), but this seems very hacky and convoluted to me.

Does anyone have a better way of setting up a postgres server with a known configuration inside a docker-compose based system?

# Answer
> 5 votes
First, use alpine images if possible.

Second, use ENV variables for prepare database configuration. More info.

Third, use `/docker-entrypoint-initdb.d` directory for extend image. More info.

Example stack with devlopment environment.

```
version: '3.2'
services:
  example:
    image: "${EXAMPLE_SERVICE}:${EXAMPLE_SERVICE__VERSION}-development"
    build:
      context: .
      dockerfile: Dockerfile.development
    networks:
      - net
    ports:
      - 5000:5000
    depends_on:
      - example__migrator
    stdin_open: true
    tty: true
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
      FLASK_APP: app.py
      FLASK_DEBUG: 1
    volumes:
      - .:/home/example
  example__migrator:
    image: "${EXAMPLE_SERVICE}:${EXAMPLE_SERVICE__VERSION}-development"
    build:
      context: .
    command: sh -c './wait-for pg.example_net:5432 -- python manage.py db upgrade'
    networks:
      - net
    depends_on:
      - pg
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
  pg:
    image: postgres:10.0-alpine
    networks:
      - net
    ports:
      - 5432:5432
    volumes:
      - pg_data:/var/lib/postgresql/data/pg_data
    environment:
      POSTGRES_PASSWORD: example
      POSTGRES_DB: example
      POSTGRES_USER: example
      POSTGRES_HOST: pg.example_net
      POSTGRES_PORT: 5432
      PGDATA: /var/lib/postgresql/data/pg_data
networks:
  net:
volumes:
  pg_data:

```

---
Tags: docker, docker-compose, postgresql
---


# Question
Title: How do I install Terraform libvrit_cloudinit?
I have searched all day trying to find a document on how to install the `libvirt_cloudinit` provider for Terraform.

When I execute a `terraform plan` I encounter this error message:

```
Error: libvirt_cloudinit.commoninit: Provider doesn't support resource: libvirt_cloudinit

```

# Answer
Try to use libvirt\_cloudinit\_disk instead of libvirt\_cloudinit. I think it is deprecated.
> 1 votes

---
Tags: terraform, cloud-init
---


title: Stolon vs CrunchyData for HA Postgresql

text:
# Question
Title: Stolon vs CrunchyData for HA Postgresql
I'm using **helm** charts with **kubernetes** and I'm trying to deploy High Availability Postgresql database and found this article mentioning three project that provide HA Postgresql:

**1- Stolon**

**2- CrunchyData**

**3- Patroni**

I didn't find helm chart example for Patroni, but I installed both helm charts for Stolon and Crunchy. However, I wasn't able to find the differences between them except in the structure, both use statefulset from kubernetes in order to deploy master service and a number of replicas(ReadOnly slaves),stolon has this proxy service that points to the master service which is like an extra layer, makes the stolon look a little more complex than crunchy.

But what I really need to know is:

1. How the replicas will synchronize with the master? synchronous or
asynchronous.
2. Whats the failover mechanism provided if any?
3. persistence volumes why is there a volume per pod? How is this related to the replication?

# Answer
> 2 votes
I ended up using stolon for its automatic leader election and fail over handling. Crunchy Data offers small building blocks so there is a lot of work to do to put them all together while stolon has a ready helm chart. But I didn't try patroni since stolon worked just fine.

---
Tags: kubernetes, helm, postgresql
---