# Question
Title: Downloading Docker Images from Docker Hub without using Docker
I want to manually download a Docker Image from Docker Hub. More specifically, I want to download a Docker Image from Docker Hub on a machine in a restricted environment which does not (and cannot) have the Docker client software installed. I would have thought that this would be possible using the official API, but this does not appear to be the case - see the following discussion:

Is it really the case that the API doesn't support downloading images? Is there a way to work around this?

---

**UPDATE 1:**

I came across the following ServerFault post:

The accepted solution uses the `docker save` command, which doesn't help in my situation. But another solution posted there cites the following StackOverflow post:

One of the solutions there refers to a command-line tool called docker-registry-debug which, among other things, can generate a `curl` command for downloading an image. Here is what I got:

```
user@host:~$ docker-registry-debug curlme docker ubuntu

# Reading user/passwd from env var "USER_CREDS"
# No password provided, disabling auth
# Getting token from https://index.docker.io
# Got registry endpoint from the server: https://registry-1.docker.io
# Got token: signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read
curl -i --location-trusted -I -X GET -H "Authorization: Token signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read" https://registry-1.docker.io/v1/images/ubuntu/layer

user@host:~$ curl \
-i --location-trusted -I -X GET \
-H "Authorization: Token signature=1234567890abcde1234567890abcde1234567890,repository="library/docker",access=read"

https://registry-1.docker.io/v1/images/ubuntu/layer
HTTP/1.1 404 NOT FOUND
Server: gunicorn/18.0
Date: Wed, 29 Nov 2017 01:00:00 GMT
Expires: -1
Content-Type: application/json
Pragma: no-cache
Cache-Control: no-cache
Content-Length: 29
X-Docker-Registry-Version: 0.8.15
X-Docker-Registry-Config: common
Strict-Transport-Security: max-age=31536000

```

So unfortunately it looks like the `curl` command generated does not work.

---

**UPDATE 2:**

It looks like I'm able to download layer blobs from Docker Hub. Here is how I'm currently going about it.

Get an authorization token:

```
user@host:~$ export TOKEN=\
"$(curl \
--silent \
--header 'GET' \
"https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/ubuntu:pull" \
| jq -r '.token' \
)"

```

Pull an image manifest:

```
user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq '.'

```

Pull an image manifest and extract the blob sums:

```
user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq -r '.fsLayers[].blobSum'

sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
sha256:be588e74bd348ce48bb7161350f4b9d783c331f37a853a80b0b4abc0a33c569e
sha256:e4ce6c3651b3a090bb43688f512f687ea6e3e533132bcbc4a83fb97e7046cea3
sha256:421e436b5f80d876128b74139531693be9b4e59e4f1081c9a3c379c95094e375
sha256:4c7380416e7816a5ab1f840482c9c3ca8de58c6f3ee7f95e55ad299abbfe599f
sha256:660c48dd555dcbfdfe19c80a30f557ac57a15f595250e67bfad1e5663c1725bb

```

Download a single layer blob and write it to a file:

```
user@host:~$ BLOBSUM=\
"sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"

user@host:~$ curl \
--silent \
--location \
--request GET \
--header "Authorization: Bearer ${TOKEN}" \
"https://registry-1.docker.io/v2/library/ubuntu/blobs/${BLOBSUM}" \
> "${BLOBSUM/*:/}.gz"

```

Write all of the blob sums to a file:

```
user@host:~$ curl \
--silent \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
'https://registry-1.docker.io/v2/library/ubuntu/manifests/latest' \
| jq -r '.fsLayers[].blobSum' > ubuntu-blobsums.txt

```

Download all of the layer blobs from the manifest:

```
user@host:~$ while read BLOBSUM; do
curl \
--silent \
--location \
--request 'GET' \
--header "Authorization: Bearer ${TOKEN}" \
"https://registry-1.docker.io/v2/library/ubuntu/blobs/${BLOBSUM}" \
> "${BLOBSUM/*:/}.gz"; \
done < blobsums.txt

```

Now I have a bunch of layer blobs and I need to recombine them into an image - I think.

---

Related Links:

# Answer
> 54 votes
It turned out that the Moby Project has a shell script on the Moby Github which can download images from Docker Hub in a format that can be imported into Docker:

The usage syntax for the script is given by the following:

```
download-frozen-image-v2.sh target_dir image[:tag][@digest] ...

```

The image can then be imported with `tar` and `docker load`:

```
tar -cC 'target_dir' . | docker load

```

To verify that the script works as expected, I downloaded an Ubuntu image from Docker Hub and loaded it into Docker:

```
user@host:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user@host:~$ tar -cC 'ubuntu' . | docker load
user@host:~$ docker run --rm -ti ubuntu bash
root@1dd5e62113b9:/#

```

In practice I would have to first copy the data from the internet client (which does *not* have Docker installed) to the target/destination machine (which *does* have Docker installed):

```
user@nodocker:~$ bash download-frozen-image-v2.sh ubuntu ubuntu:latest
user@nodocker:~$ tar -C 'ubuntu' -cf 'ubuntu.tar' .
user@nodocker:~$ scp ubuntu.tar user@hasdocker:~

```

and then load and use the image on the target host:

```
user@hasdocker:~ docker load ubuntu.tar
user@hasdocker:~ docker run --rm -ti ubuntu bash
root@1dd5e62113b9:/#

```

# Answer
> 20 votes
There is a tool called Skopeo which can retrieve Docker images from a repository and save them in several formats.

For example:

1. Download the image and save the layers as a tarball: `skopeo copy docker://ubuntu docker-archive:/tmp/ubuntu.tar:ubuntu`
2. Transfer `/tmp/ubuntu.tar` to another machine if desired.
3. Load the image on a Docker instance which does not have internet connection: `docker load --input /tmp/ubuntu.tar`

It is available in CentOS 7 repo with the package name `skopeo`. There are no Debian or Ubuntu packages at this time (but it is easy to compile).

# Answer
> 5 votes
thanks for motivation. I made a powershell version of it. Check it out... With it you can move in dockerhub containers to a restricted docker networks with a windows desktop and an ssh-scp tool to docker machine without root or administrator rights

https://gitlab.com/Jancsoj78/dockerless\_docker\_downloader a new hacker tool :)

```
$image = "ubuntu"
$tag = "latest"
$imageuri = "https://auth.docker.io/token?service=registry.docker.io&scope=repository:library/"+$image+":pull"
$taguri = "https://registry-1.docker.io/v2/library/"+$image+"/manifests/"+$tag
$bloburi = "https://registry-1.docker.io/v2/library/"+$image+"/blobs/sha256:"

#token request
$token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token

#pull image manifest
$blobs = $($(Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $taguri | ConvertFrom-Json | Select -expand fsLayers ) -replace "sha256:" -replace "@{blobSum=" -replace "}")

#download blobs
for ($i=0; $i -lt $blobs.length; $i++) {
    $blobelement =$blobs[$i]

    Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$blobelement -OutFile blobtmp

    $source = "blobtmp"
    $newfile = "$blobelement.gz"

#overwrite
Copy-Item $source $newfile -Force -Recurse
#source blobs
ls *.gz
}
#postprocess
echo "copy these .gz to your docker machine"
echo "docker import .gz backward one by one"
echo "lastone with ubuntu:latest"
echo "after docker export and reimport to make a simple layer image"

```

# Answer
> 3 votes
Here is an adapted python script, thus having an OS independant solution: docker-drag

Use it like that, and it will create a TAR archive that you will be able to import using docker load :

```
python docker_pull.py hello-world
python docker_pull.py alpine:3.9
python docker_pull.py kalilinux/kali-linux-docker

```

# Answer
> 1 votes
To me it is not completely clear what you are trying to achieve and why the attempts are not a solution for the problem. If I would need to solve this issue I would like @Tensibai and other Q&As indicated, do a docker pull first on a system with internet connectivity, save the docker image, copy it to the machine without internet connectivity, load the image and run it.

**Demonstration**

There are no images on system A:

```
userA@systemA ~ $ docker images
REPOSITORY        TAG               IMAGE ID          CREATED             SIZE
userA@systemA ~ $

```

Pull an image from dockerhub:

```
userA@systemA ~ $
docker pull nginx
Using default tag: latest
latest: Pulling from library/nginx
bc95e04b23c0: Pull complete
f3186e650f4e: Pull complete
9ac7d6621708: Pull complete
Digest: sha256:b81f317384d7388708a498555c28a7cce778a8f291d90021208b3eba3fe74887
Status: Downloaded newer image for nginx:latest
userA@systemA ~ $ docker images
REPOSITORY        TAG               IMAGE ID            CREATED             SIZE
nginx             latest            9e7424e5dbae        10 days ago         108MB

```

Save docker image:

```
userA@systemA ~ $ docker save nginx -o nginx.tar

```

Copy docker image to systemB and load it.

```
userB@systemB ~ $ docker load -i nginx.tar
cec7521cdf36: Loading layer  58.44MB/58.44MB
350d50e58b6c: Loading layer  53.76MB/53.76MB
63c39cd4a775: Loading layer  3.584kB/3.584kB
Loaded image: nginx:latest
userB@systemB ~ $ docker images
REPOSITORY        TAG               IMAGE ID            CREATED             SIZE
nginx             latest            9e7424e5dbae        10 days ago         108MB

```

# Answer
> 1 votes
I didn't really understand Jancsó's way of postprocessing, so I've spent some time to modify his script and here's what I came with:

https://github.com/meetyourturik/dockerless-docker-downloader

upd: apparently, 'a link only answer...' is something bad, so here's a whole script:

```
# Workaround for SelfSigned Cert an force TLS 1.2
add-type @"
    using System.Net;
    using System.Security.Cryptography.X509Certificates;
    public class TrustAllCertsPolicy : ICertificatePolicy {
        public bool CheckValidationResult(
        ServicePoint srvPoint, X509Certificate certificate,
        WebRequest request, int certificateProblem) {
            return true;
        }
    }
"@
[System.Net.ServicePointManager]::CertificatePolicy = New-Object TrustAllCertsPolicy
[System.Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

# use 'library/' prefix for 'official' images like postgres
$image = "atlassian/jira-software"
$tag = "8.13.2"

$imageuri = "https://auth.docker.io/token?service=registry.docker.io&scope=repository:${image}:pull"
$taguri = "https://registry-1.docker.io/v2/${image}/manifests/${tag}"
$bloburi = "https://registry-1.docker.io/v2/${image}/blobs/"

# generating folder to save image files
$path = "$image$tag" -replace '[\\/":*?<>|]'
if (!(test-path $path)) {
    New-Item -ItemType Directory -Force -Path $path
}

# token request
$token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token

# getting image manifest
$headers = @{}
$headers.add("Authorization", "Bearer $token")
# this header is needed to get manifest in correct format: https://docs.docker.com/registry/spec/manifest-v2-2/
$headers.add("Accept", "application/vnd.docker.distribution.manifest.v2+json")
$manifest = Invoke-Webrequest -Headers $headers -Method GET -Uri $taguri | ConvertFrom-Json

# downloading config json
$configSha = $manifest | Select -expand config | Select -expand digest
$config = ".\$path\config.json"
Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$configSha -OutFile $config

# generating manifest.json
$manifestJson = @{}
$manifestJson.add("Config", "config.json")
$manifestJson.add("RepoTags",@("${image}:${tag}"))

# downloading layers
$layers = $manifest | Select -expand layers | Select -expand digest
$blobtmp = ".\$path\blobtmp"

#downloading blobs
$layersJson = @()
foreach ($blobelement in $layers) {
    # making so name doesnt start with 'sha256:'
    $fileName = "$blobelement.gz" -replace 'sha256:'
    $newfile = ".\$path\$fileName"
    $layersJson += @($fileName)

    # token expired after 5 minutes, so requesting new one for every blob just in case
    $token = Invoke-WebRequest -Uri $imageuri | ConvertFrom-Json | Select -expand token

    Invoke-Webrequest -Headers @{Authorization="Bearer $token"} -Method GET -Uri $bloburi$blobelement -OutFile $blobtmp

    Copy-Item $blobtmp $newfile -Force -Recurse
}

# removing temporary blob
Remove-Item $blobtmp

# saving manifest.json
$manifestJson.add("Layers", $layersJson)
ConvertTo-Json -Depth 5 -InputObject @($manifestJson) | Out-File -Encoding ascii ".\$path\manifest.json"

# postprocessing
echo "copy generated folder to your docker machine"
echo "tar -cvf imagename.tar *"
echo "docker load < imagename.tar"

```

after sctipt downloads blobs and generates config and manifest jsons download it to docker machine and execute two following commands:

```
tar -cvf imagename.tar *
docker load < imagename.tar

```

first creates an archive, 2nd uploads image archive to docker

# Answer
> 0 votes
Another tool that could be used to download docker images is P2IWD.

# Answer
> -1 votes
If your company has nexus or a similar repository, they may already have set up nexus to pull in what you need. You could try adding your repo's host/ip in front of the relative docker hub path.

Examples:

```
docker pull nexus.example.com:18443/nginx
docker pull nexus.example.com:18443/selenium/node-chrome
docker pull nexus.example.com:18443/postgres:9.4

```

---
Tags: docker, dockerhub
---


# Question
Title: Best practice put nginx + django in AWS subnet private?public?
I want to put the `fargate` container which contains `django` \+ `nginx`

However what subnet should be used for?

At first I put this in public subnet, of course it works well.

However, is it possible to put this in private subnet with `NAT gateway`?

If so, How can users access to the private subnet from outside?

A few users(admin) need to access to this django, it doesn't need to get access from other users.

# Answer
> 0 votes
You would put it ideally into a private subnet behind an AWS Load balancer of some sort. The benefit of this is that any security issues in your nginx installation are offloaded to AWS, therefore are no longer your concern.

Some detractions:

* slightly more complex setup
* minimum cost of ~25USD per month for load balancer (minimum cost of around 25USD per 25 client certificates if you terminate them on Application Load Balancer)

People would access the nginx servers via a jump/ bastion hosts. Access to this host would probably be limited by IP address and/or VPN.

The AWS provided NAT gateway does not allow inbound traffic, unless it was initiated by outbound traffic.

---
Tags: amazon-web-services, fargate, aws-subnets
---


title: Merge trains in Azure DevOps

text:
# Question
Title: Merge trains in Azure DevOps
Is it possible to somehow implement a merge trains in Azure DevOps, similar to what GitLab does?

> Merge train example
>
> Three merge requests (A, B and C) are added to a merge train in order, which creates three merged results pipelines that run in parallel:
>
> ```
> The first pipeline runs on the changes from A combined with the target branch.
> The second pipeline runs on the changes from A and B combined with the target branch.
> The third pipeline runs on the changes from A, B, and C combined with the target branch.
>
> ```

# Answer
No, Azure DevOps does not support merge trains. I think in your case, you can create a new branch (like a release branch) from the target one. Then merge your PRs one by one into it. If all goes well then create one PR from the release branch to the target branch.
> 2 votes

---
Tags: azure-devops, azure, azure-pipelines
---


# Question
Title: How can I grant access to my cloud storage for clients?
I am developing a webshop for large digital products (multiple .tif image files). These are currently stored in an S3 bucket.

What is a recommended way to grant access to my customers once they have purchased the product? I am trying to determine how to grant access to clients who may not have their own S3 account (otherwise I could set up an IAM Policy to suit), so they can access a specific folder and download all files within. Adding password protection is important, I don't want to store the data and make it public access.

Would Google Cloud Services have additional features which might meet this requirement? Could I use an online cloud storage browser to grant access to customers without S3 details?

# Answer
> 1 votes
Use **S3 Pre-signed URLs** \- here's a demo: https://aws.nz/best-practice/s3-presigned-url/

When the customers purchase the file you'll generate a unique pre-signed URL for the file in S3 that will expire in, let's say 1 hour. You don't have to keep track of it or record it anywhere, simply generate the URL and give it to the customer. If they need to download the file again later on and still have a valid subscription (if that's a concept on your website) you can generate them a new Pre-signed URL on demand.

Don't make the expiration too long to prevent a risk of leaking through forums, reddit, etc. It's better to make it short and generate on-demand when they need it.

Hope that helps :)

---
Tags: google-cloud-platform, amazon-s3
---


title: Healthcheck: CMD vs CMD-SHELL

text:
# Question
Title: Healthcheck: CMD vs CMD-SHELL
It's not clear to me what are the differences between `CMD` and `CMD-SHELL` in Docker Compose healthcheck.

What are the benefits of using `CMD-SHELL` compared to `CMD`? What are the use cases for `CMD` and `CMD-SHELL`?

# Answer
Docker can either run the command using `CMD` which is the equivalent of the OS exec syscall directly in the kernel, or it can run using `CMD-SHELL` which runs a shell to call your command *(typically `/bin/sh`.)*

A shell provides functionality you are accustomed to from the command line, including:

* I/O redirection and pipes (`>`, `>>`, `<`, etc)
* Command chaining (`&&` and `||`)
* Variable expansion (`$var`)

However, a shell may also intercept signals, particularly when running as pid 1, which can prevent a graceful shutdown of a container *(not much of an issue for a `healthcheck`).*

In addition, to run a the shell syntax, your container's image must include a shell, which means that won't work for images based on `scratch` that do not include a `/bin/sh`.
> 23 votes

# Answer
As per Docker Documentation

> RUN has 2 forms:
>
> * `RUN <command>` (*shell* form, the command is run in a shell, which by default is `/bin/sh -c` on Linux or `cmd /S /C` on Windows)
> * `RUN ["executable", "param1", "param2"]` (*exec* form)
>
> The `RUN` instruction will execute any commands in a new layer on top of the current image and commit the results. The resulting committed image will be used for the next step in the `Dockerfile`.
>
> Layering `RUN` instructions and generating commits conforms to the core concepts of Docker where commits are cheap and containers can be created from any point in an image’s history, much like source control.
>
> The *exec* form makes it possible to avoid shell string munging, and to `RUN` commands using a base image that does not contain the specified shell executable.

And the second form, `exec` was created for cases when there is not any `shell` in containers.

There is appropriate issue for healthchecks: Use CMD instead of CMD-SHELL for --healthcheck-cmd by vdemeester · Pull Request #28679 · moby/moby

Motivation for the issue is clear: Use CMD instead of CMD-SHELL for --healthcheck-cmd by vdemeester · Pull Request #28679 · moby/moby

UPD: both the issue and the PR are closed without fix.

> Using `CMD-SHELL` will not work on any image that do not ship with a
> shell (`/bin/sh`) and/or that do not define a default shell using
> `SHELL` in `Dockerfile`.
> 6 votes

# Answer
Based on docs for healthcheck both of them are equal with a tiny difference:

```
healthcheck:
  test: ["CMD", "curl", "-f", "http://localhost"]
healthcheck:
  test: ["CMD-SHELL", "curl -f http://localhost || exit 1"]

```

As you see there is no difference between them. the former form - `CMD` \- just allow you to pass params separately.
> 2 votes

---
Tags: docker, docker-compose
---


# Question
Title: Kubernetes Failing with Self Signed Docker Registry Certificate
We have a private Docker registry and we're attempting to get a our Kubernetes cluster to pull images from it, but it's failing with the following error:

```
  Type     Reason     Age                From               Message
  ----     ------     ----               ----               -------
  Normal   Scheduled  104s               default-scheduler  Successfully assigned namespace/service-8599c54df8-592wm to fat2
  Normal   Pulling    67s                kubelet            Pulling image "192.168.100.178:5000/service:ds-1.2"
  Warning  Failed     67s                kubelet            Error: ErrImagePull
  Warning  Failed     67s                kubelet            Failed to pull image "192.168.100.178:5000/service:ds-1.2": rpc error: code = Unknown desc = failed to pull and unpack image "192.168.100.178:5000/service:ds-1.2": failed to resolve reference "192.168.100.178:5000/service:ds-1.2": failed to do request: Head "https://192.168.100.178:5000/v2/service/manifests/ds-1.2": x509: certificate signed by unknown authority (possibly because of "x509: invalid signature: parent certificate cannot sign this kind of certificate" while trying to verify candidate authority certificate "192.168.100.174: hostname")
  Warning  Failed     65s (x2 over 66s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    65s (x2 over 66s)  kubelet            Back-off pulling image "192.168.100.178:5000/service:ds-1.2"

```

I have copied the certificates from the Docker registry to `/etc/docker/certs.d/192.168.100.178:5000/` and I have copied the secret verbatim from another cluster that has absolutely no problem pulling images from our Docker registry. Regardless, here it is (as a Helm template):

```
apiVersion: v1
kind: Secret
metadata:
  name: {{ .Values.global.image_pull_secret }}
  namespace: {{ .Values.global.namespace }}
  labels:
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/release-name: {{ .Release.Name }}
    app.kubernetes.io/release-namespace: {{ .Release.Namespace }}
data:
  .dockerconfigjson: {snipped}
type: kubernetes.io/dockerconfigjson

```

I have verified I can successfully connect with a `docker login`:

```
$ docker login https://192.168.100.178:5000
Username: myuser
Password:
WARNING! Your password will be stored unencrypted in /home/myuser/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

```

I have completely run out of ideas here, but it seems the new cluster doesn't trust the CA that signed the certificate from the Docker registry. I'm at my wits end, so any help here is greatly appreciated.

# Answer
> 0 votes
Finally got it working. I had to install the root certificate by copying it to `/etc/ssl/tls/certs`. This is the root certificate that was used when creating the certificate for the Docker registry. In the error message, this is the certificate for the CA, which is referenced at the end: `"192.168.100.174: hostname"`.

---
Tags: kubernetes, docker-registry
---


# Question
Title: Use one Helm chart for ALL microservices?
This is a follow up question to my post Parameterized Kubernetes manifest files?.

I understand more about Helm now, after lots of reading, including the Learn Helm book (which I recommend). I still have this understanding that each microservice will have its OWN Helm chart? This defeats the purpose of what I wanted, where I just have ONE helm chart, and every time I want to use it, I just update the `Chart.yml` and `values.yml` files for each microservice, everything else is the same. For example, the microservice name, versions, and repo will change of course per microservice. Seems like I'm looking for a "template" for `Chart.yaml` and `values.yaml`. Is this possible to do? I don't know if it's recommended or not (I may not care).

I come from a CloudFoormation background, where I just have ONE parameterized CloudFormation template, and I just pass it parameters pertinent to each microservice.

**UPDATE:** I'm aware of Helm library charts, and I don't think they can provide the functionality I want?

# Answer
> 15 votes
Subcharts are the way to go.

I've done a very similar thing as follows:

* Create one or more base charts which cover microservices that have similar configuration (e.g. one for backend microservices, another for frontend servers).
* Create an overall chart for your application. This may be a bare bones chart without any resource configurations, but just a `Chart.yaml` and `values.yaml`. Or, you may also include some resources specific to your application overall and not a particular microservice (e.g. networkpolicy or ingress).

You could put everything in one git repo e.g.:

```
my-deployment-repo/
|- base-microservice/
   |- templates/
      |- deployment.yml
      |- service.yml
   |- Chart.yaml
   |- values.yaml
|- base-ui/
   |- templates/
      |- deployment.yml
      |- service.yml
   |- Chart.yaml
   |- values.yaml
|- myapp/
   |- Chart.yaml
   |- values.yaml

```

Then, your overall application chart (here `myapp`) can include the base charts multiple times as a dependency in its `Chart.yaml`. If you are putting all these charts in one git repo, you can use relative paths to point to them. E.g.

```
# Chart.yaml
dependencies:
  - alias: my-microservice-1
    name: base-microservice
    version: "0.1.0"
    repository: file://../base-microservice
  - alias: my-microservice-2
    name: base-microservice
    version: "0.1.0"
    repository: file://../base-microservice
  - alias: my-ui-1
    name: base-ui
    version: "0.1.0"
    repository: file://../base-ui
  - alias: my-ui-2
    name: base-ui
    version: "0.1.0"
    repository: file://../base-ui

```

The key thing here is `alias`. This allows charts to be depended on multiple times. It sets the name of the chart to the alias, so within the subcharts you can use `{{.Chart.Name}}`.

Finally, in `myapp`'s `values.yaml`, you can pass different values to the subcharts under their alias key, e.g. if you have used `{{.Values.image}}` in base-microservice, you can do the following in `myapp`'s `values.yaml`:

```
# values.yaml
my-microservice-1:
  image: foo

my-microservice-2:
  image: bar

```

When it comes to deploying `myapp`, run the following commands from within the `myapp` directory:

```
helm dependency update
helm install myapp .

```

You must always run `helm dependency update` before installing/upgrading a chart if any of the subcharts have changed.

I recommend git ignoring .tgz files, so you don't end up with .tgz files created by `helm dependency update` getting committed to your repo.

# Answer
> 7 votes
Generally, a Helm chart would cover all your microservices - ideally, you should be able to deploy whole application in one chart. It may get a little messy if you include external 3rd party dependencies, where you would need to handle dependent charts, but I assume this is out of scope of your question.

More so, if your microservices are very similar you could write for loops, such as:

```
{{- range $index, $service := .Values.myservices}}
apiVersion: apps/v1
kind: Deployment
metadata:
  name: myapp-{{ $service.name }}
# --- etc ---
{{- end }}

```

Now, where to put your chart - in most cases a preferred way is to have a separate repo for all Ops CD files, including Helm chart - so this is where Helm chart would live. This is not to be mixed with CI files, such as Dockerfile - those should live alongside microservice repositories themselves.

# Answer
> 2 votes
I believe what you're asking is: "Is there any way to create just one helm chart that can be used for all microservices in my application?". If so, then you can just use the values.yaml file to store all the values for your templates. This is not considered good practice, considering your template file needs to hold the information for each of your microservice deployments (and thus will become really difficult to manage), but it is possible.

One example: say you have two microservices, and you need one Helm chart that will create the template for both microservices. Generally, you would create separate templates for each service under the templates folder, and deploy each Helm chart for each service individually, but instead you could create multiple deployments in one template yaml file, like

```
# For service 1
apiVersion: apps/v1
type: Deployment
etc., etc... (stick in all values.yaml file values here for service 1)
---
# For service 2
apiVersion: apps/v1
type: Deployment
etc., etc... (stick in all values.yaml file values here for service 2)

```

In your values.yaml file, you would then just place in the values for each of your services, like

```
# Service 1 Keys/Values
foo: value
# Service 2 Keys/Values
bar: otherValue

```

So to answer your question, you can package all your services into one individual Helm chart using the above method, and Kubernetes will run each service as their own ReplicaSet as expected. However, when you have many services to manage, it can be tricky to manage the template YAML files and the values.yaml files when you put in values for all your services in one file, and so it's most likely not a good practice to do this.

This is just my understanding of Helm so far, as I'm still learning Helm myself. As such, I'm not 100% sure if this can be done, so you might want to double check with another person that this answer is actually correct.

EDIT: To summarize: like I mentioned above, in theory, it is possible to use one Helm chart per service. But in practice, it will be extremely messy later on to manage the values.yaml and the templates in the templates folder. So the answer would be, "Yes in theory, but not recommended at all".

---
Tags: kubernetes, helm, kubernetes-helm
---


title: Find biggest files in AWS EFS?

text:
# Question
Title: Find biggest files in AWS EFS?
I have an EFS with a crazily high number of files, many being small. The directory tree is large as well.

Listing directories can take an immensely long time.

The EFS clearly has some very large files that need to be pruned based on the size (I can say this from app architecture/file size estimations for its main use case), but I can't locate the big files given the directory listing time.

I could enable intelligent tiering to save money, but I'd like to identify and fix the root issue. Are there any options to speed up a search?

It would be nice if I could go to every directory and just get the size of the first 5 files to see if any specific directories have large files in general, but any listing seems to be exhaustive and time consuming.

# Answer
The only options I think you have are time consuming `find` and `du` commands.

You can use the `du` command to look at all files, sort the output and give you the top 5. To do this for individual directories you can update the path.

```
du -sk /path/to/efs/* |sort -nr |head -n 5

```

With `find` you can search for files greater than a certain size and dump the list to a file. You can probably parse it down to the top 5 with a little bit of piping.

```
find /path/to/efs -type f -size +200M >> /file_to_store_output.txt &

```

The size units for your usage will probably be `M`egabytes or `G`igabytes

I'd recommend running these as background processes and writing the output to a file.

**References**
find man page
du man page
> 1 votes

---
Tags: amazon-web-services
---


# Question
Title: Azure DevOps boards styles - Parent field empty
We аre trying to implement items on a board in Azure DevOps which are colored in case they have no "Parent" associated with them. Currently, we are struggling with the correct value clause here.

We have tried different things like "does not contain" "\*" or "\[Any\]" however this is not working. What would be the correct Operator and Value we need to use here?

# Answer
> 0 votes
According to the documentation (Customize cards on a Kanban board), you can use `Parent` only to show `Parent Title`:

> To show the Title of the parent work item, choose the Parent field. Choosing the Parent title from a card opens the parent work item. To change the parent work item, open the child work item and remove the link and add a different parent work item. You can filter your board based on parent work items, whether the Parent field is added to cards or not.

# Answer
> 0 votes
There is a report on the developers community that explains that behavior: Cannot set styling rule for parent field

---
Tags: azure-devops
---


# Question
Title: AWS EC2 Target groups healthchecks price
I notice AWS Application Load Balancer sends every 300 seconds(max) health check to the servers(target groups, ec2 instance).
Does these health checks are included in free tier? couldn't find the answer anywhere. https://docs.aws.amazon.com/elasticloadbalancing/latest/application/target-group-health-checks.html

Is there a way to cancel it?

# Answer
It's free

> Data transferred "in" to and "out" from Amazon Classic and Application Elastic Load Balancers using private IP addresses, between EC2 instances and the load balancer in the same AWS VPC is free.

**References**

EC2 On Demand Pricing
> 1 votes

---
Tags: amazon-web-services, healthcheck
---


# Question
Title: Why does Ansible ignore tasks after failing?
My ansible playbook has a task named 'Copy the build script to the system and run it'. It is prone to errors, so I've created a debug task to debug it:

```
- name: Copy the build script to the system and run it
  template:
    src: build3.sh.j2
    dest: "{{ yocto3_env_user_home }}/build3.sh"
    owner: "{{ yocto3_env_user }}"
    mode: 0755
- command: "/bin/su -l {{ yocto3_env_user }} -c '{{ yocto3_env_user_home }}/build3.sh'"
  register: script_output # Creating a variable from the output of the task
  async: 28800
  poll: 300

- debug:
    var: script_output # Debugging the variable created above

```

However, when the 'yocto3-env' task fails, ansible halts and doesn't proceed to the debug task, which causes me to miss out on the debugging output as it can be seen below. What could be potential reason for that? (Please note that my ansible runs in a docker container in a CI/CD pipeline)

```
TASK [yocto3-env : Copy the build script to the system and run it] *************
changed: [3.236.191.93]
TASK [yocto3-env : command] ****************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: OSError: [Errno 2] No such file or directory
fatal: [3.236.191.93]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}
ERROR! Unexpected Exception, this is probably a bug: [Errno 2] No such file or directory
to see the full traceback, use -vvv

```

# Answer
> 1 votes
Per Vladimir Botka: Fatal error terminates the thread. That's the default. You can handle the errors on your own if you want to.

# Answer
> 0 votes
Ansible is often ran against multiple hosts so it doesn't really stop execution altogether just because you ran into one failure.

You can add handle errors a few ways:

1. `ignore_errors: yes`, register the output and check the `PREVIOUS_PLAY_OUTPUT.failed == True`
2. do your work in a `block:` and place a `rescue:` at the end

As for unexpected things, handlers (even of prior modules that worked) will be ignored, so if you want to still run handlers (which is a very annoying case when you have a large playbook that fails occasionally) use `--force-handlers` in the commandline or `force_handlers: true` on a play.

---

https://docs.ansible.com/ansible/latest/user\_guide/playbooks\_error\_handling.html

---
Tags: ansible
---


title: How to create a container from an image

text:
# Question
Title: How to create a container from an image
I have this following image:

```
REPOSITORY               TAG         IMAGE ID       CREATED       SIZE
nvcr.io/nvidia/pytorch   18.04-py3   c3555fec4fe6   3 years ago   5.9GB

```

The goal is to use Ubuntu 18.04 with CUDA 10.2. Now I want to create a container with this image. How can I do that? The guides online didn't make much sense to me as I'm pretty new to docker.

# Answer
> 1 votes
Depending on what you want to do, there are a lot of options you can pass in to docker, but if you just want to run the code in the container as written:

`docker run nvcr.io/nvidia/pytorch:18.04-py3`

or

`docker run c3555fec4fe6`

or even

`docker run c3`

you can give your docker images a more reasonable name with

`docker image tag nvcr.io/nvidia/pytorch:18.04-py3 pytorch`

and run that with

`docker run pytorch`

---

see

* `man docker-run`
* `man docker-image-tag`

and https://docs.docker.com/engine/reference/commandline/run/

# Answer
> 1 votes
This would be the multi-layer approach in docker containers...

```
#Create the base OS image
FROM python:3 AS base

#Update the OS ubuntu image
RUN apt-get -y update

#Install packages
RUN apt-get -y install firefox \
&& apt-get -y install vim

#Create another image layer on top of base to install requirements
FROM base AS requirements

#Install the requirements
RUN pip3 install -r requirements.txt

#Create an intermediate image layer for testing purpose
FROM requirements as test

#Create the build context
COPY /usr/src/my-app /desktop/my-app

#Test the final app
CMD ["python3", "index.py"]

```

Take a look at this: https://www.geeksforgeeks.org/how-to-combine-multiple-base-images-using-single-dockerfile/

---
Tags: docker, containers, ubuntu
---


# Question
Title: "Retrieve a blob, and its metadata": Does the get_blob() already cause file size dependent traffic on the Google Cloud Platform?
Quote from the Python code example at View and edit object metadata --\> "Code Samples" --\> "Python": `Retrieve a blob, and its metadata`.

The full code:

```
from google.cloud import storage

def blob_metadata(bucket_name, blob_name):
    """Prints out a blob's metadata."""
    # bucket_name = 'your-bucket-name'
    # blob_name = 'your-object-name'

    storage_client = storage.Client()
    bucket = storage_client.bucket(bucket_name)

    # Retrieve a blob, and its metadata, from Google Cloud Storage.
    # Note that `get_blob` differs from `Bucket.blob`, which does not
    # make an HTTP request.
    blob = bucket.get_blob(blob_name)

    print("Blob: {}".format(blob.name))
    print("Bucket: {}".format(blob.bucket.name))
    print("Storage class: {}".format(blob.storage_class))
    print("ID: {}".format(blob.id))
    print("Size: {} bytes".format(blob.size))
    print("Updated: {}".format(blob.updated))
    print("Generation: {}".format(blob.generation))
    print("Metageneration: {}".format(blob.metageneration))
    print("Etag: {}".format(blob.etag))
    print("Owner: {}".format(blob.owner))
    print("Component count: {}".format(blob.component_count))
    print("Crc32c: {}".format(blob.crc32c))
    print("md5_hash: {}".format(blob.md5_hash))
    print("Cache-control: {}".format(blob.cache_control))
    print("Content-type: {}".format(blob.content_type))
    print("Content-disposition: {}".format(blob.content_disposition))
    print("Content-encoding: {}".format(blob.content_encoding))
    print("Content-language: {}".format(blob.content_language))
    print("Metadata: {}".format(blob.metadata))
    print("Custom Time: {}".format(blob.custom_time))
    print("Temporary hold: ", "enabled" if blob.temporary_hold else "disabled")
    print(
        "Event based hold: ",
        "enabled" if blob.event_based_hold else "disabled",
    )
    if blob.retention_expiration_time:
        print(
            "retentionExpirationTime: {}".format(
                blob.retention_expiration_time
            )
        )

```

From this `Retrieve a blob, and its metadata`, I read that blobs *might* already be more than metadata and more than the mere frame of the file object, so that they are already a loaded data object that leads to traffic on the Google Cloud Storage side at the size of the data. On the other hand, I see from Reading only the metadata of a file in a Google Cloud Storage bucket into a Cloud Function in Python (without loading the file or its data!) that there is no data loaded to the *Google Cloud Function* before calling `get_blob()` until you call `blob.download_to_filename()`, `blob.download_as_string()` or `blob.download_as_text()`. Is there still some traffic on the *Google Cloud Platform* when using `get_blob()`?

I am dealing with many large files try to log their metadata for further triggers inside and outside of GCP. I want to be sure that this get\_blob() function's traffic is independent from the filesize of the file that it takes as the parameter.

Does the `get_blob()` already cause file size dependent traffic on the Google Cloud Platform?

# Answer
There is no traffic of the file content if you just get the blob and its metadata.

```
blob = bucket.get_blob(blob_name)

```

does not yet upload the file to the cloud function since it is used for the metadata example query: View and edit object metadata --\> "Code Samples" --\> "Python". These code samples show it.

For more details, see Reading only the metadata of a file in a Google Cloud Storage bucket into a Cloud Function in Python (without loading the file or its data!).
> 0 votes

---
Tags: google-cloud-platform, google-cloud-storage
---


# Question
Title: Is the output of "Test the function" of a Google Cloud Function saved somewhere?
I have returned the database address with username and password as the plain text output of a Cloud Function during testing. The idea was to see how the db URL looks like and where the error could be that made the connection fail. Bad idea of course. Is the result of "Test the function" in tab "Testing" logged anywhere? I could not find the exposed db URL in the logs of that Cloud Function, is it perhaps stored anywhere else? Should I change my password?

# Answer
I have not seen any logs of the return of the query since the question was asked. The only thing that may get logged is prints or loggings, but the return that is shown in the `Testing` tab itself does not get logged, it is just printed once in the output of the `Testing` tab.
> 0 votes

---
Tags: security, google-cloud-platform
---


# Question
Title: Buildkit failing arm builds with missing binaries even though they are in $PATH
A few of our developers recently got some M1 machines, I want to provide them with ARM compatible images and have set my sights on Buildkit.

I set up a podTemplate in Jenkins and run the build with this command

```
buildctl build --opt platform=linux/arm64 --frontend=dockerfile.v0 --local context=. --opt build-arg:HTTP_PROXY=http://proxy-here:8080 --opt build-arg:HTTPS_PROXY=http://proxy-here:8080 --local dockerfile=. --output type=image,push=true,\\"name=\$BUILD_TAG,\$SHORT_TAG,\$MINOR_TAG,\$FULL_TAG\\"

```

When running for `linux/amd64` everything goes as planned, however `linux/arm64` will fail with a variety of errors depending on the Dockerfile, for instance:

```
RUN update-ca-certificates:
# Error while loading /usr/bin/run-parts: No such file or directory

```

I have checked that run-parts exits and that it is linked to to musl with the right architecture.

```
RUN which run-parts
# /bin/run-parts

RUN ldd `which run-parts`
# /lib/ld-musl-aarch64.so.1 (0x5500000000)
# libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0x5500000000)

RUN file `which run-parts`
# /bin/run-parts: symbolic link to /bin/busybox

RUN file `which busybox`
# /bin/busybox: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped

RUN ldd `which busybox`
# /lib/ld-musl-aarch64.so.1 (0x5500000000)
# libc.musl-aarch64.so.1 => /lib/ld-musl-aarch64.so.1 (0x5500000000)

RUN file `which update-ca-certificates`
# /usr/sbin/update-ca-certificates: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-musl-aarch64.so.1, stripped

```

The error seems to indicate that it is not running with the correct `$PATH`, but then again, `which` seems to have no issue finding the commands, `/lib/ld-musl-aarch64.so.1` also definitely exitsts.

The error will happen to all sorts of otherwise pedestrian commands `ls`, `addgroup` (but not `deluser`).

I have tried linking files from `/bin/` into `/usr/bin/` but that quickly becomes a game of whack a mole and seems too much of a hack for me.

The base image for the container is `php:7.4.28-fpm-alpine`

My Google-fu completely fails me here, what am I doing wrong?

# Answer
> 0 votes
In my case the issue was that QEMU wasn't installed and thus unavailable for emulation, apparently buildkit can get through a large amount of steps before needing emulation.

I added an init container to the pod template which installs QEMU emulators as required:

```
spec:
    initContainers:
    - name: prepare
      image: tonistiigi/binfmt:latest
      args: ["--install", "arm64"]
      securityContext:
          privileged: true

```

---
Tags: dockerfile, alpine
---


title: Hosting Wiki.js using nginx-proxy

text:
# Question
Title: Hosting Wiki.js using nginx-proxy
I am having difficulty getting Wiki.js hosted on a server.

My current setup is using `docker-compose` utilising a reverse proxy with the eventual aim to host some services. I have successfully managed to host Portainer (using a guide). And I can host Wiki.js in its own docker container, but adding the service together is failing with a `502` error. Portainer is still working, however.

Here is the `docker-compose.yml` file:

```
version: '2'

services:

  proxy:
    image: jwilder/nginx-proxy
    container_name: proxy
    restart: unless-stopped
    labels:
      com.github.jrcs.letsencrypt_nginx_proxy_companion.nginx_proxy: "true"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro
      - certs:/etc/nginx/certs:rw
      - vhost.d:/etc/nginx/vhost.d
      - html:/usr/share/nginx/html
      - ./uploadsize.conf:/etc/nginx/conf.d/uploadsize.conf:ro
    ports:
      - "80:80"
      - "443:443"
    networks:
      - "default"
      - "proxy-tier"

  proxy-letsencrypt:
    image: jrcs/letsencrypt-nginx-proxy-companion
    container_name: letsencrypt
    restart: unless-stopped
    environment:
      - NGINX_PROXY_CONTAINER=proxy
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    volumes_from:
      - "proxy"
    depends_on:
      - "proxy"
    networks:
      - "default"
      - "proxy-tier"

  portainer:
    image: portainer/portainer
    container_name: portainer
    restart: always
    environment:
      - VIRTUAL_HOST=docker.example.com
      - LETSENCRYPT_HOST=docker.example.com
      - LETSENCRYPT_EMAIL=me@example.com
    volumes:
      - ./portainer/:/data
      - /var/run/docker.sock:/var/run/docker.sock
    ports:
      - "9000:9000"
## Wiki.js specific container (db)
  db:
    image: postgres:11-alpine
    container_name: wiki
    environment:
      POSTGRES_DB: wiki
      POSTGRES_PASSWORD: wikijsrocks
      POSTGRES_USER: wikijs
    logging:
      driver: "none"
    restart: unless-stopped
    volumes:
      - db-data:/var/lib/postgresql/data
## Wiki.js specific container (app)
  wiki:
    image: requarks/wiki:2
    depends_on:
      - db
    environment:
      DB_TYPE: postgres
      DB_HOST: db
      DB_PORT: 5432
      DB_USER: wikijs
      DB_PASS: wikijsrocks
      DB_NAME: wiki
      VIRTUAL_HOST: wiki.example.ml
      LETSENCRYPT_HOST: wiki.example.ml
      LETSENCRYPT_EMAIL: me@example.com
    restart: always
    ports:
      - "8010:3000"

volumes:
  certs:
  vhost.d:
  html:
  db-data:

networks:
  proxy-tier:

```

# Answer
> 0 votes
I had the exact same issue. After investigating and looking at my nginx proxy `/etc/nginx/conf.d/default` I discovered that the nginx proxy was unable to connect to the container.

The solution was to add the nginx proxy container to the same docker network as the wikijs container.

---
Tags: docker, nginx, tls
---


title: Will my AWS usage cost me anything?

text:
# Question
Title: Will my AWS usage cost me anything?
I was exploring AWS Cloudwatch. It was first time that I tried anything on AWS. I ensured that I will turn off all instances and stacks:

* **No stacks:**
* **No instances**

But my AWS Billing Dashboard says this:

Will I need to pay anything, considering that above billing amount is forecast, I have turned off all instances and stacks and I may not have exceeded my free tier (I have used AWS first time)?

**Update**

Another screenshot:

So I have following precise questions:

**Q.** Will this cost increase further? That is, do I have to stop all the services correctly (as shown in earlier pics) to not increase the cost further, or is there something that I missed?

# Answer
Yes, you will need to pay based on what you're showing. If you have a billing question, you can create a support case in the AWS Console.

Will the cost increase depends on if you shut off or deleted services. If you shut everything off you could still get charged for EBS volume storage. If you deleted the EC2 instance(s) and deleted any storage (S3/EBS/Glacier) you should not see any additional charges.

I have been hit with surprise bills in the past by not deleting EBS volumes when I delete an instance, or forgetting about objects in S3.

If you go to **Billing Dashboard** and click on **Bills** on the left side of the page you will see an option for **Details**. You can see why you're getting charged their.

Some important things to know about the AWS Free Tier:

1. The 12-months Free starts from the day you create the account.
2. It's per Organization, if your account is linked to an Organization it will depend when the Org was created and if the free-tier limits have been reached because of other accounts in that Org.
3. It's per account and not per region.
4. There are some services that are always free (until you hit a usage level)

**References**
AWS Free Tier
AWS How Do I Check If I've Exceeded Free Tier
> 2 votes

---
Tags: amazon-web-services
---


# Question
Title: How to get all open Merge Requests using git command line?
I need to get all open Merge requests by author and if possible within a time frame. Tried using this command for Github where it works fine -

```
git ls-remote origin 'pull/*/head'

```

But in gitlab, this is not returning anything, please suggest to get all MRs, if possible status = 'opened', 'closed', 'merged' and time frame.

# Answer
> 0 votes
This is the pattern to fetch all Merge Requests from GitLab -

```
git ls-remote origin 'refs/merge-requests/*/head'

```

OR

```
git ls-remote origin 'refs/merge-requests/*/merge'

```

---
Tags: gitlab, git, github
---


# Question
Title: Docker refuses to execute commands or start containers with Ubuntu 20 image on Ubuntu 21 host, but works on Ubuntu 20 host
So I've pulled the amd64/ubuntu:latest image, which is an Ubuntu 20 image for docker and customized it a bit. Not much - zsh, vim, locales and openssh-server compiled from source (because the package wants me to download 100Mb of some stuff, whereas the compiled version is only 1.5 Mb)

In this case, I don't think those are the reason for the following problem: I did not intend to use Dockerfiles with this one, and I created and ran containers with the following command:

```
$ docker run -dit --network=my_docker_network --name=MyContainer \
  local/ubuntu:latest

```

This works fine on my Ubuntu 20.04 - notice how there isn't a command after the image name, which, as per documentation, is supposed to be there, but it looks like it isn't needed here. The `sshd` is later started manually by the `docker exec` command and works fine.

Now, to test the compatibility, I've installed Ubuntu 21 on a VirtualBox, then installed docker there and imported my custom image `local/ubuntu:latest`. Docker versions and the way they are run seem to be exactly the same - v20.10.12. When I run the same `docker run...` command on Ubuntu 21, it refuses to start start the image, saying another argument - the COMMAND - is require. Yet when I provide ANY command as the last argument, for example:

```
$ docker run -dit --network=my_docker_network --name=MyContainer \
  local/ubuntu:latest /usr/bin/sh -c 'echo OK'

```

or

```
$ docker run -dit --network=my_docker_network --name=MyContainer \
  local/ubuntu:latest /usr/bin/sshd

```

it says they're not in $PATH or file doesn't exist.

If I try and create a container first, it is indeed created with:

```
$ docker create -it --network=my_docker_network --name=MyContainer \
  local/ubuntu:latest

```

But the same problem occurs once I try actually starting it. What is this? What am I missing?

# Answer
> 1 votes
Asked on the Docker forums - apparently to correctly "import" the image, you don't user the `import` command but instead user `image load` command.

INCORRECT:

```
$ docker image save local/ubuntu:latest -o ubuntu.tgz # on Ubuntu 20
$ docker import ubuntu.tgz local/ubuntu:latest        # WRONG! on Ubuntu 21

```

**CORRECT:**

```
$ docker image save local/ubuntu:latest -o ubuntu.tgz # on Ubuntu 20
$ docker image load -i ubuntu.tgz                     # on Ubuntu 21

```

Credits go to the person who answered this question on the Docker forum here. Also note, that if you run `docker image save` the image full name (`repo/name:tag`) is actually save with along with other information, so upon running `docker image load -i ubuntu.tgz` Docker will tag that image as exactly as it was tagged when it was saved. If an image with the same name exists, it will be replaced, so be careful.

---
Tags: docker, ubuntu
---


# Question
Title: Does cloud-init store the copy of the init it ran first?
I'm sending my cloud-init file to Terraform via `user_data`. Is this stored anywhere on the machine that runs it? I know it may get merged with other stuff before it's run.

# Answer
It seems to be stored under

```
/var/lib/cloud/instances

```

Under your instance id.

```
cd /var/lib/cloud/instances/$(cloud-init query instance_id)

```

For me I found the information I wanted in `user-data.txt`.

```
sudo cat "/var/lib/cloud/instances/$(cloud-init query instance_id)/user-data.txt"

```
> 1 votes

---
Tags: terraform, configuration, cloud-init
---


# Question
Title: How to automate docker prune commands using crontab
Can someone help me on how to clean up containers and images which are 3 months older automatically by using crontab scheduling? Which type of method need to use for automation? I have manually tried deleting by using below commands and it worked but I need to automate it, to perform action for every 3 months without manual action.

```
docker image prune　 --force --filter “until=2020-01-01T00:00:00”
docker container prune　 --force --filter “until=2020-01-01T00:00:00”

```

# Answer
> 0 votes
You can add a new cron entry by prompting:

`sudo crontab -e`

You can then edit the file with your command and its frequency. Every three months is not a standard cron frequency (i.e. there's no built-in solution to transcribe this into cron) so you can use these frequencies:

30 03 01 Jan,Apr,Jul,Oct *

or else

30 03 01 \*/3 *

(Based on this answer by Richard Holloway on ServerFault)

So eventually your cron entry will look like:

`30 03 01 Jan,Apr,Jul,Oct * docker image prune　 --force --filter “until=2020-01-01T00:00:00”`

You can check out your new entry by prompting:

`sudo crontab -l`

**Edit**

chicks's comment pointed out that the goal wasn't to automate an action to be performed every three months but rather to automatically find which objects are older than months. So guessing that you would like to perform this action everyday, I would rather go for:

`0 0 3 * * find /var/lib/docker -type f -ctime +90 -delete`

This finds all Docker files older than three months and deletes them every day at 3AM.

# Answer
> 0 votes
1. create a new file in the Cron file to house our command

```
 cd /etc/cron.daily
 sudo nano docker-prune-daily

```
2. Once the file opens add the prune command we created and inform the OS it's a bash file

```
#!/bin/bash
docker image prune　 --force --filter “until=2020-01-01T00:00:00”
docker container prune　 --force --filter “until=2020-01-01T00:00:00”

```
3. Once you've added the command, hit Ctrl + O and then Ctrl + X to save and close the file
4. Now that we've created the file we need to tell the operating system that it needs to be executable with the following command:

```
sudo chmod +x /etc/cron.daily/docker-prune-daily

```

\** If you want to test out to see if it will execute correctly next time the cronjob runs, you can force the daily Cron register to run with the following command:

```
    run-parts /etc/cron.daily

```

---
Tags: docker, docker-compose
---


# Question
Title: Google Kubernetes Engine's auto-pilot mode does not grant nodes Google Storage write access
I am trying to use the workflow orchestrator Snakemake with a Google Kubernetes Engine (GKE) autopilot cluster. My workflow needs to upload files to Google Cloud Storage buckets within the same project. Both Snakemake and GKE documentations specify that a Google Storage read-write scope should be specified when the cluster is created, via `--scopes storage-rw`.

I tried doing so with an autopilot cluster:

```
gcloud container --project "project-name" clusters create-auto "default-cluster" \
    --region "asia-southeast2" --release-channel "regular" \
    --network "projects/project-name/global/networks/default" \
    --subnetwork "projects/project-name/regions/asia-southeast2/subnetworks/default" \
    --cluster-ipv4-cidr "/17" --services-ipv4-cidr "/22" --scopes storage-rw

```

(I generated this command using the Google Cloud web console, then added the `--scopes storage-rw` argument at the end.)

But I still get a Snakemake error message related to lack of permission for Google Storage read-write:

```
google.api_core.exceptions.Forbidden: 403 GET https://storage.googleapis.com/storage/v1/b/kubernetes-nccs-npc2/o/whoami.txt?fields=name&prettyPrint=false: Caller does not have storage.objects.get access to the Google Cloud Storage object.

```

It seems `--scopes storage-rw` does not work for the creation of autopilot clusters. Is this true? If so, how can I grant Google Storage read-write access to an autopilot cluster?

# Answer
I think the error has more to do with the service account permissions running your cluster. I'd make sure all the permissions are included, like *storage.ojjects.get*. The error seems very permissions related.

```
Caller does not have storage.objects.get access to the Google Cloud Storage object.

```

You might be hitting this issue Google has reported:
https://cloud.google.com/kubernetes-engine/docs/how-to/access-scopes

I'd listen to their recommendations over mine.

When managing IAM roles, you can treat a service account either as a resource or as an identity. This command is to get the iam policy of a service account resource. There are other gcloud commands to manage IAM policies for other types of resources. For example, to manage IAM policies on a project, use the $ gcloud projects commands and for services use $ gcloud iam service-accounts.

This will return your service running the cluster w/ role assignment (it's the robot one :) )

```
gcloud projects get-iam-policy <PROJECT-NAME>

```

The problem is sometimes running these commands requires special permissions too, so it becomes fools progress. I think this link may be helpful. it shows how to set permissions on a service-account.

https://cloud.google.com/sdk/gcloud/reference/iam/service-accounts/add-iam-policy-binding

Also here is Google dictionary of IAM permissions including *storage.ojjects.get*.

https://cloud.google.com/iam/docs/understanding-roles
> 0 votes

---
Tags: kubernetes, google-cloud-platform, google-kubernetes-engine, google-cloud-storage
---


# Question
Title: Jenkins Git plugin, Error: Name must follow the pattern
I receive an error after launching a Jenkins pipeline, but right at the SCM step, configure via the plugin (not by the groovy).

The message refers to a "Name" wrong spelled, but I can't figure out to which of my fields it is referring.

I looked to the source code, but it manipulates a concatenation of several inputs...

This is a git SCM.

> ERROR: Name must follow the pattern '^\[a-zA-Z0-9\]+((.|\_|\__|-+)\[a-zA-Z0-9\]+)\*$'
> at hudson.util.FormValidation.\_errorWithMarkup(FormValidation.java:268)
> at hudson.util.FormValidation.errorWithMarkup(FormValidation.java:254)
> at hudson.util.FormValidation.error(FormValidation.java:145)
> at hudson.util.FormValidation.error(FormValidation.java:170)
> at org.jenkinsci.plugins.docker.commons.credentials.ImageNameValidator.validateName(ImageNameValidator.java:289)
> at org.jenkinsci.plugins.docker.commons.credentials.ImageNameValidator.validateUserAndRepo(ImageNameValidator.java:115)
> at org.jenkinsci.plugins.docker.commons.credentials.ImageNameValidator.checkUserAndRepo(ImageNameValidator.java:141)
> at org.jenkinsci.plugins.docker.commons.credentials.ImageNameValidator$checkUserAndRepo.call(Unknown Source)
> \+ 50 lines I can show if needed

# Answer
The stack trace contains org.jenkinsci.plugins.docker.commons.credentials.ImageNameValidator, so this error is for Docker image name validation, not your Git SCM.
> 2 votes

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: Container deployment sync problem with Ansible
Imagine the following situation after you have switched from docker-compose to Ansible:

* You start a DB container which needs time to boot up
* You want to load data dump into it

(could be also you need to run tests after service containers are up)

How to tell Ansible that one task depends on finishing another one? (that is, also, Ansible needs to support a condition so that you tell how a task is done or not, if it's not default in a specific module)

# Answer
You want the `wait_for` or `wait_for_connection` modules.

If your playbook consists of a single play, which starts the container, then loads data into it, but still targetting the docker host, then you should use `wait_for` to determine when the container is up and running -- typically by polling some health check.

If however your playbook consists of two plays, *i.e.*

1. Target Docker host and start container, then add the container to the inventory
2. Target the container, using the docker transport and load the data

then you should use the `wait_for_connection` module.

In either case it will depend somewhat on what the container does and how you know that it is ready for action. If it exposes a file, port or url then you should be able to do what you need.
> 1 votes

---
Tags: docker, ansible, containers
---


# Question
Title: Ansible no user $HOME by default - so how do I run commands
I have thousands of servers that, by default, for security and space do not create a `$HOME` when you `ssh` in.

This seems to be posing a problem for Ansible as it keeps trying to `chdir to home` that does not exist. Can I get Ansible to use another directory in the `ansible.cfg` or just stop this behaviour completely?

`ansible.cfg`

```
[defaults]

inventory      = /home/welshch/.ansible/hosts
remote_tmp     = /tmp
local_tmp      = ~/.ansible/tmp
interpreter_python = auto_silent
roles_path    = /home/eekfonky/.ansible/roles
host_key_checking = False

```

Here is the error with the server name changed for security reasons;

```
☁  .ansible  ansible-playbook get_fleet_info.yml -vvvv
fatal: [ldap-corp-search-server.com]: FAILED! => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python"
    },
    "changed": false,
    "module_stderr": "OpenSSH_7.4p1, OpenSSL 1.0.2k-fips  26 Jan 2017\r\ndebug1: Reading configuration data /home/eekfonky/.ssh/config\r\ndebug1: /home/eekfonky/.ssh/config line 2: Applying options for *\r\ndebug3: /home/eekfonky/.ssh/config line 9: Including file /home/eekfonky/.ssh/bastions-config depth 0\r\ndebug1: Reading configuration data /home/eekfonky/.ssh/bastions-config\r\ndebug1: /home/eekfonky/.ssh/bastions-config line 10: Applying options for *.corp.amazon.com\r\ndebug1: /home/ekfonky/.ssh/bastions-config line 35: Applying options for *.pdx*.server.com\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 58: Applying options for *\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 27395\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\nShared connection to ldap-corp-search-server.com closed.\r\n",
    "module_stdout": "Could not chdir to home directory /home/eekfonky: No such file or directory\r\n",
    "msg": "MODULE FAILURE\nSee stdout/stderr for the exact error",
    "rc": 1
}

```

# Answer
I assume that you are referring to verbose output similar like

```
TASK [Task] *****************************************************************************************************************
task path: taskFile:<lineNumber>
...
<IP> ESTABLISH ... CONNECTION FOR USER: {{ ansible_user }}
<IP> EXEC /bin/sh -c 'echo ~{{ ansible_user }} && sleep 0'
<IP> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/{{ ansible_user }}/.ansible/tmp `"&& mkdir "` echo /home/{{ ansible_user }}/.ansible/tmp/ansible-tmp-1234567890 `" && echo ansible-tmp-1234567890="` echo /home/{{ ansible_user }}/.ansible/tmp/ansible-tmp-1234567890 `" ) && sleep 0'
Using module file /usr/lib/python2.7/site-packages/ansible/modules/files/find.py
...

```

> *Can I get Ansible to use another directory ...*

According an older documentation (v2.3), it is possible to change that value. The configuration parameter `local_tmp` might still be available in latest version.

> *When Ansible gets ready to send a module to a remote machine ... The default location is a subdirectory of the user’s home directory. If you’d like to change that, you can do so by altering this setting*

According

**Further Q&A**

it should be the variable `remote_tmp`.
> 2 votes

---
Tags: ansible, ssh
---


# Question
Title: ansible-doc command for filters & plugins
I would like to list & read documentation from the ansible-doc command for example about `ipaddr` or `flatten`.

Is it possible? Thanks in advance!

# Answer
> 1 votes
The command `ansible-doc`

> Displays information on modules installed in Ansible libraries. It displays a terse listing of plugins and their short descriptions, provides a printout of their DOCUMENTATION strings ...

from the module source.

Therefore for filters to manipulate data like `flatten` or `ipaddr` there is no such command.

**Further Information**

---
Tags: ansible
---


title: Docker build from gitlab pipeline

text:
# Question
Title: Docker build from gitlab pipeline
I do have the following "shared.gitlab-ci.yml"

```
variables:
  GIT_SUBMODULE_STRATEGY: recursive
  DOCKERFILE_BUILD: docker/build/Dockerfile
  DOCKERFILE_TEST: docker/test/Dockerfile
  DOCKERFILE_ANALYZE: docker/analyze/Dockerfile
  BUILD_NAME: $CI_PROJECT_PATH_SLUG:$CI_COMMIT_REF_NAME-$CI_COMMIT_SHORT_SHA
  PREPARE_SCRIPT: ""
  BUILD_ARGS: ""

stages:
- build
- test
- analyze

before_script:
  - echo "Project Details:"
  - echo $CI_PROJECT_PATH_SLUG
  - echo $CI_COMMIT_REF_NAME
  - echo $CI_COMMIT_SHORT_SHA
  - echo "Build Variables:"
  - echo $BUILD_NAME
  - echo $DOCKERFILE_BUILD
  - echo $DOCKERFILE_TEST
  - echo $DOCKERFILE_ANALYZE
  - echo "Directory:"
  - ls -la
  - echo "Registry info:"
  - echo $CI_REGISTRY
  - echo $CI_REGISTRY_NAME
  - echo $CI_REGISTRY_USER
  - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  - docker info

docker-build:
  image: docker:latest
  stage: build
  services:
    - docker:dind
  script:
    - docker build \
      --rm \
      --build-arg BRANCH_NAME=$CI_COMMIT_REF_NAME \
      --build-arg BUILD_ARGS=$BUILD_ARGS \
      -t $BUILD_NAME \
      $DOCKERFILE_BUILD
    - docker push $BUILD_NAME

```

Which is will be used/included in another repositories "gitlab-ci.yml" like :

```
include:
  - project: my-group/shared-pipeline
    ref: master
    file: shared.gitlab-ci.yml

```

The Dockerfile is also in this repository `/docker/build/Dockerfile`:

```
FROM unityci/editor:windows-2020.1.16f1-base-0.17.2 as unity
ARG BRANCH_NAME
ARG BUILD_ARGS

CMD echo "Hello world! This is my first Docker image. Branch: $BRANCH_NAME"

```

Somehow when I run the "gitlab" pipeline I'll get the following output:

```
Fetching changes...
Initialized empty Git repository in /builds/my-group/mygame/unity/.git/
Created fresh repository.
Checking out 3731ab28 as develop...
Updating/initializing submodules recursively...
Executing "step_script" stage of the job script
00:01
Using docker image sha256:6816f070b2395d750ce78e4fbdb7e9df0af63b7686f3063c09e008c2ef1a0036 for docker:latest with digest docker@sha256:9e13118c8f7b17349e2d350806082fdda135ebf2923376821f3eac23559b9141 ...
$ echo "Project Details:"
Project Details:
$ echo $CI_PROJECT_PATH_SLUG
my-group-mygame-unity
$ echo $CI_COMMIT_REF_NAME
develop
$ echo $CI_COMMIT_SHORT_SHA
3731ab28
$ echo "Build Variables:"
Build Variables:
$ echo $BUILD_NAME
my-group-mygame-unity:develop-3731ab28
$ echo $DOCKERFILE_BUILD
docker/build/Dockerfile
$ echo $DOCKERFILE_TEST
docker/test/Dockerfile
$ echo $DOCKERFILE_ANALYZE
docker/analyze/Dockerfile
$ echo "Directory:"
Directory:
$ ls -la
total 68
drwxrwxrwx    8 root     root          4096 Mar 15 05:32 .
drwxrwxrwx    4 root     root          4096 Mar 15 05:32 ..
-rw-rw-rw-    1 root     root          1509 Mar 15 05:32 .atomignore
drwxrwxrwx    7 root     root          4096 Mar 15 05:32 .git
-rw-rw-rw-    1 root     root          1848 Mar 15 05:32 .gitattributes
-rw-rw-rw-    1 root     root          1817 Mar 15 05:32 .gitignore
-rw-rw-rw-    1 root     root           230 Mar 15 05:32 .gitlab-ci.yml
drwxrwxrwx   13 root     root          4096 Mar 15 05:32 Assets
drwxrwxrwx    2 root     root          4096 Mar 15 05:32 Packages
drwxrwxrwx    2 root     root          4096 Mar 15 05:32 ProjectSettings
-rw-rw-rw-    1 root     root         11542 Mar 15 05:32 README.md
-rw-rw-rw-    1 root     root           947 Mar 15 05:32 Unity_lic.alf
-rw-rw-rw-    1 root     root          2485 Mar 15 05:32 Unity_v2017.x.ulf
drwxrwxrwx    3 root     root          4096 Mar 15 05:32 ci
drwxrwxrwx    3 root     root          4096 Mar 15 05:32 docker
$ echo "Registry info:"
Registry info:
$ echo $CI_REGISTRY
registry.gitlab.com
$ echo $CI_REGISTRY_NAME
$ echo $CI_REGISTRY_USER
gitlab-ci-token
$ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
WARNING! Your password will be stored unencrypted in /root/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
$ docker info
Client:
 Context:    default
 Debug Mode: false
Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.13
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 2a1d4dbdb2a1030dc5b01e96fb110a9d9f150ecc
 runc version: v1.0.3-0-gf46b6ba2
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
WARNING: API is accessible on http://0.0.0.0:2375 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
 Kernel Version: 5.4.109+
 Operating System: Alpine Linux v3.15 (containerized)
 OSType: linux
 Architecture: x86_64
 CPUs: 1
 Total Memory: 3.599GiB
 Name: b837f980368f
 ID: 6PTR:73WJ:JO7S:UML2:4FCQ:RL3P:6VTY:PV7D:37OL:UXQ2:M33X:3ZCV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine
$ docker build \ --rm \ --build-arg BRANCH_NAME=$CI_COMMIT_REF_NAME \ --build-arg BUILD_ARGS=$BUILD_ARGS \ -t $BUILD_NAME \ $DOCKERFILE_BUILD
"docker build" requires exactly 1 argument.
See 'docker build --help'.
Usage:  docker build [OPTIONS] PATH | URL | -
Build an image from a Dockerfile
Cleaning up project directory and file based variables
00:01
ERROR: Job failed: exit code 1

```

I always get: `"docker build" requires exactly 1 argument.`

I also tried for example:

```
- docker build --file="$DOCKERFILE_BUILD"
- docker push $BUILD_NAME

```

OR

```
- docker build -f $DOCKERFILE_BUILD
- docker push $BUILD_NAME

```

The result always stays the same.

# Answer
> 1 votes
I think that you forgot '.'(dot)

```
docker build -f $DOCKERFILE_BUILD .

```

(In my case) Even with this command, I got the error ...

---
Tags: docker, gitlab, gitlab-ci-runner
---


title: AWS CLI --profile with duration > 1h

text:
# Question
Title: AWS CLI --profile with duration > 1h
With the AWS CLI you can assume a role using the `--profile` parameter:

```
aws --profile <profile_name> <service> <command>

```

e.g.:

```
aws --profile admin s3 ls s3://mybucket/

```

That profile can be an AWS role that os to be assumed. My problem with that is that the session of the role is only valid for one hour by default. Is there a way of explicitly setting the session duration without using the `sts` service directly?

# Answer
> 1 votes
You can set a maximum duration of up to 12 hours in the IAM Role config:

Hope that helps.

# Answer
> 0 votes
Along with changing the MaxSessionDuration(as mentioned in the above answer) of the IAM role ,also try adding/changing "duration\_seconds" parameter value in the credentials file of the configured profile.

---
Tags: amazon-web-services, aws-cli
---


# Question
Title: Typical build size of the top 20 most popular frameworks?
"Endless list" questions are a no no, so I limit to top 20 only.

I am presenting to non-dev ops people (even though I am one myself) on how some software we use creates quite a large image (~1.5 gb), and how other popular frameworks tend to be (although aren't always) smaller.

I googled for 'typical size of a flask app' (with the intention of repeating for other frameworks until I exhausted all that I know) and had trouble even finding that (these comments were somewhat useful, although possibly more related to the particulars of that user's question).

Is there a quick an easy info source listing the typical app size of, say, the top 20 most popular app frameworks?

# Answer
> 1 votes
Maybe instead of trying to discover the "typical" or "average" size of an app built with a particular framework, since this will be purely anecdotal and difficult to search for, you could instead search for some of the most popular applications with that framework and compare those sizes? As you stated in your question, some of these frameworks tend to be smaller, but depend entirely on the development team behind them.

Aside from the finding of the supporting information, I would also encourage you to talk about the benefits of having a smaller build, especially if your audience consists of non-dev ops people. Thinks to talk about would be shorter build times, better performance, code readability, fewer dependencies, quicker deployment times, etc. I think by starting your presentation with these kinds of talking points, you are solidifying the need for moving to a different type of framework.

---
Tags: builds, images
---


# Question
Title: Is a decentralized Docker swarm possible?
I want to run services in our office in containers. The problem I have is that we do not have a spare machine available to act as the dedicated host.

So I am considering creating a swarm using everyone's desktop workstations as the workers. But if the manager node (my computer) is shutdown then the entire swarm stops.

I envisioned a swarm where every worker can act as a manager if needed. When the manager dies, another worker assumes the role as manager and the service continues running uninterrupted.

*Is it possible to create a **decentralized** swarm where any available node can assume the role as a manager as needed?*

# Answer
> 2 votes
By Docker Swarm design **it is dencentralized**, so you can *promote* any node to a manager. Normally, 3 managers are enough, for landscapes with thousands of nodes 5 are enough.

https://docs.docker.com/engine/swarm/admin\_guide/

# Answer
> 0 votes
1. Every swarm manager is/has worker functionality by default (which can be changed).
However, the number of functioning docker managers has a very important component when it comes to the ratio of the available number of managers to the active number of managers known as **"the quorum of managers"**. For more info on **the administration of the swarm** and the quorum here : https://docs.docker.com/engine/swarm/admin\_guide/#maintain-the-quorum-of-managers
2. In your particular case, instead of using actual hardware machines (desktop workstation/laptop/server) you can use virtual machines where the docker can be installed. For their automated management, you can use vagrant. However, it is all about managing the available hardware resource of your entire system. You can also add to the "docker swarm network" any number of other workstations either areas docker hosts or the virtual-machine hosts.

**Note**: I suggest you consider looking at Podman as a substitute for Docker if your project is in its early stage of development.

**P.S.** You might want to specify the idea of your project in order to get better advice than currently can be given.

Good luck and have a nice day.

---
Tags: docker, docker-swarm
---


# Question
Title: How come my hostname isn't set to the FQDN with cloud-init when using `prefer_fqdn_over_hostname`?
`prefer_fqdn_over_hostname` is documented as,

> If both `fqdn` and `hostname` are set, it is distro dependent whether `hostname` or `fqdn` is used, unless the **`prefer_fqdn_over_hostname` option is `true` and `fqdn` is set it will force the use of FQDN in all distros**, and if false then it will force the `hostname` use.

In my `cloud-init.cfg` I have,

```
hostname: "host-10-2-65-89"
fqdn: "host-10-2-65-89.openstack.build."
prefer_fqdn_over_hostname: true

```

But when I run this my hostname is set to `host-10-2-65-89`. Here is the log, and you can see it explicitly says `host-10-2-65-89.openstack.build.`

```
249 - cc_set_hostname.py[DEBUG]: Setting the hostname to host-10-2-65-89.openstack.build. (host-10-2-65-89)
250 - util.py[DEBUG]: Reading from /etc/hostname (quiet=False)
250 - util.py[DEBUG]: Read 7 bytes from /etc/hostname
250 - util.py[DEBUG]: Writing to /etc/hostname - wb: [644] 16 bytes
250 - __init__.py[DEBUG]: Non-persistently setting the system hostname to host-10-2-65-89
250 - subp.py[DEBUG]: Running command ['hostname', 'host-10-2-65-89'] with allowed return codes [0] (shell=False, capture=True)
256 - atomic_helper.py[DEBUG]: Atomically writing to file /var/lib/cloud/data/set-hostname (via temporary file /var/lib/cloud/data/tmpaxqv667c) - w: [644] 80 bytes/chars

```

The contents of `/var/lib/cloud/data/set-hostname`

```
{
 "fqdn": "host-10-2-65-89.openstack.build.",
 "hostname": "host-10-2-65-89"
}

```

The contents of `/var/lib/cloud/data/previous-hostname`

```
host-10-2-65-89.openstack.build.

```

I'm using `Cloud-init v. 21.4-0ubuntu1~20.04.1`

---

I've tried deleting all but the FQDN and it still doesn't set the `/etc/hostname` to the fqdn.

# Answer
**This is a bug in `cloud-init`, I've filed it upstream as Bug #1966533**

I was getting silent failures because the hostname was

```
host-10-2-65-89.openstack.build.

```

Instead of

```
host-10-2-65-89.openstack.build

```

What actually happens is,

```
$ sudo hostname host-10-2-65-89.openstack.build.
hostname: the specified hostname is invalid

```

---

Background, what I was actually doing was using Terraform,

```
fqdn: "${fqdn}"

```

And then setting the `fqdn` like this,

```
fqdn = openstack_networking_port_v2.port_company.dns_assignment[0].fqdn

```

But the problem is the terraform was return the actual FQDN,

```
host-10-2-65-89.openstack.build.

```

But you can't provide that to hostname. My solution was to remove the dot (`.`) with `trimsuffix`

```
fqdn = trimsuffix(
  openstack_networking_port_v2.port_company.dns_assignment[0].fqdn,
  "."
)

```
> 3 votes

---
Tags: ubuntu, cloud-init
---


# Question
Title: How to find an AMI in the new AWS console version?
I am trying to create an instance in the Launch Configuration menu, but it is impossible to find the needed AMI from the dropdown.

* I expected following menu:

* But currently AWS provides the following possibilities ()selection from the dropdown:

* I can't find the needed AMI from the list (Linux for example) The bottom line is that I can't even use the `search` dropdown. It does not find existing AMI's:

**It's confusing and I do not know how to use AWS now...**

# Answer
> 3 votes
# Ubuntu AMIs

Ubuntu maintains a page with the latest AMIs in each region for your convenience and reference.

# Rocky Linux

Rocky Linux also maintains a handy page for finding AMIs in each region.

# Amazon or other specific vendor AMIs

The AWS console can be rather limiting, but you can find the latest AMI with the awscli tool.

Get a list of Amazon images:

```
aws ec2 describe-images --owners amazon --filters "Name=name,Values=amzn*" --query 'sort_by(Images, &CreationDate)[].Name'

```

Use AWS Systems Manager paramater store to find the latest one with the name you want:

```
aws ssm get-parameters --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 --region us-east-1

```

Then make sure you always use the latest when launching an instance:

```
aws ec2 run-instances --image-id $(aws ssm get-parameters --names /aws/service/ami-amazon-linux-latest/amzn2-ami-hvm-x86_64-gp2 --query 'Parameters[0].[Value]' --output text) --count 1 --instance-type m4.large

```

---
Tags: amazon-web-services
---


# Question
Title: How can I efficiently scale a data lake?
I am running into problems with AWS Athena data lakes. They were initially very quick, but as we've scaled, they're performing poorly and are becoming expensive and inefficient. What can we do to solve this without reducing data size?

# Answer
> 1 votes
You're definitely not alone in this problem, it's well known. So much so that there's numerous articles around it. I personally have had issues with partitioning, queries failing completely and overall poor performance. The aforementioned article mentions Varada which I think could be a good solution. How it works:

> What they do is to break down a large dataset into what they call a nano block of 64k rows. Their technology looks at each nano block (thus original dataset) and automatically chooses an index for each nano block.

---
Tags: amazon-web-services, amazon-s3, databases
---


# Question
Title: What is a secure way for storing AWS IAM credentials?
When creating a new user in AWS IAM, it generates access keys for that user. Where is a secure and readily accessible location to store these credentials?

# Answer
Well there are a lot of places to keep them secure

* you can use something like hashicorp vault to keep those kind of secrets
* you can use something like lastpass and my personal favorite is enpass

But first you need to define your needs so you can choose the right tool to use and integrate with your daily use
> 1 votes

# Answer
Keepas can be another option where you can store your secret keys/passwords . Here is the document document for your reference : https://aws.amazon.com/marketplace/pp/prodview-wr72ujcv4dhmo
> 0 votes

# Answer
In case you can provision the IAM accounts programmatically, for example using Terraform, you could ask users for their PGP public keys (or fetch them from company's PGP server) and encrypt their credentials immediately upon provisioning with PGP.

You can then safely distribute the encrypted credentials to the user, however you see fit, store them in Keepass/1password/other password manager where you can share them with the users, or just leave the encrypted credentials in CI pipeline and send them via encrypted private messenger without compromising security.
> 0 votes

---
Tags: aws-iam
---


# Question
Title: Why can't my playbook find the community.docker modules
I am trying to write a playbook that will download a docker image and run it on a group of servers. I have a python 3.6 virtualenv created and installed ansible into it. I then ran:

```
ansible-galaxy collections install community.docker

```

I verified that the collection was downloaded and is in my `.ansible/collections/ansible_collections folder.`

My playbook looks like:

```
    - name: test docker
      hosts: all
      collections:
        - community.docker
      become: true

      tasks:
        - name: log into the ocir registry
          docker_login:
            registry_url: "{{ ocir_registry }}"
            username: "{{ docker_user }}"
            password: "{{ ocir_auth_token }}"
            state: present

        - name: ensure a container is running
          docker_container:
            name: LDAPServer
            state: started
            image: "{{ docker_image }}"
            pull: true
            restart_policy: always
            ports:
              - "3000:3000"

```

When I run this playbook with:

```
    ansible-playbook docker.yml

```

I keep getting the following error:

```

"/tmp/ansible_community.docker.docker_login_payload_KMQkW7/ansible_community.docker.docker_login_p
        ayload.zip/ansible_collections/community/docker/plugins/module_utils/common.py", line 33,
        in <module>
        from docker import __version__ as docker_version
    ImportError: No module named docker

```

I just double-checked and I did install docker in my virtualenv, and I have the virtualenv activated when running the playbook.

What am I doing wrong, or haven't I done to set things up correctly?

# Answer
In order to use the Docker Ansible modules, you need the Python docker module installed.

See the Ansible Docker requirements:

> Most of the modules and plugins in community.docker require the Docker SDK for Python. The SDK needs to be installed on the machines where the modules and plugins are executed, and for the Python version(s) with which the modules and plugins are executed.

So you need to do a `pip install docker` (assuming python3) before using the `community.docker` collection.
> 1 votes

---
Tags: docker, ansible
---


title: Create AWS SG and use it

text:
# Question
Title: Create AWS SG and use it
I am trying to create a security group (SG) using Terraform and then use it for an AWS instance.

My config looks like

```
resource "aws_security_group" "my_sq" {
  vpc_id = aws_vpc.mainvpc.id
  name = "my_sg"
  ingress {
    cidr_blocks = [
      "0.0.0.0/0"
    ]
    from_port = 22
    to_port = 22
    protocol = "tcp"
  }

}

resource "aws_instance" "my_new_instance" {
  ami = "AMI-ID"
  instance_type = "t2.micro"
  security_groups = ["my_sg"]
  }
}

```

I tried assigning the SG by name and id. When I ran `terraform plan` everything is all right. When I tried to apply settings `terraform apply` I see this error:

```
│ Error: Error launching instance, possible mismatch of Security Group IDs and Names.

```

How do I use the new SG which I created in the config file?

# Answer
You may not create those two things in the right order, since there is no dependency declared between them. Terraform doesn't know that the security group with name "my-sg" is the same as the security group that it is creating with that name.

You could use a reference to the sg resource in your instance declaration:

```
resource "aws_security_group" "my_sg" {
  vpc_id = aws_vpc.mainvpc.id
  name = "my_sg"
  ingress {
    cidr_blocks = [
      "0.0.0.0/0"
    ]
    from_port = 22
    to_port = 22
    protocol = "tcp"
  }

}

resource "aws_instance" "my_new_instance" {
  ami = "AMI-ID"
  instance_type = "t2.micro"
  security_groups = [aws_security_group.my_sg.name]
  }
}

```

or you can declare a dependency:

```
resource "aws_security_group" "my_sg" {
  vpc_id = aws_vpc.mainvpc.id
  name = "my_sg"
  ingress {
    cidr_blocks = [
      "0.0.0.0/0"
    ]
    from_port = 22
    to_port = 22
    protocol = "tcp"
  }

}

resource "aws_instance" "my_new_instance" {
  ami = "AMI-ID"
  instance_type = "t2.micro"
  security_groups = ["my-sg"]
  }
  depends_on = [aws_security_group.my_sg,]
}

```

If you make the reference to another resource (example 1), then Terraform can itself determine the dependency, and wait for the creation of the security group before creating the instance.
> 3 votes

---
Tags: amazon-web-services, terraform, amazon-ec2, aws-security-group
---


# Question
Title: Can't run apt-get update in an Ubuntu 16.04 docker container
Everytime I run `apt-get update -y` in my Ubuntu 16.04 docker container, I get the following error:

```
Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease
Hit:2 http://archive.ubuntu.com/ubuntu xenial InRelease
Hit:3 https://nvidia.github.io/libnvidia-container/stable/ubuntu16.04/amd64  InRelease
Hit:4 http://archive.ubuntu.com/ubuntu xenial-updates InRelease
Hit:5 https://nvidia.github.io/libnvidia-container/experimental/ubuntu16.04/amd64  InRelease
Hit:6 https://nvidia.github.io/nvidia-container-runtime/experimental/ubuntu16.04/amd64  InRelease
Ign:7 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  InRelease
Ign:8 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  InRelease
Ign:9 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  InRelease
Hit:10 http://archive.ubuntu.com/ubuntu xenial-backports InRelease
Ign:11 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Release
Ign:12 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Release
Ign:13 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Release
Ign:14 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Packages
Ign:15 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Packages
Ign:16 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Packages
Ign:14 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Packages
Ign:15 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Packages
Ign:16 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Packages
Ign:14 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Packages
Ign:15 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Packages
Ign:16 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Packages
Ign:14 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Packages
Ign:15 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Packages
Ign:16 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Packages
Ign:14 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Packages
Ign:15 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Packages
Ign:16 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Packages
Ign:14 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Packages
Ign:15 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Packages
Ign:16 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Packages
Err:14 https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Packages
  404  Not Found
Err:15 https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Packages
  404  Not Found
Err:16 https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Packages
  404  Not Found
Reading package lists... Done
W: Target Packages (Packages) is configured multiple times in /etc/apt/sources.list.d/libnvidia-container.list:2 and /etc/apt/sources.list.d/nvidia-container-runtime.list:1
W: The repository 'https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64  Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64  Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
W: The repository 'https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64  Release' does not have a Release file.
N: Data from such a repository can't be authenticated and is therefore potentially dangerous to use.
N: See apt-secure(8) manpage for repository creation and user configuration details.
E: Failed to fetch https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64/Packages  404  Not Found
E: Failed to fetch https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64/Packages  404  Not Found
E: Failed to fetch https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64/Packages  404  Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.

```

This is my `/etc/apt/sources.list.d/nvidia-docker.list`:

```
deb https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86_64 /
deb https://nvidia.github.io/nvidia-container-runtime/ubuntu16.04/x86_64 /
deb https://nvidia.github.io/nvidia-docker/ubuntu16.04/x86_64 /

```

Where am I going wrong?

# Answer
> 1 votes
This is because ubuntu16 is no longer supported by Nvidia

They have shut this repo down

visiting the error page shows they have marked it as unsupported https://nvidia.github.io/libnvidia-container/ubuntu16.04/x86\_64

check out the below for supported versions https://nvidia.github.io/libnvidia-container/

---
Tags: docker, linux, ubuntu
---


# Question
Title: How to manage Env Variables / Configs on a Helm Chart + Github Actions + Github Secrets
I have a Rest API in Python that has several Configuration variables that are setup at startup time.

This variables are read from the environment variables of the host machine.

Our production environmant is on a kubernetes cluster that is deployed with github actions and a helm chart.

The problem I have is that whenever a new config variable is added. I have to add it in 3 places.

1. On the actual python application code
2. On the helm chart's `values.yaml`
3. On the github secrets of the repo so that during the deployment it sets the `values.yaml` with the value of the secret.

How to manage it and make this process a bit less error prone? Since now it's very likely that somebody forgest to add the new config on any of the above 3 places and break the deploy pipeline.

# Answer
The process itself looks fine to me, what I would do is:

1. Add a test in the last step that runs helm template on a few targets that may get this parameter, to verify it is not failing.
2. Add the parameter in a way that will not break the pipeline if it is missing, or put a default in the code that will not cause damage.
> 1 votes

# Answer
Here's what I'd try to do:

1. Move the environment variables to .env file, which can be read by the Python application e.g. by python-dotenv and make this file the single source of truth. Depending on the info you keep in the .env file, you could have one committed to the repo (.env), which doesn't contain any sensitive info, second one .env.local, being .gitignored, containing sensitive data.
2. Write a simple pre-commit hook that would do two things:
    1. patch the .values.yaml file using e.g. yq with values from the .env file (which can be sources in a bash script),
    2. utilize GitHub CLI and set the secrets, again using the .env file.

The script that would be run by the pre-commit hook could be externalized to `bin/` so you can run it on-demand, without having to commit the code.
> 1 votes

---
Tags: configuration-management, secret-management
---