# Question
Title: Unable to understand the logic behind this terraform function
So i was going through a code of some one else and just could not make sense of it,, if may be some can of this,

`"${replace(jsonencode(list(local.cadvisor_container_definition)), "/\"(\\d{2,5}|true|false)\"/", "$1")}"`

locals {
  cadvisor_container_definition = {
    name         = "${local.cadvisor_container}"
    image        = "google/cadvisor:latest"
    portMappings = [
      {
        containerPort = "8080",
        hostPort = "8080"
      }
    ]
    mountPoints = [
      {
        sourceVolume  = "root",
        containerPath = "/rootfs",
        readOnly      = "true"
      },
      {
        sourceVolume  = "var_run",
        containerPath = "/var/run",
        readOnly      = "false"
      },
      {
        sourceVolume  = "sys",
        containerPath = "/sys",
        readOnly      = "true"
      },
      {
        sourceVolume  = "var_lib_docker",
        containerPath = "/var/lib/docker",
        readOnly      = "true"
      },
      {
        sourceVolume  = "cgroup",
        containerPath = "/sys/fs/cgroup",
        readOnly      = "true"
      }
    ]
    memory            = "${lookup(var.cadvisor_memory, var.env)}"
    memoryReservation = "${lookup(var.cadvisor_memory_reservation, var.env)}"
  }
}```
</code></pre>

# Answer
It seems like the intent of this expression is to take the result of encoding that data structure which contains numbers and boolean values in quotes and produce a new version with those values unquoted.

For example, anywhere the result contains the sequence `"true"` it would be replaced with `true`, quotes removed to produce a real JSON boolean value rather than a string.

A more direct way to achieve that result would be to set the values in `cadvisor_container_definition` to be of the intended type immediately:

```
locals {
  cadvisor_container_definition = {
    name         = local.cadvisor_container
    image        = "google/cadvisor:latest"
    portMappings = [
      {
        containerPort = 8080
        hostPort      = 8080
      },
    ]
    mountPoints = [
      {
        sourceVolume  = "root"
        containerPath = "/rootfs"
        readOnly      = true
      },
      {
        sourceVolume  = "var_run"
        containerPath = "/var/run"
        readOnly      = false
      },
      {
        sourceVolume  = "sys"
        containerPath = "/sys"
        readOnly      = true
      },
      {
        sourceVolume  = "var_lib_docker"
        containerPath = "/var/lib/docker"
        readOnly      = true
      },
      {
        sourceVolume  = "cgroup",
        containerPath = "/sys/fs/cgroup"
        readOnly      = true
      }
    ]
    memory            = var.cadvisor_memory[var.env]
    memoryReservation = var.cadvisor_memory_reservation[var.env]
  }
}

```

I changed the literal values in the above to their numeric or boolean equivalents. I can't see from what you shared whether `memory` and `memoryReservation` are also supposed to be numbers, but if they are then that can be arranged by writing `tonumber(var.cadvisor_memory[var.env])` and `tonumber(var.cadvisor_memory_reservation[var.env])`.

The above examples use Terraform 0.12 syntax. It's possible that the configuration you are looking at here is written for Terraform 0.11 instead, which would explain this unusual string substitution technique because the Terraform 0.11 version of `jsonencode` suffered from Terraform 0.11's tendency to convert all primitive-typed values to strings. If you are still using Terraform 0.11 then the strange expression you found is likely still needed, but if you are using Terraform 0.12 then a change like I described above should work to simplify the confusing expression to just a straightforward `jsonencode`:

```
jsonencode([local.cadvisor_container_definition])

```
> 1 votes

---
Tags: terraform, terraform-module
---


# Question
Title: How does one set up OpenStack Swift (object) storage container-specific API keys with Ansible?
I've got the following OpenStack commands for setting up Swift object storage containers:

```
openstack container create $CONTAINER_NAME
openstack container set --property Full-Key=$CONTAINER_API_KEY $CONTAINER_NAME

```

I'd like to convert them to Ansible so they can be run as part of a role.

For the first command, I know I can use the os\_object module for creating the container, but it doesn't appear to provide a parameter for setting a property, which in this case is for setting up key access to the container. (It's based on the documentation at Container Specific API Keys.)

So how would I execute the second command within Ansible natively, without resorting to running the shell command directly?

# Answer
> 0 votes
After further research, it's evident that it's currently impossible to do this natively. Therefore, we have to do this in the shell with the `command` module:

```
command: openstack container set --property Full-Key={{ KEY }} {{ CONTAINER }}

```

---
Tags: ansible, storage, openstack
---


# Question
Title: How to save groovy output to pipeline variable
I have the below code in the jenkisn pipeline:

```
   stage ("amd_distribution_input_transformation"){
        steps{
            script{
                    amd_distribution_input_transformation url: params.DOMAIN_DESCRIPTOR_URL, secret: params.CERDENITAL_ID
                }
            }
        }

```

amd\_distribution\_input\_transformation.groovy content:

```
def call(Map parameters)
{
    def CREDENITAL_ID = parameters.secret
    def DOMAIN_DESCRIPTOR_URL = parameters.url
    sh '''
        python amd_distribution_input_transformation.py
      '''
    }
}

```

in the amd\_distribution\_input\_transformation.py some code is running, and at the end, it returns object named 'artifacts\_list'

my question is, how can I assign the return obect from groovy file to pipeline variable. BTW, if it helps, I can write the output to json file from the python code (and here I'm stuck on how eventually assign that file to the pipeline variable)

# Answer
Have you tried setting `returnStdout` to true in the `sh()` command per the Jenkins documentation?

You can then assign that output to a variable within your pipeline:

```
def artifactsList = sh(label: 'Running amd_distribution_input transformation', returnStdout: true, script: 'python amd_distribution_input_transformation.py')

```

and check what kind of object that variable is, and act on it accordinly

```
echo(artifactsList.getClass())

```
> 1 votes

---
Tags: jenkins-pipeline, python, groovy
---


# Question
Title: Pooled Build Server and Multibranch development to stabilize the trunk?
I'm a Dev in a big (40+ Devs) old (16+Years) SW project. We still work with SVN as source control but we are getting after automation as much as we can.

One problem that hounts us is, that 40 devs check in stuff into the trunk without a fair chance of running all the tests. Our test suites consists of JUnit unit tests, Junit integration tests that run inside our app server and test business processes and some test tool blackbox tests that run after the deployment. Unittests run 10Min, Integration tests 40Mins and test tool tests run all night long.

You can imagine that for each check in, not ALL the tests and most of the time also not all of the Integration tests are run. Resulting in regular test failures on our CI environment (thats the good part: we have one!) But with 40 devs there is to much going on at the same time, so nobody knows for sure why a test failed. "Was it me? I changed something at another part of the app. This could not be the case, right...?"

I try to promote a branch based dev process with short(!) lived branches that could be testet before merging them into the trunk. Switching to Git is part of it but I would not be religious about it. Goal is that the trunk is always green. But for this I need an infrastructure that can support up to 60 branches.

I'm dreaming up a pooled build server system, where each team could request that a certain branch (maybe in a given revision/checkIn-ID) should be build and tested (maybe with a given testsuite). All the build requests get into a queue and are processed by the build server fleet. (Currently we have the problem that not enough hardware is available to scale the build infrastructure. Pooling the available hardware seems to be the best approach in my mind.)

What do you think about this approach? How would you do this? Is https://jenkins.io/doc/book/pipeline/multibranch/ the right path down the road?

Am I going down the wrong path here? I know that CI says that everybody checks in into trunk at least every day. But, damn, there are so many side effects when 40 devs check in together and the software (especially the data model) does not prevent side effects between parts of the application. We have to start somewhere. But where?

# Answer
Attempting to run your validation on branches, before being integrated into the trunk won't get you very far. The reason is that fundamentally such validations are performed in isolation, most often they will not actually reflect the trunk picture after the branch is merged.

The only case in which the branch validation is, well, valid, is if no other change (of any kind) makes it into the trunk between when the validation started (assuming the latest trunk changes are merged into the branch at that time) and when the branch is merged back into the trunk.

If any other change makes it into the trunk while the validation is in progress it won't be taken into consideration, leaving room for breakage when the branch is merged into the trunk. To prevent that you'd have to merge again the trunk into the branch - to pick up the latest trunk changes - and re-run the validation.

Effectively this means the branch validations (or at least the successful ones allowing the branch to be merged into the trunk) need to be **entirely serialized** if you want to keep the trunk stable. Which may be manageable for a while if you keep a reasonably small number of such branches (maybe multiple devs working on the same branch?). How many branches? Depends on how stable you want the trunk to be: if you want to always pass your integration tests, your serialized validations would need to run the integration tests. At 40 min each you can support max ~60 branches (assuming the merges are automated and can be performed 24x7, which isn't trivial). Without automated merges the number of branches will be a lot smaller. This won't scale, the branch lifespans will gradually increase, diverging more and more from CI and will eventually wear down your team.

The same reasoning explains why devs shouldn't actually have to run all tests if they are too lengthy - by the time they complete them the results may be invalidated by other changes committed to the trunk. Also - results tests done by developers are often not that relevant ("But it built fine in my workspace!").

There are only 2 possible approaches (that I know of) to get relevant validations:

* the ones performed by the traditional CI tools - performed **after** the changes are committed to the trunk, reflecting the actual trunk reality, with all changes integrated. The only problem is if these validations fail - effectively blocking further development on the trunk until the culprits are found and the regressions fixed. Which may be (very) difficult in larger scale projects - as you already noticed.
* the ones performed by **gating** CI tools - only 2 out there that I know of: Zuul and ApartCI (which is my baby). These validations are performed **prior** to merging changes into the trunk, but in an orchestrated manner, ensuring they remain relevant - taking into consideration all other changes in flight that will eventually be present in the trunk by the time a particular change is merged into the trunk. The actual change merging into the trunk is entirely automated, executed by the CI tool itself, immediately after the relevant validations are completed successfully. These are, IMHO, the only ones capable of guaranteeing trunk stability in larger scale projects.

Pooling resources is a good thing - give them to the CI tool - that's how you maximize their usage.
> 1 votes

---
Tags: jenkins, continuous-integration, build-pipeline
---


# Question
Title: Recommended Design - Communication among Jenkins + Vault + App
---

---

**Design A: CD tool communicate with Vault to fetch App secrets on its behalf**

* Jenkins ( *CD tool* ) computes the Vault path according to the app/project & according to which environment the app will be deployed (qa, staging, prod,...).
* Jenkins uses the computed path as payload to make a REST call to Vault.
* Jenkins get key-value pairs as an HTTP response
* Jenkins injects these values as ENV variables in the server/container where the app is deployed.
* Then, Jenkins deploys the app itself ( or restart it) after having ENV vars ready.
* The application is designed to read these values from ENV variables ( e.g.: `process.env` in case of NodeJs App ).
* If the server/container is restarted/rebooted, it is configured to call the same Vault API to bring again the key-value pairs before the app is started.

**Design B: App communicates directly with Vault to fetch its secrets**

* Jenkins ( *CD tool* ) deploys everything except app's secrets.
* Application is environment-aware. It fetches its secrets from Vault thru API call.

---

**Design A Concerns**

* In-transit Encryption of secrets while injecting them from CD tool to App environment.

**Design B Concerns**

* High Availability of Vault is crucial.If it goes down for any reason (.e.g: Network outage), All applications will not be functional.

What's the best design according to you

# Answer
> 1 votes
My choice would be design B.

It's true that the availability of the Vault would be crucial, but Jenkins' availability would similarly be crucial in design A. So from this perspective the decision would normally be driven by the comparison of the two availability figures, if available. Based on its functionality I suspect that Jenkins is likely to be a more complex tool than Vault (but I don't have any hard data supporting it). If that's correct then Jenkins would normally have a lower availability/reliability score.

A slightly different aspect of the dependency on Jenkins in design A isn't exactly appealing: Jenkins' job normally ends once it successfully deploys a certain version of the app, it isn't really designed to be around to "support" that app version during its expected lifetime (that includes restarting the app). Other tools are better suited to perform app monitoring and management in general. The app's independence from Jenkins that design B brings is, IMHO, an advantage from this perspective.

---
Tags: jenkins, continuous-deployment, hashicorp-vault, high-availability, secret-management
---


# Question
Title: jenkins-x returns error 1 when I try to create a cluster
I'm following this tutorial. I'm running the jenkins-x cli on a aws ec2 instance where I installed kubenetes and jx. I get to the part where I run the command `jx create cluster eks --cluster-name=konsek-cloud --skip-installation=true --verbose` and it errors out with

```
DEBUG: eksctl is already available on your PATH at /home/ec2-user/.jx/bin/eksctl
DEBUG: aws-iam-authenticator is already available on your PATH at /home/ec2-user/.jx/bin/aws-iam-authenticator
DEBUG: Dependencies to be installed:
DEBUG: kubectl is already available on your PATH at /usr/bin/kubectl
DEBUG: helm is already available on your PATH at /home/ec2-user/.jx/bin/helm
error: exit status 1

```

I cannot figure out how to get more information out of it or figure out what is wrong. Can someone please suggest ways to "turn up" the logging output? or know something I can try to fix this?

# Answer
> 1 votes
I just experienced the same issue. It seems the eksctl that is being installed upon first use is not working for me. I went to the eksctl site and installed it from there (https://eksctl.io/introduction/installation/)

With the jx eksctl binary :

```
➜  ~ jx create cluster eks --cluster-name=konsek-cloud --skip-installation=true --verbose
DEBUG: eksctl is already available on your PATH at /Users/<user>/.jx/bin/eksctl
DEBUG: aws-iam-authenticator is already available on your PATH at /Users/<user>/.jx/bin/aws-iam-authenticator
DEBUG: Dependencies to be installed:
DEBUG: kubectl is already available on your PATH at /usr/local/bin/kubectl
DEBUG: helm is already available on your PATH at /usr/local/bin/helm
DEBUG: brew is already available on your PATH at /usr/local/bin/brew
error: exit status 1

```

With the manually installed binary :

```
➜  ~ jx create cluster eks --cluster-name=konsek-cloud --skip-installation=true --verbose
DEBUG: eksctl is already available on your PATH at /usr/local/bin/eksctl
DEBUG: aws-iam-authenticator is already available on your PATH at /usr/local/bin/aws-iam-authenticator
DEBUG: Dependencies to be installed:
DEBUG: kubectl is already available on your PATH at /usr/local/bin/kubectl
DEBUG: helm is already available on your PATH at /usr/local/bin/helm
DEBUG: brew is already available on your PATH at /usr/local/bin/brew
Creating EKS cluster - this can take a while so please be patient...
...

```

---
Tags: amazon-web-services, kubernetes, jenkins-x
---


# Question
Title: How to install aws cli on a self-hosted azure DevOps Windows agent?
I have quite some difficulties installing **aws cli** on a self-hosted Azure DevOps Windows agent as part of a pipeline. Does anyone have a working solution?

## What I have tried so far:

### a) install aws cli via pip

Firstly, I install Python (tried both; Python2 and Python3) within the pipeline (Powershell) by downloading and extracting a tar file. Example:

```
(New-Object System.Net.WebClient).DownloadFile("$(python3_download_url)", ".\Python.tar")
Expand-7Zip .\Python.tar .

```

Secondly, I use the freshly installed Python for installing aws cli. Example:

```
$(PYTHON364EXE) -m pip install awscli

```

Aws cli is then installed in the `SCRIPTS` directory, as expected.

When I try to execute aws.cmd, I receive an error message:

```
File association not found for extension .py
'""' is not recognized as an internal or external command,
operable program or batch file

```

Similar issues are described here: , , . The suggested solution is, to associate python files with the freshly installed python executable:

```
assoc .py=pyautofile
ftype pyautofile="$(PYTHON364EXE)" "%1" %*

```

I do not feel this is appropriate for an azure agent:

1. It does not feel right to use Admin permissions for a simple build
2. I am worried that the proposed change would impact *all* agents and the entire host server, while I need it for only one specific agent.

Another option would be, to install the required tools system-wide (python, aws). That would, however, make it difficult to try out a specific (new?) version of python or aws cli.

### b) install aws cli via extracted msi package

... extracted the contents of the msi package via `lessmsi`, then run the aws cli:

```
C:\AWS> aws
failed to create process.

```

This obviously does not work either. - It actually does work properly with aws cli version 2, but not with aws cli version 1. Version 2 is not recommended for use in production yet.

### c) Use AWS Tools for Microsoft Visual Studio Team Services

This is an extension for Azure DevOps, maintained by AWS. It works, but a) we have legacy scripts of a certain complexity using aws cli we cannot migrate, and b) there might be features available in aws cli which are not available in the extension.

### d) call aws as python module

```
$(PYTHON364EXE) -m awscli --version

```

This works, but it means I would have to adapt and modify the legacy script.

# Answer
> 3 votes
The best way to have AWS CLI tooling installed to your custom agent is by installing from Visual Studio Marketplace. AWS CLI is part of AWS Tools for Azure DevOps Service and Azure DevOps Server (the on-premise Azure DevOps) under the name of "AWS Tools for Microsoft Visual Studio Team Services".

Installing this AWS Tools will enable AWS tooling on CI (build pipeline) and CD (release pipeline).

This is the official landing page of AWS Tools in Visual Studio Marketplace: https://marketplace.visualstudio.com/items?itemName=AmazonWebServices.aws-vsts-tools

Official documentation is available at: https://docs.aws.amazon.com/vsts/index.html

---
Tags: azure-devops, python, azure-pipelines, aws-cli
---


# Question
Title: docker container does not open port for php:latest
I'm working with Docker containers for a while know, but I do not notice, what I am doing wrong here:

In my current folder on the host system, I have a simple `index.php` file which contains:

```
<?php echo 'hi'; ?>

```

Why is this not working?

```
docker run --name php -v $(pwd):/app -w /app -p 8080:8080 -d --rm php:cli php -S localhost:8080 -t . index.php

```

This results in an empty reply:

```
$ curl localhost:8080
curl: (52) Empty reply from server

```

If I `exec` in the container and apply the `curl` there it succeeds in replying `hi`.

So, what am I missing?

# Answer
> 1 votes
Try to use `php -S 0.0.0.0:8080`:

```
docker run --name php -v $(pwd):/app -w /app -p 8080:8080 -d --rm php:cli php -S 0.0.0.0:8080 -t . index.php

```

*Explanation*:

The `localhost`(or `127.0.0.1`) address is attached to the container namespace, so anything outside of this namespace cannot have access to it. Furthermore, the `0.0.0.0` (in this context) is a specific address that means *all IPv4 addresses on the local machine*. If you check inside the container, you could see:

```
$ ip a
lo:
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
eth0:
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

```

The `eth0` interface is the interface used when you forward the port. As you don't specify the IP address of the container, you can use the `0.0.0.0` to listen on the `eth0` interface.

I don't know if I make my self clear, but you can (should) read this if you want to know more:

---
Tags: docker, php
---


# Question
Title: How do you reuse a specific volume in Kubernetes?
Let's say that you wanted to create a Jenkins `Deployment`. As Jenkins uses a local XML file for configuration and state, you would want to create a `PersistentVolume` so that your data could be saved across `Pod` evictions and `Deployment` deletions. I know that the `Retain` `reclaimPolicy` will result in the data persisting on the detached `PersistentVolume`, but the documentation says this is just so that you can manually reclaim the data on it later on, and seems to say nothing about the volume being automatically reused if its mounting `Pod`s are ever brought back up.

It is difficult to articulate what I am even trying to ask, so forgive me if this seems like a nebulous question, but:

* If you delete the Jenkins deployment, then later decide to recreate it where you left off, how do you get it to re-mount *that exact PersistentVolume* on which *that specific XML configuration* is still stored?
* Is this a case where you would want to use a `StatefulSet`? It seems like, in this case, Jenkins would be considered "stateful."
* Is the `PersistentVolumeClaim` the basis of a volume's "identity"? In other words, is the expectation for the `PersistentVolumeClaim` to be the stable identifier by which an application can bind to a specific volume with specific data on it?

# Answer
> 2 votes
Yes, the `PersistentVolumeClaim` is the workhorse here. You can evict pods, delete deployments, etc., to your heart's content and so long as you don't delete the PVC, your data will be there.

You don't need to use `StatefulSet` for this. From the docs:

```
Unlike a Deployment, a StatefulSet maintains a sticky identity for each of their Pods. These pods are created from the same spec, but are not interchangeable: each has a persistent identifier that it maintains across any rescheduling.

```

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

You might want that anyway, but it does nothing for keeping your PV/PVCs around.

---
Tags: kubernetes, kubernetes-volume
---


# Question
Title: AWS-CLI : to download IAM user with policy
aws : A single command(aws-cli) to get the IAM user name with the what all permission he/she have , would be great if can be downloaded as an excel or csv ?

# Answer
You can use the below command to get all the policy attached to an iam user. (You must be having permissions to list-users and list-user-policies before using the below commands)

```
aws iam list-attached-user-policies --user-name username | jq -r '.AttachedPolicies[].PolicyArn' | awk -F'policy/' {'print $NF'}

```

(Make sure to replace `username` with some actual iam username in the above command)

If you want to get the list of all users and user-policies attached to those users, you can use the below one line command. The below command will get all the `iam users` and then will find the `user-policies` attached to that user and will put in a CSV file. The first column will contain `username` and second column will contain \`policies separated by spaces.

```
for u_name in $(aws iam list-users | jq -r '.Users[].UserName') ; do permissions="$(aws iam list-attached-user-policies --user-name ${u_name} | jq -r '.AttachedPolicies[].PolicyArn' | awk -F'policy/' {'print $NF'} | tr '\n' ' ')" ; echo "${u_name}, ${permissions}" >> iam_users.csv ; done

```
> 1 votes

# Answer
If you are looking to get the results for a single user then you want list-attached-user-policies. However, this will result in the value being returned as json. To get it into a csv style format check out the guide on how to control the CLI output. You should be able to get the output into a delimited format for programmatic parsing. I do not believe you'll be able to generate an .xlsx file directly from the AWS CLI.
> 0 votes

---
Tags: amazon-web-services, aws-cli
---


# Question
Title: How to set Terraform Cloud environment variables from the command line?
Terraform Cloud is a SaaS platform for managing Terraform runs, state and variables. Once you have created a project you can set environment variables via the following page:

```
https://app.terraform.io/app/[organization]/workspaces/[workspace]/variables

```

These environment variables are then set when Terraform executes a run. Out of an abundance of a desire to automate this process, I have scripted the process of generating these environments variables:

```
$SubscriptionId = "<subscription-id>"
$PrincipalName = "http://<service-principal-name>"
$SubscriptionUrl = "/subscriptions/$SubscriptionId"

az cloud set --name "AzureCloud" | Out-Null
az account set --subscription $SubscriptionId | Out-Null

$ServicePrincipal = az ad sp create-for-rbac --role="Owner" --scopes=$SubscriptionUrl --name $PrincipalName | ConvertFrom-Json

$env:ARM_CLIENT_ID=$ServicePrincipal.appId
$env:ARM_CLIENT_SECRET=$ServicePrincipal.password
$env:ARM_SUBSCRIPTION_ID=$SubscriptionId
$env:ARM_TENANT_ID=$ServicePrincipal.tenant

```

This works, however, I would like to extend the script to be able to push the variables to the appropriate Terraform Workspace. I am aware that there is a RESTful Variables API that I can call although I was hoping for a solution that used the `terraform` command-line interface rather than relying on parsing `backend.tf` and calling a REST API from a script.

# Answer
You can use the Terraform Cloud provider to manage workspace configuration in Terraform Cloud in the same way as you might manage any other infrastructure objects with Terraform. For historical reasons the provider is named `tfe`, but it can be used to manage both Terraform Enterprise and Terraform Cloud.

In particular, you can use the `tfe_variable` resource type to manage both Terraform variables and environment variables for a workspace:

```
variable "environment_variables" {
  type = map(string)
}

variable "workspace_id" {
  type = string
}

provider "tfe" {
  hostname = "app.terraform.io" # Terraform Cloud
}

resource "tfe_variable" "test" {
  for_each = var.environment_variables

  key          = each.key
  value        = each.value
  category     = "env"
  workspace_id = var.workspace_id
}

```

To integrate this with the script you've already written, you could perhaps use PowerShell's JSON encoder to produce a file `azure.tfvars.json` with content like this:

```
{
  "workspace_id": "organization/workspace",
  "environment_variables": {
    "ARM_CLIENT_ID": "...",
    "ARM_CLIENT_SECRET": "...",
    "ARM_SUBSCRIPTION_ID": "...",
    "ARM_TENANT_ID": "..."
  }
}

```

You can then run the above Terraform configuration using a command like this:

```
terraform apply -var-file=azure.tfvars.json

```

When considering automation of this sort it's worth considering where the Terraform state snapshots for this "early" Terraform configuration would live. You could potentially create an additional Terraform Cloud workspace for this special configuration and be sure to configure it to *not* use remote operations, so that a local run of this one workspace can then arrange for your other workspace to support remote operations.
> 2 votes

---
Tags: terraform, automation, terraform-workspace
---


# Question
Title: Accessing TPM from inside a Docker Container
I want to implement Vault as part of the DevOps build pipeline to store all the secrets like passwords and certificates. I understand that Vault keeps the data encrypted but **I have been asked to look for a way to store the Vault's data in TPM as "it would be more secure and temper-protected". First off, is this argument valid?** Vault already stores the encrypted data so is this additional layer of encryption really needed?

I read the Hashicorp's Vault Storage Documentation and it does not directly support any storage over TPM but if I could mount TPM on the container then I could probably use `filesystem` storage to store data inside it.

So, is it possible to mount the host system's Trusted Platform Module (TPM) onto a docker container?

# Answer
I think that the statment "to store the Vault's data in TPM" is *not* valid. The TPM is generally not for storage. There are some slots for storing some values and likely some persitent keys, but it is only a handful. I think TPM 2.0 maybe has more storage than 1.2, which I'm more familiar with, but it is *not* a storage device. It really depends on what and how much of it you need to store.

So while there might be a solution using "on-chip" TPM memory, it is scarce and really slow. The more common pattern is the use the TPM to wrap the keys or data, such that it can be stored in an encrypted blob on actual storage, like disk or network, and only safely unwrapped on THAT MACHINE.

The TPM is not really a "crypto co-processor" nor is it a storage device. It's slow and has minimal resources, for cost.

Having clarified that, it probably is valid to use the TPM, in the proper way, to help securely store the application data, it just depends on how much work it is for you.

Also, if the data doesn't need to be specifically tied to that specific machine, or that specific container, or the installation is on a VM, and the VM might spin up on different hardware, then using the TPM doesn't really make sense because it is a root-of-trust for a specific platform, a specific board or machine.

So to be semantically picky:

1. Storing the Vault data ***in*** the TPM: not valid
2. Storing the Vault data ***with*** the TPM: valid (potentially)

Disclaimer: I don't know anything about Vault or the data you want to store.
> 3 votes

---
Tags: docker, containers, security
---


title: Weavenet for kubernetes is showing 0/2

text:
# Question
Title: Weavenet for kubernetes is showing 0/2
I'm very new to kubernetes. I thought i would first get a basic by making hands dirty by "the hard way" I followed the Linux academy, but at the networking side using weave net i am now stuck ! completely stuck !

kubectl get pods -n kube-system --\>getting the following crash loop back

weave-net-zngp6   1/2       Error              15         37m    ````

some logs for investigating
````kubectl -n kube-system logs weave-net-px6z7 -c weave-npc````

got
```` Get https://10.32.0.1:443/apis/networking.k8s.io/v1/networkpolicies?limit=500&resourceVersion=0: dial tcp 10.32.0.1:443: i/o timeout````

Not sure what it means ? any one can save me ?

# Answer
> 1 votes
weave-net is trying unsuccessfully to talk to the apiserver at 10.32.0.1 (port 443). Verify your master node is indeed at that IP address, it has network connectivity to the node where this is barfing, and that the firewall on the master node has port 443 open.

If they are all running on the same host, verify the apiserver is running.

---
Tags: kubernetes, kubectl
---


# Question
Title: Is web delivery for Node.js apps a thing?
A browser fetches some HTML via HTTP/HTTPS or a file, which may result in fetching a bunch of other resources, including some JavaScript, and then it runs it in a browser environment. I would describe this as *a mechanism for delivering code to the browser*.

Does a similar web-based delivery mechanism exist for Node.js apps? Is there a way to get everything that a node.js app needs to run via HTTP(S) and run in some kind of "node" environment? Sort of like a headless browser, but no HTML?

I'm asking this in the context of updating node.js apps. I really like how a server is in charge of the code that the browser is running. I think the same thing would be great for a certain class of node.js app. It would be great if a node.js could auto-update itself in a straightforward and secure way. This was the first thing that came to mind.

After thinking about this in detail, it seems necessary to *containerize* the application and use a mechanism like Docker to deliver the application in its entirety to the machine that will run the app. There just needs to be some supporting mechanism for ensuring that the app is always up to date. I know there are docker-compose and kubernetes, and things like Watchtower, but is this really the best way for just keeping the application up to date? During development it seems like the mechanism is copying code (like through git) and installing dependent packages (like via NPM). But this is a very developer-centric workflow. Internally, I guess it makes sense to use docker. But for web-delivery, I think a workflow for deploying a node.js-centric application and keeping it up to date would use some other mechanism. What is that mechanism? If docker is *the way*, then maybe I just need to refocus my question to find out how to auto-update docker containers. But I'm fairly certain that exposing the docker API directly isn't really a secure option for web-delivery. I also don't even know how node-based apps are typically delivered. The few that I have actually get installed via the OS's installer mechanism (Like an .msi on Windows) - and that seems very native. Surely there must be a more web-centric approach.

So, how should a Node application auto-update itself (or receive a push update) via the web? (like a browser-based app does)

# Answer
> 1 votes
I'm not aware of a project that provides that out of the box. For local development, it is common to have hot reload. If something at the file level changes, it restarts the server. That is somewhat related, as you could trigger the updates on the file system remotely. Installing libraries could also be simulated by directly updating the `node_modules` folder.

Beside that NodeJs itself provides ways to run arbitrary code without process restarts. One way is to use `eval` (as part of JavaScript). The other is to reload a file. Normally, `require` caches the file after it executed it, but you can purge this cache (`require.cache`), and `require` the file again. That is what the decache library does.

In theory, NodeJs provides all the tools to allow remote code being applied. Of course, from a security perspective, you open yourself to all kind of attacks. Make sure requests come from a trusted source.

> But for web-delivery, I think a workflow for deploying a node.js-centric application and keeping it up to date would use some other mechanism. What is that mechanism? If docker is the way, then maybe I just need to refocus my question to find out how to auto-update docker containers

I would also see it less as a matter of updating a NodeJs application (driven by the application itself), but rather as of a good deployment pipeline (driven from outside). In the end, deployment is nothing else than pushing changes to production.

There are a variety of deployment approaches for NodeJs applications:

If you are not sure, the easiest to get started with are serverless deployments (supported on all major providers: AWS Lambdas, Google Cloud Functions, Azure Functions), or platform as a service solutions (e.g., Heroku). Kubernetes (as a popular framework for deploying Docker containers) is powerful, but has a steep learning curve.

---
Tags: docker, deployment, node.js
---


# Question
Title: Every jenkins build shows error: ln builds/lastSuccessfulBuild /root/.jenkins/jobs/Build/lastSuccessful failed
I migrate my jenkins to new machine succesfully,but after migrate all jenkins job run succesfully. but every build give following error:

```
ln builds/lastSuccessfulBuild /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastSuccessful failed
java.nio.file.DirectoryNotEmptyException: /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastSuccessful
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
    at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
    at java.nio.file.Files.deleteIfExists(Files.java:1165)
    at hudson.Util.createSymlink(Util.java:1193)
    at hudson.model.Run.createSymlink(Run.java:1957)
    at hudson.model.Run.updateSymlinks(Run.java:1938)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:286)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)

ln builds/lastStableBuild /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastStable failed
java.nio.file.DirectoryNotEmptyException: /root/.jenkins/jobs/BeepnRide/jobs/43-Server/jobs/BR1.1.2QA-Build/lastStable
    at sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:242)
    at sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:108)
    at java.nio.file.Files.deleteIfExists(Files.java:1165)
    at hudson.Util.createSymlink(Util.java:1193)
    at hudson.model.Run.createSymlink(Run.java:1957)
    at hudson.model.Run.updateSymlinks(Run.java:1939)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:286)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)

```

# Answer
> 1 votes
I had a similar issue with Jenkins running under Kubernetes with Azure-storage-backed PersistentVolumes.

Azure storage doesn't support symlinks by default.

As per: https://docs.microsoft.com/en-us/azure/storage/files/storage-troubleshoot-linux-file-connection-problems#cannot-create-symbolic-links---ln-failed-to-create-symbolic-link-t-operation-not-supported

To resolve, I added the following to my PersistentVolume yaml:

```
apiVersion: v1
kind: PersistentVolume
metadata:
  name: jenkins-pv
spec:
  storageClassName: azurefile
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteOnce
  azureFile:
    secretName: azure-secret
    shareName: jenkins
  mountOptions:
    - dir_mode=0777
    - file_mode=0777
    - mfsymlinks

```

The **mfsymlinks** allows symlinks to be created on Azure storage.

# Answer
> 1 votes
1. Log on on the master where you have the issue: `ssh jenkins-machine`
2. Shutdown the jenkins master (take care that nothing is running) `sudo service jenkins stop`
3. Find all erroneous directories: `find /var/lib/jenkins/jobs -type d \( -name "last*Build" -o -name "lastStable" -o -name "lastSuccessful" \)`
4. Review the list of erroneous directories
5. Rename them `find /var/lib/jenkins/jobs -type d \( -name "last*Build" -o -name "lastStable" -o -name "lastSuccessful" \) -exec mv {} {}.err \;`
6. Restart jenkins: `sudo service jenkins start` 1 delete these err files if jenkins server is successful.

---
Tags: jenkins
---


# Question
Title: Prevent ${PIPELINESTATUS[0]} from being evaluated in Jenkinsfile
Below is a script I am running in the deploy of a docker container. The `docker ps -a | grep $APP_CONTAINER_NAME` command is returning non-zero when the container was not correctly built/tested, hence the `|| true`.

How can I prevent Jenkins from interpreting the ${PIPELINESTATUS\[0\]} as a variable replacement in the script?

```
sh label: 'Stop and Remove Old Docker Container', script: '''
                    docker ps -a
                    echo $APP_CONTAINER_NAME

                    docker ps -a | grep $APP_CONTAINER_NAME || true
                    DOCKERCODE="${PIPESTATUS[0]}"
                    GREPCODE="${PIPESTATUS[1]}"
                    echo "Docker Command Code: $DOCKERCODE"

                    echo "Grep Command Code: $GREPCODE"

                    if [ $DOCKERCODE -eq 0 ] then;
                        docker stop $APP_CONTAINER_NAME
                        docker rm $APP_CONTAINER_NAME
                    else
                        echo "WARNING: Docker command was empty or had an error"
                    fi
                '''

```

EDIT1: To clarify - this command gives the error `/mnt/data/jenkins/workspace/Test@tmp/durable-4c7b4748/script.sh: 7: /mnt/data/jenkins/workspace/Test@tmp/durable-4c7b4748/script.sh: Bad substitution`

I've tried triple double quotes, escaping the quotes in the shell script, and single quoting the PIPESTATUS command prevents execution.

UPDATE: I tried escaping quite a few things. A single slash results in the above error as well. A double slash results in printing the line and not running the bash command.

```
        sh label: 'Stop and Remove Old Docker Container', script: '''
                docker ps -a
                echo $APP_CONTAINER_NAME

                docker ps -a | grep $APP_CONTAINER_NAME || true
                echo "HERE"
                echo \\${PIPESTATUS[@]}
                DOCKERCODE="\\${PIPESTATUS[0]}"
                GREPCODE="\\${PIPESTATUS[1]}"
                echo "Docker Command Code: \$DOCKERCODE"

                echo "Grep Command Code: \$GREPCODE"

                if [ \\$DOCKERCODE -eq 0 ] then;
                    docker stop $APP_CONTAINER_NAME
                    docker rm $APP_CONTAINER_NAME
                else
                    echo "WARNING: Docker command was empty or had an error"
                fi
            '''

```

Output from above version:

```
TEST
+ docker ps -a
+ grep TEST
+ true
+ echo HERE
HERE
+ echo ${PIPESTATUS[@]}
${PIPESTATUS[@]}
+ DOCKERCODE=${PIPESTATUS[0]}
+ GREPCODE=${PIPESTATUS[1]}
+ echo Docker Command Code: ${PIPESTATUS[0]}
Docker Command Code: ${PIPESTATUS[0]}
+ echo Grep Command Code: ${PIPESTATUS[1]}
Grep Command Code: ${PIPESTATUS[1]}
/mnt/data/jenkins/workspace/Test@tmp/durable-4a5608fa/script.sh: 17: /mnt/data/jenkins/workspace/Test@tmp/durable-4a5608fa/script.sh: Syntax error: "else" unexpected (expecting "then")

```

FINAL ANSWER: The shebang needed to be on the same line as the script: ''' because of how just the first few characters are read by the system to check for #!.

```
sh label: 'Stop and Remove Old Docker Container', script: '''#!/usr/bin/env bash
                docker ps -a
                echo $APP_CONTAINER_NAME

                docker ps -a | grep $APP_CONTAINER_NAME || true
                status=("${PIPESTATUS[@]}")
                DOCKERCODE=${status[0]}
                GREPCODE=${status[1]}
                echo "Docker Command Code: $DOCKERCODE"

                echo "Grep Command Code: $GREPCODE"

                if [ $DOCKERCODE -eq 0 ]; then
                    docker stop $APP_CONTAINER_NAME
                    docker rm $APP_CONTAINER_NAME
                else
                    echo "WARNING: Docker command was empty or had an error"
                fi
            '''

```

# Answer
> 3 votes
There seems to be some syntax issues and also I would suggest to use `bash` instead if using `sh` shell.

Try the below code and see if this works properly. (EDIT - Put shebang on first line to accept as answer)

```
sh label: 'Stop and Remove Old Docker Container', script: '''#!/usr/bin/env bash
                docker ps -a
                echo $APP_CONTAINER_NAME

                docker ps -a | grep $APP_CONTAINER_NAME || true
                status=("${PIPESTATUS[@]}")
                DOCKERCODE=${status[0]}
                GREPCODE=${status[1]}
                echo "Docker Command Code: $DOCKERCODE"

                echo "Grep Command Code: $GREPCODE"

                if [ $DOCKERCODE -eq 0 ]; then
                    docker stop $APP_CONTAINER_NAME
                    docker rm $APP_CONTAINER_NAME
                else
                    echo "WARNING: Docker command was empty or had an error"
                fi
            '''

```

In the above code, I have made changes in the way you were trying to get output of command `docker ps -a | grep $APP_CONTAINER_NAME || true` using `PIPESTATUS` variable. Since `PIPESTATUS` is a special variable, it's value gets updated after every command, so you were not getting any value of `${PIPESTATUS[1]}`

The other change I have made is fixing the syntax issue in the line `if [ $DOCKERCODE -eq 0 ] then;` to `if [ $DOCKERCODE -eq 0 ]; then`

You can test out the code further.

---
Tags: jenkins-pipeline, jenkinsfile, shell-script
---


# Question
Title: How to deal with docker compose's naming convention of named volumes?
When defining named volumes in `docker-compose.yml`, their names are prepended with their parent folder name. This causes a problem when scripts outside of docker compose have to interact with them. The question is, what is the best way to deal with this?

An example scenario would include the following `docker-compose.yml`:

```
version: "3"
services:
  nginx:
    build: ./nginx
    container_name: nginx
    ports:
      - "80:80"
    volumes:
      - jekyll-data:/usr/share/nginx/html:ro

networks:
  backend:

volumes:
  jekyll-data:

```

Where the `jekyll-data` named volume is populated by the following bash script:

```
docker run \
  --name helper \
  --volume="parent_folder_jekyll-data:/web" \
  -it busybox \
  true
docker cp jekyll/web/. helper:/web
docker rm helper

```

In the above case, `parent_folder` is the name of the parent folder. This means that moving the contents to a different folder would break the application. Is there a proper way to deal with this situation?

The abridged output of `docker volume ls` where unnamed volumes have been removed:

```
DRIVER              VOLUME NAME
local               flaskthymedata_grafana-data
local               flaskthymedata_influxdb-data
local               flaskthymedata_postgres-data
local               veleda_grafana-data
local               veleda_influxdb-data
local               veleda_jekyll-cache
local               veleda_jekyll-data
local               veleda_postgres-data
local               veledaio_grafana-data
local               veledaio_influxdb-data
local               veledaio_jekyll-cache
local               veledaio_jekyll-data
local               veledaio_postgres-data

```

# Answer
Docker Prepends the current folder name with all the components name created using docker compose file.

Eg : If the current folder name containing the docker-compose.yml file is test, all the volumes,network and container names will get test appended to it. In order to solve the problem people earlier proposed the idea of using **-p** flag with docker-compose command but the solution is not the most feasible one as a project name is required just after the -p attribute. The project name then gets appended to all the components created using docker compose file.

The **Solution** to the above problem is using the **name** property as in below.

```
volumes:
  data:
    driver: local
    name: mongodata

networks:
  internal-network:
    driver: bridge
    name: frontend-network

```

This volume can be referred in the service section as

```
services:
  mongo-database:
      volumes:
        - data:/data/db
      networks:
        - internal-network

```

The above **name** attribute will prevent docker-compose to prepend folder name.

Note : For the container name one could use the property **container\_name**

```
services:
  mongo-database:
    container_name: mongo

```
> 10 votes

# Answer
What about creating docker volumes yourself instead of letting docker to that for you? https://docs.docker.com/engine/reference/commandline/volume\_create/

If one creates docker volumes yourself then the parent folder will not be automatically concatenated.
> 3 votes

---
Tags: docker, docker-compose, volume
---


title: Kubernetes api service unit file

text:
# Question
Title: Kubernetes api service unit file
i was going through the kubernetes the hard way(kelsey), i see in the API server unit file with a cluster ip range as --service-cluster-ip-range=10.32.0.0/24 \ is it a default ip range ? my VPC CIDR is 172.31.0.0/16 this so should i need to provide this ip instead of default one ?

# Answer
> 1 votes
The `--service-cluster-ip-range` will define the range of IP address that Kubernetes will assign when you create a `ClusterIP` Service. It's different from your VPC CIDR which correspond to your nodes (EC2) IPs. You can let the default `10.32.0.0/24` as is or change it but make sure that it does not overlap with your VPC CIDR.

---
Tags: kubernetes, amazon-ec2, aws-cli, kubectl
---


# Question
Title: Assign Directory Read/Write Privileges to Azure Service Principal from the command line?
I am trying to provision a user in Azure Active Directory using Terraform's AzureAD provider. At the top of the `azuread_user` documentation, it states:

> **NOTE**: If you're authenticating using a Service Principal then it must have permissions to `Directory.ReadWrite.All` within the `Windows Azure Active Directory API`.

I have created the Azure Active Directory User using the `az` command line:

```
az ad sp create-for-rbac --role="Owner" --scopes=$SubscriptionUrl --name $PrincipalName

```

However, this doesn't have `Directory.ReadWrite.All` permissions, as that would be a fairly insane security risk.

How can I add the `Directory.*` permissions to the Service Principal using the `az` command line (by preference) or even the Azure Powershell Cmdlets, I would even be willing to craft a WebRequest if that would help me solve the problem!

# Answer
> 2 votes
What you need here are the az ad app permission commands.

Here is a complete example (assuming powershell):

```
$apiId = "00000002-0000-0000-c000-000000000000" # Windows Azure Active Directory
$apiPermissionsId = "78c8a3c8-a07e-4b9e-af1b-b5ccab50a175" # Directory.ReadWrite.All
$spn = az ad sp create-for-rbac --role="Owner" --name blah | ConvertFrom-Json
az ad app permission add --id $spn.appId --api $apiId --api-permissions ("{0}=Scope" -f $apiPermissionsId)
az ad app permission grant --id $spn.appId --api $apiId
az ad app permission admin-consent --id $spn.appId

```

For reference, Microsoft has a list of the GUIDs used for various applications.

On some occasions I have been unable to find documentation for the magic GUIDs required for a particular app/permission, in which case I would resort to adding them through the portal manually (Azure Active Directory ➔ App Registrations ➔ Required Permissions) and then reading the JSON manifest back to reveal the required GUIDs which can then be plugged into automation.

---
Tags: azure, security, permissions, active-directory
---


title: Kubelet config file -- > clusterDNS

text:
# Question
Title: Kubelet config file -- > clusterDNS
clusterDNS:

## \- "10.32.0.10"

The above is a flag passed to the kubelet file, i dont know what it refers to clsuter DNS ? Is it refers to VPC CIDR or any subnet CIDR ?

# Answer
It is the address of the DNS server (probably CoreDNS but possibly kube-dns) for the cluster.

Kubernetes uses a DNS server within the cluster so pods can find each other using service names. This is the "cluster DNS" server. Every time a service is created, it gets registered in the DNS server.

In Linux, the `/etc/resolv.conf` file is where the DNS server is configured. If you want to use Google's public DNS server (8.8.8.8) your `/etc/resolv.conf` file has the entry `nameserver 8.8.8.8`.

In Kubernetes, the kubelet process on a worker node configures each pod. Part of the configuration process is to create the file `/etc/resolv.conf` and specify the correct DNS server. That server is spec'ed by the `clusterDNS` configuration option.

How you set it (if you even *need* to set it yourself) depends on how you're installing kubernetes. It ultimately depends on the CNI provider you're using, but also if you're doing "kubernetes the hard way" versus `kubeadm`.
> 1 votes

---
Tags: kubernetes, networking, kubectl, network-monitoring
---


# Question
Title: How to automate installing an ISO in a VM?
We're trying to develop CI/CD processes to automate testing and installing our builds. Ya know, as one does in DevOps.

The end result of our build process are ISOs that use Red Hat Kickstart to install RHEL + our software. Today we burn the ISOs to DVD and install them on bare metal hardware. I would like to some how install and test these ISOs on virtual machines as part of our CI process. Something like:

1. Push commit to GitLab.
2. GitLab CI process runs.
3. Generates an ISO.
4. Creates a half dozen VMs via Ansible/Vagrant/something else??
5. Installs ISO in each VM.
6. Runs a smoke test on each VM to verify that it boots and our software runs.

I'm not really sure how to do 4, 5, and 6. What tools should we be using? Welcome any tips or advice... even just names of automation tools to look into.

# Answer
If you're into RHEL you could look into running oVirt ( https://ovirt.org/ ). This will enable you to manage host machines, VMs and virtually anything in between through Ansible.

What you need is oVirt on top of CentOS or similar, on a baremetal machine. This/these machine(s) will host your new VMs and your so-called Hosted Engine (i.e. the oVirt virtualization management engine run from within a VM running on the hosts themselves to ensure high availability and flexibility), and then you just need Ansible and the oVirt Python SDK.

Once you have this, it's "as simple as" creating Ansible playbooks and using the oVirt Ansible modules https://docs.ansible.com/ansible/latest/modules/list\_of\_cloud\_modules.html#ovirt
> 1 votes

---
Tags: continuous-integration, gitlab, gitlab-ci-runner, virtualization
---


title: What is Verdaccio's default login?

text:
# Question
Title: What is Verdaccio's default login?
Back in the day `admin:admin` worked, but that results in an `access denied`. `verdaccio:verdaccio` does not work as well. The Dockerfile does not seem to contain a default password as well.

**Command**

```
docker run -it --rm --name verdaccio -p 4873:4873 verdaccio/verdaccio

```

**Log**

```
 http <-- 401, user: null(172.17.0.1), req: 'POST /-/verdaccio/login', error: ba
d username/password, access denied
 http <-- 401, user: null(172.17.0.1), req: 'POST /-/verdaccio/login', error: ba
d username/password, access denied
 http <-- 401, user: null(172.17.0.1), req: 'POST /-/verdaccio/login', error: ba
d username/password, access denied

```

**GitHub**

https://github.com/verdaccio/verdaccio/issues/1599

**adduser**

When http://localhost:4873 is opened, an instruction is shown how to create a new user. However, this fails:

```
npm adduser --registry http://localhost:4873
Username: (admin) wow
Password: (<default hidden>)
Email: (this IS public) (jane@example.com) admin@admin.admin
npm ERR! code E403
npm ERR! 403 403 Forbidden - PUT http://localhost:4873/-/user/org.couchdb.user:wow
npm ERR! 403 In most cases, you or one of your dependencies are requesting
npm ERR! 403 a package version that is forbidden by your security policy.

```

**Perhaps the company's proxy is causing the issue?**

After issuing `npm config set https-proxy http://company-proxy` and running the `npm adduser` again, a 403 is still returned.

# Answer
There is no default password, actually none at all. You must create a new user with

```
➜ npm adduser --registry  http://localhost:4873/
Username: wow
Password:
Email: (this IS public) verdaccio.does.ignore.email@whatever.com
Logged in as wow on http://localhost:4873/

```

Then you are logged in.
> 5 votes

---
Tags: verdaccio
---


# Question
Title: Bridged network adapter for Packer + VirtualBox?
So, I load an .ova image with packer of a minimal ubuntu linux installation. I realized after everything went through (Provisioners etc) and it successfully exported the output file. However, when loading the output image with VirtualBox I can't communicate with the box, because the box doesn't have a valid network adapter that allows me to ping it etc.

So I tried adding a bridged adapter one during the build process:

```
"vboxmanage": [[ "modifyvm", "{{.Name}}", "--nic2", "bridged", "--bridgeadapter2","enp0s31f6" ]]

```

However the machine is still not reachable. This is how it looks on the VM:

How can I ensure that my box is always reachable from the outside? Is adding a network bridge adapter even the right way? What am I doing wrong? I think I need the VM to have an IP like 192.168.x.x

# Answer
With a bridged adapter, your VM would get its IP from the host machine's network DHCP server. This would allow it to be pinged from your internal network.

I'd also specify in your configuration that your cable is connected as so:

```
VBoxManage modifyvm "VM" --cableconnected1 on

```

Port forwarding in the VM's configuration would also work. It would allow access from the host machine's IP by just specifying the forwarded port.

To forward port 22 to 2222 you would do as so:

```
VBoxManage modifyvm "VM" --nic1 nat
VBoxManage modifyvm "VM" --natpf1 "guestssh,tcp,,2222,,22"
VBoxManage modifyvm "VM" --cableconnected1 on

```

If you want it to be accessible by the internet, you would need do port forwarding on your router to route access from your public ip:port to your internal ip:port.
> 3 votes

---
Tags: networking, packer, virtualbox
---


# Question
Title: Issue connecting to ECR using Spinnaker on Kubernetes
I installed Spinnaker onto a Kubernetes cluster via Helm. I have tried to add the ECR registry as per the docs and so far not working. I found that the aws cli is missing even though in issue 2394 it was adding into the Dockerfile, looks like as of master it is included still. Wondering what helm used that was so wrong to not have it?

Am I missing something?

# Answer
Update for anyone else who runs into this issue, when Spinnaker is installed using helm it's using an older version of the Halyard container which doesn't have the AWS CLI tool. This seems to be hard to resolve as just updating the values file for helm to the latest version doesn't seem to work and I didn't dig much into the specifics as didn't want to go down that rabbit hole.
> 2 votes

---
Tags: amazon-web-services, kubernetes, cicd
---


title: How to use edited inventory file?

text:
# Question
Title: How to use edited inventory file?
I have workflow with 2 templates. First I add new host to my hosts.ini file:

```
- name: Add to host file
     blockinfile:
       path: /var/lib/awx/projects/_52__glusterfs/hosts.ini
       block: |
         gluster1 ansible_user=<user> ansible_host=<ip>

```

And second I wait for host up:

```
---
- name: Weit befor all hosts ssh up
  hosts: localhost
  tasks:
   - name: Test 1
     shell: cat hosts.ini
   - name: Test 2
     shell: cat /var/lib/awx/projects/_52__glusterfs/hosts.ini
   - name: Test 3
     shell: cat ansible.cfg
   - name: Wait for hosts up
     wait_for:
       host: gluster1
       port: 22

```

But i get eror:

```
 [WARNING]: provided hosts list is empty, only localhost is available. Note
that the implicit localhost does not match 'all'

```

So output from tests in template 2 is (pseudocode output):

```
shell: cat hosts.ini
>>>
[gluster]

```

---

```
shell: cat /var/lib/awx/projects/_52__glusterfs/hosts.ini
>>>
[gluster]
gluster1 ansible_user=<user> ansible_host=<ip>

```

---

```
shell: cat ansible.cfg
>>>
[defaults]
inventory = /var/lib/awx/projects/_52__glusterfs/hosts.ini
host_key_checking = false

[ssh_connection]
ssh_args = -o ControlMaster=auto -o ControlPersist=60s -o UserKnownHostsFile=/dev/null

[inventory]
enable_plugins = host_list, script, auto, yaml, ini, toml

```

Why I have empty hosts.ini in my execute directory and why ansible use it instead of `/var/lib/awx/projects/_52__glusterfs/hosts.ini` in my `ansible.cfg` file?

# Answer
It sounds like you need to gather facts again once you've updated the inventory. You start off with

> First I add new host to my hosts.ini file

However, **Ansible is not notified about this** until you tell it that the inventory has changed. If this is done *during* a play, you should use the add\_host module:

```
- name: Add host gluster1 to group gluster
  add_host:
    name: gluster1
    groups: gluster

```

This will update Ansible's facts, *i.e.* it will contact the new host and find out everything about it, now that it knows it is in the `gluster` group.

Note that then you should start a new play, against this host:

```
---
- name: First play to build hosts
  hosts: localhost
  tasks:
    - name: Add host gluster1 to group gluster
      add_host:
        name: gluster1
        groups: gluster
    - name: Persist inventory
      blockinfile:
        path: /var/lib/awx/projects/_52__glusterfs/hosts.ini
        block: |
          gluster1 ansible_user=<user> ansible_host=<ip>
        state: present

- name: Second play to do gluster things
  hosts: gluster  # This group now exists in Ansible's in-memory inventory
  tasks:
    - name: Wait for hosts up
      wait_for:
        host: gluster1
        port: 22
    - name: Be awesome
      debug:
        msg: "Awesomeness Achieved"

```
> 1 votes

# Answer
You have this in your config file:

```
[inventory]
enable_plugins = host_list, script, auto, yaml, ini, toml

```

Do you really need all these plugins? If not, they'll just add time to your processing. There is also a small chance that one of them is successfully parsing your inventory file as an empty inventory. And the documentation states that

> Once an inventory plugin succeeds at parsing a source, any remaining inventory plugins will be skipped for that source.

If you do not need those plugins, I'd comment that line (or rework it to remove some of the items).

Also, are you sure that it is using the local `hosts.ini`? The command `ansible-config` might help confirming that.

Regarding your first question ("*Why I have empty hosts.ini in my execute directory*"), there is nothing in your playbooks pointing to that file being created or modified. Did you mean to have it modified? Or you don't know how it was created? The question there is not clear.
> 2 votes

---
Tags: ansible, centos
---


# Question
Title: What is the name of Jenkins plugin for GitHub plugin configuration?
We need below plugin in Jenkins:

##

Currently I have installed below list of plugins:

```
git:latest
git-client:latest
git-server:latest
github:latest
github-api:latest
github-branch-source:latest

```

---

Am using Jenkins 2.190.2

I could not find GitHub plugin configuration in Jenkins after navigating `Manage Jenkins` -\> `Configure System`

---

What is the name of the GitHub plugin that needs to be added in plugin list to install using `/usr/local/bin/install-plugins.sh`?

# Answer
> 2 votes
The plugin that you are looking for is: https://github.com/jenkinsci/github-plugin.

You need to go in `Manage Jenkins` \> `Global Configuration` to find the GitHub plugin configuration (according the documentation).

---
Tags: jenkins, jenkins-plugins, github
---


# Question
Title: What's the equivalent of `heroku container:release web` without using heroku-cli?
I have a bitbucket pipeline that almost working:

```
image: atlassian/default-image:2

pipelines:
  default:
    - step:
        services:
          - docker
        script:
          #Build the image
          - docker build -t $IMAGE_NAME ./backend
          #Login to heroku's docker registry with an API token
          - docker login --username=_ --password $HEROKU_TOKEN registry.heroku.com

          #Tag and push our built image to the heroku docker registry
          - docker tag $IMAGE_NAME registry.heroku.com/$HEROKU_APPLICATION_NAME/web
          - docker push registry.heroku.com/$HEROKU_APPLICATION_NAME/web

```

The final step is - I would run something like `heroku container:release web -a=$HEROKU_APPLICATION_NAME`

How can I do that without actually having the heroku-cli (ie. from the bitbucket pipeline).

# Answer
> 1 votes
You can install the Heroku CLI via `npm` \- which might be the simplest.

Note that your Heroku API key has to have the env var name `HEROKU_API_KEY` for the Heroku CLI to accept it as an auth token (ie. to avoid having to do `heroku login`).

```
          - npm install -g heroku

          # build the Docker image (this will use the Dockerfile in the root of the repo)
          - docker build -t $HEROKU_APPLICATION_NAME ./backend
          # authenticate with the Docker Hub registry
          - docker login --username=_ --password $HEROKU_API_KEY registry.heroku.com

          - docker tag $HEROKU_APPLICATION_NAME registry.heroku.com/$HEROKU_APPLICATION_NAME/web
          - docker push registry.heroku.com/$HEROKU_APPLICATION_NAME/web
          - heroku container:release web -a=$HEROKU_APPLICATION_NAME

```

---
Tags: continuous-deployment, bitbucket-pipelines
---


# Question
Title: elasticsearch-fluentd helm chart on Openshift 3.11 pod failing
I install a fluentd chart on OKD 3.11 with:

```
helm install kiwigrid/fluentd-elasticsearch \
--name fluentd \
-f ../../tools/fluentd/values.yaml \
--namespace logging

```

The pod is created, I can enter in it and I can curl to external elasticsearch successfully. However the liveness probe keeps failing and the pods on timeout get recreated:

```
Normal   Killing    10m (x3 over 36m) kubelet node1.qa1.isp.ezbob.com
Killing container with id docker://fluentd-fluentd-elasticsearch:Container failed liveness probe.. Container will be killed and recreated.

```

And I can't find any indexes being created in ES.

The liveness config is the default for the chart:

```
 Liveness:       exec [/bin/sh -c LIVENESS_THRESHOLD_SECONDS=${LIVENESS_THRESHOLD_SECONDS:-300}; STUCK_THRESHOLD_SECONDS=${STUCK_THRESHOLD_SECONDS:-900}; if [ ! -e /var/log/fluentd-buffers ]; then
  exit 1;
fi; touch -d "${STUCK_THRESHOLD_SECONDS} seconds ago" /tmp/marker-stuck; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-stuck -print -quit)" ]; then
  rm -rf /var/log/fluentd-buffers;
  exit 1;
fi; touch -d "${LIVENESS_THRESHOLD_SECONDS} seconds ago" /tmp/marker-liveness; if [ -z "$(find /var/log/fluentd-buffers -type d -newer /tmp/marker-liveness -print -quit)" ]; then
  exit 1;
fi;
] delay=600s timeout=1s period=60s #success=1 #failure=3

```

The OS is CentOS 7.6. I can't get any info about this issue and no trace to catch, it looks like some permission issues on mounting points.

# Answer
Changing logging driver form journald to json has fixed it. I edited `/etc/sysconfig/docker`, and set

```
OPTIONS='--selinux-enabled --log-driver=json-file --signature-verification=false'

```

Then restarted docker on the nodes I needed:

```
systemctl daemon-reload && systemctl restart docker

```
> 2 votes

---
Tags: elastic, openshift, fluentd
---


# Question
Title: What is the proper way to fix `error: did not detect an --insecure-registry argument on the Docker daemon` while running `oc cluster up`?
According to this blog one has to change the docker daemon:

> ```
> /usr/bin/dockerd --insecure-registry 172.30.0.0/16 -H fd://
>
> ```

but I wonder why the error occurs and how to fix it properly?

# Answer
This error happens because the `oc cluster up` command spawns a Docker registry. The `172.30.0.0/16` IP range is probably corresponding to your Docker bridge network. As the registry will run in a container with an unpredictable IP address, you must indicate the IP range (`--insecure-registry 172.30.0.0/16`).
> 2 votes

---
Tags: openshift
---


# Question
Title: How to get the elastic stack-docker running on a laptop?
https://github.com/elastic/stack-docker

Running `docker-compose -f setup.yml up` resulted in:

```
Creating network "stack-docker_default" with the default driver
Pulling setup (docker/compose:1.21.2)...
1.21.2: Pulling from docker/compose
605ce1bd3f31: Pull complete
c57f5dd9da06: Pull complete
4869756a2c04: Pull complete
Digest: sha256:6fb081a4ff825a3b9311aa532589bf98b3e9d3331e4f9de69fac31770c8cb108
Status: Downloaded newer image for docker/compose:1.21.2
Creating stack-docker_setup_1 ... done
Attaching to stack-docker_setup_1
setup_1  | Found orphan containers (stack-docker_setup_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
Recreating setup_elasticsearch ... done
Attaching to setup_elasticsearch
setup_1  | setup_elasticsearch    | Determining if x-pack is installed...
setup_1  | setup_elasticsearch    | === CREATE Keystore ===
setup_1  | setup_elasticsearch    | Elastic password is: WCqGii/TGf2banI6JsJF4w==
setup_1  | setup_elasticsearch    | Created elasticsearch keystore in /usr/share/elasticsearch/config
setup_1  | setup_elasticsearch    | Setting bootstrap.password...
setup_1  | setup_elasticsearch    | === CREATE SSL CERTS ===
setup_1  | setup_elasticsearch    | Remove old ca zip...
setup_1  | setup_elasticsearch    | Creating docker-cluster-ca.zip...
setup_1  | setup_elasticsearch    | CA directory exists, removing...
setup_1  | setup_elasticsearch    | Unzip ca files...
setup_1  | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster-ca.zip
setup_1  | setup_elasticsearch    |    creating: /config/ssl/ca/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.crt
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/ca/ca.key
setup_1  | setup_elasticsearch    | Remove old docker-cluster.zip zip...
setup_1  | setup_elasticsearch    | Create cluster certs zipfile...
setup_1  | setup_elasticsearch    | Unzipping cluster certs zipfile...
setup_1  | setup_elasticsearch    | Archive:  /config/ssl/docker-cluster.zip
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/elasticsearch/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.crt
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/elasticsearch/elasticsearch.key
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/kibana/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.crt
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/kibana/kibana.key
setup_1  | setup_elasticsearch    |    creating: /config/ssl/docker-cluster/logstash/
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.crt
setup_1  | setup_elasticsearch    |   inflating: /config/ssl/docker-cluster/logstash/logstash.key
setup_1  | setup_elasticsearch    | Move logstash certs to logstash config dir...
setup_1  | setup_elasticsearch    | Move kibana certs to kibana config dir...
setup_1  | setup_elasticsearch    | Move elasticsearch certs to elasticsearch config dir...
setup_1  | setup_elasticsearch exited with code 0
setup_1  | Found orphan containers (stack-docker_setup_1) for this project. If you removed or renamed this service in your compose file, you can run this command with the --remove-orphans flag to clean it up.
setup_1  | Pulling setup_kibana (docker.elastic.co/kibana/kibana:6.6.0)...
setup_1  | 6.6.0: Pulling from kibana/kibana
setup_1  | Digest: sha256:7c6eadb13667e36d21b7e96809a02d14527c15bbfc5b92631168a58ed5f9fd33
setup_1  | Status: Downloaded newer image for docker.elastic.co/kibana/kibana:6.6.0
setup_1  | Pulling setup_logstash (docker.elastic.co/logstash/logstash:6.6.0)...
setup_1  | 6.6.0: Pulling from logstash/logstash
setup_1  | Digest: sha256:5fd22fa584ae6bd9b9e2d23014f23b415ccf8c66cba35cbacb68999f7ee4e2f5
setup_1  | Status: Downloaded newer image for docker.elastic.co/logstash/logstash:6.6.0
Recreating elasticsearch ... done
Creating setup_logstash  ... done
Creating setup_kibana    ... done
Attaching to setup_logstash, setup_kibana
setup_1  | setup_logstash         | -rw-rw-r-- 1 root root 1200 Dec  3 15:35 /usr/share/logstash/config/ca/ca.crt
setup_1  | setup_kibana           | -rw-rw-r-- 1 root root 1200 Dec  3 15:35 /usr/share/kibana/config/ca/ca.crt

```

and navigating to http://localhost:5601/ does not return anything.

When `docker-compose up` is run then it fails:

```
Creating logstash        ... error
Creating kibana          ...

ERROR: for logstash  Cannot create container for service logstash: invalid mount config for type "binCreating kibana          ... error

ERROR: for kibana  Cannot create container for service kibana: invalid mount config for type "bind": bind source path does not exist: /home/ben/dev/stack-docker/config/kibana/kibana.keystore

ERROR: for logstash  Cannot create container for service logstash: invalid mount config for type "bind": bind source path does not exist: /home/ben/dev/stack-docker/config/logstash/logstash.keystore

ERROR: for kibana  Cannot create container for service kibana: invalid mount config for type "bind": bind source path does not exist: /home/ben/dev/stack-docker/config/kibana/kibana.keystore
ERROR: Encountered errors while bringing up the project.

```

# Answer
Elastic was not able to start. The logging indicated:

```
[2019-12-09T10:22:15,124][INFO ][o.e.b.BootstrapChecks    ] [kViKuCL] bound
or publishing to a non-loopback address, enforcing bootstrap checks
ERROR: [1] bootstrap checks failed
[1]: max virtual memory areas vm.max_map_count [65530] is too low,
increase to at least [262144]

```

The latter is also depicted in the README.md:

```
sudo sysctl -w vm.max_map_count=262144

```

Once this was changed and

```
docker-compose -f setup.yml up --remove-orphans

```

was run, the logging continued:

```
setup_1  | setup_packetbeat       | Loading dashboards (Kibana must be running and reachable)
setup_1  | setup_heartbeat        | Loaded dashboards
setup_1  | setup_heartbeat        | Copy keystore to ./config dir
setup_1  | setup_heartbeat exited with code 0
setup_1  | setup_auditbeat        | Loaded dashboards
setup_1  | setup_auditbeat        | Copy keystore to ./config dir
setup_1  | setup_auditbeat exited with code 0
setup_1  | setup_packetbeat       | Loaded dashboards
setup_1  | setup_packetbeat       | Copy keystore to ./config dir
setup_1  | setup_packetbeat exited with code 0
setup_1  | setup_metricbeat       | Loaded dashboards
setup_1  | setup_metricbeat       | Copy keystore to ./config dir
setup_1  | setup_metricbeat exited with code 0
setup_1  | setup_filebeat         | Loaded dashboards
setup_1  | setup_filebeat         | Loaded machine learning job configurations
setup_1  | setup_filebeat         | Copy keystore to ./config dir
setup_1  | setup_filebeat exited with code 0
setup_1  | Setup completed successfully. To start the stack please run:
setup_1  |   docker-compose up -d
setup_1  |
setup_1  | If you wish to remove the setup containers please run:
setup_1  |  docker-compose -f docker-compose.yml -f docker-compose.setup.yml down --remove-orphans
setup_1  |
setup_1  | You will have to re-start the stack after removing setup containers.
setup_1  |
setup_1  | Your 'elastic' user password is: mnQ9c5FCGT09sRgHPnw27A==
stack-docker_setup_1 exited with code 0

```

Subsequently, running `docker-compose up -d`, resulted in:

```
Recreating elasticsearch ... done
Recreating kibana        ... done
Creating logstash        ... done
Creating apm_server      ... done
Creating packetbeat      ... done
Creating metricbeat      ... done
Creating filebeat        ... done
Creating heartbeat       ... done
Creating auditbeat       ... done

```

Navigating to http://localhost:5601 returns the kibana UI.
> 0 votes

---
Tags: docker, elastic
---


# Question
Title: Having some trouble getting service running with Docker and Terraform
I'm very new to DevOps but I'm trying to learn some of it a bit. I'm working with Terraform, trying to get a service to run on AWS using Fargate and Docker. I've successfully set up my ECR repositories, and I walked through this tutorial on getting a service set up in Fargate. However, I'm having a bit of a problem. When I walked through the tutorial, everything worked more or less fine (there were some small issues but it was mostly pilot error), but when I tried to deploy my own application everything kind of exploded, and I can't figure out why.

* The error I'm getting is 502 Bad Gateway.
* The application is a Java Spring application with embedded Tomcat. I haven't changed any run configurations from the default Spring config.
* I am able to build and Dockerize my application locally and run it through Docker on my local machine and everything works fine.
* The only changes from the tutorial code (linked above) that I made was changing the Docker image and changing the app\_port from 3000 to 8080 (I also enabled Access Logging and added an S3 bucket for that purpose)
* The application has some auth security enabled, and when I try to hit the LB I get a login screen, so I know at least my Docker image is being pulled correctly
* One of the errors in the task logs of my ECS instances says something about failing a health check, although I'm not able to figure out anything more than that.

To be honest I'm a total newbie at DevOps anything so I have no idea what I'm doing. Can anyone point me in some direction to help me get started?

# Answer
> 1 votes
Seems like, because the application has security enabled, the health check was not able to hit the / endpoint and the health check was failing. When I removed the health check block I was able to access the application. I'll add a globally-open health check endpoint and that should fix the problem I think.

---
Tags: docker, amazon-web-services, terraform, spring-boot
---


title: Pulling windows base images fails

text:
# Question
Title: Pulling windows base images fails
I tried pulling the windows/nanoserver image.
After that failed I looked around and found this site
Every one of them fails with the same error message

```
#docker pull mcr.microsoft.com/windows/nanoserver:1909
ltsc2019: Pulling from windows/servercore
no matching manifest for unknown in the manifest list entries

```

Additional Info:
Docker version: 18.09.2, build 6247962
Win10 Enterprise 2016 LTSB: Version 10.0.14393 Build 14393

# Answer
The problem is twofold:

* The Win10 LTSB is on its newest version and Docker has stopped supporting this version.
This stops Docker from updating.
* Microsoft has moved to its own container registry and in doing so abandoned support for older Docker versions. So while I was pulling from the right registry it could not find a fitting image.

**Solution:** get an up-to-date, non-LTSB version of Windows

Additional Info:
While the error message did not help in finding the cause the docker log did:

```
[WindowsDaemon  ][Info   ] debug: a Windows version 10.0.18363-based image is incompatible with a 10.0.14393 host

```
> 3 votes

# Answer
seems support ended for above mentioned image its no longer in their registry maybe

https://devblogs.microsoft.com/dotnet/net-core-container-images-now-published-to-microsoft-container-registry/
> 0 votes

---
Tags: docker, windows
---


# Question
Title: Error while using AzureFile storage with CockroachDB
I'm getting an error trying to run CockroachDB (v19.2.1) on AKS. I'm able to spin up cluster using the default storage class. However, when I attempt to specify AzureFile storage, I get errors.

Storage class:

```
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: cockroach
  namespace: cockroach
provisioner: kubernetes.io/azure-file
mountOptions:
  - mfsymlinks
  - dir_mode=0777
  - file_mode=0777
  - uid=1000
  - gid=1000
  - nobrl
  - cache=none
parameters:
  skuName: Standard_LRS

```

Helm command:

```
helm install cockroach stable/cockroachdb `
    --namespace cockroach `
    --set StorageClass="cockroach" `
    --set Storage="100Gi"

```

The first node starts up, but the second node crashes with the following:

```
F191203 17:07:35.042824 214 storage/store_raft.go:322  [n3,s3,r10/2:{-}] while applying snapshot: IO error: while link file to /cockroach/cockroach-data/000049.sst: /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst: Operation not supported
while ingesting [/cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/1.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/2.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/3.sst]

```

Full log:

```
CockroachDB node starting at 2019-12-03 17:07:34.142182444 +0000 UTC (took 6.1s)
build:               CCL v19.2.1 @ 2019/11/18 23:23:55 (go1.12.12)
webui:               http://0.0.0.0:8080
sql:                 postgresql://root@cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257?sslmode=disable
RPC client flags:    /cockroach/cockroach <client cmd> --host=cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257 --insecure
logs:                /cockroach/cockroach-data/logs
temp dir:            /cockroach/cockroach-data/cockroach-temp261131495
external I/O path:   /cockroach/cockroach-data/extern
store[0]:            path=/cockroach/cockroach-data
status:              restarted pre-existing node
clusterID:           dd364f3f-7900-45f2-9a65-7a105cd7a147
nodeID:              3
I191203 17:07:34.153647 232 server/server_update.go:53  [n3] no need to upgrade, cluster already at the newest version
I191203 17:07:34.191164 234 sql/event_log.go:130  [n3] Event: "node_restart", target: 3, info: {Descriptor:{NodeID:3 Address:cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257 Attrs: Locality: ServerVersion:19.2 BuildTag:v19.2.1 StartedAt:1575392852575521409 LocalityAddress:[] ClusterName: SQLAddress:cockroach-cockroachdb-1.cockroach-cockroachdb.cockroach.svc.cluster.local:26257} ClusterID:dd364f3f-7900-45f2-9a65-7a105cd7a147 StartedAt:1575392852575521409 LastUp:1575392800598285198}
I191203 17:07:34.360523 214 storage/replica_raftstorage.go:794  [n3,s3,r10/2:{-}] applying LEARNER snapshot [id=ea03bb31 index=18]
I191203 17:07:34.487170 214 storage/engine/rocksdb.go:119  [db/version_set.cc:3086] More existing levels in DB than needed. max_bytes_for_level_multiplier may not be guaranteed.
I191203 17:07:34.508684 84 gossip/gossip.go:1531  [n3] node has connected to cluster via gossip
I191203 17:07:34.508951 84 storage/stores.go:259  [n3] wrote 2 node addresses to persistent storage
I191203 17:07:34.711691 214 storage/engine/rocksdb.go:119  [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file  failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711725 214 storage/engine/rocksdb.go:119  [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file  failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711873 214 storage/engine/rocksdb.go:119  [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file  failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711893 214 storage/engine/rocksdb.go:119  [db/external_sst_file_ingestion_job.cc:262] AddFile() clean up for file  failed : IO error: No such file or directorywhile unlink() file: No such file or directory
I191203 17:07:34.711990 214 storage/replica_raftstorage.go:815  [n3,s3,r10/2:{-}] applied LEARNER snapshot [total=351ms ingestion=4@-9223372036855ms id=ea03bb31 index=18]
E191203 17:07:35.042651 214 util/log/crash_reporting.go:537  [n3,s3,r10/2:{-}] Reported as error 2cfee4bb679a4beb8a489e30d87ab8ea
F191203 17:07:35.042824 214 storage/store_raft.go:322  [n3,s3,r10/2:{-}] while applying snapshot: IO error: while link file to /cockroach/cockroach-data/000049.sst: /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst: Operation not supported
while ingesting [/cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/0.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/1.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/2.sst /cockroach/cockroach-data/auxiliary/sstsnapshot/10/ea03bb31-e11e-40e5-99c3-053049d1f903/3.sst]
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).applySnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raftstorage.go:908
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:526
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:321
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
while applying snapshot
github.com/cockroachdb/cockroach/pkg/storage.(*Replica).handleRaftReadyRaftMuLocked
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:528
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:321
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:1337
goroutine 214 [running]:
github.com/cockroachdb/cockroach/pkg/util/log.getStacks(0xc000438300, 0xc000438360, 0x0, 0xc0031ba280)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:1017 +0xb1
github.com/cockroachdb/cockroach/pkg/util/log.(*loggingT).outputLogEntry(0x7508900, 0xc000000004, 0x6cd15ce, 0x15, 0x142, 0xc00051ac00, 0x1087)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/clog.go:871 +0x95b
github.com/cockroachdb/cockroach/pkg/util/log.addStructured(0x4a54f40, 0xc003763560, 0xc000000004, 0x3, 0x40d626f, 0x7, 0xc0035b5170, 0x2, 0x2)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/structured.go:66 +0x2cc
github.com/cockroachdb/cockroach/pkg/util/log.logDepth(0x4a54f40, 0xc003763560, 0x2, 0xc000000004, 0x40d626f, 0x7, 0xc0035b5170, 0x2, 0x2)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:69 +0x8c
github.com/cockroachdb/cockroach/pkg/util/log.FatalfDepth(...)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/log/log.go:199
github.com/cockroachdb/cockroach/pkg/storage.maybeFatalOnRaftReadyErr(0x4a54f40, 0xc003763560, 0x4104925, 0x17, 0x49e0600, 0xc003a98600, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/replica_raft.go:859 +0x1de
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest.func1(0x4a54f40, 0xc003763560, 0xc003694000, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:322 +0x25f
github.com/cockroachdb/cockroach/pkg/storage.(*Store).withReplicaForRequest(0xc000c48300, 0x4a54f40, 0xc003763560, 0xc00077dbc8, 0xc0035b5418, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:200 +0x16b
github.com/cockroachdb/cockroach/pkg/storage.(*Store).processRaftSnapshotRequest(0xc000c48300, 0x4a54f40, 0xc0035d5bf0, 0xc00077db80, 0xe5401ee131bb03ea, 0x3f9d1493005c399, 0xc000545340, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:262 +0x1a4
github.com/cockroachdb/cockroach/pkg/storage.(*Store).receiveSnapshot(0xc000c48300, 0x4a54f40, 0xc0035d5bf0, 0xc00077db80, 0x7f30769bb2c8, 0xc000369540, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_snapshot.go:854 +0x6d3
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot.func1(0x4a54f40, 0xc0035d5bf0, 0xc00054d320, 0x411c3aa)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:59 +0x1d9
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunTaskWithErr(0xc00054d320, 0x4a54f40, 0xc0035d5bf0, 0x411c3aa, 0x1e, 0xc00331bc30, 0x0, 0x0)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:300 +0xeb
github.com/cockroachdb/cockroach/pkg/storage.(*Store).HandleSnapshot(0xc000c48300, 0xc00077db80, 0x7f30769bb298, 0xc000369540, 0xc000369540, 0x30)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/store_raft.go:49 +0xe6
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1.1(0x4aa3720, 0xc000369540, 0xc000ca40f0, 0x4a54f40, 0xc0035d5b30, 0x78c3df, 0xc000544ff0)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:415 +0x138
github.com/cockroachdb/cockroach/pkg/storage.(*RaftTransport).RaftSnapshot.func1(0x4a54f40, 0xc0035d5b30)
        /go/src/github.com/cockroachdb/cockroach/pkg/storage/raft_transport.go:416 +0x5d
github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask.func1(0xc00054d320, 0x4a54f40, 0xc0035d5b30, 0xc000075780, 0x32, 0x0, 0x0, 0xc0035d5b60)
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:321 +0xe6
created by github.com/cockroachdb/cockroach/pkg/util/stop.(*Stopper).RunAsyncTask
        /go/src/github.com/cockroachdb/cockroach/pkg/util/stop/stopper.go:316 +0x131

```

Do I need to specify different options in my storage class?

# Answer
> 1 votes
AzureFile didn't support that operation, so I tried AzureDisk and it worked. This is what I used:

```
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: cockroach-disk
  namespace: cockroach
provisioner: kubernetes.io/azure-disk
parameters:
  storageaccounttype: Standard_LRS
  kind: Managed
  #resourceGroup: CockroachStorage
reclaimPolicy: Retain
allowVolumeExpansion: true

```

---
Tags: kubernetes, azure-kubernetes-service
---


# Question
Title: How to ensure EC2's OS/application is loaded?
Recently I was playing around a bit with AWS as a part of a course that I am doing. I came across a situation that to be honest is keeping be in vilo.

I made a web application which connects to a backend that basically launches a EC2 t2.micro instance on AWS. From time to time, I pull the status of the machine to check whether it is on `running` state or note. The AMI I am using it is a configured wordpress from the marketplace.

As soon as the instance gets to the `running` state, I show a message indicating the server is up and running and provide the user with the instance's public ip address, the problem is that, it is not after a few minutes later when indeed the IP is resolved/accessible. The security group attached to the instance has the inbound and outbound traffic configured to allow TCP traffic on port 80, 22 and 443.

My question is, the `running` state simply means the instance is ready to work but, as far as I know, it does not mean OS has booted nor the X application (in this case wordpress and its configured server) are ready to or finished the bootstrapping. How can I check that indeed the application/OS has booted is I can ensure that when the user is trying to access the IP it will not timeout.

My first attempt was to ping the instance bu to be honest I don't think it's a reliable request since, I'd also need to enable the ICMP protocol but at the end, it's working in another internet layer.

# Answer
What you are talking about is known as a health check. Typically these are handled by load balancers, to decide whether instances are "healthy", and thus can serve (usually HTTP) traffic. The load balancers will poll and endpoint that you define to decide whether the instances are healthy, usually by seeing if the HTTP response code is some known-good one, like 200.

One solution therefore, could be to front your application with a load balancer, and enable health checks on the relevant urls. This will check whether your application is serving traffic properly, when polled by the load balancer. While this adds a level of complexity (and cost) to the app, it is also quite easy to do.

Of course, you could mimic this functionality without an AWS load balancer, by having your app poll the url that the app is supposed to respond on, by using the DNS or IP name returned by AWS after the instance becomes "running".

Another option would be to set up an actual monitoring system to check whether the application is serving content. A very simple and free tool that would do the job is *e.g.* Freshping.

You could also write an Inspec Compliance profile to describe a known good state of the machine. Inspec can connect to the machine via SSH, which you specify as being open. This may sound like overkill, and quite similar to the ICMP test you considered before, but it's nevertheless an option :)
> 0 votes

---
Tags: amazon-web-services, amazon-ec2
---