title: Single Build with Multiple Artifacts

text:
# Question
Title: Single Build with Multiple Artifacts
I'm a little confused on the Azure Pipeline steps. My solutions has many projects and one *UI* project that contains both an ASP.Net-Core WebApi/MVC with Angular.

the `Build Solution` builds similar to the

```
steps:
- task: VSBuild@1
  displayName: 'Build solution'
  inputs:
    solution: '$(Parameters.solution)'
    msbuildArgs: '/p:DeployOnBuild=true
      /p:WebPublishMethod=Package
      /p:PackageAsSingleFile=true
      /p:SkipInvalidConfigurations=true
      /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'

```

The default `Publish Artifact` is:

```
steps:
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: drop'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'
  condition: succeededOrFailed()

```

I'd like to do something with 2 artifacts (this doesn't work):

```
steps:
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: webapi'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)\\MVCProject\bin\$(BuildConfiguration)\netcorapp2.2'
    ArtifactName: webapi
  condition: succeededOrFailed()

steps:
- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: angular'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)\\MVCProject\clientapp\dist'
    ArtifactName: angular
  condition: succeededOrFailed()

```

What am I missing / What the *right/better* way to do this. (moving angular out of the project does not solve this problem, and we aren't ready for that yet)

# Answer
> 10 votes
Here is my YAML for multiple artifacts from a single build.

```
pool:
  name: Hosted Windows 2019 with VS2019
  demands:
  - msbuild
  - visualstudio
  - vstest
  - npm

steps:
- task: NuGetToolInstaller@0
  displayName: 'Use NuGet 4.4.1'
  inputs:
    versionSpec: 4.4.1

- task: NuGetCommand@2
  displayName: 'NuGet restore'
  inputs:
    restoreSolution: '$(Parameters.solution)'

- task: VSBuild@1
  displayName: 'Build solution WebApi'
  inputs:
    solution: '$(Parameters.solution)'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:PackageLocation="$(build.artifactstagingdirectory)\\"'
    platform: '$(BuildPlatform)'
    configuration: '$(BuildConfiguration)'
    msbuildArchitecture: x64

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: WebApi'
  inputs:
    PathtoPublish: '$(build.artifactstagingdirectory)'
    ArtifactName: 'dotnet-webapi'
  condition: succeededOrFailed()

- task: DeleteFiles@1
  displayName: 'Delete build.artifactstagingdirectory for Angular'
  inputs:
    SourceFolder: '$(build.artifactstagingdirectory)'
    Contents: '**'

- task: Npm@1
  displayName: 'npm install'
  inputs:
    workingDir: project/ClientApp
    verbose: false

- task: Npm@1
  displayName: 'npm run prod-build'
  inputs:
    command: custom
    workingDir: project/ClientApp
    verbose: false
    customCommand: 'run prod-build'

- task: Npm@1
  displayName: 'npm run dev-build'
  inputs:
    command: custom
    workingDir: project/ClientApp
    verbose: false
    customCommand: 'run prod-dev'
  enabled: false

- task: CopyFiles@2
  displayName: 'Copy Angular/Dist to build.artifactstagingdirectory'
  inputs:
    SourceFolder: project/ClientApp/dist/ClientApp
    Contents: |
     **
     !config.json
    TargetFolder: '$(build.artifactstagingdirectory)'

- task: PublishBuildArtifacts@1
  displayName: 'Publish Artifact: Angular'
  inputs:
    ArtifactName: angular

```

# Answer
> 7 votes
I managed to get Erik's approach working which is great.

I then wondered if I could simplify it so instead of deleting the content how about put each artifact in a sub folder of the **$(Build.ArtifactStagingDirectory)** So by just appending /Api or /App I could create specific publish folders that I could then push onto the azure pipeline.

You can then have as many artifacts as you need :)

The important parts to take from this are that **publishWebProjects** needs to be set to **false** or it defaults to true and then ignores the projects line below, and the **output** path puts the content in a sub folder.

```
 # This need to be false in order for the specific project to be published
 publishWebProjects: False
 projects: '**/DevOpsApp.csproj'
 arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)/App'

```

and **pathtoPublish** need to then point to the sub folder

```
pathtoPublish: '$(Build.ArtifactStagingDirectory)/App'

```

Full example file below

```
# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:
- develop

pool:
  vmImage: 'ubuntu-latest'

variables:
    buildConfiguration: 'Release'
    versionNumber: 1.0.0

name: $(versionNumber)-wip-$(Rev:r)

#----------------------------------------------------------
# Web API Build and Artifact
#----------------------------------------------------------

steps:
- task: DotNetCoreCLI@2
  displayName: 'Build'
  inputs:
    command: 'build'
    projects: '**/DevOpsApi.csproj'
    arguments: '--configuration $(buildConfiguration)'

- task: DotNetCoreCLI@2
  inputs:
    command: 'test'
    projects: '**/*Tests.csproj'
    arguments: '--configuration $(buildConfiguration)'
    testRunTitle: 'Unit Tests'

# Publish the artifact
- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    # This need to be false in order for the specific project to be published
    publishWebProjects: False
    projects: '**/DevOpsApi.csproj'
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)/Api'
    zipAfterPublish: True

# Publish the artifact for the pipelines to use
- task: PublishBuildArtifacts@1
  displayName: 'Publishing WebAPI Artifact'
  inputs:
    pathtoPublish: '$(Build.ArtifactStagingDirectory)/Api'
    artifactName: 'TestWebApi-Wip'

#----------------------------------------------------------
# Web APP Build and Artifact
#----------------------------------------------------------

- task: DotNetCoreCLI@2
  displayName: 'Build'
  inputs:
    command: 'build'
    projects: '**/DevOpsApp.csproj'
    arguments: '--configuration $(buildConfiguration)'

# Publish the artifact
- task: DotNetCoreCLI@2
  inputs:
    command: 'publish'
    # This need to be false in order for the specific project to be published
    publishWebProjects: False
    projects: '**/DevOpsApp.csproj'
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)/App'
    zipAfterPublish: True

# Publish the artifact for the pipelines to use
- task: PublishBuildArtifacts@1
  displayName: 'Publishing WebAPP Artifact'
  inputs:
    pathtoPublish: '$(Build.ArtifactStagingDirectory)/App'
    artifactName: 'TestWebApp-Wip'

```

# Answer
> 3 votes
Can I suggest building and publishing the artefacts in separate jobs? That would mean that each job is simpler (no need to specify an artefact subfolder), and might give you some speed-up by running them in parallel. Something like this:

```
jobs:
  - job: web_api
    displayName: Web API Build and Artifact
    steps:
      - checkout: self

      - task: DotNetCoreCLI@2
        displayName: 'Build'
        inputs:
          command: 'build'
          projects: '**/DevOpsApi.csproj'
          arguments: '--configuration $(buildConfiguration)'

      - task: DotNetCoreCLI@2
        inputs:
          command: 'test'
          projects: '**/*Tests.csproj'
          arguments: '--configuration $(buildConfiguration)'
          testRunTitle: 'Unit Tests'

      # Publish the artifact
      - task: DotNetCoreCLI@2
        inputs:
          command: 'publish'
          # This need to be false in order for the specific project to be published
          publishWebProjects: False
          projects: '**/DevOpsApi.csproj'
          arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
          zipAfterPublish: True

      # Publish the artifact for the pipelines to use
      - task: PublishBuildArtifacts@1
        displayName: 'Publishing WebAPI Artifact'
        inputs:
          pathtoPublish: '$(Build.ArtifactStagingDirectory)'
          artifactName: 'TestWebApi-Wip'

  - job: web_app
    displayName: Web APP Build and Artifact
    dependsOn: []
    steps:
      - checkout: self

      - task: DotNetCoreCLI@2
        displayName: 'Build'
        inputs:
          command: 'build'
          projects: '**/DevOpsApp.csproj'
          arguments: '--configuration $(buildConfiguration)'

      # Publish the artifact
      - task: DotNetCoreCLI@2
        inputs:
          command: 'publish'
          # This need to be false in order for the specific project to be published
          publishWebProjects: False
          projects: '**/DevOpsApp.csproj'
          arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
          zipAfterPublish: True

      # Publish the artifact for the pipelines to use
      - task: PublishBuildArtifacts@1
        displayName: 'Publishing WebAPP Artifact'
        inputs:
          pathtoPublish: '$(Build.ArtifactStagingDirectory)'
          artifactName: 'TestWebApp-Wip'

```

Also: if each job tends to have a similar set of tasks, you could refactor them into a template to keep the pipeline code terser.

---
Tags: azure-devops, azure-pipelines, artifacts
---


# Question
Title: Is it possible to define capacity per Area path in Azure Devops?
Background:

We have 3 teams in one Azure devops project. I would like these teams to be stable, and not shift people around when working on new projects.

Each team work on multiple clients, and we've managed to differentiate the PBIs by area path. Each area path is connected to a specific client.

This way it's been very easy to have 1 sprint planning per team, and also being able to keep these teams intact.

The problem we have atm is that it's unclear how much capacity each member can have on each client. Since capacity is planned per sprint.

**The question is**: Is there any way to plan capacity per area path? I know there is a way to plan capacity by activity type, but we cannot map activity type to clients and that would be wrong imo.

Example: Team 1 consists of 5 developers.

Each developer is assigned to work on a couple of clients.

When planning the sprint, it's easy to miss the fact that dev 1 should work 20h on client 1 and 20h on client 2. So when PBIs are added to the sprint, we might book dev 1 to work 40h on client 1 and miss client 2 totally.

# Answer
Capacity is a Team allocation concept. Areas and Iterations are a backlog concept, but they overlap because each Team is assigned one or more Area Paths.

One thing that isn’t obvious in Azure DevOps is that each Team gets its own set of Agile tools. It’s just a different view of the project backlog. Each team/backlog has their own capacity. It’s important to note that the capacity in each backlog is not limited to the members of the team and can also vary each iteration. When creating the “Team” you don’t have to put developers in it.

My suggestion would be to align your Teams per client and assign partial capacity for each developer on each (client) Team. For example, developer #1 is split 50% between client 1 and on client 2, so you’d set their capacity to 20h on both (client) Teams. You would also set up their capacity on the project level (the default Team named after the project) to 40h.

Because the backlog is shared, the sprint view per team will reflect their capacity for that Area Path which would allow you to see an overall allocation at the project level for each developer and to see if they are over allocated per client by toggling the Sprint view to the Client Team. Lots of toggling but it would help to provide insights into which client is being overlooked.
> 0 votes

---
Tags: azure-devops
---


# Question
Title: Where does the string docker.example.com in my GitLab CI job come from?
I am trying to have a Docker image deployed to my server by GitLab CI, but it only worked once¹ and then dies with the error message below. Here is the `.gitlab-ci.yml` snippet:

```
deploy-image:
  image: docker:latest
  services:
    - docker:dind
  variables:
    DOCKER_HOST: ssh://gitlab-ci@$DEPLOY_HOST
  before_script:
    - eval $(ssh-agent -s)
    - echo "${SSH_KEY}" | ssh-add -
    - mkdir -p ~/.ssh
    - '[[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config'
  script:
    - echo $CI_JOB_TOKEN | docker login --username gitlab-ci-token --password-stdin $CI_REGISTRY;
    - docker pull "$CI_REGISTRY_IMAGE"
    - docker run --name mycontainer --detach $CI_REGISTRY_IMAGE
  after_script:
    - docker logout $CI_REGISTRY

```

The error in the job log looks like that:

```
Executing "step_script" stage of the job script 00:04
Using docker image sha256:51453dcdd9bd51e503f75e6d42a4071469ad2ba816321781985041b6bc7776db for docker:latest with digest docker@sha256:ddf0d732dcbc3e2087836e06e50cc97e21bfb002a49c7d0fe767f6c31e01d65f ...
$ eval $(ssh-agent -s)
Agent pid 18
$ echo "${SSH_KEY}" | ssh-add -
Identity added: (stdin) (gitlab-ci@jacks)
$ mkdir -p ~/.ssh
$ [[ -f /.dockerenv ]] && echo -e "Host *\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config
$ echo $CI_JOB_TOKEN | docker login --username gitlab-ci-token --password-stdin $CI_REGISTRY;
error during connect: Post "http://docker.example.com/v1.24/auth": command [ssh -l gitlab-ci -- xxx.xxx.xxx.xxx docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host xxx.xxx.xxx.xxx port 22: Connection refused

```

While I x-ed out the IP address of the destination server in this post (which is the value of the `DEPLOY_HOST` env var), it literally says **docker.example.com** in the error message.

So my question would be, where that example string comes from.

This is on a self-hosted GitLab instance and the runner is also self-hosted on a server that is neither the destination host, nor the one where GitLab runs on.

¹ The connection issue itself seems to be a problem with the SSH connection and not related to the CI/CD pipeline config.

# Answer
It's a placeholder domain name for connecting over SSH. It doesn't mean that it's connecting to docker.example.com.

I ran into a similar error when using a Docker client over SSH. The error message was:

```
ERROR: Cannot connect to the Docker daemon at http://docker.example.com. Is the docker daemon running?

```

Reading the Docker source code, it looks like "docker.example.com" is a placeholder host name when connecting over SSH, resulting in the confusing error message.

In my case, the error was caused by the user I was logging in as not being part of the docker group, and therefore not having permission to use docker.
> 10 votes

---
Tags: docker, gitlab, deployment, cicd
---


# Question
Title: Colored logging messages from Python script within GitLab CI
I am using a docker container for my GitLab CI and would like to have any output/logging messages in color inside the CI. What I mean: the commands from `.gitlab-ci.yml` are in color, but the outputs of the scripts running `python3 colored_logging.py` are not.

It is so cumbersome to dig through bigger logs without any colors :-(

Here is an example project on GitLab: https://gitlab.com/ziggyler.ziegler/non-colored-ci/-/jobs/1873125969

Does somebody know how to get colored logging messages in lines 113-117? Any hints?

Thank you!

Actual output

Desired output

My `Dockerfile`:

```
FROM python:3.6

ENV TERM xterm-256color

RUN apt-get update && rm -rf /var/lib/apt/lists/*

RUN pip3 install loguru

```

My `.gitlab-ci.yml`:

```
image: docker:19.03.12

services:
  - docker:19.03.12-dind

stages:
  - build
  - run

build:
    stage: build
    rules:
      - if: $CI_PIPELINE_SOURCE == "push" && $CI_COMMIT_BRANCH
        changes:
          - Dockerfile
    before_script:
        - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
    script:
        - docker build --cache-from $CI_REGISTRY_IMAGE:latest --tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA --tag $CI_REGISTRY_IMAGE:latest .
        - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
        - docker push $CI_REGISTRY_IMAGE:latest

run:
    stage: run
    image: $CI_REGISTRY_IMAGE:latest
    script:
        - ls -la
        - python3 colored_logging.py

```

My Python code to create the logging messages:

```
from loguru import logger

logger.debug("This is a debug message!")
logger.info("This is an info logging message!")
logger.warning("This is a waring and should appear in yellow color!")
logger.error("This is an error in red color!")
logger.critical("This is a very critical message!")

```

# Answer
Set the `LOGURU_COLORIZE` variable in your Docker container.

```
run:
    stage: run
    image: $CI_REGISTRY_IMAGE:latest
    variables:
        LOGURU_COLORIZE: "true"
    script:
        - ls -la
        - python3 colored_logging.py

```
> 7 votes

---
Tags: docker, continuous-integration, gitlab
---


title: how to schedule container in GCP

text:
# Question
Title: how to schedule container in GCP
I have 5 tasks in my project that need to be run periodically. Some of these tasks are run on a daily basis, some on a weekly basis.

I try to containerize each task in a Docker image. Here is one illustrative example:

```
FROM tensorflow/tensorflow:2.7.0
RUN mkdir /home/MyProject
COPY . /home/MyProject
WORKDIR /home/MyProject/M1/src/
RUN pip install pandas numpy
CMD ./task1.sh

```

There are a list of Python scripts that need to be run in the task1.sh file defined above. This is not a server application or anything similar, it will run the task1.sh, which will run all the python scripts defined in it one by one, and the entire process will be finished within minutes. And the same process is supposed to be repeated 24 hours later.

How can I schedule such Docker containers in GCP? Are there different ways of doing it? Which one is comparably simpler if there are multiple solutions?

I am not a dev-ops expert by any means. All examples in documentation I find are explained for server applications which are running all the time, not like my example where the image needs to be run just once periodically. This topic is quite daunting for a beginner in this domain like myself.

# Answer
> 2 votes
Add any tiny http-server wrapper to your script and deploy your docker image as a Cloud Run service and use Cloud Scheduler that will invoke it by cron.

https://cloud.google.com/run/docs/triggering/using-scheduler

---
Tags: docker, kubernetes, google-cloud-platform, gcp, google-kubernetes-engine
---


# Question
Title: SubnetsHaveNoServiceEndpointsConfigured error when deplying cosmosdb account
I'm trying to deploy cosmosdb account with mongodb on azure with terraform. But 'terraform apply' times out with SubnetsHaveNoServiceEndpointsConfigured error.

> Error: creating CosmosDB Account "geo-db-2pv0c" (Resource Group "geo"): waiting for the CosmosDB Account "geo-db-2pv0c" (Resource Group "geo") to finish creating/updating: Code="BadRequest" Message="Database account creation failed. Operation Id: b2f838e87b6c, Error : {\r\n "error": {\r\n "code": "SubnetsHaveNoServiceEndpointsConfigured",\r\n "message": "Subnets geo-internal-1 of virtual network /subscriptions/b5a3431d77a5/resourceGroups/geo/providers/Microsoft.Network/virtualNetworks/geo-network do not have ServiceEndpoints for Microsoft.AzureCosmosDB resources configured. Add Microsoft.AzureCosmosDB to subnet's ServiceEndpoints collection before trying to ACL Microsoft.AzureCosmosDB resources to these subnets.",\r\n "details": \[\]\r\n }\r\n}\r\nActivityId: 95817ee8-801f-415d-ad06-06fd9af4130d, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0, Microsoft.Azure.Documents.Common/2.14.0" │ │ with azurerm\_cosmosdb\_account.db, │ on azure\_cosmosdb.tf line 9, in resource "azurerm\_cosmosdb\_account" "db": │ 9: resource "azurerm\_cosmosdb\_account" "db" {

There is no info on this error and how to that endpoint to a subnet. Anyone knows how to do that?

```
resource "azurerm_cosmosdb_account" "db" {
  name                = "geofriends-db-${random_string.random-name.result}"
  location            = azurerm_resource_group.geofriends.location
  resource_group_name = azurerm_resource_group.geofriends.name
  offer_type          = "Standard"
  kind                = "MongoDB"

  enable_automatic_failover = true

  is_virtual_network_filter_enabled = true

  virtual_network_rule {
    id = azurerm_subnet.subnet-internal-1.id
  }

  capabilities {
    name = "EnableMongo"
  }

  capabilities {
    name = "MongoDBv3.4"
  }

  consistency_policy {
    consistency_level       = "BoundedStaleness"
    max_interval_in_seconds = 300
    max_staleness_prefix    = 100000
  }

  geo_location {
    prefix            = "cosmosdb-s${random_string.random-name.result}-failover"
    location          = var.failover_location
    failover_priority = 1
  }

  geo_location {
    prefix            = "cosmosdb-s${random_string.random-name.result}-main"
    location          = azurerm_resource_group.geofriends.location
    failover_priority = 0
  }
}

```

# Answer
> 1 votes
The service endpoint is an Azure requirement; there's no change you could make to the CosmosDB deployment which will get around that.

Assuming you are also deploying the vnet and subnet through Terraform, then I guess you need to specify the service endpoint when you do so; see azurerm\_subnet.

---
Tags: terraform, azure, mongodb
---


# Question
Title: From inside a Docker container, how can I consume an API that is on the machine's localhost?
First of all, please forgive me if any information is missing or if my question does not meet the standards required by the site. I have very basic knowledge of docker, linux and networking and I am lost.

I use Ubuntu version 20.2 and I have docker version 20.10.10.

I have a system (CakePHP) running in a docker container, and an API (LARAVEL) running on my machine's local host (outside of the docker container), and I need to consume that API on the system from inside the docker container.

This is the content of docker-compose.yml

```
version: '2'
services:
  server-c10-php56:
    build:
      context: .
      dockerfile: Dockerfile
    image: php-c10-56
    volumes:
      - /srv/:/srv/
      - /var/www/html/:/var/www/html/
    ports:
      - "9056:80"
      - "9443:443"
    networks:
      - net30
    container_name: "docker-php56"
networks:
  sdnet:
    driver: "bridge"
  net30:
    ipam:
      driver: default
      config:
        - subnet: 172.30.0.0/16

```

As I understand it, the system (CakePHP) is running inside this 172.30.0.0/16 subnet. And the system (CakePHP) to connect to PostgreSQL, connects through IP 172.30.0.1 (I think it's the gateway to access the local host of my machine).

The API I can consume normally by Postman. Here is a screenshot of a test.

And on the system (CakePHP) from inside the docker container, I'm trying to consume like this (just testing for now).

It takes a while trying to connect, but without success. This is the return obtained ( sorry for the array format).

```
Array
(
    [response] =>
    [info] => Array
        (
            [url] => http://172.30.0.1:8000/api/queue/integracaoEcommerce
            [content_type] =>
            [http_code] => 0
            [header_size] => 0
            [request_size] => 0
            [filetime] => -1
            [ssl_verify_result] => 0
            [redirect_count] => 0
            [total_time] => 130.124754
            [namelookup_time] => 1.8E-5
            [connect_time] => 0
            [pretransfer_time] => 0
            [size_upload] => 0
            [size_download] => 0
            [speed_download] => 0
            [speed_upload] => 0
            [download_content_length] => -1
            [upload_content_length] => -1
            [starttransfer_time] => 0
            [redirect_time] => 0
            [redirect_url] =>
            [primary_ip] => 172.30.0.1
            [certinfo] => Array
                (
                )

            [primary_port] => 8000
            [local_ip] =>
            [local_port] => 0
        )
)

```

Here is the method used to consume the API (in case that information is important).

```
private function callAPI($options){
        extract($options);

        $ch = curl_init();

        $header = array(
            "Content-Type: application/json",
            "Authentication: bearer {$integrador['token']}"
        );

        curl_setopt($ch, CURLOPT_URL, $url);
        curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
        curl_setopt($ch, CURLOPT_HEADER, FALSE);

        if(!empty($data) OR $verb == 'POST'){
            curl_setopt($ch, CURLOPT_POSTFIELDS, $data);
            curl_setopt($ch, CURLOPT_POST, TRUE);
        }

        if($verb)
            curl_setopt($ch, CURLOPT_CUSTOMREQUEST, $verb);

        curl_setopt($ch, CURLOPT_HTTPHEADER, $header);

        $response = curl_exec($ch);
        $info = curl_getinfo($ch);
        curl_close($ch);

        return array('response' => $response, 'info' => $info);
    }

```

# Answer
> 4 votes
I'm not sure I can help you, but this might :

https://stackoverflow.com/questions/24319662/from-inside-of-a-docker-container-how-do-i-connect-to-the-localhost-of-the-mach

tldr (bridge mode):

to get the ip of the container (in container CLI) : `ip addr show eth0`

to get the ip of the host's docker interface (host CLI) : `sudo ip addr show docker0`

---
Tags: docker, docker-compose, api
---


# Question
Title: Public/Private Access from Home Machine without VPN
I'm working from home, have a personal acct at MS Azure. My internet connection is Google Fiber 1Gb just like many other people. I'm an IT pro but totally green on Azure. While setting up some Azure services, Network connectivity section, `Public access`, `Firewall rules` says Inbound connections from the IP addresses specified below will be allowed, `+ Add current client IP address (123.123.123.1)`. Since I'm using a home router believe this IP address is not static. If `Private access` is chosen, then to use my own `Vnet`. Anyway, how do I set up for personal connection from home?

# Answer
I wouldn't recommend trying to use vnet integration from home; azure vnets are complex and can involve expensive connectivity services to connect to them from your home PC.

Instead, I'd recommend defining a suitable ip address range to cover all the addresses your ISP might assign to your router. You will have to do some detective work to find out precisely what ip addresses are controlled by your ISP, but you should end up with a range which you can express in CIDR format, for example 123.123.0.0/16.
> 1 votes

# Answer
As per @Vince's mention, I will add the comment as an optional answer. The answer that Vince gave is valid as both of the things we gave depend on the need, time spending, and pricing.

Answer: I would recommend setting up OpenVPN on an Azure VM that's in the same VNET as the instance you want. Also, you can make that OpenVPN VM have a static IP and use it for other purposes, the size of the VM can be really small, and get it for cheap. The machine for OpenVPN can be found in the marketplace in Azure. There are a lot of YouTube videos for this and you have 2 free clients and you get a pretty nice UI. I would invest in that and you can have that.. VPN for a very long time, if you are ok to pay for a small VM, which you can set to turn off at night and spend even less.

I used this solution for work cases where we needed specific access without spending a huge amount of money. If you don't want a UI and are fine to work with cli, you can have much more users connected.
> 1 votes

---
Tags: azure
---


# Question
Title: Ansible to get AWS SSM Parameter Store value but fail to decrypt secure strings
I need to lookup a value from AWS SSM Parameter Store, but it needs to fail if the parameter type is secure string.

I do NOT want secure string parameters to be decrypted, and I want to detect their type and fail gracefully.

```
  - name: Get the value from AWS Parameter Store
    set_fact:
      var_value: "{{ lookup('aws_ssm', var_path, decrypt=False, region=aws_region, aws_access_key=aws_access_key, aws_secret_key=aws_secret_key) }}"

- name: Show the value
  debug:
    msg:
      - "VALUE:  {{ var_value | default( 'FAILED_TO_DECRYPT' ) }}"

```

However, the `aws_ssm` module does not fail when told not to decrypt a secure string. It returns the encrypted string (not helpful). I don't see a way to detect parameter type.

```
TASK [Show the value] **********************************************************
ok: [127.0.0.1] => {
    "msg": [
        "VALUE:  CRAZY_LONG_ENCRYPTED_STRING_AND_NOT_THE_REAL_VALUE"
    ]
}

```

What is a best practice for intentionally failing to decrypt secure strings?

References:

1. https://docs.ansible.com/ansible/latest/collections/amazon/aws/aws\_ssm\_lookup.html
2. https://docs.ansible.com/ansible/latest/collections/community/aws/aws\_ssm\_parameter\_store\_module.html

# Answer
> 1 votes
One way is to get the value twice, with `decrypt=false` and `decrypt=true`, and compare them.

```
- name: Get the value from AWS Parameter Store
  set_fact:
    var_value_decrypted: "{{ lookup('aws_ssm', var_path, decrypt=true, region=aws_region, aws_access_key=aws_access_key, aws_secret_key=aws_secret_key) }}"
    var_value_as_is: "{{ lookup('aws_ssm', var_path, decrypt=false, region=aws_region, aws_access_key=aws_access_key, aws_secret_key=aws_secret_key) }}"

- name: Fail since the value is encrypted and permission_to_decrypt is false
  debug:
    msg:
      - "VALUE:  FAILED_TO_DECRYPT"
  when: var_value_decrypted != var_value_as_is and not permission_to_decrypt | bool

- name: Show the value
  debug:
    msg:
      - "VALUE:  {{ var_value_decrypted | default( 'KEY_NOT_FOUND' ) }}"
  when: var_value_decrypted == var_value_as_is or permission_to_decrypt | bool

```

If they are different and `permission_to_decrypt` is false, fail with an error

```
"VALUE:  FAILED_TO_DECRYPT"

```

If they are the same or `permission_to_decrypt` is true, show the value

```
"VALUE:  super secret string"

```

---
Tags: amazon-web-services, ansible, amazon-ssm
---


# Question
Title: Helm how to use variables in values.yaml
In the values.yaml we have the below env values and they are refered by `templates/deployment.yaml {{- toYaml .Values.env | nindent 12 }}`

```
env:
  - name: APPLICATION_PORT
    valueFrom:
      configMapKeyRef:
        name: application-properties
        key: application_port
  - name: POSTGRES_AUTH_URL
    valueFrom:
      configMapKeyRef:
        name: application-properties
        key: postgres_business_auth_url
  - name: POSTGRES_DATA_URL
    valueFrom:
      configMapKeyRef:
        name: application-properties
        key: postgres_verification_data_url

```

Trying to change the above and make a template of the ConfigMap name without any success, by receiving the following error:

`cannot load values.yaml: error converting YAML to JSON: yaml: invalid map key: map[interface {}]interface {}{"tpl .Values.myConfigMapName .":interface {}(nil)}`

```
myConfigMapName: application-properties

env:
  - name: APPLICATION_PORT
    valueFrom:
      configMapKeyRef:
        name: {{ tpl .Values.myConfigMapName . }}
        key: application_port
  - name: POSTGRES_AUTH_URL
    valueFrom:
      configMapKeyRef:
        name: {{ tpl .Values.myConfigMapName . }}
        key: postgres_business_auth_url
  - name: POSTGRES_DATA_URL
    valueFrom:
      configMapKeyRef:
        name: {{ tpl .Values.myConfigMapName . }}
        key: postgres_verification_data_url

```

How can I consume a variable from values.yaml and use it within values.yaml ?

# Answer
Try renaming values.yaml to values.yaml.j2 (jinja2) and remove the "tpl", the template should just be as follows.

```
  configMapKeyRef:
    name: {{ .Values.myConfigMapName }}
    key: application_port

```

I know that works with bazel builds using the following:

`load("//build/helmcharts:helpers.bzl", "release_chart")`
> -2 votes

---
Tags: helm
---


# Question
Title: Jenkins pipeline : How can I display a downstream / upstream relationship?
I have a pipeline that ends with a step which call build() to run another pipeline, not waiting for its end. Even when the downstream pipeline is finished, there is no downstream/upstream indication on each job page or build page. Is there some trick to retrieve such a relationshi ? Also, could it be possible to use the Downstream Build View plugin ?

Thanks for your help. Best Regards J.L.P.

# Answer
The upstream/downstream relationship is for Freestyle jobs only and does not work for Pipeline. This is because the internal (Java) Freestyle job API that the upstream/downstream feature uses to discover job relationships does not exist for Pipeline jobs. This also means that any 3rd-party upstream/downstream-related plugins such as the Downstream Build View plugin will not work for Pipeline jobs.

Since I also wanted to visualize upstream/downstream relationships for my Pipeline jobs, I ended up writing a primitive plugin to do so (it's a very bare-bones plugin written for my particular use case and I provide no support whatsoever; I'm just including this as an example of one way to visualize upstream/downstream relationships for Pipeline jobs).
> 2 votes

# Answer
Yet Another Build Visualizer shows up- und downstream builds for pipeline jobs, too.

> A Jenkins plugin that allows visualization of the build flow of upstream and downstream builds in Jenkins.
> 1 votes

---
Tags: jenkins, jenkins-plugins, pipeline
---


# Question
Title: It seems like the kubelet isn't running or healthy
What did I want to do here?

I want to make 3 node Kubernetes cluster First I install Ubuntu server 20.04.3 LTS in my VMware and this way too I make 3 servers. After I install docker and Kubernetes and Kubernetes dependencies and install kubeadmn with all dependencies. when I try to make the first master node with kubeadmn command `kubeadm init`

after I face `kubelete isn't running or healthy` this error.

Then I follow a few solutions in stack overflow

```
 sudo systemctl daemon-reload
 sudo systemctl restart docker
 sudo systemctl restart kubelet
 kubeadm reset
 systemctl restart kubelet
 kubeadm init

```

still, I haven't solution found

# Answer
> 0 votes
here is the solution I found in stack overflow link here

what I did here

please create `vi /etc/docker/daemon.json` and inside added below:

```
{
    "exec-opts": ["native.cgroupdriver=systemd"]
}

```

after run this command and solve the problem

```
sudo systemctl daemon-reload
sudo systemctl restart docker
sudo systemctl restart kubelet

```

---
Tags: kubernetes, kubectl, virtual-kubelet
---


# Question
Title: cannot ssh first vm created by multi vm vagrantfile and vm name net getting set & public network not getting created for second vm
I have written a multi-vm vagrant file, first I am not able to ssh using vagrant ssh dnsmasq to the first vm and for second vm the public network is not getting created i.e. after I ssh to pihole vm, it only had private network.

```
# -*- mode: ruby -*-
# vi: set ft=ruby :

VAGRANTFILE_API_VERSION = "2"
PUBLIC_NET_BRIDGE = 'Realtek PCIe GbE Family Controller #5'
PIHOLE_PUBLIC_IP =  "192.168.0.111"
DNSMASQ_PUBLIC_IP = "192.168.0.121"

Vagrant.configure(VAGRANTFILE_API_VERSION) do |config|
  config.vm.synced_folder ".", "/vagrant", mount_options: ["dmode=700,fmode=600"]
  config.vm.box = "ubuntu/xenial64"
  config.vm.provider "virtualbox" do |v|
    v.memory = 1024
    v.cpus = 1
    v.customize ["modifyvm", :id, "--usb", "off"]
    v.customize ["modifyvm", :id, "--usbehci", "off"]
  end

  config.vm.define "dnsmasq" do |d|
    # d.provider.name = ""
    d.vm.provider :virtualbox do |dvb|
      dvb.customize ["modifyvm", :id, "--name", "dnsmasq"]
    end
    d.vm.hostname = "dnsmasq"
    d.vm.network "public_network", ip: DNSMASQ_PUBLIC_IP, bridge: PUBLIC_NET_BRIDGE

    d.vm.provision "shell", inline: <<-SHELL
      apt-mark hold ubuntu-advantage-tools
      apt-get update
      apt-get upgrade -y
      apt-get install dnsmasq -y
    SHELL

    d.vm.provision "shell", path: "provision.sh"

    d.vm.provision "shell", inline: <<-SHELL
      systemctl enable consul
      systemctl restart dnsmasq
      systemctl start consul
    SHELL
  end

  config.vm.define "pihole" do |p|
    # p.provider.name = "pihole"
    # p.vm.hostname = "pihole"
    p.vm.provider :pihole do |pvb|
      pvb.customize ["modifyvm", :id, "--name", "pihole"]
    end
    p.vm.network "public_network", ip: PIHOLE_PUBLIC_IP, auto_config: false, bridge: PUBLIC_NET_BRIDGE

    p.vm.provision "shell", inline: <<-SHELL
      apt-mark hold ubuntu-advantage-tools
      apt-get update
      apt-get upgrade -y
      apt-get install net-tools dnsutils -y
    SHELL
    end
end

```

# Answer
> 0 votes
auto\_config: false, this is the difference between two vm public network configuration Public Networks you can configure static IP on public network with two methods; auto\_config needs inline provisioning, easy way to do this is just define static IP along with public network

---
Tags: vagrant, virtualbox
---


# Question
Title: Specify image in Dockerfile based on building client platform/architecture
I have a somewhat specific, Swift-based problem.
It seems that none of the official Swift images on Dockerhub support the M1, and instead the only seemingly reliable arm64 Swift images are found at swiftarm/swift. This means that manifest querying will always fail for an M1 for something like `FROM swift:5.5.2`. Additionally, trying to get around this with `--platform=x86_64` the particular project I'm trying to run fails to compile due to an error with the Swift compiler in QEMU.
Is there a way to do something like

```
IF platform == arm64 FROM swiftarm/swift
ELSE FROM swift:5.5.2

```

or should I just build and push my own multiplatform image

# Answer
-`--build-arg Set build-time variables` it is possible to build with arch parameter depending on this arch is whether host arch or docker image arch.
> 1 votes

# Answer
It would be hacky, but you could `sed` it in with a wrapper script.

```
#! /usr/bin/env bash

platform=$(uname -p)

if [[ $platform == "arm64" ]]; then
    sed -i 's/FROM swift\:5\.5\.2/FROM swiftarm\/swift/' Dockerfile
fi

docker build ...

```
> 0 votes

---
Tags: dockerfile, macos
---


# Question
Title: ipfs-update isn't showing the expected version list
I ran

```
go get -u github.com/ipfs/ipfs-update

```

and got in the end of what it looks like an installation going on the following

> go get: installing executables with 'go get' in module mode is deprecated.
>
> Use 'go install pkg@version' instead.
>
> For more information, see https://golang.org/doc/go-get-install-deprecation or run 'go help get' or 'go help install'.
>
> github.com/ipfs/ipfs-update
>
> go/pkg/mod/github.com/ipfs/ipfs-update@v1.7.1/main.go:318:21: cannot use lib.NewIpfsFetcher(distPath, 0) (type \*lib.IpfsFetcher) as type migrations.Fetcher in argument to migrations.NewMultiFetcher:
>
> \*lib.IpfsFetcher does not implement migrations.Fetcher (missing Close method)

Then, used

```
ipfs-update versions

```

but didn't see the latest version.

# Answer
> 0 votes
To get that working I've used instead

```
go install github.com/ipfs/ipfs-update@latest

```

Then once running

```
ipfs-update versions

```

I got

Then could simply run

```
ipfs-update install v0.11.0

```

---
Tags: go, ipfs
---


# Question
Title: Error: error loading plugins: loading plugin /data/ipfs/plugins/go-ds-s3.so: not built with cgo support
After

1. cloning this repo
2. run `make build` (getting as output `s3plugin.so`)
3. run `make install`
4. make the appropriate changes in both `/data/ipfs/config`

```
"Datastore": {
    "BloomFilterSize": 0,
    "GCPeriod": "1h",
    "HashOnRead": false,
    "Spec": {
      "mounts": [
        {
         "child": {
           "type": "s3ds",
           "region": "us-east-1",
           "bucket": "ipfs-test2-node",
           "accessKey": "******",
           "secretKey": "******"
        },
        "mountpoint": "/blocks",
        "prefix": "s3.datastore",
        "type": "measure"
       },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },

```

and `/data/ipfs/datastore_spec`,

```
{"mounts":[{"bucket":"ipfs-test2-node","mountpoint":"/blocks","region":"us-east-1","rootDirectory":""},{"mountpoint":"/","path":"datastore","type":"levelds"}],"type":"mount"}

```

I restarted ipfs `sudo systemctl restart ipfs`

and then while checking that didn't have errors

```
systemctl status ipfs

```

then got

> Error: error loading plugins: loading plugin /data/ipfs/plugins/go-ds-s3.so: not built with cgo support

# Answer
> 0 votes
After raising an issue in the GitHub of go-ds-s3 plugin, got to know from Stebalien that

> Ah, I see. At the moment, plugins only work on 64bit Linux for the pre-built binaries (because they're built on 64bit linux and building go-ipfs with CGO support for foreign architectures is tricky). For now, you'll have to build go-ipfs from source or use the 64bit version (or use the go-ipfs docker container).
>
> I've filed an issue to support this in our build system (ipfs/distributions#546).

---
Tags: amazon-s3, ubuntu, go, ipfs
---


# Question
Title: Can a kubernetes pod be forced to stay alive after its main command fails?
After starting a long running kubernetes job we've found that the final result will fail to upload to its final location. We would like to force the pod to stay open after the main process fails so we can exec in and manually process the final results. If the main process fails and the pod exits before uploading the final result we will lose a large amount of time to re-process the job.

Is there a way to ensure the pod stays alive manually?

# Answer
> 3 votes
Keeping your pod alive after its main process exited is an anti-pattern and not possible in K8s.

The entire concept of a K8s Job is to run till completion. So what you are looking to do is neither ideal nor recommended.

It would be better to change the entrypoint command of your container to include a retry/error handling mechanism in case of failure.

With that said, you do have ways in which you can **debug** the failed pod.

If your Pod's container process exits due to error, the Pod will be marked with `Status: Failed`

But you can still debug the failed pod by attaching an Ephemeral container: https://kubernetes.io/docs/tasks/debug-application-cluster/debug-running-pod/#debugging-with-ephemeral-debug-container

**Note:** Ephemeral containers are available as K8s alpha feature only from v1.16

# Answer
> 4 votes
I want to add a quick and dirty solution to this. I often add `sleep 600` to the end of a command that's failing unexpectedly so that I can `exec` in and poke around to see what happened. Yaml example (note: `- >-` enables multi-line concatenation, just a convenience):

```
        command: ["bash", "-c"]
        args:
          - >-
            python myscript.py; sleep 600;

```

With that said, Karthick's answer is better, this is just a simple solution that doesn't follow best practices. I encourage you **not** to use `sleep` or `sleep infinity`, because it's very easy to forget that your pod is running and holding onto resources indefinitely.

---
Tags: kubernetes
---


# Question
Title: Check TZ in Container Image without creating instance?
I have a requirement to assure that the container image we build runs in UTC only. I'd like to avoid starting it up just to check the link on /etc/localtime. I thought maybe a tool like `dive` that evaluates the contents of the image would do the trick. Clearly dive has that info, but it is outside it's scope to provide it. I also looked at syft and grype, but it seems they focus on packages, not files.

All those tools are looking at the file system so it seems like this isn't very far from reality. Is there another tool that would list out specific files and corresponding file system links?

I'm also open to alternative methods of checking the timezone in a container image that doesn't involved needing to create an instance of the image.

# Answer
> 1 votes
In order to get single `/etc/localtime` symlink or file from image, you have to download the whole layer anyway. To deterimine which layer contains latest `/etc/localtime`, you have to download **all** layers of image.

When you downloaded all layers, it's really simpler to just run container with an entrypoint changed to `date +%z`:

```
docker run --rm --entrypoint /bin/date <target_image> +%z
+0000

```

It will not consume many resources.

---
Tags: docker, continuous-integration, containers
---


# Question
Title: Deployment group not showing in Azure DevOps release pipeline
I am trying to follow these instructions: https://docs.microsoft.com/en-us/azure/devops/pipelines/apps/cd/deploy-webdeploy-iis-deploygroups

I get to step 6 under "Define your CD release pipeline" and get stuck. I have a Deployment group created and can see my Windows VM in the groups targets. DevOps says that the target is online. But when I'm in the pipeline editor and trying to select the deployment group for the IIS Deployment job, no deployment groups appear in the dropdown. When I type in the name of my deployment group it just says "Deployment Group does not exist". Does anyone know why my deployment group is not detected?

# Answer
We figured this out. The deployment group was created by someone else and I had only been granted Reader and not User permissions for the group.
> 4 votes

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: Is it a good practice to run unit tests inside build job in Gitlab CI?
* We want to run unit tests against a built Docker image and not by installing the environment inside Gitlab CI runner
* We don't want to push the docker image to a registry and pull it back in a test job to run unit tests - it's slower.

But it contradicts the separation of concerns principle - now our build job will also run unit tests.

The question for the community: is it okay to sacrifice this principle in this case or there are better alternatives?

# Answer
TL;DR: See **bold** sentence below.

---

I know the *separation of concerns (SoC) principle* mostly from a "software code organisation" point of view. I never explicitly encountered it for build job definitions up to now (which does not mean it is bad to follow it in this context too).

From my point of view: *Principles* and *good* (or even *best*) *practices* exist for a reason. Following them can help you guide your project to success, because they seem to appear often in successful projects\[citation needed\].

This does not mean that every successful project follows all best practices. On the other hand, following them does not automatically make your project successful.\[citation needed, too\]

Please note that I don't know building Docker images with GitLab well enough to give you advice in this direction. Maybe GitLab provides some special mechanism directly for this use case?

---

In your case:

The main goals of SoC are to manage complexity, increase modularity and facilitate maintenance.

I summarise the points you give in your question as *increase pipeline efficiency*.

So the questions are:

* Does "test during build" increase the performance so much that it pays the additional maintenance costs?
* How complex are the building and testing logics for themselves? How easy is it to split them apart again if you put them together now?
* To put in the two additional principles "*Premature Optimization*" and "*You Aren’t Gonna Need It*": Do you *really* need this efficiency now? On the other hand, do you *really* need the separation now?
* Maybe it is possible to do both, separate building and testing each in their own script files but call them both from one job?

**If you cannot find strong arguments in favour of one or the other now, do as you like and be prepared to refactor later when your needs become clearer.** Whatever you do, document your decision and the reasons for it together with the alternative as comments in your pipeline definition. Example:

> Build and test are done in a single job to avoid pushing/pulling the Docker image to/from registry to increase performance. If the complexity of build/test logic increases too much, split them in separate steps and check if performance is still acceptable.

This way you and your colleagues in the future can better understand *why* it is the way it is currently and whether it makes sense to change it.
> 2 votes

---
Tags: continuous-integration, gitlab
---


# Question
Title: how can I commit and push only the files what I need while using vscode?
I started using vscode, have two branches like main and test,after finishing the work on test branch I switch to main branch and what I want to do is to merge test branch but only to commit&push specific files/directories on main branch.Because I do not need to commit&push all files coming from test branch.Problem is that vscode does not allow me to switch to test branch if I commit&push specific files.So I have to delete the specific files I do not need on main branch and then to commit&push on main branch.I want to commit&push on main branch only successfully tested files coming from test branch.

# Answer
> 1 votes
Before you change to main, you need to run:

`git stash push`

in vscode - first you access the ... menu at the top of the source control window, then select stash-\>stash.

According to the docs, this will stash away all remaining changes after the check in on the test branch. Once you stash, you will be allowed to change over to main without vscode or git complaining.

When you switch back to test branch to continue your work, just run

`git stash pop`

In vscode, its the ... menu in the source control window, then stash-\>pop latest stash

# Answer
> 1 votes
@hakkican maybe this could help

> git checkout branch\_name path\_file/path\_folder

then commit the changes.

or check this link for more explanation https://stackoverflow.com/questions/449541/how-can-i-selectively-merge-or-pick-changes-from-another-branch-in-git

---
Tags: github, vscode
---


# Question
Title: docker-compose exec does not execute command even when container is running
I have a docker-compose-custom.yml file like this:

```
version: '3.8'

services:
  myapp-back-preproduction:
    build:
      context: .
    container_name: myapp-back-preproduction
    image: 'myapp-back-preproduction:latest'
 ....

```

I want to execute a simple command within the container after this one started to run using `docker-compose exec`:

`docker-compose -p myapp-back-preproduction -f docker-compose-custom.yml exec npm install`

I expect `npm install` to be executed within the container.

But it displays nothing in the console, it simply returns.

`docker exec` instead works but it logically lacks all environment variables declared in my `docker-compose-custom.yml` file; I want to use `docker-compose exec`.

My `docker-compose` version is: **1.29.2**

How to make it work?

# Answer
> 2 votes
You messed up with the syntax.

```
$ docker-compose exec -h
Execute a command in a running container

Usage: exec [options] [-e KEY=VAL...] [--] SERVICE COMMAND [ARGS...]

```

Your command should be like:

```
docker-compose -f docker-compose-custom.yml exec myapp-back-preproduction npm install

```

`-p` switch is for specifying a project name: identifier that used to construct container name like `<project_name>_<service_name>_1`

# Answer
> 1 votes
You can use `docker exec -it <container> <command>` to execute a command in a running container (see docs). Notably, this enables you to start an interactive shell session within the container, which is really handy for all sorts of things.

I do wonder why you would need to call `npm`, though. If this is necessary to run your application, it should be done in the Dockerfile. (Alternatively it could be done in a start-up script.)

Do note that your container will only stay up while it is actually running an application (that is, while the entry point call did not return), so starting up an inactive container and then manually running applications won't work. The container would simply be stopped immediately.

# Answer
> 0 votes
this is sample command

> docker-compose exec nginx apk add nodejs

it that command i try to install nodejs, so the command you need to execute

> docker-compose exec container\_namme npm install xxx

---
Tags: docker, docker-compose
---


# Question
Title: How to dockerize a jupyer notebooks project?
I have a project composed of 6 different Jupyter Notebooks and 4 CSV files. I would like also my colleagues can try to use this project, even if they don't have python installed and maybe with another csv. Is it possible to do this by creating a docker image of this project and letting them "play" with the notebooks and the data?

# Answer
> 2 votes
Yeah, you can totally do this. We actually do a step up above that. We run https://zero-to-jupyterhub.readthedocs.io/ over kubernetes. It gives each user their own notebook server on kubernetes, but its based on a given docker image. So, we build notebooks and some data files into the images so that each user starts with default examples/tutorials.

More recently, we also found "The Littlest JupyterHub". This is much simpler and lets you host a multi user JupyterHub server which you can do similar things to -\> https://tljh.jupyter.org/en/latest/index.html.

The two above options are nice as you can "host" the service for everyone on a server/site so they don't have to set up their local computers. But they still each get their own area/sandbox.

If you really just want a docker image, then start your docker file with `FROM` the image here -\> https://jupyterhub.readthedocs.io/en/stable/quickstart-docker.html. Then you can `COPY` the notebook and CSV files to the user home directory. Then people can use your image instead, if you host it on DockerHub or some org registry like ECR/GitLab/Artifactory/etc.

This last option is less ideal though as it requires users to actually have Docker installed and have some knowledge of using it (pull image/create/manage containers). Many people that use Jupyter/Python for some data tasks often have no idea about Docker.

The service alternatives above let you deal with the set-up so your users can focus on the product.

---
Tags: docker, containers, jupyter-notebooks
---


# Question
Title: How to resolve merge conflicts in Azure DevOps current UI
I have followed the instructions on command line: https://docs.microsoft.com/en-us/azure/devops/repos/git/merging?view=azure-devops&tabs=visual-studio

all I want to do is resolve this merge conflict its complaining about.

To be clear, I have done a `git pull`, it says `Already up to date.`

I also tried:

```
git pull origin feature/ENGA-2514
From ssh.dev.azure.com:v3/nfib/NfibEnterprise/engage.mobile
 * branch                feature/ENGA-2514 -> FETCH_HEAD
Already up to date.

```

I have also tried:

```
git log --merge
fatal: --merge without MERGE_HEAD?

```

Where is this elusive merge conflict?

# Answer
Try this solution: Pull Request Merge Conflict Extension. It helps to resolve conflicts inside the pull request page.
> 14 votes

# Answer
I had this problem too, coming from github where you can usually resolve text file changes in browser.

On your local repo, you want to get the latest then reverse-merge, so in your case

```
// Precursor to ensure your local is the same as origin
git pull master
git checkout feature/ENGA-2514
git pull feature/ENGA-2514

// include all of the other branches changes into yours.
git merge master

// This is where you'll be able to resolve the conflicts locally
// and commit the merge into feature/ENGA-2514

git push feature/ENGA-2514

```

Then your branch wont conflict with master in devops, since youve just merged master into your branch.
> 7 votes

# Answer
I am not sure this tab appears 100% of the time, you still have to merge some cases locally, but you can merge *some* PRs without any extensions in latest UI update.
> 4 votes

# Answer
For me, I cannot resolve these conflicts.

**To PREVENT THEM:**

```
  i) Make backups outside the folder system for the repository

 ii) Totally delete your 'local' folder for the repository.  (Seems
     counterintuitive, but it is all that works for me).

iii) Re-load 'local' folder from the web-repository.

 iv) Eventually, check-in, then it will work.

  v) Repeat step i for next time.

```
> 0 votes

# Answer
Pull/Check out the branch with these merge conflicts. Edit the file with reported merge conflicts locally. Push the changes and it will be resolved.
> 0 votes

---
Tags: git, azure-devops
---


# Question
Title: How to resolve a problem "certificate signed by unknown authority" in GKE on pulling image (a private registry) when a pod is created?
I generated a CA certificate, then issued a certificate based on it for a private registry, that located in the same GKE cluster. Put the server certificates to the private registry and the CA certificate to all GKE nodes and run:

```
update-ca-certificates && systemctl restart docker

```

Images are building and putting into the private registry without problems. When a pod tries to pull the an image from the repository I get an error:

```
x509: certificate signed by unknown authority

```

Also I tried to put the CA certificate to the docker `certs.d` directory (10.3.240.100:3000 — the IP address of the private registry) and restart the docker on each node of the GKE cluster, but it doesn't help, too:

```
/etc/docker/certs.d/10.3.240.100:3000/ca.cert

```

How to solve this problem? Am I understand correctly that the GKE nodes' docker is responsible for pulling images when creating a pod?

# Answer
> 1 votes
I found a solution. The problem is actual for Kubernetes version `1.19+` and `COS`/`Ubuntu` images based on `containerd` for GKE nodes. Before the 1.19 version Kubernetes used to use `Docker` for building images, but now it uses `containerd`. More details could be found in the official Google Cloud documentation.

You need to create and put an CA certificate to each GKE node. Here you can find an answer how to do it correctly https://stackoverflow.com/a/67724696/3319341.

But for `containerd` solution you should replace command

```
nsenter --target 1 --mount systemctl restart docker

```

by

```
nsenter --target 1 --mount systemctl restart containerd

```

A more detailed answer: https://stackoverflow.com/a/67990395/3319341

# Answer
> 1 votes
If your on Standard Mode for your GKE cluster, go into Computer Engine \> SSH into every node and put into /etc/docker/certs.d/\<ip:port\>/ca.cert. Create folder if it does not exists.

This works for me. The method will not work for Autopilot Mode.

My exact steps on each node:

```
mkdir -p /etc/docker/certs.d/1.2.3.4:5000
cd /etc/docker/certs.d/1.2.3.4:5000
vi /etc/docker/certs.d/1.2.3.4:5000/ca.crt

```

Lastly, create and use the imagepullsecrets in your deployment

```
kubectl create secret docker-registry my-secret \
--docker-server=1.2.3.4:5000 \
--docker-username='testuser' \
--docker-password='testpassword' \
--docker-email='test@example.com'

cat <<EOF | kubectl apply -f
...
 spec:
   containers:
   - name: hello
     image: 1.2.3.4:5000/test:7.12.0
   imagePullSecrets:
   - name: my-secret
EOF

```

# Answer
> 0 votes
You can now use the DaemonSet to load self-signed CAs when using containerd: https://github.com/samos123/gke-node-ca-importer

Credit to @nstogner: https://github.com/samos123/gke-node-ca-importer/commit/756674ee595e1cf86df4b3181bb3cf687bd72c97

---
Tags: docker, kubernetes, google-cloud-platform, ssl, google-kubernetes-engine
---


# Question
Title: Azure DevOps User CAL License for TFS 2013
As per the below document dated 03-Dec-2020, if we buy monthly access to Azure Devops, we can have user CAL to use Azure DevOps Server (TFS 2015 to Azure DevOps 2020) . When we check for TFS 2013, it says "The requested page is not available for Team Foundation Server 2013. You have been redirected to the newest product version this page is available for."

We would like to know as per this link -- -- if we have monthly access to Azure Devops, then can we have access to both Azure DevOps Services and Azure DevOps Server User CAL (TFS 2013) ?

# Answer
> 0 votes
No, only Azure DevOps Server 2019 supports that behavior. https://azure.microsoft.com/en-us/pricing/details/devops/server/

---
Tags: azure-devops, tfs
---


# Question
Title: What is the purpose of running PHP-FPM in its own container instead in the same container as apache2/nginx?
I'm fairly new to Docker so excuse any obvious ignorance or misunderstandings. That said, I've been coding and configuring web applications for a long time now. I have recently been dabbling with some more sophisticated (Docker and "traditional") web/application server setups and experimenting with performance enhancements and simplifying deployments.

My personal favorite configuration to use for most projects thus far is nginx as a (mostly) "static file" web server &&|| caching mechanism &&|| Load Balancer in a reverse proxy config with Apache v2.4+ as the "backend" which runs PHP (also Perl &&|| Python) in a "FastCGI" (PHP-FPM) configuration (with mod\_php disabled) and mpm\_events (instead of mpm\_prefork). We can always add in something else also such as REDIS or memcached, where applicable (but I rarely see a significant performance gain when using Cloudflare caching).

I've been dabbling with a few different ways to keep my "favorite web server composition" flexible and "ephemeral" enough to accommodate any and all possible options for further performance gain or resource load management. What I keep seeing is that there seems to have been a shift towards using PHP-FPM in its own separate "stand-alone" Docker container sometimes around late 2019.

Why?

While I can appreciate keeping resources isolated and separate for an easier to debug/config/secure configuration(s), I don't fully understand what the benefits are to having PHP-FPM in a separate Docker container that is implicitly REQUIRED by the application that the Docker containers are comprising.

If anything, having a separate PHP-FPM container seems like additional configuration, additional resource consumption and an even more complicated build/deploy consideration in a CI/CD pipeline.

I can even get onboard with "simple preference", such as instead of using Apache2 and nginx on the same Ubuntu/Debian or RHEL/CentOS container, break off nginx into its own container(s) and simply config your hostname(s) &&|| IPs and ports appropriately.

But what is a practical use case and advantages for having PHP-FPM separated from Apache or nginx containers for any additional benefit beyond perhaps using more Dockerfile &&|| docker-compose.yaml and less ba.sh scripting to get the same job done? If any part of the application fails or falters, then functionality will suffer. The additional network &&|| docker-network communications and hostname resolutions seems like it would just be a performance penalty over combining certain aspects of the docker-composer-ed application together, rather than splitting them up simply for the sake of splitting them up.

# Answer
Im heavily investigating this, and like many people, im discovering that in 2020 there doesn't seem to be so much logic in separating tightly coupled webserver + PHP + app process/code from each other:

**Two main arguments for separation are scalability and separation of concerns (one process per container).**

Logically the webserver should only act as webserver and distribute traffic to PHP nodes behind a service that uses a LB.

This way one container/service serves webserver purpose and serves static files, while proxying the php requests to the php pod which only concerns PHP.

And webserver and php pods can scale differently - ie you may need 50-100 PHP pods for a workload, but only 3-4 webserver nodes. So you save on resources. Great!

**Except in respect to separation of concerns...**

If the webserver is not going to serve requests to different apps, there is little logic for the separation of concerns to different containers - leave aside services. If the app is coupled with the webserver, you just end up spawning extra webserver pods, with accompanying services and load balancers to go with them.

On top of that you will have to distribute the app code to both containers or mount them from a NFS bind, or non NFS mount. You will have to deal with all the relevant access and permission stuff, if you put the app code in both pods you will have to deal with static + dynamic file separation, or even end up having to distribute both types of files to both pods. You could put two containers and make one of them mount directories on the other and that would work, but that actually puts two containers in the same logical host in regard to network, host etc so you are still combining them. This still provides some separation of concerns though. But the communication in between the two containers in the same pod is a concern - it has to either happen through intra-host network over loopback, which uses the network stack and is less performant. Maybe sockets could work.

Even worse if your app may require webserver being present in the same container - for example if you are doing ssl termination and have to carry over the visitor's actual ip to the php pod, you may end up having to jump through some hoops. While both webserver and php are available in the same container, this can be done through various means.

When everything is in one container, none of these is necessary. Everything is neatly packed in the same place, no need to track different deployments, services, app codes, or replication controllers.

This kind of singular container/deployment is very advantageous today especially if you are using ingress-nginx or nginx-ingress and hosting many different apps/sites in the same cluster. Each app/site is contained in its own container (imagine different WordPress sites), and each site is completely contained in its permissions, access, deployment, version, requirements etc.

So much that you could even allow apps/sites to customize their runtime environments by providing configuration files for webserver or php from NFS shares or other mounts - ie .htaccess, php.ini etc. Which is a must for hosting different apps since people tend to have different requirements.

**In respect to the performance**

Today NGINX or Apache require very few resources with event-based request handling. For example in my benchmarks, i saw Apache 2 with event mpm requiring only 3-4 MB memory, and a grand total of ~2-3% cpu (out of 1000m requested cpu, ie 1 vcpu) while handling 50-100 concurrent requests per second while serving WordPress.

The weights of webserver vs PHP pods would differ depending on the app or website of course, and there could definitely be apps that would require such a separation, but for most common web workloads this does not seem necessary.

So spawning a few Apache pods and then spawning hundred or more PHP pods does not seem to bring much gain. Aside from creating internal cluster network traffic as pods need to communicate with each other and it also creates more complications in the app configurations (deployments, services, load balancers etc).

Another disadvantage of separating containers is that you create two times the services per app. This will reflect on the limits of the cluster you create since services, pods require ip addresses.

When webserver + PHP +app are in the same container, the webserver can just communicate with php-fpm via file sockets. Which is faster, and creates less network overhead even on the loopback interface inside the container, leaving aside creating network traffic inside the cluster. Internal network and cpu load saved.

**In conclusion...**

For specific purposes, with one Apache 2 or NGINX server distributing requests to different apps on different PHP-FPM clusters, separating concerns would likely be a necessity.

But in the era of ingress-nginx, this really doesn't seem necessary.

Especially reduced complexity, increased portability, saved performance and internal cluster noise, better customization, and other benefits gained from packaging tightly coupled server + PHP + app deployments in one single container seem to be too good to pass.

Furthermore, this enables even shared-hosting-like formats in Kubernetes clusters, which seems to open many possibilities.

I would very much appreciate any input from anyone who had experience in dealing with this choice between selecting singular containers for a tightly coupled app vs multiple containers, or anyone who did benchmarks on these.
> 11 votes

# Answer
Separation of concerns is quite a strong reason to keep web server and php-fpm on separate containers, especially in Kubernetes.

1. Containers can be created and maintained by separated teams (PHP by Dev, Nginx by IT-Ops/DevOps), for example, different base images
2. You can change configuration of Nginx without rebuilding the app
3. Same as point 2 with upgrading Nginx server

If you have microservices in the Kubernetes cluster I would prefer to have two containers per pod. For example for 30 microservices similar PHP services you really don't want to rebuild all of them only to change some Nginx logs format. Of course, you can keep some Nginx configuration as a config map and share it in this way.
> 3 votes

---
Tags: docker, docker-compose, nginx, php, apache-httpd
---


# Question
Title: CDK: How to solve chicken and egg problem of Lambda backed with ECR
In CDK, if I have something like this:

```
const repository = new ecr.Repository(this, 'SomeRepository', {
  repositoryName: 'SomeRepository',
});

const lambda = new lambda.DockerImageFunction(this, 'SomeLambda, {
  code: lambda.DockerImageCode.fromEcr(repository, { tag: 'latest' },
  functionName: 'SomeLambda',
  vpc: this.vpc,
  vpcSubnets: this.vpcSubnets,
});

```

CDK will fail to deploy, because there is no image present in the ECR that was just barely created. The idea is to set everything up, and then setup github actions to upload the image, and tell the lambda where to get the code.

How can I get everything set up so that it is ready for CD like this?

# Answer
You can use DockerImageAsset to initialize ECR with a base image like scratch or alpine, or even the most basic lambda container that would get your lambda function to execute.

https://docs.aws.amazon.com/cdk/api/v2/docs/aws-cdk-lib.aws\_ecr\_assets.DockerImageAsset.html

https://hub.docker.com/\_/scratch

Later, when you get the github actions going you just replace the image
> 1 votes

---
Tags: amazon-web-services, aws-cloudformation
---


# Question
Title: Is there an equivilent of a Kubernetes ingress for Docker Compose?
I'm more of a software developer than a networking/ops person to give you some context.

I'm currently creating an application that is a standard containerised microservices architecture, where my frontend is some static assets in an nginx container, and my backend APIs are expressjs servers running on node containers.

When my frontend makes a call to a backend endpoint, it's going to look like:

```
const response = await fetch("/api1/v1/items");

```

Because by default the browser is going to make this request to the frontend container, we need some way of redirecting that traffic to the correct backend container.

One standard way to do this, is that I can setup a reverse-proxy in the frontend container's `nginx.conf` to direct all `/api1/*` traffic to another service.

ie. you would do something like this:

```

location /a {
   proxy_pass         http://api1;

```

Where you have defined api1 as a service in your `docker-compose.yml`.

I'm a bit uncomfortable with this, at the cluster configuration is mixed up with the implementation.

I am aware with Kubernetes ingresses, this routing can be done from outside of the container - where essentially the traffic is routed as it enters the cluster.

This is a much nicer solution in my view.

But the reason I'm using docker-compose is just for simplicity - it's more for running the cluster on your local machine and just getting the whole thing up and running.

Is there a similar configurable ingress concept for docker-compose?

# Answer
Kubernetes ingress is implemented by changing the configuration of another container or the cloud load balancer to route traffic to the service port/ip. Docker by itself is fairly lightweight, only runs the containers you start, and does not have external dependencies on things like your cloud load balancer. The solutions to setup a reverse proxy inside a container are almost identical to the kubernetes nginx based ingress controllers, with the key difference being the configuration of that reverse proxy.

There are a variety of docker based reverse proxies that configure themselves, including docker-flow-proxy, nginx-proxy, and Docker-EE ships with interlock that configures HA proxy. Probably the easies solution I've seen to implement is traefik which can be configured to monitor the docker socket for changes and automatically reconfigure based on labels set on your containers.

One important difference from Kubernetes is that docker networks are not flat, you will need to explicitly place the proxy container on the same docker network as your application in order for them to be able to route traffic between each other.
> 1 votes

# Answer
`jwilder/nginx-proxy` image simulates that excatly. Use VIRTUAL\_HOST & VIRTUAL\_PATH env vars in the backend containers.

example:

```
version: '3.4'

services:
  ingress:
    image: jwilder/nginx-proxy
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/tmp/docker.sock:ro

  frontend:
    image: nginx:1.16
    ports:
      - "81:80"
    environment:
      - VIRTUAL_HOST=test.com

  api:
    image: nginx:1.16
    ports:
      - "82:80"
    environment:
      - VIRTUAL_HOST=api.test.com

```
> 1 votes

# Answer
I ended up using `dns-proxy-server` in combination with nginx reverse proxy (serving as an ingress) on the host using a docker bridge network (not the default one). I wrote Ansible roles and playbooks to run Docker containers in that one network.

In this formation, dns-proxy-server would act as a DNS server for anything running on the host trying to connect to a Docker container (including nginx acting as an ingress). This setup helped with the requirement where you'd wanna have nginx blocks as mentioned in the question.

I have since moved to k3s on a single node for all my container needs.
> 0 votes

# Answer
Not a lot of difference between setting up a backend in nginx or an ingress in kubernetes. They work essentially the same way and that's commonly known as reverse proxying, as you said.

What you need in that case is not just a reverse proxy, but an API gateway. We can spend days arguing about your application design, I'm more of a subdomain instead of /prefix guy. But regardless of the decision making, you need something to handle that.

Option 1 Kong Gateway:

https://konghq.com/install/#kong-community

Option 2 Traefik:

https://doc.traefik.io/traefik/

Both work in compose and kubernetes, both are reverse proxies with API gateways capabilities and both have their own kubernetes ingress controllers.

There are loads more, those are the ones I know, trust and have used in production with no awkward surprises.
> 0 votes

---
Tags: docker-compose, microservices, ingress
---


# Question
Title: Ansible playbook error: Collection' object is not callable
I am learning ansible on my local vagrant vm machine and I am setting up mongodb using ansible and I keep getting the error:

```
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: 'Collection' object is not callable. If you meant to call the 'authenticate' method on a 'Database' object it is failing because no such method exists.
fatal: [192.168.56.11]: FAILED! => {"changed": false, "module_stderr": "Shared connection to 192.168.56.11 closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1640759955.2153473-3678-226245830655855/AnsiballZ_mongodb_user.py\", line 107, in <module>\r\n    _ansiballz_main()\r\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1640759955.2153473-3678-226245830655855/AnsiballZ_mongodb_user.py\", line 99, in _ansiballz_main\r\n    invoke_module(zipped_mod, temp_path, ANSIBALLZ_PARAMS)\r\n  File \"/home/vagrant/.ansible/tmp/ansible-tmp-1640759955.2153473-3678-226245830655855/AnsiballZ_mongodb_user.py\", line 47, in invoke_module\r\n  ...

```

The anisble playbook configuration:

```
# tasks file for mongodb
- include_vars: secrets.yml
- name: Installing mongodb
  apt:
   name: mongodb
   state: present
- name: Start and enable the engine
  service:
    name: mongodb
    state: started
    enabled: yes
- name: Configure Mongodb to accept outside connections
  lineinfile:
    path: /etc/mongodb.conf
    regexp: '^#?bind_ip = 127.0.0.1'
    line: '#bind_ip = 127.0.0.1'
  notify:
     - restart mongodb
- name: Update Ubuntu
  apt:  update_cache=yes force_apt_get=yes cache_valid_time=3600
- name: Install Python pip
  apt:
    name: python3-pip
    state: present
- name: Install pymongo package
  pip:
    name: pymongo
    state: present
- name: Is authentication enabled?
  command: grep "^auth = true" /etc/mongodb.conf
  register: auth
  ignore_errors: yes
- name: Add an administrator to the db
  mongodb_user:
    database: admin
    name: root
    password: "{{ dbpass }}"
    roles: root
    state: present
  when: auth.rc == 1
- name: Enable authentication
  lineinfile:
    path: /etc/mongodb.conf
    regexp: '^#?auth \= true'
    line: 'auth = true'
  notify:
    - restart mongodb
- name: Create the application user
  mongodb_user:
    login_user: root
    login_password: "{{ dbpass }}"
    database: "{{ dbname }}"
    name: "{{ appdbuser }}"
    password: "{{ appdbpass }}"
    roles: dbOwner
    state: present

```

This part from my analysis of the error message is the issue:

```
- name: Create the application user
  mongodb_user:
    login_user: root
    login_password: "{{ dbpass }}"
    database: "{{ dbname }}"
    name: "{{ appdbuser }}"
    password: "{{ appdbpass }}"
    roles: dbOwner
    state: present

```

My anisble machine specifications are:

* Ubuntu vm:

```
 Linux ubuntu 5.4.0-80-generic #90-Ubuntu SMP Fri Jul 9 22:49:44 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
 Ubuntu 20.04.2 LTS

```
* Installed ansible version on Ubuntu vm:

```
 ansible [core 2.12.1]
    config file = /etc/ansible/ansible.cfg
    configured module search path = ['/home/vagrant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    ansible python module location = /usr/lib/python3/dist-packages/ansible
    ansible collection location = /home/vagrant/.ansible/collections:/usr/share/ansible/collections
    executable location = /usr/bin/ansible
    python version = 3.8.10 (default, Jun  2 2021, 10:49:15) [GCC 9.4.0]
    jinja version = 2.10.1
    libyaml = True

```

# Answer
*Note: this answer will very likely and quickly become obsolete when a new version of the `community.mongodb` collection is released*

---

The current version (i.e. *v1.3.2* at time of this writing) is not compatible with the python module `pymongo` version *4.x*. You need to downgrade the library to version *3.x* until the collection is fixed.

The current latest commit on the collection master branch is actually fixing the library version to *v3.12.2* but has not yet been relased. Hopefully this will soon make it into a patch version for the collection.

In the mean time, it seems that the developers have already started a branch for pymongo4 compatibiliy. There is also a specific issue related to this problem.
> 3 votes

---
Tags: ansible, mongodb
---


title: Run a dev server in CI pipeline

text:
# Question
Title: Run a dev server in CI pipeline
I have a CI pipeline setup using Github Action/Workflows, where I want to run Cypress Automated tests; however I am having some problems of when trying to run my dev server. Let me show you my pipeline:

```
name: Nuxt CI Pipeline

on:
  push:
    branches: [ CI-pipeline ]
  # pull_request:
  #   branches: [ master ]

jobs:
  build:

    runs-on: ubuntu-latest

    strategy:
      matrix:
        node-version: [ 14.x ]
        # See supported Node.js release schedule at https://nodejs.org/en/about/releases/

    steps:
    - uses: actions/checkout@v2
    - name: Use Node.js ${{ matrix.node-version }}
      uses: actions/setup-node@v2
      with:
        node-version: ${{ matrix.node-version }}
        cache: 'npm'
    - name: Make envfile
      uses: SpicyPizza/create-envfile@v1
      with:
        envkey_ENV: staging
        file_name: .env
    - run: npm ci
    - run: npm run dev
    - run: |
        cd e2e
        ls -l
        npm ci
        npx cypress run

```

Now I want to spin up the devserver and run the tests on that port usually 3000; however the problem is when the command `npm run dev` is executed, the pipeline keeps on waiting there and doesn't move forward to the next commands, which makes sense as devserver doesn't return a response – it listens on that port until explicitly closed. I'm stuck here. My knowledge of devops is the bare minimum; can someone point out what I am missing?

# Answer
The pipeline is configured correctly, you were just missing one piece of the puzzle called service containers.

Basically, the service containers also known as sidecars will run alongside your "main" pipeline logic and support your commands. In your case, you will run the `npm run dev` in a service container once the build job starts and when the `npx cypress run` runs it will have a dev server to connect to.

Follow the example GitHub added for Redis
> 1 votes

---
Tags: continuous-integration, github-actions
---


# Question
Title: What is the difference between Fluent Bit and Prometheus?
From what I've seen, Prometheus scrapes data and metrics and displays them and Fluent Bit collects logs and metrics and can output them.

So, are they doing the same thing, i.e. should I choose one over the other or am I missing something?

# Answer
This is a little off. You are comparing a "Time Series Database" to a "Log Shipper".

**How Prometheus Works**

Any service/entity prometheus wants to run monitor has to provide an "exporter". An exporter is just a http endpoint that has metrics in a simple text format. The exporter can be a standalone app, a web sever linking to a file, or something like spring boot literally just adding a url to serve prometheus metrics. Lots of other systems can pickup and use this format, not just prometheus.

You configure prometheus to "scrape" these metrics and store them in its own Time Series Database, usually at 30 second intervals. Then prometheus is basically your metrics database. It also integrates with alert manager to help you trigger alerts to slack, pager duty, etc.

**How Fluent Bit Works**

Fluent Bit is a log shipper. It is better compared with, say, logstash. It can basically take any kind of log data (text lines) and ship/route/filter/etc them to a destination system. It is not itself a time series database like prometheus though. Also prometheus is not a log shipper, it can't go make arbitrary routing decisions on the data being sent to it.

This may be worth a skim - https://logz.io/blog/fluentd-logstash/.
> 1 votes

---
Tags: kubernetes, prometheus, logging, fluentd
---


# Question
Title: How to start a container with NFS volume if NFS server is offline?
This issue is similar to one posted a couple of years ago but with a slight difference. I'm trying to identify how to spin up a Docker container which has an NFS Volume mount whether or not the NFS server is online.

I have a Docker volume that I have created from an NFS mount to a device which comes on and off the network at regular intervals:

```
docker volume create --driver local --opt type=nfs --opt o=addr=192.168.2.237,rw,soft,timeo=10,bg --opt device=:/media raspberryPi

```

In my mount options, I have intentionally specified a soft mount such that if the mount is not available, an error will return to the process running within the container and the application will handle the I/O error.

In theory, my setup works fine IFF the container is started when the NFS mount is present.

```
[eric@docker1 tmp]$ docker run --rm -it -v raspberryPi:/pi ubuntu bash
root@f8fd958740da:/# ls /pi
lost+found  x  y

```

If remove the device from the network, the NFS access fails from within the container (as expected):

```
root@f8fd958740da:/# ls /pi
ls: cannot access '/pi': Stale file handle

```

If I re-attach the device to the network, then the resource becomes available again, and everything continues to work.

The problem arises, however, if the NFS volume is NOT present at container start time. If it isn’t present, then the container fails to launch altogether:

```
[eric@docker1 tmp]$ docker run --rm -it -v raspberryPi:/pi ubuntu bash
docker: Error response from daemon: error while mounting volume '/var/lib/docker/volumes/raspberryPi/_data': error while mounting volume with options: type='nfs' device=':/media/movies' o='addr=192.168.2.237,rw,soft,timeo=10': operation not supported.

```

Is there anyway I can configure docker to “ignore” the volume mount error and continue to mount it none-the-less? I tried using a background mount in the options, but that did not help. Essentially, I would like the container to launch with a mount configured to /pi even though the NFS Server is offline (similar to behaviour if the server is online and taken offline while the container is live).

# Answer
No can do. You need to have all available bindings on container start. Imagine if the same happened with a port binding? Or a network binding even?

You can attempt to mount the nfs folder after the container boots, have the NFS IP and paths as an env var, craft a script and bake it into the container or in some trusted storage that you can pull, like github.

This solution will boot the container but won't mount the files, which is what I'm assuming you need.

You would commit to a public repo or gist something like this:

```
#! /bin/bash

mount -t nfs -o proto=tcp,port=$NFS_PORT $NFS_IP:$NFS_PATH_REMOTE $NFS_PATH_LOCAL
bash

```

Then you could run

```
docker run --rm -it -e $NFS_PORT=2049 -e $NFS_IP=192.168.2.237 -e $NFS_PATH_REMOTE=/media -e $NFS_PATH_LOCAL=/pi ubuntu curl https://raw.githubusercontent.com/eric-b/docker-scripts/master/mount-nfs.sh | bash

```

That's all because I'm assuming you're running this from a remote environment, very likely a raspberrypi if you're running this from your local machine, you could have a file called .env-nfs

```
NFS_PORT=2049
NFS_IP=192.168.2.237
NFS_PATH_REMOTE=/media
NFS_PATH_LOCAL=/pi
NFS_SCRIPT=https://raw.githubusercontent.com/eric-b/docker-scripts/master/mount-nfs.sh

```

And run

```
docker run --rm -it -env-file ./env-nfs ubuntu curl $NFS_SCRIPT | bash

```
> 1 votes

---
Tags: docker
---


title: Internal proxy for external system

text:
# Question
Title: Internal proxy for external system
I have a web server in a Kubernetes `Deployment` which uses an ElasticSearch instance on ElasticCloud.

I have parameterised (thanks to environment variables) my container, from a `Secret`, with this kind of url: `https://elastic:***@xxx.yyy.eu-west-3.aws.elastic-cloud.com:9243`

I'm really dissatisfied for these reason:

* My container have access to the credentials
* I have downtimes whenever I change the password

Is there a kind of `Service`, or any good practise, which would be able to proxy ElasticCloud (adding credentials)? If it is possible, I'd like to have a mechanism to update credentials.

# Answer
I suggest you create a Deployment of `caddy:2` Pods which a Caddyfile mounted from a Secret, something like this:

```
http://elastic-proxy.default.svc.cluster.local:9243 {
    reverse_proxy https://xxx.yyy.eu-west-3.aws.elastic-cloud.com:9243 {
        header_up Authorization "Basic Zm9vOmJhcgo="
        header_up Host {upstream_hostport}
        header_up X-Forwarded-Host {host}
    }
}

```

The create a Service named `elastic-proxy` in the namespace (`default` in this example). Point the app Pods to this service instead of the Elastic Cloud.

Explanation:

This will create a Caddy 2 server which reverse proxies everything coming to port 9243 with virtual host `elastic-proxy.default.svc.cluster.local` to the Elastic Cloud and adds an `Authorization` header with the Basic Auth credentials (foo:bar here).

You can create the password with `base64` like this:

```
$ echo 'elastic:password' | base64
ZWxhc3RpYzpwYXNzd29yZAo=

```

Every time you change the password (why do you even change it?), just modify the Caddyfile in the Secret.
> 1 votes

---
Tags: kubernetes, elasticsearch
---


# Question
Title: access secret Azure DevOps environment variables in SpecFlow tests
I have a SpecFlow test suite running some Selenium UI tests for our web app in ADO, and we need to access some sensitive information to be able to log in to various places. They are currently stored in normal environment variables so they are in plain text and pretty easy to access, but too insecure to stay that way (it is currently a test user on a dev box, but that will change when the tests are production ready).

ADO secret env vars are not automatically set up in a way that an executable can access, and it does not look like I can pass parameters through `vstest.exe` to the individual tests.

I suggested something like `Azure Key Vault`, but that is not an option. I also got `dotnet user-secrets` working locally, but setting that up on all the agents is apparently not an option either.

I think that secret environment variables should be the way to go, but I am using .NET Framework 4.6.1 and all of the documentation I read to set those up with some sort of config file are for v5+ or Core. If anybody has experience dealing with this, any ideas would help!

Update: I found out it is supposedly possible to use a `.runsettings` file with empty parameters, and set them from within ADO, but it seems like setting the MSTest attributes conflicts with some of the SpecFlow stuff I need.

# Answer
Ok, I think it does what I want, although I'm waiting on new db permissions to completely test it...

I found some information on using `.runsettings`, but I couldn't use the MSTest attributes, because the compiler only allows one initialization, which SpecFlow uses under the hood. I found SF docs to grab the context surrounding the test, but I had to annotate the class and the setter method with `[Binding]` and `[BeforeScenario]`, respectively, which the docs didn't mention.

For anybody interested in the details, I created a `.runsettings` file in my solution directory with empty values:

```
<RunSettings>
  <TestRunParameters>
    <Parameter name="api_token" value="" />
    <Parameter name="password" value="" />
  </TestRunParameters>
</RunSettings>

```

And added to my `.csproj` file:

```
  <PropertyGroup>
    <RunSettingsFilePath>$(MSBuildProjectDirectory)\..\.runsettings</RunSettingsFilePath>
  </PropertyGroup>

```

And in ADO, I edited the **Visual Studio Test** task, setting new params in **Execution options \> Override test run parameters**:

```
-api_token: $(token) -password: $(password)

```

And wrote a C# class to get the values:

```
using Microsoft.VisualStudo.TestTools.UnitTesting;
using TechTalk.SpecFlow;

namespace UITests
{
    [Binding]
    public class Context
    {
        private static TestContext testContext;

        [BeforeScenario]
        private static void SetContext(ScenarioContext scenarioContext)
        {
            testContext = scenarioContext.ScenarioContainer.Resolve<TestContext>();
        }

        public static string GetSecret(string key)
        {
            if (testContext.Properties.Contains(key))
            {
                return testContext.Properties[key].ToString();
            }
            return null;
        }
    }
}

```

Using that class, the I can get the DevOps secrets from that `testContext.Properties` dictionary, after they overrode the empty values in my `.runsettings` file.
> 0 votes

---
Tags: azure-devops, azure-pipelines, dotnet
---


# Question
Title: How can you map DNS/host names to their IPs at scale?
We're trying to look at a fairly large httpd config and find the all the IPs/ports we'd have to allow through a new firewall.

Ports are easy, but resolving all the DNS names to the (multiple) IPs they may have and outputting it in easy-to-consume manner seems hard.

What's the best way?

# Answer
> 2 votes
There may be other ways to achieve this. But the most elegant way we found was using python's socket library.

You can get all the info in a one liner, and extract it into a list in another line. It's fast, short, readable, and easy to put in a loop against a ton of records.

**The Code**

```
import socket

net_info = socket.getaddrinfo("stackoverflow.com", None)
ip_list = set([x[0] for x in [x[4] for x in net_info]])

print(ip_list)

```

**Output**

```
{'151.101.129.69', '151.101.1.69', '151.101.193.69', '151.101.65.69'}

```

---
Tags: dns, ip, network
---


title: K8S Init Container CRD/Operator

text:
# Question
Title: K8S Init Container CRD/Operator
I want to inject an init container into some pods. I know how to specify init containers in a pod's spec, but I don't want to do that directly because these sidecars are invisible to my engineers.

I don't want to go through the complexity of developing my own mutating web hook or operator, so does anyone know of a good package/service that can help?

# Answer
> 0 votes
Answering my own question here. Eventually, I went with OpenKruise, which is a pretty simple but cool open-source framework for, amongst other things, creating a centralised way of managing sidecars and init-containers in a cluster, without needing to concern application developers.

# Answer
> 2 votes
Tumblr and Salesforce have open sourced their generic injectors:

Both of these should fit your simple use case but offer a lot more.

---
Tags: kubernetes
---


title: Access to k3s DNS from host

text:
# Question
Title: Access to k3s DNS from host
I have a single node k3s "cluster" with few `Service`s on it.

Whenever I want to access to one of them, I'm forced to either do a `port-forward` or a `describe` (and get the endpoint, but it changes each time). From what I have read, `ingress` (with a local `nginx` `ingress controller`) suffers from the same issue.

Isn't there a way to reliably (ie. resisting to reload/reboot), plug my host machine DNS resolution on k3s DNS?

# Answer
Actually, I should have searched more: NodePort is what I want, letting me expose ports on my local machine.
> -1 votes

---
Tags: kubernetes, dns, minikube
---


# Question
Title: Helm: Long Running Jobs vs Long Running Hooks
I was wondering which would be more *appropriate / idiomatic* when writing a Helm-Chart:

1. Perform long running operations (such as db-migration) as part of the helm install process via **helm hooks** and inform the users that a longer *--timeout* is needed
2. Perform long running operations as a **Kubernetes-Job**

Edit: I'm talking about db-operations that can take many hours to complete

# Answer
A helm hook pretty much has to be a Job or Pod. Thus choosing between using a hook or using a Job doesn't make much sense - you are usually best off using a Job which is annotated as a pre-upgrade hook.

And yes, the timeout probably needs to be increased by the users. Unfortunately it's not possible to define the timeout in the chart itself.

In some cases it makes sense to have a db upgrade happen after the helm chart has been applied and in that case it can be a "normal" Job. This pattern however requires coordination between the Pods accessing the DB and the upgrade process such as a table in the DB which declares the version(s) of the schema.
> 0 votes

---
Tags: kubernetes, helm, kubernetes-helm
---


# Question
Title: is the Docker Certified Associate exam still endorsed by docker?
I'm trying to find some certifications and I found some info about the DCA but I can't find something about it on Docker's actual website. The only exception is a blog post from 2018.

What certifications (if any) are endorsed by docker or have the most clout/legitimacy?

# Answer
> 2 votes
That certification covered experience with enterprise components, like the Universal Control Plane (UCP) and Docker Trusted Registry (DTR). These all moved over to Mirantis. While they've rebranded the products, for some reason the certification itself has not been rebranded. However, it has little to do with Docker's OSS offerings, that remained with Docker Inc., other than being a shared based for the enterprise tooling. You're probably better off getting certified in the cloud you'll be supporting, and consider one of the Kubernetes certifications, unless you intend to support a Mirantis Enterprise project.

---
Tags: docker
---