# Question
Title: PROXY is not working inside windows container
I'm using windows containers in windows 10. I've set proxy variables(HTTP\_PROXY,HTTPS\_PROXY) in environmental variable setup and using VPN. I also configured resources -\> Proxies.

I was trying to build docker image with following Dockerfile content.

```
FROM mcr.microsoft.com/windows/servercore:ltsc2019
SHELL ["cmd", "/S", "/C"]

RUN mkdir c:\temp \
    && curl -SL --output c:\temp\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe

```

When this docker image build command is executed it shows following error.

```
Step 5/5 : RUN mkdir c:\temp     && curl -SL --output c:\temp\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe
 ---> Running in 0c134b2d004a
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:12 --:--:--     0curl: (6) Could not resolve host: aka.ms
The command 'cmd /S /C mkdir c:\temp     && curl -SL --output c:\temp\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe' returned a non-zero code: 6

```

I tried to build this with build arguments, and added dns: 8.8.8.8 in docker deamon config file

```
docker build --build-arg HTTP_PROXY --build-arg HTTPS_PROXY --tag=tswin .

```

But above mentioned error is not solved.

When the same curl command is executed in Windows's CMD it is not throwing any error

```
C:\Users\mshanm6x>curl -SL --output c:\temp\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
100 1432k  100 1432k    0     0   238k      0  0:00:06  0:00:06 --:--:--  332k

```

Also the same Dockerfile was able to built by without HTTP\_PROXY, HTTPS\_PROXY env variables and VPN off.

What is the real issue. Why it is not working inside of container with proxy configuration.

# Answer
I had the same problem and I found that you have to specify the proxy address also in the curl command.

So your Dockerfile become

```
FROM mcr.microsoft.com/windows/servercore:ltsc2019

ENV HTTP_PROXY=$HTTP_PROXY
ENV HTTPS_PROXY=$HTTPS_PROXY

SHELL ["cmd", "/S", "/C"]

RUN mkdir c:\temp \
    && curl -SL --output c:\temp\vs_buildtools.exe https://aka.ms/vs/16/release/vs_buildtools.exe --proxy %HTTP_PROXY%

```

and you will have to call the build command with this instruction

```
docker build --build-arg HTTP_PROXY --build-arg HTTPS_PROXY --tag=tswin .

```
> 0 votes

---
Tags: docker, dockerfile
---


# Question
Title: Is it possible to ask AWS-EFS to automatically remove those files that never been read in a period of time?
I would like to create an online painter web app.

The app would allow users to draw anything they like and save their drawings on the server.

However, many "guest" users may just come and draw some random strokes then never use those randomly-made drawings.

But I have to kept the files because some user may want to get their drawings a few days later.

So is it possible to set a logic for A

# Answer
You can't do this natively in AWS EFS. You can move it to cheaper storage natively in EFS by setting up `Lifecycle Management` to move the files last accessed X days ago to Infrequently Accessed (IA) EFS storage. This will reduce your storage cost but not eliminate them. This feature will also move the file back if they are accessed. See the reference below for more details.

You could run a command like the find below to find files not accessed for 30 days and delete them. NOTE: this will take up a lot of CPU cycles depending on the amount of files in the path. YMMV.

```
find /path/to/efs/* -atime +30 -exec rm {} \;

```

**References**
EFS Infrequent Access
Finding Files by Access Modification Date/Time
> 3 votes

---
Tags: amazon-efs
---


# Question
Title: What would be a better option, to migrate our server to an Azure VM or to migrate our database to an Azure SQL instance?
My current company is looking to benefit from the cloud by hosting our different web apps and sql servers on Azure. We're not a tech company and I'm the sole developer here and I need some guidance on the topic.

We've got a Windows Server 2019 machine on-premises that hosts our MSSQL databases. What would be the best way to perform a migration to azure? Migrate the Windows Server 2019 to an Azure VM instance, or to migrate the MSSQL database to Azure SQL instead? As far as I'm aware, I don't see the need for a VM as the server **only** hosts our database and does nothing else, but I want to get more information.

# Answer
From my inexperienced perspective, Azure SQL would be the go.

That is at least what Microsoft emphasised during the Certification I passed with them.

Database as a managed service would free your time of maintenance and further upgrade of the VM, mysql releases etc.

Maybe someone with more experience on cloud migrations could give a clearer answer.
> 1 votes

# Answer
I will concur with tintin. migrating from your on-prem sql server to an Azure SQL Server instance would be the way to go. Microsoft has outlined a tutorial to go through this exact thing here, though it is complete may be a bit overkill if you only wanted your data migrated: Overview Tutorial.

If you only wanted your data, you can do backup and restores of you data. you can use sql server management studio (SSMS) or Azure Data Studio to help manage the migration to/from your server. Some quick searches brings up Microsoft docs on many ways (including writing you own) to migrate between on-prem and Azure.

I would also like to point out that there is a hybrid between the two, Managed SQL instance. It is for a niche use case, so knowing your requirements is important, but it sounds like you just need an Azure SQL DB.

When spinning up your instance, know what performance you need what pricing scale you want. There is one that uses a SKU (basic, standard, premium) for size, and you pay for use using that SKU. There is another pricing model where you can scale your DB CPU and Memory independently of one another. It is a bit more expensive but allows for more flexibility.
> 1 votes

---
Tags: azure, sql, vm
---


# Question
Title: CI pipeline including upload to pypi repo
How should a ci pipeline be set up that publishes a python wheel to a private pypi server?

Say I have a publish job like this one:

```
publish:
  image: $PYTHON_IMAGE
  stage: publish
  before_script:
    - poetry install
    - poetry config repositories.nexus-pypi http://my-nexus.com/pypi-internal/
  script:
    - poetry build -f wheel
    - poetry publish -r nexus-pypi -vvv -u $NEXUS_PYPI_USERNAME -p $NEXUS_PYPI_PASSWORD

```

With this approach I have a problem with my workflow: Imagine I create a feature branch where I do some changes. I push my changes, pipeline runs and eventually publishes the package to the pypi server. However, once I want to merge that MR it will trigger the pipeline once again and publishing will fail as one cannot reupload packages more than once to pypi.

This comes along with quite some problems so what I'm asking for is how other do things like that or what best practices are known for this scenario.

Feel also free to point me to another \*overflow if this is not the right place for it.

# Answer
> 1 votes
You can use `rules` in gitlab ci to control when you want to stage to be executed: https://docs.gitlab.com/ee/ci/yaml/index.html#rules

e.g. set rules to only publish package when branch name is `master`.

You can even use regex for filtering branch name.

Hope it helps.

---
Tags: continuous-integration, gitlab-ci-runner, python, nexus
---


# Question
Title: Can you run chown on a root volume folder in a Dockerfile setup?
I have a docker container with a volume /foo defined in the docker-compose.yml. The Dockerfile then does:

```
RUN useradd -d /home/newuser -m -s /bin/bash newuser && \
    usermod newuser -g www-data
RUN chown -R newuser:www-data /foo

```

When the container is setup I can see it run the two commands with no errors, but the folder /foo is unchanged. When I run a shell on the container I can see the user is created. I can run chown on other directories and files fine. I'm not calling VOLUME anywhere in my Dockerfile for reference.

# Answer
> I have a docker container with a volume /foo defined in the docker-compose.yml. The Dockerfile then does...

The logic here is inverted. First you build an image with a Dockerfile. Then you run that image as a container with volumes and any other configuration. So you will not be able to modify the volume during the image build. Options to solve this are:

* Move the logic into your entrypoint. This requires your container to start as root, but it's possible to do something like `gosu` to later drop to your user before running your app.
* Have docker set the permissions for you as part of the named volume creation. When docker creates a new named volume, it initializes it with files from the image, including their ownership and permissions. This only happens when the named volume is empty, so once created and in use, changes to the image will not be seen in your volumes.
* Configure the volume permissions on the host before mounting it into the container. This is probably most appropriate for a host volume (aka bind mount).
> 0 votes

---
Tags: docker
---


title: Ansible Tower on a Free Linux Distro?

text:
# Question
Title: Ansible Tower on a Free Linux Distro?
Red Hat's Ansible Tower's documentation states that it can only be run on Red Hat Enterprise Linux. I can see once upon a time it could be run on other distros, such as Ubuntu. Is there a free Linux distro that Tower can still be installed upon?

# Answer
> 1 votes
The Ansible Tower Requirements still states

> CentOS 7.7 or later 64-bit (x86)

however

> you must have valid subscriptions attached before installing and running

An other option might be to use Ansible AWX

> AWX provides a web-based user interface, REST API, and task engine built on top of Ansible. It is one of the upstream projects for Red Hat Ansible Automation Platform.

---
Tags: ansible
---


# Question
Title: How to avoid disclosing account ID in AWS ARN
I'm new to the AWS ecosystem and have what might be a naive question. While trying to create a `GlueRunner` `Lambda` stack with `CloudFormation` (using `pynt`), from the `Cloud9` shell of an account with all the relevant permissions set up through `IAM`, `CloudWatch` showed me an error that occurred because I was using a "truncated" `ARN` without the region and the account ID:

```
[ERROR] InvalidArn: An error occurred (InvalidArn) when calling the GetActivityTask operation: Invalid Arn: 'Invalid ARN prefix: GlueRunnerActivity'

```

In a different CI/CD pipeline, however, I had used the truncated pattern, and it had worked without issues. Specifically, I had used `"executionRoleArn": "ecsTaskExecutionRole"` in the task definition config JSON in a pipeline to deploy a docker image with `ECS` using `GitHub Actions` with access to the required access key ID and secret access key through the repository `Secrets`.

Is there a way to use a pattern similar to the latter, which does not disclose the account ID, in the former workflow?

# Answer
> 1 votes
Your concern about obscuring your account ID is as valid as guarding your personal email address: if someone is trying to hack you, they could use your account id or email address to profile you or try and gain direct access to your account in a malicious way.

However, unmasked ARNs are ubiquitous within AWS (especially CLI inputs, outputs and CloudWatch logs). There is no easy way to automatically obscure them everywhere, and so far, AWS has not suggested we do this as a security best practice. They imply having unmasked ARNs in the logs is normal and needed for audits, troubleshooting, etc.

So the answer is: *its a risk that you must mitigate as you see fit*.

AWS does offer by default CloudWatch log encryption. They also advise a shared responsibility model when it comes to log security and suggest you can take further steps: like encryption with your own keys, doing regular key rotations, MFA depending on your risk assessment.

Reference to similar questions on devops and infosec.

---
Tags: amazon-web-services, aws-iam, aws-cloudformation, devsecops
---


# Question
Title: How can I force Docker Compose to rebuild containers when dockerfile changes?
I am using docker-compose to start 2 containers using the following **docker-compose.yaml**.

```
version: '2.4'

services:
  myservice1:
    container_name: mycontainername
    build:
       context: .
       dockerfile: ./my-dockerfolder/Dockerfile
       args:
       - MY_ARG=${MY_ARG}
    environment:
       - MY_ARG=${MY_ARG}
    mem_limit: 3500M
    ports:
    - "9090:9090"
    extra_hosts:
    - "one.mydomain.net:127.0.0.1"
    - "two.mydomain.net:127.0.0.1"

  myservice2:
    container_name: myothercontainername
    build:
       context: .
       dockerfile: ./other-dockerfolder/Dockerfile
       args:
       - BUILD=${BUILD}
       - MY_ARG=${MY_ARG2}
    environment:
       - MY_ARG2=${MY_ARG2}
    ports:
    - "2023:22"

```

My problem is that when I run **docker-compose up** again, it is using the same image to create the container, regardless of changes to the docker file.

How can I force it to take changes to the docker file into account?

Are other docker objects being reused that I need to be worried about? I want the second time I run **docker-compose up** to be as clean an environment as the first time.

note: I do not want to delete images that I do not need to delete. I have a slow connection to the docker repo, and removeing all the images would make my **docker compose up** take ~22 minutes.

# Answer
Why not `docker-compose build` ?

https://docs.docker.com/compose/reference/build/

You can also do `docker-compose up --build` to force a rebuild.
> 24 votes

# Answer
I've found sometimes I need `docker-compose build --no-cache` if I have a bad deploy.
> 12 votes

# Answer
Use this command:

```
docker-compose up --build

```
> 8 votes

# Answer
Another way that worked for me was to remove all images and then recreate all images.

Step 1. This will remove all images

`docker-compose down --rmi local/all`

https://docs.docker.com/compose/reference/down/

Step 2. This will create all images as part of up

`docker-compose up --build`

https://docs.docker.com/compose/reference/up/
> 0 votes

---
Tags: docker, docker-compose
---


# Question
Title: Should I expect EC2 instances to act up?
I recently had an issue with an EC2 instance. Briefly, the site was unavailable for 2 hours, a peak of CPU usage for no apparent reason, network traffic dropped close to 0, docker containers restarted, issues with the clock. The fault might have been not on their side, but...

**Do you expect** that EC2 instances might occasionally act up? After all, they seem to sort of guarantee no more than 0.5% downtime (3.6 hours per month?). Which means there might be downtime.

What do you do to avoid that? In my case I don't even know if that counts towards downtime, because technically the server was running, but network traffic was close to 0. And I'm not sure, but it seems their health/status checks didn't notice anything unusual (how do I check?)

**UPD** Most likely the physical server where it was running failed and the instance migrated to another server. Because there's the following line in the log:

> Jan 12 21:31:29 ip-172-xx-x-xx.xx-yy-z.compute.internal chronyd\[24287\]: Forward time jump detected!

# Answer
> 2 votes
Short answer: Yes, you should expect any IaaS, PaaS, SaaS platform to have issues from time to time.

There's no bullet-proof system. Sometimes the vendor has issues with the underlying hardware that can affect your server/container. My first step if I think AWS is having a hardware/virtualization issue is stop and start the instance. This moves your instance to a different AWS server. You should rarely need to do this though.

What you can do is monitor and alert on metrics that would indicate your service/server is not responding. In your example, Network traffic (bytes in) below a certain threshold for a certain period of time could indicate an issue.

To further protect the availability of your site you can (depending on your use case) create a second (or multiple) instance and use an application or network load balancer to check host health and send traffic accordingly. (ALB/NLB).

Bottom line, you should always expect issues (whether using a cloud provider or your own hardware) and build monitors to alert when an issue may be present and take appropriate actions.

**References**
AWS High Availability White Paper

---
Tags: amazon-ec2
---


# Question
Title: How do I configure my AWS SSO-managed user with my public SSH key?
My AWS identity is SSO-managed, not an IAM user.

I wish to upload my public SSH key (so as enable client UI access to CodeCommit) but am not the AWS account/organisation admin so cannot see what the options are or even if this is possible?

If there are any AWS SSO admins out there I'd be grateful for input.

Thanks and kind regards, Andrew

# Answer
Originally I wrote a nice answer about how to do it for IAM users, because I think it is the only way how you can use SSH public keys with CodeCommit. The key is uploaded to the IAM user profile and without an user you have nowhere to put the key. For SSO I would suggest to consider the AWS credential helper https://docs.aws.amazon.com/cli/latest/reference/codecommit/credential-helper/index.html

SSO usually allows you to assume some kind of a role in AWS so using that role for accessing any AWS service is what you are expected to do.

If you really have to use SSH Keys and there is no other way, I would consider to utilize the fact that `git` is a fully distributed VCS system, that means that no repository is the "main" one from the `git` perspective. You can create a git repository on an EC2 instance. The instance will have an IAM role which will allow writing back to CodeCommit and you can synchronize these two via hooks.
> 0 votes

---
Tags: amazon-web-services, git, ssh
---


# Question
Title: Ansible: How to run local powershell script on remote hosts
I'm trying to write a play that will take a locally stored powershell script from my Ansible server and run it on Windows hosts. I have been using the "win\_shell" module to run scripts placed inside my playbook files, but for testability I want to have the scripts being executed stored outside of my playbook yaml files.

Here's what I'm trying right now:

```
- hosts: "AnsibleDevTestingHosts"
  tasks:
    - name: "showing playbook_dir"
      debug:
        msg: "{{ playbook_dir }}"

    - name: Testing Ansible
      ansible.builtin.script: "{{ playbook_dir }}/scripts/calibrate_flux_capacitor.ps1"

```

When I run this, I'm currently getting back an error message:

```
TASK [Testing Ansible] *********************************************************
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: IndexError: list index out of range
fatal: [delorian-car-1.local]: FAILED! => {"msg": "Unexpected failure during module execution.", "stdout": ""}

```

I've been looking at the Ansible docs for ansible.builtin.script and it seems pretty simple to use (and supports Windows). So, I'm unsure what the problem here is.

# Answer
> 1 votes
This is easy if you reference the module in Ansible correctly.

```
- hosts: "TimeMachines"
  tasks:
    - name: Calibrate Flux Capacitors
      script: "scripts/calibrate_flux_cap.ps1 -some_arg 'some_value'"

```

This works, along with the arg and value. In this example, the arg is consumed by the powershell script using the params() ability, which is a powershell feature.

I think the Ansible docs should be updated with an example that indicates this type of use is possible. Weird that their doc's examples give that full name and that Ansible blows up if you actually use it.

Update: I added an example for the Ansible docs that is currently in review in GitHub, here: https://github.com/ansible/ansible/pull/76980

# Answer
> 1 votes
I've found a half-solution to this. You can load a file in to a Ansible variable and use that. So instead of having an inline script, you could store your local script file in to a variable string and then pass it to a shell module like win\_shell.

Example:

```
- hosts: "DocsTimeMachines"
  tasks:
    - name: Calibrate Flux Capacitors
      win_shell: "{{ lookup('file', 'scripts/calibrate_flux_capacitor.ps1') }}"

```

In the above example, under the directory with my playbook, there's a script "scripts" directory with the referenced script file.

How this kinda sucks still is I want to be able to call the script and use named arguments, for instance, the script call might look like:

```
scripts/calibrate_flux_capacitor.ps1 --mrfusion_enabled=false --hover_systems_enabled=false --retro_1950s_transistor_hack_enabled=true

```

I don't think I can call it like this with my string variable solution here. I'll leave this open for other answers and accept one that can actually call the script, if it comes up.

---
Tags: ansible, powershell
---


# Question
Title: What is happening in an Azure Devops build and release?
I'm finding the Azure devops documentation pretty poor - or maybe I'm just not familiar enough with it for it to make any kind of sense to me. Having said that, I've managed to create a build pipeline that runs tests and published my project, then I created a release pipeline and have successfully released to an Azure App Service.

However, I honestly don't know what is happening behind the scenes...

I have this build pipeline yaml:

```
# ASP.NET Core
# Build and test ASP.NET Core projects targeting .NET Core.
# Add steps that run tests, create a NuGet package, deploy, and more:
# https://docs.microsoft.com/azure/devops/pipelines/languages/dotnet-core

trigger:
- master

pool:
  vmImage: 'Ubuntu-16.04'

variables:
  buildConfiguration: 'Release'

steps:
- script: dotnet restore
  displayName: Restoring Dependencies

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'

- task: DotNetCoreCLI@2
  displayName: Unit Tests
  inputs:
    command: test
    projects: '**/*Tests/*.csproj'
    arguments: '--configuration $(buildConfiguration)'

- task: DotNetCoreCLI@2
  displayName: Publishing
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True

- task: PublishBuildArtifacts@1

```

I'm not sure if I need both the `DotNetCoreCLI@2` and `PublishBuildArtifacts@1` publishing tasks. Are they doing the same thing? Where are the files being published to? Can I see them?

And I've created a Release Pipeline where artifacts points at the above release pipeline:

```
#Your build pipeline references an undefined variable named ‘Parameters.ConnectedServiceName’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab. See https://go.microsoft.com/fwlink/?linkid=865972
#Your build pipeline references an undefined variable named ‘Parameters.WebAppKind’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab. See https://go.microsoft.com/fwlink/?linkid=865972
#Your build pipeline references an undefined variable named ‘Parameters.WebAppName’. Create or edit the build pipeline for this YAML file, define the variable on the Variables tab. See https://go.microsoft.com/fwlink/?linkid=865972

steps:
- task: AzureRmWebAppDeployment@4
  displayName: 'Deploy Azure App Service'
  inputs:
    azureSubscription: '$(Parameters.ConnectedServiceName)'
    appType: '$(Parameters.WebAppKind)'
    WebAppName: '$(Parameters.WebAppName)'

```

How is the release pipeline moving the files to the Azure App Service, and can I look to see what is going to the App Service?

Bit of a noob with devops as you might have guessed

# Answer
Lets take this line by line:

My additions are added as comments.

```
trigger:
- master
# The branch you are building.

pool:
  vmImage: 'Ubuntu-16.04'
# The type of VM your code will run on.

variables:
  buildConfiguration: 'Release'
#This ends up defining which webconfig file to use.

steps:
- script: dotnet restore
  displayName: Restoring Dependencies

- script: dotnet build --configuration $(buildConfiguration)
  displayName: 'dotnet build $(buildConfiguration)'
# Builds your application
- task: DotNetCoreCLI@2
  displayName: Unit Tests
  inputs:
    command: test
    projects: '**/*Tests/*.csproj'
    arguments: '--configuration $(buildConfiguration)'
#Runs unit tests for your application
- task: DotNetCoreCLI@2
  displayName: Publishing
  inputs:
    command: publish
    publishWebProjects: True
    arguments: '--configuration $(BuildConfiguration) --output $(Build.ArtifactStagingDirectory)'
    zipAfterPublish: True
#Pushes your application as an artifact (You'll be able to see it in your list of artifacts in Azure DevOps)
- task: PublishBuildArtifacts@1

```

As for your release script, you are forgetting to pass in the values so your release script isn't doing anything at the moment. But when it does work, it will take the objects from your artifacts or container images and deploy them into your webapp the same way it would do it if you ran those commands from your local machine.
> 1 votes

# Answer
First to answer the difference between DotNetCoreCLI@2 Publish and PublishBuildArtifacts@1

In DotNetCoreCLI@2 Publish task you are only publishing your build output\ files built by your code to $(Build.ArtifactStagingDirectory) which is a predefined variable for the build artifact location. This need not even be $(Build.ArtifactStagingDirectory). Here you are just telling the build pipeline where to place the files built from your code.

In PublishBuildArtifacts@1 task, it lets you publish your $(Build.ArtifactStagingDirectory) or your built binaries (if generated elsewhere) or even your source code to a file share or Azure pipelines. Basically PublishBuildArtifacts@1 is also what defines the content or artifact that gets passed to your release pipleline from your build pipeline.

Hope this answers the question.If not the below documentation from microsoft should.

https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#publish

Publish Build Artifacts
> 0 votes

---
Tags: azure, azure-devops, azure-pipelines
---


# Question
Title: ".format(name, e)) ... google.protobuf.json_format.ParseError: Failed to parse textPayload field: expected string or bytes-like object." GCF logging
I changed the logging in a Google Cloud Function from the built-in Python module `logging` to `google.cloud.logging` and I cannot seem to export exceptions and error messages since they have the wrong type - expected string or bytes-like object.

A shortcut of the error:

> \_ConvertFieldValuePair raise ParseError('Failed to parse {0} field: {1}.'.format(name, e)) google.protobuf.json\_format.ParseError: Failed to parse textPayload field: expected string or bytes-like object.

The full error from the Google Cloud Logs:

> <pre><code>"/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py",
> line 2073, in wsgi_app response = self.full_dispatch_request() File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py",
> line 1518, in full_dispatch_request rv = self.handle_user_exception(e)
> File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py",
> line 1516, in full_dispatch_request rv = self.dispatch_request() File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/flask/app.py",
> line 1502, in dispatch_request return
> self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
> File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/functions_framework/__init__.py",
> line 99, in view_func return function(request._get_current_object())
> File "/workspace/main.py", line 518, in get_sql_and_save_as_csv_in_gcs
> store_duplicated_alids(conn_for_delete_insert, logger) File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/dryable/__init__.py",
> line 34, in _decorated return function( * args, ** kwargs ) File
> "/workspace/main.py", line 368, in store_duplicated_alids
> logger.log_text(e) File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/logging_v2/logger.py",
> line 189, in log_text self._do_log(client, TextEntry, text, **kw) File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/logging_v2/logger.py",
> line 158, in _do_log client.logging_api.write_entries([api_repr]) File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/logging_v2/_gapic.py",
> line 151, in write_entries log_entry_pbs =
> [_log_entry_mapping_to_pb(entry) for entry in entries] File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/logging_v2/_gapic.py",
> line 151, in <listcomp> log_entry_pbs =
> [_log_entry_mapping_to_pb(entry) for entry in entries] File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/cloud/logging_v2/_gapic.py",
> line 542, in _log_entry_mapping_to_pb ParseDict(mapping, entry_pb)
> File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/protobuf/json_format.py",
> line 445, in ParseDict parser.ConvertMessage(js_dict, message) File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/protobuf/json_format.py",
> line 476, in ConvertMessage self._ConvertFieldValuePair(value,
> message) File
> "/layers/google.python.pip/pip/lib/python3.9/site-packages/google/protobuf/json_format.py",
> line 594, in _ConvertFieldValuePair raise ParseError('Failed to parse
> {0} field: {1}.'.format(name, e))
> google.protobuf.json_format.ParseError: Failed to parse textPayload
> field: expected string or bytes-like object.  ```
> </code></pre>

How can I get rid of this error?

# Answer
#### Workaround

You can cast all of your exceptions (and errors) to string.

Example

```
    except Exception as e:
        logger.log_text(str(e))
        raise

```

This is still a strange workaround, I would have expected the module to have a better way for this. I tried StackDriver Logging python exception in protobuf json\_format.py and it did not work. Any other ideas welcome.
> 2 votes

---
Tags: google-cloud-platform, logging
---


# Question
Title: Does Metrics Server kill pods that use too much memory?
I have a pod (in my on premises Kubernetes cluster) that I setup to eventually try to exceed its memory limit. When it did that, my pod started throwing an error saying it was out of memory.

It stayed like that. With its memory maxed out. I can kill it manually, but otherwise it will stay just like that.

However, when I installed Metrics Server and repeat this test, these pods are killed fairly quickly with "out of memory" (once they get near the max memory allowed by the pod).

So to prove that it was Metrics Server causing it, I uninstalled it, and the previous functionality returned (out of memory errors).

Lastly I installed Metrics Server again and.... the out of memory errors continued....

So I am now confused. What would have been killing my containers when they ran out of memory? (I want it to happen, but I need to understand it so I can rely on it.)

**Is it really Metrics Server doing the killing here? And is this configurable?**

# Answer
> 2 votes
I felt a little bit odd by reading your question, because this would break the kubernetes design principles.

So checked the RBAC rules of the metric server provided manifests. There is no role assignment, which would allow the metrics server to delete pods:

```
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: system:metrics-server
rules:
  - apiGroups: [""]
    resources:
      - nodes/metrics
    verbs:
      - get
  - apiGroups: [""]
    resources:
      - pods
      - nodes
    verbs:
      ## ONLY read verbs allowed here!!!
      - get
      - list
      - watch

```

There is another ClusterRoleBinding for the `system:auth-delegator` ClusterRole, but this doesn't allow this eighter:

```
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: system:auth-delegator
rules:
  - verbs:
      - create
    apiGroups:
      - authentication.k8s.io
    resources:
      - tokenreviews
  - verbs:
      - create
    apiGroups:
      - authorization.k8s.io
    resources:
      - subjectaccessreviews

```

This is only to integrate the API Service provided by Metrics-Server.

So there seems to be no direct way for Metrics-Server to kill pods.

---
Tags: kubernetes
---


# Question
Title: When do I have to renew an SSL certificate in Chef?
In the annual process of renewing SSL certificates, I have a list of infrastructure that needs updating.

This usually included Chef. This is followed by an arduous and painful process of getting every node to re-fetch the self-signed chef-server certificate regenerated from the renewed certificate. I always assumed Chef would simply break if I didn't renew my certificate.

This time I looked a bit closer and noticed the self-signed certificate that chef-server actively uses (specifically the nginx component) doesn't expire for another ten years.

Which leaves me wondering:

* Is it unnecessary to update the SSL certificate for chef?
* Will chef break and halt all deploys, chef-client runs, and bootstraps if I let the annual certificate expire?

**What I know:**

* I'm using chef server 13.0.17 on Ubuntu Linux 18.04
* chef uses a self-signed certificate that is distributed to all nodes that are bootstrapped. This is based off of your cert in /etc/ssl/private and generated to /var/opt/opscode/nginx/ca -- this is done by `chef-server-ctl reconfigure`
* `cat <the_crt> | openssl x509 -noout -enddate` reports the expiration date as ~10 years from now
* When the chef server's certificate is updated, every node must fetch the certificate through `knife ssl fetch`. I've never had a good way of automating this
* configuration is explained here, but I couldn't find any information on expiry: https://docs.chef.io/server/server\_security/ and https://docs.chef.io/chef\_client\_security/

**What I don't know**

* Does the functionality of chef actually depend on the expiration date of the cert in /var/opt/opscode/nginx/ca -- making the update to this years certificate unnecessary?
* Do I simply need to regenerate the certificate when the chef self-signed certificate expires? Seems like this would make it an incredibly rare event.

It logically follows that I shouldn't actually have to update the certificate, but I've never encountered something like this and it feels quite odd.

I'm asking here because I want to avoid having all my systems suddenly become inaccessible and fail all deploys. It would also be a huge time saver if I don't actually need to go through the update process every year.

# Answer
> 1 votes
You only need to renew this certificate when it expires unless you configured the clients to ignore ssl verification with `ssl_verify_mode :verify_none` in client.rb.

A `chef-server-ctl reconfigure` should take care of generating a new self-signed certificate if you don't use a specific certificate signed by an internal or external Authority.

> ```
> ssl_verify_mode
>
> ```
>
> Set the verify mode for HTTPS requests. Use `:verify_none` for no validation of SSL certificates. Use `:verify_peer` for validation of all SSL certificates, including the Chef Infra Server connections, S3 connections, and any HTTPS remote\_file resource URLs used in Chef Infra Client runs. This is the recommended setting.
>
> Depending on how OpenSSL is configured, the `ssl_ca_path` may need to be specified. Default value: `:verify_peer`.

Related about certificates I've made this Q/A as well: How do I include my internal CA certificate to validate SSL exchanges in Chef?

# Answer
> 0 votes
It turns out you don't have to update the SSL certificate for Chef every year. My certificate expired recently and my chef infrastructure is still working. It appears the expiration date set is based on that self-signed certificate it distributes to all the nodes.

---
Tags: chef, ssl
---


title: How to backup Amazon Cognito?

text:
# Question
Title: How to backup Amazon Cognito?
Amazon Cognito is a user management and authentication service provided by Amazon through Amazon Web Services:

> Amazon Cognito lets you easily add user sign-up and sign-in to your mobile and web apps. With Amazon Cognito, you also have the options to authenticate users through social identity providers such as Facebook, Twitter, or Amazon, with SAML identity solutions, or by using your own identity system. In addition, Amazon Cognito enables you to save data locally on users devices, allowing your applications to work even when the devices are offline. You can then synchronise data across users devices so that their app experience remains consistent regardless of the device they use.

What you are effectively doing is storing your customer data in a closed system hosted by Amazon. If you are a company, such as a social network, who's data represents a significant asset within your organisation how to you ensure that this data is protected from:

* Amazon Cognito being shut down unexpectedly, i.e. at request from a law enforcement or government agency.
* Your access keys being compromised and some or all records being corrupted.
* The relationship between your organisation and Amazon breaking down to the point Amazon withdraws all services with immediate effect.

Amazon Cognito is a huge benefit to organisations and developers as it takes a particularly hard piece of software and makes it easy to implement; from a service operability perspective, we need to find ways to mitigate, remediate or accept risks to the companies assets at all times.

# Answer
> 12 votes
The Amazon Cognito streams feature can be used to backup data.

Currently, Amazon does not provide a solution to backup their Cognito user Pools. You can use the following NPM package called "**cognito-backup**":

## Install:

```
npm install -g cognito-backup

```

## Usage

```
cognito-backup backup-users <user-pool-id> <options>  Backup all users in a single user pool
cognito-backup backup-all-users <options>  Backup all users in all user pools for this account

```

## Examples

```
cognito-backup backup-users eu-west-1_1_12345
cognito-backup backup-users eu-west-1_1_12345 --region eu-west-1 --file mypool.json
cognito-backup backup-all-users eu-west-1_1_12345 --region eu-west-1 --dir output

```

Source: https://www.npmjs.com/package/cognito-backup

# Answer
> 1 votes
Amazon has released a Cognito User Profiles Export Reference Architecture for exporting/importing users from a user pool. There are limitations:

* Passwords not backed up; users will need to reset
* Pools using MFA are not supported
* Cognito sub attributes will be new, so if the system depends on them, they need to be copied to a custom user attribute
* Federated users also pose challenges wrt sub
* Advanced security - no user history is exported
* No support for pools that allow the option of either phone or email usernames
* No support for tracked devices

# Answer
> 0 votes
To backup Cognito, we use AWS Glue.

# Answer
> 0 votes
As said in other answers, there are two main packages used for this purpose.

None of them seems to have maintainance at this time, but I recommend to use the first one as it has more functionalities implemented. It allows to use the cli and to import the npm package to be used in your own code.

One thing to have in mind is that both of them only allow to export users from a UserPool, but don't allow to export groups. In the case of cognito-backup-restore, there is an issue open and a pull request with changes that implements that functionality, but it is not merged.

---
Tags: amazon-web-services
---


title: Jenkins Build periodically

text:
# Question
Title: Jenkins Build periodically
I currently have a job running from 7AM-2PM every 15 minutes Monday-\>Friday. It is defined so:

```
H/15 7-14 * * 1-5

```

How can I update this to start at 7:30AM instead of 7AM? I still want it to run every 15 minutes and run M-\>F.

# Answer
You will need 2 expressions for this, something like this should work:

30-59/15 7 * * 1-5

\*/15 8-14 * * 1-5
> 2 votes

---
Tags: jenkins
---


# Question
Title: Where is located mysqldump log file on Linux/Ubuntu
I am neither a DB administrator, nor a DevOps but I need to migrate a very large table (~500M rows) from one MySQL schema to another. For doing so, I use `mysqldump`backup tool. Also, since the DB is an AWS DB, I establish an SSH connection to the Ubuntu server from terminal and then run the command from command line.

Everything works as needed and I can see the process being output on my terminal, but once my computer sleeps or I have an internet connection interruption, I am facing a problem: `client_loop: send disconnect: Broken pipe`.

That's probably ok, but after reconnection to the Ubuntu server I don't see that process output on the terminal, nor I can see the corresponding process in the processes list.

I don't know if that can be, but once my terminal looses the connection with the server mysqldump terminates on the server without completion.

In order to be sure, I would like to know:

* if that can happen or no?
* Where mysqldump logs its process so I could take a look at the log file and see what happened?

# Answer
Yes, when your terminal closes the stdout file descriptor for that process closes as well and it does not have where to write.

First thing which would make me nervous is that you do not store the output to a file, so you are NOT doing any backups. You have to store the backup somewhere.

One way how to do it to redirect the output to a file: `mysqldump > backup_file_name.sql` \- you will see not output on the terminal, but you will see the file growing until the backup is ready.

You can also use programs like `screen` or `tmux` for "keeping the term open" for the program. When you connect back, you can re-attach your terminal to that output. But again - do not backup to screen. When you want to see the output while storing it also to a file you can use `tee`: `mysqldump | tee backup_file.sql` \- it will split the output and write it both to terminal and to file.

Writing to terminal is really slow and you should not do it anyway. If you backup an AWS RDS instance, it is already has some automatic backup and you can trigger manual ones via UI or programatically. If you backup mysql running on your instance and you backup it onto the same instance it is not a very good backup after all. You should then copy that file to s3 or somewhere.
> 0 votes

---
Tags: amazon-web-services, linux, mysql, ubuntu
---


title: Terraform apply output only

text:
# Question
Title: Terraform apply output only
I'm trying to apply only the output variable definition. Here is the "output" I want to apply.

```
output "environment_information" {
   value = {
      foo = "value1"
      bar = "value2"
   }
}

```

Altough I added the "bar" as a new variable, I cannot see anything when I applied it by using the following Terraform command:

```
terraform apply -target=output.environment_information

```

Any idea? Is it possible to apply output without any resource/module definition? Thank you.

# Answer
> 24 votes
Running `terraform apply -refresh-only` should take care of any new outputs. It will read the latest data from each resource and then update all of the outputs in terms of those updates, which includes re-evaluating your output expressions to incorporate any changes.

# Answer
> 2 votes
You cannot use resource targeting in this way as an output is not a resource, you can however run a simple apply, as terraform is idempotent that should not change any existing infrastructure but simply put in your output into terraform state.

you can test this with a simple terraform plan. Tested with terraform 1.0. With nothing else in your statefile you will end up with this, after apply:

```
{
  "version": 4,
  "terraform_version": "1.0.0",
  "serial": 1,
  "lineage": "d79eacc7-51eb-2bf4-5ec6-cf8dca57348d",
  "outputs": {
    "environment_information": {
      "value": {
        "bar": "value2",
        "foo": "value1"
      },
      "type": [
        "object",
        {
          "bar": "string",
          "foo": "string"
        }
      ]
    }
  },
  "resources": []
}

```

If you have other non applied/half applied updates you can do one of two things, roll back your code to when they were not there and then remove any associated resources from state (keep a backup) then run apply again, or manually edit the statefile, either way update your hash in the locking db. If its only a couple of outputs i'd go with editing the statefile, just follow the syntax and always always have yourstate versioned and pull a copy just in case.

---
Tags: terraform
---


title: ansible mysql query not showing results

text:
# Question
Title: ansible mysql query not showing results
I have used community.mysql collection and was able to run a query against the database.

```
---
- name: Get stats of a file
  hosts: localhost
  become: true

  tasks:
    - name: run mysql query
      community.mysql.mysql_query:
        login_user: testuser
        login_password: testpass
        login_db: test_db
        query: SELECT * FROM STATION WHERE ID = %(id_val)s AND STATE = %(story_val)s
        named_args:
          id_val: 12
          story_val: AZ
      register: r_query

```

The playbook runs fine without errors

```
root@ip-172-31-83-231:/home/ubuntu# ansible-playbook sql_query.yml
[WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all'

PLAY [Get stats of a file] ***************************************************************************************************************************************************************

TASK [Gathering Facts] *******************************************************************************************************************************************************************
ok: [localhost]

TASK [run mysql query] *******************************************************************************************************************************************************************
ok: [localhost]

PLAY RECAP *******************************************************************************************************************************************************************************
localhost                  : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored

```

could not find the documentation on how to display the queries results. Can you help me in the right direction.

# Answer
> 1 votes
The `community.mysql.mysql_query` module documents its return values as:

* `executed_queries`
* `query_results`
* `rowcount`

So, if you're registering `r_query`, and you want to see the result, you can call `r_query.query_result`.

A common approach when developing playbooks initially is to send to `stdout` with a `debug` task:

```
- name: Get stats of a file
  hosts: localhost
  become: true

  tasks:
    - name: run mysql query
      community.mysql.mysql_query:
        login_user: testuser
        login_password: testpass
        login_db: test_db
        query: SELECT * FROM STATION WHERE ID = %(id_val)s AND STATE = %(story_val)s
        named_args:
          id_val: 12
          story_val: AZ
      register: r_query

    - name: what happened
      ansible.builtin.debug:
        msg: "{{ r_query.query_result }}"

```

---
Tags: ansible, mysql, ansible-galaxy
---


# Question
Title: When do I scale services on Azure DevOps?
I have created azure devops with a few organizations/projects in it. Do I need to scale or replicate it across different datacenters so all the developers across the globe can access it? When would I need to scale or replicate a service? Any references appreciated.

# Answer
Azure DevOps is already replicated across multiple Azure Data Centers. They did this after the south east data center hosting Azure Devops (then VSTS) went down for a few days, bringing down the service. By going with a cloud provider, this is one of the benefits you (assume) you get. Automatic (geo)replication/redundancy of SAAS products. You don't have to manage this, but if something goes wrong, you have no control (or responsibility) to fix it.
> 1 votes

# Answer
Manual intervention is not required for your Azure Devops organisation to be accessible to different developers accross the globe.

The on-premises version of Azure Devops can be scaled with multi-server deployments. However it's common to only deploy these servers to a single datacenter.
> 0 votes

# Answer
> Do I need to scale or replicate it across different datacenters so all the developers across the globe can access it?

* No, you do not need to scale or replicate your data or projects to be accessable globally; however, some countries may have laws in place that prevent access to your project that reside outside of their region.
* The notion of renting someone else's infrastructure is that typically the infrastructure provider offers some degree of replicating your organization's data. Meaning replication of your data generally isn't needed.
* Applications/projects having failures or going down can happen and Microsoft's Azure documentation has suggestions on managing backups and disaster recovery. They also have services on Site Recovery in general.

> When would I need to scale or replicate a service?

* When you want to minimize the effects of any single service having down time or being in a none recoverable state.
* When Availability Zones becomes something needed in respects to the size of your projects. An example is needing to scale and replicate your organizations/projects across multiple data centers to ensure high availability is maintained.
> 0 votes

---
Tags: azure-devops, azure, autoscaling, high-availability
---


# Question
Title: Is it possible to make the minikube VM use the VPN configuration from the host machine?
I am currently writing a script that deploys a number of applications to a local minikube cluster for development purposes. This script is working great on Linux, but then I tried it on Mac OSX.

When starting minikube on Mac OSX instead of using the host systems kernel, it spins up a VM that runs the minikube cluster instead.

An important part of this setup is that all the docker images we deploy to the minikube cluster are hosted in a private docker repository that is hosted on a private network. We use VPN to access the image repo when needed, which solves the problem nicely when running on Linux, since the container shares the host machines network config and is forced through the VPN tunnel.

When running the script on Mac OSX though, the minikube cluster is unable to reach the private docker repository since it is now run in a VM that is not being forced through the VPN tunnel. Thus resulting in the cluster being unable to pull the images.

Is there any good way to make minikube on Mac OSX use the host machines VPN configuration as well? I have been searching for a good solution to this, but am having problems finding a solution that works for me.

Best would be if the solution did not depend on any specific VPN software, but instead solved the problem on an OS-level (I hope you understand what I mean here) so that the developers may use any VPN client software they want.

I appreciate any help. I just want this script to run the same on all machines.

# Answer
In my case this problem was when using OSX, so this answer may not work on other operating systems.

Adding the flag `--hyperkit-vpnkit-sock=auto` on `minikube start` solved the problem for me. This limits you to using the hyperkit driver. If that tradeoff is worth it or not is up to you.

My complete `minikube start` command is as follows:

`minikube start --hyperkit-vpnkit-sock=auto --vm=true --driver=hyperkit`
> 1 votes

# Answer
An easy somewhat hacky solution is to disable split tunneling on your VPN, which will force all traffic from your host to go through the VPN itself.

How to do it depends on your specific client.

In some cases, you may need to ask the VPN's administrator (i.e. your company's IT department) to enable this option on the VPN appliance itself.
> 0 votes

---
Tags: docker, docker-registry, minikube, macos, vpn
---


# Question
Title: Terraform errors openstack_compute_floatingip_v2: Resource not found
When I run terraform I get,

```
openstack_compute_floatingip_v2.sandbox: Creating...
╷
│ Error: Error creating openstack_compute_floatingip_v2: Resource not found
│
│   with openstack_compute_floatingip_v2.sandbox,
│   on main.tf line 31, in resource "openstack_compute_floatingip_v2" "sandbox":
│   31: resource "openstack_compute_floatingip_v2" "sandbox" {

```

But it does seem that there is a `openstack_compute_floatingip_v2` resource. This seems to be the line triggering the error,

```
resource "openstack_compute_floatingip_v2" "sandbox" {
        pool = "public"
}

```

# Answer
All I had to do to resolve this was migrate to using `openstack_networking_floatingip_v2` instead of the deprecated `openstack_compute_floatingip_v2`. This requires an explicit association with `openstack_compute_floatingip_associate_v2`.

```
resource "openstack_networking_floatingip_v2" "myip" {
        pool = "pool_name"
}

resource "openstack_compute_floatingip_associate_v2" "bridge" {
        floating_ip = "${openstack_networking_floatingip_v2.myip.address}"
        instance_id = "${openstack_compute_instance_v2.test-server.id}"
}

```
> 1 votes

---
Tags: terraform
---


# Question
Title: Error: Error creating openstack_compute_floatingip_associate_v2
When I try to create a `openstack_compute_floatingip_associate_v2` in terraform I'm getting the following error,

> Error: Error creating openstack\_compute\_floatingip\_associate\_v2: Bad request with: \[POST https://nova-cc.hou-01.cloud.prod.acme.net:8774/v2.1/servers/REDACTED\_UUID/action\], error message: {"badRequest": {"code": 400, "message": "Unable to associate floating IP 10.2.66.173 to fixed IP 10.2.65.167 for instance REDACTED\_UUID. Error: External network REDACTED\_UUID is not reachable from subnet REDACTED\_UUID. Therefore, cannot associate Port REDACTED\_UUID with a Floating IP.\nNeutron server returns request\_ids: \['req-a35d140e-3061-4d2d-a88b-7dfb634340c7'\]"}}

# Answer
For me, this ended up being because my system administrator disabled Floating IPs. I was getting the same error in OpenStack Horizons (the web interface).
> 0 votes

---
Tags: terraform, openstack
---


# Question
Title: Where does openstack_networking_port_v2 show up in OpenStack Horizons (web interface)?
I'm creating an instance with Terraform and I'm using the Terraform resource `openstack_networking_port_v2`. I can see this corresponds to a `openstack.network.v2.port` OpenStack resource.

I'm using this,

```
data "openstack_networking_network_v2" "network" {
        name = "public"
}

data "openstack_networking_subnet_v2" "subnet" {
        name = "hou-prod-external-ipv4"
}

resource "openstack_networking_port_v2" "myport" {
  name           = "port"
  network_id     = data.openstack_networking_network_v2.network.id
  admin_state_up = "true"
  fixed_ip {
      subnet_id  = data.openstack_networking_subnet_v2.subnet.id
  }
}

```

Creating this instance works in Terraform, but when I log into to Horizons I don't see this reflected in the UI anywhere. Does the `openstack_networking_port_v2` show up anywhere in Horizons?

# Answer
To get the ports set up with `openstack_networking_port_v2` in Horizons you'll want to,

* On the Navigation bar on the left, click `Network` to open the drop down.
* Under the drop down click on `Networks`
* Click on the `openstack_networking_network_v2` in the case above `public`
* Click on the `openstack_networking_port_v2` in the case above `myport`
* You can now verify it's right by checking the ID against the ID recorded by your teraform state with something like this,

```
terraform show -json |
  jq '.values.root_module.resources[] | select(.type=="openstack_networking_port_v2") | .values.id'

```
> 1 votes

---
Tags: openstack
---


# Question
Title: Equivalent of dependsOn in azure-pipelines?
In azure-pipelines one can use dependsOn to indicate what jobs or deployments should be completed before another job will be started.

At the moment, the dependsOn array of the final job has to be added everytime if some job is added.

The question is whether it is possible to define a sort of `before` expression to ensure that this only has to be defined once in a azure-pipeline template.

# Answer
> 2 votes
No, there's no such thing as a 'before' setting or anything like it.

There are only two ways of configuring job dependencies:

1. don't define dependsOn at all: this is an easy way to let the pipeline run the jobs one after another, but doesn't let you use advanced features
2. define dependsOn for each job: this is more complicated (as you've found out) but allows fine-grained control and extra features such as reusing output variables

---
Tags: azure-pipelines
---


# Question
Title: How can I find the INSTANCE_ID from a provisioned machine?
How can I find the Instance ID referred to by Cloud-Init as provided by OpenStack?

# Answer
> 0 votes
You can run,

```
cloud-init query instance_id

```

Or, you can find the `INSTANCE_ID`, by looking in this file,

```
/var/lib/cloud/data/instance-id

```

Or, this one

```
/run/cloud-init/.instance-id

```

---
Tags: openstack
---


# Question
Title: How to access a container/service port in swarm without publishing the port to the outside world?
It is possible to docker-compose to forward a port to the container from `localhost` only, by writing the desired `ports` configure in `docker-compose.yml`, e.g. `127.0.0.1:8080:80`.

In docker swarm this does not work and the published port is visible from the internet.

How then can I access the container port only from localhost without opening the port to the internet?

I tried configuring iptables to deny connection to port `iptables -A INPUT -p tcp --dport 5432 -j DROP`, but because of the settings from docker for iptables, that didn't work.

# Answer
The container can be accessed via the ip which belongs to the `docker_gwbridge` interface.

I was able to find out the ip address of the container on the `docker_gwbridge` interface by running the `ifconfig` command in the container itself:

```
$ docker exec <container-id> ifconfig

```

This way the container can be accessed from `host machine` for example by `172.18.0.2:5432`, without having to publish port `5432`.

---

Trying to find out the ip address via `docker inspect <container-id> (or <service-name> or <network-id>)` in swarm services did not lead to the desired result, because the ip from the `docker_gwbridge` interface was not displayed there. And the "internal" ip (something like 10.0.4.6) was not available from `host machine`.
> 2 votes

---
Tags: docker-compose, networking, docker-swarm
---


# Question
Title: Can I see the values set in `ml2_conf.ini` as a regular OpenStack user (non-admin)?
My company uses a managed instance of OpenStack. I am **NOT** an administrator on this instance. There is functionality that OpenStack offers behind the `enable-ml2-dns` configuration options, this is set in the `ml2_conf.ini`.

> In order to enable internal DNS resolution, the user must set the `enable-ml2-dns` to True. The default value is False in order to provide backwards compatibility with existing deployments.

Is it possible to find out this setting without being an administrator?

# Answer
> 1 votes
I don't think this is possible. The \*.ini files are usually only configured by the cloud operators.

---
Tags: networking, openstack
---


# Question
Title: Does an FQDN on OpenStack mean there must be a resolver for it?
When I look at different resources on OpenStack, I see a FQDN's like this,

* `host-10-2-67-96.openstacklocal.`
* `host-2620-0-28a4-4140-f816-3eff-fefd-6381.openstacklocal.`

I see this for ports and for NOVA instance. Does this mean there is a resource in OpenStack that can resolve this? If there is nothing that can resolve this, what does the FQDN's even do? What purpose do they serve?

I've tried under Network Topology → Networks → (only option) → Subnet Tab → ipv4, which lists two different DNS servers (`10.0.18.100` and `10.0.18.200`) but neither of them actually resolve these FQDNs. I've tried to `dig <fqdn> @ip` where `ip` is each of the above, and I've tried setting my `resolve.conf` to,

```
domain openstacklocal
search openstacklocal
nameserver 10.0.18.100
nameserver 10.0.18.200

```

It's like those two IPs are not getting the entries for the FQDN.

---

For more information I can see the subnet that my hosts are connected on, but it seems I can find more information about the associated project.

```
(openstack) subnet show 2fa15774-4724-4796-9a84-813f7a4079d3
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| Field                | Value                                                                                |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| allocation_pools     | 10.2.64.11-10.2.95.254                                                                                |
| cidr                 | 10.2.64.0/19                                                                                |
| created_at           | 2021-02-25T16:39:29Z                                                                                |
| description          |                                                                                |
| dns_nameservers      | 10.0.18.100, 10.0.18.200                                                                                |
| dns_publish_fixed_ip | None                                                                                |
| enable_dhcp          | True                                                                                |
| gateway_ip           | 10.2.64.1                                                                                |
| host_routes          |                                                                                |
| id                   | 2fa15774-4724-4796-9a84-813f7a4079d3                                                                                |
| ip_version           | 4                                                                                |
| ipv6_address_mode    | None                                                                                |
| ipv6_ra_mode         | None                                                                                |
| location             | cloud='', project.domain_id=, project.domain_name=, project.id='0fd8601348b7412da9347a19d49fc4be', project.name=, region_name='RegionOne', zone= |
| name                 | hou-prod-external-ipv4                                                                                |
| network_id           | 990d1b13-88b2-453e-975c-d757b0ed99fb                                                                                |
| prefix_length        | None                                                                                |
| project_id           | 0fd8601348b7412da9347a19d49fc4be                                                                                |
| revision_number      | 0                                                                                |
| segment_id           | None                                                                                |
| service_types        |                                                                                |
| subnetpool_id        | None                                                                                |
| tags                 |                                                                                |
| updated_at           | 2021-02-25T16:39:29Z                                                                                |
+----------------------+--------------------------------------------------------------------------------------------------------------------------------------------------+

```

But when I try to get more information on it, I get

```
(openstack) project show 0fd8601348b7412da9347a19d49fc4be
You are not authorized to find project with the name '0fd8601348b7412da9347a19d49fc4be'.

```

So it seems that default subnet for my nova instances is on a subnet which is associated with a project that I can not find more information on. Moreover, that subnet has for DNS servers "10.0.18.100, 10.0.18.200" and it seems neither of them can resolve the hostnames for the machines on that subnet. Why is this?

# Answer
> 0 votes
Usually, depending on how the cloud was configured, Openstack should let you resolve internal DNS names (inside the openstacklocal domain).

This means you will need :

1. A VM inside a private internal network.
2. A private network configured to use the internal dnsmasq agents as DHCP (and not any external servers)

It's hard to say in your case but you might be using two different networks. Your DNS servers seem to be in a different subnet than the IP that is assigned to your server.

---
Tags: dns, openstack
---


# Question
Title: Is there a way for a user to audit their permission levels in OpenStack?
In a comment on the site someone has asked me,

> Can you give more info on what kind of permissions \[on OpenStack\] are we talking about?

I want to answer that question, but I don't know how. Given credentials to Horizon (OpenStack Dashboard) is there a method to tell what my user can and can't do? When I go to the `project/api_access/` I can see only the endpoints I have access to, but not the level of access on each end-point?

# Answer
Openstack permissions are not super fun to play around. There are a default set that limits what member/user can do.

Cloud operators are free to customize roles for each Openstack component. Only the operator can tell you if there are custom roles.
> 1 votes

# Answer
IIRC tenants usually don't have privileges to see actual policy definitions, but depending on your configuration you should be able to see some info with `openstack quota`

Reference doc
> 1 votes

---
Tags: openstack
---


# Question
Title: What is the supported method for installing Kubernetes on OpenStack?
If I want to install Kubernetes on OpenStack, is there a supported method to do this?

# Answer
> 1 votes
Yes, there is a supported method of doing this. You can install Magnum (which is itself an OpenStack project). This will allow you to install any Container Orchestration Engine (COE), including Swarm, Kubernetes, Mesos, and DC/OS. It does this by provisioning an infrastructure cluster of raw OpenStack resources like Nova (the OpenStack compute resource) per-tenant, and installing the COE on the top of it giving you access to the cluster through `kubectl`.

## Links

# Answer
> 1 votes
Just to be clear, Magnum requires cloud operators to install it first on the Openstack first. Then, users are able to provision kubernetes clusters.

If you are looking to run your own k8s clusters inside VMs, there is Rancher Openstack operator

---
Tags: kubernetes, docker-swarm, openstack
---


# Question
Title: Random delay in port binding for distributed_virtual_router and router_centralized_snat in openstack neutron
I have created a private network called "Private\_Network" in the range of 192.168.220.0/24 plus a virtual router called "Virtual\_Router" inside openstack which is connected to the external network. Then I connect the default gateway of "Private\_Network" i.e. 192.168.220.1 to the "Virtual\_Router" so that I all VMs connected to "Private\_Network" can access the Internet via SNAT.

I used vxlan as overlay network, and a flat provider network.

By connecting "Private\_Network" to "Virtual\_Router", two ports are immediately created: **router\_interface\_distributed** with the IP address of 192.168.220.1 and **router\_centralized\_snat** with IP address of 192.168.220.45, however, both ports are in the DOWN state for a bit **long and random time** like 2 hours, or 45 minutes, or 20 minutes. I should mention, that rarely the ports get UP as soon as (less than a minute) I connect the "Private\_Network" to "Virtual\_Router".

I have searched a lot to find out the main reason behind this issue. I am not convinced that server configuration is wrong because I have a few cases in which the two ports gets up right after I connect the "Private\_Network" to "Virtual\_Router". So, I tried to look at the log files and noticed there are three main phases to pass in order to get each of those ports to UP state: DHCP, port binding, and L2 provisioning. I changed the log level to DEBUG and investigate the log files in details.

I run the following process several times:

1. create a brand new project in Horizon.
2. create a brand new virtual network (called "Private\_Network") in the range of 192.168.220.x/24.
3. create a brand new virtual router (called "Virtual\_Router") connected to external network.
4. connect port 192.168.220.1 (default gateway) of "Private\_Network" to "Virtual\_Router".
5. cat /var/log/neutron/\* | grep snat\_port

From more than ten cases I experimented, neutron either stuck at the "port binding" phase or "L2 provisioning" phase. When It stuck at "port binding" it takes random time to finish, like 45 minutes, 20 minutes or 10 minutes and Once "port binding" phase is done, "L2 provisioning" phase will be done in less than a minute and port state changes to "UP". However, In case that "L2 provisioning" phase get stuck, the previous two phases finished in less that 1 minutes, but "L2 provisioning" get stuck for hours. Its so confusing to me why do I see this much delay in getting ports "UP".

I would appreciate if anybody can assist me to resolve this issue.

# Answer
Usually, issues with port spawning will require admin access/logs to troubleshoot fully.

It's harder with DVR, but you might have to trace each port creation and see where the delay happens.

In Openstack, events are transiting through rabbitmq to provide the end to end orchestration. If one part of the process takes time, the overall process will be delayed.
> 0 votes

---
Tags: networking, openstack
---


# Question
Title: Missing Red Hat agent specification in Agent Specification drop down in building Azure Pipeline
In Azure DevOps, I want to use an agent running Red Hat.
How do I do that ? It does not list in the Agent Specification drop down:

This link shows how to add a Self-Hosted Linux Redhat agent. Is this the way to go?

Not everyone in my team seems to agree on the definition of what an Agent is in Azure DevOps. My understanding is that Agents in Azure DevOps are images of VMs that can be deployed as you wish, and that Azure DevOps provides a group of them out-of-the-box. The ubuntu agent I see in the drop down is likely part of this group of built-in agent images Azure DevOps offers.

Others in my team say this is not what we want – because what this does relies in a Red Hat physical host and cannot be done in the New Build.

Bottom line, what is the way to go here in order to have Redhat agent just like the ubuntu that is shown in the drop down by default ?

# Answer
> 1 votes
The Azure Pipelines pool uses Microsoft "Hosted" agents. So as you guessed rightly these are VMs with bundled software (SDKs, CLIs, etc).

If you want to use your own self-hosted, you need to create a new agent pool. Go to your project settings \> pipelines \> agent pools and click the "Add pool" button.

You'll see the option to select the pool type of "Self-hosted".

Once you've created the pool, you can select it from the Agent pool and begin to add your own self-hosted agents by clicking the "New agent" button. There are instructions provided as part of the set up wizard.

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: What do you call variables like "%(ec2_region)" in my cloud-init?
What do you call these variables in my `cloud-init` file,

```
- http://%(ec2_region)s.ec2.ports.ubuntu.com/ubuntu-ports/
- http://%(availability_zone)s.clouds.ports.ubuntu.com/ubuntu-ports/
- http://%(region)s.clouds.ports.ubuntu.com/ubuntu-ports/

```

You can see here variables like `%(region)` are these available to all things in `cloud-init`? Where are these documented?

# Answer
> 1 votes
This is called instance metadata. Cloud providers expose metadata to cloud instances which cloud-init then stores and can be used in a few different contexts, such as jinja-templated cloud-init userdata.

Cloud-init instance metadata is documented here: https://cloudinit.readthedocs.io/en/latest/topics/instancedata.html

---
Tags: cloud-init
---


# Question
Title: How does a new network and subnet have FQDNs without DNS Nameservers?
This is an extension and a simplification of my prior problem. I have created a

* Network
* Subnet
* A port attached to the above network and subnet
* A compute instance on that port.

With the following Terraform script,

```
provider "openstack" {
  tenant_name = "admin"
  cloud       = "openstack"
}

resource "openstack_compute_instance_v2" "test-server" {
  name        = "test-server"
  flavor_name = "m1.medium"
  key_pair    = "cp-2021"
  # Ubuntu
  image_name = "ubuntu-20.04"
  user_data  = data.template_file.user_data.rendered
  network { port = openstack_networking_port_v2.port_project.id }
}

## Here we're trying to create a project association
resource "openstack_networking_network_v2" "net_project" {
  name           = "net_project"
  admin_state_up = "true"
}

resource "openstack_networking_subnet_v2" "subnet_project" {
  name       = "subnet_project"
  network_id = openstack_networking_network_v2.net_project.id
  cidr       = "192.168.199.0/24"
  ip_version = 4
}

resource "openstack_networking_port_v2" "port_project" {
  name           = "port_project"
  network_id     = openstack_networking_network_v2.net_project.id
  admin_state_up = "true"
  fixed_ip {
    subnet_id = openstack_networking_subnet_v2.subnet_project.id
  }
}

```

In Horizon's under `project/networks/ports/384f70d3-360b-478f-9c39-23dc804a3b5f/detail`, this subnet does not show a DNS server,

```
Name: port_project
ID: 384f70d3-360b-478f-9c39-23dc804a3b5f
Network Name: net_project
Network ID: 33091875-5a4b-40e0-9e79-efac82cbd324
Project ID: ce9d50b729aa4a7a95c21e534105725c
MAC Address: fa:16:3e:e0:74:d7
Status: Active
Admin State: UP
Port Security Enabled: True
DNS Name: None  <- **NAMESERVER FIELD SHOWS NONE**

```

But it still shows an Assignment (same page)

```
**DNS Assignment**
Hostname host-192-168-199-158
IP Address 192.168.199.158
FQDN host-192-168-199-158.openstacklocal.

```

What is that DNS Assignment with? What can resolve this FQDN?

# Answer
This is something that can really only be answered by your Cloud administrators.

1. The openstacklocal domain is a premade domain that Openstack uses for internal DNS names.
2. It does not mean that it will be usable out of the box.
3. To test, you can try to ping 192.168.199.2-3-4 from your VM to see if the dhcp agent responds. You can then try to query the dhcp agent with `dig`

Now, there seem to be deeper DNS support with ml2.

3. The ML2 driver will need to include dns Link to doc
4. You will need to use the DHCP agent IP as DNS servers. Usually, it's the first two/three IP of your subnet.
5. It will create DNS names for your subnet/ports.
> 2 votes

---
Tags: terraform, openstack
---


# Question
Title: Install a RPM from build context without copying it first
During dockerbuild, I'd like to install a (rather big) RPM from build context/host without copying it first into the image in order to keep my image small. Ideally, there would be a way to read a file directly from build context.

What are my options (beside using a HTTP server on the host)? What are the best practices for this use-case?

# Answer
> 3 votes
Using buildkit, you can mount from the build context with `type=bind` to access files without including them in your resulting image.

```
# syntax=docker/dockerfile:1.3
FROM base
RUN --mount=type=bind,source=./rpms/,target=/rpms/ \
    yum install /rpms/package.rpm

```

See the buildkit docs for more details.

---
Tags: docker, dockerfile, images
---


# Question
Title: How to know if a certain container has a software/package installed or not?
I use Gradient notebooks, they grab the container from a URL and files from a Github repo and combine them to make Notebook instance.

Something like this one.

However, the container image from this URL (nvcr.io/nvidia/tensorflow:21.09-tf2-py3), which I use to learn Tensorflow does not have sklearn and matplotlib.

I need to install sklearn and matplotlib every single time using pip. I thought why not grab an images with all basic packages and Tensorflow. But on Docker's website (https://hub.docker.com/search?q=tensorflow&type=image&image\_filter=store), how do I check the softwares inside the images. I know if I search Tensorflow I get images of Tensorflow, but how do I check if they have matplotlib and sklearn.

I know I can make my own container, but I just want to save my time with verified publishers.

# Answer
You can see the packages in a container by exec to it: docker exec pip list
> 0 votes

---
Tags: docker
---


title: How to caching the docker images?

text:
# Question
Title: How to caching the docker images?
I try to improve the performance of my pipeline stages in gitlab cicd. The Gitlab is self hosted and the Runner runs as docker with docker executor. I recognized, that for both jobs in my stages (test, build) they begin with pulling the images from docker hub:

```
Running with gitlab-runner 14.8.0 (565b6c0b)
  on Shared gitlab-runner on ARWS. 3yyScM1M
Preparing the "docker" executor 00:03
Using Docker executor with image python:3.9 ...
Pulling docker image python:3.9 ...
Using docker image sha256:4819be0df94257e1e31cd64dda12d46ff8b2180f9576ad9eaf98dcac9d70d9f9 for python:3.9 with digest python@sha256:57274f3fe86ad3d010025e9019bc41dcf9ee5833cfe4ba7b6773ef49672d855f ...
Preparing environment
.
.
.

```

This part seems to take most of the pipeline time. At the moment not much is happening in both jobs. I already started to implement caching in the pipelinbe:

```
image: python:3.9

stages:
  - test
  - build

variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"

cache:
  key:
    files:
      - poetry.lock
      - .gitlab-ci.yml
    prefix: ${CI_JOB_NAME}
  paths:
    - .venv
    - .cache/pip

before_script:
  - python --version  # For debugging
  - pip install poetry
  - poetry config virtualenvs.in-project true
  - poetry install
  - source `poetry env info --path`/bin/activate

testing:
  tags:
    - docker
    - linux
  stage: test
  script:
    - echo "Launching Job 'testing' in stage 'test'"
    - pytest -v --junitxml=report.xml
  artifacts:
    when: always
    reports:
      junit: /builds/$CI_PROJECT_PATH/report.xml

building:
  tags:
    - docker
    - linux
  stage: build
  script:
    - echo "Launching Job 'building' in stage 'build'"

```

And there is a huge possibility I do some stupid stuff. But the real question bothers me: Is it possible to cache the environment instead of pulling and preparing it at the start of every job?

# Answer
GitLab has some docs on how to speed up builds, and one of the suggestions does involve caching runner images.

https://docs.gitlab.com/runner/configuration/speed\_up\_job\_execution.html#use-a-proxy-for-containers

## Use a proxy for containers

You can speed up the time it takes to download Docker images by using:

* The GitLab Dependency Proxy or
* A mirror of the DockerHub Registry
* Other open source solutions

There are other non-container related suggestions too, as well as information on how to use these up at that link.

You can also host your own runners, in which case you can run them on docker or just centos, etc. I assume, in that case, that you could already have the images available on the runner node to avoid new pulls. We run ours in kubernetes and are looking at harbor as an option to speed these things up too, but we haven't tested it yet. I imagine self-hosted k8s and harbor would be similar to GitLab using its own dependency proxy solution.
> 0 votes

---
Tags: docker, gitlab, gitlab-ci-runner, cache
---


# Question
Title: How do you enable the DNS extension to ml2 in OpenStack?
In one doc it says,

> **To enable the internal DNS resolution**, the neutron-server service must be configured with the dns\_domain config option set to a value other than openstacklocal in neutron.conf file **and the dns extension driver must be enabled in the `ml2_conf.ini` file. In order to enable internal DNS resolution, the user must set the `enable-ml2-dns` to `True`.** The default value is `False` in order to provide backwards compatibility with existing deployments.

In another doc it says this

> Add dns to extension\_drivers in the `[ml2]` section of `/etc/neutron/plugins/ml2/ml2_conf.ini`. The following is an example:
>
> ```
> [ml2]
> extension_drivers = port_security,dns
>
> ```

Which method is the correct method for enabling the ml2 dns extension? Are we setting the `enable-ml2-dns` to true, or adding a `dns` driver in the `extension_drivers` list?

# Answer
The first link charm \- is for an Openstack deployed with Charm.

The second doc is correct. The file at `/etc/neutron/plugins/ml2/ml2_conf.ini` should be changed on the controllers. But the exact location will depend on how Openstack was deployed.
> 1 votes

---
Tags: networking, dns, openstack
---


# Question
Title: How can I find out what ml2 extension drivers are enabled?
According to this document on *"The Networking service internal DNS resolution"*, a feature I need requires the `dns` extension driver in OpenStack,

```
[ml2]
extension_drivers = port_security,dns

```

How can I find what extension drivers my OpenStack's ml2 has?

# Answer
I was able to test this in my lab.

* Deployed with kolla ansible.
* Currently running Ussuri

If I add the `dns` driver to the ml2 config, I am now able to resolve network ports DNS through the `dnsmasq` inside my private network.

```
(venv) laurentdumont@openstack-deployer:/etc/kolla$ openstack port show 046068ee-4b1a-47b5-bfdb-91af916de1ee -c dns_assignment
+----------------+------------------------------------------------------------------------------------------+
| Field          | Value                                                                                |
+----------------+------------------------------------------------------------------------------------------+
| dns_assignment | fqdn='test-dns.os.coldnorthadmin.com.', hostname='test-dns', ip_address='10.100.100.112' |
+----------------+------------------------------------------------------------------------------------------+

```

In this case, the VM has one port and the port takes the name of the VM.

```
# nslookup -type=A test-dns.os.coldnorthadmin.com 10.100.100.2
Server:     10.100.100.2
Address:    10.100.100.2:53

Name:   test-dns.os.coldnorthadmin.com
Address: 10.100.100.112

```
> 1 votes

---
Tags: networking, dns, openstack
---


# Question
Title: What does "extension list" show in the OpenStack cli?
Are these extensions shown with OpenStack's client `openstack extension list` extensions that are *available* or extensions that are *enabled*? I ask, because there is a notion of an extension that is not enabled. And I'm wondering if this is showing that the extension exists and is enabled, or that it *can* be enabled by an administrator?

Also somethings mention `ml2` extension drivers, are these drivers listed in the output of `extension list`?

# Answer
> 1 votes
Yes, the `extension` command will list the plugins that are enabled

```
openstack extension list | grep dns

```

```
| FloatingIpDns                                                                                | os-floating-ip-dns                    | Floating IP DNS support.                                                                                |
| DNS Integration                                                                                | dns-integration                       | Provides integration with DNS.                                                                                |

```

---
Tags: openstack
---


# Question
Title: How can I get more information about an OpenStack extension?
There is *something* (still not exactly sure because of the documentation) that allows DNS internal resolution in OpenStack through an extension. I'm wondering if any of these do that deed (from the output of `extension list`)

* `os-floating-ip-dns` \- Floating IP DNS support.
* `dns-integration` \- Provides integration with DNS.
* `dns-domain-ports` \- Allows the DNS domain to be specified for a network port.

These are all very interesting to me. Is there anyway to find more information about what these do than the 4 or 5 vague words their described with with `extension list`?

# Answer
From what I can see

* os-floating-ip-dns - Floating IP DNS support. This seems to be limited to integrating floating IP DNS with external DNS services. Not a DNS inside Openstack.
* dns or dns\_domain\_ports : Allows you to add a FQDN to a port in Openstack. Then, if it's an internal network, the dnsmasq instance (that acts as DHCP and DNS) will be able to resolve the zone and the associated hostname.

`dns_domain_ports` seems to be the new plugin name for Queens and up. Change in config

There doesn't seem to be any configuration for `dns-integration`. It's either `dns` or `dns_domain_ports`
> 1 votes

---
Tags: openstack
---


title: set loadbalancing to true round-robin

text:
# Question
Title: set loadbalancing to true round-robin
What is the default behaviour when a kubernetes service is called over and over from the same clientip? I ran some tests and it seems that always the same pod is called and after a while it switches and then that pod gets called every time. Is that correct?

How can we setup kubernetes that every time a service is called it switches to the next pod?

# Answer
Got it working by using ip virtual server.

It's a service/module which hands out ip's of the next pod to schedule/call

it's likely installed already.

```
lsmod | grep ip_vs
ip_vs_sh               16384  0
ip_vs_wrr              16384  0
ip_vs_rr               16384  19

```

Have your cni properly setup and edit kube-proxy's ConfigMap

```
kubectl edit cm -n kube-system kube-proxy

```

edit the ipvs section

set mode to ipvs

```
mode: "ipvs"

```

and the ipvs section

```
ipvs:
      excludeCIDRs: null
      minSyncPeriod: 0s
      scheduler: "rr"

```

As always there are lots of variables biting each other with k8s, but it is possible with ipvs.

https://kubernetes.io/blog/2018/07/09/ipvs-based-in-cluster-load-balancing-deep-dive/
> 0 votes

---
Tags: kubernetes
---


title: Terraform single-file modules

text:
# Question
Title: Terraform single-file modules
I've been reading up the Terraform standard module structure and I see they want a main, outputs, and variables file in every one.

I have a pair of modules where everything fits in a single tmux window, is it really worth breaking out into three files for such a small module or is this a rule that is really only applied at scale?

# Answer
> 3 votes
While it's best practice to use `variables.tf` and `outputs.tf`, both variables and outputs can be declared in any file - so in a way it just boils down to both personal preference along with any formatting rules that may exist in your project or team.

Personally, I prefer to always use these files as it helps me keep things tidy and organized.

---
Tags: terraform, terraform-module
---


# Question
Title: Why is my GCP managed Cert not working with kubernetes
I have the following that seems to work...

```
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test
  annotations:
    kubernetes.io/ingress.global-static-ip-name: thanos-static
  labels:
    app: test-app
spec:
  defaultBackend:
    service:
      name: test-app
      port:
        number: 8000

```

I am trying to add ssl so I change to the following...

```
apiVersion: networking.gke.io/v1
kind: ManagedCertificate
metadata:
  name: managed-cert
  namespace: test
spec:
  domains:
    - me.mine.co
---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: test-ingress
  namespace: test
  annotations:
    kubernetes.io/ingress.global-static-ip-name: thanos-static
    networking.gke.io/managed-certificates: managed-cert
    kubernetes.io/ingress.class: "gce"
  labels:
    app: test-app
spec:
  defaultBackend:
    service:
      name: test-app
      port:
        number: 8000

```

I set everything up on the DNS side and can access `http` but when I try to hit `https` I get...

```
Secure Connection Failed
An error occurred during a connection to me.mine.co. PR_END_OF_FILE_ERROR

```

What am I missing?

# Answer
> 2 votes
Worked after a while. It apparently needed time to propagate.

---
Tags: google-cloud-platform, gcp, google-kubernetes-engine, cert-manager
---


title: How to create an Ubuntu image for 16.04

text:
# Question
Title: How to create an Ubuntu image for 16.04
I am just slowly getting into Docker. I want to create an Ubuntu 16.04 image. I know that docker run -it ubuntu creates an ubuntu image but when I do that, I get 20.04

# Answer
I wouldn't suggest making a 16.04 image since that was EOL as of April 2021. You can search the docker hub for "ubuntu 16.04" and find several instances of it

```
$ docker pull ubuntu:16.04

Trying to pull repository docker.io/library/ubuntu ...
16.04: Pulling from docker.io/library/ubuntu
58690f9b18fc: Pull complete
b51569e7c507: Pull complete
da8ef40b9eca: Pull complete
fb15d46c38dc: Pull complete
Digest: sha256:0f71fa8d4d2d4292c3c617fda2b36f6dabe5c8b6e34c3dc5b0d17d4e704bd39c
Status: Downloaded newer image for docker.io/ubuntu:16.04

```

and then run it with:

```
$ docker run -it --rm ubuntu:16.04 /bin/bash
root@b912d2c0db04:/# cat /etc/os-release
NAME="Ubuntu"
VERSION="16.04.7 LTS (Xenial Xerus)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 16.04.7 LTS"
VERSION_ID="16.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"
VERSION_CODENAME=xenial
UBUNTU_CODENAME=xenial

```

and you can use `FROM ubuntu:16.04` in your docker files to reference it.

The `16.04` after the colon in the name identifies which version you're pulling down, if you don't specify it, there's some convention that defaults to latest, which would be 20.04, until sometime in April 2022.
> 4 votes

---
Tags: docker, ubuntu
---


# Question
Title: Terraform wanting to redeploy azure vms for 'no' reason
I got a VM defined in terraform like this:

```
resource "azurerm_linux_virtual_machine" "vm" {
  name                            = var.vm-name
  resource_group_name             = var.RG
  location                        = var.location
  size                            = var.vm-size
  admin_username                  = data.azurerm_key_vault_secret.username.value
  disable_password_authentication = true
  proximity_placement_group_id    = data.azurerm_proximity_placement_group.proximity-group.id

  tags = var.tags
  network_interface_ids = [
    azurerm_network_interface.nic.id,
  ]

  admin_ssh_key {
    username   = data.azurerm_key_vault_secret.username.value
    public_key = data.azurerm_key_vault_secret.pub.value
  }

  os_disk {
    caching              = "ReadWrite"
    storage_account_type = "Premium_ZRS"
  }

  source_image_reference {
    publisher = "Canonical"
    offer     = "0001-com-ubuntu-server-focal"
    sku       = "20_04-lts-gen2"
    version   = "latest"
  }
}

```

Every time I run tf apply it thinks that the admin\_username has been changed and wants to redeploy the machine. This is happening to several environments I'm working with so logically I'm doing something wrong but I can't figure out what..

# Answer
So, in short, there is a bug/feature in terraform, and the posted solution was to output the keyvault\_id from the keyvault module, then declaring the output value as a variable to pass into other modules. So when getting the secrets now, I use the output value as key\_vault\_id in the data-block for the secrets like this:

```
data "azurerm_key_vault_secret" "username" {
  name         = var.username
  key_vault_id = var.keyvault_id
}

```

instead of this:

```
data "azurerm_key_vault_secret" "username" {
  name         = var.username
  key_vault_id = data.azurerm_key_vault.kv.id
}

```
> 1 votes

---
Tags: terraform, azure, linux, vm
---


# Question
Title: Can we run github actions inside a docker container?
Is it possible to run github actions inside docker container like jenkins?

# Answer
> 0 votes
There are two aspects:

1. can github actions run containers?
2. Can github actions run container *like jenkins*

To answer the former, by default Github actions do run in containers. From the docs:

> Your workflow contains one or more jobs which can run in sequential order or in parallel. Each job will run inside its own virtual machine runner, **or inside a container**, and has one or more steps that either run a script that you define or run an action, which is a reusable extension that can simplify your workflow.

(emphasis my own)

There is also a dedicated page for creating container-based runners

For the latter part of the question, it's not quite clear what you mean by "like Jenkins", but the Github Actions guide should provide the relevant context.

---
Tags: docker, github-actions
---


title: What is a golden image?

text:
# Question
Title: What is a golden image?
Is there a defacto understanding of what a golden image is? I've had it come up in requirements to satisfy cloud audits. What's the meaning of "Golden" when it comes to audit and compliance, especially for cloud services?

It seems like the word "Golden" has different meanings when it comes to different platforms, for instance you can mark a "Golden Snapshot" in UCS Director.

# Answer
> 3 votes
As noted in the comments there are several applications for the term. This answer focuses on the term as it relates to a compliance audit.

In the context of an audit, a `golden image` refers to an original image that is used when creating servers in your environment. Based on the reference below:

> The term itself comes from the old practice of recordable CDs being manufactured with gold film. Hence the gold colored CD actually being the source

The NIST Risk Management Framework (800-53) refer to `golden images` as `baseline configurations`. In 800-53 CM-2:

> Baseline configurations serve as a basis for future builds, releases, or changes to systems and include security and privacy control implementations, operational procedures, information about system components, network topology, and logical placement of components in the system architecture. Maintaining baseline configurations requires creating new baselines as organizational systems change over time. Baseline configurations of systems reflect the current enterprise architecture.

An example that an auditor would be looking for is a hardened image of a specific OS version that is used to build out other servers in your environment. It demonstrates that your base-build a) has been reviewed, b) has not been altered, and that c) its use is repeatable.

**References**

What is the Genesis of the Term Golden Master
Golden Image
NIST 800-53 CM-2

---
Tags: terminology, virtualization, images
---


# Question
Title: Making blue green deployment work for dependent ECS services
I have two ECS services. UI(`http://<LB>/SS_UI/`) and API(`http://<LB>/webservice/`). UI calls API and it's config file will have references to the API e.g.

```
public static springURL = "http://<LB>/webservice/"

```

The documentation for blue green deployment for ECS(https://cicd-pipeline-cdk-eks-bluegreen.workshop.aws/en/ecsbg/serviceupd.html) suggests creating two listener on the load balancer and two target groups

```
<LB>:80 -> Original set with live traffic
<LB>:8080 -> Replacement set for testing

```

This works if I want to test changes to individual projects. I commit to API. Old version of API is on port 80. New version is on port 8080. After testing `http://<LB>:8080/webservice/` the switchover can happen after pre-defined time(pic)

But it doesn't work if I want to test from the UI because UI references the old version of API on port 80

If I commit changes to the UI project by referencing `http://<LB>:8080/webservice/`, even though I can test from UI at `http://<LB>:8080/SS_UI/`, I cannot allow the replacement set for UI to become the original set because API on port 8080 will eventually get deleted. How can I ensure that as soon as replacement set for API at port 8080 moves to port 80, the UI also starts referencing API of port 80?

# Answer
I don't think CodeDeploy is designed to include these type of use cases (also because the ramifications and options of allowing that flexibility are pretty much infinite). CodeDeploy is designed to manage B/G deployments for a single micro-service and not for a complex app like you are describing (sure your app isn't very complex but where do you stop supporting complex configurations?). In the context of CodeDeploy these would be two separate B/G deployments (and you'd need to test the back end service via API).

The only alternative I can think of to make this work is to create an additional (hidden/secured?) path in your front-end (e.g. `/SS_UI-8080` or perhaps passing a parameter to the `/SS_UI` path?) that allows you to modify (in your code) the port to use to connect to the back end service via the LB.
> 0 votes

---
Tags: amazon-web-services, amazon-ecs, bluegreen
---