# Question
Title: Package installed using Dockerfile not available on Jenkins pipeline stage
I'm having sort of a weird problem with my Jenkins pipeline. I've defined a Dockerfile which installs some tools I need for my CI/CD pipeline, such as node, npm, sfdx, and sfpowerkit (a plugin for sfdx). This is the Dockerfile:

```
FROM node:14.14-alpine

#add usefull tools
RUN apk add --update --no-cache  \
  git \
  findutils \
  bash \
  unzip \
  curl \
  wget \
  nodejs-npm \
  openjdk8-jre \
  openssh-client \
  perl \
  jq

# install Salesforce CLI from npm
RUN npm install sfdx-cli --global
RUN sfdx --version

# install sfpowerkit - https://github.com/Accenture/sfpowerkit
RUN echo y | sfdx plugins:install sfpowerkit
RUN sfdx plugins

```

My Jenkinsfile is pretty simple as well:

```
pipeline {
    agent {
        dockerfile true
    }

    stages {
        stage("build") {
            steps {
                echo "build step"
                sh 'node --version'
                sh 'npm --version'
                sh "sfdx --version"
                sh "sfdx plugins --core"
                sh "sfdx plugins"
                sh "sfdx commands"
                sh 'sfdx sfpowerkit --help' <-- this command fails
            }
        }
    }

}

```

The problem is that even though I install the plugin sfpowerkit and it shows on the job output that it was successfully installed, the plugin is not available inside the build stage.

This is during the Dockerfile image construction:

And this is during the build stage on the pipeline:

I can't find out what the problem is, and why the plugin is not available after constructing the container through the Dockerfile.

Would anyone be kind to help?

Thank you!!

---

**EDIT 4 Sep:**

This is the job output: https://justpaste.it/4jnjn

# Answer
> 0 votes
I was able to solve this issue by using the following docker image instead of the Dockerfile:

**"dxatscale/sfpowerscripts"**

Here is an example:

```
pipeline {
    agent {
        docker { image 'dxatscale/sfpowerscripts' }
    }

    stages {
        stage("build") {
            steps {
                echo "build step"
                sh 'node --version'
                sh 'npm --version'
                sh "sfdx --version"
                sh "sfdx plugins --core"
                sh "sfdx plugins"
                sh "sfdx commands"
                sh 'sfdx sfpowerkit --help'
            }
        }
    }

}

```

# Answer
> 0 votes
I have same issue, adding `mkdir .local && chmod g+rwx .local` before instalation helped me.

```
RUN mkdir .local && chmod g+rwx .local \
&& echo 'y' | sfdx plugins:install sfpowerkit

```

or

install it whithin jenkins 'steps'. it might not be very clever or wise but it should work.

---
Tags: docker, jenkins, jenkins-pipeline, dockerfile, jenkinsfile
---


# Question
Title: How can I access additional services in my container?
A decision was made by my company to containerise the entirety of a solution with multiple services into a single monolithic container (best practices aside please as I am well aware this goes against the microservice approach of 1 container = 1 service). In essence the following ports should in theory be listening for traffic:

* Port 3000 (static React frontend)
* Port 8081 (backend services)
* Port 8545 (additional service) \[can deal with this later as I feel illumination of the problem with the Port 8081 service will lead to a similar resolution for this one\]

When I host this container on an Azure Web App for Containers instance the frontend service comes up on it's own, no problem, (route is `/`). However, the backend service is mapped to `/services/*` (actually `:8081/services/*` as above) and contains a config file the frontend needs to be truly "functional". However, this appears to be inaccessible as the web app attempts to proxy to `localhost:8081/services/*` in order to get it (as it is localhost WITHIN the container itself).

What it is the best approach to allow the web app to call out to the additional service(s) on these different ports?

I initially thought I could place an Azure Application Gateway in front of the web app to act as a reverse proxy and do a hostname rewrite to rewrite requests on Port 8081 to (FQDN/IP):8081, but this doesn't seem to be working correctly (no difference in error codes or visually). I also thought about placing a reverse proxy like Nginx WITHIN the container, but was interested in seeing if I could do it outside first to stop increasing the size of this already large image or if this would even be an option.

# Answer
Azure Web App for Containers expose one port from your container. Azure Web App for Containers detect port 3000 first (3000\<8081).

Revers proxy in container will resolve your problem. Test outside of container will not work.

For test you may be change port 3000 to 8082. Backend /services/\* will available, but frontend won't.
> 1 votes

---
Tags: docker, azure
---


# Question
Title: What is a standard way to quickly deploy a LAMP installation with a virtual host and certbot?
Say I rent an IaaS environment and want to quickly deploy a LAMP installation with a virtual host and SSL certificate by Bash alone (no programs such as Ansible would be used).

Is there some standard way to do so?

# Answer
It depends on the details. What do you understand on "LAMP installation", it is a specific set on configs of apache/mysql/php, or it is a custom system image?

Considering that it is an IaaS setting, logic dictates you have a LAMP system (as a VM image or a compressed fs) and you want to deploy it into some VM container. To my best knowledge, there is not even an usual way to do that. All it depends on your VM provider. Very likely, you need to get some command line interface from your provider, and you need to deploy things to it.

Another possible interpretation of your question, that you have an actually running virtual machine, and you want to deploy your LAMP configuration to that. There, things depend on the distro, however in the current devops world, typically containers are deployed.
> 0 votes

---
Tags: apache-httpd
---


# Question
Title: Work item rules in Azure DevOps board: Parent STATUS (custom field) dependent on child STATUS
I'm working in Azure DevOps and have some organisational restrictions so instead of using "State" we will use a custom field "Status" with 6 options:

* On Track
* Delayed
* Pending
* Done
* Removed
* Blocked

The rule we want is: Parent status is restricted and dependent on child Status. If any of the children of a parent item have the Status as Blocked, the parent item is changed to Blocked. Same with Delayed. The Status of the parent cannot be changed from Blocked or Delayed until all children are in one of the other states. If all children are in any other state, the parent changes to on track and can be manually changed.

Help!

# Answer
I think the simplest way to do this would be to utilise a webhook on your Azure DevOps Boards and have it updated via the DevOps API. With a simple serverless instance like Azure Functions you should be able to do this quickly and painlessly.
> 0 votes

---
Tags: azure-devops, automation
---


# Question
Title: Is there a safe way to archive Azure App Services application settings?
I successfully deployed a fastAPI onto Azure App Services.

However, the environmental variables that I specified into the **Configurations** \> **App Settings** pane got unexpectedly deleted, thus preventing my app from running correctly and I had to manually recreate the application settings from scratch.

I could save these settings onto my local computer but I am wondering if there is a better way to archive these environment variables.

# Answer
> 1 votes
Ideally, you would be injecting these via an `appsettings.json`, Key Vault reference or something and not manually setting these in the Configuration panel itself. In terms of archiving the actual values etc, this is probably most easily accomplished via the App Service API. There are several CRUD operations that can GET, UPDATE, etc configurations/application settings which should be able to allow you to accomplish what you have asked here.

---
Tags: azure, environment-variables
---


title: Gathering timespan statistics from Git

text:
# Question
Title: Gathering timespan statistics from Git
I would like to easily query git to answer a question like the following:

How much time passed from when a developer made a git commit to when that commit was merged to a default branch from a feature branch (if that pattern is being used) or was tagged with a particular tag (for trunk-based development)?

At a higher level, my goal is to see how much time it takes for code to leave a developer's fingers to actually reach production.

Thanks!

# Answer
git-time-metric is a set of tools that allow to track time on Git.

The latest release can be installed automatically using brew or manually by installing the executables listed here.

gtm must first be initialized (`gtm init`) from with the git project directory to allow time tracking. When you commit your work, gtm then saves the time associated with your commit.

The time can be displayed using the `gtm report` command; `format` argument `timeline-hours` allows to create a Gantt chart.

On a side note, I am not sure whether `gtm` can help with *existing* projects. I tried running it onto one of my repos and it logged `0s` for all my previous commits.
> 2 votes

---
Tags: git
---


# Question
Title: Is there a way to use a Jenkinsfile from a git-submodule in a multibranch pipeline?
**Background:** The nature of our project's source involves multiple long-living branches, each one has its own Jenkins job (manually created).

**Multibranch limitation:** Unfortunately we do not use a Multibranch Pipeline to organize the project, because when (for example) a change is needed in the Pipeline, it would need to be added (manually) to each of the long-living branches' Jenkinsfiles.

**Current implementation:** We store the Jenkinsfile in a separate repository (let's call it *ci-repo*), which is added as a submodule to the project's repository (let's call that *dev-repo*).

**What still hurts:** We need to manually monitor branches of the *dev-repo* and manage job creation. Multibranch pipelines support submodule checkout for source files, but branch discovery requires presence of a Jenkinsfile under the parent repository (in our case *dev-repo*).

**The question:** Jenkinsfile from git-submodule of Multibranch Pipeline - Is there a way to do this? Perhaps using an initial/mock Jenkinsfile in the *dev-repo*, with one stage in which is calls the Jenkinsfile from the submodule? Can a Jenkinsfile be over-written or replaced somehow during runtime?

# Answer
I agree with Omri that you are heading the wrong way. I would recommend using a shared library which implements the pipeline. You would then have a simple Jenkins file which looks all the same. See https://jenkins.io/blog/2017/10/02/pipeline-templates-with-shared-libraries/

Let's say you have a shared library which contains a `vars/commonpipeline.groovy` which contains something like this:

```
def call(body) {
    // evaluate the body block, and collect configuration into the object
    pipelineParams= [:]
    body.resolveStrategy = Closure.DELEGATE_FIRST
    body.delegate = pipelineParams
    body()

    pipeline {
        agent {
            label pipelineParams.nodes
        }
        options {
            buildDiscarder(logRotator(artifactNumToKeepStr: '1', numToKeepStr: '10'))
            disableConcurrentBuilds()
            timestamps()
            timeout(time: pipelineParams.timeout, unit: 'MINUTES')
        }

        stages {
            stage('Build') {
                steps {
                    //TO SOMETHING
                }
            }
        }
    }
}

```

Then you can have a `Jenkinsfile` in your dev project which loads the library and uses the common pipeline like this:

```
import org.apache.commons.io.FileUtils
library identifier: 'shared-pipeline-library@master', retriever: modernSCM(
  [$class: 'GitSCMSource',
       remote: 'https://bitbucket/jenkins/hared-pipeline-library.git',
       credentialsId: 'bitbucket.service.user'
  ])

commonpipeline {
    nodes     = 'BUILD' /* label of jenkins nodes*/
    timeout   = '60'
}

```
> 3 votes

# Answer
There may not be a way to make your idea work with submodules. When a Jenkins multibranch pipeline polls SCM for changes, it has no visibility into submodules. I verified this recently. The result in "Scan Multibranch Pipeline Log" was a `Communication error for url` response. The Jenkinsfile was not found, and so no build was triggered.

The good news is that I discovered a straightforward way to achieve what we're after *without* using submodules. Have a look at the Remote Jenkinsfile Provider Plugin.

When a pipeline is polling a repository to trigger new builds, it looks for two things for each branch:

1. The polled branch has new SCM changes since the last build (HEAD is different).
2. A Jenkinsfile exists in repository X on branch Y at path Z.

In a multibranch pipeline's Build Configuration section, normally only one Mode is available to tell Jenkins where to find the Jenkinsfile: "by Jenkinsfile". This mode allows us to specify Z (path) only; X and Y are *implicitly* set to the repository and branch being polled.

What the Remote Jenkinsfile Provider Plugin gives us is a new Build Configuration Mode: "by Remote Jenkins File Plugin". This mode allows us to *explicitly* set the repository, branch, and path for the Jenkinsfile - X, Y, *and* Z. In your example, X would be `ci-repo`, but *any* Jenkinsfile in *any* repository is fair game. You could have `ci-repo` be a submodule of `dev-repo` if it's convenient, but no such relationship is required to make this work.

If desired, a "Match branches" option uses the *polled branch name* for Y, looking for a *remote* Jenkinsfile on a branch of the same name. For example, while polling `dev-repo/feature-123`, it would look for a Jenkinsfile on `ci-repo/feature-123`.

As long as the Jenkinsfile exists at the specified location, the second criteria for triggering new builds will be satisfied. You'll see something like this in the pipeline's polling logs:

```
Looking up DR/dev-repo for branches
Checking branch master from DR/dev-repo
No local file defined. Skipping Source Code SCM probe, since Jenkinsfile will be provided by Remote Jenkins File Plugin
    Met criteria
Scheduled build for branch: master

```

Instructions for how to set this up, with screenshots, can be found on the plugin's page.

**Caveat:** Pushing changes to a *remote* Jenkinsfile (`ci-repo`) will *not* trigger any builds of the polling repository (`dev-repo`). If this behavior is desired, one solution would be to use webhooks. Another would be to create a pipeline that polls the *remote* Jenkinsfile's repository and builds other pipeline jobs as desired using the build step, like so:

```
build(job: "dev-repo/${BRANCH_NAME}", wait: false)

```
> 2 votes

# Answer
I believe you are heading in the wrong direction. The way to implement what you want is to separate the "devops infrastructure logic" (placed in the 'ci-repo') from the pipeline trigger - a jenkinsfile inside the dev-repo, which includes the "devops infrastructure logic" as a tagged artifact.

Pros:

* You have a central place to develop (and test) devops code
* When changes are backward compatible, you can update all pipelines by moving the tag
* When changes are NOT backward compatible, repos & specific branches can gradually move to a new tag
* There is room for specific repos & branches to create their unique process if they have to

Cons:

* Yes, sometimes people will need to update Jenkinsfiles in multiple places as they are accepting a new version. I actually think it's good.
> 1 votes

# Answer
We have the exact same problem, and wanted to run a multibranch pipeline using a Jenkinsfile located in a git submodule. Our setup is a bit different because the submodule contains also libraries that are shared amongst several projects. So this is more than a *ci-repo*, but the principle stay the same. We prefer this approach because it allows linking easily the version of the project to the version of the pipeline that was used at release (instead of manually pointing to some revision of a shared library).

Our solution is actually a direct application of https://stackoverflow.com/questions/37800195/how-do-you-load-a-groovy-file-and-execute-it.

Let's say we have following project tree:

```
build/
    Jenkinsfile
src/
    ...
common/           <-- a git submodule
    src/
        ...

```

Our Jenkinsfile is something like

```
    // file: build/Jenkinsfile

    def doThings()
    {
        // ...
    }

    try {
        node {
            // ...
        }
        // ...
    }
    catch(e) {
        // ...
    }
    // ...

```

So basically some utility functions and some pipeline stages (here in a try-catch for instance).

Two actions are required to move this file to our `common` submodule:

* Move the file to submodule and adapt it slightly.
* Replace the original jenkinsfile with a small boilerplate version.

Our moved Jenkinsfile becomes

```
    // file: common/build/Jenkinsfile

    def doThings()
    {
        // ...
    }

    def call() {
        try {
            // ...
        }
        // ...
    }
    return this

```

So basically we surround the pipeline in a call function, and don't forget to return `this` at the end of the file.

The original Jenkinsfile is then replaced with this small boilerplate code as explained in the original post:

```
// file: build/Jenkinsfile

def myPipeline
node {
    checkout scm // mandatory
    myPipeline = load "common/build/Jenkinsfile"
}
myPipeline()

```

Of course this can be tuned further to pass parameters, use several jenkinsfile, etc. For instance reusing ideas from the documentation:

```
def call(Map params)
{
    println "Building for project " + params.projectName
    // ...
}

```

```
// ...
myPipeline(projectName: "MyProject")

```

Also note that this checkout the whole project, which might be inefficient. A better approach would be to checkout only the submodule that contains the Jenkinsfile.
> 0 votes

---
Tags: jenkins-pipeline, git, jenkinsfile
---


# Question
Title: how to automate helm deployments in github actions
I've got a private github actions runner with a repo that contains a helm chart for logstash

```
runner@runnerdeploy-elkstack-6pkp9-45kpw:
/runner/_work/elastic-logstash-kibana/elastic-logstash-kibana/elkhelmconfig/logstash$ ls
    Chart.yaml  templates  values.yaml

```

The values.yaml is in elkhelmconfig/logstash

so manually I would go to the elkhelmconfig directory and run

```
helm install logstash logstash/

```

but how to do it from within the private runner automatically? There is no k8s config and the helm action fails on aws-cli not installed.

Can it be done scripted by curl https://\***helm && chmod a+x helm && helm upgrade logstash

or something?

# Answer
> 2 votes
Got it working with this helm github action

https://github.com/marketplace/actions/helm-3

add secret to github as your .kube/config

and this workflow step:

```
steps:
      - name: 'check it out'
        uses: actions/checkout@v3
      - name: helm-deploy
        uses: WyriHaximus/github-action-helm3@v2.0
        with:
          exec: helm upgrade logstash /github/workspace/elk/logstash/ --install --wait --atomic --namespace=default --set=app.name=logstash --values=/github/workspace/elk/logstash/values.yaml
          kubeconfig: '${{ secrets.KUBECONFIG }}'

```

---
Tags: kubernetes, github, helm
---


# Question
Title: Running this groovy jenkins script output this error "WorkflowScript: 17: expecting '}', found ':' @ line 17, column 11"
```
#!groovy

env.BEARER_TOKEN = "uuuuuuuuuu"

node {
    publishModule()
}

def publishModule() {
    stage('Deploy Modules') {
  }
}
def buildPayload() {
    def payload =
{
    "data": {
        "attributes": {
          "vcs-repo": {
            "identifier":"nnnnnn",
            "oauth-token-id":"hhhhhh",
            "display_identifier":"nnnnnn"
          }
          "no-code": true
        },
        "type":"registry-modules"
     }
 }
}

def deployModules() {
    def payload = buildPayload()
    def response = httpRequest(
        customHeaders: [
                [ name: "Authorization", value: "Bearer " + env.BEARER_TOKEN],
                [ name: "Authorization", value: "application/vnd.api+json" ]
            ],
        httpMode: 'POST',
        responseBody: "${payload}",
        url: "https://app.terraform.io/api/v2/organizations/my-organization/registry-modules/vcs"

    )
}

```

```
WorkflowScript: 17: expecting '}', found ':' @ line 17, column 11.
       "data": {
             ^

```

I am trying to build this code in a Jenkinsfile so I am getting this error. **WorkflowScript: 17: expecting '}', found ':' @ line 17, column 11.** Any suggestion will be appreciated.

# Answer
> 0 votes
According to the Groovy language docs, when defining a map literal, the keys should not be surrounded by double quotes. I.e.:

```
def payload = {
  data: {
    attributes: {
      some_key: "some_value"
    }
  }
}

```

---
Tags: jenkins, jenkins-pipeline, groovy
---


# Question
Title: Missing some subscriptions in Azure DevOps UI when using automatic service principal
I am trying to select a subscription I have access to in another tenant from my Azure DevOps UI where I am connected to the Azure tenant AAD as a member with external login and certain permissions/roles.

In this case I want to select a subscription that I have created a resource group and an app service in so that I can create the deployment pipeline using a pre-configured template in Azure DevOps.

Previously, my account on Azure DevOps was user1@company.com and the account in Azure portal was user1@company.onmicrosoft.com as it was a different AAD. I have since added user1@company.com to the AAD of the Azure portal where the subscription resides and given it some permissions to access these subscriptions. MFA is set up on both accounts.

The really frustrating thing about this is that I did get it working temporarily last night and could both select the subscription in AzureDevOps and login when prompted with the user1@company.com account but today it seems to have reverted back to be missing the subscriptions from the additional tenant.

It is also an issue when I try to set up a new service connection but assuming it depends on same permissions in place.

Thank you for any help you can provide.

# Answer
> 2 votes
Is there a particular reason you can't just use the manual SP approach? I have also had issues in the past using the `automatic` flow as well, so I usually just add in my SP creds and get on with it rather than hope all my default subscriptions have been exposed for each tenant etc.

---
Tags: azure-devops, azure, azure-pipelines, active-directory
---


# Question
Title: How to upgrade nodes in a kubernetes cluster?
For keeping the nodes of a kubernetes cluster (e.g. AWS EKS) up to date with kernel bugfixes etc, **how should the existing nodes be replaced** once an updated image becomes available (i.e. once the ASGs have been reconfigured so new nodes spawn with a more recent AMI)?

For example, is there a command to cordon off *all* existing (or old) nodes from pod scheduling? Is there a benefit to performing rolling restarts of deployments etc, before otherwise attempting to drain the nodes? Will fresh nodes automatically spin up in proportion to pod scheduling demand, even when other remaining nodes (cordoned or partway drained) are nearly idle/unoccupied? Or is it better to disengage the autoscaler and perform manual scaling? How soon after a node is drained would the instance be automatically terminated? Will manually deleting a node (in kubernetes) cause the aws cluster autoscaler to terminate that instance *immediately*? Or should termination be done with the aws cli? Will persistent data be lost if a node is deleted (or terminated) before it fully drained? Also, can some pods be granted exceptions from getting evicted (e.g. stateful long-running interactive user sessions, such as jupyterhub), while still ensuring their host node does get refreshed *as soon as* those pods finish? And if so, can this be overridden (when there is an urgent security patch)?

# Answer
Given how kubernetes is intended for high availability online services, I'm surprised the topic isn't more clearly consolidated in the kubernetes documentation (or more prominent in both the CKA and CKS curriculum). I did find docs from Amazon EKS and Google GKE, covering different strategies (and technicalities) for node pool upgrades. There are also a number of related ongoing kubernetes feature proposals, to more intelligently move pods.

OS patches do not strictly necessitate a new AMI (machine image), but kernel and glibc patches *do* require a restart (disrupting all pods anyway) and a fresh AMI ensures consistency between nodes.

To **drain** a node has the effect of *cordoning* it off (that is, tainting it with an annotation to prevent scheduling of new pods) and (gracefully) evicting its current pods. Note, it does not affect static/mirror pods (which are outside of API control) and *daemonset* pods (since their controller ignores the taint). The `kubectl drain` command blocks until complete (requiring parallel invocations to drain multiple nodes), and requires a `--force` argument if the node hosts any bare pods (pods with no controller to replace them).

**Pod disruption budgets** are objects created to specify how many of an application's pods should be kept available during voluntary disruptions. For a long-running stateful application, a restrictive PDB can be used to prohibit certain pods from being evicted (which prevents drain from succeeding).

The **cluster autoscaler** deployment is responsible for instructing ASGs (AWS EC2 autoscaling groups) to launch additional (suitable) instances if there exist (pending) pods with no place (node) to be scheduled to. It also drains and terminates instances, but only after several conditions are all satisfied for an extended period (10min): the node has low utilisation (less than half its capacity requested by its hosted pods); there is suitable space available on other nodes (accounting for affinities, etc); the autoscaler would dare to evict the pods (no inhibiting disruption budgets or annotations, no `kube-system` pods other than static or daemonset or with explicit disruption budgets, no bare pods, no local storage); and the cluster did not need more nodes. (Note cluster autoscaling is intentionally not concerned with actual CPU load. Also differs from default ASG behaviour by not preferentially terminating the instances with oldest launch templates.)

You could manually remove an instance with the command `aws autoscaling terminate-instance-in-auto-scaling-group --instance-id <value> --should-decrement-desired-capacity`. (Note, may have to use --*no*-should-.. if the ASG is already at its MinSize.) The kubelet should attempt (using systemd) to detect whenever an instance is shutting down, and to delay it by a grace period so that remaining pods have an opportunity to shut down cleanly.

The **node controller** is responsible for creating node objects which represent the currently available instances, and deleting those nodes when the corresponding instance is terminated. Note that deletions in kubernetes generally cascade. There *may* be cases where node deletion could discard data that a pod had claimed from a dynamically provisioned "persistent" volume. Obviously any data stored locally on the node will be jettisoned.

When a set of replica pods is scaled down (such as for deployments using *horizontal pod autoscalers*) the **replicaset controller** preferentially deletes a pod that has been *ready* for a shorter time (or none), as well as trying to leave the pods more evenly spread between nodes. This means that load fluctuations tend to only relinquish the youngest nodes, and will not organically migrate pods from cordoned older nodes to newer nodes unless the older pods were manually annotated with negative *deletion costs*.

During a **rollout restart**, the deployment controller maintains pod availability within a tolerance (by default ±25%, rounded upward). Thus, for deployments with less than 4 replicas, a rollout restart will ready a healthy replacement pod *before* it deletes each original pod. This surge mechanism avoids disruption to the application (whereas draining relies on rectification by other controllers after a disruption). Rollout is an alternative approach that could be used to relocate pods from cordoned nodes onto other (new) nodes.
> 2 votes

---
Tags: kubernetes, amazon-web-services
---


# Question
Title: Can I define a CodePipeline with Terraform that deploys my Terraform resources?
I'm trying to figure out the best way to automate the deployment of infrastructure resources defined in Terraform. Ideally, I'd like to deploy all my code — including resource definitions — in a CI/CD manner.

So, if I define an AWS CodePipeline that reads my Terraform code from GitHub, can I have that CodePipeline...deploy itself (+ any other AWS resources defined in the repo)?

**Update:**

I built this and tried to push a change through the pipeline that updated the CodeBuild image. It seems like the CodeBuild step to run Terraform succeeded in updating the pipeline, but then the "Build" stage that did this ended up in a "Cancelled" state with the message: `Pipeline definition was updated`.

So it seems like it sort of worked, but I may have to kick off a manual release of the most recent change after that. Any downsides/risks?

# Answer
Step 1: Sign into AWS Console

Step 2: Navigate to CodePipeline service

Step 3: Click on the “Create Pipeline” button

Step 4: Enter the Pipeline name and click on the “Next” button

Step 5: Select “GitHub” as Source provider and enter the following values

Repository: https://github.com/cloudacademy/cloud-academy-terraform-pipeline.git

Branch: master

Click on the “Next” button

Step 6: Select the following Build provider

AWS CodeBuild

Project name:

Click on the “Next” button

Step 7: Select the following Deploy provider

AWS CodeDeploy

Deployment provider: S3

Click on the “Next” button

Step 8: Review the values and click on the “Create Pipeline” button

Step 9: Select the following Build provider

AWS CodeBuild

Project name:

Click on the “Next” button

Step 10: Select the following Deploy provider

AWS CodeDeploy

Deployment provider: S3

Click on the “Next” button

Step 11: Review the values and click on the “Create Pipeline” button

Step 12: Select the following Build provider

AWS CodeBuild

Project name:

Click on the “Next” button

Step 13: Select the following Deploy provider

AWS CodeDeploy

Deployment provider: S3

Click on the “Next” button

Step 14: Review the values and click on the “Create Pipeline” button

Step 15: Select the following Build provider

AWS CodeBuild

Project name:

Click on the “Next” button

Step 16: Select the following Deploy provider

AWS CodeDeploy

Deployment provider: S3

Click on the “Next” button

Step 17: Review the values and click on the “Create Pipeline” button

Step 18: Review
> 0 votes

---
Tags: terraform, github, aws-codepipeline
---


# Question
Title: Trunk Based Development Deployment Pipeline
We are currently working on transitioning to Trunk Based Development and starting to look at our deployment pipeline and how we can improve.

**Our current workflow:**

1. All engineers work on the trunk, frequently committing, which is automatically deployed to our dev environment
2. When QA sign-off on dev, we generate a release by cutting a release branch (releases/v1.0) from trunk which then will get deployed to our UAT environment. This is a manual approval step. Once approved, the release branch is generated dynamically and pushed to the repo, and also deployed.
3. Test team (QE) then essentially performs E2E testing in UAT and they request no new code is merged apart from only cherry-picks for P1 defects identified. We cherry pick from trunk. Devs also continue working on trunk fixing any other P2/P3 defects or adding new features.
4. Once QE sign-off on UAT, that snapshot can then be pushed along to the next environments if applicable, and eventually PROD.

**Questions/Problems**

In this approach, UAT is updated from the release branch. How can we handle getting sign-off on future work (P2, P3, features) in UAT? Do we need a separate stage for deploying from DEV?

Basically, I am trying to figure out the best approach to handle releases when there is a long gap between sign-off and actual release. This will change in the future as we have plans to introduce feature flags and eventually aim to get to true CI/CD

We currently use Azure DevOps.

# Answer
> 1 votes
There are a few changes that I'd make.

First, I'd get rid of the QA sign-off on dev before cutting a release branch. I'd look at methods to instill a culture of developer-led testing (especially if that means developing automated tests) on your trunk. Of course, this doesn't mean that your testers shouldn't be using the trunk deployed to the development environment - they can be practicing any manual test cases or doing exploratory testing and giving feedback.

Second, if you haven't, I'd look at automating the end-to-end testing done in the UAT environment, at least from a regression standpoint. You may want to do some manual testing, especially of an exploratory nature, in UAT, but you want to reduce the burden of manual testing, especially as the system increases in complexity.

It may not matter much, but I would recommend fixing defects in the release branch and merging back into the trunk. You could also cherry-pick or rebase or some other method, as well. But I've found that going from release branch to development branch is more intuitive.

Once you have a sign off on the release, apply a tag. You can either get the state of the code back into trunk and deploy the tag in the trunk or you can deploy the head of the release branch. Both are the same thing.

If you have multiple UAT in progress at once, then you may need multiple environments. However, you're also introducing complexity around a defect found in UAT2 that also impacts UAT1 and managing getting the fixes synchronized. I'd want to understand what makes UAT take so long and what can be done to get an accepted system into production faster to reduce parallel UATs.

---
Tags: continuous-integration, deployment, azure-pipelines
---


# Question
Title: Stage Parallelization in Jenkins declarative pipelines
I am trying to get a Jenkins (2.204.2) declarative pipeline to run **parallel stages generated into a map** on different machines. I am aware it can be done by mixing in the `script` block and I have done it that way in the past, but from documentation and other Stack\____ questions I cannot figure out why this format does not work.

I have stripped everything down as far as possible and I am trying to just create the map statically outside the pipeline to figure out the syntax I need.

```
#!/usr/bin/env groovy

def map = [:]
map['Passed Stage'] = {
    stage("Passed Map Inner Stage"){
        agent{ label "nodeLabel" }
        steps{
            echo "PASSED MAP STEP"
        }
    }
}

pipeline {
    agent { label "nodeLabel" }
    stages {
        stage("Non map stage"){
            agent{ label "nodeLabel" }
            steps{
                echo "NON MAP STEP"
            }
        }
        stage("Direct Map Outer Parallel Stage"){
            parallel{ direct :
                stage("Direct Map Inner Stage"){
                    agent{ label "nodeLabel" }
                    steps{
                        echo "DIRECT MAP STEP"
                    }
                }
            }
        }
        stage("Passed Map Outer Parallel Stage"){
            parallel{ map }
        }
    }
}

```

The first two stage methods work if I comment out the mapped one but "Passed Map Outer Parallel Stage" always fails with:

```
Running in Durability level: MAX_SURVIVABILITY
org.codehaus.groovy.control.MultipleCompilationErrorsException: startup failed:
WorkflowScript: 33: Expected a stage @ line 33, column 23.
               parallel{ map }
                         ^

WorkflowScript: 33: No stages specified @ line 33, column 13.
               parallel{ map }
               ^

WorkflowScript: 33: No stages specified @ line 33, column 13.
               parallel{ map }
               ^

3 errors

    at org.codehaus.groovy.control.ErrorCollector.failIfErrors(ErrorCollector.java:310)
    at org.codehaus.groovy.control.CompilationUnit.applyToPrimaryClassNodes(CompilationUnit.java:1085)
    at org.codehaus.groovy.control.CompilationUnit.doPhaseOperation(CompilationUnit.java:603)
    at org.codehaus.groovy.control.CompilationUnit.processPhaseOperations(CompilationUnit.java:581)
    at org.codehaus.groovy.control.CompilationUnit.compile(CompilationUnit.java:558)
    at groovy.lang.GroovyClassLoader.doParseClass(GroovyClassLoader.java:298)
    at groovy.lang.GroovyClassLoader.parseClass(GroovyClassLoader.java:268)
    at groovy.lang.GroovyShell.parseClass(GroovyShell.java:688)
    at groovy.lang.GroovyShell.parse(GroovyShell.java:700)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.doParse(CpsGroovyShell.java:142)
    at org.jenkinsci.plugins.workflow.cps.CpsGroovyShell.reparse(CpsGroovyShell.java:127)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.parseScript(CpsFlowExecution.java:561)
    at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.start(CpsFlowExecution.java:522)
    at org.jenkinsci.plugins.workflow.job.WorkflowRun.run(WorkflowRun.java:327)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:427)
Finished: FAILURE

```

The stage format seems fine from everything I've spent reading all day, and passing the same stage directly to `parallel` outside the map works fine.

What am I missing here? why won't `parallel` accept my map? Does declarative `parallel` only accept stages statically passed? Is my Jenkins version too low?

# Answer
The Scripted version of `parallel` and the Declarative version of `parallel` are different functions. The Scripted version takes a Map as an argument, whereas the Declarative version expects a block with calls to the `stage` function within that block.
> 0 votes

---
Tags: jenkins, jenkins-pipeline, pipeline
---


# Question
Title: Multistage docker build for Python distroless image
This is my Dockerfile for distroless image similar to this example

```
FROM python:3.9-slim AS build-venv
RUN python3 -m venv /venv
# other installation steps go here
RUN /venv/bin/pip install --upgrade pip setuptools wheel
# installing from requirements.txt etc.

# Copy the virtualenv into a distroless image
FROM gcr.io/distroless/python3-debian11
COPY --from=build-venv /venv /venv
ENTRYPOINT ["/venv/bin/python3"]

```

I'm trying to just get into Python shell (with all the dependencies installed), but `docker run -it my-distroless` gives me this error

```
docker: Error response from daemon: failed to create shim task:
OCI runtime create failed: runc create failed:
unable to start container process: exec: "/venv/bin/python3":
stat /venv/bin/python3: no such file or directory: unknown.

```

But when replacing base image with `debian:11-slim` everything works as expected.

```
FROM debian:11-slim AS build
RUN apt-get update && \
    apt-get install --no-install-suggests --no-install-recommends --yes python3-venv gcc libpython3-dev && \
    python3 -m venv /venv
# the rest is the same

```

Are there only "compatible" base images for *distroless* that I should use for my builds or what is the possible reason?

# Answer
> 2 votes
You're trying to copy just the `/venv` directory, but that's not going to work. Take a look at the *contents* of that directory, for example:

```
root@52bdcc57abd8:/# python3 -m venv /venv
root@52bdcc57abd8:/# ls -l /venv/bin/
total 48
-rw-r--r-- 1 root root 8834 Oct 22 03:52 Activate.ps1
-rw-r--r-- 1 root root 1880 Oct 22 03:52 activate
-rw-r--r-- 1 root root  829 Oct 22 03:52 activate.csh
-rw-r--r-- 1 root root 1969 Oct 22 03:52 activate.fish
-rwxr-xr-x 1 root root  222 Oct 22 03:52 pip
-rwxr-xr-x 1 root root  222 Oct 22 03:52 pip3
-rwxr-xr-x 1 root root  222 Oct 22 03:52 pip3.9
lrwxrwxrwx 1 root root    7 Oct 22 03:51 python -> python3
lrwxrwxrwx 1 root root   22 Oct 22 03:51 python3 -> /usr/local/bin/python3
lrwxrwxrwx 1 root root    7 Oct 22 03:51 python3.9 -> python3

```

As you can see `/venv/bin/python3` isn't actually a *file*, it' just a symlink to whichever version of Python was used to create the virtual environment.

You won't get away with just copying the binary, either, because you need the full set of runtime files from `/usr/local/lib/python3.9`, as well as any shared libraries that are by required by Python.

---

> But when replacing base image with debian:11-slim everything works as expected.

In this case, you're installing a compatible version of Python...but if you're going to do that, you might as well just stick with the `python:3.9-slim` image.

---
Tags: docker, dockerfile, python, images
---


# Question
Title: Docker exec environment different to container shell
I have a docker image that looks something like this:

```
FROM ubuntu:16.04

# Install packages for building ruby
RUN apt-get update && \
apt-get -y install build-essential git curl wget net-tools apt-utils sudo man \
libssl-dev libreadline-dev libyaml-dev nano gcc rubygems ruby-dev

# Install rbenv and ruby-build
ENV PATH /root/.rbenv/bin:$PATH
RUN git clone https://github.com/sstephenson/rbenv.git /root/.rbenv && \
git clone https://github.com/sstephenson/ruby-build.git /root/.rbenv/plugins/ruby-build && \
/root/.rbenv/plugins/ruby-build/install.sh && \
echo 'eval "$(rbenv init -)"' >> /etc/profile
echo 'eval "$(rbenv init -)"' >> ~/.profile && \
echo 'eval "$(rbenv init -)"' >> ~/.bashrc

# Install ruby and bundler
# Install Bundler
ENV CONFIGURE_OPTS --disable-install-doc
RUN rbenv install 2.4.1 && \
echo 'gem: --no-rdoc --no-ri' >> ~/.gemrc && \
rbenv global 2.4.1 && \
rbenv local 2.4.1 && \
gem install bundler

ENTRYPOINT /bin/bash

```

My problem is that when I build the image and start a container

```
docker build -t test .
docker run -it --name test-container test

```

If I run the commands directly in the container shell,
ruby is 2.4.1 (as I'd expect after running `rbenv global 2.4.1`)

```
ruby -v    # 2.4.1p111
which ruby # /root/.rbenv/shims/ruby

```

But if I open a new shell and use docker exec,
ruby is 2.3.1, which is the system default

```
docker exec test-container ruby -v    # 2.3.1p112
docker exec test-container which ruby # /usr/bin/ruby

```

---

I think it could have something to do with the bash profile not starting rbenv properly when using `docker exec`. The `env` command seems to have less output when using `docker exec` too, which is why I suspect something to do with .bashrc, .profile, or even /etc/profile not loading somehow.

How can I ensure docker exec produces the same result as running the command directly on the container?

# Answer
> 2 votes
Please refer to bash initialization process. https://www.gnu.org/software/bash/manual/html\_node/Bash-Startup-Files.html

In your `Dockerfile`, at lines 12-14 you are patching files, but do not activate any of rbenvs. Later at your last `RUN` call, you finally ARE activating an rbenv. This starts pointing at that specific version of ruby (I thin this patches `root` files) At that point you have different global shell initialization files vs. user initialization files.

Difference

* entry point: "non-login" initialization process happens, then the file: `/root/.bashrc` is executed (a.k.a. "sourced"), it points to your cool ruby
* `exec` of `ruby`: ruby executable is located using your default `PATH`. without running `/root/.bash*` files.

To validate all this you can append printouts to your shell's initialization files, and see what gets printed upon 2 cases.

Regards.

# Answer
> 0 votes
I had a similar issue and was able to solve it as follows:
(Side-note: I had not much time to look deep into the topic, so I can not explain in detail why it works, but it worked)

When executing `sudo docker -it exec container /bin/bash` I believe the shell gets started in interactive mode. This mode will source the .bashrc and so on (See link in answer from mvk\_il).

But with `sudo docker exec -it container /bin/bash -c '<cmd>'` the bash is executed in a different mode (I think even using " or ' can make kind of a difference).

**My solution:**
Execute the bash specificly in interactive mode and as a login shell. by that, it should behave the same way as if you enter the container and type the command by hand.
`sudo docker exec -it container /bin/bash -ilc '<cmd>'`

---
Tags: docker, bash
---


# Question
Title: How to get gcp project name by project id
Using cloud function - trying to make get call to get the project name and with the id - https://cloud.google.com/resource-manager/docs/creating-managing-projects#listing\_projects

```
 from googleapiclient import discovery
 from oauth2client.client import OAuth2Credentials as creds
 crm = discovery.build(
     'cloudresourcemanager', 'v3',
      http=creds.authorize(httplib2.Http()))

 project = crm.projects().get(name="projects/"+projectId).execute()

```

\`

# Answer
```
from googleapiclient import discovery
from oauth2client.client import OAuth2Credentials as creds
crm = discovery.build(
 'cloudresourcemanager', 'v3',
  http=creds.authorize(httplib2.Http()))

project = crm.projects().get(name="projects/"+projectId).execute()
project_name = project['displayName']

```
> 0 votes

---
Tags: python, google-cloud-platform, cloud
---


# Question
Title: How to connect from Docker container to the host?
My problem is that im trying to connect to a Docker host(Im running on Ubuntu) from a container. Ive tried several solutions including adding extra\_hosts: host.docker.internal:host-gateway, but still im not able to establish the connection. Im running docker with docker-compose up.

So for example when i run script the `requests.get("http//:host.docker.internal:8000")` in the container then receive an error

```
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8000): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f18ac13dd90>: Failed to establish a new connection: [Errno 111] Connection refused'))

```

.

This is my docker-compose.yml

<pre><code> version: 3.7

services:
  web:
    container_name: schedule_model_service-api
    build: .
    ports:
      - "8001:8001"
    command: startserver
    volumes:
      - ./:/app
    env_file:
      - .env
    depends_on:
      - postgres
      - redis
    extra_hosts:
    - "host.docker.internal:host-gateway"

  worker:
    container_name: schedule_model_service-worker
    build: .
    command: celeryworker
    volumes:
      - ./:/app
    env_file:
      - .env
    depends_on:
      - web
      - postgres
      - redis

  redis:
    container_name: schedule_model_service-redis
    image: redis:6.2.6
    volumes:
      - ./docker-data/redis:/var/lib/redis
    env_file:
      - .env
    restart: on-failure

  postgres:
    container_name: schedule_model_service-db
    image: postgres:14.1
    volumes:
      - ./docker-data/postgresql:/var/lib/postgresql/data
    env_file:
      - .env
    restart: on-failure```

</code></pre>

# Answer
If the container needs to connect to a host or a remote service, you need his ip or domain. **Any other workaround only will work on your laptop or is just for academic purposes**, not valid in the real scenarios with real users. For example: a container in aws needs to connect to an azure mysql.

In your case, you need the local ip. As I explained here https://stackoverflow.com/a/52213178/3957754 you could get the ip before the `docker-compose up`:

```
export MACHINE_HOST_IP=$(hostname -I | awk '{print $1}')
docker-compose up ...

```

And in your compose file use this value. For example of you have a mysql on your host and your container need it, you could use:

```
version: '3.2'
services:
  wordpress:
    environment:
      DB_HOST : ${MACHINE_HOST_IP}:3306
      DB_USER : usr_wordpress

```

Review:
> 0 votes

# Answer
The problem is that you've got an extra colon in `requests.get("http//:host.docker.internal:8000")`. It should be `requests.get("http//host.docker.internal:8000")`.

You might have hit a bug in requests' URL parsing; from the exception message you can see it's trying to connect to `localhost:8000` when it would make more sense to complain about the malformed URL.
> 0 votes

---
Tags: docker, docker-compose, ubuntu
---


# Question
Title: Minimum laptop requirements for devops learning
I started learning devops technologies, but I encountered a problem with my laptop on the first labs, I cant provision more than 3 VMs in Vagrant, and I found out that my processor is not sufficient, so I cant build labs of 6 or more servers ...

Is there a minimum requirement if I have to buy a new laptop ? Am I going to need less/more requirements when I'm going to jump to containers or newer technologies ?

My actual configuration : CPU : i7-7500U (4xCores 2.7Ghz) RAM : 8Go

# Answer
> 1 votes
It'll probably really depend on your budget. 8GB RAM seems like very little, as bahrep suggested I would first try to see if your laptop can get a RAM upgrade.

If you can afford it you could also do your experiments in the cloud. It depends on what you want to learn (DevOps is a large field) but chances are you'll have to get acquainted with the cloud. If your current hardware does not allow you to run multiple infrastructures at once I would look into VM/VPS purchasing: you'll get the resources you need and get to learn about major cloud providers in the meantime.

---
Tags: docker, vagrant, virtualization
---


# Question
Title: How to hide/mask credentials stored at terraform state file
In terraform, when we create a resourcekey for eg ibm\_resource\_key, all the credentials created are stored in plain text in state file. Question is how to hide/mask the credential section/ secret sections in state file.

Options are to store the state file itself in some safe place like S3 , vault etc, but is there a way to hide the info in state file itself?

# Answer
> 1 votes
The tfstate file can be thought of as your "executable". So no, you cannot hide/remove sensitive values from it.

What you can do, however, is to store it safely. Terraform offers tutorials on how to configure your backend to store the tfstate (because this file must never make it to your Git repository). Usually it consists of an S3 bucket (to store the contents) and a DynamoDB table (for version management).

# Answer
> 0 votes
An alternative strategy is having your passwords outside of terraform's management and use a data query to collect which ever passwords are required. This way, if you need to password rotations, the terraform won't reconfigure the resource with the former password

Good options are using AWS's Secrets Manager, or HashiCorp Vault

# Answer
> -1 votes
If you are storing credentials in your terraform state file, you can use the terraform state show command to mask the sensitive fields:

terraform state show -no-color module.web | sed -e 's/sensitive\_string.*/sensitive\_string = "*\**"/'

This will replace the sensitive string field with asterisks.

---
Tags: terraform, ibm-cloud
---


# Question
Title: How to determine which files are ignored by a .helmignore file?
I'm installing a local Helm chart; however, I keep getting an error `Error: UPGRADE FAILED: create: failed to create: Request entity too large: limit is 3145728`. From searching other SO/Stack Exchange questions, this is typically caused by unnecessary files mistakenly included in the chart. The way to resolve this, is to add those entries to a `.helmignore` file.

My chart has a `.helmignore` file, which should be excluding all unnecessary artifacts within my chart, but I'm still getting the error. So my thought is that my `.helmignore` entries aren't quite targeting the files that they should be.

I've tried running them with the `--debug` flag (didn't show anything more interesting):

```
upgrade.go:139: [debug] preparing upgrade for chartname
upgrade.go:520: [debug] copying values from chartname (v11) to new release.
upgrade.go:147: [debug] performing update for chartname
upgrade.go:319: [debug] creating upgraded release for chartname
Error: UPGRADE FAILED: create: failed to create: Request entity too large: limit is 3145728
helm.go:88: [debug] Request entity too large: limit is 3145728ffff
create: failed to create
helm.sh/helm/v3/pkg/storage/driver.(*Secrets).Create
        helm.sh/helm/v3/pkg/storage/driver/secrets.go:164
helm.sh/helm/v3/pkg/storage.(*Storage).Create
        helm.sh/helm/v3/pkg/storage/storage.go:69
helm.sh/helm/v3/pkg/action.(*Upgrade).performUpgrade
        helm.sh/helm/v3/pkg/action/upgrade.go:320
helm.sh/helm/v3/pkg/action.(*Upgrade).RunWithContext
        helm.sh/helm/v3/pkg/action/upgrade.go:148
main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:200
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.2.1/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.2.1/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.2.1/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_arm64.s:1133
UPGRADE FAILED
main.newUpgradeCmd.func2
        helm.sh/helm/v3/cmd/helm/upgrade.go:202
github.com/spf13/cobra.(*Command).execute
        github.com/spf13/cobra@v1.2.1/command.go:856
github.com/spf13/cobra.(*Command).ExecuteC
        github.com/spf13/cobra@v1.2.1/command.go:974
github.com/spf13/cobra.(*Command).Execute
        github.com/spf13/cobra@v1.2.1/command.go:902
main.main
        helm.sh/helm/v3/cmd/helm/helm.go:87
runtime.main
        runtime/proc.go:255
runtime.goexit
        runtime/asm_arm64.s:1133

```

I also tried it with the `--dry-run` flag and the chart succeeded. So at this point I'm not sure how to find what's bloating my chart.

How can I tell which files are actually getting ignored (or which are included) when I run a `helm install` or `helm upgrade`?

# Answer
Figured out a way, but it might not be the most elegant.

First, we render the chart locally and then use `tree` to print out the subdirectories with their sizes in a human-readable format.

```
helm template . --output-dir=file-size-test
tree --du -h file-size-test

```

This produces an output similar to:

```
[340M]  testing123
└── [340M]  chart_name
    ├── [ 91K]  charts
    │   ├── [ 58K]  sub-chart
    │   │   └── [ 58K]  templates
    │   │       ├── [1.3K]  deployment.yaml
    │   │       ├── [ 411]  config.yaml
    │   │       ├── [ 579]  service.yaml
    │   │       └── [ 359]  serviceaccount.yaml
<truncated>
    └── [340M]  templates
        ├── [ 68M]  deployment.yaml
        ├── [136M]  config1.yaml
        ├── [136M]  config2.yaml
        └── [1.3K]  ingress.yaml

```

So now I've found that the issue seems to be the massive `config1.yaml` and `config2.yaml` files.

While this doesn't exactly answer the question of *how* to find which have been ignored, it at least points to which might not have been ignored.
> 1 votes

---
Tags: kubernetes, helm, kubernetes-helm
---


# Question
Title: calculating the size of objects in AWS S3 buckets
I ran into an problem recently, my AWS S3 bucket shows total size of 30 TB. However when i try to get the total size of individual folders, it is not summing up more than 2 TB.

I used the below command

```
aws s3 ls --summarize --human-readable --recursive s3://BUCKETNAME

```

also, in the AWS console, I used the calculate total size option, am I missing something. I feel that there might be sizes and number of objects based on the versions, if this is the case, I do not get a query that can show the total size by individual file with versions. Is there some query or command that I can use to get the right size and objects that are taking huge size.

# Answer
You might have incomplete multipart upload in your S3 bucket which does not show up in the aws console.

> If the complete multipart upload request isn’t sent successfully, Amazon S3 will not assemble the parts and will not create any object. The parts remain in your Amazon S3 account until the multipart upload completes or is aborted, and you pay for the parts that are stored in Amazon S3. These parts are charged according to the storage class specified when the parts were uploaded.

You can follow the guide here to check if that's what taking up the storage in your S3 bucket.

If you do not have a lifecycle policy to abort incomplete multipart upload, you should probably do so.
> 1 votes

---
Tags: amazon-web-services, amazon-s3, aws-cli
---


# Question
Title: How do I configure a Readiness Probe for Selected Services?
I have 2 pods and my application is based on a cluster i.e. application synchronizes with another pod to bring it up. Let us say in my example I am using appod1 and appod2 and the synchronization port is 8080.

I want the service for DNS to be resolved for these pod hostnames but I want to block the traffic from outside the apppod1 and appod2.

I can use a readiness probe but then the service doesn't have endpoints and I can't resolve the IP of the 2nd pod. If I can't resolve the IP of the 2nd pod from pod1 then I can't complete the configuration of these pods.

E.g.

#### app1\_sts.yaml

```
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    cluster: appcluster
  name: app1
  namespace: app
spec:
  selector:
    matchLabels:
      cluster: appcluster
  serviceName: app1cluster
  template:
    metadata:
      labels:
        cluster: appcluster
    spec:
     containers:
       - name: app1-0
         image: localhost/linux:8
         imagePullPolicy: Always
         securityContext:
          privileged: false
         command: [/usr/sbin/init]
         ports:
         - containerPort: 8080
           name: appport
         readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 120
            periodSeconds: 30
            failureThreshold: 20
         env:
         - name: container
           value: "true"
         - name: applist
           value: "app2-0"

```

#### app2\_sts.yaml

```
apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    cluster: appcluster
  name: app2
  namespace: app
spec:
  selector:
    matchLabels:
      cluster: appcluster
  serviceName: app2cluster
  template:
    metadata:
      labels:
        cluster: appcluster
    spec:
     containers:
       - name: app2-0
         image: localhost/linux:8
         imagePullPolicy: Always
         securityContext:
          privileged: false
         command: [/usr/sbin/init]
         ports:
         - containerPort: 8080
           name: appport
         readinessProbe:
            tcpSocket:
              port: 8080
            initialDelaySeconds: 120
            periodSeconds: 30
            failureThreshold: 20
         env:
         - name: container
           value: "true"
         - name: applist
           value: "app1-0"

```

#### Check the Statefulset

```
[root@oper01 onprem]# kubectl get all -n app
NAME             READY   STATUS    RESTARTS        AGE
pod/app1-0       0/1     Running   0               8s
pod/app2-0       0/1     Running   0               22s

NAME                        READY   AGE
statefulset.apps/app1       0/1     49s
statefulset.apps/app2       0/1     22s

kubectl exec -i -t app1-0 /bin/bash -n app

[root@app1-0 ~]# nslookup app2-0
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find app2-0: NXDOMAIN

[root@app1-0 ~]# nslookup app1-0
Server:         10.96.0.10
Address:        10.96.0.10#53

** server can't find app1-0: NXDOMAIN

[root@app1-0 ~]#

```

I understand the behavior of the readiness probe and I am using it as it helps me to make sure service should not resolve to app pods if port 8080 is down. However, I am unable to make out how can I complete the configuration as app pods need to resolve each other and they need their hostname and IPs to configure. DNS resolution can only happen once the service has end points. Is there a better way to handle this situation?

# Answer
> 0 votes
You want to use two services.

One of your services should set some `spec.publishNotReadyAddresses: true`. That one you would use for internal communications, initializing your cluster while Pods could still be not Ready.

And the other service exposing your cluster to its clients, observing the defaults readiness.

Eg, for redis, I would use something like:

```
apiVersion: v1
kind: Service
metadata:
  name: redis-kube
spec:
  clusterIP: None
  ports:
  - name: tcp-6379
    port: 6379
    protocol: TCP
    targetPort: 6379
  - name: tcp-26379
    port: 26379
    protocol: TCP
    targetPort: 26379
  publishNotReadyAddresses: true
  selector:
    name: redis-kube
  sessionAffinity: None
  type: ClusterIP

```

Also: when trying to resolve name for the other containers in your cluster, instead of `nslookup app-0`, try `nslookup <statefulset-pod-name>.<statefulset-spec.serviceName>.<statefulset.namespace>`

to resolve app1-0, you would use `nslookup app1-0.app1cluster.app` or `nslookup app1-0.app1cluster.app.svc.cluster.local`

---
Tags: kubernetes
---


# Question
Title: Deploy A War/Ear To Container Marked build As failure When Deploying To Tomcat 9 Server
When I was Deploying A Sample Application On Tomcat9 I Faced This Issue.

This is the Tomact9 Users File where I added credentials do I need to add anything more,

Is There Any Configuration Which I have do in Tomcat or In Jenkins

What could be the point I might be missing? Can someone help me out with this?

# Answer
> 0 votes
You need to add the "manager-script" role in the roles section. I figured this out by googling:

> the username you used is not allowed to use the text based tomcat manager

and the first result was this StackOverflow post.

---
Tags: jenkins, deployment, testing, maven
---


# Question
Title: Terraform conditional block inside a map
I have an aws\_lambda\_function resource like below:

```
resource "aws_lambda_function" "mylambda" {

    #...

    environment {
        variables = {
            FOO = 1
        }
    }

}

```

I'm tring to add some environment variables dynamically based on my `var.enable_vars`

```
variable "enable_vars" {
  type        = bool
  default     = false
}

resource "aws_lambda_function" "mylambda" {

    #...

    environment {
        variables = {
            FOO = 1
    #### if var.enable_vars == true
    #       BAR = 2
    #       BAZ = 3
        }
    }

}

```

How to achieve that? Is not clear to me if a `dynamic` block can be used there.

# Answer
> 3 votes
This could be done with a `dynamic` block but its pretty complicated. This is how I would do it but there are other ways.

```
variable "enable_vars" {
  type        = bool
  default     = false
}

locals {
  default_lambda_vars = {
    FOO = 1
  }
  extra_vars = {
    BAR = 2
    BAZ = 3
  }

  final_lambda_vars = var.enable_vars ? merge(local.default_lambda_vars, local.extra_vars) : local.default_lambda_vars
}

resource "aws_lambda_function" "mylambda" {

    #...

    environment {
        variables = local.final_lambda_vars
    }

}

```

---
Tags: terraform, terraform-module
---


title: how to delete branch with vscode

text:
# Question
Title: how to delete branch with vscode
I created three branches on github.How can I delete one branch on vscode? just to workaround this, I delete the branch on github, go back to vscode to pull/synch etc but even though deleted branch does not exist on github, vscode seems to still have it. How can I properly delete a branch on vscode?

# Answer
Go to View --\> select Command Platted option --\> select Delete Branch --\> and select the local branch which we have to delete
> 4 votes

# Answer
Menu - View - SCM, choose the project then, More actions - Branch - Delete Branch. If branch is deleted on github, on vscode, More actions - Pull,Push-Sync, otherwise vscode still is on old cache.
> 1 votes

---
Tags: github, vscode
---


# Question
Title: Docker Compose: How do you build an image while running another container?
I'm trying to build a Nuxt 3 application using docker compose. However, I'm having trouble with docker compose and dockerfile configuration.

## Context

To give background on the project that I'm working on, here are the stack that I'm using:

* **Nuxt 3** for SSG
* **Strapi** for CMS
* **Nginx** for serving static contents generated from Nuxt 3

So, in Nuxt 3 I can run `npx nuxi generate` to pre-render/generate static contents for the website. After generating the static HTML files, nginx will serve the static contents. Strapi will serve as headless CMS, providing contents for the website.

## Problem

The problem is that, when pre-rendering a static contents in Nuxt 3, it requires Strapi backend to run so that it can fetch the contents from Strapi CMS and generate the html files.

So in terms of command line, I need to make sure that Strapi server is running (`strapi start`) and only then generate the static files (`npx nuxi generate`) or else there would be an error during build process because Nuxt application cannot fetch resource from the strapi server.

When using docker, I need to make sure that Strapi server container is running while building the Nuxt/node.js image. However, I don't know how I'm supposed to run a container (strapi server) while building an image (Nuxt SSG contents).

## Files

**Dockerfile.client**

```
# Nuxt Image
FROM node:16-alpine as build
WORKDIR /app
COPY ./client .
RUN npm install
RUN npm run generate

# Nginx Image
FROM nginx:1.15 as publish
COPY --from=build /app/.output/public/ /usr/share/nginx/html
COPY ./nginx/nginx.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
ENTRYPOINT ["nginx", "-g", "daemon off;"]

```

**Dockerfile.server**

```
# Strapi Image
FROM node:16-alpine
# Installing libvips-dev for sharp Compatability
RUN apk update && apk add  build-base gcc autoconf automake zlib-dev libpng-dev nasm bash vips-dev
ARG NODE_ENV=production
ENV NODE_ENV=${NODE_ENV}
WORKDIR /opt/
COPY ./server/package.json ./server/yarn.lock ./
ENV PATH /opt/node_modules/.bin:$PATH
RUN yarn config set network-timeout 600000 -g && yarn install
WORKDIR /opt/app
COPY ./server .
RUN yarn build
EXPOSE 1337
CMD ["yarn", "start"]

```

**docker-compose.yml**

In `docker-compose.yml` file, I specified three services (client, server, database) and all of them belong to one network. I made sure that client `depends_on` server and database.

---

I just started learning docker few days ago and it's my first time doing DevOps so I might be missing the mark... Any help will be appreciated.

---

**Edit 1:**

I tried separating `docker-compose.yml` into two:

1. `docker-compose.yml` for strapi server and database service
2. `docker-compose.client.yml` for Nuxt service

I ran `docker compose up` for running Strapi server first. After Strapi container was running, I tried running `docker compose -f docker-compose.client.yml up --build` to build Nuxt image and hoping it would refer to the running strapi container. Unfortunately, Nuxt image was not referring to the running strapi container.

# Answer
> 1 votes
You could try using the `--profile` option during the build.

```
version: "3"
services:
  strapi:
    profiles: ["strapi"]
    build:
      context: ./
      dockerfile: strapi/Dockerfile
    container_name: "strapi"
    restart: always
    ports:
      - "3000:3000"

  nuxt:
    build:
      profiles: ["nuxt"]
      context: ./
      dockerfile: nuxt/Dockerfile
    image: nuxt:latest
    container_name: "nuxt"
    restart: always
    ports:
      - "3000:3000"

```

Then pass the profile switch during the build:

`docker compose --profile nuxt build`

Please refer to the official documentation for more information:

https://docs.docker.com/compose/profiles/

---
Tags: docker, docker-compose, dockerfile, nginx, node.js
---


title: Can I found a small company on DevOps?

text:
# Question
Title: Can I found a small company on DevOps?
Suppose I am en entrepreneur with little seed money for my Great Idea (TM) and I start a company initially planning to implement my Great Idea (TM) on my own and hire additional help on a basis dictated by the rate of company growth.

Frequently, answers on this site center around the idea that DevOps is not a role, but a new way of doing things with a new philosophy of operation that allows a team of roles to actually do DevOps. This prevents burnout and people from feeling like a lone wolf

As a fledgling and budding company, then, should I even bother to pursue DevOps as a philosophy, or is this only appropriate for a more mature operation capable of having a development team of sufficient size?

# Answer
DevOps has three parts: *Tooling/automation, Organizational, and Cultural.*

You're basing your organizational structure and process' on your business' maturity. Which is smart. My experience in a startup and with a fortune 1000 company has afforded me the experience to denote that starting with DevOps is better for company growth. It's efficient and fits nicely with agile.

## Tooling

Being your maturity is really young, you should buy your DevOps process as saas products to reduce time to market. Instead of Jenkins use circleCI or similar product. Instead of VM's and docker, use Heroku. Try and incorporate the toolset but do it in a manner that works for the size of your business. If deploying code manually takes 4 hours a week and automating it in 1 hour saves that. Do it.

## Organizational

Also from an organizational standpoint, you will naturally start with all developers as the responsible party for operations/features even product development. Take a look at http://web.devopstopologies.com/ for organizational structures. Our startup started with type 2, and as we mature will move toward type 7.

## Cultural

DevOps is simply teamwork. Work together and everything gets easier. It usually is a problem because leadership doesn't keep this in mind so as you start out, ensure that everyone is apart of the business. This is also reflected in agile.

So in short, yes pursue DevOps but do so at the level that meets your business maturity.
> 10 votes

# Answer
The one (starting a small 1-2 person company and growing as money comes in) has nothing to do with the other (using a DevOps philosophy).

Even a 1-man business can be DevOps based. You can start implementing your Great Idea (tm) with a functioning fully containerized CI/CD pipeline, in fact there is no better time than at the beginning, when there is no cruft to take care of, yet, and *before* you hit a significant number of users. It's not going to take months of work; you can hack together a basic Jenkins/Docker pipeline in a few hours (if you know a lot already) or maybe days (but then the days will have been productively spent learning that stuff as you go).

Starting out with the CD part in a greenfield project is *much* easier than adding CD to an existing application which used manual deploys before.

You can start out with full test coverage at the start (there is no better time, for obvious reasons), including tools that verifies test coverage as part of the CI/CD.

You can easily start out with some small security measures, like checking your test suite for superfluous open ports or whatever.

You can, and should start out with cattle-not-pet right away (meaning IaC, IaaS, PaaS and all that good stuff, hosted on one of the pretty streamlined "big three" providers). Start small, scale a needed.

When you add more personel, they will start out right in that fruitful athmosphere, and hopefully many of their cheaper errors will immediately be caught by the CI/CD.
> 5 votes

# Answer
I think it's a feasible idea. You can set up as a service provider to companies that need services of DevOps but don't want to employ a staff for such services either because they can't afford to keep the person on a salary or because they're not into a business that permanently needs someone on hand. Such companies can engage your services or even put you on some form of retainers.
> 1 votes

# Answer
I have found developing a new venture can actually be made EASIER by adopting the DevOps philosophy and practices. After all, a small (1-2) person company does not have the resources (or luxury) to hire a developer, QA, operations and even customer service.

So almost by necessity the early players will have to cover multiple roles, development and operations (features and releases) can be mapped directly to a "DevOps" styled business.

That what I did with my previous micro-agency, we eventually became 2 full time (I mostly developer, my partner primarily ops backgrounds, but we started cross pollenating to hybrid ourselves :) and 3-4 regular contractors that would be called in as needed. Eventually we started also ongoing managing (operating) some projects for clients in addition to adding features.

Though we did not use the term "DevOps" (I don't think I knew it existed at the time), we were focused on improving feature to deployment timeframe, and making the client experience as painless as possible.

Also, a DevOps model may make it easier for people to take time off, since their role can be covered by somebody else!

I am just now starting a venture (providing DevOps services) to small business.

So in a nutshell, not only do I think it is *possible*, but I believe it is the **BEST** way to start!

Good Luck!
> 1 votes

# Answer
Yes because you can use advanced automation to create value and establish reusable foundations of your future bigger company. Depends, of course on how often you are going to update your digital products. But you will value the possibility to as much manual work as necessary on repetitive exercises.
> 0 votes

# Answer
**Don't forget:** *the money comes from your implemented great idea (tm) - not from the best CI/CD pipeline or best DevOps practices*.

If your idea is a nothing burger, the best CI/CD pipeline does not save you. Probably you lose time in setting it up, administrating it, and monitoring it while other people have already built your idea. Or you lose time to develop a PoC or a MVP to test it. Because you are still building a CI/CD pipeline. So my recommendation: first grow, then make DevOps so you get any benefits and deliver faster and better stuff to your customers. Otherwise, it just satisfies your technical interests. Nothing else! Don't be an ignorant nerd!
> 0 votes

---
Tags: culture, business
---


# Question
Title: Creating a hostgroup from a super-set of hosts
I have two host files with different hosts, hostGroups and super-set in each file like below.

/ets/hostFiles/TestBoxes

```
[TestBox:children]
groupA
groupB

[groupA]
...
...

[groupB]
...
...

```

/ets/hostFile/ProdBoxes

```
[ProdBox:children]
groupPA
groupPB

[groupPA]
...
...

[groupPB]
...
...

```

Now i would like to create a Primary host group with TestBox and ProdBox

Please let me know if this is possible.

Note: This is just an example... I have 100+ files like this. and I want to create primary group with few of the files. `hosts=all` is not my requirement.

# Answer
Use the *constructed* inventory plugin. See

```
shell> ansible-doc -t inventory constructed

```

Create the inventory files

```
shell> tree inventory/
inventory/
├── 01-TestBoxes
├── 02-ProdBoxes
└── 03-Primary.yml

0 directories, 3 files

```

```
shell> cat inventory/01-TestBoxes
[TestBox:children]
groupA
groupB

[groupA]
host_1

[groupB]
host_2

```

```
shell> cat inventory/02-ProdBoxes
[ProdBox:children]
groupPA
groupPB

[groupPA]
host_P1

[groupPB]
host_P2

```

```
shell> cat inventory/03-Primary.yml
plugin: constructed
groups:
  Primary: (group_names | intersect(['ProdBox', 'TestBox'])) | length > 0

```

The playbook

```
shell> cat pb.yml
- hosts: Primary
  gather_facts: false
  tasks:
    - debug:
        var: inventory_hostname

```

reads files from the directory *inventory* in alphabetic order. The file *Primary.yml* creates the group *Primary* from hosts that are members of the groups *ProdBox* and *TestBox*

```
shell> ansible-playbook -i inventory pb.yml

PLAY [Primary] *******************************************************************************

TASK [debug] *********************************************************************************
ok: [host_1] =>
  inventory_hostname: host_1
ok: [host_2] =>
  inventory_hostname: host_2
ok: [host_P1] =>
  inventory_hostname: host_P1
ok: [host_P2] =>
  inventory_hostname: host_P2

PLAY RECAP ***********************************************************************************
host_1: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
host_2: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
host_P1: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
host_P2: ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

```

Q: ***" I have 100+ files like this. and I want to create primary group with few of the files."***

A: Create links

```
shell> tree inventory/
inventory/
├── 01-TestBoxes -> ../hostFiles/TestBoxes
├── 02-ProdBoxes -> ../hostFiles/ProdBoxes
└── 03-Primary.yml

```
> 1 votes

---
Tags: ansible, ansible-inventory
---


# Question
Title: How do I provide a config file (.env) when starting a container?
I'm trying to install a GUI on Bitcoin Lightning Network Testnet. I have 2 lightning nodes running and connected together. Both in containers.

Now I would like to have a GUI interface to manage my nodes. I used this image: https://hub.docker.com/r/apotdevin/thunderhub

I can start the container and access it on my local computer on port 3000.

My question is how do I configure Thunderhub to connect to my nodes ?

If I follow the instructions on the Thunderhub home page, they refer me to this page:

https://docs.thunderhub.io/setup/#server-accounts

But the instructions are:

```
You can add accounts on the server by adding this parameter to the .env file:
.....

```

How should I proceed from there ? Does it means I must rebuild the image ? How do I provide a '.env' config file when starting a container

# Answer
I would recommend running containers using docker-compose and use "environment" directive. https://docs.docker.com/compose/gettingstarted/

From this link above is a good example, see step 5. Also you can create .env file in the same directory as the the docker-compose.yml See examples in this link below.

https://docs.docker.com/compose/environment-variables/

If you don't want use docker-compose, you can use --env flag:

```
docker run --env VAR1=value1 --env VAR2=value2 ubuntu env

```
> 1 votes

---
Tags: docker
---


# Question
Title: How to override global "environment {}" Jenkins Variables in a stage?
I'm trying to set up an automated Jenkins pipeline for the `Development` & `Staging` branches within our Git repository. I have most of the pipeline working. But I do not know how to override globally set `environment {}` variables depending on the git branch.

This is my Jenkinsfile:

```
pipeline {
  agent any

  environment {
    dev_mongo_url = credentials('DEV_MONGO_URL')
    stg_mongo_url = credentials('STG_MONGO_URL')
  }

  stages {

    stage('Env Setup') {
      steps {

        script
        {
        if (env.BRANCH_NAME == 'bugfix/*' || env.BRANCH_NAME == 'feature/*' || env.BRANCH_NAME =='development' || env.BRANCH_NAME =='hotfix/*') {
            env.environment = 'development'
            echo 'Build & deployment to development environment'
            }

        if (env.BRANCH_NAME == 'staging'){
            env.environment = 'staging'
            echo 'Build & deployment to staging environment'

        }
  }
}
}

    stage('Migration') {
      when {
        anyOf {
          branch 'development';
          branch 'feature/*';
          branch 'staging'
         }
      }
      steps {
        echo "Installing the project NodeJS dependencies..."
        sh 'npm ci'
        echo "Executing the MongoDB migration script & displaying the status..."
        dir('db-migrations') {
          sh 'npm ci'
          sh('npm run migrate --silent -- --mongodb-url=$mongo_url')
          sh 'npm run migrate:status'
        }

      }
    }

}
    }

```

Within the above pipeline, I have the `Env Setup` stage, which assigns the `environment` type variable (development/staging) depending on the git branch. We have two MongoDB instances - one for development & one for the staging environment. I have defined both of the MongoDB server URL's under the `environment {}` block. Now, I want to dynamically connect to the dev/staging mongo servers in the `Migration` stage, depending on the environment type, and apply the migrations. I'm trying to automatically inject `$mongo_url` value from the `dev_mongo_url` & `stg_mongo_url` global variables depending on the environment. I'm stuck, and I have no clue how to achieve it.

Any help?

# Answer
> 1 votes
See the "using environment variables dynamically" section of the official docs. I think this will answer your question. Here is the snippet of example code from those docs:

```
pipeline {
    agent any
    environment {
        // Using returnStdout
        CC = """${sh(
                returnStdout: true,
                script: 'echo "clang"'
            )}"""
        // Using returnStatus
        EXIT_STATUS = """${sh(
                returnStatus: true,
                script: 'exit 1'
            )}"""
    }
    stages {
        stage('Example') {
            environment {
                DEBUG_FLAGS = '-g'
            }
            steps {
                sh 'printenv'
            }
        }
    }
}

```

So your code might end up looking something like this:

```
pipeline {
    agent any
    environment {
        mongo_url = """${
            switch(env.BRANCH_NAME) {
                case 'bugfix/*'
                case 'feature/*'
                case 'development'
                case 'hotfix/*'
                    'YOUR_DEV_MONGO_URL_HERE'
                    break
                case 'staging'
                    'YOUR_STAGING_MONGO_URL_HERE'
                    break
            }"""
    }
}

```

---
Tags: jenkins, jenkins-pipeline, jenkinsfile
---


# Question
Title: Value of succeeded() in Azure DevOps pipeline before first stage is run
In an Azure DevOps pipeline, what is the value of Agent.JobStatus before any stages have been run? Is it initialized to 'Succeeded' or is it initially undefined or set to null or something like that, only to be set to 'Succeeded' after the first successful execution?

In other words:

* If I include `succeeded()` (which is the same thing as, `in(variables['Agent.JobStatus'], 'Succeeded', 'SucceededWithIssues')`) in a condition for running the first stage, is it going to return true even though there have been no previous stages to return a successful result?
* If I include `succeeded()` in a conditional for running the second stage, where the first stage has a condition that evaluated to false so the first stage was skipped, will it be true even though no previous stages have been *executed* returning a successful result?

The way I'd like to think it works is such that `succeeded` will return `true` in both places where I use it in the following code, where I want the first stage to run if the original triggering repo branch was 'main' and the second stage to run if the original branch was 'release/\*'. (This is a slightly simplified version of the actual logic.) Am I correct?

```
stages:
  - template: my-template.yml
    condition: and(succeeded(), eq(variables['resources.pipeline.previousPipeline.SourceBranch'], 'refs/heads/main'))
    parameters:
      ...
  - template: my-template.yml
    condition: and(succeeded(), startsWith(variables['resources.pipeline.previousPipeline.SourceBranch'], 'refs/heads/release/'))
    parameters:
      ...

```

Alternatively, if `succeeded()` doesn't work in this manner, would I get what I need if, instead, I were to use `not(failed())`?

# Answer
> 0 votes
Five minutes later I found this dealt with at https://github.com/MicrosoftDocs/azure-devops-docs/issues/7738. Indeed, it appears that the Succeed outcome is not set until a stage has actually been executed. One respondent in the discussion indicates that this has worked for them:

> As a work-around, our team has had success (no pun intended) using:
>
> `condition: not(or(failed(), canceled())`
>
> This works because the previous stage being skipped does not set a failed status.

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: bitbucket pipeline to push commits to another repo
So this is the scenario i have right now, we have a repo (boilerplate) that contains our infrastructure code, docker files, and pipeline scripts, each time we create a new project we copy this repo to a new repo.

the issue is this when we make a change in the boilerplate repo, we have to manually make the change in all our projects and i'd like to automate this process.

is it possible to create a git hook or a bitbucket pipeline to automate this process.

# Answer
Don't know if you have found an answer already, but yes, it is possible.

there are many possible solutions here for the automation tools, but one of the obvious is to create automation code that updates your repositories as needed (for example by using terraform with git provider) and then running this terraform code in bitbucket pipeline on your boilerplate repo.

At this simple scenario - you need to get a list of all repositories that you need to update in some way, by reading it from some file/api or anything else.
> 0 votes

---
Tags: git, bitbucket, bitbucket-pipelines, git-hooks
---


# Question
Title: Does `helm upgrade` use rolling restarts for `deployments`, yes/no? if not then what is the default?
I ask because:

* I want to know what is the default `helm upgrade` behavior
* I might need to change the default `helm upgrade` behavior

### Does `helm upgrade` use rolling restarts for `deployments`? If not then what is the default?

If `helm upgrade` is not the thing that controls the behavior for `deployments`, please say what does. (I suspect the `deployment` controls the behavior of what happens during a `helm upgrade` but I am not sure so I am asking.)

# Answer
> 1 votes
Helm upgrade is making revisions to the deployment which means, you can roll back to the last deployment. regarding the restart of current pods, it will happen based on your definition of Deployment.spec.strategy.rollingUpdate will recreate the pods!

---
Tags: kubernetes, deployment, helm
---


# Question
Title: Azure DevOps, get the triggering branch of the triggering pipeline
I've got two (YAML) pipelines in Azure DevOps, call them B (application build) and D (application deployment). D deploys to Dev, QA, and Test, each in a separate stage. The QA and Test environments are configured in DevOps to require approval before their respective stages in D are executed. The YAML files behind B and D are in the same DevOps project.

The code repository is Azure DevOps Git. B is triggered by completed merges to the `main` branch. D is triggered by successful completion of B. In case it matters, the means by which I've configured D to be triggered by successful completion of B is via

leading to

So far, this arrangement has worked well.

Now I want B to be triggered by not only feature pushes to `main` but by hotfix pushes to any branch named `release/*`. So,

```
  trigger:
    - main
    - release/*

```

The difference is that the hotfixes should be deployed only to Test, not to Dev or QA.

Therefore, in D, I want to make execution of the Dev and QA deployment stages conditional on the triggering branch of B having been `main`. Is there some way in D to access from B the value that *in* B can be referenced as `$(Build.SourceBranch)`?

UPDATE: I now learn that the manner I described above for having D triggered by B is itself outmoded, and I should be using something like

```
resources:
  pipelines:
    - pipeline: sourcePipeline
      source: 'B'
      trigger:
        branches:
          - main
          - release/*

```

Is that the correct way to set this up? Do I need to specify the branches here or are they relevant? (I saw one example that simply has `trigger: true`, which I'm guessing means that the second pipeline should *always* be run after the first completes. Perhaps branches are specified above only when B may be triggered by lots of branches but D should run after B only when B was triggered by a subset of those.)

If this is the preferred approach and I switch to it, does the answer to my question become that I can now access B's triggering branch in D through `$(resources.pipeline.sourcePipeline.SourceBranch)`?

# Answer
> 0 votes
In the absence of a response I continued with my project and engaged in some painful experimentation. In the end, my guesses turned out to be correct:

* Replace the Classic method of triggering D off of B with the YAML method, like this:

```
  resources:
    pipelines:
      - pipeline: sourcePipeline
        source: 'B'
        trigger: true

```

And this requires replacing Build variables like Build.SourceBranch with pipeline variables: `resources.pipeline.sourcePipeline.SourceBranch` is correct.

Finally (bonus answer to a question I hadn't asked here), it turns out you can't use `condition:` on stages that are template invocations. So instead I had to create a `canRun` parameter for the template being invoked, setting it to the same expression I would have used in a `condition:`, and then referencing `canRun` in `condition:`s in the template.

---
Tags: azure-devops, azure-pipelines
---


# Question
Title: How to list security groups for the specific instance via CLI?
I'm looking for syntax like:

```
aws ec2 describe-security-groups --instance-id i-0xyz

```

however, the above sub-command doesn't support `--instance-id` parameter.

I've checked and there are `--filters` and `--query` parameters, but I'm not sure about the syntax.

What would be the easiest way to display the description of the security group of the instance?

# Answer
You can use `aws ec2 describe-instances` instead, you can specify the id `--instance-ids <value>` and it will output the security group.
> 4 votes

# Answer
The following one-liner in shell works for me:

```
aws ec2 describe-security-groups --group-ids $(aws ec2 describe-instances --instance-id $id --query "Reservations[].Instances[].SecurityGroups[].GroupId[]" --output text) --output text

```

Where `$id` is my instance-id.
> 5 votes

# Answer
you can use

```
aws ec2 describe-instances --instance-ids (your Instance id) --query "Reservations[].Instances[].SecurityGroups[].GroupId[]" --output text
```

replace

```
(your instance id)
```
with the specific instance id you are looking for.
> 0 votes

# Answer
Even better check this out:

```
aws ec2 describe-instances --output=json | jq '.Reservations[] | .Instances[] | {PrivateAddress: .PrivateIpAddress, Tags: .Tags, KeyName: .KeyName, Tags: .Tags, RunningState: .State, InstanceId: .InstanceId, SecurityGroups: .SecurityGroups}'

```

You don't need to know anything this way ... but you need to install `jq`, to chomp the JSON o/p just right.

A lot of the elements there could be removed I guess ... tags, key names, running state, etc.
> 0 votes

# Answer
If you want to run this on an EC2 instance and find out its own security groups, you can do:

```
id=$(wget -q -O - http://169.254.169.254/latest/meta-data/instance-id)

for sg in $(aws ec2 describe-instances --instance-id $id \
  --query "Reservations[].Instances[].SecurityGroups[].GroupId[]" \
  --output text)
do
 echo
 aws ec2 describe-security-groups --group-id $sg --output text
done

```
> 0 votes

---
Tags: security, aws-cli
---


# Question
Title: Validating kubernetes manifest with --dry-run and generateName
We're using ArgoCD to manage deployments, and I'm in the process of sorting out the config repository. I'm planning on having pull requests for any change to the config, and I want to validate the configuration to ensure it's not breaking anything. I've done some looking around, and it looks like the main options are `kubeval`, `kubeconform` or using `--dry-run` with `kubectl`.

Due to kubectl actually connecting to the cluster, and have the cluster perform the validation I prefer this approach as it should catch every possible error, however I'm running into a problem.

One of the resources uses `generateName` which is not compatible with `kubectl apply`, so if I try and validate using `kubectl apply -f manifest.yaml --dry-run=server` I get the error `cannot use generate name with apply`. To get around this, I tried to use `kubectl create -f manifest.yaml --dry-run=server` but instead I get a load of errors about resources already existing (understandable).

So how can I do this? I can't use apply, and I can't use create. Are there any other options? Does anyone know what Argo uses to validate, because if I push something invalid it presents an error before it is told to sync.

# Answer
After a lot of playing around, I came to a working solution that I briefly mentioned in a comment in the original question. The CI is now creating a namespace on the cluster, running the dry run `apply` and then deleting the namespace when finished. Not sure if this is the perfect solution but it's working as I hoped.

```
helm template . \
  --values common/values-common.yaml \
  --values variants/$VARIANT/values-$VARIANT.yaml \
  --name-template=github-actions-test \
  --set image.tag=github-actions-test \
  --namespace $NAMESPACE \
  --debug > dry-run.yaml

kubectl create namespace $NAMESPACE --dry-run=client -o yaml | kubectl apply -f -
echo "errors=$(kubectl create -f dry-run.yaml -n $NAMESPACE --dry-run=server -o yaml 2>&1 > /dev/null)" >> $GITHUB_OUTPUT
kubectl delete namespace $NAMESPACE

```
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Deploying environment secrets to services
I know I can use CD pipelines to deploy an app to a given environment (dev/stage/prod)

Given that each environment should have its own environment variables/secrets for each app, how can I streamline the process of securely setting those variables/secrets in each environment without having to ssh into the environment server and create a .env file for the specific app/environment that's being deployed?

I've heard of KeyVaults but I'm not sure if that's overkill for a single set of environments.

# Answer
> 1 votes
You can achieve this in multiple ways.

You can set your credentials as secrets if you use a CI/CD system such as Jenkins or Gitlab pipelines. Then reference them within your pipeline script as variables and inject the credentials into your application during the build. This will not expose the credentials in the CI/CD system logs. Please refer to the following articles for more information:

https://docs.gitlab.com/ee/ci/variables/

https://www.jenkins.io/doc/book/using/using-credentials/

https://devops.com/how-to-securely-manage-secrets-within-jenkins/

Depending on your application security requirements, you can set up an identity-based secret and encryption management system such as `Hashicorp Vault.` Vault provides encryption services that are gated by authentication and authorization methods. You can create a role for your CI/CD system within the Vault and get the secrets during the build to auto-generate a `.env` file. Learning the vault concepts may take a decent amount of time.

https://docs.gitlab.com/ee/ci/secrets/

---
Tags: continuous-deployment, environments, environment-variables
---


# Question
Title: which path has to be specified in httpGet handler in kubernetes?
I just started learning Kubernetes and I am confused about the 'path' parameter in httpGet handler in kubernetes probes. can anyone explain about that please?

# Answer
> 0 votes
`path` is used to declare the service's `routing address` / `interface address` / `API`

When using `livenessProbe.httpGet` as a health checker, a health query is required, i.e. it calls a uri to confirm the service health based on the returned status code

The composition of this uri: `scheme://podIp:port/path` == `http://10.244.171.183:8080/index.html`

---
Tags: kubernetes
---


title: What is the PF field in k9s?

text:
# Question
Title: What is the PF field in k9s?
In `k9s` (kubernetes management tool) what is the PF column (for pods)? What does the Ⓕ value mean?

# Answer
> 5 votes
The "PF" field refers to port forwarding.

You may setup port forwarding, ("shift-f" in pod view). This indicator probably means you have setup some port forwarding -- in my case, it prints some weird character.

Although to be honest ... I only figured it out, searching their git repository history .... Introduced here: https://github.com/derailed/k9s/commit/f1111174aa8df815e25b0a029ba1465d3a855b49#diff-5b19eaab395254884c7dff045b0e50a599c29794e80a81fe42c9667450b6e9d4

---
Tags: kubernetes
---