# Question
Title: How to access environment variable in post-build access for jira plugin
I installed `JIRA Plugin` for the jenkins. It add some action in `Post-build Action` menu. One of the action is **JIRA: Update relevant issues**.

When I try to access Environment Variable, it gives error.

```
WARNING: jira rest client get issue from jql search error. cause: RestClientException{statusCode=Optional.of(400), errorCollections=[ErrorCollection{status=400, errors={}, errorMessages=[The issue key '$MY_ISSUE_ID' for field 'issue' is invalid.]}]}
java.util.concurrent.ExecutionException: RestClientException{statusCode=Optional.of(400), errorCollections=[ErrorCollection{status=400, errors={}, errorMessages=[The issue key '$MY_ISSUE_ID' for field 'issue' is invalid.]}]}
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:262)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
    at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
    at com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:107)
    at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:177)
    at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:136)
    at hudson.plugins.jira.selector.JqlIssueSelector.findIssueIds(JqlIssueSelector.java:49)
    at hudson.plugins.jira.Updater.perform(Updater.java:85)
    at hudson.plugins.jira.JiraIssueUpdater.perform(JiraIssueUpdater.java:64)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
    at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1749)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:421)
Caused by: RestClientException{statusCode=Optional.of(400), errorCollections=[ErrorCollection{status=400, errors={}, errorMessages=[The issue key '$MY_ISSUE_ID' for field 'issue' is invalid.]}]}
    at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient$2.apply(AbstractAsynchronousRestClient.java:176)
    at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient$2.apply(AbstractAsynchronousRestClient.java:170)
    at com.atlassian.httpclient.api.ResponsePromiseMapFunction.apply(ResponsePromiseMapFunction.java:67)
    at com.atlassian.httpclient.api.ResponsePromiseMapFunction.apply(ResponsePromiseMapFunction.java:11)
    at com.atlassian.util.concurrent.Promises$Of$3.apply(Promises.java:268)
    at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:158)
    at com.google.common.util.concurrent.Futures$6.run(Futures.java:975)
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
    at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
    at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
    at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
    at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
    at com.atlassian.util.concurrent.Promises$Of$3.apply(Promises.java:268)
    at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:158)
    at com.google.common.util.concurrent.Futures$6.run(Futures.java:975)
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
    at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
    at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
    at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
    at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$1$1.run(SettableFuturePromiseHttpPromiseAsyncClient.java:46)
    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$ThreadLocalDelegateRunnable$1.run(SettableFuturePromiseHttpPromiseAsyncClient.java:197)
    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.runInContext(SettableFuturePromiseHttpPromiseAsyncClient.java:90)
    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$ThreadLocalDelegateRunnable.run(SettableFuturePromiseHttpPromiseAsyncClient.java:192)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

```

`$MY_ISSUE_ID` is valid environment variable, and I am able to access it on other section of the build.

I tried `issue=${XXXX}`, `issue=XXXX`, `issue="$XXXX"` etc... but all gives error.

After answer I tried `env. MY_ISSUE_ID` but it give same error.

```
WARNING: jira rest client get issue from jql search error. cause: RestClientException{statusCode=Optional.of(400), errorCollections=[ErrorCollection{status=400, errors={}, errorMessages=[The issue key 'env. MY_ISSUE_ID' for field 'issue' is invalid.]}]}
java.util.concurrent.ExecutionException: RestClientException{statusCode=Optional.of(400), errorCollections=[ErrorCollection{status=400, errors={}, errorMessages=[The issue key 'env. MY_ISSUE_ID' for field 'issue' is invalid.]}]}
    at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:289)
    at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:262)
    at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:91)
    at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:69)
    at com.atlassian.jira.rest.client.internal.async.DelegatingPromise.get(DelegatingPromise.java:113)
    at hudson.plugins.jira.JiraRestService.getIssuesFromJqlSearch(JiraRestService.java:177)
    at hudson.plugins.jira.JiraSession.getIssuesFromJqlSearch(JiraSession.java:132)
    at hudson.plugins.jira.selector.JqlIssueSelector.findIssueIds(JqlIssueSelector.java:46)
    at hudson.plugins.jira.Updater.perform(Updater.java:89)
    at hudson.plugins.jira.JiraIssueUpdater.perform(JiraIssueUpdater.java:64)
    at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:81)
    at hudson.tasks.BuildStepMonitor$3.perform(BuildStepMonitor.java:45)
    at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:744)
    at hudson.model.AbstractBuild$AbstractBuildExecution.performAllBuildSteps(AbstractBuild.java:690)
    at hudson.model.Build$BuildExecution.post2(Build.java:186)
    at hudson.model.AbstractBuild$AbstractBuildExecution.post(AbstractBuild.java:635)
    at hudson.model.Run.execute(Run.java:1749)
    at com.tikal.jenkins.plugins.multijob.MultiJobBuild.run(MultiJobBuild.java:76)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:421)
Caused by: RestClientException{statusCode=Optional.of(400), errorCollections=[ErrorCollection{status=400, errors={}, errorMessages=[The issue key 'env.MY_ISSUE_ID' for field 'issue' is invalid.]}]}
    at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient$2.apply(AbstractAsynchronousRestClient.java:176)
    at com.atlassian.jira.rest.client.internal.async.AbstractAsynchronousRestClient$2.apply(AbstractAsynchronousRestClient.java:170)
    at com.atlassian.httpclient.api.ResponsePromiseMapFunction.apply(ResponsePromiseMapFunction.java:67)
    at com.atlassian.httpclient.api.ResponsePromiseMapFunction.apply(ResponsePromiseMapFunction.java:11)
    at com.atlassian.util.concurrent.Promises$Of$3.apply(Promises.java:295)
    at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:185)
    at com.google.common.util.concurrent.Futures$6.run(Futures.java:975)
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
    at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
    at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
    at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
    at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
    at com.atlassian.util.concurrent.Promises$Of$3.apply(Promises.java:295)
    at com.atlassian.util.concurrent.Promises$2.onSuccess(Promises.java:185)
    at com.google.common.util.concurrent.Futures$6.run(Futures.java:975)
    at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:253)
    at com.google.common.util.concurrent.ExecutionList$RunnableExecutorPair.execute(ExecutionList.java:149)
    at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:134)
    at com.google.common.util.concurrent.AbstractFuture.set(AbstractFuture.java:170)
    at com.google.common.util.concurrent.SettableFuture.set(SettableFuture.java:53)
    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$1$1.run(SettableFuturePromiseHttpPromiseAsyncClient.java:46)
    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$ThreadLocalDelegateRunnable$1.run(SettableFuturePromiseHttpPromiseAsyncClient.java:197)
    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient.runInContext(SettableFuturePromiseHttpPromiseAsyncClient.java:90)
    at com.atlassian.httpclient.apache.httpcomponents.SettableFuturePromiseHttpPromiseAsyncClient$ThreadLocalDelegateRunnable.run(SettableFuturePromiseHttpPromiseAsyncClient.java:192)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

```

What is the way to pass `environment variable` in **jql** for `post build action`.

When I gave `issue=MYPROJECT-1234`, it works fine. But, I want to set `MYPROJECT-1234` as environment variable.

Job **config.xml**

```
<?xml version="1.0" encoding="UTF-8"?><com.tikal.jenkins.plugins.multijob.MultiJobProject>
  <actions/>
  <description><!-- Managed by Jenkins Job Builder --></description>
  <keepDependencies>false</keepDependencies>
  <disabled>false</disabled>
  <blockBuildWhenDownstreamBuilding>false</blockBuildWhenDownstreamBuilding>
  <blockBuildWhenUpstreamBuilding>false</blockBuildWhenUpstreamBuilding>
  <concurrentBuild>false</concurrentBuild>
  <assignedNode>linux</assignedNode>
  <canRoam>false</canRoam>
  <properties>
    <com.coravy.hudson.plugins.github.GithubProjectProperty plugin="github">
      <projectUrl>https://myuser@mygithub.mydomain.com/myteam/myproject</projectUrl>
      <displayName/>
    </com.coravy.hudson.plugins.github.GithubProjectProperty>
  </properties>
  <scm class="hudson.plugins.git.GitSCM">
    <configVersion>2</configVersion>
    <userRemoteConfigs>
      <hudson.plugins.git.UserRemoteConfig>
        <name>origin</name>
        <refspec>+refs/pull/*:refs/remotes/origin/pr/*</refspec>
        <url>https://myuser@mygithub.mydomain.com/myteam/myproject.git</url>
        <credentialsId>github_username_with_token</credentialsId>
      </hudson.plugins.git.UserRemoteConfig>
    </userRemoteConfigs>
    <branches>
      <hudson.plugins.git.BranchSpec>
        <name>${sha1}</name>
      </hudson.plugins.git.BranchSpec>
    </branches>
    <disableSubmodules>false</disableSubmodules>
    <recursiveSubmodules>false</recursiveSubmodules>
    <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
    <remotePoll>false</remotePoll>
    <gitTool>Default</gitTool>
    <submoduleCfg class="list"/>
    <reference/>
    <gitConfigName/>
    <gitConfigEmail/>
    <extensions/>
  </scm>
  <triggers class="vector">
    <org.jenkinsci.plugins.ghprb.GhprbTrigger>
      <spec>H/5 * * * *</spec>
      <adminlist/>
      <allowMembersOfWhitelistedOrgsAsAdmin>false</allowMembersOfWhitelistedOrgsAsAdmin>
      <whitelist/>
      <orgslist/>
      <whiteListLabels/>
      <blackListLabels/>
      <cron>H/5 * * * *</cron>
      <triggerPhrase>^\s*check\s+everything\s*$</triggerPhrase>
      <skipBuildPhrase/>
      <onlyTriggerPhrase>false</onlyTriggerPhrase>
      <useGitHubHooks>true</useGitHubHooks>
      <permitAll>true</permitAll>
      <autoCloseFailedPullRequests>false</autoCloseFailedPullRequests>
      <whiteListTargetBranches>
        <org.jenkinsci.plugins.ghprb.GhprbBranch>
          <branch>staging</branch>
        </org.jenkinsci.plugins.ghprb.GhprbBranch>
      </whiteListTargetBranches>
      <extensions>
        <org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus>
          <commitStatusContext>pull request validation</commitStatusContext>
          <triggeredStatus>starting pull request validation...</triggeredStatus>
          <startedStatus>running pull request validation...</startedStatus>
          <completedStatus>
            <org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage>
              <message>All is well.</message>
              <result>SUCCESS</result>
            </org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage>
            <org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage>
              <message>Something went wrong. Investigate!</message>
              <result>FAILURE</result>
            </org.jenkinsci.plugins.ghprb.extensions.comments.GhprbBuildResultMessage>
          </completedStatus>
        </org.jenkinsci.plugins.ghprb.extensions.status.GhprbSimpleStatus>
      </extensions>
    </org.jenkinsci.plugins.ghprb.GhprbTrigger>
  </triggers>
  <builders>
    <com.tikal.jenkins.plugins.multijob.MultiJobBuilder>
      <phaseName>JIRA ID Validation</phaseName>
      <continuationCondition>SUCCESSFUL</continuationCondition>
      <executionType>PARALLEL</executionType>
      <phaseJobs>
        <com.tikal.jenkins.plugins.multijob.PhaseJobsConfig>
          <jobName>cicd-myproject-prtest-jiraid</jobName>
          <currParams>true</currParams>
          <configs/>
          <abortAllJob>false</abortAllJob>
          <enableRetryStrategy>false</enableRetryStrategy>
        </com.tikal.jenkins.plugins.multijob.PhaseJobsConfig>
      </phaseJobs>
    </com.tikal.jenkins.plugins.multijob.MultiJobBuilder>
    <com.tikal.jenkins.plugins.multijob.MultiJobBuilder>
      <phaseName>Tox Validation</phaseName>
      <continuationCondition>SUCCESSFUL</continuationCondition>
      <executionType>PARALLEL</executionType>
      <phaseJobs>
        <com.tikal.jenkins.plugins.multijob.PhaseJobsConfig>
          <jobName>cicd-myproject-prtest-unittest</jobName>
          <currParams>true</currParams>
          <configs/>
          <abortAllJob>false</abortAllJob>
          <enableRetryStrategy>false</enableRetryStrategy>
        </com.tikal.jenkins.plugins.multijob.PhaseJobsConfig>
      </phaseJobs>
    </com.tikal.jenkins.plugins.multijob.MultiJobBuilder>
    <hudson.tasks.Shell>
      <command>#!/bin/bash
echo "MY_ISSUE_ID=`echo ${ghprbPullLongDescription} | grep -e 'MY-[0-9]\+' -o`" > MY_issue.property
</command>
    </hudson.tasks.Shell>
    <EnvInjectBuilder>
      <info>
        <propertiesFilePath>$WORKSPACE/MY_issue.property</propertiesFilePath>
      </info>
    </EnvInjectBuilder>
    <hudson.plugins.jira.JiraIssueUpdateBuilder plugin="jira@3.0.0">
      <jqlSearch>issue=${MY_ISSUE_ID}</jqlSearch>
      <workflowActionName>Ready for Review</workflowActionName>
      <comment>[Unit Testing|${BUILD_URL}] Done</comment>
    </hudson.plugins.jira.JiraIssueUpdateBuilder>
    <hudson.tasks.Shell>
      <command>#!/bin/bash
curl -X POST --data-urlencode 'payload={"text": "'"${ghprbPullLink}"' is ready for review"}' <slackurl>
</command>
    </hudson.tasks.Shell>
  </builders>
  <publishers>
    <hudson.plugins.jira.JiraIssueUpdater plugin="jira@3.0.0">
      <issueSelector class="hudson.plugins.jira.selector.JqlIssueSelector">
        <jql>issue=${MY_ISSUE_ID}</jql>
      </issueSelector>
      <labels/>
    </hudson.plugins.jira.JiraIssueUpdater>
  </publishers>
  <buildWrappers>
    <hudson.plugins.ansicolor.AnsiColorBuildWrapper>
      <colorMapName>xterm</colorMapName>
    </hudson.plugins.ansicolor.AnsiColorBuildWrapper>
  </buildWrappers>
</com.tikal.jenkins.plugins.multijob.MultiJobProject>

```

# Answer
You normally would access environment variables using the following syntax:

```
env.MY_ISSUE_ID

```

Typically you'll see the following in practice regarding envvars:

```
pipeline {
  agent any
  environment {
    MY_ISSUE_ID = 'PROJECT-1234'
  }
  stages {
    stage('Build') {
      steps {
        echo env.MY_ISSUE_ID
      }
    }
  }
}

```

or

```
node{
  stage('Build') {
    withEnv(["MY_ISSUE_ID=PROJECT-1234"]) {
      echo env.MY_ISSUE_ID
    }
  }
}

```
> 1 votes

---
Tags: jenkins, jenkins-plugins, jira
---


# Question
Title: tools to gather variant information from thousands of servers and update into database
Recently, I use `ansible` to gather information periodically from thousands of servers(which already installed `collectd` to gather CPU, memory, disks, netowrking info) in our company.

For example: I need to get those information daily

* kernel version
* installed packages versions(JDK, docker, etc)
* running process info
* some commands output (lldp, docker ps, etc)
* some files content (cat /etc/some/config | grep word)

I use ansible `shell` module to execute those variant commands and parse the output into our inventory NOSQL.

By using ansible which use SSH/WINRM to access thousands servers seems inefficient.(which fine tune with more `forks`, enabled `ControlMaster` `pipeline`, etc).

I are trying to find a better way to replace ansible for those information gathering and parsing.

Below are some solutions which I am surveying.

`facebook/osquery` is flexible and powerful, but maybe not easy(comparing to bash script) to run a customized commands and parsing the output by writing an extension.

`collectd with exec plugin` to run customized scripts seems straightforward. in this way I need to deploy new scripts to all servers from day to day.

Are there some tools which is more suitable for such variant commands output gathering? and easy to `deploy`, `reporting`, or may with some `dashboard` and `API` for operation.

# Answer
> 1 votes
Doing something with `collectd` feeding into Prometheus or datadog is definitely a good idea for gathering metrics, but it doesn't help you with things like the kernel version. Handling process monitoring might be better handled with nagios or datadog as well. But for the bulk of what you're doing I'd say that it makes the most sense to stick with ansible.

What you're doing sounds very similar to the ansible-cmdb project:

> Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page (and other things) containing system configuration information.
>
> It supports multiple types of output (html, csv, sql, etc) and extending information gathered by Ansible with custom data. For each host it also shows the groups, host variables, custom variables and machine-local facts.

I've been using this for the last couple of years and it continues to evolve and improve. I hope you'll find it useful as well.

---
Tags: ansible, monitoring, systems
---


# Question
Title: Why use EXPOSE in Dockerfile -- since you can bind to all ports anyways
I can `docker run -p 3000:3000 image` *without* `EXPOSE`ing that port in the container (see below). If that's true, then why bother putting EXPOSE in the Dockerfile? Is it just for communication to image users? Because I don't know of a functional reason to EXPOSE ports if they are all bindable anyways.

---

Here are the steps showing me binding to a port in a container despite the fact it is not EXPOSEd

```
$ cat Dockerfile
FROM alpine
RUN apk add nodejs npm vim
COPY webserver /webserver
CMD [ "node", "/webserver/index.js" ]

$ docker build .
Sending build context to Docker daemon  1.931MB
Step 1/4 : FROM alpine
 ---> 11cd0b38bc3c
Step 2/4 : RUN apk add nodejs npm vim
 ---> Using cache
 ---> 4270f8bdb201
Step 3/4 : COPY webserver /webserver
 ---> Using cache
 ---> 67f4cda61ff0
Step 4/4 : CMD [ "node", "/webserver/index.js" ]
 ---> Using cache
 ---> 1df8f9024b85
Successfully built 1df8f9024b85

$ curl localhost:4400
curl: (7) Failed to connect to localhost port 4400: Connection refused

$ docker run -d -p 4400:3000 1df8f9024b85
7d0e6c56f8ad8827fe72830a30c1aac96821104b8ea111291ca39e6536aad8fd

$ curl localhost:4400
Hello World!

$

```

# Answer
Docker's EXPOSE documentation addresses this specific point:

> The `EXPOSE` instruction does not actually publish the port. It functions as a type of documentation between the person who builds the image and the person who runs the container, about which ports are intended to be published. To actually publish the port when running the container, use the `-p` flag on `docker run` to publish and map one or more ports, or the `-P` flag to publish all exposed ports and map them to high-order ports.

Pay attention to the last sentence, if you expose multiple ports then `-P` becomes useful to avoid setting multiple `-p` on the command line.
> 32 votes

# Answer
This is done for automation sake. You can have a universal command that runs `docker run -P` to start a container and the Dockerfile itself is used to specify which container exposes which port. In case you are dealing with dozens or hundreds of containers being built through a pipeline, this is quite useful. Passing external details not contained in Dockerfile together with the container through pipeline from stage to stage is quite difficult at scale.
> 5 votes

---
Tags: docker, dockerfile, ports
---


# Question
Title: Getting data from crashed docker container
I am having docker container with Ubuntu installed which was running with Nvidia docker But now I connected that HDD in a different computer. Now After typing following command in newly assembled system

> sudo docker start container

I am getting error something like Unable to create /etc/nvidiactl

It seems like it needs Nvidia GPU to run that container which I don't have. So how can I get the data which is there in the container if i am unable to start it?

# Answer
> 2 votes
So what you can do is you list all of your containers with command:

```
docker container ls -a

```

And the second field is an image name, you can run this command to get all the read-only layers with their respective directories to search for the file in:

```
docker image inspect <image_id> | jq '.[0].GraphDriver.Data.LowerDir' -r | tr ":" "\n"

```

If the file was something written inside the container replace **Lower** with **Upper**.

In case you are on Mac the /var/lib/docker is inside the virtual machine and so you have to first enter that before you can get to those directories described in the command above. You can do it using nsenter like this:

```
docker run -it --privileged --pid=host debian nsenter -t 1 -m -u -n -i sh

```

# Answer
> 0 votes
*This answer assumes a completely broken installation, i.e. no `docker xxx` commands work (though the OP did not say that explicitely). If I guessed wrong and they do work, then please refer to Jiri Klouda's answer instead.*

Docker manages all the data in `/var/lib/docker`. Look around, you should immediately be able to find it. The top level directory names are hashes, and thus look pretty garbled, but below that you have the regular file system(s) of your images and containers. All those "onionized" filesystem layers will still be there if you didn't delete them. You can search using the usual command line tools (`find` etc.), filtering by date, file size, file name, etc.

---
Tags: docker, containers, linux
---


# Question
Title: docker compose & azure key vault - passing secrets to a container on startup
I am using docker-compose to create 5 containers that run my application. Two containers need credentials retrieve from Azure Key Vault (web.config passwords to access 3rd party service).

As I am committing this to a public repo I do not want to store any azure credentials in plain text in the docker-compose.yaml file when starting up each container.

Is there a better way to do this?

I am thinking to use https://github.com/mariotoffia/FluentDocker to grab the secrets and dynamically create each container - but wonder if there is a better way?

Can I use Azure Key Vault with Docker?

Is it easier to use Docker secrets?

Or something else?

# Answer
> 3 votes
Found this --\> Passing secrets to a Docker container

Environment variables appear the preferred method - will need to run a prebuild script to fetch the secrets and then follow this

https://docs.docker.com/compose/environment-variables/

@Mods feel free to close as a duplicate

---
Tags: docker, docker-compose, azure, key-management
---


# Question
Title: Backing up an ephemeral Elasticsearch installation
I have an installation of Elasticsearch without persistence running and I need to back it up so we can move it to a persistent setup.

What are safe ways to do this and keep the data in the ES database?

It is running and I want to avoid shutting it down.

It is running in a container and I think the entrypoint starts the server. So stopping the cluster my stop my pod and get rid of all my data...

# Answer
> 1 votes
ElasticSearch can backup itself when told to do so via an API call. You just need to configure a destination, and optionally credentials, for example you make ElasticSearch create snapshots of its database in AWS S3.

https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-snapshots.html

You can also automate these calls using a schedule with the Elastic `curator` tool.

https://www.elastic.co/guide/en/elasticsearch/client/curator/5.5/snapshot.html

---
Tags: elastic
---


# Question
Title: Terraform plan -var-file="myvars.vars" does not run
I try to lean terraform for usage with our AWS environment. I have created a .tf file and terraform init was successful.

In the .tf file I have a number of variables that are populated via .var files (not **.tfvar**) because I want to assign values per instance/project etc.

The problem is that when I run the command

```
terraform plan -var-file="myvars.var"

```

all I get is the terraform usage message.

Where is my mistake?

# Answer
> 3 votes
You define variables in `variables.tf` file. However, you can define inputs to these defined variables in a `.tfvars`(recommended) or any other files.

> Values for the input variables of a root module can be gathered in variable definition files and passed together using the -var-file=FILE option.
>
> For all files which match terraform.tfvars or \*.auto.tfvars present in the current directory, Terraform automatically loads them to populate variables. If the file is located somewhere else, you can pass the path to the file using the -var-file flag. It is recommended to name such files with names ending in .tfvars. more...

Make sure you are running `terraform plan` from the directory where the variable file resides.

# Answer
> 0 votes
The problem was a typo in the var-file I had a double " in a value:

```
varname = ""value"

```

---
Tags: terraform
---


# Question
Title: what environment does a build "RUN" run in?
When docker build does a RUN command, what "environment" is it running in?

Explaining the question, does it run at build time, or when the container starts? Is it using the operating system mentioned in the FROM?

# Answer
> 3 votes
In the Dockerfile, the `RUN` step is performed during the `docker build` process. The operating system is shared with the host (containers != VMs). The filesystem and rest of the configuration used to define the temporary container is based on the state of the image resulting from the previous step. Each step in the Dockerfile extends the image with additional metadata and filesystem layers.

The `RUN` step itself is performed inside a temporary container. The output of the `RUN` step is any filesystem changes that exist in the container at the time the command exits successfully. Changes to the shell (like a `cd` command or setting an environment variable inside the shell), spawned processes, and changes to files inside a Volume are all discarded. The changes to the container filesystem are captured as a filesystem layer that is included in the resulting image.

---
Tags: docker, dockerfile
---


# Question
Title: Dockerfile COPY command with three input parameters
I've encountered a Multi-stage Build example, where the first stage builds a ngnix server, and the second stage creates a ngnix service from that image.

The second stage looks like this:

```
...
# second stage
FROM scratch
COPY --from=build /etc/passwd /etc/group /etc/
COPY --from=build /usr/local/nginx /usr/local/nginx
COPY index.html /usr/local/nginx/html
...

```

This is a working example.

Now, notice the first COPY command. It received three parameters(!)

What is the meaning of 3 parameters in the COPY command? do you have a reference to documentation regarding this input?

Notice that the second COPY command uses 2 parameters, which I assumed are src and dest folders.

# Answer
> 2 votes
As per this docker documentation the copy instruction may have multiple sources but only one destination.
Your first copy is coping two file `/etc/passwd` and `/etc/group` to `/etc`.

---
Tags: docker, dockerfile, builds, nginx
---


# Question
Title: How do you handle the huge number of possible permutations of Dockerfiles / Teraform files?
We have adopted Docker and Terraform as the foundation forour devOps. We have a basic version of it up and running today.

Dockerfiles live in each projects git repository and Terraform files are more centralized.

The number of projects we manage is increasing as do the number of environments, data seeding To make things more interesting we are also moving toward splitting services up into microservices where that makes sense.

So for each project, we have options like:

* What destination are you targetting (dev local, dev cloud, test (several variations), staging prod)
* What version / tag of the source code should it build from.
* What data should be seeded?
* Who should be notified?
* What infrastructure should it use? (We want to be able to allow "prod infrastructure) to be available for developers to spin up in dev cloud for debugging and testing) (along with several other combination)

There are a few other parameters we are considering and all in all the number of possible permutations become very high.

There is no way we can cleanly keep order of that many Dockerfiles.

So I am looking for a tool that helps us keep track of all the different configurations and makes it easy to find them.

Ideally, inside my head, I am picturing a wizard-based web app, where you enter in the parameters and it executes the necessary scripts automagically. This would also contain logs of what has been done, what environments are running and a few other things.

So far I havent had much luck. I know a lot of companies must have solved it and I must be thinking about it wrong, or there is a big opportunity here.

Is there such a tool? How do large companies solve this?

# Answer
> 5 votes
So let's imagine you have 20 microservices, each in a separate repository. Each microservice needs to be deployed into 5+ environments: dev local, dev cloud, test, staging, prod. Understandably all configuration will be different in each environment. However, you don't need to put it into the Dockerfile - in fact you don't put any configuration into the Dockerfile. Your app gets all required parameters from environment variables, or may be from a config file, if it gets too complex.

So you would have one Dockerfile per project, and if your developers need to spin it up locally, they will just use docker-compose, which has all environment variables set up, and maybe a config file mounted as volume. In stage envs the configuration will be totally different, but the Docker image will be the same. Your app will need to be able to read this configuration at start up time, and bail out early if something doesn't work.

Read https://12factor.net/config for some inspiration on how to configure your app.

Now, provisioning the infrastructure with Terraform would follow the same approach. You would have a generic module which provisions a set of some resources, and it would accept a bunch of parameters which would depend on environment. You can use workspaces to manage this quite easily.

As for the last question, large companies solve this with a CI/CD system. This can be Jenkins, or Spinnaker, or any of the other tools, whichever you prefer.

---
Tags: docker, terraform, configuration-management
---


# Question
Title: Using Google IAM for GKE service web access
I am hosting an application on GKE and would like to be able to let users from my organization access this application from the web. I would like them to be able to log-in using their Google Account IAM credentials.

Is there a way to configure a service exposing the clusters web endpoint such that to access this service the user simply needs to login with their google account?

For example, when testing a service I can easily do a web-preview in the cloud-shell and then access the web application in my browser.

Is there a way to configure this such that any users authorized in my organization can access the web interface of my application?

# Answer
> 0 votes
Okay, I managed to make it work perfectly. But it took a few steps. I am including the manifest here that is required to setup the IAP using an ingress. It requires a few things which I listed in the manifest below. Hopefully this can help others since I could not find a single source that had all of this put together. Essentially all you need to do is run `kubectl apply -f secure-ingress.yaml` to make everything work (as long as you have all the depenedencies) and then you just need to configure your IAP as you like it.

---

**`secure-ingress.yaml`**

```
# Configure IAP security using ingress automatically
# requirements: kubernetes version at least 1.10.5-gke.3
# requirements: service must respond with 200 at / endpoint (the healthcheck)
# dependencies: need certificate secret my-secret-cert
# dependencies: need oath-client secret my-secret-oath (with my.domain.com configured)
# dependencies: need external IP address my-external-ip
# dependencies: need domain my.domain.com to point to my-external-ip IP
# dependencies: need an app (deployment/statefulset) my-app
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: my-secure-ingress
  namespace: default
  annotations:
    kubernetes.io/ingress.class: "gce"
    kubernetes.io/ingress.allow-http: "false"
    kubernetes.io/ingress.global-static-ip-name: my-external-ip
spec:
  tls:
  - secretName: my-secret-cert
  backend:
    serviceName: my-service-be-web
    servicePort: 1234
---
kind: Service
apiVersion: v1
metadata:
  name: my-service-be-web
  namespace: default
  annotations:
    beta.cloud.google.com/backend-config:
      '{"default": "my-service-be-conf"}'
spec:
  type: NodePort
  selector:
    app: my-app
  ports:
    - protocol: TCP
      port: 1234
      targetPort: 1234
      name: my-port-web
---
apiVersion: cloud.google.com/v1beta1
kind: BackendConfig
metadata:
  name: my-service-be-conf
  namespace: default
spec:
  iap:
    enabled: true
    oauthclientCredentials:
      secretName: my-secret-oath

```

---
Tags: kubernetes, google-cloud-platform, google-compute-engine
---


title: Bamboo equivalent to Jenkins color

text:
# Question
Title: Bamboo equivalent to Jenkins color
I'm currently building out a radiator screen to bridge a few different areas of our CI\CD pipeline, pulling in metrics as well as build statuses. We're also migrating from Jenkins to Bamboo

In Jenkins, when you do a rest call to the ***job/api/json*** it, one of the response fields is a color, which acts as an initial health. Great for colour coding, especially when things go red!

However, as far as I can see, there isn't a bamboo equivalent. The two calls I'm using so far in bamboo are

```
/result/{projectKey}-{buildKey}/{buildNumber : ([0-9]+)|(latest)}

/result/status/{projectKey}-{buildKey}-{buildNumber : ([0-9]+)}

```

Any clues or pointers as to where I can get the info from?

p.s. Writing this app in React \ Node

# Answer
Bamboo doesn't have a color status like Jenkins per say, but it does have a very similar build status indicated by a circle with a symbol within it. This status is available for both the entire build plan and for an individual build.

For your GET request, you will want to use the same api calls that you are using to get the *buildstate* value to get a SUCCESSFUL or FAILED state.
> 2 votes

---
Tags: jenkins, bamboo, node.js
---


# Question
Title: lc_collate values for database "postgres" do not match: old "C.UTF-8", new "en_US.UTF-8"
I am working on testing a method for postgres 9.3 to 10 upgrade. I currently am at the dry run phase and ran into a few issues. The current blocking one is `lc_collate values for database "postgres" do not match: old "C.UTF-8", new "en_US.UTF-8"`, I get this when I run the below command:

```
time ${PGBINNEW}/pg_upgrade --old-bindir=${PGBINOLD} --new-bindir=${PGBINNEW} --old-datadir=${PGDATAOLD} --new-datadir=${PGDATANEW} --old-port=${PGPORT} --new-port=$5433 --jobs=$(nproc | awk '{ {print ($1 > 4) ? 4 : ($1 == 1 ? 1 : $1 - 1)} }') --link --username=postgres --check

```

Any ideas on what might be happening here?

# Answer
> 0 votes
I solved this thanks to Reddit post. It was fairly simple when combined with another part of my testing.

I used the code `{PGBINNEW}/initdb --pgdata="$PGDATANEW" --encoding=UTF8 --no-locale` to reinit the DB AFTER I did the below command:

```
sudo rm -rf ${PGBASEDIR}/data10

```

What happened was I cleared the old DB init config and then put in the config I needed without the localization of the database

---
Tags: upgrade, postgresql
---


# Question
Title: jenkins fatal: ambiguous argument, unknown revision or path not in the working tree
In jenkins pipeline, execute git command :

```
git log --format=%B --no-merges origin/master..origin/${currentBranch}

```

jenkins output error :

```
fatal: ambiguous argument 'origin/master..origin/XOPS-42_AddResultToJIRA': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'

```

but in my local computer i execute above git command, it work well, how to resolve above error in jenkins

# Answer
From the git documentation around the command `git rev-list`:

**`git rev-list A...B`**

Try your command syntax like the above (i.e. `origin/master...origin/${currentBranch}`)
> 1 votes

---
Tags: jenkins, git
---


# Question
Title: Control order of container termination in a single pod in Kubernetes
I have two containers inside one pod. One is my application container and the second is a CloudSQL proxy container. Basically my application container is dependent on this CloudSQL container.

The problem is that when a pod is terminated, the CloudSQL proxy container is terminated first and only after some seconds my application container is terminated.

So, before my container is terminated, it keeps sending requests to the CloudSQL container, resulting in errors:

```
could not connect to server: Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432

```

That's why, I thought it would be a good idea to specify the order of termination, so that my application container is terminated first and only then the cloudsql one.

I was unable to find anything that could do this in the documentation. But maybe there is some way.

# Answer
> 2 votes
> But maybe there is some way.

Order of termination (or start) as such is not something you can easily define, but you can try a workaround with Container Lifecycle Events.

General idea is that on a `PreStop` handler for proxy container you can check if app container finished working (say localhost:appport is not active or whatever api check you can muster to determine when app pod is down) and then continue with proxy pod shutdown normally.

---
Tags: kubernetes
---


# Question
Title: How to keep configuration files automatically during apt-get upgrade or install?
When performing and `apt-get update; apt-get upgrade -y` on a server I encountered the message:

```
Setting up sudo (1.8.16-0ubuntu1.5) ...

Configuration file '/etc/sudoers'
==> Modified (by you or by a script) since installation.
==> Package distributor has shipped an updated version.
What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
The default action is to keep your current version.
*** sudoers (Y/I/N/O/D/Z) [default=N] ?
Setting up ubuntu-minimal (1.361.1) ...

```

I know that `debconf` can be used to configure packages, like MySQL, to answer questions to prompts.

Which Linux Program/Module/Function/etc should I use to prepare an answer to this prompt?

I would really like to know a solution that be performed via Ansilbe if one exists. In addition to a BASH script.

Please note: ALL my distributions are done via IaC and thus automation is critical.

# Answer
There are options you can pass through apt-get to dpkg that will handle the config choices. We usually do something like:

`apt-get install -y --no_install_recommends -o Dpkg::Options::='--force-confdef' -o Dpkg::Options::='--force-confold' <package_name>`

When using ansible you can do:

```
  apt:
    pkg: "{{ item }}"
    only_upgrade: yes
    install_recommends: no
    force: yes
    dpkg_options: 'force-confdef,force-confold'
    state: latest
  with_items:
    - <package_name>

```

For further details on the ansible **apt** module options look here
> 17 votes

---
Tags: ansible, linux, automation, bash
---


title: Single Page Apps and DevOps Pipeline?

text:
# Question
Title: Single Page Apps and DevOps Pipeline?
I am new to the whole DevOps process.

As a software developer with some experience in building single-page applications (SPAs) using Webpack which already takes care of the unit checks, code validation, etc, how does a SPA development process fit into the DevOps workflow/pipeline?

Do you do all the code checks on the client-side and then push compiled code into the pipeline process via a `Static` buildpack (ie, `/dist` folder) or do I push my `build` files into Git with a Jenkins file for CI/CD and then go through the pipeline with a server-side `NodeJS` buildpack?

I am wondering how much work needs to be done locally before going through the pipeline. I don't like the idea of having extra dependencies to rely on (it seems there are always issues on the server side preventing apps from getting deployed -- outdated buildpacks, errors in servers, downtimes, etc). I wonder if simply offloading all that to the client side is a good way to get an app out quicker (just push the static files via a Static buildpack).

Thanks

# Answer
> 0 votes
Here is how we handle the pipeline process:

Push your code to github ignoring the **build** folder.

The Jenkins will then pull the code from github and run tests against it. After all the tests have passed, it would build the application for production, creating the **build** folder. The **build** folder would then get deployed.

# Answer
> 0 votes
When starting from scratch first build a set of scripts (bash and golang) which start with a `git pull` to get the fresh code for a given branch then the script issues all the command line steps you can do yourself to compile the code into executable bundles. This includes issuing Docker commands to cut fresh docker images for the various microservices. Then a script launches all the various Docker containers. Now the app can get fresh code pushed to domains where people can login and use the app from their client (mobile or browser) whenever a developer requests a domain needs a refresh.

The next step for us was to setup a github webhook server on each server for each domain which subscribes to all `git push` performed by our developers. If a given git push matches the github code branch a domain is running the webhook would trigger launch of the above script to compile then deploy the app

All of above is just automation to replace what someone was originally manually issuing themselves at the command line. As need appeared a new step was issued manually until it got crafted into above tooling.

---
Tags: node.js, javascript
---


# Question
Title: Scheduling a Docker Container or Installing Scheduler in Container
I have an interesting question related to scheduling my Docker Container.

I have a virtual machine with a docker image inside of it. I can spin up a container from the image that has an entrypoint (jar).

I am in a dilemna on how I should schedule the jobs to run against the jar.

Here are my 2 options:

1) If I run a cron on the VM then I can call the docker run commands to produce a container and run different jobs against the jar. The issue here is that once a single job is run, the container automatically terminates and the second job has no previous histroy of the previously run jobs. I believe theres a way to keep a container running forever with an entry point but I haven't explored/tested this yet.

2) I can install a cron inside of the container and through the Dockerfile provide the cron for updating - however, the issue here is how would I launch the container and keep it running for the cron jobs to run?

If you need any further details please do let me know.

Thanks!

# Answer
You don't need cron. Most server OS-es (CoreOS, RedHat, CentOS, Ubuntu, etc.) these days run systemd, and systemd unit files can run your container to execute the job, for example:

Two unit files. One is mytimedstuff.service and mytimedstuff.timer. The .timer defines when the service should be executed.

In mytimedstuff.service

```
[Unit]
Description=Executes mystuff
After=default.target docker_network_apps.service
Requires=default.target docker_network_apps.service

[Service]
Type=oneshot
User=root
ExecStartPre=/usr/bin/docker pull mystuff:stable
ExecStartPre=-/bin/bash -c "/usr/bin/docker rm -f mystuffcontainer 2>/dev/null"
ExecStart=/bin/bash -c "/usr/bin/docker run --name mystuffcontainer mystuff:stable mystuff"

```

In mytimedstuff.timer:

```
[Unit]
Description=My stuff runs at 00:10 sharp.

[Timer]
OnCalendar=00:10:00

[Install]
WantedBy=multi-user.target

```
> 5 votes

# Answer
My understanding is that you want to run multiple services in your container.

The best practice to run several jobs is to create several containers for your different processes and manage them through docker compose this allows you to define in one file all the containers needed for your app and launch in one single command.

Otherwise, docker docs provide two simple solutions for your situation.

One thing to remember when doing Docker: one container, one service.
> 2 votes

---
Tags: docker, containers
---


# Question
Title: How to install a private managed Kubernetes on-premise?
Where do you start in case you need something like a managed Kubernetes solution to setup a multi-tenant orchestration solution on-premises?

* Any entity with its own container infrastructure has an access to the centrally managed orchestration service
* Entities **do not see** each other's containers

Note: if "managed Kubernetes" is kind of reserved term by AWS, maybe it's better to say "orchestration as a service"?

# Answer
Take a look at `OpenShift` container platform (https://www.openshift.com/products/container-platform/). They seem to provide on-premise options. You might want to start with `OpenShift Origin` \- the community edition (https://github.com/openshift/origin) to test for free if it fits your plan.
> 3 votes

# Answer
I don't think "Managed Kubernetes" is reserved for AWS (EKS), there is Microsoft's Azure Kubernetes Service (AKS) and Google Kubernetes Engine (GKE) also, besides Red Hat's OpenShift. Those work as SaaS (or KaaS if you will).

OpenShift has an on-premise version you can try called MiniShift in the OpenShift.org site, while there are other alternatives as Apache Mesos based Mesosphere DC/OS or Rancher with RancherOS containers
> 1 votes

---
Tags: kubernetes, cloud, orchestration, organization
---


# Question
Title: CodeBuild - multiple input artifacts in AWS Pipeline
I am building an AWS CodePipeline which depends on two source repositories. If the 'CodeBuild' step is specified with multiple Input artifacts, then you get the error: `.. declares 2 input artifacts which is more than the maximum count`. Documentation has a strict requirement for CodeBuild to have only a single input artifact.

Lambdas, on another hand, do not have this limitation. I've been looking for a way how a Lambda could "merge" two artifacts into a single one which I can pass to the CodeBuild then. Unfortunately, information on this subject is quite limited.

Does anyone have a working CodePipeline Lambda which would merge two Artifacts into one?

**Note**: Documentation about How to access artifacts.

# Answer
As of Sep 4 2018 Amazon added support for multiple input sources:

https://aws.amazon.com/about-aws/whats-new/2018/08/aws-codebuild-adds-ability-to-create-build-projects-with-multiple-input-sources-and-output-artifacts/

From documentation:

```
{
"name": "sample-project",
"source": {
  "type": "S3",
  "location": "bucket/sample.zip"
},
"secondarySources": [
  {
    "type": "CODECOMMIT",
    "location": "https://git-codecommit.us-west-2.amazonaws.com/v1/repos/repo"
    "sourceIdentifier": "source1"
  },
  {
    "type": "GITHUB",
    "location": "https://github.com/awslabs/aws-codebuild-jenkins-plugin"
    "sourceIdentifier": "source2"
  }
],

```

---

```
version: 0.2

phases:
  build:
    commands:
      - cd $CODEBUILD_SRC_DIR_source1
      - touch file1
      - cd $CODEBUILD_SRC_DIR_source2
      - touch file2

artifacts:
  secondary-artifacts:
    artifact1:
      base-directory: $CODEBUILD_SRC_DIR_source1
      files:
        - file1
    artifact2:
      base-directory: $CODEBUILD_SRC_DIR_source2
      files:
        - file2

```
> 3 votes

# Answer
We've had exactly the same problem - our build stage needs to use information from two separate locations.

To solve this in a generic manner, I've created two general purpose Lambdas - one that will merge artifacts, and one that will extract specific files from an artifact. In both cases a new artifact is created that can be passed on to a subsequent CodePipeline stage.

See https://github.com/tooltwist/codepipeline-artifact-munge

If it does not meet your exact requirement, it should be easy enough to tweak to do what you need.
> 6 votes

# Answer
I have actually solved the problem by cloning repository directly in codebuild:

Pass GitHub token:

```
      - { Name: GITHUB_TOKEN, Value: {Ref: GitHubToken } }
      - { Name: GITHUB_BRANCH, Value: {Ref: GitHubBranch } }

```

\# Execute git clone in install step:

```
      - git clone --single-branch --depth=1 -b $GITHUB_BRANCH https://gitorgname:$GITHUB_TOKEN@github.com/gitorgname/reponame.git  src/reponame

```

I found that this solution is easier to implement, does not rely on Lambdas and even though performs two pulls, works quite reliably.
> 4 votes

# Answer
You can review CodeBuild. The BuildScpec file can be put in the CodeBuild project.
> -1 votes

---
Tags: amazon-web-services, build-pipeline
---


# Question
Title: Which password database system work with ansible?
Which password database system work with ansible to manage password of network devices and servers?

# Answer
> 3 votes
Any system would do. You might need to write a wrapper. For example:

```
> cat get-passwd-01.yml
- hosts: localhost
  gather_facts: no
  tasks:
    - command: ~/bin/get-passwd user9
      register: password
    - debug:
        msg: "passwd:{{ password.stdout }}"

```

.

```
> ansible-playbook get-passwd-01.yml | grep msg
    "msg": "passwd: T4gpJtZ69R5cZD9zQh"

```

---
Tags: ansible, passwords
---


# Question
Title: How to assign static ip in docker for Windows for linux container?
I want to assign static ip to my docker container in docker for Window for linux container. I have achieved same in linux container using macvlan but macvlan doesn't support for docker for Windows. Does any one have work around how can i assign static ip to my linux container so it directly expose to outside world without port mapping. I have also tried to use transperent driver in window container with LCOW enabled but as my application need privileged and bind mount to cgroup window container can not be used. Note:I have specific scenario to use static ip as port mapping is not working for my application.

# Answer
You need create user define network

```
docker network create -d transparent --subnet=10.1.1.0/24 --gateway=10.1.1.1 TransparentNet3

```

Then you need run container with key --ip

```
docker run -it --name nano03 --network=TransparentNet3 --ip 10.1.1.1 someimage:sometag

```
> 1 votes

---
Tags: docker, containers, windows
---


# Question
Title: Which SCM solutions exist for managing mainframe software?
Imagine a company using mainframes to run (part of) their (often mission critical) business applications, and using z/OS (also known as OS/390, or MVS).

What are the typical softwares they use to facilitate Software Change and Configuration management, for software that is deployed/used on those mainframes?

# Answer
> 9 votes
From my own experience, these are some the typical softwares packages:

All of these packages can manage, more or less out-of-the-box, anything that's stored in regular "PDS" components (a typical file structure used in z/OS).

When it comes to a company evaluating which one best fits for them, it often boils down to these criteria:

* IBM SCLM is perceived as free of any license/maintenance fees (actually it's included in the z/OS license, which itself is not free). So if no dedicated budget is available, then this is often the software package that gets selected (better then no package at all). If there is a budget, then this one is often the one that doesn't make it to the shortlist.
* CA Endevor has the highest install base. Its key strength, IMO, is the way that you can trace down for each executable how it has been compiled/linked using which version of which building blocks (copybooks, etc).
* SERENA ChangeMan ZMF's install base is quite a bit below the one from CA Endevor. Some of its key strengths are:

+ the notion of "packaging" related software changes, which is at the core of it.
    + its capabilites to deploy softwares towards physically remote sites.
* Compuware ISPW is like the "new kid in town" (as compared to the CA Endevor or SERENA ChangeMan ZMF alternative). It is commonly perceived as the solution where "any custom SCM requirements can be implemented with it, with a relatively low effort to do so".

Looking at it from an architecture standpoint, then SERENA ChangeMan ZMF and Compuware ISPW appear to have the most open architecture, which is what you'll need if you want to tune it to make it manage software components written in a 4GL language that is (what some call) more exotic, as compared to 3GL languages such as COBOL, PL/I, etc. I.e. because the software components are stored in file systems that are not stored in standard PDSs. Some examples of those languages are:

Attention: having an "open architecture" is great to have it fit your custom requirements (the sky is the limit). However when it comes to upgrading to new releases, it comes with a price to get those custom requirements upgraded also.

**Note**: rather accidentally, during a CA Endevor training for SERENA ChangeMan ZMF experts, we discovered that CA Endevor and SERENA ChangeMan ZMF appear to have the same roots (from somewhere in the late 1980s ...). For those who are a bit familiar with both of them: go check what the functionality is of these utility programs, with similarly named names ... (you'll be shocked ...):

* PGM=CONWRITE versus PGM=CMNWRITE.
* PGM=CONPRINT versus PGM=SERPRINT.

# Answer
> 2 votes
The answers above assume managing source code for z/OS has to be different from any other platform. The answer 10 years ago might have been the above. But z/OS has evolved along with the z hardware and it is no longer separate. You can use a modern source code manager such as Git for all your source code including any COBOL or PL/I or assembler you might have. Git was updated to handle the ASCII to EDBCIC translation if you get the port from Rocket Software. It's still free and open source, they just did the compile to run on the platform. Having your z/OS source code in the same SCM also allows you to have your test cases and other artifacts along side them. You might be surprised at the number of open source tools you can use with z/OS.

If you have a DevOps pipeline it probably just works with z/OS as well, as an example Jenkins runs on the platform. With a current PTF to z/OS you can even store you build artifacts in Artifactory or Nexus as you do any other platform. The process and practices that are used on other platforms also work for z/OS so there is no reason it has to separate or different.

# Answer
> 0 votes
There's a company from Belgium who've been in the SCM (now they label their product as DevOps) market for over 12 years. But since they're not a giant like IBM or CA they are less known.

However they have their product (IKAN ALM) running at large banks and insurance companies, mostly as a replacement for Changeman. They support Mainframe and Distributed, this means companies will be able to manage Mainframe and, for instance Java development (and deployment) using the same tool.

They have a brand called BlueBridge, which is actually their main product already configured for Mainframe.

---
Tags: scm, mainframe, software-recommendation
---


# Question
Title: Which plugin should I use to get started with Jenkins to manage mainframe components in PDS format?
Assume somebody familiar with DevOps and mainframes, but new to Jenkins, wants to get started with Jenkins, e.g. to:

After some kind of (minimal) Jenkins installation and configuration got completed, the actual question becomes "**`Which of the typical Jenkins plugins, if any, would be needed?`**". From the various Jenkins plugins that appear to make sense in my case, it seems that these are possible candidates (quotes are from the linked pages):

* IBM zOS Connector.

> ... provides its functions via FTP connection to IBM z/OS LPAR. You can configure your SCLM project on z/OS and then check for the changes via Jenkins.
>
> Features include:
>
> + Submission of user JCL job (with optional log collected upon finish).
>     + Introduction of SCLM as SCM for your projects allowing to checkout SCLM changes.
>     + The ability to build SCLM projects currently can be performed only via 'Submit zOS Job' build action.
* Compuware Source Code Download for Endevor, PDS, and ISPW Plugin.

> ... allows Jenkins users to download Endevor, PDS, or ISPW members from the mainframe to the PC. Source can then be accessed on the PC, for example, for SonarQube analysis and reporting.

Since the 1st plugin is about SCLM (which is all about PDSes), and the 2nd plugin explicitly states support for PDSes (in its name), I trust both are possible candidates.

So assuming my shortlist is complete (is it?), what are the pros and cons of either one that should help me decide which one is best for my case?

Ps: Something like a "Jenkins Kickstart" package doesn't seem to exist (at least I've not found it yet).

# Answer
> 2 votes
I set up jenkins to manage mainframe tables. Essentially utilizing groovy script plugin. I wrote code that handles movement of files on the mainframe and call them though the plugin. Essentially leveraging the mainframes API to connect and the vsc's commands to move the files.

# Answer
> 2 votes
Jenkins slave runs in unix system services of z/OS and can run any script or function from there. We use the Git plugin to pull the source code onto z/OS then you have different choices of how to work with it. You can use IBM Dependency Based Build which is Groovy ported to z/OS plus some extensions to specifically work with native z/OS functions such as copy to PDS (or PDSE) or copy back to zFS, as well as the ability to run MVS, TSO, ISPF commands and submit JCL.

Also with the new Zowe announcement Zowe.org for a new open source project on z/OS you also have access to a number of REST based APIs work with files and perform other tasks on z/OS such as calling zOSMF.

# Answer
> 1 votes
In general, there are two ways to automate actions from Jenkins:

1. Install a plugin for Jenkins and use it in steps to automate tasks.
2. Write a script (shell, python, other) and call it from a step to automate tasks.

There are drawbacks / benefits to both approaches. While #1 - the plugin approach - gives you out of the box functionality and does not require you to script, it may be rigid and limited in its abilities. #2 - the scripting approach - requires you to write scripts, as well as maintain them, but may provide much more flexibiity and freedom in automating things.

The scripting approach generally uses a lot of CLIs - aws cli, docker cli, etc.

If #2 - the scripting approach - is viable and you're comfortable with writing shell scripts or other scripts like python to interact with the mainframe, then Brightside can help.

Here's an example of using Brightside (free CLI) to interact with PDS on z/OS from a shell script:

```
    #!/bin/sh
    echo 'bright files create classic "USER.PUBLIC.SAMPLE.PDS"'
    bright files create classic "USER.PUBLIC.SAMPLE.PDS"
    echo ''

    echo 'bright files download all-members "USER.LIB.SAMP" -d MyPDS'
    bright files download all-members "USER.LIB.SAMP" -d MyPDS
    echo ''

    echo 'bright files upload dir-to-pds MyPDS "USER.PUBLIC.SAMPLE.PDS"'
    bright files upload dir-to-pds MyPDS "USER.PUBLIC.SAMPLE.PDS"
    echo ''

    echo 'bright files list all-members "USER.PUBLIC.SAMPLE.PDS"'
    bright files list all-members "USER.PUBLIC.SAMPLE.PDS"
    echo ''

    echo 'bright files delete data-set "USER.PUBLIC.SAMPLE.PDS(ALLOCLIB)" -f'
    bright files delete data-set "USER.PUBLIC.SAMPLE.PDS(ALLOCLIB)" -f
    echo ''

    echo 'bright files list all-members "USER.PUBLIC.SAMPLE.PDS"'
    bright files list all-members "USER.PUBLIC.SAMPLE.PDS"
    echo ''

    echo 'bright files delete data-set "USER.PUBLIC.SAMPLE.PDS" -f'
    bright files delete data-set "USER.PUBLIC.SAMPLE.PDS" -f
    echo ''

```

Here's the output from that script:

Easy enough to call these scripts from Jenkins and parse the output for whatever purpose.

Full disclosure: I work for CA who created Brightside but we do use this CLI internally for the exact purpose of integrating our mainframe application testing with Jenkins.

# Answer
> 1 votes
@Pierre.Vriens, wrt your ask about “investigate the feasibility of managing data stored in personal files (PDSes, i.e. partitioned data sets) on the mainframe.”, you can use zOS Management Facility (zOSMF) which comes with the z/OS operating system to access APIs for that. In Sujay's reply, he suggests using Brightside and that tool taps into the zOSMF APIs to simplify that task. So then if you want to tie Jenkins to the Mainframe to manage Partitioned Data Sets (PDS) and Data Sets (PS), you can use these zOSMF APIs or you could use Brightside.

In the spirit of disclosure, I work as a UX Designer for CA Technologies who builds Brightside.

---
Tags: jenkins, jenkins-plugins, mainframe
---


# Question
Title: Bitbucket Pull Request Builder Jenkins job is not getting invoked
I have enabled web hooks pull request in Bitbucket like below -

And my Jenkins job configurations are as follows -

Bitbucket hooks are getting invoked as soon as I create pull request but Jenkins job is not getting invoked. Can anyone help please?

# Answer
It took me a while to understand this as well as I had experience with Bitbucket Server and not Bitbucket Cloud which is different.

You do not need to enable the webhook for the Bitbucket Pull Request Builder plugin in Jenkins.

Just ensure you set the *RepositoryName* and *RepositoryOwner* properly from the URL of your repo as such: `https://bitbucket.org/RepositoryOwner/RepositoryName`

Once you have that set, a new PR or commit to branch of existing PR will trigger your job.

P.S. There's a bug in the plugin where you can't use git for other repos such as workflow library or pipeline from scm. Git will try to use the same hash to checkout all repos which will fail the build.

P.P.S. The webhook in Bitbucket cloud uses a POST with Json data so that can't be used with the git plugin in Jenkins unless you don't care about the hash or branch.

EDIT: One last note, I've since moved to use the multibranch pipeline with bitbucket branch plugin and that works great with the webhook as well as allows us to use a workflowlib library
> 1 votes

# Answer
if that is the error you're getting, it is clearly an issue with BitBucket Authentication as that is in the stack trace... the specific API call which is failing is displayed and you can see it is in the authentication process... as to why it is running successfully when invoked manually - it would have to do with a configuration upstream somewhere - check the format of your repo and make sure you use this pattern:

https://user:password@bitbucket.org/user/project\_name.git
> 0 votes

---
Tags: jenkins, bitbucket
---


# Question
Title: Aggregating JUnit test results from multiple projects
We have a Jenkins setup which utilizes multiple projects for running various suites of tests for our application.

There is one 'parent' project, which is triggered by Git commits. This project in turn builds our 'child' project. Each child projects runs a series of defined tests.

Each project is setup with JUnit to output onto the build page the results of the tests.

In order to get a comprehensive view of commits across our entire project, you need to go to the build page of EACH of our 'child' projects to see the results.

What I would like to do is aggregate ALL the JUnit reports, from builds triggered by out 'parent' project across all the 'child' projects to one place so we can view all the test results together.

Does anyone have experience with something like this?

# Answer
> 4 votes
The JUNIT plugin has a built-in publisher for aggregating test results called "Aggregate Downstream Test Results". You need to make sure that the 'parent' and 'child' jobs are linked together with a downstream/upstream relation. This is usually done by archiving an artifact in the parent build and copying it to the children builds using the 'Archive Artifact' publisher and the 'Copy Artifact' buildstep respectively.

# Answer
> -1 votes
If you are able to use gradle, maybe use something like mentioned here:

Multi-module projects or this

Also you'll get coverage :)

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: How to deploy multiple dependent micro services
We want to deploy multiple microservices on AWS ECS.

The issue that we need to solve is how to deploy them in an atomic way:

Let says we have the front-end services which requires the user service version 2.0.

How to make sure the user service is available before the front-end one is deployed.

We have the idea to replicate the whole infra, deploy everything then make it available. But it seems to me complex. How do you make these spare instances, deploy the services, and make the switch?

We are using two ALB and target groups for each services (dynamic port mappings).

# Answer
Such question could be an indication of a poor architectural slicing into microservices. From What are Microservices?:

> These services are **built around business capabilities** and **independently deployable** by fully automated deployment machinery.

The key point missed in such case would be their **independently deployable** aspect.

The point could also be simply misunderstood: it doesn't necessarily mean that each microservice should be capable to perform its entire functionality by itself, it just means that it's able to gracefully bail when some other microservice its functionality depends on isn't available, maybe keeping track of "todo" stuff for later execution, when that microservice becomes available.

Seen from a different perspective: handling unavailability of external service dependencies should actually be part of each microservice functionality - so one could consider the microservice fully functional even when its runtime dependencies aren't satisfied :)

It's worth noting, I think, that in the above comments I'm referring to microservices unavailability regardless of its reason, be it deployment dependency order or something else (outages, for example).

Another thing to consider would be the finer difference between deploying the microservices and bringing them into action by switching the traffic to them. Switching traffic is often faster than the deployment itself, so the availability at the entire system level can be increased by first deploying all microservices and only after that performing the traffic switch for all of them.
> 8 votes

# Answer
We had a similar situation wherein one of our applications had to be built, deployed and be up & running before the second one was. This was because application A has some functionality that should be executed prior to it being referenced via application B. The way we accomplished this was by building it into a single delivery pipeline. That way, all the dependencies were deployed either sequentially or parallely as needed. We also had our database scripts in sequence along with rollback capabilities, just in case. Rewording to your case, you could deploy your api prior to your front-end service. Hope this helps.
> 1 votes

# Answer
The solution here is to deploy the micro services using the blue/green deployment pattern.
> 1 votes

---
Tags: docker, deployment, microservices, amazon-ecs
---


# Question
Title: What's the difference between checkin and checkout?
When teaching SCM classes to students that are new to Software Configuration Management, it happens that a question comes up like "**`What's the difference between checkin and checkout?`**".

And a variation of it is that such students get confused about these SCM concepts (they understand them as the other way around).

So what kind of metaphor can you use to explain this crucial SCM concept to such audience?

# Answer
To explain something to anybody, try to compare it to something they are (hopefully) familiar with already.

So that's why I just answer such question like so:

> Think of it as arriving at a place to stay (a hotel, a resort, etc):
>
> * the very *first* thing you do (when you arrive) is to **`checkin`**.
> * the very *last* thing you do (when you leave) is to **`checkout`**.
>
> A similar SCM concept applies when you want to apply changes to software components ... except that ***it applies the other way around***:
>
> * the very *first* thing you do (when you start) is to **`checkout`** (or think of it like borrowing it).
> * the very *last* thing you do (when you finish) is to **`checkin`** (or think of it like giving it back).

**Note**: this applies to centralized systems (such as the ones used in mainframe environments ...). In systems such as the "*`checkout`*" concept has a completely different meaning (which IMO is also why in those systems there is hardly any confusion about both concepts).
> 8 votes

# Answer
It's important to note that the terms "checkin" and "checkout" have different meanings depending on the type of SCM system.

Centralized systems like TFVC, Subversion, and Clearcase use "exclusive" checkouts. This is like Pierre's book borrowing metaphor, where only one user can have a file checked out at one time.

Distributed systems like git have a "checkout" command, but it means something completely different. `git checkout` is used to switch between branches when working with a local repository.
> 6 votes

# Answer
For centralized systems, think of it like a technical library. (might be a stretch of the imagination how this hypothetical library functions...)

If you are an author of a document, you might `checkout` the library copy, make changes, return it `check it back in` to the library for the world to see.

This can become an issue if the library has digital copies, and I `checkout` a document, someone else also `checks out` a document, we both make changes, there will be a conflict (merge conflict) that might be difficult to resolve. When then the initial "fix" for this is exclusive `checkout` functionality...

---

Of course for large projects the chances of a critical merge conflict issue is reduced (people will be working on different parts of the system) so `checkout`/`checkin` is not needed nearly as much. And since distributed systems by design somewhat require good merge functionality, along with many other benefits, that concept doesn't really exist in git and other DVCS
> 4 votes

# Answer
With the *SCM repository* as the main subject then'

* **checkout** is getting changes **out** from the local or remote repository (into your local working directory).
* **checkin** is putting changes back **into** the the local or remote repository (from your local working directory).
> 4 votes

# Answer
> * **Checkout** is an exclusive lock on modifying a branch of object in a repository.
> * **Checkin** is a release of exclusive lock.

There are two kinds of source control systems depending on what is the smallest unit of branching.

1) **Per repository** branching (CVS, SVN, GIT, Perforce, ... etc)

In products where you branch the entire repository, **checkout** will usually either create or enable modifications to local branch (copy) of the entire repository. In those products **checkin** is often unused and becomes a part of **commit** operation, which is at once **checkout** of remote branch, applying of local **patch** and **checkin** of remote branch in single operation. You do not **checkin** your local branch as it is permanently checked out. (Note: *In GIT you don't commit to remote branch, you push your local commit to it. Strictly syntactic difference.*)

2) **Per object** branching (ClearCase, AccuRev, Oracle ADE)

In products where you branch individual objects, like directories, files, etc. The concept of **checkout** and **checkin** applies per object per branch. You will lock the object to modify it with **checkout** and release it with **checkin**. In those products you often work on a private branch where locks do not hold anyone from working and at the time of merge of your local branch into a shared branch, the objects are also **checkedout** on shard branch (main, master, feature branch, etc) conflicts of merge are resolved and object is **checkedin** on the shared branch. This allows multiple people to "commit" at the same time to shared branch as long as they do not modify the same objects.
> 1 votes

---
Tags: terminology, scm
---


# Question
Title: Do not show all docker containers to a particular user
I want to use one server to host multiple docker-containers. I want to give another user a possibility to manage new docker-containers, but I do not want to give him access to all other containers, which I started, since they may contain sensitive data.

Is there a way to create a secure configuration which doesn't require creating virtual machine?

# Answer
> 1 votes
Hi and welcome to DevOps SE!

To enable Role-Based Access Control (RBAC) for Docker Engine, you need the UCP (Universal Control Pane) available through Docker EE.

https://success.docker.com/article/rbac-example-overview

A free alternative could be also the Kubernetes module ***rbac.authorization.k8s.io***.

Side note: To control docker commands available to a user, use ***AuthZPlugin*** for the Docker engine (there are open source implementations of this API).

---
Tags: docker, virtualization, role-based-access-control
---


# Question
Title: Dealing with ambiguity of the terms publish vs. deploy
DevOps terminology sometimes originates in products as introduced by their authors.

As a result, there are some standard actions described by ambigous terms. In mixed teams and large heterogeneous environments we need to agree on terms to undertstand what we actually do.

* *publish*: term specific to Ant/Ivy meaning putting an artefact into binary repository. Also, some people would say "publish content updates to website".
* *deploy*: while "in the Maven world" this word describes putting an artefact to binary repository afaik, more "opsy" people deploy their servers also without having Maven in place.

How to deal with this ambiguity (ideally, please refer to "success stories")? An explorative talk on terms meaning is not always welcome in delivery oriented stressful environments; maybe there are already better terms around?

For terms definitions, please refer to sources.

# Answer
> 10 votes
The terms are actually quite well defined and some authors are simply making a mess of things by using a term like 'publish' for deployment, in cases where publishing a website for example constitutes for their own process also a deployment, as there is no extra deployment step after publishing. Sometimes they use word publishing both for uploading and publishing together as all of their artifacts are public. That creates unnecessary confusion. Just to make it clear, here are the definitions:

Environment Definitions:

> * **build environment** \- Environment setup for build process.
> * **runtime environment** \- Environment setup for execution process (test/staging/production).

Artifact Lifecycle Stages:

> * **building** \- process of creating an artifact in build environment
> * **uploading** \- moving artifact from build environment to repository.
> * **publishing** \- making artifact in repository public.
> * **downloading** \- moving artifact from repository to runtime environment
> * **installing** \- applying artifact to runtime environment
> * **deploying** \- using artifact in runtime environment
> * **delivery** \- making artifact (effects) available to customer

Note: *People often use any of those terms to mean one or more of the previous terms together.*

# Answer
> 1 votes
First things first: let us make clear what we are talking about. We are dwelling on the action of **storing an artifact in an artifact repository**.

I personnaly would rather avoid using *deploy*, because:

* **publish** implies that you make an artifact available for others to consume. The term seems to do the job. It weakens a bit when you push the artifact to a *private* repository. You would then do a *private publication*. It's a lesser evil. It still makes sens.
* **deploy** is definitely not appropriate. Pushing a single artifact in a single repository is in no way a *deployement*. Deploy implies some sort of spreading, distributing an artifact on several environments or machines. What we are dealing with, here, is a pre-requisite to deploying; not deploying itself.

**\[Success?\] Story**

For what it's worth, last time I was faced with naming the job that pushes the artifacts to the artifact repository, I was working in a Kanban context. The team came to an agreement to call the job

```
version

```

because every package that was built following a merge request was a potential release candidate, therefore it was *versioned* then pushed.

**After Thoughts**

As of today, I would use the term **publish** (possibly privately) laking of a better term.

Still I don't find this satisfactory. Please feel free to leave your thoughts in the comments below!

---
Tags: terminology, culture, term-disambiguation
---


title: Jenkins job using VPN?

text:
# Question
Title: Jenkins job using VPN?
Is there a way to connect to a VPN for the duration of a Jenkins job? (freestyle or pipeline, I can work with either).

I scoured the plugin list figuring someone might have developed something, but the only page found was this OpenConnect page with no actual code developed, so that seems to be out.

We're running Jenkins on Ubuntu AWS boxes, if that changes anything. I'd prefer the ability to connect and disconnect in a single job, rather than making everything run on VPN.

# Answer
I think that it's very difficult, if not impossible, to achieve something like this.

The reason is that once the machine running Jenkins is connected to the VPN it is also effectively disconnected from the original network it was connected to, which means Jenkins on that machine will loose connection with the rest of your Jenkins setup.

The only way that might work would be if using completely standalone one-machine Jenkins setups.
> 2 votes

# Answer
If you want to VPN for a jenkins task you would want to use the OpenConnect plugin as you mentioned.

Here are the steps to do that:

On linux (debian based)

```
sudo apt-get install openconnect

```

Add the following lines to the bottom of /etc/sudoers (ubuntu configuration)

```
jenkins ALL=NOPASSWD:/usr/sbin/openconnect*
jenkins ALL=NOPASSWD:/bin/kill*

```

The you will get an option in the Jenkins run task called:

```
Connect to Cisco AnyConnect VPN

```

Can also be found here: https://stackoverflow.com/questions/35151072/deploy-with-jenkins-to-vpn/43357784#43357784
> 3 votes

---
Tags: jenkins, vpn
---


# Question
Title: Chef resource to register gitlab runner automatically
We would like to automate installation of gitlab runner using chef. we are successful with it. Now, the Issue is regarding registering gitlab runner with chef resource. we have gone through chef resource. and official cookbook https://gitlab.com/chef-platform/gitlab-ci-runner but looks a bit complicated for us. we wonder if we have any pre-built chef resources to register gitlab runner. if yes, can we know how we can register with a example resource. Additionally, I have tried looking at resource https://github.com/TYPO3-cookbooks/site-gitlabcirunnertypo3org/blob/master/recipes/default.rb

```
name = "#{node['fqdn']}-docker"
  gitlab_ci_runner name do
      options(
        registration_token: registration_token,
        url: node['site-gitlabcirunnertypo3org']['gitlab_url'],
        executor: 'docker',
        :'docker-image' => 'docker:1',
        :'tag-list' => ['docker']
      )
end

```

But I dont understand how it is registering the runner. Thanks for help!

# Answer
> 1 votes
Your second link just use the official cookbook. This official cookbook still uses the LWRP syntax, so you need to check in the provider directory for the resource action code -- which for register is basically just two execute commands (extract from the file linked above):

```
converge_by("Register runner #{new_resource.description}") do
  execute "#{env} gitlab-runner register --non-interactive #{options}"
  # giving time to gitlab-ci to manage the request
  execute "sleep #{new_resource.sleep}"
end

```

So mostly, your exemple ends up calling the `gitlab-runner register` command with the options defined in the options hash (with a little transformation of the hash into command parameters).

There's no 'core' chef resource to register gitlab runners out of using an `execute` resource to run the command.

A basic example of this resource is present in the cookbook Readme:

```
gitlab_ci_runner 'my runner' do
  options({
    registration_token: '1234567890',
    url: 'http://gitlab-ci.myinstance'
  })
end

```

---
Tags: chef, gitlab, gitlab-ci-runner
---


# Question
Title: Using Artifactory plugin in Jenkins declarative pipeline
I am using Jenkins Declarative Pipeline to automate my build process. We want to publish our artifacts to a remote JFrog repository only if certain conditions (Sonar,Checkmarx) pass.

After a bit of research, I found that Artifactory plugin is useful for this. But I am unable to find any document on how to integrate in declarative pipeline. Below is the code snippet from Jenkinsfile

```
stages{

    stage('Pre-Build'){
        steps{

             script{
                def server = Artifactory.server 'LocalJfrog'
                def rtGradle = Artifactory.newGradleBuild()
                rtGradle.resolver server: server, repo: 'gradle-dev-local'
                rtGradle.deployer server: server, repo: 'gradle-release-local'
                rtGradle.useWrapper = true
            }

        }
    }
}

```

The conditional publish is not possible with above code as I cannot reuse the **server** variable even if I disable auto publish.

# Answer
> 3 votes
You can have conditionals in your declarative pipeline by using the **when**-block inside a stage. There is a plugin called "environment injector" which lets you set variables outside of the pipeline-script which is nice. Also if you put the step below the other steps, it won't execute if they fail.

```
 when {
    environment name: 'pushArtifact', value: 'true'
  }
  steps{
     //push artifact
  }

```

# Answer
> 2 votes
I think your issue is rooted in the **server** variable not being reusable outside the pre-build stage block.

In Jenkins declarative, you can define variables like that using the `script { ... }` block, but once you leave the stage those variables are inaccessible to other to stages.

With the previous suggestions, I'd recommend this:

House the artifactory deploy code to a shared library.

```
*gradle_artifactory.groovy*

    def call (Map parameters = [:]) {  //optional parameters mapping

        def server = Artifactory.server 'LocalJfrog'
        def rtGradle = Artifactory.newGradleBuild()
        rtGradle.resolver server: server, repo: 'gradle-dev-local'
        rtGradle.deployer server: server, repo: 'gradle-release-local'
        rtGradle.useWrapper = true
        def buildInfo = rtGradle.run rootDir: "projectDir/", buildFile:
            'build.gradle', tasks: 'clean artifactoryPublish'

    }

```

Then to keep your declarative pipelines `D.R.Y`

```
@Library('my-shared-lib')
...
stage('Artifactory Upload') {
    when { <some expression with sonarqube/checkmarx> }
    steps {
        script {
            gradle_artifactory()
        }
    }
}

```

refs:

https://jenkins.io/doc/book/pipeline/syntax/#when

https://jenkins.io/doc/book/pipeline/shared-libraries/

# Answer
> 1 votes
If you want to embed logic in the Jenkinsfile, the declarative syntax may not be the best method as it is not always easy to reflect it in the code.

If you switched to the Jenkinsfile scripted pipeline, you would be able to define and use conditions in an easier way.

---
Tags: jenkins, jenkins-plugins, jenkins-pipeline, jenkinsfile
---


title: Dynamically grep with less/tail

text:
# Question
Title: Dynamically grep with less/tail
when tailing a log file, we might use:

```
tail -f foo.log | grep bar

```

if we want to change the grep argument, we have to ctrl-c and then change the argument and then restart the command.

my question is - is there a way to dynamically change what we are grepping for without having to kill the tail/grep commands?

Also looking for a way to do this with `less` instead of `tail` if possible.

# Answer
I implemented this in node.js - I don't know of a unix utility that can dynamically change grep. Here is how it works:

*before dygrep:*

```
many-child-procs > logfile.log                  # in terminal 1
tail -f logfile.log | grep <expression>          # in terminal 2

```

*after dygrep:*

```
many-child-procs | dygrep                    # in terminal 1

```

then you connect to dygrep with a tcp-client and control which expressions are filtered on.

The following node.js process reads from stdin, and uses a tcp connection to listen for expressions to filter on. For each line of stdin, it only forwards that to stdout if it matches one of the current regular expressions stored in memory.

```
#!/usr/bin/env node
'use strict';

import * as readline from 'readline';
import * as net from 'net';
import chalk from 'chalk';
import {createParser} from "./json-parser";
import log from './logger';

let port = 4900;

const rl = readline.createInterface({
  input: process.stdin.resume()
});

const regex = new Map<string, RegExp>();

rl.on('line', l => {

  if (regex.size < 1) {
    process.stdout.write(l + '\n');
    return;
  }

  for (let [k, v] of regex) {
    if (v.test(l)) {
      process.stdout.write(chalk.magenta(' (filtered) ') + l + '\n');
      break;
    }
  }

});

interface IncomingTCPMessage {
  command: {
    add: string,
    remove: string,
    list: boolean,
    removeall: boolean
  }
}

const server = net.createServer(s => {

  const sendMessage = (m: any) => {
    s.write(JSON.stringify({message: m}) + `\n`);
  };

  s.pipe(createParser()).on('data', (d: IncomingTCPMessage) => {

    if (!d.command) {
      log.error('No "command" field was found:', d);
      return ''
    }

    const c = d.command;

    if (c.list) {
      sendMessage({regexes: Array.from(regex.keys()).map(k => ({regex: regex.get(k), str: k}))});
      log.info('Listing all regex for the client.');
      return;
    }

    if (c.removeall) {
      regex.clear();
      sendMessage(`Cleared all regex.`);
      log.info('Cleared all regex.');
      return;
    }

    if (c.add) {
      regex.set(c.add, new RegExp(c.add));
      sendMessage(`Added regex: ${c.add}.`);
      log.info('Added regex:', c.add);
      return;
    }

    if (c.remove) {
      regex.delete(c.remove);
      sendMessage(`Deleted regex: ${c.remove}.`);
      log.info('Removed regex:', c.remove);
      return;
    }

    log.error('No matching field was found:', d);
    log.info('Regex:', regex);

  });

});

server.listen(port);

```

to see the complete code, see: https://github.com/ORESoftware/dygrep
> 3 votes

---
Tags: logging
---


# Question
Title: What's the practice to perform a releasing with Git?
I'm using **GitLab CE** with **Maven** project, and try to follow the Environment branches with GitLab flow and The 11 Rules of GitLab Flow

The step for releasing is as the following: -

```
feature   +---   +---          +---
         /    \ /    \        /    \
master  + ---- + ---- + ---- + ---- +
                        \            \
qa-tmp (chage ver to RC) +---         + ---
                             \             \
qa      + ------------------- + ---------- +
                             (v1.0-RC1)   (v1.1-RC1)
                              \            \
prod-tmp (chage ver to final)  +---         + ---
                                   \             \
prod    + ------------------------ + ------------ +
                                 (v1.0)         (v1.1)

```
1. Create a `feature` branch and merge into the `master` so that it is ready to be released
2. Create the `qa-tmp` branch e.g. `release-v1.0-RC1` from `master`

2.1 Update the pom version to `1.0-RC1`

2.2 Merge the `release-v1.0-RC1` to `qa`

2.3 Tag `v1.0-RC1` from merged `qa` branch.
3. When it pass the QA, create `prod-tmp` branch e.g. `release-v1.0` from tag `v1.0-RC1`

3.1 Update the pom version to `1.0`

3.2 Merge the `release-v1.0` to prod

3.3 Tag `v1.0` from merged `prod` branch

3.4 Bump `master` to version `1.1-SNAPSHOT`
4. Develop the version `1.1-SNAPSHOT` in the features & master when it is ready to be released, repeat the step 2 and 3.

The trouble is since the `version` is changed during merging the `qa` branch and `prod` branch, when I perform a next release there is a *merge conflict* for the `version` inside the pom. This cause me to resolve this conflict manually.

I'm not sure if I'm doing something wrong or misunderstanding about the releasing procedure or not.

Furthermore, do we have any practice to make this releasing to be an automation, e.g. via script?

Could you please help to advise?

# Answer
> 4 votes
Consider to substitute the version in POM by the `git describe` upon build without hardcoding the version in your repository, you could do this with e.g. `sed` from below `version_slug`.

```
export git_version=`git describe --tags --always`
export version_slug="$(sed s/-/\./g <<<$git_version)"

```

and then substitute the version\_slug in pom.

In that way for each commit and/or tag in gitlab, you'll create a new version, with out actually changing the repository.

this is what i did for a ruby project: jekyll-plantuml-url, where the version is not coded in the source, but is set upon build, and after testing deployed as the git tag.

```
  script:
   - bundle install
   - rake set_git_tag
   - rake
   - gem build jekyll-plantuml-url.gemspec

```

Furthermore don't understand the need of all those branches, with CI/CD we continuously build, test and deploy, that means before accepting a merge to master you have a deployment for QA to do the testing, code review, functional testing ...etc.

Next add QA as approver for a merge to master, thus make it mandatory that QA signs of a merge request to master and integrate quality in the development cycle or have a master that is always ready for release.

One could opt to work from master and for a release tag the master branch with a version tag, as per above example.

OR

like gitlab does it, to create a branch for each release, cherry pick the features from master that are ready or planned and then tag the branch with a version tag.

where as a version tag creates a deployment for a commit, release or a RC.

---
Tags: git, gitlab
---