# Question
Title: Pod's state "pod has unbound immediate PersistentVolumeClaims" when deploy Jenkins with Helm chart
I try to deploy Jenkins in local Kubernetes cluster with Helm from repo stable/jenkins:lts. But after deployment I get pod which has status "`pod has unbound immediate PersistentVolumeClaims`". PVC has status "bound" without any errors.

My environment: virtual machine with Linux Ubuntu 18.x where there is installed microk8s. When I deploy I change only node type and persistence size.

Helm chart config:

```
NAME:   inclined-eagle
REVISION: 1
RELEASED: Fri Jan 11 21:35:47 2019
CHART: jenkins-0.28.6
USER-SUPPLIED VALUES:
Master:
  ServiceType: NodePort
Persistence:
  Size: 4Gi

COMPUTED VALUES:
Agent:
  AlwaysPullImage: false
  Component: jenkins-slave
  CustomJenkinsLabels: []
  Enabled: true
  Image: jenkins/jnlp-slave
  ImageTag: 3.27-1
  NodeSelector: {}
  PodRetention: Never
  Privileged: false
  resources:
    limits:
      cpu: 200m
      memory: 256Mi
    requests:
      cpu: 200m
      memory: 256Mi
  volumes: null
Master:
  AdditionalConfig: {}
  AdminUser: admin
  CLI: false
  CSRF:
    DefaultCrumbIssuer:
      Enabled: true
      ProxyCompatability: true
  Component: jenkins-master
  CustomConfigMap: false
  DisabledAgentProtocols:
  - JNLP-connect
  - JNLP2-connect
  ExtraPorts: null
  HealthProbeLivenessFailureThreshold: 12
  HealthProbeReadinessPeriodSeconds: 10
  HealthProbes: true
  HealthProbesLivenessTimeout: 90
  HealthProbesReadinessTimeout: 60
  HostNetworking: false
  Image: jenkins/jenkins
  ImagePullPolicy: Always
  ImageTag: lts
  Ingress:
    Annotations: {}
    ApiVersion: extensions/v1beta1
    TLS: null
  InitScripts: null
  InstallPlugins:
  - kubernetes:1.14.0
  - workflow-job:2.31
  - workflow-aggregator:2.6
  - credentials-binding:1.17
  - git:3.9.1
  LoadBalancerSourceRanges:
  - 0.0.0.0/0
  Name: jenkins-master
  NodeSelector: {}
  NumExecutors: 0
  OverwriteConfig: false
  PodAnnotations: {}
  ServiceAnnotations: {}
  ServiceLabels: {}
  ServicePort: 8080
  ServiceType: NodePort
  SlaveListenerPort: 50000
  SlaveListenerServiceAnnotations: {}
  SlaveListenerServiceType: ClusterIP
  Tolerations: {}
  UsePodSecurityContext: true
  UseSecurity: true
  resources:
    limits:
      cpu: 2000m
      memory: 2048Mi
    requests:
      cpu: 50m
      memory: 256Mi
NetworkPolicy:
  ApiVersion: networking.k8s.io/v1
  Enabled: false
Persistence:
  AccessMode: ReadWriteOnce
  Annotations: {}
  Enabled: true
  Size: 4Gi
  mounts: null
  volumes: null
backup:
  annotations:
    iam.amazonaws.com/role: jenkins
  destination: s3://nuvo-jenkins-data/backup
  enabled: false
  env:
  - name: AWS_REGION
    value: us-east-1
  extraArgs: []
  image:
    repository: nuvo/kube-tasks
    tag: 0.1.2
  resources:
    limits:
      cpu: 1
      memory: 1Gi
    requests:
      cpu: 1
      memory: 1Gi
  schedule: 0 2 * * *
rbac:
  install: false
  roleBindingKind: ClusterRoleBinding
  roleKind: ClusterRole
  roleRef: cluster-admin
  serviceAccountName: default

HOOKS:
---
# inclined-eagle-ui-test-1mhfp
apiVersion: v1
kind: Pod
metadata:
  name: "inclined-eagle-ui-test-1mhfp"
  annotations:
    "helm.sh/hook": test-success
spec:
  initContainers:
    - name: "test-framework"
      image: "dduportal/bats:0.4.0"
      command:
      - "bash"
      - "-c"
      - |
        set -ex
        # copy bats to tools dir
        cp -R /usr/local/libexec/ /tools/bats/
      volumeMounts:
      - mountPath: /tools
        name: tools
  containers:
    - name: inclined-eagle-ui-test
      image: jenkins/jenkins:lts
      command: ["/tools/bats/bats", "-t", "/tests/run.sh"]
      volumeMounts:
      - mountPath: /tests
        name: tests
        readOnly: true
      - mountPath: /tools
        name: tools
  volumes:
  - name: tests
    configMap:
      name: inclined-eagle-jenkins-tests
  - name: tools
    emptyDir: {}
  restartPolicy: Never
MANIFEST:

---
# Source: jenkins/templates/secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: inclined-eagle-jenkins
  labels:
    app: inclined-eagle-jenkins
    chart: "jenkins-0.28.6"
    release: "inclined-eagle"
    heritage: "Tiller"
type: Opaque
data:

  jenkins-admin-password: "enkyZlZqdVdqMw=="

  jenkins-admin-user: "YWRtaW4="
---
# Source: jenkins/templates/config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: inclined-eagle-jenkins
data:
  config.xml: |-
    <?xml version='1.0' encoding='UTF-8'?>
    <hudson>
      <disabledAdministrativeMonitors/>
      <version>lts</version>
      <numExecutors>0</numExecutors>
      <mode>NORMAL</mode>
      <useSecurity>true</useSecurity>
      <authorizationStrategy class="hudson.security.FullControlOnceLoggedInAuthorizationStrategy">
        <denyAnonymousReadAccess>true</denyAnonymousReadAccess>
      </authorizationStrategy>
      <securityRealm class="hudson.security.LegacySecurityRealm"/>
      <disableRememberMe>false</disableRememberMe>
      <projectNamingStrategy class="jenkins.model.ProjectNamingStrategy$DefaultProjectNamingStrategy"/>
      <workspaceDir>${JENKINS_HOME}/workspace/${ITEM_FULLNAME}</workspaceDir>
      <buildsDir>${ITEM_ROOTDIR}/builds</buildsDir>
      <markupFormatter class="hudson.markup.EscapedMarkupFormatter"/>
      <jdks/>
      <viewsTabBar class="hudson.views.DefaultViewsTabBar"/>
      <myViewsTabBar class="hudson.views.DefaultMyViewsTabBar"/>
      <clouds>
        <org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud plugin="kubernetes@1.14.0">
          <name>kubernetes</name>
          <templates>
            <org.csanchez.jenkins.plugins.kubernetes.PodTemplate>
              <inheritFrom></inheritFrom>
              <name>default</name>
              <instanceCap>2147483647</instanceCap>
              <idleMinutes>0</idleMinutes>
              <label>inclined-eagle-jenkins-slave </label>
              <nodeSelector></nodeSelector>
                <nodeUsageMode>NORMAL</nodeUsageMode>
              <volumes>
              </volumes>
              <containers>
                <org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
                  <name>jnlp</name>
                  <image>jenkins/jnlp-slave:3.27-1</image>
                  <privileged>false</privileged>
                  <alwaysPullImage>false</alwaysPullImage>
                  <workingDir>/home/jenkins</workingDir>
                  <command></command>
                  <args>${computer.jnlpmac} ${computer.name}</args>
                  <ttyEnabled>false</ttyEnabled>
                  # Resources configuration is a little hacky. This was to prevent breaking
                  # changes, and should be cleanned up in the future once everybody had
                  # enough time to migrate.
                  <resourceRequestCpu>200m</resourceRequestCpu>
                  <resourceRequestMemory>256Mi</resourceRequestMemory>
                  <resourceLimitCpu>200m</resourceLimitCpu>
                  <resourceLimitMemory>256Mi</resourceLimitMemory>
                  <envVars>
                    <org.csanchez.jenkins.plugins.kubernetes.ContainerEnvVar>
                      <key>JENKINS_URL</key>
                      <value>http://inclined-eagle-jenkins:8080</value>
                    </org.csanchez.jenkins.plugins.kubernetes.ContainerEnvVar>
                  </envVars>
                </org.csanchez.jenkins.plugins.kubernetes.ContainerTemplate>
              </containers>
              <envVars/>
              <annotations/>
              <imagePullSecrets/>
              <nodeProperties/>
              <podRetention class="org.csanchez.jenkins.plugins.kubernetes.pod.retention.Default"/>
            </org.csanchez.jenkins.plugins.kubernetes.PodTemplate></templates>
          <serverUrl>https://kubernetes.default</serverUrl>
          <skipTlsVerify>false</skipTlsVerify>
          <namespace>default</namespace>
          <jenkinsUrl>http://inclined-eagle-jenkins:8080</jenkinsUrl>
          <jenkinsTunnel>inclined-eagle-jenkins-agent:50000</jenkinsTunnel>
          <containerCap>10</containerCap>
          <retentionTimeout>5</retentionTimeout>
          <connectTimeout>0</connectTimeout>
          <readTimeout>0</readTimeout>
          <podRetention class="org.csanchez.jenkins.plugins.kubernetes.pod.retention.Never"/>
        </org.csanchez.jenkins.plugins.kubernetes.KubernetesCloud>
      </clouds>
      <quietPeriod>5</quietPeriod>
      <scmCheckoutRetryCount>0</scmCheckoutRetryCount>
      <views>
        <hudson.model.AllView>
          <owner class="hudson" reference="../../.."/>
          <name>All</name>
          <filterExecutors>false</filterExecutors>
          <filterQueue>false</filterQueue>
          <properties class="hudson.model.View$PropertyList"/>
        </hudson.model.AllView>
      </views>
      <primaryView>All</primaryView>
      <slaveAgentPort>50000</slaveAgentPort>
      <disabledAgentProtocols>
        <string>JNLP-connect</string>
        <string>JNLP2-connect</string>
      </disabledAgentProtocols>
      <label></label>
      <crumbIssuer class="hudson.security.csrf.DefaultCrumbIssuer">
        <excludeClientIPFromCrumb>true</excludeClientIPFromCrumb>
      </crumbIssuer>
      <nodeProperties/>
      <globalNodeProperties/>
      <noUsageStatistics>true</noUsageStatistics>
    </hudson>
  jenkins.model.JenkinsLocationConfiguration.xml: |-
    <?xml version='1.1' encoding='UTF-8'?>
    <jenkins.model.JenkinsLocationConfiguration>
      <adminAddress></adminAddress>
      <jenkinsUrl>http://inclined-eagle-jenkins:8080</jenkinsUrl>
    </jenkins.model.JenkinsLocationConfiguration>
  jenkins.CLI.xml: |-
    <?xml version='1.1' encoding='UTF-8'?>
    <jenkins.CLI>
      <enabled>false</enabled>
    </jenkins.CLI>
  apply_config.sh: |-
    mkdir -p /usr/share/jenkins/ref/secrets/;
    echo "false" > /usr/share/jenkins/ref/secrets/slave-to-master-security-kill-switch;
    yes n | cp -i /var/jenkins_config/config.xml /var/jenkins_home;
    yes n | cp -i /var/jenkins_config/jenkins.CLI.xml /var/jenkins_home;
    yes n | cp -i /var/jenkins_config/jenkins.model.JenkinsLocationConfiguration.xml /var/jenkins_home;
    # Install missing plugins
    cp /var/jenkins_config/plugins.txt /var/jenkins_home;
    rm -rf /usr/share/jenkins/ref/plugins/*.lock
    /usr/local/bin/install-plugins.sh `echo $(cat /var/jenkins_home/plugins.txt)`;
    # Copy plugins to shared volume
    yes n | cp -i /usr/share/jenkins/ref/plugins/* /var/jenkins_plugins/;
  plugins.txt: |-
    kubernetes:1.14.0
    workflow-job:2.31
    workflow-aggregator:2.6
    credentials-binding:1.17
    git:3.9.1
---
# Source: jenkins/templates/test-config.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: inclined-eagle-jenkins-tests
data:
  run.sh: |-
    @test "Testing Jenkins UI is accessible" {
      curl --retry 48 --retry-delay 10 inclined-eagle-jenkins:8080/login
    }
---
# Source: jenkins/templates/home-pvc.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: inclined-eagle-jenkins
  labels:
    app: inclined-eagle-jenkins
    chart: "jenkins-0.28.6"
    release: "inclined-eagle"
    heritage: "Tiller"
spec:
  accessModes:
    - "ReadWriteOnce"
  resources:
    requests:
      storage: "4Gi"
---
# Source: jenkins/templates/jenkins-agent-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: inclined-eagle-jenkins-agent
  labels:
    app: inclined-eagle-jenkins
    chart: "jenkins-0.28.6"
    component: "inclined-eagle-jenkins-master"
spec:
  ports:
    - port: 50000
      targetPort: 50000

      name: slavelistener
  selector:
    component: "inclined-eagle-jenkins-master"
  type: ClusterIP
---
# Source: jenkins/templates/jenkins-master-svc.yaml
apiVersion: v1
kind: Service
metadata:
  name: inclined-eagle-jenkins
  labels:
    app: inclined-eagle-jenkins
    heritage: "Tiller"
    release: "inclined-eagle"
    chart: "jenkins-0.28.6"
    component: "inclined-eagle-jenkins-master"
spec:
  ports:
    - port: 8080
      name: http
      targetPort: 8080

  selector:
    component: "inclined-eagle-jenkins-master"
  type: NodePort
---
# Source: jenkins/templates/jenkins-master-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inclined-eagle-jenkins
  labels:
    heritage: "Tiller"
    release: "inclined-eagle"
    chart: "jenkins-0.28.6"
    component: "inclined-eagle-jenkins-master"
spec:
  replicas: 1
  strategy:
    type: Recreate
  selector:
    matchLabels:
      component: "inclined-eagle-jenkins-master"
  template:
    metadata:
      labels:
        app: inclined-eagle-jenkins
        heritage: "Tiller"
        release: "inclined-eagle"
        chart: "jenkins-0.28.6"
        component: "inclined-eagle-jenkins-master"
      annotations:
        checksum/config: 4eebbff9187d5eadf249ff0106423ec0b571659e014ed75f2abef5619afc24f9
    spec:
      securityContext:
        runAsUser: 0
      serviceAccountName: "default"
      initContainers:
        - name: "copy-default-config"
          image: "jenkins/jenkins:lts"
          imagePullPolicy: "Always"
          command: [ "sh", "/var/jenkins_config/apply_config.sh" ]
          resources:
            limits:
              cpu: 2000m
              memory: 2048Mi
            requests:
              cpu: 50m
              memory: 256Mi

          volumeMounts:
            -
              mountPath: /var/jenkins_home
              name: jenkins-home
            -
              mountPath: /var/jenkins_config
              name: jenkins-config
            -
              mountPath: /var/jenkins_plugins
              name: plugin-dir
            -
              mountPath: /usr/share/jenkins/ref/secrets/
              name: secrets-dir
      containers:
        - name: inclined-eagle-jenkins
          image: "jenkins/jenkins:lts"
          imagePullPolicy: "Always"
          args: [ "--argumentsRealm.passwd.$(ADMIN_USER)=$(ADMIN_PASSWORD)",  "--argumentsRealm.roles.$(ADMIN_USER)=admin"]
          env:
            - name: JAVA_OPTS
              value: ""
            - name: JENKINS_OPTS
              value: ""
            - name: ADMIN_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: inclined-eagle-jenkins
                  key: jenkins-admin-password
            - name: ADMIN_USER
              valueFrom:
                secretKeyRef:
                  name: inclined-eagle-jenkins
                  key: jenkins-admin-user
          ports:
            - containerPort: 8080
              name: http
            - containerPort: 50000
              name: slavelistener
          livenessProbe:
            httpGet:
              path: "/login"
              port: http
            initialDelaySeconds: 90
            timeoutSeconds: 5
            failureThreshold: 12
          readinessProbe:
            httpGet:
              path: "/login"
              port: http
            initialDelaySeconds: 60
            periodSeconds: 10
          # Resources configuration is a little hacky. This was to prevent breaking
          # changes, and should be cleanned up in the future once everybody had
          # enough time to migrate.
          resources:

            limits:
              cpu: 2000m
              memory: 2048Mi
            requests:
              cpu: 50m
              memory: 256Mi

          volumeMounts:
            -
              mountPath: /var/jenkins_home
              name: jenkins-home
              readOnly: false
            -
              mountPath: /var/jenkins_config
              name: jenkins-config
              readOnly: true
            -
              mountPath: /usr/share/jenkins/ref/plugins/
              name: plugin-dir
              readOnly: false
            -
              mountPath: /usr/share/jenkins/ref/secrets/
              name: secrets-dir
              readOnly: false
      volumes:
      - name: jenkins-config
        configMap:
          name: inclined-eagle-jenkins
      - name: plugin-dir
        emptyDir: {}
      - name: secrets-dir
        emptyDir: {}
      - name: jenkins-home
        persistentVolumeClaim:
          claimName: inclined-eagle-jenkins

```

pod's log:

```
container "inclined-eagle-jenkins" in pod "inclined-eagle-jenkins-65c8f989d-lglfc" is waiting to start: PodInitializing

```

There is also warning message among events:

```
AssumePod failed: pod 6bf15a11-168a-11e9-b322-0800277359c7 is in the cache, so can't be assumed

```

# Answer
In my case problem was related with droped IPtables FORWARD policy. I had investigated it with the help of "microk8s.inspect" command. InitContainer couldn't get access to the Internet and it stopped deployment of whole pod with main container. Solving was easy - "sudo iptables -P FORWARD ACCEPT" and redeployment with Helm.
> 0 votes

---
Tags: jenkins, kubernetes, deployment, helm
---


# Question
Title: How to automatically remove k8s namespaces every week?
In our company multiple stakeholders create test systems in k8s that they forget to remove. The aim is to remove namespaces that are older than one week automatically. Is there a certain operator that is able to do this?

# Answer
I recommend that you have an expire-date as label or annotation on the namespaces used for test. Then you can have a Job to clean-up expired namespaces.
> 3 votes

# Answer
While I don't know of any operator that allows for this directly, it can be done with a bash one liner with `awk`, `xargs`, `sed`, and gnu `date`. I've taken it from this post and modified it to do namespaces, and to not delete the `default` or `kube-*` namespaces.

```
kubectl get namespaces -o go-template --template '{{range .items}}{{.metadata.name}} {{.metadata.creationTimestamp}}{{"\n"}}{{end}}' | awk '$2 <= "'$(date -d '1 week ago' -Ins --utc | sed 's/+0000/Z/')'" && $1 !~ /(default|kube.*)/ { print $1 }' | xargs --no-run-if-empty kubectl delete namespace

```

Be careful using this though, it's worth running it without the final xargs call to verify that it is selecting the namespaces you are interested in.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Bash Tokenization Errors on Jenkins JNLP Slave
Why might the below bash commands give me an error on a Jenkins JNLP slave in Kubernetes cluster? Each work on a local machine.

---

**Error:** `Syntax error: redirection unexpected`

```
IFS='/' read -r -a branch <<< "${1}"

```

---

**Error:** `Syntax error: "(" unexpected (expecting "}")`:

```
IFS='/'
branch=($1)

```

---

# Environment Details

# Answer
I cannot say this is a the best answer so I do hope someone can provide a better clarification.

It seems the default shell for `sh()` in Jenkins is Bourne. By updating the `sh()` with the correct shell this allows the below usage:

```
sh'''#!/bin/bash
    IFS='/'
    branch=($1)
    # Use it.
'''

```

This doesn't resolve the the redirection error. Also I would think by updating the default shell in the image with `chsh -s /bin/bash` I wouldn't need to set it but I still do.
> 1 votes

---
Tags: jenkins, shell-script, bash
---


title: What happens when a PAT expires

text:
# Question
Title: What happens when a PAT expires
In azure devops when you create a PAT you can set it's lifetime up to a year. What happens when it expires? Does it auto renew? Do I need to crate a new one and for every service that uses it, I need to track down and update it's credentials?

# Answer
I had expirienced the same problem, I needed to renew manually my PAT.

VS

MSDN
> 1 votes

---
Tags: azure-devops
---


title: Cookbook get and set dynamic URL

text:
# Question
Title: Cookbook get and set dynamic URL
There is a static Artifactory URL being used to grab a semantically versioned artifact and deploy it to JBoss. The deployment aspect of the cookbook is already fleshed out. However, I would like to make the URL "dynamic". In other words, get the most recently published artifact from Artifactory.

**Example URL** https://myartifactory.com/releases/com/package/helloworld/1.0.0/helloworld-1.0.0.war

A Jenkins pipeline exists for this artifact that deploys to Artifactory. So, if 1.1.0 were to be released, it's URL would look like the following:

**1.1.0 URL** https://myartifactory.com/releases/com/package/helloworld/1.1.0/helloworld-1.1.0.war

You can see that the directory in Artifactory changes, as well as the name of the artifact. How can I set up the cookbook to always grab the latest version from Artifactory?

# Answer
> 1 votes
You should use of the Retrieve Latest Artifact endpoint. It uses the repository layouts to know what is the version in your artifact path, so you want to make sure those are set right for you. In this case, it's a standard Maven layout, which is the default for Maven repo, so you're all good.

---
Tags: chef, artifactory, curl
---


# Question
Title: Should developer wait for CI pipeline to complete or start next task after pushing
My company is integrating CI/CD, so far we have implemented CI from what I understand. Currently when a developer pushes code to our git repo, the CI pipeline runs.

Currently our CI pipeline include building the project and the doing static code analysis to make sure it meets our coding standards. We will be implementing testing next. The build and static code analysis take about 3mins right now. From what I have read fixing issues right away is critical to CI/CD. I expect when we add in unit tests that the pipeline could take around 10mins to run.

So my question is when a developer make a pull/merge request should they wait for the CI pipeline to complete or just move on to the next task and come back to fix pipeline issues if they exists? Or should they sit and watch the pipeline run?

# Answer
> Sit and watch the pipeline run?

No, that is not how you work efficiently.

Developers push their commits to the source control repository and then the CI/CD pipeline is triggered.

Developers may post a well-written pull request anytime they want. There is usually a visual mark representing an "on-going build"/"failed build"/"successful build".

Typically a brach can be merged whenever **all** these criteria is fulfilled:

* at least one approver / or as many that is required to approve has approved.
* a "successful build"
* no unresolved merge conflicts

If any of these criteria not is fulfilled, they need to be fixed, but the developer does it whenever he has time or priority to do this task.
> 7 votes

# Answer
I recommend trying your best to ensure the pipeline is less than 10 minutes. You can execute your tests in parallel to enable this. As jonas answered they can then spend time creating a pull request (while the pipeline is running).

Context switching is bad for productivity so rather than picking up another task, I recommend the developer use that time to work on anything else related to the change he just made. Maybe there’s some documentation that could be updated related to this. If it’s a noteworthy feature he could create a short gif showing how to work with it. Even looking at their code change again may help them think how they could improve it next time. This time could also be used in reviewing other developers pull requests and code changes.

If they move onto developing another feature in that 10 minutes it’s likely it’ll be longer than 10 minutes before they get back to it and the work will be less fresh in their mind. If the CI then fails, it will be harder for them to remember what could have gone wrong and fix the code.
> 1 votes

# Answer
Ok so let's assume you have version control tool as Git and CI tool Jenkins. So Dev creates a feature branch for a issue. You have a multibranch pipeline or a workflow in your CI tool which detects this feature branch in next scan and executes the CI steps.

So things which must be ensured are:

1. Before raising PR/MR, the CI job is green. If it is not green, PR/MR should not be raised. Obviously the developer can perform other tasks and then come back and fix issue on this task, that is their choice depending upon task priority. You can even control any PR from being raised by checking it's CI parameters.(If you do not trust your dev that much and they continuously raise PRs for broken builds)
2. Once PR is raised, a code reviewer will review and merge if everything is OK. Code reviewer can be any other developer. His/her responsibility is to review the code, see if it is within the criteria of "Definition of Done". DoD is mainly an agile term but agile and DevOps go hands in hand.
3. Once code is merged the CI for main branch should be green. If not, code should be rolled back and issue should be fixed because generally main branch is also deployed to environments and not rolling back means whole environment will be broken.

Obviously, post build actions will notify the committer that Hey, you have broken the build, so Devs can perform their other tasks, they will get emails anyways.
> 0 votes

---
Tags: continuous-integration, git, continuous-delivery
---


# Question
Title: Can I change the mount path and port number of a built container?
Two containers (A,B) were created from the same image.

The two containers have the same mount path (-v) and port number (-p).

The "A" container is currently in the "stop" state. The "B" container is currently running.

Can I change the mount path and port number of the "A" container?

# Answer
Containers are designed to be disposable. Delete the "A" container and recreate it with the new volume mount and port number. Note that deleting the container will delete any changes made within the container specific RW filesystem layers. If you have changes in there that you need to preserve, then you really want to make those volume mounts. In the end, the workflow looks like:

```
docker container rm ${old_container_id}
docker container run -v ${new_volume} -p ${new_port} ...

```

As for changing an existing container, there is:

```
docker container update ...

```

however, changing volumes and port bindings are not part of the settings designed to be changed from this. Going under the hood of the docker engine to try to change these settings manually is unsupported and may result in unexpected behavior or a broken engine.
> 1 votes

# Answer
First thing first - two containers cannot run on the same host port, while one container is running or stopped, the port is still attached to it, so that same host port scenario won't come in your case.

Regarding changing the volume mount and to run under another port, you can remove the container A and then start another container with either same name or different name and mount point and ports(free ports) as you want them too.

```
docker stop A;docker rm A;
docker run --ti \
           --name A \
           -p <desrired host port:desired container port> \
           -v <desired mount point:container mount point> \
           -d your_image:tag

```
> 0 votes

---
Tags: docker
---


# Question
Title: Entering to docker container of my elasticsearch google kubernetes pod - CONTAINER ID is changing
I have a kubernetes cluster on GCP, in which I have a elasticsearch service. I have a `elasticsearch-0` pod which is running a docker container of elasticsearch.

```
~ ❯❯❯ kubectl describe pod elasticsearch-0
Name:           elasticsearch-0
Namespace:      default
Node:           gke-jaeger-persistent-st-default-pool-xxxxx-phvx/10.166.0.4
Start Time:     Mon, 07 Jan 2019 14:21:19 +0100
Labels:         app=jaeger-elasticsearch
                controller-revision-hash=elasticsearch-8684f69799
                jaeger-infra=elasticsearch-replica
                statefulset.kubernetes.io/pod-name=elasticsearch-0
Annotations:    kubernetes.io/limit-ranger: LimitRanger plugin set: cpu request for container elasticsearch
Status:         Running
IP:             10.36.2.27
Controlled By:  StatefulSet/elasticsearch
Containers:
  elasticsearch:
    Container ID:  docker://5212b8c223401355bd29a5639caea5097074a8a8101ceb10300f76465e4a6536
    Image:         docker.elastic.co/elasticsearch/elasticsearch:5.6.0
    Image ID:      docker-pullable://docker.elastic.co/elasticsearch/elasticsearch@sha256:f95e7d4256197a9bb866b166d9ad37963dc7c5764d6ae6400e551f4987a659d7
    Port:          <none>
    Host Port:     <none>
    Command:

```

And I need access to the node which manage my pod in order to get the container ID of my elasticsearch container.

I get the name of my instances with `gcloud compute instances list` and enter via `gcloud compute ssh <node-name> --zone=<zone-name>` command and I've enter to the elasticsearch docker container with root privileges `docker exec -it -u root containerID bash` (following these steps)

I need perform this because I need add a new mountPath named `/jaeger-data` in order to associate it to a persistent volume claim that I have prepared.

I enter to my docker container and create the mountPath

```
[root@elasticsearch-0 elasticsearch]# mkdir /jaeger-data
[root@elasticsearch-0 elasticsearch]#

```

But I can see that I am disconnecting continously from my docker container because the `containerID` is changed of a continous way. Each time that I enter to my node to get my `containerID` with `docker ps -a` command, this `containerID` is different and when enter to the new container making use of the new ID, my `/jaeger-data` mountPath is not found, this mean that there isn't, may be because I am entering to a new container ...

What is the reason to my containerID change all time? Is this related with Kubernetes node that contain my pod which have my elasticsearch docker container?

Is related with the **Service** situation?

> Kubernetes Pods are mortal. They are born and when they die, they are not resurrected. ReplicaSets in particular create and destroy Pods dynamically (e.g. when scaling out or in). While each Pod gets its own IP address, even those IP addresses cannot be relied upon to be stable over time. This leads to a problem: if some set of Pods (let’s call them backends) provides functionality to other Pods (let’s call them frontends) inside the Kubernetes cluster, how do those frontends find out and keep track of which backends are in that set?
>
> Enter Services.

How to can I address this situation?

I need create my mountPath in order to associate it to my `elasticsearch-0` pod configuration

# Answer
If I understand you question clearly, you're trying to mount `/jaeger-data` inside the elastic-search container or pod. To achieve this, what you can do is put this mount details inside the statefulset yaml of the Kubernetes pod for elastic-search. Using the volumeMounts inside the statefulset, you will not need to ssh or exec inside the container or bother about the change or container ID for each pod restart/recreation. You can use the below example as a reference for your **PVC** and **volumeMounts**, just edit your statefulset using `kubectl edit sts elasticsearch-0 -n default`, this will open an editor and you can replace you PVC details in the below yaml and append and save and exit will recreate your pod and attach the mountpoint:

```
          volumeMounts:
            - mountPath: /es-storage
              name: jaeger-data
  volumeClaimTemplates:
  - metadata:
      name: es-storage
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi

```
* Container ID will change with each recreation of containers;
* If in case you need to exec into the container, the better way is to use `kubectl exec -ti <pod-name> -n <namespace> bash` to exec into the container from any of the master node(similar command as docker exec, just replace pod-name and namespace) or from your local laptop if you have kubeconfig configured.
* Using **Service**, you can expose this application on a particular port, Kubernetes creates an abstraction and itself takes care of the change of pod IP
> 1 votes

---
Tags: docker, kubernetes
---


title: Docker Persisting Data During Build

text:
# Question
Title: Docker Persisting Data During Build
I am trying to create a custom container which also contains MongoDB (hence not using the official MongoDB container). I am trying to add authentication to the database, and I *think* I am having an issue in persisting authentication data, i.e. in `dockerfile` I run the following command at some point:

```
RUN (nohup mongod &) && mongo < /data/mongo.js

```

and `mongo.js` is:

```
use admin
db.createUser(
  {
    user: "admin",
    pwd: "abcd",
    roles: [ "root" ]
  }
)

```

which yields below during build:

```
Successfully added user: { "user" : "admin", "roles" : [ "root" ] }
Removing intermediate container...

```

So after build, when I login to the container and start mongo with --auth flag, still can't login:

```
# mongo -u admin -p abcd --authenticationDatabase "admin"
MongoDB shell version: 3.2.11
Enter password:
connecting to: abcd
exception: login failed

```

What is missing?

# Answer
Instead of writing your own Dockerfile, you can simply pass auth through environment variables to official images, I use compose file something like this:

```
version: '3.1'

services:
  mongo:
    image: mongo
    restart: always
    container_name: mongo
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: abcd
    ports:
      - "27017:27017"
    volumes:
      - ./databases:/data/db

  mongo-express:
    image: mongo-express
    restart: always
    container_name: mongo-express
    ports:
      - 8081:8081
    environment:
      ME_CONFIG_MONGODB_ADMINUSERNAME: root
      ME_CONFIG_MONGODB_ADMINPASSWORD: abcd

```

you can have your database persisted in ./databases folder and nice in browser ui at port :8081
> 1 votes

---
Tags: docker, containers
---


title: How to unseal a Vault in practice?

text:
# Question
Title: How to unseal a Vault in practice?
It is possible to force that multiple keys have to be entered in order to unseal the vault. How does this work in practice? If someone has entered a key and there is another key required to unseal the vault should one ask a colleague to enter another key and how should this be done to prevent that colleague A or someone else sees that key?

# Answer
I assume your vault would be being accessed over a network, so you would ask a colleague to use their own command line on their own workstation to enter their key. Mine requires 3 keys to unseal so I just have 3 users independently enter the `vault unseal` command with their key.
> 4 votes

# Answer
I agree with @alitheg answer, and to add two my cents, sequence of commands to be executed by you or any 3 colleagues of yours who has possession of the vault unseal token

```
export VAULT_ADDR=https://vault.yourcompany.com  # Address of your Vault
export VAULT_SKIP_VERIFY=true                    # this can be skipped, if you use a proper certificate and not self-signed certificate
vault status                                     # to check the vault status
vault unseal                                     # this will ask for the vault unseal token

```
> 0 votes

---
Tags: hashicorp-vault
---


title: Trigger a cross project GitLab trigger

text:
# Question
Title: Trigger a cross project GitLab trigger
I am trying to call a trigger on one GitLab project using a job on another.

Let’s call one A, and the other B. I want a job in A to trigger a job in B.

In A, this is what my `.gitlab-ci.yml` looks like:

```
deploy:staging:
  image: alpine
  variables:
    GIT_STRATEGY: none
  script:
    - apk add --no-cache curl
    - curl -X POST -F "token=$CI_JOB_TOKEN" -F "ref=$CI_COMMIT_REF_NAME" https://gitlab.com/api/v4/projects/1234/trigger/pipeline

```

In B, this is what my `.gitlab-ci.yml` looks like:

```
image: google/cloud-sdk

deploy:staging:
  only:
    - triggers
  script:
    - echo "$CI_ENVIRONMENT_SLUG"

```

However, when A runs, I get the following result from curl:

```
{"message":{"base":["Reference not found"]}}

```

When I hardcode the ref to master, I get this message:

```
{"message":{"base":["No stages / jobs for this pipeline."]}}

```

How can A be configured, so that the `deploy:staging` job in B is triggered on master?

Thinking ahead, I would like to add more deployment targets, so it should **only** trigger the `deploy:staging` job.

# Answer
In your job, I noticed the below curl command, where two things need to be corrected:

* `token=$CI_JOB_TOKEN` \- this TOKEN should be generated in project B under Settings -\> CI/CD -\> Pipeline triggers (Add triggers)
* `"ref=$CI_COMMIT_REF_NAME"` \- this is pointing to a branch with name $CI\_COMMIT\_REF\_NAME on project B and this will only execute if it find a project with that branch name, so if you are trying to trigger it on master branch or any specific branch, you should replace it with something like `-F "ref=master"` in your curl command

```
 - curl -X POST -F "token=$CI_JOB_TOKEN" -F "ref=$CI_COMMIT_REF_NAME" https://gitlab.com/api/v4/projects/1234/trigger/pipeline

```

I use the same trigger with few variables to execute one of my job in another project

```
- "curl -k -X POST -F token=$PROJECT_B_TOKEN -F ref=master -F variables[PROJECT_NAME]=$CI_PROJECT_NAME -F variables[TAG]=$CI_BUILD_REF_NAME -F variables[SOME_VARIABLE_IDENTIFIER]=DevOps -F variables[USER]=$GITLAB_USER_LOGIN https://gitlab.mycompany.com/api/v4/projects/1234/trigger/pipeline"

```

The person who created in token in project B, needs to be owner in project A to execute this trigger, else it fails with a 404 message.
> 2 votes

---
Tags: gitlab
---


# Question
Title: How to keep original and new repositories in sync during a migration? (including the commit history)
We have a git repository that is currently hosted on Azure Devops and have to migrate it to our in house TFS 2017 git (the move is due to a business decision).

But during this setup we need to run both the original and new repositories in parallel for a time and synchronise them regularly until the new repository is signed off by the Dev leads. Only after the new one is approved by the team lead will the devs switch over and then we will make the original read-only. Throughout this process, we need to ensure the developers remain productive.

So how do we perform the initial migration from the Azure Devops repository to the new TFS 2017 git? And then how do we synchronise them regularly so that the new one does not get too far out of sync with the original?

We want to include the commit message history when migrating and syncing.

# Answer
> 6 votes
Simply keep pushing stuff between the repositories. Git pushes are designed for exactly that.

So, if `origin` is the old repository, and `new`is the new one, and you want to move new stuff in the `master` branch from `origin` to `new`, then, locally:

```
git checkout master && git reset --hard    # to clean up any local changes, optional
git pull origin
git push new

```

That's it. `master` is a branch like any other, so if you have multiple branches, rinse and repeat.

# Answer
> 0 votes
Check this out - SO question on moving git repo

You probably would need to add automation around this since you want to move all repos.

---
Tags: git, azure-devops, tfs
---


# Question
Title: How to get list of env variables from the root folder to jenkins job?
> Using configure option, when we check the environment variables, we get only the variables which are available with in the folder or job. But, I want to get complete list of env variable from the root folder to the folder/job, which I want to see.

# Answer
> 1 votes
I recommend that you use the **printenv** command for list system Variable:

```
printenv

```

and also An easy way to obtain the Jenkins environment variables list from your local installation is to append env-vars.html to the server's URL. the URL would be

`http://{jenkins_url}/env-vars.html`

# Answer
> 0 votes
AFAIK, ***environment variables*** are not folder/directory specific, but user specific, that means, for any user its loaded when the user logins into the shell or source the profile which contains the added environment variables (like ~/.bash\_profile). So, if your environment variables are present at root directory, it should be present at the folder/job level as well unless your new bash where this job is running, doesn't loads the environment variable, for which you can test them using the below commands in the job to see if those variables are present or not:

```
pwd
echo $PATH
env

```

if not present, you can source your profile in the job using `. .bash_profile` and check if all variables are present or not

---
Tags: jenkins, jenkins-pipeline, configuration, environment-variables
---


title: Jenkins plugin recomendation

text:
# Question
Title: Jenkins plugin recomendation
My company wants to visualize the % of failed/successful deployments of jenkins jobs.

And the way jenkins is arranged complicates the task, each team or project is a folder name and inside the project folder the jobs are called same.

Folder1 samejobname subfolder1 jobname folder2 samejobname subfolder1 jobname . . . folderN samejobname subfolder1 . . . subfolderN

So I need a plugin (there must be one) which can give me this stats of each job so I can take it from API and pull them to prometheus somehow.

I will be very very happy to be pointed in the right direction so thanks in advance for all the help.

# Answer
Do you use grafana for the dashboard? If so, you can customize with the prometheus plugin for Jenkins See also: https://piotrminkowski.wordpress.com/2017/08/29/visualizing-jenkins-pipeline-results-in-grafana/
> 1 votes

# Answer
Have a look at the Summary Display plugin. Some more details about it (= quote from the linked page):

> ... allows an easy and fully customized build report display.
>
> Features
>
> * Allow a rich summary report visible from both project and build page
> * Reports must be written in an XML format according to the syntax described in section Syntax Description
> * This plugin allow the parsing of several XML files
> * Reports displayed are ordered according to the XML file names
> * Element displayed are based on

In general, the answer to such questions is usually found by searching the endless Jenkins plugin index.
> 2 votes

---
Tags: jenkins, jenkins-plugins, prometheus
---


# Question
Title: How to deploy to Appengine standard from gitlab CI?
How to deploy from Gitlab CI to AppEngine Standard environment?

# Answer
AFAIK there are only 2 ways of deploying GAE apps:

> gcloud app deploy - deploy the local code and/or configuration of your app to App Engine

> To deploy a version of your app with the Admin API:
>
> 1. Upload your app's resources to Cloud Storage.
> 2. Create a configuration file that defines your deployment.
> 3. Create and send the HTTP request for deploying your app.

Both `gcloud` (used in the 1st method) as well as `gsutil` (used in the 1st step of the 2nd method) are part of the Google Cloud SDK, so you'll need to have the SDK installed on your runner.

Summarizing the required steps from a GAE deployment example using the 1st approach that I found in Getting started with Angular 6, GitLab CI/CD and Google App Engine (probably a good read for details):

* in gitlab set env vars with your GAE service account and project ID info
* in your `.gitlab-ci.yml` file's deployment section add a `before_script` section for installing the SDK (update for current SDK version)

> ```
>         before_script:
>           - wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-218.0.0-linux-x86.tar.gz
>           - tar -xzf  google-cloud-sdk-218.0.0-linux-x86.tar.gz
>           - ./google-cloud-sdk/install.sh
>
> ```

and a `script` section for the actual deployment, referencing those env vars:

> ```
>         script:
>           - echo $GAE_KEY_FILE > gae_auth.json
>           - ./google-cloud-sdk/bin/gcloud auth activate-service-account --key-file gae_auth.json
>           - ./google-cloud-sdk/bin/gcloud app deploy --project=$GAE_PROJECT_ID
>
> ```
> 1 votes

---
Tags: cicd, gitlab-ci-runner
---


# Question
Title: Additional push event syncing between Artifactory instances
We have two instances of Artifactory for Prod and Non-Prod.

Normal syncing happens via a pull mechanism by the Prod instance.

Is it possible to also have an additional push event sync from Non Prod instance to the Prod that we would use for hotfixes.

# Answer
Yes, you can set up push event-driven replication. Please note that while pull replication prepopulates remote repository in server A from a local repository in server B, push replication replicates between two local repositories. Once you set up a virtual repository it doesn't really matter because you'll have a single point of access anyway, but just something for you to be aware of.

You can read more about push replication and pull replication in the user guide. Also, please find here a whitepaper on different replication topologies and a webinar on the same topic.
> 2 votes

---
Tags: artifactory
---


# Question
Title: DevOps tools and automation of manual processes
This is a general question, but couldn't find a better place to ask. I'm a new DevOps member in a company, and I've been ask to "update" our DevOps technologies.

Generally speaking, in what areas do we need to provide tools, and what are some latest and advanced technologies/tools to do that?

For instance, we need container management (e.g. `docker, kubernetes`), log analysis (e.g. `ELK Stack`), Application monitoring (e.g. `JMX`), `Weblogic`, database management, `Jenkins`, etc.?

What else, and what tools? Which processes are probably needed to be automated? It can be anything. I just need some advice.

# Answer
Alright... you're not going to implement every tool or automation at the same time, as some of them have quite deep impact on your development processes (and I daresay, development culture). Take a step-by-step approach; research each individual tool. Figure out what it actually does, what it is useful for. Play around with them, install them locally, go through tutorials, and see what they do, or whip up a quick VM somewhere (AWS, etc.) if you need more than one.

Then, take a look at the work your team/company is doing right now. What *are* the manual process you wish to automate? Ideas should spring up pretty quickly.

A nice, simple(-ish), bog-standard stack to start out with:

* Vagrant to automate setting up VMs for your developers, or test environments.
* Ansible to automate configuration management (relatively lightweight compared with Chef or Puppet)
* Gitlab or Jenkins as your automated CI/CD driver, mostly depending on taste
* Docker for your containers
* docker-compose, docker-swarm, Kubernetes, OpenShift to automate container deployment
* ELK or Splunk for your logging/monitoring, again depending on taste

Have fun!
> 5 votes

# Answer
Hi and welcome to DevOps SE!

While DevOps is not self-purposed, maybe it is worth to step back for a moment to find out the actual problem before trying out solution.

As you have asked a more or less generic question, I'll give you sort of a "*DevOps primer*" stick more to methodology rather than give you a list of tools.

How detailed you would tailor the implementation of this depends on many factors like business needs, complexity of teams, products and systems. .. and the time you have got/are allowed to think.

* **Why** did they ask you to update DevOps techologies?
* **What** is the business objective i.e. what will change after you are done - find out the management perspective. Will be software delivered faster/in shorter periods, or are deployment problems a real issue?
* If there is an update of DevOps technologies, what is the current DevOps toolchain (please read on this Wikipedia article to understand families of tools by their purpose in the toolchain) your colleagues use?
* **Who** i.e. which team members have knowledge/daily usage of which of these tools?
* **What** are technically software products your manufacture and deliver?

Now you know the problems and the existing toolchain and products it produces:

* Are there now some quick fixes to achieve quick results or do you need more time to analyze data you have now, and find answers to questions you have to the team and management?
* Which tools do you think are missing and how which change would bring the company closer to the set objective?
* Will be free open source tools enough or some enterprise licenses could be required?
* Is it possible to design a DevOps strategy and generate iterative tasks to deliver changes and improve the situation in baby steps (design thinking with your team as end users or do you have a complex situation in large enterprise - system thinking)?

Side note: regarding to container management, depending on existing expertise you might consider using a solution bundling Kubernetes, e.g. OpenShift as already mentioned or RancherLabs etc. Regarding Jenkins, after you know what is happening, skip through its plugins, Jenkins-as-is is just a beginning.
> 4 votes

# Answer
I've been responsible for setting up DevOps processes in two companies from scratch so hopefully this will help.

Others have asked questions about business objectives but I'm going to give you a brief rundown of some of the problems we had and the technologies I used to fix it.

1. Infrastructure Creation/Management was slow and error prone - I started creating and managing our infra (particularly K8's clusters) using **Terraform**
2. Kubernetes CI/CD becomes difficult as the number of services grows - to solve this issue we started using **Helm** which bills itself as the 'missing package manager' for Kubernetes, but it also helps manage Kubernetes .yaml files as templates which helps simplify CI/CD
3. CI/CD - Building/testing and deploying applications was also slow and cumbersome, so we started using **Jenkins (Blue Ocean)** to define our deployment pipelines in a Jenkinsfile. Highly recommend this if you are still using the old version of Jenkins.
4. Managing configurations of mutable infrastructure - we still have some 'mutable' infrastructure which was prone to drift, and you could use **Chef** or **Ansible** to manage those to minimise configuration drift or to bootstrap new servers quickly.
> 3 votes

# Answer
If you looking for a fun idea try adding a chatbot to drive your devops automation:

I wrote a short presentation with a link at the end to a video of settings up a chatbot on slack but botkit supports a lot of different chat solutions.
> 0 votes

---
Tags: amazon-web-services, deployment, containers, automation, infrastructure-as-code
---


# Question
Title: How to change a statefulset in OpenShift?
What's the best way to change a statefulset (e.g., modify the ENV parameters, modify PVCs) in OpenShift? `oc edit statefulset xyz` only allows to change the scaling, nothing else.

# Answer
Shameless self-answer:

So it turns out the only way to edit the statefulset, and this is by design, is to delete it and recreate it with the new values.

Doing that does not delete or stop the pods or the the PVCs - those will re-attach to the new statefulset spun up later, so you won't lose anything. N.B.: you can delete the pods, and the PVCs will still stick around. When the pods belonging to this statefulset restart later, they will re-attach to the PVCs.

There is a discussion on one of the OpenShift/Kubernetes task trackers (which I cannot find anymore) that explains the rationale: in contrast with stateless Deployments, where OpenShift can simply scale up a new set of pods, then scale down the old one, while routing traffic to the new one, this is not easily possible with statefulsets, since those have persistent PVCs, so there usually is a very defined number of pods that can/should run at a time.

So, if OpenShift would do an upgrade for you, due to an edit, it would need to decide how to do that (spawn up new pods parallel to the old one would require spinning up PVCs which may wreak havoc with your quota; replacing them in-place could wreak havoc if your pods need to talk to each other to synchronize). Hence, they require you to do this on your own. For example, you can decide to delete and recreate the statefulset, then delete one of the pods after the other; they will be automatically recreated immediately, as usual. You can then wait until the next pod has reached a state where it is synced with the other ones (in whatever way your software does it), then delete the next one etc., until you have upgraded them all.
> 2 votes

---
Tags: openshift
---


# Question
Title: Set container registry auth key via command line/env variable
I'm trying to set up Jenkins CI pipeline which will be used to build docker images. I have my auth key stored in Jenkins credentials and I want it to be loaded on start in placed in some environment variable (already done).

Is it possible that Docker will read that environment variable and use the key for authenticating to my container registry? Or maybe I can pass it somehow via command line?

I've tried searching for that in the docs, but only thing I found is that I can specify that values in `config.json` and specify the location of that. This is not the best option IMO.

Any ideas? Thanks in advance.

# Answer
You mentioned that the auth key is stored in Jenkins credentials and are available as environment variables, if so, then your half task is already done; what you can probably do from here is, use those environment variables to perform docker login (assuming USERNAME and PASSWORD are the Jenkins credentials variable), use the below command to login

```
docker login your.docker-registry.com -u $USERNAME -p $PASSWORD

```

This assumes, that your registry has SSL enabled, if so, don't forget to place the registry certificates under the below name on your Jenkin host

```
/etc/docker/certs.d/your.docker-registry/ca.crt

```

Post this, you would be able to push image to the registry
> 1 votes

---
Tags: docker, registry
---


# Question
Title: Chef remote_file set maximum number of re-directions to zero
I am trying to implement the below `wget` download command using chef `remote_file` resource. But I couldn't find a way to avoid the re-directions. The URL that I am trying to access may sometimes be redirected to my ISP bill reminder page. And chef `remote_file` resource downloads this bill reminder HTML page as a zip file.

`wget -N --max-redirect=0 http://www.someurl.com/file.zip`

The chef recipe resource is

```
remote_file "#{node['download-zip-path']}/#{zip}" do
    source "http://www.someurl.com/#{zip}"
    action :create
    notifies :run, 'execute[unzip_file]', :delayed
end

```

The `--max-redirect=0` flag in `wget` makes sure that there isn't any redirection to ISP page. I can achieve the same using ruby's `open-uri` or `execute` command resource but is there any native chef way to achieve this no/zero re-direction?

# Answer
> 1 votes
Found out that the remote\_file resource couldn't handle redirects. Hence I had to write a ruby\_block resource that makes use of 'Down' gem.

```
ruby_block 'download_openvpn_zip' do
    block do
        attempt = 2
        begin
            retries ||= 0
            tempfile = Down::NetHttp.download("http://www.someurl.com/#{zip},max_redirects: 0)
            FileUtils.mv tempfile.path, "#{node['openvpn-conf-path']}/#{tempfile.original_filename}"
        rescue Down::TooManyRedirects => e
            puts "\n \t ERROR: #{e.message}"
            retry if (retries += 1) < 1
        end
    end
    action :run
    notifies :run, 'execute[unzip_file]', :delayed
end

```

---
Tags: chef
---


# Question
Title: Terraform - aws_security_group_rule doesn't support import
I'm having a strange problem when trying to import an `aws_security_group_rule` into terraform.

I'm running the latest version of TF (0.11.11)

For example:

`terraform import module.ec2.aws_security_group_rule.control_elb sg-securitygroupid_ingress_tcp_8080_8080_sg-sourcesgid`

Returns `resource aws_security_group_rule doesn't support import`

However, looking at the documentation, this does seem possible?

https://www.terraform.io/docs/providers/aws/r/security\_group\_rule.html

# Answer
Did you try to import in flat setup than modular setup? It seems that in modular setup there is a bug with Terraform. https://github.com/terraform-providers/terraform-provider-aws/issues/559
> 1 votes

---
Tags: amazon-web-services, terraform
---


# Question
Title: Gitworkflow and testing short lived feature branches in Openshift/Kubernetes
Recently my team started adapting git workflow for agile-based delivery, we've been using JIRA, TeamCity and various other CI tools to complete the pipeline. The workflow works great for monolith or SOA applications in JAVA.

The problem comes with microservices delivery pipeline targeted for enterprise environments such as Openshift.

With git-workflow, the expectation is `feature` branches once developed and tested will be merged to the `develop` branch, however, when there are numerous features being developed a developer needs a platform to test the service feature he just built before submitting a pull request.

With openshift/K8s we expose a service, let's assume each service built out of the feature branch looks like below.

```
myapplication-MIM-158848-feature1.myco.internal.au
myapplication-MIM-158849-feature2.myco.internal.au

```

Internally openshift should be able to allocate dedicated pods to test the services and they should be disposed once merged with `develop` branch, another problem with this approach is running integrated test or if services have dependency with consumer apis. Modifying the service name based on `feature` branch is not a good idea.

Does anyone know a better way of handling this delivery model in microservices based platform. Thanks

# Answer
I'm not altogether sure if there actually is a problem you're trying to solve, or whether you are looking for confirmation that you are on the right track. Some thoughts:

> Internally openshift should be able to allocate dedicated pods

Sure. Add a post-build step in your CI pipeline which does the following:

* Fashion a .yaml or .json description of that pod (or rather, DeploymentConfig). How/where you do that depends on your architecture - if each of your microservices has exactly the same port (i.e., all listening on 80 or 443), then you only need a central template for that, and basically only inject the name (containing the microservice name and the feature branch name) and the git source URL for it. The file can of course contain multiple OpenShift objects (i.e., the DeploymentConfig and a Route, maybe).
* `oc apply` it, OpenShift will build an image and start a pod for it.

> and they should be disposed once merged with develop branch

Add a post-merge step which does the same in reverse, i.e. `oc delete ...` for every component of the original `.yaml`. Probably this is easiest if you give a branch-based label to each of the components, so you don't need to keep track of which you have applied in the first place.

> another problem with this approach is running integrated test or if services have dependency with consumer apis.

If a service involved in an integrated test has state, then you indeed need to fire up a new pod/volume for each test run. If your integration test depends on external state, then you also need to manage that somehow. Both of these aspects are true anyways, no matter how you architect your solution.

In these cases, you would probably do well to not start deployments right out of your individual CI pipelines, but do a "big" deployment when you start your integration test (i.e., deploy all individual microservices like described above, but also inject an ID for your current integration test run into all their names, so everything stays unique).

> Modifying the service name based on feature branch is not a good idea.

Why not? If you wish to have a pod running for every feature branch, then you need to do it that way, period.

It would certainly force you to do some kind of dependency injection (i.e., if you are testing one service A which needs another service B, then you have to tell A which B-feature it should take. But you need to do that anyway, in some form or fashion; as far as I can understand, you are trying to make the services (their branches) independent from each other, and I also assume from your question that you have one service per git repository (i.e., one service per branch) so you *need* to have them running per-branch.
> 4 votes

# Answer
We are in the process of making open source our openshift continuous deployment setup over on GitHub that we call OCD. It is actually driven by git tags and Helmfile yaml within git. As it’s driven by git tags it is agnostic as to which branches are being use. It is also agnostic to which CI/CD tools you are using.

The idea is that developers can “self-service” spinning up new apps on openshift by changing yaml files in git. So a developer setting up a feature branch can “self-service” spinning up that code by simply tagging the code and changing some yaml to have openshift run that code.

It could be used to meet your use case as follows:

1. OCD builds from git tag (or git release) events and makes a container image of the source code in the git tag, and tags the container image with the same tag. It doesn’t need to know which branches those tags belong to.
2. A team that needs a new feature branch deployment would add a new folder in the git repo that holds all the configuration to build and run their tagged code.

OCD is implemented as generic helm charts. One sets up BuildConfig that build from git tags and tag the image with the same tag. Another will setup DeploymentConfig to run tagged images. Another manages Secrets encrypted in git. Another manages ConfigMaps. There are installed using Helmfile that lets you declare a bunch of helm release in a yaml file. That is run by adnanh/webbook a small go app that can catch git webhook events and run shell scripts.

OCD was created with micro services in mind. The idea being teams can spin up new services very easily. Using it to spin up temporarily apps to test is easy. It doesn’t current delete apps if they are removed from git. That wouldn’t be hard to implement when testing it I delete apps with a single command line.

Disclaimer: it’s currently pre-release but feature complete and we are in the process of migrating our own apps into it.
> 0 votes

---
Tags: kubernetes, git, microservices, cicd, openshift
---


# Question
Title: Handling request based on source ip with traefik in k8s
I have a k8s cluster (three vms on my own hardware; no aws, google cloud, ...) that uses *traefik* (https://traefik.io/) as a reverse proxy to address services/deployments in the background.

For this I use the deployment-variant from this part of the documentation: https://docs.traefik.io/user-guide/kubernetes/#deploy-trfik-using-a-deployment-or-daemonset

Now I have multiple applications deployed in the cluster, which all have some ingresses assigned, which are read by the *traefik-ingress-controller*. Some of those applications are internal ones, like *kibana* or the *traefik-web-ui*, and some others are external, like the applications themselves. I distinguish those two by having different dns entries (like `https://dashboard.internal.mycoolapp.com` and `https://app1.external.mycoolapp.com`) and the internal dns is not resolved from the outside world (=the internet) whereas the external is (like from google dns).

That's for the setup. Now, let's come to the problem:

A couple of days ago, I thought: What happens, if I create a wildcard dns entry for `*.internal.mycoolapp.com` on a machine, that is outside my network, and just resolve it to the same ip(s) the external dns entry resolves to. Et voila, my internal services are accessible from the outside!

So this is, of course, a state which is not acceptable. So I'm searching for solutions on this.

What first came to mind was to block all incoming requests on rules for internal services, if the host ip of the requester is outside of our network:

```
...
kind: Ingress
metadata:
  name: app1
  namespace: default
  annotations:
    traefik.ingress.kubernetes.io/whitelist-source-range: "10.0.0.0/8"
    ingress.kubernetes.io/whitelist-x-forwarded-for: "true"
...

```

Theoretically this should work. But as I found out later on, before reaching the *traefik-ingress-controller*, all requests are handled by *kube-proxy* and their host addresses are translated to local addresses (*(S)NAT*), so every request has an internal host address set.

So this is the point I am currently looking for a solution.

One solution supposedly is to deploy the *traefik-ingress-controller* not as a deployment, but as a daemon set and bind it to the ports on the host directly (as said here https://docs.traefik.io/user-guide/kubernetes/#deploy-trfik-using-a-deployment-or-daemonset). I tried that yesterday by just changing my traefik configuration to a daemon set and add the `NET_BIND_SERVICE` capability to it, but it didn't really change anything there. So has someone any idea what I might have done wrong there? Or has someone a good how-to/tutorial/... on how to pass the actual requester host through to the ingress controller?

Here's my current configuration file for traefik:

```
---
kind: Deployment
apiVersion: extensions/v1beta1
metadata:
  name: traefik-ingress-controller
  namespace: kube-system
  labels:
    k8s-app: traefik-ingress-lb
spec:
  replicas: 1
  selector:
    matchLabels:
      k8s-app: traefik-ingress-lb
  template:
    metadata:
      labels:
        k8s-app: traefik-ingress-lb
        name: traefik-ingress-lb
    spec:
      serviceAccountName: traefik-ingress-controller
      terminationGracePeriodSeconds: 60
      containers:
      - image: traefik:v1.6.4
        name: traefik-ingress-lb
        volumeMounts:
        - mountPath: /ssl/external
          name: ssl-external
        - mountPath: /ssl/internal
          name: ssl-internal
        - name: traefik-toml
          subPath: traefik.toml
          mountPath: /config/traefik.toml
        ports:
        - name: http
          containerPort: 80
        - name: https
          containerPort: 443
        - name: admin
          containerPort: 8080
        args:
        - --configfile=/config/traefik.toml
        - --api
        - --kubernetes
        - --logLevel=INFO
      volumes:
      - name: ssl-external
        secret:
          secretName: external.mycoolapp.com.cert
      - name: ssl-internal
        secret:
          secretName: internal.mycoolapp.com.cert
      - name: traefik-toml
        configMap:
          name: traefik-toml
---
kind: Service
apiVersion: v1
metadata:
  name: traefik-ingress-service
  namespace: kube-system
spec:
  selector:
    k8s-app: traefik-ingress-lb
  ports:
  - protocol: TCP
    port: 80
    name: web
  - protocol: TCP
    port: 443
    name: sweb
  externalIPs:
  - 10.2.3.1
  - 10.2.3.2
  - 10.2.3.3

```

The toml-file just contains http-to-https redirect and the paths to the certificate files.

I have some other solutions in mind, but they do all have the downside that they need more *things* deployed and maintained, like a second ingress controller on another port, that handles only specific ingress objects, or a new entrypoint for external requests in front of the whole cluster, that deletes host headers from requests, when it is not my external address (does this even work with ssl?).

So I really need some help on this. Thanks in advance!

# Answer
> 1 votes
I think your ingress rule is fine but I think your Traefik service needs to be changed. Check out this article https://kubernetes.io/docs/tutorials/services/source-ip/

I think what you are looking for is `nodePort` with `externalTrafficPolicy` set to `local`.

This should allow Traefik to enforce `whitelist-source-range` annotation on the ingress.

EDIT:

I was going to go into more detail but it was so old I thought you might have already found a solution. Ingress controllers are not allowed to bind to 80 and 443 but most organizations are using the load balancer service to load balance traffic to the Ingress controller and then out to the kubernetes services and then to the pods.

```
DNS    HostName Routing       Port routing
LB --> Ingress nodePort --> Service ClusterIP --> Pod

```

I don't usually use the cloud providers LB. So usually I will wildcard my dns `*.mycluster.com` and point that to nginx. In nginx I will then have a couple of configs to redirect that traffic to either the ingress controller or some of my services that are running nodePort because they are fragile.

Hope this helps

---
Tags: docker, kubernetes, rproxy
---


# Question
Title: In Docker, is there any advantage or disadvantage in using 'CMD cmd1 && cmd2' versus 'CMD cmd1 && exec cmd2'?
In Docker, when chaining several startup commands (CMD) in shell form, is there any advantage in using the shell's exec feature to replace it with the last command of the chain instead of just running it as a child process of the startup shell?

Here is an example:

```
CMD npm install && npm start

```

versus:

```
CMD npm install && exec npm start

```

I think in the latter example, the `npm start` command will replace the shell as the PID 1 process. But I do not fully understand which advantages or disadvantages this would have.

# Answer
> 6 votes
When you replace pid 1 with a new executable, rather than forking it as a child pid, you reduce overhead (slightly). More importantly, you get signals directly to your application instead of having them passed to the shell running as pid 1. By default, the shell running in pid 1 will ignore the SIGTERM sent by docker to gracefully stop your application. After 10 seconds of no response, docker will send a SIGKILL which will immediately terminate the shell and child processes without giving them a chance to drain network connections or finish writing to any files. This can result in things like database corruption depending on what the app is inside the container.

Even when this doesn't apply to your application (if there's no state), you can save up to 10 seconds every time you stop the container by switching to the exec syntax (assuming your application exits when it receives a SIGTERM). Therefore the exec syntax is the one I prefer for both data safety and speed.

---
Tags: docker
---


# Question
Title: Specifying branch name for jenkins job through AnyStatus
I am wondering if anyone else has integrated before a parameterized Jenkins build with AnyStatus.

When removing the parameter above, I am able to run builds using the branch specified previously from the browser, but that's not the idea. There is a post explaining that I can use a script, but that wouldn't be integrated with AnyStatus in the existing Jenkin's plugin.

# Answer
Oh, I figured it out. I just went to the job configuration, activated the "build is parameterized" checkbox, added a string parameter called BRANCH, and in the "branches to build" section added `${BRANCH}`. Pretty much what is explained here:
> 0 votes

---
Tags: jenkins, monitoring
---


title: Mesos-slave not picking up attributes

text:
# Question
Title: Mesos-slave not picking up attributes
I am running a x-mesos-slave container with these attributes:

```
"Cmd": [
    "--attributes=farm:app,region:eu-west-1,zone:eu-west-1b"
],

```

When I run an application in Marathon with farm:UNIQUE, this works fine.

However when I try to use zone:UNIQUE, the application gets stuck in waiting.

This makes it seem there is no region/zone configured.

# Answer
> 1 votes
As per https://mesos.readthedocs.io/en/0.22.2/attributes-resources/ states, attributes are separated by ';' not by ','.

---
Tags: docker
---


# Question
Title: Can I check whether ansible is on last batch of rolling update (serial: 25%)?
I'm using Ansible to update a pool of EC2 servers with a rolling update, in batches of 25%. In between batches, I'm pausing long enough for the server health checks to start to pass again (since they'll fail while the application server restarts).

The question is, can I skip the delay on the final pass so the playbook will finish sooner? Here is the script. The last line is the API I want:

```
---
- hosts: tag_deploy_environment_development
  serial: "25%"
  remote_user: ec2-user
  become: no
  vars:
    health_check_healthy_threshold: 2
    health_check_interval: 10
    server_port: 80
  tasks:
    #- debug: var=vars
    #- debug: var=ansible_facts

    - name: "Copy the current server version from disk"
      include_role:
        name: deploy-code
        tasks_from: push-files

    - name: "Wait for {{ server_port }} to become available"
      wait_for:
        port: "{{ server_port }}"
        timeout: 60
        state: started

    # I would like to check whether we are on the final batch,
    # and don't pause in that case:
    - pause:
        seconds: "{{ health_check_healthy_threshold * health_check_interval }}"
        prompt: Waiting to allow AWS health checks to pass
      when: ansible_facts['remaining_batches'] > 0

```

# Answer
> 3 votes
To solve this, I printed `vars` and `ansible_facts`, and eventually stumbled on the `ansible_play_batch` variable, which is a list of hostnames in the current batch. `ansible_play_hosts_all` gives the full list of hostnames. (Do not use `ansible_play_hosts` for this purpose, because as of Ansible 2.7, it works but the docs say it gives something completely different.)

The question: Is this the final batch?
Rephrasing based on the data: Is this batch == the last part of the host list?
Or to make it simpler: Is the end of this batch == the end of the host list?

In Ansible, the check for that is:

```
tasks:
  - pause:
      seconds: "{{ health_check_healthy_threshold * health_check_interval }}"
      prompt: Waiting to allow AWS health checks to pass
    when: ansible_play_batch[-1] != ansible_play_hosts_all[-1]

  - debug: msg="This is the last batch."
    when: ansible_play_batch[-1] == ansible_play_hosts_all[-1]

```

---

Side note: I used the following playbook for testing, to simulate having many/few hosts without bothering to set them up. I couldn't see how to specify the hosts as a list in the playbook, so I added them to a group, then executed the group:

```
---
- hosts: localhost
  connection: local
  tasks:
    - add_host: { name: host1, groups: test_hosts }
    - add_host: { name: host2, groups: test_hosts }
- hosts: test_hosts
  connection: local
  serial: "25%"
  tasks:
    - debug: var=ansible_play_batch

```

---
Tags: amazon-web-services, ansible, amazon-ec2, ansible-inventory
---


# Question
Title: What are the advantages of dockerizing nginx and php in different containers?
I just started working with Docker and Kubernetes and I've been watching a lot of stacks, in which some people build nginx+php in a single image and some build an image with nginx and another one with php (mounting the same path and enclosing both containers in the same deployment in Kubernetes).

What would be the advantages of building two docker images instead of installing both nginx+php in the same one?

# Answer
PHP with nginx is usually done using php-fpm which is a separate processus.

Keeping the core idea of docker of one process (see end of answer for more details on this point) per container this makes sense to have the nginx process and php-fpm process in separate containers.

As the communication between nginx and php-fpm arise through fastcgi the php-fpm container can also be on a separated host and this allows using a cluster of php-fpm containers behind nginx.

After the wall of comment here's a little more background, docker documentation have paragraph about the idea that a container should have only one concern.

The main idea of a linux container (**lxc**) is to run a process in an isolated namespace at the cpu and memory level, docker add on top of this an isolation at the filesystem level.
The advantage is that the compromission of a process within this namespace won't allow to read memory of other processes and as such should prevent other compromission on the host.

While talking about nginx and php-fpm, they work in pair but each has it's own concern, nginx will do the HTTP part, routing, headers validation, etc. and php-fpm will do the code interpretation and return the html part to nginx. While it's usual to have both together serving a single application that's not mandatory.

Depending on context it may be easier to have a container including the whole stack for an application, on a developer workstation for exemple. But ideally for production use, try to keep the fewer interaction inside the container, having separated processes in the same container with supervisord brings its share of problem in term of zombie process and log handling (exemple story here for illustration purpose only).

So finally I'll quote the docker page with some emphasis:

> While “one process per container” is frequently a good rule of thumb, it is not a hard and fast rule. **Use your best judgment to keep containers as clean and modular as possible**.

There's no "silver bullet rule" which apply to everything, it's always a balance between the complexity within the container and the complexity orchestrating the containers themselves.
> 22 votes

# Answer
There is no meaningful benefit that outweighs having to manage two containers. As long as you have a 1:1 relationship between the processes and they serve a single purpose, put them in the same container.
> 5 votes

# Answer
Actually, one missing point here is the horizontal scalability. There's an article from Jamie Alquiza long time ago addressed this:

http://archive.is/pDzz0

In short, you scale your php-fpm horizontally for reaching higher performance. Scaling Nginx+php-fpm together does not bring you any benefit. I encourage you do some stress testing (e.g. Tsung, Gatling, etc.; please don't do Apache ab, that is a very old toy) yourself to verify what the article stated. I personally have several real world experiences proved the article is true in general.

But there're two drawback (maybe not for Kubernetes) for bare metal machines / VMs:

1. How to configure Nginx dynamically discover php-fpm container changes? This is the easy part.
2. How do we share the same volume / file systems after scaling? Nginx and php-fpm containers should read exactly the same content to reach consistency. This leaves you the least complicated puzzle piece (and the most fun part) to work on.

EDITED: Now it's almost halfway into the of year 2019. Old model, php-fpm+nginx in the same pod, has different usage. If you're familiar with service mesh, then nginx (or what so call Nginmesh) serves as a sidecar to handle east-west bound traffic. East-west bound traffic mostly used to authenticate among services, or other fancy functionalities, whereas pure "php-fpm only" containers could not do that alone.
> 4 votes

# Answer
The advantage is : you can run multiple php-fpm containers in back-end, we call it PHP cluster, via number of ports. Example port 9000, 9001, 9002 .. and so on
> -1 votes

---
Tags: docker, dockerfile
---


# Question
Title: How to pull all alternative tags of a Docker image?
I administer a GitLab instance with a build pipeline. All components are encapsulated in Docker images from the official GitLab maintainer.

Whenever I update - usually once a week - I need to check whether the `gitLab/gitlab-runner-helper` still works for the current latest version of GitLab. This can only be checked by executing a pipeline. If it does not work, the log tells me exactly what image it needs an I proceed to pull it.

The image in question is also tagged with a latest tag, which I cannot use, due to the hard dependency to the non-volatile tag.

```
$docker image ls
REPOSITORY                    TAG                 IMAGE ID            CREATED             SIZE
gitlab/gitlab-runner-helper   x86_64-8af42251     1ee5a99eba5f        20 hours ago        43.7MB
gitlab/gitlab-runner-helper   x86_64-latest       1ee5a99eba5f        20 hours ago        43.7MB

```

**To automate my update process, I'd like to know, how I could pull the latest image with all alternative tags?**

The man page of Docker pull says, there is a `--all-tags` option, to load any tagged image from the repository, but this cannot be combined with a tag.

# Answer
> 1 votes
I posted the same question at stackoverflow. The question is answered there and I have posted my script. For the sake of completeness, I will post it here as well.

```
#!/bin/sh

TOKEN=`curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:gitlab/gitlab-runner-helper:pull" | jq '.token' | sed 's/"//g'`
TAGS=`curl -s https://registry.hub.docker.com/v2/gitlab/gitlab-runner-helper/tags/list -H "Authorization: Bearer $TOKEN" | jq ".tags[]" | sed 's/"//g' | grep x86_64`

for tag in $TAGS;
do
  # is $tag an old entry?
  if grep -Fxq $tag tags.list
  then
    # already processed
    continue
  else
    echo "new tag found: $tag"
    newSHA=`curl -s https://registry.hub.docker.com/v2/gitlab/gitlab-runner-helper/manifests/$tag -H "Authorization: Bearer $TOKEN" | jq ".fsLayers[] .blobSum" | sed 's/"//g'`
    latestSHA=`curl -s https://registry.hub.docker.com/v2/gitlab/gitlab-runner-helper/manifests/x86_64-latest -H "Authorization: Bearer $TOKEN" | jq ".fsLayers[] .blobSum" | sed 's/"//g'`
    if [ "$newSHA" = "$latestSHA" ]
    then
      echo "$tag is new latest version"
      docker pull gitlab/gitlab-runner-helper:$tag
      echo $tag >> tags.list
    fi
  fi
done

```

The above script utilizes a file named `tags.list`, that is placed next to it. This file contains the older tags, to prevent issuing 500+ HTTP requests. If a tag from the `TAGS` is not yet present in the file, it does not mean, it is the latest. Sometimes tags appear, that eventually will become the latest version. Those tags are probed, but will not be inserted into the file. This might become an issue in the future, if those versions will be skipped as latest.

**Note:** The script above only focuses on a specific subset of tags `(x86_64)`.

# Answer
> 0 votes
You can list all alternative image using the reg tool.

---
Tags: docker, gitlab, gitlab-ci-runner
---


# Question
Title: How to pass a /dev/disk device on macOS into Linux Docker?
I've connected external HDD via USD on macOS, and I'm running Ubuntu's Docker container as:

```
docker run -it --device=/dev/disk3 --privileged ubuntu bash

```

but the device is not present in the container:

```
# ls /dev/disk3
ls: cannot access '/dev/disk3': No such file or directory
# fdisk -l
Disk /dev/sda: 59.6 GiB, 63999836160 bytes, 124999680 sectors
Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1  *     2048 124999679 124997632 59.6G 83 Linux

```

Here is another attempt:

```
$ docker run -it --device=/dev/disk3:/dev/sdb --privileged ubuntu bash
# ls /dev/sdb*
ls: cannot access '/dev/sdb*': No such file or directory
# fdisk -l
Disk /dev/sda: 59.6 GiB, 63999836160 bytes, 124999680 sectors
Device     Boot Start       End   Sectors  Size Id Type
/dev/sda1  *     2048 124999679 124997632 59.6G 83 Linux

```

Here is the device which I'm trying to add to the container (as reported on macOS):

```
$ diskutil list /dev/disk3
/dev/disk3 (external, physical):
   #:                       TYPE NAME                    SIZE       IDENTIFIER
   0:     FDisk_partition_scheme                        *1.0 TB     disk3
   1:                      Linux                         1.0 TB     disk3s1

$ sudo fdisk /dev/disk3
Disk: /dev/disk3    geometry: 121601/255/63 [1953525167 sectors]
Signature: 0xAA55
         Starting       Ending
 #: id  cyl  hd sec -  cyl  hd sec [     start -       size]
------------------------------------------------------------------------
 1: 83 1023 254  63 - 1023 254  63 [         2 - 1953525165] Linux files*
 2: 00    0   0   0 -    0   0   0 [         0 -          0] unused
 3: 00    0   0   0 -    0   0   0 [         0 -          0] unused
 4: 00    0   0   0 -    0   0   0 [         0 -          0] unused

```

Similar issues are posted here:

However, I cannot mount it on macOS host, as the partition is formatted to Linux file system (ext4) which is not supported. That's why I'm trying to pass through the device to the Linux Docker container, but the device isn't shown.

What am I missing?

# Answer
> 5 votes
Docker was built for linux, it runs on Mac through HyperkIT, which is a lightweight hypervisor. This means Docker only sees devices connected to the hyperkit hypervisor, rather than the devices connected to the Mac.

Unfortunately, Hyperkit has issues with USB device passthrough from your MacOS to the hypervisor (read more here) so the --device command won't be much use to you here. You have three options,

Option 1: Mount the ext4 drive to your Mac (not recommended)

Whilst your Mac doesn't natively support ext4, there are ways around this. I have installed ext4fuse on your Mac by following this tutorial. When you have mounted your device, add it to your ubuntu container with the -v flag,

```
docker run -v <Your_volume>:<dest_volume> -it ubuntu bash

```

Making sure that the directory of your volume has been added to the file sharing folders in Docker.

Option 2: create a docker-machine using the virtualbox driver and add the device to the virtualbox vm (still not recommended)

Follow this great tutorial here which goes through installing a docker machine with a virtualbox driver and mounting the USB stick.

Option 3: Run Docker on linux (recommended!)

I wouldn't recommend using Docker for Mac. It was built for linux and if you don't switch to linux you'll waste your time troubleshooting issues like these for no real reason.

I've got a mac, but run a few ubuntu cores in virtualbox and then install docker on top of them, that way I can play around with docker starm too (which you can't with docker Mac unless you use docker-machine!

# Answer
> 2 votes
As per @pgayvallet comment on GitHub:

> As the daemon runs inside a VM in Docker Desktop, **it is not possible to actually share a mac host device with the container inside the VM**, and this will most definitely never be possible.

---
Tags: docker, linux, macos
---


# Question
Title: How to deal with untested features in the develop branch before deployment?
Our current software deployment lifecycle looks like this (using git-flow);

1. We test locally or on a development environment.
2. We submit a pull-request to merge the feature into the develop branch
3. Our testers test the feature.
4. \** If rejected \** we create a new bugfix, which we test locally and submit a new pull-request to merge with the develop branch.
5. At the end of the sprint we create a release from the develop branch.
6. Deploy the release to the acceptation environment.
7. \** When accepted by all the stakeholders \** We finish the release, thus merging the release branch into the master branch.
8. The master branch is then deployed to the production environment.

This used to work fine, but now we’re forced to create a new release at the end of each sprint, which means that the develop branch might contain rejected features awaiting a bugfix at the time of release.

To make it more complicated, we have two repositories, one for front-end and one for back-end and they have to be in sync. i.e. our front-end cannot contain features dependent on, an api endpoint that has been rejected by a tester and won't be finished before the release.

I'm at a loss and could really use some advice.

\[*Sorry for the long comment, but I thought this question needed some context.*\]

# Answer
> 2 votes
You could use the same tools used in Trunk Based Development (TBD) to carry work-in-progress in the trunk (the only non-release TBD branch with a longer lifespan):

> "Branch by Abstraction" is a technique \[1\] for making a large-scale change to a software system in gradual way that allows you to release the system regularly while the change is still in-progress.

These would allow merging a feature in develop/master at any time, without the feature being actually (completed/functional and) enabled.

If technically these are implemented using configurable means then at any time the software can be tested both with and without a feature enabled, simply by flipping the config flag in the respective test environment. Otherwise special builds may be needed to enable them (still a lot easier than doing a branch merge with the feature's full/partial code).

And you could then run all your feature testing in develop/master to get your features enabled, one at a time.

But if you go in this direction - why would you even struggle with git\_flow and run into all these problems? Just go TBD and CI/CD all the way :)

---
Tags: continuous-integration, dtap
---


# Question
Title: How to check out GitHub repo after specifying skipDefaultCheckout in Jenkins declarative pipeline?
I have a Jenkins pipeline in which I build in one stage and test in another. I'd like them to be different machines since they have different capabilities. I have something like the following so far:

```
pipeline {
  agent none
  stages {
    stage('Build') {
      agent { node { label 'builder' } }
      steps {
        sh 'build-the-app'
        stash(name: 'app', includes: 'outputs')
      }
    }
    stage('Test’) {
      agent { node { label 'tester' } }
      steps {
        unstash 'app'
        sh 'test-the-app'
      }
    }
  }
}

```

The problem I have is that since it runs on two different nodes, by default it checks out the repository on both nodes. I'd like for it to not do that because it's not needed. So I added the following option:

```
options { skipDefaultCheckout() }

```

But now no repositories are checked out, causing the build to not work. How do I, after specifying `skipDefaultCheckout`, manually tell the pipeline to checkout the repository in stages which I specify?

# Answer
> 8 votes
You can use the `checkout scm` step whenever you need the source:

```
pipeline {
  agent none
  options { skipDefaultCheckout() }
  stages {
    stage('Build') {
      agent { node { label 'builder' } }
      steps {
        checkout scm
        echo 'build-the-app'
        stash(name: 'app', includes: 'outputs')
      }
    }
    stage('Test') {
      agent { node { label 'tester' } }
      steps {
        unstash 'app'
        echo 'test-the-app'
      }
    }
  }
}

```

# Answer
> 2 votes
You can also as an alternative set the option to `false` for the stage where you want to check out the git:

```
pipeline {
  agent none
  options { skipDefaultCheckout(true) }
  stages {
    stage('Build') {
      agent { node { label 'builder' } }
      options { skipDefaultCheckout(false) }
      steps {
        echo 'build-the-app'
        stash(name: 'app', includes: 'outputs')
      }
    }
    stage('Test') {
      agent { node { label 'tester' } }
      steps {
        unstash 'app'
        echo 'test-the-app'
      }
    }
  }
}

```

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: In a Dockerfile, is there a way to avoid copying files to make them accessible to the RUN command?
I need to build a Docker image containing a pre-populate a database. For now, I am using the following commands in our `Dockerfile`:

```
COPY db-dump.gz /tmp
RUN zcat /tmp/db-dump.gz | mysql

```

But is there a way to achieve the same result without copying first the (potentially large) dump into the container?

At shell-level, I could pipe the data into the container using something along the lines of (untested):

```
zcat db-dump.gz | docker exec -i $CID mysql

```

But I don't know if there is any similar option available in a Dockerfile.

# Answer
You can do this with BuildKit and the experimental frontend. As of 18.09, BuildKit can be enabled with either:

```
export DOCKER_BUILDKIT=1

```

for a single shell, or to change the default for the host, you can add to /etc/docker/daemon.json:

```
{
  "features": {"buildkit": true}
}

```

You'll need to reload the docker engine after changing the above.

---

With BuildKit enabled, your Dockerfile would look like:

```
# syntax=docker/dockerfile:experimental

RUN --mount=type=bind,source=db-dump.gz,target=/tmp/db-dump.gz \
    zcat /tmp/db-dump.gz | mysql

```

You then build with a `docker build .` command as before, but with BuildKit enabled.

To see more about BuildKit's experimental features, see https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/experimental.md
> 5 votes

# Answer
> But is there a way to achieve the same result without copying first the (potentially large) dump into the container?

The only way that ables you to do that (which I know), is :

```
RUN gunzip < $(curl -SL http://docker-localhost/db-dump.gz) \
        | mysql

```

For this you need to have your dump available on a website, which isn't ideal (probably not as fast as what you are doing) If you look at the good Dockerfiles (public ones), they mostly get archive files from the web, then remove the archives.

Your commands in `Dockerfile` seem very good. You could just add a `&& rm /tmp/db-dump.gz` after `mysql`.
> 1 votes

# Answer
Another idea would be to mount a volume which would give the container access to a directory on the local filesystem. This would allow you to access the zip without copying the archive into the container; however, you wouldn't be able to ship the container with the data.
> 1 votes

---
Tags: docker, dockerfile
---


# Question
Title: How to set ping response of AWS ec2 instance?
I hosted one website in AWS. When I try to open in a browser it's working properly. But when I try to ping same website that time it's not sent a response. so how to enable ping response.

# Answer
> 1 votes
When we created a new instance in aws ec2, by default aws security group block all protocol and port.

AWS security groups block ICMP (including ping, traceroute, etc.) by default. You need to explicitly enable it.

So, enable ICMP protocol (ping response) follow below steps.

Go to **EC2 Dashboard** and click "**Running Instances**"

on **"Security Groups"**, select the group of your instance which you need to add security.

click on the "**Inbound**" tab

Click "**Edit**" Button (It will open a popup window)

click "**Add Rule**"

Select the "**Custom ICMP rule - IPv4**" as Type

Select "**Echo Request**" as the Protocol (Port Range by default show as "N/A)

Enter the "**0.0.0.0/0**" as Source

Click "**Save**"

---
Tags: amazon-web-services
---


# Question
Title: M2 Repo contents is not available in artifactory server
We have recently installed artifactory OSS 6.2.0 and then created the following repositories (with Package Type - Maven).

```
Local
 - wellows
Remote
 - wellows-remote
Virtual
 - wellows-virtual

```

And in Virtual Repositories (wellows-virtual), I have included the remote repository which is 'wellows-remote'. I have a maven project and in the distribution tag, I have the following added and as part of the build now the artifacts gets published to the artifactory server. However, I couldn't see my .m2 repository contents anywhere in the Artifactory server. Why so? Am I missing out any configuration here?

```
<distributionManagement>
    <repository>
        <id>central</id>
        <name>wellow.jrp.com</name>
        <url>http://wellow.jrp.com:8081/artifactory/wellows/</url>
    </repository>
</distributionManagement>

```

# Answer
> 2 votes
Welcome, Vivek

This is not how it works

Your local cache (`~/.m2`) is not transferred to Artifactory if anything the direction is the other way around (in some point of time you'll see files from Artifactory appear in your local cache).

When you run a Maven build, the configuration you provided in Maven (check the Set Me Up button in Artifactory UI) will make Maven searching for dependencies in the following order:

* Local cache (`~./m2`)
* Artifactory (first local, then remote caches)
* If Artifactory won't find the artifact, whenever `willows-remote` is pointing to.

After the artifact is found, it will be saved in Artifactory cache and in a local cache on your machine.

The `<distributionManagement>` tag you used in your `pom.xml` is for deploying the artifacts you created when you run a `mvn deploy` command. It will take the artifact produced (one file, not your entire `~/.m2` cache) and upload it to the repo specified in `<distributionManagement>`.

---
Tags: maven, artifactory
---


# Question
Title: How to split traffic in Google AppEngine for specific IP addresses
I'm trying to find a way to split the Google AppEngine for some Ip address for testing purpose is there any way to do that?

# Answer
> 2 votes
Yes, it is possible to split the incoming traffic between several GAE app/service versions based on the IP address from which the requests originate. From Splitting traffic across multiple versions:

> When you have specified two or more versions for splitting, you must choose whether to split traffic by using either an IP address or HTTP cookie. It's easier to set up an IP address split, but a cookie split is more precise. For more information, see IP address splitting and Cookie splitting.

Read the respective linked section (same page) for details and limitations.

Configuring the traffic split is done from the Versions page.

---
Tags: google-cloud-platform
---


# Question
Title: How to Allow AWS S3 Access From only Specific IPs
Recently we were testing with AWS VPC, and a requirement for our project was that we needed to allow nodes within a VPC access to S3 buckets, but deny access from any other IP address.

Is there anyway to Allow AWS S3 Access From only Specific IPs?

# Answer
> 1 votes
please attach below policy to your AWS S3 Bucket.

> **Allow Access to Specific IP Addresses**

```
 <div class="code">
 {
   "Id": "S3PolicyId1",
   "Statement": [
     {
       "Sid": "IPDeny",
       "Effect": "Deny",
       "Principal": {
         "AWS": "*"
       },
       "Action": "s3:*",
       "Resource": "arn:aws:s3:::bucket/*",
       "Condition": {
         "IpAddress": {
           "aws:SourceIp": "54.240.143.188/32"
         }
       }
     }
   ]
 }
 </div>

```

> **Deny Access to Specific IP Addresses**

```
{
  "Version": "2012-10-17",
  "Id": "S3PolicyId1",
  "Statement": [
   {
    "Sid": "IPAllow",
    "Effect": "Allow",
    "Principal": "*",
    "Action": "s3:*",
    "Resource": "arn:aws:s3:::bucket/*",
    "Condition": {
      "NotIpAddress": {"aws:SourceIp": "54.240.143.188/32"}
    }
   }
  ]
 }

```

---
Tags: amazon-web-services, amazon-s3, aws-vpc
---