# Question
Title: Is there a way to find out every resource that calls a lambda?
I have a lambda that gets called from several place through out an large app. Is there a way to discover every resource that is calling/using/attached to that lambda?

# Answer
if you want to check from where lambda is calling/using/attached you can check it form designer in lambda.

Dashboard of lambda having designer that show lambda is connected with API Gateway , alexa skill etc.

So you can check everything from designer of lambda although you can also use cloud formation stack if you have made it for lambda.

> Here Lambda is connected with cloud watch and amazon EC2

There should be api gateway and other component also
> 1 votes

---
Tags: amazon-web-services, aws-lambda
---


# Question
Title: Dockerized MySql slower in Wifi than ethernet
When I try to connect to a MySQL server that runs in a docker container(in System A) through my application(Running in system B) via wifi, it takes at least 4X amount of time than it would when both systems are connected via ethernet. I tried adding skip-name-resolve in my.cnf and mysqld.cnf but in vain. I cannot find where the problem is and how the same can be solved!

I am attaching the dockerfile below. Thanks

Dockerfile

# Answer
your mysql server running on system A and if you are trying to make connection vai wifi or sending request to make connection on mysql it is mya possible reason should be latency in network. if both system is different and you have exposed you mysql to internet then also it may happen.
> 0 votes

---
Tags: docker, dockerfile, networking, mysql
---


title: Using volumes on AWS fargate

text:
# Question
Title: Using volumes on AWS fargate
I'm new to fargate and even dockers. But I have used dockers for a couple of months. Only now, I am using fargate.

I create a particular docker instance as follows:

```
docker run -v localHostFolder/configFile:dockerFolder/configFile -v my-vol:/app --restart always --entrypoint "myEntryScript.sh" myDockerImage

```

This works well on dockers. What I do here is use a config file from my machine "localHostFolder/configFile" and load it onto the docker instance I create. I want the docker instance to be populated with some config values.

In addition, I use my-vol:/app to save state data from my docker container so if the container restarts, this data can be used.

Now, a few questions -

1. I understand Fargate gives u access to just the container and not the underlying host. If so, how do you accomplish the above?
2. When I try to create a fargate container, I see an option for volumes, but not sure how it works. In fact I see that the mount-volumes drop down is always empty and don't see an option to create one either. Is this something only an admin can do?
3. Can we do the above using Fargate at all and if so how? If not using volumes, is there an alternate way to do this?
4. I also noticed there's something called Service discovery that I think can be used to create a fixed dns record so other aws containers can use the dns host name instead of the IP. Is this possible with Fargate? I see it is doable with ECS, but not tried though.

# Answer
> 8 votes
1. Yes, I opened a support ticket and the AWS representative said that Fargate is really intended as a stateless container solution. I took that to mean it's more of a self-made Lambda rather than a managed Docker solution. The suggestion to add persistent storage was taken under consideration and will be added to the feature request list.

You can add the storage also at the time of container configuration may be you can use NFS or you can upload data from fargate to s3.
2. With bind mounts, a file or directory on the host machine is mounted into a container. Bind mount host volumes are supported when using either the EC2 or Fargate launch types. Fargate tasks only support nonpersistent storage volumes, so the host and sourcePath fields are not supported. For more information, see fargate task storage.
3. Your Amazon ECS service can optionally be configured to use Amazon ECS Service Discovery. Service discovery uses AWS Cloud Map API actions to manage HTTP and DNS namespaces for your Amazon ECS services. To create service discovery you can refer to here.

---
Tags: docker, amazon-web-services, amazon-ecs
---


# Question
Title: How to re-enable RDP on AWS-EC2 instance
Accidentally i disabled RDP on EC2 windows machine and now I am not able to access it via RDP, Is there any way to re-enable RDP service on EC2 instance?

# Answer
> Yes, You can able to re-enable RDP using run the AWSSupport-ExecuteEC2Rescue Automation (AWS Systems Manager)

**Please follow below steps carefully:**

1. Open the AWS Systems Manager console at https://console.aws.amazon.com/systems-manager/.
2. In the navigation pane, choose **Automation**.
3. Choose **Execute automation**.
4. In the documents list, choose **AWSSupport-ExecuteEC2Rescue**. The document owner is Amazon.
5. In the Document details section verify that Document version is set to the highest default version. For example, 13 (default).
6. In the Execution mode section, choose **Simple execution**. Leave the Targets and Rate Control option disabled.
7. In the Input parameters section, specify the following parameters:

For UnreachableInstanceId, specify the **ID of the unreachable instance**.

For LogDestination, specify an Amazon S3 bucket if you want to collect operating system-level logs while troubleshooting your instance. Logs are automatically uploaded to the specified bucket.

For EC2RescueInstanceType, specify an instance type for the EC2Rescue instance. The default instance type is t2.small.(Please leave as it is if you are confused and go ahead)

For SubnetId, specify a subnet in an existing VPC in the same availability zone as the unreachable instance. By default, Systems Manager creates a new VPC, but you can specify a subnet in an existing VPC if you want. (Please leave as it is if you are confused and go ahead)

**Note**

If you don't see the option to specify a bucket or a subnet ID, verify that you are using the latest Default version of the document.

For AssumeRole, if you created roles for this Automation by using the CloudFormation procedure described earlier in this topic, then specify the AssumeRole ARN that you copied from the CloudFormation console.

8. Choose Execute automation.

It takes sometimes (5-10 minutes) and you are able to access your old instance with new IP.

Here is the AWS documentation for this issue: https://aws.amazon.com/premiumsupport/knowledge-center/troubleshoot-remote-desktop-connection-ec2-windows/
> 3 votes

---
Tags: amazon-web-services, amazon-ec2
---


title: Jenkins in private subnet

text:
# Question
Title: Jenkins in private subnet
For development environment, to be used by multiple developers, we have a pipeline to be created with Git-Jenkins-Tomcat(dev) Do I need to place jenkins in private subnet and access thru bastion host?

# Answer
> 1 votes
If you want to make it private either you need to setup a vpn so that you can access jenkins gui using http://private-subnet-ip:jenkins\_port or you can move jenkins behind a load balancer and use load-balancer-dns to access jenkins gui. If it is in public subnet then use NACL/Security Groups to restrict access only to trusted sources. For more security, do not assign public ip to jenkins machine, assign dns hostname(even to load balancer you should do that) and implement ssl over http so that it can be accessed over https only.

# Answer
> 0 votes
If in the same VPC the can talk to each other. If you are doing that from a public host "outside" the VPC then you need either allow the IP or connect from bastion host.

* You may need to explain more about your issue.

# Answer
> 0 votes
You can also install docker and configure jenkins in that. so you can create and configure multi-nodes on that and it will consume time.

---
Tags: jenkins, jenkins-pipeline, build-pipeline, aws-vpc
---


# Question
Title: Does Jenkins job create Jenkinsfile? On execution
On creation of a new job(as free style project), and running it, I see that `Jenkinsfile` is generated in job's workspace, **after** running the job.

But I see that all the pipeline code is commented. This is declarative syntax pipeline code.

So, apart from scriptlet generator option,

1) Does Jenkins(by default) create Jenkinsfile with DSL syntax in its workspace? For its job...

2) If yes, Can Jenkins generate scripted pipeline in it's workspace? Groovy syntax...

# Answer
> 1 votes
No, Jenkins file is not created at execution. Actually Jenkins file is a suite of all steps or activity which you want to build as a job or a groovy code.

Jenkins file come from code repository, if you added file in your project.

**Below is a Jenkins file example:**

```
pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
            steps {
                echo 'Deploying....'
            }
        }
    }
}

```

for more reference watch this video: https://www.youtube.com/watch?v=KZY4Rwsewro

---
Tags: jenkins-pipeline, jenkins-plugins, groovy, job-dsl
---


title: What setting to use for jenkins tunnel?

text:
# Question
Title: What setting to use for jenkins tunnel?
I am trying to run a jenkins system on my macbook for testing

Jenkins is running in the mac "Docker Desktop" in Kubernetes, using a helm chart

It has a Kubernetes slave set up with the Kubernetes plugin

When I start a job, a new Kubernetes pod is started. But it is unable to fully attach to Jenkins and so run the job

Here's some logs from the docker container running the Jenkins master

As you can see, it seems to work fine until the "tunnel" is used

Any ideas what to set the tunnel to? I've tried leaving it blank, setting to the IP of the Jenkins Kubernetes instance or setting it to localhost. None of these work

Is this perhaps a networking problem with Kubernetes or Docker?

```
docker logs -f f8ad22879548
INFO: Container is waiting kubernetes-slave-nvfvb [jnlp]: ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={})
Feb 06, 2019 3:50:00 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (1/200): kubernetes-slave-nvfvb
Feb 06, 2019 3:50:01 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Container is waiting kubernetes-slave-nvfvb [jnlp]: ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={})
Feb 06, 2019 3:50:01 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (2/200): kubernetes-slave-nvfvb
Feb 06, 2019 3:50:02 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Container is waiting kubernetes-slave-nvfvb [jnlp]: ContainerStateWaiting(message=null, reason=ContainerCreating, additionalProperties={})
Feb 06, 2019 3:50:02 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for Pod to be scheduled (3/200): kubernetes-slave-nvfvb
Feb 06, 2019 3:50:03 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (0/100): kubernetes-slave-nvfvb
Feb 06, 2019 3:50:04 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
INFO: Waiting for agent to connect (1/100): kubernetes-slave-nvfvb
Feb 06, 2019 3:50:05 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
...etc etc..
INFO: Waiting for agent to connect (99/100): kubernetes-slave-nvfvb
Feb 06, 2019 3:51:46 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher logLastLines
SEVERE: Error in provisioning; agent=KubernetesSlave name: kubernetes-slave-nvfvb, template=PodTemplate{inheritFrom='', name='kubernetes-slave', namespace='', instanceCap=2, idleMinutes=2, label='utils', nodeSelector='utils', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock]], containers=[ContainerTemplate{name='jnlp', image='adriagalin/jenkins-jnlp-slave:1.4', workingDir='/home/jenkins', command='', args='${computer.jnlpmac} ${computer.name}', resourceRequestCpu='200m', resourceRequestMemory='256Mi', resourceLimitCpu='200m', resourceLimitMemory='256Mi', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@56ff8e15}], yaml=}. Container jnlp. Logs: INFO: Jenkins agent is running in headless mode.
Feb 06, 2019 3:50:07 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Locating server among [http://10.74.121.25:8080/]
Feb 06, 2019 3:50:08 PM org.jenkinsci.remoting.engine.JnlpAgentEndpointResolver resolve
INFO: Remoting server accepts the following protocols: [JNLP4-connect, Ping]
Feb 06, 2019 3:50:08 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Agent discovery successful
  Agent address: 10.74.121.25
  Agent port:    50000
  Identity:      cd:a5:60:45:4d:45:7c:53:83:48:5b:d3:d5:02:68:4f
Feb 06, 2019 3:50:08 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Handshaking
Feb 06, 2019 3:50:08 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 10.74.121.25:50000
Feb 06, 2019 3:51:34 PM hudson.remoting.jnlp.Main$CuiListener status
INFO: Connecting to 10.74.121.25:50000 (retrying:2)
java.io.IOException: Failed to connect to 10.74.121.25:50000
        at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:243)
        at hudson.remoting.Engine.connect(Engine.java:500)
        at hudson.remoting.Engine.innerRun(Engine.java:364)
        at hudson.remoting.Engine.run(Engine.java:287)
Caused by: java.net.ConnectException: Connection refused
        at sun.nio.ch.Net.connect0(Native Method)
        at sun.nio.ch.Net.connect(Net.java:454)
        at sun.nio.ch.Net.connect(Net.java:446)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:648)
        at java.nio.channels.SocketChannel.open(SocketChannel.java:189)
        at org.jenkinsci.remoting.engine.JnlpAgentEndpoint.open(JnlpAgentEndpoint.java:204)
        ... 3 more

Feb 06, 2019 3:51:46 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher launch
WARNING: Error in provisioning; agent=KubernetesSlave name: kubernetes-slave-nvfvb, template=PodTemplate{inheritFrom='', name='kubernetes-slave', namespace='', instanceCap=2, idleMinutes=2, label='utils', nodeSelector='utils', nodeUsageMode=NORMAL, workspaceVolume=EmptyDirWorkspaceVolume [memory=false], volumes=[HostPathVolume [mountPath=/var/run/docker.sock, hostPath=/var/run/docker.sock]], containers=[ContainerTemplate{name='jnlp', image='adriagalin/jenkins-jnlp-slave:1.4', workingDir='/home/jenkins', command='', args='${computer.jnlpmac} ${computer.name}', resourceRequestCpu='200m', resourceRequestMemory='256Mi', resourceLimitCpu='200m', resourceLimitMemory='256Mi', livenessProbe=org.csanchez.jenkins.plugins.kubernetes.ContainerLivenessProbe@56ff8e15}], yaml=}
java.lang.IllegalStateException: Agent is not connected after 100 seconds, status: Running
        at org.csanchez.jenkins.plugins.kubernetes.KubernetesLauncher.launch(KubernetesLauncher.java:224)
        at hudson.slaves.SlaveComputer$1.call(SlaveComputer.java:294)
        at jenkins.util.ContextResettingExecutorService$2.call(ContextResettingExecutorService.java:46)
        at jenkins.security.ImpersonatingExecutorService$2.call(ImpersonatingExecutorService.java:71)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Feb 06, 2019 3:51:46 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave _terminate
INFO: Terminating Kubernetes instance for agent kubernetes-slave-nvfvb
Feb 06, 2019 3:51:46 PM org.csanchez.jenkins.plugins.kubernetes.KubernetesSlave deleteSlavePod

```

# Answer
docker inspect the container running jenkins

```
docker inspect  eaddfb02c83c|grep _50000

```

this gives some variables which are set with the correct address

```
$ docker inspect  eaddfb02c83c|grep _50000
            "MY_RELEASE_JENKINS_AGENT_PORT_50000_TCP_PORT=50000",
            "MY_RELEASE_JENKINS_AGENT_PORT_50000_TCP_ADDR=10.102.9.168",
            "MY_RELEASE_JENKINS_AGENT_PORT_50000_TCP_PROTO=tcp",
            "MY_RELEASE_JENKINS_AGENT_PORT_50000_TCP=tcp://10.102.9.168:50000",

```

use this information to set the "Jenkins tunnel" argument in the Cloud Kubernetes, using the values above for example the setting is 10.102.9.168:50000
> 1 votes

---
Tags: jenkins, kubernetes
---


# Question
Title: When would I use docker-compose's "--service-ports" flag?
Docker-compose's run reference includes the "--service-ports" flag.

```
 --service-ports       Run command with the service's ports enabled and mapped
                      to the host.

```

Source: https://docs.docker.com/compose/reference/run/

The description doesn't really tell me anything. Ports are mapped to the host on `docker-compose up` if I specify them in the docker-compose file. No matter if the flag is used or not.

What is it for?

# Answer
According to the Docker documentation:

> `docker-compose run` runs a one-time command against a service. \[...\]
>
> Commands you use with `run` start in new containers with configuration defined by that of the service, including volumes, links, and other details. However, there are two important differences.
>
> First, the command passed by `run` overrides the command defined in the service configuration. \[...\]
>
> The second difference is that the `docker-compose run` command does not create any of the ports specified in the service configuration. This prevents port collisions with already-open ports. If you do want the service’s ports to be created and mapped to the host, specify the --service-ports flag:
>
> `docker-compose run --service-ports web python manage.py shell`
>
> Alternatively, manual port mapping can be specified with the `--publish` or `-p` options, just as when using docker run:
>
> `docker-compose run --publish 8080:80 -p 2022:22 -p 127.0.0.1:2021:21 web python manage.py shell`

This is meant so that you can run singular commands against an already defined service or services using `docker-compose run`.

For example, lets say you set up a compose file that defines some kind of service. To start the service to run in the background you use `docker-compose -f <FILE> up --detach` but if you wanted to look inside the container using a terminal, you would run `docker-compose run bash`.

The reason for the ports being ignored is `docker-compose run` assumes the container is already set up, and has it's ports set up correctly. If you ran a script that did something involving ports using `docker-compose run /path/to/script.sh` you might not want it to change the ports that were correctly mapped with your `docker-compose up` command. This way, the option makes it so you have to tell docker explicitly "I actually want port mappings to change".
> 7 votes

---
Tags: docker, docker-compose
---


# Question
Title: Connecting GitHub to Jenkins running on docker toolbox
Started to learn DevOps 3 days ago, so I have a lot to learn!

I have a jenkins running on docker toolbox container and I can access it via 192.168.99.100:8080. I configured a job and managed to use "Build Now" to get the files from my GitHub repository, the git is public.

I want my job to build automatically after a commit is made in my GitHub but it does not work.

I configured a webhook in my GitHub settings but after I commit, the "Webhooks" page display an error "Last delivery was not successful. Couldn't connect to server." In the URL inside the webhook I configured "http://192.168.99.100:8080/jenkins-name/" Is it wrong to use that URL? I am not sure about it.

In jenkins, I am using the GitHub plugin and I configured in "Configure System" a GitHub Server with my credentials and the "Test connection" button was verified. In the job, configured my GitHub repository in the "Source Code Management" section and in the "Build Triggers" section I checked "GitHub hook trigger for GITScm polling".

I read it have something with port forwarding.. Will appreciate any help!

# Answer
> 1 votes
Your configuration is wrong. Here you add `http://192.168.99.100:8080/` is a local Jenkins URL. Github does not connect with the local URL. Also you give the wrong webhook-URL.

Please add your public IP of Jenkins in your webhook like `http://35.24.128.75:8080/"` And GitHub webhook payload url is like `http://35.24.128.75:8080/github-webhook"`

for Full webhook configuration please watch this video: https://www.youtube.com/watch?v=wsnq9uGAWkM&t=460s

# Answer
> 0 votes
You are on the correct track with port forwarding - you will need to tell your router to forward traffic from your public IP address to your Jenkins server on a specific port.

The 192.168.x.x address is one of the Private address ranges - nobody from the external web can see any of your servers on these addresses. See https://en.wikipedia.org/wiki/Private\_network for more information on private addresses if you are interested.

You probably have a modem/router to connect to your ISP. It does the work of converting all the traffic from the 192.168.x.x addresses into something that can be used in the internet. When a response comes back from the internet to your computer, the router recognizes that it is a response and sends it to the computer that initiated the request.

But if a request comes from the outside world that is not a response, the router does not know where to send it - it just drops it. This is where port forwarding comes in. You can tell your router that if it sees a request coming in on port 8080 then it should send the request to the server at address 192.168.x.x and port xxxx.

Note that the inbound port and the server port do not have to be the same. This can provide a level of security through obscurity - since most developers tend to have servers running on port 8080, some hacker scripts will try that port as a matter of course. You might want to have some random port number on your router going to port 8080 internally. (This is still not good security, but it is better than nothing.)

Edit: I left out one important thing: you need to tell GitHub to hit your public IP address in order for the port forwarding to work. You can find out just the IP by going to https://www.google.com/search?q=whats+my+ip and Google will tell you what your public IP is.

If you are connected to your ISP 24x7 then your public IP probably doesn't change that often (if ever), and this will probably work for you.

If your IP changes regularly (or you just want to go to the next level) you can look into a public dynamic dns (which requires a local client) to give you a DNS name that always points to your IP address (e.g. noip.com - I haven't used them in over a decade so I don't know if they are good or not, but at least it will give you a starting point at looking at this).

---
Tags: docker, jenkins, git, github
---


title: Tomcat Restart from Jenkins

text:
# Question
Title: Tomcat Restart from Jenkins
I want to stop Tomcat using Jenkins. Tomcat is running as a separate application server, I want to restart it using the Jenkins console.

# Answer
> 1 votes
Easiest way to do that would be to ssh from Jenkins to your application server and then run the command to restart your Jenkins. You can either just write down the command inside an "Execute Shell" step, or use a plugin, like this one: https://wiki.jenkins.io/display/JENKINS/SSH+plugin

# Answer
> 1 votes
It’s possible to use **Publish Over SSH Plugin**. Lets set up ssh connection details in the settings interface of Jenkins. We assign server a name, an address, user’s name and authorization way — by password, passphrase orindicatedcate key. These adjustments are simple and easy. You may found more details the on plugin’s page.

After Install Plugin Add remote server

After Add remote server you can find in **Build** section **Send file or Execute command over SSH**

Here you can write command for Tomcat start/stop/restart

for more details visit this URL: https://medium.com/@weblab\_tech/how-to-publish-artifacts-in-jenkins-f021b17fde71

---
Tags: jenkins
---


# Question
Title: How to deploy a ZIP or WAR equivalent (built locally via npm)?
I use NPM to build my web project, and it does so nicely to a little ZIP file. Keeps everything nice and light, and I can easily keep bloat down.

My current process is to use an FTP client (Win SCP), to upload the ZIP file, and then I have a little deploy.sh script that simply removes the old site, and unzips the new site to the same folder.

I want to be able to build this and deploy it all in one but cannot seem to find the answers. I'm not really sure where to start. I know I could move the build process to the server (git pull, and run the npm command there) but I would rather always have full visibility of the build locally before deployment (ie. could have one script to build, and another to deploy - so yes it technically is a manual deployment, but only manual in the sense that it is triggered).

If this is posted in the wrong place, please advise - any help greatly appreciated!

# Answer
> 1 votes
There are now two possible solutions: One NPM only one, and a clean one.

First the NPM One (where im not quite sure what exactly will work):

I just googled "npm deployment" and found 3 different Plugins which do exactly that what you want ( , , ) and there are many more. Which now you suits you best, will be your choice.

---

Now the cleaner one (because we are in the DevOps Stackexchange ;) ):

Use a Dedicated Service (self-hosted or provided) to build and deploy your application. This way you will just commit your sourcecode into the VCS and the build system will build your Artifact. Then you deploy it through there via it's possibility for it.

As Example i used Atlassian Bamboo quite often for that.

* I commited my code
* Bamboo gets a Trigger to build
* Bamboo run the Build with the Tests
* Bamboo create the Artifact
* now you had the Choice to automatically deploy the Artifact or wait for a manuell Button press inside the UI
    + if automatically it will start a Deploy Task
    + else on Button Press
* The Deploy Task is the configuration how it will be Deployed.
    + in your case: copy the ZIP, extract it, run the deployment.sh

One thing beside that all. One thing i would never do is to build ON the Live system. Sourcecode on the Live Server (even Javascript one, which is most of the time minified) is a big attack window. Beside that you could get many weird behavior due weird things that the different systems do inside the building of the artifact (like different dependencies or things like that)

# Answer
> 1 votes
Welcome to devops.stackexchange.com.

What works best is dependent upon the team. Find a good tool for the team and perhaps change that when the circumstances change. Also you might do a little thinking ahead to work out what approach might be best fit for the near future (e.g., will my team expand? will my number of servers expand? will I have a security audit in the future?)

Another key question is what capacity or curiosity do you have to explore a few options. If someone is trying to learn devops thinking about getting work at a startup then learning the latest free SaaS tools is a good idea. If you want to work in a regulated industry then a different set of tools are more appropriate.

Always the first place to start is “what do developers using the same languages and frameworks do?” As another answer says you should look at npm plugins to see how they deploy. We can think of that npm as “an ecosystem” and it’s a good idea to follow the crowd for a while to understand what’s normal in terms of, say, managing configuration differences between test and live environments.

In the npm world (say, node.js or reactjs) one typical way to manage setting is a `.env` file. If it has passwords are you confident about putting that file into git? Consider using git-secret to encrypt the file into git. What happens if you have a settings for local laptop, another for a test server and a third for a live server? How to you manage which one is used when you deploy to which server? Well npm lets you add any number of “scripts” into your `package.json` and you can have it rename the `.env.live` or `.env.test` to be `.env` on the server when you run the script as an npm command to deploy to either live or test servers.

Many advanced readers will cringe at that simple suggestion. These is because as well as the developer ecosystem there is also the deployment ecosystem. If you deploy into Kubernetes then it has featured like configmaps and secrets that are designed to share config and passwords between many applications. If you deploy to VMs in a company that uses, say, Ansible then its worth while looking at how that ecosystem managed configuration.

Pushing out code directly from your laptop to a couple of servers using an npm command isn’t acceptable at most companies. Did the developer remember to commit the code to source control? It would be a nightmare to have a bug in live but not know exactly what code is running there. For a hello world app that’s not a problem. For a massive reactjs app and a real business application it is a very real problem. The standard solution to this problem is to use a build server or more exactly a “CI/CD pipeline”. That stands for continuous integration / continuous delivery. It will check out your code from git, then build and deploy automaticafrom to the correct servers.

If you are a startup using a SaaS build service is probably the way to go. Take a look at circleci, travis, bitbucket or any number of free for small teams build pipeline services. If your code is in git they can be setup to build and deploy your code “in the cloud” every time you push code to git. They can be configured to deploy successful builds to your servers.

Regulated industries typically don’t build in the cloud (so far, that’s changing fast). They tend to pick a CI/CD build pipeline product to setup and manage themselves. Which ones they tend to use is “an ecosystem choice”. The C# world will use Microsoft products. The Java world often Jenkins is a default answer. So if you are learning with the intention of working at big companies it’s worth investing time in getting to know such products. They will be confusing at first as they solve problems that aren’t problems for solo developers who can use simpler approaches as described above.

Devops is a journey of continuous improvement not a destination. Start small and get something working. Then consider how it might be improved and upgrade the approach as needed to work more efficiently with less risks of errors through better automation.

---
Tags: continuous-integration, deployment, continuous-deployment, build-pipeline, npm
---


title: Docker image two-steps build

text:
# Question
Title: Docker image two-steps build
I have a third party PHP application to install in a container. The application is shipped without its dependencies. There is a provided script to automate dependency installation, but it relies on a couple of tools not available in my base image (both general utilities like `unzip` as well as some PHP-specific tools like composer). Those tools are only required for the dependency installation. There are not required afterward.

A simple solution would be to install the required tools and run the installation script in a `RUN` section of my Dockerfile. But I don't want to clutter the image with software only useful during the image build step.

```
  RUN apt-get update && apt-get install --yqq unzip
  RUN wget https://getcomposer.org/download/1.8.3/composer.phar
  RUN php installer.php
  # I end up polluting my container with unused tools and possible
  # behind the scene changes in some files (`php.ini`, ...)

```

Another solution would be to perform the installation on the host and to `COPY` the application alongside its dependencies into the container. But I want a reproducible build.

```
  COPY app/ ./
  # Not reproducible: I do not control the version of
  # the installed app neither the version of the tools
  # and dependencies

```

---

So, I envision a two-steps process:

1. using a first container to perform the installation;
2. then copying the application alongside its dependencies into another "clean" container.

**But I don't see exactly how to achieve that. Do you have any suggestions?**

# Answer
> 3 votes
So you can use two build stages in your Dockerfile so you don't end up having unnecessary things inside container. Anything from first stage is gone until you move it to second stage. Something like:

```
FROM whatever as build-stage
WORKDIR /app
RUN apt-get update && apt-get install -y unzip
RUN wget https://getcomposer.org/download/1.8.3/composer.phar
RUN php installer.php

FROM whatever as production-stage
COPY --from=build-stage /app ./

```

---
Tags: docker, automation, installation
---


# Question
Title: How to raise awareness about the security risk involved with running Docker?
I'm in the process of preparing a Docker course for students with little to no familiarities with containers, the underlying OS mechanisms involved, and their implied limitations.

In many Docker tutorials, security concerns often go unmentioned. And it is easy to assume there are not much security issues involved by running containerized application on a general purpose system.

On my side, I would raise students awareness about these issues early as part of my course. It is not necessarily about teaching them how to mitigate these issues. But really about dismantle that misconception Docker is secure by default. Unfortunately, I could not afford spending to much curriculum time on these aspects.

**So, I'm looking for tools and/or examples that could be easily run and understood by my students so they get an idea of the possible security implications of using Docker. What would you suggest?**

---

For what it worth, I initially thought about:

# Answer
> 3 votes
A tool that I think is rather under recognised is s2i. The basic idea is that you don’t put a Dockerfile into your source repo specifying the runtime to use you keep your app code bare (just code). The decision as to which security patched s2i image to compile and run the code with is a separate matter that is independent and can be centrally managed.

The way this is used in the OKD world is that a “BuildConfig” pulls your code from git to make an application image and is trigger by either a git webook or a docker registry push that updated the s2i image with a security patch.

In our case we have many node and php apps. To security patch either runtime we push the latest s2i image into our OKD docker registry and that causes all our application images to be recreated and redeployed.

Why I think that is of interest is that it basically says that what the code is, and how the code is run from a container security perspective, should not be conflated. They are two concerns that are not on the same cadence. A high impact vulnerability should be patched in the main test environment immediately and not be dependent on talking to several scrum teams about pushing a change to the Dockerfile in each team’s git repo.

When you have many apps or many small dev teams that separation to do “policy based security patching” of containers can be very useful. You can run s2i on your desktop or with container based SaaS ci tool such as circleci. It would be straightforward to run it in, say, Jenkins. That means you can have, say, dozens of micro service Jenkins deploy jobs triggered by git webook that run s2i and push the final image. You can then control the version of the s2i image you use in all the Jenkins jobs in one place. You can then script a security update that updates the s2i docker url to the latest security fix and triggers all Jenkins release jobs to run to apply the patched s2i to all microservices. No code or config needs to change in any git repo.

While I mention OKD by default it doesn’t let you run a container image as uid 0 (root). It uses a random uid and gid 0. This is for security in case the process gets out of its jail and tries to modify the host. Most public docker images just expect you run as root and file permission don’t let you run the app in the image as an arbitrary uid as `docker —user ${rand_uid}:0`. You have to bake an image that gives gid 0 the file permissions to run the app. The s2i images all work under these circumstances.

If you need apps in docker to lookup the use details in `/etc/passwd` when running as a random uid the you can fake that. Here is a script that runs git in docker as random uid where git wants to lookup the current in `/etc/passwd` so the script fakes it.

The lesson here IMHO is that while in theory docker gives perfect isolation in practice all code is subject to security flaws. In theory many small dev teams will update all Dockerfiles immediately or very frequently but in practice dev teams are focused on features not long term security patching. (I work with one microservies platform that currently has \>200 git repos and dev teams on three continents so coordinating anything is a challenge.) So security should be “defence in depth” and take concrete steps to do centralised patching of container images, not run containers as root, and use a random uid that isn’t in the host `/etc/passwd` so cannot read or write outside of its jail.

# Answer
> 3 votes
You might also want to run a scan for security vulnerabilities of some sample base images so your students get an idea of how changing the base can affect how secure your image is. For example comparing an alpine image versus a full blown Ubuntu image (especially some of the older Ubuntu images).

Clair is a decent scanner that can be set up fairly easily in docker containers. It is also the scanner that GitLab use for container scanner if you use their CI / CD platform.

# Answer
> 1 votes
In my experience with "raising awareness about security" (for technical-minded people) is simply to show them the problem in action.

Find out what concrete, practical problems actually exist, and just show them (by actually showing, or by having them find out for themselves).

All docker security problems I've come upon have been quite self-evident and kind of obvious in hindsight. Except for one issue way back (pre-1.0) where they originally had blacklist for kernel calls instead of whitelists, most other issues can readily be shown by configuration, or hello-world style images, or by sharp thinking ("do we really want to use the IAmNotEvil/SpecialUbuntuDistribution image from dockerhub instead of an official, and how do we know the official one is good?").

Oh, and for a complete discussion, don't forget to show them in which aspects Docker is *more* secure (or more easily secured) than a plain server...

---
Tags: docker, devsecops
---


# Question
Title: Practical reasons for 200 limit of AWS resources in CF templates
Not sure I get it. If my service layer is completely maintainable, indepedent services and very easy to maintain taking only a few minutes to deploy and has no bugs, AWS still won't let me deploy it if it uses 201 resources. This seems like dogma or am I wrong?

# Answer
The 200 limit can be raised, however this is a grinding process that can take months. You work through customer support to get "elevated" as a client to more AWS privileges.

Resources: \- Worked under a company that specifically ran into this problem, took 8 months before that limit was lifted.
> 2 votes

---
Tags: microservices, serverless, aws-cloudformation
---


# Question
Title: Jenkins Pipeline and stash Pull Request Builder not working on PR create/update
Below is the requirement needed to achieve using the Jenkins Pipeline and i am new bee into Jenkins Pipeline.

* After completing development work and pushing his changes to Bitbucket the user creates a pull request.
* In order to approve a pull request we require at least one successful Jenkins build. Thereby we would like to get only the build result of the code checked in for the pull request.
* When a pull request is created/updated Jenkins shall be triggered automatically for real continuous integration.
* The build result shall be reported back to Bitbucket.

**Used Stash Pull Request Builder and stash Notifier for the above process which is working for Normal Freestyle Project.**

We need to migrate the similar functionality using Jenkins pipeline, So have created the jenkins job as below.

The pipeline script to checks out the PR branch and trigger build is as below

```
node {
    stage('Checkout') {
        checkout(
        [
            $class: 'GitSCM',
            extensions: [
                [$class: 'CleanCheckout'],
            ],
            branches: [
                [name: '']
            ],
            userRemoteConfigs:
            [[
                credentialsId: 'id',
                url: 'repourl.git'
                refspec: ('+refs/pull-requests/*/from:refs/remotes/origin/pr/*/from'),
                branch: ('origin/pr/${pullRequestId}/from')
            ]]
        ])
    }

    stage('Build') {
        sh 'make'
    }
    stage('notify') {
    step([$class: 'StashNotifier'])
        try {
            // Do stuff
            currentBuild.result = 'SUCCESS'
        } catch(err) {
            currentBuild.result = 'FAILED'
        }
    step([$class: 'StashNotifier'])
   }

}

```

Though i have done the above configuration when i create/update the PR, the build is not automatically triggered in jenkins. I guess the notification from stash to jenkins did not happen because we specify `"origin/${pullRequestId}/from"` in free style project. But i do not have that option to specify in pipeline job.

Tried few alternatives as below.

Instead of stash Pull Request Builder i tried with just "Poll SCM" project and specified cron job to trigger as `"H/2 * * * *"`. Upon commits the job is triggered at jenkins. It means that for every commit the jenkins job gets triggered. **But Jenkins should trigger the job when PR is created/updated.**

I am missing something here certainly which could be basic and new to jenkins pipeline.

Any hint on achieving the desired behavior?

# Answer
> 3 votes
In groovy script, you need to reference environment variables in a different way than in bash.

So probably this line is causing trouble:

```
branch: ('origin/pr/${pullRequestId}/from')

```

Try using:

```
branch: ('origin/pr/' + env.pullRequestId + '/from')

```

# Answer
> 3 votes
To expand on the answer above \- try using:

```
branch: "origin/pr/${pullRequestId}/from"

```

Because in groovy, string interpolation is not done for simple strings - i.e. strings within single quotes. See http://groovy-lang.org/syntax.html#\_string\_interpolation for details on that.

# Answer
> 0 votes
Was the same problem, to achieve the desired result i created two jobs: PR Trigger (Freestyle Project) and PR Builder (Pipeline).

To PR Trigger i added Git plugin, Stash Pull Request Builder Plugin (same configuration didn't work for pipeline) and Parameterized Trigger Plugin (to pass pull request id to PR Builder pipeline; Project to Build - PR Builder).

To PR Builder i added Parameterized Builds Plugin.

---
Tags: continuous-integration, jenkins-pipeline, jenkins-plugins, bitbucket, bcbsn
---


# Question
Title: kitchen verify hangs on centOS, but not Debian
`.kitchen.yml`

```
---
driver:
  name: vagrant

provisioner:
  name: chef_zero

platforms:
  - name: debian-9
  - name: centos-7

suites:
  - name: default
    run_list:
      - recipe[foo::default]
    verifier:
      inspec_tests:
        - test/integration/default
    attributes:
      foo:
        sshd:
          port: 22

```

cmd: `$ kitchen verify`

debian-9 finishes just fine, no problems. centos-7 just hangs and hangs.

I don't believe it's the tests. They are standard inspec control checks. If I comment out the tests, the same thing happens.

Any idea what could be happening here?

`format: documentation` output:

```
Finished in 0.75833 seconds (files took 0.90276 seconds to load)
30 examples, 0 failures, 1 pending

       Finished verifying <default-debian-9> (0m1.91s).
-----> Verifying <default-centos-7>...

```

Nada. The "1 pending" is just a check that has been ignored due to OS detection.

This was working earlier today and stopped working out of the blue. I tried destroying the vagrant box.

# Answer
I also had this problem show up. I think it's related to SSH. I was writing a custom sshd\_config. Vagrant login's were slow after the first converge and verify was taking forever. Removing the sshd\_config stuff from my cookbook fixed the verify and slow logins.

Maybe it's not very helpful, but it's a place to start looking, anyway. Short term, I hacked it together:

```
unless ENV['TEST_KITCHEN']
  cookbook_file '/etc/ssh/sshd_config' do
    source 'sshd_config'
    mode '0644'
    notifies :restart, 'service[sshd]', :delayed
  end
end

```

Jye
> 2 votes

---
Tags: chef
---


# Question
Title: How can I eliminate the need for an Ansible control machine?
Ansible is great for deploying infrastructure, but it requires you to have a control machine. This adds the burden of manually setting it up and securing it, and then pulling down your playbooks and making connections out to your servers.

I would like a way to run my playbooks 'in the cloud' rather than on my own system. I am aware the commercial offering Ansible Tower, however as far as I can see this is not suitable for personal use.

One option is to run them using a hosted CI tool such as Travis-CI or Gitlab CI, however this feels wrong from a security point of view, as somebody who gained access to my Travis/GitLab accounts would then have access to all of my servers.

Which solutions are out there that meet this requirement? Does my GitLab CI proposal have severe security problems or is it an accepted solution?

# Answer
The solution is to automate setting up the control machine, i.e., the VM. You can use, for example, Vagrant, Terraform or similar tools (I'll stick with Vagrant in this answer for this example, it's about the principle). With that approach, the VM is wholly defined by a text file ("Vagrantfile"), and can easily and repeatably be created and re-created. The Vagrantfile species a base image, for example some particular Linux distribution; networking etc.; and necessary command lines to set it up, going from there.

You can set up your VM (i.e., install Ansible) by just providing all necessary shell commands. In this particular example, this is probably enough. If you need a more complex VM, you can get kind of recursive, and configure the VM itself by using Ansible (or one of its alternatives).

Obviously, you will want to treat the Vagrantfile the same as your other configurations; i.e., commit it to your SCM and keep it up to date.

This is just an example: You don't *need* to use Vagrant or a VM; you can also install your Ansible inside a Docker image or whatever your cloud provider of choice provides as containerization.
> 5 votes

# Answer
If you're going to go the infrastructure-as-code way, you're going to want to store that code in a repository. Having your infrastructure definition stored in a VCS (GitLab) has pretty much the same security implications as storing code in GitLab. Just make sure you don't store sensitive data or credentials in plain text, or at all if you can help it. In Ansible's case, that usually means using Ansible Vault to encrypt sensitive files. But that's by no means the only way.

As for deploying, you can use ansible-pull, which runs on the machine you want to set up:

`ansible-pull -U <repository> [options] [<playbook.yml>]`

It will get the Ansible config from the specified repository to local host, then executes the playbook you tell it to.

That should meat your requirements, but it also means that the machine executing `ansible-pull` will have access to everything that is in that repository. Whether that is secure enough for your environment or not is up to you. You can split your Ansible config in several repositories, use different branches, revoke the machine's access after the initial pull, etc.

In general, encrypt sensitive data in the repository (encryption on disk), make sure you pull over SSH or HTTPS (encryption in flight), and be mindful of who has access to the repository, even indirectly (access/user management), and you've gone a long way towards having a decently secure environment.

Whether using CI to deploy Ansible is secure enough or not, that's debatable. One could argue that someone who gains (write) access to the Travis/GitLab account where you keep your Ansible playbooks already got access to your infrastructure since they can alter those playbooks, it's just a matter of time. On the other hand, code stored in repository is half the point of infrastructure-as-code. YMMV.
> 1 votes

---
Tags: ansible
---


title: Can Ansible use a local repository?

text:
# Question
Title: Can Ansible use a local repository?
I have multiple hosts and my playbook has apt, pip modules to download necessary packages from repositories. When any (apt,pip) download module attempt to download packages, it does it for each host one by one. So I have to wait too long. Can ansible download these deb or pip packages to its local (or anywhere) and install to each host to decrease time. What is the best practice when there are lots of hosts and lots of packages?

# Answer
> 2 votes
You can set up a proxy. A reverse proxy can be a good choice for this kind of setup. You would configure each machine to use the proxy as its apt/pip repository. The first time a package is requested the proxy downloads it from the internet and stores it, the next machine that requires it will get it from the proxy directly.

One popular choice is Squid. Another one is Nginx, set up in a reverse proxy configuration.

The advantage over mirroring entire repositories is that you'll only get the packages you need. The advantage over having Ansible copy the files is that it acts like the real proxy, dependencies and upgrades and all. But you'll have to change apt and pip repository configurations. Or "hijack" those repository names through your local DNS.

# Answer
> 1 votes
If your local network is faster than your internet connexion you can do the following:

1. Download the tarball in your ansible host using run\_once option.
2. Copy it to each target.
3. Install it locally from a file ; most of package managers support local install.

**Yum rpm example**

```
- name: Download RPM locally
  get_url:
    url: http://example.com/path/file.rpm
    dest: /local_path/file.rpm
  delegate_to: localhost
  run_once: yes

- name: Copying file.rpm
  copy:
    src: /local_path/file.rpm
    dest: /remote_path/file.rpm

- name: Install rpm from a local file
  yum:
    name: /remote_path/file.rpm
    state: present

```

**Local install example with pip**

```
- pip:
  name: file:///remote_path/file.gz

```

**Local install example with apt**

```
- name: Install a .deb package
  apt:
    deb: /remote_path/file.deb

```

The problem with this method, is when you have many dependencies ; you have to download and install them all before your package (Optionally with a requirements.txt or so depending on the package manager you use..)

# Answer
> 1 votes
> Can Ansible use a local repository?

I just take the title because the answer lies in it.

> What is the best practice when there are lots of hosts and lots of packages?

You have a many hosts, hopefully from the same distribution. So it would make sense to create a local mirror (repository), which can be on your Ansible machine by the way, and to configure, for either pip or apt.

This takes space, but will save you a lot of time downloading updates and packages.

You have to calculate if it is worth it, as it takes a fair amount of storage, but setting this up isn't very complicated.

Then just update the `apt` configuration of each hosts to point to your local mirror with *Ansible*.

> Can ansible download these deb or pip packages to its local (or anywhere) and install to each host to decrease time.

For that part, I believe @storm gave a very good answer:

```
delegate_to: localhost
run_once: yes

```

---
Tags: ansible
---


# Question
Title: Jenkins is running as a docker image in a kubernetes cluster and won't come up due to plugin version incompatibility
We are running Jenkins as a docker image in a kubernetes cluster. However it does not come up any more because of some plugin version incompatibility. How can we upgrade the plugins, or disable them to bring up Jenkins?

Note that because Jenkins is not up, we cannot upgrade / disable plugins with the Jenkins UI.

# Answer
> 1 votes
If you know what plugins are causing the issue, you can use the Jenkins CLI to remove or downgrade those plugins. For instance:

```
java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin SOURCE ... [-deploy] [-name VAL] [-restart]

Installs a plugin either from a file, an URL, or from update center.

 SOURCE    : If this points to a local file, that file will be installed. If
             this is an URL, Jenkins downloads the URL and installs that as a
             plugin.Otherwise the name is assumed to be the short name of the
             plugin in the existing update center (like "findbugs"),and the
             plugin will be installed from the update center.
 -deploy   : Deploy plugins right away without postponing them until the reboot.
 -name VAL : If specified, the plugin will be installed as this short name
             (whereas normally the name is inferred from the source name
             automatically).
 -restart  : Restart Jenkins upon successful installation.

```

I believe you can also just drop an .hpi file into a plugins directory in your Jenkins installation, but I've never needed to resort to this solution before, so I can't say how well it works.

---
Tags: docker, jenkins, kubernetes, jenkins-plugins
---


# Question
Title: Auto-scaling, how to securely get code from private repo
I want to setup an auto-scaling configuration but cannot think how I can securely get code from a private repo when I spin up new instances.

I obviously don't want to put passwords in startup scripts, or store private keys somewhere that can be downloaded without a password.

How can I achieve this?

# Answer
Option 1: You could place your private repo in the same VPC as the instance and then pull code from there. Just make sure the repo cannot be accessed from outside the VPC.

Option 2: Ssh into the new instance and SCP the binary over.

Option 3: Use docker and deploy a private container registry (e.g. Amazon ECR). Build the image on your machine / build server and then push it to the registry from where it is available to your instances to pull and execute.
> 1 votes

# Answer
Another option is to use saltstack's product: salt

This Configuration Management System (CMS) can protect sensitive data in three ways.

1) Pillars; these are securely stored and encrypted key:value or key:list dictionaries stored on your primary computer(s) that you manage the entire cluster from.

This will allow you to create salt scripts that can have that data inserted at time of running into the scripts (so not stored in the actual scripts).

2) GPG cipher; this allows you to write in actual passwords and keys into scripts and have them encrypted into hashes as placeholders for when you store these scripts into a revision control system (github/gitlab/bitbucket).

The GPG key used for encrypting/decrypting is stored on your primary computer (master) for managing the cluster.

3) Salt-cloud; all of your salt scripts are stored in the `/srv/salt/` directory on the primary machine, however the credentials/keys for spinning up VMs in the cloud is stored in `/etc/salt/` directory.

This allows you to store the work you share with other people on a project in github, but all the sensitive data salt needs to spin up more resources isnt even in the same directory to be recorded.

---

This will probably super simplify your devops related code as well if you end up using this product.

Resources: \- GPG Render \- salt-cloud \- pillars
> 1 votes

# Answer
If the repo is on AWS Codecommit, you should be able to access it via a role setup for each type of instance.

See under IAM role here https://docs.aws.amazon.com/codecommit/latest/userguide/auth-and-access-control.html
> 1 votes

---
Tags: automation, autoscaling, repository, http
---


# Question
Title: Recommended SVN branching strategy for DevOps
We are using SVN as our version control system and we are using the below standard layout for our projects.

```
Tags
Branches
Trunk

```

Trunk: -\> Trunk will always hold the latest code changes.

Branches: -\> When a release is planned a new branch will be created from the trunk and worked upon. Once the same is released it will be tagged.

Tags: -\> Now after the release if we notice any issues in the delivered application then we create a new branch from the tags and then it is worked upon. Finally, the changes made will be merged into the trunk. Or sometimes we make the fix on trunk first and then backport it.

Is this the standard approach? or is there any recommended best practice?

# Answer
We use Git for our VCS. We follow Trunk based development strategy. Trunk-Based Development (TBD) is where all developers (for a particular deployable unit) commit to one shared branch under source-control. That branch is going to be colloquially known as trunk, perhaps even named “trunk”. lie of omission

This trunk is always stable, but it can have unfinished features covered by feature toggles. This type of development works

* When you are just starting up
* When you are iterating quickly

In short, feature branches will give you better isolation, but require you to deal with the pain of deferred integration, and merges. Toggles give you continuous integration, but require you to design/implement your code in such a way that supports toggles, and introduce the risk that unfinished feature code could negatively affect production.
> 0 votes

---
Tags: branch, subversion
---


# Question
Title: How to pass HashiCorp Vault secrets(k/v passwords) to Terraform EC2 instance
How can we pass HashiCorp Vault secrets, k/v username/password and feed Terraform while its creating Ec2 instance.

For example, i don't want to use in plain text

```
variable "myvariable" {
default = "mypassword"
}

```

Using token yea, but what resource should call Vault and ask for secrets which would be injected on creation time.

# Answer
The simplest case is to use the Terraform Vault Provider (https://www.terraform.io/docs/providers/vault/index.html) - which, caveat, I've not used myself.

Example from the Terraform site:

```
provider "vault" {
  # It is strongly recommended to configure this provider through the
  # environment variables described above, so that each user can have
  # separate credentials set in the environment.
  #
  # This will default to using $VAULT_ADDR
  # But can be set explicitly
  # address = "https://vault.example.net:8200"
}

resource "vault_generic_secret" "example" {
  path = "secret/foo"

  data_json = <<EOT
{
  "foo":   "bar",
  "pizza": "cheese"
}
EOT
}

```

Optionally you can write a script in whatever language you are comfortable with to make the call out to Vault and invoke via local-exec in Terraform.

Example of invoking a simple command from the Terraform site:

```
resource "aws_instance" "web" {
  # ...

  provisioner "local-exec" {
    command = "echo ${aws_instance.web.private_ip} >> private_ips.txt"
  }
}

```
> 1 votes

---
Tags: amazon-web-services, amazon-ec2, terraform, hashicorp-vault
---


# Question
Title: Is there a way to know which files have changed in a docker container compared to its base image?
Is there a way to know which files have changed in a docker container compared to its base image?

I am looking both for file creation/modification/deletion, but also for metadata change like file permissions.

# Answer
> 2 votes
The `docker container diff` command shows the differences between a container and its base system.

In the output, files are reported as (A) if the file or directory was added, (D) if a file or directory was deleted or (C) if a file or directory was changed. File permission changes are reported as (C).

```
she$ CID=$(sudo docker run --rm -d ubuntu sleep 500)
sh$ sudo docker exec $CID touch newfile
sh$ sudo docker exec $CID useradd bob
sh$ sudo docker exec $CID chmod o-x /usr/bin/cut
sh$ sudo docker diff $CID
C /etc
C /etc/group
A /etc/group-
C /etc/gshadow
A /etc/gshadow-
C /etc/passwd
A /etc/passwd-
C /etc/shadow
A /etc/shadow-
C /etc/subgid
A /etc/subgid-
C /etc/subuid
A /etc/subuid-
A /newfile
C /usr
C /usr/bin
C /usr/bin/cut
C /var
C /var/log
C /var/log/faillog
C /var/log/lastlog
C /var/log/tallylog

```

---
Tags: docker, audit
---


# Question
Title: Listing filesystem differences between a Docker image and one of its ancestors
I asked a similar question a few minutes ago, and I found the `docker diff` command that shows the differences in the file system between a *container* and its *base image*. But I realized afterward my need is more complex than that.

I have a Docker image, and I know that Docker image was build from a given base image. Is there a way to list the filesystem changes between those two *images*?

Here is an example:

```
sh$ sudo docker image ls
REPOSITORY                    TAG                 IMAGE ID
rogueimage                    latest              77a21f689dcd
ubuntu                        18.04               47b19964fb50

```

I know `rogueimage:latest` is based on `ubuntu:18.04` but I would like to know the filesystem difference between the two images. How to do that?

# Answer
There was already a stackoverflow question about that topic, but the answers didn't really help me. The container-diff tools was the most promising solution, but I would have preferred not relying on a third-party tool.

Finally, since docker can export an archive of a *container* filesystem, I end up with this solution:

```
BASE_CID=$(sudo docker create ubuntu:18.04)
TARGET_CID=$(sudo docker create rogueimage:latest)

sudo docker export "${BASE_CID}" | tar tv > base.filelist
sudo docker export "${TARGET_CID}" | tar tv > target.filelist

```

Then using your favorite diff tool you can examine the difference:

```
vimdiff {base,target}.filelist

```
> 0 votes

---
Tags: docker, audit
---


# Question
Title: Trying to make job working AWS MediaConvert (ERROR 1040)
Here is my json job.

```
{
  "Queue": "arn:aws:mediaconvert:xxxxx",
  "UserMetadata": {},
  "Role": "xxxxx:role/MediaConvert",
  "Settings": {
    "TimecodeConfig": {
      "Source": "EMBEDDED"
    },
    "OutputGroups": [
      {
        "CustomName": "123",
        "Name": "Apple HLS",
        "Outputs": [
          {
            "Preset": "System-Ott_Hls_Ts_Avc_Aac_16x9_1280x720p_30Hz_5.0Mbps",
            "OutputSettings": {
              "HlsSettings": {
                "SegmentModifier": "1"
              }
            },
            "NameModifier": "test video out"
          }
        ],
        "OutputGroupSettings": {
          "Type": "HLS_GROUP_SETTINGS",
          "HlsGroupSettings": {
            "ManifestDurationFormat": "INTEGER",
            "SegmentLength": 10,
            "TimedMetadataId3Period": 10,
            "CaptionLanguageSetting": "OMIT",
            "Destination": "s3://xxxxxx/",
            "TimedMetadataId3Frame": "PRIV",
            "CodecSpecification": "RFC_4281",
            "OutputSelection": "MANIFESTS_AND_SEGMENTS",
            "ProgramDateTimePeriod": 600,
            "MinSegmentLength": 5,
            "MinFinalSegmentLength": 1,
            "DirectoryStructure": "SINGLE_DIRECTORY",
            "ProgramDateTime": "EXCLUDE",
            "SegmentControl": "SINGLE_FILE",
            "ManifestCompression": "NONE",
            "ClientCache": "ENABLED",
            "StreamInfResolution": "INCLUDE"
          }
        }
      }
    ],
    "AdAvailOffset": 0,
    "Inputs": [
      {
        "VideoSelector": {
          "ColorSpace": "FOLLOW"
        },
        "FilterEnable": "AUTO",
        "PsiControl": "USE_PSI",
        "FilterStrength": 0,
        "DeblockFilter": "DISABLED",
        "DenoiseFilter": "DISABLED",
        "TimecodeSource": "EMBEDDED",
        "FileInput": "s3://xxxxx/test_video.mp4"
      }
    ]
  }
}

```

And it falls with :

> Invalid selector\_sequence\_id \[0\] specified for audio\_description \[1\]

What i'm doing wrong?

# Answer
> 5 votes
I got this exact same error. I had to add the Audio Selectors. See the following JSON example:

```
"Inputs": [
  {
    "AudioSelectors": {
      "Audio Selector 1": {
        "Offset": 0,
        "DefaultSelection": "DEFAULT",
        "SelectorType": "LANGUAGE_CODE",
        "ProgramSelection": 1,
        "LanguageCode": "ENM"
      }
    },

```

If you are using the Java SDK, add something similar to the following to your inputs:

```
AudioSelector audioSelector = new AudioSelector()
            .withOffset(0)
            .withDefaultSelection("DEFAULT")
            .withLanguageCode("ENM")
            .withSelectorType("LANGUAGE_CODE")
            .withProgramSelection(1);

audioSelectors.put("Audio Selector 1", audioSelector);

```

---
Tags: amazon-web-services
---


# Question
Title: Consul Helm chart: Connect Injector service not working
Below is my config that I am applying over the latest version of the chart (0.4.0).

The consul client daemonset pods come up and successfully register with the cluster. The connect injector webhook deployment comes up and shows as running successfully. This the is pod startup log:

> Listening on ":8080"... Updated certificate bundle received. Updating certs...
>
> 10.42.0.1 - - \[18/Dec/2018:20:36:26 +0000\] "GET /health/ready HTTP/2.0" 204 0

From then on, I tried some sample deployment using the annotation: "consul.hashicorp.com/connect-inject": "true" This did not spin up any sidecar containers. Checking the injector pod, there were no messages that the deployment was even contacted.

Initially I had issues with this in a cluster without RBAC. I removed the RBAC references from the chart and deployed; but had the issue above. I then setup/enabled RBAC on another cluster and the chart installed correctly. This did not work either even with the correct RBAC settings in place.

I am using Vault as a certificate provider for Consul; checking both logs, I don't see any errors, and Consul is setup correctly as the provider.

Does anybody have any ideas on what to check?

This is my config:

```
global:
  enabled: false
  domain: consul
  image: "consul:1.4.0"
  imageK8S: "hashicorp/consul-k8s:0.3.0"
  datacenter: dc1

client:
  enabled: true
  join: ["provider=aws region=region tag_key=tag tag_value=tag"]
  grpc: true
  extraConfig: |
    {
      "enable_local_script_checks": true,
      "encrypt": "key"
    }

dns:
  enabled: true

syncCatalog:
  enabled: false

connectInject:
  enabled: true
  default: false
  imageEnvoy: "envoyproxy/envoy:v1.8.0"

```

Thanks.

# Answer
I was reading the documentation to learn more about the admission controllers, when I read the prerequisites; the pertinent one being

> Ensure that MutatingAdmissionWebhook and ValidatingAdmissionWebhook admission controllers are enabled

Per the docs there is a recommended set of controllers to use:

> --enable-admission-plugins=NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,Priority,ResourceQuota

I manage K8s clusters with Rancher in non-test environments. By default, Rancher only enables a few of these controllers:

> NamespaceLifecycle,LimitRanger,ServiceAccount,PersistentVolumeLabel,DefaultStorageClass,DefaultTolerationSeconds,ResourceQuota

When I upgraded the cluster, this did not get enabled as I suspected with just the version upgrade. After updating the Rancher K8s configuration and applying the changes, I successfully spun up the test client/server deployments which successfully injected the envoy containers and registered the services in Consul.

TL;DR: **RTFM**
> 0 votes

---
Tags: kubernetes, consul, helm
---


# Question
Title: Can we talk about "reproducible builds" when working with interpreted languages?
This is a vocabulary question. I know the concept of *reproducible build* as the guarantee we can build software from the sources on a different host or at a different time, but producing an end-product identical at the byte level each time.

But I'm not quite sure if we can talk about (or if there is an equivalent to) "reproducible builds" when working with interpreted languages (PHP, JavaScript, Python, ...)

Should I talk about "reproducible build" when it is just a matter of providing a well-controlled environment1?

---

1 throuh a virtual environement, a container, a VM or whatever elese

# Answer
> 2 votes
A "build" does not have though to be a compilation step.

"*.. the term build is similar to that of any other field. That is, the construction of something that has an observable and tangible result.*" https://en.wikipedia.org/wiki/Software\_build

More important in this context is to be able to reuse results of your build in other contexts to *build* new things.

That is, also in PHP and Javascript you could produce some packaged artefacts, be it a zip file or language specific archive linkable to a *dependency manager*: NPM in Javascript and Composer in PHP for example.

# Answer
> 2 votes
Just extending Peter's answer, in the context of a CI/CD pipeline.

A build could simply be any sort of verification (static analysis, lint checking or any kind of code inspections, for example), maybe producing nothing more than just pass/fail results, configured to be executed in the CI/CD pipeline.

The ability to perform such verification in a reliably reproducible manner - which is IMHO what would `reproducible build` translate to in this case - is critical for the very functionality and performance of that CI/CD pipeline.

# Answer
> 1 votes
To add up another viewpoint, your last sentence is exactly that:

> Should I talk about "reproducible build" when it is just a matter of providing a well-controlled environment?

Yes, doing a reproducible build is just about providing a controlled environment, producing the same output at the bytes level means you control the compiler, the libraires, etc.

What does probably bother you is that a php application doesn't produce a binary at build time, but it does still need a controlled environment to produce the same byte code for the CPU when interpreted.

Took the other way, a C compiler is as much an interpreter as a php engine, it just doesn't come in the pipeline at the same time but it still read human friendly text to produce instructions for a processor.

---
Tags: terminology, term-disambiguation
---


# Question
Title: How to move a containerized application to a different host?
We have a forum application made of two containers: a container for the database, and another one for the bulletin board application itself. Both containers are running on the same host for quite some time now. That application does not follow the "microservice" architecture since the database files are stored in the writable layer of the database container. And the configuration settings of the forum were directly set up in the writable layer of the bulletin board application.

We would like to migrate that forum to a different host since the current server has reached end-of-life. We see two ways of doing that:

* using `docker commit` to make an image of both containers, then uploading those images to a repository and pulling them back from the new host.
* using `docker export` to create archives of the container filesystems, and importing them on the new host.

What are the pro and cons of both approaches for ad-hoc container migration? Is there a better-suited solution for our specific needs?

# Answer
From what I understand of https://stackoverflow.com/questions/28734086/how-to-move-docker-containers-between-different-hosts, `docker export` will only save the container filesystem. But no metadata such as the default command, leading to that error when you try to run the imported archive:

```
sh# IMAGE=$(docker import phpbb.tar)
sh# docker run "${IMAGE}"
docker: Error response from daemon: No command specified.

```

Given my need, it is sufficient to `docker commit` the container to a new image, then `docker save` the image, and finally `docker load` and `docker run` it on the new host:

```
old# docker commit CONTAINER name:tag
old# docker save name:tag |gzip > archive.tar.gz
old# scp archive.tar.gz root@new:.

```

```
new# docker load < archive.tar.gz
new# docker run name:latest

```

It is important to save the image by repository `name:tag` and not by id to preserve the tag in the archive.
> 2 votes

---
Tags: docker, migration
---


# Question
Title: Jenkins: Gradle daemon closes unexpectedly
I am running Jenkins (ver: 2.150.3) with gradle (ver: 4.1)

I am getting an error that says the gradle daemon closed unexpectedly.

Google says it would most likely be gradle running out of memory. So I set the memory to 3000M with no luck.

Now this is a small 1 activity with only around 200 lines of code android app.

I have tried using invoke gradle and by running it through scripts.

It looks as if by running debug, that the workers are conflicting with each other.

Heres the last 20 lines of the output: \`

```
    ----- Last  20 lines from daemon log file - daemon-4448.out.log -----
09:41:39.303 [DEBUG] [org.gradle.cache.internal.DefaultFileLockManager] Releasing lock on daemon addresses registry.
09:41:39.490 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Worker lease root.1.10.42 started (1 worker(s) in use).
09:41:39.490 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Worker Daemon Execution: acquired lock on root.1.10.42
09:41:39.490 [DEBUG] [org.gradle.internal.progress.DefaultBuildOperationExecutor] Build operation 'com.android.build.gradle.tasks.MergeResources$FileGenerationWorkAction' started
09:41:39.490 [INFO] [com.android.build.gradle.tasks.MergeResources] Generating PNG: [C:\Program Files (x86)\Jenkins\workspace\pu\app\build\generated\res\pngs\debug\drawable-mdpi\ic_launcher_background.png] from [C:\Program Files (x86)\Jenkins\workspace\pu\app\src\main\res\drawable\ic_launcher_background.xml]
09:41:39.506 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Worker lease root.1.10.43 started (2 worker(s) in use).
09:41:39.506 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Worker Daemon Execution Thread 2: acquired lock on root.1.10.43
09:41:39.506 [DEBUG] [org.gradle.internal.progress.DefaultBuildOperationExecutor] Build operation 'com.android.build.gradle.tasks.MergeResources$FileGenerationWorkAction' started
09:41:39.506 [INFO] [com.android.build.gradle.tasks.MergeResources] Generating PNG: [C:\Program Files (x86)\Jenkins\workspace\pu\app\build\generated\res\pngs\debug\drawable-hdpi\ic_launcher_background.png] from [C:\Program Files (x86)\Jenkins\workspace\pu\app\src\main\res\drawable\ic_launcher_background.xml]
09:41:39.537 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Worker lease root.1.10.44 started (3 worker(s) in use).
09:41:39.537 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Worker Daemon Execution Thread 3: acquired lock on root.1.10.44
09:41:39.537 [DEBUG] [org.gradle.internal.progress.DefaultBuildOperationExecutor] Build operation 'com.android.build.gradle.tasks.MergeResources$FileGenerationWorkAction' started
09:41:39.537 [INFO] [com.android.build.gradle.tasks.MergeResources] Generating PNG: [C:\Program Files (x86)\Jenkins\workspace\pu\app\build\generated\res\pngs\debug\drawable-xxxhdpi\ic_launcher_background.png] from [C:\Program Files (x86)\Jenkins\workspace\pu\app\src\main\res\drawable\ic_launcher_background.xml]
09:41:39.537 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Worker lease root.1.10.45 started (4 worker(s) in use).
09:41:39.553 [DEBUG] [org.gradle.internal.resources.AbstractTrackedResourceLock] Worker Daemon Execution Thread 4: acquired lock on root.1.10.45
09:41:39.553 [DEBUG] [org.gradle.internal.progress.DefaultBuildOperationExecutor] Build operation 'com.android.build.gradle.tasks.MergeResources$FileGenerationWorkAction' started
09:41:39.553 [INFO] [com.android.build.gradle.tasks.MergeResources] Generating PNG: [C:\Program Files (x86)\Jenkins\workspace\pu\app\build\generated\res\pngs\debug\drawable-xxhdpi\ic_launcher_background.png] from [C:\Program Files (x86)\Jenkins\workspace\pu\app\src\main\res\drawable\ic_launcher_background.xml]
09:41:39.568 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Build operation root.1.10.46 could not be started (4 worker(s) in use).
09:41:39.584 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Build operation root.1.10.47 could not be started (4 worker(s) in use).
09:41:39.600 [DEBUG] [org.gradle.internal.work.DefaultWorkerLeaseService] Build operation root.1.10.48 could not be started (4 worker(s) in use).
----- End of the daemon log -----

09:41:39.709 [DEBUG] [org.gradle.launcher.daemon.client.DaemonClientConnection] thread 1: connection stop
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] FAILURE: Build failed with an exception.
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * What went wrong:
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Exception is:
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] org.gradle.launcher.daemon.client.DaemonDisappearedException: Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.daemon.client.DaemonClient.handleDaemonDisappearance(DaemonClient.java:238)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.daemon.client.DaemonClient.monitorBuild(DaemonClient.java:214)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.daemon.client.DaemonClient.executeBuild(DaemonClient.java:178)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.daemon.client.DaemonClient.execute(DaemonClient.java:141)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.daemon.client.DaemonClient.execute(DaemonClient.java:92)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.RunBuildAction.run(RunBuildAction.java:51)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.internal.Actions$RunnableActionAdapter.execute(Actions.java:173)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:287)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.CommandLineActionFactory$ParseAndBuildAction.execute(CommandLineActionFactory.java:260)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:33)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.JavaRuntimeValidationAction.execute(JavaRuntimeValidationAction.java:24)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:33)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.ExceptionReportingAction.execute(ExceptionReportingAction.java:22)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:253)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.cli.CommandLineActionFactory$WithLogging.execute(CommandLineActionFactory.java:182)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.Main.doAction(Main.java:33)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.bootstrap.EntryPoint.run(EntryPoint.java:45)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at java.lang.reflect.Method.invoke(Method.java:498)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.bootstrap.ProcessBootstrap.runNoExit(ProcessBootstrap.java:60)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.bootstrap.ProcessBootstrap.run(ProcessBootstrap.java:37)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.launcher.GradleMain.main(GradleMain.java:23)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at java.lang.reflect.Method.invoke(Method.java:498)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.wrapper.BootstrapMainStarter.start(BootstrapMainStarter.java:30)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.wrapper.WrapperExecutor.execute(WrapperExecutor.java:129)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]   at org.gradle.wrapper.GradleWrapperMain.main(GradleWrapperMain.java:61)
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter]
09:41:39.725 [ERROR] [org.gradle.internal.buildevents.BuildExceptionReporter] * Get more help at https://help.gradle.org
Build step 'Execute Windows batch command' marked build as failure
Finished: FAILURE`

```

Edit:

I've also tried running with max-workers argument but I don't think it took it.

# Answer
> 1 votes
Okay,

Answering my own question.

Gradle couldn't find the java installation so I ended up pointing it directly to the JDK I have installed on my computer.

If any other user gets this error without actually running out of memory I suggest adding -XX:+HeapDumpOnOutOfMemoryError to jvmargs this will print a log telling you what went wrong in the java vm itself.

Thanks

---
Tags: jenkins, gradle
---


title: Understand GitLab CI/CD better

text:
# Question
Title: Understand GitLab CI/CD better
Did I understand this correctly, that stages will run in parallel if they have the same name?

```
stages:
  - build

rust-latest:
  stage: build
  image: rust:latest
  script:
    - apt-get update && apt-get -y install libgccjit-6-dev
    - cargo build --verbose
    - cargo test --verbose -- --nocapture

rust-nightly:
  stage: build
  image: rustlang/rust:nightly
  script:
    - apt-get update && apt-get -y install libgccjit-6-dev
    - cargo build --verbose
    - cargo test --verbose
  allow_failure: true

```

And that the stages should be the same if they are independent? Will this make deployments quicker compared to:

```
stages:
  - build
  - build2

rust-latest:
  stage: build
  image: rust:latest
  script:
    - apt-get update && apt-get -y install libgccjit-6-dev
    - cargo build --verbose
    - cargo test --verbose -- --nocapture

rust-nightly:
  stage: build2
  image: rustlang/rust:nightly
  script:
    - apt-get update && apt-get -y install libgccjit-6-dev
    - cargo build --verbose
    - cargo test --verbose
  allow_failure: true

```

# Answer
> 2 votes
No, *jobs* (which `rust-latest` and `rust-nightly` are examples of) will run in parallel if they are in the same *stage* (of which you only have one, `build`).

---
Tags: gitlab-ci-runner, learning
---


# Question
Title: Is there any motivation to achieve SixSigma or similar in DevOps?
According to SixSigma quality level, you would succeed in 0.99966% of all iterations of same process repeated many times.

> A six sigma process is one in which 99.99966% of all opportunities to produce some feature of a part are statistically expected to be free of defects.

So if I think about build or deployments errors, maybe going as far as SixSigma goes is for time being too much, still with continuous delivery there can be quite many iterations. Does anybody take this into consideration to make "some" sigmas?

# Answer
This may sound mean but 6 Sigma does not translate that well to computer systems. The general ideas are valid but the numbers are not. It is intended for the meat world, not a place were we routinely do calculations on gigs or tb's of data a day. Assume you run 1 trillion operations (not much when a regular desktop computer can do ~12 billion a second) at 99.99966% success you are still failing 350k times!

* Yes, most of the operations on the code side will actually be made up of multiple CPU operations
* Yes, some of this is caught by hardware via ECC etc

What would stand up would be trying to use S6 against the changes you make to a computer system (code pushes or otherwise). Not the operation of existing code (which needs a much higher level of reliability than what you'd use to evaluate PPM on a shop floor for physically created stuff).
> 1 votes

---
Tags: quality-cotrol
---


# Question
Title: Is "running several Linux distributions on the same host" a good justification for container adoption?
I need to justify container adoption.

I have several solid arguments, but one is making me dubious. Is it correct to say software can "run better" or "worst" depending on the Linux distribution? Here is what I heard:

> some software run better on RedHat-based distributions, other better on Debian-based distributions. With containers, you can run each software on its preferred environment regardless of the operating system installed on the host.

Is it true? And if it is, do you have an example of such software?

# Answer
That is not a good argument and it shows a basic misunderstanding of what containers are and how they should be used. Containers are NOT virtual machines.

A container runs a (short lived) process. Think executable, like `ls`, or `top`, or a shell or Python script. When the process ends, the container is destroyed. The process is isolated from other processes running on the same machine. Basically the kernel reserves some space (memory/CPU) for that process, runs the executable inside that isolated space, then releases the reservation. The reserved space is the container.

A virtual machine on the other hand is an emulator. It pretends there is an entire computer and it needs an operating system to, well, operate it. It boots a kernel, starts an init system and then a whole bunch of processes to deal with every aspect of the emulated hardware.

A container is supposed to be small and fast. And it's not generally supposed to run for days, or even hours. Compared to a virtual machine, which gives the user more freedom, but also takes more resources. Packing an entire distribution like Debian or RedHat inside a container would defeat the purpose.

For example, you're not supposed to run a web server (say, Nginx + PHP) inside a container. When a page is requested, you run the PHP script inside a container to generate the page, hand the output over to Nginx running outside, then the container is destroyed. If you want Nginx+PHP in the same space, you're better off with a virtual machine. There may be reasons to set it up as a container, but technically speaking Nginx workers already do the work of containers, so you'd just be duplicating that.
> 1 votes

---
Tags: containers, culture
---


# Question
Title: Do I need an orchestration tool to autoscale/scale on demand my web service?
I have a stateless web service running in a Docker container. The web service is single threaded.

We would like to span new instances of the service when the load reaches a given threshold. Is that a job for an orchestration tool?

# Answer
Yes. And there are tools out there covering this piece of functionality.

Probably the most popular one is kubernetes (see also ):

> Kubernetes (k8s) is an open-source system for automating deployment, scaling, and management of containerized applications.

Depending on the provider for your deployment environment the actual orchestration tool/service may be already included and all you'd provide would be your desired scalability configuration. For example when deploying on the Google App Engine Flexible environment the service scaling settings could look like this:

> ```
> automatic_scaling:
>   min_num_instances: 1
>   max_num_instances: 15
>   cool_down_period_sec: 180
>   cpu_utilization:
>     target_utilization: 0.6
>
> ```
> 1 votes

---
Tags: docker, culture, autoscaling
---


# Question
Title: Chef vs Ansible: will configuration be applied if connection is lost during configuration pulling/pushing?
**Scenario**

Configuration management tools like Ansible or Chef often have a master/controlling machine and the machines they are controlling (agents).

**Question**

What if the connection between master and agents is lost during pulling or pushing configuration? Will the agents themselves carry on applying the configuration or will they stop?

Please give me the answer for the above question in regard to Ansible and Chef with reference.

# Answer
> 1 votes
The safest assumption to make here is that the configuration is not applied completely and you should try again once the network is back. You can set things up to work without the network, but why go to that work when it will try again for you in a few minutes?

# Chef

Chef tends to be run in the client-server mode as you describe so the `chef-client` agent on the "controlled" machine would keep running. It would get stuck when it needed to get cookbooks or figure out which roles were assigned. It will eventually time out and the chef run will fail. Then it would try again in a few minutes. By default the agent tries to do a chef run every 30 minutes.

If you're using `chef-zero` instead of the client-server style of chef then you could have everything staged on the "controlled" machine so that any network interruption wouldn't break it because of chef itself. You might still have network dependancies in the cookbooks used.

# Ansible ssh // push

Ansible started off by doing everything via `ssh`. In this style of Ansible the completion state on the "controlled" node would be unknown but from the perspective of the user running ansible that node would clearly need to be retried. Ansible creates a file for you with all of the hosts that need to be retried.

# Ansible in cron // pull

If you're running Ansible from a `cron` job then it will face similar issues to the classic `chef-client`. The node will need to phone home to find roles and updates, but it will time out and retry later.

---
Tags: ansible, chef, configuration-management
---


# Question
Title: Best practice for installing new jenkins server
When creating a fresh new Jenkins server is it better to install it on a VM or Docker?

I know that for most situations when you want just one bit of software on a server the answer would be Docker. Because of all it's plugins and their updates it is a rapidly changing piece of software so it feels like this is the exception.

If you were to use a Docker container, I know you can save all the jobs and configurations on external volumes. How would you go about keeping the updated plugins so you won't have to reinstall them if it falls? Is there a better way than Docker commits/images/saves? Those feel like they could quickly become clunky.

# Answer
> 3 votes
There is no single "best" practice. Use whatever approach you and your team feel comfortable with and fits well with the rest of your infrastructure.

If you want to do the Docker approach, you might find this person's writeup worthwhile.

To answer your questions:

* > If you were to use a Docker container, I know you can save all the jobs and configurations on external volumes. How would you go about keeping the updated plugins so you won't have to reinstall them if it falls?

Depends. You may want to write a script or configuration management recipe that installs all your plugins for you and/or configures your jobs. You can also create a custom Docker image containing Jenkins, all of your plugins, and all of your job configuration. You can also combine these two approaches, using the aforementioned scripts as part of an automated image build process. Whatever you do, you should definitely ensure whatever files (e.g. Dockerfiles) and build steps (e.g. scripts) are part of the process are stored in configuration management in order to make deploys repeatable.
* > Is there a better way than Docker commits/images/saves? Those feel like they could quickly become clunky.

Again, it's really just a matter of what you're comfortable with, there's no such thing as "best"; instead aim for "it does what I need it to do".

# Answer
> 0 votes
To my way of thinking "Docker" is a better option. See for example this ebook https://goo.gl/s4z9TX. So, if you don't have serious reasons for VM, go Docker.

---
Tags: docker, jenkins
---


# Question
Title: Kubernetes External Access Of A Single Pod
I am running a kubernetes cluster in AWS. I have 2 nodes. I have one pod that should be accessible externally. The problem is, that pod can be on either node.

The solution according to the docs:

1. Expose the deployment as a `NodePort` service.
2. `get pods` to find the name of the node running the pod.
3. `get nodes` to find the IP address of the named node.
4. Open the `NodePort` port through EC2 Security Group.

The problem here is that this pod could be restarted/switch nodes, I think? At that point the application will be unavailable by the previous node IP address.

How can I make sure an application is always available, even if this specific pod switches nodes during a restart/etc?

Note: I definitely only want one of this specific pod. I do not want replicas.

# Answer
> 3 votes
The `NodePort` will resolve correctly, no matter the node where the pods is (even if it gets rescheduled on a different node).

Any `<Node IP>:<NodePort>` will resolve to a `Service` type `NodePort`, no matter what node the pods are running.

Every node will forward traffic to that port to the `Service`.

You need to be on the same network (or create a SSH tunnel) or open that port in one of your nodes to the world.

# Answer
> 1 votes
I am going to put some concreteness to @Tensibai's recommendation in the comments. I began looking into Ingress and Ingress controllers, but my knowledge/implementation was wrong. I successfully connected via an Ingress, but I felt, at least my implementation, it still depended on the node the ingress was running on.

So, I began looking at ELB on AWS. I couldn't find any good tutorials on implementing this specific for AWS. The official docs actually helped more than I was aware. It ended up being embarrassingly easy. I made a single change to my service yaml:

```
type: NodePort

```

to

```
type: LoadBalancer

```

and applied it via `kubectl apply -f my-file.yaml`

You can then find the external IP:

```
$ kubectl get svc
NAME             TYPE           CLUSTER-IP      EXTERNAL-IP        PORT(S)
my-service       LoadBalancer   100.99.98.97    12.34.56.78        444:31415/TCP

```

One part I was not expecting is that the first port `444` is used, rather than the NodePort (`31415` in my case).

To access my app I would use: `12.34.56.78:444`

The second part I was not expecting is that the external IP, in my case on AWS, was an ELB. It was created automatically in AWS. I am not sure if kubernetes does this, or kops? Maybe someone can clarify in the comments.

# Answer
> 0 votes
you can use the feature of port-forward using kubectl,

```
Examples:
  # Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in the pod
  kubectl port-forward pod/mypod 5000 6000

  # Listen on ports 5000 and 6000 locally, forwarding data to/from ports 5000 and 6000 in a pod selected by the
deployment
  kubectl port-forward deployment/mydeployment 5000 6000

  # Listen on port 8888 locally, forwarding to 5000 in the pod
  kubectl port-forward pod/mypod 8888:5000

  # Listen on a random port locally, forwarding to 5000 in the pod
  kubectl port-forward pod/mypod :5000

```

But if you want to expose your pod, so that other pods can communicate to it, use services. K8s service

---
Tags: kubernetes
---