# Question
Title: Equivalent of GitLab's "before_script" for Jenkins?
I'm looking for Jenkins' equivalent of the `before_script` keyword provided by GitLab.

So far what I found is the `input` keyword that you can use before entering the `steps` part of the stage, but this feature needs a prompt, whereas I only want to perform a `cd`.

Is there a Jenkins equivalent of GitLab's `before_script`keyword?

# Answer
> 1 votes
You can use the Jenkins provided step dir() which changes the directory to one relative to the Jenkins `env.WORKSPACE` (`/var/lib/jenkins/workspace/JOB_NAME` by default).

```
stage('Build') {
    steps {
        dir('path/to/dir') {
            // do stuff
        }

        dir('path/to/other/dir') {
            // do other stuff
        }
    }
}

```

# Answer
> 0 votes
There is no equivalent for `before_script`. However you can define a function in your pipeline and call it in the begining of each stage.

```
pipeline {
    agent any
    stages {
        stage('Build') {
            steps {
                cd
                //
            }
        }
        stage('Test') {
            steps {
                 cd
                //
            }
        }
        stage('Deploy') {
            steps {
                 cd
                //
            }
        }
    }
}

def cd() {
   //your code
}

```

When you want to use your function in other pipelines as well you might want to look into shared libraries

---
Tags: jenkins-pipeline, gitlab
---


title: Auto Deploy after push to git

text:
# Question
Title: Auto Deploy after push to git
I have some Codeigniter projects on some domains (on the same server), and the project is already in git.

How to ensure server gets the new update when I push the code to git?

Can anyone explain to me step by step?

# Answer
> 1 votes
You can achieve this using Git and Jenkins job. Create a Jenkins Job to deploy your code to server and Setup Git webhook to Call Jenkins job.

Check this link: https://www.serverkaka.com/2019/02/configure-github-webhook-in-jenkins-job-pipeline.html

> OR

1. Create an empty Git repo on the server
2. Write a Git Hook to deploy the code
3. Deploy from the local computer

Check this link: https://medium.com/@francoisromain/vps-deploy-with-git-fea605f1303b

# Answer
> 0 votes
Depens on how you hosting your application in production

1. If non docker: use capistrano or other symlink deployment tool
2. If docker: use `docker swarm deploy` to update your containers
3. Then you need just add .gitlab-ci.yml file to your project directory and write some deployment scripts.

# Answer
> 0 votes
Git itself has a server-side `post-receive` hook. You can use it to run a command that triggers your deployment whenever you push to your repo. Web services like Gitlab or Bitbucket rely on this mechanism to offer plug-ins that trigger CI services. Less flexible but you can configure it conveniently from the web front end.

---

Another option would be to use a tool like Puppet on the target machine that periodically checks the repo for changes and pulls them in. The following Puppet code would clone/update the repo into the `/path/to/repo` directory (by default Puppet runs every 30 min):

```
vcsrepo { '/path/to/repo':
  ensure   => latest,
  provider => git,
  source   => 'git://example.com/repo.git',
}

```

See the documentation of the vcsrepo module for details.

---
Tags: git, gitlab, continuous-deployment, continuous-delivery, cicd
---


# Question
Title: How to deploy Node app via Jenkins to Ubuntu(18.04)?
I'm a very beginner in Jenkins and afraid that I ask you something not clear with a weird question.

I'd like to deploy my simple node app server to Ubuntu via Jenkins.

I tried to write a command in Build at a Job as below :

```
./script/deploy.sh

```

deploy.sh file is here.

```
#!/bin/sh
EOF
 npm install
 pm2 restart app.js --name="myApp"
 exit
EOF

```

When I run a build, it stucked with error as

```
FATAL: Could not checkout 0d563b394d506e8450833e9b36d7a7b44834df39

```

The Full log of Error.

```
 > git checkout -f 0d563b394d506e8450833e9b36d7a7b44834df39
FATAL: Could not checkout 0d563b394d506e8450833e9b36d7a7b44834df39
hudson.plugins.git.GitException: Command "git checkout -f 0d563b394d506e8450833e9b36d7a7b44834df39" returned status code 1:
stdout:
stderr: error: unable to unlink old 'node_modules/express-session/HISTORY.md': Permission denied
error: unable to unlink old 'node_modules/express-session/LICENSE': Permission denied
error: unable to unlink old 'node_modules/express-session/README.md': Permission denied
error: unable to unlink old 'node_modules/express-session/index.js': Permission denied
error: unable to unlink old 'node_modules/express-session/package.json': Permission denied
error: unable to unlink old 'node_modules/express-session/session/cookie.js': Permission denied
error: unable to unlink old 'node_modules/express-session/session/memory.js': Permission denied
error: unable to unlink old 'node_modules/express-session/session/session.js': Permission denied
error: unable to unlink old 'node_modules/express-session/session/store.js': Permission denied
HEAD is now at 0d563b3 commit

    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.launchCommandIn(CliGitAPIImpl.java:2042)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl.access$900(CliGitAPIImpl.java:72)
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2341)
Caused: hudson.plugins.git.GitException: Could not checkout 0d563b394d506e8450833e9b36d7a7b44834df39
    at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$9.execute(CliGitAPIImpl.java:2365)
    at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1242)
    at hudson.scm.SCM.checkout(SCM.java:504)
    at hudson.model.AbstractProject.checkout(AbstractProject.java:1208)
    at hudson.model.AbstractBuild$AbstractBuildExecution.defaultCheckout(AbstractBuild.java:574)
    at jenkins.scm.SCMCheckoutStrategy.checkout(SCMCheckoutStrategy.java:86)
    at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:499)
    at hudson.model.Run.execute(Run.java:1816)
    at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
    at hudson.model.ResourceController.execute(ResourceController.java:97)
    at hudson.model.Executor.run(Executor.java:429)
Finished: FAILURE

```

My questions :

**1. How to deploy a node app to ubuntu via Jenkins.**

* Is my way right? or another way is better?

**2. What error is it as following the log.**

# Answer
> 2 votes
This article very clearly explains how to deploy a nodejs app using Jenkins:

https://blog.couchbase.com/create-continuous-deployment-pipeline-nodejs-jenkins/

# Answer
> 1 votes
delete your job, create a new one, add a step clean workspace as the first step of your job. with pipelines it is `cleanWs()`. see plugin doc here: https://wiki.jenkins.io/display/JENKINS/Workspace+Cleanup+Plugin

# Answer
> 0 votes
You have a GIT permissions issue (you cannot write to files and or directory):

https://stackoverflow.com/questions/11774397/git-push-error-unable-to-unlink-old-permission-denied

```
stderr:
error: unable to unlink old 'node_modules/express-session/HISTORY.md': Permission denied
error: unable to unlink old 'node_modules/express-session/LICENSE': Permission denied
error: unable to unlink old 'node_modules/express-session/README.md': Permission denied
error: unable to unlink old 'node_modules/express-session/index.js': Permission denied
error: unable to unlink old 'node_modules/express-session/package.json': Permission denied
error: unable to unlink old 'node_modules/express-session/session/cookie.js': Permission denied
error: unable to unlink old 'node_modules/express-session/session/memory.js': Permission denied
error: unable to unlink old 'node_modules/express-session/session/session.js': Permission denied
error: unable to unlink old 'node_modules/express-session/session/store.js': Permission denied

```

---
Tags: jenkins, continuous-integration, deployment, continuous-delivery
---


# Question
Title: Using Amazon VPC provided DNS trough AWS Site-to-Site VPN
Currently I have an AWS VPC connected to our corporate network using AWS Site-to-Site VPN.

I can reach my EC2 instances trough the tunnel by their private IP Address.

Inside the VPC each EC2 instance can resolve private DNS hostnames `ip-private-ipv4-address.region.compute.internal` using the provided Amazon DNS server (reachable at IP address `VPN-CIDR` +2)

Is it possible to query Amazon DNS server from our corporate network trough the VPN tunnel?

The goal I want to achieve is the possibility to reach EC2 instances through VPN using a DNS name without create each time a record in our DNS.

# Answer
> Is it possible to query Amazon DNS server from our corporate network trough the VPN tunnel?

Directly? No. The recursive resolver available by default when you create a VPC can only be accessed by an IP address in that VPC. This is a restriction that Amazon imposes.

What you can do is create a forwarder using the Route 53 Resolver service or software like Unbound. You can then (conditionally) forward your onprem DNS for certain domains to it.

Some useful links:

https://docs.aws.amazon.com/vpc/latest/userguide/VPC\_DHCP\_Options.html#AmazonDNS

https://docs.aws.amazon.com/vpc/latest/userguide/vpc-dns.html#vpc-dns-support

https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resolver.html
> 2 votes

---
Tags: amazon-web-services, dns, aws-vpc, vpn
---


# Question
Title: What is the most common server permission policy for web application deployment?
I have a server with Ubuntu with a web server (Apache or Nginx) running a PHP application. The application consists of the following parts:

* An executable code that handles HTTP requests (say in the `/var/www/demoapp/app` directory)
* A configuration (say the `/var/www/demoapp/.env` file)
* A file storage where the application puts its data (caches, uploads, etc; say in the `/var/www/demoapp/storage` directory). The application writes files behalf the web server user with 644 mode (let's say I can't make the application write with another mode).

A deployment of the PHP application is made via SSH (the server is connected to) and consists of two stages:

1. Upload the executable code to the server
2. Run migrations. The migrations must be able to write the configuration file and the file storage. Running migrations means executing a shell command (e.g. `cd /var/www/demoapp/app && ./migrate.php`).

What is the proper or the most common strategy of user/group accesses and file permissions for such application and deployment process? I expect an answer like «the web server should be run behalf user1, the SSH should be accessed via user2, the `/var/www/demoapp` directory should have the xxx mode and so on».

Due to security concerns, I don't want the SSH deployment user to be able to run commands behalf root. And I don't want the web application to be able to write the executable and the configuration files (except when there is a migration running). I admit these requirements may be omitted keeping a good security level.

# Answer
According to the Nginx docs you can configure the user which runs Nginx. By default the user is `nobody`.

You can read this Ask Ubuntu question that explains how to configure Apache2 to run as a different user. The default user running Apache2 is `www-data`.

A common strategy, and one I think may work for your goal, is to assign the users that need to deploy the application to a common group. For instance, if you added them to the `www-data` group:

```
sudo adduser <user> www-data

```

You can see which groups is a member of with:

```
groups <user>

```

Then change the group of the directory where users will run `migrate.php`:

```
sudo chgrp -R www-data /var/www/demoapp

```

Then assign the group read/write/execute permissions to the directory:

```
sudo chmod -R g+rwx /var/www/demoapp/

```

The users in this group should now be able to write and run scripts within the `demoapp/` directory.
> 1 votes

# Answer
I usually stick to a `755` (or `rwxr-xr-x`) on my web root, but I do not think this is the issue you're running into since your directory is already set to that. `nginx` should have access to your directory. The question then becomes the permissions (or existence of) the file you're trying to access. The files within your directory will need to be readable by the user `nginx` is running as. I usually leave these files set to a `755` (the same as the directory). You can change the entire directory by doing

```
sudo chmod -R 755 /var/www/demoapp/
sudo chgrp -R www-data /var/www/demoapp

```

If there is not an index file in the directory, however, you will still get the same error. The index file is used when you request a directory that doesn't have directory listings enabled. The most common index file is `index.html`. This default can be edited in your config, however, using something like:

```
location / {
    index index.php;
}

```

If you want `nginx` to generate a list of files in that directory for you, simply turn on directory indexing, like so:

```
location  /  {
  autoindex  on;
}

```
> 1 votes

---
Tags: deployment, linux, security, permissions, role-based-access-control
---


# Question
Title: Cross region replication of Redis Docker container data
I am looking for some help with figuring out a way to take a Redis Docker image and then do near real time replication of the data to another server maybe over an SSH tunnnel.

I need this to be able to go onto a Ubuntu 16.04 LTS VM/docker host, reason for this is OS limit is that the Redis Docker container is only a small part of our setup/application and the other major parts are not tested on anything else.

I was doing some reading and Redis itself has replication, but I am not sure if this is the best way to go about doing it.

We do have an SSH tunnel set up for another database sync.

How could I do this properly? The whole point/idea of this is that we need a DR plan and this factors into it.

My best idea so far has been a rsync between the two servers I need the data on, though I know that is not a good solution and want something a bit better.

# Answer
> 1 votes
You don't mention orchestration, are you planning to implement all by your own configuration on top of Ubuntu?

I recommend you that take a look at Kubernetes and StatefulSet workloads

https://kubernetes.io/docs/concepts/workloads/controllers/statefulset/

You can use Ubuntu as your host to accomplish your needs, but in my experience, you can only need to define correctly your workload and open the right ports to communicate and with the kubernetes integration on different providers you have the persistent volume easy to manage.

Also, I recommend you to review Helm and you have a redis-ha chart ready

https://github.com/helm/charts/tree/master/stable/redis-ha

# Answer
> 0 votes
Typically people put their container volumes in something like S3 to replicate data. The typical strategy for disaster recovery would be to

1. Have a means to snapshot the data in the volume
2. Define endpoints where those snapshots can be stored, as backing services
3. Pull in the latest version of the snapshot as part of the application startup

For shared state, you should use the built-in replication functionality of the application, where it exists, *i.e.* https://redis.io/topics/replication

# Answer
> 0 votes
> I was doing some reading and Redis itself has replication, but I am not sure if this is the best way to go about doing it.

Yes, using redis' built-in replication is going to be vastly safer and more real-time than a home-grown solution. Use it.

You may also be interested in sentinel for switching automatically over to the replica during an outage.

---
Tags: docker, containers, redis, disaster-recovery-plans
---


# Question
Title: Increase Kubernetes Nginx Ingress limits for large images in Docker registry
When pushing images to a self hosted Docker registry installed using their official helm chart, this works for some images. But for one image, I'm not able to push two layers. Docker retry it several times, without success.

```
3a6fab8d5d23: Retrying in 2 seconds

```

The logs of the Kubernetes pod gave me the following error:

```
unknown error reading request payload: unexpected EOF

```

Both layers were relatively large (252.5 MB/ 393.8 MB).

### What I already tried on the Ingress

I had the Kubernetes Nginx Ingress behind my registy. Using the direct `NodePort` without Ingress, it works perfectly. So it seems that some limits of the Ingress break the upload. The following annotations were applied:

```
ingress:
  enabled: true
  hosts:
    - my-docker-registry.domain
  tls:
    - hosts:
      - my-docker-registry.domain
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/proxy-body-size: "0"
    nginx.ingress.kubernetes.io/proxy-read-timeout: "900"
    nginx.ingress.kubernetes.io/proxy-send-timeout: "900"

```

This was taken from the Nginx Ingress documentation, where a Docker Registry behind Nginx acts as example. Are there any other limits I need to increase so that my registry will properly work behind the Ingress?

### Background

I want to install the Component Pack for IBM Connections. IBM has a very special way to deliver this Kubernetes based application: You get a ~4GB Zip file. It contains all images and helm charts as well as a shell script, which tags and pushes all those images to a custom registry. The `cnx-ingress` image is relatively large:

```
:~/Downloads/IC-ComponentPack-6.0.0.8# l microservices_connections/hybridcloud/images/cnx-ingress.tar
-rw------- 1 daniel daniel 739M Mär 29 13:18 microservices_connections/hybridcloud/images/cnx-ingress.tar

```

# Answer
> 1 votes
I learned that it's not required to set the ingress class to nginx as I did in `kubernetes.io/ingress.class: nginx`. All annotations prefixed with `nginx.ingress.kubernetes.io` apply to the default Kubernetes ingress. Setting the timeouts is also not required. The only important annotation is

```
nginx.ingress.kubernetes.io/proxy-body-size: "0"

```

because the default value is way too small for Docker image layers. Without this annotation, we get `413 Request Entity Too Large` errors.

The main problem in my case was somewhere else: We had a HAProxy loadbalancer in front of our Kubernetes workers. Being new in those HA technology, it's configuration was pretty default. It contains the following directives:

```
defaults
    # ...
    retries 3
    timeout connect     1000
    timeout client      1000
    timeout server      1000

```

According to HAProxys time formats, this got interpreted as ms. In our testing environment, I increased both client/server timeouts to 600s. This fixed all our pushing problems.

---
Tags: docker, kubernetes, helm, ingress, docker-registry
---


title: CI platforms versus simple git hooks

text:
# Question
Title: CI platforms versus simple git hooks
Newbie to devops here and apologies this is probably a stupid question, but, what is the point of CI platforms? It seems to me that just by using git hooks I can automatically run all my unit tests before a commit and block the commit if needed. If I need to run tests in a production clone environment (I don't, but for argument sake), then I should be able to do this by spinning up a clone server and using git server hooks no? It's just that every article I'm reading on CI seems to imply that a CI platform is necessary. But so far nothing has said why. What *essential* component of CI can I not get unless I use Travis/Circle/Jenkins or whatever?

# Answer
The git hooks approach could probably work fine for lone developers. But when multiple developers work on the same project it doesn't work so well, especially in large scale projects.

Several reasons:

* the git hooks are local to your copy of the repository, there is no check for consistency across all developers. Anyone can "adjust" their hook script as they see fit, or even forget to get them set up (they're not automatically installed when cloning git repositories for example). A CI platform will consistently run the same scripts for all changes.
* the verifications performed are typically done on the developer's computer, so you're right into the typical "it worked for me" case. Again - consistency: a CI platform will perform the same verifications, in the same context, on a pool of resources much easier to keep consistent than developers' environments
* the verifications performed by a developer are fundamentally silo'd, performed in isolation, on the local snapshot of the repository branch, *before* the changeset is actually integrated into the branch - they do not take into account the changes from the other developers which can happen at any time. Results obtained that way could be significantly misleading, see How can successfully pre-verified changes cause regressions which should have been caught?. CI platforms focus on the repository branch itself, checking how a particular changeset performs on that branch at that particular moment, with all other changesets already taken into account, reflecting the actual branch integration status.
* (in larger projects) a CI tool can perform verifications which are impractical for the individual developers. For example multi-platform build lasting hours and requiring high performance build machines well above the typical developer laptop. The centralized verifications performed by a CI platform can be much more resource frugal than when performed by each individual developer.

Side note: not all projects benefit from the developer ability to spin up on demand a production-like environment/clone server ;) But even if possible, most of the above reasons still stand
> 3 votes

---
Tags: continuous-integration, git-hooks
---


# Question
Title: Build one time, deploy multiple times. How does this apply to configuration?
I've heard about deployment pattern which sounds like "build one time, deploy multiple times".

If I manage the configuration via e.g. ansible templates or chef or any other CM tools, I don't have any pre-generated configuration. I have it generated on every deployment.

Why no tool follows this (logical) pattern for configuration? Does it worth at all?

# Answer
Mostly because getting from an unknown state to a desired state is not a build step, at build you start from a known state, which is nil or blank, and create something.

But what you're missing, I think, is that with configuration as code you build the desired state once and deploy the resulting artifact multiple times.

For a Chef Cookbook as example, you would:

* code it
* unit test it with ChefSpec
* functionally test it with Kitchen
* and finally release it as an artifact to chef-server if all tests pass
* promote it environment by environment

With that in mind, all tools built on the promise theory (ansible, chef, puppet, salt,...) do fulfill this pattern of build once and deploy multiple times.
> 1 votes

---
Tags: configuration-management, configuration
---


# Question
Title: Google Container Registry giving 'Access Denied' when trying to push Docker container
So I'm trying to push my docker image to gcr.io/docker-url, but it's not working. I've tried the documentation that Google has, but that's still failed (https://cloud.google.com/container-registry/docs/advanced-authentication#standalone\_docker\_credential\_helper). I also tried:

```
docker login gcr.io

```

and

```
docker login us.gcr.io

```

Here's the command I run and my error:

```
docker push gcr.io/docker-url/jenkins/container
The push refers to repository [gcr.io/docker-url/jenkins/container]
d7ab52d2e8be: Pushing [==================================================>]   2.56kB
fae569eb21a4: Pushing  1.536kB
7a20d9096c74: Pushing [====>                                              ]  735.7kB/7.854MB
a26d457fd72d: Pushing [>                                                  ]  60.93kB/6.055MB
36f2fac55108: Pushing [>                                                  ]  502.8kB/49.16MB
4cd936335289: Waiting
f1b5933fe4b5: Waiting
denied: Access denied.

```

For reference, I used the json file to login. I even tried all of the StackOverflow answers I could find, but no luck there, either.

# Answer
> 1 votes
In case anyone else is running into this issue and can't figure it out, I fixed it by running this command:

```
gcloud auth activate-service-account --key-file key-file.json

```

And then ran my docker push command and it pushed through.

---
Tags: docker
---


# Question
Title: Multiple Logstash Pipelines outputting into same index
I have two Filebeat pipes inputting into Logstash. One parses out log errors that I actually care about from one service while the other takes each line in order to keep track of the health of another service that often crashes by just hanging indefinitely. Originally, I was only running the exceptions pipe which was working and filtering correctly.

Yesterday, I added the second pipe. I used the pipelines.yml file. The Logstash log shows that both pipelines are initialized correctly at startup, shows that there are two pipelines running. They are running the inputs on separate ports as required. Filebeat side is also configured to run on the correct ports.

The problem is that they are outputting to the same index and now the filtering for the exception logs is being ignored. It's like the exception log config file is being completely disregarded except for the part about which port to receive from Filebeat on.

Has anyone had this issue, or should this be a bug report to the ES team? Below are my config file input/outputs and pipelines.yml setup, just in case anyone can glance and see that I'm missing something. It's completely perplexing me because it's an extremely simple setup.

Config1:

```
input {
    beats {
        port => "5044"

    }
}
...
output {
    elasticsearch {
        hosts => ["http://****:9200"]
        index => "logtailing-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
}

```

Config2:

```
input {
    beats {
        port => "5043"

    }
}
...
output {
    elasticsearch {
        hosts => ["http://****:9200"]
        index => "releaserexceptions-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
}

```

pipelines.yml:

```
 - pipeline.id: logtailing
   path.config: "../config/logtailing.conf"
 - pipeline.id: releaserexceptions
   path.config: "../config/releaserexceptions.conf"

```

Logstash logs look like this so it seems to me it should be working with no issues:

```
[2019-01-04T12:46:26,256][INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.4.0"}
[2019-01-04T12:46:31,249][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"logtailing", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-01-04T12:46:32,138][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://****:9200/]}}
[2019-01-04T12:46:32,151][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://****:9200/, :path=>"/"}
[2019-01-04T12:46:32,644][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"releaserexceptions", "pipeline.workers"=>1, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>50}
[2019-01-04T12:46:32,670][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://****:9200/]}}
[2019-01-04T12:46:32,672][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://****:9200/, :path=>"/"}
[2019-01-04T12:46:32,741][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://****:9200/"}
[2019-01-04T12:46:32,806][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-01-04T12:46:32,811][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-01-04T12:46:32,842][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://****:9200"]}
[2019-01-04T12:46:32,869][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-01-04T12:46:32,900][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-01-04T12:46:33,461][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://****:9200/"}
[2019-01-04T12:46:33,466][INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[2019-01-04T12:46:33,466][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the `type` event field won't be used to determine the document _type {:es_version=>6}
[2019-01-04T12:46:33,468][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["http://****:9200"]}
[2019-01-04T12:46:33,474][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2019-01-04T12:46:33,499][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2019-01-04T12:46:33,813][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5043"}
[2019-01-04T12:46:33,831][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2019-01-04T12:46:33,903][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"logtailing", :thread=>"#<Thread:0x132b2cfb run>"}
[2019-01-04T12:46:33,903][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"releaserexceptions", :thread=>"#<Thread:0xaf8b522 run>"}
[2019-01-04T12:46:33,998][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:logtailing, :releaserexceptions], :non_running_pipelines=>[]}
[2019-01-04T12:46:34,013][INFO ][org.logstash.beats.Server] Starting server on port: 5043
[2019-01-04T12:46:34,013][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2019-01-04T12:46:34,453][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

```

# Answer
> 1 votes
"I have two Filebeat pipes inputting into Logstash." ---I assume this means that you have conditional logic and/or prospectors in your filebeat config to ship to the multiple logstash (5044 and 5043)ports? I'd be interested to see that config.

Here's another approach that might acheive your goal: Use the source value in a conditional output. An example of logstash conditional outputs can be found here: https://discuss.elastic.co/t/conditional-statement-in-output/74672

If, for some reason filebeat's 'source' field will not suffice: you might inject your own fields, per prospector, as described by andrewkroh midway through the thread here: https://discuss.elastic.co/t/how-to-tag-log-files-in-filebeat-for-logstash-ingestion/44713/6. Then you could do conditional output on your custom fields. The example Andrew provided for adding custom fields by prospector is as follows:

```filebeat:
```  prospectors:
```    - paths:
```        - /path/to/logs/access.log
```      fields:  {log_type: access}
```    - paths:
```        - /path/to/other/logs/errors.log
```      fields: {log_type: errors}

---
Tags: logstash
---


# Question
Title: Purchased AWS reserved instance in wrong region
I purchased AWS reserved instance by mistake in Mumbai Region. actually, I need that in N.virginia region.

I contact AWS support center and they told "switching region of AWS reserved instance is not possible"

Is there any way to switch region? Please help.

# Answer
> 1 votes
If you have done payment of your reserved instance then you are unlucky. now you have only one option **Sell your reserved instance in AWS marketplace**.

But If you till not done payment then you lucky, I suggest don't do payment till next billing cycle. AWS by default cancel your reserved instance if you don't do payment after the end of the month.

---
Tags: amazon-web-services
---


# Question
Title: What options are available to encrypt the content of git repositories?
git-crypt, git secret were found. What is your personal preference and why?

A couple of months ago, I have applied git-crypt. It was quite hard to set it up and the secrets are stored as binaries. I wonder whether git-secret is less cumbersome to set up. What other options are available?

# Answer
I personally am a fan of Mozilla SOps, we've introduced it to two teams and it's gone down well both times, it has proven easy to implement both with local development and our CI/CD pipelines.

It has support for PGP/GPG to encrypt secrets but also supports Amazon KMS, GCP KMS and Azure KeyVault thus works well in an enterprise cloud-native infrastructure.
> 2 votes

# Answer
If your use case is Kubernetes, I like Sealed Secrets

It is what Weave Cloud uses to store configuration and do GitOps.
> 0 votes

# Answer
I have spent a good amount of time lately looking into the GoDaddy open-source CRD called ExternalSecrets. It's been pretty good to work with, it keeps the actual secret material in AWS Secrets Manager, and polls for changes. When a change is detected it creates/updates a Secret in your Kubernetes cluster. The only issue I've encountered is that you are mostly limited to using key/value pairs.
> 0 votes

---
Tags: git, encryption
---


# Question
Title: Does K8s ClusterIP service supports custom timeout?
Report from Stack Overflow due to off-topic

**Base knowlege**

Since this question deals particularly with various timeouts configuration in `Nginx`, I thought giving their references here would make it easier for everyone to help me. These are `Nginx` time out directives I use:

1. proxy\_connect\_timeout: Defines a timeout for establishing a connection with a proxied server. It should be noted that this timeout cannot usually exceed 75 seconds.
2. `proxy_read_timeout`: Defines a timeout for reading a response from the proxied server. The timeout is set only between two successive read operations, not for the transmission of the whole response. If the proxied server does not transmit anything within this time, the connection is closed.
3. `proxy_send_timeout`: Sets a timeout for transmitting a request to the proxied server. The timeout is set only between two successive write operations, not for the transmission of the whole request. If the proxied server does not receive anything within this time, the connection is closed.

**Scenario**

Machine from Internet (A) – **opens HTTP connection** – \> NginxPod (B) – **reverses proxy** – \> a ClusterIP service (C).

```
    location /abc {
        proxy_pass http://service1.test.svc.cluster.local:3000;
        # This is a dns name given to a `ClusterIP` service by K8s.

        proxy_connect_timeout 2m;
        proxy_read_timeout 60m;
        send_timeout 60m;
    }

```

K8s will do some magic internally (e.g. *load balancing* my requests to specific pods which are serving the service1).

Because there's another layer of load balancing imposed by K8s which I'm not in charged of, I wonder if my timeout settings for `Nginx` will be honored by K8s.

# Answer
All cookies and headers are a construct of the HTTP protocol. Kubernetes Services operate on TCP/UDP packets therefore it will pass the payload as-is.

It is not that there isn't an `nginx.conf` file but rather the configuration is managed via annotations. See this link https://github.com/kubernetes/ingress-nginx/blob/master/docs/user-guide/nginx-configuration/annotations.md
> 1 votes

---
Tags: kubernetes, nginx
---


# Question
Title: What are the levels of monitoring and their importance?
We are going through a process of change in our IT Operations Management. One of the items we are trying to improve is our monitoring of our infrastructure.

So far I've been thinking of only 3 levels of monitoring and I believe that one tool can not do all very well.

To hire such tools in the market, we are in the process of listing these levels of monitoring, what is the relevance of this monitoring for the business and applications that achieve our objectives.

I believe to date that we have 3 macro levels of infrastructure monitoring:

1) Application Level

* APM softwares

2) Asset Level (Hardware, VMs, OS)

3) Network Level

* NMS softwares

Is my thinking correct? Are there other levels of monitoring? We do not have microservices or containers, but I would like this to be a generic post. What tools are currently on the market that will monitor each level?

# Answer
> 1 votes
In our environment we use Appdynamics and logz.io for all levels of monitoring and it's a best tool,

Basically it monitor it for all levels,

1. Databse
2. Application especially works well on Java based services
3. Servers, containers
4. remote services
5. Business transactions
6. service endpoints, etc.,

Also it monitors all the components like, CPU, Memory, Load average, Network errors per minute, Network utilization, top 10 process usage of CPU and Memory.

All monitoring you can get it in the graph format.

---
Tags: monitoring, network-monitoring
---


# Question
Title: kubernetes statefulset - communicate addresses of other pods when pod launches?
Is there some metadata that I can apply, or some DNS that I can query, such that when a pod is started, the addreses of all the other pods in that statefulset will be somehow made available to the newly started pod?

I want to run a stateful application (Riak) where every node (pod) in the Riak cluster needs to be aware of the addresses of every other node (pod).

Using the kubernetes statefulset controller, pods are started one by one.

I also wonder how I can somehow listen for some event in each Riak pod so that as instances are started, all the preceeding instances will be made aware of the new instance?

---

Following on from the information provided by @kaizenCoder - I modified the service so it had a configuration of :

```
spec:
  clusterIP: None

```

I then scaled up to two replicas, exec'd to one of the containers and ran `dig` again as mentioned.

```
root@grafana-0:/usr/share/grafana# dig SRV grafana.default.svc.cluster.local

; <<>> DiG 9.10.3-P4-Debian <<>> SRV grafana.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 27002
;; flags: qr aa rd; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 3
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;grafana.default.svc.cluster.local. IN  SRV

;; ANSWER SECTION:
grafana.default.svc.cluster.local. 5 IN SRV     0 50 55555 10-1-0-77.grafana.default.svc.cluster.local.
grafana.default.svc.cluster.local. 5 IN SRV     0 50 55555 10-1-0-78.grafana.default.svc.cluster.local.

;; ADDITIONAL SECTION:
10-1-0-78.grafana.default.svc.cluster.local. 5 IN A 10.1.0.78
10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77

;; Query time: 0 msec
;; SERVER: 10.96.0.10#53(10.96.0.10)
;; WHEN: Fri Jun 28 12:12:32 UTC 2019
;; MSG SIZE  rcvd: 372

```

Much better - but unlike @kaizenCoder - I don't get back a name for each of the pods - it just uses the IP address - the names don't match - I would have expected the first pod to have had a name of `grafana-0.grafana.default.svc.cluster.local` \- that nodes hostname is correct though - `grafana-0`.

Any suggestions on how to get the DNS to match the hostname ?

---

Update again - more weirdness - the other pods don't seem to be resolvable by hostname.

```
root@grafana-0:/usr/share/grafana# ping grafana-1
ping: unknown host

```

Presumably hostname only resolves in the pod as hostname resolves to 127.0.0.1 ... Which makes it a bit worthless tbh.

I feel like I've made some progress but still missing a piece. When a container starts up, how can software within it resolve :

1. It's own fully qualified name
2. The fully qualified names of all the other statefulset pods

I could do something like this to determine the node name and the other node names but it feels ugly.

```
root@grafana-0:/usr/share/grafana# dig SRV *.default.svc.cluster.local | grep 0-77
*.default.svc.cluster.local. 5  IN      SRV     0 11 55555 10-1-0-77.grafana.default.svc.cluster.local.
10-1-0-77.grafana.default.svc.cluster.local. 5 IN A 10.1.0.77
root@grafana-0:/usr/share/grafana# ip addr | grep eth0 | grep inet
    inet 10.1.0.77/16 scope global eth0

```

# Answer
> 1 votes
Yes you can by doing a DNS SRV lookup. Here's an example:

```
$ kubectl run -it srvlookup --image=tutum/dnsutils --rm --restart=Never -- dig SRV jira-headless.default.svc.cluster.local

; <<>> DiG 9.9.5-3ubuntu0.2-Ubuntu <<>> SRV jira-headless.default.svc.cluster.local
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 56846
;; flags: qr aa rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 2

;; QUESTION SECTION:
;jira-headless.default.svc.cluster.local. IN SRV

;; ANSWER SECTION:
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-0.jira-headless.default.svc.cluster.local.
jira-headless.default.svc.cluster.local. 30 IN SRV 10 50 0 jira-1.jira-headless.default.svc.cluster.local.

;; ADDITIONAL SECTION:
jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188

;; Query time: 6 msec
;; SERVER: 10.233.0.3#53(10.233.0.3)
;; WHEN: Fri Jun 28 00:18:14 UTC 2019
;; MSG SIZE  rcvd: 143

```

In the example above I have a headless Service called `jira-headless.` If you pay attention to `ADDITIONAL SECTION:` you'll notice it provides the SRV record in the following format:

`_service._proto.name. TTL class SRV priority weight port target.`

Which maps to the Pods (along with their IPs) backing the Service:

```
jira-0.jira-headless.default.svc.cluster.local. 30 IN A 10.244.11.152
jira-1.jira-headless.default.svc.cluster.local. 30 IN A 10.244.0.188

```

---
Tags: kubernetes, dns
---


# Question
Title: Ansible: Exclude a parameter if variable is not set
Is there a way to exclude a parameter entirely if a variable is *not* set? I do not want to use a default value for this, I simply do not way the parameter referenced at all.

For example,

```
---
- name: manage_aix_lvm | creating new VG
  # https://docs.ansible.com/ansible/latest/modules/aix_lvg_module.html#aix-lvg-module
  aix_lvg:
    force: "{{ item.force | default('no') }}"
    pp_size: "{{ item.ppsize }}"
    pvs: "{{ item.disks | join(',') }}"
    vg: "{{ item.vgname }}"
    state: present
  become: true
  loop: "{{ lvm_groups }}"

```

In the event that variable item.ppsize is *not* set I want to exclude that entire parameter from being included. The reason for this is that this is an optional parameter, and if not set, the Operating System will automatically calculate the value.

Is this possible within a single task? Or will I need to create multiple Tasks/Resources to check if item.ppsize is set and redirect to different tasks/resources?

Thanks.

# Answer
Q: ***"If the variable item.ppsize is not set exclude that parameter."***

A: Set the default value to the special variable *omit*

```
    pp_size: "{{ item.ppsize|default(omit) }}"

```
> 11 votes

---
Tags: ansible
---


# Question
Title: Amazon S3 bucket does not redirect folder URLs to index.html
I have uploaded my HUGO static site into AWS S3 Bucket. I have also added the domain in cloudrfront.

The home index.html page works fine, but I cannot access my resources. For example `https://www.example.com` works fine, but I cannot access `https://www.example.com/posts/`. I have to redirect links to the html file every time.

On localhost `https://www.example.com/posts/` displays automatically the page ( without redirecting `https://www.example.com/posts/` to `https://www.example.com/posts/index.html`.

Is there some property in cloudfront or Amazon S3 to replicate the behaviour of localhost and not have to change the links of the site?

# Answer
CloudFront does not actually have this feature. You can redirect the root object `https://www.example.com/` to `https://www.example.com/index.html` by setting `DefaultRootObject`, but there's nothing for subdirectories such as `https://www.example.com/posts/`.

However, you can run arbitrary code on CloudFront using Lambda@Edge or CloudFront Functions, so it's possible for us to add this feature ourselves. With the "origin request" trigger we can rewrite the request to S3, changing `/posts/` to `/posts/index.html`. https://github.com/CloudUnder/lambda-edge-nice-urls has some good handling of paths that works well with Hugo. The Amazon documentation has an example of how you can do the same thing with CloudFront Functions.

Another option is to enable `uglyurls` in your config so that all links within you site have `index.html` hardcoded in to them.
> 3 votes

---
Tags: amazon-s3, go, amazon-cloudfront
---


title: Ansible nested playbooks with variables

text:
# Question
Title: Ansible nested playbooks with variables
I have an ansible playbook `add_dns_record.yml` that register the A record and relative PTR of an host to our MS DNS server.

```
- hosts: my_ad_dns
  remote_user: "{{ lookup('env', 'USER') }}@MYDOMAIN.COM"
  gather_facts: false
  tasks:
  - name: Create A record
    win_dns_record:
      computer_name: "{{ inventory_hostname }}"
      name: "{{ dns_a_name }}"
      type: "A"
      value: "{{ dns_ip_address }}"
      zone: "MYDOMAIN.COM"
      state: present

  - name: Create PTR
    ...

```

I have another playbook `debian_base.yml` used to prepare debian servers:

```
- hosts: debian
  tasks:

    - name: install python
      raw: test -e /usr/bin/python || (apt -y update && apt install -y python-minimal)

    - name: Set hostname
      hostname:
        name: "{{ inventory_hostname }}"

    - name: Update repositories cache and install pre-requirements
      apt:
        name: "{{ packages}}"
        update_cache: yes
      vars:
        packages:
        - sudo
        - apt-transport-https
        - cloud-guest-utils
        - open-vm-tools
        - screen
        - dirmngr
        - openjdk-8-jdk-headless
        - dos2unix
        ....

```

I want that every debian server deployed using the playbook `debian_base.yml` get registered on the MS DNS.

I had a look to `import_playbook` but it doesn't accept variables.

What are the best practices to achieve that?

# Answer
This looks like you are trying to solve a problem (how express a desired state using appropriate OS-specific strategies), but selecting the "wrong" approach.

I would suggest that instead of maintaining separate playbooks (one for Windows, one for Debian, *etc*), you should write an OS-independent role. The role will use facts from the target machines to apply the relevant configuration. This can be done in a few ways :

1. Set OS-specific variables and include them in the tasks using a `include_vars: "{{ ansible_os_family }}.yml"` which will pick up a `Debian.yml` set of vars *e.g.*. You could then write `when` conditions on tasks to execute OS-specific tasks
2. Write OS-specific tasks to be included in the `tasks/main.yml` *e.g.*

```
# tasks/main.yml
- import_tasks: "{{ ansible_os_family }}.yml"

```

This will afford you some benefits:

1. You will be able to test the role independently for different scenarios (using molecule *e.g.*
2. You will be able to maintain the configuration (vars, tasks, etc) independently in separate files for respective OS's.
3. You won't have to maintain two separate playbooks
4. Your role will be easier to re-use

See writing re-usable playbooks on the Ansible documentation.
> 1 votes

# Answer
the best way I think is to create a var yml file and include it in your ansible-books:

```
    - name: Include vars of stuff.yaml into the 'stuff' variable (2.2).
      include_vars:
        file: stuff.yaml
        name: stuff

```

see include\_vars\_module
> 1 votes

---
Tags: ansible
---


# Question
Title: How to clean up orphaned AWS EC2 snapshots?
We end up with a fair amount of AWS EC2 snapshots where the AMI has been deleted, but the snapshot is left behind to rot. I'd like a non-manual way of **identifying and deleting** these orphans to save us money and space.

Ideally I'm thinking a **bash script leveraging the CLI**, but my AWS-fu is weak. I assume someone's done this before but I can't find a script that actually works.

In the best-case scenario this will also check volumes and clean those as well, but that may be better suited for a second question.

# Answer
> 16 votes
Largely inspired by the blog posts and gist already linked in the other answers, here is my take to the problem.

I did use some convoluted JMESpath functions to get a list of snapshots and not require `tr`.

*Disclaimer*: Use at your **own risks**, I did my best to avoid any problem and keep sane defaults, but I won't take any blame if it cause problem to you.

```
#!/bin/sh
# remove x if you don't want to see the commands
set -ex

# Some variable initialisation with sane defaults
DRUN='--dry-run'
DO_DELETE=${1:-'no'}
REGION=${2:-'eu-west-1'}
ACCOUNTID=${3:-'self'}

# Get two temporary files
SNAP_FILE=$(mktemp)
IMAGE_FILE=$(mktemp)

# Get the snapshot list and the volume list
aws --region "$REGION" ec2 describe-snapshots --owner-ids "$ACCOUNTID" --query 'Snapshots[*].[SnapshotId]' --output text > "$SNAP_FILE"
aws --region "$REGION" ec2 describe-images --owners "$ACCOUNTID" --filters Name=state,Values=available --query 'Images[*].BlockDeviceMappings[*].Ebs.[SnapshotId]' --output text > "$IMAGE_FILE"

# Check if the outputed command should be dry-run (default) or not
if [ "$DO_DELETE" = "IAMSURE" ]
then
 DRUN=''
fi

# count each snapshot id, decrease when a volume reference it, print delete command for those with no volumes
awk -v REGION="$REGION" -v DRUN="$DRUN" '
FNR==NR { snap[$1]++; next } # increment snapshots and get to next line in file immediately

{ snap[$1]-- } # we changed file, decrease the snap counter when a volume reference it

END {
 for (s in snap) { # loop over the snapshots
   if (snap[s] > 0) { # if we did not decrese under 1 that means there is no volume referencing this snapshot
    cmd="aws --region " REGION " " DRUN " ec2 delete-snapshot --snapshot-id " s
    print(cmd)
  }
 }
}
' "$SNAP_FILE" "$IMAGE_FILE"
# Clean up the temp files
rm "$SNAP_FILE" "$IMAGE_FILE"

```

I hope the script itself is commented enough.

Default usage (no-params) will list delete commands of orphaned snapshots for the current account and region eu-west-1, extract:

```
aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-81e5856a
aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-95c68c7e
aws --region eu-west-1 --dry-run ec2 delete-snapshot --snapshot-id snap-a3bf50bd

```

You can redirect this output to a file for review before sourcing it to execute all the commands.

If you want the script to execute the command instead of printing them, replace `print(cmd)` by `system(cmd)`.

Usage is as follow with a script named `snap_cleaner`:

for dry-run commands in us-west-1 region

```
./snap_cleaner no us-west-1

```

for usable commands in eu-central-1

```
./snap_cleaner IAMSURE eu-central-1

```

A third parameter can be used to access another account (I do prefer to switch role to another account before).

Stripped down version of the script with awk script as a oneliner:

```
#!/bin/sh
set -ex

# Some variable initialisation with sane defaults
DRUN='--dry-run'
DO_DELETE=${1:-'no'}
REGION=${2:-'eu-west-1'}
ACCOUNTID=${3:-'self'}

# Get two temporary files
SNAP_FILE=$(mktemp)
IMAGE_FILE=$(mktemp)

# Get the snapshot list and the volume list
aws --region "$REGION" ec2 describe-snapshots --owner-ids "$ACCOUNTID" --query 'Snapshots[*].[SnapshotId]' --output text > "$SNAP_FILE"
aws --region "$REGION" ec2 describe-images --owners "$ACCOUNTID" --filters Name=state,Values=available --query 'Images[*].BlockDeviceMappings[*].Ebs.[SnapshotId]' --output text > "$IMAGE_FILE"

# Check if the outputed command should be dry-run (default) or not
if [ "$DO_DELETE" = "IAMSURE" ]
then
 DRUN=''
fi

# count each snapshot id, decrease when a volume reference it, print delete command for those with no volumes
awk -v REGION="$REGION" -v DRUN="$DRUN" 'FNR==NR { snap[$1]++; next } { snap[$1]-- } END { for (s in snap) { if (snap[s] > 0) { cmd="aws --region " REGION " " DRUN " ec2 delete-snapshot --snapshot-id " s; print(cmd) } } }' "$SNAP_FILE" "$IMAGE_FILE"
# Clean up the temp files
rm "$SNAP_FILE" "$IMAGE_FILE"

```

# Answer
> 5 votes
I used the following script on GitHub by Rodrigue Koffi (bonclay7) and it works pretty good.

https://github.com/bonclay7/aws-amicleaner

Command:

```
amicleaner --check-orphans

```

From the documentation blog post it does some more things:

> It actually does a bit more than that, at of today it allows:
>
> * Removing a list of images and associated snapshots
> * Mapping AMIs:
>     + Using names
>     + Using tags
> * Filtering AMIs:
>     + used by running instances
>     + from autoscaling groups (launch configurations) with a desired capacity set to 0
>     + from launch configurations detached from autoscaling groups
> * Specifying how many AMIs you want to keep
> * Cleaning orphan snapshots
> * A bit of reporting

# Answer
> 3 votes
Here is one script which can help you find orphaned snapshots

```
comm -23 <(echo $(ec2-describe-snapshots --region eu-west-1 | grep SNAPSHOT | awk '{print $2}' | sort | uniq) | tr ' ' '\n') <(echo $(ec2-describe-images --region eu-west-1 | grep BLOCKDEVICEMAPPING | awk '{print $3}' | sort | uniq) | tr ' ' '\n') | tr '\n' ' '

```

(from here)

Also you can check this article from serverfault

P.S. Of course you can change the region to reflect your

P.P.S. Here is updated code:

```
 comm -23 \
<(echo $(aws ec2 describe-snapshots --region eu-west-1 |awk '/SNAPSHOT/ {print $2}' | sort -u) | tr ' ' '\n') \
<(echo $(aws ec2 describe-images --region eu-west-1 |  awk '/BLOCKDEVICEMAPPING/ {print $3}' | sort -u) | tr ' ' '\n') | tr '\n' ' '

```

The sample exaplanations what the code do is:

```
echo $(aws ec2 describe-snapshots --region eu-west-1 | awk '/SNAPSHOT/ {print $2}' | sort -u) | tr ' ' '\n')

```

send to STDOUT the list of snapshots. this construction:

```
<(...)

```

create virtual temporary filehandler to make `comm` command read from two "files" and compare them

# Answer
> 2 votes
Here is a GitHub Gist code snippet of exactly what you are asking for by Daniil Yaroslavtsev.

It uses the list of all images and their snapshots and compares the IDs to list of all snapshot IDs. Whatever remains are the orphaned ones. The code works in the same principle as the answer above, but is better formatted and slightly more readable.

The code takes advantage of the JMESPath with `--query Snapshots[*].SnapshotId` option (you can also use jp command line utility for that, if its already in your distribution. The formats the output as text with `--output text`. Here is a link to API reference and few examples. It is slightly more elegant than a long chain of grep/awk/sort/uniq/tr pipes.

**Warning by Todd Walton**: Don't mistake with 'jq' utility which uses different query language to parse json documents.

# Answer
> 0 votes
I've written snapshots.py script which iterates over all snapshots (in defined list of regions) and generates `report.csv`. This file contains information about instance, AMI and volume referenced by all snapshots.

There is also command to interactively remove dangling snapshots.

---
Tags: amazon-web-services, amazon-ec2
---


# Question
Title: Jenkins won't substitute variables when trying to check out from perforce
I want to generalize my code so that it can pull from perforce for any files marked with a type "perforce" because some files need to downloaded from the artifactory instead. I want to sub in the variable perf\[0\].ID into the 'view' parameter so that I can pull different files depending on the ID. However, when I run it, it's passed in as an empty string. Does the checkout plugin not have any support for variables? If there is no support, would having a spec file give me this generality I want? Or do I need to bite the bullet, abandon the checkout plugin and make a batch script?

--\>I have an array perf with a type,path,ID, and size field. Here perf\[0\].ID = "Linak".

stage ('Perforce binaries') {

```
        steps {
        //can do this "for perforce[].size()" on specified path
          script {
            for (int i = 0; i < perf.size(); i++ ) {
              dir("raw/${perf[i].ID}") {
                checkout perforce(credential: 'perforce_credentials', populate: autoClean(delete: true, modtime: false, parallel: [enable: false, minbytes: '1024', minfiles: '1', threads: '4'], pin: '', quiet: true, replace: true, tidy: false), workspace: manualSpec(charset: 'none', cleanup: false, name: 'jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}', pinHost: false, spec: clientSpec(allwrite: false, backup: false, changeView: '', clobber: false, compress: false, line: 'LOCAL', locked: false, modtime: false, rmdir: false, serverID: '', streamName: '', type: 'READONLY', view: '"//depot/Software/${perf[i].ID}/..." //jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}/...')))
              }
            }
          }
       }
    }

```

# Answer
Your string isn't being interpolated at the correct time because it's in single quotes. Instead it's being passed to the checkout method as an un-interpolated string. By the time the checkout method interpolates the string, the `perf` variable is out of scope and so interpretation of this variable falls back to a blank string. Something like this is what you (probably) want instead:

```
view: "\"//depot/Software/${perf[i].ID}/...\"" + '//jenkins-${NODE_NAME}-${JOB_NAME}-${EXECUTOR_NUMBER}/...'

```

Note how this concatenates two strings, a double-quoted string and a single-quoted string. This ensures that each string gets interpolated at different times.
> 1 votes

---
Tags: jenkins, environment-variables, versioning
---


title: How to implement DTAP in k8s?

text:
# Question
Title: How to implement DTAP in k8s?
Dev, Tst, Acc, Prd (DTAP) is about isolating multiple environments to ensure that data will not be mixed, e.g. production data in development or lost, e.g. database removal removes all production data. DTAP is also about environments that are identical, e.g. if a database in prd, then there should be a database in dev as well, but the data should not be mixed. It is about prediction. The more the environments are identical the bigger the change that if a certain change works in dev, then it will work in tst, acc and prd.

**Discussion**

There are at least two options, i.e. one k8s cluster or multiple.

*one k8s cluster*

* Lower costs
* Dangerous as pods from production environment could be removed, while the intention was to removed dev pods.

*multiple k8s clusters*

* Higher costs
* Lower change that data will be removed accidentally due to isolation

# Answer
Kubernetes supports namespaces.

> multiple virtual clusters backed by the same physical cluster

These namespaces can allow you to use one cluster for all your environments. You can also do fancy things in your CI/CD pipeline so that each branch or each commit gets its own namespace. Those pods in prod namespace can still speak to pods in dev namespace unless you segment with NACL's. Just make sure you CNI supports that.

That being said most large companies I have worked for are running a cluster per environment.

I don't think there are right or wrong answers here. Do what you think is best for your company/team/workload/environment etc.
> 2 votes

# Answer
Supplementary to @Levi's answer I would also advise you to consider where you plan to test changes to your Kubernetes cluster (any change that can impact how your cluster operates e.g: lifecycle). To satisfy this and provide another alternative to the mix, you could operates 2 clusters. One for your non-prod environment (dev, test and acc) and one for the prod environment. This has the added benefit of protecting your production cluster from getting clobbered from non production workloads.
> 1 votes

---
Tags: kubernetes, dtap
---


# Question
Title: What is the performance difference between running Docker on Windows versus Linux?
We are looking at containerizing some of our applications and servers (Red Hat servers to be specific), and there was an idea to run Docker on Windows instead of Linux so the end user has an easier time operating the system. The first thought I had was "what is the performance difference between using Docker in Windows vs in Linux?"

# Answer
The performance difference between **linux** containers running on Windows, Linux, or Mac will depend entirely upon how much ram you allocate to the Docker Daemon. The difference should not be noticeable. Additionally, you will be able to take advantage of the Alpine base image (~5 Mb) for small, compact containers.

If you need to run IIS in your container, then you should look at **windows** containers. The disadvantage is that the containers will be significantly larger in size, and you will lose cross-platform compatibility. Docker windows containers.
> 1 votes

---
Tags: docker, containers
---


# Question
Title: Can we reuse Variables across variable groups in Azure DevOps?
I'm using DevOps to deploy Azure Data Factory ARM template

I'm using a deployment pipeline to take artifacts from a prior build pipeline and deploy to various environments (TEST, PROD)

I've created two variable groups: `TEST` and `PROD`. I've created two deployment "stages" `TEST` and `PROD`

Within the `TEST` variable group I've created a bunch of variables reflecting the `TEST` specific configuration.

As far as I understand, I now have to manually recreate all of these identically named variables in my `PROD` variable group.

Not only that I have to make a copy of my TEST stage

So I basically have two duplicate configs and two duplicate stages

It would make more sense if I had a single stage definition that could be run against different variable groups rather than repeating code and variable group definitions.

It seems strange and redundant to have to do this. I also find it strange that the concept of environments is not specifically called out in doco anywhere

So the specific question is: Is it the correct approach to define a variable group per environment then manually create identical variables in both variable groups?

and a more general question: am I missing something fundamental here or is the product not mature enough yet.

I have not used any other deployment tool in any detail before this and I'm not using YAML (just the UI) so maybe that's the piece I'm missing.

Another "What Am I missing" thing:

In the *Resource Group deployment* task in the *override template parameters* field I used the UI to assign variables to parameter values. I had to type the variables in and had to make sure I got the magic notation right. `$(myVariable)` Then I read another stack overflow that indicated if your variable value has a space in it you need to surround it by quotes. This is the second "you need quotes for spaces" issue I have found.

# Answer
> 1 votes
My understanding of the variable groups is that you would create this to leverage the same variables across multiple pipelines. If you needed to update the value, you'd only have to update it on the library instead of updating it on dozens of pipelines. If you are only going to use the variable group on a single pipeline definition, variables groups appear to be overkill.

If you have concerns about duplicating code and maintaining the exact same deployment steps across your stages, I would recommend you group them into a task group. That way you can manage versioned updates and will also have it reflected across all pipelines and stages where it is referenced.

I think you would have some name resolution issues if you tried to use the different variable groups that have the same name on a single pipeline. Instead, you should probably leverage the variable scope on the pipeline. It will grab the appropriate value staged to the scope you are executing.

---
Tags: azure-devops, azure-pipelines, environment-variables
---


# Question
Title: Terraform aws_ecr_repository resource always attempts to destroy
I have been banging my head against this seemingly simple issue for awhile now. I have some Terraform code which sets up an ECR repository, policy and lifecycle policy.

```
provider "aws" {
    region = "${var.region_id}"
}

terraform {
   required_version = "0.11.13"
}

resource "aws_ecr_repository" "frontend_ecr_repo_creation" {
  name = "${var.app_image}"
}

resource "aws_ecr_repository_policy" "frontend_ecr_policy" {
  repository = "${aws_ecr_repository.frontend_ecr_repo_creation.name}"

  policy = <<EOF
{
    "Version": "2008-10-17",
    "Statement": [
        {
            "Sid": "new policy",
            "Effect": "Allow",
            "Principal": "*",
            "Action": [
                "ecr:GetDownloadUrlForLayer",
                "ecr:BatchGetImage",
                "ecr:BatchCheckLayerAvailability",
                "ecr:PutImage",
                "ecr:InitiateLayerUpload",
                "ecr:UploadLayerPart",
                "ecr:CompleteLayerUpload",
                "ecr:DescribeRepositories",
                "ecr:GetRepositoryPolicy",
                "ecr:ListImages",
                "ecr:DeleteRepository",
                "ecr:BatchDeleteImage",
                "ecr:SetRepositoryPolicy",
                "ecr:DeleteRepositoryPolicy"
            ]
        }
    ]
}
EOF
}

resource "aws_ecr_lifecycle_policy" "frontend_ecr_lifecycle_policy" {
  repository = "${aws_ecr_repository.frontend_ecr_repo_creation.name}"

  policy = <<EOF
{
    "rules": [
        {
            "rulePriority": 1,
            "description": "Keep last 3 images",
            "selection": {
                "tagStatus": "any",
                "countType": "imageCountMoreThan",
                "countNumber": 3
            },
            "action": {
                "type": "expire"
            }
        }
    ]
}
EOF
}

```

This runs perfectly fine, however subsequent runs fail with:

```
 aws_ecr_repository.frontend_ecr_repo_creation: error creating ECR repository: RepositoryAlreadyExistsException: The repository with name 'cwyo/master' already exists in the registry with id '720734997891'
    status code: 400, request id: cf59815c-7c9d-11e9-965d-0b326e93dee5

```

I would expect Terraform plan no changes, instead it attempts to recreate the repository. This shows all of the signs of an incorrect remote state, however I run a Terraform init in my pipeline beforehand and I am satisfied that it is correct.

I have been working around this by adding GitLab CI's inbuilt $COMMIT\_SHA variable as a prefix, this works well except lifecycle policies cannot be applied therefore I ended up with 100's of useless repositories.

I have looked in to a data structure to check if the repository exists and add a depends\_on to the resource but no luck...

**How do I ensure the aws\_ecr\_repository resource uses an existing repository if it already exists?**

# Answer
> 1 votes
In your `terraform {...}` section you need a backend directive to store your state somewhere: https://www.terraform.io/docs/state/remote.html (OP later explain that the code above is just a module imported from a main file).

S3 backends might be the easiest to use and probably to most commonly used backends to store remote state across different executions of Terraform:

```
terraform {
  backend "s3" {
    bucket = "mybucket"
    key    = "path/to/my/key"
    region = "us-east-1"
  }
}

```

Please note that Terraform init will not import existing resources into a state (there's Terraform import for that). It will therefore try to create new ones and you will get a conflict on the name. The Terraform documentation gives some pointers on how to use Terraform in automation and that includes using remote state.

It is possible to prevent the creation of the ECR through a data block as you hinted (with `count = ...`), but doing so would feel like a hack to me.

Once you have your remote backend configured, you can double check that everything works as expected by using the `terraform state` command in your pipeline. I would suggest using the show subcommand on a resource you expect to be there in order to validate that the state indeed contains the resource:

```
terraform init -input=false
terraform state show module.name-of-your-module.aws_ecr_repository.frontend_ecr_repo_creation

```

---
Tags: amazon-web-services, terraform, amazon-ecs
---


# Question
Title: How to configure bitbucket pipeline and deploy to alibaba clouds
I have a repo on my bitbucket. I want to deploy it via bitbucket pipeline to my alibaba clouds ecs. I dont know what are steps required to manage it.

# Answer
You can be done this task using below steps:

1: Creating the Bitbucket Git Repository And Adding our Application and Docker Configurations in it

2: Integrating Bitbucket with ACR

3: Deploying The Application

4: Securing our Application with Let’s Encrypt SSL/TLS

for full details check this:
> 1 votes

---
Tags: bitbucket-pipelines
---


# Question
Title: How can I 'aws s3 sync' two buckets, which are located in different accounts
I'm trying to use 'aws s3 sync' on the awscli between two accounts.

Account A, I own. Account B, Owned by a third party.

Account B has given a user:jon on account A permission to a bucket through a role:assumeDevOps assumption.

Jon assumes assumeDevOps to access bucket on Account B. But now I have to sync to a bucket back on account A.

I'm getting an access denied. Possibly because that role that Jon assumed has no permissions to the bucket back on my account.

How do I do this?

Is there documentation on this kind of situation?

# Answer
**Attach a policy to the source bucket**

1. Get the AWS account ID number of the destination account.
2. From the source account, attach a policy to the source bucket that allows the destination account to get objects, similar to the following:

**Important:** For the value of **Principal**, be sure to enter the AWS account ID number of the destination account.

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "DelegateS3Access",
            "Effect": "Allow",
            "Principal": {"AWS": "222222222222"},
            "Action": ["s3:ListBucket","s3:GetObject"],
            "Resource": [
                "arn:aws:s3:::sourcebucket/*",
                "arn:aws:s3:::sourcebucket"
            ]
        }
    ]
}

```

**Attach a policy to a user or group in the destination account**

Attach a policy to the destination account's IAM user or group that allows the user to copy objects from the source bucket to the destination bucket. The policy can be similar to the following example:

```
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::sourcebucket",
                "arn:aws:s3:::sourcebucket/*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:PutObject",
                "s3:PutObjectAcl"
            ],
            "Resource": [
                "arn:aws:s3:::destinationbucket",
                "arn:aws:s3:::destinationbucket/*"
            ]
        }
    ]
}

```

**Copy objects from the source bucket to the destination bucket**

After you set up the policies on the source bucket and the destination account, the destination account can copy objects from the source bucket to the destination bucket. Then, the destination account owns the objects copied into the destination bucket.

To synchronize all content from the source bucket to the destination bucket, you can run this command:

```
aws s3 sync s3://sourcebucket s3://destinationbucket

```

for more details check this link OR this link
> 2 votes

---
Tags: amazon-web-services, amazon-s3, aws-cli, aws-iam
---


# Question
Title: Azure virtual machine stalls in R due to parallel package
I am writing an R package with tests using the `testthat` package. The tests pass locally and on Travis.

I want to plot the benefit of parallelisation on up to 24 cores, so I set up a virtual machine programmatically on Azure:

```
az vm create \
   --resource-group <resource group> \
   --name <name> \
   --image microsoft-dsvm:linux-data-science-vm-ubuntu:linuxdsvmubuntu:18.12.01 \
   --size Standard_NV24 \
   --admin-username <azure user> \
   --generate-ssh-keys

```

I call `devtools::test()` and the virtual machine gets stuck at `testthat` for hours with 0% CPU usage:

```
✔  checking for unstated dependencies in ‘tests’ ...
─  checking tests ...
   Running ‘testthat.R’

```

`devtools::test()` has no specific arguments to print some output, and it calls `testthat::test_dir()`, which also has no arguments to print output.

When I remove the tests, the code stalls in a function call. I stopped and relaunched multiple times, so I think it's a problem with Azure.

Other sizes of machines, `Standard_F16s_v2` and `Standard DS3 v2`, have the same problem.

How can I fix it?

## Update

I stumbled upon the culprit when I was running another piece of code. The VM hangs when calling the R `parallel` package on multiple cores (it works fine on just one core).

I added some printouts around that call:

```
[1] "num_cores = 2"
[1] "Entering parallel"
[1] "Drawing 1"
[1] "Drawing 2"

```

and then it hangs. Following the suggestion of `strace`, I launched again and see this output around this last printout:

```
[pid  8571] 21:41:48.718625 write(1, "\n", 1[1] "num_cores = 2"
) = 1
[pid  8571] 21:41:48.719486 write(1, "[1]", 3) = 3
[pid  8571] 21:41:48.719550 write(1, " \"num_cores = 2\"", 16) = 16
[pid  8571] 21:41:48.719615 write(1, "\n", 1) = 1
[pid  8703] 21:41:48.730707 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4032, NULL <unfinished ...>
[pid  8705] 21:41:48.730745 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4032, NULL <unfinished ...>
[pid  8701] 21:41:48.730757 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4032, NULL <unfinished ...>
[pid  8704] 21:41:48.730794 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4032, NULL <unfinished ...>
[pid  8702] 21:41:48.730841 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4032, NULL <unfinished ...>
[pid  8700] 21:41:48.730852 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4032, NULL <unfinished ...>
[pid  8706] 21:41:48.731199 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4032, NULL <unfinished ...>
[pid  8571] 21:41:48.742710 futex(0x498fcf4, FUTEX_WAKE_PRIVATE, 2147483647) = 7
[pid  8706] 21:41:48.742831 <... futex resumed> ) = 0
[pid  8705] 21:41:48.742847 <... futex resumed> ) = 0
[pid  8704] 21:41:48.742858 <... futex resumed> ) = 0
[pid  8703] 21:41:48.742868 <... futex resumed> ) = 0
[pid  8702] 21:41:48.742878 <... futex resumed> ) = 0
[pid  8701] 21:41:48.742888 <... futex resumed> ) = 0
[pid  8700] 21:41:48.742899 <... futex resumed> ) = 0
[pid  8700] 21:41:48.742931 futex(0x87da514, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[1] "Entering parallel"
[pid  8700] 21:41:48.743001 futex(0x87da514, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  8571] 21:41:48.743411 write(1, "[1]", 3) = 3
[pid  8571] 21:41:48.743485 write(1, " \"Entering parallel\"", 20) = 20
[pid  8571] 21:41:48.743545 write(1, "\n", 1) = 1
[pid  8571] 21:41:48.743946 pipe([4, 11]) = 0
[pid  8571] 21:41:48.744019 pipe([12, 13]) = 0
[pid  8571] 21:41:48.744096 rt_sigaction(SIGCHLD, {0x7fdc52d45d40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fdc693d1390}, NULL, 8) = 0
[pid  8571] 21:41:48.744160 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fdc6f002a90) = 8708
strace: Process 8708 attached
[pid  8708] 21:41:48.746022 set_robust_list(0x7fdc6f002aa0, 24) = 0
[pid  8571] 21:41:48.746478 close(11)   = 0
[pid  8708] 21:41:48.746529 rt_sigaction(SIGCHLD, {SIG_DFL, [CHLD], SA_RESTORER|SA_RESTART, 0x7fdc6902b4b0},  <unfinished ...>
[pid  8571] 21:41:48.746549 close(12 <unfinished ...>
[pid  8708] 21:41:48.746559 <... rt_sigaction resumed> {0x7fdc52d45d40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fdc693d1390}, 8) = 0
[pid  8571] 21:41:48.746571 <... close resumed> ) = 0
[pid  8708] 21:41:48.746581 close(4)    = 0
[pid  8708] 21:41:48.746643 dup2(12, 0) = 0
[pid  8708] 21:41:48.746704 close(12)   = 0
[pid  8708] 21:41:48.746764 rt_sigaction(SIGUSR1, {0x7fdc52d45a90, [USR1], SA_RESTORER|SA_RESTART, 0x7fdc6902b4b0}, {0x7fdc6ce94550, [USR1], SA_RESTORER|SA_RESTART, 0x7fdc6902b4b0}, 8) = 0
[pid  8571] 21:41:48.748205 pipe([11, 12]) = 0
[pid  8571] 21:41:48.748280 pipe([14, 15]) = 0
[pid  8571] 21:41:48.748350 rt_sigaction(SIGCHLD, {0x7fdc52d45d40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fdc693d1390}, NULL, 8) = 0
[pid  8571] 21:41:48.748410 clone([1] "Drawing 1"
 <unfinished ...>
[pid  8708] 21:41:48.749158 write(1, "[1]", 3) = 3
[pid  8708] 21:41:48.749242 write(1, " \"Drawing 1\"", 12) = 12
[pid  8708] 21:41:48.749300 write(1, "\n", 1) = 1
[pid  8571] 21:41:48.750225 <... clone resumed> child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0x7fdc6f002a90) = 8709
strace: Process 8709 attached
[pid  8709] 21:41:48.750313 set_robust_list(0x7fdc6f002aa0, 24) = 0
[pid  8571] 21:41:48.750630 close(12)   = 0
[pid  8571] 21:41:48.750695 close(14)   = 0
[pid  8709] 21:41:48.750829 rt_sigaction(SIGCHLD, {SIG_DFL, [CHLD], SA_RESTORER|SA_RESTART, 0x7fdc6902b4b0}, {0x7fdc52d45d40, [], SA_RESTORER|SA_RESTART|SA_SIGINFO, 0x7fdc693d1390}, 8) = 0
[pid  8709] 21:41:48.750908 close(11)   = 0
[pid  8709] 21:41:48.750976 dup2(14, 0) = 0
[pid  8709] 21:41:48.751050 close(14)   = 0
[pid  8709] 21:41:48.751119 rt_sigaction(SIGUSR1, {0x7fdc52d45a90, [USR1], SA_RESTORER|SA_RESTART, 0x7fdc6902b4b0}, {0x7fdc6ce94550, [USR1], SA_RESTORER|SA_RESTART, 0x7fdc6902b4b0}, 8) = 0
[1] "Drawing 2"
[pid  8571] 21:41:48.752325 wait4(-1, 0x7ffd95fddc44, WNOHANG, NULL) = 0
[pid  8571] 21:41:48.752392 select(12, [4 11], NULL, NULL, {1, 0} <unfinished ...>
[pid  8709] 21:41:48.753053 write(1, "[1]", 3) = 3
[pid  8709] 21:41:48.753138 write(1, " \"Drawing 2\"", 12) = 12
[pid  8709] 21:41:48.753202 write(1, "\n", 1) = 1
[pid  8708] 21:41:48.755113 futex(0x498fcf4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  8704] 21:41:48.756406 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4040, NULL <unfinished ...>
[pid  8703] 21:41:48.756424 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4040, NULL <unfinished ...>
[pid  8701] 21:41:48.756557 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4040, NULL <unfinished ...>
[pid  8702] 21:41:48.756625 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4040, NULL <unfinished ...>
[pid  8705] 21:41:48.756769 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4040, NULL <unfinished ...>
[pid  8700] 21:41:48.757066 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4040, NULL <unfinished ...>
[pid  8706] 21:41:48.757184 futex(0x498fcf4, FUTEX_WAIT_PRIVATE, 4040, NULL <unfinished ...>
[pid  8709] 21:41:48.758569 futex(0x498fcf4, FUTEX_WAKE_PRIVATE, 2147483647) = 0
[pid  8708] 21:41:48.768419 futex(0x88b9a04, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid  8709] 21:41:48.772260 futex(0x88b9a04, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
[pid  8602] 21:41:48.993871 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid  8602] 21:41:48.993914 clock_gettime(CLOCK_REALTIME, {1558734108, 993928682}) = 0
[pid  8602] 21:41:48.993978 futex(0x10e40a8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  8602] 21:41:48.994032 clock_gettime(CLOCK_REALTIME, {1558734108, 994048584}) = 0
[pid  8602] 21:41:48.994096 futex(0x10e40dc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 25, {1558734109, 994048584}, ffffffff <unfinished ...>
[pid  8571] 21:41:49.753515 <... select resumed> ) = 0 (Timeout)
[pid  8571] 21:41:49.753623 wait4(-1, 0x7ffd95fddc44, WNOHANG, NULL) = 0
[pid  8571] 21:41:49.753688 select(12, [4 11], NULL, NULL, {1, 0} <unfinished ...>
[pid  8602] 21:41:49.994210 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid  8602] 21:41:49.994299 clock_gettime(CLOCK_REALTIME, {1558734109, 994341104}) = 0
[pid  8602] 21:41:49.994442 futex(0x10e40a8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  8602] 21:41:49.994590 clock_gettime(CLOCK_REALTIME, {1558734109, 994614207}) = 0
[pid  8602] 21:41:49.994663 futex(0x10e40dc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 27, {1558734110, 994614207}, ffffffff <unfinished ...>
[pid  8571] 21:41:50.754833 <... select resumed> ) = 0 (Timeout)
[pid  8571] 21:41:50.754971 wait4(-1, 0x7ffd95fddc44, WNOHANG, NULL) = 0
[pid  8571] 21:41:50.755117 select(12, [4 11], NULL, NULL, {1, 0} <unfinished ...>
[pid  8602] 21:41:50.994707 <... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)
[pid  8602] 21:41:50.994748 clock_gettime(CLOCK_REALTIME, {1558734110, 994763726}) = 0
[pid  8602] 21:41:50.994816 futex(0x10e40a8, FUTEX_WAKE_PRIVATE, 1) = 0
[pid  8602] 21:41:50.994879 clock_gettime(CLOCK_REALTIME, {1558734110, 994894827}) = 0
[pid  8602] 21:41:50.994944 futex(0x10e40dc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 29, {1558734111, 994894827}, ffffffff <unfinished ...>

```

The rest of the log is a repetition of the last few lines.

I think that the parent process is 8571, the first spawned process is 8708, which prints `Drawing 1`, and the second spawned process is 8709, which prints `Drawing 2`. Then the parent process lists a `wait4(-1, 0x7ffd95fddc44, WNOHANG, NULL) = 0`, process 8602 lists a `<... futex resumed> ) = -1 ETIMEDOUT (Connection timed out)`. I have never used `strace` and could not go beyond that.

I have pushed the code to this Github branch, commit 1b3d1f6, if you want to reproduce the problem. Pull the code from the branch, and run `azure-parallel/deploy_azure.sh` on a terminal (if you have the Azure Command-Line-Interface and are logged in), or change the code a bit to use the IP address of a existing virtual machine.

Has anyone succeeded in running parallel on Azure recently? If so, what were the virtual machine and image?

# Answer
The problem is with the image `microsoft-dsvm:linux-data-science-vm-ubuntu:linuxdsvmubuntu:18.12.01` ("Linux Data Science VM" (on the Azure portal). This is an old image with `R 3.4` instead of the release version of `3.5`.

I succeeded with the following virtual machine (Ubuntu Server 19.04.19, size D64 v3):

```
az vm create \
       --resource-group <resource group> \
       --name <name> \
       --image Canonical:UbuntuServer:19.04:19.04.201906280 \
       --size Standard_D64_v3 \
       --admin-username <azure user> \
       --generate-ssh-keys

```

and installed `R`, `Rstan`, and an SSL library for `curl` and `devtools` with:

```
sudo apt update
sudo apt -y install r-base
sudo apt -y install r-cran-rstan

# Add LibSSL for installing curl and devtools, see:
# https://stackoverflow.com/questions/44228055/r-rstudio-install-devtools-fails
sudo apt-get install libcurl4-openssl-dev libssl-dev

```

and the machine can use multiple cores:

```
Welcome to PosteriorBootstrap, a parallel approach for adaptive non-parametric learning
[1] "Speedup performance"
[1] "n_bootstrap = 100"
[1] "num_cores = 1"
[1] "Finished sampling"
[1] "num_cores = 2"
[1] "Finished sampling"
...
[1] "num_cores = 64"
[1] "Finished sampling"

```

I also checked the duration and the speedup is as I expected.
> 1 votes

---
Tags: azure
---


# Question
Title: Is containerized deployment of jenkins on ECS better than running jenkins inside EC2 instance to secure pipeline information?
I am new to AWS, docker and jenkins. I want to run a jenkins build pipeline and I have tried using following two ways on AWS:

1. Running jenkins inside the EC2 instance following the steps at this AWS link. I created a new EC2 instance and installed jenkins inside the ec2 instance (ssh into the instance). I configured jenkins and ran a build pipeline successfully.
2. Running jenkins inside the ECS cluster following the steps at this AWS link. I created two container instances for Jenkins and data volume, jenkins is installed and running inside one of the instances. (I could not move ahead this point because I was unable to not find the initial password for jenkins)

I could not find much information about the second process of containerized deployment on internet. As I am moving ahead in this process, I am unable to understand which process is better to run a build pipeline securely.

# Answer
> 2 votes
Running Jenkins on EC2 is the better option. It's better to keep the functions that manage your containers separate from the containers themselves. I know this is different than Kubernetes, but this isn't Kubernetes.

Build an ELK Stack on EC2 as well and send your build logs to that instance so you can go through them when troubleshooting. It sounds like your setup is good with data volumes and such. I would recommend that you use Fargate instead of EC2 nodes for your workers though. Not clear which you are using.

There are simpler solutions than Jenkins as well. You may want to consider CircleCI, GitLab Runner, TeamCity, Bamboo, and others. Jenkins is made to do all things and is a good place to learn the process, but if an alternative does the job just as well and more easily, then try that. My current favorite is GitLab Runner. It's just simpler and it integrates your Git repo with the CI tool in addition to having its own image repo for you to use (in certain versions).

You can also give EKS a shot. I haven't tried it but Kubernetes is a superior product. It's far more complex, but definitely a better orchestration tool.

# Answer
> 0 votes
There are other options- Containers on EC2 is what we do. A lot of people are not thrilled with ECS, and EKS is maturing as a better alternative, they say.

It's hard to really devopsify the current Jenkins, without treating the persistent configuration as a one big persistent filesystem. By the time you've installed jobs, and plugins, and credentials, and configuration you may as well just maintain the whole designated storage filesystem, back it up to S3, and attach it to the container as you docker run. This practice is independent of the container-orchestration choice.

Another problem that plagued us was/is maintaining build logs. Jenkins emails build logs in plain text, but I have yet to figure out how to ship them off to a filesystem where I can grep through them when debugging. Our current practice only keeps a few weeks worth. Maybe I missed something? Please correct me!

There is a new JenkinsX aimed at Kubernetes, but not ready as of Feb 2019.

That said, keeping your jobs as jenkinsfiles in GitHub works really well.

---
Tags: jenkins-pipeline, amazon-ec2, amazon-ecs
---


title: Jenkinsfile for monorepo & monobranch

text:
# Question
Title: Jenkinsfile for monorepo & monobranch
I'm currently working on a Microservice architecture-based project and writing Jenkins Pipeline for it.

The project structure is as follows:

Web Service (master)

* Jenkinsfile
* Service 1/
    + Jenkinsfile
* Service 2/
    + Jenkinsfile
* Service 3/
    + Jenkinsfile

Within each Service, there will be a Jenkinsfile that contains information on building and testing for the specific service.

**How do I use the Jenkinsfile on the base directory to build and test each individual Services before running integration test on a cross service level ?**

I've been trying to look for ways to implement this, but the build seems to target project level, it is said to be able to process relative file path but I haven't had much success with it.

I have tried loading, but load seems to be made for groovy file, which doesn't seem to fit the requirement.

# Answer
> 0 votes
I have sometimes used the pattern where all pipeline things go in a `.pipeline/` directory of the app, including the Jenkinsfile. I would, in your case, add several pipelines (ie, Jenkinsfiles) - one per microservice as you propose, and one for the web app.

Then, you define independent multibranch pipeline jobs, based on the location of the pipelines. In other words, you create 5 multibranch pipeline jobs, (one for the web app, one for each microservice), and attach each job to the relevant pipeline file in your repsitory. In this way, they will be tested independently.

The important thing is to properly specify the correct triggers. The microservices will be triggered on changes to code in their path. They may also have as post-build actions the triggering of the web app pipeline, for integration tests, *etc*.

---
Tags: jenkins, jenkins-pipeline, jenkins-plugins, jenkinsfile
---


# Question
Title: Preventing an IAM user from overriding launch template parameters on RunInstance
I'm trying to set up an IAM policy that allows a user to run an instance (`ec2:RunInstances`), but only from a specific launch template - *and* without making any changes to that launch template's parameters.

The relevant section of my policy looks like this:

```
{
  "Effect": "Allow",
  "Action": "ec2:RunInstances",
  "Resource": "*",
  "Condition": {
    "ArnLike": {
      "ec2:LaunchTemplate": "arn:aws:ec2:ap-southeast-2:xxxxxxx:launch-template/lt-xxxxxxx"
    },
  }
}

```

This works great. Running the following using the user's credentials works:

```
aws ec2 run-instances --launch-template LaunchTemplateName=test

```

...and specifying a different launch template fails. So far so good.

However, the user can override values in the launch template, and pretty much do whatever they like:

```
aws ec2 run-instances --launch-template LaunchTemplateName=test --instance-type t2.micro

```

This is despite me setting the instance type in the launch template as `t2.nano`.

The documentation has a solution for this: require the `ec2:IsLaunchTemplateResource` condition to be `"true"`, which apparently only is so if the user has not overridden the launch template.

Perfect - exactly what I was after! The Condition statement in my IAM policy now looks like this:

```
  "Condition": {
    "ArnLike": {
      "ec2:LaunchTemplate": "arn:aws:ec2:ap-southeast-2:xxxxxxx:launch-template/lt-xxxxxxx"
    },
    "Bool": {
      "ec2:IsLaunchTemplateResource": "true"
    }
  }

```

However, with this additional condition present, the `RunInstances` call fails *even if the user makes no changes*. After decoding the authorization failure message, I can see that AWS is reporting `ec2:IsLaunchTemplateResource` was not `"true"`:

```
{
  "key": "ec2:IsLaunchTemplateResource",
  "values": {
    "items": [
      {
        "value": "false"
      }
    ]
  }
}

```

---

**EDIT**: After posting this, I continued to iterate, and noticed that the specific resource that was hitting up against this issue was the subnet - which wasn't defined in the launch template. Taking cues from additional examples in the documentation I linked to, I tried adding another statement to allow the subnet resource even without `ec2:IsLaunchTemplateResource` being true, and this worked. I also tried, alternatively, explicitly defining the subnet in the launch template: this worked too.

*However,* now I am not getting access denied *even when changing the instance type*! So it appears that it was the subnet all along that was triggering the original denial. While I've sort of taken a step forward, the symptom of the issue has now flipped from what I was initially experiencing... and now although the user can launch, they can also launch instances I don't want them to. So my initial question below still remains. :)

---

So, my question is... have I done something wrong? Does `ec2:IsLaunchTemplateResource` work in the way I have understood it (which I think is fairly clear in the documentation)?

Alternatively, is there another way I could allow an IAM user to launch an instance in line with a template, while preventing them from making changes (such as choosing a super-expensive instance type)?

# Answer
`Alternatively, is there another way I could allow an IAM user to launch an instance in line with a template, while preventing them from making changes (such as choosing a super-expensive instance type)?`

I've encountered this problem last year and have circulated up to our AWS account's architect, who contacted their IAM division.

AWS's response is that this is **intended behavior**. Reason given was that **"An instance-type spec on a launch template is not a resource, or conditional that can be used to granted/deny permissions to"**. Notice that inside the IAM policy, specifying a launch template is a condition to action "RunInstance", and the resource of said policy is an ec2 instance.

According to AWS IAM team, in order for you to restrict instance-type (a condition on the resource, "instance"), you must specify the restriction as part of the condition clause to be applied to the instance, like the following:

```
{
  "Effect": "Allow",
  "Action": "ec2:RunInstances",
  "Resource": "*",
  "Condition": {
    "ArnLike": {
      "ec2:LaunchTemplate": "arn:aws:ec2:ap-southeast-2:xxxxxxx:launch-template/lt-xxxxxxx"
    },
    "Bool": {
      "ec2:IsLaunchTemplateResource": "true"
    },
    "StringNotLikeIfExists": {
      "ec2:InstanceType": [
        "c5.*xlarge",
        "p3.*xlarge"
      ]
    }
  }
}

```

Obviously, it is near impossible to list all of the expensive instances we don't want our users to use, and kinda defeats the purpose of using a launch template. But in a messed-up way, in the IAM world AWS created, this actually makes sense.

Consider the docs here: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ExamplePolicies\_EC2.html#iam-example-runinstances-launch-templates

Note the language: `The policy uses the ec2:IsLaunchTemplateResource condition key to prevent users from overriding any of the launch template *resources* in the RunInstances request.` So that clause only applies to resources (like NICs, IPs, etc), but not instance-type, which is not considered as a resource.

I asked that they revise their docs to explain this, and of course, nothing's changed.
> 1 votes

---
Tags: amazon-ec2, aws-iam
---


title: How to download a VM image from GCP?

text:
# Question
Title: How to download a VM image from GCP?
I do not see a download button. I would like to download a VM image that was created on GCP using Packer and I would like to run it locally in Virtualbox.

# Answer
I don't know of a way to get the actual GCE VM images.

But it's possible to export the images to GCS (as a `tar.gz` archive):

> If you need to move your Compute Engine boot disk data outside of your Compute Engine project, you can export a boot disk image to Cloud Storage as a `tar.gz` file.
>
> You can export a custom image as backup or for sharing by exporting the image to Cloud Storage. This method is ideal for sharing individual images with other projects that do not have access to your images.

You can then download this `tar.gz` file and, I guess, create a VM image from it to run on your local machine. I imagine the process would contain steps similar to those detailed in Migrating VMs to Compute Engine and Importing Boot Disk Images to Compute Engine, only seen from your VirtualBox perspective instead of the GCE one.

These guides are making it easy to transfer VM loads **to** GCE. I guess they don't have incentives to do the same for transfers **from** GCE, which probably explains why there is no guide for it ;)
> 3 votes

---
Tags: packer, gcp, virtualbox, images, vm
---