# Question
Title: what do you think of making fast decisions?
I am trying to convince my fellow workers and management as part of DevOps transformation to adopt "fail fast" in almost everything. Amazon has two principles "bias for action" & "are right a lot" which I belive are quite similar. my dilemma is espically when adopting a new solution. as an Enterprise, they tend to evaluate several products first, then run many proofs of concepts, then an open bid. which is a waste of time and resources. my take is that decide on one solution quickly based on technical and financial merits, start using it if you fail, you can select another technology, reversing a decision on technology is not a big deal, isn't DevOps about being Agile and having quick feedback loops? I would like to hear your input.

# Answer
> start using it if you fail, you can select another technology

This approach has two major drawbacks:

1. Technical debt

While it may be tempting to apply "fail fast" mentality to architecture and design, skipping or abbreviating the design phase would be a mistake. fast-fail should be used in conjunction with good design practices, not in lieu of them. Making hasty decisions can often result in poorly scaled solutions. While using MySQL seems like a good idea when you are working with a 100mb database to generate reports, it scales poorly into the terabyt range for doing your archival reports. Using DNSMasq to run an overlapping DNS Zone might seem like a good political decision at the time, migrating off of that and resolving the disparity to between your DNS Zone and the overlapping upstream zone a few years later turns into a nightmare.
2. Expense

Technology can be expensive. Buying a solution only to discard it later cares with it hugely wastful expenses. But even more expensive is the time expended by personnel. Salaries are every companies' largest expense. Wasting time developing a solution only to discard it and develop a replacement is wildly inefficient.

This however should not be construed to mean that decision paralysis or analysis paralysis isn't a problem. It is! Fail-fast just isn't the best way to combat it. There are other, much more tried and true ways to combat this without comprimising good design principles.
> 4 votes

# Answer
I believe in 'fail fast' but it's also important to tackle understanding an opportunity in the right way.

If you come up with an idea, code in a vacuum for a bit, then release it to people, you will undoubtedly fail fast, but not in a way you would like. A better approach is if you come up with an idea, clearly articulate your hypotheses on why you think it's a great idea, then validate that idea with potential clients to see what **they** think. This will give you a great opportunity to clarify the idea, refine it, and possibly reject it outright.

I never cease to be amazed at how much my initial idea improves from just a little collaboration with people that I would want to 'sell' it to. And - to be honest, I've discovered why many of my ideas really weren't all that great - and shelved them fast (before any code was written often).
> 2 votes

---
Tags: project-management, agile, practices
---


title: How to deal with authority bias?

text:
# Question
Title: How to deal with authority bias?
https://en.wikipedia.org/wiki/Authority\_bias

> Authority bias is the tendency to attribute greater accuracy to the opinion of an authority figure (unrelated to its content) and be more influenced by that opinion. The Milgram experiment in 1961 was the classic experiment that established its existence.

I have experienced that some people that have a lot of influence promote silos and are anti DevOps. How to deal with this?

# Answer
I bet you're not the only one with such experiences :)

One possible approach today (which wasn't available in the early DevOps days) is to counter-argument with the many research-based publications from industry-recognized influencers. Like Accelerate or 2017 State of DevOps Report to name a few.

Another approach could be to propose and implement DevOps-based pilot projects to actually measure and compare performance parameters against the existing silo'd solutions. This *could* be tricky, though, as those influencers could insist on requirements cornering the pilot projects. For example insisting on individual/small team development speed which could be an obstacle for optimizing and speeding up the overall larger/entire project team's development.
> 1 votes

---
Tags: culture
---


# Question
Title: What are the expected effects of the acquisition of GitHub by Microsoft?
When one navigates to github.com it is announced that Microsoft has acquired Github.

There are some rumors that multiple projects were migrated to other git providers like gitlab and bitbucket. What effect will the acquisition have from a DevOps perspective. To be more specific, the integration of Travis and GitHub works out of the box? Should one worry? E.g. will users have to buy licenses in order to use GitHub? Will the travisci integration be neglected?

# Answer
Microsoft has heavily invested in their Git Virtual File System. It is unlikely then that such a key system central to Microsoft's internal development processes will be going anywhere any time soon, and GitHub is a smart dovetail with Microsoft's portfolio and strategic direction.

Microsoft has recently been moving towards more freemium services \- which GitHub already is - and moving towards open source \- the company was one of the top contributors to open source in 2017. Other notable examples are things like:

This also provides a strategic opportunity to integrate Visual Studio (which already has pretty good integration with Git) more tightly.

In short, Microsoft will probably do very little to harm GitHub - it will only work to make it solvent. This acquisition may have been very much about preserving GitHub to ensure that Microsoft will be able to continue to leverage its' features - the very same features you probably use. Nonetheless, guardianship of GitHub will be a difficult balancing act, as Microsoft must be careful not to alienate developers but they may have to end the gravy train as GitHub has been bleeding money for some time. Ultimately this acquisition is likely to be a good thing for the developer community as it will help to preserve and continue the public services that GitHub provides, but at worst things will look like the migration from SourceForge to whatever the new and shiny hot young thing is.
> 11 votes

---
Tags: github, business
---


# Question
Title: Code coverage versus functional coverage
We are working on a big Java project where we use JUnit tests. Our Jenkins helps us to monitor the code coverage and shows us the progress we make with this.

But there seems to be something lacking here. In our case, one of the apps results in a zip file with data files generated based on different source files.

All code to generate these data files is covered, but that doesn't tell us if the content of the generated files itself is OK. Is this just a matter of generating additional JUnit tests and rely on the knowledge of the team to know what needs to be tested? Or can this be controlled and monitored in some way?

# Answer
> 1 votes
The way I would approach this would be to have a second part of the junit test that reads the generated zip file and then verifies it is correct. You are right that if you are only ensuring you can generate the file, but the file is incorrect, the unit test is incomplete.

# Answer
> 1 votes
This is my opinion: Coverage says nothing about correctness, so if you are writing unit tests for coverage, you are doing it wrong. Unit tests shall originate in requirements, the resulting coverage is a measure for completeness of your test suite and your code quality. If the coverage is low, chances are that

* your unit tests don't cover all requirements.
* there is dead code.

---
Tags: jenkins, testing, builds, code-coverage
---


# Question
Title: after automatic update jenkins fails with `InvalidBuildsDir: .../${ITEM_FULL_NAME}/builds does not resolve to an absolute path`
I had a working Jenkins running as a Service on a Windows-2000 Server

I updated my Jenkins from 2.120 to 2.122 via Jenkins own update mechanism.

In the same run I updated some plugings too.

Now my Jenkins fails to (re-) start with this error:

```
jenkins.model.InvalidBuildsDir: /development/workspace/jenkins/${ITEM_FULL_NAME}/builds does not resolve to an absolute path
    at jenkins.model.Jenkins.checkRawBuildsDir(Jenkins.java:3060)
    at jenkins.model.Jenkins.loadConfig(Jenkins.java:3009)
Caused: java.io.IOException
    at jenkins.model.Jenkins.loadConfig(Jenkins.java:3012)
    at jenkins.model.Jenkins.access$1200(Jenkins.java:304)
    at jenkins.model.Jenkins$14.run(Jenkins.java:3104)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:296)
    at jenkins.model.Jenkins$5.runTask(Jenkins.java:1068)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:214)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused: org.jvnet.hudson.reactor.ReactorException
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:282)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:48)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:1102)
    at jenkins.model.Jenkins.<init>(Jenkins.java:904)
    at hudson.model.Hudson.<init>(Hudson.java:85)
    at hudson.model.Hudson.<init>(Hudson.java:81)
    at hudson.WebAppMain$3.run(WebAppMain.java:233)
Caused: hudson.util.HudsonFailedToLoad
    at hudson.WebAppMain$3.run(WebAppMain.java:250)

```

I suspect that there is a problem to resolve the property `${ITEM_FULL_NAME}`.

But on the other Hand I'm missing the drive letter (`D:`) in the path reported too. The Jenkins instance is running from the same drive (`D:`).

What are my Options to track the Problem and fixing it?

# Answer
I'm not very familiar with Jenkins on Windows, but I imagine your issue lies in the fact installing Jenkins on drives other than `C:\` are more difficult to deal with. Since Jenkins is not running, you'll have to configure `config.xml` by hand.

Here's a post on stackoverflow.com that may help you. As they state, you'll probably need to find the **Workspace Root Directory** to `D:\Jenkins\${ITEM_FULL_NAME}\workspace` and **Build Record Root Directory** to `D:\Jenkins\${ITEM_FULL_NAME}\builds` (or whatever values are specific to your environment) in your `config.xml`.

The post also mentions the *bubble help* info that is useful:

> This value may include the following variables:
>
> * `${JENKINS_HOME}` — Absolute path of the Jenkins home directory
> * `${ITEM_ROOTDIR}` — Absolute path of the directory where Jenkins stores the configuration and related metadata for a given job
> * `${ITEM_FULL_NAME}` — The full name of a given job, which may be slash-separated, e.g. foo/bar for the job bar in folder foo
>
> The value should normally include `${ITEM_ROOTDIR}` or `${ITEM_FULL_NAME}`, otherwise different jobs will end up sharing the same workspace.
> 1 votes

# Answer
The main problem I faced with a similar jenkins.model.InvalidBuildsDir exception is, that the given path was embedded in white spaces between tag markers and the check then thinks the path starts with white space, thus declaring it non-absolute.
> 0 votes

---
Tags: jenkins2, installation, startup
---


# Question
Title: Other tools to provision virtual machines
I've been using vagrant for provisioning virtual machines (Linux only) and so far I've been happy with it. The two things I dislike about it - the hardcoded dependency on the `user vagrant` as well as making the very first interface a virtual box NAT - this causes some routing problems when communicating outside my LAN - it has eth0 created by vagrant and eth1 in bridged mode. As far as I understand, this was a design decision so there is no way around it.

I currently use vagrant

* To create a Virtual Box virtual machine,
* Set Static IP/MAC addresses
* Create my own secondary user, and then treat it as a standalone "physical" host. I then use this user for any additional provisioning.

I can delete the `user vagrant` \- the problem is that if I do vagrant up (or vagrant halt I believe to gracefully shutdown) it tries to ssh and the command hangs while it retries. The other problem is that I need to delete it each time I spin up a VM and I **cannot** when doing the first time provisioning because I am logged in as user vagrant.

I've been wondering if there is any other similar software/tool that just does a one time provision, sets up a static IP/MAC address, and allows for easy start/resume/shutdown when testing/deploying/maintaining.

I've been looking at Hashicorp's packer but I don't think it is possible to automatically configure IP/MAC addresses.

I am mainly looking for creating a virtual machine from scratch as if it was installed on a bare metal desktop computer by hand using a ISO on a CD.

In the end, using packer to provision and then using VirtualBox command line tools to recreate some of the functionality that vagrant does will suffice (sett IP/MAC address) - but I wonder if there are any other tools used for creating **secure** (no vagrant user and ssh only with private key) vms.

# Answer
Have you considered to use Ansible?

You have some resources about using Ansible with VirtualBox and also vagrant
> 2 votes

# Answer
> I've been wondering if there is any other similar software/tool that just does a one time provision, sets up a static IP/MAC address, and allows for easy start/resume/shutdown when testing/deploying/maintaining.

Sure... a virtualbox VM is simply a XML file. You can copy/modify it as such easily. Use `vboxmanage` to clone the hdd images as necessary.

So, I don't think it would be too far fetched to just create a template VM in virtualbox once, and then use a little script to do the rest. This way, you have complete control over everything. As you are using Ansible to provision the fresh VM anyway, you don't lose that much by ditching Vagrant.
> 1 votes

# Answer
This is targeted for something a little more industrial strength, but have you looked at Foreman? I use the Foreman bootdisk plugin to provision my VMs and bare metal machines via kickstart using iPXE. You will need a DHCP server for that though...
> 0 votes

---
Tags: vagrant, provisioning, virtualization
---


# Question
Title: Copying a file with special character & spaces
I have been trying to copy a file from my remote server to AWS s3, but the file name is quite weird and I am not able to figure it out what can I do. I tried using escape characters but that did not work. Filenames:-

```
# R?glement de zzzzzxxxxx v1.0 #.pdf
# R?glement de xxxxxyyyyy v1.0 #.docx

```

My locale is like this :-

```
LANG=en_US.UTF-8
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL=

```

I am using below command

```
aws s3 sync file_name s3://bucket_name/raw_files/ --sse

```

Please suggest

# Answer
Note that the suspect filename character may not actually be `?`, that could be just a filename displaying artifact if your system somehow doesn't properly display that character. BTW, `R?glement` sounds awfully close to the French `Règlement`.

One possible approach to accessing those files without worrying too much about typing in those characters or escaping them would be to try to wildcard them instead. In your case try something along these lines (use a more specific pattern if there are other files matching this one besides the 2 files of interest):

```
aws s3 sync R*glement*pdf s3://bucket_name/raw_files/ --sse

```
> 0 votes

---
Tags: amazon-web-services, linux, amazon-s3
---


title: Storage gateway reset cache

text:
# Question
Title: Storage gateway reset cache
I have a problem in the storage gateway: the cache volume is full. When we look at the operating system there is still space available on that volume.

Based on the referenced aws model, only the files accessed regularly should be kept in the cache, but apparently it is keeping all the files, thus filling the cache and causing the requests of the users and applications to be made directly in aws. This causes slowness: it takes 2 minutes to open a 2 MB pdf file.

Our local storage is very large (4TB volume) and it is full, in addition to other smaller volumes that are also almost full.

How can I check if it is actually retaining only the recently used files or if it is retaining all the files?

And how could I perform the cache cleaning and start a new storage cycle in a safe way in order to avoid increasing the cache?

# Answer
After much tryings, i was look for problem with implementation of solution and reads documentation of resource, and to my surprise that requirements are not attends.Só i change that configurations mistakes and the storage gateway is flying.

The misconfiguration was about requirements, the minimo indicated by AWS is 16GB of memory, and at least 20% local storage used to cache all data used loaded to AWS storage.
> 0 votes

---
Tags: amazon-web-services
---


# Question
Title: Way to Connect to all ec2-instances one by one
Is there any way through which I can connect to all ec2-instances one by one through Python script to extract some server information from there. Please note that all ec2-instances lies behind a bastion host. Using shell script I can do that but how about python. Is there any specific library?

# Answer
You can benefit from EC2 inventory script provided by Ansible - click It will allow you to to connect to all ec2-instances, one by one (in `serial: 1` mode)
> 0 votes

---
Tags: amazon-web-services, amazon-ec2, python
---


# Question
Title: Health monitoring of web pages without using GET check
I am looking for a solution to monitor the health (`DOWN` or `UP`) of my web pages. Let's say I have three services,

```
site.com/data
site.com/user
site.com/id

```

Now, what I can do is that I can create a `GET` request which will be served every second and check if `response status code` is `200 OK`. If it is anything else than `200 OK`, then it will trigger an alert. But, I don't want to hit the server continuously, as it will create unnecessary logs. I am using a `mongodb logger`. What is the recommended way to achieve this?

# Answer
I see two way to solve this problem:

1. Check your logs often and ensure there was an access with code 200 in the last N seconds/minutes and no code 5xx meaning there's a server side error. (the often should be coherent with how long you are ok with no entries)
2. Keep an active check but:

* use a HEAD request (so there's less data to return by the server)
    * use a custom user agent for the check
    * configure your log exporter to ignore this user agent.
    * take a larger check interval, 1 second is probably over obsessive

Depending on how often your sites are called you may wish to mix both methods to have a proper check even if there's no traffic from real clients.
> 5 votes

# Answer
One possibility would be to monitor the responses to requests from other site users. For example I'd check if there were `200 OK` responses and skip hitting the server is such responses were sent in the most recent X minutes (up to you to determine which time interval is best fit for your application).

Maybe checking on the (server side) errors as well and hitting the server as well if a sudden change in the error occurrence rate is detected - just to be sure.
> 1 votes

# Answer
While I like the other great answers already here, I'd like to add:

# use GET against a different port

You could have your app listen on a different port which isn't logged. `haproxy` and such are happy to check for health on port A for a service that is live on port B.

# don't log canary checks

You could designate certain data items for monitoring and other tests to retrieve and not log requests against those designated items.

# set a lower TTL for canary entries

It is a bit scary to cut off logging, so maybe you could delete the entries related to logging more aggressively since it is going into a database? This avoids checking a code path that turns out to not fail when the user experience is failing.

# end-to-end checks are pretty important

Having something that is able to

* see whether it can get to you
* and whether your thing is doing its thing
* and whether your thing can talk to everything it needs

often ends up being pretty vital to detect issues that you wouldn't catch by focusing on any one part of this. Paying the price for logging is usually negligible compared to the value and peace of mind you get from end-to-end checks.
> 1 votes

# Answer
You can use Uptime Robot, it offers a very good free layer (up to 50 sites free).

You can configure your sites (they are called monitors) in many ways. A simple `GET` (your case), `ping` and more. In case you want to have an idea, the screen to add a monitor look like this:

You can also configure your email to receive alerts, it has more options but I haven't explored which one of them are on the free tier, but in the other options you can find SMS, slack, hipchat and more.

It has a very good dashboard (I never use it though, I just get the email notifications) where you can see the status of the health checks and the type of error if it's the case.

The only downside on the free tier is the five minutes interval, I would prefer to use a two or three minutes interval but most of the sites don't need one second accuracy on the health check anyway.
> 0 votes

# Answer
One method is that you do TCP-only monitoring. In this scenario, you establish a three way handshake (or use tcp-half-open monitoring in which no connection is established - you merely look for an \[syn, ack\] to your \[syn\], and don't \[ack\] the \[syn, ack\] back) and then close the connection assuming it is the web server listening on that port.

A better option however, is to figure out how to exclude your health monitor traffic from logging, though I am unable to give specifics on doing that with rails/mongodb logger.
> 0 votes

---
Tags: monitoring
---


# Question
Title: How not to break Jenkins Pipeline when adding new parameters (declarative pipeline)
An Issue I have been struggling with is whenever I add a new parameter to a Jenkins Pipeline, the next job will not correctly get this parameter. This is especially an issue if those parameters are required for certain pipeline procedures like determining agent. This issue is further frustrating because on a job that already exists, when the default parameter is changed the new job will still use the old one; but when creating a new job it will take none.

In order to solve the parameter issue from the first run of a job, and also to solve the parameter for changes to the parameters for jobs that already ran we need a solution. (At the moment, I have a workaround, which I am posting as an answer, but I would like to know if there is a better solution)

# Answer
> 7 votes
Here is a solution which seems almost like a workaround that I have found which works for declarative pipelines.

First, we need to add some code at the start of our file before we enter into the `pipeline{}` section. The code we add will run before the pipeline section, and will check if the env/param exists, and if it does will take the value, otherwise we will give it a default and assign that to a variable that we can access during the pipeline. Then we need to change our pipeline code references from the parameter to the variable.

Below is a sample that defines an arbitrary parameter, and also an agent label to be used. As you can see, the variables can be used during stages and also during procedures such as determining the agent.

```
def MY_VARIABLE = null
if (env.my_parameter) {
  MY_VARIABLE = env.my_parameter
} else {
  MY_VARIABLE = "my default value"
}
echo "MY VARIABLE IS SET: ${MY_VARIABLE}"

def MY_AGENT = null
if (env.my_agent) {
  MY_AGENT = env.my_agent
} else {
  MY_AGENT = "agent_label"
}
echo "MY AGENT IS SET: ${MY_AGENT}"

pipeline {
    parameters {
        string(defaultValue: "my default value", description: 'This is my parameter', name: 'my_parameter') ;
        string(defaultValue: "agent_label", description: 'This is my agent label', name: 'my_agent') ;
    }
    agent {
      label "${MY_AGENT}"
    }
    stages {
        stage('my stage') {
            steps {
                sh 'echo ${MY_VARIABLE}'
            }
        }
    }
}

```

---
Tags: jenkins, jenkins-pipeline, jenkinsfile
---


# Question
Title: Terraform Issue - Can't find Data Source in Sub Module?
I'm fairly new to Terraform and have been following this example on creating an EKS cluster - https://github.com/terraform-providers/terraform-provider-aws/tree/master/examples/eks-getting-started

Only difference is that I've broken the structure up into modules:

```
-main.tf
-vpc
   vpc.tf
   vars.tf
-eks
   eks-cluster.tf
   eks-worker-nodes.tf
   vars.tf
   outputs.tf

```

In 'main.tf' I've got the following

```
provider "aws" {
  region     = "us-east-1"
}

# Using these data sources allows the configuration to be
# generic for any region.
data "aws_region" "current" {}

data "aws_availability_zones" "available" {}

```

When I run a 'terraform plan' I get the following error which comes from eks-worker-nodes.tf :

```
 * module.eks.local.demo-node-userdata: local.demo-node-userdata:
 Resource 'data.aws_region.current' not found for variable
'data.aws_region.current.name'

```

Is there any reason why it wouldn't be able to access the data source? I'm fairly new to Terraform but I've done some googling and couldn't find a clear answer :/

# Answer
> 2 votes
You need to pass that info to your module or also declare it in your module again. For example, for the region, pass it as a variable to the module:

```
module "foo" {
  ...
  current_region = "${data.aws_region.current.name}"
  ...
}

```

Or add it inline in your module, just like you did in your `main.tf`

---
Tags: amazon-web-services, kubernetes, terraform
---


# Question
Title: Ansible Dynamic Inventory - generate inventory with ip addresses under specific tag with underscore instead of dot
I am new to Ansible. When I am using `ec2.py` dynamic inventory to generate inventory, after the playbook is finished to run, the results are shown as list of IP addresses under specific tag with underscore instead of dot. For example I want to run a playbook on instances with specific tag, I wonder how other people work with IP addresses?

```
---
- hosts: tag_test_staging
  sudo: true
  tasks:
  - name: Make sure that we can connect to the machine
    ping:

    PLAY RECAP *********************************************************************
    10_80_20_47                : ok=0    changed=0    unreachable=1    failed=0

```

For private address, ec2.ini:

```
hostname_variable = private_ip_address
destination_variable = private_ip_address
vpc_destination_variable = private_ip_address

```

The results from ec2.py:

```
 "tag_test_staging": [
    "10_80_20_47"
  ],

```

For Public Address, ec2.ini:

```
hostname_variable = ip_address
destination_variable = public_dns_name
vpc_destination_variable = ip_address

```

The results from ec2.py:

```
 "tag_test_staging": [
    "52_28_11_11"
  ],

```

# Answer
It works if I comment out the hostname\_variable, because it allow to override the inventory\_name with an ec2 variable, instead of using the destination\_variable

```
#hostname_variable = private_ip_address
#hostname_variable = ip_address
destination_variable = public_dns_name
vpc_destination_variable = private_ip_address

```

For Private IPs:

```
destination_variable = private_ip_address

```
> 5 votes

# Answer
I had the same issue. I was getting back ip addresses with underscores instead of dots. But I have several times now noticed that changing the ini file and re-running ec2.py won't immediately return the correct information. Once I changed to private\_ip\_address and it kept returning private\_dns\_name for a while. This time, after changing to private\_ip\_address and re-running it several times -- trying it and trying it and trying it -- after about 20 minutes it suddenly started returning IP addresses with dots instead of underscores! I dunno. Weird AF, right?
> 0 votes

---
Tags: ansible
---


# Question
Title: How does jenkins publish over ssh store passphrases?
When I look in jenkins.plugins.publish\_over\_ssh.BapSshPublisherPlugin.xml I can see several entries, that have secretPassphrase populated with a string 44 characters long. By using hudson.util.Secret.fromString in the /script tool, I can decrypt them and see that password. Fine so far.

However... I also have an entry there where the secretPassphrase is populated with a string only 24 characters long, which cannot be decrypted. Trying gets me an empty result, which is strange, because decrypting "asdfasdf" returns "asdfasdf". So there must be something special about this string which is preventing the decryption for working, or from returning the default. But deployments to this server work; so the passphrase is correct. But the secretPassphrase is NOT the actual password on the server.

Which means jenkins must be using some other method of decrypting that passphrase, besides the standared hudson.util one. What is it? How can I recover this password?

# Answer
According to the source code for the Publish Over SSH plugin, the passphrase is encrypted and decrypted using `hudson.util.Secret`, and the encrypted value is stored in the plugin's XML configuration file.

Supposedly there is ongoing work to switch to using Jenkins' built-in credentials store; however, I don't see any active pull requests for such a change.
> 1 votes

---
Tags: jenkins, jenkins-plugins, passwords
---


# Question
Title: Heroku Slack integration without using Github
Heroku has this great Slack chatbot (chatops) that lets you send messages if a deployment pipeline succeeded or failed. Unfortunately, while setting it up and logging in to Heroku, it asks for your github account and will not let you chose a different DVCS provider.

Since I am using Bitbucket, this leaves me without a solution for pipeline notifications. Has anyone solved this in the past?

# Answer
> 4 votes
ChatOps uses GitHub status checks, so it's decidedly GitHub specific and won't work with another provider.

However, if you're just after notifications, what about Heroku's App webhooks? They have an 'entity' for new app releases (`api:release`) which might partially help you... in that you can set up a notification for successful releases, at least.

Alternatively, you could listen to the `api:build` entity's `create` event to know when a build starts, and the `update` event to know when its status changes - which I haven't used but presume would include a field for whether it succeeded or failed.

To subscribe to a webhook, you use the Heroku CLI:

```
heroku webhooks:add --include api:build --level sync --url https://example.com/hooks

```

This is outlined in more detail in the documentation here. The payload you receive will look like this.

This isn't going to get it straight into Slack for you though, so you'll need a piece of middleware to do that. Here's a tiny Sinatra app I forked from a fork. It converts Docker Hub notifications into the format Slack expects for an incoming webhook. It should be pretty easy to edit it to make it convert Heroku webhooks instead. Then, voila, the tiny app itself deploys on Heroku, and that's the URL you stick in the webhook subscription above.

This is probably a lot more steps than you wanted. Someone may have already done it somewhere... but I can say that at least doing it the above way will give you a lot more flexibility to create your own customised workflows where you need them!

---
Tags: git, slack, authentication
---


title: How to track changes to AWS settings?

text:
# Question
Title: How to track changes to AWS settings?
Is there a way to track changes we make to the AWS system?

For example, changes to a subnet setting, from using *nat* to *iwg* \- these display a message, and then disappear.

Is there a way to get AWS to create a log so that one can track which changes were made to what and when?

The closest thing we have now is the ElasticBeanstalk events - but even that only tells you what AWS did, not which settings were changed to cause the events.

# Answer
AWS has the CloudTrail service that tracks most of the API calls done in your account. Then stores these in files in your specified S3 bucket.

https://aws.amazon.com/cloudtrail/

Using CloudTrail you can see who, or which service, called which change - in many cases also including which arguments were used.

Unfortunately it doesn't support ALL services at the moment.
> 12 votes

# Answer
There are multiple AWS services that can help with this, and it depends really on your exact needs as to which would suit you best. Different features (and costs) apply to each.

CloudTrail has been mentioned, but the the first option that comes to mind for me given your question (and it's mentioned in a comment on @Evgeny's answer) is the **AWS Config Service**. It stores 'snapshots' of your AWS configuration at points in time (in an S3 bucket), but helpfully also sends any changes to an SNS topic. You can then deal with these however you like. For example, on a low traffic account I have these going straight into Slack; on a high traffic account I am tracking the `NumberOfMessagesPublished` metric on that SNS topic to notice if a larger number of changes are made than usual.

AWS Config also offers a 'rules' service; these are a bit more pricey than I would expect them to be but depending on your needs they could be useful. Just don't activate them all at once like I did when I was playing around... the month's charge for each rule applies right away. ;) (But you can use Config without using the rules - that is what I am doing at the moment).

There's also **Trusted Advisor**, which doesn't do what you asked for exactly but might be handy to perform certain checks against how engineers configure your infrastructure, such as whether anyone's left S3 buckets open. It's most useful with a Business level Support plan or above, as many of its checks are blocked otherwise.

Then there's third party tools like **CloudCheckr**, which combine aspects of AWS Cost Explorer, Trusted Advisor, Config, CloudTrail, CloudWatch, Inspector and GuardDuty. Useful if you want to get in really deep but save time configuring everything yourself.

Alternatively, you could manage your infrastructure using a tool such as Terraform or CloudFormation, and mandate that all changes must be made through those. You can then commit your config files/templates to source control, and even test them against the live infrastructure in CI and fail the build if someone's made non-tracked changes. That way, your commit history becomes your audit log - but it does require your engineers to be disciplined!
> 5 votes

---
Tags: amazon-web-services
---


# Question
Title: What's a good devops approach for a single developer writing python web applications?
I'm guessing this question will seem incredibly trivial for some readers, but as someone who is a developer but with little experience of deploying apps in anything other than a manual, hit and hope sort of a way, I hope you will understand that it's quite daunting to see the number of different approaches and tools there are, so I could do with a bit of advice to get me started in the right direction.

I am a developer, now only in my spare time, which is limited. Up to now I have worked with Java, building webapps, and have been reasonably happy with deploying a war file to a Tomcat environment which keeps things nicely encapsulated.

I am now working in Python and Django, but as I get closer to the point where I need to deploy, I want to set up a solid devops workflow to automate as much as I can and ensure I can deploy reliably, but given that my use case is relatively simple, I want to avoid learning a big fat toolset which is over-engineered for my needs and which requires a big investment of time I would rather use coding my app.

So I am looking for a middle ground which allows me reliably to deploy and manage my app(s) without investing a huge amount of time setting up and learning a big devops ecosystem.

Some more details...

### Context

1. I develop on a Mac, using PyCharm to build Django 2, Python 3.
2. I use git (but not on github) to manage software versioning.
3. I am comfortable with other languages and scripting languages and have written a few (probably fairly amateurish) bash scripts, although I don't enjoy bash. I've also dabbled with Perl, which I realised isn't really a language for dabbling (i.e. you need to spend time learning it properly)
4. I intend to deploy on a VPS environment, probably DigitalOcean.
5. My app isn't mission critical but it is important that I know if the site goes down, and need to have a way of reliably recovering if it does, whether this be restarting the app, restarting the server, or moving to another server (or other).

### Specific Requirements

1. Ability to set up a new environment to receive the app.

Up to now while I am learning, this has been manual, and every time I have done it I have started from scratch with a new Droplet. I would like this to be much simpler (automated) so that if I have to set up a new environment in an emergency I can do so reliably.
2. Ability to deploy the app to a staging environment which is as identical to the live as possible, ideally as an automated process triggered by a git push using a continuous integration approach (which I have never done before).
3. Ability to "press the button" when I am happy with the app in the staging environment to push to a live environment ideally automatically.
4. Way to monitor the site (just a poll to a page will do)
5. Way to switch live site to another server if I need to recover from an app or server failure on the live site. I think maybe a Blue-Green approach would work for me?

### What have I tried or considered?

* Manual set up of live environment with Django app, then manually copy new codebase to it when there is a change. This feels prone to human error and I fear making a mistake in a deploy causing an un-recoverable failure.
* Docker. I admit when I found out about Docker it seemed like a dream come true but after a bit of experimenting and research I am daunted by how much I need to learn and know to get this up and running and to manage it. It may be that this is worth it because once it's working it is very low risk but at the moment it feels like a larger investment of my time than I am hoping for.
* Bash scripts. Use them to set up the original environment and for specific tasks like updating the app. My worry about this is that the scripts will be code which needs testing and I fear it would take a lot of time to build a reliable toolset this way.
* I've looked at Digital Ocean's options for floating IP addresses and the ability to have two servers for a "blue green" approach which seems quite sensible. If I go this route I still need to be able to automate the deployment.

So... I am looking for advice on a devops approach which finds the right balance between minimising risk (e.g. the risk of breaking the live app with an update, or the risk of being unable to recover from a failure) and minimising the time investment I need to make to set up the environments and workflow.

# Answer
The posted answers have been very helpful in allowing me to re-think my problem and various approaches. I haven't implemented a solution yet but I have decided on an approach so I am documenting that and selecting it as the answer. In summary it is this:

### My Chosen Approach

* For the live environment I will use two Virtual Machines (probably using DigitalOcean droplets) running Ubuntu and configured exactly the same.
* I will employ a Blue-Green approach using the Floating IP facility within DO to maintain my two identical server as Live and Pre-Prod/Backup.
* I will create a VM (probably using VirtualBox) in my development set up for use as a staging environment. This VM will be set up exactly the same as my two live servers.
* I will write a single common script in Python for setting up an environment from scratch and I will use this to configure my staging environment and my live/pre-prod pair.
* I will employ git hooks to trigger updates to the environments (probably manually triggered).

### Considerations Which Drove This Approach

* Docker: I have decided against it. Although I take seriously the responses (thank you @Levi and @Dan) which say I should re-visit and it shouldn't be that bad, I have had too many experiences in the past of embarking on something new and realise I have fallen down a rabbit hole which eats up time and takes an age to get going. I think it would be different if I was even working with one other person but as someone working completely alone every minute is precious.
* Virtual Machines: I hadn't really considered this until I started working with some of the Docker tutorials which use VMs to demonstrate the Swarm functionality. The idea of being able to create a brand new environment which I have complete control of is very appealing.
* Scripting: Prompted by @AnoE's helpful reply I have done a bit more digging and it seems that Python is recognised as a viable option for scripting and as that is what I am writing my app in it seems there should be some synergy (If I need to learn something new for my script it will be knowledge I may use in writing my app)

I will update once I have made some progress with this and if it goes horribly wrong I will acknowledge I maybe made the wrong choice!).

### Update on 20th October 2018.

I set out to write a Python script but this often involved invoking a bash command from Python and then getting the response back and I found this added to the development time quite a lot. After a couple of weeks of slow progress I looked elsewhere. I admit I may have been approaching it wrongly but I needed something which was going to be faster.

I eventually settled on Vagrant / Ansible / VirtualBox and after more months than I like to admit got something that worked well, but after a lot of work learning some new skills (Vagrant and Ansible were completely new to me). I then applied the Ansible script to provision a DigitalOcean Droplet and found this worked really well. I have become a fan of Ansible but even though I agree (with reviewers) that it is relatively easy to use, it's still a new paradigm and quite a steep learning curve.

At the time of writing, I have provisioned two separate Droplets on DigitalOcean in a blue-green configuration, using DO Floating IP addresses to switch between the two, and on each the application is within a git working copy so I just need to refresh the Master to update an environment.

I am having a problem getting Floating IPs to work as I was expecting but I expect to resolve that soon and then I will have a working DevOps environment.

The big advantage of this approach is that the way Ansible works, once you have something working it's relatively easy to make it work in a different environment and this may not be so easy to achieve with a python script (or at least you have to build this in which is extra work).

I think the big lesson is that things take longer than I expect them to and learning a new technology always brings unknown unknowns. This shouldn't be a surprise to me - but it always is and working as a lone developer, this happens to me a lot.
> 0 votes

# Answer
I am not familiar with Python development nor DigitalOcean, so I'll just offer a few pointers:

* The goal is to automate. Everything. How you achieve that is really up to you, and creating your own tooling is not far-fetched, many do it that way. One concrete and pretty low(ish) hanging fruit is to get a git post-receive hook running which deploys and restarts your test environment. If you have that, the rest should be simple.
* "My worry about this is that the scripts will be code which needs testing" - that worry is unfounded. You *are* testing those scripts every time you deploy to your test environment, after all. Especially coupled with a blue-green approach it will be fine to have bash scripts.
* "I don't enjoy bash." - then find another scripting language that you enjoy. Maybe try Ruby? The language and core libraries are very clean and well documented, and I'd say, rather easy to learn. Or, just for fun, Go(lang), which seems to be well-suited to devops tooling tasks. And finally, as you seem to like Python, you certainly can do installation tasks with that as well. From these, Go has the benefit that it creates standalone binaries and does not need a complex environment installed first, itself, so bootstrapping may be easier.
* "a staging environment which is as identical to the live as possible" - if you have a script that spins up an environment from scratch, i.e. from a more or less empty base image, then your environments *will* be identical, save for deltas encoded in your script. That's the point of all of this.
* "Way to switch live site to another server" - the only thing to ponder is what happens with your persistent data. I.e., you will want to make it so you can link your applications with different persistent volumes/stores on the fly, to be able to switch back and forth.
* "Docker - daunted" - to be honest, it should not be that bad. If you know how to build an environment from scratch with command line tools (no GUI tools), then placing those in a Dockerfile should be rather easy. The worrisome details appear when it's time to tune (i.e., reduce image sizes), but apart from that it should not be too bad. First get it to work *somehow*, then find out how to make it beautiful. The good thing is that the knowledge you gain will transfer to lots of other environments.

Good luck!
> 5 votes

# Answer
Thanks for the great question. Nothing is really trivial the first time you do it and we all were new to something once.

My first recommendation is to revisit docker. Try some different guides and tutorials. It's really simple. You have a docker file that gets "built", literally just commands you want ran on the "container" or "image". You push that image to a registry which can be public or private. You then run that image on a host machine. Docker is really important with node.js and python where you have lots of dependencies and it can really be hard to manage them sometimes. If you are using pip, and you should be, you can generate a `requirements.txt` file to feed to your docker container.

Now you said you are using git, so I would use local git hooks. You can use these to build the docker container, run automated tests and then deploy your container. You can look up lots of different guides and tutorials on this subject.

For managing your infrastructure I would you use Terraform. Terraform is great because you can spin up an environment on demand and delete it when done. My recommendation would be to start out simple and once you mastered docker and terraform you can try blue/green deployments.

Now if you are using Gitlab or willing to switch, it also offers a free ci/cd service. This includes so many cool features and is really easy to use. I use it personally for all my apps. You could completely skip the local git hooks and test in the gitlab pipeline or keep them for testing each commit locally and using gitlab to build and deploy.

I hope this was somewhat helpful.
> 3 votes

---
Tags: continuous-integration, git, python
---


# Question
Title: Jenkins fails to start after a restart with "hudson.util.HudsonFailedToLoad: org.jvnet.hudson.reactor.ReactorException"
A week ago I had to restart our Jenkins, I did so via "jenkins url"/restart At first it was stuck on "please wait while Jenkins is restarting" for like...1.5 days, so I figured something was wrong.

I troubleshooted and restarted the service (daemon) itself as this post suggested. After the manual restart the jenkins page spat out this error

```
    hudson.util.HudsonFailedToLoad: org.jvnet.hudson.reactor.ReactorException: java.lang.RuntimeException: Unable to read history for build2
    at hudson.WebAppMain$3.run(WebAppMain.java:239)
Caused by: org.jvnet.hudson.reactor.ReactorException: java.lang.RuntimeException: Unable to read history for build2
    at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:269)
    at jenkins.InitReactorRunner.run(InitReactorRunner.java:44)
    at jenkins.model.Jenkins.executeReactor(Jenkins.java:969)
    at jenkins.model.Jenkins.<init>(Jenkins.java:850)
    at hudson.model.Hudson.<init>(Hudson.java:85)
    at hudson.model.Hudson.<init>(Hudson.java:81)
    at hudson.WebAppMain$3.run(WebAppMain.java:227)
Caused by: java.lang.RuntimeException: Unable to read history for build2
    at hudson.plugins.jobConfigHistory.FileHistoryDao.getRevisions(FileHistoryDao.java:748)
    at hudson.plugins.jobConfigHistory.FileHistoryDao.hasDuplicateHistory(FileHistoryDao.java:804)
    at hudson.plugins.jobConfigHistory.ComputerHistoryListener.onChange(ComputerHistoryListener.java:94)
    at hudson.plugins.jobConfigHistory.ComputerHistoryListener.onConfigurationChange(ComputerHistoryListener.java:52)
    at hudson.model.AbstractCIBase.updateComputerList(AbstractCIBase.java:209)
    at jenkins.model.Jenkins.updateComputerList(Jenkins.java:1293)
    at jenkins.model.Nodes$7.run(Nodes.java:304)
    at hudson.model.Queue._withLock(Queue.java:1312)
    at hudson.model.Queue.withLock(Queue.java:1189)
    at jenkins.model.Nodes.load(Nodes.java:295)
    at jenkins.model.Jenkins$17.run(Jenkins.java:2739)
    at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:169)
    at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:282)
    at jenkins.model.Jenkins$8.runTask(Jenkins.java:958)
    at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:210)
    at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:117)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to read /Users/Shared/Jenkins/Home/config-history/nodes/build2/2018-06-13_11-41-13/history.xml
    at hudson.XmlFile.read(XmlFile.java:144)
    at hudson.plugins.jobConfigHistory.FileHistoryDao.getRevisions(FileHistoryDao.java:746)
    ... 18 more
Caused by: com.thoughtworks.xstream.io.StreamException:  : only 1.0 is supported as <?xml version not '1.1' (position: START_DOCUMENT seen <?xml version=\'1.1\'... @1:19)
    at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:124)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.readRealEvent(AbstractPullReader.java:148)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.readEvent(AbstractPullReader.java:141)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.move(AbstractPullReader.java:118)
    at com.thoughtworks.xstream.io.xml.AbstractPullReader.moveDown(AbstractPullReader.java:103)
    at com.thoughtworks.xstream.io.xml.XppReader.<init>(XppReader.java:63)
    at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:54)
    at com.thoughtworks.xstream.io.xml.AbstractXppDriver.createReader(AbstractXppDriver.java:65)
    at com.thoughtworks.xstream.XStream.fromXML(XStream.java:1053)
    at hudson.XmlFile.read(XmlFile.java:142)
    ... 19 more
Caused by: org.xmlpull.v1.XmlPullParserException: only 1.0 is supported as <?xml version not '1.1' (position: START_DOCUMENT seen <?xml version=\'1.1\'... @1:19)
    at org.xmlpull.mxp1.MXParser.parseXmlDeclWithVersion(MXParser.java:2608)
    at org.xmlpull.mxp1.MXParser.parseXmlDecl(MXParser.java:2592)
    at org.xmlpull.mxp1.MXParser.parsePI(MXParser.java:2466)
    at org.xmlpull.mxp1.MXParser.parseProlog(MXParser.java:1447)
    at org.xmlpull.mxp1.MXParser.nextImpl(MXParser.java:1395)
    at org.xmlpull.mxp1.MXParser.next(MXParser.java:1093)
    at com.thoughtworks.xstream.io.xml.XppReader.pullNextEvent(XppReader.java:109)
    ... 28 more

```

I guess it has to do with a slave not restarting correctly but I ain't sure, does anyone have any guesses what my next steps should be? I have already removed all the plugins for jenkins to no avail. I have also tried to start the jenkins.war file through Java, but it just says that putty can't be started and aborts. The jenkins is running on a MAC

Sorry if the solution is super obvious, I haven't got that much experience with Jenkins. Thanks!

# Answer
> 1 votes
So I feel stupid now, but I solved it. I took another look at the error and found `Caused by: java.io.IOException: Unable to read /Users/Shared/Jenkins/Home/config-history/nodes/build2/2018-06-13_11-41-13/history.xml
 at hudson.XmlFile.read(XmlFile.java:144)`

I went to the file and the found that the first row said `<?xml version='1.1' encoding='UTF-8'?>`. I also noticed that the error message said

```
Caused by: org.xmlpull.v1.XmlPullParserException: only 1.0 is supported as <?xml version not '1.1' (position: START_DOCUMENT seen <?xml version=\'1.1\'... @1:19)
    at org.xmlpull.mxp1.MXParser.parseXmlDeclWithVersion(MXParser.java:2608)

```

I changed the `<?xml version='1.1>` to `<?xml version='1.0>`, saved the file and restarted jenkins using

```
sudo launchctl stop org.jenkins-ci
sudo launchctl unload /Library/LaunchDaemons/org.jenkins-ci.plist
sudo launchctl load /Library/LaunchDaemons/org.jenkins-ci.plist

```

---
Tags: jenkins
---


# Question
Title: Chef username/passwords best practice for multiple nodes
I've been playing around with Chef to evaluate it as a potential CD tool for the company I work for.

So far I've been using it with Vagrant to provision a VM. In my recipes I've been retrieving the username/password from data bags.

I'm now looking to re-use these recipes across multiple nodes which all have different credentials so pointing to one data bag for credentials is no longer sufficient.

The only way I can see to get around the issue is to store the username and password in the node attributes, however is this best practice/secure?

I've done some googling around this and can't get a clear answer...

# Answer
Storing sensitive data in attributes is definitely not a good practice. I once used a strategy of a chef-vault per environment (dev,stg,prod), and it worked well. If you use the sensitive data within templates you can set the `sensitive` property on the template resource to true to keep the password in plain text form from showing up in the run logs and associated template file diffs

We eventually moved to Hashicorp vault, and while it took more time upfront to learn and get up and running, it was definitely a better long term solution for secrets management and automation around managing sensitive data
> 4 votes

# Answer
Storing secrets in node attributes is usually not a good practice as all nodes attributes are readable by all nodes in an organization.

There's multiple way to store secrets, Hashicorp's vault, chef-vault (a kind of more secure encrypted databag) or home made system.

You don't say a word about what are those username/passwords for, so it's not really possible to give any advice on a solution for you case.

For general perspective you can check Coderanger's talk about it here
> 3 votes

---
Tags: chef
---


# Question
Title: Git Workflow with User Acceptance and Staging Environments
I'm trying to streamline/standardize a git workflow in my workplace. The current delopment process is as follows:

1. Pull 'development' branch
2. Write code and commit and push to 'development' branch
3. Cherry pick from 'development' into 'testing' branch for User Acceptance Testing
4. Cherry pick from 'testing' into 'staging'
5. 'staging' is merged into 'master' once a week

I want to adopt a 'feature branch' work flow with testing, staging and master. Thus I don't think there is any need for 'development'. From waht I gather, feature branches should be taken from 'master'.

What I'm having trouble understanding is how to fit in the 'testing' and 'staging' environments into a 'feature branch' git workflow.

Can someone explain how that might work?

# Answer
> 2 votes
My favourite workflow is the `development`-less "branch per feature" workflow from Adam Dymitruk (http://dymitruk.com/blog/2012/02/05/branch-per-feature/). It has these branches:

* Exactly one `master` in an unbroken line from day 0 up to the version of your app that is currently in production.
* Many `feature-...` branches.
* One (or if you are so inclined more) `qa` branch.

The gory details can be found on Adam's site, but in short:

* *Every* branch goes off of `master`.
* All branches that are in active testing are merged into `qa`.
* `qa` *only* ever contains merge commits (i.e. feature branches merged on top of each other, starting from `master`), *never* direct manipulations.
* Whenever a feature branch gets new commits, and you want those in `qa`, you rebuild `qa` from scratch.
* Whenever you want to deploy a bunch of features, you create a `qa` from all of them, test to your heart's content, and then simply rename that branch to the new `master` (which, as it started out from the old `master`, will be a direct, unbroken line of merge commits from day 0). Then, all feature branches are rebased on that.
* One fundamentally important part for this workflow is to enable the `git rerere` cache feature, and distribute that cache between all developers. Hence, every merge conflict needs to be resolved only once, globally.

It's trivial to have more than one `qa` branch at a time to test different subsets of features; in your case, you could have `qa` (with everything nearly close to finished, i.e., compiles without error), `qa-acceptance`, `qa-staging` or whatever you like.

In my experience, there is always a relatively large main, ongoing `qa` branch (recreated from the master and features regularly), and when some features are picked for release, the will be a temporary smaller `qa-...`.

It is trivial to add and remove individual features from the set of `qa` or release-candidate branches on the fly, as those are recreated constantly and only ephemeral.

You will find a lot of people very much objecting to `git rebase`, but if one wraps their mind around the fact that this is just a tool like any other, it will be fine. You need good tooling for this of course, but this is not complicated to program yourself; the actual `git` commands are all pretty simple and self-explanatory.

We have extended the workflow (or rather our tooling) to even allow arbitrary sub-features (i.e., feature branches going off of another feature branch, not `master`), so multiple branches can go off of a common ancestor which is not yet in production; works like a charm. But you can get pretty far without this.

# Answer
> 0 votes
well, gitflow is your friend here. http://nvie.com/posts/a-successful-git-branching-model/

---
Tags: deployment, git, continuous-deployment, gitlab, workflow
---


# Question
Title: What Golang function could be used to enforce a minimum code coverage and return an Exit1 if false?
Follow-up to this Q&A.

How to ensure that a certain minimal code coverage is reached in Golang? E.g. if the CI runs, the build should fail if the code coverage is less than X percent.

In Java, one could use Jacoco and define a minimum code coverage. Other language like Python support minimum code coverage as well.

One option is to simply create a bash script that parsers the output of `go test -cover`, but I would like to know if there is a off the shelf solution.

# Answer
> 2 votes
There aren't any off the shelf solution currently that allows you to specify a variable number of code coverage percentage.

There is the tool Courtney which enforces 100% code coverage. This tool also allows you to comment on blocks of code which do not need to be covered.

---
Tags: code-coverage, go
---


# Question
Title: Trigger a job and get the job log on the same page in Jenkins
I would like to trigger the job in Jenkins and also get the log on the same page.

**E.g.** Kind of IDE where we can run the code in one side and get the output on the other side.

Are there any tools exists to fulfill this, or do we need to create it?

# Answer
It looks like there exists a plugin to do exactly what you want: Display console output plugin.

However, I haven't tried this plugin myself, and it looks like this plugin is not well-maintained, so I cannot vouch for the quality of this plugin.
> 2 votes

---
Tags: jenkins, jenkins-pipeline, build-pipeline, jenkinsfile, jenkins2
---


# Question
Title: Ansible service module running state utility
Ansible v2.5 introduced a new "running" state for the service module, the doc is not saying anything about it.

> **started/stopped** are idempotent actions that will not run commands unless necessary. **restarted** will always bounce the service. **reloaded** will always reload. At least one of state and enabled are required. Note that **reloaded** will start the service if it is not already started, even if your chosen init system wouldn't normally.

I presume it checks if the service is running and starts it if not, but this is exactly what the started state does.

What differs the running state from the started state ?

# Answer
> 5 votes
It's deprecated in favor of *started*.

```
TASK [Test postfix is running]
**************************************************************
[DEPRECATION WARNING]: state=running is deprecated. Please use
state=started. This feature will be removed in version 2.7.
Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.

```

---
Tags: ansible
---


title: Issue integrating acunetix with jenkins

text:
# Question
Title: Issue integrating acunetix with jenkins
The first of all, thank you for reading my question, I really appreciate it! I am trying to integrate jenkins and acunetix (a vulnerability scanner software), but it seems to be near impossible...My situation is as follows:

There are two machines, a debian 9 machine \[DEB\] and a win10 machine \[WIN\]. In \[WIN\] I installed Acunetix v11 and jenkins in \[DEB\], and I configured acunetix to be accessible from outside (following this tutorial: https://www.acunetix.com/blog/docs/use-acunetix-host-localhost/). After that, I followed another tutorial (https://www.acunetix.com/blog/docs/install-acunetix-root-certificate-another-computer/) to connect jenkins with acunetix. I managed to access to acunetix webpage from \[DEB\] via web (Not issues in network or configuration in that sense).

However, I could not connect with acunteix from jenkins despite the fact I followed every single step explained in acunetix web page. I got this error: "Please add the Acunetix scanner certificate to Java CA store" when I tested the connection from jenkins (obviously, acunetix plugin is installed in jenkins). I checked that the certificate obtained from acunetix installation (in \[WIN\]) "ca.cer" was correctly added in the keystore of \[DEB\]. Then, I made a custom keystore to be used by jenkins (as it is explained in this other tutorial: https://support.cloudbees.com/hc/en-us/articles/203821254-How-to-install-a-new-SSL-certificate-) and I got the same result...At that time, I thought that maybe something was wrong in \[DEB\], so I used another windows 10 machine \[WIN2\] to repeat the process...and I got the same result... The acunetix certificate was also included in the OS apart from java keystore in all cases.

My company contacted Acunetix support about this issue but no solution has been provided so far... Currently, we don't what to do right now... Have you dealt with this? Have you found a solution?

Thank you in advance, mates!

See you!

# Answer
We've configured the plugin properly (we're able to invoke acunetix from jenkins although we're having issues getting the report copied into the Jenkins workspace). The issue he might have run into is that you might have more than one jre installed and you have to register the certificate on the keystore JRE that Jenkins is using (you can check which one is being used by jenkins by looking at the "`<executable>`" tag in the `jenkins.xml` config file)

In a default installation the jre being used by jenkins and its keystore are located inside the jenkins folder: `.\jenkins\jre\bin\keytool` and `.\jenkins\jre\lib\security\cacerts`

With this configuration, you can verify if the certificate is in the keystore by using the command: `keytool -list -keystore "jenkins\jre\lib\security\cacerts"`
> 1 votes

---
Tags: jenkins, jenkins-plugins, security, integration, devsecops
---


# Question
Title: Data Science pipelines and monolithic model blobs
Normally, one important topic in DevOps is how we take care of automated creation and delivery of software artefacts.

With the rise of data science there is a new type of artefact - monolithic binary blobs representing a trained neural net for example or other machine learning models. Such a blob can be many GB in size and its creation is not yet standardized AFAIK which brings organizations back to the pre-CI age. Nevertheless, such blobs have their version and associated collections of training data (corpora) which tend to grow rapidly as well.

What are best practices to address this new challenge using DevOps methods - if even possible?

# Answer
> 9 votes
Personally I don't see any reason for which an Artefact Repository \- the recommeneded DevOps tool of managing artefacts - wouldn't be applicable to trained neural nets or other artefacts.

The artefact size *might* have some upper limit for a particular artefact repository, but in such case it would be a technical or policy limitation one, not a fundamental/principial one.

As for applying DevOps methodologies for the process producing these artefacts, I think most if not all of them can be applied equally well, as long as the artefacts:

* are produced from some sort of specification which supports change versioning (equivalent to software source code)
* are built via a repeatable and automatable process
* are validated using some sort of repeatable and automatable verification (similar to QA), eventually using some supporting data (training data in this case, equivalent to DB snapshots, for example)

Side note: monolithic software code delivery is still a big deal and is perfectly maintainable with DevOps methodologies (with a bit of care), not everything can be split in microservices. Size doesn't matter enough to make DevOps not applicable.

# Answer
> 4 votes
I would recommend taking a look at DVC \- an open source version control system for data science projects.

One of the basic things that it perfectly handles is managing data files (along with code) - inputs, outputs (models), intermediate results. Semantically it's similar to `git-lfs` but unlike `git-lfs` it is capable of managing files like 100GB and what is more important it does not rely on proprietary storage/format. It's completely open-source and is compatible with any network storage as a server to keep data files - S3, GCP cloud storage, SSH, FTP, etc.

---
Tags: continuous-integration, continuous-delivery, automation, methodology
---


# Question
Title: Jenkins "internal" pipeline script producing errors when running sh steps
I've been experimenting with creating a new pipeline, using the built-in pipeline editor (i.e. not from SCM). The strange bit is that, when I run *any* `sh` step, I receive the following error:

> sh: 1: Syntax error: end of file unexpected (expecting "done")

This comes from the following MWE:

```
pipeline {
    agent any

    stages {
        stage('Test') {
            steps {
                sh 'echo "FOO!"'
            }
        }
    }
}

```

The odd bit is that my pipeline scripts that *are* coming from SCM are working just fine, despite the fact that they have loads of shell scripts.

Running Jenkins 2.73.2.

# Answer
Apparently this was caused by the pipeline name having an apostrophe character in it. Renaming the build solved the issue.
> 2 votes

---
Tags: jenkins, jenkins-pipeline
---


title: Python Dictionary

text:
# Question
Title: Python Dictionary
I am getting the output from my previous executions something like this:

```
[{u'Value': '2018-06-21', u'Key': 'Deletion_Date'}]

```

I want to get only 2018-06-21 saved in a variable.

I tried this

```
deletion_date = response1['TagList'][0]['Value']

```

But I am getting error

```
deletion_date = response1['TagList'][0]['Value']
IndexError: list index out of range

```

Could someone help me with this or some alternative solution

# Answer
> 1 votes
You're not showing how the response example you shown relates to `response1` (there is no `'TagList'` in your example). It looks like `response1['TagList']` is an empty list.

Otherwise you're on the right track (you can always check your exact case directly in the python shell):

```
$ python
Python 2.7.12 (default, Jul 01 2016, 15:36:53) [GCC] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> response = [{u'Value': '2018-06-21', u'Key': 'Deletion_Date'}]
>>> response[0]
{u'Key': 'Deletion_Date', u'Value': '2018-06-21'}
>>> response[0]['Value']
'2018-06-21'

```

---
Tags: amazon-web-services, python, amazon-rds
---


# Question
Title: How to add ECS attributes to an instance using terraform
I heavily use ECS Attributes on our containerized infrastructure. I couldn't find terraform docs to achieve this. Do I need to execute aws cli commands manually to apply those attributes after creating the infrastructure?

# Answer
> 2 votes
Terraform docs are pretty comprehensive, so if it’s not in the docs it’s probably not implemented yet.

However, they’re very responsive to feature requests posted as GitHub issues, particularly those that indicate that the feature is available via the AWS CLI (or more specifically, the AWS Go SDK). In my experience, feature requests I’ve raised have usually been available in a new AWS provider release within a month or so.

I’d suggest doing up a new issue here: https://github.com/terraform-providers/terraform-provider-aws/issues

(I did do a quick search to see if ECS attributes have been raised before and couldn’t see any at a quick glance, but you may like to as well!)

# Answer
> 1 votes
In Terraform, **value\_specs** are a way of injecting extra key/value pairs into the create request to help with random vendor-specific data. Anyhow this is a workaround option and also it works well with OpenStack provider but I haven't tried with AWS. Could you please try this approach and post your comments.

```
value_specs {
      Attrkey = AttrValue
}

```

---
Tags: terraform
---


# Question
Title: Using docker-compose, what's the recommended way to handle different sets of containers for dev and prod?
I'm creating a webapp with a python backend, Nginx as the server, and Vue as my frontend web framework. When deploying to production, I build an nginx container that, on build, compiles all my frontend code into minified static files it can serve (takes ~2minutes), and on run, serves those static files and routes non-static-file HTTP requests to a different API Container (running the python app). (So two containers: nginx, and api server).

The problem I'm facing is that during development, I want to use a third container to run a hot-reloading webpack-dev-server for the frontend files, instead of having to wait two whole minutes for webpack to rebuild the frontend bundle every time I make a change. For reference, building the frontend code into servable static files takes around two minutes, whereas the webpack-dev-server can reflect changes in \<3 seconds)

My idea is to have a production nginx config that handles the production case of serving static content directly from the nginx container, and a dev nginx config that handles the dev case of routing those static requests to the webpack-dev-server container. But having different docker-compose files for different environments seems to go against docker philosophy. Am I approaching this incorrectly?

# Answer
> 4 votes
**docker-compose.override.yml, docker-compose.prod.yml and `extend`**

An outline answer would be to have a `docker-compose.yml` file for your common base configuration, a `docker-compose.prod.yml` for production-specific sets of containers and for other specific environments, `docker-compose.override.yml`. It is possible to use the docker-compose `extend` keyword/command/statement in a docker compose yml file to inherit from another docker compose yml file.

The above approach is on the basis that one would first consider only the things common across all environments as the pure, base setup and then dev and prod environments would be based off of this (inherited).

Your dev-specific packer setup could be configured in the `docker-compose.override.yml` file, which inherits from the base `docker-compose.yml`. Configuration that is common across all environments would reside in `docker-compose.yml`. Your production setup could be in `docker-compose.prod.yml`, which again can inherit from the base `docker-compose.yml`.

**References**

This seems to be the best practice approach, based on the following references:

**Some things you may or may not need to consider for your specific case**

* There may or may not be some bespoke config code required in *addition* i.e. in your deployment itself, e.g. bash scripts etc., should the above approach provide most of what you need but not quite everything. These scripts can be run from docker compose.
* Research what can be overridden when writing `docker-compose.override.yml` when "inheriting" from `docker-compose.yml`. Your particular project may require a configuration in each of the dev and prod environments that override the same thing in the base environment, thus the need to override what is in the "parent"/base docker-compose.yml environment. Consider this issue: https://github.com/docker/compose/issues/3729 \- which suggests there are limitations as to what can be overridden, particularly if you want to move a configuration item that was present in the parent `docker-compose.yml`. A better way could be to only have items common across all environments in `docker-compose.yml` and have the specifics in files for specific enviroments, such as `docker-compose.prod.yml`

---
Tags: docker, docker-compose
---


# Question
Title: Is there any relationship between "production engineering" and DevOps?
Production engineering: "Production engineering is a combination of manufacturing technology, engineering sciences with management science." (Wikipedia)

Is there any relationship to/with DevOps?

To avoid an opinionated discussion, please establish, formalize with facts and use references in your answers.

# Answer
> 1 votes
This is opinionated I feel. But for me I think the relation between the 2 is actually a massive mistake made by a lot of companies. There are a ton of software-systems hybrid jobs out there, and I've worked both ends of the spectrum, and in between. DevOps is a buzzword and it's new and shiny, so everything that falls in that realm is instantly deemed DevOps. I would argue that the same description you might give for a Production Engineer would sound exactly like what most people think DevOps is, when in reality DevOps(beware of opinions) is providing tooling for development to leverage and deliver to infrastructure. You are developing the operations of software shipping to and running on a platform. This makes a very diverse role, and that's why I think DevOps is more so a position than a profession. I consider myself a Software Engineer, but I understand the systems side of things very well so I apply my profession to a specialty. While I think, and have seen, that applied inversely, people who used to be sysadmins with knowledge of development and development processes.

I think a production engineer is similar loosely in the realm of work that is being done, but ultimately DevOps delivers solutions to and shapes the way the Development team works. Where as a production engineer is likely a systems software engineer who writes software to support infrastructure. They are writing software services which run on infrastructure in support of other services and not like DevOps might write Infrastructure as Code which is usually driving some sort of systems automation(puppet, chef, terraform, etc).

# Answer
> 2 votes
According to Facebook

> Production Engineering at Facebook is a hybrid between software and systems engineering; it keeps Facebook running smoothly and scaling efficiently. The Production Engineering team works within Facebook’s product and infrastructure teams to make sure our services are reliable and scalable. The team writes code and debugs hard problems in live production that impact more than 2 billion people around the world. From backend services like our Hadoop data warehouses, to frontend services like News Feed, to infrastructure components like our caching infrastructure, load balancing and deployment systems, the Production Engineering team keeps Facebook running.

as *team works within product and infrastructure* meaning it works along with developers and operations.

---
Tags: methodology
---


# Question
Title: AWS CloudWatch Logs Publisher plugin hangs on job completion
We use the AWS CloudWatch Logs Publisher plugin on our Jenkins and most of the time the logs are sent within 2-5 seconds of the request being sent out. However, sometimes the step will hang for up to 5-10 minutes, with no errors in the console or in Jenkins' log. The job hangs with the following status message;

> AWS CloudWatch Logs Publisher is waiting for a checkpoint on \[...\]

It always completes successfully, even with the delay.

Has anyone seen this before? Is there anything I can do to prevent the step hanging?

# Answer
> 0 votes
This is conjecture, but after watching this for almost six months here's my theory;

I think the jobs have to post their logs in sequential order. If you have multiple jobs, or even multiple runs of the same job that all push logs it will wait for them to complete in the order they were started. This can be troublesome if you have one hour-long build that started before a 10-minute job. The shorter one started second even if it finishes first, so will wait for the hour long one to finish before finishing itself.

There doesn't seem to be any fix for it, but we've learned to just ignore it and count the jobs as done if they're hanging at the logs phase.

# Answer
> 0 votes
Googling returns only this Q&A:

It looks like that it is not a reported issue. Also combined with your remark, i.e.:

> It always completes successfully, even with the delay.

I wonder whether it is really an issue as I have seen a lot of logs and also weird messages while everything worked. What I do in such situation is talk with the developer of the code. In this case I would create an issue in the Jenkins bug tracker and discuss whether this is an issue and whether the developers should change the code that triggers this message.

---
Tags: jenkins, jenkins-plugins, amazon-cloudwatch
---


# Question
Title: Reflect GitLab repo changes in VSTS code
I have a repository in GitLab that is used by our development team. On the other hand, there is a VSTS setup (at xyz.visualstudio.com) that we use to manage our scrum board.

As of today, we have to manually import the repository in VSTS every time a feature is finalized. Once the repository is imported, it does not pull the latest commits by itself.

How do I ensure that every time a developer commits code to GitLab repository, same is displayed in VSTS automatically? Am I missing something when I use "import repository" feature in VSTS?

# Answer
I'm using https://docs.gitlab.com/ee/workflow/repository\_mirroring.html#pushing-to-a-remote-repository to push everything into a repository in VSTS, easily done via access token with the right permissions.
> 2 votes

# Answer
> I have a repository in GitLab that is used by our development team. On the other hand, there is a VSTS setup (at xyz.visualstudio.com) that we use to manage our scrum board.

First impression is that you should either choose one repository or check the API of both products to investigate whether the tools could be integrated. From a DevOps perspective, repetitive processes like this that are done manually over and over again should be automated.
> 1 votes

---
Tags: gitlab, azure-devops
---


# Question
Title: kubernetes cluster node with 4cpu can't hold two pods with 2cpu
I have a Kubernetes Cluster on GCE (Google Cloud Engine) (Also called GKE - Google Kubernetes Engine).

The Cluster is configured to have **4cpu** per node. The pods I am using request **2cpu**. I expect to be able to have **2 pods per node**. But instead each time I launch a new pod, it also provisions a new node (am *only* getting **1 pod per node**).

What is happening, How can I more efficiently use my resources? (I don't want to pay for nodes that are under-utilized)

Node:

> Machine type
> n1-highcpu-4 (4 vCPUs, 3.6 GB memory)

Pod (Yaml):

```
resources:
  limits:
    cpu: "2"
    memory: 1000Mi
  requests:
    cpu: "2"
    memory: 1000Mi

```

**EDIT:**

As mentioned below in the comments; it seems there are some 'default' pods that run on the node which utilize some small amount of CPU. For me there were two service pods taking up 100mcpu each. Additionally it seems there is a small overhead regardless of those small pods when viewing my node resources I see:

```
Resource type   Capacity   Allocatable
CPU             4 CPU      3.92 CPU

```

This means that each node in my cluster will only have 3.72 CPU that I can distribute to my pods. (3.92 allocatable, - 100m CPU per service pod (x2) = 3.72)

**Knowing this, I can understand why my desired deployment will not work.**

However, now this raises new questions; How can I most efficiently allocate pods that need 2cpu. Since GCE only lets me provision nodes with even numbers of cpu (2,4,6,etc) I will always have almost 2 wasted CPU. Is there a better solution?

**EDIT2:**

Settings my pods to use 1.8 CPU allowed me to deploy 2 pods per node. (and for now is good enough) It would be desirable to be able to anticipate the overhead on nodes. For example, in my use-case above, allocating 4CPU per node, I should be able to know ahead of time that after the service pods and allocatable CPU are considered I will only be able to use 3.72 for my own pods. But if I now want to provision a new Node with 8CPU I will need to check these numbers all over again.

# Answer
It seems like you found your answer but let me add a little color since we went through the same thing.

Your yaml for the pods can have both a request and limit for both CPU and memory. Since you are discussing CPU here I'll stick with that. What you can do in order to utilize your resources efficiently is set your resource request to the minimum requirement to get scheduled. Say 1.8. But also set the limit to a higher limit, say 2 or more. What this will do is make your pod be schedulable on the machine and if CPU is not completely consumed, your pod will utilize the available CPU up to 2 or the limit you set. This is the way kubernetes can make good use of the resources by "sharing" CPU to the application that needs it.

See here for requests and limits: https://kubernetes.io/docs/concepts/configuration/manage-compute-resources-container/
> 1 votes

---
Tags: kubernetes, google-compute-engine
---


title: Pod object and Pod’s health

text:
# Question
Title: Pod object and Pod’s health
Couple of questions here: a) what the concept of Pod object satisfies? There is the pause container which holds the namespace for the actual ‘app’ container(s) that will get added later and use that namespace. So, what operations in real life qualify to have more than one containers running in a Pod ? Can you provide to the point examples? What are considered sidecar operations after all?

b) what is the threshold for e.g. N containers in the Pod that master will be able to tell that Pod is not healthy? So basically, how many containers in a Pod need to be down to mark the Pod down?

# Answer
the most common example would be some sort of logging sidecar. Depending on your implementation or your service you may want a service that manipulates the files for the pod etc. Some databases can use sidecars for log shipping in replication, or you might see some process to clean up log files as well. Although available, those options are not suggested specifically because there are more alternatives.
> 1 votes

---
Tags: kubernetes
---


# Question
Title: Adding a persistent volume to a Kubernetes deployment from Gitlab Auto Dev-ops
Recently, GitLab partnered up with Google Cloud to provide automatic deployment to Kubernetes clusters directly from project repos.

I enabled Auto Dev-Ops on my project, which just contains some Node.js and a Dockerfile, and as if by magic, I have a working deployment (after figuring out that I needed to make sure the exposed port was 5000 rather than anything else!)

But now I need to add some persistent storage to this. I'm aware of how to do this with pure Kubernetes, but I'm struggling to find out how to do this as part of my continuous deployment.

GitLab uses this script for Auto Dev-Ops, and I presume I might need to fork this to make my own `.gitlab-ci.yml` to add the custom volume, but I just cannot see anywhere in which this could be done.

The only other option I can work out is to use the provided PostgreSQL installation which *does* have a persistent volume to store my data, but this would just be skirting around the basic issue.

Do I need a custom buildpack? Do I just do it via the Google Cloud Dashboard?

# Answer
> 2 votes
The answer is that all Auto Dev-Ops deployments at present use Helm to package and deploy your application.

In order to make changes to the deployment, such as adding a persistent volume claim, you should download and modify the Helm chart that GitLab uses.

The files in this repo should go into a new directory `chart/`, or you can link to another location by setting the `AUTO_DEVOPS_CHART` project variable to a URL.

This page has more details.

---
Tags: docker, kubernetes, gitlab, gitlab-ci-runner
---


# Question
Title: Jenkins shows "No changes in any of the builds."
# Description

Jenkins job was made and configured to build release branch which already existed and had changes on it before first build. Job also monitors for push on BitBucket and starts building if something is pushed to the release branch.

Problem is we didn't made any changes on release branch, instead we were rebasing it on development and pushing it which triggered builds but it shows *"No changes in any of the builds."* in **Changes** menu.

# Question

Is there special configuration for Jenkins which allows it to handle rebase commits and show changes that occurred to that commit comparing to previous build?

# Answer
> 4 votes
# Temporary workaround

Instead of calculating changes between builds compare changes against specific branch.

1. Open job configuration
2. Go to **Source Code Management** -\> **Git** -\> **Additional Behaviours**
3. Add **Calculate changelog against a specific branch**
4. Enter name of repository (origin in my case), and branch (master in my case)

Now instead of showing only new changes, each build will show all commits in release branch that are different comparing to master.

---
Tags: jenkins, git, bitbucket
---


# Question
Title: Passing a jenkins credential (global) to a python script
I have a global jenkins credential and a config script that includes the line

```
API_TOKEN = "Token XXX"

```

For security reasons we can't keep the token in the script itself as its in version control. Is there a way to pass the jenkins credential to the python config script from jenkins?

# Answer
Specifically for credentials, CloudBees recommends using the Credentials Plugins, which inject the credentials in the environment available to the invoked scripts. From Injecting Secrets into Jenkins Build Jobs:

> * Credentials plugin \- provides a centralized way to define credentials that can be used by your Jenkins instance, plugins and build jobs.
> * Credentials Binding plugin \- allows you to configure your build jobs to inject credentials as environment variables.
> * Plain Credentials plugin \- a plugin dependency required by the Credentials Binding plugin.

Inside your script you'd pick up the variable from the script's environment:

```
API_TOKEN = os.getenv('API_TOKEN')

```

In a similar way you can pass any arbitrary variables (other than credentials), using the EnvInject Plugin (watch for the warnings, tho). See related:

Another possible approach is to configure Jenkins to pass the value as an argument to your script. Inside your script you'd pick up the value using your preferred method of processing the script arguments.
> 6 votes

# Answer
If you are using Pipeline (i.e. Jenkinsfile), use `withCredentials`. To give a quick and dirty example:

```
withCredentials([[
  $class: 'UsernamePasswordMultiBinding',
  credentialsId: 'my-credentials',
  usernameVariable: 'user',
  passwordVariable: 'pw',
]]) {
  sh("./my-python-script.py --user ${user} --password ${pw}")
}

```

Your credentials will be automatically redacted from the console output to help prevent sensitive information from leaking.
> 4 votes

---
Tags: jenkins, python
---


title: Mysql with Kubernetes

text:
# Question
Title: Mysql with Kubernetes
I am converting a webapp from monolithic to kubernetes based architecture. But how can mysql be persistent with data when we have 2 mysql pods in each of 2 differnet nodes? I think we can mount a volume but still if my server with the mounted volume fails, this will be a single point of failure. How can I avoid this? I want to acheive no single point of failure using kubernetes without cloud. How can I acheive this?

# Answer
I would suggest to only run stateless apps in k8s. One should run statefull apps like databases in dedicated database solutions like AWS RDS.
> 2 votes

---
Tags: docker, kubernetes, microservices, mysql, high-availability
---


# Question
Title: Problem Restore EF Core 2.1.1 in VSTS, Hosted VS2017
I have a project with `Asp.Net Core 2.1.1`, so I have to force the runtime to 2.1.1, when I added the `EF Core 2.1.1` packages (`SqlServer, Design and Tools`). Locally the project performs well, however in the process of Build in `VSTS` using the `Hosted VS2017` the build process fails, it does not restore the packages by dependency conflict. I installed the packages he asks for:

`Microsoft.EntityFrameworkCore` After New Failure `Microsoft.EntityFrameworkCore.Abstractions` and then `Microsoft.EntityFrameworkCore.Analyzers` ...

This is the way? Go running and failing build a build until you discover all the packages that need to be placed as dependencies? Anyone have any tips to solve this another way?

My build process is like the .net core of VSTS.

This is my .csproj file

```
<Project Sdk="Microsoft.NET.Sdk.Web">
  <PropertyGroup>
    <TargetFramework>netcoreapp2.1</TargetFramework>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <RuntimeFrameworkVersion>2.1.1</RuntimeFrameworkVersion>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.AspNetCore.App" />
    <PackageReference Include="Microsoft.EntityFrameworkCore" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Abstractions" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Analyzers" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Design" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.SqlServer" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools" Version="2.1.1" />
    <PackageReference Include="Microsoft.EntityFrameworkCore.Tools.DotNet" Version="2.0.3" />
  </ItemGroup>
</Project>

```

# Answer
I also could not restore packages after upgrading to 2.1.1. I got multiple errors similar to this:

```
Detected package version outside of dependency constraint: Microsoft.AspNetCore.App 2.1.0 requires Microsoft.EntityFrameworkCore (= 2.1.0) but version Microsoft.EntityFrameworkCore 2.1.1 was resolved.

```

I solved my problem by explicitly specifying version 2.1.301 in the build definition.
> 0 votes

---
Tags: azure-devops, azure-pipelines, nuget
---


# Question
Title: What tools should I deploy to support AWS for a Managed Services company?
I work for a managed services company and am working out a plan to retrofit our customer support machines to support AWS in a general way.

We use Ansible and lots of ssh port forwarding to connect to traditional, non-cloud, boxes (Windows, Linux, AIX, Firewalls, routers, UC, other Cisco stuff).

Beyond the AWS CLI, what tools should I deploy to my Linux and Windows support boxes for managing, monitoring and troubleshooting AWS that are different than the traditional tools?

# Answer
Installing a box is probably not the best solution, mostly because there's no real need for it.

You should ask for an access to AWS console in customer account with proper rights fitting your responsibilities and maybe setup AWS config, Cloudwatch for the monitoring and alerting part, Cloudtrail and maybe Amazon Inspector for the audit part.

To ensure machines and AWS services themselves are properly configured you may rely on AWS Config/Amazon Inspector or tools like inspec (and here you may or may not need a bastion host on your customer VPC to check other machines, this mostly depends on the agreement you have with customer about direct links/VPN)
> 1 votes

---
Tags: amazon-web-services, toolchain
---


# Question
Title: How to update frontend servers behind a load balancer
I have multiple frontend servers churning out the HTML pages , behind a load balancer.

Just wondering what is the best practice to update the frontend app while users are surfing the site.

The objectives:

1. ensure each user see either the new or old version, but not flip between both.
2. ensure load is still evenly distributed amongst servers that are not being updated.
3. no disruption to service.

My thoughts:

1. Divide the servers into groups.
2. Take out a group from load balancing and wait for it to finish off current requests.
3. Update the app and OS, etc.
4. put the group back to load balancing.
5. Pick another group and repeat steps 2-5.

While in process, some servers will have the new version, so to ensure users do not flip between the old and new ones, do I enable sticky session?

# Answer
> 1 votes
What you have above seems like a reasonable update procedure, however you can also use persistence or a sticky session so that when a user goes to a web server, they will be routed to the same web server for the life of their browser session (or for the next hour, or whatever you decide to configure). This also allows for some A/B testing if you need to get user feedback on a new feature or just allow for the possibility of a bug (just pull server group B from the pool if an issue is discovered during A/B testing with that group).

There are some concerns you should be aware of with sticky sessions: Eg, source address persistence can be problematic if you have a thousand users behind a firewall - then you have a huge group that always goes to a single web node. Maybe you really don't want a cookie to do persistence, etc.

But, there really is no agreed "best practice" on this - the best practice is going to the the one that meets your business need in this case. You should just collect all of your options and choose the workflow that best meets your needs.

---
Tags: deployment, load-balance
---


title: Make instance with newuser by terraform

text:
# Question
Title: Make instance with newuser by terraform
I want to make new instance with "user1" rather than ec2-user(AWS) default.i want achieve this by terraform. Like in azure terraform it can be done by "os\_profile{}", but how can i do in aws by terraform scripts?

# Answer
> 2 votes
AWS AMIs come with a default username - which username it is depends on the specific AMI that is launched. Amazon Linux AMIs use `ec2-user`, CentOS uses `centos`, and there's other combinations too.

If you want to create your own user on instance launch, you can use EC2 User Data for this, which essentially entails passing a custom script that will be run at launch time.

The `user_data` argument is supported by Terraform's `aws_instance` resource. You can specify a script inline, but for most cases you'll probably want to use an external file or template.

For example:

**create-user.sh**:

```
#!/usr/bin/env bash

useradd mytestuser

# Anything else you want to do to set up your user account

```

**main.tf**:

```
resource "aws_instance" "main" {
  ...
  user_data = "${file("create-user.sh")}"
}

```

Alternatively, there's an example on rendering a template here, which is useful if you want to pass through custom values rather than just provide a static file.

Or, as an alternative to writing your own shell script, you could try rendering a cloudinit config file using `template_cloudinit_config`. Some resources that might be useful for this include this blog post and this part of the cloudinit documentation.

---
Tags: amazon-ec2, terraform
---