# Question
Title: In the HashiCorp stack, where's the appropriate place to add users?
I'm in the process of building some custom Linux images using HashiCorp's Packer, which will later be deployed to Azure using HashiCorp's Terraform. The VMs created from these images will need to have a set of users created; in this particular case I'm concerning myself with user accounts for employees that may need to ssh into these VMs.

For this kind of configuration, does it make more sense to add these user accounts to the base image in the Packer script, or to add them when the VM is created via Terraform? It seems to me that handling it in Packer makes more sense, but are there reasons not to do it there?

# Answer
I am less familiar with what Azure offers, but other cloud providers offer a way to manage users external to the VMs the provide (normally IAM permissions).

I would not bake user into the system but look for an external way to manage them. This allows you to more easily remove users that leave your company or add new users that join.

If Azure doesn't provide a way I would look at setting up Active Directory or LDAP to manage user permissions to hosts. But I would be shocked if you needed to do this.

**Note**: Don't confuse infrastructure and data. Users are data, not infrastructure. Data (ie adding a user) should be able to change without changing the infrastructure. If you need to remove a user you do not want to rebuild your whole infrastructure.
> 3 votes

# Answer
I would agree with previous statements: avoid baking users into your base image with Packer. Likewise, you should avoid baking them with `Terraform` as well. Terraform does not provide a mechanism to perform *on-going* changes after the server is provisioned. It only has those *initial* provisioning scripts that get run only one time. If you manage users with Terraform provisioning scripts, those users will never get updated again by Terraform. You might as well be baking these changes into Packer at that point.

I would honestly suggest you venture outside the Hashicorp ecosystem for this task. For *on-going* changes of machines, this is exactly what a configuration management system is designed for. Something like Anisble, Chef, Puppet, or Salt, *real* configuration management tools, are best suited for this job.

Packer lets you set a base image to be used by many machines. Terraform maintains the state of your architecture as code. However, what happens when you on-board or off-board new users? Or you need to change their individual SSH keys because they lost their computer? This should not be a code change. Nor would you want something like User secrets stored as code in your Terraform provisioning script.

At my company the stack is: Packer images, Terraform bootstrapping, and Chef management. They work well together. I'll admit, I love putting more things into Packer and Teraform as it makes spin up times quicker. But a good CM tool still has its place in the DevOps world.
> 3 votes

# Answer
I avoid "baking" user accounts into my OS image with Packer *unless* there is an OS specific need for it. For example, a service or system account that is needed across all of my deployments.

My suggestion is to use a Terraform provisioner to create the user accounts. That way you can change them when deploying the VMs via Terraform without having to create a new Packer image. This also allows you to reuse the same Packer image for different purposes that may have different user accounts.

Ultimately it depends on how your workflow is set up. Do you want to make those changes on the VM image or during the deployment?
> 2 votes

# Answer
I agree with others not to bake user accounts into the OS images. Especially **personal** user accounts of your infrastructure team should be maintained with a decent user management. There are only very rare circumstances where you want to system/service account to be installed everywhere.

Reason:

Getting users into the systems is most times easy because people ask for it. Reliably cleaning up obsolete user accounts is the much harder problem because the employees are gone.

Often I get asked by customers whether it's possible with an IAM / login solution to deactivate user accounts everywhere without any delay. So even if you plan to mitigate the user removal issue by defining a short life-time of your VMs it will probably not be short enough for dealing with urgent cases.
> 1 votes

# Answer
I wouldn't introduce another infrastructure tool/CM (like Ansible) just for this purpose, instead of that, I would consider these two options:

1. If your instance lifecycle is short enough, like maybe you replace instances often with recently built images with Packer, you can make use of "user data" of your EC2 instances or launch configurations on your Terraform configuration to describe your users. Next time your instances are launched, your changes will be applied. I think this is the simplest option for a small company, yet might not be suitable for all environments for security or lifecycle reasons.
2. You can make use of an external authentication source like IAM (see: https://github.com/widdix/aws-ec2-ssh) or LDAP.
> 0 votes

# Answer
Just to add to the existing answers a bit, for "system" users, or users that applications run as, packer would be a fine option, as would using a Terraform provision, or Ansible directly.

For human users, I'd **highly** suggest using something like FreeIPA. FreeIPA does all the heavy lifting in regards to setting up LDAP, Kerberos, sudo rules, general access rights, all with a shiny web interface.

I mean, do you really want to rebuild the servers if someone wants to change their password?
> 0 votes

---
Tags: terraform, linux, azure, packer
---


title: Jenkins Job vs Jenkins Pipeline

text:
# Question
Title: Jenkins Job vs Jenkins Pipeline
**BACKGROUND :** We have one Selenium automation project which is shared in BitBucket repository. Project is maven project. we have 10 `testng.xml` files for all modules

**OVERVIEW:** Now we want to trigger the automation build once any changes committed in AUT application code. Also want to trigger the build for failed testcase in Post Step as well, So we will be sure with actual test failure. Once this execution done then it should send the email with report attachment.

**PROBLEM**: We are unable to decide What would be feasible Jenkins Job or Jenkins Pipeline ?

**TRIALS :** I have idea regarding Jenkins Job like setting up maven job and configuring post build trigger, email configuration but not clear enough as a layman.

# Answer
*Jenkins Job* and *Jenkins Pipeline* are basically the same. In a pipeline you define the steps of your job as groovy code (actually it is CPS https://github.com/jenkinsci/workflow-cps-plugin, but that should in general just be a custom groovy interpreter).

The point, that is making pipelines "better", form my perspective, is, that you can add those in so-called `Jenkinsfile`s alongside your code. So you have your build job versionized alongside your application code.

So I would recommend: Take the time to learn how these Jenkins pipelines work and use those. But if you do not want or if you do not have the time, then doing it with plain-old Jenkins jobs is fine as well.
> 6 votes

---
Tags: jenkins, continuous-integration, jenkins-pipeline, automation, selenium
---


# Question
Title: Where to obtain the token to register a GitLab runner?
I have a GitLab repository. I want to add a runner to it. I was able to install it, with the instruction here without Docker (step 3).

Now I have to register it according to the instructions here. In the beggining of that file it is wrriten that I have to obtain a token, but I don’t understand where do I obtain the token from.

In this link it is wrriten that I can get the token from `CI/CD -> settings` but `CI/CD` does not have a `settings` option.

Also, in the registration instruction in step 2, it is written: `Please enter the gitlab-ci coordinator URL` but there is no documentation anywhere about what is this URL

# Answer
For a specific GitLab runner tied to a project, the token can be found in Step 3; in `Project -> Settings -> CI/CD -> Runners` in the `Setup a specific Runner manually` section.

You will also be able to find the URL which you need to use to register the runner in Step 2.
> 2 votes

---
Tags: continuous-integration, gitlab, gitlab-ci-runner
---


# Question
Title: Running unmanaged Hardware Security Modules (HSMs) in the Cloud
I have to admit to never having asked, or been asked, the question if it is possible to have a Hardware Security Module in a public cloud, by which I mean Google, Amazon or Azure. *Has anyone found any techniques for enabling organizations to use HSMs that they fully manage?*

It seems to me that the two concepts, Cloud and HSMs, are fundamentally at odds with each other - because cloud generally involves "outsourcing" or transferring the risk of operating hardware to the cloud service provider.

There is clearly a middle ground in terms of fully managed HSMs as you find in Azure and AWS:

* Azure KeyVault: Use Key Vault and you don’t need to provision, configure, patch and maintain HSMs and key management software. Provision new vaults and keys (or import keys from your own HSMs) in minutes and centrally manage keys, secrets and policies.
* AWS CloudHSM: AWS CloudHSM is a cloud-based hardware security module (HSM) that enables you to easily generate and use your own encryption keys on the AWS Cloud. With CloudHSM, you can manage your own encryption keys using FIPS 140-2 Level 3 validated HSMs.

In addition there are some non-HSM based solutions to key management:

* Cloud Key Management (Google): Cloud KMS is a cloud-hosted key management service that lets you manage encryption for your cloud services the same way you do on-premises. You can generate, use, rotate and destroy AES256 encryption keys. Cloud KMS is integrated with IAM and Cloud Audit Logging so that you can manage permissions on individual keys, and monitor how these are used. Use Cloud KMS to protect secrets and other sensitive data which you need to store in Google Cloud Platform.
* Various Security Appliances available in all of the cloud marketplaces.

Has anyone found any techniques for enabling organizations to use HSMs that they fully manage?

# Answer
> 1 votes
So having gone backwards and forward over this for a couple of weeks, Azure has confirmed to me in-person that the only way to utilise FIPS-140 Level 2 certified hardware security modules in Microsoft Azure is to use Azure Key Vault.

# Answer
> 1 votes
Richard, you are right that Cloud and HSMs are two contradictory concepts.

To fulfill availability and elasticity requirements for key management and cryptographic operations a middleware is needed controlling all the hardware. This is basically done by the cloud KMS available now.

With AWS Cloud HSM, there is no fully managed availability and elasticity any longer. Instead, you can add HSMs to the cluster if needed (scalability) and if they are located in different availability zones you achieve a certain level of availability as well (but there is no guaranteed SLA as opposed to KMS). Moreover, AWS is doing backups. However, you have to trust AWS and Cavium that they never collaborate and calculate the Backup Encryption Key from the AKBK and MKBK (see page 10 of the AWS whitepaper) outside of the HSM.

When you really want to have full control over the HSM, you have to host the HSM yourself and connect it to the cloud using VPN. However, depending on the Internet connection of your data center you might suffer from low latency and your external Internet connection could be vulnerable to DoS attacks. Therefore, solutions like Utimaco CryptoServer Cloud include private connections from their data centers to your cloud. Depending on the location of the data center and the cloud region, these connections can have very low latency. Since the HSMs do not belong to any cloud service provider you are able to switch CSPs more easily since you don't have to move keys (which can be quite hard to impossible with KMS). Moreover, multi-cloud scenarios are possible. On the other hand, you have to bear all the management tasks like backups or firmware upgrades. Also, to achieve high availability you have to rent at least two of the HSMs, and scaling granularity is a full HSM only. But that's the price for being very secure by having full control over the HSM.

# Answer
> 0 votes
I just wanted to point to you that there are solutions out there to benefit from HSM aaS dedicated. We are piloting this at Interxion (colocation provider Europe) : https://www.interxion.com/products/key-guardian/

---
Tags: security, cloud, key-management
---


# Question
Title: AWS Security Group Creation Using AWS CLI
I have created a security group in AWS. I have added the ingress rule using the below command

```
aws ec2 authorize-security-group-ingress --group-id sg-07xxxx4d --ip-protocol tcp --from-port 22 --to-port 22 --cidr-ip 0.0.0.0/0

```

But instead of IP, if i want to give the source as another existing security group, how can i do that?

# Answer
Ok I got the answer after exploring few things and looking at the error more attentively. Here is the way:

```
aws ec2 authorize-security-group-ingress --group-id sg-07xxxx24d  --protocol tcp --port 22 --source-group sg-d7dyyyy5

```
> 2 votes

---
Tags: amazon-web-services, aws-cli
---


# Question
Title: What is the difference between modules and workspaces in Terraform
I currently use Terraform workspaces to configure different environments like dev, test, prod, and my code works fine. And I believe I can use workspaces to configure different regions as well.

However, I think Terraform modules can do the same thing, right?

If that's so, when should I use workspaces and when should I use modules? Or they are doing a similar thing in different ways?

# Answer
> 4 votes
I think a key difference between Terraform Modules and Workspaces is that modules can be inherited and used by other modules and configurations. Terraform Workspaces are intended to be a collection of configurations that represent a single environment, whereas Modules are components that can be utilized by one or more modules/configurations. I think you could use Modules in a similar way to how you are using Workspaces, but you would be contradicting the intent of the Modules.

For example, say that you want to create a base configuration file for all your company's EC2 instances that allows traffic over SSH. You would create a module that spins up a VM, modifies your firewall to allow SSH, and any other configuration settings that you would want all of your instances to have. You would then have every VM Terraform config inherit this file. This would minimize the amount of code that you would have to write and allow for any changes that you would want across all of your TF configs to be made in one central module.

# Answer
> 4 votes
Terraform modules are a way to encapsulate reusable business logic in order to be DRY. For example, you might use a module to define a jenkins setup, then invoke that module every time you want a jenkins server. In other words, you can instantiate modules as many times as you need them to achieve the same outcome.

On the other hand, Terraform workspaces are a way to isolate terraform state between environments. For example, you don't want to mingle state between a production and staging environments.

In practice, we *highly* recommend writing modules for *everything*. This is why we have over 100+ modules. Writing modules is like writing "functions" in other languages. Generally, it's considered a bad practice to stick all logic in a single function or scope; modules help us avoid that.

Workspaces are nice in principle, but in practice we recommend to share *nothing* between production and staging environments, including terraform state. Workspaces violate this principle. Thus we architect things so that production state remains in the production account and staging state in the staging account. We accomplish this with our `terraform-aws-tfstate-backend` module.

———

For the record, “we” refers to Cloud Posse, a DevOps professional services company. I am the founder. The views expressed are our own and may not represent the views of the community at large. :)

---
Tags: terraform, terraform-module, terraform-workspace
---


# Question
Title: How can I get terraforms extern to execute `ssh-keygen -y -f ~/.ssh/id_rsa`?
I seem to be confused about how external works. I tried:

```
data "external" "local_key" {
  program = [
    "ssh-keygen", "-y", "-f ~/.ssh/id_rsa"
  ]
}

```

This gives me:

> failed to execute "ssh-keygen": ~/.ssh/id\_rsa: No such file or directory

Which presumably happens because `~` expansion doesn't. `ssh-keygen -y -f ~/.ssh/id_rsa` works normally. So instead I tried invoking bash like:

```
data "external" "local_key" {
  program = [
    "bash"
  ]

  query {
    "-c" = "ssh-keygen -y -f ~/.ssh/id_rsa"
  }
}

```

I'm still getting the same issue. I realize I can just pass the output as a var to terraform from the outside, but I'm still curious what the solution is.

# Answer
> 4 votes
The best way to achieve what you're trying to do (generate an openssh public key from an `id_rsa` private key) is using the `tls_public_key` data provider built-in to terraform. Using the `external` provider should be considered an option of last resort because it introduces dependencies on the local OS which may not be portable. It's like an escape hatch.

You can use it like this:

```
data "tls_public_key" "example" {
  private_key_pem = "${file("~/.ssh/id_rsa")}"
}

```

Then to get the public key suitable for openssh, you can access

```
"${data.tls_public_key.example.public_key_openssh}"

```

Lastly, if you are open to generating a new SSH key pair, you can use the `terraform-tls-ssh-key-pair` module which will write the keys to a new file and exposes a `public_key` output with the openssh public key.

# Answer
> 6 votes
So what happens in first case is, as Dan's already said, there's no shell used and as such nothing to expand the `~`. Quoting the documentation about program:

> Terraform does not execute the program through a shell

On the second case, bash receive in stdin something like this:

```
{ "-c": "ssh-keygen -y -f ~/.ssh/id_rsa" }

```

And this looks like a command block for bash, but -c is not a valid command.

What could work could be this kind of program (assuming no specific input):

```
jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{"pubkey":$pubkey}'

```

So something like this should work to get the key in `local_key["pubkey"]` if I understand the documentation properly:

```
data "external" "local_key" {
  program = [
    "bash", "-c jq -n --arg pubkey \"$(ssh-keygen -y -f ~/.ssh/id_rsa)\" '{\"pubkey\":$pubkey}'"
  ]
}

```

There's a need to use bash for a one liner to take advantage of command subsitution. You can also do a .sh script like:

```
#!/bin/sh
jq -n --arg pubkey "$(ssh-keygen -y -f ~/.ssh/id_rsa)" '{\"pubkey\":$pubkey}'

```

And call this script in the `program` parameter.

# Answer
> 1 votes
Note: this answer is based solely on documentation, I didn't try it.

From External Data Source (emphasis mine):

> The following arguments are supported:
>
> * program \- (Required) A list of strings, whose first element is the program to run and whose subsequent elements are optional command line arguments to the program. **Terraform does not execute the program through a shell**, so it is not necessary to escape shell metacharacters nor add quotes around arguments containing spaces.

Expanding `~` to the home directory is a capability of the shell, not of `ssh-keygen` and since in the 1st example `ssh-keygen` is invoked, it literally attempts to open a file named `~/.ssh/id_rsa`. Which fails because a `~` directory doesn't exist (I'm not sure if that's what you mean by `presumably happens because ~ expansion doesn't`).

In the 2nd example you're passing the `ssh-keygen` command as a query, not as arguments to `bash`. Probably what's executed is not what you're expecting.

I'd try:

```
data "external" "local_key" {
  program = [
    "bash", "-c", "ssh-keygen", "-y", "-f ~/.ssh/id_rsa"
  ]
}

```

But I'm not sure if `bash` itself (or `ssh-keygen` for that matter) follows the External Program Protocol required (also on the above referenced page). You may need to write your own script to wrap the cmd you desire while also providing the data according to the protocol. Invoke that script as the program instead of `bash` or `ssh-keygen`.

Finally, the entire external data source page doesn't appear to suggest that `export` would be designed for interactive programs and `ssh-keygen -y -f ~/.ssh/id_rsa` asks for the user passphrase interactively. YMMV.

# Answer
> 0 votes
Have you considered other options? e.g. A discussion my peers and I engaged recently was "time and place". We use mostly Ansible for deploying servers. Yet some of our team wanted to use Terraform and others argued Cloud Formation. Our discussion concluded why not let Terraform do what it does best and leave the rest to Ansible? In that within the Terraform user\_data section you can call your Ansible Playbooks. My '**limited**' understanding of Terraform is that its purpose is to deploy services to a limited extent and not be a full on automation drop-in replacement for services like Ansible, Chef, Puppet, etc. As a side note, Ansible has a module to handle the ssh keys generation and plenty more.

---
Tags: terraform
---


# Question
Title: pod to pod can not be reached through service clusterIP running in the same minion
few days ago, i am facing a strange issue which is "a pod can not reach another pod running in the same minion/host through servide ClusterIP whereas they can reach each other skipping going through service clusterIP"

i am running kubernetes 1.9.6 on top of openstack (ubuntu 16.04 hosts) with:

* cni 0.7.0
* cri-containerd 1.0.0-beta
* flannel 0.9.1
* etcd 3.0.13

the other scenarios were working fine. even between pods running in different hosts, they can reach each other through service clusterIP.

what is going on here !!! any support will be more than welcome

# Answer
The response from the called service is not DNAT'ed back to set src to the service clusterIP. So the response packet is going directly via cni0 bridge and not getting into iptables, preserving pod's IP.

to fix this i added the following option to kube-proxy and issue was gone.

--masquerade-all=true
> 0 votes

---
Tags: kubernetes, networking
---


# Question
Title: How to implement the Load Balancing in Openshift ?
I have deployed my application on cloud using Openshift. I tried few blogs but could not find out the best ways to implement the Load Balancing . Please help.

# Answer
The simplest way is to find the icon that scales the application up and click it. In the current UI it's in Project --\> Overview --\> expand the application info ("\>" icon to the left), --\> click "^" next to the blue circle to the right.

This gives you a default load-balancing across as many PODs (instances) of the application as your resources and imagination allow.

From the command line, this would scale it to 3 PODs:

```
oc scale --replicas=3 rc yourapplicationname

```

Another way is to deploy the same application with a different name, and then edit the route (Applications --\> Routes --\> RouteName) that points to it. There should be an option:

> Alternate Services / Split traffic across multiple services

This gives you control over how much traffic you want through each version. Good for A/B testing or canary releases.
> 1 votes

---
Tags: docker, openshift, loadbalancing
---


title: Initialize database with data

text:
# Question
Title: Initialize database with data
I'm setting up Build Automation for a MS-SQL project. One of the requirements is to initialize the database with data. I have them as SQL scripts. I'm trying to figure out what the best way to inizialize a database after it is deployed using VSTS build and release automation.

# Answer
Depends on where the source data is when considering how you want to initialise the database.

1. If the data pre-exists in another SQL-Azure Database you could create a BACPAC and export the file to a blob store in Azure. Then using PowerShell in your pipeline import the data into your created database. Bare in mind this will be time costly dependent on the amount of data and the size of the database.
2. If the data is only small, an approach we have used before to build and populate tables is a DACPAC. Here we had the create scripts for our tables and then stored procedures with static reference data stored in insert statements which were run after the DAC built the tables & indexes etc. Here is an example in how to create a DACPAC project in Visual Studio and populate from an exisiting database. Alternatively if you're building from scratch you can add the required database objects to the solution and create them using SQL Statements.
3. The third option would be streaming data from a source using an Azure Function and blob storage however without knowing more about your set up I can't advise more.

Hope that helps
> 1 votes

---
Tags: azure-devops
---


# Question
Title: Are Route 53 DNS records aliasing ELBs guaranteed for their lifetime?
Given a Route 53 DNS record aliasing a (classic) load balancer, and the potential for the load balancer ip to change:

Is it possible for the DNS record to guarantee that the provided ips are valid for the record's lifetime? If so, is there a specific TTL value I need to set?

# Answer
> 2 votes
If you’re talking about using a Route 53 ALIAS record, you cannot set the TTL on it, for this reason. It defaults to a low value, and is managed by AWS for you (to the world it appears as an A record, and it will change often).

If you’re using a CNAME, then you can set whatever TTL you like. The TTL of the DNS record for the load balancer’s domain will be low, but you don’t need to worry about that. Your only concern in this circumstance is whether you think you’ll be changing the load balancer name or repointing the name elsewhere. If you don’t plan to do that, you can set a high TTL to save a bit of money and milliseconds at load time.

Either way, the IP addresses themselves will change.

For help choosing between an ALIAS and CNAME record, see https://docs.aws.amazon.com/Route53/latest/DeveloperGuide/resource-record-sets-choosing-alias-non-alias.html. I’d usually recommend a CNAME unless you’re pointing your root domain, in which case you have to use an ALIAS.

# Answer
> 0 votes
No. The ELB IPs may always change without notice. The only way to get a stable IP is an EIP and do the proxy yourself, or their new NLBs.

If you mean can you set a DNS entry going straight to the IPs yourself, no (well you can, but don't). Amazon maintains the DNS entry they put atop it, but if you try to set your own with its own TTL you can absolutely have a broken IP for however long your TTL is. CNAME to their provided DNS name as necessary.

---
Tags: amazon-web-services, dns, aws-elb
---


# Question
Title: Why don't the instances in the auto-scaling group get assigned public ip addresses unless it's a default vpc?
I'm trying to migrate our deployment process over to aws code deploy (by following instructions here:https://pionect.nl/autoscaling-aws-laravel-codedeploy-opsworks-chef-automation/).

When I'm creating the auto-scaling group, I selected the same vpc as my other instances belong to. However when I do that, it says that it the ec2 instance will not be assigned a public ip. How come? My other instances have the same vpc id and are assigned an ip address just fine.

# Answer
I got the answer from AWS support:

The reason why you are seeing that notification is because it is in accordance to your launch configuration (staging-1-deploy) settings. During the creation of a launch configuration, the assignment of public IP addresses to EC2 instances defaults to `Only assign a public IP address to instances launched in the default VPC and subnet. (default)` \[1\]. To support automatic assignment of public IP addresses to your EC2 instances as they are launched, you will have to change the launch configuration settings. Since a launch configuration cannot be edited \[2\], you can create a new one by copying your current launch configuration. Follow the steps below to achieve this:

1. Go to your AWS console, select `Launch Configurations`, select `staging-1-codedeploy`
2. At the bottom of the details, click the grey `Copy Configuration` button on the right.
3. Click`Edit Details` link, then click `Advanced Details` on the resulting page.
4. Go to `IP Address Type` and select the second option, `Assign a public IP address to every instance.`
5. \[optional\] Change the name of the launch configuration if need be and
6. Save launch configuration

You can now proceed to use your new launch configuration for your auto-scaling group and all your EC2 instances will be assigned public IP addresses as desired. I have attached a few screenshots to show some of the steps described above.
> 1 votes

---
Tags: amazon-ec2, aws-vpc, aws-codedeploy
---


# Question
Title: How to update pods if version hasn't changed?
During development we are continually building a Docker image with the same version (say 0.0.1-development) as there is no need to bump for each minor change. We do eventually bump when additional features are added.

We've discovered that `kubectl set image deployment/ourdeployment` doesn't force the updated container to be pulled from Dockerhub and run if the version in the image hasn't changed.

In the deployment we have `imagePullPolicy: Always` but this doens't seem to have any effect.

Is there a way we can achieve the desried behaviour of pulling down the image even if the version hasn't bumped, replacing the old containers each time?

# Answer
The main problem here is, that from a Kubernetes point-of-view nothing has changed to your deployment. So nothing gets updated.

The best solution here would be, that you add something to your image tag, like the git short hash, for example. The your update will just work. Also you will make your deployments re-creatable, when you do it this way.

If you really want to keep it the way it is, you could add some other information to the deployment, that changes on each deployment, but is not used, like an annotation. But I would not recommend that, since I saw this leading to strange behavior with *old* pods no being shutdown.

On a side note: The `imagePullPolicy` is just evaluated, when there *is* change to your deployment. So basically this flag says: If you do deploy something (e.g. bc there is a new deployment), no matter if you have the image already locally, reload it from the registry.
> 1 votes

---
Tags: docker, kubernetes, deployment
---


# Question
Title: Git - Everyone Pushing to Same Branch in Different Files
Database team is implementing code changes, using Visual Studio SSDT database projects with Git source control. Everyone is pushing to main Release branch with code review (only 5 developers on the team). All the database coworkers are only allowed pushing to different files only (tables, sprocs, functions), etc. The way work is assigned, none of us push or work the same sql file. Eventually all good changes from Release (currently in Work) are merged into Master branch (Production Ready).

Code Review ---\> Push to Release Branch (Currently in work during Sprint) ---\> Merge to Master Production Ready Branch

(a) What are the negative consequences of utilizing this strategy in Git?

(b) For cleaner history, should everyone Rebase ReleasePublic Remote into ReleaseLocal, or conducting Pull? (Fetch/Merge), I would think Rebase is answer for cleaner history.

Note: I agree, it would be annoying only if we are working on the same file and pushing changes. Alternative Strategy to create different feature branches and then merge into main branch. We are refraining from this strategy since each developer has 10 dba admin related changes a day, creating many branches and merges is time consuming and cumbersome-

https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow

# Answer
This is not a "strategy". This is a sure way toward chaos.

The idea behind "trunk branch", which means having one branch for integration, does not mean that everyone should be pushing their changes to this branch directly. It only means that his branch is the place where integration happens.

Why do you even need two branches, `Release` and `Master`? What benefit does this bring? Probably just the benefit of confusing git history into an incomprehensible mess so that no one knows what is happening.

Stop this madness!

* Make each change on its own pull-request branch (call it feature branches if you like).
* Make sure that these pull-requests are not merged into the main integration branch before the code there goes through some minimal code-review.
* Restrict from pushing directly to your integration branch.
* For cleaner history, make sure that the pull-request branch if first rebased on top of integration.
* Make sure that merges of pull-requests are not fast-forwarded and clearly visible in history.

The above will allow for several benefits,

* Higher quality and less "human error" mistakes.
* Learning and teaching each other cool dba tricks (during code reviews).
* Keeping the integration branch **green** most of the time (it works!).
* High trust that you are not shooting yourself in the leg using your process.
* Way less "oops, I overwrote your change" kind of merges.
* High trust that new team members are not going to mess things up.
* Git history that is a thing of beauty and a joy forever.

And more ...

Prettier explanation of the "Feature Branch Workflow" is available on the Atlassian Git website here https://www.atlassian.com/git/tutorials/comparing-workflows/feature-branch-workflow
> 5 votes

# Answer
I believe there are several anti-patterns in your workflow.

**1.** Having different developpers pushing to different files to avoid conflicts is to hard of a constraint to be sustainable in the long run.
**Consequences:**

* you will not be able to propose coherent tasks to your devs for long
* there will certainly be complexity in the code introduced merely because of the process and not because of business requirements

**2.** Late merging to `master` branch
**Consequences:**

* you will discover merge issues, technical and business bugs late
* chances are you may not be able to keep your schedule

Your development process should be **business** oriented, not **tech** oriented. You should ask yourself: *How can we bring value to users faster?*

The answer to this question is actually the alternative strategy that you talk about! But this cannot be implemented without some change in your workflow and very probably in your organisation.

Also, please note that trunc based development cannot be a standalone thing: it has to be paired with feature toggling.

So this is nothing easy here, and nothing to rollout without a serious methodology.

**Bottom line is:**

You ask yourself very good questions, and you really should be going forward on this line of thought. However what seems to be ahead of you looks like a team transformation; something to be done with some fire power and / or team mates that are consumed agilists.
> 1 votes

---
Tags: git, sql
---


# Question
Title: Should we name names in a blameless post-mortem/retrospective?
We want our post-mortems to be blameless.

Recently someone included this statement in a post-mortem report:

```
On DATE, PERSONMAME did code a fix (link), but forgot to merge it to master

```

It was suggest that to be "blameless" we shouldn't include PERSONNAME. The suggested replacement was:

```
On DATE, the code fix (link) was created but the engineer failed to merge it to master.

```

In a blameless culture, nobody should be worried about having their name listed in a report. In the first example, it feels blame-full to point out their name.

On the other hand, in the second example, someone could follow the link and see who the engineer was. This seems to be hiding someone's name without really hiding it. It feels like something we'd do in an organization that is having a problem with blame/shame.

What should we do?

# Answer
> 3 votes
Usually for a blameless postmortem, the best idea is to go further than the human error (which for a proper 5 whys should not arise, that's rule 11, but we're just humans :)) and complete the 5 Whys with the 5 Hows.

To follow on this particular case for a blameless postmortem I'd go few step further whith those iterations (For the sake of the example) of How *possible follow up interpretation* (possible answers):

* How this fix has been left unmerged after creation ? (it wasn't reviewed and didn't follow the usual path)
* How *did it miss review* ? (That was urgent and straightforward, review wasn't necessary and decision was taken to deploy)
* How *this process has been defined* ? (It wasn't)
* How *was the bug report closed without merge* (No validation/no defined process for this case)
* How *to prevent the problem arise again* ? (Define the process and set validation/review even for quick patches)

Now you don't have someone who have forget to merge something, you have an analysis of how it was possible and more important, you have something actionnable to improve which is the main, if not the sole, reason to do a postmortem.

A more detailled criticism of the 5 whys (which as you show end often with a Who instead of a Why) can be found on O'reilly's blog post The infinite hows

To address the specific point of naming in the analysis, I'd argue neither the name nor the fix itself are of value, wording the step in the analysis as "An engineer wrote a fix and didn't merge it" without any link would be enough for the report.

# Answer
> 3 votes
Yes, you should include names.

"Blameless" postmortems do need to have detailed information and not be bowdlerized. Let me quote from one of the most definitive sources on blameless postmortems, the Etsy blog post that started it all.

> **A Blameless Post-Mortem**
>
> What does it mean to have a ‘blameless’ Post-Mortem?
>
> Does it mean everyone gets off the hook for making mistakes? No.
>
> Well, maybe. It depends on what “gets off the hook” means. Let me explain.
>
> Having a Just Culture means that you’re making effort to balance safety and accountability. It means that by investigating mistakes in a way that focuses on the situational aspects of a failure’s mechanism and the decision-making process of individuals proximate to the failure, an organization can come out safer than it would normally be if it had simply punished the actors involved as a remediation.
>
> Having a “blameless” Post-Mortem process means that engineers whose actions have contributed to an accident can give a detailed account of:
>
> * what actions they took at what time,
> * what effects they observed,
> * expectations they had,
> * assumptions they had made,
> * and their understanding of timeline of events as they occurred.
>
> …and that they can give this detailed account without fear of punishment or retribution.

The ENTIRE POINT of blameless postmortems is not to hide names, but so that people can boldly state what they did in the open without fear of punishment or retribution.

# Answer
> 1 votes
So, the second example is clearly better - but surely they're missing the whole point of blameless culture?

Firstly, it's not relevant who did it or if anyone really wants to follow the link and discover the person's name. People make mistakes and forget things.

More importantly, nothing you've written is the root cause! Dig deeper into the "why"s.

I'd speculate that the root cause is that you have systems and processes where it can happen in the first place, and when it does, it's not caught in time. Could the merge/PR process be improved? Would more reviews help? Should each merge be matched against your bug list and vica-versa?

Edit: If that's just one line of a six page document then, I assume you're not naming said person as the root cause. So, fairer. But I'd disagree that "hiding" his name is something a company having blame/shame problems would do. They shouldn't include the name because, one also assumes, they don't want to **start** having a blame/shame culture?

# Answer
> 0 votes
We also struggled with this balance. Having a name attached to certain records does tend to reveal some process faults. Things like:

* If a given person did not do something that they arguably should have
* What role a person played through the entire post mortem
* What a given persons contributions to each process in the post mortem were

All of which is useful when attempting to process change. For example, it might be that a given person did not know that they were responsible for ensuring the safety of a system, or a person did not feel that they were adequately qualified to solve a problem.

The thing is, to understand the problem it does not need to be allocated to a specific person - just to the same person throughout the duration of the report. Accordingly, we do not name people, but rather their role in their role in the management process:

* at 10:24:22 "John" reported that the system was inaccessible → At 10:24:22 the project owner reported the system ...
* at 10:38:00 "Bill" restarted MySQL to attempt to free the socket pool → at 10:38:00 Operator A restarted MySQL ...

This allows this sort of internally consistent reference, and we can evaluate the role each team member played. It additionally means that "gut reaction" to each team members failure is lost.

We do keep a key as to who is what, at the bottom of the report. But it's so we can ask for more information, and has no bearing on the reports content.

# Answer
> 0 votes
In the post-mortems I've taken part in, one of the components of the report is:

> What happened, and when did it happen?

Which includes a timeline. Timelines are incredibly useful for debugging process faults and providing context to people reading it long after it happened. Since humans are responding to whatever happened (if not kicked it off entirely), there will be references to humans in it. However, indirections that mention a role like *an operations engineer* are preferred.

Depending on the severity of the incident undergoing review, there may be chat-logs, tickets, and code-commits attached to the post-mortem as artifacts. Those will by definition name-names. The point of the blameless post-mortem is to keep the tone of it from assigning blame directly, and to avoid where possible assigning it indirectly. Indirect blaming is entirely cultural, though.

You can't stop people from assigning blame in their heads, but you can avoid assigning blame in the report. Also, by not including extensive membership in the post-mortem archive during performance reviews. If someone wants to assign blame, make them *work* for it.

The other area where names sometimes come up in the report itself is in the short and long descriptions of what happened.

> NAME merged to master before authorized, triggering a CD pipeline. While attempting to prevent the unauthorized change from reaching production, NAME caused the CD system to be mistakenly recycled stopping all deploy pipelines for six hours.

Which is clearly blamey. Compare this to:

> A Release Manager merged to master before authorized, triggering a CD pipeline. The mistake was realized immediately. While attempting to prevent the change from deploying, the CD system was destroyed by accident. Restoration by the CD team took six hours.

A certain kind of nerd will notice that this is very passive-voice. You can tell those nerds that *this is very much intentional*. Passive-voice has been the voice of deflecting blame for a very long time now, so go ahead and passive it up.

> Mistakes were made.

Yes, indeed. And which nebulous, not well defined people were involved in this?

> A Release Manager.

Doing what, vaguely?

> Merging to the master branch of a project.

And what, unspecifically, did they do in response to this?

> Broke the deploy system.

What, generally, was the impact of this?

> The deploy pipleine system was unavailable for a period of about six hours.

And how was this rectified?

> The CD team rebuilt the pipelines.

Very good. Now give me some action items.

> * A process should be put in place to prevent merges to master before authorization.
> * A circuit-breaker should be introduced in the deploy pipelines to allow a build to be safely blocked.

Thank you.

# Answer
> 0 votes
Professionals are responsible. Name names.. It sounds a bit hipster.. but we issue a special award for mistakes. Only in this way can mistakes be corrected promptly. The problem is not blame.. the problem is potential retribution. If your company is in the habit of punishing people for mistakes then you should leave. Companies that behave like that eventually find it very difficult to hire professionals. What you need to foster is an environment where people own up to.. and fix their mistakes. Blaming is not personal.. it's transparency and done correctly promotes camaraderie.

---
Tags: culture, operations, post-mortem
---


# Question
Title: Where are dunder dictionaries stored in salt?
I am new to Salt-Stack wanted to know where are all the dunder dictionaries stored?

For example `__salt__`, `__opts__`, `__grains__`, `__beacons__`).

# Answer
There are currently 5 dunder dictionaries. They are available at runtime and generally not stored statically.

`__opts__` \- Master or Minion configuration options; stored in configuration files of master and minion, collected at startup

`__salt__` \- Execution module functions (i.e. `__salt__['test.echo']('foo')`); from built-in and custom execution modules stored in `_modules/`

`__grains__` \- Grains (i.e. `__grains__['os']`); collected from minions, dynamically generated, or stored in minion config or `/etc/salt/grains`

`__pillar__` \- Pillar data (i.e. `__pillar__['foo']`); collected from the pillar

`__context__` \- Context defined in state modules and execution modules

You generally don't need to use them, unless you are developing Salt or writing custom modules, runners, returners, or external pillars.
> 3 votes

---
Tags: deployment, configuration-management, configuration, salt, storage
---


# Question
Title: Why unable to find the exact path for grains storage in master?
I am trying to completely remove the grains from the system. I have one master and one minion setup(minion id= minion1). I am doing

> salt minion1 grains.ls

that is listing me all the grains

Now I removed data.p which is the cache memory for the grains in master

```
path : /var/cache/salt/master/minions/minion1/data.p

```

Without running a salt\_sync or any other command for sync I am again doing.

```
 `salt minion1 grains.ls`,

```

It's returning the same result.(The grains are still somewhere stored.

Where exactly it is storing the grains and how to delete that?

# Answer
These are built-in grains, which are collected at minion startup. Here's the code responsible for generating them.

You cannot remove them, but you can overwrite them, for example to an empty value in `/etc/salt/grains`:

```
osfinger: null

```

or with grains execution module: `salt minion1 grains.delval osfinger`.

Keep in mind that some of the core grains are essential for the correct functioning of salt, e.g. module providers for `pkg` module rely on them.
> 2 votes

---
Tags: salt, devsecops
---


# Question
Title: how to automatically create a subdomain for every pull request
# Background

I got a team of untechnical QA's who have to do testing on iOS/android apps for every Pull Request (PR) that gets created by my backend team.

# Question

This is what I want to do: everytime a backend engineer creates a PR on bitbucket, I would like a script to automatically deploy the code of that PR git branch into a subdomain of our dev server that matches the JIRA issue created.

For example suppose the jira issue that the PR addresses is BAC-421, then as soon as the engineer creates a PR, the script deploys the code they created into AWS EC2 so that the QA can point their apps to www.bac421.mydevdomain.com

What is the best way to do this? I'm a devops technical nube.

# Update - Environment Specs

so here is a breakup of our env \- the backend uses laravel 5.3 \- it's deployed on AWS EC2 \- we use forge for auto deployment (nothing fancy.. we just run this script:

```
cd /home/forge/default
git fetch --tags
git pull origin master
git describe
composer install --no-dev --no-interaction --prefer-dist --optimize-autoloader
echo "" | sudo -S service php7.1-fpm reload

if [ -f artisan ]
then
    php artisan migrate --force
    php artisan config:cache
    php artisan queue:restart
fi

```

that we run as soon as we merge dev unto master branch) \- besides that we don't use any CI/CD tools although I'm open for recommendations \- DNS provider is GoDaddy \- our application server is nginx \- our database is in a separate RDS instance

# Answer
> 3 votes
this setup worked perfectly for me using aws code deploy:

# Answer
> 7 votes
We do this at work.

We have a small server, let's call it the receiver, it's the target of the GitHub webhook events. It runs a small application that parses the payload and incorporates logic around how to proceed e.g. create a new server on the infrastructure provider, update the load balancer, deploy to an existing server, destroy the server etc. This could be a traditional web application serving up an API or it could be a serverless application, up to you how you want to approach it.

The receiver is relatively straightforward to handle, the other necessary supporting systems you'll need are an approach to configuration management/provisioning (how does the server have the packages necessary to run the application), secrets management (how does the server get access to sensitive information) and routing (how do the subdomains get updated and route to the correct server).

It could be worthwhile looking at preparing an AMI with the necessary services configured, a CloudFormation template with the infrastructure provisioning logic and CodeDeploy could handle deployments for you.

**Configuration Management**

This is really up to you and your team, there are a multitude of tools you can use or you can simply rely on shell scripting. At what point in the servers lifecycle you apply the changes is what's discussed in the AMI design article I linked.

**Secrets Management**

This is a challenging topic to address with the information at hand, in the interest of brevity I'll leave that up to you and your team.

**Routing**

There are a few ways you can handle routing, the Application Load Balancer (not to be confused with the ELB/NLB) offered by AWS supports host based routing. Alternatively you could use a reverse proxy like NGINX or HAProxy, when you provision a new environment you will need to update this routing (ideally automatically) irrespective of what approach you take.

Don't forget to consider how you'll handle the database/persistence layer & zero down time deployments. The question to ask with the persistence layer is whether the team will share a database and how that will interact with things like migrations. On the topic of zero downtime deployments, CodeDeploy should handle that nicely for you. One more thing, you mentioned a single mobile application pointing to different environments, how will you point these applications to the environments.

---
Tags: git, deployment, bitbucket, git-hooks
---


# Question
Title: How to write logs for certain tasks only?
I am using Ansible and created a Playbook for deployment. I defined `log_path` in `ansible.cfg` and now tasks output are going to this log file. I want to send only few task's log into this log file. How can I achieve this ?

One option is to use `no_log: True` attribute but this is not displaying the logs on console.

So my requirement is to display all task's logs in console but I want to send only few task's log to my log file.

# Answer
It's in the wish list, I think. Quoting from the doc log\_plays - write playbook output to log file

```
* This callback writes playbook output to a file per host in the /var/log/ansible/hosts directory
* TODO: make this configurable

```

IMHO required functionality is not available at the moment. Take a look at the available plugins. "You can only have one plugin".

```
ansible-doc -t callback -l

```

There is an option to customize a callback. context\_demo looks promising.
> 1 votes

---
Tags: ansible, ansible-inventory
---


# Question
Title: Is Docker compose installation required everytime I start a new session on docker CLI
I have downloaded docker toolbox to learn docker and I have started with docker compose topic. I had run few docker-compose command and next time when I restart the docker CLI(here docker quickstart terminal) everytime I have to run below docker-compose install commands

```
sudo curl -L https://github.com/docker/compose/releases/download/1.21.2/docker-compose-$(uname -s)-$(uname -m) -o /usr/local/bin/docker-compose

sudo chmod +x /usr/local/bin/docker-compose

```

Is there any way to permanently install docker compose.

# Answer
When installing docker on windows or mac, there is a virtual machine called 'docker machine' that runs Linux OS. Docker has a command `docker-machine` used to start this virtual machine using `docker-machine start`. It is likely that every time you restart the toolbox, you start a new VM with new fresh copy of the OS. If you do `docker-machine ls` you should see multiple docker machines, one of them running. If instead on restart you always start the same docker-machine, it will have the previous changes, like previously installed packages, including the docker compose. You can give `docker-machine start <arg>` with the name of the docker machine you want to start, which should be always the same one.
> 2 votes

# Answer
According to this documentation, Docker Toolbox is a legacy product and one should use docker-for-X.

> Legacy desktop solution. Docker Toolbox is for older Mac and Windows systems that do not meet the requirements of Docker for Mac and Docker for Windows. We recommend updating to the newer applications, if possible.
> 1 votes

---
Tags: docker, docker-compose
---


# Question
Title: CD (Continuous Deployment) best practices for managing configuration
My CD pipeline:

1. Get docker image (created and pushed during CI stage)
2. Copy docker-compose .yml template to target server
3. Run the app

Let's go straight to the 3rd point. I can run the app using docker-compose up e.g., but I have no idea how to handle configuration. Target server is a Ubuntu Server created by Chief on AWS, I have an access to all AWS services like Parameter Store etc.

So here are my ideas about managing configurations:

1. save secrets to configuration file before running the app and attach this file to specific location in docker-compose template (I don't like this solution, I can handle this by some bash script that inject secrets to file)
2. set environment variables before running the app (I talked with some devops and basically they like this solution)
3. get secrets from AWS Parameter Store (unfortunately I can't have an access by role management, only by key which I need to set on server by env or in config file)

How managing configuration looks in real case scenarios? I'm worried about case of server crash or something similar. 1st solution will survive this, but in case of 2nd and 3rd ones the environment variables will be lost after crash/reboot. What is the best solution here?

# Answer
> 1 votes
If you start a container with an environment variable, that variable is part of the container definition. As a pro, when the server is rebooted, the same variable will be there when the container restarts. But as a con, anyone with access to the docker API can view that secret in clear text, and some apps log their environment when debugging.

With swarm mode (Kubernetes has their own similar solution), you can define your secret on the manager and it will be given to nodes running the container on a need to know basis. The secret is accessible inside the container as a file, but that file is only stored in memory on the worker node, and you can use an entrypoint script in your container to load that into the environment if that's how the application expects it. The swarm mode secrets and configs have the advantage of being portable as the container is rescheduled on other nodes in the event of a worker crash. Migrating from a docker-compose install to a single node swarm cluster and deploying a stack is a very simple task.

There are also external solutions for storing configs and secrets in key/value stores like CoreOS's etcd and Hashicorp's Vault. Both Swarm Mode and Kubernetes use a key/value store for their own settings, giving you an idea of how the community sees it as a best practice. The downside is you often want to configure the application to read and write configuration from one of these key/value stores directly. If you use something like swarm mode secrets and configs, these will be stored in docker's internal key/value database, so you get a similar solution with little effort on your part.

---
Tags: docker, continuous-integration, docker-compose, continuous-deployment, configuration-management
---


title: Docker: strategy-advise for a rookie

text:
# Question
Title: Docker: strategy-advise for a rookie
I am VERY new to Docker and I looking for a confirmation of my views and my usage of Docker in my use-cases.

I'll try it with examples:

* I have a T3 application. Every tier is a basic requirement (lets say persistence with postgres, application with java, presentation with php). Because I find for any tier an image on docker-hub I use them and compose them to have 3 running containers which exchange their data via network. Correct?
* I have an application that is in java and needs to work on the files of centos (e.g. read logs). There is not interaction - only stdout or files. Docker-images for centos are available, for java too. There is no easy way to create a Dockerfile with a double FROM. So I build my own image from centos and do ADD/COPY/RUN to get a java JDK into it. Correct?
* I can do in docker-compose.yml also actions that I can have in dockerfile too (e.g. workdir and command). Is it better to keep it in dockerfile and see docker-compose only for assembling multiple containers OR should I put as many things into docker-compose as I can?

# Answer
* Yes, one would usually have minimal docker images, i.e., one responsibility per image/container, with appropriate networking between them.
* Yes, images are linear, there is only ever one unbroken line of `FROM`s. To merge two images, get the two `Dockerfile`s and see what they are doing; it should be pretty self explanatory. Then you can either craft a single new `Dockerfile` mixing and merging lines from the sources, or keep one of the original dependencies as your `FROM`, and just copy lines from the other original to a new `Dockerfile`.
* You would use `command` in a `docker-compose.yml` in the same cases when you would give a command on the `docker run` line, in case you were running the container manually. That is, if you have an image which conveniently supports different commands, and were it would just be a waste to have a different image per command. So... if you find that you run your container with the same command in 99% of all cases, then put that `CMD` into the `Dockerfile` and remove it from the `docker-compose.yml`. If you find, instead, that you are using different commands all the time, then keep them in `docker-compose.yml`.
> 4 votes

---
Tags: docker, docker-compose, dockerfile
---


title: Dockerfile base image

text:
# Question
Title: Dockerfile base image
I have a Python application that I want to containerize with Docker. Should I use `FROM python:3` or `FROM python:3.7`?

Does the `python:3` tag automatically use the latest minor version of Python? Would there be a practical reason to pin to version 3.7 explicitly?

I could not find a clear answer to these questions in the official documentation.

# Answer
> 2 votes
Generally, when referencing 3rd party packages (Docker base images, Maven dependencies, Ruby Gems etc.) it is worthwhile to be as specific as possible. You want to avoid randomly upgrading dependencies at a later time, just because there's a newer version out there, just because you only specified a prefix of the actual 3rd party release number.

I personally would do whatever it takes to make sure that I get the same upstream version on every build, forever, and that upgrading always is an activity I have to d, that is never done for me. For example, in Ruby, the `bundler` keeps specific versions in the `Gemfile.lock`. While the `Gemfile` can specifiy vague versions (like "~\> 3.1" giving you anything "\>= 3.1 and \< 4.0"), whenever a version is picked by the automatic resolving, the actual immutable version is fixed in the `Gemfile.lock` and will never change on its own.

The reasoning is very simple - we use tools like Docker specifically to get out of the upgrade/version hell we had in the past; to make sure that we have the same versions on all environments, and to make changes visible (in the form of `git` commits etc.). Introducing the same all over, by having "floating" upstream version numbers, would really be a shame.

There is really nothing elaborate like that Ruby `Gemfile.lock`, for Docker base images, so in this case you have to look at the images you are pulling; they should document which tags are considered permanent. Then just specify those. And stay clear of `latest`, obviously.

For your particular example, to pick at random, a version like `3.6.6-alpine3.7` seems to be pretty fully specified. I'd expect not much more than bugfixes from upstream. But if I were especially paranoid on that day, I might specificy a `@digest` for the FROM (I'm not that paranoid normally...).

# Answer
> 1 votes
From the official Docker hub repository at https://hub.docker.com/\_/python/.

> Shared Tags
>
> * 3.7.0, 3.7, 3, latest:
>     + 3.7.0-stretch (3.7/stretch/Dockerfile)
>     + 3.7.0-windowsservercore-ltsc2016 (3.7/windows/windowsservercore-ltsc2016/Dockerfile)
>     + 3.7.0-windowsservercore-1709 (3.7/windows/windowsservercore-1709/Dockerfile)

Based on that, I assume version X will be a tag of latest X.Y.Z.

---
Tags: docker
---


# Question
Title: Jenkins Pipeline Kubernetes Agent shared Volumes
I have a Jenkins Pipeline using Kubernetes Agent.

I schedule multiple containers in my pod template.

I recently wanted to add volumes that would be shared some of the containers and am having difficulties.

I need to know how to create a volume that will be shared between 2 or more containers in a pod. That volume should mount to a specific path on each container.

**Specifics**:

* Using GCE (Google Cloud Engine) with Kubernetes Jenkins
* Using emptyDir volumes

There also seems to be a *severe lack of documentation* for this. So any direction on that would also be appreciated. The documentation I did find says that the way to do this is to define volumes in the pipeline that will be shared by all containers.

**Sample Jenkinsfile**:

Below is not my actual pipeline, but it recreates the issue.

```
pipeline {
    agent {
    kubernetes {
      label 'test-pod'
      defaultContainer 'jnlp'
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: frontend-test
    image: centos:7
    command:
    - cat
    tty: true
  - name: backend-test
    image: centos:7
    command:
    - cat
    tty: true
  volumes:
  - name: sharedvolume
    emptyDir: {}
    mountPath: '/opt/app/shared'
"""
        }
    }
    stages {
        stage('Test Pipeline Configuration') {
            steps {
                container('frontend-test') {
                    sh 'touch /opt/app/shared/test_file'
                }
                container('backend-test') {
                    sh 'ls /opt/app/shared/test_file ; echo $?'
                }
            }
        }
    }
}

```

**Result**:

```
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘test-pod’ are offline
Agent test-pod-5t0gj-wln8d is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (test-pod):

Running on test-pod-5t0gj-wln8d in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test Pipeline Configuration)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ touch /opt/app/shared/test_file
touch: cannot touch '/opt/app/shared/test_file': No such file or directory
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE

```

# Answer
To solve this, you need to include the volume mounts in the containers described here:

**Sample Jenkinsfile**:

```
pipeline {
    agent {
    kubernetes {
      label 'test-pod'
      defaultContainer 'jnlp'
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: frontend-test
    image: centos:7
    command:
    - cat
    tty: true
    volumeMounts:
    - mountPath: '/opt/app/shared'
      name: sharedvolume
  - name: backend-test
    image: centos:7
    command:
    - cat
    tty: true
    volumeMounts:
    - mountPath: '/opt/app/shared'
      name: sharedvolume
  volumes:
  - name: sharedvolume
    emptyDir: {}
"""
        }
    }
    stages {
        stage('Test Pipeline Configuration') {
            steps {
                container('frontend-test') {
                    sh 'touch /opt/app/shared/test_file'
                }
                container('backend-test') {
                    sh 'ls /opt/app/shared/test_file ; echo $?'
                }
            }
        }
    }
}

```

**Result**:

This solved the problem.

```
[Pipeline] podTemplate
[Pipeline] {
[Pipeline] node
Still waiting to schedule task
All nodes of label ‘test-pod’ are offline
Agent test-pod-wv12q-brfxj is provisioned from template Kubernetes Pod Template
Agent specification [Kubernetes Pod Template] (test-pod):

Running on test-pod-wv12q-brfxj in /home/jenkins/workspace/pipeline-test
[Pipeline] {
[Pipeline] container
[Pipeline] {
[Pipeline] stage
[Pipeline] { (Test Pipeline Configuration)
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ touch /opt/app/shared/test_file
[Pipeline] }
[Pipeline] // container
[Pipeline] container
[Pipeline] {
[Pipeline] sh
[pipeline-test] Running shell script
+ ls /opt/app/shared/test_file
/opt/app/shared/test_file
+ echo 0
0
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // container
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Finished: SUCCESS

```
> 4 votes

---
Tags: kubernetes, jenkins-pipeline, jenkinsfile
---


# Question
Title: Continuous Deployment best practices for managing application configuration
My CD pipeline:

1. Get Docker image (created and pushed during CI stage).
2. Copy docker-compose .yml template to target server.
3. Run the app.

Let's go straight to the 3rd point. I can run the app using docker-compose up e.g., but I have no idea how to handle configuration. Target server is a Ubuntu Server created by Chief on AWS, I have an access to all AWS services like Parameter Store etc.

So here are my ideas about managing configurations:

1. save secrets to configuration file before running the app and attach this file to specific location in docker-compose template (I don't like this solution, I can handle this by some bash script that inject secrets to file).
2. set environment variables before running the app (I talked with some DevOps and basically they like this solution).
3. get secrets from AWS Parameter Store (unfortunately I can't have an access by role management, only by key which I need to set on server by env or in config file).

How does managing configuration look like in real case scenarios? I'm worried about case of server crash or something similar. 1st solution will survive this, but in case of 2nd and 3rd ones the environment variables will be lost after crash/reboot. What is the best solution here?

# Answer
\[Copying answer over from duplicate post\]

If you start a container with an environment variable, that variable is part of the container definition. As a pro, when the server is rebooted, the same variable will be there when the container restarts. But as a con, anyone with access to the docker API can view that secret in clear text, and some apps log their environment when debugging.

With swarm mode (Kubernetes has their own similar solution), you can define your secret on the manager and it will be given to nodes running the container on a need to know basis. The secret is accessible inside the container as a file, but that file is only stored in memory on the worker node, and you can use an entrypoint script in your container to load that into the environment if that's how the application expects it. The swarm mode secrets and configs have the advantage of being portable as the container is rescheduled on other nodes in the event of a worker crash. Migrating from a docker-compose install to a single node swarm cluster and deploying a stack is a very simple task.

There are also external solutions for storing configs and secrets in key/value stores like CoreOS's etcd and Hashicorp's Vault. Both Swarm Mode and Kubernetes use a key/value store for their own settings, giving you an idea of how the community sees it as a best practice. The downside is you often want to configure the application to read and write configuration from one of these key/value stores directly. If you use something like swarm mode secrets and configs, these will be stored in docker's internal key/value database, so you get a similar solution with little effort on your part.
> 2 votes

# Answer
Our scenario is very similar to yours. Our configuration is a plaintext YAML file that is stored on the host that is bound mounted into the container.

It's not ideal because secrets are in plaintext. It's also problematic to maintain parity between 2 nodes.

As someone mentioned in the comments; k8s is one potential solution, but it feels like an overkill for a 2 node system. I haven't found anything between `docker-compose` and k8s yet.
> 0 votes

# Answer
I guess the point here is that you want to keep the deployment approach the same across all environments and update configuration in one place.

If you have had a look at the 12 factor app it states application config should live in the environment e.g. vault, config store, AWS Parameter Store. Kubernetes secrets scoped to a namespace.

Environment variables are also not considered very secure as they can be inspected at runtime.

I prefer option *3* for a number of reasons:

1. Application would pull configuration from config store at runtime (AWS Parameter store).
2. Application config can be changed dynamically (no need to re-deploy the app)
3. No need to add logic to your deployment scripts that pulls secrets from a config store and injects then into the compose file during deployment.
4. Your deployment orchestration tool does not need access to your applications secrets.

On the other hand:

1. Application will need to be updated to support pulling config from a store. This might make the application not cloud agnostic.

On a side not (not related to your question), if you're just deploying a few containers for a simple application and k8 is not suitable why don't you have a look at ECS Fargate? No need to manage any underlying infrastructure.
> 0 votes

---
Tags: docker, deployment, docker-compose, continuous-deployment
---


# Question
Title: Multi platform test on Travis-CI using Docker
I am trying to write my `.travis.yml` to perform multi platform test on Travis-CI using Docker. The platforms I want to test on:

1. Ubuntu 14.04
2. Ubuntu 16.04
3. Debian 8
4. Debian 9

The whole concept in a nutshell is (for all 4 distros):

1. `docker pull ubuntu:trusty`
2. `docker run ubuntu:trusty`
3. Run `make test` inside the container (on my repo's clone)
4. Copy the `result.xml` back to Travis
5. Run `scp result.xml ${SERVER_USER}@${SERVER_ADDR}:/home/${SERVER_USER}/result/`

I know how to run scripts inside a docker container but how am I going to get the current branch inside the container and run `make test` on it make me go crazy.

Any help will be much appreciated!

# Answer
> 1 votes
This functionality is built straight into travis. Have a look at here.

Secondly you should not need to specify a branch as part of your make script as travis will triggered on a specific branch and therefore that revision would already be checked out.

---
Tags: docker, travis-ci, docker-in-docker
---


# Question
Title: What is the Maven versioning strategy for Continuous Delivery or Deployment?
At the moment, we're doing "traditional" deployment, where every couple of weeks, we roll a release. We rely on Semantic Versioning, which has worked well for us.

However, we now want to release faster, using Jenkins Pipelines etc, but I'm not sure what the best versioning strategy would be. Not having any versions (only Snapshots) is a step backwards, creating new versions all the time seems to be a lot of work.

So, if you are doing CD with Maven, what's your strategy?

# Answer
> we now want to release faster

And from your comment:

> Basically, since the artifacts depend on each other, we need to release them in the correct sequence. And since there's a lot of them, it takes some time.

I think you're having an X/Y problem. Yes, versioning plays a role, but it seems like you would benefit from solving the source of your problem first, which is that you have to release something in the *correct sequence*; presumably you also have to release everything at the same time, with a large downtime of the whole system.

Try to reach a state where you can release new versions of each artifact in arbitrary order. This basically means:

* Employing some form of Dependency Injection (which is a big word, but can be something pretty simple, depending on how your software works).
* You need to be able to have multiple versions of each artifact on the server at the same time, and to switch between them without much effort. That "switch" should be be on the side of the "user" of an artifact, not on the artifact itself, if you can do so at all.
* Wherever you can, it also is great to be able to run multiple instances simultaneously (obviously this is restricted for data management stuff, databases and such).

I'm not telling you how to do that or that it is easy - I don't know your architecture or technical constraints. But this should give you a general roadmap.

When you have that capability in place (at least conceptionally), then you will be able to release smaller parts more often, and a little more independently. And - to answer your actual question - sticking with SemVer is then not the worst you can do.
> 1 votes

# Answer
I've just done this and so far it's working well for us.

Basically we use a Jenkinsfile and retrieve the version from the POM (which is using semantic versioning) and remove the snapshot.

```
stage('Build Docker Container') {
        steps {
             script {
                pom = readMavenPom file: 'pom.xml'
                pom_version_snapshot = pom.version

                version = pom_version_snapshot.split("-")[0]
                echo version

            }
            sh "docker build -f 'Dockerfile' -t repo/container:${version}-${env.BRANCH_NAME} ."
        }
    }

```

We then do a Docker build and use that version to tag the image, which is later deployed to a Kubernetes cluster.
> 1 votes

# Answer
Before this can be answered I think we need understand what does a version mean to you?

* Is it a customer facing product version (version 8.1.2 contains these features and bug fixes)?
* Or is it just for highlighting what version of the code is deployed where?

If it is the later you could just use the commit id and tag the commit with Jenkins build number. However, if you loose your Jenkins build history it gets slightly confusing therefore some form of CMDB managed by a container platform like k8 would be best.
> 0 votes

---
Tags: maven, versioning
---


# Question
Title: What are the correct volume maps for zabbix running on Docker?
I want to run zabbix inside Docker and there are images for MySQL, Nginx and zabbix server ready to go.

I know that to keep the containers stateless I need to map a volume on the host to the container. I know that for MySQL the important container volume to map is

`/var/lib/mysql`

But what are important parts of zabbix to backup so that the container can stay truest stateless?

From what I can tell it seems to be all the directories under `/var/lib/zabbix`.

# Answer
> 1 votes
There is a list of all the important paths Zabbix containers use: https://www.zabbix.com/documentation/3.0/manual/installation/containers (scroll down a bit until you find "Volumes" part).

If I were you, instead of bind mounting these directories I would build a derivative image from the official Zabbix image and run statements like "COPY my\_alertscript.sh /var/lib/zabbix/alertscripts/" in the build phase.

If, for some reason, baking your changes to the image itself is not possible, then you may still want to prefer named volumes instead of bind mounting. See: https://docs.docker.com/storage/volumes/ to read about its advantages.

---
Tags: docker, docker-compose, dockerfile
---


# Question
Title: In Jenkins, how can parameters that contain spaces be referenced?
Google has failed to even lead me to a proper answer as to why spaces are permitted in names of Jenkins parameters, let alone how to reference them. Assuming it's not just a major oversight, how does one access these parameters?

For example, I have a string parameter named "IP Address". I could rename it "IP\_Address" but I'm trying to avoid the underscores\_as\_spaces if I can.

I've tried to refer to this parameter using "${IP Address}", "IP Address" and even "IP\ Address" but nothing seems to work. Is there any way to actually reference these?

# Answer
You can use `params['IP Address']`.

Think of `params` as a Map containing a key `'IP Address'`. If the key didn't have a space, then you could use `params.IPAddress` or `params['IPAddress']`, but when there's a space, you can only use the latter syntax.

That syntax is also useful when you're computing the key name to look up, and have stored it in a variable:

```
final String myParam = "param${i}"
// can't say `params.myParam`
echo "Value is ${params[myParam]}"

```
> 5 votes

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: CloudWatch trigger for a range of GuardDuty severities
I want to edit my CloudWatch rule so that it only triggers an SNS topic for "GuardDuty findings" that fall in the *High severy* range, defined by AWS as `the value of the severity parameter in the GetFindings response falls within the 7.0 to 8.9 range`.

The sample Event Pattern in the documentation only shows how to create a trigger for severities 5 and 8 as follows:

```
{
  "source": [ "aws.guardduty" ],
  "detail-type": [ "GuardDuty Finding" ],
  "detail": {  "severity": [ 5, 8 ]  }
}

```

How do I change this `{ "severity": [ 5, 8 ] }` to be a range between 7.0 to 8.9?

# Answer
I used this post to get our Guard Duty alerts working, thanks! But a recent change at AWS caused us to stop getting any alerts. Turns out we needed to add the integer values along with floating point numbers. We used a CLI command like the following to do that:

`aws events put-rule --name Test --event-pattern "{\"source\":[\"aws.guardduty\"],\"detail-type\":[\"GuardDuty Finding\"],\"detail\":{\"severity\":[7.0,7.1,7.2,7.3,7.4,7.5,7.6,7.7,7.8,7.9,8.0,8.1,8.2,8.3,8.4,8.5,8.6,8.7,8.8,8.9,7,8]}}"`

Note the "`7`" and "`8`" at the end. Alerts are working again.
> 3 votes

# Answer
Here's the answer I got from AWS Support, slightly edited. (I would downvote them, if I could).

By design, CloudWatch events do not support ranges, so you need to explicitly set the values that are in that range. Additionally, the console has an issue that we are currently addressing, regarding the **.0** values.

So, the way to achieve this at the moment is only via the CLI. For more information regarding the CLI command, you can read this documentation.

The workaround is this:

1. Create the rule through the console by selecting as a service name : Guard Duty and as an Event type: Gard duty finding.
2. Add the targets
3. Configure details
4. Use the AWS CLI to run the following command. I took the initiative and added all the values in the range you mentioned. You can change it if needed:

`aws events put-rule --name Test --event-pattern "{\"source\":[\"aws.guardduty\"],\"detail-type\":[\"GuardDuty Finding\"],\"detail\":{\"severity\":[7.0,7.1,7.2,7.3,7.4,7.5,7.6,7.7,7.8,7.9,8.0,8.1,8.2,8.3,8.4,8.5,8.6,8.7,8.8,8.9]}}"`

In case you need further assistance regarding the above please let me know.

Best regards,

Amazon Web Services
> 2 votes

---
Tags: amazon-cloudwatch, amazon-guardduty
---


# Question
Title: How to notate ipv6 addresses in inventory file?
I have a machine, that is accessible via ipv6 only. It is setup for ssh with rsa key. Now I'd like to manage it with ansible. Unfortunately, this is not covered in the inventory documentation.

So I tried the following permutations:

* `docker_test ansible_host=fe80::88ea:e002:b409:4be0`
* `docker_test ansible_host=[fe80::88ea:e002:b409:4be0]`
* `docker_test ansible_host=\[fe80::88ea:e002:b409:4be0\]`

None of this worked out.

For comparison I add my `.ssh/config` entry:

```
Host docker_host
    AddressFamily inet6
    HostName fe80::88ea:e002:b409:4be0
    BindAddress fe80::3e6b:335a:d2f7:3a75%enp3s0

```

I found two closed bug in the issue tracking system,which indicate the problem being at least partially solved: 3879, 3888

Any advise on this topic is highly appreciated.

# Answer
> 6 votes
This blog post covers all of the methods:

*excerpt - Host names and patterns in Ansible 2*
> ### Inventory hostnames
>
> Ansible 2 requires inventory hostnames to be valid IPv4/IPv6 addresses or hostnames (i.e., x.example.com or x, but not x..example.com or x--). As an extension, it accepts Unicode word characters in hostname labels. Any mistakes result in specific parsing errors, not mysterious failures during execution.
>
> Inventory hostnames may also use alphabetic or numeric ranges to define more than one host. For example, foo\[1:3\] defines foo1 through foo3, while foo\[x:z:2\] expands to fox and foz. Addresses may use numeric ranges: 192.0.2.\[3:42\].
>
> ### IPv6 addresses
>
> A number of problems with the parsing of IPv6 addresses have also been fixed, and their behaviour has been made consistent across the inventory (.ini files) and in playbooks (e.g., in hosts: lines and with add\_host).
>
> All of the recommended IPv6 address notations (from spelling out all 128 bits to the various compressed forms) are supported. Addresses with port numbers must be written as \[addr\]:port. One can also use hexadecimal ranges to define multiple hosts in inventory files, e.g. 9876::\[a:f\]:2.
>
> A couple of small but necessary bugfixes go hand-in-hand with the parsing changes, and fix problems with passing IPv6 addresses to ssh and to rsync. Taken together, these changes make it possible to use IPv6 in practice with Ansible.

Therefore patterns such as these IPv6's would be allowed:

* fully specified 128 bit address
* \[addr\]:port
* hexadecimal ranges - 9876::\[a:f\]:2
* various forms of compressing ranges - FF01::\[1:6\]

Simplified examples:

```
# IPv6 Address                                # Simplified Notation
-----------------------------------------     ----------------------------
- FF01:0000:0000:0000:0000:0000:0000:0001     - FF01::1
- 2031:0000:130F:0000:0000:09C0:876A:130B     - 2031:0:130F::9C0:876A:130B
- 0000:0000:0000:0000:0000:0000:0000:0001     - ::1
- FE80:0000:0000:5EFE:0192.0168.0001.0123     - FE80::5EFE:192.168.1.123
- FE80:0000:0000:0000:1585:4868:495F:D521     - FE80::1585:4868:495F:D521

```

### References

---
Tags: ansible, ansible-inventory
---


# Question
Title: Jenkins Pipeline GitLab API Credentials Management
I have a Jenkins pipeline that executes a build whenever a change is pushed to the appropriate branch. Jenkins uses an API token to authenticate with the GitLab server and checkout the repository. Besides creating a "Jenkins" user account in GitLab and using that user's API token, is there any other way I can manage GitLab authentication for an automated CI/CD pipeline? Each user of Jenkins currently has their own API token.

# Answer
> 1 votes
Are you doing the building using web hooks or just polling? I think it's inevitable that you'll have to create a Jenkins account to manage API access. This Gitlab link explains how to use the Jenkins Gitlab plugin to manage building when changes are pushed. You'll see they use a Jenkins account as well. This is common practice in my organization to remove single points of failure, such as a departing employee.

---
Tags: jenkins, jenkins-pipeline, gitlab, authentication, cicd
---


title: Jenkins slave not able to find dotnet

text:
# Question
Title: Jenkins slave not able to find dotnet
When I check for dotnet version it is throwing an error

```
D:\jenkins\workspace\csharp-windows-test>dotnet --version
17:58:06 System.ArgumentNullException: Value cannot be null.
17:58:06 Parameter name: path1
17:58:06    at System.IO.Path.Combine(String path1, String path2)
17:58:06    at Microsoft.DotNet.Configurer.CliFolderPathCalculator.get_DotnetUserProfileFolderPath()
17:58:06    at Microsoft.DotNet.Configurer.FirstTimeUseNoticeSentinel..ctor(CliFolderPathCalculator cliFolderPathCalculator)
17:58:06    at Microsoft.DotNet.Cli.Program.ProcessArgs(String[] args, ITelemetry telemetryClient)
17:58:06    at Microsoft.DotNet.Cli.Program.Main(String[] args)
17:58:06

```

but when I search for 'where' it shows the location

```
17:58:06 D:\jenkins\workspace\csharp-windows-test>where dotnet
17:58:07 C:\Program Files\dotnet\dotnet.exe

```

This is causing build failures. Can someone suggest a best way to resolve this

Also I was able to execute the script on the machine directly without issues. This is only happening when called from Jenkins

# Answer
When I did `echo %USERPROFILE%` from Jenkins it returned null. So I had to manually set `USERPROFILE` value from Jenkins as environment variable. Also I had to set these

`APPDATA=%USERPROFILE%\AppData\Roaming`

`LOCALAPPDATA=%USERPROFILE%\AppData\Local`
> 0 votes

---
Tags: jenkins, continuous-integration, windows, builds, dotnet
---


# Question
Title: DNS hosting choices - how to programmatically update DNS settings
I manually update the DNS settings at the registrar for my domains, such as

```
https://domains.google.com/registrar

```

What are people using to automate updates to DNS settings? which registrars have an API ?

Specifically I need to manually update :

* TXT field for the DKIM setting
* TXT field for spf

@ TXT 1m "v=spf1 ip4:100.22.22.100 include:blah.com -all"
* A record everytime I change external IP

To run my own DNS host servers seems like overkill. Ideally I would just make some secure REST call to perform programmatic updates.

# Answer
Amazon web services provides such a service, called Route53. You can actually use REST calls, but it is much easier to use their command line tools or bindings for PHP, Python or JavaScript.

Here is an entry point to get you started, the route53 change resource record sets call can be used to update or insert one or many records on a domain. https://docs.aws.amazon.com/cli/latest/reference/route53/change-resource-record-sets.html

Of course you need to create an account, set up permissions and delegate name servers to route53.

It's also not free, but fairly cheap at $0.50 for first 25 domains, then $0.10 after. All charges per month.
> 1 votes

# Answer
You can use dns.resolver or dns.rdatatype and update in Python like below:

```
import dns.resolver
import dns.rdatatype
import dns.update

for i in dns.resolver.query('blah.com', 'TXT').answer.response:
   update.replace(% i , 60, dns.rdatatype.TXT, '" NEW RECORD"')

```
> 1 votes

---
Tags: dns
---


# Question
Title: Is artifact promotion in this pipeline correct?
I have been tasked with rebuilding my DevOps pipeline and I need some help with a few parts.

We will have our code in BitBucket and have something trigger a server to spin up and validate our PR's. I don't fully understand how to do this at the moment, but its a pretty common pattern so it shouldn't be too bad. We will run our Unit Tests, do some static code analysis (test coverage, complexity, code quality, etc.).

Next we will have a Jenkins job monitoring master. It will run the unit test suite again (I am thinking that this is necessary because when merging the PR's we could introduce some errors possibly). Next it will build the bits and push it to an artifact repository. This will then start the staging test job.

The staging test job will pick up the bits built by the previous job, or allow any user to start it with a valid staging artifact. It will deploy it to a staging environment and then Run the automated tests here. If they pass it will somehow mark this artifact as acceptable. This is my first big question, is it common to move a compiled artifact between repositories to let the team know when it has passed certain quality thresholds?

Next we will have a job that allows anyone to take an artifact that has passed the automated tests and deploy it to some UAT environment where they can do any manual testing or demo's as required. My big question here is how can we mark this artifact as passed/failed?

Finally we have a job that can push an artifact that has passed manual testing out to production. My thinking is that this would allow us to push anything deemed acceptable to go to the customer, and would give us rollback if we build it correctly.

To summarize my questions:

1. How do we mark artifacts that have passed certain stages in the pipeline in an automated fashion?
2. How do we mark artifacts as passing final stages in a manual way?
3. Is this missing anything major or have you run in to issues with a similar pattern?

# Answer
> 7 votes
It depends on the artifact repository you are using but in general, you tag the artifacts in the repository to indicate their state. This can specify it has passed some level of quality gate, some approval, or what ever is required to move through the stages.

I would say you tag the artifact as to it's state, not move between artifact repositories. A single artifact repository can be used to hold the artifacts at multiple versions, and multiple states. This provides a single audit point and a single way of controlling the deployment which can be very helpful.

With tagging of the artifacts you can be clear if they are ready for deployment, and if so to which environments. The tags can be updated in multiple ways, but I would recommend automated processes, such as when an artifact is deployed, the artifact is tagged. When it set of tests pass the end of that marks the artifact as ready. If you have manual testing, that testing will be tracked in some tool or dashboard or approval process, when it is approved, that process should tag the artifact as ready to move on.

# Answer
> 0 votes
I think the way to think about the pipeline is to have an automated system with decision points - gates - that require a human to approve. Your pipeline might have one or many of these gates. There should be people who watch the gates, e.g. QA people waiting for a new build, release management people deciding whether to deploy.

The record of what the pipeline did should be logged somewhere. We are an Atlassian shop, and intend to use JIRA tickets with some special software at these gates. In general we record our DevOps work in JIRA tickets. There are lots of other ways to do the same thing, you could use GitHub, or Jenkins, or CircleCI or Google docs etc.

---
Tags: build-pipeline, artifacts
---


# Question
Title: What is good and/or bad about Accenture's Multi-speed IT approach to DevOps?
Accenture are talking a lot about their "Multi-speed IT" approach.

It appears in their strategy consulting here:

and on blog posts in their DevOps blog:

As well as in many other places that deal with various aspects of a technology organisation.

Their own "head of DevOps" is talking about how Multi-speed is a bad practice https://www.youtube.com/watch?v=yGcXLwsnDgg

Does this idea of "Multi-speed IT" has merit and can be used for continuous improvement in-line with DevOps? Or does it go against some of the common DevOps practices?

# Answer
> 4 votes
There are reasons different parts of the environment might change at different frequencies. One example might be a calendaring service, most large organizations have one, they provide information about the current day, if it is a holiday etc. The calendaring service does not need to change frequently because it provides a stable function. This calendaring service may be used by many different kinds of apps that also move at different speeds.

Multi speed exists because you need to deal with making changes that deliver value. Many people indicate multi speed is to deal with the font end system of engagements and the systems of record. Yes, generally user facing systems change more often because of the desire to update for the end users and experiment with different ways of presenting data to get the best value. However, the systems of record may need to be changed as fast, in order to satisfy a new business requirement.

If all parts of the application are built using a pipeline with appropriate automated testing, and deployment, and their are ways to test independently but also test together, then any part can move as needed. It is similar to the gears thought that the different parts move but need to be integrated such they you know they work together as they are deployed.

# Answer
> 2 votes
Based on this

> The smaller gear moves much faster than the larger one, but where the two gears interlock they remain aligned to not stop the motion.

and this quote

> The whole idea of the Multi-Speed IT is to make the delivery of functionality less interdependent. On the flip side, you need to spend more effort on getting the right practices and tools in place to support this. For example you want to make sure that you can quickly test the different interface versions with automated testing, you need to have good version control to make sure you have in place the right components for each application, you also want to make sure you can manage your codeline very well through abstractions and branching where required. And the basics of configuration management, packaging and deployment will become even more important as you want to reduce the number of variables you have to deal with in your environments. You better remove those variables introduced through manual steps by having these processes completely automated.

from this post one could say that the multi-speed IT approach could be aligned with DevOps as it is aware about multiple departments and processes in an organization and that these do not need to align in terms of velocity as long as these get less dependent. For example, multiple DevOps teams could create a different amount of features every sprint without thwarting each other as long as the components are versionized, backward compatible and well documented.

---
Tags: culture
---


# Question
Title: Software/devops engineer's personal dashboard recomendations
Can you recommend a free tool or GitHub project that is designed for an individual to keep up/aware of their daily tasks and operations?

For example:

* Show perf metrics from running azure VMs in graphs or tables
* Show events from application insights
* Ability to configure workflows that execute some code (like javascript on node) with a click of a button
* Call a rest API with parameters with a click of a button
* List my work items, PRs, build status from VSTS
* Show a filtered view of my mailbox on Office 365 that I configure to show important mails from the team or boss

I'm hoping that this tool would either be browser based or work as a Windows client.

I'm sure no tool exists that meets all these requirements. So if I had to build something myself then are there any good platforms that allow easy customization to make these things happen?

Any pointers?

# Answer
For all of your read-only dashboard needs, there is Grafana with multitude of plugins, data sources and visualizations (tables, charts etc). You can also program your custom data sources pretty easily with official "simple-json-datasource" plugin.

By default, Grafana does not support buttons or other kind of form/action stuff on its own. Grafana is all about visualizing data, not modifying it. Yet, while I haven't tried it myself, Grafana supports Text/HTML/Markdown panels, I think you can implement any JavaScript logic inside those panels.
> 2 votes

# Answer
It sounds to me like you might get some good help by changing what you ask for. The tools that support Agile programming tick off many of these boxes. The Agile world is pretty mature, and it's unlikely you have a problem not encountered already. If you look at those tools, you'll see how they lay out kanban boards, and tickets, and software metrics, and visualizations.

GitHub ( and probably GitLab) also have a lot of developer tools. I am using Grafana, but its definitely not an execution environment, nor is JIRA without a paid extension. But Jenkins can be used for many types of tasks to kickoff whatever. For your email, create some filters for Outlook, or start using Slack, even better.
> 0 votes

---
Tags: software-engineering
---