# Question
Title: Reverse Proxy and TCP Optimization Inside the Network
Need to make some optimizations and have been reading articles such as linux-tcp-tuning

In a world of load balancers and revers proxy is there any value in tuning TCP connections that are inside the network? As I understand (please correct me if I'm wrong) there is almost zero latency and no packet loss between the load balancer and the servers.

So I wonder if changing parameters such as net.ipv4.tcp\_window\_scaling, net.ipv4.tcp\_timestamps, net.ipv4.tcp\_sack will have any effect if not set on the load balancer. Thanks!

**EDIT:**

Other parameters improved my server in simulation. I set net.ipv4.tcp\_fin\_timeout=5, net.ipv4.tcp\_keepalive\_probes=5 and it served more clients (without a load balancer). I just don't know if it will have the same effect with a load balancer. There are so many TCP parameters under /proc/sys/net/ipv4/. Testing them on a single server is one thing. Testing them with a load balancer and many servers is a huge effort. I just don't know if it worth the effort of testing those parameters with many severs and a load balancer.

# Answer
> 5 votes
Tuning the parameters you have identified will certainly have an effect - to *degrade* performance. For example, net.ipv4.tcp\_window\_scaling is an on or off option. Similarly, net.ipv4.tcp\_sack is either on or off. Both default to a value of on on Linux and on all load balancers. Both of these were defined in RFC1323 which was published in order to give us the ability to achieve 10/100 Ethernet networks. Without this RFC, networks were limited to 10 Mbps a second, so disabling these will result in a huge, immediate and noticable performance hit.

These performance benefits come because there is a huge difference between "almost" zero latency and zero latency. Using a sliding TCP window allows equipment to use Selective Acknowledgements (SACKs) to acknowledge the receipt of hundreds of packets simultaneously. So, if latency is 10ms on a network, without SACKs, every packet has to be acknowledged and that 10ms latency adds up very quickly - even at theoretical minimum latencies. This means that I send packet A, then have to wait for a response to packet A before sending packet B. But as it turns out I can send the rest of the alphabet and then some while waiting for the ACK to packet B. Enter SACKs - These allow the receiver to signal that it is has successfully received up to and including packet Z. Meanwhile, the sender is already sending packet AA-ZZ.

This results in something known as a "TCP Window Size" which is the maximum amount of data the client can buffer/handle. Since memory can be a concern, if the recipient is only capable of storing and processing Packets A-F, if we send A-Z some amount of data is going to be lost. Therefore, it becomes necessary for the recipient to signal their ability to receive data - that is "I can only buffer this many packets). This is really CPU dependent on how fast the recipient can empty their buffers and transmit the data up to layer 7. It is not uncommon to see TCP windows scale from empty to full (Max window size goes from 0 indicating the window is completely used to max indicating the window is completely available) and this is one way throughput bottlenecks can be identified on your network.

TCP Window Scaling offers two primary benefits:

1. Less unused time on the wire - we aren't sitting around waiting for an ACK and we have successfully defeated the prop. delay (latency).
2. Less time devoted congestion control = more time for data. Congestion control messages can eat into your transmission time and bandwidth. Even with TCP Window Scaling, Congestive Collapse can still occur in which an entire TCP window may need to be retransmitted and congestion control may be required.

Finally, the only real reason to update net.ipv4.tcp\_timestamps is to provide additional security as it is otherwise based off of server uptime. Since you are proxying behind a load balancer anway, adjusting this setting is of little value unless you are using something like a Performance Layer 4 profile. I doubt adjusting this will make much difference, though again it is only likely to degrade, not improve performance.

Far and away, what is more likely to help is adjusting your send and receive buffer sizes and using Jumbo Frames. this will allow you to process more data with the same amount of overhead (a 75% reduction in processing over a standard MTU of 1500. These are the primary changes required to reach a 10 Gbps connection. Otherwise, just make sure you have enough CPU to ingest your data from the TCP stack and be aware of the pattern of your traff (eg, avoid things like the C10K problem).

**EDIT:**

`tcp_fin_timeout` doesn't actually improve performance, instead it defeats resource constraints. This setting could actually be a major player in a load balancer implementation. The problem is that for any given client and server, the server will be hosting a web server (for example) on 192.168.1.5:80. Every TCP connection must have both a source IP and source port as well as a destination port and destination IP. Source ports are chosen by the client from a pool of "ephemeral" ports. This results in a theoretical maximum of 65534 connections (the maximum number of ports) between a given client to port 80 on that server. When using load balancers, this can result in "port exhaustion" when the pool of ephemeral ports is depleted when SNAT (Secure Network Address Translation) is used on the load balancer, which results in a theoretical maximum of 65534 *between the load balancer and the server.* There are two basic solutions to this problem: 1) use a SNAT pool where 2 source IPs for the load balancer * 65534 = 131068 connections and 4 IPs X 65534 = 262136 max connections, and so forth or 2) don't use a SNAT. For this to work, the load balancer needs to be part of the default route back to your clients. If you are testing with a single actual client simulating multiple clients, this is a poor real-world test and you may actually support many more clients than you think because you are exhausting your ephemeral ports, where real clients would never do that because they come from unique IPs and two clients can use the same ephemeral port *if you are not using SNAT on the load balancer*.

Now, I keep mentioning that this is the *theoretical maximum* but the reality is that on Linux servers, there is a defined range of ephemeral ports and so typically you will see port exhaustion at around half of the theoretical maximum. You can see results similar to adjusting `tcp_fin_timeout` by adjusting the ephemeral port range in /proc/sys/net/ipv4/ip\_local\_port\_range.

This brings me to the final piece, the TIME\_WAIT state governed by `tcp_fin_timeout` and defaults to 60 seconds. This gives every port used a 60-second "cooling off period" before it is returned to the ephemeral port pool. This is why you saw a performance boost in your testing. This setting is *supposed* to be 2 times your maximum segment lifetime as defined by the TCP protocol, as it is theoretically possible that for a FIN packet with this setting below below 1 X maximum segment lifetime (30 seconds) to still be in transit. In reality, this is very unlikely, but technically you are breaking RFC compliance by adjusting this setting. If you adjust this setting downward however, you would want to be sure to adjust it downward on the Load Balancer's TCP profile (so tcp\_fin\_timeout corresponds to Time Wait, not FIN Wait on an F5, for example).

Ultimately, this is a fantastic example of where testing with a load balancer and many servers would result in a huge performance difference - because your testing probably doesn't reflect the real-world architechure. It isn't a good simulation of the customer experience.

However, tweaking both ephemeral port settings and using keepalives are two divergent traffic management strategies. In fact, if you use a setting like OneConnect, there simply is no longer any need to tweak your ephemeral port ranges and timeouts because connections are recycled and multiple clients can be funneled over the same connection (note, this only works for HTTP traffic, but similar can be achieved with `tcp_keepalive_probes` if you want to configure it that way because clients no longer have to wait for the three-way-handshake - they can simply start transmitting.)

## Conclusion

There is no pat answer here. You are asking "Is it worth doing all this work?" - I don't know. Is it? What is the acceptable level of business risk? What does it cost your business if you get it wrong and bottleneck?

Ultimately, I feel you are basically asking two things 1) what do all these settings do and what is the effect of tweaking them? And 2) are there any shortcuts I can take?

Unfortunately, you have not provided enough information to answer either question (which is why your last question was closed and this one will be downvoted.) First, I don't have the time to explain what every TCP setting in proc does (nor do I know off the top of my head, I'm actually googling these on the fly.) but secondly (and more importantly) it is impossible to determine their effect without knowing how your service is architected. There is probably performance that can be squeezed out of your system, but kernel TCP parameters are pretty much tuned to the best possible configuration that can be safely made without breaking the TCP/IP RFC. What you are doing is pushing into territory that *can* be safe, but only if you understand the architecture and the TCP/IP protocol. And it would behoove you to learn it. If you want to be the smartest guy in the room, learn TCP/IP. Pick up a copy of TCP/IP Illustrated and/or Internetworking with TCP/IP and commit the basics to memory. Learn how HTTP requests and SSL handshakes work. Learn how to read a pcap in wireshark and tell the story of what is happening according to these protocols. This will take you very far in life whether you go in to System Administration, Networking, or Programming (I have used this knowledge in all three realms). Simply put, *there is no substitute for knowing what the hell you are doing* and this just isn't something you are going to be able to answer off of Stack Exchange. This is why you pay consultants - because if they are good they know the ins-and-outs of this. Once you understand 1) your architecture and 2) what the settings actually do and what effect they will have you will better be able to tell if you can take any shortcuts (that is, I don't need to tweak this setting; it is irrelevant. This testing will be worthless because it doesn't simulate the customer experience.

---
Tags: load-balance, servers
---


# Question
Title: Kubernetes create multiple jobs based on template and list
We are currently migrating many of our production data ingress/processing scripts/processes into a kubernetes cluster. Each "customer" or company that we interface with has a bunch of data and we want to process all of this data on a per customer basis. I have created a container with a script where I pass a single customerID into. `E.G. One container will run and process one customers data.` This will need to happen for one or more customers on a schedule.

I have been using the following definition to work specifically for customer 756, and it is working as expected.

```
apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: image-puller-756
spec:
  schedule: "30 * * * *"
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: image-puller-756
            image: index.docker.io/my_company/image_puller:latest
            imagePullPolicy: Always
            env:
              - name: ALLOW_TTY
                value: "true"
              - name: PYTHONUNBUFFERED
                value: "0"
              - name: prodDbUser
                valueFrom:
                  secretKeyRef:
                    name: image-creds
                    key: username
              - name: prodDbPass
                valueFrom:
                  secretKeyRef:
                    name: image-creds
                    key: password
            command: ["/bin/bash"]
            args: ["-c", "/usr/bin/python3 -u /go/bin/main.py -c 756"]
          restartPolicy: Never
          imagePullSecrets:
          - name: myregistrykey

```

Once scheduled it will run hourly which is perfect, however I now need a way to duplicate this definition for ~500 different customers and apply it to the cluster. Currently I am using a template file and doing a Find and replace with a list of the customers I'm testing, but I'm looking for a better way to manage it. The customers will only have certain processes ran for them based on some logic, and I would like to be able to run another process that will act as a schedule and dynamically add or remove jobs for the customers.

I thought that there might be a way using yaml or k8s to pass a list and a template and have it create them. The approach just seems very antiquated and I'm struggling to think that this is the correct one.

# Answer
Have you checked the documentation for cron-jobs? You can use the `run` command.

```
$ kubectl run hello --schedule="*/1 * * * *" --restart=OnFailure --image=busybox -- /bin/sh -c "date; echo Hello from the Kubernetes cluster"
cronjob "hello" created

```

This way you can loop through your customers in a bash script to deploy your containers.

Not knowing the big picture its hard to answer completely, but I also think ansible templates would work for this. You could turn your above config file into a jinja2 template. You could pass your customers in as a JSON dict and use with\_items to loop through them or treat your customers as an inventory. This would give you an easy way to deal with certain "snowflake" customers.

EDIT: as far as doing this with Vanilla Kubernetes, I don't think it's possible. You could add your own custom controller and extend the kubernetes API. This would allow you to creat an API object that defines a customer, and then have a custom controller pick that up and create your cron job. Check out https://github.com/coreos/prometheus-operator as they have implemented such a solution to allow multiple teams to run there own Prometheus solution on the same cluster.
> 1 votes

---
Tags: kubernetes, automation, yaml
---


# Question
Title: Local Prometheus and Grafana Installations to Monitor a Kubernetes Cluster
We want to start our monitoring off the cluster on a local server. Is this practical and will it be feasible and reasonable to hook local Prometheus and Grafana installations into a Kubernetes cluster?

# Answer
It's possible but after doing many replatforming projects for clients I can't say I would advise it. Have you seen kube-prometheus? It's a one-shot install that will get you 90% of where you need to be for your cluster.

The only way I can advise it would be if you're going to run 5 or more clusters and want central monitoring, but I'm a bigger fan of having a Grafana per cluster.

Edit: Getting started with kube-prometheus is a one command install. `kubectl apply -f manifests/`
> 3 votes

---
Tags: kubernetes, prometheus, grafana
---


# Question
Title: Create multiple EC2 instances using python script (in loop iteration)
I Have JSON file in which email address and count of instances are specified.I'm using python script to iterate through that file and get username/count number (count number will be used as number of EC2 instances), output from JSON file:

```
DJukes
1
JWilson
2
eflame
3

```

So, for DJukes one instance will be created,JWilson 2,eFlame 3, also these names will be usernames for new instances.

Idea is to create as many instances as it's specified in JSON file based on username and count values

When creating manually i got

```
Terraform will perform the following actions:

-/+ aws_instance.win-example (new resource required)
      id:

                   "i-073bda5b30f10d3ba" => <computed> (forces new resource)

```

I passed these variables to python-terraform wrapper (https://github.com/beelit94/python-terraform)

If only one user/count exists in JSON file, no issues, but as soon i add more entries in JSON, then machine for DJukes is created then terminated,same for JWilson, only last created machine remains running. Can someone point out what's wrong with this code ?

```
#!/bin/python
import json
from pprint import pprint
from python_terraform import *

def myfunc():

  tf = Terraform(working_dir='/home/ja/terraform-course/demo-2b', variables={'count':count,'INSTANCE_USERNAME':user})
  tf.plan(no_color=IsFlagged, refresh=False, capture_output=False)
  approve = {"auto-approve": True}
  print(tf.plan())
  print(tf.apply(**approve))
  return

    json_data=open('./my.json')
    data = json.load(json_data)

    json_data.close()

    for i in range (0, len (data['customers'])):
        #print data['customers'][i]['email']
        k=data['customers'][i]['email']
        #print(k.split('@')[0])
        user=k.split('@')[0]
        #print(user)
        count=data['customers'][i]['instances']
        #print(count)
        #enter = int(input('Enter number of instances: '))
        myfunc()

```

windows.tf:

```
resource "aws_key_pair" "mykey" {
  key_name = "mykey"
  public_key = "${file("${var.PATH_TO_PUBLIC_KEY}")}"
}

resource "aws_instance" "win-example" {
  ami = "${lookup(var.WIN_AMIS, var.AWS_REGION)}"
  instance_type = "t2.medium"
  count="${var.count}"

  key_name = "${aws_key_pair.mykey.key_name}"
  user_data = <<EOF
<powershell>
net user ${var.INSTANCE_USERNAME} '${var.INSTANCE_PASSWORD}' /add /y
net localgroup administrators ${var.INSTANCE_USERNAME} /add

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

netsh advfirewall firewall add rule name="WinRM 5985" protocol=TCP dir=in localport=5985 action=allow
netsh advfirewall firewall add rule name="WinRM 5986" protocol=TCP dir=in localport=5986 action=allow

net stop winrm
sc.exe config winrm start=auto
net start winrm
</powershell>
EOF

  provisioner "file" {
    source = "test.txt"
    destination = "C:/test.txt"
  }
  connection {
    type = "winrm"
    timeout = "10m"
    user = "${var.INSTANCE_USERNAME}"
    password = "${var.INSTANCE_PASSWORD}"
  }

tags {
Name="${var.INSTANCE_USERNAME}"
}
}

```

# Answer
The main issue I see is that you are calling Terraform at the end of every python loop. This is creating new "state" file. That will make it really hard to manage what you are creating. This is how I would do it.

```
users = []
count = 0
for customer in data['customers']:
    username = customer['email'].split('@')[0]
    instances = customer['instances']
    count += instances
    users.extend([username] * instances)
    #print(Added {} to build list with {} instances.".format(username, instances))
myfunc(count, users)

```

then I would change...

```
def myfunc(count, users):

  tf = Terraform(working_dir='/home/ja/terraform-course/demo-2b', variables={'count':count,'INSTANCE_USERS':users})
  tf.plan(no_color=IsFlagged, refresh=False, capture_output=False)
  approve = {"auto-approve": True}
  print(tf.plan())
  print(tf.apply(**approve))
  return

```

and finally the Terraform.

```
resource "aws_key_pair" "mykey" {
  key_name = "mykey"
  public_key = "${file("${var.PATH_TO_PUBLIC_KEY}")}"
}

resource "aws_instance" "win-example" {
  ami = "${lookup(var.WIN_AMIS, var.AWS_REGION)}"
  instance_type = "t2.medium"
  count="${var.count}"

  key_name = "${aws_key_pair.mykey.key_name}"
  user_data = <<EOF
<powershell>
net user ${var.INSTANCE_USERS[count.index]} '${var.INSTANCE_PASSWORD}' /add /y
net localgroup administrators ${var.INSTANCE_USERS[count.index]} /add

winrm quickconfig -q
winrm set winrm/config/winrs '@{MaxMemoryPerShellMB="300"}'
winrm set winrm/config '@{MaxTimeoutms="1800000"}'
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'

netsh advfirewall firewall add rule name="WinRM 5985" protocol=TCP dir=in localport=5985 action=allow
netsh advfirewall firewall add rule name="WinRM 5986" protocol=TCP dir=in localport=5986 action=allow

net stop winrm
sc.exe config winrm start=auto
net start winrm
</powershell>
EOF

  provisioner "file" {
    source = "test.txt"
    destination = "C:/test.txt"
  }
  connection {
    type = "winrm"
    timeout = "10m"
    user = "${var.INSTANCE_USERNAME[count.index]}"
    password = "${var.INSTANCE_PASSWORD}"
  }

tags {
Name="${var.INSTANCE_USERS[count.index]}"
}
}

```

What I did was change your code so we now create a list of the users, that match the length of the count. That way we can call Terraform just once, and use the `count.index` on our users list to make sure we create the correct amount of instances per user.

You can read more about `count.index` here. I have always found reading and understanding more complicated code makes me better. Try reading some of the modules from the Terraform Registry. The vpc module gives good examples of using count.

If you still need help drop me a comment.
> 1 votes

---
Tags: amazon-web-services, amazon-ec2, terraform, automation, python
---


# Question
Title: Github plugin triggering to push event only for changes in specific branches
I am using Jenkins2 and GitHub plugin for triggering **Push** events.

I have the next service (Jenkins Github): http://myjenkinsmaster/github-webhook/

I also have 3 branches: `dev`, `stage`, `master` which need to be built.

But I require only the branch with changes to be built.

For example if changes were pushed to `dev`, only `dev` branch will be built, and others ignored.

However, Jenkins builds all 3 branches which I specified in branch field in job config.

How can I achieve a Jenkins build that builds only the branch where changes were pushed?

# Answer
> 1 votes
Finally I have resolved it with the help of "strategy of choosing what to build" with "Default behavior" for each branch.

---
Tags: jenkins, jenkins-plugins, github
---


# Question
Title: Is there a reason not to use Bamboo to run repetitive tasks instead of cron?
Is there a reason it would be a bad decision to use Bamboo ( or Jenkins ) to execute a scheduled task ( say every 15 minutes ) rather than using a cron job?

# Answer
Using task schedulers like Bamboo or Airflow have a lot of advantages over cron, particularly in their ability to recognize errors and process dependent tasks accordingly. By far the most common reason not to use them is that you simply haven't set up the infrastructure yet: some form of cron is installed on every Linux machine, and it's easy to add a new job in, whereas these more advanced schedulers require setting up fleets of machines, installing and configuring various pieces of software, and learning how to write new tasks.

Assuming that you already have this in place, though, there are a few usecases that cron handles better:

1. **Running a job on a specific machine.** You can do this with other schedulers, but they're generally designed for tasks that can be run on any one of a set of machines; when you want to do something like run a log rotation on every server in your fleet, that's much easier to set up with cron.
2. **Running time-sensitive tasks at exact times.** Distributed schedulers will schedule a job to run, but it may not actually run for some period of time, depending on what else is in the queue. Cron doesn't have a queue, and so a task that needs to run every minute will run every minute (or at least as close to that as the OS scheduler will allow).
> 3 votes

# Answer
Possible reasons for using cron over jenkins

I am assuming you mean cron on a Linux system and not a Kubernetes CronJob Controller (https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/)

1) User permissions. Although you can work around this with Jenkins, it is possibly easier to run commands as arbitary users in cron

2) Provisioning of commands. You might want to be able to set up the commands with Puppet using a cron type. Again, I'm sure Jenkins can be configured in this way but it's possibly easier in some circumstances to configure cron

3) Lightweight. All Linux/Unix hosts run cron as a basic service that is always available. Whereas Jenkins requires extra non standard items to be installed and running
> 1 votes

---
Tags: jenkins, bamboo
---


# Question
Title: How to automate rolling upgrades for ec2 running linux behind elastic and/or application load balancer?
Would like to ask for recommendations on ready automated solutions or templates for rolling ec2 instance upgrade on AWS, requiring a reboot with ELB/ALB. Specifically required to perform the maintenance upgrade for the kernel on hundreds of Linux machines.

For each ELB/ALB steps are:

1. Disconnect ec2 instance from ELB/ALB wait for connection draining
2. Upgrade kernel and reboot machine (not part of automation, e.g. running external ./upgrade.sh)
3. Wait for instance to be online, connect to ELB/ALB, wait for health check status
4. Continue to next instance in the load balancer

Note: not looking for migration to the solution similar to Netflix Spinnaker or immutable deployment strategy implementation, but for general and simple on-demand maintenance tool.

# Answer
> 2 votes
AWS have a specific service to deal with this type of use case, EC2 Systems Manager

This allows making base patch sets and then checking they are applied to target systems.

> "Patch Manager is a fully-managed service (provided at no additional cost) that helps you simplify your operating system patching process, including defining the patches you want to approve for deployment, the method of patch deployment, the timing for patch roll-outs, and determining patch compliance status across your entire fleet of instances. It’s extremely configurable with some sensible defaults and helps you easily deal with patching heterogeneous clusters."

-- "Amazon EC2 Systems Manager Patch Manager now supports Linux"

You'll need to install an SSM agent on each system and add a role but other than that it should be unobtrusive

# Answer
> 1 votes
As people in the comments are saying, you can do this with aws-cli in bash or in the python boto3 library.

What about ansible? It has the ability to see your ec2 instances and it could handle the automation for maintenance tasks. If you are not using ansible I highly recommend you add that tool to your DevOps toolbox.

Here is how you might structure your playbook to do what you are wanting.

---
Tags: amazon-web-services, amazon-ec2, automation, linux, aws-elb
---


# Question
Title: How do I run Ansible Azure playbooks while avoiding storing credentials in files?
## Background

1. We use Ansible to provision and manage Azure infrastructure. At the moment we run Ansible "manually" i.e. we *manually execute playbooks* for various automated tasks. No CI infrastructure.
2. Probably not relevant but we manage our inventory using dynamic script `azure_rm.py`.
3. We are encouraged to be as secure as possible i.e.
    1. Don't store Vault passwords in `~/.vault_pass` or in any local file
    2. Don't store Azure secrets in `~/.azure/credentials`
    3. Don't store anything secure in `.bashrc`.

In such a scenario, I am having trouble coming up with a coherent strategy to ensure that my playbooks can access Azure secrets, while following the guidelines above.

## Question

How can I avoid storing Ansible Vault and Azure credentials on files, while still ensuring my playbooks can access them?

## What I've tried

So far I have come up with a wrapper script that

1. asks the user for Vault password
2. Uses that to decrypt a Vaulted Shell script
3. Evaluates the script, which loads Azure environment variables into the environment;
4. Runs the playbook on the environment that has been thus set.

Any better (more elegant, less complicated, more "Ansible") solutions out there?

# Answer
> 8 votes
### Vault password

First of all, you should get familiar with the fact that vault password file can be executable script. In this case Ansible executes it and expects to receive password as its output.

For example you can use `gpg-agent` or `keychain` to store your actual password and unlock it when required. Read more in this blog post: https://benincosa.com/?p=3235

If you are a bit paranoid, you can add notification when your password script is called, like this:

```
#!/bin/bash
PARENT_PROCESS=$(ps -p $PPID -o args | tail -n 1)
osascript -e "display notification \"Vault password used by ${PARENT_PROCESS}\" with title \"Ansible\" sound name \"default\""
gpg --batch --use-agent --no-tty --decrypt key.gpg 2>/dev/null

```

This vault password script uses `key.gpg` as actual vault key and also shows popup notification (for MacOS) with parent process name when script is used. Gpg-agent caches unlock password for some time, so there is no need to enter password every time you start playbook.

Just set `vault_password_file = ./vault_pass.sh` in your `ansible.cfg`.

### Environment

You said that you use `azure_rm.py` as dynamic inventory script. This means that you have to set credentials into your environment variables before you start ansible-playbook for it to be able to use them.

You can make two files:

`secure_env` (encrypted with vault):

```
export AZURE_SECRET=xxx;export AZURE_SUBSCRIPTION_ID=xxx;

```

`set_env` (plain text):

```
echo -n "Setting secure vars... "
eval $(ansible-vault view secure_env)
echo "done."

```

When you open new terminal to execute your automation tasks, you have to run:

```
source set_env

```

At this moment, bash evaluates `set_env` and `secure_env` (decrypted via ansible-vault). After this command you have Azure credentials defined for the current shell, so you can execute playbooks as usual:

```
ansible-playbook provision-my-azure-instances.yml

```

So using this two approaches, you can store `key.gpg` and `secure_env` in your repository; then in the new terminal call `source set_env` once, enter gpg password once (to unlock future use of key.gpg); then call `ansible-playbook` as many times as you like without any passwords.

# Answer
> 2 votes
Please read https://docs.ansible.com/ansible/2.4/vault.html Since Ansible 2.4 one could use `--vault-id @prompt`.

Encrypt a file using ansible-vault:

```
ansible-vault encrypt /path/to/encrypted/file

```

Run the playbook and it will result in:

```
fatal: [localhost]: FAILED! => {"msg": "A vault password or secret must be
specified to decrypt /path/to/encrypted/file"}

```

There are multiple options to decrypt files, including `@prompt`:

```
ansible-playbook some-playbook --vault-id @prompt

```

will prompt:

```
Vault password (default):

```

Once the vault password has been entered, the playbook should succeed.

---
Tags: ansible, ansible-vault
---


# Question
Title: How to explain the Twelve Factors to non techy people?
The Twelve Factors is a sort of manifesto seen as a road-map for developers to follow when building modern web-based applications.

How would you explain each of the 12 factors in less than three sentences to random people?

# Answer
As far as I can read between the lines, you want your upper management to support you in introducing the 12-factor approach to your company's software development.

In that respect, you do *not* want to explain each of the 12 factors to them. If they really were interested in that, then you could just send them the 12-factor website you have linked in your question - it already has 1-sentence summaries.

I'll answer the question you wrote in the title:

> How to explain the Twelve Factors to non techy people?

The Twelve Factors are not a soft "culture". It is a set of concrete, practical and pretty low-level techniques for solving problems relating to configuration management, deployment and high availability of applications. Instead of "culture" or "methodology", I would rather call it a pattern, or a kind of handbook for developers. It is obviously related to DevOps, cloud and similar buzzwords, but not necessarily so - you can also employ the techniques if you only have one developer doing his own thing, and even if there is no cloud or otherwise "modern" environment involved at all.

It is not even necessary to involve management in the question whether to employ some or all of the techniques mentioned in the 12-factors. Many of them are simply best practices which everybody who develops ("service"-like) software will end up over the years. This is pretty evident in the *I. Codebase*, *III. Config*, *XI. Logs*. No manager is required to give a green light to employ these practices.

Other of the principles, namely *II. Dependencies*, *IV. Backing services*, *VI. Processes*, *VII. Port binding*, *VIII. Concurrency*, *IX. Disposability* are neatly combined into the keyword *"microservices"*. Don't explain to your management those specifics, but tell them that you are going to make a move from some presumably monolithic application architecture you had before to a microservice based one. On a management level, this can be sufficiently done with a graphical representation (one big blob of complex software on the one hand - several individual small blobs of software on the other hand). You can explain why the later can be better for the job at hand with nontechnical terms; every manager can understand the concepts of dividing a big, convoluted mass of code into smaller pieces that can be handled (and scaled) much easier.

Your management will want to know what it costs and how long it takes. 12-factor tells you nothing at all about that. While there is a slight cost involved in some of the factors (e.g., establishing a central log stash), the cost is minor and will easily be caught up by savings later. Other costs may be the time needed for your developers to embrace some of the more tougher parts (i.e., II, VII, IX, depending on how much existing software you already have).

Very important: it is pretty simple to work in an environment with a big monolithic application, and only add *new* stuff with the full 12-factor approach, or convert bits and pieces from the original software as you go. This might be more sensible if you have time or money constraints.
> 6 votes

# Answer
> **I. Codebase**
>
> One codebase tracked in revision control, many deploys

Version Control System (VCS)

> **II. Dependencies**
>
> Explicitly declare and isolate dependencies

Use a package manager tool

> **III. Config**
>
> Store config in the environment

Environment variables

> **IV. Backing services**
>
> Treat backing services as attached resources

The code should not change if another service like a database changes.

> **V. Build, release, run**
>
> Strictly separate build and run stages

* unique release id
* a new release must be created for every change

> **VI. Processes**
>
> Execute the app as one or more stateless processes

...

> **VII. Port binding**
>
> Export services via port binding

...

> **VIII. Concurrency**
>
> Scale out via the process model

...

> **IX. Disposability**
>
> Maximize robustness with fast startup and graceful shutdown

Disposable. It does not matter whether services go down. The platform that consists of multiple services, is able to continue if services go down, e.g. data will not be lost. When the services are rebooted the platform can continue without data loss.

> **X. Dev/prod parity**
>
> Keep development, staging, and production as similar as possible

* Save time: one does not have to think, where does the acceptance database reside or where does this app exist in the testing environment. If one applies the same structure for systems in dev, test and acc then one does not have to think every time when such an environment is used and this will save time
* Decrease number of bugs: if a development environment is similar to production, the number of bugs will be decreased as there is a higher chance that the bugs will be solved in the dev phase proactively

> **XI. Logs**
>
> Treat logs as event streams

* stdout
* store all logs in a log manager like splunk

> **XII. Admin processes**
>
> Run admin/management tasks as one-off processes

...

---

**Discussion**

> we are a startup of 6 devs our boss is a former sportsman

...

> **Who should read this document?**
>
> Any developer building applications which run as a service. Ops engineers who deploy or manage such applications.

...

https://en.wikipedia.org/wiki/Software\_rot

> slow deterioration of software over time that will eventually lead to it becoming faulty \[or\] unusable" and, importantly, that "the software does not actually decay, but rather suffers from a lack of being updated with respect to the changing environment in which it resides.

...

**Manager:** Why do you want to apply 12factor?

**Team:** The 12factor prevents software rot, i.e. more bugs, unstable software resulting in less customer satisfaction, less revenue.

---

**Conclusion**

Instead of explaining the 12factor it is better to explain what will happen if the 12factor is not applied, i.e. software rot. The 12factor is for Devs and Ops people, not for managers.

---

**References**

1. http://www.clearlytech.com/2014/01/04/12-factor-apps-plain-english/
2. https://12factor.net/
3. https://thenewstack.io/12-factor-app-streamlines-application-development/
> 4 votes

---
Tags: culture, methodology
---


title: Git Files Encryption for Business

text:
# Question
Title: Git Files Encryption for Business
I am using git and have the following problem: In the company where I work you now want to protect the source code from theft. So if for some reason a developer lets the source code of a project go along and sells it, for example. To prevent this, I thought about it for a long time and came to the conclusion that encrypting the source code is the best choice. Like git-crypt does, that's a very good idea. Only with one mistake: If I add a user's GPG key, he can decrypt all files, and the protection measure would be worthless. The files of the developer should be decrypted locally and only remotely. (This way the developer does not always have to encrypt, push, decrypt the files)

I tried to re-program git-crypt in Python (because of cross-platform) but then failed due to the git attributes/smudge, clean filter drivers.

Do you know a possibility or an application that takes up exactly my problem?

# Answer
There really is no way to take on your problem with git, which is to deny a developer with access to the code (required to work with it) the possibility to steal the code. This is because as soon as your developer has the code in a readable format (to program on it), he can do with it whatever he wants, anyways.

git-crypt has a totally different use, and is not related to your problem at all.

To solve your problem, there is basically only one possibility: lock the network in which the code resides down completely - make it so that there is no physical way to transfer the code to the outside. This could mean:

* No direct internet connection, obviously.
* No http/https proxy either, no mail clients etc.
* No way to add new devices to the network (unless through official ways). E.g., no unused network ports to plug anything in; no guest DHCP; MAC address filtering; no WIFI, smartphones; etc.
* No way to plug an USB device or other transfer mechanism into any devices.

This may sound harsh, but really is not. It could be as simple as keeping your code strictly within a set of VMs, accessed by something like Vmware Horizon or something like that, with cut&paste and file transfer disabled, of course. Be nice to your developers and allow them to transfer stuff *into* the environment (their favourite editor, for example).

The measures so far do not require any physical changes, i.e. your developers still can work remotely or in their usual unsecure office. Obviously, they still can go to extreme measures (screenshots on their client PC...) to get specific parts of the code page-by-page.

To go a step further, in increasing severity (although we are going into James Bond territory here...):

* Strict policy with severe punishment. Setup a specific contract which each emplyee which contains clauses as harsh as the laws of your country allow.
* Switching access to the limited network to a physical area in your office.
* Security measures like locked doors accessible to only specific personnel, video cameras with 24/7 human supervision and so on.
* 2-factor access (either 2 persons required to checkout code, or at least a physical TAN generator or something along those lines).

Note that in your case you want to make it hard for a thief who knows that he is doing something illegal and already has committed to doing a crime. He will not respect any "weak" protection (like rules/laws) anyways.
> 9 votes

---
Tags: git, python, encryption, git-hooks
---


title: Consul with Nginx, not using docker

text:
# Question
Title: Consul with Nginx, not using docker
I am from development background and totally new to DevOps, I want to setup **consul for service discovery** for my micro-services, services are reverse proxied by **Nginx**. Now I want to configure Nginx with the consul, we are not using dockers as of now.

I have gone through many tutorials but I didn't get much about that how can I configure Nginx with the consul.

Any lead would be appreciated.

# Answer
> 2 votes
I have done this in the past using https://github.com/hashicorp/consul-template.

What `consul-template` does it generate a configuration file (for nginx) based on a certain template you provide. And the values that it fills into this template are coming from the configuration stored in `consul`.

Each time your micro-services register themselves in `consul` and need to be addressed in the nginx configuration file, `consul-template` does it for you by changing the file and sending a `SIGHUP` (`kill -1`) to the nginx process that makes it reload nginx configuration.

The repository for `consule-template` contains an example of managing `nginx` https://github.com/hashicorp/consul-template/blob/master/examples/nginx.md

---

If you are using *Nginx Plus*, which is the commercial version of Nginx. Then it has support for Consul built-in. https://www.nginx.com/blog/service-discovery-with-nginx-plus-and-consul/ and https://www.nginx.com/blog/service-discovery-nginx-plus-srv-records-consul-dns/ go into depth on this subject.

---
Tags: microservices, nginx, consul
---


title: AWS REST API Query

text:
# Question
Title: AWS REST API Query
How to authenticate against AWS to make API query ?

```
https://iam.amazonaws.com/?Action=ListAttachedUserPolicies&UserName=ja&Version=2010-05-08&accesskey

```

and getting:

```
<Code>MissingAuthenticationToken</Code>
<Message>Request is missing Authentication Token</Message>

```

# Answer
> 2 votes
Got it working by aws curl https://github.com/okigan/awscurl

```
awscurl --service iam 'https://iam.amazonaws.com/?Action=ListUsers&Version=2010-05-08'

```

# Answer
> 0 votes
The referred documentation from the question https://docs.aws.amazon.com/IAM/latest/APIReference/API\_ListAttachedUserPolicies.html

has this example of a request:

```
https://iam.amazonaws.com/?Action=ListAttachedUserPolicies
&UserName=Alice
&Version=2010-05-08
&AUTHPARAMS

```

AUTHPARAMS are explained https://docs.aws.amazon.com/general/latest/gr/signature-version-2.html

(version 4 is preferred) https://docs.aws.amazon.com/general/latest/gr/sigv4-calculate-signature.html

So you have to calculate a digest from the request together with your secret key and you have to do it exactly as it is explained in the documentation - which means be careful about every letter including spaces.

---
Tags: amazon-web-services, api
---


# Question
Title: How to find out the used Key pair name via AWS CLI?
I'm aware of `get-password-data` command which can display PasswordData data like:

```
$ aws ec2 get-password-data --instance-id i-0123456789
{
    "InstanceId": "i-0123456789",
    "PasswordData": "\r\nAOh...xg==\r\n",
    "Timestamp": "2018-03-27T16:52:04.000Z"
}

```

however I'd like to know which *Key pair name* was used to launch the instance, so I can pass it using `--priv-launch-key` in order to decrypt the password.

# Answer
For a given instance, you would first use `aws ec2 describe-instances` to get the information JSON for your instance.

The information also contains the keypair name used to create that instance.

E.g. for an instance `i-0e2x8xd7xxx` (Note: I use the awesome tool `jq` to do JSON parsing but you can use any other solution)

```
aws ec2 describe-instances --instance-ids i-0e2x8xd7xxx | jq '.Reservations[].Instances[].KeyName'

```

Output:

```
"my_key_name"

```

You can store that in a variable, say `$keypair_name` and then pass it into your `aws ec2 get-password-data` command.

You would also need to pass in the path on your machine where your keypairs are located e.g. `$keypair_path`.

For example:

```
aws ec2 get-password-data --priv-launch-key $keypair_path/$keypair_name .....

```
> 7 votes

# Answer
Based on @Vish answer, I've created the following shell script:

```
#/usr/bin/env bash
# Script to show password data of the EC2 instance.
[ $# -eq 0 ] && { echo "Usage: $0 instance_id ..."; exit; }
keyname=$(aws ec2 describe-instances --query 'Reservations[].Instances[].KeyName' --output text --instance-ids $1)
pemfile=$(find ~/.ssh -name "*$keyname*.pem" -print -quit)
if [ -z "$pemfile" ]; then
  aws ec2 get-password-data --instance-id $1
else
  aws ec2 get-password-data --instance-id $1 --priv-launch-key $pemfile
fi

```

which aims to decrypt the password based on the key pair name. By default it's looking for the PEM file in user's `~/.ssh` folder.

Usage:

```
./show_ec2_password_data.sh i-instance_id

```
> 4 votes

---
Tags: amazon-ec2, aws-cli
---


# Question
Title: How can I clean/delete a Jenkins workspace after merge?
I would like to keep a workspace in Jenkins until after the pull request is merged. Meaning, I don't want to clear the workspace after every build, but I also don't want to clear the workspace only when the branch gets deleted.

How can I use the github webhooks to trigger a workspace cleanup whenever a pull request gets resolved as merged?

Ideally, the solution would involve using my JenkinsFile/Pipelines.

edit: Background.

We have long-lived branches and short-lived branches. For brevity, I'll call the long-lived branches "team branches". Our workflow is to take short-lived branches, such as feature branches and make a PR request to pull them into the team branches.

On Jenkins, we then build the feature branch and the PR request and the team branch. Normally this works fine, but sometimes when multiple teams are merging to qa each team and feature branch needs to update(pull) and then get built on Jenkins again. Currently, we are always cleaning the workspace which makes the update builds take longer than they need to. So I want to be able to preserve the workspace until all updates are done and the PR request actually gets merged, only then do I want to delete the workspace for the feature branch and the PR request and the team branch.

# Answer
I am assuming you are using Jenkinsfile. You can do for example:

```
pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                echo 'Building..'
            }
        }
        stage('Test') {
            steps {
                echo 'Testing..'
            }
        }
        stage('Deploy') {
             when { branch 'master' } /* when in master branch */
             steps {
                echo 'Deploying....'
                deleteDir() /* clean up our workspace */
            }
        }
    }
}

```
> 4 votes

# Answer
Gitlab seems to have a solution although the github webhook on MR should trigger an event in Jenkins. It does on travis. Configuring the Jenkins job seems to be well documented in the gitlab
> 0 votes

# Answer
There are ways to detect using Groovy if a build is say part of a multibranch/non master branch and then you can say on successful build output perform a workspace delete.

How about adding more info around what you want to preserve?
> 0 votes

---
Tags: jenkins, jenkins-pipeline, github
---


title: consul throwing an error

text:
# Question
Title: consul throwing an error
Trying to run consul on my system.

Have followed link to download consul:

consul and consul-template setup

```
consul agent -server -bootstrap-expect 1 \
    -data-dir /tmp/consul -node=agent-one \
    -bind=127.0.0.1:8500 \
    -client=0.0.0.0 \
    -config-dir /etc/consul.d \
    -ui-dir /opt/consul/

```

But getting an error:

```
WARNING: BootstrapExpect Mode is specified as 1; this is the same as Bootstrap mode.
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
==> Error starting agent: Failed to start Consul server: Failed to start lan serf: Failed to create memberlist: Failed to parse advertise address!

```

# Answer
> 3 votes
It looks like that the official Consul documentation is more up to date than the link that was posted in the question.

If one starts with the Getting Started guide, then one could start a consul agent in no time and the documentation also explains how to proceed.

```
user@localhost /tmp $ ./consul agent -dev
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v1.0.6'
           Node ID: '777c5475-bc78-6e38-75a6-42038eef7cb8'
         Node name: 'localhost.localdomain'
        Datacenter: 'dc1' (Segment: '<all>')
            Server: true (Bootstrap: false)
       Client Addr: [127.0.0.1] (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: 127.0.0.1 (LAN: 8301, WAN: 8302)
           Encrypt: Gossip: false, TLS-Outgoing: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2018/04/09 21:33:04 [DEBUG] Using random ID "777c5475-bc78-6e38-75a6-42038eef7cb8" as node ID

```

---
Tags: consul
---


# Question
Title: Chef Provisioning for canary deployment and orchestration
I am looking for best tool that, does below jobs for deployment.

**Configuration:**

1) Configuration of deployment node machines in specific environment

2) Configure no of service instances to be alive in environment at all time

**Deployment:**

Now, Just doing above configuration. When I trigger deployment of N services. It will randomly pick up nodes from deployment environments and will start total N services.

**Multiple Services:**

If I have 2 nodes and I want to bring up 4 services, it should bring up 2 services on each node.

**Service Failure recovery:**

If any machine goes down or any service goes down in any node. It will bring up a new service in any of environment node.

**Scenario to handle:**

Choosing nodes at random used in distributed system with system discovery/failure.

**1) Service deployment :**

I required 2 services of calculator, and other services as well. And I have total 4 machines. Now 2 calculator services will be deployed in any node.

**2) Service discovery/failure:**

These 2 calculator services will be registered with service registry. Now my application will query to service registry, and service registry will provide node details with service running. Same in failure case, service will be started in any node, and discovered by application through service registry.

# Answer
> 1 votes
You should take a look at BOSH. Its the tool that is used by CloudFoundry, its services and a distro of Kubernetes called Kubo for installation, management and "Day 2" operations.

It's basically a declarative, cloud-agnostic orchestration tool that features rolling updates, canary deployments, scaling, monitoring and self healing. It can monitor VMs as well as processes on those VMs (i.e. services) and will make sure that the deployment is running as you specified it to in the deployment manifest.

In order to do all of this (especially the monitoring and self-healing bits) it has a client-server architecture which is deployed with a cut-down version of BOSH itself called bosh bootloader or `bbl` for short. You use this to deploy the BOSH director. You talk to the bosh director by installing the bosh-cli (brew install bosh-cli on a mac).

For you deployment you would first need to create what's called a BOSH release and this can, admittedly, be a little daunting if you are not familiar with BOSH but as CF, its services and Kubo are all open source there are tons of references out there. There are also lots of pre-backed releases and stemcells (OSes).

After creating your release you upload it to your bosh director and `bosh deploy`. To upgrade it you upload the next version of your release and `bosh deploy`. To patch a security vulnerability you upload the latest blessed stemcell from bosh.io and `bosh deploy`. I am sure you get the picture.

If you want to go the next level then there is a good getting started guide here.

---
Tags: chef, orchestration
---


# Question
Title: Apache Mesos - cases pro and contra containers?
My current impression is, Apache Mesos was introduced before the LXC container technology became *easily* available (via Docker).

So you could implement "your own Apache Mesos Java framework" for example, or use Marathon to do that.

Now, Marathon seems to support containers as well.

Question - when would you deploy Java nowadays on/with Mesos/Marathon without contauners? And if you use containers, why should you opt for Mesos instead of for example Kubernetes?

For example, here they say in my understanding, k8s ecosystem deployment is easier while with Mesos you would have to opt for the enterprise version of DC/OS.

# Answer
Your impression is false, linux containers (LXC) exists since 2008, mesos and docker have started later. All three makes use of cgroups available since kernel 2.6.28.

For your overall question, mesosphere have a blog post about it.

But mainly you're looking at it from the wrong point of view, choosing the underlying orchestration system should not be done from 'how to deploy this application?' point of view, but more on 'What would be the best orchestration for what I have to deploy as a whole ?'. Making the choice between swarm/kubernetes/mesos/k8s on mesos/swarm on mesos/whatever is very dependent on your environment, your existing knowledge, your team skills etc. and we can"'t really help you choose without an extended description on what you're willing to manage.
> 1 votes

---
Tags: containers, orchestration
---


# Question
Title: What are Design Patterns for creating images?
In Object Oriented Programming (OOP) there are Design Patterns (DP). DP are best practices for tackling certain problems. For example, a singleton is acceptable to be used in a logger class.

What about applying DPs to creation of images, e.g. docker. In our situation we have the core, i.e. the application, application configuration (customization per customer), and different artifacts per customer.

At the moment, a bash script is used that contains various if-else statements and it is getting more and more a spaghetti.

# Answer
The general design pattern is one container = one running service - see the initial comment at https://docs.docker.com/config/containers/multi-service\_container/

Now, after spending about a year developing a multi-service application relying heavily on docker this is what works for me:

Avoid creating files inside the container unless it is your storage engine - in which case use volumes (recommended by Docker and also common sense) - we use Elasticsearch for that.

If I need multiple custom configurations I am trying to either load it from somewhere (DB, Environment) or I mount the configuration as a volume like

```
-v config.yml:/usr/src/app/config.yml

```

In general, we have one service = one repository, but sometimes you have to copy all the dependencies in order to just create one script. So we have one repository where these scripts are aggregated and I change the CMD with an environment variable like this:

```
CMD cd /usr/src/app && python -u $RUN_SCRIPT

```

It can be probably improved with a combination of ENTRYPOINT and CMD as described here http://goinbigdata.com/docker-run-vs-cmd-vs-entrypoint/

Then we use various `docker-compose.yml` files for getting the setup we need.

For some operations (like DB setup) we either run one container like

```
docker run --network ournetwork init-profiling

```

or I can run a command inside that container

```
docker run -it --network ournetwork init-profiling bash
# run_command.sh
^D

```

We spent some time minimizing the footprint of containers, they can quickly get bloated. For example, we use a common base image for our python containers. Also, things like `pip freeze` do make sense only if you use a separate virtual environment for each repository/container.

We do not use swarm/kubernetes for now, so I cannot add any additional guidelines for that.
> 3 votes

---
Tags: architecture, images
---


title: use of nginx server_name

text:
# Question
Title: use of nginx server_name
I am new to nginx, I have configured my consul with nginx.

It's working but I have few doubts in my configuration.

here is my configuration

```
upstream consul {
  server 127.0.0.1:8500;
}

server {
  listen 80 default_server;
  server_name localhost;

  location / {
    proxy_pass http://consul;
  }
}

```

this is working fine if hit http://localhost, in my machine I am able to see consul UI.

But If I change my location to **/consul**, it is giving 404.

and what is the use of server name, if I change it to **server\_name mylocalserver;**

I think I should be able to run my gttp request on this address http://mylocalserver. but that is also not working.

# Answer
> 1 votes
1. You need to add a host entry of `mylocalserver` in the host file against `127.0.0.1`, like `127.0.0.1 mylocalserver`.
2. For `/consul`, there should be a `/consul` endpoint in your code to serve your request.

As per your configuration, your `http://localhost` will be served because you have set proxypass for `"/"(all)`. It will send all requests to the application/server and will be served.

---
Tags: nginx
---


# Question
Title: How to get Ansible playbook ouput in one line?
In Ansible we are running a playbook using `ansible-playbook playbookname.yaml`. The output will be multiple lines (like below). I need output in single line. How can I get that?

```
PLAY [localhost] **************************************************************
GATHERING FACTS ***************************************************************
localhost                  : ok=3    changed=1    unreachable=0    failed=0

```

# Answer
> 5 votes
That's still a XY problem IMHO, why do you want an output in a single line at all ? what's the need for it ?

If you can't explain more and wish to footgun yourself by replacing newlines with spaces the use `command | tr '\n' ' '` (you'll probably end up loosing the end of the log).

If your goal if to parse the last line then

`command | awk '/ok=/ { for (i=3; i < 7; i++) { split($i,a,"="); printf a[1]":"a[2]" " } }'`

will give `ok:3 changed:1 unreachable:0 failed:0`

This select the lines containing `ok=` and split each `key=value` field (3 to 6) on the `=` char, thus giving the key as first element of array `a` and the value as the second element.

use `print` instead of `printf` to get each state on separate lines.

# Answer
> -1 votes
I am not sure if that is possible or not but you can write same command in bash script

```
#!/bin/bash

ansible-playbook playbookname.yaml > output.txt

# Then grep from output.txt what you want

grep -A 5 "PLAY" output.txt

```

May not be good way but will achieve what you want.

---
Tags: ansible
---


# Question
Title: How do I make read only mount in Docker container writable?
When developing code on a very large repository, I would like to run multiple builds/tests in parallel in containers. I need to get my repository or some other large directory into the container. If I mount it read/write, the parallel processes will fight each other and I will need to be able to clean after them, there is also issue with privilege escalations on those files. If I mount it read/only then I cannot run processes that write files inside the directory.

Docker suggest to build a new container with a COPY of the repository, save it as new layer and then start several containers from this new image. Having to make a copy of megabytes of data every time you want to run tests is making it take too long though.

How can I solve this efficiently?

# Answer
Solution to this problem is to create an overlay filesytem over a read-only mount, but if you try to do it directly, overlay will refuse to put upper and work directories on another overlay filesystem. The trick is to create a tmpfs for the upper and work directories like this:

Create a script called run-in-c.sh

```
#!/bin/bash
NAME=$1
shift
HOSTNAME=dock${NAME}
CONTAINER=dc-${USER}-${NAME}
REPOSITORY=${HOME}/repository
BASEIMAGE=hub.docker.io/my-org/my-base-container
OVERLAY=/mnt/overlay
LOWERDIR=/mnt/lower
UPPERDIR=${OVERLAY}/upper
WORKDIR=${OVERLAY}/work
TARGET=/mnt/repository
# PRIVILEGED="--privileged"
PRIVILEGED="--cap-add SYS_ADMIN"
docker container create --name $CONTAINER $PRIVILEGED --hostname $HOSTNAME --volume ${CO}:${LOWERDIR}:ro $BASEIMAGE
docker container start $CONTAINER
docker container exec $CONTAINER mkdir -p $OVERLAY
docker container exec $CONTAINER mount -t tmpfs tmpfs $OVERLAY
docker container exec $CONTAINER mkdir -p $WORKDIR $UPPERDIR $TARGET
docker container exec $CONTAINER mount -t overlay overlay -o lowerdir=${LOWERDIR},upperdir=${UPPERDIR},workdir=${WORKDIR} $TARGET
docker container exec -it --workdir $TARGET $CONTAINER $*
docker container stop $CONTAINER
docker container rm $CONTAINER

```

Then you can run commands in the container as:

```
run-in-c.sh test01 'cd dir && command args'

```

or simply get interactive shell with:

```
run-in-c.sh naming-stuff-is-too-hard bash

```
> 2 votes

---
Tags: docker, testing
---


# Question
Title: Who is responsible for running and maintaining shared services in DevOps
As a member of a small agile development team developing an application heavily relying on Kafka I wonder what implications the DevOps paradigm of *You build it, you run it* has regarding running and maintaining a shared service like Kafka used by other teams as well.

Does the *You build it, you run it* not only revolve around our application?

Does DevOps really mean *you are using another service so you have to manage that service as well*?

# Answer
> 5 votes
I think we need to view this in the larger context of Flow and Systems Thinking which is one of the core values of DevOps.

1. Specifically, DevOps Values require an understanding that each individual is part of a larger system and work toward increasing the flow of value through that system to the customer.
2. DevOps also focuses on helping the organization set a shared vision and work incrementally and collaboratively on achieving that vision.

In such a scenario, the focus needs to shift from "technology-specific silos" to "cross functional teams". And each DevOps practitioner needs to achieve a T-Shaped Skillset with core as well as shared responsibilities.

**So, to answer your question:**

> Does DevOps really mean you are using another service so you have to manage that service as well?

I think it depends.

If your being able to manage the service leads to *increased flow of work through the pipeline to the customer* then yes.

If however, it leads to you becoming a bottleneck or constraint to flow (perhaps due to you becoming overloaded with additional responsibilities), then no.

If you are able to share this responsibility with others than manage the service (and have a Systems and Flow mindset), and *collaborate* with them so that none of you get too overloaded with work to become a constraint to flow, then yes.

References:

1. DevOps Values
2. YBI/YRI
3. What is DevOps
4. Cross-Functional Teams and DevOps

---
Tags: terminology, sre
---


# Question
Title: Making thousands of microservice/dockers with different IPs
I need to simulate many connections. Currently using curl. What is the best way to deploy thousands of such requests with different IPs using some script?

# Answer
> 6 votes
# Linux, without docker

In Linux, you can create arbitrarily many IP address for each interface. A comment in https://serverfault.com/questions/328146/max-number-virtual-ip-addresses-per-nic reports success with 2000, and hearsay of 5000 successful IPs.

So. Pick any Linux box on your intranet, which could of course be a VM, and create as many IPs on its single ethernet interface (MAC address) as you wish.

Say you're using the 10.0.0.0/8 private network, then you can execute lots of these:

```
ifconfig eth0:1 10.0.0.2 netmask 255.0.0.0
ifconfig eth0:2 10.0.0.3 netmask 255.0.0.0
...
ifconfig eth0:260 10.0.1.6 netmask 255.0.0.0
...

```

You should be able to use `curl --interface eth0:3` (or maybe `--interface 10.0.0.3`) to use a specific source address. For `wget`, it's `--bind-address=10.0.0.3` etc. Start the curls in the background ("&") and they will run more or less concurrently - or look for some test driver (different question) to run them for you.

This will give you lots of requests from separate IP addresses, but all conveniently running on one machine.

# Docker, trivially

With docker: just make an image, as small as humanly possible, which connects to your server. You can use `curl` without any special options.

Then fire up your 1000 docker containers, just like you usually would. Each will automatically get an IP address on the internal docker network (172.x.0.0/16 by default).

https://stackoverflow.com/questions/21799382/is-there-a-maximum-number-of-containers-running-on-a-docker-host indicates that it is possible with hardware that is not too uncommon today; a naive 1000 docker containers used roughly 3 GB of RAM for them. That question also has some hints on options you can use to decrease the memory needs. Aside from memory, there won't be too much overhead, i.e. CPU wise it should comparable.

# Docker, with special networking

`docker network` opens up manually created networking for you. You should be able to cobble something together (just check the individual `--help` pages, it's all pretty self explanatory), and combine that with the `ifconfig` approach explained above. This would mean that you only need one container for your `curl`s, and start 1000's of them inside that container.

I would try to just run your 1000 containers, first, though, unless you are limited in the RAM department, for ease of use.

# About the `curl`...

Whatever you do, don't just start a single `curl` in each instance, but have each runner be an endless loop of back-to-back `curl` calls. This way, you avoid long gaps inbetween where your individual jobs are being started and stopped; this is obviously especially important for the "1000 docker containers" variation. In the answer linked above, they note that it took them some good amount of minutes to start up those thousands of containers, so you do not want to do that over and over again.

And you might as well get rid of `curl` altogether and create small networking app in the language of your choice (Perl, Ruby, C, Java...) to avoid process startup overhead.

---
Tags: docker, microservices
---


title: How can I become a DevOps Engineer?

text:
# Question
Title: How can I become a DevOps Engineer?
Please suggest me a right path to this DevOps career. From blogs Im not getting a well defined path. What all courses I have to take.

# Answer
**DevOps** is not a role and in words of **@Tensibai** here:

> DevOps is roughly an organization pattern aiming at breaking the silo between developers and sysadmins.

So, you could already be on the DevOps moto and if you don't think you are, you can probably just try to implement the DevOps pattern on your organization or move to another organization with that pattern already implemented.

You may want to become an Infrastructure Engineer, a SysAdmin Engineer, an Automation Engineer, a Site Reliability Engineer, a Performance Engineer... but surely you can not become a DevOps Engineer.

You can get a good sense of your path by reading some questions and answers on this site. Here are a few ones:

What is DevOps?

What is the difference between DevOps engineer Vs DevOps consultant Vs DevOps architect?

What is the difference between SRE and DevOps?

What could be a valid definition of DevOps to introduce it to a novice?

What is the difference between the traditional Development and Operations Model and Site Reliability Engineering?

What is the difference between Sysadmin and DevOps Engineer?
> 5 votes

# Answer
There is a nice hands-on path into the DevOps world:

* Read books to set the mental direction. Starting out with "The Phoenix Project" is not the worst bet. We can discuss the literary qualities of the story, but it gives a very sound idea on what are common pitfalls during software development (or in the larger IT world), and how to avoid. It is not too specific about a method (yes, it does use Kanban as a basic scheme, but Kanban itself is quite unspecific, compared to others like Scrum). If that book is too cheesy for you, take "The Devops Handbook" from the same authors.
* Learn basic tools. Build a basic DevOps toolset. You can start with Git, Gitlab, Docker, Ansible, Vagrant, to name a few concrete choices (except for Git, there are plenty of alternatives, but those are nice). This gives you Code Control, Pipelines, Containers, Configuration Management, VM Management.
* For your next application, setup a very basic CI/CD pipeline using Gitlab.
* Read more practical things like the "12-Factor Apps" article, and try to understand what all of the 12 factors are about.
* Learn orchestration. Install MiniShift on your own PC, use it to develop/locally deploy a few small applications. Try to apply the concepts from 12-Factor Apps to MiniShift. Learn more about OpenShift, play around with their free online demo area. Dive into Kubernetes.
* Get your CI/CD pipeline running in MiniShift/OpenShift. Think about how you would do certain things (e.g. what happens if you scale the Pod containing your database... etc.).
* Get an AWS demo/cheap account and play around with their offering. Look into all their parts that are not trivial VMs - play with Lambdas, etc.

All of this is easy and fun - each single step is more or less self contained, and if you understand what you are doing along the way, then you *are* a DevOps guy.

If you let it be known in the working environment that you have all this know-how, then DevOpsy work should come your way. If it doesn't, find a new employer and strongly emphasize your toolbox and pay attention during your interviews, whether they have CI/CD etc. in their process. No need to look for specific "DevOps Engineer" jobs.
> 5 votes

---
Tags: amazon-web-services, culture, learning
---


title: Why can I not run docker daemon?

text:
# Question
Title: Why can I not run docker daemon?
After instalating Docker from repo I can't run it. Problem is with daemon:

```
dockerd
INFO[2018-04-16T09:35:46.054942906+02:00] libcontainerd: started new docker-containerd process  pid=21656
INFO[0000] starting containerd                           module=containerd revision=773c489c9c1b21a6d78b5c538cd395416ec50f88 version=v1.0.3
INFO[0000] loading plugin "io.containerd.content.v1.content"...  module=containerd type=io.containerd.content.v1
INFO[0000] loading plugin "io.containerd.snapshotter.v1.btrfs"...  module=containerd type=io.containerd.snapshotter.v1
WARN[0000] failed to load plugin io.containerd.snapshotter.v1.btrfs  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module=containerd
INFO[0000] loading plugin "io.containerd.snapshotter.v1.overlayfs"...  module=containerd type=io.containerd.snapshotter.v1
INFO[0000] loading plugin "io.containerd.metadata.v1.bolt"...  module=containerd type=io.containerd.metadata.v1
WARN[0000] could not use snapshotter btrfs in metadata plugin  error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter" module="containerd/io.containerd.metadata.v1.bolt"
INFO[0000] loading plugin "io.containerd.differ.v1.walking"...  module=containerd type=io.containerd.differ.v1
INFO[0000] loading plugin "io.containerd.gc.v1.scheduler"...  module=containerd type=io.containerd.gc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.containers"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.content"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.diff"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.events"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.healthcheck"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.images"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.leases"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.namespaces"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.snapshots"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.monitor.v1.cgroups"...  module=containerd type=io.containerd.monitor.v1
INFO[0000] loading plugin "io.containerd.runtime.v1.linux"...  module=containerd type=io.containerd.runtime.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.tasks"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.version"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] loading plugin "io.containerd.grpc.v1.introspection"...  module=containerd type=io.containerd.grpc.v1
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd-debug.sock" module="containerd/debug"
INFO[0000] serving...                                    address="/var/run/docker/containerd/docker-containerd.sock" module="containerd/grpc"
INFO[0000] containerd successfully booted in 0.007348s   module=containerd
ERRO[2018-04-16T09:35:46.318603395+02:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.
ERRO[2018-04-16T09:35:46.321272231+02:00] 'overlay' not found as a supported filesystem on this host. Please ensure kernel is new enough and has overlay support loaded.
INFO[2018-04-16T09:35:46.334031676+02:00] Graph migration to content-addressability took 0.00 seconds
WARN[2018-04-16T09:35:46.337419439+02:00] Your kernel does not support swap memory limit
WARN[2018-04-16T09:35:46.337438494+02:00] Your kernel does not support memory reservation
WARN[2018-04-16T09:35:46.337452567+02:00] Your kernel does not support oom control
WARN[2018-04-16T09:35:46.337465274+02:00] Your kernel does not support memory swappiness
WARN[2018-04-16T09:35:46.337479319+02:00] Your kernel does not support kernel memory limit
WARN[2018-04-16T09:35:46.337488145+02:00] Unable to find cpu cgroup in mounts
WARN[2018-04-16T09:35:46.337501250+02:00] Your kernel does not support cgroup blkio weight
WARN[2018-04-16T09:35:46.337515483+02:00] Your kernel does not support cgroup blkio weight_device
WARN[2018-04-16T09:35:46.337528836+02:00] Your kernel does not support cgroup blkio throttle.read_bps_device
WARN[2018-04-16T09:35:46.337542675+02:00] Your kernel does not support cgroup blkio throttle.write_bps_device
WARN[2018-04-16T09:35:46.337556361+02:00] Your kernel does not support cgroup blkio throttle.read_iops_device
WARN[2018-04-16T09:35:46.337571999+02:00] Your kernel does not support cgroup blkio throttle.write_iops_device
WARN[2018-04-16T09:35:46.337589881+02:00] Unable to find cpuset cgroup in mounts
WARN[2018-04-16T09:35:46.337656042+02:00] mountpoint for pids not found
Error starting daemon: Devices cgroup isn't mounted

```

I installed cgroup manualy:

```
apt-get install cgroupfs-mount

```

But still this same issue. Is really a kernel from 2016 year too old?

uname -a retruns:

> Linux xxx 3.16.0-042stab127.2 #1 SMP Thu Jan 4 16:41:44 MSK 2018 x86\_64 GNU/Linux

# Answer
> 1 votes
Try the following as root:

* Edit the /etc/default/grub file and set the GRUB\_CMDLINE\_LINUX value as follows:

`GRUB_CMDLINE_LINUX="cgroup_enable=memory swapaccount=1"`
* Update GRUB.

`sudo update-grub`
* Reboot your system.

As per this docker's documentation.

---
Tags: docker
---


title: One time/set up jobs on gitlab ci?

text:
# Question
Title: One time/set up jobs on gitlab ci?
I'm setting up a pipeline that compiles and builds new docker images, and then deploys those images to testing kubernetes cluster.

I'm using a shell runner.

I need to setup my `kubectl` client to point to the testing kubernetes cluster. (ie. setting the cluster and context).

As I understand it - I should only need to run this once. Is there a way to get this to run once, as defined in the `.gitlab-ci.yaml`? Or should I just repeat the set up steps everytime in `before_script`?

# Answer
> 3 votes
I dealt with the same issue. Some people recommended using the Gitlab group "secrets" and using `before_script`. Since I knew I was also going to deploy and would need other tools on my runner like "helm", I made my own docker container. I still keep my config for my clusters in a base64 encoded group secret but I set it like this in the dockerfile.

```
ENV kube_config=$kube_config
RUN echo -n ${kube_config} | base64 -d > ~/.kube/config

```

Now just set an environment variable for your gitlab runner to point to your new image.

```
KUBERNETES_IMAGE: registry.gitlab.com/MY_USERNAME/MY_REPO_NAME/MY_IMAGE_NAME:latest

```

Don't forget to test your new container!

```
stages:
  - build
  - test
  - release

variables:
  CONTAINER_TEST_IMAGE: $CI_REGISTRY_IMAGE/gitlab_runner:$CI_COMMIT_REF_NAME
  CONTAINER_RELEASE_IMAGE: $CI_REGISTRY_IMAGE/gitlab_runner:latest

before_script:
  - apt-get update && apt-get install docker.io -y
  - docker login -u gitlab-ci-token -p $CI_JOB_TOKEN $CI_REGISTRY

Build Image:
  stage: build
  script:
    - docker build --build-arg kube_config=${kube_config} -t ${CONTAINER_TEST_IMAGE} .
    - docker push ${CONTAINER_TEST_IMAGE}
  except:
    - master

Test Kubectl:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm ${CONTAINER_TEST_IMAGE} kubectl get deployments -n kube-system
  except:
    - master

Test Helm:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm ${CONTAINER_TEST_IMAGE} helm ls
  except:
    - master

Test Docker:
  stage: test
  script:
    - docker pull ${CONTAINER_TEST_IMAGE}
    - docker run --rm -v /var/run/docker.sock:/var/run/docker.sock ${CONTAINER_TEST_IMAGE} docker images
  except:
    - master

release-image:
  stage: release
  script:
    - docker pull $CONTAINER_TEST_IMAGE
    - docker tag $CONTAINER_TEST_IMAGE $CONTAINER_RELEASE_IMAGE
    - docker push $CONTAINER_RELEASE_IMAGE
  only:
    - master

```

Edit: Missed this part.. **I'm using a shell runner.**

So post of what I said is probably not relevant. How ever I do think there is some advantages to building and deploying with a gitlab runner directly on a cluster.

---
Tags: kubernetes, gitlab-ci-runner
---


# Question
Title: How effective is ELK than other Monitoring tools?
"ELK" is the acronym for three open source projects: Elasticsearch, Logstash, and Kibana. Elasticsearch is a search and analytics engine. Logstash is a server‑side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and then sends it to a "stash" like Elasticsearch. Kibana lets users visualize data with charts and graphs in Elasticsearch.

```
 How effective is ELK than other Monitoring tools?

```

# Answer
ELK is pretty ineffective as a monitoring tool in a default configuration as it doesn't do monitoring - only reporting. ELK can only provide monitoring and alerting abilities if you configure additional add-ons such as ElastAlert.

Then, you are impacted by your ability to log errors via syslog. If messages come in via, say, SNMP traps, you will be unable to alert on these if you do not configure the appropriate plugin.

Like most monitoring systems, your effectiveness is determined mostly by 1) your input method and 2) if you have told the system to alert off of all of the correct messages.
> 1 votes

---
Tags: monitoring, logstash, elastic
---


# Question
Title: Switching between blue / green services in Kubernetes using UI page
The project is going with a new technology stack and I need to do a new CI / CD implementation. I started to play with Kubernetes and how to do blue / green deployment via command line. I managed to do switch manually via command line changing label selector in the service to filter green or blue pods. Now, what exactly I am looking for is kind of UI page where QA guys could do a switch between blue and green instances once they finished to test the application on staging server.

So:

1. Web site is deployed to staging environment (i.e. green);
2. QA test it and confirm that everything is fine;
3. QA navigate to some UI to make green instance live and blue to staging;

Are there any tools on the market for this?

Thank you.

# Answer
Jenkins allows you to construct a CI/CD pipeline that meets your requirements. Here's an example of a pipeline using an input step to decide which environment to deploy to. All you would need to do is wire the stages together.

Here's an example of the pipeline code (declarative) used to create the example:

```
pipeline {
  agent any
  stages {
    stage('Deploy Staging') {
      steps {
        echo 'deploying to staging'
      }
    }
    stage('Test Staging') {
      steps {
        echo 'testing staging env'
      }
    }
    stage('Blue/Green?') {
      steps {
        input(message: 'Blue/Green?', id: 'deploy', ok: 'deploy?', parameters: [choice(choices: 'blue\ngreen', description: 'Select an environment', name: 'PROD_ENV')])
      }
    }
  }
}

```

Other tools that provide similar capabilities are Rundeck, Stackstorm or Ansible Tower.
> 1 votes

# Answer
After a while, I decided to proceed with TeamCity+Octopus as I did for the previous .NET stack. QA / Dev teams got accustomed to the way they are doing deployments. Even though Octopus doesn't have a first-class Kubernetes support, I implemented my own way to do releases to Kubernetes clusters via custom linux target with Calamari. Blue / Green deployments were implemented using manual intervention step in Octopus.
> 1 votes

---
Tags: kubernetes
---


title: Running Docker images as Jenkins slave

text:
# Question
Title: Running Docker images as Jenkins slave
I have a Jenkins Master running in Windows as Local system. I have Jenkins slaves running as Docker containers in a VM in the same Machine.

Everything is working OK, I just want to know are there any gains in memory consumption or ressources if I'm going to run parallel jobs in the master and slaves or should I just use parallel steps using pipeline in Master?

# Answer
> 1 votes
In general, you can go with the assumptions that running something inside docker does not add a significant amount of resource consumption, neither CPU nor RAM, compared to starting the same process directly.

Also, it is usually good to optimize only if necessary.

So pick the Jenkins configuration that seems cleanest/nicest/most elegant to you. If you run into trouble, *then* look for a solution. As you are running everything on a single physical machine, thinking too much about performance is a moot point anyways, it's not like the solution will scale in any direction, anyways. But getting to know Jenkins, and finding good configuration solutions which you may be able to apply in a "real" environment later is certainly possible in this way.

---
Tags: docker, jenkins
---


title: Saving time on an EC2 instance

text:
# Question
Title: Saving time on an EC2 instance
I have started playing around with AWS (privately). I have one free tier Amazon Linux 2 VM running right now, which serves a "fat" service (in a docker container) to myself and one other person. It has an Elastic IP associated with it.

We use the service 1-2 hours every few days.

Is there, in the rather large AWS zoo of services, a smart way to have the VM shut down when not in use, and come up when we need it? "Need it" could be measured by counting network traffic, i.e. I could in principle write a little script which uses `tcpdump` or something like that to watch for 15 minutes of no traffic on the port of the service and then somehow trigger a shutdown; but how would we get it up again?

I am happy to change it in any way necessary; but the Docker image has to be considered a black box for all intents and purposes. There is one docker volume (realised as a directory in the home of the `ec2-user` right now) if that makes a difference. Timing is not critical.

So far I'm doing shutdowns/startups through the EC2 Management Console, but would like to end that.

# Answer
AWS has something built in called cloudwatch alarms that can handle the basics.

You could also use a Lambda Function. Give it access to the ASG, and then have cloudwatch configured so once the min amount of traffic starts coming in it sends an Event to the Lambda.

```
{
  "ASG": "MyASGName"
  "Action": "Start"
}

```

And once the traffic goes below the min.

```
{
  "ASG": "MyASGName"
  "Action": "Stop"
}

```

You could also build on this over time to control Vertical and Horizontal scaling.

EDIT: I actually prefer the Lambda. I always seem to run into these weird use cases where we can't shut down any of the instances, we have to do some math or investigation on which instances to shut down. In these situations a python lambda running boto3 is really helpful.
> 3 votes

---
Tags: amazon-web-services, amazon-ec2, startup
---


title: Open source microservice visualization

text:
# Question
Title: Open source microservice visualization
We run various microservices on rancher and kubernetes, and I am wondering if there are any open source projects that assist in visualizing network traffic between services to help detect dependencies. Similar to the project below

https://docs.netsil.com/

# Answer
I believe some of the monitoring tools like Dynatrace and Scope show some neat stuff. This really depends on your platform (I'm assuming Kubernetes).

However, if you are running something else checkout Visceral. Here is a neat video of it in action. It's more complicated to get up and going.
> 1 votes

---
Tags: docker, monitoring, network-monitoring
---


# Question
Title: Limits of, or preconditions for of Blue-Green deployment?
I find a lot information about the blue green deployment pattern but maybe miss a point. Please help me to understand.

Could it be scenarios where you can't successfully duplicate the information flow?

Examples:

* Captcha request from a local API; can you ensure same captcha goes to blue and green?
* Randomly generated hardware-based UUID's: blue and green wouldn't get same random number, or?

Or does blue/green mean, that exactly such considerations have to flow as well into the software design?

# Answer
I must admit that I don't quite get the examples your giving. Sure, random numbers will be different between the two environments.

Blue-green means simply to have two nearly identical production environments with some kind of router/load balancer in front. Clients talk to the load balancer, and the LB routes requests to one of the two environments.

To deploy a new version, you provision all machines/services/etc. in the inactive environment, test to your heart's content, and then do the instant switch (on the router) if you're satisfied.

Actually, I don't see anything that requires any precondition at all about this. Obviously, one precondition is that you are at all able to create a second, fully fledged production environment in the first place; and that there is a technical solution that works as a load balancer in the above sense. That may get very complicated if you have a widely distributed system which you want to switch completely at once. That may just become to complex.

But aside from that, nothing special compared to, for example, running a separate test environment next to your dev/prod environments.
> 4 votes

---
Tags: deployment, analysis, patterns
---


# Question
Title: How to generate Code Churn in VSTS? (Azure DevOps)
In this article I noticed that is it possible to generate code churn reports for TFS:

I would like to generate the similar report using Visual Studio Team Services (VSTS). Is that possible and, if so, how?

# Answer
> 1 votes
For now you can't. But you have Analytics and there is a Code part that is in the roadmap. Take a look there anyway. It has some awesome widgets so far.

---
Tags: project-management, azure-devops
---


# Question
Title: Does Jenkins support cloning git submodules with "git@" (not "https") URLs?
Had some issues with Jenkins when trying to checkout a project with Git nested submodules. It failed with a permission error when cloning the submodules:

```
hudson.plugins.git.GitException: Command "git submodule update --init --recursive some/path" returned status code 128:
stdout: Submodule path 'some/path': checked out '34e663c11661b5153076c1d624ead25137df0783'

stderr: Cloning into 'some/other/path'...
Host key verification failed.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
fatal: clone of 'git@github.com:foo/foo.git' into submodule path 'some/other/path' failed
Failed to recurse into submodule path 'some/path'

```

In the Jenkins job, checking out submodules is enabled: In the Git plugin under `Advanced sub-modules behaviours`, the options `Recursively update submodules` and `Use credentials from default remote of parent repository` are enabled. All git repositories are private Github repositories, but the Jenkins user has read access to them.

The only way to get it working as by replacing `git@...` URLs by `https://...`. For instance, this here would fail

```
[submodule "foo"]
    path = some/path
    url = git@github.com:foo/foo.git

```

... but this would work

```
[submodule "foo"]
    path = some/path
    url = https://github.com/foo/foo.git

```

I cannot rule out that there is a misconfiguration on our end. But still I have some generic understanding question:

* Does the Jenkins Git plugin support `git@` URLs? Or is it a known limitation that you need to use `https://...` URLs?
* Why does the choice of the protocol make a difference on Jenkins? Is it because `git@` assumes that SSH keys of Jenkins were added to the Github repository?

# Answer
As a reference, it works for our system, submodules over SSH. We use deploy keys for this purpose. In Gitlab they can be found under *Settings* --\> *Repository* --\> *Deploy Keys*. The used key has to be marked as *Enable* in each used Git repository (main and submodules).
> 2 votes

# Answer
Are the credentials you are using a username/pass or token? If so, that will work for http cloning, but will fail for git@ since that uses ssh keys to authenticate.

There is another option you can select called "Checkout over SSH" that comes from the Github Branch Source Plugin. You will need to add a new "SSH Username with Private Key" credential to Jenkins where you can select to either upload a private key, use a file, or use the one on jenkins master (usually `~/.ssh/id_rsa`). The corresponding public key will need to be added to your Github profile or repo settings.
> 1 votes

---
Tags: jenkins, git, jenkins-plugins
---


# Question
Title: How to increase the size of a disk on a running instance on Amazon EC2
I got a running instance of a EC 2 server setup with Ubunto. What's the best way to increase the disk size without any downtime and minimal risk?

Reading through the guides, one way would be to create a new disk, migrate data, turn off the instance swap disk and turn it back on. This approach sounds a bit risk will require some downtime. I wonder if there is a better approach?

# Answer
> 14 votes
Amazon AWS just released (on 13th February 2017) a new feature that allows to change a size of an EBS volume.

source: https://aws.amazon.com/blogs/aws/amazon-ebs-update-new-elastic-volumes-change-everything/

This allows to increase the size of an EBS volume on an existing instance, while it is running.

It is important to note that changing the volume size, does not change the size of the filesystem on the volume (for most filesystems). Additional steps might be required (depending on filesystem) in the operating system itself. For example a `resizefs` on `ext4` filesystem used by most Linux OS today.

Full documentation from AWS describing the process can be found at http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-expand-volume.html

In some cases the block device has mapped partitions, and only then one (or more) of the partitions include a filesystem. In this case the partition will need to be resized first, and only then the filesystem. This process is described in the documentation as well.

The new "online" resize feature described in the blog only applies to current generation instances, and there are some other considerations and limitations that need to be checked before attempting a volume resize.

# Answer
> 2 votes
What I do (and that's not exactly answering your question) is as follow:

1. Create an EBS volume and attach it to the instance doc is here
2. Rescan the scsi buses `echo '- - -' > /sys/bus/scsi/devices/host1/scsi_host/host1/scan` (you may have to adapt the host number)
3. Create a physical volume with `pvcreate` on the new disk found (`fdisk -l` to list all disk)
4. Create a volume group and then logical volume on it (`vgcreate` and `lvcreate`)
5. Format the logical volume with your desired filesystem
6. `tar` the target mountpoint to restore it.
7. Mount this volume where you need new space.
8. Restore the tar into this newly mounted space.

Steps 6 and 8 are optional if you use a new space before installing something. If you want to replace an existing directory you obvisouly have to avoid something writing there between the end of the archive and the restoration.

You can repeat the steps 4 to 8 for different mountpoints, this allow to extend the space needed and then to resize those volumes on line without interruption.

# Answer
> 1 votes
# ZFS Zero Downtime filesystem storage scaling on AWS (or elsewhere)

## upsize

1. Install ZFS on EC2.

http://serverascode.com/2016/09/05/aws-zfs-user-data.html
2. Make a zpool for your bulk data using an EBS volume.
3. Add another EBS to get more block storage. (or set the pool autoexpand=true and just grow your EBS)
4. Add the new EBS to your zpool to make the space available. (unless you used autoexpand and increased the EBS size)

## Downsize

1. Make a new zpool on new EBS big enough to hold the shrunken data. (doesn't need to be mounted yet, or even on the same EC2)
2. snapshot the old too-big zpool
3. zfs-send the snapshot to the new zpool
4. promote the received snapshot on the new pool and mount it
5. destroy the old pool
6. destroy the old pool's EBS

# Answer
> 0 votes
After resizing the EBS volume, here is what I just executed when I needed to expand a ZFS pool:

```
parted -l # Get the list of partitions
parted /dev/xvdf rm 9 # Remove the buffer partition
parted /dev/xvdf resizepart 1 100% # Resize the partition
zpool online -e <zfs partition name> /dev/xvdf # Expand the zpool and the filesystem it holds

```

---
Tags: amazon-ec2, amazon-ebs
---


# Question
Title: How should we organize VSTS repositories?
We are a small company with the following systems:

* External Web Site for pre-customers (already created in ASP.NET Framework),
* External Mobile Application for customers (in progress in Flutter),
* Internal custom CRM that work on items collected by this External Web Site and Mobile App (already created in ASP.NET Core),
* Internal DataBase and REST API Server (already created in ASP.NET Core this will NOT be changed in a long time).

So External Web Site and Mobile App talks to Internal Database by API, and then Internal CRM uses this API to retrieve data collected by those External Systems. We are moving from mono repo in Subversion to GIT in VSTS.

How should we divide projects into the repository? Should we have one repository to manage all products and then later build CI/CD configured to release few projects at once? This will be easier to run development on a single machine with API running in the background. Or should we have few repositories with Test environment to manage API and Database version? This will require setting up an additional server for Web API and Database.

I am asking from a project manager and developer perspective what will be easier and more convenient to use and manage?

# Answer
> 1 votes
From the perspective of build automation, I find that it is better to have multiple repositories. This allows for smaller configurations and more granular control of your build/release process. You can allow references and pull in source code to build or release (of course you can also ignore source code as well in VSTS build pipeline).

This leads to being able to release just the project you want or need to your desired environment. You can, of course, pull in artifacts from other builds and include them in your release if you need to.

To summarize, I tend to go as small as possible and only group things into the same repository if they are indeed part of the same project and no part of the project can be used elsewhere.

---
Tags: git, project-management, azure-devops, monorepo
---