title: How to login to k8s proxy nowadays?

text:
# Question
Title: How to login to k8s proxy nowadays?
Back in the day when I ran `kubectl proxy`, I could login to http://localhost:8001 immediately, but know a login prompt is shown. When `skipped` is clicked the following message is shown:

> configmaps is forbidden: User "system:serviceaccount:kube-system:kubernetes-dashboard" cannot list configmaps in the namespace "default": Unknown user "system:serviceaccount:kube-system:kubernetes-dashboard"

**Attempt to solve the issue**

When the kubeconfig is copied, then the login fails as well

# Answer
> 4 votes
What is the Kubernetes version? Is it \>1.8 ? (which has RBAC enabled by default).

The error message says that the Kubernetes Dashboard service account is not allowed to list configmaps in default namespace. RBAC by default is deny-all. You need to explicitly grant permissions.

You can create the following `ClusterRoleBinding` to grant dashboard full admin privileges.

```
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard
  labels:
    k8s-app: kubernetes-dashboard
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: kubernetes-dashboard
  namespace: kube-system

```

Save the above block into `dashboard-rolebinding.yaml` and create it on the cluster:

```
kubectl create -f dashboard-rolebinding.yaml

```

**IMPORTANT**: Make sure your dashboard is secured from outside before doing this. Otherwise anyone can have complete access to your cluster. Here is a great article about securing kubernetes dashboard, from Heptio: https://blog.heptio.com/on-securing-the-kubernetes-dashboard-16b09b1b7aca writted by Joe Beda himself.

---
Tags: kubernetes, authentication
---


# Question
Title: Handle OS and Software maintenance/updates on Hardware distributed to Customers
In the last weeks I have tried to find a solution to the following scenario assuming that someone else had similar requirements and documented his solution but all my searches were unsuccessful. So now I'll just ask away and hope that someone with enough experience in this field can give me directions so that I don't take the wrong turn too many times.

The simplified scenario:

Let's say I have written a piece of software with a web interface and REST API for my customers which absolutely has to run in the local network at the customer's premises and will be provided to them as a "black box" hardware appliance. Apart from that I will essentially provide it "as a service" to them, meaning that maintenance and updates are my responsibility. I come to them with a small headless pre-installed and pre-configured PC, connect it to their network, perform only networking configuration and I'm done. The appliance is ready to communicate with any other software components running in the customer's network.

To be a bit more precise my software product runs under Linux (and also Windows), it should also be possible to build it as a Docker container (if that makes any sense for the proposed solution, although my know-how with Docker is still limited). I expect to set up CentOS 7 on the appliances.

Now I have two challenges:

* Keeping CentOS packages up-to-date
* Keeping my software package up-to-date

To prevent all customers from experiencing the same bug at the same time I want to be in full control which customers update which packages at which point in time on the appliance (essentially a staged rollout). In regards to my own software package this could very well be in form of different update channels (beta, stable, etc.). Ideally I am also able to fetch and push configuration and other files from/to the appliance. It would be great if I could use existing solutions for everything instead of rolling my own mechanisms.

Also, the solution should scale to hundreds or even thousands of customers.

What I have thought of until now:

**Spacewalk, Puppet, etc.**

For managing the CentOS systems. However, I have no idea whether any one is good at managing machines that are not running inside a single company network.

**RPM packaging my software component** or **My software as a Docker container**

I don't know which one of these would be better regarding updates. Docker may be overkill, it's just a single software application on a single machine for each customer, I only thought that it would ease application updates. Maybe an RPM package would be preferable, that way I could handle update channels as different RPM repositories and control the updates themselves with Spacewalk/Puppet.

**An OpenELEC/LibreELEC JeOS approach**

I don't know what this is called exactly or whether there are pre-built solutions for this but I have first encountered it with OpenELEC. Essentialy the whole OS and application are packaged in a single non-modifiable and updatable (kernel) image which is installed on the target machine and there's a separate partition for user and local configuration data.

I know this is a pretty big question, but any pointers would be very appreciated.

# Answer
> 2 votes
Red Hat used to brand Spacewalk as "Red Hat Satellite 5" and decided to scrap it and build a new toolset from the ground up. Red Hat then based what they now brand as "Red Hat Satellite 6" on the Katello plugin for The Foreman.

In addition, the Foreman also has plugins for Docker, Puppet, and several other related systems.

But to be honest, the system is a bit rough and needs some polish. I frequently bump up against bugs and have to expend quite a bit of time making all the pieces work together properly - though it is still very cool and powerful.

You may, however, have an issue with any client networks that have strict security requirements. At my employer, we are not allowed to directly connect to the internet and are barely able to use proxies which require a rigorous security review and then we are still quite limited, so that might need to be a consideration for your updates model - You will be a no-go on those types of customers. On the other hand, that's part of the advantage of the "As-a-service" model. You don't have to worry about it or maintain it, so you will pick up more small-and-medium business as customers.

# Answer
> 2 votes
I can provide some personal experience here. I worked at a company that had the same need. The product was a BeagleBone (like an Arduino or Raspberry Pi) that ran our software and sent data back to our SaaS. It had to be installed inside customer networks, and be a little black box that they didn't touch.

When I walked into this company, management of the systems was a nightmare. Networking was a nightmare. OS updates, keeping them online, shipping new versions of the software, etc... it was a nightmare.

The path we ended up choosing ended up working really well: The entire application and supporting packages was shipped as a Docker container. That allowed us to easily ship application code changes and control versions of any supporting software (ffmpg, gcc, etc...).

With the application abstracted, that simply left making the OS as minimal as possible. All we needed was a dead simple OS that could run docker. One of our low level engineers actually made our own in-house fork of Debian (if I remember) and we shipped that embedded directly into the devices.

We already had a service contract with all of our customers to service these machines yearly, so when our field technician went on-site, the procedure was to replace the entire unit with a fresh hardware that came with the latest embedded OS. We didn't have to make many OS changes just to run Docker, so often times swapping wasn't even necessary. But out of procedure, we always swapped hardware whether it was an emergency service call or the yearly maintenance.

I can't say if this would be a standard approach or not, but just that it worked really well for us and was a life saver. It sounds like your idea of shipping the OS as an immutable image is along the same lines.

**Edit**: When I left, we were still using home grown bash scripts to automate the Docker container deployments, but the idea was to move toward a config management tool (Like Chef, Salt, or Ansible). The basic premise is that you have a *private* Docker registry somewhere, then on the devices you would simply run a couple commands:

```
docker pull my-private-registry/my-custom-image:latest
docker run my-custom-image:latest

```

If you are going to always be using the 'latest' tag, you could then have your devices ship with a cronjob that has then always shut down their local running Docker container, and re-pull the 'latest' tag every week during a scheduled maintenance window.

---
Tags: deployment, configuration-management, installation
---


# Question
Title: How to deploy applications to k8s using bitbucket pipelines?
The aim is to implement Continuous Deployment (CD) to a k8s cluster.

It works to deploy a Nginx image to the k8s.

https://github.com/kubernetes/kubernetes/blob/master/examples/simple-nginx.md `kubectl run my-nginx --image=nginx --replicas=2 --port=80`

When the same command is run in a CD tool then it complains that it cannot find the cluster. Off course it is possible to copy the `~/.kube/config`, but is that recommended?

When the Kubeconfig is stored in a variable and kubectl is run by bitbucketpipelines then the following error is thrown:

```
error: Error loading config file "/root/.kube/config": yaml: mapping values are not allowed in this context

```

Inspecting the files turns out that the whole file consists of one line instead of multiple.

# Answer
You need to separate your daily authentication method of the CD authentication.

You should start understanding the different methods that K8S offers https://kubernetes.io/docs/admin/authentication/

On the other hand if you use some cloud solution like GCP you can benefit of the service account that the platform puts in all your instances, setting the right permissions to a service account allow you to to controll any cluster inside your project like in your local machine.
> 1 votes

---
Tags: kubernetes, bitbucket-pipelines
---


# Question
Title: Windows Docker in swarm with Linux nodes connection issue
I have the following setup for my docker swarm:

* three Linux master nodes
* one Linux slave nodes
* Windows slave node

They are connected with an overlay network. Within the linux nodes everything works as expected.

But if I'm in a windows container(e.g. microsoft/nanoserver) and I try to ping a linux container with the internal dns name of the container. The DNS resolution works just fine, but the request times out. So far so bad. But it's getting stranger, because my linux container has two IP-addresses one is global loopback and another one which is in the same network.

Let's say the linux container has xx.xx.xx.2/32 as global lo and xx.xx.xx.3/24. The internal dns resolves to the ip xx.xx.xx.2. This cannot be pinged from windows containers. But the address xx.xx.xx.3 can be pinged. If I'm in a linux container, I can ping the xx.xx.xx.2 address. I also can curl the service(port 8080) of the windows container, within a linux container.

It's just weird. Has anyone experienced similar issues or can I manipulate the internal dns resolution of docker?

Also fun to note, it worked earlier. But after a swarm leave and join of the windows node it doesn't anymore.

Versions:

* Windows Server 2016 - Docker version 17.06.2-ee-6, build e75fdb8
* RedHat 7.4 (Maipo) - Docker version 17.12.0-ce, build c97c6d6

UPDATE: The windows node cannot reach a container which runs on the linux slave node all other containers can be pinged normally, but from the linux master nodes it is possible to ping the containers which run on the linux slave nodes.

# Answer
> 1 votes
~~I think Windows Server 2016 needs at least 17.09 Docker EE to support overlay networking and VIP's. I recommend updating it.~~

I was wrong, the latest version is indeed `17.06.2-ee-6` and should support overlay and VIP.

Second, ping isn't a good test, as it may not be available on the VIP or container IP depending on the setup (Docker has changed over time if the VIP does or does not respond to ping). I recommend using something that tests the Exposed container port, like curl. For testing you can just use something like bretfisher/netshoot which has common utilities like that installed.

Third, you could be resolving Swarm services on the network using the service name, not the internal hostname of containers. Depending on which network endpoint mode you use (VIP or DNSRR) in the service, this will resolve the correct VIP or the correct container IP for other containers on the same Docker network.

Also, you want a minimum of 3 manager nodes to follow raft consensus rules in a Swarm. 2 managers actually makes it less fault-tolerant then 1 manager.

---
Tags: windows, linux, docker-swarm
---


title: Launch jenkins agent from jenkins job?

text:
# Question
Title: Launch jenkins agent from jenkins job?
**Is it possible to launch a jenkins agent from within a script in another job?** I'm open to either shell or pipeline options.

To be clear I don't mean the agent the job is running on itself. I'm talking about spinning up a completely separate agent.

Sometimes our master gets slammed with unit test requests and will make jobs wait in the queue for 15-20+ minutes instead of just spinning up more agents. I want to see if there's a way to spin an agent up intentionally so I can tie it into a job that polls the queue for wait times.

# Answer
What you are looking for is called **dynamic provisioning** of Jenkins' slaves, and yes this is doable. Setting it up depends upon your desired infrastructure provider/resource. The easiest example in my opinion is using AWS EC2 instances. Install the Amazon EC2 Plugin for Jenkins, and configure it with the appropriate access keys, IAM roles, etc. to allow it to provision EC2 instances in a specified VPC. Here is a short (but slightly outdated) guide.

If you are moving to K8's and Docker, you may want to consider looking into the Docker Plugin for Jenkins. If you are using bare bones infrastructure, this should allow you to more effectively utilize what current hardware you have.
> 1 votes

# Answer
We have a jenkins master which has like 10 agents connected via ssh, usually they are down. But when a build starts, jenkins starts an agent automatically(It launches the jar file over ssh).

So I would believe jenkins should solve that problem with the build in logic. If the queue is full it starts up agents which are currently offline.
> 0 votes

---
Tags: jenkins
---


# Question
Title: What's the best way to restrict an environment to a specific CDN?
We have had problems a couple times now where the links to the CDN in our components don't get properly updated between the team, QA, and staging environments.

Is there a way to force an error if the QA environment tries to access the team CDN or vice-versa? Or some other well-known solution to this problem?

# Answer
If the different CDNs are hosted at different IP addresses one possibility might be to configure the local `hosts` table for the machines in a certain environment in such way as to overwrite the DNS resolution for the CDNs which shouldn't be accessible and pointing them to either:

* a non-existent address, causing connection timeout failures if the invalid CDNs are referenced
* a server where the respective files cannot be found, causing 404 errors (faster than the connection timeouts)
> 2 votes

---
Tags: environments, cdn
---


# Question
Title: How to pass the version to docker-compose?
When I would like to update an app at the moment, `docker-compose down`, the version in the `docker-compose.yaml` is changed, subsequently the app is started and the latest version is running.

Now I would like to update it automatically. It it possible to update the version by passing a variable?

**Workaround**

At the moment I use `sed` to update the docker-compose file, but I wonder whether there is an easier approach.

# Answer
You could use environment variables e.g.:

```
export VERSION=2

```

docker-compose.yml:

```
...
  serviceName:
    image: "imageName:${VERSION}"
  ....

```
> 2 votes

# Answer
Another way to execute the @golfNintyNine answer is putting inside docker-compose.yml the environment variable:

```
serviceName:
image: "imageName:${VERSION}"

```

And then executing simply:

```
$ VERSION=2 docker-compose up

```

Also you can put all together inside script file like this:

```
#!/bin/bash

function usage {
    cat << EOF >&2
Version argument is required, please use -v or --version.
EOF
    exit 1
}

OPTS=`getopt -o v: -l version: -- "$@"`

if [ $? != 0 ] ; then usage ; fi

eval set -- "$OPTS"

while true; do
    case "$1" in
        -v | --version ) export VERSION="$2"; shift 2;;
        -- ) shift; break ;;
    esac
done

if [ -z $VERSION ]
then
    usage
fi

docker-compose up

```
> 3 votes

---
Tags: docker-compose
---


# Question
Title: How do you test changes to Jenkins plugins before deploying them?
If you ever been bitten by a plugin upgrade that broke some functionality you must have given some thought about this problem: What should be the Jenkins plugins upgrade policy? How do you test changes before deploying them?

Has anyone gone as far as having a test instance running dummy jobs to test the new versions or do you just pray that upgrading the versions won't break anything?

# Answer
> 4 votes
According to the policies of the company where I work we have dev, preprod and prod environments (on some service dev can be missing). And the path of new version preprod-\>tests-\>validation-\>prod.

In our case jobs in preprod are heavy and complex enough to be sure we do not need to pray when implement in prod :)

**Note**: We use svn to maintain and deliver configuration. We do not make changes inplace.

# Answer
> 2 votes
We needed a 100% HA Jenkins environment. we often upgrade plugins / Jenkins itself.

This causes a big headache if the build breaks after the upgrade.

The safest way to sort this is really getting a Demo Jenkins setup. Maybe on the same machine using multiple Tomcat application, you can achieve this cheaper.

What we did is create a separate (Demo)VM and replicated the prod set-up on the Demo VM. Before changing / upgrading anything we would take a snapshot of both VMs. Then we would test the upgrades on Demo VM. If it works good, change it on Prod.

I guess you can look up the community (like SE/SO) if anyone has faced any problems with the plugin you are planning for.

# Answer
> 0 votes
I'd always manually trigger a re-run or two on at least one recent green (or almost green) label on every relevant project/branch that uses the respective plugin and check that I get the same results. Just to be on the safe side.

Any result discrepancy would need to be investigated to determine it they're caused by the plugin update or not. Maybe a few more re-runs with both the old and new plugins?

---
Tags: jenkins, jenkins-plugins
---


# Question
Title: AWS Security group rule for certbot renewal
I want to add a rule to my `aws` security group to allow for incoming traffic from the acme server (I guess it is `acme-v01.api.letsencrypt.org`) to issue and automate the renewal of my cert;

What is the best way of doing this, given that sg rules do not support hostnames but only IPs?

# Answer
You just can **not** because from Let's Encrypt FAQ:

> * **What IP addresses does Let’s Encrypt use to validate my web server?**
>
> We don’t publish a list of IP addresses we use to validate, because they may change at any time. In the future we may validate from multiple IP addresses at once.

You will need a reverse proxy (usually a WAF), routing /.well-know/acme to a machine and answering the challenges (from anywhere) and doing the IP filtering before forwarding to your application all other requests.
> 3 votes

---
Tags: amazon-web-services, certbot
---


# Question
Title: Does terraform support blue green deployments and roll backs out of the box
I have been trying use terraform for a while . any suggestions/tips on how I can handle blue green deployments . I see how terraform enables it with auto scale groups on aws . But I’d like run some Cucumber acceptance tests on the new Asg before routing traffic to it . Is there a way in terraform to retain the old asg for a period , if In case I need to do a rollback . Can this be accomplished with terraform ... thanks in advance .

# Answer
> 8 votes
It all depends on how you are deploying your code.

(First, IMHO, I don't think that Terraform is the right hammer for this operation. Terraform responsibility is to build the necessary infrastructure for your application, not to be the tool used for deploying it. But, as always, "the right/best one" will depend on lots of factors and, depending on the case/scale, it may be the "good enough" solution)

My suggestion would be, in summary:

1. Deploy your application in the "unused" environment
2. Test it
3. Change the domain name to point to it

Assuming you have 2 AutoScaling groups already configured. One called `asg-blue` and the other called `asg-green`. And that the current version resides on `asg-blue` and in terraform you have an `aws_route53_record` pointing to it. You would need to follow these steps, each one with a `terraform apply` before the next:

* Deploy the newer version of your application on `asg-green`
    + I.e. update everything that is needed to deploy your application version. It may be the AMI you point on your `Launch Configuration` or any other way that you do exclusively using Terraform.
* Test that everything is running as expected
    + Here you could use the domain-name provided for each ASG/ELB or one already configured on terraform
* Update the domain name to pont to `asg-green` ASG/ELB on the `aws_route53_record`
* Reduce the number of machines needed on `asg-blue` to don't use too many resources
    + Depending on your case you can configure for 0 machines

This way you can have your B/G deployment, but for every deployment you'd need at least two manual interventions in your terraform (the deployment of the new version and updating the domain name).

But with this strategy you should **take a lot of care** to not deploy on the current ASG being used.

Of course if your terraform/aws fu is good, instead of reusing ASGs, you could delete and create them as needed.

Hope that this help in having a "big picture" of the entire process. But I'd suggest that you research and start using tools more focused on deployment, like AWS CodeDeploy.

# Answer
> 2 votes
I had a similar question and this answer was quite helpful. It sketches pattern on how you can do blue-green deployments in Terraform.

But having said that there is no clearly best solution as far as I know. I think the most common approach for your use case would be to have a dedicated "staging" environment that uses a setup that is very close to the production one (sharing the same Terraform modules, but with smaller auto scaling groups). You can use it for the final tests, maybe even route some fraction of the production traffic over it.

If there are no errors in staging, you simplify roll it out in production and hope that you found most critical bugs before.

---
Tags: terraform
---


# Question
Title: How to dimmension VMWare VM linux servers to run kubernetes container clusters?
I'm designing a web apps and web services hosting platform using kubernetes clusters of VMs running docker containers (web services, web apps, databases).

This application hosting platform will be set on top of an existing VMWare infrastructure. We have 3 physical hosts powered by ESX.

Today a network of some 10s of VLANs runs approx 100 to 200 virtual servers with ~1 to 4 cores and 1 to 8 GB of RAM, with some 10s of GB of hard disks (classical HD, no-SSD).

Today our system is managed the old way, with named VM servers running one app or one database.

What I have to do is to create a new VLAN of some VMs (maybe 5 or 6 in a first time), running a kubernetes cluster to host web applications, web services and databases docker containers (developed and deployed by our gitlab on-premise instance).

Once this VLAN and kubernetes cluster will be up and running, tested for a while and validated by me and the direction, the project is to extend this architecture to the whole VMWare infrastructure, VLAN after VLAN.

The question is, how to dimension (vertical scaling) the underlying linux VMs composing the kubernetes cluster ?

How much cpus, RAM and disk space should I allocate by virtual host to be able to maximize the benefits of horizontal scaling allowed by the containerized architecture, the nginx load balancing and the mongodb sharding ?

# Answer
> 1 votes
There are several related system engineering trade-offs and choices involved in answering your question. It may be helpful going forward to tackle this question by first prioritizing your objectives and then making the selection and trade-off decisions about VLAN structure, cluster(pod) sizes, etc. to decide which options give you the most bang for the buck in vertical scaling.

For example, you didn't explain how cpu intensive your web apps are or how much load balancing (based on the front end API management and back end messaging) you expect you will need. While load balancers may not seem relevant to your Docker/K8s design question, the impact this can have on how you size your cluster nodes is obvious if you consider what impact too many nodes and a poorly distributed or overworked load balancer could have on your VLAN design.

The 'one app per VM' design you described suggests your app's workloads are pretty big (for an 8GB VM). Again need more info to delve into this more deeply, but assigning a DB VM per App VM seems a bit wasteful in terms of K8s worker/manager efficiency unless there's a severe reliability and resiliency requirement.

Next, what are your options to replace the magnetic HD with SSD? Costs for 100GB and 500GB SSD have dropped significantly and replacing your mag HD with SSD could certainly give your clusters a good deal of disk performance headroom when you scale up. In addition to performance, despite the added costs, overall system reliability (MTTF, MTTR) will increase dramatically. New SSD drives have 6GB data channels which is probably at least double what your mag HD have now. In building a Docker/K8s cluster of any size, disk channel bandwidth and processor core count can cover a multitude of design sins.

Finally, not sure what your project delivery timeframe is for this, but it might help your thinking all this over and making the best choices if you can take a step back and model this with SysML (using MS Visio) and look at your design and then apply Architectural Tradeoff Analysis Method.

ATAM and the theory behind it, despite its name, is really rather simple both in theory and in practice. ATAM gets its name because it not only reveals how well an architecture satisfies particular quality goals (such as performance or "modifiability" or scaling in your case), but it also provides insight into how those goals interact with each other — how they trade off against each other. Doing ATAM upfront does point out critical design decisions with the most far-reaching consequences and are the most difficult to change after a system has been implemented.

SysML and ATAM is also a kind of insurance policy against painting yourself into a corner when you are asked to cost justify your design choices or forecast TCO when a change in either the web app design or your VMWare infrastructure occurs in the future.

ATAM info here \>\>\>

SysML info here \>\>\>

---
Tags: docker, kubernetes, load-balance, vmware
---


# Question
Title: Installing Ansible Controllers to build agents which best practice to use?
Trying to implement **infrastructure as code**. I need the ansible controller machine which pulls the updated playbooks from the git and run the playbooks to create the infrastructure. The issue is I have tried VSTS and AppVeyor and their build agents do not have Ansible installed.

Should I use my build agent for this task? I am looking for open source and free solutions for the open source project.

# Answer
> 4 votes
Infrastructure as code means that the infrastructure configuration is stored in a version control system (VCS). If changes will be applied, it is traceable who changed the code and when.

Although it is unclear to me what the aim is of your question, it is possible to use a continuous integration (CI) tool like, Jenkins, gitlab, circleci that pulls the code from git, installs ansible or use an image that contains ansible in order to run a playbook.

---
Tags: ansible, azure-devops, appveyor, azure-pipelines
---


# Question
Title: What is a suitable strategy for provisioning Chef Server through code?
**Background**

* I am tasked with setting up a Chef server on our Azure Cloud for a new DevOps automation project.
* We are using the Resource Manager model for Azure, in case it's relevant.
* We will likely use free versions of Chef i.e. Chef Automate is not an option.

**Objective**

I am new to Chef and Azure, but I would like to *try* to do it the "right" way from the start. Specifically: Given the Chef server will eventually become an indispensable part of the DevOps infrastructure I want to make it provisionable through code.

To move toward achieving these objectives I have explored the following options.

**Option 1: Chef to provision Chef Server**

Use Chef Server cookbooks and Chef solo to provision Chef server.

In such a scenario, my main question is: how do I *create* the Chef server VM and make Chef Solo discover it?

**Option 2: Use Azure CLI + RM Templates**

1. Manually create Azure VM
2. Manually install Chef server
3. Save it as an image and Azure RM template
4. Use Azure CLI to provision from the RM Template and Image.

I am angling toward this option, but I have some anecdotal opinions from teammates, that Azure RM Templates wll not be declarative. Are they?

**Option 3: Use Vagrant + Azure Provider + Chef Solo**

1. (This option is primarily because I am comfortable with Vagrant)
2. The main advantage of this option is that I think it will eliminate the question I had in Option 1. Is my assumption here correct?
3. However, I think this option does not work for a permanent infrastructure like Chef Server, I think Vagrant is more useful when you have *disposable cloud-based virtualized environments.* Is my understanding here correct?

*Note: Obviously these are not the only 3 options. Please help with any strategies I might have missed.*

# Answer
> 2 votes
'Suitable' can honestly be whatever works for you. I personally feel its perfectly reasonable to do a manual install of Chef server. It is the CM tool to help provision the rest of your fleet. You will not be setting up Chef server over and over again. You do it once, and then never again, in *theory*.

That theory breaks down however, in respect to server outages. Eventually, one day, your Chef server *will* go down. If you are running Chef in HA mode, or you have a very good backup system that allows you to restore the entire server, then you are fine. I don't know what it is for Azure, but with Amazon, you can take nightly snapshots of the entire server and simply restore from the AMI if the server ever goes "hard down".

Let's pretend that you don't run Chef in HA, or you aren't very confident in your backup system, or you just want to tinker... my recommendation would be to leverage Terraform w/provisioner scripts, or Packer with Terraform.

You can get the basics of Chef server setup with some simple commands that only need to be run once. This is perfect for Terraform and it's provisioner scripts. Obviously, it runs the provisioner scripts only once (during server creation) and that's all you need for Chef. If something happens to your Chef server, Terraform can rebuild it from scratch. Although, you'll have to re-key your local client and tie all your instances to the new Chef server. This is a huge headache and I would not recommend ever doing that if you can avoid it. :P

---
Tags: chef, azure, vagrant, infrastructure-as-code
---


# Question
Title: How good are heartbeat, elasticsearch and kibana as a monitoring/alerting solution as a replacement for Nagios?
Since we already have filebeat, metricbeat and heartbeat configured and running.

I wonder if people have successfully implemented the ELK stack for actual monitoring with alerting/notifications instead of using traditional software as

* Nagios
* Icinga
* Zabbix

Especially since ElasticSearch now has had watchers and alerting for a while.

# Answer
I haven't tried it myself, but in Alarms management with Elasticsearch and Kibana someone recommended Elastalert
> 1 votes

---
Tags: elastic
---


# Question
Title: What is the best strategy for running an ansible playbook within an ansible playbook?
I have playbook that I've been hacking on for a year and it works OK, but I know it's not great, part of it runs ansible on the target machine with a "command" task. Because of that, I don't see the output of this task in my ansible playbook.

I need to run this play in the middle of the playbook because it is dependent on a prior task and a subsequent task depends on it (otherwise I'd break it up differently)

I'd like to be able to see the output of all my plays, is there a better way to branch off plays in the middle of a play.

For instance: given a playbook running on HOST\_A and HOST\_B at TASK 3, suspend running plays, run next plays on HOST\_C and HOST\_D then HOST\_E and HOST\_F?

```
- name: update other machines
  command: "/usr/bin/ansible-playbook /etc/ansible/playbooks/other-machines.yml -e 'ansible_user={{ item.ansible_user }}' -l {{ item.ansible_hostname }}"
  with_items: "{{ other_machine_connections }}"

```

# Answer
> 5 votes
If I've understood your problem correctly:

* the `ansible` shell command in the middle could be converted to a simple playbook using the `command` task - let's call this `middle.yml`
* your existing playbook could then directly import this `middle.yml` play (or even a more complex playbook). In Ansible 2.4+, you would use import\_playbook, specifically `import_playbook: middle.yml`, or in earlier versions, just `include: middle.yml`.

This processes the whole `middle.yml` playbook as if it was part of the main playbook - you have all the features of Ansible plays (`hosts` etc) and the output appears as part of main playbook.

This answer may need updating depending on your response to my comment above.

# Answer
> 3 votes
Take a look at Delegation \- you can delegate the execution of tasks to a different host from within a playbook:

```
  - name: enable the server in haproxy
    haproxy: 'state=enabled backend=myapplb host={{ inventory_hostname }} socket=/var/lib/haproxy/stats'
    delegate_to: "{{ item }}"
    with_items: groups.lbservers

```

---
Tags: ansible
---


# Question
Title: Docker MongoDB image - How to specify credentials other than in the compose file?
I'm using this docker image https://hub.docker.com/\_/mongo/. That image provides a way of initializing a mongodb root database user with env variables which I can specify in the compose file as follows:

```
environment:
  - MONGO_INITDB_DATABASE=db
  - MONGO_INITDB_ROOT_USERNAME=root
  - MONGO_INITDB_ROOT_PASSWORD=pass

```

It seems storing the password in config files is not very secure. How can I use more secure ways (i.e. docker secrets) to init the root user in this case? Same thing for initdb files, which I need to copy to the container and may as well contain more credentials.

I wonder if anyone here found a more secure way of handling this.

# Answer
Secrets support for mongo is built in. You can see in that image's entrypoint script that if you add `_FILE` to end of username/password values it will pull from those secret files.

As for .js files, if you're storing decrypted passwords in those files, you'd need to make a custom entrypoint script or something that will take envvars you're passing in for which secrets to use and run a `jq` or `sed` on the file to replace I guess. You can store the .js files themselves in a secret as longas its smaller than 500Kb (max size of a secret).
> 3 votes

# Answer
Here's a solution I've used in cases like this - I utilize Ansible to manage Docker containers, and Ansible Vault to store secrets for those containers.

### Ansible Playbook to run MongoDB container

Your `playbook.yml` may look something like this:

```
- name: run mongodb docker container
  docker_container:
    name: mongo-container
    image: mongo
    ports:
      - "27017:27017"
    env:
      MONGO_INITDB_DATABASE: "{{secret_db_name}}"
      MONGO_INITDB_ROOT_USERNAME: "{{secret_db_user}}"
      MONGO_INITDB_ROOT_PASSWORD: "{{secret_db_pass}}"

```
* As you might notice, the `docker_container` syntax looks a lot like what you'd write in a Docker Compose YML file.
* The difference is that your *secrets* are managed in variables (the `{{}}` is Ansible Jinja2 variables).
* Here's a list of Ansible Modules for interacting with Docker.

### Vault file to manage your `mongodb` secrets

The `vault.yml` file would contain the definitions of your secrets in an encrypted form.

```
secret_db_name: foodb
secret_db_user: foo
secret_db_pass: bar@123

```

You can use `ansible-vault` commands to create encrypted files (for e.g in version-control).

### Getting it all together

When you want to run your Docker container, you would run an Ansible command

`ansible-playbook playbook.yml --ask-vault-pass`

Which would

* ask you for your vault password file,
* decrypt the vault file and pass the variables to the playbook;
* run the docker mongodb container with your secret credentials

All whilst ensuring your credentials are not publicly visible.

### Notes

* I've used this approach to provision Dockerized databases on remote managed servers in a declarative way.
* You introduce one more tool, a 'wrapper' around Docker. In my experience DevOps *toolchains* work better than utilizing single tools. However, this may or may not be a constraint for you.
> 2 votes

---
Tags: docker, docker-compose, security, docker-swarm, databases
---


# Question
Title: Why does Helm time out on almost every action?
I'm trying to set up `helm` for the first time and I'm having troubles.

First, I've created account with `cluster-admin` role. (According to https://github.com/kubernetes/helm/blob/master/docs/rbac.md#example-service-account-with-cluster-admin-role ).

After that I've initialized brand new `helm` tiller by using `helm init --service-account=tiller` and it was successful.

Now when I'm trying to install something:

* First try:

```
$ helm repo add gitlab https://charts.gitlab.io
$ helm install --name gitlab-runner -f gitlab-runner-values.yaml gitlab/gitlab-runner

```

where `gitlab-runner-values.yaml` looks like this:

```
gitlabUrl: https://my-gitlab.domain.com
runnerRegistrationToken: "MY_GITLAB_RUNNER_TOKEN"
concurrent: 10

```
* Second try (as I was not sure if there is an issue with custom repo, so I tried from official):

```
$ helm install stable/kibana

```

I'm getting this error:

```
Error: forwarding ports: error upgrading connection: error dialing backend: dial tcp 192.168.0.18:10250: getsockopt: connection timed out

```

I noticed that `192.168.0.18` is visble on pod list:

```
kube-system   kube-proxy-kzflh                        1/1       Running   0          7d        192.168.0.18   kube-worker-7
kube-system   weave-net-jq4n4                         2/2       Running   2          7d        192.168.0.18   kube-worker-7

```

and that `tiller` is running on the same node:

```
kube-system   tiller-deploy-5b48764ff7-qtv9v          1/1       Running   0          3m        10.38.0.1      kube-worker-7

```

I was told that I probably don't have permission to `pods/port-forward` and `list pods`, but `kubectl auth can-i create pods/portforward` tells me that I can do this (the same with `list pods`)

Also `helm list` is throwing the same error as `install`.

# Answer
> 4 votes
The problem was that nodes, while registered with `kubeadm init` were providing their private IPs to the cluster master. This caused problems, because master was trying to reach `192.0.*.*` addresses which were not resolved as nodes from it's point of view.

I needed to edit `/etc/systemd/system/kubelet.service.d/10-kubeadm.conf` and specify public IP of the node on `--node-ip=<public-node-ip>` parameter. The reload the service and restart it as mentioned in https://github.com/kubernetes/kubeadm/issues/203#issuecomment-335416377

Then I registered nodes again and everything was working fine.

# Answer
> 3 votes
You have found a Kubernetes issue #22770, where there is a workaround mentioned here and it goes like follows:

> What you're experiencing is a known issue with k8s where for some operations it expects to be able to resolve your node names in the global DNS.
>
> And suggested a work around would be to:
>
> * Add entries to /etc/hosts on the master mapping your hostnames to their public IPs
> * Install dnsmasq on the master (e.g. apt install -y dnsmasq)
> * Kill the k8s api server container on master (kubelet will recreate it)
> * Then systemctl restart docker (or reboot the master) for it to pick up the /etc/resolv.conf changes

As last comment on the issue #22770 there is

> I can confirm that adding the names of the our nodes to /etc/hosts on the kubernetes master resolves this problem for us too :)

, so there is not a fix in k8s coming in any upcoming version. At least from this particular ticket.

---
Tags: kubernetes, helm
---


# Question
Title: Unable to run AWS CLI commands from Rundeck-No module named botocore.session
From terminal all works fine, botocore is installed

```
Traceback (most recent call last):
      File "/bin/aws", line 19, in <module>
        import awscli.clidriver
      File "/usr/lib/python2.7/site-packages/awscli/clidriver.py", line 17, in <module>
        import botocore.session
    ImportError: No module named botocore.session

 pip install botocore
Requirement already satisfied: botocore in /root/.local/lib/python2.7/site-packages
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /root/.local/lib/python2.7/site-packages (from botocore)
Requirement already satisfied: docutils>=0.10 in /root/.local/lib/python2.7/site-packages (from botocore)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /usr/lib/python2.7/site-packages (from botocore)
Requirement already satisfied: six>=1.5 in /root/.local/lib/python2.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore)

```

env:

```
SUDO_USER=ja
SUDO_UID=1000
USERNAME=root
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAIL=/var/spool/mail/ja
PWD=/tmp
LANG=en_US.UTF-8
SHLVL=1
SUDO_COMMAND=/bin/su
HOME=/root
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
SUDO_GID=1000
_=/bin/env

```

# Answer
Managed to get rid of error by

```
pip install awscli --force-reinstall --upgrade

```

then from rundeck interface:

```
aws configure set aws_access_key_id default_access_key
aws configure set aws_secret_access_key default_secret_key
aws configure set default.region us-west-2

```
> 1 votes

---
Tags: python, rundeck
---


# Question
Title: Error starting cloned VM in Vsphere Client
I cloned a VM in Vsphere Client as described here. When I try to start it, it unexpectedly reboot over and over itself. I get this error in console:

```
Call "EnvironmentBrowser.QueryTargetCapabilities" for object "45-envmgr" on ESXi "192.168.100.172" failed.

```

Which also shows another status message saying:

```
The object has been deleted or not created completely.

```

# Answer
> 1 votes
Have a look at this page which documents `EnvironmentBrowser` for clues of which setting to edit.

Possibly something has a **relative path that is now modified**, or a **resource that's fully used up** or no longer available (or no longer valid, but that makes little sense if it's the same VM host).

---
Tags: virtualization, vmware
---


# Question
Title: Robocopy is Running Slowly on Jenkins Server
I'm working to automate the deployment process for a Fullstack application using Jenkins. The app is deployed on a Windows Server. The .cmd script I have to automatically deploy the app is this:

```
@ECHO OFF
REM SET THE PARAMETERS FOR THE SCRIPT TO RUN
SET CONFIG_ENV=%1
SET SERVER_DOMAIN=%2
SET ADMIN_LOGIN=%3
SET ADMIN_PASS=%4
SET ARTIFACT_DIR=%~5
SET DROPFOLDER_PATH=%~6
SET DEPLOY_PATH=%~7
SET SERVICE_NAME=%8

REM ECHO THE PARAMETERS OUT TO THE CONSOLE (SO WE CAN SEE WHAT'S GOING ON)
ECHO ...
ECHO RUNNING WCF AUTO DEPLOYMENT. PARAMETERS ARE:
ECHO CONFIG_ENV      : %CONFIG_ENV%
ECHO SERVER_DOMAIN   : %SERVER_DOMAIN%
ECHO ADMIN_LOGIN     : %ADMIN_LOGIN%
ECHO ADMIN_PASS      : %ADMIN_PASS%
ECHO ARTIFACT_DIR    : %ARTIFACT_DIR%
ECHO DROPFOLDER_PATH : %DROPFOLDER_PATH%
ECHO DEPLOY_PATH     : %DEPLOY_PATH%
ECHO SERVICE_NAME    : %SERVICE_NAME%
ECHO ...

REM SELECT THE LATEST SET OF RELEASE ARTIFACTS FROM THE DSL
FOR /F "delims=" %%i IN ('dir "%ARTIFACT_DIR%" /b /ad-h /t:c /od') DO SET latest_build=%%i
ECHO DEPLOYING ARTIFACTS IN: %ARTIFACT_DIR%\%latest_build% (latest build)
ECHO DEPLOYING TO "\\%SERVER_DOMAIN%\%DROPFOLDER_PATH%"
ECHO ...
REM LOGIN TO DEPLOY TO THE SERVER
net use \\%SERVER_DOMAIN% %ADMIN_PASS% /USER:%ADMIN_LOGIN%

REM COPY ALL THE RELEASE ARTIFACTS TO THE DROP LOCATION ON THE SERVER
ECHO COPYING ARTIFACTS TO DROP FOLDER
ROBOCOPY "%ARTIFACT_DIR%\%latest_build%" "\\%SERVER_DOMAIN%\%DROPFOLDER_PATH%" /MIR /is /it /np /r:1 /w:1 /eta /tee /copy:DT
ECHO ...

REM STOP THE SERVICE SO THAT WE CAN UPDATE THE FILES
ECHO STOPPING THE SERVICE
SC \\%SERVER_DOMAIN% STOP %SERVICE_NAME%
ECHO ...

REM COPY ALL THE .DLLs and .EXEs FILES
ECHO COPYING .DLL AND .EXE FILES
ROBOCOPY "\\%SERVER_DOMAIN%\%DROPFOLDER_PATH%" "\\%SERVER_DOMAIN%\%DEPLOY_PATH%" *.dll *.exe /is /it /np /r:1 /w:1 /eta /tee /copy:DT
ECHO ...

REM COPY THE APPROPRIATE CONFIGURATION FILE FOR THE ENVIRONMENT
ECHO COPYING CONFIGURATION FILE
ROBOCOPY "\\%SERVER_DOMAIN%\%DROPFOLDER_PATH%\Configuration_Files\%CONFIG_ENV%" "\\%SERVER_DOMAIN%\%DEPLOY_PATH%" *.config /is /it /np /r:1 /w:1 /eta /tee /copy:DT
ECHO ...

REM START THE SERVICE BACK UP AGAIN
ECHO STARTING THE SERVICE
SC \\%SERVER_DOMAIN% START %SERVICE_NAME%
ECHO ...

REM AND WE ARE DONE
ECHO SCRIPT COMPLETE

```

If I run the script directly from where it's located on the Jenkins Server, the script finishes in a matter of seconds. However, when I queue the script to run from a Jenkins job, the ROBOCOPY command takes roughly a minute to copy each file.

If I just run the same script directly on the (exact same) Jenkins server, the process runs a lot faster.

What is causing the process to run slowly and how can I make the script run faster when executed from the Jenkins Job?

# Answer
> 2 votes
This issue is resolved. Robocopy has a lot of smart features and functions that allow it to check and see if the files being copied are actually different. Unfortunately that comparison is where the time is really going. Once we changed the script to delete the files from the destination folder first (every time) - we should have a repeatable process that runs in ~10-15 seconds.

---
Tags: jenkins
---


# Question
Title: Is it possible to load an image into S3 via an URL?
I'm going through some steps documented in How Do I Upload Files and Folders to an S3 Bucket? to get my ova into S3. Seems like a waste of time for me to download it and then upload it, is there a mechanism to import an S3 object from a web resource?

I have a UCS Director job that builds OVAs for me, uploads them to a Cassandra big data cluster and serves out obfuscated links for me and customers to download them - I'd like to provide said obfuscated link to AWS instead of downloading it on my computer and uploading it to S3. In the end, I think I could just make my images go straight from UCS Director to AWS, but for the time being, I'd like to save a step.

# Answer
> 3 votes
Nope, credit goes to this SO answer, still valid today: How to upload files directly to Amazon S3 from a remote server?.

But if you have a server in the cloud, ideally in AWS, it'll probably go faster to execute the copy operations on that server - higher bandwidth and lower latency to/from such server when compared to your computer.

---
Tags: amazon-web-services, amazon-s3
---


# Question
Title: VPN container providing access to host's LAN?
I'm kinda new to this Docker thing and I'm interested in getting a biffed up VPS for some containers (websites/data, mysql, nginx, rocket.chat, etc) but I wouldn't like to access them in the wild (even through SSH) and some VPS hosts's console access are plain cumbersome. So, here's my question:

I've got around 3 separate VPS sharing a common network: 10.0.0.1 thru 10. I'd like to know if, in the host (10.0.0.5) I can set up a docker server, then, inside I can set up an OpenVPN container (or any other kind of VPN) so I can connect from outside to the other VPSes (10.0.0.3, 10.0.0.8, and so on).

Bear in mind that I'm not looking for a privacy-related VPN, I don't want to route traffic nor go outside with the VPS's IP, I just want to "get inside" the internal network I have shared with my VPS (Vultr in case you're wondering, they have a private IPv4 service only for the VMs in my account so it's safe for them) through a VPN for added security and commodity, so I can access a few hosts naturally (and maybe containers as well?) using SSH/RDP/etc. with a single connection instead of having 10 profiles in PuTTY for each server.

Any pointers would be greatly appreciated. Please also take note I'm a newbie regarding docker, I'm a developer & IT guy so I know my way around, but I don't have much idea about networking so I'm not sure if it's possible - and if it is, then can you please tell me how?

Thanks!

# Answer
> 4 votes
I am accessing my home network with a VPN running in a docker container in my home server.

That sounds similar to what you want to do.

Here's how I configured it (using this docker image \- note that the documentation of the docker image should be enough)

1. Use a “convenience” environment variable to store the path to your persistent storage location that will be bind-mounted to the container.

```
OVPN_DATA="/n7wings/openvpn/"

```
2. Run an ephemeral instance (–rm) of the image to initialize the data directory of the container (ovpn-host should be the hostname of your openvpn server)

```
docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_genconfig -u udp://ovpn-host

```
3. Run an interactive ephemeral instance of the image to generate the opevnpn CA certificate and server key (you will have to type your passphrase for the private key)

```
docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn ovpn_initpki

```
4. Run the VPN service: start and detach the container (-d) and map a host port to the UDP container port where the openvpn server process is listening (1194). In this example the host port will be 1195

```
docker run -v $OVPN_DATA:/etc/openvpn -d -p 1195:1194/udp --cap-add=NET_ADMIN kylemanna/openvpn

```
5. Generate client configuration (i.e., add a user to the VPN). If you omit the nopass option, the client key will be encrypted with a passphrase.

```
docker run -v $OVPN_DATA:/etc/openvpn --rm -it kylemanna/openvpn easyrsa build-client-full vince nopass

```

The client key will be in `${OVPN_DATA}/pki/private` and the certificate in `${OVPN_DATA}/pki/issued`

1. Retrieve the client configuration to a local file:

```
docker run -v $OVPN_DATA:/etc/openvpn --rm kylemanna/openvpn ovpn_getclient vince > vince.ovpn

```

If you need to add more users, just repeat the last two steps to create a user configuration on the server and retrieve the ovpn file.

# Answer
> 0 votes
> I just want to "get inside" the internal network I have shared with my VPS \[...\] using SSH/RDP/etc. with a single connection instead of having 10 profiles in PuTTY for each server.

For what it worth you can achieve that pretty simply using your ssh client as a socks server and configuring your clients (RDP/Browse/Ssh client) to use a localhost: as a socks proxy and then access your inner network. A SSH tunnel is a tunnel as secure as a VPN tunnel; main difference with a VPN is there's no routing involved.

> through a VPN for added security and commodity

Forget this idea a VPN is for security, you won't add security to your network with a VPN, that's no more security than a SSH tunnel and usually even less when not properly configured. The commodity point is absolutely valid and a good reason to go this way :)

---

So clearly you can, I wonder why you want the VPN server to run within docker as this add a routing complexity you should avoid if you're not comfortable with the overall routing involved before.

Anyway the generic scheme for a request would be:

```
client stations --> client station routing table --> client VPN interface --> VPN host server --> target network

```

The problem involved is understanding a VPN is made of a tunel **AND** routing configuration, this needs some configuration to avoid breaking your connection to the VPN server once the tunnel is established (usual caveat is the vpn client setting up a new default route and thus breaking your workstation ability to talk with the VPN server through local gateway).

The VPN client job is to establish the tunnel and then configure your client station routing table to route packets for the destination network through the tunnel interface.

Vincenzo's answer to your question give the methodology to start an openVPN server container so I'll let you start from there but I highly encourage you to read more about the subject to avoid a false sense of security.

---
Tags: docker, containers, networking, vpn
---


# Question
Title: How to set a Git repository when creating a job in Jenkins from a Groovy script?
I'm trying to setup a Jenkins server with Groovy script:

```
def repository = 'git@somerepo'
import hudson.tasks.Shell;

job = Jenkins.instance.createProject(FreeStyleProject, 'TestJob')
job.setDescription("Some description")
job.displayName = 'SomeTestJob(TESTING groovy)'

job.scm = new hudson.plugins.git.GitSCM(repository)
job.save()

```

Now I need to set a repo branch - How do I do this?

# Answer
```
def repository = 'git@somerepo'
import hudson.tasks.Shell
job = Jenkins.instance.createProject(FreeStyleProject, 'TestJob')
job.setDescription("Some description")
job.displayName = 'SomeTestJob(TESTING groovy)'
job.scm = new hudson.plugins.git.GitSCM(repository)
job.scm.branches = [new BranchSpec('*/master')]
job.save()

```

Thanks a lot Michael Durrant for giving that link !
> 1 votes

---
Tags: jenkins, continuous-integration, git, release-management, groovy
---


# Question
Title: Ansible Expect Module hung on attempting a simulated return keystroke
I would like to use the expect module in an ansible playbook to telnet through a termserver to reach a NetApp filer. I tried the telnet module, but using the port argument after the ip address doesn't seem to work. Manually I would type the following:

```
[sansible@scspr0415857001 ansible]$ telnet 10.233.82.7 2033
Trying 10.233.82.7...
Connected to 10.233.82.7.
Escape character is '^]'.
Connected to port 33.

login: admin
Password:

rtpsb-f6280-02a-1::>

```

~~~~~~~~~~~~~~~~~~~

I think my issue is that the playbook is getting caught up on simulating a return keystroke after "Connected to port 33."

I've also tried using echo -ne '\n'

Playbook:

```
---
- hosts: localhost
  connection: local
  gather_facts: false

  tasks:
    - name: Expect Attempt
      expect:
        echo: yes
        command: telnet 10.233.82.7 2033
        responses:
          Question:
            - Connected to port* "echo"
            - login: "admin"
            - Password: "password"

```

The full traceback is:

```
  File "/tmp/ansible_piMb3N/ansible_module_expect.py", line 190, in main
    encoding='utf-8')

fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": "telnet 10.233.82.7 2033",
    "delta": "0:00:30.122472",
    "end": "2018-03-08 13:08:12.487220",
    "invocation": {
        "module_args": {
            "chdir": null,
            "command": "telnet 10.233.82.7 2033",
            "creates": null,
            "echo": true,
            "removes": null,
            "responses": {
                "Question": [
                    "Connected to port* \"echo\"",
                    {
                        "login": "admin"
                    },
                    {
                        "Password": "password"
                    }
                ]
            },
            "timeout": 30
        }
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2018-03-08 13:07:42.364748",
    "stdout": "Trying 10.233.82.7...\r\r\nConnected to 10.233.82.7.\r\r\nEscape   character is '^]'.\r\r\nConnected to port 33. ",
    "stdout_lines": [
        "Trying 10.233.82.7...",
        "",
        "Connected to 10.233.82.7.",
        "",
        "Escape character is '^]'.",
        "",
        "Connected to port 33. "
    ]
}
        to retry, use: --limit @/home/sansible/ansible/expect-r1.retry

PLAY RECAP   *********************************************************************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1

```

Any help would be greatly appreciated!

# Answer
> 1 votes
I figured out the issue. The "Question:" parameter needs to be omitted:
---
\- hosts: localhost
connection: local
gather\_facts: false

```
  tasks:
    - name: Expect Attempt
      expect:
        echo: yes
        command: telnet 10.233.82.7 2033
        responses:
          Connected to port 33. :  "echo"
          login: "admin"
          Password: "password"

```

---
Tags: ansible
---


# Question
Title: salt-stack: capture group for regex include_pat
is it possible to have a capture group with include\_pat when copying files ?

I would like to do something like that:

```
copy_template:
    file.recurse:
    - name: '{{ service_path }}\$1'
    - template: jinja
    - include_pat: E@^(.*)\.template$
    - source: {{ pillar['locations']['systemtoolsfolder'] }}\

```

# Answer
A capture group is a type of backreference. Saltstack, being python based, uses the re.match object to do regular expression matching, which appears to support backreferences, so these backreferences are allowed.

However, you be unable to pass the value of the backreference `\1` in the `- include_pat` field back to `- name` because the `- name` field is never processed by the `re.match` object. Even if it were, variable scope would prevent you from sharing the backreference between `re.match` calls.

Instead, you will need to do one of the following:

* Rename all the files in `- source` from `myfile.conf.template` to `myfile.conf`
* Use cmd.run to do a batch file rename after your `file.recurse` and add to your `file.recurse` an onchanges pointed to a previous event in order to defeat the idempotency (that is, to prevent it from writing out the .template files every time salt runs). You will also be sure to use a require and `onchanges` in your cmd.run that references `copy_template`

This will have the unfortunate side-effect that you will never be able to update your jinja templates. You will only be able to deploy them the first time - at least not without first deleting what is there, and then re-writing and re-copying from the source.
* Use `file.managed` for every template file.

For example:

```
copy_template1:
  file.managed:
    - name: '{{ service_path }}\template1.conf'
    - template: jinja
    - source: {{ pillar['locations']['systemtoolsfolder'] }}\template1.conf.template
copy_template2:
  file.managed:
    - name: '{{ service_path }}\template2.conf'
    - template: jinja
    - source: {{ pillar['locations']['systemtoolsfolder'] }}\template2.conf.template
...
and so forth

```
> 2 votes

---
Tags: salt
---


title: SVN Externals in Git

text:
# Question
Title: SVN Externals in Git
I have a handful of projects that utilize SVN externals. How should these externals be handled in Git? For reference, these externals are shared by multiple repositories and are updated very infrequently, but typically the changes are very impactful (non-backwards compatible).

# Answer
> 1 votes
One could `git svn clone` a subversion repository and use submodules.

---
Tags: git, subversion
---


# Question
Title: Continuous delivery workflow: Build script for PHP Composer (Drupal) on Docker
I'd like to set up a continuous delivery workflow for our new Drupal site. When someone pushes to our Git repo (typically) master branch, a build job on our CI server is triggered. This job will check out the source code, import the database from production, and build the Drupal project running in a Docker container.

I'm now looking to create the actual script that the CI job will run. I come from a Jenkins background, and there one would create a Jenkinsfile for this. We're probably not going to be using Jenkins for this project, so I'm looking for a tool that can be run regardless of the build server.

The most straight forward way would probably be to script this build using bash or python or something. I'm quite new to both Drupal and PHP, so I'm wondering if there are any tools out there similar to Jenkinsfile I can make use of, which I can run on Gitlab or whatever other CI server we go for?

# Answer
Each CI/CD system is different when it comes to performing the job(s) that the execution pipeline consists of. In most (if not all) cases the details are part of a configuration which is specific to the particular CI/CD system used. As you mentioned Jenkins uses a Jenkinsfile, GitLabCI uses `.gitlab-ci.yml`, others use "live"/GUI-based configurations, etc.

So I don't think you can find a common solution *directly* pluggable in an arbitrary CI/CD system. But you can get close to that: fundamentally each of these configurations are simply adaptation layers between the CI/CD system and the automated task(s) to be performed.

So if you can create a (set of) standalone script(s) to perform your automated step(s) in as portable way you need as possible you *should* be able to integrate with in almost any CI/CD system without too much trouble: any decent CI/CD system should support a custom script invocation mode to cover for whatever special case that doesn't properly fit into its pre-defined cases.

Using python/bash is generally quite portable, up to you to decide if that indeed covers all possible execution environments you'd be willing to consider.

I'd recommend structuring such script itself as a wrapper around smaller scripts implementing its logical/functional steps - some CI systems may require individual control of the steps or may offer advantages if such control is provided. In your case I'd have separate scripts for:

* checking out the source code
* importing the database from production
* building the Drupal project running in a Docker container (maybe this can/should be further split if it makes sense?)

And I'd add a wrapper script tying them together for one-shot end-to-end execution, either by CI/CD systems not interested in the individual steps or even for manual invocation.
> 1 votes

---
Tags: docker, continuous-deployment, php, drupal, composer
---


# Question
Title: Expose kubernetes deployment on master's IP
I have simple deployment which has 3 replicas. I want this deployment to be load balanced between this 3 replicas, but available under IP of my `master` and on some specified port.

This cluster was created by using `kubeadm`, I have one master and 20 worker nodes.

* I tried to expose the service as `LoadBalancer`, but this only works when you have cloud load balancer configured. I don't have such thing and `EXTERNAL-IP` is always `<pending>`.
* I tried to expose the service as `NodePort`, but this only works on nodes where `pods` were created. So for example after rollout, when `pods` will be created on different nodes, it will stop working. Also I guess this is not load balanced properly.
* I tried to create `ingress` to make it work, but unsuccessfully.

```
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: registry-ingress
  namespace: registry
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  rules:
  - http:
    paths:
      - path: /registry
        backend:
          serviceName: registry-service
          servicePort: 5000

```

What is the proper approach here?

---

If the above is possible then second question: is it possible to expose it on 2 masters if I will want to create 2nd one?

# Answer
I was wrong about the `NodePort` approach. It was firewall blocking the connection to my master on ports `30000-32767`.

After unblocking this ports everything was working fine with `NodePort` service.
> 1 votes

---
Tags: kubernetes
---


title: What is Systems Thinking?

text:
# Question
Title: What is Systems Thinking?
The "First Way" in Gene Kim's explanation of DevOps requires "Systems Thinking" https://itrevolution.com/the-three-ways-principles-underpinning-devops/

Where can one learn the ability for "Systems Thinking"? What are some of the resources that can take someone who has no familiarity with "Systems Thinking" and enable that person to start doing "Systems Thinking" in his daily work?

# Answer
> 4 votes
Googling "system thinking" returned a lot of resources that are about "Systems thinking" rather than "System thinking". This source indicates that "Systems thinking" is one of the three ways that could be applied in DevOps.

After reading several references "Systems thinking" is about interactions rather than silos. If one translates this to DevOps then "Systems thinking" is aligned with the theory and also important for DevOps as this is the interaction between QA, Dev and Ops.

In order to apply "Systems thinking" one should be able to communicate with people from Dev, Ops and QA, listen to them, understand their problems and find ways to let them work together instead of getting a "through it over the wall" mentality and unconnected departments in the company.

An element that really requires "Systems thinking" is Continuous Delivery (CD). Multiple departments are involved, while CI is mostly used by Dev, QA and Ops will be definitely need to be involved in this process as one does not want to release a newer version if integration or manual tests failed and Ops want to be informed if there are changes to the system as they monitor systems and try to prevent and solve issues on the platform.

In summary, "Systems thinking" is required for DevOps engineers. If one only sees Dev, Ops and QA as individual parts, but no interdependency then this is not aligned with the DevOps theory, i.e. intersection between Dev, Ops and QA.

**References**

https://medium.com/disruptive-design/tools-for-systems-thinkers-the-6-fundamental-concepts-of-systems-thinking-379cdac3dc6a

http://www.thwink.org/sustain/glossary/SystemsThinking.htm

> a discipline for seeing the 'structures' that underlie complex situations, and for discerning high from low leverage points.

https://thesystemsthinker.com/connecting-systems-thinking-and-action/

> The iceberg is one of the most frequently used models to explain systems thinking (see “The Iceberg”). Thanks to movies like “Titanic,” many people recognize that most of an iceberg sits hidden beneath the water; that is, out of sight.

https://www.accenture.com/us-en/blogs/blogs-devops-evan-erwee-system-thinking

> System Thinking allows people to make their understanding of social systems explicit and improve them in the same way that people can use engineering principles to improve their understanding of mechanical systems.
>
> ...
>
> We can therefore state that System Thinking for DevOps is required to understand Software Engineering as a social construct using engineering principles.
>
> From a System Thinking approach, DevOps thinking is fundamentally different from the traditional form of analysis. Traditional analysis focus on separating the individual software engineering pieces, e.g., architecture, coding, testing, tooling etc. In contrast, the DevOps approach focuses on how all aspects of the software life cycle interact with other constituents of the system. Circling back to the definition of DevOps we can easily identify the ‘behavior traits’ observed in the DevOps movement, namely: adaptable/flexible (improve relationship), cooperative (better communication), diplomatic (collaboration), etc.

http://www.systemicleadershipinstitute.org/systemic-leadership/theories/basic-principles-of-systems-thinking-as-applied-to-management-and-leadership-2/

> Systems thinking is a management discipline that concerns an understanding of a system by examining the linkages and interactions between the components that comprise the entirety of that defined system.

https://thesystemsthinker.com/introduction-to-systems-thinking/

> What exactly is systems thinking? In simplest terms, systems thinking is a way of seeing and talking about reality that helps us better understand and work with systems to influence the quality of our lives. In this sense, systems thinking can be seen as a perspective. It also involves a unique vocabulary for describing systemic behavior, and so can be thought of as a language as well. And, because it offers a range of techniques and devices for visually capturing and communicating about systems, it is a set of tools.

https://www.burgehugheswalsh.co.uk/Systems-Thinking/Approach.aspx

> Fundamentally, Systems Thinking is straightforward. We treat the situation of interest as a system and examine it using system properties and characteristics like purpose, context and structure to discover more.

http://study.com/academy/lesson/systems-thinking-in-management-definition-theory-model.html

> System thinking is a method of critical thinking by which you analyze the relationships between the system's parts in order to understand a situation for better decision-making. In simpler terms, you look at a lot of the trees, other plants and critters living around the trees, the weather, and how all these parts fit together in order to figure out the forest.

# Answer
> 3 votes
I believe that systems thinking can be found within the three DevOps ways. For me, systems have properties that DevOps addresses very well. I've written something about in https://opensource.com/article/18/3/how-apply-systems-thinking-devops

# Answer
> 0 votes
The author of that article has misappropriated the term "systems thinking" which has its origins in biology and sociology with Ludwig von Bertalanffy and his General Systems Theory (1968), and was later applied to cybernetics (in its original meaning of feedback loops in living and nonliving contexts - cells, machines, organisations, before "cybersecurity" and similar buzzwords hijacked it). In short you cannot learn about it because it is not a thing that exists in the way he uses it, just something he made up to sound clever - but if you are interested in *real* systems thinking, I'd say start from GST and work your way forwards.

(Source: I have a Master's in Systems Analysis, before that term too was purloined to mean "IT support")

---
Tags: terminology
---


title: Conditionally run certbot

text:
# Question
Title: Conditionally run certbot
I am running an `ansible` task that will among others install a let's encrypt certificate.

I want the task to only run if the certificate has not been installed.

What is the best way to check (via cmd) if my host has a valid secure connection over https?

# Answer
One could use the `apt` module of Ansible. http://docs.ansible.com/ansible/latest/apt\_module.html

The `apt` module uses `apt` and apt itself is able to check whether a package has already been installed and will prevent that it will be installed as it has already been installed.

I use this module myself as well. As the idempotence test passes it indicates that it will not install certbot if it has been installed already.

https://github.com/030/ansible-certbot/blob/master/tasks/Debian.yml#L10

```
- name: certbot installed
  apt:
    name: certbot
    state: latest

```
> 1 votes

---
Tags: ansible, ssl
---


title: How set and get var with Path?

text:
# Question
Title: How set and get var with Path?
I write script for deploy files by ssh via Jenkins

```
#!groovy

node('superhost01'){
    String HOSTNAME="host01"
    String USERNAME="tech_user"
    withEnv(['PATH=/data/jdbc_connector'])
    stage('Prepare') {
        checkout scm
    }
    stage('Deploy') {
        sh """
        scp -r config.yaml ${USERNAME}@${HOSTNAME}:$PATH
        """
}

}

```

BUT Jenkins returned error in line

```
withEnv(['PATH=/data/jdbc_connector'])

```

error:

```
[Pipeline] node
Running on superhost01 in /data/jenkins/workspace/jdbc_connector
[Pipeline] {
[Pipeline] withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
[BFA] Scanning build for known causes...
[BFA] No failure causes found
[BFA] Done. 0s
java.lang.IllegalStateException: There is no body to invoke
    at org.jenkinsci.plugins.workflow.cps.CpsStepContext.newBodyInvoker(CpsStepContext.java:283)

```

how to correctly assign the value of a variable?

# Answer
`withEnv` doesn't persist environment variables. Instead, you need to tell `withEnv` what steps to run with that environment by passing it a block. Here is what I think you want your code to look like:

```
#!groovy

node('superhost01'){
  String HOSTNAME="host01"
  String USERNAME="tech_user"
  withEnv(['PATH=/data/jdbc_connector']) {
    stage('Prepare') {
      checkout scm
    }

    stage('Deploy') {
      sh """
      scp -r config.yaml ${USERNAME}@${HOSTNAME}:$PATH
      """
    }
  }
}

```

As a side note, I suspect you actually want to set `PATH` to `/data/jdbc_connector:$PATH`, otherwise your checkout step will not be able to find git and your deploy step will not be able to find scp.
> 0 votes

---
Tags: jenkins, jenkins-pipeline
---


# Question
Title: How do I get Openshift to let Tomcat retain it's hosted site's changes
I have a pod in Redhat Openshift running Tomcat and inside Tomcat I have Ozone Widget Framework, my site hosted in Tomcat. I have executed the `oc` command to use my existing configuration so that my pod won't disappear and lose my session and data. I ran

`oc cluster up --host-data-dir=/path_to_data/openshiftdata --use-existing-config`

But after making changes within OWF, when I take down the cluster and bring it back up my pod lives on, but my OWF instance is reverted back to it's default state.

How do I get Openshift to let Tomcat retain OWF's state?

# Answer
You need to use a Persitent Volume.

Your `/path_to_data/openshiftdata` is probably on a disk that belongs to the pod and is being destroyed each time.
> 1 votes

---
Tags: docker, kubernetes, openshift
---


# Question
Title: How to help DevOps Engineers feel less like a lone wolf?
I have just been speaking to a DevOps guy who raised some really good points about the struggles of being a DevOps Engineer and feeling like a one-man army sometimes, even though he is in a team of 16 Engineers.

He wears lots of different hats, but sits in the Development team doing infrastructure work. He loves the cool tech that he gets to work with- loads of automation, cloud, containerization etc. But he struggles that he is the only person doing `ops`, in a `dev` team. He reports to the Development Manager, but works more closely with the Infrastructure Manager.

This seems to be the case with lots of DevOps professionals I speak to. What could be done to help DevOps Engineers feel less like a lone wolf?

# Answer
> 41 votes
My first thought is "why is the he the only person doing ops, on a dev team, especially when he gets to work with loads of automation?". I think there's an opportunity there to address the lone wolf syndrome; particularly in a dev environment, infrastructure-as-code provides a great framework for sharing the load. Ops people should be experts in determining and defining infrastructure needs for application, but they should also be able to build a platform to let dev roles do as much as they can independently.

It sounds like a silo within a team; old habits die hard. A coder may not feel comfortable spinning up and hardening a server, but in a pure devops model, they should have the tools to do so. An ops person in a devops team shouldn't responsible for delivering infrastructure for the app itself, but they should provide some insight into what is needed and some guidance on how the app developers can do it themselves. It's almost a meta-infrastructure model; ops engineers build infrastructure that can build infrastructure on demand when requested by the development team.

Consultation, communication, and blending of responsibilities are all crucial to the success of a devops team.

# Answer
> 31 votes
I think the first flaw is in this sentence:

> He reports to the Development Manager, but works more closely with the Infrastructure Manager.

DevOps is a cultural shift aiming at removing silos. If silos remain, then this engineer is whatever you want to name call him; an engineer doing operational development, an automation expert, a developer automating infrastructure - but this engineer is not a DevOps engineer.
In fact, "DevOps Engineer" is not a real role, it's more a 'chapeau' as it can encompass developers, system administrators, QA testers and architects working in a common team.

A problem I often see is that people fall into the buzzword usage of DevOps, seeing it as a job title, but they don't really understand what DevOps is. By viewing DevOps this way, they often end up isolated and feel alone, blaming failures and shortcomings on being a "lone wolf" without managerial and organizational buy-in.

As you describe it this engineer is the only one doing Ops in a Dev team. That doesn't make him a "DevOps engineer". (Whatever that means in his organization) He's working in isolation because the job is presented as "DevOps Engineer" but it seems tha other people in his team doesn't wish to work on operations.

Let's be honest, there will always be ops and dev, the main idea behind devops is sharing the responsibilities such that there's no handoff of a product from dev to ops or just supplying a platforms by ops for dev. The primary goal is bringing more collaboration into a team. Calling this role a "DevOps Engineer" is breaking this idea by suggesting in the name you can do both at the same level of expertise - which is rarely true.

The first thing to do in my opinion would be to present the operational tools to the team and give everyone a basic knowledge on the tools, then transfer the responsibility of configure/code the operational tools to the whole team. The main idea behind this is moving from "the one doing all the ops things" to "the one which supports and give references implementations to the team".

This complements the other answers by providing something actionable in an simpler way as a first step than a management reorganization.

# Answer
> 18 votes
The most important thing for DevOps Engineers in this kind of situations, is to get (a) ***Management Commitment*** and (b) ***Required Budgets***. Read on for some more details on both ...

# Get Management Commitment

Once that is in place, things become easy for such DevOps engineers. Especially whenever resistance (from all sorts of parties) comes into the game. Trust me, there will be such resistance, which challenges such as:

* Why do we have to change? I want to keep doing what I did for X years already!
* I do not want to loose the (technical) power I have, and complete all sorts of workflow procedures, to get a silly fix in production that should take me like 5 minutes instead of 5 hours (or days ...).
* ... (I could add another dozen bullets here ...).

Whenever those challenges come up, all a DevOps engineer should have to say is like:

> ***I'm sorry, I'm just doing my job ... based on instructions from upper management.***

# Get Required Budgets

An effective way to get Required Budgets, is to create/submit an appropriate business case which explains the tangible and intangible benefits of various DevOps practises by applying them to some real world cases that apply to the company itself.

Below are some real world cases I experienced myself, as an SCM consultant hired by some companies where these things had happened. I know, SCM is only part of DevOps, but it's the area where I have *some* expertise ...

### 1. Benefits of automation

Due to some strike from only 2 (!!!) computer operators (who did not type the console commands anymore that they where expected to type), trains had to be halted somewhere half the way between 2 factories (since the system at the factory where the train was heading to was down, crucial data about handling the train was not available).

By implementing an SCM system, many operator commands were automated.

### 2. Reduce software license costs

Some software vendor had decided to increase some yearly fees for the (outdated) SCM software, which management didn't agree to. Therefor they created a special project to get it replaced by some alternative SCM software.

The budget of the project was equal to the yearly fee they didn't want to keep paying. That included flying in engineers from other continents (like me) to make the project succeed.

### 3. Reduce operating costs

Some major insurance company was using some FTP software to transfer software fixes to about 13.000 midrange computers (AS/400s) throughout the country, and this whenever "a" fix became available. The cost of 1 such transfer was about 4 USD (13.000 x 4 = 52.000 USD for a single transfer ...). The software consisted of 120.000 components, developed/maintained by about 150 developers. Do the math about the probability that 1 developer made 1 (tiny) mistake in any of these 120.000 components, which made it to production, and required an urgent fix, which would cost another 52.000 USD (just for the transfer!).

By implementing an adequate SCM system (with managed test environments, approvals, etc), this company achieved a major cost reduction. Think about it, if the SCM system could prevent the need for only 20 transfers of urgent fixes, it resulted in a cost reduction of 52.000 x 20 = 1.040.000 USD (quite a budget to implement an SCM system, they only needed a fraction of that amount to get the job done).

### 4. Reduce costs of unavailability

If the above cases are not convincing enough, then think of the system(s) of a major credit card company being unavailable around the world. I've been told that 1 second of unavailability costs them 1.000.000 USD.

That's probably also the reason why, for a very long time, such companies have sophisticated DevOps tools in place, for many decades already. Because each second they are not in business costs them a fortune.

# Answer
> 5 votes
TL;DR: Since upper management is usually fickle and prone to anger, I'd suggest trying to bend his mind a bit to get a different perspective, while changing things for the better, gradually.

(I am assuming that his trouble is mainly with the reluctant *devs*, not his other *ops* colleagues who seem to do classical operations mainly.)

IMO, even if you have DevOps going, that does not necessarily mean that every developer needs to be a full fledged DevOps guru. I find it rather normal that there will be one or two real experts in a given group of people, and the rest more or less tag along. As long as the workload is not too big for that guy, and as long as he manages to encapsulate his knowledge in scripts etc. instead of building his own silo, it is fine with me.

The one thing that should *not* be happening is that the DevOps guy does his automation, and everybody else tries their hardest to avoid said automation (i.e., go past the CI/CD pipeline and do stuff manually in one of the environments). This, IMO is the main thing that has to stop. One solution for that would be to push very hard for the cattle-not-pet approach, i.e. relentlessly tear down VMs or containers left and right as soon you can, and spin up new ones continuously.

Second, of course everybody needs to be *aware* of what the automation is doing, and at least in theory, with some digging around maybe, be able to start the automation machinery (i.e., if everything runs from a commit/push, then the devs need to be aware and very much up to date with the fact that there will be stuff happening in the background when they commit). The CI(/CD) pipeline needs to be quite visible and should be a thing everybody is constantly aware of (i.e. when a dev breaks it).

Third, the "one guy" of course has to watch out that he does not do menial, everyday tasks for his colleagues (e.g., creating Dockerfile after Dockerfile for *their* artifacts...).

Fourthly, the solutions the DevOps guy creates of course have to *actually be superior* to the manual approaches of the past in some measurable way. In that case, it should be possible for him to *demonstrate* his improvements; i.e., demonstrate how things get easier for everybody, or how it seems to be impossible to introduce bugs in the later stages of the pipe etc. If this does not seem possible, then the DevOps guy needs to take a good long hard look at what he is doing. If it is possible, that calls for brownbag sessions and plenty of evangelisation in his team.

Obviously, in such a reluctant environment, you will probably not be arriving at a wholly automated CD solution, or at trunk based development anytime soon. But I would not worry too much about being singled out. He is the expert, and if he is doing his job well, the whole team will very gradually improve.

And finally, if after years and years of toiling there is just no visible improvement with his colleagues, it is always possible to look for other avenues (be it inside or outside the company). Having all that DevOps experience under his belt is excellent base for looking for a job, these days...

# Answer
> 5 votes
I see a lot of great answers here talking about DevOps as a culture, suggestions about how to work with management, and help defining the do’s-and-do-nots of a DevOps team or engineer. I think each of them is great, and really illustrative of many answer can be 100% right, and still be very different, or even completely ambiguous, from each other... That’s DevOps!

This answer is just my unique perspective from experience, and may not be indicative of norms or best practice...

But what your DevOps colleague was complaining about **is the very nature of what makes DevOps challenging and difficult**, especially when appointed the role of DevOps engineer, and not simply being a cultural mindset.

Personally, I ***like*** being a lone wolf, because I still make valuable contributions, but also get to set my own limits, whilst persuading others to help themselves, thereby helping me, breakdown IT silos.

**Some silos remain intact**, and that’s okay, it’s DevOps mission to work around that and try to make that silos as insignificant or invisible as possible.

It’s possible your coworker may be realizing, or hasn’t yet realized, that **he or she does not like being a DevOps engineer**.

# Answer
> 3 votes
Relatively speaking, the concept of devops is new and still defining itself in my opinion. I currently fulfill a devops engineer role. For me, this means I facilitate and develop the tools and processes used by both our dev and ops teams freeing them to focus on the product that generates revenue for the company. The ops and dev teams spin up their own servers and such as needed. I just hook up the CI for our products, ensure our processes makes sense and seek out what process can be improved/automated. I meet with all of our departments, from sales, to warehouse, to developers and operations (QA and release managers) to see what they are doing and how I can improve their process.

# Answer
> 2 votes
For me, DevOps means that the development and operation of a software system becomes the responsibility of one team, instead of separate dev and ops teams. **This is a two-way street.** The best teams are made up of "T-Shaped" people who are experts in one field and familiar with several related fields.

* Team members with Ops experience are expected to do what they do best (i.e. Ops), to teach the fundamentals of what they do best (i.e. Ops) to the others, and to learn and do jobs in related fields (i.e. Dev tasks).
* Team members with Dev experience are expected to do what they do best (i.e. Dev), to teach the fundamentals of what they do best (i.e. Dev) to the others, and to learn and do jobs in related fields (i.e. Ops tasks).

So to make the DevOps engineer feel less like a lone wolf, invite him to teach the developers how to **run** the systems, while acknowledging that he is the expert how to **design** the infrastructure.

Get him involed in high-level architecture from the beginning, so that he can introduce the concerns of his specialty. (Before we had DevOps, our architecture drawings always glossed over "little things" like load balancers and redundant servers. Now such things are part of the very first sketches.)

Expect the Devs to take some of the daily routine Ops tasks, both to build redundancy into the team and to spread the "drudge" jobs fairly.

Expect him to contribute to the Dev effort if there are no Ops-like tasks to be done. Some DevOps I know seem to find database work a natural extension of their area of expertise, not sure if that can be generalized.

# Answer
> 1 votes
> What could be done to help DevOps Engineers feel less like a lone wolf?

To paraphrase - **what can a DevOps Engineer do *himself*/*herself* to feel less like a lone wolf?**

Lack of culture and management support is only one part of the equation. The other part is in my opinion that the detail DevOps knowledge often refers to complex contexts and it is important to have advice and reference to working examples.

Therefore - do not feel like a lone wolf; participate in DevOps communities like this one here or tool-specific groups and GitHub - the feeling is at least then you are not the only lone wolf ;-)

---
Tags: culture
---


# Question
Title: Bugzilla 4.2 to 5.0.4 'Can't locate parent.pm in @INC'
I've just updated Bugzilla from 4.2 to 5.0 on our Centos 6 vps, using the tarball method:

```
bash$ cd /var/www/html
bash$ wget https://ftp.mozilla.org/pub/mozilla.org/webtools/bugzilla-5.0.4.tar.gz
bash$ tar xzvf bugzilla-5.0.4.tar.gz
bash$ cd bugzilla-5.0.4
bash$ cp ../bugzilla/localconfig* .
bash$ cp -r ../bugzilla/data .
bash$ cd ..
bash$ mv bugzilla bugzilla.old
bash$ mv bugzilla-5.0.4 bugzilla

```

I'm getting the following error from `./checksetup.pl`:

> Can't locate parent.pm in @INC (@INC contains: . lib /usr/local/lib64/perl5 /usr/local/share/perl5 /usr/lib64/perl5/vendor\_perl /usr/share/perl5/vendor\_perl /usr/lib64/perl5 /usr/share/perl5) at Bugzilla/Constants.pm line 14.
>
> BEGIN failed--compilation aborted at Bugzilla/Constants.pm line 14.
>
> Compilation failed in require at ./checksetup.pl line 27.
>
> BEGIN failed--compilation aborted at ./checksetup.pl line 27.

What did I do wrong and/or how to fix?

# Answer
> 1 votes
I needed to install a new perl module

```
yum install perl-parent

```

then I had to install `make` to install other dependencies:

```
cpan
o conf make '/usr/bin/make' #or path to your make
o conf commit

```

---
Tags: sysadmin
---


# Question
Title: Cassandra persistence with Docker - is an example of a comprehensive, up-to-date and facts-based guideline for a deployment architecture available?
Being new to this topic, I have found quite many pros and cons about making data persistent while working with dockerized Cassandra. However, most of it was posted in 2016.

Now, I understand that it can depend on the specific case, but already googling the very question reveals a lot of debate which appears sometimes opiniated (and maybe also historically related to earlier versions of Docker and Cassandra which might have been improving over time and will in future, but maybe there is a conceptual flaw).

So I post this question to differentiate the pros and cons in a non-opiniated way in order to support finding a source or an answer which allows an architectural decision based on facts (or draws a working example, even for a specific case).

Myself is quite biased to Docker but as a DevOps engineer I think I have to learn which things work and which not, also understanding the WHY behind *both*.

*Pros or seems to be quite ok but to prove:*

1. Production, yes or no: is there an real-world example of a large-scale Cassandra in a Swarm/Kubernetes environment? I thought, initially, yes?
2. Guidelines like "Cassandra and Docker - Lessons learnt" (Slide 20)
3. Docker Official Image packaging for Cassandra on GitHub

*Cons/Criticism:*

1. Articles like "Thou shall not run a database in container" or "Why database are not for containers" (related yc debate)
2. A colleague told me, Cassandra would store data associated with IP addresses and this makes usage of Docker/Cassandra not possible if you need data persistence - can't yet google up this but if there is no workaround this will make persistence impossible in my opinion.

**Side note.** Sometimes they say also, "don't do it in production #1, but integration environment is okay also without persistency #2". I want to learn more about #1 through this question and can't agree with #2 because sometimes you can have a huge effort to prepare already the sample data you need in the integration environment because the production data is yet bigger/complexer.

# Answer
Since you posted your original question, Datastax has come out with dockerized images of datastax enterprise, along with a comprehensive guide. I'd start with this reference and maybe check out this older "Best Practices" doc, although some details might have changed.
> 2 votes

---
Tags: docker, cassandra
---


# Question
Title: fatal: ambiguous argument 'master': unknown revision or path not in the working tree
when I modified one of my cookbook that I tried to add a cookbook dependency in berks file

```
cookbook name, '=0.1.0', git: 'ssh://url:xxx/cookbook', tag: '0.0.1'

```

and then tried to converge it which throws an error

```
'fatal: ambiguous argument 'master': unknown revision or path not in the working tree.use '--' to separate paths from revisions, like this:git <command> [<revision>...] -- [<file>...]'on test

```

I don't have much clue to explain even..but I can post my explanation as we go. I appreciate your help thank you

# Answer
The error looks like you're using a non-existent reference (branch/tag). Try specifying the correct branch and tag using the `branch/tag` options. You can also use the commit hash directly using `ref` option.

Also ensure that your git URL is correct (your code does not contain the xxx.git part).
> 5 votes

# Answer
I recently ran into the same error. In my case, I had rebased a branch and was attempting to push to the remote. One of the pulled commits had changed the structure of a few directories, and git refused to push after the rebase, throwing:

```
fatal: ambiguous argument 'path/to/scripts': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
Cannot open directory /absolute/path/to/scripts: No such file or directory
error: failed to push some refs to 'git@host:repo'

```

I tore apart all of my git configs and could not find the cause of this anywhere. The culprit turned out to be a pre-push git hook, left over from some previous config we used to use, which included some operations on the now-defunct directory.

While this clearly wasn't the cause of the OP's issue, thought I'd post this here in case someone else gets bitten. Beware the git hooks...
> 3 votes

---
Tags: git, chef, gitlab, github
---