# Question
Title: What practices or tools enable Continuous Deployment of Databases?
Continuous Delivery or Continuous Deployment of infrastructure and code is comparatively simple compared to trying the same approaches for databases, specifically RDBMSs. Code and infrastructure will not change or evolve once deployment has completed. Databases, however, will have new data added to them making the data if not the schema inherently mutable components.

I am aware that there are practices such as only adding database objects, i.e. tables and columns, never modifying or removing them - this purely additive way of approaching database schemas has the advantage of ensuring that schemas are backwards compatible at the cost of successively more complicated schemas.

Equally there are products such as Flyway and Ready Roll which assist in writing migrations to be written between versions of a schema.

What other practices and tools exist currently to allow for continuous deployment of database schemas into a production environment where data integrity is of a concern?

# Answer
Pramod Sadalage and Scott Ambler wrote a book Refactoring Databases: Evolutionary Database Design that is an incredibly solid primer to the subject of DBs in a CD org/team.
> 15 votes

# Answer
## The Challenges

---

> I am aware that there are practices such as only adding database objects, i.e. tables and columns, never modifying or removing them

At one company I worked for, a rolling window of raw data equated to about 6 months of data and ate up 10 TB. The data was then processed into an RDBMS format, with some data portions hitting the cutting room floor. After, this dataset still weighed in at 6 TB of usable data which accounted for about 10 years of reportable data.

The point being that at scale, these kinds of practices (never modifying or removing data) simply isn't practical. Storage is expensive - probably the greatest compute expense of most companies. This provides several interesting challenges:

1. **Backups** \- the InnoDB plugins are great and all, but the backup times on that much data just isn't that practical.
2. **Restore times** \- for large datasets - especially if you need replication to catch up after a restore getting back to an operational state can take days or even weeks.
3. **Creating/seeding new instances** \- Often the work you may be doing in dev/test involves ETL (Extract, Transform and Load) jobs on your dataset. These jobs need to be validated using QA testing units, but this needs to be done in a non-destructive manner so that the original production dataset is preserved. While in a disaster, you may be willing to deal with a long restore time on the understanding that backups are an insurance policy (and the intent is to avoid disasters) the DevOps development workflow requires that, essentially, you be able to perform a restore or copy of your data on a regular basis (perhaps multiple times a day).
4. **Capacity** \- Slinging around that much data at the scale I just described can be very I/O intensive. Not only do you need to address the problems described in 1-3, but you need to do it in a way that doesn't cause outages or performance slowdowns to your production systems.

## While the above considerations may not be a concern at smaller scales, at larger scales, these become huge problems to tackle. This means that it is ***extremely important*** that you define your requirements and forecast the size of your dataset.

Defining requirements

---

As a result of that excercise, you need to define several requirements:

* **RTO** \- RTO or Restore Time Objective for backups is one of the most important drivers of database backup solutions. While at first it might not appear relevant to most of the other problems described, it becomes extremely relevant when you ask "What if I used my backup solution for creating or seeding new instances?" (I'll cover some techniques for doing that in the next section)
* **RPO** \- RPO or Restore Point Objective for backups defines A) how far back you are able to restore to (minutes, hours, days, weeks, months, or years) B) The backup interval or frequency at different tiers and C) how granularly you can restore. For example, for E-mail databases, Message Level Backups - restoring a specific E-mail - are often sought. Similarly, you may find that data older than a few days is completely useless - so there is no point restoring back a year. It depends on the kind of data you are working with.
* **Size of your dataset** \- This is important because for a 1MB database, your RTO may be achieved with most backup products and solutions. For a 10TB database however, you will find that a full, row level backup to LTO 3 tape *probably* won't achieve your RTO and could interfere with your RPO because backups begin exceeding your backup window. You can't exactly just do a mysqldump on this large of a dataset, but could probably get away with that on the 1MB database.
* **Database consistency** \- A final thing that makes an immense difference in continuous deployment, site reliability, scalability and high-availability when working with databases is your need (or lack thereof) for consistency. There are three basic types: immediate consistency, Just-In-Time (JIT) consistency and eventual consistency.

In addition to the above major considerations, you also need to consider:

* Licensing and support requirements (open source or closed source; in house support, third party support or vendor support, and does this fit your budget?)
* Application/language requirements (the connectors for many databases can be important; is your app compiled? Do you have access to the source code? Can you recompile it, or is it provided by a vendor? Does it run on an interpreted language?)
* Political requirements (Does your organization only trust Oracle? Do they hate Oracle? Do they not trust MySql? How do you feel about MariaDB or Postgres? Do we require an EAL certified DB product due to contracts?)
* Database engine (innoDB? MyISAM? Blackhole? NDB Cluster? Spider?)
* Historical or compatibility requirements (we used PL/SQL for years and half our code is built into the oracle engine! How could we ever port over to MariaDB?!?)

## All of these things (significantly) impact the tools available to you.

Some Options for in-house data management

---

*Note: The following is in no way exhaustive, and other SE users should chime in with additional suggestions.*

With the generic considerations out of the way, let me provide you with some techniques and technologies for addressing some of the above. First, ask yourself if you really *need* to use an RDBMS or if unstructured data with something like Hadoop, CouchDB, KV pairs, or even Object Oriented Storage (something like swift) is an option.

Second, consider looking into a cloud based solution. This outsources some of this headache and leaves the complicated problems to highly qualified (and paid) individuals. At scale however, you can find this really eats into your budget (cloud providers DO make a profit at this, and at a certain scale, you can just afford to employ these experts yourself) or if you are working under specific security or political requirements (read: we can't do clouds). If you need to run in-house, consider a hybrid NFS/Fibre Channel Filer. Most of these filers, such as those by NetApp, Pure Storage and Tegile support a delta-based snapshotting and cloning technique that can be very handy for A) taking backups, B) restoring backups and C) seeding new backups.

At this point, I need to note that I am not a backup and storage expert, so there some portions of this problem that I never quite was able to solve before I moved on to other problems (and greener pastures).

*But* that being said, some of these products allow you to take differential snapshots underneath your database. You will need to script out a full "lock tables with read lock" on one of your database instances (a read-only slave is recommended) and dump your binlog position or GTID for InnoDB based engines. But, when using these filers, once you script that, you will be able to use these snapshots to create new instances of your database. You will want to put binlogs on a separate partition and put only your database data on the other partition. With this, you will be able to clone these partitions (on NetApps, this is know as a "FlexClone") The only issue with this technique is that FlexClones cannot span datastores which means that all your I/O is still impacting the same datastore/volume and calculating the deltas on these volumes can cause some overhead in terms of CPU resources.

This is because for each block read, the filer must determine if the data resides in the frozen original snapshot, or in the delta. For volumes/stores with multiple snapshots, this might need to be checked multiple times. You can overcome this by refreshing the data (meaning, discard your snapshot and clone it again periodically - which might happen naturally and organically for a good continuous deployment environment) or by permanently splitting the volume (known as a "Flex Split" in NetApp terminology) which will take a moment to permanently resolve the deltas and create an entirely new and separate volume.

These delta clones have the added benefit of reducing your overall storage requirement - you can spawn several clones or instance of your production database to do your development, testing and validation. If you are only keeping one copy of your large dataset plus the (what are likely to be) small deltas, you reduce your overall storage cost and footprint.

The only trick here is that this may not constitute a full backup solution as the "backup" still resides on your filer. For this you may need to use something NetApp calls a Snap Mirror which will mirror data (using rsync-style technology) between filers and even datacenters, or use some type of integrated backup solution which can backup to tape one of your delta snapshots or a flex-clone.

This however has one major flaw: All of your data - dev, test and prod is *still* using I/O on the same filer and storage head. To work around this, consider creating a slave database instance on a second filer which can be the seeding point for you Test and/or dev filer, or consider using a load balancer/applcation delivery controller for your application layer to mirror production requests into your testing (and/or dev) environment(s). This has the added benefit of throwing prodcution traffic at your QA/Test environment before promoting deployments and code to production and allows you to test for issues that might not be immediately noticed otherwise. You can then check your logs for errors based on production traffic and user behavior in test.

## This should allow you to use a few script to programatically spawn and destroy entire (and large) datasets for use with continuous deployment methodologies.

Scalability and High Availability

While you asked about continuous deployment, DevOps is conserned with more than *just* continuous deployment - so I am going to include some bits about redundancy, scalability and high availability.

I mentioned, JIT, immediate and eventual consistency. This is where varous RDBMS engines come in. Eventual consistency is relatively easy by simply configuring circular asynchronous replication. This can cause some collisions however \*(what if your application layer updates data on one side of the cluster and on the other side of the cluster before replication is completed?) For immediate consistency, look at Galera cluster which will force synchronous replication, but causes scalability issues (how will you replicate to your Disaster Recovery site or load balance geographically without incurring significant latency due to propigation delay at the network layer?) You can also see if you can do synchronous replication within the datacenter and asynchronous replication between sites, but this seems the worst of both worlds.

Typically however, most people do not need fully synchronous replication - this is *usually* only needed for very specific (and exotic) high-write environments where multi-master is needed with table sharding. Most apps can deal with Just-In-Time consistency using a database proxy. For example, ScaleArc will monitor replication status and track where writes just went (to send subsequent read requests to that location until replication catches up) in order to provide Just-In-Time consistency and the *appearance* of database consistency. ScaleArc is compatible with Postgres, MySQL, MariaDB, Oracle and MSSQL and can use regular expressions to shard/partition your databases for applications that can't use shard keys. It also has a robust REST API for your configuration management software to interact with - and their support team is outstanding.

Similarly, you might wish to consider a free alternative, MaxScale developed by the MariaDB team for MariaDB. It lacks the GUI and some of the caching features of ScaleArc however.

Finally, MySQL fabric (and the in-RAM only MySQL Cluster - if you can afford that much RAM) are other potentials - especially with MySQL's new proxy. This can provide the scalability and redundancy component to your environment.

Postgres and Oracle (Eg, GoldenGate and RAC Cluster) should have the replication and sharding features you need, but ScaleArc will pair well if you need a proxy.

Ultimately, all these pieces and technologies add up to a highly flexible environment suitable for continuous deployment and development if you are unable to simply use a cloud based environment and let your cloud provider deal with the above problems for you.
> 15 votes

# Answer
We use Flyway at work for managing Postgres schemas in the app, and Pillar for managing Cassandra schemas. We have found it best if the app manages its own schema.

We had a horrible experience having manage schemas before the apps managed their own schemas.
> 7 votes

# Answer
I'd argue a tool alone won't really help unless you shift the schema responsibility to the application team.

We do use liquibase or flyway at work, where the application team is responsible to create the changesets.

Along with this,you can avoid a purely additive way.
Each application is required to be compatible with its precedent version, when an alteration of schema has to be done, then the application will integrate a new column in the schema, write to it and still read and write from/to previous column (allowing the previous version to still have the same data).
The next version of the application is allowed to stop updating the old column and only the third version will be allowed to remove the now unused column.

Obviously, regular backups are needed in case something goes wrong.
A proper subset of data likely to create problems in integration tests to avoid them in production is also a good idea. The ideal case get an anonymous subset of production data.
> 7 votes

# Answer
We use liquibase at our work and I'll speak highly for it. It's also used by our QA tool QASymphony.

We're utilizing it against MSSQL and Oracle databases internally and QASymphony uses/has used it with both postgres + mysql instances.
> 4 votes

# Answer
To answer this question in the context of a mainframe environment, and specific to DB2® databases, there are typically 2 commonly used (not cheap ...) alternatives to pick from:

* Object Administration for DB2®, from BMC. Here are some details about it (quote from the linked page):

> Making changes to objects in your database—or even just performing routine administrative tasks—can be difficult, risky work. There are dozens of tasks to keep track of, and a single misstep could have a disastrous impact on availability and data integrity. You can cut back on both effort and risk with BMC Object Administration for DB2 11, a collection of tools to help you:
>
> + Reduce the time required to administer complex and disparate DB2 environments.
>     + Automate routine tasks throughout the application lifecycle for improved integrity.
>     + Improve productivity with simplified DB2 catalog navigation and change management.
>     + Enhance application availability by performing changes and maintenance with minimal outages.
* DB2 Administration Tool for z/OS, from IBM.

> It simplifies the complex tasks associated with safely managing DB2 objects and schema throughout the application lifecycle with the least possible impact to availability.
>
> + Allows users to navigate the DB2 catalog quickly and easily
>     + Builds and executes dynamic SQL statements without knowing the exact SQL syntax
>     + Manages and tracks changes made to DB2 object definitions resolving any potential conflicts prior to execution
>     + Helps build DB2 commands to execute against databases and tables
>     + Enables users to create, alter, migrate, drop and reverse engineer DB2 objects
>     + Builds and executes utility jobs, allowing users to take advantage of LISTDEFs and TEMPLATEs for increased productivity

### Integration with SCM tools

Some time ago I was challenged by a customer to actually "integrate" the BMC alternative in their SCM lifecycle (managed by SERENA ChangeMan ZMF). The idea behind such integration being "*Consider our DB2 DBA-team (doing stuff with DDLs) as a special case of a development team, accidentally using some exotic editor (the BMC tool) to produce the code (DDL) to be migrated*".

About the only (but *real*!) challenge about this integration was to also facilitate "restartability", which is one of the key benefits of such DBA tool:

* *Restartability* means that when this DBA tool is doing its thing (via sometimes long running jobs, as per the nature of the work to be completed), unexpected things may happen (deadlocks, time abends, etc).
* If any of those things happen, and you don't want to restart from your backup (before you started), then the DBA tool expects you to restart from the (check-)point from which things started to go wrong (and from where you want everything to be re-executed).
* Better still (or should I say "even worse"?), if a newbee to the DBA tool doesn't real know how to restart from such checkpoint, and simply tries again (from the beginning), then the DBA tool will simply fail with some kind of user error. And this with an indication with something like "*Until you tell me how you want me to proceed after my last point of failure, I refuse to do anything (to not make things even worse)*".
* In the end, the real clue to implement this restartability in the SCM tool being used, required the SCM tool to be tuned also. Actually to allow it to be used for *restarting failed SCM procedures* (typically related to deliveries to target test/prod environments) ... instead of just *submitting the failed SCM procedure all over again* (which is how typically those SCM tools recover from such situations).

Bonus: after the SCM integration of the BMC alternative got completed, the customer decided to change their DBA tool to the IBM alternative. And even though both alternatives don't really look the same, the impact of it to the SCM integration was rather minimal: simply a matter of replacing (in some reusable SCM customization) some calls to the BMC alternative by the equivalent calls to the IBM alternative ... which (using DevOps terminology) got implemented by / .
> 4 votes

---
Tags: continuous-deployment, continuous-delivery, databases, flyway
---


# Question
Title: How does using packer and terraform in a CD pipeline compare to docker images built from git?
At work we have a Go CD pipeline for producing docker images, and scheduling docker containers with rancher.

This works quite well. You can commit a change to the Docker image in Git, the pipeline will pick it up, and upgrade the container. (The CD pipeline tags the version of the docker image with the build number of the pipeline, and this provides continuity throughout the upgrade.)

We have a similar arrangement with AWS AMIs using Hashicorp Packer and terraform. You can make a change to a packer json file in git, the pipeline will rebuild a new AMI. Given the user's approval in Go (we have some stage-gates in Go) this can then in theory to stop the existing ec2 instance and start a new one based on the AMI that was built. (The ID of the AMI is passed through the pipeline.)

In practice terraform doesn't work quite as well in a pipeline. (Compared to docker). Sometimes it is just the sheer complexity of things associated with an EC2 instance and corresponding VPC/routing arrangements can tangle it. Sometimes it is that you need to custom-write your own tests to see if the EC2 instance was stopped, and then started.

Is there a broader consensus on this issue? Are there some patterns I'm missing? Or is it just that when you're on the bleeding edge, things hurt a bit?

My question is: **How does using packer and terraform in a CD pipeline compare to docker images built from git?**

# Answer
> 2 votes
You're comparing apples with oranges according to your description. A strict comparison could be packer toward dockerfile, and/or terraform toward kubernetes yaml for example.

Main difference between your docker pipeline and your packer/terraform one is the architecture behind them.
What is putting the burden here is that your AWS infrastructure doesn't take advantage of auto-scaling groups rolling updates, or your 'stage-gates' in go does disrupt them.

You're not comparing docker with terraform, you're comparing rancher with your AWS infrastructure choices. If you don't use the equivalent bricks of rancher load balancing and deployment groups (kubernetes wording here) which are Elastic Load Balancers and Auto Scaling Groups in AWS you can't compare them.
That's saying a car and an bicycle does the same thing (transportation) with a better result for the car, put a motor on the bicycle and you'll have a far different appreciation and something less biased.

Your observation is pointing out that your usage of AWS facilities is sub optimal, terraform (or whatever provisionning system) can't help you more than running things, if your architecture is not made (in the terraform file) to allow rolling updates from this facility, no automatic system will fix it for you as you'll have to code it again (test instances status, etc.).

Of course rancher pre-bake most of those, making them invisible to you. This also mean you are enforced in rancher methodology (well, swarm, kubernetes or messos architecture to be exact).
That's the main difference, on one side you have an infrastructure already ready to serve with rancher and you can't run outside it, and on the other side you have to think your architecture before hand. If you were to run docker containers on a bare docker host (without any orchestrator on top) you'll end up with the same architecture design to think before hand and the same caveats you're seeing on AWS.

AWS is not strict on how you use it, it's an infrastructure as a service with no opinion on how you use it, whereas rancher enforce a bunch of services around your own container to save you the design time as long as you accept the constraint within it.

Hope this help seeing the problem from an upper level and as such I think the answer to your questions is that you were missing a pattern ;)

For the records: we're doing deployments on AWS with cloudformation (native version of terraform, with its own caveats) from a push on a gitlab server and it works nicely, the work to get proper cloudformation templates and the script to run it was non negligible. This to say the tools are not the problem, it's all a design choice at a point, same results can be achieved, arncher comes with its own caveats and design choices to make when you integrate it into an existing infrastructure.

---
Tags: docker, amazon-ec2, terraform, packer, gocd
---


# Question
Title: Why shouldn't I try to hire a 'DevOps Engineer'?
The idea of having a DevOps Engineer has become quite popular recently, and it seems appealing to just have a person who can slot in and provide many of the benefits of DevOps, as described in the Puppet blog:

> Organizations using DevOps practices are overwhelmingly high-functioning: They deploy code up to 30 times more frequently than their competitors, and 50 percent fewer of their deployments fail, according to our 2015 State of DevOps report.

However, I've noticed a lot of vocal opposition to the idea of a DevOps Engineer to try and make these improvements:

> Even with broad agreement about core DevOps attributes, controversy surrounds the term “DevOps engineer.” Some say the term itself contradicts DevOps values. Jez Humble, the co-author of Continuous Delivery, points out that just calling someone a DevOps engineer can create a third silo in addition to dev and ops — "...clearly a poor (and ironic) way to try and solve these problems."

Why might it not be such a great idea for a business to hire a DevOps Engineer to try and 'implement DevOps', as opposed to the organisational change advocated by blogs like this? Will the benefits be negated by just having an isolated DevOps role?

# Answer
**TL;DR**: You should **never** try to hire a **DevOps Team**

---

There are essentially three most common roles to hire for:

1. DevOps Architect / Evangelist
2. DevOps Engineer
3. CI/CD Engineer

These roles are distinct from your 6 essential software development roles that traditionally compose the software engineering organization:

1. Product Management
2. Software Development
3. Tools Development
4. Security and Compliance
5. Quality and Testing
6. System Operations (SRE)

Lets go through the three roles one by one and see how they fit

---

## DevOps Architect or Evangelist

* **Why**: If you are lost, slow, broken and don't know what to do.
* **When**: At the start of the process in planing stages.
* **What**: Management level role to guide all managers and leads in the entire Software Engineering org. This person will plan the entire transformation of your engineering organization to a highly functioning state.
* **Who**: Consulting member well versed in theory, management practices, culture topics and operations who reports directly to VP of Software Engineering.

In some cases and for smaller and mid-sized companies you might start the process instead with hiring a consulting organization, like DORA.

## DevOps Engineer

* **Why**:
    1. To bridge the gaps between teams if they are organized along the functional roles mentioned above to ensure cross functional level cooperation.
    2. To embed with product oriented teams, which have each of the 6 traditional roles included in the team, to help bridge the knowledge gaps and to help with implementation and adoption of the novel practices and tools.
* **When**: Once you've laid out your plans and the organizational transformation starts and the entire management team is on board.
* **What**: Enable cross function cooperation, ensure that team boundaries are broken down, that local optimizations inside teams are not creating a barrier to high throughput of work throughout the value chains all the way from customer wishes to customer deliveries.
* **Who**: Experienced engineer with skills both in software development and system operations. He should be skilled in the best practices, process and culture changes related to DevOps transformation.

## CI/CD Engineer

* **Why**: To help implement CI/CD pipelines, integrate your tool chain, bring in the tools that will enable better working of the company.
* **When**: During the transition in larger organization, while the above roles have been already filled.
* **What**: Engineer, which is essentially part of the tools team that will be able to setup CI/CD pipelines and start integrating internal systems in a way that will remove friction from the throughput of work.
* **Who**: Engineer experienced with Tools, Integration process, Release Management and DevOps practices. Someone who understands they are replacing human gating in release process by Automation.
> 26 votes

# Answer
I'd argue Devops Engineer as described in your question link is mainly a sysadmin role. Quoting the skills here for background to this answer:

> Your climbing gear.
>
> * Strong background in Linux/Unix Administration experience with automation/configuration management using either Puppet, Chef or an equivalent
> * Ability to use a wide variety of open source technologies and cloud services (experience with AWS is required)
> * Strong experience with SQL and MySQL (NoSQL experience is a plus, too, since we also use Redis)
> * A working understanding of code and script (PHP, Python, Perl and/or Ruby)
> * Knowledge of best practices and IT operations in an always-up, always-available service

In this sample job description DevOps Engineer is just a buzz word for a sysadmin comfortable with cloud based infrastructure, automation, able to read code to help in diagnostic, and aware of high availability practices and solutions.

This is loosely related to DevOps practices and silo breaking culture between dev and ops as seen in this question What is the difference between Sysadmin and DevOps Engineer?

It won't be a good idea because a sysadmin, how comfortable he/she can be with devops practice and culture, is not the right person to drive a company transformation. You won't be hiring this person with a culture change in mind but with a tool configuration view, which won't really help breaking the processes. This may also be badly received by his/her colleagues and you'll just bring resistance to the change if there's no culture change planned beforehand

For a successful pattern toward the gains of devops, @Jiri Klouda's answer gives a great overview on an acceptable DevOps Engineer role along with the step in the change it would bring value and help toward success.
> 10 votes

# Answer
I realise this answer might not be a perfect fit for you, but here's what I did

I was the first developer working at a very busy ecommerce startup, with incredibly high traffic. I realise the company was still young, and that, for a while, I would be the only technical in-house resource.

Knowing this, I decided to structure my infrastructure in such a way, that I would have to do ZERO systems administration.

I decided to host in the cloud, because doing so relieved me of systems maintenance. I looked for a AWS engineer, with puppet experience. Together, we architected an autoscalable, infrastructure, written as code in cloudformation. All configuration files were versioned within puppet.

This allowed me, as a developer, to assume this devops role. I built code release tools, in python, fabric. I used the same script to bootstrap my application onto new servers that are autoscaled.

This worked very well, and today, 3 years later, I still don't do any system maintenance. We have a systems admin (the same AWS engineer) for 10 hrs a month, and I try to structure his sprints in such a way that I don't become an annoyance for him. In that way I respect his time, and manage his sprints in the best way I can.

If a system has a degraded performance I simply terminate it, and another one spins up in his place.

I hope this answer could somehow benefit you
> 6 votes

# Answer
You shouldn't hire a DevOps engineer because DevOps encompasses such a wide variety of disciplines that one individual cannot possibly be an expert in all aspects of these disciplines. By hiring a jack of all trades, you would be hiring a master of none.

DevOps is necessarily a *team* based endeavor and you cannot possibly expect one individual to support the expectations of an entire team. Consider the scope of DevOps. One person cannot possibly:

* Be a rockstar developer in \[language\]
* Be a networking guru, knowing all requisite RFCs
* Be an exert Systems Administrator
* Be an expert QA tester
* Be a Database Administrator
* Specialize in storage and backup
* Know Site Reliability Engineering
* Potentially other disciplines as well

Some of the above even have *sub* disciplines, such as Windows Systems Admin vs. Linux/Unix System admin or maybe you use more than one coding language in your.

No one person can possibly be an expert in all this which means that if you are adverting for a DevOps engineer, when the weakest area on your DevOps team is Networking, you are not doing a very good job of advertising your need for a networking *specialist* for your DevOps *team*. While no individual should be pigeon-holed to a specific role in the DevOps team, you would do your team a disservice by pretending there are no specialists or subject matter experts (SMEs) within the scope of DevOps. Swinging the pendulum from one extreme to the other - from silo-ing to pretending like every role on your DevOps team is the same - can cause just as many problems.

While having team members cross-train in more than one discipline - particularly in the overlapping areas is good, expecting them to be able to be proficient in such a wide volume of knowledge simply isn't practice.

This means that anyone who tells you they know all aspects of DevOps is probably lying to you. Hire a specialist in the area you are weakest in who has worked on a DevOps team - not a "DevOps Engineer."
> 4 votes

# Answer
I had this exact challenge when implementing at ASOS. The goal for us is to have teams who are self sufficient and so having a dedicated role can be an anti-pattern, however we live in the real world and for many developers thinking about good DevOps practices is not their primary thing so they need help getting there.

What we did was:

* lose the term of DevOps engineers, DevOps is something we should all do, not our job title so we called them something different.
* rolled them out to teams but only 1 per 3, this meant they could not be a doer but had to be seen as a competency to help teams get themselves better and solve their own problems (with guidance)
* maintained a central function as well to act as competency centre and handle the enterprise considerations, anything that affects all teams

As we evolve we review this model but for us it is working effectively so far
> 3 votes

# Answer
I don't think you will be able to get a definitive answer for this because it seems like a lot of things factor in.

* How on board the company is with the DevOps practice
* What sort of application or service the company provides
* The structure of your company

I just got done interviewing for positions and ended up with a title of DevOps engineer, but some of what I am doing is Sys Admin work. That is just out of necessity because of the size of the company and the nature of what is being managed application wise. Some positions I interviewed for had a similar title and they were looking more for someone from the development side of things experience wise. They expected more writing of code and not a sys admin who does automation. SRE seems to be a title gaining popularity so that might be the way to go. I had myself as System Administrator and Automation Engineer as my last job because I was writing ansible and chef stacks part of the time. The company was following a pretty good devops model where everyone was on board and devs worked alongside ops but I think their future didn't have dedicated infrastructure people in mind.

Now that I am in this position I am trying to shoe horn in some automation and we have some people problems we will have to sort out. People come and go and some of the workflows were designed because someone else designed it that way and not because it fits how people work.

So basically I think you should figure out the job description and not worry so much about the title unless its tied to pay somehow or you think one or the other would attract the right kind of people.
> 3 votes

# Answer
If you are concerned with the meaning of DevOps, and following a "one true path". You shouldn't hire a DevOps Engineer. You should hire an Automation Engineer, or A Deployment engineer, or Platform Architect, or a number of other roles that do what you need.

If being a true DevOps practitioner isn't important to you, then you can call it whatever you want.
> 1 votes

---
Tags: terminology, culture, job-role
---


# Question
Title: Are configuration management tools appropriate to use as deployment tools?
Off the back of my answer to the question: How can DevOps help to improve Software Escrow procedures? Tensibai had the question:

> What would necessitate Capistrano on top of puppet or chef?

My response was to post a link to Noah Gibbs' article "Do We Need Both Capistrano and Chef?". Personally, I still subscribe to Noah's view that it is most appropriate to:

* use a specialist deployment tool such as Capistrano for deployments.
* use a specialist configuration management tool such as Chef for configuration management.

The fundamental approach that each type of tool uses to complete its task is very different:

* **Configuration Management Tools** \- are about creating and maintaining the desired state of a system, they are inherently idempotent in nature. Examples of configuration management tools are Chef, Puppet, Ansible, PowerShell DSC, Salt Stack.
* **Deployment Tools** \- are about delivering *versions* of software into a hosting environment, they provide functionality to maintain multiple versions of the software on multiple machines and manage which version is "current", they are inherently imperative in nature. Examples of deployment tools are Capistrano, Octopus Deploy, Deployer and Command.io.

I do believe that Configuration Management Tools can do the job of deployment tools and in the case of Immutable Infrastructure they are the most appropriate tool for the job as software versions on the target don't need to be maintained.

**Question:** Have configuration management tools such as Chef, Ansible and Puppet matured to the point that they are capable of fulfilling both the idempotent and imperative models?

# Answer
In such context the typical advice should be immediately applicable: use the right tool for the job.

But then you also cannot ignore nowadays the almost virulent tendency of software *tools* to extend functionality into more or less related fields and actually become *toolsets* for various reasons: cool feature(s) to have, expand customer base, amass more revenue, etc.

For example many file management tools include image viewing features and many image processing tools include file management features. You can move files around and you can view images with either of the tools, often equally well.

Because of this it's quite possible to have entire portions of the software development process covered/overlapped by multiple tool(sets) even if their main feature/capability differs.

So it really boils down to the exact functionality *you* want to achieve in *your particular process* and how well one tool or the other does the job *in your context*. Subjectivity/preferences/convenience included.

Making this question primarily opinion-based ;)
> 10 votes

# Answer
Configuration management tools are used to get a system into a known state. Deployment tools deploy new program files and program data to a system. At the end of the day, both types of tools do some combination of:

* Determine the current state of the system.
* Transfer files to the system.
* Add or change persistent data (e.g. configuration files, database data, registry settings)
* Start or restart programs.

Configuration management tools have declarative languages which specify the state of the system. Deployment tools have imperative languages which tell the system to do things. A DevOps person needs to do both.

Using the deployment tool Capistrano, it is clumsy to use its language to tell the system to ensure that the web server is active. You have to issue a command to restart the web server, and another to check to see if the web server is up. It is a kludge to get the web server into a known state.

Using the configuration management tool Ansible, it is clumsy to restart a web server. The language lets you tell the web server to be "up", but if you specifically want it to be restarted, you have to set its state to "restarted". But there is no easy way to check if the web server has been restarted. This is a kludge in Ansible to enable one-off operations.

Some folks prefer doing both types of jobs with one tool, and working around the rough edges. Other folks prefer having two tools to do almost the same thing, but without rough edges. To answer the question, "appropriateness" is a matter of taste. This answer explains why.
> 6 votes

# Answer
**TL;DR**: Just use Ansbile, it is **both** configuration and deployment tool :)

There are several types of deployment:

* **Application** based (files, archives packages)
* **Container** based (includes VMs, Habitat, LXC, Docker)
* **Function** based (Micro services / Lambdas / Functions)

I assume in this case we speak only about application updates on server(s).

---

For deployment you need to have two things happen:

1. Correct files or packages need to move to the server.
2. Configuration and service states need to change.

---

Now for (1) you can use multiple strategies:

* Artifact Repositories / Syncing
* Package Repositories / Package Managers
* Version Control System / Updating + Compiling (optionally)
* File transfer protocols (scp, rsync, ftp)
* Deployment Tools

For the (2) you can use:

* Configuration management tools
* Deployment Tools

So while the Deployment Tools are a way to do deployment all in one, they are not always the best strategy. Sometimes you want to use combination of these ways for deployment. You most likely already use package managers at least on your servers already. You most likely run configuration tools anyway. The problem with some of the configuration tools was a proper orchestration among multiple servers, but now even Chef and Puppet can do that quite well. Ansible has been always good at this.

From *personal experience*, I've used all combinations, but currently we use Capistrano for deployment and Ansible sync for configuration management, and VCS and package repositories for file transfers, but there are issues with Capistrano and we are planing to move away from it to unify on Ansible for both deployment, maintenance and configuration management.
> 4 votes

# Answer
Application deployment is a hard thing to pin down because it has a lot of sub-problems. Config management systems are excellent at modeling tasks which are convergent and work with "what is the desired state of the system". In the context of app deployment, this is great for things like deploying bits to a machine, managing configuration files, and setting up system services. What it is extremely bad for is things that are inherently procedural, notably database migrations and service restarts. I usually try to put the convergent logic in Chef and let an external procedural tool (usually Fabric in my case) handle the few remaining bits as well as sequencing the actual converges.

So, basically, you should use both for the pieces they are best at.
> 3 votes

# Answer
For software and deploying code to an existing server or inside a Docker container, the answer is relatively simple - No, you don't *need* both, but you might want both if another tool or utility adds value and is the right tool for the job, however things get more complicated when you are deploying servers and operating systems.

One value-add of a DevOps mentality is treating infrastructure as code and frequently deploying or destroying virtual machines or even bare metal in a highly elasticized environment. Your configuration management system cannot easily netboot and kickstart your server for you and cannot manage repositories, packages and updates/patching for you during and after deployments or in some cases, licensing and entitlements.

For Amazon Web Services, this is rather conveniently manageable by APIs for the most part, but for those of us who have to manage our own data centers, this is not an option. For this reason The Foreman project (and Red Hat which re-brands this) have found it necessary to bundle Katello, Candlepin and a configuration manager such as Ansible, Foreman or Puppet together when deploying the Katello Scenario.

So while you might be able to get away with using configuration management tools for software code deployments on the Dev side of the house, over on the Ops side, there are several cases where the answer is a resounding "no, configuration management tools are *not* appropriate to use as deployment tools" Doing so would require a serious re-invention of the wheel and is impractical. You must instead use your configuration management tools to initiate deployments in another tool.
> 3 votes

---
Tags: ansible, deployment, chef, configuration-management, capistrano
---


# Question
Title: How to insert a minion Powershell variable into Salt Pillar or Mine?
Say I have a Powershell script, named `test.ps1`, that has this:

```
$VariableForPillar = 5

```

I can call it from a Salt state with -

```
test powershell run:
 cmd.run:
   - name: 'C:\Windows\System32\WindowsPowershell\v1.0\powershell.exe  -ExecutionPolicy ByPass -File  d:/tmp/Powershell/test.ps1'

```

But what can I do to retrieve that value and insert it into a Pillar (or a Mine)?

# Answer
> 4 votes
You can only insert the return into the mine.

Just specify the cmd.run (or possibly cmd.script) in your mine\_functions and you will get the whole return back.

If you want to have a script that sets up extra config on the minion, I would recommend writing a custom grain.

https://docs.saltstack.com/en/latest/topics/grains/#writing-grains

---
Tags: configuration-management, windows, salt
---


# Question
Title: How to examine Jinja output in Saltstack?
I have a templated `SLS` in Salt I'm trying to build, but it's emitting invalid syntax, which is resulting in errors such as:

```
my-minion-id:
    - State 'system' in SLS 'network' is not formed as a list

```

In principle, it should be possible to, *somehow* examine the output of the Jinja template before it attempts to parse the output as an `SLS` file. There exists a Python module for the Jinja renderer`salt.renderers.jinja`, but if I attempt to execute it on the CLI, I get an error:

```
# salt my-minion-id salt.renderers.jinja.render /srv/salt/network/init.sls
my-minion-id:
    'salt.renderers.jinja.render' is not available.
ERROR: Minions returned with non-zero exit code
# salt my-minion-id renderers.jinja.render /srv/salt/network/init.sls
my-minion-id:
    'renderers.jinja.render' is not available.
ERROR: Minions returned with non-zero exit code

```

How can I see the output of my template? It seems absurd it should be this difficult to debug.

# Answer
Check out the slsutil.renderer module.

This should do what you want

```
salt my-minion-id slsutil.renderer /srv/salt/network/init.sls 'jinja'

```

This module just calls the compile\_template function directly for you.

Edit: /srv/salt/network/init.sls is the path on the minion, if you are not targeting the master as your minion, you will probably need to do the following.

```
salt minion-id cp.cache_file salt://network/init.sls
salt minion-id slsutil.renderer /var/cache/salt/minion/files/base/network/init.sls

```

or point to whatever file that cache\_file spits out.

If you are on 2018.3 or newer, you can just specify `salt://network/init.sls`
> 14 votes

# Answer
Given how much time I spent weeks ago struggling with a closely-related issue, I wish I'd figured this out sooner.

The solution appears to be to use `salt.modules.cp.get_template` to have the Salt minion retrieve the file, render it through the templating engine and place it in a readable place:

```
# salt my-minion-id cp.get_template salt://network/init.sls /root/network.sls template=jinja
my-minion-id:
    /root/network.sls

```

From there, you connect to the `my-minion-id` host and examine the file you placed at `/root/network.sls`.

This makes sense; `salt.renderers.jinja` is in the `salt.renderers` namespace, while the modules you have access to from the CLI are in the `salt.modules` namespace.

It also makes sense from a data visibility standpoint; template rendering happens on the *minion*, where grains and such are available, and I've yet to see a module that executes minion code return arbitrary output to the master (for view on the CLI, for example); the returned data is invariably well-structured and concise. (There may be such a module, but I don't know what it is. It would be a preferable solution to dropping test files onto a minion.)

**edit:** @gtmanfred's answer is far better and more direct, and I've accepted that one. I'm leaving this one here for informative purposes. It's not the best solution, but it does still work.
> 14 votes

---
Tags: salt, jinja
---


# Question
Title: Jenkins Promote Build across different pipeline?
In my workplace, we are using a monolithic repo. It contains many project and micro services that run on different Jenkins pipeline. I know that using Promoted Build Plugin, we can promote build of tested artifact.

However as our projects depend on projects that run on a separate pipeline, I am wondering if the Promoted Build Plugin is able to promote artifact to a different pipeline. Or can the pipeline that is running head over to a separate "place" to take the last tested artifact and use it as part of its own pipeline criteria?

The reason we want to do this is because of integration test that depend on the existent of another build artifact. We want to be able to test on the latest build artifact that is on another pipeline. Would be even better if we can select base on the tag/version.

# Answer
> 2 votes
In my opinion you should treat each pipeline dependency as any other dependency.

Set up an artifact repo like nexus or artifactory to be used as source of artifacts.
This way you can ask for latest or tagged version of an artifact inside your pipeline and push to this repo your own artifacts when the build succeed.

---
Tags: jenkins, jenkins-plugins, jenkins-pipeline
---


title: Rerun of Bamboo jobs on commit

text:
# Question
Title: Rerun of Bamboo jobs on commit
Imagine the following setup: a build plan has a set of jobs (J), each builds a module (M). Latest successful build for all of them is also there. (BX) E.g. (imagine there are ten or so; modules are interchangeable i.e. will be never linked to each other)

```
J1->M1->BM1-X
J2->M2->BM2-Y
J3->M3->BM3-Z

```

Now imagine there are changes in just one of repos, say M1. We know that Bamboo runs jobs in parallel.

Questions:

* will Bamboo completely rerun builds also J2 and J3, even if it finds out there are no changes in repos M2 and M3?
* If yes (I think so by a test setup but not sure), what is the purpose of doing so? (at least in terms of Bamboo logic)

# Answer
> 3 votes
I think you have fundamentally misunderstood the relationship between plans, stages and jobs in bamboo.

A good rule of thumb is to have a single repository of code, which is linked to a single build plan, which produces a single independent module.

You may initially start out creating a single stage with a single job inside each single plan where you just run a single maven command and the unit tests built into your maven build may be perfectly sufficient to exercise your modules completely in this extremely simple use case you probably don't need a tool like bamboo or Jenkins to deal with your builds.

However, very quickly your conceptualisation of a "build" gets very complicated. As you mentioned in your question you might quickly amass multiple modules that are interrelated, or you might need to support your code on multiple platforms or in multiple browsers, and if you get all the way to the CD end of the work flow you might even need to deploy binaries into environments as the result of a code change.

This is where you would want to use the extra orchestration capabilities of your CI tool.

For example, if you are developing a module that you expect to load in a browser then your might have a build pipeline that looks like:

Check in Code -\> Build/Compile/Unit Test -\> Deploy to a CI environment -\> Stand up a half dozen browsers and verify that they all still work correctly.

In the above work flow you would have a single stage to handle your compilation and unit testing which would spit out a compiled or built artifact. Then you have a stage that does whatever needs doing to make your module useful, ie it might create your final shippable product by combining your module with other modules built from other plans, or it might package some software into an RPM, a DEB and so on, depends what you're actually building. If you need to ship to multiple platforms then this is your first stage where parallelism becomes meaningful.

Alternatively, if you are building for a single platform but for multiple consumers you might follow up this build of your component with a bunch of headless browsers kicking off test cases to validate that your changes didn't break anything. These can all happen in parallel on different agents running in different OS's, depends how you have bamboo configured.

Then finally, assuming you are shopping something more complex than a single module, you will want to do a bit of integration testing with all the other components that you need to interact with. This step will usually be separated into a build plan of its own, triggered by the successful completion of the plans which are spitting out individual modules. Here you get a chance to run all the same tests but with your latest version running along side everyone else's module so you can be sure that there are no breakages caused by dependencies.

So at any one time you only have a single repo triggering a single plan. If you have multiple repositories feeding into the same plan and they are completely unrelated but use the same build pipeline then I'd recommend taking advantage of bamboos plan branches so that you can reuse the same build process but with multiple different code bases (if that fits your use case)

---
Tags: bamboo
---


title: SaltStack and Vagrant adding files

text:
# Question
Title: SaltStack and Vagrant adding files
I use file.managed to add a file:

```
/home/vagrant/.bash_profile:
  file.append:
    - name: /home/vagrant/.bash_profile
    - source: salt://config/user/.bash_profile
    - user: vagrant
    - group: vagrant
    - template: jinja
    - mode: 0775
    - replace: true

```

The file is added; however the file is owned by root/root and does not change to by owned by vagrant.

Using Vagrant 1.8.5. Salt is configured to be masterless and file client is local.

# Answer
> 1 votes
You aren't actually using file.managed, you are using file.append - so salt is expecting to add onto the end of an existing file. This may be the cause of your issue. Instead try:

```
/home/vagrant/.bash_profile:
  file.managed:
    - name: /home/vagrant/.bash_profile
    - source: salt://config/user/.bash_profile
    - mode: 0775
    - user: vagrant
    - group: vagrant
    - template: jinja
    - replace: true

```

Also, be sure you are using the latest version of Salt Stack, 2016.11

---
Tags: vagrant, salt
---


# Question
Title: How can I share a (global) variable between salt states?
I am using Salt for deployment and configuration management. In order to attach to a FreeIPA server, I need to randomly generate an OTP token and then use it to attach to IPA. How can I share a variable between SLS files?

For example,

**init.sls**:

```
{%- from "freeipa/map.jinja" import client, ipa_host with context %}
include:
  - freeipa.example

freeipa_client_install:
  cmd.run:
  - name: >
    ipa-client-install
    --server server.example.com
    --domain example.com
    --realm EXAMPLE.COM
    --hostname client.example.com
    -w {{ otp }}                                   <===== !!!
    --unattended

```

**./freeipa/example.sls**:

```
{% set otp = salt['random.get_str'](20) %}         <===== !!!

get_ticket:
  cmd.run:
    - name: kinit service-account@EXAMPLE.COM -kt /tmp/salt-service.keytab
ipa_host_add:
  cmd.run:
    - name: >
      curl -k
      -H referer:https://{{ client.get("server", {}) }}/ipa
      --negotiate -u :
      -H "Content-Type:application/json"
      -H "Accept:applicaton/json"
      -c /tmp/cookiejar -b /tmp/cookiejar
      -X POST
      -d '{
        "id": 0,
        "method": "host_add",
        "params": [
          [
            "client.example.com"
          ],
          {
            "all": false,
            "force": false,
            "no_members": false,
            "no_reverse": false,
            "random": false,
            "raw": true,
            "userpassword": "{{ otp }}",          <==== !!!
            "version": "2.156"
          }
        ]
      }' https://server.example.com/ipa/json

```

What I find is that everything looks good in example.sls, but I cannot reference this variable in init.sls. Do I need to be doing `salt['random.get_str'](20)` inside of init.sls instead? If so, how can I reference the otp variable in example.sls - or vice-versa - how can I reference the otp variable set in example.sls inside of init.sls?

# Answer
> 2 votes
When I encountered a related problem, I wound up having Jinja run execute Salt modules.

In my case, it was `cmd.run` to run `mktmp` (and, at the end of the template, `rm` for cleanup), with `file.append` and `file.grep`, though in your case, you might be able to use Salt's `load_yaml` extension to Jinja.

So, something like this (rough, untested):

```
{% set tmpfile = salt['cmd.run']('mktmp') -%}
{% salt['file.append']('my_key: ' ~ your_key) -%}
{% set my_key = load_yaml(tmpfile)['my_key'] -%}
{% salt['file.remove'](tmpfile) %}

```

This certainly isn't the best place to put data, but it demonstrated using calling into Salt modules from Jinja to store data. I don't know of any existing Salt modules suitable for storing ephemeral data that only lives as long as the templates compile. I'll check back later and see what other solutions might work.

---
Tags: deployment, salt
---


# Question
Title: Disable a Jenkins pipeline on failure? Issue with a central deployment job
Consider a structure of X micro services and 2 environments; staging and production.

Each of these micro services is represented by a separate git repository, when a merge to master happens on any of these services, a jenkins job is triggered to test the code locally and if it's successful - trigger a deployment pipeline.

The deployment pipeline is a single job with the following steps:

`build and test locally -> deploy to staging -> run overall system tests on staging -> deploy to production`

Again, this downstream pipeline can be triggered by any of the X micro services.

The issue I have comes from the following scenario:

`service A code is updated and merged to master, the code is built and tested -> deployment pipeline is triggered -> code is again built and tested successfully -> service is deployed to staging -> tests against staging fail -> pipeline is aborted and deployment to production is avoided`

this scenario causes a difference of versions between staging and production. The larger problem happens when this failure goes unnoticed and isn't rolled back, then subsequent services that trigger the deployment pipeline will fail the tests on staging because of the previous deployment and the gap between staging and production will just keep growing until someone rolls back/fixes the failing tests and either triggers all the deployments again, or manually syncs between the environments.

This is of course not ideal, and the immediate solution seems to be "disabling" the deployment pipeline after a certain threshold has been crossed (weather it be a single failure, or a number of them), but this doesn't seem to be possible to achieve using Jenkins.

Is there actually a way to disable a jenkins job dynamically? Is there another obvious solution that comes to mind with this architecture?

# Answer
There is a `"Jenkins Disable Job"` plugin, which you can use in your pipeline to disable the job on failure.

You will be able to use this plugin for single failure or to disable after multiple failures.

https://wiki.jenkins-ci.org/display/JENKINS/Disable+Failed+Job+Plugin
> 2 votes

---
Tags: jenkins, deployment, jenkins-pipeline
---


# Question
Title: Who redirect ports on Windows to access http container?
When we use Linux, Docker can create a rule using iptables to do a port redirection to connect in the http of a container.

When we use Windows, how does this work?

# Answer
> 3 votes
it uses the Hyper-V Virtual Switch (https://docs.microsoft.com/en-us/virtualization/windowscontainers/manage-containers/container-networking)

---
Tags: docker, windows
---


# Question
Title: In what all cases would one use an external load balancer over the Ingress routing mesh in a Docker cluster?
Also, would the routing mesh act as an application load balancer?

Like for example, if I am having a Docker cluster in AWS, how does the ELB added to it, work? And how would that be different from what the internal load balancer is doing?

Or, did I get the concept of the routing mesh completely wrong?

# Answer
> 3 votes
You missed a point that there's still a need in balancing traffic before nodes in Docker cluster, and ELB is great for it (managed cluster, healthchecks, you don't pay for number of ELB instances per VPC, you can spawn spare nodes and register them in ELB automagically with Cloudwatch), great discussion on ServerFault. Basically, external balancer will split traffic between *all* nodes, while internal balancers will route it to nodes where service containers are running (which is where routing mesh is used).

---
Tags: docker, docker-swarm, load-balance
---


# Question
Title: Does the "cattle not pets" distinction apply as equally to machine instances as to containers?
There is a great discussion of the Cattle vs Pets distinction from Randy Bias here. Martin Fowler talks about a SnowFlakeServer.

In a series of talks, Adrian Cockcroft talks about how they moved toward Cattle Servers for solving a scalability problem at Netflix.

The challenge with this distinction is always managing persistent state. Does it make sense to treat your database servers as Cattle? It does if you (a) manage the state outside of your cattle model (external volumes for your docker containers), or (b) use a distributed database like Cassandra that allows for individual nodes to fail, but still maintain state in the cluster.

I get that you can get very close to the 'disposability with persistent state' of Docker containers mounting a shared volume, with launching AMIs to machine instances mounting a shared drive. You can get closer this this idea of scheduled cluster management by having an autoscaling group that recreates machines that you've blown away.

To me - the machine instances lack the granularity of a docker container. They gravitate more towards the 'pets' end of the spectrum.

My question is: **Does the "cattle not pets" distinction apply as equally to machine instances as to containers?**

# Answer
While technologically, containers and virtual machines are very different, there is no apparent difference from the perspective of your software. It seems like the argument in your question is that data is special and will always be a unique snowflake, so your question basically boils down to what to do about it in terms of DevOps, CI and Automation.

This is the perpetual challenge of the DevOps model. Ultimately, what you are asking is, either A) your data will be in a database or B) a datastore , so how do you manage that?

The answer is that yes, your database servers can and should also be treated as cattle - and I recently detailed several techniques and technologies for doing so in house. If you don't treat your data storage (databases, storage filers) as cattle you will find you have a single points of failure within your data(base) infrastructure and a lack of scalability and redundancy. These techniques I discuss in my linked answer basically allow you to distribute and cluster your relational database in a manner similar to Cassandra.

Managing your data(bases) and making data(bases) redundant is probably the most difficult challenge facing DevOps. The easiest way to solve the problem: outsource the headache to a cloud provider.

So in short, yes the "cattle not pets" distinction apply as equally to machine instances as to containers.
> 9 votes

---
Tags: docker, containers, autoscaling, amazon-ami, cattle-vs-pet
---


# Question
Title: When would I choose IPTables over an Ambassador pattern for port forwarding?
In an environment of Secured Virtual Private Clouds, in a Cluster of Docker Containers, we have to setup some routes.

We can do that with the Ambassador pattern, which is simple and easy to maintain. (Docker does have some usage of IPTables under the hood - but from what I can see the Ambassador pattern uses socat, and not IPTables to achieve its forwarding.

We can setup NAT rules with IPTables to achieve a similar goal.

My question is: **When would I choose IPTables over an Ambassador pattern for port forwarding?**

# Answer
> 1 votes
`iptables` will probably scale better than `socat` because `socat` runs completely in user space. There's a common impression that `iptables` is hard to deal with, but I find it easier that most routers to fiddle with. There are a number of `iptables` wrappers available to make it easier to handle.

If you're willing to accept the performance hit of running in user space then consider `haproxy` over `socat` because it is intended to run over longer periods and provides statistics, failover, SSL termination, and lots of other conveniences.

---
Tags: docker, ports
---


# Question
Title: Changing the build intermediate paths for gitlab-runner
I have a server pointed as ab12.xy24.com in which the projects are placed as:

```
 Path-1
 /home/ab12/public_html/project1/*
 /home/ab12/public_html/project2/*
 ................................
 /home/ab12/public_html/projectn/{project-files}

```

I am using gitlab-runner for CI and CD. The build is being built as:

```
 Path-2
 /home/ab12/build/{runner-token}/{commit-id}/{group-name}/project1/{project-files}
 /home/ab12/build/{runner-token}/{commit-id}/{group-name}/project1/{project-files}

```

I want the build path to be as in Path-1 style instead of path-2 style.

---

What have I tried?

I tried changing the `build_path="path1"` variable which only changed:

```
/home/ab12/build/{runner-token}/{commit-id}/{group name}/project1/{project-files}

```

to

```
/home/ab12/path1/{runner-token}/{commit-id}/{group-name}/project1/{project-files}

```

I cannot also use docker as due to its virtualization concept and takes much space, which is not desired.

# Answer
Conceptually, this approach is not the way to go; the build directory is not a deployment directory, it's a temporary directory, to build or to deploy **from**, whereas on a shell executor this could be fixed.

So what you need is to deploy from that directory with a script as per `gitlab-ci.yml` below, to the correct directory of deployment.

```
stages:
- deploy

variables:
  TARGET_DIR: /home/ab12/public_html/$CI_PROJECT_NAME

deploy:
  stage: deploy
  script:
     mkdir -pv $TARGET_DIR
     rsync -r --delete ./ $TARGET_DIR
  tags:
    - myrunner

```

This will move your `projectfiles` in /home/ab12/public\_html/

naming your projects as `project1` .. `projectn`, all your projects could use this same `.gitlab-ci.yml` file.
> 10 votes

---
Tags: continuous-integration, continuous-deployment, gitlab
---


# Question
Title: Force script to fail if "npm install" has network issues
With the current Github outage, I noticed a problem with my packer setup. In a nutshell, I have a packer setup where I call a shell script that looks like this:

```
#!/bin/bash
set -e
...
npm install
...

```

I expected that if there is any error during `npm install`, the script should fail and packer should abort. What I found out is that the script continued, even though the output looks like this:

```
amazon-ebs: + npm install
amazon-ebs: npm ERR! fetch failed http://github.com/component/emitter/archive/1.0.1.tar.gz
amazon-ebs: npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 192.30.253.112:80
amazon-ebs: npm ERR! fetch failed https://github.com/LearnBoost/node-XMLHttpRequest/archive/0f36d0b5ebc03d85f860d42a64ae9791e1daa433.tar.gz
amazon-ebs: npm WARN retry will retry, error on last attempt: Error: fetch failed with status code 502
amazon-ebs: npm ERR! fetch failed https://github.com/component/global/archive/v2.0.1.tar.gz
amazon-ebs: npm WARN retry will retry, error on last attempt: Error: connect ETIMEDOUT 192.30.253.113:443
amazon-ebs: npm WARN deprecated node-uuid@1.4.8: Use uuid module instead
amazon-ebs:
amazon-ebs: > ws@0.4.31 install /opt/socks-proxy/node_modules/tomahawk/node_modules/socket.io/node_modules/engine.io/node_modules/ws
amazon-ebs: > (node-gyp rebuild 2> builderror.log) || (exit 0)
amazon-ebs:
amazon-ebs: make: Entering directory '/opt/socks-proxy/node_modules/tomahawk/node_modules/socket.io/node_modules/engine.io/node_modules/ws/build'
amazon-ebs: CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
amazon-ebs: bufferutil.target.mk:96: recipe for target 'Release/obj.target/bufferutil/src/bufferutil.o' failed
amazon-ebs: make: Leaving directory '/opt/socks-proxy/node_modules/tomahawk/node_modules/socket.io/node_modules/engine.io/node_modules/ws/build'
amazon-ebs:
amazon-ebs: > ws@0.4.31 install /opt/socks-proxy/node_modules/tomahawk/node_modules/socket.io/node_modules/socket.io-client/node_modules/engine.io-client/node_modules/ws
amazon-ebs: > (node-gyp rebuild 2> builderror.log) || (exit 0)
amazon-ebs:
amazon-ebs: make: Entering directory '/opt/socks-proxy/node_modules/tomahawk/node_modules/socket.io/node_modules/socket.io-client/node_modules/engine.io-client/node_modules/ws/build'
amazon-ebs: CXX(target) Release/obj.target/bufferutil/src/bufferutil.o
amazon-ebs: bufferutil.target.mk:96: recipe for target 'Release/obj.target/bufferutil/src/bufferutil.o' failed
amazon-ebs: make: Leaving directory '/opt/socks-proxy/node_modules/tomahawk/node_modules/socket.io/node_modules/socket.io-client/node_modules/engine.io-client/node_modules/ws/build'
amazon-ebs: proxy-socks-server@1.0.0 /opt/socks-proxy
amazon-ebs: npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
amazon-ebs: npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
amazon-ebs: npm WARN proxy-socks-server@1.0.0 No repository field.
amazon-ebs: ├─┬ cluster@0.7.7
amazon-ebs: │ ├── log@1.4.0
...
amazon-ebs: └─┬ ipv6@3.1.1
amazon-ebs: ├── cli@0.4.5
amazon-ebs: └── sprintf@0.1.3
amazon-ebs:
...
(packer continues)

```

So, it seems that `npm` does not return an error code if there are timeouts. The result is that packer will not notice it and in the end, it will result in a potentially broken AMI.

**Is there some way to make `npm install` less risky?**

When in doubt, I would prefer that it breaks better too much than too little when there is a chance that the installation was not successful. Then you would at least see it and then you can react on it. Otherwise, in the worst case, you will end up deploying a broken AMI.

---

To simulate the error handling, I also made a local experiment where I disabled my network connection and run `npm install`. It took some while, but eventually gave up. Still it continued silently and returned a non-error return code:

```
npm install && echo "SUCCESS"
npm WARN optional SKIPPING OPTIONAL DEPENDENCY: fsevents@^1.0.0 (node_modules/chokidar/node_modules/fsevents):
npm WARN notsup SKIPPING OPTIONAL DEPENDENCY: Unsupported platform for fsevents@1.1.1: wanted {"os":"darwin","arch":"any"} (current: {"os":"linux","arch":"x64"})
npm WARN proxy-socks-server@1.0.0 No description
npm WARN proxy-socks-server@1.0.0 No repository field.
SUCCESS

```

Here, I would rather see it fail when it is run in an automated environment.

# Answer
Try piping your output to grep and then failing based off of the return code of grep:

```
npm install 2>&1 | grep "Error: connect ETIMEDOUT"

```

Per the grep documentation,

> the exit status is 0 if a line is selected, 1 if no lines were selected, and 2 if an error occurred.

If needed, you can "not" the return code or just invert the conditional so that:

```
if [ "$?" -ne "0" ]; then
    echo "Packer failed!"
    exit 1
fi

```

becomes instead:

```
if [ "$?" -ne "1" ]; then
    echo "Packer failed!"
    exit 1
fi

```
> 4 votes

---
Tags: packer, npm
---


# Question
Title: What is the ec2 cli command to modify the minimum number of nodes in a scaling group?
I've got an ec2 autoscaling group that starts in the morning and finishes in the evening. I'd like to create a 'button' that people can click to warm up the autoscaling group to run jobs in the middle of the night (only on an on-demand basis).

My question is: **What is the ec2 cli command to modify the minimum number of nodes in a scaling group?**

# Answer
I'd go with `aws autoscaling set-desired-capacity` following the documentation

But if you're up to make a button you should look into the various AWS sdk (on top right is the list of them) to call the related API instead of forking to a system command.
> 0 votes

---
Tags: amazon-ec2, autoscaling, aws-cli
---


# Question
Title: How to ensure that removed allowed ports will be removed using Ansible's UFW module?
The UFW module document has been read, but it is unclear how to let ansible remove ports that are not defined anymore.

It works to add new ports, but now 5 ports have been removed and I do not want to login to each of the more than 50 servers.

It is possible to write some custom exec, but I would like to use the Ansible's UFW module's functionality. Does it support such a scenario?

# Answer
Based on:

> ```
> # Allow OpenSSH. (Note that as ufw manages its own state, simply removing
> # a rule=allow task can leave those ports exposed. Either use delete=yes
> # or a separate state=reset task)
>
> ```

The following snippet was added to the top of the file:

```
---
- name: Reset UFW
  ufw:
    state: reset

```

Once ansible was run:

```
TASK [rolename : Reset UFW]

```

the undefined ports were removed
> 4 votes

---
Tags: ansible
---


# Question
Title: Jenkins pushing to branch overwrites commits made in interim
A while ago I had written about how commits were being merged (successfully) into our develop branch in Git, but would later go missing, with no evidence of the missing code other than all the Bitbucket logs proving the merge happened.

We just figured out yesterday that what's happening is Jenkins clones our repo, then holds it for up to 40+ minutes while jobs run, then pushes back with `git push origin +develop`. This is essentially overwriting any commits we've made in that 40+ minutes as Jenkins pushes back the branch exactly as it saw it on the clone, and minus our additions.

We thought the solution would be to change to

```
git pull --rebase origin
git push origin develop

```

but that's returning

```
+ git pull --rebase origin
error: Cannot pull with rebase: You have unstaged changes.

```

To clarify, Jenkins' "branch" has files on it that we do need merged into the repo. The ideal situation is to find the right way to do a fresh pull of develop into Jenkins' copy before pushing back.

What's the correct git procedure to avoid this issue?

**Note**: My original post was deleted for being too vague, and rightfully so since at the time I had zero idea why things weren't working. In this case what happens is Jenkins checks out the repo, runs a script that makes parameter files needed for a second job, then checks it back in so the files (and associated logs) exist in the repo for use/auditing. Not having Jenkins do the process isn't an option.

# Answer
It really depends on your situation, but I'll go with the generic/naive approach and say do exactly what your developers do in the same situation: I pulled an hour ago, I made some changes, now I want to push my changes, but other changes have been pushed in the meantime. I commit, pull/rebase, then push. Assuming that whatever Jenkins is committing isn't tied to the commit it built from, this is the same process it should follow to get the same result.
> 3 votes

# Answer
From a high level, why do you have both jenkins and developers pushing to the branch? If you have enough CI testing for jenkins to force-push to the branch, then you have enough CI testing to revoke push access from the developers. At first it sounds scary to only have robots pushing to branches (and you should also have a small core team with push access which is only used in the direst of situations), but once you take the plunge into branches gated by CI -- meaning CI must pass before merges are allowed -- operational confidence in the code will quickly increase.
> 1 votes

---
Tags: jenkins, git
---


# Question
Title: What is the proper relationship between rollback/rollforward and MTTR metrics?
I'm trying to understand the best way to capture data to start measuring Mean Time To Repair (MTTR) metrics, and I need to wrap my head around how "rollback" impacts MTTR positively or negatively.

## Scenario 1

Assuming that solid monitoring is in place, code is deployed that causes an incident that is detected rather quickly (low MTTI). At the point of identification, there are two main possible paths forward (yes, I'm oversimplifying for purposes of discussion):

1. Rollback the deployment, returning stability quickly, but without the intended features in production.
2. Roll-forward with additional changes that resolve the incident and keep the intended features live.

In this scenaro, MTTR is pretty darn low, given that site stability can come back pretty quickly. That said, the intended outcome of the change isn't live, and thus the code/feature/change is still stuck in process. If a goal is low MTTR, it seems to incentivize roll-back as a recovery mechanism.

## Scenario 2

In this scenario, MTTR is strictly measured by how long it takes the expected code/feature/change to be working properly in production. Even if I rollback, until my "fixed" code change goes into prod, the MTTR timer is still running. In this case, MTTR seems tied to business outcome stability instead of just pure "hey, things are stable."

Now, the answer may be as simple as MTTR not being used as a metric in a vacuum, but rather in conjunction with Change Failure Rate - a super-low MTTR caused by frequent rollbacks could point to a sky-high Change Failure Rate. That said, there's something that doesn't seem right to me in the idea of divorcing the MTTR measurement from business outcome.

I may be way overthinking this, but I'm curious how others are measuring MTTR and what the end point-in-time is for "recovery." Are you using it simply as stability, or do other factors come into determining what "recovered" means?

# Answer
> 2 votes
Yes, MTTR is/should always be tied to the business outcome: if things are not stable the very business is at risk.

The fact that the expected code/feature/change is still stuck in process in scenario 1 is irelevant: the feature is not stable, so it doesn't bring new business, rolling back is the best you can do at that time from the business prospective.

The rollforward is a gamble: keeps the business at risk waiting for a *potential* fix that in fact has statistically lower changes of success (due to the instability it will always be rushed compared to the change that caused the instability in the first place without even having such pressure on it). The rollforward is a yet another version of the code which hasn't been checked before.

If you want to keep the MTTR low you rollback immediately, without debate. This removes the business risk and gives you a chance to check that the fix is actually working before attempting to deploy it. I'd strongly suggest making it a policy as yes, almost always there will be someone asking for a fix instead of the rollback and calling a meeting to negociate/decide on it - all while business remains at risk.

Side note: if you're concerned with a high Change Failure Rate then I'd suggest checking the the actual rollback rate instead of deriving it from a low MTRR. Maybe you'd like to add a gate check before deployment for the most frequent failures. If you have such check already automated - why not include it in the CI verification? If you don't have one - maybe its time to start thinking about it? :)

# Answer
> 1 votes
The mean time to recover has an implied subject - the mean time to recover *what*? Defining this is key to using the metric effectively.

Are you recovering the general availability of your production website? Are you recovering the functionality of a particular feature that has a bug in it? Once you know what you're actually trying to measure, it's much easier to measure it!

The general thrust of your question seems to actually be surrounding the competing goals of shipping features and maintaining reliability, which is an ages-old battle. Traditionally it is developers' jobs to implement new things, and sysadmins' jobs to prevent things from breaking, and this leads to departmental conflict, as change tends to cause breakage. One of the philosophies oft associated with DevOps is the idea that developers and ops engineers should work closely together so as to ease this tension.

You may also be interested in Google's approach to that problem, which is to have "error budgets" for development teams to spend; once they've penalized stability too much, they must spend the rest of the quarter only working on stability. Along with this, the site reliability engineers have available goals, and if they *over* shoot, they are encouraged to let more changes through; the idea here is that their goal must not simply be to maintain reliability as high as possible, as then they'd be motivated to fight change in every situation.

---
Tags: metrics
---


# Question
Title: What is the difference between the traditional Development and Operations Model and Site Reliability Engineering?
> "SRE is what happens when you ask a software engineer to design an operations team." – *Site Reliability Engineering*

Since Google's Site Reliability Engineering Book was released, on more than one occasion I have been told that SRE is an extension of the existing Operations or Application Support model.

We've had a couple of questions that defined differences between Sys. Admins, DevOps Engineers and Site Reliability Engineers:

However none of these questions or their answers describe the **differences between a Systems Administrator and a Site Reliability Engineer**.

In broader terms: what are the *key differences* between Google's practice of Site Reliability Engineering and the traditional separated Development and Operations functions within a business.

# Answer
> 7 votes
Thankfully, since Site Reliability Engineering developed internally at Google and only recently has started to make its way into the broader community, it is fairly well-defined. What *isn't*, though, is web operations (or "systems administration" - as an example of the lack of clarity, you use both in your question). It's difficult to discuss the differences between two things when you're not altogether sure what one of them is.

But I'm an adventurous fellow, so I'll give it a shot.

---

In very traditional shops, developers and sysadmins are very siloed from each other. The devs build an app, then consider their job complete as soon as their code has been committed. The sysadmins take the build artifacts (which may be just the code, if it's an interpreted language) and deploy it to production servers. It's the sysadmins' job to keep the application running smoothly, and in general manage the production environment. However, often performance problems come from architecture issues in the app; the sysadmins don't have the programming knowledge to know what the app is doing, and the developers don't know how the app acts in the production topology with production traffic, so no one is equipped by themselves to solve the problem.

Additionally, the developers are usually judged on how quickly they can produce new features, while the sysadmins are judged on how infrequent the app breaks in production. Since change is one of the leading causes of breakage, this puts the two departments at odds with each other - an old rivalry that hurts the business and the people involved.

At some point, some developer-centric companies got so annoyed at this that they began practicing "NoOps" - they eliminated their operations departments and the perceived roadblocks that came with them. In reality, this meant that developers took on operations roles, but maintained their old titles.

In a discussion surrounding NoOps, John Allspaw, then VP of Technical Operations at Etsy and an editor of the well-respected Web Operations book, defined roles at Etsy this way:

> Etsy Operations is responsible for:
>
> * Responding to outages, takes on-call
> * Alerting systems thresholding, design
> * Architecture design and review
> * Building metrics collection
> * Application configuration
> * Infrastructure buildout/management
>
> Etsy Development is responsible for:
>
> * Responding to outages, takes on-call
> * Alerting systems thresholding, design
> * Architecture design and review
> * Building metrics collection
> * Application configuration
> * Shipping public-facing code
>
> Neither of those lists are comprehensive, I'm sure I'm missing something there. While Etsy Ops has made production-facing application changes, they're few but real (and sometimes quite deep). While Etsy Dev makes Chef changes, they're few but real. If there's so much overlap in responsibilities, why the difference, you might ask? Domain expertise and background. Not many Devs have deep knowledge of how TCP slow start works, but Ops does. Not many Ops have a comprehensive knowledge of sorting or relevancy algorithms, but Dev does. Ops has years of experience in forecasting resource usage quickly with acceptable accuracy, Dev doesn't. Dev might not be aware of the pros and cons of distributing workload options across all layers1-7, maybe only just at 7, Ops does. Entity-relationship modeling may come natural to a developer, it may not to ops. In the end, they both discover solutions to various forms of Byzantine failure scenarios and resilience patterns, at all tiers and layers.

In his world, developers and ops engineers had very similar high-level skill sets and responsibilities; where they differed was in their expertise. Their differing specialties encouraged them to work together to solve problems, and their common base-level skills gave them a language in which to do that.

This is generally the definition of web operations that I land on for most cases. So it's the one we're going to continue along with.

---

So then, what is Site Reliability Engineering?

The Google SRE book opens with a definition of SRE... and then another one... and then spends a chapter continuing to define the role and an entire book covering the specifics. Even when developed in one organization, it seems that it's difficult to condense the job down to one single agreed-upon definition.

To start with, we need to walk back to 2003, when Ben Traynor joined Google and founded what came to be the first Site Reliability Engineering team. Recall that a few paragraphs ago we were in the early 2010s; but in 2003, the industry was still pretty set on the sysadmin/developer divide as the natural way of things. So when Ben says that SRE was what would happen if a software engineer created an operations team, this was a much more radical melding of the two worlds than it appears now.

The definition given in the preface emphasizes each of the three words individually:

* **Engineering** \- the use of computer science and engineering concepts to solve problems
* **Reliability** \- a focus on making systems more scalable, more reliable, and more efficient
* **Service** \- the later evolution of "site", emphasizing that SREs are responsible for networked services

The introduction chapter lists the tenets of Site Reliability Engineering as:

* **Ensuring a durable focus on engineering** \- taking pre-emptive action to avoid frequent pages and other "toil"
* **Persuing maximum change velocity without violating a service's SLO** \- a subject that can easily have its own several-hundred word answer, but roughly summarized as *helping* developers make changes, as long as they don't cause *too many* issues
* **Monitoring** \- automatic alerts when things go wrong
* **Emergency response** \- fixing things when they're broken
* **Change management**
* **Capacity planning**
* **Provisioning**
* **Efficiency and performance** \- ensuring that a service performs *at* an expected level - bottlenecking hurts users, but excess capacity costs money

---

I'd categorize Site Reliability Engineering as a specialized subset of modern Web Operations. An SRE organization focuses heavily on automating *everything*, to a degree that is only cost-effective in fairly large companies. Ideas like error budgets can only work when your service has many, many requests, as otherwise you lose granularity (for a smaller service, a particular error could affect 0-20% of your requests, depending on the minute). Related areas like security are absent from the SRE definition because companies large enough to have true SRE teams have dedicated teams for security.

The SRE program, as defined by Google, is web ops developed for the specific needs of Google, and not necessarily applicable elsewhere.

However, Site Reliability Engineering has been expanding in broader industry use recently. My current job title is an SRE, even though I work at a much smaller company and my job description fits pretty well with John Allspaw's 2012 Etsy web ops definition. My theory is that we've been progressing through titles as a shorthand for espousing the evolution of a single field:

* We started as **sysadmins**.
* Then as web sites became more of a "thing", job postings started to refer to **web operations engineers** to distinguish sysadmins who specialized in the web from those who also handled general office IT.
* Then **DevOps** was supposed to separate out those who were comfortable using programming to reduce their web ops workload.
* But as DevOps got muddled by the lack of a clear definition, we adopted **Site Reliability Engineering** to specify that we're looking for people who are on-call supporting production services.

So what is the difference between a sysadmin and an SRE? The year in which they received their title. What is the difference between traditional operations and site reliability engineering? SRE is merely the current incarnation of ops, using new tools (hello, containers!) and, as networked programs continue to become more larger and more important, an increased focus on practices that allow one engineer to do *more*.

---
Tags: sre, operating-model, sysadmin
---


# Question
Title: How to get the ARN for an arbitrary Amazon Resource?
I am creating an AWS IAM policy for a Serverless Framework project, I am applying the Principle of Least Privilege to ensure the policy is as tight as possible. To restrict the scope of the policy I need to be able to identify the ARN for each of the resources, i.e.:

```
arn:aws:logs:us-east-1:123456789012:log-group:my-log-group:*

```

Looking through the CloudWatch and CloudFormation UIs I can't actually see any mention of the resulting ARN for a created log group.

*Question: For any resource within Amazon Web Services how can I obtain the ARN?*

# Answer
> 5 votes
https://docs.aws.amazon.com/general/latest/gr/aws-arns-and-namespaces.html lists the schema for (I believe) all of the ARNs that Amazon supports. Text in italic red indicates where you should substitute in your own value, rather than the literal string, and there are examples in each section to help figure this out.

Be warned that IAM is less usefully restrictive than it appears on first glance. To start with, most services have only partial IAM support. Secondly, often the things that are supported at a resource level are supported only in very specific ways; for instance, granting all permissions on a subnet ARN does not grant the attached users the ability to fully manage instances in that subnet, as `AttachVolume` only supports restricting by volume id, etc. Thirdly, if the users are going to use the Amazon web console at all, they must have broad viewing permissions, as the console was not written with PoLP in mind and issues non-resource-restricted API calls (and often merely dumps errors in the javascript console!).

---
Tags: amazon-web-services, aws-iam
---


title: Vendoring - what is it?

text:
# Question
Title: Vendoring - what is it?
In various tools there is the concept of "vendoring". i.e. Ruby's bundle, and Chef's Berkshelf.

"vendor" is not a verb and in documentation it's often assumed that I know what it means. Can someone help shed some light on what it is and the benefits of doing it.

# Answer
> 7 votes
Vendoring is the concept of downloading/installing a specific version of dependencies and making it available elsewhere (typically, within a local repo/folder of your supporting application)

Vendoring is usually done to prevent breakage occurring when dependences are no longer available. Vendoring also prevents your applications from breaking when dependencies are no longer available( for ex: the leftpad incident)

For example, let's consider you have your Chef cookbook that depends on another third party cookbook, say mysql2 chef gem. Now while mentioning in your `metadata.rb`, you didn't pin the version or vendor the cookbook. Lets consider the cookbook has had a version upgrade, breaking all previous APIs. The previous versions have also been taken down. In such a case, your cookbooks are suddenly broken as well and if you have a urgent deploy, your Chef run will fail.

Vendoring prevents this by making available the dependency on your local chef server, so even if the dependency has gone ahead, you can continue referring to the previous version and your cookbook will not break

(Also goes without saying, you must keep updating the dependencies and cookbooks, else you might get bit with some bad bugs)

---
Tags: terminology, chef
---


title: How to hook up Jenkins to Endevor?

text:
# Question
Title: How to hook up Jenkins to Endevor?
I'm not DevOps, I only come from a .net background, but I've been asked how to hook up Jenkins to kick off jobs, listening to Endevor mainframe?

I only found this which talks about SCLM which as I understand is a competitor company.

# Answer
Apart from your "this", there is also Compuware Source Code Download for Endevor, PDS, and ISPW Plugin. Some details about it (from the linked page):

> ... allows Jenkins users to download Endevor, PDS, or ISPW members from the mainframe to the PC. Source can then be accessed on the PC, for example, for SonarQube analysis and reporting.

So you may want to give "that" a try ...

PS: I'm assuming your question is about Endevor SCM, and not Endevor DB (that a completely different story, and a challenging one ...).
> 1 votes

---
Tags: jenkins, continuous-integration, mainframe
---


# Question
Title: Securely grab minion ID in pillar top file template?
I found this SLS file in a pillar top.sls targeting example:

`top.sls`:

```
base:
  '*':
    - hostspecificsls

```

`hostspecificsls`:

```
# See https://groups.google.com/forum/#!topic/salt-users/UWfP6bMe6F8
{% include 'hosts/' + salt['grains.get']('fqdn') + '.sls' ignore missing %}

```

This strikes me as distinctly insecure, as it runs counter to one of the Salt Best Practices:

> Don't use grains for matching in your pillar top file for any sensitive pillars.

Why is this insecure? Well, grains are evaluated *on the minion*, so if a minion is sufficiently compromised, it can return whatever an attacker wishes for an arbitrary grain call. A compromised minion might claim to be an SSL (or VPN!) terminator to get at private key material, for example.

Fortunately, there's a (theoretical) solution; use the minion ID. Minion IDs are unique, and they're verified on the master cryptographically.

Except I can't find how to get at the minion ID without depending on grains. Remember, the grains are executed on the minion, so even if the Salt master knows at the transport layer what the minion ID is, if you query for the minion ID via grains, you're asking the minion for the ID, not the master.

So how can I get at the master's concept of the minion ID from Jinja on the master?

# Answer
> 3 votes
There is currently one safe way to do this.

You can use `{{opts.id}}` as an explicit way to grab the minion ID. `opts` is technically an implementation detail; the `opts` dictionary is a Salt internal structure that doesn't appear to be formally specified. Its candidacy for the purpose was pointed out to me in a bug comment.

In the next version of Salt (so, later than 2016.11.5) the `id` grain, will have been special-cased, making it possible to use `{{grains.id}}` safely.

---
Tags: salt, jinja, pillar
---