Net Eng Skills Gap Redux: Entry Routes into Network Engineering

While I was recently at the LINX meeting in London, I ended up having a side-discussion about entry routes into the Internet Engineering industry, and the relatively small amount of new blood coming into the industry.

With my UKNOF Director’s hat on for a moment, we’re concerned about the lack of new faces showing up to our meetings too.

Let me say one thing here and now:

If you work in any sort of digital business, remember that you are nothing without the network, nothing without the infrastructure. This eventually affects you too.

Yes, I know you can just “shove it in the Cloud”, but this has to be built and operated. It has real costs associated with it, and needs real people to keep it healthily developing and running.

I’ve written about this before here, almost 3 years ago. But it seems we’re still not much better off. I think that’s because we’ve not done enough about it.

One twitter correspondent said, “I didn’t know the entry route, so ended up in sysadmin, then internet research, and not netops.”

This pretty much confirmed some of my previous post, that we’d basically destroyed the previous entry route through commoditisation of first-line support, and that was already happening some time around 1998/1999.

It’s too easy to sit here and bleat, blaming “sexy devops” for robbing Net Eng and Network Infrastructure of keen individuals.

But why are things such as devops and more digital and software oriented industries attracting the new entrants?

One comment is that because a large number of network infra companies are well established, there isn’t the same pioneering spirit, nor the same chance to experiment and build, with infrastructure compared to the environment I joined 20 years ago.

My colleague, Paul Thornton, characterised this pioneering spirit in a recent UKNOF presentation titled “None of us knew what we were doing, we made it up as we went along” – note that it is full of jargon and colloquialism, aimed at a specific techie audience, but if you can excuse that, it really captures in a nutshell the mid-90’s Internet engineering environment the likes of he and I grew up in.

Typing “debug all” on a core router can liven up your afternoon no end… But I didn’t really know what I wanted to do back then, I was green and wet behind the ears.

Many infrastructure providers are dominated by obsessions with high-availability, and as a result resistance to change, because they view a stable and available infrastructure as the utopia. An infrastructure which is being changed and experimented upon, by implication, is not as stable.


Has a desire to learn (from mistakes if necessary!) become mutually exclusive from running infrastructure?

In many organisations, the “labs” – the development and staging environments – are pitiful. They often aren’t running the same equipment as that which exists in production, but are cobbled together from various hand-me-down pieces of gear. This means it’s not always possible to compare apples with
apples, or exactly mimic conditions which will exist in production.

Compare this to the software world, where everything is on fairly generic compute, and the software is largely portable from the development and staging environments, especially so in a world of virtualisation and containerisation. There’s more chances to experiment, test, fail, fix and learn in this environment, than there is in an environment where people are discouraged from touching anything for fear of causing an outage.

This means we Network Engineering types need to spend a lot of time on preparation and nerves of steel before making any changes.

Why are the lab environments often found wanting? Classically it’s because of the high capital cost of network gear, which doesn’t directly earn any revenue. It’s harder to get signoff, unless your company has a clear policy about lab infrastructure.

I’m not saying a blanket “change control is bad”, but a hostile don’t touch anything” environment may certainly drive away some of the inquisitive folks who are keen to learn through experimentation.

Coupled with the desire of organisations to achieve high availability with the lowest realistically achievable capital spend, it means that when these organisations hire for Network Engineering posts, they often want seasoned and experienced individuals, sometimes with vendor specific certifications. You know how I hold those in high esteem, or not as the case may be, right?

So what do we need to do?

I can’t take all the credit for this, but it’s partly my own opinions, mixed in with what I’ve aggregated from various discussions.

We need to create clear Network and Infra Engineering apprenticeship and potential career paths.

The “Way In” needs to be clearly signposted, and “what’s in it for you” made obvious.

There needs to be an established and recognised industry standard for the teaching in solid basic network engineering principles, that is distinct from vendor-led accreditations.

In some areas of the sector, the “LAIT” (LINX Accredited Internet Technician) programme is recognised and respected for it’s thoroughness in teaching basic Internet engineering skill, but it’s quite a narrow niche. Is there room to expand the recognition this scheme, and possibly others have?

A learning environment needs to exist where we enable people to make mistakes and learn from them, where failure can be tolerated, and priority placed on teaching and information sharing.

This means changing how we approach running the network. Proper labs. Proper tooling. Proper redundant infrastructure. No hostile “change control” environment.

Possibly running more outreach events that are easier for the curious and inquisitive to get into? That’s a whole post in itself. Stay tuned.