Improving the IT Security Industry – A Top-Down Effort

Wednesday, August 10, 2011

Don Eijndhoven


By Don Eijndhoven 

The ever ongoing debate about quality IT staff once again received a nudge, this time by an article of J.Oquendo.

In his article he takes another brutally honest stab at the Industry by pointing out that the new Shady RAT attacks aren't that new and would have been easily caught by capable personnel.

I agree with that view very strongly and would also like to point out that Shady RAT is really no different than Night Dragon in that both attack waves used techniques that have been known for a decade or more. Obviously someone is asleep at the wheel, but who?


In several articles I've seen about this topic, I have seen in-depth descriptions of the observed failures of the staff itself as well as the certifications that should have tested their skills.

These seem to me to be symptoms rather than a cause, and one that I don't see in many other industries.

Most industries have some kind of self-correcting function built in. In the Medical profession there is a Medical Board that reviews its members and is able to punish shoddy work.

Lawyers can be disbarred by the Bar Association in their district. A bad carpenter may well find himself nailed upside-down to a wall if he doesn't pull his weight during a large construction project. All of these are examples of Peer Review. What makes the IT industry so different?

Two major differences immediately came to mind:

  • Cost of mistakes are hard to quantify (or even detect) in IT and;
  • Line- and Project management are much less skilled in IT than other industries are in theirs.

Cost of mistakes are hard to detect and quantify

Compared to other industries, mistakes made by IT personnel aren't always obvious. Systems may keep on working and may even work properly when its poorly configured. If a system does crash, its often very hard to quantify exactly how much damage there is and what it has cost the company. 

If a surgeon makes a mistake, the effect is often immediate (e.g. a patient keels over). If a construction worker makes a mistake, a building may collapse. In either case a problem is usually clearly visibly detectable and peer review takes place. Lack of visibility and immediate effects inhibit such peer review in the IT industry.  

Line- and Project Management personnel are not sufficiently skilled in IT to manage its staff

The fact that IT is still somewhat of an ethereal topic to most people is reflected in the poor choices made when hiring management personnel. You wouldn't believe how often I've heard it said that 'IT managers don't need to know IT, they just need to manage the people'. This is just plain wrong.

Yes they need to be skilled in managing people, but they also have to make regular professional judgement of the quality of work provided by the staff they are managing. Virtually every other profession does this better than we in the IT industry.

I believe this has a lot to do with the fact that there are less IT-savvy managers to begin with and so management accepts second-best as its defacto standard. There also seems to be less promotion from the ranks than in other industries.

Maybe the stigma of IT personnel having less social skills (think Geek or Nerd) has its part in this problem, I don't know and wouldn't care to judge its veracity. What is evident is that there aren't nearly as many well-educated (in IT!) CIO's as we should have.

We need those proper CIO's to hire proper IT managers, who in turn hire proper personnel instead of the pseudo-specialists that are so often the topic of negative discussion.

Of course you could say that its up to the IT professionals to get themselves skilled, but we've tried that and it doesn't work. And why would they? Many of them skate by excellently with a minimum of effort because of that 'peoplemanager' with the bachelor degree in napkin folding you thought would do just fine (and wasn't he cheap!).

As an organization, try the following:

  • Stop assuming that 'any bachelor/master degree' will suffice for an IT position. The higher up the manager is going to be, the more skill you can ask for the position. That includes the CIO position! Although their knowledge has to be scoped broader, it must still be present and relevant.
  • Promote from the ranks where possible. The pecking order in an IT department is established fairly quickly and its almost always based on skill and knowledge. Leverage that information in getting the right people promoted. If you choose right, they'll be perfectly capable of hiring their own replacement.
  • When hiring technical personnel, have each applicant vetted by your best tech(s), even if it is a contractor. Listen to their advice.
  • Don't let certifications dazzle you. Many certifications don't mean much anyway. Look to match certifications with practical experience and you'll fare better.
  • Remember: If you pay peanuts, you'll get monkeys. If you don't have money, find other ways to entice new personnel such as exciting projects or nice perks.
  • Recruiting agencies often play it fast and loose with matching your needs to their staff. Don't assume their personnel is any better - verify! Remember: You're paying a premium and deserve quality. Ask them about the training their staff receives. If they're any good, it should be at least a periodically recurring thing. I know companies that demand an x-amount of study a year per employee.

Cross-posted from

Possibly Related Articles:
Information Security
Certification Training Standards Information Security Infosec CIO Shady Rat J.Oquendo
Post Rating I Like this!
J. Oquendo Very well written and I figured I'd sort of counter-yet-expound on some of your key points.

Promotion and Collaboration

Promoting from within will almost always work best in the long run as insiders know and will ALWAYS know the environment in far greater detail. When it comes to security, there needs to be more collaboration between insiders on all levels. This means that the chest thumping (that's not my department/you don't have the right to) needs to cease.

I cannot count how many times where I have had to deal with department heads where, when information is needed, there is the "job preservational" response of "we can't let you", "you're not authorized" and so on. From a security engineering perspective, there should be no reason in the world why I should suffer from red-tape when trying to baseline security from another function (networking dept, application devel dept, etal). Collectively there is no reason why collaboration is not a key function in any industry. There is always going to be the need for separation of duties however, separation of duties and collaboration are two different beasts.

Candidate vetting

Vetting candidates can be a trick task and I say this as having had to vet candidates myself in the past. As a candidate comes in for a technical interview, there is a high likelihood of nervousness for the candidate. This nervousness should not be confused with "unqualified." Those responsible for vetting a candidate need to take a collaborative approach as opposed to the typical geek slash arrogant approach of "no one knows it better than me." The approach needs to be "non template" driven and by this I mean, questions need be broad yet focused on what is really necessary.

For example, if hiring say a network engineer, he may not know much about say web application security. Should he be disqualified just because of this? The questions need to be specific to the position being filled and broad in the sense of the topic on a generic level working its way down to the granular levels. One can get a better grasp of how much a candidate knows. It is always vital to remember though that a candidate is likely to be nervous to some extent and interpretations can sometimes differ. When interviewing candidates for security where Linux was a concern, in the past I have asked questions such as: Name three different ways to block an attacker from reaching the machine WITHOUT using a firewall. I did so to see their methods of thinking and versatility not only on the networking/security scale, but to see if they have enough Linux know how to accomplish a task under differing circumstances.

Certification versus Experience

Far and long argued. If someone has say 3 years experience and more than say 2 certs? The interview will not go far if I am interviewing a candidate. When it comes to certifications I am torn as many equate to a candidate understanding how to take tests. That statement was not written to incur the holy wrath of many certified individuals, on the contrary it was written to make people think. I myself have +5 certifications with 95% of them being all technical certs and 90% of that 95% being written + practical exams.

In the days of yesterDecade, one usually got the training, experience, then aimed for a certification. Nowadays, everyone is strictly looking for a paper however, having the certification does not equate to having experience. Everyone should be vetted no matter what certification they hold.
Don Eijndhoven Thanks for that reply J. I especially agree on the Vetting part and had this in mind when writing the article, but going into that would have taken it away from what I was aiming at. Regardless I fully agree and fully recognise that "no one knows it better than me" attitude. It somehow can become somewhat of an ego thing for the vetting party and thats NOT what should happen.

I worked at a place where the engineers wrote up 30 questions based in reality and fired that at the candidate. Some were multiple choice, but not all. I liked that. Many failed and fell through even though they weren't hard questions. Those people failed because they were good at memorizing Microsoft questions and these were ours :)
