The Biggest Problem in Computer Security

People tend to focus on various areas as being important for computer security such as memory corruption vulnerabilities, malware, anomaly detection, etc. However the lurking and most critical issue in my opinion is staffing. The truth is, there is no pool of candidates out there to draw from at a certain level in computer security. As an example, we do a lot of consulting, especially in the area of incident response, for oil & gas, avionics, finance, etc. When we go on site we find that we have to have the following skills:

1. Soft skills. (often most important) The ability to talk to customers, dress appropriately, give presentations or speak publicly, assess the customer staff, culture and politics, and determine the real goals. I can't stress enough how important this is. It's not the 90s anymore, showing up with a blue mohawk, a spike in the forehead and leather pants, not a team player, cussing and surfing porn on the customers system doesn't cut it no matter how good you are technically. If you are that guy then you get to stay in the lab and I guarantee you will make far less money. Even if you can write ASLR bypass exploits and kernel rootkits.

2. Document. This ties with the above for number 1. If you didn't document it, you didn't do it. I don't care how awesome an 0day you discovered, or what race condition in the kernel you found. If you cant clearly document it, the customer doesn't care and sees no value in what you did. The documentation has to be clean, clear, layed  out so that an executive can understand it and so that the other security firm the customer hires to validate your results doesn't make fun of you.

3.) The ability to mine disparate sets of data. This means taking in apache logs, windows Event logs, proxy logs, full packet captures. Handling, splitting and moving terabytes of data. Writing data mining code in sed/awk/bash/perl/python/ruby. Correlating events, cutting out desired fields, reassembling binary files from packets, etc. Using graphics visualization packages to map out an intruders connections on a network based on netflow data.

4.) Reverse Engineering. This means disassembling binaries in IDA, running binaries in a debugger such as Ollydbg, WinDBG, IDA, memory forensics, and especially de-obfuscation. Can you unpack a binary? How about if the packer is multi-stage and does memory page check summing? What if the packer carries its own virtual machine? Do you know what breakpoints to set, when to change the Z flag, or how to hot patch a binary in memory?

5.) Understanding programming. To be good at this stuff you need to know C, C++, .NET, VB, HTML, ASP, PHP, x86 assembly and another dozen languages, at least well enough to look up APIs, understand standard libraries, discover which imports are important.

6.) Operating systems. You should know the ins and outs including file systems, memory management, kernel, library system and key command line tools of at least half a dozen OS's, especially as they are used in enterprise environments. Domains, NFS, NIS, kerberos, LDAP. So not only windows, linux and OS X, but also solaris, AIX and some embedded or mobile systems.

7.) Exploit development. Often on engagements you run across an exploit or even an 0day that you must reverse engineer, replicate safely and test on the customers particular environment. You have to be able to take it apart, analyse the shellcode, understand everything its doing and re-write your own version of it.

8.) Versatility with a wide variety of tools, many of which are not easy to access outside of the enterprise. At a minimum enough technical base knowledge to use whatever tool is put in front of you. Examples include wireshark, splunk, fireeye, netwitness, arcsight, tippingpoint, snort / sourcefire, bluecoat, websense, TMI, Encase.

All of the members of your team whether you are a consulting shop or an internal incident response team need to be able to do these things and overlap with each other. Some can be stronger in RE than network forensics but everyone has to be able to do all of it to some extent, especially 1 and 2.

The problem with this? These people don't exist, they are unicorns. Those who can do this are either already employed, well payed and tackling more interesting problems than you can offer, or they are running/partners in their own company that you could (and should) outsource to. </shameless self promotion>. But even small boutiques that can do the above are rare, heavily booked, and are charging close to high powered lawyer hourly rates. (when people question rates I point out that big name IR shops are around $400/hr and even the BestBuy geek squad charges $120/hr to reload your OS).

A lot of big contractors are trying to approach security like they did IT in the 90s and 00's. Bid low, win a huge contract, then put out job ads for anyone who knows how to use a computer. The problem is, while you can come up to speed for a help desk or to admin a windows server relatively quickly, the above list of skills takes a decade + to master. So big contractors are failing, badly, and trying to buy up the small guys. But there is another problem there as well.

People who are able to do the above 1.) Value freedom highly and don't want to work 9 to 5 in a cube farm and 2.) Don't want to live or work long periods of time onsite where you are. They don't want to live in Houston or in Cleaveland or in Indianapolis or probably even in the DC area. They want to live in La Jolla and San Francisco and New York and someone, somewhere is willing to pay them a lot to do it, and probably do it remotely most of the time, so you are going to lose there.

In response, many companies try to follow the old plan of recruiting at colleges. In a lot of cases these students come out knowing some Office and probably some Java and that's about it. You might luck out and get a good RIT, Georgia Tech, New Mexico Tech student who knows more but most likely these have already been recruited to the government or somewhere else. And the learning curve time is long enough that by the time they are really good, they have already moved on. This kind of work is PRIME for remote. Let people come in for a week every other month. If you require internal security people to be on site all the time in some crappy city you will fail.

On the security company side you have the same problem, no one to hire. So many security companies, in order to grow (because the way you make money in services is via higher staffing levels) hire whatever they can find and field them. This continues the trend in mediocre security, companies getting owned, PCI, etc. Boutiques cannot grow to the size necessary to win the bigger contracts because there is no one to hire.

The solution many companies have been trying out is to focus on buying appliances and contracting pro services to set them up and hope that automation can solve the problem. It cannot. Here is a perfect example. A customer has a box that detects malware in email attachments. It flagged a PDF as highly malicious. We decided to check it out and at first glance it looked very bad. It had all the classic signs of an exploit, heap spray, etc. You couldn't tell the difference between it and another verified malicious PDF. However, upon further inspection we discovered that a popular autocad type program generated legitimate PDFs that looked this way. This is something that is not automatible. You must have an experienced and skilled analyst to do this. No amount of rack mount, fancy logo appliances will help you. And the bigger your enterprise the more you need. Every enterprise block of 30 - 50k IPs needs a team of 5 - 10 people.

Which leads me to the next issue. How you perceive your staffing resources. Example: One company I saw told they had a staff of 12 analysts to deal with security detection and response. I thought wow pretty good! Lets break the team down:

• A manager, full time in meetings, paperwork, etc.
• An assistant to the manager, secretarial work, etc.
• 3 senior advisers, i.e. guys about to retire, smart guys who give great advice and hold institutional knowledge, but not analysts
• 5 people involved in tool testing, stand up and maintenance (all those boxes I mentioned before). Great guys, not analysts or really involved in analysis
• 1 Developer mostly focused on designing queries and interfaces for the tools.
• 1 Actual analyst. 

While management believes they have 12 people and doesn't understand why things take so long they actually have 1 person. This situation is very common in big companies. 1 good analyst for an enterprise is not NEARLY enough. And you can't be reliant on a specific person unless you want to set yourself up for a disaster (while at the same time you must cultivate and care for those star players).

That's my case for why staffing is the most important issue we face in computer security.  What is the solution? Some would say training, but lets be honest, were you back home writing rootkits for work after taking Hoglund and Butler's class at Blackhat? Probably not. Have you found piles of valuable 0day after completing Halvar's most excellent course in Vegas? I doubt it. A 2 day - 1 week course isn't doing it. Going through the entire SANS curriculum isn't doing it and CISSP sure as hell isn't doing it.

You have to spend around 6hrs a day, after work, highly focused on coding, reversing, etc. for a minimum of 2 years to be decent. That is how the adversary does it. That's how the big name researchers and best staff does it,  and unfortunately you only need a couple of attackers for every 10 defenders out there.

V.