Friday, December 11, 2015

Thoughts on the skills shortage


Sorry no meterpreter shells on this one.

Reading Trey Ford's article https://community.rapid7.com/community/infosec/blog/2015/11/19/ciso-guidance-on-building-the-team led me to want to put some ideas onto the blog that I've discussed at work and over beers but never here. So here it goes.

I'm not going to address each point rather I'm going to just share a few observations and opinions on the subject from my life/career.

1. I don't do any hiring but I can agree that there may be a lack of skilled mid to senior people in the market.  At every place I've worked it was always difficult to find qualified people to just to interview let alone hire. The fix, we/us/you/me need to grow them (more below).

1a. What I don't see is a shortage of INTERESTED junior people. There are tons and tons of people that want to get into infosec but sadly everyone wants mid to seniors and they don't want to train juniors.

2. It CAN be hard to afford people, especially  in expensive places like SFO/Silicon Valley, DC Metro Area, NYC, etc.  However, there is a real reluctance to allow remote workers, so when you base your HQ in an expensive area, or a place with a crappy commute AND don't allow remote employees then you don't get to complain that people are asking for lots of $$$.  Valsmith touched on this in a post as well; (http://carnal0wnage.attackresearch.com/2015/06/hard-to-sprint-when-you-have-two-broken.html).

That being said, I know a lot of people want to make a difference and do cool shit and they are willing to take slight pay cuts to do this (also mentioned in Trey's article). Management should keep this in mind.  Also, maybe its less the pay and more the sense that it's going to be impossible to make impact in your organization. That's what keeps me from wanting to go back to doing gov work.

3. There is a clear problem with senior people getting upset that people "get trained" and leave the company.  Bottom line, we shouldn't get upset.  Every person that goes from junior to mid or mid to senior and moves on to another company brings those skills with them and improves the other company and Security as a whole. Less companies getting pwned or more companies being able to react better/faster to attacks is a good thing.

We should reframe our thinking of not wanting to pay to train someone else's employee and more on we need to grow literally as many security people as we possibly can for our industry. Every company should think this way.

4. Have a FORMAL plan to grow your security people. An unamed CISO mentions this in Trey's article but saidly no details are given; 
“I like to work with entry level candidates on a 2-5 year growth path. I realize they may not be here forever, but I want to focus on giving them the right tools and a good experience.”
 I've  never had a job outside of the military that had a written plan to grow a security engineer/pentester from junior to mid or mid to senior. No required tasks or knowledge identified, no listed skills for my job role, no specific training to take, books to read, or anything to prove I was ready for that next level.  It has always been On the Job Training (OJT).  To be fair there is no replacement for OJT and its absolutely required to gain experience but there is no "growth path" when you rely on the whatever pentest comes in as what guides a person's development or whatever internal projects come up or fires to put out. I think we have attempted to rely on certifications to do some of this, and it does to an extent, but its general knowledge and not going to be organization or position dependent. Not to mention the whole value of certifications dilemma. 

You know who does have a plan to grow people from zero to competency? The military.  They take someone with aptitude (usually) but zero experience (well... assumes zero experience) and put them through training and testing with specific objectives and at the end they demonstrate proficiency in those specified tasks.

I'm not saying we need to get THAT formalized in our training but we need SOME plan on how to take someone with aptitude (and i'm going to make the assumption that if you got through college with a CS degree or demonstrate aptitude some other way) and repeatably train and grow that person from one level to another.

I don't know if we can do this collectively in a broad security community/PTES type sense (maybe we should try?) but i'd certainly like to see it implemented at a team level inside companies.

The second part of the article is also worth a read:
https://community.rapid7.com/community/infosec/blog/2015/12/07/ciso-guidance-on-building-the-team-part-ii


Thoughts?

CG
CG

11 comments:

mubix said...

Leadership tracks in companies have had these processes for years. Most even have a "leadership program". Where you are specifically groomed for management. Maybe we should adopt the same processes or at least steal some ideas from them. The only hitch in this idea is that most managers WANT to climb the latter. Most InfoSec don't want to go any further than "Senior", which most managers can't even comprehend. Just a suggestion on how to start fixing things...

dre said...

I don't care what org you are (or came from) or what structure you have: only one thing is important. We, i.e., the experienced crew that has been doing infosec (or cyber) for years -- we must start measuring ourselves by our ability to allow others around us to succeed. We should not focus on our own success, but rather by how many people we bring to the discussion and how far we take them with highly-technological, geographical-crossing, and efficiently-chained processes, models, frameworks, and platforms that grow themselves.

CG said...

well put Dre

Anonymous said...

I've been involved in hiring web application testers for around 8 years. 3 of those from within a consulting company and 5 from within a (non?) typical software development company. During my years consulting, we absolutely had a process and a plan for junior people to come up to speed to mid level technical people. However, the amount of junior people we were able to employ was about 1/8th of the number of mid to senior people we were hiring. The problem being you can't put junior people out into the field directly, thus limiting their billing potential, thus limiting your immediate return on hiring them. Unfortunately, the company revolves around making money so it wasn't possible to break that cyclical process.

Now, at a typical software development company we have tried to hire junior people. I have the same process we used in the consulting company at my disposal (because I developed it) however, it's been a huge task to find any decent junior people. If memory serves I'd say we interviewed 20 junior candidates this year. Of those, one wasn't asking for senior salary. And that one was so weak on skills but so egotistical about what they thought they knew that I decided it would be far too much trouble, based on personality, to re-train them properly.

So, in short, I have experienced a completely different issue. I haven't been able to find junior people who acknowledge that they are junior and want and are willing to be trained. I certainly feel like the "I used to walk up hill to school in a foot of snow" crowd at this point. But, I've gone through the interviews with a very young mid level technical person and they are in 100% agreement with my assessment of the candidates, even before I voice concerns. What still holds true is that the more paper certifications you have the less you actually know about actual testing. Of course, those coming from consulting have the standard impress-the-client-certs, so they don't count.

It would be more interesting for me to hear about other peoples experiences in trying to hire web app and penetration testers over the course of the last few years. Maybe I'm just a jerk.

dre said...

@ Anonymous : Regarding your statement "one wasn't asking for senior salary", let's put numbers to that. What zip code is the organization in? In SF or NYC, e.g., a "senior" penetration tester or appsec architect, including appsec-focused developers, should be making 185K base with over 20K bonus and 25K in paid-time off (or other figures to bring total compensation to that level). Entry-level in those fields for SF/NYC would be 135K base, 5K bonus, and 20K paid-time off (or adding up for total comp). Even entry-level blue teamers should be making 90K in those metro areas. Go to places like Seattle (or the Beltway and Chicago areas) and the numbers aren't far off, maybe 10K less on base ; Atlanta or Houston would be 20K less than SF/NYC on base. Louisville would be 30K less the SF/NYC base, but it doesn't get much lower anywhere in the US.

I know that people in our industry do not know what they are worth. I know that HR "advocates" and hiring managers get bonuses based on how well they negotiate down starting salaries. The nastiest of the HR recruiters and advocates will not even allow a candidate's total compensation to go above 30-40 percent of their current (or most-recent) salary -- and they absolutely will not move forward until they get IRS tax records tied to a background check for the previous decade.

Hey HR teams, I have a message for you: Organized Crime makes twice the money that funds all militaries the world over. Since we penetration testers and appsec professionals are the only people capable of changing that landscape, I think you should consider upping our total compensation packages. We need executive-level benefits including VP or C-level healthcare packages, more paid-time off than other technology workers, pensions (even when not appropriate), and 6 or even 12-month paid severance. How's that for egotistical? Welcome to the 21st century.

The barrier to entry should be a Security+ certification, not a CISSP. "If you have a Security+, we will hire you for those entry-level base salaries quoted above". That's what you should be saying. Then you should pay for Offensive-Security labs and every local conference as well as conferences in nearby cities and states. Career conversations towards OSCE and/or CISSP can happen gradually over time. Hiring managers must stay connected to these Security+ entry-levelers after they leave the company -- at least until the career professional attain OSCE or CISSP (e.g., after they put 5 years in anywhere, not just at one single company). Make promises and keep commitments.

Anonymous said...

I think that finding the right security people is hard. Good technical skills are important, but if the person can't get along with the users or his team, for whatever reasons, then he's worthless.

In my experience, there's a lot of ego in the IT field (and Info Security, in particular). Trying to find someone who realizes there's always something to learn, and who is willing to share his knowledge is really tough.

sho luv said...

Seems the first step is to quantify what it means to be junior, mid level, and senior. This isn't all about technical skills I think. Soft skills should be considered. I feel like identifying these issues is sometimes the easy part.

dre said...

The last two commenters are right: it's not all about expertise, but also professionalism and leadership qualities. However, actual studies (e.g., HBR) do indicate that technical competency is ultimately much more important than authority-proving -- https://hbr.org/2015/11/leadership-qualities-vs-competence-which-matters-more

In other words, the almost-common-sense logic that leadership behaviors are more valuable than technical skills is instead flatly wrong. However, if the highly-competent candidate cannot build enough credibility within his or her own teams (let alone outside of their team), then the competency can certainly be valued less. If you find someone with a strong combination of expertise and credibility, though, then this person should be your next Cyber Risk VP or Director. I would say for CISOs, this is even more important.

Business skills can be learned or trained by 20-55 percent of the population in nearly every country in the world. Technical skills are among an elite and ever-changing few. That's why recruiters try to hire 50K CISSPs every year when there's only 40K CISSPs in the whole world.

NIST has a program called the National Initiative for Cybersecurity Education (NICE) -- http://csrc.nist.gov/nice/workforce.html -- and produced some great work including the NICE Cybersecurity Workforce Framework. NICE links out to several other projects including the NICCS (DHS) work, which I found to be especially empowering because the Cybersecurity Workforce Development Toolkit embodies a lot of strong principles for workforce planning, hiring, and training -- including sections on behavioral questions for interviewing. NICE also worked to produce the Cybersecurity Competency Model -- http://www.careeronestop.org/competencymodel/competency-models/cybersecurity.aspx -- which elaborates on how different types of competencies fit into a bigger-picture model and subsequent frameworks.

dre said...

For general information, please see this report -- http://burning-glass.com/research/cybersecurity/

Anonymous said...

Stopped at "*their* may be a lack of skilled people". *THERE* !! Yes, skills, tell me more, retard.

CG said...

@anonymous

Fixed. Sorry you couldn't get past the grammar issues and provide something useful to the discussion.

CG