So the latest article by Brian Chess didnt stir up quite the controversy that that his pentesting dead in 2009 interview/article but this one is worth a read:
http://securitysa.com/news.aspx?pklNewsId=31945
Its a short article and not near as controversial as the dead in 2009 one but three quotes...
"People are now spending more money on getting code right in the first place than they are on proving it is wrong. However, this does not signal the end of the road for penetration testing, nor should it, but it does change things. Rather than being a standalone product, it is going to be more like a product feature. Penetration testing is going to cease being an end unto itself and re-emerge as part of a more comprehensive security solution."
"2009 will be the year this strategy comes together, and when we look back, it will be the year when most of the world began thinking about penetration testing as part of a larger offering."
All that is good news (I think), secure coding is where things need to go but I personally dont feel any amount of secure code will ever completely replace pentesting as long as its possible to mis-configure it or set it up insecurely. So Microsoft Windows at some point may be free of stack overflows (or any memory corruption exploits) but that wont stop some system admin setting up their domain in some insecure fashion. That will still need to be pentested to discover and help remediate. Which leads me to the last quote...
"More than ever before, people understand the software security challenge, and penetration testing deserves credit for helping spread the word. But knowing a security problem exists is not the same as knowing how to fix it. In other words, penetration testing is good for finding the problem but does not help in finding the solution – and that is why it must take a long hard look at itself and then make a change. Just like the venerable spell-checker, it is going to die and come back in a less distinct but more pervasive form and I, for one, cannot wait."
I dont agree with this. Penetration testing/testers should never leave you without a fix to security issues. I know alot of pentesters and I dont know any that dont give the customer recommendations for remediations and a customer shouldn't accept a pentest that doesnt have recommended fixes. I suspect that what Chess meant here were "problems" like SQL injection vulns or code bugs that a source code scanning tool could help find and recommend the secure way to code it where a pentester may say "recode it", "have your developers find and fix the code" or "you may have improper parameter checking in this public function", etc.
I do agree that pentesting should evolve, but I think it should begin to look more at assessing an organization from many angles and taking the path of least resistance than pentesting the network side one quarter, the web app side the next, physical security the next, etc. When we begin to identify what makes us money, then look at how we are protecting it across the enterprise, then testing all those defenses at the same time, then we are evolving in the right direction. The evolution should be Full Scope pentesting and not the way most shops do it now.
Anyone else have thoughts on the article?
@ Chris:
ReplyDeleteBut knowing a security problem exists is not the same as knowing how to fix it. In other words, penetration testing is good for finding the problem but does not help in finding the solution
Brian Chess means "software weaknesses; not vulnerabilities". Fixing vulnerabilities is nonsense. Finding the root-cause of vulnerabilities and you'll find a software weakness. Now fix that!
I don't know of very many pen-test shops that advocate going into the software weakness layer to solve problems... including "full scope" shops.
As far as the "Evolution of Pen-Testing" is concerned: my major problem with pen-testing is that repeatable exercises are almost always left-out over time. It's difficult to take the findings and criteria that went into last year's pen-test and use it for the current year's one.
In the case of software security using an assessment factory or secure SDLC program -- metrics can be gathered in various places and used to trend. Even in the case of breach data this is true.
Pen-test must ALLOW itself to evolve, but right now it's stuck with crawling. You guys haven't even made it to baby-steps yet -- while some software security programs are actively walking around. Certainly, software security is relatively new in comparison to pen-testing, right?
I think pen testing is bordering on a major evolution in focus. Almost everyone is focused on two flawed concepts. The first is that pen testing can be cleanly divided into discrete services (web pen test, external, internal, phishing, exploit dev, etc). The second flawed concept is that if you pass any one of these services you aren't secure and even if you test them all individually you're still missing the problem.
ReplyDeleteBanks are the worst for testing everything individually and then declaring the whole secure. I've assessed numerous systems where components X and Y pass security testing individually. Of course the purpose of X is to talk to Y and they weren't tested together... if they were tested together they'd fail the audit, so of course the test isn't like that.
The same is true for externals/internals/phishing. It usually goes as follows.
1. Customer: "We know we're vulnerable to phishing we accept that so we're not going to do the testing."
2. External pen test: nothing is open except web and well patched MS services.
3. Internal is wide open. Own the domain in under a day.
4. Customer: "Great job. Is there any way you can tone down the internal findings? There's no way a real attacker could do that... and we trust our employees".
5. Six months or later, Customer: "We had a major security incident from phishing. There appears to be a hacker in our network"
I just don't think that trend is going to continue indefinitely. I believe it's going to continue to get worse. After it gets sufficiently bad, customers will implement more effective security measures. This is similar to organizations taking perimeter security seriously only after they were owned repeatedly by worms. Phishing and bots will get sufficiently bad and we'll hit that point before long.
I agree with the idea that pen testing has to evolve or die. I teach this, and live doing this as well, and there are a lot of folks out there who call themselves pen testers, but well, you know how this is going to end.
ReplyDeleteThere are many changes that the information security industry has to commit to, we need to evolve, adapt or die as an industry. It is not just one vertical in the industry, but as long as we say no, as long as we do not provide viable options and solutions, as long as we continue to work in our own private cube farms, we will miss the boat.
This is not a throw beer and pizza over the wall and let the security engineer do their thing. We need to become tightly integrated with the organization, including pen testing, and be seen as helpful rather than a stumbling block.
my 2 cents, but nice article along the way.
r/dan
"Tattle-tale reports aren't helping things"
ReplyDeleteI agree with this sentiment towards pentesting. The bug parade seems like it is never going to let up and it's way past the time to start addressing this through secure coding/design.
But you still are going to want that pen test to make sure that your secure coding/design program is WORKING and producing output that is actually secure.
There's a long way to go before pen testing will be obsolete (and so much left to test still!)
for the love of god, will people please define whether or not they are talking about secure coding practices or pentesting. They are different. Sure there are areas where they map together but all the SDLC programs in the world will not prevent bad network design or a user entering credentials into a fake site (no exploit required). These are things that pentesting addresses.
ReplyDeleteAnd yes, we all know that the final deliverable needs to have actionable items and provide some means for the organization to track their progress. The responsibility lies with the organization to implement these recommendations, with outside help if necessary. It may take more than a few years until it's at a point where metrics that came out of the original pentest actually start being useful or even understood too.
Good security is a process and is developed over time. Software or infrastructure. I agree though that most pentesters fail at education and providing any real actionable items for a client in general. My personal opinion on this is that it stems from the fact that folks are going straight into 'security' without any real background in systems, networking and infrastructure design today. This means we have people performing sub par pentests that generate no value for the client because the pentester does not understand the needs and challenges of the client or the vertical the client operates in. I've been in environments where I've seen things I thought I never seen in an enterprise but the reality is that everything is assigned a certain level of risk, be it qualitative or quantitative, and that risk is either accepted or mitigated. It's part of the responsibility of a pentester to assess the security of the client in relation to this risk and if new avenues of attack are found assess them according to the risks the client is willing to accept.
I question the comment of a the client who refused the phishing portion of a pentest because they knew they were susceptible. I fail to see the issue with that. If the client's I.T. team knows the risks associated with a user compromise and knows whether or not it will lead to critical data being stolen then why should they have a pentest to prove what they already know? The responsibility is now with the client to attempt to mitigate the risk associated with that threat. If they are compromised via that avenue of attack how would a pentest have prevented that?
Either way I think there is room and a necessity for both. Software security looks at individual applications and pentesting addresses the infrastructure. That's oversimplified I know.
Watts Humphrey of Carnegie Mellon's Software Engineering Institute has some interesting research on software defects. According to Humphrey, even the best developed software has one thousand defects per million lines of code. That's a single error per 30 printed pages.
ReplyDeleteGiven that even the best shops are still delivering defective software, pen testing should be a component of healthy info sec programs for a long time to come.
Quality SDLC initiatives include application specific pen testing along with security reviews during requirements gathering, architecture and design planning, implementation and deployment.
In my experience, pen testing still uncovers flaws that were missed during earlier phases of the SDLC. For that matter, repeat pen tests uncover items that were missed previously.
To think that we can completely solve the vulnerable software issue through improving code quality overestimates human abilities. Even DJBDNS recently had a vulnerability.
And like you said Chris, I've never seen a pen test that didn't offer information on remediating discovered vulnerabilities. I wonder if Chess has ever seen the results of a professional pen test.
Penetration Testing of course is not going to die because of secure coding. And when we are saying secure coding of course we mean what we know now as attacking vectors in software. Buffer overflows or Format Strings. New class of bugs might come soon if hackers and researchers try to be more innovating in their research methods. New technologies are coming, of course more complicated so you have to expect new things. Actually something tells me that new things are already out but most of us are still stucked with overflows...
ReplyDelete