Data Driven Pentests...Don't You mean Vulnerability Assessments?

So first a disclaimer, i didnt listen to the referenced podcast, this is based solely of this blog post:

http://newschoolsecurity.com/2011/04/data-driven-pen-tests

So I’m listening to the “Larry, Larry, Larry” episode of the Risk Hose podcast, and Alex is talking about data-driven pen tests. I want to posit that pen tests are already empirical. Pen testers know what techniques work for them, and start with those techniques.

What we could use are data-driven pen test reports. “We tried X, which works in 78% of attempts, and it failed.”

We could also use more shared data about what tests tend to work.

Thoughts?

Dre's response to the post was surprising to me, he listed a bunch of tools that seem to do correlating of pentest results into a portal so you can trend over time. Cool idea, i'll give the people that. But to me when we start jumping into repeatable metrics driven stuff we are in Vulnerability Assessment land, not pentesting land.

Here is the comment I left:

I like the idea and i think it could be useful.

However, they need to drop the pentest part. you are solidly into the vulnerability assessment part of things when you are talking about “ok, i tried 1,2,3,4,5 and 1 & 3 worked” ok on to the next set of tests… thats vulnerability assessment (with exploitation if you want to get technical) and not pentesting.

pentesting is about that human looking at the problem and figuring out how to break it, not some scanner, thats going to be very hard to standardize and put hard numbers on and i dont think its going to be possible without tying up your tester’s time with bullshit.

I'm all for "repeatable" pentests. You should have a methodology for each type of test, but when you are paying for human's time you should be paying for them to go after the site like a human would and not how a scanner would or not in a way where i'm worried about religiously following some checklist because if i don't the metrics get all fucked up. Your pentest should come after you have thrown the kitchen sink at it scanner wise.

as an added bonus this post was right below the new school post in my Google reader:

http://coding-insecurity.blogspot.com/2011/04/developing-good-methodology-part-3.html

This post and really any methodology document you will ever read or write will have gaps, because no document on this subject can ever really be 100% all inclusive of every vulnerability and the myriad of variations that exist for many of these.

I think it drives the point home as well.

-CG