Friday, April 15, 2011

Data Driven Pentests...Don't You mean Vulnerability Assessments?

So first a disclaimer, i didnt listen to the referenced podcast, this is based solely of this blog post:

So I’m listening to the “Larry, Larry, Larry” episode of the Risk Hose podcast, and Alex is talking about data-driven pen tests. I want to posit that pen tests are already empirical. Pen testers know what techniques work for them, and start with those techniques.

What we could use are data-driven pen test reports. “We tried X, which works in 78% of attempts, and it failed.”

We could also use more shared data about what tests tend to work.


Dre's response to the post was surprising to me, he listed a bunch of tools that seem to do correlating of pentest results into a portal so you can trend over time. Cool idea, i'll give the people that. But to me when we start jumping into repeatable metrics driven stuff we are in Vulnerability Assessment land, not pentesting land.

Here is the comment I left:

I like the idea and i think it could be useful.

However, they need to drop the pentest part. you are solidly into the vulnerability assessment part of things when you are talking about “ok, i tried 1,2,3,4,5 and 1 & 3 worked” ok on to the next set of tests… thats vulnerability assessment (with exploitation if you want to get technical) and not pentesting.

pentesting is about that human looking at the problem and figuring out how to break it, not some scanner, thats going to be very hard to standardize and put hard numbers on and i dont think its going to be possible without tying up your tester’s time with bullshit.

I'm all for "repeatable" pentests. You should have a methodology for each type of test, but when you are paying for human's time you should be paying for them to go after the site like a human would and not how a scanner would or not in a way where i'm worried about religiously following some checklist because if i don't the metrics get all fucked up. Your pentest should come after you have thrown the kitchen sink at it scanner wise.

as an added bonus this post was right below the new school post in my Google reader:

This post and really any methodology document you will ever read or write will have gaps, because no document on this subject can ever really be 100% all inclusive of every vulnerability and the myriad of variations that exist for many of these.

I think it drives the point home as well.



dre said...

Yeah they all do pretty much the same thing. Call it vulnerability analytics. Call it pentest management. Call it appsec risk management. I'm not sure what any of these words mean anymore, but I know that I want access to these tools and I want to upload my results from testing things to them so that I can understand what I've done in the past.

CG said...

agree...the idea is solid, for repeat clients seems pretty much like a must do. time for an open source version ;-)

Jhaddix said...

oh you two and your quarrels ;)

I really like SpiderLabs flash pentest manager, its drag and drop, graphs trends, and can upload video for findings.

Time to get some underpaid grad student to code it! Where's Joe's interns?

CG said...

hey! we actually agree on this one i think