I'm not the most articulate person, especially in writing, and while I thought by the people that bothered to comment on the blog, I got my point across other people make me think I didn't.
So, I'll try again, if this doesn't work I'll resign myself to just not being as l33t and skilled as other people in the community.
I'm not against ALL automation, by its very nature everything involved with "hacking" or penetration testing is automated but I'll try to restate. From the orginal post:
Automation is a good thing but when it comes to pentesting I really think there can be too much automation. Too much automation leads to "fire and forget" activities and lack of TLC.
Believe me, I'm all for bash running my nmap scan and rolling IPs for me all nite rather than stay there and do the stuff myself. I'm all for automating a tool that needs to run a set of commands on a subnet or group of hosts given the appropriate scenario. Hell I'm even all for running scripts that will log into every box and "do something" (go tebo!) also given the appropriate scenario.
But what I'm not for (but I do concede there are plenty of times when the following is perfectly acceptable and maybe regarded as "good" pentest):
Rolling in and run my full port nmap scan, nessus scan, core impact rapid pent test, connect to a couple of agents, take screenshots, high five with my team mates on how l33t I am, then head out and write the report.
I'm of the opinion that for the above scenario, a couple weeks of training and a couple of licenses later any in-house shop can do that for themselves.
I will propose that there is an alternate type of pentest where my goal is not to root as many boxes and I can but its too see if I can gain access to "what makes the company money". I am unannounced, trying to not get caught is part of the test, and where the customer actually has a great patching program, an IDS, possibly and IPS, an outbound proxy, somebody actively monitoring the previous technologies, and asking them to turn all that shit off is not going to happen....
how would our above scenario fair for that pentest?
how fast would your pentest be over if you did say....
"For example, I enumerate open shares on the entire subnet, then pull down all .doc documents, then search them for interesting information from the recon phase (i.e. the name of the CFO)."
I'd give it about 2 minutes before any SOC that doesn't totally suck blocks your ass and you have to go begging for them to unblock your IP...that's never any fun.
I wont address everything from the pauldotcom show notes, because frankly I think they completely missed the point of the post (because I am not against all automation), but I will address this.
"[PaulDotCom] - While I agree, automation can have a negative effect on risk identification,
its a vital part of every penetration test. Much of the post talked about how automated tools "Don't have the love" and "you need TLC". That's all well and good, but how do you show risk when you've got a meterpreter shell on 30 hosts? What are those hosts? Do you spend 3 weeks of your time and the customer's money to demonstrate risk? No, you automate the mundane tasks and pieces and parts that can be automated. "
I guess my response would be why do I need 30 shells to demonstrate a point? how would I have gotten those shells without being really sure about what/where the host was before I launched anything that would have gotten me a shell? but if ./autohack ran on a subnet and gave me 30 shells then I guess I might find myself wondering what are all those hosts and how did I get in. As for risk identification, if I got 30 shells the risk identification is an enterprise patching problem in this scenario.
"This does not make you any less skilled or a script kiddie, in fact, it makes you more of a master."
umm how? did I have to run a nessus with credentials to find those vulnerable hosts after checking 100's of KB's and throwing tons of traffic at the hosts or did core impact throw 20 exploits at each box before I got lucky and popped a few? Or did I limit my exposure and try to enumerate what services were running and what service pack the host was and try the latest exploit at one host to see what would happen?
I'll spare you the rest, like I said I think the point of the post was completely missed.
Amen brother.. he totally missed the point.
ReplyDeletePaul Assadorian lags behind the point in most of his podcasts and posts usually. I would be surprised if he had a moment of clarity. What does surprise me is his reading your blog. I thought only intelligent people did that.
ReplyDeletesome people are slow; news at 11.
ReplyDeleteEvading active monitoring and IDS aside, TLC is necessary with all blackbox testing.
Web app testing is just one area for example, where manual testing and TLC is required; even more so if your testing is restricted to the DMZ.
Web servers are a breeding ground for bugs, with old systems often being patched and hacked together with flawed, handwritten code. As such, successful fuzzing results can vary and could even DoS the server in the case of hilariously bad code.
Not to mention the web server may connect to servers outside the DMZ, such as corp intranet, mail, DNS or databases.
Accidentally crashing a critical server, while it does demonstrate a flaw, causing loss of business is probably not something you want to justify to the client.
More to the point, if our tertiary education system continues to push automation in all areas of IT and produce certified people who have no real experience then IT security is going to continue up shit creek.
I'm a little surprised some of the big names in IT security haven't lost faith and gone postal. You might be making lots of money and donating to charity, but you can't bang your head against the wall until you wake up before "web 2.0", and I bet that keeps you up at night - just sayin'
Automation is the result of commercial pentesting. When you need to test 10 clients in 2 days, all that left is to automate. At least the majority of security companies do such thing in Baltic states, where I live. Just to get more cash
ReplyDeleteI would rather hire script kiddies to perform my yearly PCI pentest than CG. Which option would make me look better and probably be less expensive? ;)
ReplyDeleteFrom a risk management perspective you're on the money.
PDC beef aside.. the point was missed..
ReplyDeleteA serious test is approached with tactics, stealth, and intelligent target selection.
As CG mentioned, your customer can probably buy all the tools you run and run them themselves, they bring you in for your expertise and experience, aka the intangible TLC you add to the process of testing.
There really are a lot of minor leaguers out there, which is why CG and the rest of the all-stars aren't worried about losing any business to them :]
CG, you had it right the first time...F'em if they "don't get it" Compliance/Yearly Pentest...great I got it. Plenty of CISA's to hook that up for you, but I'm pretty sure the real baddies you need to worry about aren't running ./autohack (fun as it is)
ReplyDeleteSorry I've got to cut this comment short, need to go post a nonsense retort on my own blog instead of posting constructive, intelligent criticisms here:| That will make me l33t, Right?
Hey CG,
ReplyDeleteI didn't miss the point, I just took the opposite opinion :)
"Too much automation leads to "fire and forget" activities and lack of TLC."
I totally agree.
"I will propose that there is an alternate type of pentest where my goal is not to root as many boxes and I can but its too see if I can gain access to "what makes the company money"."
This was part of my point as well, and should be the goal of every pen test. How you get there may differ, and working with your customer on what that looks like is important. It can go either way, automate too much or focus on wicked cool super-sexy techniques and you can miss the purpose of the test. I guess my point, which I clarify on the podcast, is that each test is slightly different and you need to work with the customer to find the right balance and level/depth of testing.
"I'll spare you the rest, like I said I think the point of the post was completely missed."
I apologize for that, I think its fun to debate. You are welcome to come on the show anytime to clarify your point, we'd love to have you on as a guest.
Cheers,
Paul
Hello Paul and CG – this looks like a good debate and I can’t resist jumping in. [Full Disclosure: I work for Core Security, makers of Core Impact]
ReplyDeleteI like this debate because I think a lot of people are unclear about what constitutes automation. Some people believe that if you use a framework that can generate an exploit for a target and package-in a payload, then you are automating the process -- in that case, any Penetration Testing framework is automated. Others think that products like Core Impact Pro only run in an automated fashion and offer no way for the user to apply their own experience to how the product performs the test.
I personally believe that automation is when you let the framework make decisions about what attacks to run against a target with little or no guidance from the tester. That’s fine in some cases, such as when your test is internal to a network with nothing monitoring your traffic – in that scenario, a rapid fire test by clicking a wizard in Impact Pro would work. However, in a lot of tests, you have to walk softly and carry a big (pointed) stick. Here the experience and training of the tester is just as important as the quality of the framework.
And that to me is the key point of this debate: While Core has created a product that is very powerful and (I believe) can take any user a long way down the path of a penetration test, we still expect the person performing the test to apply their experience and knowledge of working in this environment or working with the limitations and/or goals that are part of the current scenario. Only in this way can the product perform the test that best matches the user’s (or their client’s) needs. I believe it is OK for people who perform penetration tests to use a commercial tool (with the potential for automation) provided they are not just clicking wizards, but are rather using the product as part of their overall testing methodology.
Alex, i think you are exactly right!
ReplyDeletePaul after listening to the podcast you are right, we do agree more than we disagree. My point is that some situations require a bit more finesse and as organization's security programs mature some penetration testing will move more towards black box testing and away from the whitebox, asset identification, use nessus to id all my vulnerabilities type testing.
Things change drastically when you approach a test of exploiting one flaw to gain as much access as I can or obtaining the "thing that makes that company money" without getting caught versus identifying all the boxes missing patches, exploit those for proof, and passing that off for mitigation.
An important point that I think your original post touched on but largely missed and may have lead to some of the confusion was the point about stealth. You're right in that running autohack scripts and large automata will likely give you away, and quite quickly. In a true Penetration Test you likely need to be stealthy (see my blog post linked below for a rant on engagement terminology), but if your initial reconnaissance phase indicates that there's likely nothing to catch you should you make a ruckus on the network, there's nothing wrong with applying some automata to your processes. If you're not providing a PenTest however but instead an Assessment or an Audit (again, see the rant), automation is likely your only path to success in a reasonable amount of time, but since you focused solely on Penetration Tests I believe your original point is completely valid.
ReplyDeletehttp://dtrammell.wordpress.com/2008/08/19/penetration-test-audit-assessment/
Unfortunately, I have been swamped lately and haven't had time to listen to PDC for a few weeks so I've totally missed the debate between two individuals that I admire and respect.
ReplyDeleteIn my own experience as a web application pen tester, I can attest that there are entire classes of vulnerabilities that are overlooked by the popular automated web app vuln scanners.
But we continue to use both approaches for our engagements. Automated tools catch the bulk of the issues, while our manual reviews continue to catch critical flaws that are missed entirely by the automated tools.
As for flying under the radar, I haven't had an engagement yet where that was a concern, but I can't imagine using an automated tool with success in those circumstances.
Debate is good. Remember the old proverb: As iron sharpens iron, so one man sharpens another.
Good on ya both.
Don't be too hard on automation. Some service droid-like functionaries can (and should) be replaced by shell scripts, particularly in government agencies and other large bureaucracies.
ReplyDeleteI wanted to post a comment, but my auto_post.py was broke.
ReplyDeleteAs I sit on the lanai (balcony of the houli's in da house) at the Hilton Village, I have to ponder what would happen if the customer did automate me...Damn it!!!
There is a time and place, but in the end it is the art not the science that makes the impact. No script can demonstrate operational impact and how the residual risk can be used against a customer.
It is all tools, in the toolbox.
Ok..time for Mai Tia's...
Miss ya CG!
I think many pentesting activities either start from just looking around from the user/admin perspective or at some point end up being in the "admin" mindset. This cannot be automated. Some pentests cannot even rely on automated or manual exploits. For example legitimate pathways via information disclosure. And really, as much as would like for a bot to go out and search word docs, etc., it cannot forsee all the possible constructs to pase within documents a human eye can easily spot. Years of experience "admining" or "coding" translate into "hunch" which is not automatable in many cases. Plus, automation fails to span technologies involved in penetration logic. I have yet to see a tool that autohacks a web application, downloads a tar file with a year -old configuration, recovers passwords for FTP, logs in, realizes that a backup CVS is available, parses the startup code in /etc/init.d and inserts code to give you root upon restart of the box.
ReplyDelete