Grabbing Index Pages Of Webservers

Grabbing the index pages of web servers seems like a no brainer and something every pentester is going to perform on a test. The problem I ran into is how do you get this info once your inside and using meterpreter as your pivot into the network.

Your current options are to port forward to each host or set up a route via your meterpreter session and run some sort of auxiliary module. You can tcp port scan and find open ports or use the http_version module to see server version but you don't get a feel for whats actually on the site.

I opted to write something that would scan a range, perform a HTTP GET of / on the ip, then take the resulting body from the response, which should be html, and save it to a file to look at afterwards.

Looks like this when it runs...

msf auxiliary(http_index_grabber) > set RHOSTS carnal0wnage.com/24
RHOSTS => carnal0wnage.com/24
msf auxiliary(http_index_grabber) > run
[+] Received a HTTP 200...Logging to file: /home/cg/.msf3/logs/auxiliary/http_index_grabber/209.20.85.4_20100904.4426.html
[+] Received a HTTP 200...Logging to file: /home/cg/.msf3/logs/auxiliary/http_index_grabber/209.20.85.5_20100904.4429.html
[*] Received 301 to http://drumsti.cc/ for 209.20.85.10:80/
[-] Received 403 for 209.20.85.8:80/
[+] Received a HTTP 200...Logging to file: /home/cg/.msf3/logs/auxiliary/http_index_grabber/209.20.85.12_20100904.4432.html
...
[*] Received 302 to http://209.20.85.57/apache2-default/ for 209.20.85.57:80/ [+] Received a HTTP 200...Logging to file: /home/cg/.msf3/logs/auxiliary/http_index_grabber/209.20.85.56_20100904.4503.html
[*] Received 302 to http://209.20.85.51/session/new for 209.20.85.51:80/

you can then check out the folder with the results


code is here:
http://carnal0wnage.googlecode.com/svn/trunk/msf3/modules/auxiliary/admin/random/http_index_grabber.rb