Using WebRipper to Examine Website Content
Using WebRipper to Examine Website Content
A skilled attacker usually begins an attack by performing detailed reconnaissance. Recon is critical to a comprehensive attack as it yields information about the target network, hosts, personnel, publicly accessible assets, and a host of other data.
One of the easiest targets is a website. Websites are, by design, highly accessible and loaded with data. Sometimes the data is hidden in fields, behind various tags, with obfuscated URLs, etc. Attackers want to see all of that data without tipping off the target that an attack may be imminent.
As a result, an initial step of an attacker is usually to copy an entire website to their local computer. In fact, when I teach a class in Ethical Hacking & Countermeasures, this is one of the very first techniques I demonstrate. Students mirror a website to their local computer where they can index the contents, perform exhaustive searches, and analyze both visible and hidden content without continuing to communicate with the target.
One of my favorite tools for this is WebRipper.
I often recommend to system administrators to use actual attacker tools to determine the security and resilience of a connected system. WebRipper is a great and simple tool for website mirroring that enables the administrator to examine the content for undesired or dangerous information.
WebRipper can be downloaded from Calluna Software. I’ll demonstrate it here using the default installation as shown in Figure 1.
Figure 1. WebRipper startup screen.
To test a server, click the Create new targeted rip job button and fill in the blanks. After providing a URL, you have several options screens that allow you to specify:
- Types of content to collect including video, audio, and images
- Whether to follow all URLs listed and also rip those
- How many layers deep to rip
- Keywords to search for, collecting only results with matching keywords
- File minimum and maximum sizes
Once those parameters are specified, you just click Start and it is off and running as shown in Figure 2.
Figure 2. Ripping the first level of www.interfacett.com with WebRipper.
When the job is finished I can click the folder icon to browse the downloaded files as shown in Figure 3. Because this is a local browse, I can use whatever applications I want and can spend as much time as necessary without indicating any kind of attack.
Figure 3. The fruits of the attack.
What Do I Do With This Test?
Once you’ve conducted the test on your own systems you know exactly what an attacker will and will not find with a web crawling tool. This gives you a fairly clear idea of the information that this technique will yield. For example, the crawler often finds hidden URLs and images that are not visible to normal browsing behavior. These may contain confidential information or pointers to internal resources that you want to remove.
So now you know how a basic web-ripping recon works and what information it may divulge. And knowing is half the battle!
Mike Danseglio teaches Security classes at Interface Technical Training. His classes can be attended in Phoenix, Arizona or online from anywhere in the world with RemoteLive.
- CISSP (Certified Information Systems Security Professional)
- Certified Ethical Hacking and Countermeasures v8
- CompTIA Security + Certification Skills
You May Also Like
Mark Jacob, Cisco Instructor, presents an introduction to Cisco Modeling Labs 2.0 or CML2.0, an upgrade to Cisco’s VIRL Personal Edition. Mark demonstrates Terminal Emulator access to console, as well as console access from within the CML2.0 product. Hello, I’m Mark Jacob, a Cisco Instructor and Network Instructor at Interface Technical Training. I’ve been using … Continue reading A Simple Introduction to Cisco CML2
In this SharePoint training video, I want to talk about the Navigation Controls in SharePoint. They tend to fall into two kind of different categories; one with the navigation controls in a typical Collaboration Site such as a Team Site or a Project Site. These are Sites that are based on the Team Site Template … Continue reading Using Navigation Controls in a Collaboration Site in SharePoint
See our class schedule for complete Course Schedule Training. Classes are held in Phoenix, AZ and can be attended online from anywhere in the world with RemoteLive™. Instructor: Rick Trader Video Transcription: One of the things that we might have to do in our corporate network is to take a class of IP addresses and then subnet that into … Continue reading Subnetting a TCP/IP Network using the Magic Box Method