Using WebRipper to Examine Website Content
Using WebRipper to Examine Website Content
A skilled attacker usually begins an attack by performing detailed reconnaissance. Recon is critical to a comprehensive attack as it yields information about the target network, hosts, personnel, publicly accessible assets, and a host of other data.
One of the easiest targets is a website. Websites are, by design, highly accessible and loaded with data. Sometimes the data is hidden in fields, behind various tags, with obfuscated URLs, etc. Attackers want to see all of that data without tipping off the target that an attack may be imminent.
As a result, an initial step of an attacker is usually to copy an entire website to their local computer. In fact, when I teach a class in Ethical Hacking & Countermeasures, this is one of the very first techniques I demonstrate. Students mirror a website to their local computer where they can index the contents, perform exhaustive searches, and analyze both visible and hidden content without continuing to communicate with the target.
One of my favorite tools for this is WebRipper.
Using WebRipper
I often recommend to system administrators to use actual attacker tools to determine the security and resilience of a connected system. WebRipper is a great and simple tool for website mirroring that enables the administrator to examine the content for undesired or dangerous information.
WebRipper can be downloaded from Calluna Software. I’ll demonstrate it here using the default installation as shown in Figure 1.
Figure 1. WebRipper startup screen.
To test a server, click the Create new targeted rip job button and fill in the blanks. After providing a URL, you have several options screens that allow you to specify:
- Types of content to collect including video, audio, and images
- Whether to follow all URLs listed and also rip those
- How many layers deep to rip
- Keywords to search for, collecting only results with matching keywords
- File minimum and maximum sizes
Once those parameters are specified, you just click Start and it is off and running as shown in Figure 2.
Figure 2. Ripping the first level of www.interfacett.com with WebRipper.
When the job is finished I can click the folder icon to browse the downloaded files as shown in Figure 3. Because this is a local browse, I can use whatever applications I want and can spend as much time as necessary without indicating any kind of attack.
Figure 3. The fruits of the attack.
Recon complete!
What Do I Do With This Test?
Once you’ve conducted the test on your own systems you know exactly what an attacker will and will not find with a web crawling tool. This gives you a fairly clear idea of the information that this technique will yield. For example, the crawler often finds hidden URLs and images that are not visible to normal browsing behavior. These may contain confidential information or pointers to internal resources that you want to remove.
So now you know how a basic web-ripping recon works and what information it may divulge. And knowing is half the battle!
Stay safe!
Mike Danseglio -CISSP / MCSE / CEH
Interface Technical Training – Technical Director and Instructor
Mike Danseglio teaches Security classes at Interface Technical Training. His classes can be attended in Phoenix, Arizona or online from anywhere in the world with RemoteLive.
- CISSP (Certified Information Systems Security Professional)
- Certified Ethical Hacking and Countermeasures v8
- CompTIA Security + Certification Skills
You May Also Like
CISSP, ECH, Ethical Hacking & Countermeasures, IT Security, obfuscated URLs, Recon, WebRipper
A Simple Introduction to Cisco CML2
0 3774 0Mark Jacob, Cisco Instructor, presents an introduction to Cisco Modeling Labs 2.0 or CML2.0, an upgrade to Cisco’s VIRL Personal Edition. Mark demonstrates Terminal Emulator access to console, as well as console access from within the CML2.0 product. Hello, I’m Mark Jacob, a Cisco Instructor and Network Instructor at Interface Technical Training. I’ve been using … Continue reading A Simple Introduction to Cisco CML2
Cable Testers and How to Use them in Network Environments
0 692 1This content is from our CompTIA Network + Video Certification Training Course. Start training today! In this video, CompTIA Network + instructor Rick Trader demonstrates how to use cable testers in network environments. Let’s look at some tools that we can use to test our different cables in our environment. Cable Testers Properly Wired Connectivity … Continue reading Cable Testers and How to Use them in Network Environments
Configuring Windows Mobility Center and How to Turn it On and Off
1 1433 1Video transcription Steve Fullmer: In our Windows training courses, we often share information about the Windows 8.1 Mobility Center. Mobility Center was introduced for mobile and laptop devices in Windows 7. It’s present and somewhat enhanced in Windows 8. Since we don’t have mobile devices in our classrooms, I decided to take a little bit … Continue reading Configuring Windows Mobility Center and How to Turn it On and Off