Using WebRipper to Examine Website Content
Using WebRipper to Examine Website Content
A skilled attacker usually begins an attack by performing detailed reconnaissance. Recon is critical to a comprehensive attack as it yields information about the target network, hosts, personnel, publicly accessible assets, and a host of other data.
One of the easiest targets is a website. Websites are, by design, highly accessible and loaded with data. Sometimes the data is hidden in fields, behind various tags, with obfuscated URLs, etc. Attackers want to see all of that data without tipping off the target that an attack may be imminent.
As a result, an initial step of an attacker is usually to copy an entire website to their local computer. In fact, when I teach a class in Ethical Hacking & Countermeasures, this is one of the very first techniques I demonstrate. Students mirror a website to their local computer where they can index the contents, perform exhaustive searches, and analyze both visible and hidden content without continuing to communicate with the target.
One of my favorite tools for this is WebRipper.
I often recommend to system administrators to use actual attacker tools to determine the security and resilience of a connected system. WebRipper is a great and simple tool for website mirroring that enables the administrator to examine the content for undesired or dangerous information.
WebRipper can be downloaded from Calluna Software. I’ll demonstrate it here using the default installation as shown in Figure 1.
Figure 1. WebRipper startup screen.
To test a server, click the Create new targeted rip job button and fill in the blanks. After providing a URL, you have several options screens that allow you to specify:
- Types of content to collect including video, audio, and images
- Whether to follow all URLs listed and also rip those
- How many layers deep to rip
- Keywords to search for, collecting only results with matching keywords
- File minimum and maximum sizes
Once those parameters are specified, you just click Start and it is off and running as shown in Figure 2.
Figure 2. Ripping the first level of www.interfacett.com with WebRipper.
When the job is finished I can click the folder icon to browse the downloaded files as shown in Figure 3. Because this is a local browse, I can use whatever applications I want and can spend as much time as necessary without indicating any kind of attack.
Figure 3. The fruits of the attack.
What Do I Do With This Test?
Once you’ve conducted the test on your own systems you know exactly what an attacker will and will not find with a web crawling tool. This gives you a fairly clear idea of the information that this technique will yield. For example, the crawler often finds hidden URLs and images that are not visible to normal browsing behavior. These may contain confidential information or pointers to internal resources that you want to remove.
So now you know how a basic web-ripping recon works and what information it may divulge. And knowing is half the battle!
Mike Danseglio teaches Security classes at Interface Technical Training. His classes can be attended in Phoenix, Arizona or online from anywhere in the world with RemoteLive.
- CISSP (Certified Information Systems Security Professional)
- Certified Ethical Hacking and Countermeasures v8
- CompTIA Security + Certification Skills
You May Also Like
Mark Jacob, Cisco Instructor, presents an introduction to Cisco Modeling Labs 2.0 or CML2.0, an upgrade to Cisco’s VIRL Personal Edition. Mark demonstrates Terminal Emulator access to console, as well as console access from within the CML2.0 product. Hello, I’m Mark Jacob, a Cisco Instructor and Network Instructor at Interface Technical Training. I’ve been using … Continue reading A Simple Introduction to Cisco CML2
This content is from our CompTIA Network + Video Certification Training Course. Start training today! In this video, CompTIA Network + instructor Rick Trader demonstrates how to use cable testers in network environments. Let’s look at some tools that we can use to test our different cables in our environment. Cable Testers Properly Wired Connectivity … Continue reading Cable Testers and How to Use them in Network Environments
In this video, Security Instructor Mike Danseglio demonstrates how to use BitLocker in Window 10 to secure files on a USB Flash drive that adhere to stricter data protection requirements as found inside Government entities. BitLocker 2-day instructor-led training is now available at Interface: BITLOCK: Planning and Deploying BitLocker Drive Encryption Training Video Transcription: Hi. … Continue reading Government Edition – Encrypting a USB Flash Drive in Windows 10