Like This Blog 0

Added by Mike Danseglio September 22, 2014

A skilled attacker usually begins an attack by performing detailed reconnaissance. Recon is critical to a comprehensive attack as it yields information about the target network, hosts, personnel, publicly accessible assets, and a host of other data.

One of the easiest targets is a website. Websites are, by design, highly accessible and loaded with data. Sometimes the data is hidden in fields, behind various tags, with obfuscated URLs, etc. Attackers want to see all of that data without tipping off the target that an attack may be imminent.

As a result, an initial step of an attacker is usually to copy an entire website to their local computer. In fact, when I teach a class in Ethical Hacking & Countermeasures, this is one of the very first techniques I demonstrate. Students mirror a website to their local computer where they can index the contents, perform exhaustive searches, and analyze both visible and hidden content without continuing to communicate with the target.

One of my favorite tools for this is WebRipper.

Using WebRipper

I often recommend to system administrators to use actual attacker tools to determine the security and resilience of a connected system. WebRipper is a great and simple tool for website mirroring that enables the administrator to examine the content for undesired or dangerous information.

WebRipper can be downloaded from Calluna Software. I’ll demonstrate it here using the default installation as shown in Figure 1.

Figure 1. WebRipper startup screen.

To test a server, click the Create new targeted rip job button and fill in the blanks. After providing a URL, you have several options screens that allow you to specify:

• Types of content to collect including video, audio, and images
• Whether to follow all URLs listed and also rip those
• How many layers deep to rip
• Keywords to search for, collecting only results with matching keywords
• File minimum and maximum sizes

Once those parameters are specified, you just click Start and it is off and running as shown in Figure 2.

Figure 2. Ripping the first level of www.interfacett.com with WebRipper.

When the job is finished I can click the folder icon to browse the downloaded files as shown in Figure 3. Because this is a local browse, I can use whatever applications I want and can spend as much time as necessary without indicating any kind of attack.

Figure 3. The fruits of the attack.

Recon complete!

What Do I Do With This Test?

Once you’ve conducted the test on your own systems you know exactly what an attacker will and will not find with a web crawling tool. This gives you a fairly clear idea of the information that this technique will yield. For example, the crawler often finds hidden URLs and images that are not visible to normal browsing behavior. These may contain confidential information or pointers to internal resources that you want to remove.

So now you know how a basic web-ripping recon works and what information it may divulge. And knowing is half the battle!

Stay safe!

Mike Danseglio -CISSP / MCSE / CEH
Interface Technical Training – Technical Director and Instructor

Mike Danseglio teaches Security classes at Interface Technical Training. His classes can be attended in Phoenix, Arizona or online from anywhere in the world with RemoteLive.

• CISSP (Certified Information Systems Security Professional)
• Certified Ethical Hacking and Countermeasures v8
• CompTIA Security + Certification Skills