Using WebRipper to Examine Website Content

Home > Blogs > Security > Using WebRipper to Examine Website Content

Using WebRipper to Examine Website Content

Like This Blog 0 Mike Danseglio
Added by September 22, 2014

A skilled attacker usually begins an attack by performing detailed reconnaissance. Recon is critical to a comprehensive attack as it yields information about the target network, hosts, personnel, publicly accessible assets, and a host of other data.

One of the easiest targets is a website. Websites are, by design, highly accessible and loaded with data. Sometimes the data is hidden in fields, behind various tags, with obfuscated URLs, etc. Attackers want to see all of that data without tipping off the target that an attack may be imminent.

As a result, an initial step of an attacker is usually to copy an entire website to their local computer. In fact, when I teach a class in Ethical Hacking & Countermeasures, this is one of the very first techniques I demonstrate. Students mirror a website to their local computer where they can index the contents, perform exhaustive searches, and analyze both visible and hidden content without continuing to communicate with the target.

One of my favorite tools for this is WebRipper.

Using WebRipper

I often recommend to system administrators to use actual attacker tools to determine the security and resilience of a connected system. WebRipper is a great and simple tool for website mirroring that enables the administrator to examine the content for undesired or dangerous information.

WebRipper can be downloaded from Calluna Software. I’ll demonstrate it here using the default installation as shown in Figure 1.

001-start-screen-WebRipper-to-Examine-Website-Content

Figure 1. WebRipper startup screen.

To test a server, click the Create new targeted rip job button and fill in the blanks. After providing a URL, you have several options screens that allow you to specify:

  • Types of content to collect including video, audio, and images
  • Whether to follow all URLs listed and also rip those
  • How many layers deep to rip
  • Keywords to search for, collecting only results with matching keywords
  • File minimum and maximum sizes

Once those parameters are specified, you just click Start and it is off and running as shown in Figure 2.

002-first-level-screen-WebRipper-to-Examine-Website-Content

Figure 2. Ripping the first level of www.interfacett.com with WebRipper.

When the job is finished I can click the folder icon to browse the downloaded files as shown in Figure 3. Because this is a local browse, I can use whatever applications I want and can spend as much time as necessary without indicating any kind of attack.

003-first-level-screen-WebRipper-to-Examine-Website-Content

Figure 3. The fruits of the attack.

Recon complete!

What Do I Do With This Test?

Once you’ve conducted the test on your own systems you know exactly what an attacker will and will not find with a web crawling tool. This gives you a fairly clear idea of the information that this technique will yield. For example, the crawler often finds hidden URLs and images that are not visible to normal browsing behavior. These may contain confidential information or pointers to internal resources that you want to remove.

So now you know how a basic web-ripping recon works and what information it may divulge. And knowing is half the battle!

Stay safe!

Mike Danseglio -CISSP / MCSE / CEH
Interface Technical Training – Technical Director and Instructor

Mike Danseglio teaches Security classes at Interface Technical Training. His classes can be attended in Phoenix, Arizona or online from anywhere in the world with RemoteLive.

 

Videos You May Like

Agile Methodology in Project Management

0 130 0

In this video, you will gain an understanding of Agile and Scrum Master Certification terminologies and concepts to help you make better decisions in your Project Management capabilities. Whether you’re a developer looking to obtain an Agile or Scrum Master Certification, or you’re a Project Manager/Product Owner who is attempting to get your product or … Continue reading Agile Methodology in Project Management

Creating Users and Managing Passwords in Microsoft Office 365

0 570 3

In this Office 365 training video, instructor Spike Xavier demonstrates how to create users and manage passwords in Office 365.   For instructor-led Office 365 training classes, see our course schedulle: Spike Xavier SharePoint Instructor – Interface Technical Training Phoenix, AZ 20347: Enabling and Managing Office 365    

Detailed Forensic Investigation of Malware Infections – April 21, 2015

4 525 3

How does an investigator hunt down and identify unknown malware? In this recording of our IT Security training webinar on April 21, 2015, Security expert Mike Danseglio (CISSP / CEH) performed several malware investigations on infected computers and identify symptoms, find root cause, and follow the leads to determine what’s happening. He demonstrated his preferred … Continue reading Detailed Forensic Investigation of Malware Infections – April 21, 2015

Write a Comment

Share your thoughts...

Please fill out the comment form below to post a reply.