HttpWatch: An HTTP Viewer and HTTP Sniffer for IE and Firefox

Automated Testing with HttpWatch, Ruby And Watir

Introduction

This article demonstrates how HttpWatch can be used to gather HTTP level information from a Ruby/WATIR automated test. A simple web spider program is used to 'walk' around the pages on a web site. The data recorded in HttpWatch is then analyzed and a report is displayed with network statistics and a list of URLs that caused an error.

The software and source code used are available as a free download.

What is HttpWatch?

HttpWatch is a widely used web development tool that shows you HTTP and HTTPS traffic from within IE; allowing you to quickly debug, fix and optimize your website. It displays headers, cookies, status codes, compression, content and more as you browse a web site. You can use it to:

This article uses HttpWatch Basic Edition, which is available as a free download.

What is Ruby?

"Ruby is the interpreted scripting language for quick and easy object-oriented programming. It has many features to process text files and to do system management tasks (as in Perl). It is simple, straight-forward, extensible, and portable."

Yukihiro Matsumoto, The Creator of Ruby

What is WATIR?

"Watir" (pronounced water) stands for "Web Application Testing in Ruby". Watir is an automated test tool which uses the Ruby scripting language to drive the Internet Explorer web browser. Watir is a toolkit for automated tests to be developed and run against a web browser.

Installing the Software

In order to use WATIR you will first need to install Ruby: using the windows installer:

Ruby Windows Installer

and then install WATIR itself:

WATIR Windows Installer

A free version of HttpWatch can be downloaded from our website:

HttpWatch Basic Edition

The source code used in this article is available in this zip file:

Site Spider - Ruby Source Code

Running the Automated Test

To run the test, open a command prompt in the directory where you unzipped the Ruby source files. Then type:

site_spider<Enter>

You will be prompted to enter a domain name (pressing Enter will use www.httpwatch.com). The script will then start IE and navigate to the first page. You should see output like this in the command prompt:

If you use www.httpwatch.com there will be a delay of a minute or two on the httpgallery/errors page. This is due to the intentional HTTP errors that are demonstrated by this page.

Once the test has completed you will see an analysis of the HTTP data gathered by HttpWatch during the test:

In this case, it shows the six errors from the Status Codes & Errors page in our online HTTP samples.

How Does it Work?

The web spider is implemented by creating an instance of the WATIR::IE class and using its goto() method to visit each URL. The WATIR::IE class maintains a copy of Internet Explorer and issues commands to it as required. On each page, the Ruby code examines the links() collection to find URLs on the same site that have not yet been visited. Any new links are added to a list of pending URLs that will be visited.

The spidering process continues until no more links are found or a hard-coded limit on the number of pages is reached. The variable MAX_NO_PAGES controls this limit and is set to 20 by default.

HttpWatch has a COM based automation interface ,with sixteen classes and over a hundred methods, that allows it to be controlled programmatically and provides access to the HTTP data that it records. In this sample, the HttpWatch Plugin object is attached to the instance of Internet Explorer that is being used by Ruby / Watir. The Ruby script is able to directly call its Record() and Stop() methods to control when data is recorded during the test. At the end of the spidering process, the HttpWatch Log and Entry classes are used to compile information about the URLs visited.

This diagram shows the relationship between the Ruby/WATIR, HttpWatch and Internet Explorer:
 

For more detail information about the implementation please have a look at the Ruby source file site_spider.rb in the zip file.

What else can WATIR and HttpWatch do?

This sample uses only a small part of the WATIR framework. It is possible, to interact with HTML controls on a web page, fill our forms, extract text and much more. For more information about WATIR please visit http://wtr.rubyforge.org/.

If you purchase HttpWatch Professional Edition you can gather and analyze data about all aspects of the HTTP and HTTPS traffic generated by an automated test. For example, you could look at cookies, headers, HTTP compression, redirects and caching.

Please let us know if you have any feedback about this article or any questions about HttpWatch.