The Web Robots Pages

Guidelines for Robot Writers

This document contains some suggestions for people who are thinking about developing Web Wanderers (Robots), programs that traverse the Web.

Reconsider

Be Accountable

Identify your Web Wanderer

HTTP supports a User-agent field to identify a WWW browser. As your robot is a kind of WWW browser, use this field to name your robot e.g. "NottinghamRobot/1.0". This will allow server maintainers to set your robot apart from human users using interactive browsers. It is also recommended to run it from a machine registered in the DNS, which will make it easier to recognise, and will indicate to people where you are.

Identify yourself

HTTP supports a From field to identify the user who runs the WWW browser. Use this to advertise your email address e.g. "j.smith@somehwere.edu". This will allow server maintainers to contact you in case of problems, so that you can start a dialogue on better terms than if you were hard to track down.

Announce It

Post a message to comp.infosystems.www.providers before running your robots. If people know in advance they can keep an eye out. I maintain a list of active Web Wanderers, so that people who wonder about access from a certain site can quickly check if it is a known robot -- please help me keep it up-to-date by informing me of any missing ones.

Announce it to the target

If you are only targetting a single site, or a few, contact its administrator and inform him/her.

Be informative

Server maintainers often wonder why their server is hit. If you use the HTTP Referer field you can tell them. This costs no effort on your part, and may be informative.

Be there

Don't set your Web Wanderer going and then go on holiday for a couple of days. If in your absence it does things that upset people you are the only one who can fix it. It is best to remain logged in to the machine that is running your robot, so people can use "finger" and "talk" to contact you

Suspend the robot when you're not there for a number of days (in the weekend), only run it in your presence. Yes, it may be better for the performance of the machine if you run it over night, but that implies you don't think about the performance overhead of other machines. Yes, it will take longer for the robot to run, but this is more an indication that robots are not they way to do things anyway, then an argument for running it continually; after all, what's the rush?

Notify your authorities

It is advisable to tell your system administrator / network provider what you are planning to do. You will be asking a lot of the services they offer, and if something goes wrong they like to hear it from you first, not from external people.

Test Locally

Don't hog resources

Walk, don't run

Make sure your robot runs slowly: although robots can handle hundreds of documents per minute, this puts a large strain on a server, and is guaranteed to infuriate the server maintainer. Instead, put a sleep in, or if you're clever rotate queries between different servers in a round-robin fashion. Retrieving 1 document per minute is a lot better than one per second. One per 5 minutes is better still. Yes, your robot will take longer, but what's the rush, it's only a program.

Use If-modified-since or HEAD where possible

If your application can use the HTTP If-modified-since header, or the HEAD method for its purposes, that gives less overhead than full GETs.

Ask for what you want

HTTP has a Accept field in which a browser (or your robot) can specify the kinds of data it can handle. Use it: if you only analyse text, specify so. This will allow clever servers to not bother sending you data you can't handle and have to throw away anyway. Also, make use of url suffices if they're there.

Ask only for what you want

You can build in some logic yourself: if a link refers to a ".ps", ".zip", ".Z", ".gif" etc, and you only handle text, then don't ask for it. Although they are not the modern way to do things (Accept is), there is an enourmeous installed base out there that uses it (especially FTP sites). Also look out for gateways (e.g. url's starting with finger), News gateways, WAIS gateways etc. And think about other protocols ("news:", "wais:") etc. Don't forget the sub-page references (<A HREF="#abstract">) -- don't retrieve the same page more then once. It's imperative to make a list of places not to visit before you start...

Check URL's

Don't assume the HTML documents you are going to get back are sensible. When scanning for URL be wary of things like <A HREF=" http://somehost.somedom/doc>. A lot of sites don't put the trailing / on urls for directories, a naieve strategy of concatenating the names of sub urls can result in bad names.

Check the results

Check what comes back. If a server refuses a number of documents in a row, check what it is saying. It may be that the server refuses to let you retrieve these things because you're a robot.

Don't Loop or Repeat

Remember all the places you have visited, so you can check that you're not looping. Check to see if the different machine addresses you have are not in fact the same box (e.g. web.nexor.co.uk is the same machine as "hercules.nexor.co.uk" and 128.243.219.1) so you don't have to go through it again. This is imperative.

Run at opportune times

On some systems there are preferred times of access, when the machine is only lightly loaded. If you plan to do many automatic requests from one particular site, check with its administrator(s) when the preferred time of access is.

Don't run it often

How often people find acceptable differs, but I'd say once every two months is probably too often. Also, when you re-run it, make use of your previous data: you know which url's to avoid. Make a list of volatile links (like the what's new page, and the meta-index). Use this to get pointers to other documents, and concentrate on new links -- this way you will get a high initial yield, and if you stop your robot for some reason at least it has spent it's time well.

Don't try queries

Some WWW documents are searcheable (ISINDEX) or contain forms. Don't follow these. The Fish Search does this for example, which may result in a search for "cars" being sent to databases with computer science PhD's, people in the X.500 directory, or botanical data. Not sensible.

Stay with it

Log

Make sure it provides ample logging, and it wouldn't hurt to keep certain statistics, such as the number of successes/failures, the hosts accessed recently, the average size of recent files, and keep an eye on it. This ties in with the "Don't Loop" section -- you need to log where you have been to prevent looping. Again, estimate the required disk-space, you may find you can't cope.

Be interactive

Arrange for you to be able to guide your robot. Commands that suspend or cancel the robot, or make it skip the current host can be very useful. Checkpoint your robot frequently. This way you don't lose everything if it falls over.

Be prepared

Your robot will visit hundreds of sites. It will probably upset a number of people. Be prepared to respond quickly to their enquiries, and tell them what you're doing.

Be understanding

If your robot upsets someone, instruct it not to visit his/her site, or only the home page. Don't lecture him/her about why your cause is worth it, because they probably aren't in the least interested. If you encounter barriers that people put up to stop your access, don't try to go around them to show that in the Web it is difficult to limit access. I have actually had this happen to me; and although I'm not normally violent, I was ready to strangle this person as he was deliberatly wasting my time. I have written a standard practice proposal for a simple method of excluding servers. Please implement this practice, and respect the wishes of the server maintainers.

Share results

Keep results

This may sound obvious, but think about what you are going to do with the retrieved documents. Try and keep as much info as you can possibly store. This will the results optimally useful.

Raw Result

Make your raw results available, from FTP, or the Web or whatever. This means other people can use it, and don't need to run their own servers.

Polished Result

You are running a robot for a reason; probably to create a database, or gather statistics. If you make these results available on the Web people are more likely to think it worth it. And you might get in touch with people with similar interests.

Report Errors

Your robot might come accross dangling links. You might as well publish them on the Web somewhere (after checking they really are. If you are convinced they are in error (as opposed to restricted), notify the administrator of the server.

Examples

Examples of how not to do it

The robot run from phoenix.doc.ic.ac.uk in Jan 94. It provides no User-agent or From fields, one can't finger the host, and it is not part of a publicly known project. In addition it has been reported to retrieve documents it can't handle. Has since improved.

The Fish search capability added to Mosaic. One instance managed to retrieve 25 documents in under one minute.

Better examples

Jumpstation: the results are presented in a searchable database, the author announced it, and is considering making the raw results available. Unfortunately some people complained about the high rate with which documents were retrieved.

Why?

This page has been contributed to by Jonathon Fletcher JumpStation Robot author, Lee McLoughlin (L.McLoughlin@doc.ic.ac.uk), and others.