Re: Is a robot visiting?

Tim Freeman (tim@infoscreen.com)
Sat, 2 Nov 1996 15:42:06 -0800


nabil@teleport.com said:
>He wants to give index-fodder to the search engine, while sending the
>"real" page to the user.
>
>Another entry for my score-manipulator blacklist.

The purpose is what I said: my software normally gives different URL's
to different folks at the main entry to the site, so I don't want a
search engine causing a bunch of different folks to have the same URL.

You should wait until you see the score manipulation happening before
you make such accusations. Successful spamdexing is easy to detect if
you ever search the database generated by your web crawler, so there
is little lost by waiting until you have real evidence. Your
blacklisting technique is an appropriate and effective deterrent, if
you have the patience to wait until the infraction has actually happened.
There's also a technical problem with doing it prematurely -- what
were you going to enter into your blacklist, anyhow?

Tim Freeman