Re: Is a robot visiting?

Aaron Nabil (nabil@teleport.com)
Sun, 3 Nov 1996 01:14:05 -0800 (PST)


Tim Freeman writes...
> nabil@teleport.com said:
> >He wants to give index-fodder to the search engine, while sending the
> >"real" page to the user.
> >
> >Another entry for my score-manipulator blacklist.
>
> The purpose is what I said: my software normally gives different URL's
> to different folks at the main entry to the site, so I don't want a
> search engine causing a bunch of different folks to have the same URL.

Actually, you didn't say. I'll forward you your original request back
if you like. I guess enough people asked you "why" and then you decided to
explain your objectives in a later message.

We just had a "announcement" forwarded to list last month about a product
that does just that, ie, returns different results to search engines
in an attempt to inflate scores.

> You should wait until you see the score manipulation happening before
> you make such accusations. Successful spamdexing is easy to detect if
> you ever search the database generated by your web crawler, so there
> is little lost by waiting until you have real evidence. Your
> blacklisting technique is an appropriate and effective deterrent, if
> you have the patience to wait until the infraction has actually happened.
> There's also a technical problem with doing it prematurely -- what
> were you going to enter into your blacklist, anyhow?

You are missing the point.

The objective isn't to actually correct score inflation retroactively, it's
to scare people into not taking the chance in the first place.

The whole idea is defeated is you have rules written down like "if your
page has the same keyword repeated 20 times, the whole page will be
discarded". All they do is repeat the keywords 19 times.

You need to say things like "if we detect what we think is score inflation,
we may delete your page or your entire site from our index. We decide
what 'score inflation' is, and our criteria may change anytime. If you try
it, you might sneak by, but if we catch you..."

As for you illustration of "what would someone blacklist", how about anything
with your name on it, now, or in the future? Any sites of any clients of
yours. Any sites I even think are related to yours. Your mother's site.
You brother's site. Your neighbor's site. Your gym teacher's site.

I think you get the idea. The fear of such an "irrational response" would
no doubt make you think twice. Since a search engine is private property,
the owners can index, or not, anything they please. You have to play by
their rules, and they can change the rules without even showing them to
you.

-- 
Aaron Nabil
nabil@teleport.com