Over the next year I plan to incrementally start posting API's and
components for robot builders. I've been pondering hard-coding in
limits, restrictions, etc. These limits could be overridden if say the
target robots.txt allowed it. For example, can't collect mailto's
unless "Mail-To: OK" were found or something. But I'm not sure if I
should take that approach or if I should just let the violators pay the
penalty.
Another issue is: since people will be using my API's. should I
hard-code in a line that will always be packaged with a request, like:
User-API: WebDroid/1.x ...? On the one hand I like it, on the other
hand, some twit spams Congress using my API and it's Steve Jackson Games
all over again. Plus from a programmers perspective I don't think I'd
like an API sticking a line in that I don't have control over.
But say for example that there's some unique problem with my API where
it just mangles Apache servers. So then 10 different robots get created
by ten different programmers, but they all mangle the same server type.
Knowing the root of the cause, all Apache owners could disallow robots
using my API. Though we know, being the wicked incredible programmer I
am that could never happen. But still, I think it may be a good
feature.
Thoughts?
Also where is the official keeper/Web/FAQ on robots.txt these days?
WARNING: Digression alert (topic purists - bail now!):
Richard Gaskin wrote:
> 1) His last name is not "Returned".
And I don't think his first name is Robert:
Whitall, James (JW3426) hipcrime@POST1.COM
41 Sutter Street, Apt. #1225
San Francisco, CA 94104
I fed it into http://www.mapblast.com and confirmed it this time. If
you can stroll over there by Golden Gate University, check the garbage.
If you can the see the remains of Spam in the can, we have our man.
Brad Fox wrote:
>
> Richard Gaskin <Ambassador@FourthWorld.com> wrote:
>
> > Look it up.
>
> The tone of "Look it up" (or "do the math" or "connect the dots") is
> unpleasantly superior and condescending. It seems particularly
Yikes! Chill! Chill! I sware I don't feel slighted! ;)
-- ================================================================== Mitch Allen WebDroid http://www.webdroid.com P.O. Box 6569 mailto:mitch@webdroid.com Boston, MA. 02114 _________________________________________________ This messages was sent by the robots mailing list. To unsubscribe, send mail to robots-request@webcrawler.com with the word "unsubscribe" in the body. For more info see http://info.webcrawler.com/mak/projects/robots/robots.html