Robots Mailing List Archive by thread
Starting: Wed 00 Jan 1970 - 16:31:48 PDT
Ending: Thu 18 Dec 1997 - 14:33:60 PDT
Messages: 2106
- Announcement Michael G=?iso-8859-1?Q?=F6ckel
- RE: The Internet Archive robot Sigfrid Lundberg
- The robots mailing list at WebCrawler Martijn Koster
- Something that would be handy Tim Bray
- Site Announcement James
- How do I let spiders in? Roger Dearnaley
- Unfriendly robot at 205.177.10.2 Nick Arnett
- CORRECTION -- Re: Unfriendly robot Nick Arnett
- Looking for a spider Alain Desilets
- Is it a robot or a link-updater? David Bakin
- Unfriendly robot owner identified! Nick Arnett
- Really fast searching Nick Arnett
- Sorry! Alain Desilets
- Re: Unfriendly robot at 205.252.60.50 Nick Arnett
- re: Lycos unfriendly robot Murray Bent
- Re: Unfriendly robot at 205.252.60.50 Nick Arnett
- Re: Unfriendly robot at 205.252.60.50 Nick Arnett
- Re: Proposed URLs that robots should search Nick Arnett
- Re: Proposed URLs that robots should search Kim Davies
- lycos patents Murray Bent
- re: Lycos patents Murray Bent
- Patents? Martijn Koster
- meta tag implementation Davide Musella
- Preliminary robot.faq (Please Send Questions or Comments) Keith Fischer
- BOUNCE robots: Admin request
- wwwbot.pl problem Andrew Daviel
- yet another robot Paul Francis
- Q: Cooperation of robots Byung-Gyu Chang
- Smart Agent help Michael Goldberg
- harvest John D. Pritchard
- How frequently should I check /robots.txt? Skip Montanaro
- McKinley Spider hit us hard Christopher Penrose
- Mail failure Adminstrator
- Mail failure Adminstrator
- Mail failure Adminstrator
- Mail failure Adminstrator
- Small robot needed Karoly Negyesi
- New robot turned loose on an unsuspecting public... and a DNS question Skip Montanaro
- inquiry about robots Cristian Ionitoiu
- MacPower Lance Ogletree
- Re: Returned mail: Service unavailableHELP HELP! Julian Gorodsky
- Re: Returned mail: Service unavailableHELP AGAIN HELP AGAIN! Julian Gorodsky
- Indexing two-byte text Harry Munir Behrens
- Indexing two-byte text Harry Munir Behrens
- Either a spider or a hacker? ww2.allcon.com Randall Hill
- Indexing two-byte text Mark Schrimsher
- RE: Indexing two-byte text smadja@netvision.net.il
- RE: Indexing two-byte text Mark Schrimsher
- Freely available robot code in C available? ecarp@tssun5.dsccc.com
- Freely available robot code in C available? Kevin Hoogheem
- Harvest question Jim Meritt
- Announcement and Help Requested Simon.Stobart
- Announcement and Help Requested Simon.Stobart
- Re[2]: Harvest question Jim Meritt
- Robot on the Rampage mschrimsher@twics.com
- [1]RE>Checking Log files Roger Dearnaley
- [2]RE>Checking Log files Roger Dearnaley
- [3]RE>Checking Log files Roger Dearnaley
- [4]RE>Checking Log files Roger Dearnaley
- [5]RE>Checking Log files Roger Dearnaley
- [5]RE>Checking Log files Mark Schrimsher
- [1]RE>[5]RE>Checking Log fi Roger Dearnaley
- [2]RE>[5]RE>Checking Log fi Roger Dearnaley
- Wobot? Byung-Gyu Chang
- Announcing NaecSpyr, a new. . . robot? Mordechai T. Abzug
- [3]RE>[5]RE>Checking Log fi Roger Dearnaley
- [1]Wobot? Roger Dearnaley
- [1]Announcing NaecSpyr, a n Roger Dearnaley
- [2]Announcing NaecSpyr, a n Roger Dearnaley
- [2]Wobot? Roger Dearnaley
- [4]RE>[5]RE>Checking Log fi Roger Dearnaley
- [3]Wobot? Roger Dearnaley
- [3]Announcing NaecSpyr, a n Roger Dearnaley
- [1]RE>[3]RE>[5]RE>Checking Roger Dearnaley
- [5]RE>[5]RE>Checking Log fi Roger Dearnaley
- [1]RE>[2]RE>[5]RE>Checking Roger Dearnaley
- [4]Wobot? Roger Dearnaley
- [4]Announcing NaecSpyr, a n Roger Dearnaley
- [2]RE>[3]RE>[5]RE>Checking Roger Dearnaley
- Dearnaley Auto Reply Cannon? Micah A. Williams
- Dearnaley Auto Reply Cannon? Kevin Hoogheem
- [2]RE>[2]RE>[5]RE>Checking Roger Dearnaley
- [1]Contact for Intouchgroup Roger Dearnaley
- Re: [2]RE>[5]RE>Checking Lo Saul Jacobs
- [5]Wobot? Roger Dearnaley
- Re: [2]RE>[5]RE>Checking Lo Bonnie Scott
- Vacation wars Martijn Koster
- Re: [2]RE>[5]RE>Checking Lo David Henderson
- New Robot??? David Henderson
- test; please ignore Martijn Koster
- Unfriendly Lycos , again ... Murray Bent
- Inter-robot Comms Port David Eagles
- Re: Unfriendly Lycos , again ... Steven L. Baur
- Inter-robot Communications - Part II David Eagles
- unknown robot (no name)
- RE: Inter-robot Communications - Part II David Eagles
- RE: Inter-robot Communications - Part II Martijn Koster
- please add my site gil cosson
- Please Help ME!! Dong-Hyun Kim
- Infinite e-mail loop Roger Dearnaley
- Up to date list of Robots James
- Web Robot Matthew Gray
- Re: Web Robots James
- Re: Web Robots Jakob Faarvang
- Does this count as a robot? Thomas Stets
- RE: avoiding infinite regress for robots David Eagles
- Recursion David Eagles
- Duplicate docs (was avoiding infinite regress...) Nick Arnett
- RE: Recursion David Eagles
- MD5 in HTTP headers - where? Skip Montanaro
- robots.txt extensions Adam Jack
- Does anyone else consider this irresponsible? Robert Raisch, The Internet Company
- FAQ again. Martijn Koster
- Robots / source availability? Martijn Koster
- Re: Does anyone else consider... Mark Norman
- Re: Does anyone else consider... Mark Schrimsher
- (no subject) Alison Gwin
- Robots not Frames savy David Henderson
- Re: Spam Software Sought Mark Schrimsher
- Re: Does anyone else consider... smadja@netvision.net.il
- Horror story
- Gopher Protocol Question Hal Belisle
- New Robot Announcement Larry Burke
- Re: robots.txt extensions Steven L Baur
- Alta Vista searches WHAT?!? Ed Carp @ TSSUN5
- BOUNCE robots: Admin request Martijn Koster
- RE: Alta Vista searches WHAT?!? Ted Sullivan
- robots.txt , authors of robots , webmasters .... savron@world-net.sct.fr
- Web robots and gopher space -- two separate worlds savron@world-net.sct.fr
- [ANNOUNCE] CFP: AAAI-96 WS on Internet-based Information Systems Alexander Franz
- Robot Research Bhupinder S. Sran
- Re: Re: robots.txt , authors of robots , webmasters .... Larry Burke
- re: privacy, courtesy, protection John Lammers
- Server name in /robots.txt Martin Kiff
- Polite Request #2 to be Removed form List AJAJR@aol.com
- un-subcribe AJAJR@aol.com
- RE: Server name in /robots.txt David Eagles
- RE: Server name in /robots.txt Tim Bray
- Who sets standards (was Server name in /robots.txt) Nick Arnett
- HEAD request [was Re: Server name in /robots.txt] Davide Musella
- Activity from 205.252.60.5[0-8] Martin Kiff
- test. please ignore. Mark Norman
- Any info on "E-mail America"? Bonnie Scott
- www.pl? The YakkoWakko. Webmaster
- New URL's from Equity Int'll Webcenter Mark Krell
- Requesting info on database engines Renato Mario Rossello
- News Clipper for newsgroups - Windows Richard Glenner
- Wanted: Web Robot code - C/Perl Charlie Brown
- Perl Spiders Christopher Penrose
- Here is WebWalker Christopher Penrose
- The "Robot and Search Engine FAQ" Keith D. Fischer
- algorithms Kenneth DeMarse
- The Robot And Search Engine FAQ Keith D. Fischer
- Money Spider WWW Robot for Windows John McGrath - Money Spider Ltd.
- robots.txt changes how often? Tangy Verdell
- robots.txt Tangy Verdell
- Re: Commercial Robot Vendor Recoomendations Request Michael De La Rue
- fdsf Davide Musella
- Robots and search engines technical information. Hayssam Hasbini
- Tutorial Proposal for WWW95 http
- URL measurement studies? Darren R. Hardy
- about robots.txt content errors savron@world-net.sct.fr
- Dot dot problem... Sean Parker
- Robot Databases Steve Livingston
- Anyone doing a Java-based robot yet? Nick Arnett
- RE: Robot Databases Ted Sullivan
- Ingrid ready for prelim alpha testing.... Paul Francis
- Robot for Sun David Schnardthorst
- url locating Martijn De Boef
- Re[2]: Anyone doing a Java-based robot yet? Tangy Verdell
- Altavista indexing password files John Messerly
- RE: Altavista indexing password files John Messerly
- BSE-Slurp/0.6 Gordon V. Cormack
- Can I retrieve image map files? Mark Norman
- Robots available for Intranet applications Douglas Summersgill
- "What's new" in web pages is not necessarily reliable Mordechai T. Abzug
- verify URL Jim Meritt
- libww and robot source for Sequent Dynix/Ptx 4.1.3 (no name)
- Re[2]: verify URL Jim Meritt
- robot authentication parameters Sibylle Gonzales
- RE: verify URL Debbie Swanson
- Re: Robots available for Intranet applications jon madison
- How to...??? Francesco
- image map traversal Mark Norman
- ReHowto...??? Jared Williams
- Info on authoring a Web Robot Jared Williams
- image map traversal frizzlefry@nucleus.atom.com
- Links Jared Williams
- Limiting robots to top-level page only (via robots.txt)? Chuck Doucette
- Image Maps Thomas Merlin
- Request for Source code in C for Robots ACHAKS
- robots that index comments Dan Gildor
- Re: UNSUBSCRIBE ROBOTS Pinano@aol.com
- keywords in META-element Detlev Kalb
- Announce: ActiveX Search (IFilter) spec/sample Lee Fisher
- Re: Links This Site is about Robots Not Censorship Keith
- Re: Links (don't bother checking; I've done it for you) Darrin Chandler
- Re: Links (don't bother checking; I've done it for you) David Henderson
- Re: Links This Site is about Robots Not Censorship Rob Turk
- Re: Links (don't bother checking; I've done it for you) Martin Kiff
- Re: Links (don't bother checking; I've done it for you) Darrin Chandler
- The Letter To End All Letters Jared Williams
- Heuristics.... Martin Kiff
- unscribe christophe grandjacquet
- unsubscibe christophe grandjacquet
- Admin: how to get off this list Martijn Koster
- Search accuracy Nick Arnett
- Clean up Bots... Andy Warner
- Re: (Fwd) Re: Search accuracy Colin Goodier
- VB and robot development Mitchell Elster
- Problem with your Index Mike Rodriguez
- Handling keyword repetitions Mr David A Weeks
- word spam chris cobb
- http directory index request Mark Norman
- RE: http directory index request David Levine
- Returned mail: Can't create output: Error 0 Mail Delivery Subsystem
- Re: word spam
- Web Robot Jared Williams
- Robots in the client? Ricardo Eito Brun
- General Information Ricardo Eito Brun
- Magic, Intelligence, and search engines Tim Bray
- default documents Jakob Faarvang
- Mailing list Jared Williams
- RE: Mailing List Mitchell Elster
- About Mother of All Bulletin Boards Ricardo Eito Brun
- search engine Jose Raul Vaquero Pulido
- Quiz playing robots ? Andrey A. Krasov
- Try robot... Jared Williams
- (no subject) Jared Williams
- "Good Times" hoax Nick Arnett
- (no subject) Kevin Hoogheem
- RE: "Good Times" hoax David Eagles
- Re: Re: Bill Day
- About integrated search engines Ricardo Eito Brun
- [ MERCHANTS ] My Sincerest Apologies Jared Williams
- Re: Apologies || communal bots Rob Turk
- Re: communal bots Rob Turk
- To: ???? Robot Mitchell Elster
- RE: To: ???? Robot chris cobb
- Private Investigator Lists Scott W. Wood
- [Fwd: Re: To: ???? Robot] Rob Turk
- Re: To: ???? Robot] Terry Smith
- Re: To: ???? Robot Mitchell Elster
- Looking for a search engine Bhupinder S. Sran
- Admin: List archive is back Martijn Koster
- topical search tool -- help?! Brian Fitzgerald
- cc:Mail SMTPLINK Undeliverable Message
- Indexing a set of URL's Fred Melssen
- Political economy of distributed search (was topical search...) Nick Arnett
- Re: Political economy of distributed search (was topical search Robert Raisch, The Internet Company
- (OTP) RE: Political economy of distributed search (was topical search...) David Levine
- Re: Political economy of distributed search (was topical Terry Smith
- Re: (OTP) RE: Political economy of distributed search (was topical Kevin Hoogheem
- Re: Political economy of distributed search (was topical Steve Jones
- Meta-seach engines Harry Munir Behrens
- any robots/search-engines which index links? Alex Chapman
- alta vista and virtualvin.com chris cobb
- VB. page grabber... Marc's internet diving suit
- Re: Lead Time Myles Olson
- RE: VB. page grabber... Victor F Ribeiro
- Re: Political economy of distributed search (was topical Brian Ulicny
- RE: VB. page grabber... chris cobb
- RE: any robots/search-engines which index links? Louis Monier
- Re[2]: verify URL Jim Meritt
- ANNOUNCE: Don Norman (Apple) LIVE! 15-May 5PM UK = noon EDT Marc Eisenstadt
- Web spaces of strange topology. Where? Michael De La Rue
- Somebody is turning 23! Eric
- robots and cookies Ken Nakagama
- Accept: Thomas Abrahamsson
- Defenses against bad robots kathy@accessone.com
- Re: Image Maps jon madison
- That wacky Wobot
- ??: reload problem Andrey A. Krasov
- Robot-HTML Web Page? J.Y.K.
- Robot Exclusion Standard Revisited Charles P. Kollar
- McKinley robot Rafhael Cedeno
- McKinley robot Dave Rothwell
- RE: alta vista and virtualvin.com Louis Monier
- RE: alta vista and virtualvin.com Louis Monier
- RE: alta vista and virtualvin.com Louis Monier
- Specific searches The Wild
- RE: alta vista and virtualvin.com Louis Monier
- RE: alta vista and virtualvin.com Bakin, David
- RE: alta vista and virtualvin.com Paul Francis
- A new robot...TOPjobs(tm) USA JOBbot 1.0a D. Williams
- Test server for robot development? D. Williams
- Re: Robot Exclusion Standard Revisited (LONG) Martijn Koster
- Content based robot collectors Scott 'Webster' Wood
- Robot to collect web pages per site Henrik Fagrell
- Recherche de documentation sur les agents intelligents ou Robots. Mannina Bruno
- www.kollar.com/robots.html John D. Pritchard
- Tagging a document with language Tronche Ch. le pitre
- Re: Tagging a document with language Gen-ichiro Kikui
- RE: Tagging a document with language Henk Alles
- RE: Tagging a document with language Robert Raisch, The Internet Company
- implementation fo HEAD response with meta info Davide Musella
- robots, what else! Fred K. Lenherr
- Finding the canonical name for a server Jaakko Hyvatti
- Re: (book recommendation re: net agents) Rob Turk
- Content based search engine Scott 'Webster' Wood
- (no subject) Digital Universe Inc.
- (no subject) Larry Stephen Burke
- (no subject) Fred K. Lenherr
- BackRub robot L a r r y P a g e
- robots.txt: allow directive Leslie Cuff
- looking for specific bot... Brenden Portolese
- Looking for News robot Michael Goldberg
- robot.polite Brian Hancock
- Looking for good one JoongSub Lee
- More dangers of spiders... dws
- RE: Tagging a document with language Henk Alles
- Keyword indexing David Reilly
- Robot books boogieoogie goobnie
- Allow/deny robots from major search services boogieoogie goobnie
- Robot logic? marco@specialnet.cmt.it
- robots.txt usage Wiebe Weikamp
- robot vaiable list Marco Genua
- WebAnalyzer - introduction Craig McQueen
- (no subject) Martijn Koster
- RE: Introducing myself Anthony D. Thomas
- RE: WebAnalyzer - introduction Gregg Steinhilpert
- PERL Compilers & Interpretive Tools MPMC-Manhattan Premed Council & Hunter PBPMA-Post Bac PreMed Assoc 212-843-3701 Ext 2800
- RE: robots.txt usage David Levine
- RE: WebAnalyzer - introduction Craig McQueen
- RE: Tagging a document with language Robert Raisch, The Internet Company
- Microsoft Tripoli Web Search Beta now available Lee Fisher
- RE: Tagging a document with language Henk Alles
- (no subject) Kevin Lew
- in-document directive to discourage indexing ? Denis McKeon
- (no subject) edas@micros.et.byu.edu
- Client Robot 'Ranjan' Brenden Portolese
- multiple copies Steve Leibman
- RE: Client Robot 'Ranjan' chris cobb
- Inter-robot communication David Reilly
- Re: RCPT: Re: Introducing myself Steve Jones
- Re: RCPT: Re: Introducing myself Fred K. Lenherr
- Looking for... Steven Frank
- Re: RCPT: Re: Introducing myself Rob Turk
- Re: RCPT: Re: Introducing myself Daniel Williams
- RE: Looking for... Craig McQueen
- Collected information standards Scott 'Webster' Wood
- Re: RCPT: Re: Introducing myself Ross Finlayson
- Re: RCPT: Re: Introducing myself Chris Crowther
- Re: RCPT: Re: Introducing myself Chris Crowther
- Java Robot Fabio Arciniegas A.
- Re: RCPT: Re: Introducing myself Hauke Loens
- Report of the Distributed Indexing/Searching Workshop Martijn Koster
- RE: Java Robot David Levine
- RE: Java Robot David Levine
- Re: Java Robot John D. Pritchard
- unscribe Jeffrey Kerns
- Dead account Anthony John Carmody
- web topology Fred K. Lenherr
- A modest proposal...<snip> to discourage indexing ? Rob Turk
- Re: Unsubscribing from Robots (was "your mail") Ian Samson
- Robot's Book. Mannina Bruno
- ADMIN: unsubscribing (was: Re: Martijn Koster
- Search Engine end-users Stephen Kahn
- loc(SOIF) John D. Pritchard
- ADMIN: Archive Martijn Koster
- roverbot - perhaps the worst robot yet dws
- we should help spiders and not say NO! Daniel Lo
- Advice Alyne Mochan & Warren Baker
- Robot Mirror with Username/Password feature (no name)
- (no subject) K.E. HERING
- Re: Robot Mirror with Username/Password feature Mannina Bruno
- (no subject) K.E. HERING
- Announcement Michael G=?iso-8859-1?Q?=F6ckel
- Harvest-like use of spiders Fred Melssen
- (no subject) Ronald Kanagy
- Re: hey man gimme a break Martijn Koster
- Identifying identical documents Daniel T. Martin
- Description or Abstract? G. Edward Johnson
- Should I index all ... CLEDER Catherine
- HTML Parser Ronald Kanagy
- desperately looking for a news searcher Saloum Fall
- Re: Unsubscribing from Robots (was "your mail") Ian Samson
- A modest proposal...<snip> to discourage indexing ? Rob Turk
- Re: your mail Ed Carp
- Blackboard for Discussing Domain-specific Robots Fred K. Lenherr
- robots.txt unavailability Daniel T. Martin
- nastygram from xxx.lanl.gov Aaron Nabil
- Re: nastygram from xxx.lanl.gov Aaron Nabil
- nastygram from xxx.lanl.gov Peter.Vogt@tokai.sprint.com
- RE: nastygram from xxx.lanl.gov Elias Sideris
- RE: nastygram from xxx.lanl.gov Frank Wales
- htaccess Steve Leibman
- RE: robots.txt unavailability Louis Monier
- RE: On the subject of abuse/pro-activeness Louis Monier
- NCSA Net Access_log Analysis Tool for Win95 MPMC-Manhattan Premed Council & Hunter PBPMA-Post Bac PreMed Assoc 212-843-3701 Ext 2800
- Proxies Larry Stephen Burke
- NCSA Net Access_log Analysis Tool for Win95 MPMC-Manhattan Premed Council & Hunter PBPMA-Post Bac PreMed Assoc 212-843-3701 Ext 2800
- RE: Should I index all ... David Levine
- How long to cache robots.txt for? Aaron Nabil
- Re: psycho at xxx.lanl.gov Rob Turk
- Re: Alta Vista getting stale? Denis McKeon
- RE: nastygram from xxx.lanl.gov Frank Wales
- RE: nastygram from xxx.lanl.gov Paul Francis
- RE: nastygram from xxx.lanl.gov Rob Hartill
- Updating Robots Ian Samson
- Re: nastygram from xxx.lanl.gov Aaron Nabil
- Prasad Wagle: Webhackers: Java servlets and agents John D. Pritchard
- RE: nastygram from xxx.lanl.gov Istvan
- RE: nastygram from xxx.lanl.gov Tim Bray
- Re: nastygram for xxx.lanl.gov David Levine
- RE: nastygram from xxx.lanl.gov Istvan
- Re: nastygram from xxx.lanl.gov Aaron Nabil
- dumb robots and xxx Rob Hartill
- ADMIN: Spoofing vs xxx.lanl.gov Martijn Koster
- RE: nastygram from xxx.lanl.gov Frank Wales
- RE: nastygram from xxx.lanl.gov Istvan
- forwarded e-mail Paul Ginsparg 505-667-7353
- the POST myth... a web admin's opinions.. Rob Hartill
- xxx.lanl.gov - The thread continues.... Chris Crowther
- xxx.lanl.gov a real threat? John Lammers
- Re: Alta Vista getting stale? Nick Arnett
- crawling accident Rafhael Cedeno
- xxx.lanl.gov/robots.txt Chris Crowther
- Stupid robots cache DNS and not IMS Roy T. Fielding
- Suggestion to help robots and sites coexist a little better Rob Hartill
- PS Rob Hartill
- Linux and Robot development... root
- Re: Suggestion to help robots and sites coexist a little better Mark J Cox
- interactive generation of URL's Fred Melssen
- Newbie question Dan Hurwitz
- Q: meta name="robots" content="noindex" ? John Bro, InterSoft Solutions, Inc
- Possible MSIIS bug? Jakob Faarvang
- HOST: header Rob Hartill
- Robot Research Kosta Tombras
- Safe Methods Istvan
- robots source code in C Ethan Lee
- Re: How long to cache robots.txt Daniel T. Martin
- *Help: Writing BOTS* Shiraz Siddiqui
- Re: How long to cache robots.txt Mike Agostino
- Search Engine article Danny Sullivan
- Unusual request - sorry! David Eagles
- Re: How long to cache robots.txt Daniel T. Martin
- Last message David Eagles
- Re: Social Responsibilities (was Safe Methods) David Eichmann
- AltaVista's Index is obsolete Ian Samson
- RE: AltaVista's Index is obsolete; but what about the others Ted Sullivan
- RE: AltaVista's Index is obsolete; but what about the other Ian Samson
- Re: AltaVista's Index is obsolete; but what about the other Daniel T. Martin
- hoohoo.cac.washington = bad Andy Warner
- PHP stops robots kathy@accessone.com
- netscape spec for RDM Jim Ausman
- Stop 'bots using apache, etc. or php? kathy@accessone.com
- Anyone know who owns this one? Betsy Dunphy
- Robot Gripes forum? (Was: Anyone know who owns this one?) Betsy Dunphy
- RE: Anyone know who owns this one? Martin.Soukup
- I vote NO (Was: Robot Gripes forum?) Nick Arnett
- Re: I vote NO (Was: Robot Gripes forum?) - I vote YES Betsy Dunphy
- What is your favorite search engine - a survey Bhupinder S. Sran
- robots on an intranet Adam Gaffin
- grammar engines Ross A. Finlayson
- Re[2]: robots on an intranet Stacy Cannady
- Add This Search Engine to your Results. Thomas Bedell
- Search Engine Tutorial for Web Developers Edward Stangler
- RE: How can IR Agents be evaluate ? Jim Harris
- RE: How can IR Agents be evaluate ? Dan Quigley
- Project Aristotle(sm) Gerry McKiernan
- Re: robots on an intranet (replies to list...) Martijn Koster
- RE: Search Engine System Admin
- Re: robots on an intranet (replies to list...) Operator
- Re: Search Engine Operator
- Re: Search Engine Stimpy
- Lynx. The one true browser. Ian McKellar
- email spider Richard A. Paris
- RE: email spider Jim Harris
- RE: How can IR Agents be evaluate ? Nick Arnett
- Re: Search Engine Nick Arnett
- Offline Agents for UNIX A. Shiraz Siddiqui
- mini-robot Chiaki Ohta
- Webfetch Edward Ooi
- Library Agents(sm): Library Applications of Intelligent Software Agents Gerry McKiernan
- Mail robot? Christopher J. Farrell IV
- depth first vs breadth first Robert Nicholson
- Quakebots Ross A. Finlayson
- articles or URL's on search engines Werner Schweibenz, Mizzou
- HEAD Anna Torti
- The Internet Archive robot Mike Burner
- The Internet Archive robot ACHAKS@inf.com
- (no subject) Stacy Cannady
- "hidden text" vs. META tags for robots/search engines Todd Sellers
- Looking for subcontracting spider-programmers Richard Rossi
- RE: The Internet Archive robot Ted Sullivan
- (no subject) Robert Stober
- RE: The Internet Archive robot (fwd) Brewster Kahle
- pointers for a novice? Alex Strasheim
- Extracting info from SIG forum archives Peter Small
- Conceptbot spider David L. Sifry
- Re: The Internet Archive robot (fwd) Robert B. Turk
- Re: The Internet Archive robot Phil Hochstetler
- Copyrights (was Re: The Internet Archive robot) Brian Clark
- Copyrights on the web Charlie Brown
- RE: Copyrights on the web Ted Sullivan
- More ways to spam search engines? G. Edward Johnson
- RE: copyright, etc. Richard Gaskin - Fourth World
- Copyrights, let them be ! Joao Moreira
- crawling FTP sites Greg Fenton
- RE: Copyrights, let them be ! William Dan Terry
- RE: crawling FTP sites William Dan Terry
- Re: The Internet Archive robot (fwd) Brewster Kahle
- RE: crawling FTP sites William Dan Terry
- Re: Public Access Nodes / Copywrited Nodes Ross A. Finlayson
- FAQ? Frank Smadja
- RE: The Internet Archive robot David Levine
- RE: Copyrights on the web Bryan Cort
- RE: The Internet Archive robot William Dan Terry
- RE: The Internet Archive robot David Levine
- InfoSpiders/0.1 Filippo Menczer
- info for newbie Filippo Menczer
- Bad agent...A *very* bad agent. Benjamin Franz
- RE: The Internet Archive robot Nick Arnett
- A few copyright notes Nick Arnett
- Topic drift (archive robot, copyright...) Nick Arnett
- Netscape Catalog Server: An Eval Eric Kristoff
- file retrieval Eddie Rojas
- Preferred access time Martin.Soukup
- White House/PARC "Leveraging Cyberspace" Nick Arnett
- RE: Preferred access time Martin.Soukup
- robot to get specific info only? Martin.Soukup
- Unregistered MIME types? Nick Arnett
- A bad agent? Nick Dearnaley
- Use of robots.txt to "check status"? Ed Costello
- Robot exclustion for for non-'unix file' hierarchy Hallvard B Furuseth
- How to get listed #1 on all search engines (fwd) Bonnie
- RE: How to get listed #1 on all search engines (fwd) Ted Sullivan
- Cannot believe it "Morons" Cafe
- Possible robot? Chad Zimmerman
- Bye Bye HyperText: The End of the World (Wide Web) As We Know It! Gerry McKiernan
- Image Maps Harold Gibbs
- Bug in LibWWW perl + Data::Dumper (libwwwperl refs are strange) Michael De La Rue
- do robots send HTTP_HOST? Joe Pruett
- The End of The World (Wide Web) / Part II Gerry McKiernan
- Seeing is Believing: Candidate Web Resources for Information Visualization Gerry McKiernan
- CitedSites(sm): Citation Indexing of Web Resources Gerry McKiernan
- Netscape Catalog Server Eric Kristoff
- Netscape Catalog Server Eric Kristoff
- Topic-specific robots Fred K. Lenherr
- robots.txt syntax Fred K. Lenherr
- Another rating scam! (And a proposal on how to fix it) Aaron Nabil
- META tag standards, search accuracy Nick Arnett
- ADMIN (was Re: hypermail archive not operational Martijn Koster
- ActiveAgent HipCrime
- non 2nn repsonses on robots.txt Aaron Nabil
- servers that don't return a 404 for "not found" Aaron Nabil
- Re: servers that don't return a 404 for "not found" Aaron Nabil
- Re: servers that don't return a 404 for "not found" Aaron Nabil
- Re: Another rating scam! (And a proposal on how to fix it Nick Dearnaley
- Returned mail: Host unknown (Name server: webcrawler: host not found) Mail Delivery Subsystem
- Re: META tag standards, search accuracy Eric Miller
- ActiveAgent Larry Steinberg
- Re: ActiveAgent Benjamin Franz
- Re: META tag standards, search accuracy Eric Miller
- Returned mail: Host unknown (Name server: webcrawler: host not found) Mail Delivery Subsystem
- RE: ActiveAgent and E-Mail Spam Bryan Cromartie
- Re: META tag standards, search accuracy Eric Miller
- Re: ActiveAgent William Neuhauser
- agents ignoring robots.txt Rob Hartill
- Re: agents ignoring robots.txt Rob Hartill
- Disallow/Allow by Action (Re: robots.txt syntax) Brian Clark
- CyberPromo shut down at last!!! Richard Gaskin - Fourth World
- infoseek Fred K. Lenherr
- Comparing robots/search sites Fred K. Lenherr
- Re: infoseek Rob Hartill
- McKinley -- 100% error rate Jennifer C. O'Brien
- Re: robots.txt buffer question. Brent Boghosian
- What to rate limit/lock on, name or IP address? Aaron Nabil
- RE: What to rate limit/lock on, name or IP address? Greg Fenton
- RE: What to rate limit/lock on, name or IP address? Brent Boghosian
- Filtering queries on a robot-built database Fred K. Lenherr
- sockets in PERL HipCrime
- Re: sockets in PERL Otis Gospodnetic
- Re: Showbiz search engine Showbiz Information
- Is a robot visiting? Tim Freeman
- RE: Is a robot visiting? Greg Fenton
- Tim Freeman Aaron Nabil
- Thanks! Tim Freeman
- Possible robots.txt addition Ian Graham
- RE: Is a robot visiting? Hallvard B Furuseth
- download robot LIAM GUINANE
- A new robot -- ask for advice Hrvoje Niksic
- Re: Possible robots.txt addition Martin Kiff
- technical descripton[D[D[D LIAM GUINANE
- Domains and HTTP_HOST Brian Clark
- Source code Stephane Vaillancourt
- Back of the envelope computations Francois Rouaix
- Re: Possible robots.txt addition Ian Graham
- Re: Possible robots.txt addition Ian Graham
- Possible robots.txt addition - did I say that? Martin Kiff
- Re: Possible robots.txt addition Klaus Johannes Rusch
- Re: Possible robots.txt addition (fwd) Ian Graham
- Belated notice of spider article Adam Gaffin
- Re: Possible robots.txt addition Klaus Johannes Rusch
- ActiveAgent HipCrime
- Matching the user-agent in /robots.txt Hrvoje Niksic
- Re: ActiveAgent Aaron Nabil
- Re: ActiveAgent ROBERT-LF_HUANG@HP-China-om1.om.hp.com
- Re: ActiveAgent Rob Hartill
- We need robot information Juan
- anti-robot regexps Hallvard B Furuseth
- anti-robot regexps Hallvard B Furuseth
- An extended verion of the robot exclusion standard Captain Napalm
- Re: An extended version of the Robots... Martijn Koster
- Re: Domains and HTTP_HOST Klaus Johannes Rusch
- robot algorithm ? Otis Gospodnetic
- itelligent agents Wanca, Vincent
- Get official! Hallvard B Furuseth
- Regexps (Was: Re: An extended version of the Robots...) Hallvard B Furuseth
- Regexps (Was: Re: An extended version of the Robots...) Hallvard B Furuseth
- Is their a web site... z960849@rice.farm.niu.edu
- Re: Regexps (Was: Re: An extended version of the Robots...) Martijn Koster
- Re: An extended verion of the robot exclusion standard Captain Napalm
- Re: An extended version of the Robots... Captain Napalm
- An updated extended standard for robots.txt Captain Napalm
- Notification protocol? Nick Arnett
- Re: An extended version of the Robots... Hallvard B Furuseth
- Re: An extended version of the Robots... (fwd) Vu Quoc HUNG
- changes to robots.txt Rob Hartill
- Re: An updated extended standard for robots.txt Captain Napalm
- Re: An extended version of the Robots... Martijn Koster
- UN/LINK protocol is standardized! wasn't that quick! John D. Pritchard
- Re: UN/LINK protocol is standardized! wasn't that quick! John D. Pritchard
- [ROBOTS JJA] Lib-WWW-perl5 Juan Jose Amor
- Admitting the obvious I think therefore I spam
- how do they do that? Otis Gospodnetic
- databases for spiders Elias Hatzis
- databases for spiders Elias Hatzis
- Re: databases for spiders ROBERT-LF_HUANG@HP-China-om1.om.hp.com
- indexing intranet-site Martin Paff
- Re: An extended version of the Robots... Hallvard B Furuseth
- Re: An extended version of the Robots... Hallvard B Furuseth
- RE: databases for spiders Larry Fitzpatrick
- RE: databases for spiders Nick Arnett
- infoseeks robot is dumb Otis Gospodnetic
- RE: changes to robots.txt Scott Johnson
- Not so Friendly Robot - Teleport David McGrath
- RFC, draft 1 Martijn Koster
- Re: An extended version of the Robots... Martijn Koster
- Re: infoseeks robot is dumb DA
- Getting a Reply-to: field ... Captain Napalm
- Re: Getting a Reply-to: field ... Captain Napalm
- RE: RFC, draft 1 Keiji Kanazawa
- RE: RFC, draft 1 Keiji Kanazawa
- Re: An extended version of the Robots... Darren Hardy
- Re: indexing intranet-site Nick Arnett
- Re: RFC, draft 1 Klaus Johannes Rusch
- Hipcrime no more Martijn Koster
- RE: Notification protocol? Larry Fitzpatrick
- http://HipCrime.com HipCrime
- Re: http://HipCrime.com Nick Arnett
- RE: RFC, draft 1 Martijn Koster
- Re: Virtual (was: RFC, draft 1) Klaus Johannes Rusch
- Re: Washington again !!! Erik Selberg
- Re: Washington again !!! Rob Hartill
- SetEnv a problem Anna Torti
- User-Agent David Banes
- Broadness of Robots.txt (Re: Washington again !!!) Brian Clark
- Re: Washington again !!! Martijn Koster
- Re: Washington again !!! Gregory Lauckhart
- robots.txt HipCrime
- Re: User-Agent Klaus Johannes Rusch
- Re: Broadness of Robots.txt (Re: Washington again !!!) Captain Napalm
- Re: Broadness of Robots.txt (Re: Washington again !!!) Captain Napalm
- must be something in the water Rob Hartill
- who/what uses robots.txt HipCrime
- Re: Broadness of Robots.txt (Re: Washington again !!!) Art Matheny
- Re: Broadness of Robots.txt (Re: Washington again !!!) Rob Hartill
- Koen Holtman: Content negotiation draft 04 submitted John D. Pritchard
- Re: who/what uses robots.txt Rob Hartill
- defining "robot" HipCrime
- Re: defining "robot" Art Matheny
- define a page? HipCrime
- robots.txt (A *little* off the subject) Thaddeus O. Cooper
- Re: define a page? Rob Hartill
- robot defined HipCrime
- not a robot HipCrime
- robot? HipCrime
- another suggestion Rob Hartill
- Re: not a robot Rob Hartill
- ActiveAgent Rob Hartill
- robot definition Ross Finlayson
- make people use ROBOTS.txt? HipCrime
- Re: make people use ROBOTS.txt? Benjamin Franz
- Hip Crime Thomas Bedell
- another rare attack Rob Hartill
- another dumb robot (possibly) Rob Hartill
- ActiveAgent Rob Hartill
- Lycos' HEAD vs. GET Otis Gospodnetic
- java applet sockets John D. Pritchard
- Re: spam? (fwd) Otis Gospodnetic
- Re: spam? (fwd) Otis Gospodnetic
- hipcrime Rob Hartill
- Re: robots.txt (A *little* off the subject) Erik Selberg
- RE: make people use ROBOTS.txt? Terry Coatta
- IROS 97 Call for Papers John D. Pritchard
- Re[2]: Lycos' HEAD vs. GET Shadrach Todd
- Re: user-agent in Java Captain Napalm
- USER_AGENT and Apache 1.2 Rob Hartill
- NetJet dpp@peg.apc.org
- Standard Joseph Whitmore
- Servers vs Agents Davis, Ian
- unix robot LIAM GUINANE
- Re: Servers vs Agents Rob Hartill
- Re: Servers vs Agents Martin Kiff
- Standard? admin@superhot.com
- Cache Filler Nigel Rantor
- an article... (was: Re: Standard?) Greg Fenton
- [...]Re: Cache Filler Benjamin Franz
- Re: Cache Filler Ian Graham
- Re: an article... (was: Re: Standard?) Ian Graham
- Re: an article... (was: Re: Standard?) Rob Hartill
- Re: Servers vs Agents Art Matheny
- Re: an article... (was: Re: Standard?) Nigel Rantor
- Re: [...]Re: Cache Filler Carlos Horowicz
- Regexp Library Cook-off Tim Bunce
- Re: Standard? Nigel Rantor
- Re: robot ? Baron Timothy de Vallee
- Lets get on task! {was Re: Standard} Wes Miller
- What Is wwweb =?ISO-8859-1?Q?Alvaro_Mu=F1oz-Aycuens_Martinez?=
- Re: robot ? David Banes
- WebCrawler & Excite Otis Gospodnetic
- [Fwd: WebCrawler & Excite] Wes Miller
- Re: Standard? Nigel Rantor
- Re: WebCrawler & Excite Otis Gospodnetic
- Please take Uninvited Email discussion elsewhere Martijn Koster
- Just when you thought it might be interesting to standardize Dave Bakin
- Server Indexing -- Helping a Robot Out Ian Graham
- stingy yahoo server? Mark Norman
- IIS and If-modified-since [was Re: stingy yahoo server?] Dan Gildor
- Crawlers and "dynamic" urls David Koblas
- The Big Picture(sm): Visual Browsing in Web and non-Web Databases Gerry McKiernan
- Re: Crawlers and "dynamic" urls Klaus Johannes Rusch
- Re: IIS and If-modified-since Lee Fisher
- Merry Christmas, spidie-boyz&bottie-girlz! Santa Claus
- Merry Christmas, HipXmas-SantaSpam! Santa Claus
- USER_AGENT spoofing Rob Hartill
- Web pages being served from an SQL database Eric Mackie
- Re: Web pages being served from an SQL database Sigfrid Lundberg
- RE: Web pages being served from an SQL database Martin.Soukup
- Netscape-Catalog-Robot Rob Hartill
- It's not only robots we have to worry about ... Captain Napalm
- Re: It's not only robots we have to worry about ... Rob Hartill
- Re: Remember Canseco..... YukYukYo69@aol.com
- Re: Remember Canseco..... Lilian Bartholo
- Re: RE: Scalpers (SJPD does crack down) Tim Simmons
- FS- Sharks Tkts- Front Row (2nd Deck) Jan 13 Howard Strachman
- FS-Jan 7 Row 1 Sec 211 Anne Greene
- For Sale for 12/26! Reynolds, Cathy A
- Game tonight Karmy T. Kays
- Good HREFs vs Bogus HREFs: 80/20 mike mulligan
- Returned mail: User unknown Mail Delivery Subsystem
- We need to Shut down Roenick RaffiK98@aol.com
- RE: shot clock?!.... Mark Fullerton
- Forsberg Laura M. Sebastian
- Quick--who knows listproc? Bonnie Scott
- Error Condition Re: Invalid request listproc@plaidworks.com
- Cyclones sign MacLeod clonez1@one.net
- Error Condition Re: Invalid request listproc@plaidworks.com
- Re: WRITERS WANTED (re-post) Greg Tanzola
- ADMIN: mailing list attack :-( Martijn Koster
- RE: Netscape-Catalog-Robot Ian King
- Referencing dynamic pages Thomas Merlin
- Re: help Martijn Koster
- Excite Authors? Randy Terbush
- Do robots have to follow links ? Thomas Merlin
- Frames ? Lycos ? Thomas Merlin
- Do robots have to follow links ? Ross A. Finlayson
- Re: Frames ? Lycos ? Theo Van Dinter
- Lycos Thomas Merlin
- Meta refresh tags John Heard
- Cron <robh@us2> /usr/home/robh/show_robots (fwd) Rob Hartill
- Re: Cron <robh@us2> /usr/home/robh/show_robots (fwd) Sigfrid Lundberg
- email address grabber Dorian Ellis
- Re: email address grabber Art Matheny
- Re: email address grabber Klaus Johannes Rusch
- Re: email address grabber Jeff Drost
- Re: email address grabber Issac Roth
- robot source code v.sreekanth
- Meta Tag Article Mitch Allen
- Re: email address grabber Jeff Drost
- re Email Grabber Steve Nisbet
- Re: email address grabber Captain Napalm
- Re: email address grabber Captain Napalm
- Re: email address grabber Art Matheny
- re: email grabber Rich Dorfman
- re: email grabber Richard Gaskin
- Re: email grabber Chris Brown
- SpamBots Mitch Allen
- an image observer Martin Paff
- Re: email grabber Andy Rollins
- Re: an image observer David Steele
- Re: email grabber Wendell B. Kozak
- Re[2]: SpamBots Brad Fox
- Re: email grabber Robert Raisch, The Internet Company
- Re[3]: SpamBots
- Re: More Robot Talk (was Re: email grabber) Captain Napalm
- RE: email grabber Ian King
- Re: More Robot Talk Nick Arnett
- Too Many Admins (TMA) !!! HipCrime
- Re: More Robot Talk Theo Van Dinter
- Re: More Robot Talk bbh@xenodata.com
- escaped vs unescaped urls Dan Gildor
- Q: size of the web in bytes, comprehensive list Yossi Cohen
- "real-time" spidering by Lycos Otis Gospodnetic
- Info on large scale spidering? Nick Craswell
- Re: Info on large scale spidering? Otis Gospodnetic
- AltaVista Meta Tag Rumour Mitch Allen
- indexing via redirectors Patrick Berchtold
- fetching .map files Patrick Berchtold
- Re: indexing via redirectors Eric Miller
- Re: indexing via redirectors Theo Van Dinter
- Re: indexing via redirectors Sigfrid Lundberg
- Re: indexing via redirectors Benjamin Franz
- Re: indexing via redirectors Eric Miller
- RE: indexing via redirectors Martin.Soukup
- Meta Tags =?ISO-8859-1?Q?Alvaro_Mu=F1oz-Aycuens_Martinez?=
- Re: indexing via redirectors Jeff Drost
- Re: indexing via redirectors Captain Napalm
- Have you used the Microsoft Active-X Internet controls for Visual Basic? (Or know someone who does?) Richard Edwards
- Crawling & DNS issues Neil Cotty
- robot meta tags Dan Gildor
- Re: Crawling & DNS issues Otis Gospodnetic
- Re: Crawling & DNS issues Srinivas Padmanabhuni
- Re: Info on large scale spidering? Otis Gospodnetic
- Inktomi & large scale spidering Otis Gospodnetic
- Single tar (Re: Inktomi & large scale spidering) =?ISO-8859-1?Q?Jaakko_Hyv=E4tti?=
- Meta Tags only on home page ? Thomas Merlin
- Re: Single tar (Re: Inktomi & large scale spidering) Sigfrid Lundberg
- Re: Single tar (Re: Inktomi & large scale spidering) Martin Hamilton
- robots & copyright law Tony Rose
- Re: Single tar (Re: Inktomi & large sca Howard, Dan: CIO
- Re: Single tar (Re: Inktomi & large scale spidering) Rob Hartill
- Referencing dynamic pages Thomas Merlin
- Need help on Search Engine accuracy test. HiPromote@aol.com
- Need help again. HiPromote@aol.com
- Question about Robot.txt =?ISO-8859-1?Q?Alvaro_Mu=F1oz-Aycuens_Martinez?=
- Re: Analysing the Web (was Re: Info on large scale spidering?) Patrick Berchtold
- Robot Specifications. Paul Bingham
- Agent Specification Paul Bingham
- specialized searches Mike Fresener
- NaughtyRobot Martijn Koster
- Unfriendly robot at 192.115.187.2 Tim Holt
- Information about AltaVista and Excite Huaiyu Liu
- FILEZ bbh@xenodata.com
- Re: Single tar (Re: Inktomi & large scale spidering) Martin Hamilton
- i need a bot! Myles Weissleder
- robots: lycos's t-rex: strange behaviour Dinesh
- WININET caching Martin.Soukup
- Welcome to cypherpunks Majordomo@toad.com
- Re: Welcome to cypherpunks Skip Montanaro
- More with the Cypherpunk antics Captain Napalm
- The Metacrawler, Reborn Paul Phillips
- Java and robots... Manuel J. Kwak
- Lack of support for "If-Modified-Since" Howard, Dan: CIO
- Re: Lack of support for "If-Modified-Since" Rob Hartill
- How to get the document info ? mannina bruno
- Thanks! Manuel Jesus Fernandez Blanco
- RE: How to get the document info ? Howard, Dan: CIO
- Re: message to USSA House of Representatives autoresponder@WhiteHouse.gov
- New Site Aaron Stayton
- New Site Aaron Stayton
- Re: message to USSA Senate autoresponder@WhiteHouse.gov
- Re: More with the Cypherpunk antics Benjamin Franz
- Re: More with the Cypherpunk antics Sigfrid Lundberg
- Re: More with the Cypherpunk antics Rob Hartill
- Re: More with the Cypherpunk antics Jeff Drost
- AMDIN: The list is dead Martijn Koster
- [3]RE>[5]RE>Checking Log fi Roger Dearnaley
Last message date: Thu 18 Dec 1997 - 14:33:60 PDT
Archived on: Sun Aug 17 1997 - 19:13:25 PDT
This archive was generated by hypermail 1.02.