Crawlers, Robots and Spiders

• ht://Dig

  Indexing and searching system for a small domain or intranet.

  www.htdig.org

• BotSpot

  Directory of bots and bot resources for Windows and Mac.

  www.botspot.com

• Web Robots Pages, The

  Lists Web Robot FAQs, databases, and mailing lists.

  www.robotstxt.org/wc/robots.html

• Google Sitemaps

  Collaborative crawling system that enables webmasters to communicate directly with Google to keep the search engine informed of all their web pages, and when changes are made to these pages.

  /www.google.com/webmasters/sitemaps/login

• Googlebot

  Web-crawling robot that collects documents from the web to build a searchable index for the Google search engine.

  www.google.com/bot.html

• BotKnowledge.com

  Directory of intelligent software agents, knowbots, and bots. Includes FAQ and newsletter.

  www.botknowledge.com

• SpiderHunter.com

  Demonstrates how to write cloaking scripts and track spiders.

  www.spiderhunter.com

• Grub

  Open source, distributed Internet crawler.

  www.grub.org

• MSNBot

  Web-crawling robot developed by MSN Search that collects documents from the web to build a searchable index.

  search.msn.com/docs/siteowner.aspx

• Searchbots

  Offers bots with different search programs, or allows users to customize their own.

  www.searchbots.net

• Robotcop

  An open source module for webservers which helps webmasters prevent spiders from accessing parts of their sites they have marked off limits.

  www.robotcop.org

• Collegebot

  Dedicated to indexing and searching education and academic related pages.

  www.collegebot.com

• Webglimpse

  Search engine software that consists of Glimpse, an engine for indexing and searching text files on a Unix system and Webglimpse, the spider and manager for the files to be searched.

  www.webglimpse.net

• RoboGen

  Program to generate a robot exclusion file, robots.txt, for your web site, which controls the files appearing in search engines.

  www.rietta.com/robogen

• Peregrinator: A Web-Indexing Robot

  A robot for traversing and indexing sections of the Web.

  www.maths.usyd.edu.au:8000/jimr/pe/Peregrinator.html

• SG-Scout Home Page

  Developed for the Xerox Palo Alto Research Center as part of a project involving the development of a new form of directed WWW browser.

  www-swiss.ai.mit.edu/~ptbb/SG-Scout/SG-Scout.html

• Spider Cloak

  Offers search engine cloaking script that hides the source code to prevent theft and generates optimized marketing pages for submission.

  www.spidercloak.com

• WWWMM Robot (W3M2)

  A wanderer robot written for several experimental purposes, but more especially to study throughput/latency on paths in the net.

  tronche.com//W3M2