Open-Source Crawlers

• DataparkSearch is a crawler and search engine released under the GNU General Public License.
• GNU Wget is a command-line operated crawler written in C and released under the GPL. It is typically used to mirror web and FTP sites.
• Heritrix is the Internet Archive’s archival-quality crawler, designed for archiving periodic snapshots of a large portion of the Web. It was written in Java.
• ht://Dig includes a web crawler in its indexing engine.
• HTTrack uses a Web crawler to create a mirror of a web site for off-line viewing. It is written in C and released under the GPL.
• ICDL Crawler is a cross-platform web crawler written in C++ and intended to crawl web sites based on ICDL standard using computer’s free CPU resources only.
• JSpider is a highly configurable and customizable web spider engine released under the GPL.
• Larbin by Sebastien Ailleret
• Webtools4larbin by Andreas Beder
• Methabot is a speed-optimized web crawler and command line utility written in C and released under a 2-clause BSD License. It features a wide configuration system, a module system and has support for targeted crawling through local filesystem, HTTP or FTP.
• Nutch is a crawler written in Java and released under an Apache License. It can be used in conjunction with the Lucene text indexing package.
• Pavuk is a command line web mirror tool with optional X11 GUI crawler and released under the GPL. It has bunch of advanced features compared to wget and httrack, eg. regular expression based filtering and file creation rules.
• WebVac is a crawler used by the Stanford WebBase Project.
• WebSPHINX (Miller and Bharat, 199 is composed of a Java class library that implements multi-threaded web page retrieval and HTML parsing, and a graphical user interface to set the starting URLs, to extract the downloaded data and to implement a basic text-based search engine.
• WIRE - Web Information Retrieval Environment (Baeza-Yates and Castillo, 2002) is a web crawler written in C++ and released under the GPL, including several policies for scheduling the page downloads and a module for generating reports and statistics on the downloaded pages so it has been used for web characterization.
• LWP::RobotUA (Langheinrich , 2004) is a Perl class for implementing well-behaved parallel web robots distributed under Perl5’s license.
• Web Crawler Open source web crawler class for .NET (written in C#).
• Sherlock Holmes Sherlock Holmes gathers and indexes textual data (text files, web pages, …), both locally and over the network. Holmes is sponsored and commercially used by the Czech web portal Centrum. It is also used by Onet.pl.
• YaCy, a free distributed search engine, built on principles of peer-to-peer networks (licensed under GPL).
• Ruya Ruya is an Open Source, high performance breadth-first, level-based web crawler. It is used to crawl English and Japanese websites in a well-behaved manner. It is released under the GPL and is written entirely in the Python language. A SingleDomainDelayCrawler implementation obeys robots.txt with a crawl delay.
• Universal Information Crawler Fast developing web crawler. Crawls Saves and analyzes the data.
• Agent Kernel A Java framework for schedule, thread, and storage management when crawling.
• Spider News, Information regarding building a spider in perl.
• Arachnode.NET, is an open source promiscuous Web crawler for downloading, indexing and storing Internet content including e-mail addresses, files, hyperlinks, images, and Web pages. Arachnode.net is written in C# using SQL Server 2005 and is released under the GPL.
• dine is a multithreaded Java HTTP client/crawler that can be programmed in JavaScript released under the LGPL.