Free crawler software for linux

PyGalleryCrawler project is a Web crawler for online image galleries.
Installation:
tar -xzf pygallerycrawler.tar.gz
cd pygallerycrawler
Extra python modules
psyco @ http://psyco.sourceforge.net
- performance
Python Imaging Library aka PIL @ http://www.pythonware.com/products/pil/
- thumbnails generation
- size verification
feedparser @ http://feedparser.org
- feed parser
Use:
chmod a+x pygallerycrawler.py
./pygallerycrawler.py the_url_you_want_crawl
Personal configuration:
If you make change in config.py, your changes will be overwrite at the next update. So, you can create a personal configuration and use it with the --config (or -c) switch.
cp config.py ~/pgc_config.py
vi ~/pgc_config.py
./pygallerycrawler.py -c ~/pgc_config.py the_url_you_want_crawl
Version restrictions:
- No check if some pictures are the same after download. Some gallery have a presentation link wich is one of the pictures. So the images will be double.

pro-search is a crawler for FTP servers, SMB shares, HTTP servers, and DC++ networks.

Fast File Search is a crawler for FTP servers and SMB shares that can be found on Windows or UNIX systems running Samba.
It provides a web interface for searching files. It is optimized for searching files by a wildcard when there are some normal (not * or ?) chars specified in the beginning or in the end of the mask (for example *.iso).
Fast File Search crawler runs on UNIX (currently only Linux has been tested but I do not now any reasons why it should not work on other UNIXes). Fast File Search uses MySQL database, web interface needs a web server with PHP >= 4.0.3 and the crawler needs some perl modules.
The crawler (ffsearch.pl) crawls the network (FTP servers from the list and all reachable SMB hosts on the local network) and stores the information about files into database. It is invoked at certain times each day via crontab entries.
There are two modes of operation of the crawler: complete crawl and incremental crawl. The crawler expects a command line argument that tells crawler which mode to run (-c or --complete for complete crawl, -i or --incremental for incremental crawl). Both modes retrieve a list of the active SMB hosts in all workgroups.
The complete crawl tries to scan all active hosts and all hosts that are listed in database. The complete crawl should be run once a day.
The incremental crawl tries to scan active hosts and hosts listed in database that have not been scanned since the last complete crawl because they were unreachable. The incremental crawl should be run several times a day, for example each 3 hours.
How does the crawler get know whether the host has been crawled since the last complete crawl?
Each time the complete crawl is executed, the expire count is incremented first. When the host is crawled, expire count is set to zero. So all hosts whose expire count > 0 were not reachable since the last complete crawl. Moreover, when expire count reaches value specified in configuration (i.e. it was unreachable during the time period of complete crawls) the information about files on the "expired" host is deleted from database.
Web interface is used to search the files in database, details how to search are described in the Help section of the search page.
You can also add a FTP server to a FTP server list, edit FTP server in the list or delete FTP server from the list through the web interface. So that anybody could not do anything with the server list only the record about abcdef is editable from host abcdef. There are also admins who can edit all records in the server list. The admins login through the web interface.
Enhancements:
- fixed few bugs in the crawler
- added a possibility to exclude some SMB shares
- www: improved Russian and Ukrainian translation

Tales of Middle Earth (ToME) is a fantasy adventure game, based on the works of J.R.R. Tolkien. It is a game that emphasizes intricate, challenging, and varied gameplay over graphics.
Hundreds of different monsters in randomly-generated, unpredictable dungeons will strive to slay you by various means, and you counter - if you survive - by developing the skills of your choice and wielding mighty artifacts.
ToMEs races from Hobbit to Troll and classes from Swordmaster to Summoner allow for many different playing styles and a replay value that extends through years.
The only game so realistic that your scrolls and spell books will burn if you trudge in lava (unless you have gained immunity from some armour), you will dry up rivers to cast mighty spells, strike at orcs with blades attuned to slay them specifically, summon armies from a simple totem, and even forge your own artifacts.
There is an entire community to help you with the game, and it is far from static - ToME is a developing game, always improving. And best of all, its free.
Enhancements:
- Fix window position saving on Mac OS
- Remove buggy trap of Stair Movement
- Fix typo in one monsters flags
- Fix word wrapping in character sheet

BitTorrent Queue Manager is a console-based BitTorrent client running on top of BitTornado that provides built-in queue management functions.
BitTorrent Queue Manager also provides a remote interface compatible with ABC for Web-based control. Furthermore, peer information can be queried, including country and network names, and a built-in crawler can gather new torrents on specified trackers or catalog sites for downloading automatically.
This is the new beginning of BTQueue. By upgrading to 0.1.0, you are able to:
- Utilize DHT network compatible to Bram Cohens client and BitComet
- Query IP location from updated database plus AS number (see ip2cc)
- Change client identifier to Azureus, Bram Cohens client, or BitComet

jGetFile project is geared towards mass downloading specifically non-html files from the web. Web crawlers and website downloads are widely available and work very well. Not all web-based file crawlers are equal however.

Few file crawers handle this href scenario: < a href="http://www.foo.com?url=http://youreallywantthislink.com/files" >< /a > jGetFile was engineered to be able to handle as many extraneous href configurations as possible. Although it does not handle all possible cases, more will be supported in future releases.

jGetFile supports a highly configurable means to filter the acceptance of links within the program. A user can use the -i or -e options to have the program only traverse links that start with the addresses the user specified, or to exclude links that start with the specified addresses.

Alternatively, for the powerusers, one can specify a BeanShell script through the -als option. This allows for abritrarily complex rules to be specified for accepting links, like accept only links at depth 1 that begin with www.blah.com, exclude links at level 2 that begin with www.foo.com, and exclude links at level 2 that contain word cat. The depth variable is currently not available to custom scripts, but will be in the next release.

There is no gui in the works, and because of the intended simplicity of this program, one is not planned in the future either. Also, jGetFile is not intended to replace wget, although its initial features were based on wget. jGetFile has the single goal of downloading files fast and efficiently from the web.

MKSearch provides a Web metadata spider and search engine.

MKSearch is a metadata search engine that indexes structured metadata in Web documents instead of free text in the document body.

The data acquisition system conforms to the Dublin Core metadata in HTML recommendations, and supports other application profiles, such as the UK e-Government Metadata Standard.

It also indexes native RDF formats, including RSS 1.0. The system has five major components: a Web crawler, an HTML document validator and formatter, a set of custom indexers, an RDF storage and query system, and a public query interface, provided through a standard servlet container.

System composition

The MKSearch system is composed of several other free software components. Further details are provided in the MKSearch development plans.

JSpider
JSpider is a Java Web crawler engine that has pluggable interfaces that can be used to add custom processing and content handling. MKSearch uses custom SAX-based content handlers for extracting metadata from Web documents.

Sesame
Sesame is a set of RDF processing and storage APIs and applications that includes RDF data query facilities. MKSearch uses Sesame to store indexed metadata in RDF format and to search the repository via the public query interface.

JTidy
JTidy is a utility for correcting common HTML markup errors and is used to convert HTML documents to XHTML so they can be processed using SAX.