Free spider software for linux

TkSpider project is a game of solitaire is played with the equivalent of two packs of cards.

The last release includes several small new features, including a redo feature.

Some code, and the card images, taken from Jeff Godreys "Once in a Lifetime".

Spider, based on the card-game shipped with Windows XP

WWW::Orkut::Spider is a Perl extension for spidering the orkut community.

SYNOPSIS

use WWW::Orkut::Spider;
my $orkut = WWW::Orkut::Spider->new;
$orkut->login($user,$pass);
$orkut->get_hisfriends($uid);
print $orkut->get_xml_profile($uid);

WWW::Orkut::Spider uses WWW:Mechanize to scrape orkut.com.
Output is a simple xml format containing friends, communities and profiles for a given Orkut UID.

- Access to orkut.com via WWW::Mechanize
- Collects UIDs
- Fetches Profiles/Communities/Friends for a given UID
- Output via simple xml format

WWW::Spyder is a Perl module that acts like a web spider.

A web spider that returns plain text, HTML, and other information per page crawled and can determine what pages to get and parse based on supplied terms compared to the text in links as well as page content.

METHODS

$spyder->new()

Construct a new spyder object. Without at least the seed() set, or go_to_seed() turned on, the spyder isnt ready to crawl.

$spyder = WWW::Spyder->new(shift||die"Gimme a URL!n");
# ...or...
$spyder = WWW::Spyder->new( %options );

Options include: sleep_base (in seconds), exit_on (hash of methods and settings). Examples below.

$spyder->seed($url)

Adds a URL (or URLs) to the top of the queues for crawling. If the spyder is constructed with a single scalar argument, that is considered the seed_url.

$spyder->bell([bool])

This will print a bell ("a") to STDERR on every successfully crawled page. It might seem annoying but it is an excellent way to know your spyder is behaving and working. True value turns it on. Right now it cant be turned off.

$spyder->spyder_time([bool])

Returns raw seconds since Spyder was created if given a boolean value, otherwise returns "D day(s) HH::MM:SS."

$spyder->terms([list of terms to match])

The more terms, the more the spyder is going to grasp at. If you give a straight list of strings, they will be turned into very open regexes. E.g.: "king" would match "sulking" and "kinglet" but not "King." It is case sensitive right now. If you want more specific matching or different behavior, pass your own regexes instead of strings.

$spyder->terms( qr/bkings?b/i, qr/bqueens?b/i );

terms() is only settable once right now, then its a done deal.

$spyder->spyder_data()

A comma formatted number of kilobytes retrieved so far. Dont give it an argument. Its a set/get routine.

$spyder->slept()

Returns the total number of seconds the spyder has slept while running. Useful for getting accurate page/time counts (spyder performance) discounting the added courtesy naps.

$spyder->UA->...

The LWP::UserAgent. You can reset them, I do believe, by calling methods on the UA. Here are the initialized values you might want to tweak (see LWP::UserAgent for more information):

$spyder->UA->timeout(30);
$spyder->UA->max_size(250_000);
$spyder->UA->agent(Mozilla/5.0);

Changing the agent name can hurt your spyder b/c some servers wont return content unless its requested by a "browser" they recognize.

You should probably add your email with from() as well.

$spyder->UA->from(bluefintuna@fish.net);
$spyder->cookie_file([local_file])

They live in $ENV{HOME}/spyderCookie by default but you can set your own file if you prefer or want to save different cookie files for different spyders.

DirIndexFaker script is writen in PHP and its designed to produce fake Apache directory listings for the purpose of slowing down and overloading with false positives the Web spiders used by the RIAA, MPAA, etc.
People like the riaa, mpaa, and others are on a copyright enforcing rampage, destroying innocent victims along the way. They are using automated tools (web spiders) to find people hosting illegal content to sue. Sometimes the spiders catch innocent people in their web of evil.
Since our politicians think the RIAAa well-being is more important than ours, we must find a way to make the RIAA/MPAAs spiders too expensive to operate. Therefore our goals should be to:
- Slow the spider down, or get it stuck in a loop
- Provide soo many false positives, that sorting the actual infringers from the innocent is too expensive to allow the copywrong police to continue
These things are what a honeypot, or tarpit is designed to do. There are several available RIAA/MPAA spider trapping scripts currently available, but they all have unacceptable limitations (either requirements are too high, or they take an unacceptable toll on your server).
What was needed was a script which could generate fake apache index pages, but with links to large files with copyrighted sounding names. The server operator should not have to have root, nor should it waste excessive disk space for the server operator, IE - the files should be generated by the script, and not actually stored on the servers disk. This is what DirIndexFaker does!
The best existing script I could find which came close to meeting these criteria was the DMCA Bot Killer , but it had several problems:
- It requires the files to be generated beforehand with a perl script, the code is in the source, but commented out and a little wonky.
- It does not look like an apache index page, it looks suspicious, the **AAs spiders could be easily modified to detect this.
- It requires a list of filenames to use when generating our warez index. This list is loaded from a server at every invocation. This is innefficient, and error-prone.
So DirIndexFaker is a fixed version of the DMCA Bot Killer.
Usage:
Simply extract the contents of the .zip file to a subfolder under your PHP enabled webservers document root, then place a link to that subfolder somewhere on your site where bots can see it, and people cannot. One way to hide the link from humans is with CSS.
Note: You may be tempted to keep the search engines away with your robots.txt file, but is is well known that the RIAA is using the search engines to find infringing content, so this is not a good idea.
Enhancements:
- This release adds the ability to slow down RIAA/MPAA bots with a usleep call.
- This is enabled by default, but can be disabled by commenting out a line at the top of index.php.
- The bottom row has been fixed to make server info match the actual server version running on your host.
- The random seed has been changed so that filenames/sizes will only change once a day.
- This makes it harder to detect as a fake.

MKSearch provides a Web metadata spider and search engine.

MKSearch is a metadata search engine that indexes structured metadata in Web documents instead of free text in the document body.

The data acquisition system conforms to the Dublin Core metadata in HTML recommendations, and supports other application profiles, such as the UK e-Government Metadata Standard.

It also indexes native RDF formats, including RSS 1.0. The system has five major components: a Web crawler, an HTML document validator and formatter, a set of custom indexers, an RDF storage and query system, and a public query interface, provided through a standard servlet container.

System composition

The MKSearch system is composed of several other free software components. Further details are provided in the MKSearch development plans.

JSpider
JSpider is a Java Web crawler engine that has pluggable interfaces that can be used to add custom processing and content handling. MKSearch uses custom SAX-based content handlers for extracting metadata from Web documents.

Sesame
Sesame is a set of RDF processing and storage APIs and applications that includes RDF data query facilities. MKSearch uses Sesame to store indexed metadata in RDF format and to search the repository via the public query interface.

JTidy
JTidy is a utility for correcting common HTML markup errors and is used to convert HTML documents to XHTML so they can be processed using SAX.

PHPSPELLBOOK is a suite of tools for Web site promotion, diagnosis, and improvement.
PHPSPELLBOOK provides Web masters with tools such as advertisiment submission, a mass mailer, a broken link checker, a link exchange checker, fake click generators, anonymizer tools, a calendar, and more.
Main features:
- All tools could perform operations in anonymity relaying on Proxy/Socks4/Socks5 lists.
- Advertisement submissions tools
- SendMail: mass mailer
- Broken Link Checker
- Clicker tools [fake hits generators]
- Domain Spider
- HTTP Header Checker
- Link Exchange Checker
- Meta Tags Tools
- Page Ranking Tools [pagerank checker)
- Partners Finding Tools [find websites in a given pagerank range]
- E-Mail handling and parsing tools
- Translation Tools
- Url handling tools and generators
- Calendar
- Anonymous hosts list administration
- Anonymous host checker [proxy/socks4/socks5 checker]
- Jobs administration
- Smtp servers list administration
- Statistics
Enhancements:
- This release adds documentation updates, a code cleanup, new sources for socks4, socks5, and proxy lists, an installation script fix, a link popularity checker, and a partners finding tool.

PROPS is an open, extensible Internet publishing system designed specifically for periodicals such as newspapers and magazines who want to publish online, either exclusively or as an extension of their print publication.
PROPS is written entirely in PHP4 using a MySQL database backend, and is free software released under the GNU General Public License (GPL).
Main features:
- Easy to install
- Well-documented, robust API allowing third party developers to create plug-ins to extend base functionality
- Delivery of content to multiple target platforms (HTML, XML/XSL, WAP/WML, text, etc)
- Readers may format a story for printing, or email to a friend
- Pages are presented as static URLs so that spider type search engines may crawl and index them
- Strict separation of design and content - designers control site look and feel by developing templates using standard site editing tools such as Dreamweaver, GoLive, BBEdit, etc., while editors manage site content via a web interface
- Permissions-based multiuser site management screens allow a PROPS site to be maintained by a distributed team of reporters and editors
- Will support the XML-based News Industry Text Format
- Will speak various XML dialects for both import and export, allowing syndication in both directions, and enabling PROPS sites to participate in distributed news networks
- Is targeted at mid-sized publishers serving fewer than one million page views daily, however will be architected to scale through deployment of multiple front-end web servers, separate image servers, etc.