Free scraper software for linux

ScraperPOD is a framework for scraping results from search engines.

SYNOPSIS

use WWW::Scraper;

# Name your Scraper module / search engine as the first parameter,
use WWW::Scraper(eBay);
# or in the new() method
$scraper = new WWW::Scraper(eBay);

Classic WWW::Search mode

# Use a Scraper engine just as you would a WWW::Search engine.
$scraper = new WWW::Scraper(carsforsale, Honda, { lbxModel => Accord, lbxVehicleYearFrom => 1998 });
while ( $response = $scraper->next_result() ) {
# harvest results via hash-table reference.
print $scraper->{sellerPhoneNumber};
}

Canonical Request/Response mode (not yet implemented)

$scraper = new WWW::Scraper(carsforsale, Request => Autos, Response => Autos);
# or, since carsforsale.pm defaults to the Request and Response classes of Autos
$scraper = new WWW::Scraper(carsforsale);
#
# Set field values via field-named canonical access methods.
$scraper->scraperRequest->make(Honda);
$scraper->scraperRequest->model(Accord);
$scraper->scraperRequest->minYear(1998);
#
# Note: this is *not* next_result().
while ( $response = $scraper->next_response() ) {
#
# harvest results via field-named access methods.
print $response->sellerPhoneNumber();
}

Variant Requests to a single search engine

$scraper = new WWW::Scraper(carsforsale);
$scraper->scraperRequest->make(Honda);
$scraper->scraperRequest->minYear(1998);
#
for ( $model = (Accord Civic) ) {
$scraper->scraperRequest->model($model);
$response = $scraper->next_response() ) {
# all response fields are returned as a reference to the value.
print ${$response->sellerPhoneNumber()};
}

Single Request to variant search engines

# Set the request parameters in a Request object (sub-class Autos).
$request = new WWW::Scraper::Request(Autos);
$request->make(Honda);
$request->model(Accord);
$request->minYear(1998);
#
for ( $searchEngine = (carsforsale 1001cars) ) {
$scraper = new WWW::Scraper($searchEngine, Request => $request);
for ( $response = $scraper->next_response() ) {
# all response fields are returned as a reference to the value.
print ${$response->sellerPhoneNumber()};

Text::Scraper contains structured data from (un)structured text.

SYNOPSIS

use Text::Scraper;

use LWP::Simple;
use Data::Dumper;

#
# 1. Get our template and source text
#
my $tmpl = Text::Scraper->slurp(*DATA);
my $src = get(http://search.cpan.org/recent) || die $!;

#
# 2. Extract data from source
#
my $obj = Text::Scraper->new(tmpl => $tmpl);
my $data = $obj->scrape($src);

#
# 3. Do something really neat...(left as excercise)
#
print "Newest Submission: ", $data->[0]{submissions}[0]{name}, "nn";
print "Scraper model:n", Dumper($obj), "nn";
print "Parsed model:n", Dumper($data) , "nn";

__DATA__

< div class=path>< center>< table>< tr>
< ?tmpl stuff pre_nav ?>
< td class=datecell>< span>< big>< b> < ?tmpl var date_string ?> < /b>< /big>< /span>< /td>
< ?tmpl stuff post_nav ?>
< /tr>< /table>< /center>< /div>

< ul>
< ?tmpl loop submissions ?>
< li>< a href="< ?tmpl var link ?>">< ?tmpl var name ?>< /a>
< ?tmpl if has_description ?>
< small> -- < ?tmpl var description ?>< /small>
< ?tmpl end has_description ?>
< /li>
< ?tmpl end submissions ?>
< /ul>

ABSTRACT

Text::Scraper provides a fully functional base-class to quickly develop Screen-Scrapers and other text extraction tools. Programmatically generated text such as dynamic webpages are trivially reversed engineered.

Using templates, the programmer is freed from staring at fragile, heavily escaped regular expressions, mapping capture groups to named variables or wrestling with the DOM and badly formed HTML. In addition, extracted data can be hierarchical, which is beyond the capabilities of vanilla regular expressions.

Text::Scrapers functionality overlaps some existing CPAN modules - Template::Extract and WWW::Scraper.
Text::Scraper is much more lightweight than either and has a more general application domain than the latter. It has no dependencies on other frameworks, modules or design-decisions. On average, Text::Scraper benchmarks around 250% faster than Template::Extract - and uses significantly less memory.

Unlike both existing modules, Text::Scraper generalizes its functionality to allow the programmer to refine template capture groups beyond (.*?), fully redefine the template syntax and introduce new template constructs bound to custom classes.

WWW::Scraper::Dice Perl module contains Scrapes Dice : (skills,locations) => (title, location ,residue).

SYNOPSIS

use WWW::Search;
my $oSearch = new WWW::Scraper(Dice);
my $sQuery = WWW::Scraper::escape_query("unix and (c++ or java)");
$oSearch->native_query($sQuery,
{method => bool,
state => CA,
daysback => 14});
while (my $res = $oSearch->next_result()) {
if(isHitGood($res->url)) {
my ($company,$title,$date,$location) =
($res->company, $res->title, $res->date, $res->location);
print "$company $title $date $location " . $res->url . "n";
}
}

WWW::Scraper::FlipDog it Scrapes www.FlipDog.com.

SYNOPSIS

use WWW::Scraper;
use WWW::Scraper::Response::Job;

$search = new WWW::Scraper(FlipDog);

$search->setup_query($query, {options});

while ( my $response = $scraper->next_response() ) {
# $response is a WWW::Scraper::Response::Job.
}

FlipDog extends WWW::Scraper.
It handles making and interpreting FlipDog searches of http://www.FlipDog.com.

WWW::Scraper::Monster is a Perl module that scrapes Monster.com.

SYNOPSIS

use WWW::Search;
my $oSearch = new WWW::Search(Monster);
my $sQuery = WWW::Search::escape_query("unix and (c++ or java)");
$oSearch->native_query($sQuery,
{st => CA,
tm => 14d});
while (my $res = $oSearch->next_result()) {
print $res->company . "t" . $res->title . "t" . $res->change_date
. "t" . $res->location . "t" . $res->url . "n";
}

This class is a Monster specialization of WWW::Search. It handles making and interpreting Monster searches at http://www.monster.com. Monster supports Boolean logic with "and"s "or"s. See http://jobsearch.monster.com/jobsearch_tips.asp for a full description of the query language.

The returned WWW::Scraper::Response objects contain url, title, company, location and change_date fields.

WWW::Scraper::NorthernLight it Scrapes NorthernLight.com.

SYNOPSIS

require WWW::Scraper;
$search = new WWW::Scraper(NorthernLight);

This class is an NorthernLight specialization of WWW::Search. It handles making and interpreting NorthernLight searches http://www.NorthernLight.com.

This class exports no public interface; all interaction should be done through WWW::Search objects.

WWW::Search::Scraper::BAJobs it Scrapes BAJobs.com.

SYNOPSIS

require WWW::Search::Scraper;
$search = new WWW::Search::Scraper(BAJobs);

This class is an BAJobs specialization of WWW::Search. It handles making and interpreting BAJobs searches http://www.BAJobs.com.

This class exports no public interface; all interaction should be done through WWW::Search objects.

WWW::Search::Scraper::Dice is a Perl module that scrapes Dice : (skills,locations) => (title, location ,residue)

SYNOPSIS

use WWW::Search;
my $oSearch = new WWW::Search::Scraper(Dice);
my $sQuery = WWW::Search::Scraper::escape_query("unix and (c++ or java)");
$oSearch->native_query($sQuery,
{method => bool,
state => CA,
daysback => 14});
while (my $res = $oSearch->next_result()) {
if(isHitGood($res->url)) {
my ($company,$title,$date,$location) =
($res->company, $res->title, $res->date, $res->location);
print "$company $title $date $location " . $res->url . "n";
}
}

This class is a Dice extension of WWW::Search::Scraper. It handles making and interpreting Dice searches at http://www.dice.com.