Free search query software for linux

Search::Lemur is a Perl class to query a Lemur server, and parse the results.

SYNOPSYS

use Search::Lemur;

my $lem = Search::Lemur->new("http://url/to/lemur.cgi");

# run some queries, and get back an array of results
# a query with a single term:
my @results1 = $lem->query("encryption");
# a query with two terms:
my @results2 = $lem->query("encryption MD5");

# get corpus term frequency of MD5:
my $md5ctf = $results2[1]->ctf();

This module will make it easy to interact with a Lemur Toolkit for Language Modeling and Information Retrieval server for information retreival exercises. For more information on Lemur, see http://www.lemurproject.org/.

This module takes care of all parsing of responses from the server. You can just pass a query as a space-separated list of terms, and the module will give you back an array of result objects.

KinoSearch is a search engine library.
SYNOPSIS
First, write an application to build an inverted index, or "invindex", from your document collection.
use KinoSearch::InvIndexer;
use KinoSearch::Analysis::PolyAnalyzer;
my $analyzer
= KinoSearch::Analysis::PolyAnalyzer->new( language => en );
my $invindexer = KinoSearch::InvIndexer->new(
invindex => /path/to/invindex,
create => 1,
analyzer => $analyzer,
);
$invindexer->spec_field(
name => title,
boost => 3,
);
$invindexer->spec_field( name => bodytext );
while ( my ( $title, $bodytext ) = each %source_documents ) {
my $doc = $invindexer->new_doc;
$doc->set_value( title => $title );
$doc->set_value( bodytext => $bodytext );
$invindexer->add_doc($doc);
}
$invindexer->finish;
Then, write a second application to search the invindex:
use KinoSearch::Searcher;
use KinoSearch::Analysis::PolyAnalyzer;
my $analyzer
= KinoSearch::Analysis::PolyAnalyzer->new( language => en );
my $searcher = KinoSearch::Searcher->new(
invindex => /path/to/invindex,
analyzer => $analyzer,
);
my $hits = $searcher->search( query => "foo bar" );
while ( my $hit = $hits->fetch_hit_hashref ) {
print "$hit->{title}n";
}
Main features:
- Extremely fast and scalable - can handle millions of documents
- Incremental indexing (addition/deletion of documents to/from an existing index).
- Full support for 12 Indo-European languages.
- Support for boolean operators AND, OR, and AND NOT; parenthetical groupings, and prepended +plus and -minus
- Algorithmic selection of relevant excerpts and highlighting of search terms within excerpts
- Highly customizable query and indexing APIs
- Phrase matching
- Stemming
- Stoplists

pro-search is a crawler for FTP servers, SMB shares, HTTP servers, and DC++ networks.

Search::Tools are tools for building search applications.

SYNOPSIS

use Search::Tools;

my $re = Search::Tools->regexp(query => the quick brown fox);

my $snipper = Search::Tools->snipper(query => $re);

my $hiliter = Search::Tools->hiliter(query => $re);

for my $result (@search_results)
{
print $hiliter->light( $snipper->snip( $result->summary ) );
}

Search::Tools is a set of utilities for building search applications. Rather than adhering to a particular search application, the goal of Search::Tools is to provide general-purpose methods for common search application features. Think of Search::Tools like a toolbox rather than a hammer.

Examples include:

Parsing search queries for the meaningful keywords
Rich regular expressions for locating keywords in the original indexed documents
Contextual snippets showing query keywords
Highlighting of keywords in context

Search::Tools is derived from some of the features in HTML::HiLiter and SWISH::HiLiter, but has been re-written with an eye to accomodating more general purpose features.

Yahoo::Search::Request is a container object for a Yahoo! Search request. (This package is included in, and automatically loaded by, the Yahoo::Search package.)

Package Use

You never need to use this package directly -- it is loaded automatically by Yahoo::Search.

Object Creation

In practice, this class is generally not dealt with explicitly, but rather implicitly via functions in Yahoo::Search such as Query and Links, which build and use a Request object under the hood.

You also have access to the Request object via Request() method of resulting Response and Result objects.

To be clear, Request objects are created by the Request() method of a Search Engine object (Yahoo::Search).

Search::ContextGraph is a Perl module for spreading activation search engine.

SYNOPSIS

use Search::ContextGraph;

my $cg = Search::ContextGraph->new();

# first you add some documents, perhaps all at once...

my %docs = (
first => [ elephant, snake ],
second => [ camel, pony ],
third => { snake => 2, constrictor => 1 },
);

$cg->bulk_add( %docs );

# or in a loop...

foreach my $title ( keys %docs ) {
$cg->add( $title, $docs{$title} );
}

# or from a file...

my $cg = Search::ContextGraph->load_from_dir( "./myfiles" );

# you can store a graph object for later use

$cg->store( "stored.cng" );

# and retrieve it later...

my $cg = ContextGraph->retrieve( "stored.cng" );


# SEARCHING

# the easiest way

my @ranked_docs = $cg->simple_search( peanuts );


# get back both related terms and docs for more power

my ( $docs, $words ) = $cg->search(snake);


# you can use a document as your query

my ( $docs, $words ) = $cg->find_similar(First Document);


# Or you can query on a combination of things

my ( $docs, $words ) =
$cg->mixed_search( { docs => [ First Document ],
terms => [ snake, pony ]
);


# Print out result set of returned documents
foreach my $k ( sort { $docs->{$b} $docs->{$a} }
keys %{ $docs } ) {
print "Document $k had relevance ", $docs->{$k}, "n";
}

# Reload it
my $new = Search::ContextGraph->retrieve( "filename" );

Spreading activation is a neat technique for building search engines that return accurate results for a query even when there is no exact keyword match. The engine works by building a data structure called a context graph, which is a giant network of document and term nodes. All document nodes are connected to the terms that occur in that document; similarly, every term node is connected to all of the document nodes that term occurs in. We search the graph by starting at a query node and distributing a set amount of energy to its neighbor nodes. Then we recurse, diminishing the energy at each stage, until this spreading energy falls below a given threshold. Each node keeps track of accumulated energy, and this serves as our measure of relevance.

This means that documents that have many words in common will appear similar to the search engine. Likewise, words that occur together in many documents will be perceived as semantically related. Especially with larger, coherent document collections, the search engine can be quite effective at recognizing synonyms and finding useful relationships between documents. You can read a full description of the algorithm at http://www.nitle.org/papers/Contextual_Network_Graphs.pdf.

The search engine gives expanded recall (relevant results even when there is no keyword match) without incurring the kind of computational and patent issues posed by latent semantic indexing (LSI). The technique used here was originally described in a 1981 dissertation by Scott Preece.

WWW::Search::GoTo is Perl class for searching GoTo.com.

SYNOPSIS

require WWW::Search;
$search = new WWW::Search(GoTo);

This class is an GoTo specialization of WWW::Search. It handles making and interpreting GoTo searches www-GoTo.com.

Nothing special about GoTo: no search options. It is much like Google in that it attempts to returm relavent search results using simple queries.

This class exports no public interface; all interaction should be done through WWW::Search objects.