Free indexing tool software for linux

PDL::Indexing Perl module contains a tutorial on how to index piddles.

This manpage should serve as a first tutorial on the indexing and threading features of PDL.

This manpage is still in alpha development and not yet complete. "Meta" comments that point out deficiencies/omissions of this document will be surrounded by square brackets ([]), e.g. [ Hopefully I will be able to remove this paragraph at some time in the future ]. Furthermore, it is possible that there are errors in the code examples. Please report any errors to Christian Soeller (c.soeller@auckland.ac.nz).

Still to be done are (please bear with us and/or ask on the mailing list, see PDL::FAQ):

document perl level threading
threadids
update and correct description of slice
new functions in slice.pd (affine, lag, splitdim)
reworking of paragraph on explicit threading

Indexing and threading with PDL

A lot of the flexibility and power of PDL relies on the indexing and looping features of the perl extension. Indexing allows access to the data of a pdl object in a very flexible way. Threading provides efficient implicit looping functionality (since the loops are implemented as optimized C code).

Pdl objects (later often called "pdls") are perl objects that represent multidimensional arrays and operations on those. In contrast to simple perl @x style lists the array data is compactly stored in a single block of memory thus taking up a lot less memory and enabling use of fast C code to implement operations (e.g. addition, etc) on pdls.

pdls can have children

Central to many of the indexing capabilities of PDL are the relation of "parent" and "child" between pdls. Many of the indexing commands create a new pdl from an existing pdl. The new pdl is the "child" and the old one is the "parent". The data of the new pdl is defined by a transformation that specifies how to generate (compute) its data from the parents data. The relation between the child pdl and its parent are often bidirectional, meaning that changes in the childs data are propagated back to the parent. (Note: You see, we are aiming in our terminology already towards the new dataflow features. The kind of dataflow that is used by the indexing commands (about which you will learn in a minute) is always in operation, not only when you have explicitly switched on dataflow in your pdl by saying $a->doflow. For further information about data flow check the dataflow manpage.)

Another way to interpret the pdls created by our indexing commands is to view them as a kind of intelligent pointer that points back to some portion or all of its parents data. Therefore, it is not surprising that the parents data (or a portion of it) changes when manipulated through this "pointer". After these introductory remarks that hopefully prepared you for what is coming (rather than confuse you too much) we are going to dive right in and start with a description of the indexing commands and some typical examples how they might be used in PDL programs. We will further illustrate the pointer/dataflow analogies in the context of some of the examples later on.

There are two different implementations of this ``smart pointer relationship: the first one, which is a little slower but works for any transformation is simply to do the transformation forwards and backwards as necessary. The other is to consider the child piddle a ``virtual piddle, which only stores a pointer to the parent and access information so that routines which use the child piddle actually directly access the data in the parent. If the virtual piddle is given to a routine which cannot use it, PDL transparently physicalizes the virtual piddle before letting the routine use it.

Currently (1.94_01) all transformations which are ``affine, i.e. the indices of the data item in the parent piddle are determined by a linear transformation (+ constant) from the indices of the child piddle result in virtual piddles. All other indexing routines (e.g. ->index(...)) result in physical piddles. All routines compiled by PP can accept affine piddles (except those routines that pass pointers to external library functions).

Note that whether something is affine or not does not affect the semantics of what you do in any way: both

$a->index(...) .= 5;
$a->slice(...) .= 5;

change the data in $a. The affinity does, however, have a significant impact on memory usage and performance.

Pyndexter (pronounced poindexter) is an abstraction layer for full-text indexing engines. It presents a uniform query syntax to the user, includes a basic but functional pure-Python indexer, and has adapters for Hype, Hyperestraier, Lucene, Lupy, Pyndex, Swish-e and Xapian.
How do I install it?
Pyndexter should be installable with setuptools:
easy_install pyndexter
Enhancements:
- The API has been revamped considerably and is now much more flexible and extensible.

FTP Index provides a FTP indexer and search engine.
FTP Index is a search engine for FTP servers. It scans servers for definable filetypes and stores the results in a MySQL database.
It utilizes the ftpls tool from the ftpcopy package for indexing the servers.
It scans multiple servers at the same time by running with multiple processes.
Enhancements:
- fixed a nasty bug that caused the loss of the half all found files.

LUA is a small scripting language designed to be embedded inside your application to provide extensibility.
It is also possible to extend the Lua scripting language itself via the use of extensions written in C. This page describes such an extension which provides a small number of basic networking and filesystem primitives.
By installing this extension you will be able to write networking applications in the Lua!
Enhancements:
- Added some filesystem primitives.
- Added API documentation & examples.
- The HTTP server now falls back to a default virtual host if none is specified, or if the attempted one doesnt exist.
- The HTTPD server now supports directory indexing if no index.html file is found in a directory.
- The HTTPD server now writes access.log files to each virtualhost.

DoxMentor4J is a standalone cross platform Web/Ajax based documentation library that is fully searchable and may be hosted in the file system, in an archive or embedded in the Java classpath.

Users create the library content structure and copy any online books or manuals into directories in the library structure. DoxMentor4J then provides a dedicated web server which presents the library as a tree structure on a web page. Users can navigate through the tree to locate online books. The tree is AJAX based so there is no large initial download of all the nodes in the content structure, instead the nodes are downloaded as they are opened.

Because DoxMentor4J is a cross platform Java based application, user can write their online library to CD/DVD or flash and have it available wherever they go on different machines and operating systems.

The web server framework classes used by DoxMentor4J is called HttpdBase4J and it supports content inside archive files or in jar and zip files in the Java classpath. The content can therefore be hosted in one compressed file, and as many documentation files are often text based this means the space saving can be meaningful. Additionally it can be more convenient to have all the content in one file.

This software also supports full text indexing and searching of content using the Lucene library. Content indexing may be enabled/disabled on a node by node basis. Search indices can be accessed inside an archive or on cd/dvd. The indexing code currently supports text, html, pdf, chm and djvu.

Remote File Index is an add-on for Plone which keeps track of a document only by its url.
Did you ever find a huge pdf file that youd like to keep track of but wouldnt like to copy it entirely on your server ?
Now RemoteFileIndex indexes the content in the portal Catalog and only keeps the url of that document.
Works with:
- Plone 2.5.2
- Plone 2.5.1
- Plone 2.5
Enhancements:
- better integration with ATContentType

ruby index/search is a general indexing framework for ruby. With it, you can create collections of documents, then index and search them. Currently, both inverted indexing and LSA indexing are supported, with rudimentary result clustering in the works.

The indices may be marshalled out (with Marshal.dump()), then pulled back in with Marshal.load(), to allow for relatively quick search operations in both LSA and inverted indices.

PloneExFile provides a Plone/AT content type with an attachment, supporting preview & indexing & lock.
PloneExFile is a file like content type that provides indexing and preview support for MS Word, MS Excel, MS Powerpoint, PDF and OO Writer files.
Works with:
- Plone 2.5
- Plone 2.1
Enhancements:
- Added German translations and fixed description of upgrade procedure; provided by Carsten Kirck. Thank you!