Free pipeline software for linux

Alvis::Pipeline is a Perl extension for passing XML documents along the Alvis pipeline.

SYNOPSIS

use Alvis::Pipeline;
$in = new Alvis::Pipeline::Read(host => "harvester.alvis.info",
port => 16716,
spooldir => "/home/alvis/spool");
$out = new Alvis::Pipeline::Write(port => 29168);
while ($xml = $in->read(1)) {
$transformed = process($xml);
$out->write($transformed);
}

This module provides a simple means for components in the Alvis pipeline to pass documents between themselves without needing to know about the underlying transfer protocol. Pipe objects may be created either for reading or writing; components in the middle of the pipeline will create one of each. Pipes support exactly one method, which is either read() or write() depending on the type of the pipe. The granularity of reading and writing is the XML document; neither smaller fragments nor larger aggregates can be transferred.

The documents expected to pass through this pipeline are those representing documents acquired for, and being analysed by, Alvis. These documents are expressed as XML contructed according to the specifications described in the Metadata Format for Enriched Documents. However, while this is the motivating example pipeline that led to the creation of this module, there is no reason why other kinds of documents should not also be passed through pipeline using this software.

The pipeline protocol is described below, to facilitate the development of indepedent implementations in other languages.

Application::Pipeline is a module designed to map methods ( referred to in this role as handlers ) to different phases of an applications life cycle. By assigning methods to different phases of this pipeline, the author can concentrate on the logic for each phase and let the framework manage the application flow. Adopting the same idea as CGI::Application, writing an application with Application::Pipeline is a matter of creating a module that is a subclass of Application::Pipeline.

To build a pipeline application, it is necessary to register methods to run during each phase. This can be done one at a time, with the addHandler method. But Application::Pipeline also looks in the subclass package for the package variable %plan. This hashs keys are the names of the phases of the pipeline. Each key points to an array reference which is a list of the methods to run for that phase. The methods are either the names of the methods to run, or references to the actual methods.

This is not the be-all end-all definition of the pipeline. It is still possible to use addHandler to modify the pipeline, and as explained later, it is possible to take into account %plans defined in superclasses.

TCPpipe provides a small console-tool to transfer any data over a network.
TCPpipe is a small console-tool to transfer any data over a network. TCPpipe has to be run on both hosts. After establishing a TCP connection, the standard input on one host is sent to standard output on the other host.
This can be used to create something like a pipeline between two processes which run on different hosts.
TCPpipe is a very simple program, but it can be very useful.
To compile type:
$ make
Then copy the binary where you want, for instance:
# cp tcppipe /usr/local/bin
USAGE EXAMPLES:
For example you can transfer files without an FTP or HTTP server:
On host A:
tcppipe --listen --port 1100 --send < file-on-A
On host B:
tcppipe --connect --port 1100 --receive host.name.of.A > file-on-B
(The port 1100 is only an example, change the file- and hostnames.)
You could even transfer many files with tar:
tar -cOf file1 file2 dir/* | tcppipe -lp1100
tcppipe -cp1100 192.168.0.5 | tar -xf -
To gzip the data do this:
prog-producing-output | gzip | tcppipe --send ...
tcppipe --receive ... | gunzip | prog-reading-input
Enhancements:
- all the documentation stuff + improved Makefile
- support for resolving host names
- additional parameter checks

OSSP iselect is an interactive line selection tool for textual files, operating via a full-screen Curses-based terminal session.
It can be used either as an user interface frontend controlled by a scripting backend as its wrapper or in batch mode as a pipeline filter (usually between grep and the final executing command).
Enhancements:
- A security fix was done by upgrading GNU shtool from 2.0.1 to 2.0.3.

WebCleaner is a filtering HTTP proxy. It can be customized to your needs.
Main features:
- remove unwanted HTML (adverts, flash, etc.)
- popup blocker
- disable animated GIFs
- filter images by size, remove banner adverts
- compress documents on-the-fly (with gzip)
- reduce images to low-bandwidth JPEGs
- remove/add/modify arbitrary HTTP headers
- configurable over web interface
- usage of SquidGuard blacklists
- antivirus filter module
- detection and correction of known HTML security flaws
- Basic, Digest and (untested) NTLM proxy authentication support
- per-host access control
- HTTP/1.1 support (persistent connections, pipelining)
- HTTPS proxy CONNECT and optional SSL gateway support
Enhancements:
- Compatibility problems with Internet Explorer and the Compress
- filter have been fixed.
- The Blocker filter now allows absolute block URLs.
- The sets of blacklisted domains and advertisement servers have been updated.

Jetspeed provides a JSR-168 compliant enterprise portal.
etspeed-2 is a full implementation of the Java Portlet API. It is fully compliant with the Portlet Specification 1.0 (JSR-168). It has passed the TCK (Test Compatibility Kit) suite and is fully CERTIFIED to the Java Portlet Standard.
Notable features include security components backed by LDAP and database implementations, and some robust administration interfaces. Custom portals can be built and deployed using the Jetspeed plugin for Maven.
Developers can use the Jetspeed PSML language to assemble portlets, and the Apache Portals Bridges project to bridge portals with existing technologies including Struts, JSF, PHP, and Perl.
For GUI designers, Jetspeed comes with several built-in templates used to decorate portals and portlets.
Main features:
- Fully compliant with Java Portlet API Standard 1.0 (JSR 168)
- Passed JSR-168 TCK Compatibility Test Suite
- J2EE Security based on JAAS Standard, JAAS DB Portal Security Policy
- LDAP Support for User Authentication
- Spring-based Components and Scalable Architecture
- Configurable Pipeline Request Processor
- Auto Deployment of Portlet Applications
- Jetspeed Component Java API
- Jetspeed AJAX XML API
- Declarative Security Constraints and JAAS Database Security Policy
- Runtime Portlet API Standard Role-based Security
- Portal Content Management and Navigations: Pages, Menus, Folders, Links
- Multithreaded Aggregation Engine
- PSML Folder CMS Navigations, Menus, Links
- Jetspeed SSO (Single Sign-on)
- Rules-based Profiler for page and resource location
- Integrates with most popular databases including Derby, MySQL, MS SQL, Postgres, Oracle, DB2, Hypersonic
- Client independent capability engine (html, xhtml, wml,vml)
- Internationalization: Localized Portal Resources in 12 Languages
- Statistics Logging Engine
- Portlet Registry
- Full Text Search of Portlet Resources with Lucene
- User Registration
- Forgotten Password
- Rich Login and Password Configuration Management

Polipo project is a small and fast caching web proxy (a web cache, an HTTP proxy) designed to be used by one person or a small group of people.
I like to think that is similar in spirit to WWWOFFLE, but the implementation techniques are more like the ones ones used by Squid.
Installation:
make all
su -c make install
man polipo
polipo &
If you want Polipo to put itself into the background, you may replace the last line with:
polipo daemonise=true logFile="/var/log/polipo.log"
On SVR4 systems (Solaris, HP/UX), you will need to use one of the following (whichever works):
make PLATFORM_DEFINES=-DSVR4 all
make PLATFORM_DEFINES=-DSVR4 LDLIBS=-lsocket -lnsl -lresolv all
You can also use Polipo without installing:
make
nroff -man polipo.man | more
./polipo &
Main features:
- Polipo will use HTTP/1.1 pipelining if it believes that the remote server supports it, whether the incoming requests are pipelined or come in simultaneously on multiple connections (this is more than the simple usage of persistent connections, which is done by e.g. Squid);
- Polipo will cache the initial segment of an instance if the download has been interrupted, and, if necessary, complete it later using Range requests;
- Polipo will upgrade client requests to HTTP/1.1 even if they come in as HTTP/1.0, and up- or downgrade server replies to the clients capabilities (this may involve conversion to or from the HTTP/1.1 chunked encoding);
- Polipo has complete support for IPv6 (except for scoped (link-local) addresses).
- Polipo can optionally use a technique known as Poor Mans Multiplexing to reduce latency even further.
Enhancements:
- This version tweaks Polipos behaviour when speaking to HTTP/1.0 Web sites (such as Wikipedia) to make it more aggressive.
- It also fixes a resource leak under Windows.

Terrier project is a probabilistic Java toolkit for building search engines.
Terrier is software for the rapid development of Web, intranet, and desktop search engines.
More generally, it is a modular platform for building large-scale information retrieval applications, providing indexing and probabilistic retrieval functionalities.
It comes with a desktop search application.
Terrier has various cutting-edge features including parameter-free probabilistic retrieval approaches (such as Divergence from Randomness models), automatic query expansion/re-formulation methodologies, and efficient data compression techniques.
Terrier comes with a powerful proof-of-concept Desktop search application [Screenshots], and full TREC capabilities including the ability to index, query and evaluate the standard TREC collections, such as AP, WSJ, WT10G, .GOV and .GOV2.
Terrier is written in Java [Requirements] and has been successfully used for adhoc retrieval, Web search and cross-language retrieval, in a centralised or distributed setting.
Currently, it is also being used for running various applications.
Main features:
- Open Source (Mozilla Public Licence)
- Written in cross-platform Java
- Highly compressed disk data structures.
- Handling large-scale document collections.
- Direct file for efficient query expansion.
- Modular and open indexing and querying APIs.
- Testbed for indexing and retrieval from standard TREC test collections.
- Interactive querying application.
- Desktop search application for searching various types of documents.
- Input/output of gamma, unary and binary encoded integers for compressing streams or random access files.
- Standard evaluation of TREC ad-hoc and known-item search retrieval results.
- Indexing of tagged document collections, as well as documents of various formats, such as HTML, PDF, or Microsoft Word, Excel and Powerpoint files.
- Indexing of field information.
- Indexing of position information on a word, or a block level.
- Support for classic retrieval models, such as tf-idf, BM25 and Ponte-Croft language model, and Rocchios query expansion.
- Provides a number of Divergence From Randomness (DFR) document ranking models.
- Provides a number of parameter-free DFR term weighting models for automatic query expansion.
- Advanced query language that supports AND/NOT operators, phrase and proximity search.
- Flexible processing of terms through a pipeline of components, such as stop-words removers and stemmers.
Enhancements:
- This is a major update with improvements in indexing and retrieval functionalities, including faster indexing and retrieval, and new retrieval models (including models from Divergence from Randomness and Language modeling).
- It has support for much larger collections of documents, including TREC GOV2 collections (25M documents), merging of indices, and multi-lingual and non-English collections of documents.
- The documentation has been vastly improved.