Free external software for linux

Sort::External is a Perl module that can sort huge lists.

SYNOPSIS

my $sortex = Sort::External->new( -mem_threshold => 2**24 );
while ( ) {
$sortex->feed($_);
}
$sortex->finish;
while ( defined( $_ = $sortex->fetch ) ) {
&do_stuff_with($_);
}

Problem: You have a list which is too big to sort in-memory.
Solution: "feed, finish, and fetch" with Sort::External, the closest thing to a drop-in replacement for Perls sort() function when dealing with unmanageably large lists.

How it works:

Cache sortable items in memory. Periodically sort the cache and empty it into a temporary sortfile. As sortfiles accumulate, interleave them into larger sortfiles. Complete the sort by sorting the input cache and any existing sortfiles into an output stream.

Note that if Sort::External hasnt yet flushed the cache to disk when finish() is called, the whole operation completes in-memory.

In the CompSci world, "internal sorting" refers to sorting data in RAM, while "external sorting" refers to sorting data which is stored on disk, tape, punchcards, or any storage medium except RAM -- hence, this modules name.

Stringification

Items fed to Sort::External will be returned in stringified form (assuming that the cache gets flushed at least once): $foo = "$foo". Since this is unlikely to be desirable when objects or deep data structures are involved, Sort::External throws an error if you feed it anything other than simple scalars.

Taint and UTF-8 flags

Expert: Sort::External does a little extra bookkeeping to sustain each items taint and UTF-8 flags through the journey to disk and back.

METHODS

new()
my $sortscheme = sub { $Sort::External::b $Sort::External::a };
my $sortex = Sort::External->new(
-mem_threshold => 2**24, # default: 2**20 (1Mb)
-cache_size => 100_000, # default: undef (disabled)
-sortsub => $sortscheme, # default sort: standard lexical
-working_dir => $temp_directory, # default: see below
);

Construct a Sort::External object.

-mem_threshold -- Allow the input cache to consume approximately -mem_threshold bytes before sorting it and flushing to disk. Experience suggests that the optimum setting is somewhere between 2**20 and 2**24: 1-16Mb.
-cache_size -- Specify a hard limit for the input cache in terms of sortable items. If set, overrides -mem_threshold.
-sortsub -- A sorting subroutine. Be advised that you MUST use $Sort::External::a and $Sort::External::b instead of $a and $b in your sub. Before deploying a sortsub, consider using a GRT instead, as described in the Sort::External::Cookbook. Its probably a lot faster.
-working_dir -- The directory where the temporary sortfiles will reside. By default, this directory is created using File::Temps tempdir() command.
feed()

$sortex->feed( @items );

Feed one or more sortable items to your Sort::External object. It is normal for occasional pauses to occur during feeding as caches are flushed and sortfiles are merged.

finish()
# if you intend to call fetch...
$sortex->finish;

# otherwise....
use Fcntl;
$sortex->finish(
-outfile => sorted.txt,
-flags => (O_CREAT | O_WRONLY),
);

Prepare to output items in sorted order.

If you specify the parameter -outfile, Sort::External will attempt to write your sorted list to that location. By default, Sort::External will refuse to overwrite an existing file; if you want to override that behavior, you can pass Fcntl flags to finish() using the optional -flags parameter.

Note that you can either finish() to an -outfile, or finish() then fetch()... but not both.

fetch()
while ( defined( $_ = $sortex->fetch ) ) {
&do_stuff_with($_);
}

Fetch the next sorted item.

Mod_Auth_External and mod_authnz_external are Apache modules used for authentication. The two modules provide basically the same functionality, and differ mainly in the way they interface to other authentication code in Apache. For Apache 2.1 and later, mod_authnz_external should be prefered.

The Apache HTTP Daemon can be configured to require users to supply logins and passwords before accessing pages in some directories. Authentication is the process of checking if the password given is correct for a user. Apache has standard modules for authenticating out of several different kinds of databases. These external authentication modules provide a flexible tool for creating authentication systems based on other databases.

These modules can be used in either of two somewhat divergent ways:

External Authentication:

When a user supplies a login and password, mod_auth*_external runs a program you write, passing it the login and password. Your program does whatever checking and logging it needs to, and then returns a Accept/Reject flag to Apache.

This is slower than doing the authentication internally because it has the overhead of launching an external program for each authentication. However, there are at least two situations where it is very useful:

Rapid prototyping. Mod_auth*_external makes a very nice swiss army knife authenticator. You can quickly put together custom authentication systems for many weird authentication applications. The external authentication program can be a shell script or perl program. It can be written without knowing much about building Apache modules. Bugs in it will not endanger the overall integrity of the Apache server.
Access restrictions. There are situations where you do not want to make your user database readable to the user-id that Apache runs under. In these cases the external authentication program can be an suid program that has access to databases Apache cannot access. For example, if you want to authentication out of a Unix shadow password database, and you arent foolish enough to run Apache as root, a carefully written suid-root external authentication program can do the job for you. Such an authenticator for shadow password files and PAM (pwauth) is available separately.

Hardcoded Authentication:

Some hooks have been inserted into mod_auth*_external to make it easy to replace the call to the external authentication program with a call to a hardcoded internal authentication routine that you write.

This is sort of a half-way measure to just writing your own Apache module from scratch, allowing you to easily borrow some of the logic from mod_auth*_external. Its a bit more dangerous than using an external authenticator, as bugs in your module can crash Apache, but performance is usually better.

I think mod_auth*_external is the best current solution for authenticating out of shadow password files, and other similar applications. For rapid prototyping and for an easy way to build your own modules, mod_perl may be a better solution in some cases.

mod_authnz_external is an Apache module used for authentication. The two modules provide basically the same functionality, and differ mainly in the way they interface to other authentication code in Apache.

For Apache 2.1 and later, mod_authnz_external should be prefered.

The Apache HTTP Daemon can be configured to require users to supply logins and passwords before accessing pages in some directories. Authentication is the process of checking if the password given is correct for a user. Apache has standard modules for authenticating out of several different kinds of databases. The external authentication module provides a flexible tool for creating authentication systems based on other databases.

The module can be used in either of two somewhat divergent ways:

External Authentication:
When a user supplies a login and password, mod_auth*_external runs a program you write, passing it the login and password. Your program does whatever checking and logging it needs to, and then returns a Accept/Reject flag to Apache.
This is slower than doing the authentication internally because it has the overhead of launching an external program for each authentication. However, there are at least two situations where it is very useful:

Rapid prototyping. Mod_auth*_external makes a very nice swiss army knife authenticator. You can quickly put together custom authentication systems for many weird authentication applications. The external authentication program can be a shell script or perl program. It can be written without knowing much about building Apache modules. Bugs in it will not endanger the overall integrity of the Apache server.

Access restrictions. There are situations where you do not want to make your user database readable to the user-id that Apache runs under. In these cases the external authentication program can be an suid program that has access to databases Apache cannot access. For example, if you want to authentication out of a Unix shadow password database, and you arent foolish enough to run Apache as root, a carefully written suid-root external authentication program can do the job for you. Such an authenticator for shadow password files and PAM (pwauth) is available separately.

Hardcoded Authentication:
Some hooks have been inserted into mod_auth*_external to make it easy to replace the call to the external authentication program with a call to a hardcoded internal authentication routine that you write.

This is sort of a half-way measure to just writing your own Apache module from scratch, allowing you to easily borrow some of the logic from mod_auth*_external. Its a bit more dangerous than using an external authenticator, as bugs in your module can crash Apache, but performance is usually better.

I think mod_auth*_external is the best current solution for authenticating out of shadow password files, and other similar applications. For rapid prototyping and for an easy way to build your own modules, mod_perl may be a better solution in some cases.

Ezstream is a command line source client for the Icecast media streaming server. It can stream Ogg Vorbis and MP3 audio, as well as Ogg Theora video, either "as-is" without reencoding (which uses only very little CPU time) or it can use external decoders and encoders to convert virtually any media format into one of the supported streaming formats. The project supports metadata, streaming from standard input, playlists, and external playlist scripts or programs, and more.

etPan is a console mail user agent based on libEtPan! libEtPan ! is a mail purpose library. Its a library that handles mail at low-level: MAP/NNTP/POP3/SMTP over TCP/IP and SSL/TCP/IP, mbox/MH/maildir, message / MIME parser.
Main features:
- IMAP4rev1 / POP3 / NNTP / mbox / mh / maildir
- virtual folder tree
- multiple folder views and message views
- smart multi-threading
- PGP signing and encryption (using GnuPG as external command)
- S/MIME signing and encryption (using OpenSSL as external command)
- SPAM process (using bogofilter as external command)
- user interface for configuration edition

WebGrab is a tool that acts as a CGI proxy. It simulates a Web browser and grabs data from external programs, and can exchange individual words or whole blocks.

The project can be run standalone or in conjunction with phpCMS.

htsneak is a general-purpose HTTP tunnelling utility written in Python. It can be used for accessing external Internet resources from behind a firewall, through an HTTP proxy. It comes with a built-in proxy server that tunnels data through an HTTP proxy to external servers for pop3, imap, smtp, ymsg, etc.

Installation is very simple. Unzip the archive into your prefered installation directory.