Free in parallel software for linux

File::Find::Parallel allows you to traverse a number of similar directories in parallel.

SYNOPSIS

use File::Find::Parallel;

my $ffp = File::Find::Parallel->new( qw( /foo /bar ) );

print "Union:n";
my $union = $ffp->any_iterator
print " $_n" while $_ = $union->();

print "Intersection:n";
my $inter = $ffp->all_iterator
print " $_n" while $_ = $inter->();

File::Find is the ideal tool for quickly scanning a single directory. But sometimes its nice to be able to perform operations on multiple similar directories in parallel. Perhaps you need to compare the contents of two directories or convert files that are shared in more than one directory into hard links.

This module manufactures iterators that visit each file and directory in either the union or the intersection of a number of directories. Hmm. What does that mean?

Given two directory trees like this

foo
foo/a
foo/b/c
foo/d

bar
bar/a
bar/b
bar/e

you can choose to work with the intersection of the two directory structures:
.
./a
./b

That is the subdirectories and files that the foo and bar share.
Alternately you can work with the union of the two directory structures:
.
./a
./b
./b/c
./d
./e

Still not clear? Well, if you wanted to do a recursive diff on the two directories youd iterate their union so you could report files that were present in foo but missing from bar and vice-versa.

If, on the other hand you wanted to scan the directories and find all the files that are common to all of them youd iterate their intersection and receive only files and directories that were present in all the directories being scanned.

The any_iterator and all_iterator are built on a more general purpose method: want_iterator. If, for example, you want to make links between files that are found in more than one directory you might get your iterator like this:

my $iter = $ffp->want_iterator( 2 );

The apparently magic 2 reflects the fact that if youre going to be making links you need at least two files. No matter how many directories you are iterating over in parallel you will only see files and directories that appear in at least two of those directories.

File::Find::Parallel can scan any number of directories at the same time. Heres an example (on Unix systems) that returns the list of all files and directories that are contained in all home directories.

use File::Glob :glob;
use File::Find::Parallel;

my $find = File::Find::Parallel->new( bsd_glob( /home/* ) );

my @common = ( );
my $iter = $find->all_iterator;
while ( defined my $obj = $iter->() ) {
push @common, $obj;
}

print "The following files are common to ",
"all directories below /home :n";

print " $_n" for @common;

For a complete concrete example of its use see lncopies in the bin subdirectory of this distribution.

Iterators

The iterator returned by any_iterator, all_iterator or want_iterator is a code reference. Call it to get the next file or directory. When all files and directories have been returned the iterator will return undef.

Once created an iterator is independent of the File::Find::Parallel object that created it. If the object goes out of scope and is destroyed during the life of the iterator it will still function normally.

You may have many active iterators for a single File::Find::Parallel object at any time.

Parallel::Queue is a Perl module to fork or thread a list of closures N-way parallel.

SYNOPSIS

# example queue:
# only squish files larger than 8KB in size. figure
# that the system can handle four copies of squish
# running at the same time without them interfering
# with one another.

my @queue = map { -s > 8192 ? sub{ squish $_ } : () } @filz;

# functional: pass in the count and list of coderefs.
#
# adding runqueue exports the subroutine into
# the current package. useful for non-OO situations.
#
# run the queue 4 way parallel.

use Parallel::Queue qw( runqueue verbose fork );

my @remaining = runqueue 4, @queue;

die "Incomplete jobs" if @remaining;

# OO: generate queue manager and use without the
# runqueue arguments, construct a queue manager,
# and use it to run the jobs

use Parallel::Queue;

my $quemgr = Parallel::Queue->construct( thread );

$quemgr->runqueue( 4, @queue );

die "Incomplete jobs" if @queue;

# call Parallel::Queue with the default configuration
# (fork quietly).

require Parallel::Queue;

Parallel::Queue->runqueue( 4, @queue );

# pre-define defaults for the objects: leave
# out runqueue, set the rest, and construct
# an object. the one here gets verbose, thread,
# and debug all set to true.

use Parallel::Queue qw( verbose thread );

my $quemgr = Parallel::Queue->construct( debug );

my @remaining = $quemgr->runqueue( 4, @queue );

Given a count and an array of coderefs (most likely closures), runqueue will run the jobs in parallel. The jobs can be run via fork or detached threads [see known issues for threading]. Jobs on the queue are executed until one of them exits non-zero, the fork/thread operation fails, or all of them are dispatched (i.e., the queue is empty).

Parallel MPI BZIP2 is a parallel implementation of the bzip2 block-sorting file compressor that uses MPI and achieves significant speedup on cluster machines.

The output of this version is fully compatible with bzip2 v1.0.2 or newer (ie: anything compressed with mpibzip2 can be decompressed with bzip2). MPIBZIP2 should work on any system that has a pthreads compatible C++ compiler (such as gcc). It has been tested on: Linux and Solaris.

NOTE: If you are looking for a parallel BZIP2 that works on multi-processor/muti-core/SMP machines, you should check out PBZIP2 which was designed for a multi-threaded shared-memory architecture.

Usage:

Run mpibzip2 for the help listing.
==================================================================
Usage: mpibzip2 [-1 .. -9] [-b#cdfktvV] < filename > < filename2 > < filenameN >

-b#: where # is the file block size in 100k (default 9 = 900k)
-c : output to standard out (stdout)
-d : decompress file
-f : force, overwrite existing output file
-k : keep input file, dont delete
-t : test compressed file integrity
-v : verbose mode
-V : display version info for mpibzip2 then exit
-1 .. -9 : set BWT block size to 100k .. 900k (default 900k)

Example: mpibzip2 -b15k myfile.tar
Example: mpibzip2 -v -5 myfile.tar second*.txt
Example: mpibzip2 -d myfile.tar.bz2

PDL::Parallel::MPI Perl module contains routines to allow PDL objects to be moved around on parallel systems using the MPI library.

SYNOPSIS

use PDL;
use PDL::Parallel::MPI;
mpirun(2);

MPI_Init();
$rank = get_rank();
$a=$rank * ones(2);
print "my rank is $rank and $a is $an";
$a->move( 1 => 0);
print "my rank is $rank and $a is $an";
MPI_Finalize();

MPI STANDARD CALLS

Most of the functions from the MPI standard may be used from this module on regular perl data. This is functionallity inherited from the Parallel::MPI module. Read the documentation for Parallel::MPI to see how to use.

One may mix mpi calls on perl built-in-datatypes and mpi calls on piddles.

use PDL;
use PDL::Parallel::MPI;
mpirun(2);

MPI_Init();
$rank = get_rank();
$pi = 3.1;
if ($rank == 0) {
MPI_Send($pi,1,MPI_DOUBLE,1,0,MPI_COMM_WORLD);
} else {
$message = zeroes(1);
$message->receive(0);
print "pi is $messagen";
}
MPI_Finalize();

Parallel::Workers::Shared is a simple Perl module.

Parallel::Workers::Shared requires no configuration files or environment variables.

Parallel Bladeenc is a true parallel version of the Bladeenc MP3 encoder; it distributes work across CPUs to speed up MP3 encoding. It uses the Message Passing Interface (MPI) for parallelization across SMPs and/or multiple machines. Hence, if you have a 4-way SMP, you can encode your MP3s about 4 times as fast as the regular Bladeenc; if you have two 4-way SMPs, you can encode about 8 times as fast.
The difficult part about parallelization is typically about how to split the problem up into independent (or nearly independent) parts. The structure shown above - assuming that the encode() function was independent of previous calls to encode() - is trivial to parallelize.
For example, if we want to run on four machines, we can split the input file into four parts, give 1/4 of the file to each machine, and let each machine loop over encode() for their portion of the file. Then take the output from each machine, put it in the right order, and write it out to a single output file. Done.
With such a scheme, the more work that you throw at it, the more efficient it will become.
Hence, trying this scheme with small MP3 files will probably not result in any noticeable speedup (in fact, it may be slower than running in serial, because of the added overhead for working in parallel). It is necessary to give the parallel engine enough work to offset the overhead added by the parallel framework. Generally, this is not very much overhead (read on to find out why), but parallel is not free.
Enhancements:
- Fixed minor error that caused an error message from MPICH when shutting down. Thanks to Gary Smith for pointing this out.

Many times when one is downloading files from a ftp site, the bottleneck is at the servers end, because the server is handling many connections, even if the server has a greater bandwidth than the client. Popular programs are often mirrored at several sites, but this does not help the matter much; people tend to congregate at one site anyways.
Given that there are mirror-sites for the file one is downloading, one can in principle accomplish a much greater bandwidth by downloading from all the sites in parallel.
You can speed up your download if you are getting files from different sites in parallel. This is accomplished by dividing the file being fetched into several pieces, and by getting each piece from a different server, and then re-assembling them.
If there are enough mirror sites, this partitioning makes it so that the bottleneck is now placed at the client end, maxing the clients connection.
Hence, we arrive at paraget, a parallel ftp-fetching program. paraget is designed to not only do basic n-equal-piece partitioning of a file and sending requests out to n servers for data, but to also be dynamic during the downloading process. For example if one server is too slow, and paraget was done with faster server downloading its piece
Enhancements:
- Server statistics is now loaded correctly.
- Cleanup of temp files.
- Several undefineds in intervalmanager fixed.
- Now checks for size mismatches.