Free data transfering software for linux

Fast Data Transfer is an application for efficient data transfers that is capable of reading and writing at disk speed over wide area networks (with standard TCP).
It can be used to stream a large set of files across the network, so a large dataset composed of thousands of files can be sent or received at full speed, without the network transfer restarting between files.
The project is written in Java, runs an all major platforms, and is easy to use.
Main features:
- Streams a dataset (list of files) continuously, using a managed pool of buffers through one or more TCP sockets.
- Uses independent threads to read and write on each physical device
- Transfers data in parallel on multiple TCP streams, when necessary
- Uses appropriate-sized buffers for disk I/O and for the network
- Restores the files from buffers asynchronously
- Resumes a file transfer session without loss, when needed

Local Data Manager (LDM) is a collection of cooperating programs that select, capture, manage, and distribute arbitrary data products.
The system is designed for event-driven data distribution, and is currently used in the Unidata Internet Data Distribution (IDD) project. The LDM system includes network client and server programs and their shared protocols.
An important characteristic of the LDM is its support for flexible, site-specific configuration.
Enhancements:
- Fixes for timestamp bugs.

Data::Inherited is a Perl module with hierarchy-wide accumulation of list and hash results.

SYNOPSIS

package Foo;
use base Data::Inherited;
use constant PROPERTIES => (qw/name address/);

package Bar;
use base Foo;
use constant PROPERTIES => (qw/age/);

package main;
my $bar = Bar->new;
print "$_n" for $bar->every_list(PROPERTIES);
prints
name
address
age

This is a mixin class. By inheriting from it you get two methods that are able to accumulate hierarchy-wide list and hash results.

every_list(String $method_name, Bool ?$override_cache = 0)

Takes as arguments a method name (mandatory) and a boolean indicating whether to override the cache (optional, off by default)

Causes every method in the objects hierarchy with the given name to be invoked. The resulting list is the combined set of results from all the methods, pushed together in top-to-bottom order (hierarchy-wise).

every_list() returns a list in list context and an array reference in scalar context.
The result is cached (per calling package) and the next time the method is called from the same package with the same method argument, the cached result is returned.

This is to speed up method calls, because internally this module uses NEXT, which is quite slow. It is expected that every_list() is used for methods returning static lists (object defaults, static class definitions and such). If you want to override the caching mechanism, you can provide the optional second argument. The result is cached in any case.

Data::Region Perl module can define hierarchical areas with behaviors.

SYNOPSIS

use Data::Region;

$r = Data::Region->new( 8.5, 11, { data => PageObj->new() } );
$r->data( PageObj->new() );

foreach my $c ( $r->subdivide(2.5,3) ) {
$a = $c->area(0.25,0.25, 2.25,2.75);
$a2 = $c->area(0.25,0.25, -0.25,-0.25); # as offset from lower right

($t,$m,$b) = $a->split_vertical(2,5,1); # sequential heights
($t,$m,$b) = $a->split_vertical_abs(0,2,7); # absolute offsets
($l,$r) = $a->split_horizontal(2); # $l gets width of 2, $r gets the rest

my($x1,$y1,$x2,$y2) = $a->coords();
my $data = $a->data(); # data inherits from parent, if not set
$a->action( sub { $data->setfont("Times-Bold", 10);
$data->text($x1,$y1, "Some Text");
$data->line( $_[0]->coords() ); # the non-closure way
} );
}
$r->render(); # heirarchically perform all the actions

# Get some info about a region:
($w,$h) = ( $a->width(), $a->height() );
($x1,$y1, $x2,$y2) = $a->coords();
($x1,$y1) = $a->top_left();
($x2,$y1) = $a->top_right();
($x1,$y2) = $a->bottom_left();
($x2,$y2) = $a->bottom_right();

Data::Region allows you to easily define a set of nested (2-dimensional) areas, defined by related coordinates, and to associate actions with them. The actions can then be performed hierarchically from any root of the tree.

Data::Region was written to provide an easy way to do simple page layout, but has, perhaps, more general uses.

Data::Serializer package contains modules that serialize data structures.

SYNOPSIS

use Data::Serializer;

$obj = Data::Serializer->new();

$obj = Data::Serializer->new(
serializer => Storable,
digester => MD5,
cipher => DES,
secret => my secret,
compress => 1,
);

$serialized = $obj->serialize({a => [1,2,3],b => 5});
$deserialized = $obj->deserialize($serialized);
print "$deserialized->{b}n";

Provides a unified interface to the various serializing modules currently available. Adds the functionality of both compression and encryption.

EXAMPLES

Please see Data::Serializer::Cookbook(3)

METHODS

new - constructor
$obj = Data::Serializer->new();


$obj = Data::Serializer->new(
serializer => Data::Dumper,
digester => SHA-256,
cipher => Blowfish,
secret => undef,
portable => 1,
compress => 0,
serializer_token => 1,
options => {},
);

new is the constructor object for Data::Serializer objects.

The default serializer is Data::Dumper
The default digester is SHA-256
The default cipher is Blowfish
The default secret is undef
The default portable is 1
The default encoding is hex
The default compress is 0
The default compressor is Compress::Zlib
The default serializer_token is 1
The default options is {} (pass nothing on to serializer)
serialize - serialize reference

$serialized = $obj->serialize({a => [1,2,3],b => 5});

Serializes the reference specified.
Will compress if compress is a true value.
Will encrypt if secret is defined.
deserialize - deserialize reference

$deserialized = $obj->deserialize($serialized);

Reverses the process of serialization and returns a copy of the original serialized reference.

freeze - synonym for serialize
$serialized = $obj->freeze({a => [1,2,3],b => 5});

thaw - synonym for deserialize
$deserialized = $obj->thaw($serialized);

raw_serialize - serialize reference in raw form
$serialized = $obj->raw_serialize({a => [1,2,3],b => 5});

This is a straight pass through to the underlying serializer, nothing else is done. (no encoding, encryption, compression, etc)

raw_deserialize - deserialize reference in raw form
$deserialized = $obj->raw_deserialize($serialized);

This is a straight pass through to the underlying serializer, nothing else is done. (no encoding, encryption, compression, etc)

secret - specify secret for use with encryption
$obj->secret(mysecret);

Changes setting of secret for the Data::Serializer object. Can also be set in the constructor. If specified than the object will utilize encryption.

portable - encodes/decodes serialized data

Uses encoding method to ascii armor serialized data

Aids in the portability of serialized data.

compress - compression of data

Compresses serialized data. Default is not to use it. Will compress if set to a true value $obj->compress(1);

serializer - change the serializer

Currently have 8 supported serializers: Storable, FreezeThaw, Data::Denter, Config::General, YAML, PHP::Serialization, XML::Dumper, and Data::Dumper.
Default is to use Data::Dumper.

Each serializer has its own caveats about usage especially when dealing with cyclical data structures or CODE references. Please see the appropriate documentation in those modules for further information.

cipher - change the cipher method

Utilizes Crypt::CBC and can support any cipher method that it supports.

digester - change digesting method

Uses Digest so can support any digesting method that it supports. Digesting function is used internally by the encryption routine as part of data verification.

compressor - changes compresing module

This method is included for possible future inclusion of alternate compression method Currently Compress::Zlib is the only supported compressor.

encoding - change encoding method

Encodes data structure in ascii friendly manner. Currently the only valid options are hex, or b64.

The b64 option uses Base64 encoding provided by MIME::Base64, but strips out newlines.

serializer_token - add usage hint to data

Data::Serializer prepends a token that identifies what was used to process its data. This is used internally to allow runtime determination of how to extract Serialized data. Disabling this feature is not recommended.

options - pass options through to underlying serializer

Currently is only supported by Config::General, and XML::Dumper.

my $obj = Data::Serializer->new(serializer => Config::General,
options => {
-LowerCaseNames => 1,
-UseApacheInclude => 1,
-MergeDuplicateBlocks => 1,
-AutoTrue => 1,
-InterPolateVars => 1
},
) or die "$!n";

or

my $obj = Data::Serializer->new(serializer => XML::Dumper,
options => { dtd => 1, }
) or die "$!n";
store - serialize data and write it to a file (or file handle)
$obj->store({a => [1,2,3],b => 5},$file, [$mode, $perm]);

or

$obj->store({a => [1,2,3],b => 5},$fh);

Serializes the reference specified using the serialize method and writes it out to the specified file or filehandle.

If a file path is specified you may specify an optional mode and permission as the next two arguments. See IO::File for examples.

Trips an exception if it is unable to write to the specified file.

retrieve - read data from file (or file handle) and return it after deserialization

my $ref = $obj->retrieve($file);

or

my $ref = $obj->retrieve($fh);

Reads first line of supplied file or filehandle and returns it deserialized.

Data::CTable is a Perl module that helps you read, write, manipulate tabular data.

SYNOPSIS

## Read some data files in various tabular formats
use Data::CTable;
my $People = Data::CTable->new("people.merge.mac.txt");
my $Stats = Data::CTable->new("stats.tabs.unix.txt");

## Clean stray whitespace in fields
$People->clean_ws();
$Stats ->clean_ws();

## Retrieve columns
my $First = $People->col(FirstName);
my $Last = $People->col(LastName );

## Calculate a new column based on two others
my $Full = [map {"$First->[$_] $Last->[$_]"} @{$People->all()}];

## Add new column to the table
$People->col(FullName => $Full);

## Another way to calculate a new column
$People->col(Key);
$People->calc(sub {no strict vars; $Key = "$Last,$First";});

## "Left join" records matching Stats:PersonID to People:Key
$Stats->join($People, PersonID => Key);

## Find certain records
$Stats->select_all();
$Stats->select(Department => sub {/Sale/i }); ## Sales depts
$Stats->omit (Department => sub {/Resale/i}); ## not Resales
$Stats->select(UsageIndex => sub {$_ > 20.0}); ## high usage

## Sort the found records
$Stats->sortspec(DeptNum , {SortType => Integer});
$Stats->sortspec(UsageIndex, {SortType => Number });
$Stats->sort([qw(DeptNum UsageIndex Last First)]);

## Make copy of table with only found/sorted data, in order
my $Report = $Stats->snapshot();

## Write an output file
$Report->write(_FileName => "Rept.txt", _LineEnding => "mac");

## Print a final progress message.
$Stats->progress("Done!");

## Dozens more methods and parameters available...

OVERVIEW

Data::CTable is a comprehensive utility for reading, writing, manipulating, cleaning and otherwise transforming tabular data. The distribution includes several illustrative subclasses and utility scripts.

A Columnar Table represents a table as a hash of data columns, making it easy to do data cleanup, formatting, searching, calculations, joins, or other complex operations.

The objects hash keys are the field names and the hash values hold the data columns (as array references).

Tables also store a "selection" -- a list of selected / sorted record numbers, and a "field list" -- an ordered list of all or some fields to be operated on. Select() and sort() methods manipulate the selection list. Later, you can optionally rewrite the table in memory or on disk to reflect changes in the selection list or field list.
Data::CTable reads and writes any tabular text file format including Merge, CSV, Tab-delimited, and variants. It transparently detects, reads, and preserves Unix, Mac, and/or DOS line endings and tab or comma field delimiters -- regardless of the runtime platform.

In addition to reading data files, CTable is a good way to gather, store, and operate on tabular data in memory, and to export data to delimited text files to be read by other programs or interactive productivity applications.

To achieve extremely fast data loading, CTable caches data file contents using the Storable module. This can be helpful in CGI environments or when operating on very large data files. CTable can read an entire cached table of about 120 megabytes into memory in about 10 seconds on an average mid-range computer.

For simple data-driven applications needing to store and quickly retrieve simple tabular data sets, CTable provides a credible alternative to DBM files or SQL.
For data hygiene applications, CTable forms the foundation for writing utility scripts or compilers to transfer data from external sources, such as FileMaker, Excel, Access, personal organizers, etc. into compiled or validated formats -- or even as a gateway to loading data into SQL databases or other destinations. You can easily write short, repeatable scripts in Perl to do reporting, error checking, analysis, or validation that would be hard to duplicate in less-flexible application environments.

The data representation is simple and open so you can directly access the data in the object if you feel like it -- or you can use accessors to request "clean" structures containing only the data or copies of it. Or you can build your own columns in memory and then when youre ready, turn them into a table object using the very flexible new() method.

The highly factored interface and implementation allow fine-grained subclassing so you can easily create useful lightweight subclasses. Several subclasses are included with the distribution.

Most defaults and parameters can be customized by subclassing, overridden at the instance level (avoiding the need to subclass too often), and further overridden via optional named-parameter arguments to most major method calls.

Data::Str2Num - int str to int; float str to float, else undef.

SYNOPSIS

#####
# Subroutine interface
#
use Data::Str2Num qw(config str2float str2int str2integer);

$float = str2float($string, [@options]);
(@strings, @floats) = str2float(@strings, [@options]);

$integer = $secspack->str2int($string);

$integer = str2integer($string, [@options]);
(@strings, @integers) = str2int(@strings, [@options]);


#####
# Class, Object interface
#
# For class interface, use Data::SecsPack instead of $self
#
use Data::Str2Num;

$str2num = Data::Str2Num;
$str2num = new Data::Str2Num;

$float = $secspack->str2float($string, [@options]);
(@strings, @floats) = $secspack->str2float(@strings, [@options]);

$integer = $secspack->str2int($string);

$integer = $secspack->str2integer($string, [@options])
(@strings, @integers) = $secspack->str2int(@strings, [@options]);

Generally, if a subroutine will process a list of options, @options, that subroutine will also process an array reference, @options, [@options], or hash reference, %options, {@options}. If a subroutine will process an array reference, @options, [@options], that subroutine will also process a hash reference, %options, {@options}. See the description for a subroutine for details and exceptions.