Free storage software for linux

PicoStorage allows you to store hierarhical information (similar to "files and directories") inside a single file. The functionality offered is largely equivalent to the one offered by any filesystem, or by the Structured Storage and Compound Files.

PicoStorage can efficiently handle huge numbers of small files, with very economical disk usage; it also allows you to keep open (in RAM) simultaneously a large number of files. Transaction support guarantees data integrity.

Learn more about the distinctive advantages of PicoStorage. or look at the benchmark.

The library is available on a dual-license basis: under GPL for free, and under a commercial license for use in closed-source applications.

Using

The library contains the classes File and Dir to represent files and directories. On a File you can read or write a number of bytes from a given offset, and set/get the file size. On a Dir you can create entries (either files or subdirectories), open entries, delete entries, and iterate over the directorys content.

The storage itself (i.e. the whole hierarchical structure, contained in a filesystem file) is represented by the class Storage. Using this class, you can create or open a storage, obtain the root directory of the storage, close the storage and do commit or rollback.

cfgstoragemaker remotely generates an MRTG config file in order to graph all storage devices (disk, memory, and swap) of one or more specific host(s) via SNMP.

Twisted Storage is an application that will take your data and manage it for you, just like a file system. But Twisted Storage goes way beyond what a normal file system does.
What types of "objects" are stored in Twisted Storage : just regular files. The data stored in the system is expected to be fixed content, meaning it doesnt change at all. If you read the data back, change it, and store it, it is considered a different file. There are plenty of examples of fixed content data : emails you received, movies or mp3 files you made or own, documents you wrote, etc. In fact it is estimated that 80% of the data you have on your computer is fixed, reference data.
Your Twisted Storage system can start out small - a single computer with a few disk drives. When you need more storage you can add more disk drives and more computers. But it has been designed to store petabytes worth of data without requiring any special hardware. For example storing a petabyte worth of data, using the biggest disk drives available today (500 GB), would require 2000 disk drives. To overcome the need of special hardware, Twisted Storage is a scalable, distributed system using ordinary commodity computers. Even so that petabyte of data would require 500 computers networked together and that requires special handling.
Twisted Storage has a few features you wont find in any file system. For example once a file is added to the Twisted Storage repository it can be written in multiple locations simultaneously. The system has been designed to be compliant with Sarbanes-Oxley Act: when an object is added to the repository you can no longer delete it and the integrity of the contents are guaranteed. In addition any access to the object is recorded.
Main features:
Scalable to thousands of servers and multiple petabytes of data.
- Regardless of how much data you want to store,Twisted Storage is designed to handle it. The system can grow from one computer and a few disks to hundreds of systems and thousands of disk drivers.
Automatic data recovery without human intervention.
- Twisted Storage doesnt require backup/restore software or procedures. All data written to the system is recorded in multiple locations.
No special disaster recover procedures.
- So long as you have your systems in physically different locations and you have instructed the software to write the multiple copies to those systems, you have no-hassle disaster recover. If one of the sites goes down, the system will retrieve any data from the other systems. Additionally, once the disaster site is up and running, the system will automatically restore the data to the it.
Simplified configuration.
- To ease management of the system, you can change any one systems configuration and have it ripple through out the system.

Storage Resource Broker is client-server middleware that provides a uniform interface for connecting to heterogeneous data resources over a network and accessing replicated data sets.
SRB, in conjunction with the Metadata Catalog (MCAT), provides a way to access data sets and resources based on their attributes and/or logical names rather than their names or physical locations.
Starting with SRB 2.1.1 we now have an install script, install.pl, that can do a complete Postgres, MCAT and SRB installation. See README.MCAT.INSTALL. With SRB 3.0.0, this script can run on Solaris too, as well as the original Linux and Mac OS X.
The SEA authentication system is no longer recommended; for the secure authentication use either the ENCRYPT1 form of MDAS_AUTH authentication, or GSI. It should be noted that if the SEA authentication scheme is to be used and if the SEA library (libsea.a) does not already exist on your build platform, the SEA software that can be downloaded at the
URL must be built first.
1) Build configuration.
This version uses the configure script to configure the build. Most of the configurable parameters for building the SRB server and client can be configured using the "./configure" script. Run "./configure --help" for more information.
All configurable parameters for building the SRB server and the client library, including those set by the configure script, are defined in the mk/mk.config.in file. (The configure script automatically generates a third file, mk/mk.config, using mk/mk.config.in as a
template.)
Those parameters that cannot be modified via the configure script (because flags for those parameters have not yet been implemented) are set by directly editing the mk/mk.config.in file prior to running "./configure". Comments in the mk/mk.config.in file make it clear whether or not a particular parameter can be set through the configure
script, and if so, how to do so.
NOTE: The configure script does a number of self tests before the configuration is carried out. One of the test it does is the compiler test which it assumes "gcc" as the default compiler. If "gcc" is not installed or if the test of "gcc" failed (which happened on an AIX platform), the configure script should be re-run with the env variable CC set to cc or other compilers. This will override the default in the compiler test.
If the configure script still failed, do the following:
a) cd SRB2_0_0rel
b) ./config.rescue
c) edit the mk/mk.config file
2) Configure examples
a) Non-MCAT-enabled server and client, type in
configure
This will configure the mk.config file to build a non-MCAT enabled
server and client with the default settings.
b) Non-MCAT-enabled server and client with java enabled, type in
configure --enable-javagui=yes --enable-jdkhome=/usr/local/apps/jdk1.4.1
where /usr/local/apps/jdk1.4.1 is the directory where the JAVA JDK 1.4.1
is installed.
c) MCAT-enabled server with Oracle 8.1.5 MCAT, type in
configure --enable-oramcat --enable-oraver=815
--enable-orahome=/usr/local/apps/oracle/product/8.1.7
where /usr/local/apps/oracle/product/8.1.7 is the Oracle home
directory.
2) Parameters in the mk/mk.config file
The SRB architecture supports multiple SRB servers running on various hosts. Each SRB server may be built with different options, as set by the configure script and/or defined in the mk/mk.config.in file. For example, the SRB server on host A may include the driver for accessing HPSS and the SRB server on host B may include the driver for accessing the Lobj stored in DB2, etc.
The parameters are self-explanatory through the comments given in this file. Some of the more important parameters are discussed below:
installDir - The absolute path of the SRB install directory.
PORTNAME - The OS platform of this SRB port. Currently, the SRB software runs on 8 platforms. i.e., valid PORTNAMEs are :
PORTNAME_solaris, PORTNAME_sunos, PORTNAME_linux, PORTNAME_osx,
PORTNAME_aix, PORTNAME_alpha, PORTNAME_c90 and PORTNAME_sgi.
SRB_LARGEFILE64 - defines whether the 64 bit file size is supported by the underlining driver of this SRB server. Current, 64 bit file size is supported by the ORTNAME_solaris, PORTNAME_aix, PORTNAME_linux and PORTNAME_c90 platforms.
ORAMCAT - defines that this SRB server being built is MDAS enabled and the MCAT is stored in Oracle DBMS. Normally, only one SRB server is MDAS enabled.
DB2MCAT - defines that this SRB server being built is MDAS enabled and the MCAT is stored in Oracle DBMS. Normally, only one SRB server is MDAS enabled.
NOTE : Both ORAMCAT and DB2MCAT cannot be defined at the same time.
ADDR_64BIT - defines whether to compile for 64 bits address. This option has only been tested for the the Solaris and Linux platforms.
PARA_OPR - defines whether this SRB server support parallel operation API.
MDAS_AUTH - defines whether the plain text and encrypted password MDAS authorization scheme will be supported. If used, the user/passwd pair registered with the MDAS catalog will be used to authenticate a user. Comment it out if the SRB server does not support MDAS authorization.
NOTE : A server can be built to support either MDAS_AUTH (plain or encrypted (ENCRYPT1)) or GSI_AUTH, or both.
SEA_AUTH - defines whether SEA authorization scheme will be supported. The software can be configured to support both SEA_AUTH and MDAS_AUTH. (SEA is no longer recommended.)
LIB_SEA - Is needed only if SEA_AUTH is defined. LIB_SEA specifies where the SEA client library is located.
GSI_AUTH - defines whether the GSI authentication scheme is supported. This is set when --enable-gsi-auth is included on the configure line.
NOTE : A server can be built to support either MDAS_AUTH or GSI_AUTH,
or both.
LIB_GSI_AUTH - Set by configure when --enable-gsi-auth is included (i.e. GSI_AUTH is defined). LIB_GSI_AUTH specifies where the GSI client libraries are located. The optional configure --enable-globus-location=path can also be used specify the parent
location of the GSI libaries, and will cause LIB_GSI_AUTH to be adjusted.
JAVA_GUI and javaDir - JAVA_GUI defines whether the srbBrowser should be built. javaDir specifies the directory where the JDK software is installed. (e.g. /usr/local/apps/Java). See README.srbBrowser for more details.
3) "cd" to the main SRB directory and type in "gmake clean" and then "gmake" to make the SRB software. The Makefile contains various options to make and clean all or a subset of the build.
- gmake --- build all.
- gmake clean --- clean all.
- gmake srb --- build only the SRB server and client.
- gmake clean_srb --- clean only the SRB server and client.
- gmake util --- build only the utilities (S commands). See README.utilities for more details.
- gmake clean_util --- clean only the utilities.
- gmake browser - build only the java srbBrowser GUI. See README.srbBrowser for more details.
- gmake clean_browser - clean only the java srbBrowser.
4) (Optional) Type in "gmake install" to install the software in the $(installDir) directory. This procedure installs the following modules in the $(installDir) directory:
bin/runsrb - The script that starts the SRB
bin/srbMaster2_0_0 - The frontend server.
bin/srbServer - The backend server (forked by the srbMaster1_0 for each client connection).
bin/libSrbClient.a - The client library.
data/hostAuthConfig - The optional (needed only if HOST_BASED_AUTH in the mk.config file is set) host based authorization configuration file.
data/mcatHost - This file identifies the host on which the MCAT enabled SRB server is running.
data/hostConfig - This is the optional SRB host configuration file. It is only needed when when you want to add aliases to your local hostName.
data/hpssCosConfig - This is the optional HPSS Class of Services configuration file. It is only needed if HPSS in the mk.config file is set.
data/hpssNodceAuth - The file contains authentication info for non-dce HPSS. It is only needed if the HPSS and NO_DCE flags in the mk.config file are set.
data/MdasConfig - The MDAS configuration file.
data/metadata.fkrel - This file defines the foreign key relationship between the MDAS catalog tables and is used internally by the SRB for query generation. This file should not be changed between releases.
data/LobjConfig - The database configuration file for the DB Large Object driver. Basically, it contains the userID and password for accessing each database server.
Enhancements:
- Three vulnerabilities that allow SRB users to read/write non-Vault files that are readable/writable by the the srbadmin user were fixed.
- A bug that causes the GridFTP driver to use the wrong credential to connect to GridFtp server was fixed.
- A file descriptor lock bug was fixed.
- Uploading files larger than 2 gigabytes into GridFtp resources now works.
- Timeout bugs that could arise when sending large numbers of files were fixed.
- A core dump problem for HPSS type resources involving parallel I/O on Linux servers was fixed.
- A new option -o was added to show collection ownership in SgetColl.

BerkeleyDB Backend Storage Engine for DURUS project provides a storage engine for DURUS, a persistence system for the Python programming language.
Some advantages compared to Durus standard FileStorage:
- Startup time is negligible.
- You dont need an in-memory index, so your repository size is only limited by storage space, not RAM.
- If you change existing objects, your storage size doesnt increase.
- If you delete objects, those objects are garbage collected in background, slowly, without performance degradation.
- You can still do a full fast collection, if you need it. While this collection is in progress, Durus still serves objects. From time to time, nevertheless, it can be unresponsible for 2 or 3 seconds, while is doing a checkpointing to be able to free database logging diskspace.
- Garbage collection doesnt increase storage size. Neither RAM usage.
- Garbage collection deletes objects using nondurable transactions, very efficiently. If the collection is aborted abruptly (program or machine crashes), the collection will start again from the beginning. If the GC finishes without problems, that state is durable.
- Any object store in the storage will commit a durable transaction, including all objects released in the background garbage collector, along the way.
- Garbage collection time is proportional to garbage, not repository size.
There are some disadvantages, nevertheless:
- IMPORTANT: This backend uses reference counting to decide when an object is garbage and can be collected. So, if you have cycles in your data structures, you **MUST*BREAK** them before releasing the objects.
- Failing to do that will leak diskspace. It is possible that in a future release we can collect cycles, but try to avoid that pattern.
- Leaking objects will grow the diskspace, but **NO** corruption or malfunction will happen. No other secondary effect.
- Although this code could work on Windows, I havent checked it. Absolutely no garantee!.
- Sometimes this backend can become irresponsible for a couple of seconds. It is busy doing a checkpoint to recycle database logging space. The pause should be sort, nevertheless.
- Dont use this storage backend over NFS, at least you know what is going on.
- Since we are using BerkeleyDB as the backend:
- You should be experienced with BerkeleyDB deployments.
- Beware when updating Python or BerkeleyDB. In particular, BerkeleyDB is known by breaking (but they DOCUMENT!) binary compatibility between versions. In this case, they ALWAYS document the procedure to do a controlled upgrade, so dont worry. But take note of the risk.
- To do a trustable backup, you should follow instructions in BerkeleyDB documentation:
- http://www.sleepycat.com/docs/ref/transapp/reclimit.html
- http://www.sleepycat.com/docs/ref/transapp/archival.html
- http://www.sleepycat.com/docs/utility/db_hotbackup.html
- In Python you can use the standard "bsddb" or the up-to-date "bsddb3" bindings (which will be included in new python versions). This product will try to use always the more recent BerkeleyDB bindings. Be careful about BerkeleyDB version changes when you update the bindings.
- Since BerkeleyDB files are binary structures, a corrupt database can be unrecoverable. Be diligent and careful with your backups.
You can use this product both as a normal (local) filestorage, or a server (remote) storage system, just like the usual Durus FileStorage.
Enhancements:
- Compatibility with Durus 3.7 was added.

Class::DataStore is a Perl module for generic OO data storage/retrieval.

SYNOPSIS

my %values = ( one => 1, two => 2 );
my $store = Class::DataStore->new( %values );

# using get/set methods
$store->set( three, 3 );
my $three = $store->get( three );

# using AUTOLOAD method
$store->four( 4 );
my $four = $store->four;
my @four = $store->four; # returns a list

my $exists = $store->exists( three ); # $exists = 1
my $data_hashref = $store->dump;
$store->clear;

Class::DataStore implements a simple storage system for object data. This data can be accessed via get/set methods and AUTOLOAD. AUTOLOAD calls are not added to the symbol table, so using get/set will be faster. Using AUTOLOAD also means that you will not be able to store data with a key that is already used by a instance method, such as "get" or "exists".
This module was written originally as part of a website framework that was used for the Democratic National Committee website in 2004. Some of the implementations here, such as get() optionally returning a list if called in array context, reflect the way this module was originally used for building web applications.

Class::DataStore is most useful when subclassed. To preserve the AUTOLOAD functionality, be sure to add the following when setting up the subclass:

use base Class::DataStore;
*AUTOLOAD = &Class::DataStore::AUTOLOAD;
This module is also a useful add-on for modules that need quick and simple data storage, e.g. to store configuration data:
$self->{_config} = Class::Datastore->new( $config_data );
sub config { return $_[0]->{_config}; }
my $server = $self->config->server;
my $sender = $self->config->get( sender );

GlusterFS package contains clustered file storage that can scale to peta bytes. GlusterFS is a programmable system. With little thinking, you can even redesign the GlusterFS file system by re-arranging the GlusterFS components using translator interface. It is all achieved through volume specification file. This allows GlusterFS to be flexible for all kinds of storage needs. Even with all these advanced features, GlusterFS is very easy to setup and manage.

Gluster is a GNU cluster distribution aimed at commoditizing Supercomputing and Superstorage. Core of the Gluster provides a platform for developing clustering applications tailored for a specific tasks such as HPC Clustering, Storage Clustering, Enterprise Provisioning, Database Clustering etc.

mod_repository turns Apache into a content storage system with the benefit of storing just one copy of each file submitted to the repository. Upon storing a file a user is given a unique token representing the file.

This can be used to create content servers which are abstracted from the filesystem. Files can be placed, retrieved, and deleted from the server.