Free database clustering software for linux

Document clustering project is a data mining suite to cluster a document set. This set of tools was implemented from a series of papers: "Clustering Web Pages Semantically using Combinatorial Topology", "Data mining using granular computing", and "A fast association rule algorithm based on bitmap and granular computing".
Enhancements:
- A bug with hash table has been fixed.

Image Cluster copies and renames images based on Exif data and file number names. Image Cluster also clusters those images into directories based on a variable sliding window (with a default of 36 hours), which makes it easy to group images based on events without manual intervention.

The inspiration for this program came from recently getting a new Canon SD500 camera to replace my Canon S30 that Id had for years. The upside, the Canon SD500 rocks! The downside, I now have 2 cameras that are burning through the same sequence numbers, so my previous solution of just putting all the files in to directories by the first 2 digits of the sequence numbers was no longer going to work.

Imagecluster solves this problem, plus another grouping problem that Id been thinking about, by extracting the CreateDate and FileNumber exif tags from the images, and using that as the basis of a new filename (typically YYYY:mm:dd_HH:MM:SS_FileNumber.jpg). This ensures that 2 images taken at the same second have an even smaller chance of colliding, as their camera sequence numbers would have to also be the same at that second.

But that is just the first step. I have noticed that I am an occational photographer, so take pictures in bursts, often for a weekend of hanging out with folks, though sometimes for a vacation as well. This got me thinking. What I really needed is a tool that also creates directories that allows for some minimum tollerance between CreateDate, that is used to cluster images. For me, the optimum value seems to be 36 hours, though this is configurable via the command line.

This took me an afternoon to pull together, Im sure it could be smarter, but it is useful enough to post for others to use.

Options:

-d directory
Set the target directory for images. Defaults to /tmp/photos, which is probably not what you want.

-D
Dryrun. Tells you what the program would have done.

-h
Print out help message

-s
Seperator character. It defaults to : (i.e. 2005:10:09...), but is user configurable because my friend Clemens wants to use - (i.e. 2005-10-09) instead.

-t
Set the tollerance for image clustering. This is the maximum time between any 2 pictures in a cluster, which will cause a new cluster to be created. The name of the cluster will be YYYY:MM:DD of the first image in the cluster, even if it spans multiple days. Because this tollerance is the maximum time between any two images in the cluster, it is possible that all images you have ever taken could be in 1 cluster, if you took a picture every day of your life. Hence, this feature isnt useful to everyone. If you are that kind of person, set tollerance to 16 hours or something, and youll tend to get 1 day sized buckets.

-v
Prints verbose output

MP3DB2 is a collection of bash scripts for keeping track of large MP3 collections. It will retrieve song information from the filenames and ID3 tags of a directory or CD of MP3s and store them in a local database.
This program is a complete rewrite of my MP3 database version 1. Version 1 only stored the data that was present in the filenames while version 2 actually retrieves information from the ID3 tags of each MP3.
Please note that as of this version the database format is NOT considered carved in stone. I have already added a few fields mid-development and this will continue for a while if I think the format can be made better. See the changelog for a list of changes to the database format since V2.0.
Enhancements:
- mp3dupe: now just displays the name of the directory we are checking not the full path in the host filesystem.

Rocks Cluster is a complete "cluster on a CD" solution for x86 and IA64 Red Hat Linux COTS clusters.
Building a Rocks cluster does not require any experience in clustering, yet a cluster architect will find a flexible and programmatic way to redesign the entire software stack just below the surface (appropriately hidden from the majority of users).
Although Rocks includes the tools expected from any clustering software stack (PBS, Maui, GM support, Ganglia, etc), it is unique in its simplicity of installation.
From a hardware component and raw processing power perspective, commodity clusters are phenomenal price/performance compute engines. However, if a scalable ``cluster management strategy is not adopted, the favorable economics of clusters are offset by the additional on-going personnel costs involved to ``care and feed for the machine. The complexity of cluster management (e.g., determining if all nodes have a consistent set of software) often overwhelms part-time cluster administrators, who are usually domain application scientists. When this occurs, machine state is forced to either of two extremes: the cluster is not stable due to configuration problems, or software becomes stale, security holes abound, and known software bugs remain unpatched.
While earlier clustering toolkits expend a great deal of effort (i.e., software) to compare configurations of nodes, Rocks makes complete Operating System (OS) installation on a node the basic management tool. With attention to complete automation of this process, it becomes faster to reinstall all nodes to a known configuration than it is to determine if nodes were out of synchronization in the first place. Unlike a users desktop, the OS on a cluster node is considered to be soft state that can be changed and/or updated rapidly.
This is clearly more heavywieght than the philosophy of configuration management tools [Cfengine] that perform exhaustive examination and parity checking of an installed OS. At first glance, it seems wrong to reinstall the OS when a configuration parameter needs to be changed. Indeed, for a single node this might seem too severe. However, this approach scales exceptionally well, making it a preferred mode for even a modest-sized cluster. Because the OS can be installed from scratch in a short period of time, different (and perhaps incompatible) application-specific configurations can easily be installed on nodes. In addition, this structure insures any upgrade will not interfere with actively running jobs.
One of the key ingredients of Rocks is a robust mechanism to produce customized distributions (with security patches pre-applied) that define the complete set of software for a particular node. A cluster may require several node types including compute nodes, frontend nodes file servers, and monitoring nodes. Each of these roles requires a specialized software set. Within a distribution, different node types are defined with a machine specific Red Hat Kickstart file, made from a Rocks Kickstart Graph.
A Kickstart file is a text-based description of all the software packages and software configuration to be deployed on a node. The Rocks Kickstart Graph is an XML-based tree structure used to define RedHat Kickstart files. By using a graph, Rocks can efficiently define node types without duplicating shared components. Similiar to mammalian species sharing 80% of their genes, Rocks node types share much of their software set. The Rocks Kickstart Graph easily defines the differences between node types without duplicating the description of their similarities. See the Bibliography section for papers that describe the design of this structure in more depth.
By leveraging this installation technology, we can abstract out many of the hardware differences and allow the Kickstart process to autodetect the correct hardware modules to load (e.g., disk subsystem type: SCSI, IDE, integrated RAID adapter; Ethernet interfaces; and high-speed network interfaces). Further, we benefit from the robust and rich support that commercial Linux distributions must have to be viable in todays rapidly advancing marketplace.
Wherever possible, Rocks uses automatic methods to determine configuration differences. Yet, because clusters are unified machines, there are a few services that require ``global knowledge of the machine -- e.g., a listing of all compute nodes for the hosts database and queuing system. Rocks uses an SQL database to store the definitions of these global configurations and then generates database reports to create service-specific configuration files (e.g., DHCP configuration file, /etc/hosts, and PBS nodes file).
Enhancements:
- Rocks v4.3 is released for i386 and x86_64 CPU architectures. New features: Rocks command line - initial release of the Rocks command line which facilitates non-SQL administrative access to the database; PXE First - hosts can now be configured in BIOS with a boot order of CD, PXE, hard disk. Enhancements: based on CentOS 4.5 and all updates as of July 4, 2007; Anaconda installer updated to 10.1.1.63; performance improvement when building torrent files for the Avalanche Installer; database indirects, more flexibility with Rocks variables; Globus updated to gt4.0.4 with web services....

OSCAR version 4.0 is a snapshot of the best known methods for building, programming, and using clusters. OSCAR Cluster project consists of a fully integrated and easy to install software bundle designed for high performance cluster computing.

Everything needed to install, build, maintain, and use a modest sized Linux cluster is included in the suite, making it unnecessary to download or even install any individual software packages on your cluster.

Large Database Backup is a PHP class can be used to backup large MySQL databases into multiple files.

It can list the tables of a given MySQL database and generate SQL statements to create and insert records with the current values of the table rows, as if it was creating a new database. The class stores the generated SQL statements in files.

A full backup process can be split in multiple iterations to not exceed PHP default script execution time limit. Each iteration dumps a limited number of table rows. An iteration can be executed by a script that redirects the page request to itself at the end to proceed to the next iteration.

Media Database is a media database that uses PHP for the GUI and MySQL for data storage.

It has movie, series, and anime listings. Media Database has lending and wishlist capabilities.

Installation:

Unpack mediadb-*.tar.gz to www dir or public_html dir.
Edit config.php, change mysql_user_name, mysql_password and mysql_server_host to match your MySQL database username, password and host.
run url: http://url_to_mediadatabase/install_mediadb.php, and follow the link and log in as admin and make default yourself a user.

*******************************************************************
****** Dont use the admin user for adding films and stuff. ******
*******************************************************************

If there are any problems, please send a short description of where it occurs and the error message to my email addres: marco (at) cakebox (dot) net, marc the topic with [mediadb]: (something you find fitting)

XML::DB::Database::Exist is a Perl module with XML:DB driver for the eXist database.

SYNOPSIS

use XML::DB::Database::Exist;

This is the eXist XML-RPC driver. It is intended to be used through the XML:DB API, so that it is never called directly from user code. It implements the internal API defined in XML::DB::Database

The methods required to implement the Database interface are documented in Database.pm; only methods unique to eXist, and not required by the XML:DB API are documented here.