Free mmap software for linux

Cache::Mmap Perl module helps you to share data cache using memory mapped files.

SYNOPSIS

use Cache::Mmap;

$cache=Cache::Mmap->new($filename,%options);

$val1=$cache->read($key1);
$cache->write($key2,$val2);
$cache->delete($key3);

This module implements a shared data cache, using memory mapped files. If routines are provided which interact with the underlying data, access to the cache is completely transparent, and the module handles all the details of refreshing cache contents, and updating underlying data, if necessary.
Cache entries are assigned to "buckets" within the cache file, depending on the key. Within each bucket, entries are stored approximately in order of last access, so that frequently accessed entries will move to the head of the bucket, thus decreasing access time. Concurrent accesses to the same bucket are prevented by file locking of the relevant section of the cache file.

CONFIGURATION METHODS

These methods are used to examine/update the configuration of a cache. Most of these methods are read-only, and the value returned may be different to that passed to the constructor, since the cache may have been created by an earlier process which specified different parameters.

buckets()

Returns the number of buckets in the cache file.

bucketsize()

Returns the size of buckets (in bytes) in the cache file.

cachenegative()

Returns true if items not found in the underlying data are cached anyway.

context()

Returns the context data for reads and writes to the underlying data.

context($context)

Provides new context data for reads and writes to the underlying data.

expiry()

Returns the time in seconds cache entries are considered valid for, or zero for indefinite validity.

pagesize()

Returns the page size (in bytes) of the cache file.

strings()

Returns true if the cache stores strings rather than references.

writethrough()

Returns true if items written to the cache are immediately written to the underlying data.

libyama is a malloc implementation that bundles leak tracking by auditing allocations, array bound write detection, detection of access to freed memory, free/realloc on invalid pointers. It grew out of an attempt to build array bound overwrite detection into LeakTrac, a leak tracker I wrote earlier.
Since LeakTrac performed only book keeping and not any allocation, it was not possible to make LeakTrac detect array bound write, hence an allocator that does; and also tracks leaks.
You can think of it as merging LeakTrac and code that was inspired by ElectricFence.
If you only want to track leaks, then LeakTrac is just what you need. If you want an allocator that lets you debug also, then YaMa is the one for you. Sure, there are other similar things on the block - which is why this is Yet another Memory allocator.
Main features:
- provides malloc, calloc, realloc and free
- tracks and reports leaks
- detects accesses beyond allocated memory
- detects accesses to freed memory
- detects free on non-malloced pointers, NULL pointers
- detects realloc on non-malloced pointers
Installation:
Currently YaMa is written for Linux x86. Youll need a kernel that provides mmap etc, and gcc that can build ELF shared libraries. You also need to install binutils, if you havent already (very unlikely) And youll need the /proc filesystem.
To install YaMa
Unpack the archive
Edit Makefile; the only things you need to change are INSTALLDIR and BT.
make lib to make
make install to install. Youll need write permission on INSTALLDIR.
You may need to run ldconfig
Usage:
YaMa contains libyama.so, a shared library installed into INSTALLDIR by make install. The library provides malloc, calloc, realloc and free which can be used as replacements to their libc implementations.
A program can be linked against libyama on the command line (-lyama). A more interesting way to use libyama would be to preload it using LD_PRELOAD. The command line
LD_PRELOAD=libyama.so
would cause all calls to malloc, calloc, realloc and free originating from the program being run to be handled by the YaMa implementation. If libyama.so doesnt show up on ldconfig -p or it is not on your LD_LIBRARY_PATH, you need to specify the full path length.
When the program errs by accessing memory across an array bound, i.e, beyond the memory allocated using malloc/calloc/realloc, it receives a SIGSEGV. An access to freed memory also results in a SIGSEGV. If youve compiled your program using -g, you can locate the offending statement using any debugger on the core file. Note that overwrites on statically allocated arrays are not detected by YaMa.
If the program peforms an invalid free or realloc, an "Alert!" message is written to stderr, along with the call chain till the free/realloc. The call fails.
Upon normal program termination, either thro exit or return from main, a summary of leaks is printed on stderr. The size of each chunk of memory that remains un-freed at the end of the program is reported, along with the call chain till the allocation.
The behaviour of YaMa on malloc (0) (or calloc (x, 0)) is controlled by the environment variable ALLOWMALLOCZERO. If this variable is set to values = 2, YaMa returns a valid pointer and no warning is printed. Writes to this pointer, too, will fail. If ALLOWMALLOCZERO is not set, the behaviour is identical to ALLOWMALLOCZERO = 0.
Enhancements:
- Fixed a bug in the stack backtrace code which would crash libyama. Compile time control of backtracing (Ive found it useful at times not to have tracing).

camE is a Webcam grabber designed for video4linux devices. It is based on the xawtv webcam app, but extended to use imlib2 for applying antialised, blended truetype fonts to the image before upload.
A large number of new options have been added including scp support (in addition to ftp), image archiving, and much more. It runs as a daemon and needs no X connection to operate.
Main features:
- ttf fonts
- blended transparent text
- title text
- options in ~/.camErc
- message read from separate file (eg echo "eating my dinner" > ~/.caminfo)
- color the text as you like
- text background any color/alpha value
- scp support for uploading
- ftp support for uploading using libcurl, reconnect for each upload or reuse one connection
- proper logging to file
- user defined actions to be run before and after shots, and after upload (use to play sounds, beep, run scripts, copy files, whatever)
- file for stopping uploads temporarily (touch ~/BLOCKCAM to stop shooting, remove it to resume)
- now a proper daemon that can run without X forever if wanted
- control of brightness, contrast, gamma, hue, colour, etc
- control of framerate and use of snapshot mode for Philips cams
- image archiving (timestamped into a directory of your choice after upload), every frame or every n frames
- lag reduction (for some cams, when mmaped, the image grabbed lags behind the cam by up to 20 seconds. I use lag_reduce 5 in my ~/.camErc for example. This takes 5 shots in a row, which clears the mmap buffers, and then keeps the last shot, lag-free)

Bit-mapped Japanese font parser is a font parser. Note, this package doesnt include the actual font data. To get the font data you need to download it from the download section in the left.
Then move *.jfr into the directory where you unpacked this parser, and follow with the quick instructions.
Quick instructions:
Complete parse requires about 4 megabytes of free disk space. This is a huge improvement over the original version which required almost 45 megabytes.
1. make
2. make parse
3. watch the progress indicator
4. mv *.pcf.gz /usr/X11R/lib/X11R6/fonts/misc
5. make clean
6. HUP your font server if you use one
7. xset fp rehash
8. xlsfonts | grep kanji
/usr/X11R/lib/X11R6/fonts/misc is the standard location for all sorts of random bit-mapped fonts, but you might have a special location. Substitute that in step 4.
About:
I came across a number of these "raster fonts" a while ago. Quick look inside the files proved that they are bit-mapped fonts, and the format looked pretty straight-forward. I wrote the original parser for these just guessing the values, basically by experimenting and playing around. Later on I came across some docs on the subject - looks like these fonts were used in Windows 3.1 Japanese edition to substitute back-then low quality Japanese TTF fonts at small point sizes. These were designed using full-scale 16 bit programming techniques.
Quick info about the font format, there are some headers, then follows a "segment table" which is basically a table with pointers inside the font file where to locate a particular chunk of data. Because the 16 bit way of accessing memory is by using 65k "segments", each file is virtually split into < 65k segments which get loaded into separate memory areas, and then there is a algorithm how to assemble whatever character by using the segment number and offset. Anyway, with 32 bit access all of that doesnt really matter. In my implementation I just mmap the whole file and read it all out of memory.
Generating table.h was a LOT of work! First, I took the codearea table out of one of the jfr files (this maps shift-jis code to the character number inside the font file), and extracted the number ranges. These were shift-jis, of course, and X uses jis0208. There is no converter from a shift
jis byte into jis0208. So I had to write one. Taking iconv, and some tables from glibc 2.1.93, I hacked together something which converted the shift-jis data into ucs4 (unicode, I guess) and then from that into jis0208. The code to the converter is about 500k thanks to the huge jis->unicode->jis conversion tables, and you wont need it unless you get a jfr font with a different encoding table (unlikely). Anyway. After I got the font format figured out and converted the character table, everything else was pretty easy. Note some bit hackery in the bitmapXX() functions which was necessary to present the font data in a usable format. Also notice cool use of function pointers to select a conversion function at runtime.
Enhancements:
- This version uses correct JISX0208 tables, and is much faster.

FlipDCD is a small utility for reversing the endianism of binary DCD trajectory files from Charmm, and NAMD. This can be useful when running simulations on one architecture and visualizing or analyzing the results on another.

FixDCD is a tiny utility to modify the header of an X-PLOR DCD file to make it readable by programs expecting Charmm DCD files, at the expense of a Timestep size value in the header.

FlipDCD provides a mechanism for converting the endianism (byte ordering) of CHARMM, X-PLOR, and NAMD DCD trajectory files so that they may be loaded by visualization and analysis programs on platforms with the opposite byte ordering of the platform on which they were originally generated.

This allows one to use a Windows PC to read DCD trajectories generated on a Sun or an SGI and allows a Sun or an SGI to read trajectory files produced on a PC cluster running Linux.

FlipDCD does the endianness conversion by memory mapping the DCD file with mmap(), and converting the endianism in-place. This provides a relatively high performance method to perform endianness conversion.

FlipDCD can be used to report the endian status of DCD files or to force a particular endianness without regard for the origin of the DCD files.

FlipDCD Usage:
flipdcd [-s] [-B] [-L] file . . .
where "file" can be a list of files.

The default behavior is to flip the byte ordering. Other options are:
-s report the byte-order status of each file without changing it
-B make/keep each file big-endian
-L make/keep each file little-endian
The options are mutually exclusive; the last one read is used.

FixDCD changes the header on an X-PLOR style DCD files so that they can be read by tools which expect CHARMM formatted trajectory files. As with FlipDCD, FixDCD performs the conversion in-place. This conversion is not reversible so you may wish to make a backup copy of your file(s).

Dovecot project is an open source POP3 and IMAP server for Linux/UNIX-like systems, written with security primarily in mind. Although its written in C, it uses several coding techniques to avoid most of the common pitfalls.
Dovecot can work with standard mbox and maildir formats and its fully compatible with UW-IMAP and Courier IMAP servers implementation of them as well as mail clients accessing the mailboxes directly.
Its easy to migrate from them to Dovecot. I have also plans to support storing mails in SQL databases.
Dovecot is easy to set up and doesnt require special maintenance. Only thing you need is to get the authentication working properly - if your users are in /etc/passwd theres hardly anything you have to do.
Dovecot should be pretty fast. There are still some optimizations that could be done, but I believe it already beats most other IMAP servers in overall performance.
This is mostly because of index files that Dovecot maintains; instead of having to scan through all the data in mailbox, Dovecot can get most of the wanted information from index with little effort.
Dovecots indexes can scale to huge amount of messages per mailbox with hardly any noticeable slowdown. Ive tested only up to 367000 mails, but millions of messages should be no problem.
Dovecot takes very little memory. Most of it goes to mmap()ed index and mailbox files, meaning that if operating system is low on memory, it can simply drop those memory pages without having to store them in swap.
Connections are handled in separate processes, each one currently using around 100kB of swappable memory. Some extensions like SORT and THREAD will require more memory to work though.
Dovecot is fail safe. Indexes could potentially be quite a large problem maker, but Dovecot does sanity checks to all data before using it to avoid crashes and other problems.
Any kind of crash is considered as bug and will be fixed - even if it happens only by deliberately poking the index files.
Main features:
- Dovecot should be quite ready for use with normal IMAP clients.
- Complete IMAP4rev1 support.
- Supports THREAD and SORT extensions, required by many IMAP webmails.
- Complete TLS/SSL support
- IPv6 ready.
- Shared mailboxes arent yet supported.
- Maildir++ quota isnt yet supported. Hard filesystem quota can also be problematic.

dcache is a database library implementing a persistent first-in-first-out cache.
The database (or `cache) size limits are configured at creation time, and as soon as they are reached the oldest records are automatically removed.
dcache library has a number of commandline tools for managing the database.
dcache is a disk based caching database in a single file containing a header and fixed sized hash and data space.
The header holds variables and constants describing the database.
The hash table holds pointers into the data area and hashes of the keys. The hash function used is a crc32.
The data area consists of the data, the key, a 64bit number with application defined use, and two 32bit numbers holding key and data lengths.
The header and hash table are mapped into process address space using the mmap(2) system call. The data area is not memory mapped, thus allowing to handle multi-gigabyte files even on systems with a 32bit address space.
Main features:
- The database size is limited to 63 bits on operating systems supporting large files.
- The database size is limited to 31 bits (2 gigabytes) on operating systems not supporting large files.
- The sum of the length of key, data and record overhead is limited to 31 bits (2 gigabytes).
- The record overhead in the data area is 16 bytes.
- The number of possible keys is limited by the address space of the process, or somewhat over 200 million, whatever comes first.
- Storage of numbers is done in big endian byte order (network byte order). Databases are portable between hosts with different byte orders.
- The size of the cache and the number of records in it are set at creation time.
- The cache is cleaned up automatically, removing old records if theres not enough space for data or records.
- Deletions are supported.
- Multiple readers and writers are supported, provided that the applications lock to database.
- Keys need not be unique.
The dcache library files are published under the GNU Lesser General Public License (dont hesitate to ask me if this is a problem for you). The tools are published under the GNU GENERAL PUBLIC LICENSE.
Enhancements:
- This release fixes a few compiler and linker warnings, among them those breaking the self check when using dietlibc.
- No functional changes were done.