Free unicode implementation software for linux

Suffix tree implementation library is a C library, an implementation of the suffix trees algorithm to store/retrieve key/data pairs.
The main advantages are a linear indexing time, little memory usage, and very fast retrieving.
It has been developped on FreeBSD/gcc but should be fairly portable.
The source code "testsfx.c" show an example of how to use the library both for inserting, retrieving, and deleting data. There arent many functions and comments should be enough to give you an idea of how to use the library. (read the header of the source file)
You should edit sfxdisk.h to suit your needs: you can change the alphabet size and the offset type. It should be OK to use "long long" 64 bits ints instead of long, in fact I tested it succesfully but havent gone to the point of filling more than 2 GB of data (needless to say you need a 64 bits filesystem).
Two "tools" come with the library (new with version 1.2): dumpsfx and loadsfx. dumpsfx is used to dump the database: dumpsfx [-s separator] if you want to output the result as readable text or dumpsfx < file.sfx > -h to output it for reloading with loadsfx.
dumpsfx outputs on stdout and loadsfx reads from stdin. loadsfx < suffix tree file to create > < dumped_file
Enhancements:
- removed an useless offset incrementation in STwritenode

Fast MD5 Implementation in Java is a heavily optimized implementation of the MD5 hashing algorithm written in Java.
Fast MD5 Implementation in Java includes an optional native method for even greater speed improvements.
How Fast Is It?
Short answer:Much faster than any other Java implementation that I have tested and (surprisingly) even faster than the native, non-Java MD5 implementation on some systems.
Long answer:First of all, it is important to note that the term "fast" is used here in relative terms. The implementation of the MD5 message digest algorithm available on this page is written in Java and is fast compared with other implementations written in Java, both because it is heavily optimized by itself and because there is an optional native method that makes it even faster when the platform supports it. How it compares to a sensible implementation written in a language, such as C, that is compiled directly to machine code, is heavily dependent upon how good of a job the JIT compiler in your JVM does in compiling the code or whether you are able to use the optional native method.
Enhancements:
- Martin West contributed a bug fix and some code refactoring to make all targets work out of the box in the Ant build file. Previously, the "dist" target did not work if the "docs" directory was not present.

Bellagio is a sample implementation of OpenMAX IL for Linux.
It enables software developers and ISVs to familiarize themselves with the OpenMAX IL API and to develop their own OpenMAX multimedia and streaming media components for mobile devices, including codecs, video I/O, and audio mixers.
Included sample components comply with the OpenMAX base and interoperability profiles and can be tunnelled together.
Main features:
- a shared library with the IL core and a "reference" OpenMAX component
- a number of OpenMAX components which pass Khronos conformance tests
- a set of GStreamer plugins that use the IL API (not available yet)
Enhancements:
New video components:
- ffmpeg based MPEG4/H.264 decoder
- color converter component YUV -> RGB
- video renderer based on devFB
New audio component:
- audio file reader based on ffmpeg audio format
- volume component
Fixed known bugs:
- FFMPEG audio decoder now works on FC6 and other distributions with the latest ffmpeg release (0.4.9-0.35.20070204)
Known pending bugs:
- some ogg streams can not be decoded properly
- the tunneling between file reader, mp3 dec based on ffmpeg - alsa sink ends in a deadlock sometimes.
- This behavior has been detected some times using FC6 and UBUNTU, not with the FC4
Full list of components:
Audio:
- ogg decoder based on libvorbis (stand alone components, and multiple roles component)
- mp3 decoder based on mad decoder
- mp3 decoder based on ffmpeg (multiple roles component)
- volume component
- alsa audio sink
- ffmpeg audio file reader (to be used with mp3 ffmpeg decoder)
Video:
- MPEG4 decoder based on ffmpeg (multiple roles component)
- H.264 decoder based on ffmpeg (multiple roles component)
- Color converter based on ffmpeg
- video renderer based on devFB
- Major additions to the 0.2
- New port classes
The components are:
- multiple formats audio decoder component that supports mp3 and ogg audio formats
- alsa sink component
- all the other components are NOT compatible with the new architecture.
- They have been removed and will be ported to the new architecture in a further delivery

Unicode::Unihan is the Unihan Data Base 5.0.0.

SYNOPSIS

use Unicode::Unihan;
my $db = new Unicode::Unihan;
print join("," => $db->Mandarin("x{5c0f}x{98fc}x{5f3e}"), "n";

ABSTRACT

This module provides a user-friendly interface to the Unicode Unihan Database 3.2. With this module, the Unihan database is as easy as shown in the SYNOPSIS above.

The first thing you do is make the database available. Just say

use Unicode::Unihan;
my $db = new Unicode::Unihan;

Thats all you have to say. After that, you can access the database via $db->tag($string) where tag is the tag in the Unihan Database, without k prefix.
$data = $db->tag($string) =item @data = $db->tag($string)

The first form (scalar context) returns the Unihan Database entry of the first character in $string. The second form (array context) checks the entry for each character in $string.

@data = $db->Mandarin("x{5c0f}x{98fc}x{5f3e}");
# @data is now (SHAO4 XIAO3,SI4,DAN4)

@data = $db->JapaneseKun("x{5c0f}x{98fc}x{5f3e}");
# @data is now (CHIISAI KO O,KAU YASHINAU,TAMA HAZUMU HIKU)

Unicode::MapUTF8 is a Perl module with conversions to and from arbitrary character sets and UTF8.

SYNOPSIS

use Unicode::MapUTF8 qw(to_utf8 from_utf8 utf8_supported_charset);

# Convert a string in ISO-8859-1 to UTF8
my $output = to_utf8({ -string => An example, -charset => ISO-8859-1 });

# Convert a string in UTF8 encoding to encoding ISO-8859-1
my $other = from_utf8({ -string => Other text, -charset => ISO-8859-1 });

# List available character set encodings
my @character_sets = utf8_supported_charset;

# Add a character set alias
utf8_charset_alias({ ms-japanese => sjis });

# Convert between two arbitrary (but largely compatible) charset encodings
# (SJIS to EUC-JP)
my $utf8_string = to_utf8({ -string =>$sjis_string, -charset => sjis});
my $euc_jp_string = from_utf8({ -string => $utf8_string, -charset => euc-jp })

# Verify that a specific character set is supported
if (utf8_supported_charset(ISO-8859-1) {
# Yes
}

Provides an adapter layer between core routines for converting to and from UTF8 and other encodings. In essence, a way to give multiple existing Unicode modules a single common interface so you dont have to know the underlaying implementations to do simple UTF8 to-from other character set encoding conversions. As such, it wraps the Unicode::String, Unicode::Map8, Unicode::Map and Jcode modules in a standardized and simple API.

This also provides general character set conversion operation based on UTF8 - it is possible to convert between any two compatible and supported character sets via a simple two step chaining of conversions.

As with most things Perlish - if you give it a few big chunks of text to chew on instead of lots of small ones it will handle many more characters per second.

By design, it can be easily extended to encompass any new charset encoding conversion modules that arrive on the scene.
This module is intended to provide good Unicode support to versions of Perl prior to 5.8. If you are using Perl 5.8.0 or later, you probably want to be using the Encode module instead. This module does work with Perl 5.8, but Encode is the preferred method in that environment.

Unicode::Map8 is a mapping table between 8-bit chars and Unicode.

SYNOPSIS

require Unicode::Map8;
my $no_map = Unicode::Map8->new("ISO646-NO") || die;
my $l1_map = Unicode::Map8->new("latin1") || die;

my $ustr = $no_map->to16("V}re norske tegn b|r {resn");
my $lstr = $l1_map->to8($ustr);
print $lstr;

print $no_map->tou("V}re norske tegn b|r {resn")->utf8

The Unicode::Map8 class implement efficient mapping tables between 8-bit character sets and 16 bit character sets like Unicode. The tables are efficient both in terms of space allocated and translation speed. The 16-bit strings is assumed to use network byte order.

The following methods are available:

$m = Unicode::Map8->new( [$charset] )

The object constructor creates new instances of the Unicode::Map8 class. I takes an optional argument that specify then name of a 8-bit character set to initialize mappings from. The argument can also be a the name of a mapping file. If the charset/file can not be located, then the constructor returns undef.

If you omit the argument, then an empty mapping table is constructed. You must then add mapping pairs to it using the addpair() method described below.

$m->addpair( $u8, $u16 );

Adds a new mapping pair to the mapping object. It takes two arguments. The first is the code value in the 8-bit character set and the second is the corresponding code value in the 16-bit character set. The same codes can be used multiple times (but using the same pair has no effect). The first definition for a code is the one that is used.

Consider the following example:

$m->addpair(0x20, 0x0020);
$m->addpair(0x20, 0x00A0);
$m->addpair(0xA0, 0x00A0);

It means that the character 0x20 and 0xA0 in the 8-bit charset maps to themselves in the 16-bit set, but in the 16-bit character set 0x0A0 maps to 0x20.

$m->default_to8( $u8 )

Set the code of the default character to use when mapping from 16-bit to 8-bit strings. If there is no mapping pair defined for a character then this default is substituted by to8() and recode8().

$m->default_to16( $u16 )

Set the code of the default character to use when mapping from 8-bit to 16-bit strings. If there is no mapping pair defined for a character then this default is used by to16(), tou() and recode8().

$m->nostrict;

All undefined mappings are replaced with the identity mapping. Undefined character are normally just removed (or replaced with the default if defined) when converting between character sets.

$m->to8( $ustr );

Converts a 16-bit character string to the corresponding string in the 8-bit character set.

$m->to16( $str );

Converts a 8-bit character string to the corresponding string in the 16-bit character set.

$m->tou( $str );

Same an to16() but return a Unicode::String object instead of a plain UCS2 string.

$m->recode8($m2, $str);

Map the string $str from one 8-bit character set ($m) to another one ($m2). Since we assume we know the mappings towards the common 16-bit encoding we can use this to convert between any of the 8-bit character sets.

$m->to_char16( $u8 )

Maps a single 8-bit character code to an 16-bit code. If the 8-bit character is unmapped then the constant NOCHAR is returned. The default is not used and the callback method is not invoked.

$m->to_char8( $u16 )

Maps a single 16-bit character code to an 8-bit code. If the 16-bit character is unmapped then the constant NOCHAR is returned. The default is not used and the callback method is not invoked.

The following callback methods are available. You can override these methods by creating a subclass of Unicode::Map8.

$m->unmapped_to8

When mapping to 8-bit character string and there is no mapping defined (and no default either), then this method is called as the last resort. It is called with a single integer argument which is the code of the unmapped 16-bit character. It is expected to return a string that will be incorporated in the 8-bit string. The default version of this method always returns an empty string.

Example:

package MyMapper;
@ISA=qw(Unicode::Map8);

sub unmapped_to8
{
my($self, $code) = @_;
require Unicode::CharName;
"";
}

$m->unmapped_to16

Likewise when mapping to 16-bit character string and no mapping is defined then this method is called. It should return a 16-bit string with the bytes in network byte order. The default version of this method always returns an empty string.

rxvt-unicode is a clone of the well known terminal emulator rxvt, modified to store text in Unicode (either UCS-2 or UCS-4) and to use locale-correct input and output. rxvt-unicode also supports mixing multiple fonts at the same time, including Xft fonts.
Main features:
- Stores text in Unicode (either UCS-2 or UCS-4).
- Uses locale-correct input, output and width: as long as your system supports the locale, rxvt-unicode will display correctly.
- Daemon mode: one daemon can open multiple windows on multiple displays, which improves memory usage and startup time considerably.
- Crash-free. At least I try, but rxvt-unicode certainly crashes much less often than rxvt and its many clones, and reproducible bugs get fixed immediately.
- Completely flicker-free.
- Full combining character support (unlike xterm :).
- Multiple fonts supported at the same time: No need to choose between nice japanese and ugly latin, or no japanese and nice latin characters.
- Supports Xft and core fonts in any combination.
- Can easily be embedded into other applications.
- All documentation accessible through manpages.
- Locale-independent XIM support.
- Many small improvements, such as improved and correct terminfo, improved secondary screen modes, italic and bold font support, tinting and shading.
Version restrictions:
- Complex script support, such as arabic or tibetian - more info is needed. (use mlterm)
- Left-To-Right rendering - more info is needed. (use mlterm)
- Tabs (although a supplied perl script implements a tabbed shell). (use mrxvt)
- IIIMF (Intranet/Internet Input Method Framework) support. (use scim)
Enhancements:
- This release optionally takes advantage of libafterimage for much improved image format and transparency support: transparency is now officially supported.
- Some new options are available: "skipScroll" hides fast scrolling text, "urgentOnBell" sets urgent hints to use an ASCII bell, and the "iso14755_52" resource controls the keycap insert mode.
- Portability has been enhanced, and many minor bugs have been fixed.

Unicode Error Detector is a product for Plone used to pinpoint errors in your application leading to UnicodeDecodeErrors.

Do not use this product unless you are actively debugging a Unicode Error. Never use this product in production sites.

UnicodeDecodeErrors typically occur when you try to add a Unicode string to a non-ascii string. This product patches StringIO used by page templates to check if the appended string is a Unicode string, and if it is, it replaces the string with an error marker.

As there is some overhead associated with inspecting the strings instead of just appending to the output, this product is meant for debugging purposes only.

Usage

Put the product in your Products directory and restart Zope. Load the template causing the UnicodeDecodeError, and this tool will indicate the location by printing THIS IS WHERE THE ERROR IS in the rendered template.

You can then inspect the template and/or code more closely to figure out where the decode error happens.