Free unicode software for linux

Unicode::Map8 is a mapping table between 8-bit chars and Unicode.

SYNOPSIS

require Unicode::Map8;
my $no_map = Unicode::Map8->new("ISO646-NO") || die;
my $l1_map = Unicode::Map8->new("latin1") || die;

my $ustr = $no_map->to16("V}re norske tegn b|r {resn");
my $lstr = $l1_map->to8($ustr);
print $lstr;

print $no_map->tou("V}re norske tegn b|r {resn")->utf8

The Unicode::Map8 class implement efficient mapping tables between 8-bit character sets and 16 bit character sets like Unicode. The tables are efficient both in terms of space allocated and translation speed. The 16-bit strings is assumed to use network byte order.

The following methods are available:

$m = Unicode::Map8->new( [$charset] )

The object constructor creates new instances of the Unicode::Map8 class. I takes an optional argument that specify then name of a 8-bit character set to initialize mappings from. The argument can also be a the name of a mapping file. If the charset/file can not be located, then the constructor returns undef.

If you omit the argument, then an empty mapping table is constructed. You must then add mapping pairs to it using the addpair() method described below.

$m->addpair( $u8, $u16 );

Adds a new mapping pair to the mapping object. It takes two arguments. The first is the code value in the 8-bit character set and the second is the corresponding code value in the 16-bit character set. The same codes can be used multiple times (but using the same pair has no effect). The first definition for a code is the one that is used.

Consider the following example:

$m->addpair(0x20, 0x0020);
$m->addpair(0x20, 0x00A0);
$m->addpair(0xA0, 0x00A0);

It means that the character 0x20 and 0xA0 in the 8-bit charset maps to themselves in the 16-bit set, but in the 16-bit character set 0x0A0 maps to 0x20.

$m->default_to8( $u8 )

Set the code of the default character to use when mapping from 16-bit to 8-bit strings. If there is no mapping pair defined for a character then this default is substituted by to8() and recode8().

$m->default_to16( $u16 )

Set the code of the default character to use when mapping from 8-bit to 16-bit strings. If there is no mapping pair defined for a character then this default is used by to16(), tou() and recode8().

$m->nostrict;

All undefined mappings are replaced with the identity mapping. Undefined character are normally just removed (or replaced with the default if defined) when converting between character sets.

$m->to8( $ustr );

Converts a 16-bit character string to the corresponding string in the 8-bit character set.

$m->to16( $str );

Converts a 8-bit character string to the corresponding string in the 16-bit character set.

$m->tou( $str );

Same an to16() but return a Unicode::String object instead of a plain UCS2 string.

$m->recode8($m2, $str);

Map the string $str from one 8-bit character set ($m) to another one ($m2). Since we assume we know the mappings towards the common 16-bit encoding we can use this to convert between any of the 8-bit character sets.

$m->to_char16( $u8 )

Maps a single 8-bit character code to an 16-bit code. If the 8-bit character is unmapped then the constant NOCHAR is returned. The default is not used and the callback method is not invoked.

$m->to_char8( $u16 )

Maps a single 16-bit character code to an 8-bit code. If the 16-bit character is unmapped then the constant NOCHAR is returned. The default is not used and the callback method is not invoked.

The following callback methods are available. You can override these methods by creating a subclass of Unicode::Map8.

$m->unmapped_to8

When mapping to 8-bit character string and there is no mapping defined (and no default either), then this method is called as the last resort. It is called with a single integer argument which is the code of the unmapped 16-bit character. It is expected to return a string that will be incorporated in the 8-bit string. The default version of this method always returns an empty string.

Example:

package MyMapper;
@ISA=qw(Unicode::Map8);

sub unmapped_to8
{
my($self, $code) = @_;
require Unicode::CharName;
"";
}

$m->unmapped_to16

Likewise when mapping to 16-bit character string and no mapping is defined then this method is called. It should return a 16-bit string with the bytes in network byte order. The default version of this method always returns an empty string.

Bundle::Unicode is a Perl bundle to install Unicode modules and their dependencies.

SYNOPSIS

perl -MCPAN -e install Bundle::Unicode

CONTENTS

Unicode::Lite
Unicode::String
Unicode::Map
enum
Unicode::EastAsianWidth
ExtUtils::MakeMaker
Module::Build
Unicode::Collate
Unicode::Collate::Standard
File::Spec
Cwd
Exporter
Test
Test::More
Inline
XSLoader
Unicode::CheckUTF8
Unicode::Char
Unicode::Unihan
Fcntl
File::Path
Lingua::Han::Utils
Unicode::IMAPUtf7
Unicode::Map8
Unicode::Map
Unicode::MapUTF8
App::Info::Lib::Iconv
Encode
Module::Install
FindBin
Jcode
MIME::Base64
Unicode::UTF8simple
Unicode::Japanese
ExtUtils::Manifest
ExtUtils::Embed
Unicode::Escape
Filter::Simple
Unicode::Transliterate
AutoLoader
Clone
Unicode::Wrap
File::Copy
Unicode::Normalize
Unicode::Regex::Set
Unicode::Transform
Unicode::Decompose
Unicode::RecursiveDowngrade
String::Multibyte
String::Multibyte::Unicode
String::Multibyte::Grapheme
Acme::MetaSyntactic
Scalar::List::Util
IO::Compress::Base
Text::Unidecode
ShiftJIS::X0213::MapUTF
ShiftJIS::CP932::MapUTF
ShiftJIS::CP932::Correct
Convert::Base32
Convert::RACE
HTML::Fraction
String::Fraction
XML::Simple
YAML
Convert::CharMap
Config
File::Spec::Functions
File::Basename
Scalar::Util
AppConfig
Template
Template::Config
version
Template::Provider::Unicode::Japanese
Bundle::Encode
ShiftJIS::Collate
ShiftJIS::Regexp
encoding::warnings
Locale::Recode
TeX::Encode
Pod::LaTeX
Pod::Find
Pod::ParseUtils
Pod::Select
Locale::Maketext
Locale::Maketext::Lexicon
Locale::Maketext::Simple
i18n
Convert::ASCIInames
Apache::GuessCharset
HTML::Entities
HTML::Parser

Unicode::Collate is a Unicode Collation Algorithm.

SYNOPSIS

use Unicode::Collate;

#construct
$Collator = Unicode::Collate->new(%tailoring);

#sort
@sorted = $Collator->sort(@not_sorted);

#compare
$result = $Collator->cmp($a, $b); # returns 1, 0, or -1.

# If %tailoring is false (i.e. empty),
# $Collator should do the default collation.

This module is an implementation of Unicode Technical Standard #10 (a.k.a. UTS #10) - Unicode Collation Algorithm (a.k.a. UCA).

Constructor and Tailoring

The new method returns a collator object.

$Collator = Unicode::Collate->new(
UCA_Version => $UCA_Version,
alternate => $alternate, # deprecated: use of variable is recommended.
backwards => $levelNumber, # or @levelNumbers
entry => $element,
hangul_terminator => $term_primary_weight,
ignoreName => qr/$ignoreName/,
ignoreChar => qr/$ignoreChar/,
katakana_before_hiragana => $bool,
level => $collationLevel,
normalization => $normalization_form,
overrideCJK => &overrideCJK,
overrideHangul => &overrideHangul,
preprocess => &preprocess,
rearrange => @charList,
table => $filename,
undefName => qr/$undefName/,
undefChar => qr/$undefChar/,
upper_before_lower => $bool,
variable => $variable,
);

Unicode Utilities project are a set of programs for manipulating and analyzing Unicode text. uniname defaults to printing the character offset of each character, its byte offset, its hex code value, its encoding, the glyph itself, and its name. Command line options allow undesired information to be suppressed and the Unicode range to be added.
unidesc reports the character ranges to which different portions of the text belong. unihist generates a histogram of the characters in its input. ExplicateUTF8 is intended for debugging or for learning about Unicode. It determines and explains the validity of a sequence of bytes as a UTF-8 encoding. unirev reverses UTF-8 strings.
Enhancements:
- Adds to unidesc the option -r which causes it to list the ranges detected after reading all input rather than listing them as they are encountered, and adds to uniname the option -B which causes it to ignore characters within the Basic Multilingual Plane.

Unicode::Normalize Perl module contains Unicode Normalization Forms.

SYNOPSIS

(1) using function names exported by default:
use Unicode::Normalize;

$NFD_string = NFD($string); # Normalization Form D
$NFC_string = NFC($string); # Normalization Form C
$NFKD_string = NFKD($string); # Normalization Form KD
$NFKC_string = NFKC($string); # Normalization Form KC

(2) using function names exported on request:

use Unicode::Normalize normalize;

$NFD_string = normalize(D, $string); # Normalization Form D
$NFC_string = normalize(C, $string); # Normalization Form C
$NFKD_string = normalize(KD, $string); # Normalization Form KD
$NFKC_string = normalize(KC, $string); # Normalization Form KC

Parameters:

$string is used as a string under character semantics (see perlunicode).

$code_point should be an unsigned integer representing a Unicode code point.

Note: Between XSUB and pure Perl, there is an incompatibility about the interpretation of $code_point as a decimal number. XSUB converts $code_point to an unsigned integer, but pure Perl does not. Do not use a floating point nor a negative sign in $code_point.

Normalization Forms

$NFD_string = NFD($string)

It returns the Normalization Form D (formed by canonical decomposition).

$NFC_string = NFC($string)

It returns the Normalization Form C (formed by canonical decomposition followed by canonical composition).

$NFKD_string = NFKD($string)

It returns the Normalization Form KD (formed by compatibility decomposition).

$NFKC_string = NFKC($string)

It returns the Normalization Form KC (formed by compatibility decomposition followed by canonical composition).

$FCD_string = FCD($string)

If the given string is in FCD ("Fast C or D" form; cf. UTN #5), it returns the string without modification; otherwise it returns an FCD string.

Note: FCD is not always unique, then plural forms may be equivalent each other. FCD() will return one of these equivalent forms.

$FCC_string = FCC($string)

It returns the FCC form ("Fast C Contiguous"; cf. UTN #5).
Note: FCC is unique, as well as four normalization forms (NF*).

$normalized_string = normalize($form_name, $string)

It returns the normalization form of $form_name.

As $form_name, one of the following names must be given.

C or NFC for Normalization Form C (UAX #15)
D or NFD for Normalization Form D (UAX #15)
KC or NFKC for Normalization Form KC (UAX #15)
KD or NFKD for Normalization Form KD (UAX #15)

FCD for "Fast C or D" Form (UTN #5)
FCC for "Fast C Contiguous" (UTN #5)

Unicode::Regex::Set is a subtraction and intersection of Character Sets in Unicode Regular Expressions.

SYNOPSIS

use Unicode::Regex::Set qw(parse);

$regex = parse([p{Latin} & p{L&} - A-Z]);

Perl 5.8.0 misses subtraction and intersection of characters, which is described in Unicode Regular Expressions (UTS #18). This module provides a mimic syntax of character classes including subtraction and intersection, taking advantage of look-ahead assertions.

The syntax provided by this module is considerably incompatible with the standard Perls regex syntax.

Any whitespace character (that matches /s/) is allowed between any tokens. Square brackets ([ and ]) are used for grouping. A literal whitespace and square brackets must be backslashed (escaped with a backslash, ). You cannot put literal ] at the start of a group.

A POSIX-style character class like [:alpha:] is allowed since its [ is not a literal.
SEPARATORS (& for intersection, | for union, and - for subtraction) should be enclosed with one or more whitespaces. E.g. [A&Z] is a list of A, &, Z. [A-Z] is a character range from A to Z. [A-Z - Z] is a set by removal of [Z] from [A-Z].
Union operator | may be omitted. E.g. [A-Z | a-z] is equivalent to [A-Z a-z], and also to [A-Za-z].

Intersection operator & has high precedence, so [p{A} p{B} & p{C} p{D}] is equivalent to [p{A} | [p{B} & p{C}] | p{D}].

Subtraction operator - has low precedence, so [p{A} p{B} - p{C} p{D}] is equivalent to [[p{A} | p{B}] - [p{C} | p{D}] ].

[p{A} - p{B} - p{C}] is a set by removal of p{B} and p{C} from p{A}. It is equivalent to [p{A} - [p{B} p{C}]] and [p{A} - p{B} p{C}].

Negation. when ^ just after a group-opening [, i.e. when they are combined as [^, all the tokens following are negated. E.g. [^A-Z a-z] matches anything but neither [A-Z] nor [a-z]. More clearly you can say this with grouping as [^ [A-Z a-z]].

If ^ that is not next to [ is prefixed to a sequence of literal characters, character ranges, and/or metacharacters, such a ^ only negates that sequence; e.g. [A-Z ^p{Latin}] matches A-Z or a non-Latin character. But [A-Z [^p{Latin}]] (or [A-Z P{Latin}], for this is a simple case) is recommended for clarity.

If you want to remove anything other than PERL from [A-Z], use [A-Z & PERL] as well as [A-Z - [^PERL]]. Similarly, if you want to intersect [A-Z] and a thing not JUNK, use [A-Z - JUNK] as well as [A-Z & [^JUNK]].